MVP 8400 Locking up

AlexArtist

IS THERE CODE THAT CAN LOCK UP MVPs THAT WON'T NECESSARILY LOCK UP WIRED PANELS?

I have 3 different MVP-8400's that are locking up at various intervals. This is all at one job, there's 9 wired panels that are all running off the same master and running the same graphics as one of the MVPs; but this specific MVP occasionally locks up. Two of the other MVPs running slightly different programming and graphics also lock up. The only strange thing (on this job compared with other jobs with NON-locking up MVPs) is that all the panels locking up are in wall docking stations that are in Wall Dock Conduit Boxes that are in Concrete walls. Possibly they're over heating? i don't know. The wired panels are fine though and are inside conduit box and concrete walls aswell. One of the locking up MVPs used to take days to lock up. But seems to lock up within hours now. Another MVP always takes days and the Last MVP takes weeks possibly months.

IS THERE CODE THAT CAN LOCK UP MVPs THAT WON'T NECESSARILY LOCK UP WIRED PANELS?
Is it possible power fluctuations can affect Docked MVPs more than In-Wall panels? I've noticed non AMX devices seeming to lock up aswell. So possibly it is something with power, Hmmm...

I'm near wits end on this.

Auser

I strongly suspect that there's a memory or similar resource leak in Modero touch panels as I wasted a lot of time chasing a similar problem. We managed to ascertain that when an MVP-8400 disconnects and reconnects to a wireless network numerous times (on the order of 100 or more times), it will eventually become unresponsive and not connect to the wireless network until rebooted.

In our case the client would store the panel in a secure room on the fringe of wireless range, and the panel would constantly connect, disconnect and reconnect overnight. In the morning it wouldn't reconnect to the network, even when in close proximity to the access point, until rebooted.

Not sure how much wireless traffic there is floating around at your install, is it possible that it's jumping between access points repeatedly? The logs on the panel should give you a good idea if you can still access the setup pages or telnet interface when it locks up.

DHawthorne

I have a panel that cannot re-connect when waking from a sleep state. It comes right back when you reboot it from the setup page. I replaced the panel and still have the problem on the job, but the old one is fine on my desk. I've come to the conclusion that local wireless interference is interfering with the handshake, and it is somehow more sensitive to it on a cold boot than a warm. I am moving the access point away from what I suspect to be the primary cause of interference (a bunch of Sonos units, which all have built-in WiFi). If that doesn't do it, I'm going to shut down the encryption next.

As a general statement, they are sensitive to interference and work best in a pristine wireless environment. Of course, that is sometimes impossible to provide, so there is a bit of a learning curve in developing tricks to deal with it. Most of them involve setting up the access points so you have a stronger signal.

ericmedley

DHawthorne wrote:

I have a panel that cannot re-connect when waking from a sleep state. It comes right back when you reboot it from the setup page. I replaced the panel and still have the problem on the job, but the old one is fine on my desk. I've come to the conclusion that local wireless interference is interfering with the handshake, and it is somehow more sensitive to it on a cold boot than a warm. I am moving the access point away from what I suspect to be the primary cause of interference (a bunch of Sonos units, which all have built-in WiFi). If that doesn't do it, I'm going to shut down the encryption next.

As a general statement, they are sensitive to interference and work best in a pristine wireless environment. Of course, that is sometimes impossible to provide, so there is a bit of a learning curve in developing tricks to deal with it. Most of them involve setting up the access points so you have a stronger signal.

There has been much discussion on this matter. We deal primarily with 8400s in our systems. As you mention, there are a lot of tricks and plain-ole smoke and mirrors to getting them to work sometimes. We've actually toyed with getting an RF Spec analyzer to look at what's going on in some of our installs. They're not cheap!

In homes that have any kind of 2.4Ghz phones you can expect all kinds of issues as they tend to end up near panels. It makes sense when you consider that clients will commonly locate this kind of stuff on their night stands or on a kitchen counter or a side table in a TV room.

As it stands, the frequency band is way too crowded to expect any kind of reliability. And, the functionality of the WiFi equipment is not really built to handle interference and deal with ensuring the connection stays alive.

I would argue that the WiFi band needs to be somewhere on its own and not on the public bands. Of course, this will make the $49.95 WiFi Routers triple in price, but something has to give or the log jam will continue. (notice how I deftly mix my metaphors..)

AlexArtist

I checked the Telnet log on the most problematic MVP and its memory does seem to drop by 10K or more every 5 or 10 minutes. The signal seems fine, the WAP 250 is about 30 feet from the panel and has clear Line of Sight. There are a few strange errors though like:

23: 02-06-2008 WED 13:23:13 Info setup
Master CONNECTED 192.168.1.11

24: 02-06-2008 WED 13:23:13 WARN unknown
CMessagePipe::(writer=ICSPMessageManager): Queue overflowed, dumping messages!

25: 02-06-2008 WED 13:23:12 ERROR icsptcp
ICSPTCP::ProcessICSPPacketRX Authentication Result=0003

26: 02-06-2008 WED 13:23:12 ERROR icsptcp
ICSPTCP::ProcessICSPPacketRX Authentication Challenge:Challenge=62BA05CC
...


39: 02-06-2008 WED 13:23:10 WARN network
Error retrieving MAC information for wlan0: No such device

40: 02-06-2008 WED 13:23:09 WARN network
Error retrieving MAC information for : No such device

41: 02-06-2008 WED 13:23:09 WARN setup
Could not read touch driver setting from NvRAM: No child processes


I'll send the info to AMX and see what they say. (Although they seem a bit hush hush about revealing discovered issues sometimes, until they've got the fix.)

DHawthorne

AlexArtist wrote:

I'll send the info to AMX and see what they say. (Although they seem a bit hush hush about revealing discovered issues sometimes, until they've got the fix.)

You can say that again; it's probably policy. I have more than once reported an issue, got "I have nothing on that problem here," then found it was corrected in a subsequent firmware release. The guys and gals manning the phones sometimes may simply not know what the firmware teams are working on, but it still reflects a policy of "don't tell them anything until we know for sure." There are pluses and minuses for that kind of policy, and I can't say I disagree with it, though it does get frustrating sometimes. Heck, I can understand them not saying anything to "Joe Blow Automations Systems," but surely they could tell me .

a_riot42

Are you suer eyou aren't sending data to the panels? You can lock up a panel if you are sending strings to it non-stop. Perhaps you have checked this already though. Is the panel going offline when it locks up?
Paul

AlexArtist

a_riot42 wrote:

Are you suer eyou aren't sending data to the panels? You can lock up a panel if you are sending strings to it non-stop. Perhaps you have checked this already though. Is the panel going offline when it locks up?
Paul

Of course i'm sending DATA to the panels. But not continual strings. The main data the panel gets is lots of level data. About 60 different levels of Lighting and Audio. But most the time the Level data isn't CHANGING. Occasional when a preset is selected or when the panel turns on, about 60 zones of levels will get sent to the panel. The Wired panels receiving the same data and have never locked up yet.

I'm not sure how AMX communicates to panels, but 60 units of 2Byte data could easily fit within one IP packet to the panel.

ericmedley

AlexArtist wrote:

The Wired panels receiving the same data and have never locked up yet.

I'm not sure how AMX communicates to panels, but 60 units of 2Byte data could easily fit within one IP packet to the panel.

Wired and wireless are completely different animals. I too, could be accused of trying to send to much comm to my touch panels. I've been questioned on this forum for such practices myself. However, in my experience, if there's a hiccup on the wireless network, the panel just seems to skip over the commands it didn't quite get or overran. It usually catches up pretty quickly.

Typically, if you've tied a level in a program to a level on a touch panel, the level doesn't get re-sent if the level has not changed. However, if you have some kind of loop or timeline going that continually sends 'level 1=50%, level 2=34%, etc...' type commands out, you might be asking for trouble.

A lot depends upon how rapidly you send out the commands. For example, I've found that a good practice for me is to limit the commands I send to wireless touch panels to 2 in one cycle.

So, for example...

if(TP_Flag)
{
send_command TP_1, 'some command #1'
send_command TP_1, 'some command #2'
TP_Flag=0
}

on wireless touch panels, if I put three or more, some of those commands will get ignored.

if(TP_Flag)
{
send_command TP_1, 'some command #1'
send_command TP_1, 'some command #2'
send_command TP_1, 'some command #3' // this one might get done
send_command TP_1, 'some command #4' // this one probably won't at all.
TP_Flag=0
}

A simple way to make this work is to put a wait in. That's so old-skool but it illustrates one work around.

if(TP_Flag)
{
send_command TP_1, 'some command #1'
send_command TP_1, 'some command #2'
wait 1
  {
  send_command TP_1, 'some command #3' 
  send_command TP_1, 'some command #4' 
  }
TP_Flag=0
}

If you're donig a whole slew of commands in a FOR loop, I promise you it'll fail to get them all done consistently.

I've never had any of this lock up a panel, however, which was the original discussion.

a_riot42

AlexArtist wrote:

I'm not sure how AMX communicates to panels, but 60 units of 2Byte data could easily fit within one IP packet to the panel.

I'm not sure either but you could watch with a packet sniffer what traffic those 60 units creates. I have a feeling it would be considerably more than 1 packet. Do they lock up if you send 10 levels instead of 60? You don't include your code but you may need to do some buffering of send_levels.

23: 02-06-2008 WED 13:23:13 Info setup
Master CONNECTED 192.168.1.11

24: 02-06-2008 WED 13:23:13 WARN unknown
CMessagePipe:writer=ICSPMessageManager): Queue overflowed, dumping messages!

This would seem to indicate a queue overflow and a subsequent dump of messages. Perhaps too much data or the wireless card is toast so nothing is being sent out of the queue so it dumps it.

25: 02-06-2008 WED 13:23:12 ERROR icsptcp
ICSPTCP::ProcessICSPPacketRX Authentication Result=0003

26: 02-06-2008 WED 13:23:12 ERROR icsptcp
ICSPTCP::ProcessICSPPacketRX Authentication Challenge:Challenge=62BA05CC
...


39: 02-06-2008 WED 13:23:10 WARN network
Error retrieving MAC information for wlan0: No such device

40: 02-06-2008 WED 13:23:09 WARN network
Error retrieving MAC information for : No such device

Can't seem to find the wireless interface?

41: 02-06-2008 WED 13:23:09 WARN setup
Could not read touch driver setting from NvRAM: No child processes

Not sure what this refers to. Can't read something from non-volatile random access memory. Sounds like maybe a corrupt flash drive?
Rather bizarre messages. I haven't seen any of these before.
Paul

DHawthorne

Overloaded message queues tend to lock up the master, not the panel. I too have not seen these kind of error messages, and strongly suspect something was corrupted. Have you tried reloading the firmware?