The problem with the NI-3100 on yesterday's service call are as follows.
TX lights would flash as the program would send commands.
RX light would not flash in response making it seem all devices were dead.
RX lights would flash for unsolicitated data being sent from devices.
Only 1 device sent unsolicitated messages so it appeared everything else was dead and the 1 device that sent unsolicited responses did so, so frequently that it appeared the port was working when the program did TX to it. At first I thought that port was working but later realized I was being duped.
Connected Hyper-Termial to a NI serial port and sent strings via control device to this port to display in HT but the while TX light flashed HT didn't RX anything.
The unsolicited data sent from the 1 device that did so was received and processed normally. With HT still connected to the NI's serial port I could send strings to the master, the RX light would flash and I assume the RX data would be processed just as the data on the port with unsolicited feedback did. I didn't verify in debug though.
So something it apparently is making the strings being sent out the serial ports disappear but still flashing the TX led. The ports will RX fine but since the devices never recieve commands they don't talk so unless they send unsolicited feedback you won't see flashing RX leds.
I'm not sure if this is exactly the same thing that everyone else is seeing but I'm pretty sure this is what I'm seeing.
Of course this was on a job that I've never been to before running programs I've never seen before and I went there thinking just the music server was down based on a phone call from the client. I did have the full client file from the original programmer/installer though.
I did update the firmware to the most recent and reloaded the dev firmware and uploaded the program again but to no avail. I didn't use the Hot Fix firmware since it's supposedly for systems brought down by to much chatter, they'll go back to normal after a reboot for an unknown amount of time. Maybe they'll never fail again but they also may fail that day or the next week. That's another issue.
This is the cap crap?
That is exactly what I have experienced when the capacitor goes bad in the master. Once the cap goes bad the power supply for the 232 ports can no longer produce the voltage required to send any commands to connected devices, but the LEDs are not powered from that power rail and will continue to function normally.
I have seen this issue on quite a few master included NI-3000, NI-4000, and NI-3100 (with the old bubble style LEDs).
That is exactly what I have experienced when the capacitor goes bad in the master. Once the cap goes bad the power supply for the 232 ports can no longer produce the voltage required to send any commands to connected devices, but the LEDs are not powered from that power rail and will continue to function normally.
I have seen this issue on quite a few master included NI-3000, NI-4000, and NI-3100 (with the old bubble style LEDs).
Yeah, I shipped it back and they replaced C206. This was an NI-3100 w/ the bubble type LEDs. I wonder what the failure rate of these guys are? 100%, 50%? Are all the units with this hardware package going to fail at some point or just a random unfortunate few?
We've been putting these things all over Houston since 2003 and have had just one fail, so, I think the overall failure rate is pretty low. The one that failed was installed in 2005 and it failed in late 2010.
We've been putting these things all over Houston since 2003 and have had just one fail, so, I think the overall failure rate is pretty low. The one that failed was installed in 2005 and it failed in late 2010.
It's an aging thing. There may be some out in the wild that will still fail in the future. That said, I have only seen it happen 2-3 times, and at least one of those is iffy. That's out of dozens. So I think it's in the single-digit percentiles.
It's an aging thing. There may be some out in the wild that will still fail in the future. That said, I have only seen it happen 2-3 times, and at least one of those is iffy. That's out of dozens. So I think it's in the single-digit percentiles.
Unfortunately I am on the opposite end of the spectrum. I think I have had to send in around 7 or 8 masters in the last 18 months.
We have two spare 3100's for loaners that have pretty much been steadily rolling from house to house while the repairs are being made. AMX is nice enough to not charge us for the repair (even on out of warranty masters), but we still have to pay for the labor and shipping the unit to AMX so I can't say that I am happy about the ordeal.
With about a 250 unit visibility/exposure, we have seen this potentially only once. It was before this was clearly understood and was thought to be a lightning strike, which it could have been in fact. We've suspected it in several since, but at least one was the system-side lockup now addressed by new firmware, and the others turned out to be wiring issues (despite extreme denial by techs). So not everybody is getting hardware failures. But >YET< may eventually be the operative word.... but then, it always is - eventually is a long long time.
"Eventually" has caught up with us here. In the last 6 months, we've seen at least 5 2000/3000 units lose SERIAL control (including my own home system), and 2 3100's have the same happen to them. So this is probably going to continue to be a trend everywhere.
I had another 3100 cap fail Dec 23. I tried to fix it with a spare 4000 running duet but it didn't have enough memory to run the program. So christmas eve day I went back and tried another spare 4000 and had the same result. I didn't have a spare x100 but i did pull my personal 3100 and fortunately i took it with me so when the 4000 was a no joy i put in my 3100 and the client was back in business.
Unfortunately now i had no system and subsequently no music for Christmas eve and day so i took a ride to radio shack and bought all their caps that fit the spec. Not a very easy thing to solder in since the pads are very small and my eyesight is getting worse ever year but i did it. It worked and i had Christmas carrols.
I waisted alot of time trying to get the 4000's to run the program when 1/2 an hour with a cap and soldering hour did the trick. For some reason i never even thought about that option until i got home and wanted music.
FYI, if you load old pre-duet firmware (still available on AMX), you'll have room for your program in the 4000. Presuming the job didn't require duet...
FYI, if you load old pre-duet firmware (still available on AMX), you'll have room for your program in the 4000. Presuming the job didn't require duet...
I thought of that but there wasn't any dev firmware online that goes along with it. I also had the latest duet firmware loaded which has a catch 22. It you load the latest dev firmware the dev doesn't show up until you load the latest master firmware. If you then try to load any earlier master firmware you can't change the dev firmware cuz it doesn't show up after that and if I try to upload an earlier dev firmware first it doesn't let me. I fought the process for hours trying to get anything to work and even getting the latest and greatest to take with the dev was a chore.
These 4000s only allow setting duet memory between 3 and 8 and if i set it to anything other that 3 the program would load but not completely boot up. With 3 it would boot up but not really run.
Wow, I see both the pre-duet master and device firmwares have been removed from the online archive. This is recent, I think I last downloaded the old x000 non-duet only a month ago. Well, many of us still have the old firmware if you want to try again... just ask for it.
Added thought: Are you sure the Device side can't be seen by the old master, it may have an unexpected device number up in the 32000's. I've seen that happen, though it was lost, just hiding.
Wow, I see both the pre-duet master and device firmwares have been removed from the online archive. This is recent, I think I last downloaded the old x000 non-duet only a month ago. Well, many of us still have the old firmware if you want to try again... just ask for it.
They have or had the current x000 non duet master in with the current firmwares but no dev could be found there or in the archive.
Thanks for the offer and I will post if I ever need it but I'm not sure if I'll be able downgrade from the current 1.30.8 on the x000s. I'm pretty sure it wouldn't even let me go back to 1.20.7.
FYI, if you load old pre-duet firmware (still available on AMX), you'll have room for your program in the 4000. Presuming the job didn't require duet...
Just a word of warning, from recent experience.
When upgrading from older firmware (pre build 430) there is an issue that can cause failure during the upgrade process which bricks the master. I can be fixed but only by AMX (they have to do an onboard firmware upgrade).
The problem is caused when a chatty RS232 device is connected during the firmware upgrade.
The issue can be avoided if you physically disconnect ALL RS232 devices before the upgrade.
I personally have only seen the issue when I loaded the device firmware before the master firmware.
"Eventually" has caught up with us here. In the last 6 months, we've seen at least 5 2000/3000 units lose SERIAL control (including my own home system), and 2 3100's have the same happen to them. So this is probably going to continue to be a trend everywhere.
The very first NI-3100 and NI-2100 devices were released before the problem was resolved.
So you shouldn't see too many of them.
Installation was done 4 months with no issues. Last two weeks NI2100 processor froze twice. Jandy is connected to RS232 port number two. I was at the job site when processor was lockup up. TX and RX of Jandy RS232 LEDs were solid on. The Status led was blinking fine, I did no have my laptop with me to ftp. I just did a reset and problem went away. It happened again today. Will the hot fix solve this issue.
The x100 version is nearly too tiny to try to fix without really good surface mount soldering tools.
AMX charges $150 to fix these, I think the relative value of a x100 processor is worth that to be sure it survives. The x000 series uses larger and more easily repaired parts, and is lower value-lower risk if you fail... so worth doing yourself. IMHOO.
If you are seeing activity on both the TX and RX lights, it doesn't seem like a Capacitor issue. I believe AMX redesigned the part that was failing to make it more reliable and I have not had any issues on processors in the last few years. The fact that rebooting it allows things to communicate again tells me that it is probably not the capacitor issue as well.
Now, you mentioning that you have a pool controller on the processor tells me that most likely there is a problem with the pool controller connection that is causing extreme amounts of line noise/garbage that is filling up the incoming buffer and appearing to lockup the processor. I would start by checking all of the wiring connections in the line for the pool controller connection. Make sure nothing has corroded from exposure to moisture. I have seen this numerous times and it fits your symptoms. Also, do you have the Jandy adapter near your AMX processor or is it near the pool equipment with a longer RS232 connection to the processor? Lastly, are you plugging the Jandy adapter directly into the processor or using a 3 conductor only cable?
The early x100's share the same board design with the x000, but there are few of them. If it looks at all the same, you have the early one, and they are far more likely to have the failure than the later design. And you are able to fix it more easily.
If you are seeing activity on both the TX and RX lights, it doesn't seem like a Capacitor issue. I believe AMX redesigned the part that was failing to make it more reliable and I have not had any issues on processors in the last few years. The fact that rebooting it allows things to communicate again tells me that it is probably not the capacitor issue as well.
While the rest of your info is good, the above isn't quite what we've seen and heard. The capacitor failure drops only the carrier voltage on the RS232, the data is actually fine, and what results is similar to what RS422/485 expects, and the lights still work. And it works fine without the voltage if your device out there doesn't rely on it. And being a capacitor failing, it can respond differently across time and boots. My own 3000 for a while would work fine for a day after a rest and reboot. Then fail. After a short time, it always failed. My lights always looked fine. It was the capacitors.
Later production x100's have a different design, harder to fix, far less likely to fail, but it still can happen. The symptoms are the same. We have 2 of these waiting now, failed in use exactly as the others, with serial gradually becoming unreliable, some devices affected before others, and results varying upon reboots and cold reboots.
I believe my experience has been that the TX will flash but no traffic is sent out the serial port and that means the RX light won't flash from reponses to those commands since they were never sent. If the device sends updates on it own then you will see the RX led flash and if it's a chatty device you'll think the RX is responding to the TX like normal but you're really just getting updates and not acknowledgements to the commands sent.
I beleive the OP said the TX/RX lights were solid which to me means something is really chatty or the code may occasionally get into a loop where it's sending out strings constantly.
I beleive the OP said the TX/RX lights were solid which to me means something is really chatty or the code may occasionally get into a loop where it's sending out strings constantly.
I would agree, the cap issue has not been reported to lock the lights up.
As for replies (yellow lights) to the defective sends (red), I'd agree for devices where it has completely failed. In the half-working state, the device may be responding with a "what???". You should be able to examine the replies and see if they were understanding what was sent.
I was at the job site when processor was lockup up. TX and RX of Jandy RS232 LEDs were solid on.
AMX identified this (very sporadic) behaviour as a firmware bug a while ago and resolved it with a firmware release. At the time the firmware release was stated to prevent devices flooding the network with Bonjour packets causing the zeroconfig related firmware portions in the master to go off into the weeds though I never could figure out whether this was the same issue or an additional one. To the user, the master appeared to be running fine, but the on board control ports had locked up.
Updating the master and device firmware on your controller and turning off zeroconfig via the console should prevent it recurring.
Thank you all for the feedback.
Auser could you elaborate on "turning off zeroconfig via the console".
Jeff AquaLinkRS is near AMX and AquaLink RS came with RS232 adapter/cable which we used to make the connection to the processor. I have to check if only 3 conductors are used, we just used what it came with.
I don't think my issue is related to capacitor, it might be related to CPU usage and chatty AquaLinkRS device.
Today I was at the client's house and everything was running fine. She rebooted the processor once 4 days ago,
One this project I have one NI2100 and one NI3101.
NI2100
RS232 #1 AutoPath Video Matrix
RS232 #2 Jandy AquaLinkRS
RS232 #3 DSC
IR .....
IP Control: Lutron RadioRA2 2 systems with 150 dimmers
IP Control: Zigbee gateway with 4 R4 remotes
2 Termostates
NI2100 average cpu usage is 70% to 80%.
NI3101 average cpu usage with one iPort enabled is 10% to 15%.
AquaLink RS is polling every 3 sec(TX lights up every 3sec). DSC TX lights up every 5 sec to 10 sec.
I'm guessing my issue is distribution of processing power between two processor and AquaLinkRS chatty device. Can my code have some unknown loop which will take all processing power. Any suggestions????????
As reminder I put these compiler warnings in the beginning of the main code of every system. Otherwise I will forget and even with it I'll often ignore it.
#WARN 'TELNET MASTER & set udp bc rate, 0'
#WARN 'TELNET MASTER & set zeroconf, disable'
#WARN 'TELNET MASTER & set duet memory, 18m'
The duet memory depends on the situation but 18m is a good number since I don't use duet modules in the first place.
Technote on zeroconfig- http://www.amx.com/techsupport/techNote.asp?id=970
Description: While Zeroconf is a useful tool during system setup, it may be disabled afterwards to eliminate unnecessary traffic.
Zeroconf may be disabled on NetLinx masters by telneting into a master and issuing "zeroconf disable" command. This concept applies to non-master AMX devices such as the ZigBee gateway and others also.
This pertains mostly to large networks with many BONJOUR (Apple compliant) devices, including most AMX IP devices like masters.
Comments
That is exactly what I have experienced when the capacitor goes bad in the master. Once the cap goes bad the power supply for the 232 ports can no longer produce the voltage required to send any commands to connected devices, but the LEDs are not powered from that power rail and will continue to function normally.
I have seen this issue on quite a few master included NI-3000, NI-4000, and NI-3100 (with the old bubble style LEDs).
It's an aging thing. There may be some out in the wild that will still fail in the future. That said, I have only seen it happen 2-3 times, and at least one of those is iffy. That's out of dozens. So I think it's in the single-digit percentiles.
Unfortunately I am on the opposite end of the spectrum. I think I have had to send in around 7 or 8 masters in the last 18 months.
We have two spare 3100's for loaners that have pretty much been steadily rolling from house to house while the repairs are being made. AMX is nice enough to not charge us for the repair (even on out of warranty masters), but we still have to pay for the labor and shipping the unit to AMX so I can't say that I am happy about the ordeal.
Unfortunately now i had no system and subsequently no music for Christmas eve and day so i took a ride to radio shack and bought all their caps that fit the spec. Not a very easy thing to solder in since the pads are very small and my eyesight is getting worse ever year but i did it. It worked and i had Christmas carrols.
I waisted alot of time trying to get the 4000's to run the program when 1/2 an hour with a cap and soldering hour did the trick. For some reason i never even thought about that option until i got home and wanted music.
These 4000s only allow setting duet memory between 3 and 8 and if i set it to anything other that 3 the program would load but not completely boot up. With 3 it would boot up but not really run.
Added thought: Are you sure the Device side can't be seen by the old master, it may have an unexpected device number up in the 32000's. I've seen that happen, though it was lost, just hiding.
Thanks for the offer and I will post if I ever need it but I'm not sure if I'll be able downgrade from the current 1.30.8 on the x000s. I'm pretty sure it wouldn't even let me go back to 1.20.7.
Just a word of warning, from recent experience.
When upgrading from older firmware (pre build 430) there is an issue that can cause failure during the upgrade process which bricks the master. I can be fixed but only by AMX (they have to do an onboard firmware upgrade).
The problem is caused when a chatty RS232 device is connected during the firmware upgrade.
The issue can be avoided if you physically disconnect ALL RS232 devices before the upgrade.
I personally have only seen the issue when I loaded the device firmware before the master firmware.
The very first NI-3100 and NI-2100 devices were released before the problem was resolved.
So you shouldn't see too many of them.
Installation was done 4 months with no issues. Last two weeks NI2100 processor froze twice. Jandy is connected to RS232 port number two. I was at the job site when processor was lockup up. TX and RX of Jandy RS232 LEDs were solid on. The Status led was blinking fine, I did no have my laptop with me to ftp. I just did a reset and problem went away. It happened again today. Will the hot fix solve this issue.
I am going to order some to have on hand after reading this thread!
The C155's are 10uf 35v.
Thanks a ton, now I will have some on hand in my bag just in case I run into it on a jobsite.
AMX charges $150 to fix these, I think the relative value of a x100 processor is worth that to be sure it survives. The x000 series uses larger and more easily repaired parts, and is lower value-lower risk if you fail... so worth doing yourself. IMHOO.
Now, you mentioning that you have a pool controller on the processor tells me that most likely there is a problem with the pool controller connection that is causing extreme amounts of line noise/garbage that is filling up the incoming buffer and appearing to lockup the processor. I would start by checking all of the wiring connections in the line for the pool controller connection. Make sure nothing has corroded from exposure to moisture. I have seen this numerous times and it fits your symptoms. Also, do you have the Jandy adapter near your AMX processor or is it near the pool equipment with a longer RS232 connection to the processor? Lastly, are you plugging the Jandy adapter directly into the processor or using a 3 conductor only cable?
Jeff
It could be that they later changed the design, but in the one I opened its not the case.
While the rest of your info is good, the above isn't quite what we've seen and heard. The capacitor failure drops only the carrier voltage on the RS232, the data is actually fine, and what results is similar to what RS422/485 expects, and the lights still work. And it works fine without the voltage if your device out there doesn't rely on it. And being a capacitor failing, it can respond differently across time and boots. My own 3000 for a while would work fine for a day after a rest and reboot. Then fail. After a short time, it always failed. My lights always looked fine. It was the capacitors.
Later production x100's have a different design, harder to fix, far less likely to fail, but it still can happen. The symptoms are the same. We have 2 of these waiting now, failed in use exactly as the others, with serial gradually becoming unreliable, some devices affected before others, and results varying upon reboots and cold reboots.
I beleive the OP said the TX/RX lights were solid which to me means something is really chatty or the code may occasionally get into a loop where it's sending out strings constantly.
I would agree, the cap issue has not been reported to lock the lights up.
As for replies (yellow lights) to the defective sends (red), I'd agree for devices where it has completely failed. In the half-working state, the device may be responding with a "what???". You should be able to examine the replies and see if they were understanding what was sent.
AMX identified this (very sporadic) behaviour as a firmware bug a while ago and resolved it with a firmware release. At the time the firmware release was stated to prevent devices flooding the network with Bonjour packets causing the zeroconfig related firmware portions in the master to go off into the weeds though I never could figure out whether this was the same issue or an additional one. To the user, the master appeared to be running fine, but the on board control ports had locked up.
Updating the master and device firmware on your controller and turning off zeroconfig via the console should prevent it recurring.
Thank you all for the feedback.
Auser could you elaborate on "turning off zeroconfig via the console".
Jeff AquaLinkRS is near AMX and AquaLink RS came with RS232 adapter/cable which we used to make the connection to the processor. I have to check if only 3 conductors are used, we just used what it came with.
I don't think my issue is related to capacitor, it might be related to CPU usage and chatty AquaLinkRS device.
Today I was at the client's house and everything was running fine. She rebooted the processor once 4 days ago,
One this project I have one NI2100 and one NI3101.
NI2100
RS232 #1 AutoPath Video Matrix
RS232 #2 Jandy AquaLinkRS
RS232 #3 DSC
IR .....
IP Control: Lutron RadioRA2 2 systems with 150 dimmers
IP Control: Zigbee gateway with 4 R4 remotes
2 Termostates
NI3101
RS232 #1 Integra Receiver #1
RS232 #2 AMX Tango 12 Zones
RS232 #3 Integra Receiver #2
RS232 #4 Integra Receiver #3
RS232 #5 iPort #1
RS232 #6 iPort #2
IR ...
IP Control: 2 Panamax conditioners
5 TouchPanels.
NI2100 average cpu usage is 70% to 80%.
NI3101 average cpu usage with one iPort enabled is 10% to 15%.
AquaLink RS is polling every 3 sec(TX lights up every 3sec). DSC TX lights up every 5 sec to 10 sec.
I'm guessing my issue is distribution of processing power between two processor and AquaLinkRS chatty device. Can my code have some unknown loop which will take all processing power. Any suggestions????????
I believe the console command is but don't have a master in front of me. Typing "help" into a telnet or serial console session should point it out pretty quickly.
http://www.amx.com/techsupport/techNote.asp?id=970
Description: While Zeroconf is a useful tool during system setup, it may be disabled afterwards to eliminate unnecessary traffic.
Zeroconf may be disabled on NetLinx masters by telneting into a master and issuing "zeroconf disable" command. This concept applies to non-master AMX devices such as the ZigBee gateway and others also.
This pertains mostly to large networks with many BONJOUR (Apple compliant) devices, including most AMX IP devices like masters.