Options
ICSNet Bus Dropping Offline Periodically
Reese Jacobs
Posts: 347
in AMX Hardware
I have a relatively simple Netlinx installation consisting of the following:
- NXC-ME260/64 Master
- NXI Integrated Controller (connected with NXC-HE Hub Expander)
- NXC-COM2 (in a Module Shell)
- NXC-IRS4 (in a Module Shell)
The NXI is connected to the ICSNet Hub Out port on the Master and the 2 module shells are connected directly to ISCNet ports on the Master. The NXI controls several DSS receivers (IR), some pumps (relays), 2 Apex 6100 panels (RS232), and also monitors a number of contact closures (IO). The COM2 runs a thermostat system with 8 OPS-300 stats all connected through a Disty panel and RS422 to the COM2. Lastly, the IRS4 controls several IR devices and monitors several PCS devices. As I said, this is a simple system.
Here is the problem - Periodically all devices on the ICSNet bus go OFFLINE and then immediately come back ONLINE. There is no specific timeframe associated with the failures - sometimes failures are a week apart, several hours apart, and I have even had several failures within the span of an hour. Over a given 4-week period, the problem probably only occurs 6-8 times and it is not predictable.
When the problem occurs, the Master remains ONLINE (network) and it does not reboot. Further, all G4 panels (TPI/4 and MVP) remain ONLINE so IP based connections are not affected. Once the ICSNet devices come back ONLINE and go through their ONLINE handlers, the system returns to normal operation as if the problem never occurred. No Axlink connections are used with this system.
Here is what I have done and what is not a source of the problem:
- COM2 card has been replaced (along with module shell)
- IRS4 card has been replaced (along with module shell)
- NXI has been replaced (along with Hub Expander card)
- Cables have been replaced
- No cable runs exceed 6' (head-end installation)
- Termination has been checked and verified
- All components on UPS (generator backup after 30 seconds)
- Surge or power quality issues have been ruled out
- PSN6.5 power supplies have been swapped
- Environmental causes ruled out (temp is constant 78F, humidity 45%)
- Log files indicate no abnormal events except all ICSNet devices drop offline and immediately come back online
- Master code includes Design Express queue and threshold settings per AMX recommendations
Has anyone had similar experiences with ICSNet devices dropping OFFLINE for no reason and then coming back ONLINE immediately? I have not replaced the Master but that is the next step of course. Before arriving at the configuration noted above, an NXC-NH was used to connect the ICSNet bus to the Master and in turn to the NXI and NMS module shells. I eliminated the Hub as an effort to further troubleshoot the problem to ensure that a bad hub was not the problem and further to determine if the problem seemed to be all devices on the ICSNet or a particular device.
Note that this problem might be hard to detect in some installations. If the devices come back ONLINE and everything resets or initialized properly, it might not be noticeable. I found the problem in this installation since several relays were reverting to an open state and anytime these pump relays change state, a warning email is generated. When I realized what was occurring, I added code to the OFFLINE/ONLINE handlers to log when they go OFFLINE and to send messages accordingly. Note, the Master has never rebooted, the system is generally very light on traffic, and the system is generally lightly used except for some asynchronous events when the problem occurs.
Does anyone have any suggestions or similar experiences aside from swapping out the Master? Thanks in advance,
Reese
- NXC-ME260/64 Master
- NXI Integrated Controller (connected with NXC-HE Hub Expander)
- NXC-COM2 (in a Module Shell)
- NXC-IRS4 (in a Module Shell)
The NXI is connected to the ICSNet Hub Out port on the Master and the 2 module shells are connected directly to ISCNet ports on the Master. The NXI controls several DSS receivers (IR), some pumps (relays), 2 Apex 6100 panels (RS232), and also monitors a number of contact closures (IO). The COM2 runs a thermostat system with 8 OPS-300 stats all connected through a Disty panel and RS422 to the COM2. Lastly, the IRS4 controls several IR devices and monitors several PCS devices. As I said, this is a simple system.
Here is the problem - Periodically all devices on the ICSNet bus go OFFLINE and then immediately come back ONLINE. There is no specific timeframe associated with the failures - sometimes failures are a week apart, several hours apart, and I have even had several failures within the span of an hour. Over a given 4-week period, the problem probably only occurs 6-8 times and it is not predictable.
When the problem occurs, the Master remains ONLINE (network) and it does not reboot. Further, all G4 panels (TPI/4 and MVP) remain ONLINE so IP based connections are not affected. Once the ICSNet devices come back ONLINE and go through their ONLINE handlers, the system returns to normal operation as if the problem never occurred. No Axlink connections are used with this system.
Here is what I have done and what is not a source of the problem:
- COM2 card has been replaced (along with module shell)
- IRS4 card has been replaced (along with module shell)
- NXI has been replaced (along with Hub Expander card)
- Cables have been replaced
- No cable runs exceed 6' (head-end installation)
- Termination has been checked and verified
- All components on UPS (generator backup after 30 seconds)
- Surge or power quality issues have been ruled out
- PSN6.5 power supplies have been swapped
- Environmental causes ruled out (temp is constant 78F, humidity 45%)
- Log files indicate no abnormal events except all ICSNet devices drop offline and immediately come back online
- Master code includes Design Express queue and threshold settings per AMX recommendations
Has anyone had similar experiences with ICSNet devices dropping OFFLINE for no reason and then coming back ONLINE immediately? I have not replaced the Master but that is the next step of course. Before arriving at the configuration noted above, an NXC-NH was used to connect the ICSNet bus to the Master and in turn to the NXI and NMS module shells. I eliminated the Hub as an effort to further troubleshoot the problem to ensure that a bad hub was not the problem and further to determine if the problem seemed to be all devices on the ICSNet or a particular device.
Note that this problem might be hard to detect in some installations. If the devices come back ONLINE and everything resets or initialized properly, it might not be noticeable. I found the problem in this installation since several relays were reverting to an open state and anytime these pump relays change state, a warning email is generated. When I realized what was occurring, I added code to the OFFLINE/ONLINE handlers to log when they go OFFLINE and to send messages accordingly. Note, the Master has never rebooted, the system is generally very light on traffic, and the system is generally lightly used except for some asynchronous events when the problem occurs.
Does anyone have any suggestions or similar experiences aside from swapping out the Master? Thanks in advance,
Reese
0
Comments
The only thing you did not mention was terminating resistors on the ICSNet loop-thru connectors. I was taught that they should be used. Short of that I would suspect the Hub connector on your Master.
I have a much larger system with 5 card frames using ICSHub connections without a problem.
Both of the module shells (housing the NXC-COM2 and NXC-IRS4) are properly terminated with standard ICSNet/Phastlink terminators. These are both directly connected to separate ICSNet ports on the Master - there is no daisy chaining involved for any of the ICSNet devices. The NXI is connected to the Master on the Hub Out port since it uses a Hub Expander for its ICSNet bus connection. I originally was daisy-chaining and using hubs but I changed to this configuration to see if all devices on the bus were equally affected without them being daisy-chained or connected to the same hub. Each device essentially has its own ICSNet connection to the Master although they of course share the bus. I did not terminate the Hub Out port on the NXI but then I generally do not terminate Hub Out ports and I have never run into a problem before now.
As it stands now, if this appears to be a problem with the Master, it is affecting all of the ICSNet and Hub ports equally unless the problem is being caused by a device on the bus (even though they have all been swapped for new devices and hence it is unlikely it is the COM2, IRS4, NXI, or module shells). The other possible explanation would be that one of the devices connected to and being controlled by an ICSNet controller (one of the RS232 ports) is causing the problem and it is manifesting itself in the ICSNet bus. In other words, perhaps one of the Apex panels or the OPStat Disty panel is in fact the problem.
Thanks again for the thoughts.
Reese
Hi,
A couple of questions.
1. Are you powering the NXS-NMS shells through ICSNet or the PSN6.5?
2. Is there some particular reason you have not installed the master inside the NXI instead of using a NXC-HE? Just curious.
3. Please don't be offended with this question, but have found it to be a problem in the field before. Are the ICSNet CAT5 cables made to spec 568A or 568B? I have found some cables made "Straight Through" by installers which can cause this problem.
Regards,
Rex
Hey Rex,
Thanks for the thoughts - responses are below.
The NMS shells for the COM2 and IRS4 cards are being powered from the ICSNet and are therefore sharing the same PSN6.5 as the Master. The NXI is on a different PSN6.5 since the system is being expanded with an NXF cardframe housing a full complement of NXC-VOL4 cards once this problem is resolved.
No, not a significant reason. I did not house the Master in the NXI primarily because the NXI is in a different part of the rack and the Master is located adjacent to the ICSNet/Ethernet surge protect panel and the primary Ethernet network switch. As a general rule, I like the flexibility of replacing or upgrading the Master without having to pull it from an NXI housing and I find it more convenient when it is in a shell. It is just a matter of preference but I don't think it should have a bearing on this problem.
No offense taken -- I am looking for possible solutions so I consider anything to be a possibility. In this very small installation, commercial off-the-shelf standard 568B CAT5e cables were used. Each of the cables has been replaced as part of the troubleshooting process. The cables were also swept with a Fluke OneTouch network analysis device and checked out fine. I am fairly confident that the cables are not the problem.
Thanks for the help -- let me know if you have additional ideas to check.
Reese
Kevin D.
Yes, I have wired the ICSNet connections directly to the shells and the NXI in an effort to eliminate the surge protector as a potential problem. This does not eliminate the problem unfortunately. I am kind of running out of options other than to try a new Master and see what impact that has on the problem.
I also neglected to mention in my 1st post that the Master is running 2.31.137 and has not been upgraded to Duet firmware. The NXI as well as the COM2 and IRS4 cards are also on the latest firmware revisions available for those cards.
Thanks --
Reese
Try powering the shells directly from the PSN6.5 instead of the ICSNet buss.
Rex
Have you tried removing the ICSNet/Phastlink terminators to see if this problem still exists? I?ve had experiences with terminators sometimes creating stability problems with the ICSNet bus similar to yours.
Can't say for sure if I have tried eliminating the terminators for the module shells or not - it can't hurt to try.
The other thing I have seen wreak havoc is monmentary brownout condidtions. I had a system that made me insane for months, until one day on site I noticed the lights dim when the customers AC compressors kicked in. It was enough of a voltage drop to mess up his Phastnet (same as ICSNet nowadays) - a UPS on the processor fixed it.