NXT-CA7/CV7 Connectivity Issues in a University Environment
Gesch
Posts: 7
Hello. Sorry in advance as this is a long post. I'm trying not to rant.
Back in 2009 I helped develop an in entire business school (University of Nebraska Omaha, College of Business) with classroom systems that are driven by AMX control systems.
We brought the building online in 2010 There are around 30 controllers. There were a few issues (all programmer... me) but we worked around them in the first few months of deployment.
Then, last spring, my CV7 touch panels starting locking up... losing all communication with the master and I either needed to power cycle them or pull their network cable briefly depending on the one of three WELL documented situations we encountered.
After WEEKS of troubleshooting I had our network 'people' (not the most accurate folks) create a separate VLAN for my gear. Call it a network island. No communications in and none out... very unfortunate for my remote management capability. Suddenly, amazingly, they stopped locking. All was well... except for all the remote management I had been building for two years but hey. WTH.
Recently, I implemented Netlinx code logging a great deal of data to a syslog server I have deployed and I had them (the network people) create a private subnet that would allow me to talk to my controllers and I moved off 'the island'. It took 48 hours for my CV7's to start locking again. I am looking at a very long weekend putting my control system gear back off the 'semi-private' network... again.
I am posting here hoping and PRAYING someone else has dealt with this issue and has a better solution than 'the island'. These touch panels worked for 2+ years off 'the island' without this issue. And yes, I have rolled backwards, forwards, sideways, and out of bed with code revisions trying to find a way around this issue. There is something on my network that has appeared in the last 4 months that affects my CV7's and those only. My NXT-1700VG's, 5150's and MXT's are UNAFFECTED. I have tried everything I can think of.
Here is my short list of what I have done so far. Yes, I was IT for 15 years before converting to AV out of necessity:
Moved all control system gear to an isolated subnet that can only be accessed from in the building - N/E
Passwords on all - Always
Passwords changed - N/E
Disabled Telnet - N/E
Enabled ICSP - N/E
Enabled SSH - N/E
Disabled G4 Web/VNC - N/E
Changed G4 Web/VNC port to xxxx - N/E
Disabled my 2 year old M2M network - N/E
Rebuilt my M2M network - N/E
Isolated my Master and Touch Panel - Effective but worthless
Moved all control system gear to an isolated VLAN - Effective but worthless
Moved to a more isolated but accessible subnet - N/E
I don't know how many of you deal with a static AV environment (such as your own dedicated building) that must maintain but I am running out of ideas. My co-worker and I have probably tried other scenarios aside from the ones above but I just can't think of them right now.
Any Ideas?
Thanks in advance.
Gesch.
Back in 2009 I helped develop an in entire business school (University of Nebraska Omaha, College of Business) with classroom systems that are driven by AMX control systems.
We brought the building online in 2010 There are around 30 controllers. There were a few issues (all programmer... me) but we worked around them in the first few months of deployment.
Then, last spring, my CV7 touch panels starting locking up... losing all communication with the master and I either needed to power cycle them or pull their network cable briefly depending on the one of three WELL documented situations we encountered.
After WEEKS of troubleshooting I had our network 'people' (not the most accurate folks) create a separate VLAN for my gear. Call it a network island. No communications in and none out... very unfortunate for my remote management capability. Suddenly, amazingly, they stopped locking. All was well... except for all the remote management I had been building for two years but hey. WTH.
Recently, I implemented Netlinx code logging a great deal of data to a syslog server I have deployed and I had them (the network people) create a private subnet that would allow me to talk to my controllers and I moved off 'the island'. It took 48 hours for my CV7's to start locking again. I am looking at a very long weekend putting my control system gear back off the 'semi-private' network... again.
I am posting here hoping and PRAYING someone else has dealt with this issue and has a better solution than 'the island'. These touch panels worked for 2+ years off 'the island' without this issue. And yes, I have rolled backwards, forwards, sideways, and out of bed with code revisions trying to find a way around this issue. There is something on my network that has appeared in the last 4 months that affects my CV7's and those only. My NXT-1700VG's, 5150's and MXT's are UNAFFECTED. I have tried everything I can think of.
Here is my short list of what I have done so far. Yes, I was IT for 15 years before converting to AV out of necessity:
Moved all control system gear to an isolated subnet that can only be accessed from in the building - N/E
Passwords on all - Always
Passwords changed - N/E
Disabled Telnet - N/E
Enabled ICSP - N/E
Enabled SSH - N/E
Disabled G4 Web/VNC - N/E
Changed G4 Web/VNC port to xxxx - N/E
Disabled my 2 year old M2M network - N/E
Rebuilt my M2M network - N/E
Isolated my Master and Touch Panel - Effective but worthless
Moved all control system gear to an isolated VLAN - Effective but worthless
Moved to a more isolated but accessible subnet - N/E
I don't know how many of you deal with a static AV environment (such as your own dedicated building) that must maintain but I am running out of ideas. My co-worker and I have probably tried other scenarios aside from the ones above but I just can't think of them right now.
Any Ideas?
Thanks in advance.
Gesch.
0
Comments
There is also a hotfix for NI masters, but this generally would benefit those who have them open to the world wide web and are experiencing lock-ups.
Hotfixes are not available for web download and can only be obtained through Technical Support.