Options
Master Loss of Network Connectivity
jabramson
Posts: 106
in AMX Hardware
NI-700 with lots of connectivity to other systems and out to the internet to grab scheduling & weather data. I've been having a problem where the unit will lose network connectivity. I originally thought it was related to a master hardware that died and I've since installed new hardware. I've also rebuilt my entire program adding in a section at a time and monitoring. Got a bunch of bugs worked out and all was working well for a few weeks, but the issue has returned. Sometimes it doesn't happen, other times it takes several hours, and other times within an hour. I'm not really sure where to check next. I suspect it may be running out of IP ports, but not completely sure. As far as I can tell, the master doesn't actually lock up, it just drops the network connectivity.
There are 10 TPs directly connected to this master
5 M2M connections. Other other Masters are running their own source.
This master just monitors the other masters, also updates the 10 TPs with schedule and weather data.
I was able to run the debug until it crashed and here are the last lines:
10.44.11.207, 10.44.2.200 are another masters (M2M)
Ports 15 & 16 are IP devices which have gone offline. It attemps to reconnect every few mintues.
Any direction is appreciated
There are 10 TPs directly connected to this master
5 M2M connections. Other other Masters are running their own source.
This master just monitors the other masters, also updates the 10 TPs with schedule and weather data.
I was able to run the debug until it crashed and here are the last lines:
Line 54909 (00:23:50):: SendString to socket-local port (16) invalid Line 54910 (00:23:50):: CIpEvent::OnError 0:16:3 Line 54911 (00:23:51):: SendString to socket-local port (16) invalid Line 54912 (00:23:51):: CIpEvent::OnError 0:16:3 Line 54913 (00:23:57):: CICSPTCP Rx connection to 10.44.11.207 has been closed locally or by peer Line 54914 (00:23:57):: ICSPTCPRx15::CloseSocket: Closing Socket Line 54915 (00:24:06):: SendString to socket-local port (15) invalid Line 54916 (00:24:06):: CIpEvent::OnError 0:15:3 Line 54917 (00:24:09):: SendString to socket-local port (15) invalid Line 54918 (00:24:09):: CIpEvent::OnError 0:15:3 Line 54919 (00:24:11):: SendString to socket-local port (15) invalid Line 54920 (00:24:11):: CIpEvent::OnError 0:15:3 Line 54921 (00:24:12):: SendString to socket-local port (16) invalid Line 54922 (00:24:12):: CIpEvent::OnError 0:16:3 Line 54923 (00:24:15):: SendString to socket-local port (16) invalid Line 54924 (00:24:15):: CIpEvent::OnError 0:16:3 Line 54925 (00:24:16):: SendString to socket-local port (16) invalid Line 54926 (00:24:16):: CIpEvent::OnError 0:16:3 Line 54927 (00:24:29):: CICSPTCP Rx connection to 10.44.11.207 has been closed locally or by peer Line 54928 (00:24:29):: ICSPTCPRx15::CloseSocket: Closing Socket Line 54929 (00:24:31):: SendString to socket-local port (15) invalid Line 54930 (00:24:31):: CIpEvent::OnError 0:15:3 Line 54931 (00:24:35):: SendString to socket-local port (15) invalid Line 54932 (00:24:35):: CIpEvent::OnError 0:15:3 Line 54933 (00:24:36):: SendString to socket-local port (15) invalid Line 54934 (00:24:36):: CIpEvent::OnError 0:15:3 Line 54935 (00:24:37):: SendString to socket-local port (16) invalid Line 54936 (00:24:37):: CIpEvent::OnError 0:16:3 Line 54937 (00:24:40):: SendString to socket-local port (16) invalid Line 54938 (00:24:40):: CIpEvent::OnError 0:16:3 Line 54939 (00:24:41):: SendString to socket-local port (16) invalid Line 54940 (00:24:41):: CIpEvent::OnError 0:16:3 Line 54941 (00:24:52):: SSH connection accepted 10.44.2.200:46899 socket=4125 Line 54942 (00:24:52):: Accepted Telnet connection:socket=75 addr=10.44.2.200 port=46900 Line 54943 (00:24:52):: SSH_closeConnection instance=4125 Line 54944 (00:24:57):: SendString to socket-local port (15) invalid Line 54945 (00:24:57):: CIpEvent::OnError 0:15:3 Line 54946 (00:25:00):: SendString to socket-local port (15) invalid Line 54947 (00:25:00):: CIpEvent::OnError 0:15:3 Line 54948 (00:25:01):: SendString to socket-local port (15) invalid Line 54949 (00:25:01):: CIpEvent::OnError 0:15:3 Line 54950 (00:25:01):: CICSPTCP Rx connection to 10.44.11.207 has been closed locally or by peer Line 54951 (00:25:01):: ICSPTCPRx15::CloseSocket: Closing Socket Line 54952 (00:25:02):: SendString to socket-local port (16) invalid Line 54953 (00:25:02):: CIpEvent::OnError 0:16:3 Line 54954 (00:25:03):: Accepted Telnet connection:socket=26 addr=10.44.2.200 port=46902 Line 54955 (00:25:05):: SendString to socket-local port (16) invalid Line 54956 (00:25:05):: CIpEvent::OnError 0:16:3 Line 54957 (00:25:06):: SendString to socket-local port (16) invalid Line 54958 (00:25:06):: CIpEvent::OnError 0:16:3 Line 54959 (00:25:08):: Accepted Telnet connection:socket=26 addr=10.44.2.200 port=46903 Line 54960 (00:25:13):: Accepted Telnet connection:socket=26 addr=10.44.2.200 port=46905 Line 54961 (00:25:13):: Accepted Telnet connection:socket=26 addr=10.44.2.200 port=46906 Line 54962 (00:25:18):: Accepted Telnet connection:socket=26 addr=10.44.2.200 port=46907 Line 54963 (00:25:18):: Accepted Telnet connection:socket=26 addr=10.44.2.200 port=46908 Line 54964 (00:25:18):: Accepted Telnet connection:socket=26 addr=10.44.2.200 port=46909 Line 54965 (00:25:18):: Error setting TCP socket to blocking Line 54966 (00:25:22):: SendString to socket-local port (15) invalid Line 54967 (00:25:22):: CIpEvent::OnError 0:15:3 Line 54968 (00:25:23):: Accepted Telnet connection:socket=26 addr=10.44.2.200 port=46913 Line 54969 (00:25:23):: Accepted Telnet connection:socket=26 addr=10.44.2.200 port=46914
10.44.11.207, 10.44.2.200 are another masters (M2M)
Ports 15 & 16 are IP devices which have gone offline. It attemps to reconnect every few mintues.
Any direction is appreciated
0
Comments
You can try telnet diagnostics to see what is up on IP and if there are a lot of dead connections building up.
Show TCP List
The following TCP connections exist(ed):
1: IP=192.168.1.180:35284 Socket=23 (Active)
Last[MsgsInQ=0 RecvTime=0006892814 SendTime=0006892802]
2: IP=192.168.1.129:1271 Socket=24 (Active)
Last[MsgsInQ=0 RecvTime=0006892805 SendTime=0006892802]
IP Status
NetLinx IP Connections
TCP Server Listener on IP Port 5750 (backlog=1) (socket=34)
TCP Server waiting on IP Port 5750 LocalPort=21
I'll check it again after it's been running for a while to see if anything builds up.
The telnet commands are a strange one. That is not a piece of my equipment (non AMX or related). I've reached out to our IT group to see if they know what it is.