ICSPTCP End ???

jsnyder

Greetings,

I've been undergoing a rash of TP disconnects with no clue as to why. the NI controllers stay online but the TP's (CV10's and 1000vi's) appear to randomly lock up requiring a minion to be dispatched to which ever of the 51 classrooms this occurs in. The issue has gotten so bad I have rewired all of the TPs power through an unused relay on the controller so i can reboot them from my "Omnipanel".

Here is a sample of the log from a locked panel.

2: 02-22-2013 FRI 08:33:58 MasterUrlConnect
Connected to (Sys=3) Master 1XX.21.200.113 (URL Mode)

3: 02-22-2013 FRI 08:33:49 SocketClosure
ICSPTCP End

4: 02-22-2013 FRI 08:33:41 MasterUrlConnect
Disconnected from (Sys=0) Master 1XX.21.200.113 (URL Mode)

5: 02-22-2013 FRI 08:33:41 SocketError
ICSP TCP Socket read error: iBytesRead==0

6: 02-22-2013 FRI 08:32:26 SocketError
Socket connect attempt failed: No route to host (errno=0x94)

7: 02-22-2013 FRI 08:32:20 SocketError
Socket connect attempt failed: No route to host (errno=0x94)

8: 02-22-2013 FRI 08:32:15 SocketError
Socket connect attempt failed: No route to host (errno=0x94)

9: 02-22-2013 FRI 08:32:09 SocketError
Socket connect attempt failed: No route to host (errno=0x94)

10: 02-22-2013 FRI 08:32:04 SocketError
Socket connect attempt failed: No route to host (errno=0x94)

11: 02-22-2013 FRI 08:32:01 SocketError
Socket connect attempt failed: No route to host (errno=0x94)

12: 02-22-2013 FRI 08:31:55 SocketError
Socket connect attempt failed: No route to host (errno=0x94)

13: 02-22-2013 FRI 08:31:50 SocketError
Socket connect attempt failed: No route to host (errno=0x94)

14: 02-22-2013 FRI 08:31:02 SocketClosure
ICSPTCP End

15: 02-22-2013 FRI 08:30:54 MasterUrlConnect
Disconnected from (Sys=0) Master 1XX.21.200.113 (URL Mode)

L Mode)

79: 02-22-2013 FRI 07:53:30 SocketError
ICSP TCP Socket read error: iBytesRead==0

80: 02-22-2013 FRI 07:52:05 SocketClosure
ICSPTCP End

81: 02-22-2013 FRI 07:51:57 MasterUrlConnect
Disconnected from (Sys=0) Master 1XX.21.200.113 (URL Mode)

82: 02-22-2013 FRI 07:51:57 SocketError
ICSP TCP Socket read error: Connection reset by peer (errno=0x83)

83: 02-22-2013 FRI 07:50:33 SocketClosure
ICSPTCP End

84: 02-22-2013 FRI 07:50:25 MasterUrlConnect
Disconnected from (Sys=0) Master 1XX.21.200.113 (URL Mode)

85: 02-22-2013 FRI 07:50:25 SocketError
ICSP TCP Socket read error: iBytesRead==0

FRI 08:25:49 SocketError
Socket connect attempt failed: Connection timed out (errno=0x91)

23: 02-22-2013 FRI 08:22:40 SocketClosure
ICSPTCP End

24: 02-22-2013 FRI 08:22:32 MasterUrlConnect
Disconnected from (Sys=0) Master 1XX.21.200.113 (URL Mode)

25: 02-22-2013 FRI 08:22:32 SocketError
ICSP TCP Socket read error: iBytesRead==0

26: 02-22-2013 FRI 08:21:31 SocketClosure
ICSPTCP End

27: 02-22-2013 FRI 08:21:23 MasterUrlConnect
Disconnected from (Sys=0) Master 1XX.21.200.113 (URL Mode)

28: 02-22-2013 FRI 08:21:23 SocketError
ICSP TCP Socket read error: iBytesRead==0

29: 02-22-2013 FRI 08:20:33 SocketClosure
ICSPTCP End


Any thoughts?

Thanks in advance for any help.

John

Hedberg

NXV-300 virtual panels have had similar problems. Look at this thread, for example:
http://www.techdirt.com/articles/20130217/18381022008/attribution-troll-press-inc-now-50-less-troll-like-also-not-yelling-people-sells-more-books.shtml

Things to try (as suggested by tech support):

disable bonjour on the TP if the panels have this feature

Use static IP addresses, set ethernet mode to full, 100 (or 10 or something) on masters and tps

Disable zeroconfig on the masters

Disable UDP on the masters (baud rate to zero)

Disable SSH on the masters (port set to zero)

jsnyder

SSH port now off

all other settings were as suggested (I've been reading posts trying to resolve this for sometime)

now we wait and see.

Have a splendid weekend.
John

jsnyder

Nope not yet

had a panel drop after turning off ssh

Mkn

Are all panels connected to the same LAN switch?

Recently had indentical problem (all panels were on static LAN IP) on site, network switch (D-Link in my case) was the reason of this problem. The new switch (Netgear) solved this problem.
For test you can use any 5 (or 8 port) switch, connect controller and some panels. If it helps, the problem is network switch.

VLad.

jsnyder

Replacing 51 d-link switches with Cisco business class gig switches

We'll try this and let you know.
Thank you.

John

jsnyder

Still no joy!

from a controllers log:


19: 03-13-2013 WED 02:12:36 Interpreter
CIpEvent::OffLine 10001:1:1041
20: 03-13-2013 WED 02:12:36 ICSPTCP
ICSPTCPRx1::CloseSocket: Closing Socket

and the touch panel never regains its IP.

For goodness sake! will some linux hacker out there write a new firmware for cv-10's and 1000vi's
so they agressivly regain their IP after a loss?
oh yeah,
PLEASE
thanks
John

DHawthorne

It's not just touch panels. I have a customer with an NI-3000 that he complained was locking up daily. This was a system that had been working fine for years. When I got on site, I found the master was losing it's connection to the router, so I tried putting it on DHCP to see if there was something that just needed to be cleared in some invisible cache somewhere. It wouldn't get a DHCP address no matter what I tried, so I gave up and swapped it with an NI-3100 I just got as a trade-in when another customer upgraded to an Enova DGX. The 3100 was just fine, and has been for about a week now. I figure, "problem solved, the 3000 had a dying NIC or something of that nature." Yet, it works fine at my office. That customer has a very old and very messed up network (he's his own IT guy ... shudder ...), and I've been trying to sell him better, but he isn't going for it. I can talk him into buying the 3100 upgrade, but it still leaves me scratching my head what caused the problem in the first place.

jsnyder

ip addresses ......

Sounds like he might have "seen" it on HIS network not realizing it was valid he locked it out??

Our infrastructure varies from building to building. yesterday 10 classroom touch panels lost their IPs while the controllers and other devices were fine, all on a older commercial switch (HP400).

today its 8 panels dropping IPs in another building thats only 2 years old. multiple reboots of panels required.

arg!

John

DHawthorne

jsnyder wrote: »

Sounds like he might have "seen" it on HIS network not realizing it was valid he locked it out??
John

No, it was really locking up. For one, the touch panel, though reachable from the outside via VNC, can't see the master on the house network. For two, I keep persistent logs on my masters, and the module is set to delete logs more than 5 days old (which I can set to whatever I want, but this guy was 5). There were no logs on the master except the current day, which tells me it was not recording for 5 days and when it rebooted, the old ones were all deleted.

But to get back on topic, since the same flaky master works on *my* network, I can't help but think a noisy network environment is part (and probably only a part, since a new master fixed it) of the problem. I once had a master that was locking up like this 2-3 times a week, and I was pulling my scant hair out trying to find it (there's a big long thread on it in here), and that time a new master *didn't* fix it. It fixed itself. It just suddenly started working just fine, thank you, and has been for over a year. I strongly suspect there are things going on that the AMX devices don't like, but aren't vocal about either ... my logs never show much of anything except what you are seeing: something going off line, no idea why.

On new jobs, I use a managed network switch, and insist the customer upgrades the network as part of the job. I make sure I have a VPN router, and I divide the system into subnets so my control stuff has it's own network. Only time will tell if it's a complete solution.

MC2

network issues

Hi
AMX is not very "open" in explaining those non-documented meesages in the logs.

I have a project with 25 android panels + TPC license. They are continuously going offline and online. Cisco AP's. Tried other AP's, hiding SSID, changing channels, etc etc ...
PC's connected wireless to the same isolated network never have an issue.
The panels have just a displaying function, so I left it like it is

I had years ago a project where everything was nicely controlled and integrated on the network of the customer, a big telecom operator. For one or other reason, after a while the http and telnet part of the NI crashed.
I had the feeling at that time that the NI's don't like too many requests on their NIC (like ARP etc etc).
We created and moved them to a dedicated multimedia VLAN and things were solved.
We were the only 3thd party contractor with gear on their network they had to do that for and I can tell you that it was the last time we sold amx to them.

Numerous times I did the effort to give feedback on the java problem with NI controllers.
I.e. when you browse to the NI with a browser that has the latest version of java, the probability is high that the webbrowser in the NI will crash after a while.
Only remedy is rebooting the NI, but when the facility is in use at that moment, ...
So I proposed to foresee a function or command to reboot only the webserver via f.e. telnet.
I can understand that amx cannot foresee that every NI with older firmware is compatible with the latest java version, but amx ... foresee "something" without having to reboot the NI.

Another project.
48 rooms, 37 NI controllers
After an NI is rebooted, webserver and telnet crashes after 2 to 8 hours.
Also here investigated many hours together with network experts of the customer without resolution.
Also here we are the only vendor with that kind of problem.


But keep on smiling all ... the only way is up

best regards
Mike

ericmedley

... reminds me of the bad ole days when you'd see the same behaviour when using those awful cheap Linksys WGxxxxx wireless routers or switches. Like Dave, we had a guy who just would not upgrade his network. the solution we ended up using was a power controller that also a ping poller. When it quit seeing the NI master respond, it would power cycle the NI and everything was fine. We tracked the incidents and found that there seemed to be no rhyme or reason to the occurance other than it kinda happened every3 or 4 days. (sometimes longer, sometimes several the same day.) There was no big run up of errors in the NI log prior to it happening or anything like that.

This same behaviour occured in touch panels occasionally too.

It's kinda scary to hear that it seems to be back...

jsnyder

NXD-1000Vi gets it's IP address back when you navigate to the System Settings page!!

The controllers stay online, but the silly panels are still going off and online like nuts..
get to the panels setup and go to the "System Settings" page and within 10 seconds it rejoins the network..