Home AMX User Forum AMXForums Archive Threads AMX Hardware
Options

NI-3100 Ports(IR-RS232) sometimes not working and there is no error message.

We use 2 NI-3100 and 3 MVP 5200 for a project. Sometimes first NI-3100 device ports not working. All panel connected first controller. The NI 3100 is online, but ports is not working. No connection problem panel's connection in this time. There is no error message. I reset the NI3100 and it's work regularly. I change NI-3100 and I change first and second devices of NI's. but I have the same problems. The second NI3100 work regularly.

First Controller,
RS1 AUTOPATCH 18X18
RS2 IPORT1
RS7 BOWERS AND WILKINS PANAROMA SOUNDBAR1
IR1 SATELITTE1
IR3 SATELITTE2
IR4 SATELITTE3
IR5 SATELITTE4
IR6 SATELITTE5
IR7 CD CHANGER
IP AMX Multi-Format Dist. Hub and Rec. Kit ENDELLEO
Second Controller;
RS1 BOWERS AND WILKINS PANAROMA SOUNDBAR1
RS2 IPORT2
RS3 IPORT3
IR1 HARMAN KARDON BLURAY PLAYER1
IR2 HARMAN KARDON BLURAY PLAYER 2
IR3 TV1
IR4 TV2
IR5 SATELITTE 6
IR6 SATELITTE 7
IR7 SATELITTE 8
IP AMX KNX
«13

Comments

  • Options
    ericmedleyericmedley Posts: 4,177
    You could put a data_event with an online: / Offline: event on the ports and track to see if they fall offline that way.

    However, if all the devices fall offline, then there might be something wrong with the board itself. I'd call Tech Support and see if they can get you an RMA. It might be something simple too like the device firmware might need reloading. You can try that yourself.
    Hope that helps
    e
  • Options
    When the system locks up like this, you may also want to telnet into the 3100 that is having the problem and do a SHOW BUFFERS.

    When I have systems act this way, the counts for the Interpreter start building up. But a build-up of any of the numbers is bad. Normally, everything is 0 and you will sometimes see a number go to 1. (Unless you telnet in while it is still booting up - then it is not unusual to see numbers in the various counters).

    The serial and IR ports die first, and if you wait long enough the IP communications will die as well. So you have to telnet in after the problem starts but not before IP locks up too.

    Are you using any modules to control the serial devices? I've just started having similar issues with what had been old and stable systems after the firmware was updated to the latest version. For my systems, it looks like there is an issue between the old NetLinx Sony EVID100 camera module and the new Duet firmware. A few days after a reboot the interpreter counts start building and the system locks up. Replacing the module with the newer Duet version has eliminated the problem.
  • Options
    Pep_SDPep_SD Posts: 106
    Three days ago (yes, Christmas Eve, 8pm...) we experienced a similar situation on one site with 2 NI-3100 (latest firmware updated a month ago to 3.50.439):
    The program was still running (page flips controlled in program were working) but nothing from Rs232 to IR was getting out from the NI's - the Tx leds weren't even flashing.
    I was still able to remote into each NI through port 1319.
    Rebooting each NI solved the issue. The problem did not happen the same day on each NI.

    What's wiered is one of the NI program has not been modified for a year or so - I could suppose a 'glitch' on the other that is in full revamping, but not the second one.

    Pretty concerning...
  • Options
    AMX Investigating

    First I will say there could a number of things causing the problem. The first thing to check would be your NetLinx program, buffers, etc. Just basic debugging. I've seen a number of "lockups" that were just code related.

    Second, I will say that we have been chasing an issue related to device side ports that just seem to stop working. This has thus far been isolated to a few installations, all in Europe and the Ukraine. I say isolated because I have not been made aware of other installations. It could be that more are having this issue and I'm just not aware of it. The symptoms are that the serial ports stop communicating, IR ports, relays, IO stop working. The LEDs on the device side will either go off or stay on, I've seen both. At a low level what is happening is that communication between the master side and the device side stops. No messages go either direction. This has the utmost priority in engineering and we are working this.

    Until we find a good solution there may be things you can do to help yourselves to minimize the number of times this occurs. The times I've seen this issue are on systems where there is a lot of traffic through the serial ports, most often caused by polling devices. Large amounts of traffic seems to worsen the problem and polling is traffic. Reduce your poll times to what is necessary. If you don't need to poll 5 times a second and can get away with once a second then you've decreased the likelihood of a problem by 5x. Most sites never see this issue. A few sites experience this once every 2 months. We are trying to find what is common among the sites that do have the issue and what is different from the sites that don't have it. We have all kinds of equipment and scopes hooked up to our masters so when a problem occurs we can try to isolate the root cause. We are also using all of our black magic and voodoo powers in hopes of getting a resolution quickly to those in the field experiencing the problem.

    Obviously the amount of polling should not cause the device side to stop functioning. Nothing should. But the fact is we do have an issue we are working hard to resolve. I will try to post here when we have something we think will be ready for hot fix testing.

    I'd also like to note that we have tested with the latest build available as well as gone back to very old builds and they all have the issue. This isn't something that just popped up. This has been verified by the dealers in the field who have done their own testing.

    Stay tuned.....
  • Options
    This problem started in one of my projects as well. I have a NI-3000 that has been running fine for over a year, no changes on code or firmware either. I only have 2 serial devices (Projector and Receiver) and I am using only one way serial commands out, no pooling or receiving and the serial ports don't communicate anymore. A reboot didn't fix it either. IR control still works. I am very concerned about this problem. I have a lot of NIs out there and some projects are even out of town. I hope AMX can fix this as soon as possible; otherwise, I will have quite a few unhappy clients and non paid trips and field visits.
  • Options
    If the ports are down and the IR stills works, it's the capacitor issue. Easy fix if you know how to solder, but last time I talked to TS, they are fixing them even if out of warranty.

    Kevin D.
  • Options
    Good news?

    For the issue with the serial ports that stop working but IR still works, that is not the same issue as is being discussed in this thread. The issue here is that everything on the device side stops. I believe the previous response was correct. There is a known fix for that problem.

    Now for some possible good news. We have a possible fix for the device side lockups. We have not been able to reproduce our failures with this fix. It is undergoing our hot fix testing process right now and if it can survive that we will be able to put a hot fix out to address the problem. I am hoping like mad that the problem we reproduced is the same problem that is affecting the units in the field (and in this thread). I believe it is, but I've learned not to assume too much. :)
  • Options
    AuserAuser Posts: 506
    rgelling wrote: »
    Second, I will say that we have been chasing an issue related to device side ports that just seem to stop working. This has thus far been isolated to a few installations, all in Europe and the Ukraine.

    Rick, please add Australia to the list.

    I went out to a site immediately prior to Christmas to investigate an issue as the client was switched on enough to send through the attached image when reporting that the system wasn't functioning. As previously described the program was running, touch panels on line and page flips being handled by the code, but the device layer appeared to have stopped communicating with the master (NI-3100 running 3.50.430/1.20.7 or 3.50.439/1.20.7 - I can't remember which).

    The code for this system is very simple, the only device being being polled is a Polycom HDX codec (using the AMX comms module; PolycomHDX9002_Comm_dr2_0_0).

    Unfortunately I wasn't in a position to capture logs off the NetLinx at the time. Site staff indicated that they have seen occasional power brown outs at the site.
  • Options
    DHawthorneDHawthorne Posts: 4,584
    Interesting ... I may be seeing this on an NI-3000. The customer complains that the system is locking up, and they have to power it down to get it working again. Every indication is the master is still running, but none of the devices are responding. Since they have gotten reset-happy, it's hard for me to tell for sure; it's running again by the time I get there. However, I did have a tech on site once when it happened, and he says he saw the TX and RX lights on a comm port connected to a RS-232 controlled Marantz BluRay were lit solid, no flashing, both on at once. So he disconnected the RS-232 line, and we haven't had lockups since (though, again, it's hard to say since it's not like they happened every day .. it's only been disconnected about 24 hours). The Marantz does need to be polled (no unsolicited feedback), and I am doing so every 800ms.

    Unfortunately, I'm not going to get a chance to see if bumping up that poll rate fixes it, since the customer doesn't like that BluRay anyway and is going to replace it, so it's not going to get reconnected.
  • Options
    Hot Fix is available

    I am pleased to report that the hot fix for this problem is now available from tech support. You can call and get it. You will require master side firmware version 3.60.447 and device side firmware 1.30.4. Make sure you get the firmware versions for the right type of masters. It is available now for the 2100/3100/4100 and 700/900 masters. There are two versions of device side firmware available for the 700/900 depending on what type you have (older 32MB or newer 64MB version) so make sure you get the right one. There is no hot fix yet for the 2000/3000/4000 but we will be working on that.

    Also, I ran across an issue at a customer site that had the exact same symptoms as this device side lockup. However, it wasn't the device side. After some evaluation we believe the Zero Config client in the master was receiving wrongly formatted zero config packets from some device(s) on the network and this caused problems in many forms. All master activity stopped so it looked like the device side wasn't working. You could ping the master but could not telnet to it or do anything else with it. The program port, however, was working. We will be looking into this in more detail to understand what could cause this situation. In the meantime, if you have multiple masters on the network lock up at the same time it is not going to be the issue this thread is about. It is more likely to be the issue I just explained. There is an easy solution: disable the zero config client in the master. This can be done with the following telnet command: "zeroconfig disable". This command requires a reboot to take effect. I hope this helps some of you out there that may be experiencing this type of problem after a system wide power up (when zero config devices start broadcasting packets on the network) or at random times when zero config devices are restarted or enter the network.

    Cheers.
  • Options
    rgelling wrote: »
    I am pleased to report that the hot fix for this problem is now available from tech support. You can call and get it. You will require master side firmware version 3.60.447 and device side firmware 1.30.4. Make sure you get the firmware versions for the right type of masters. It is available now for the 2100/3100/4100 and 700/900 masters. There are two versions of device side firmware available for the 700/900 depending on what type you have (older 32MB or newer 64MB version) so make sure you get the right one. There is no hot fix yet for the 2000/3000/4000 but we will be working on that.

    Also, I ran across an issue at a customer site that had the exact same symptoms as this device side lockup. However, it wasn't the device side. After some evaluation we believe the Zero Config client in the master was receiving wrongly formatted zero config packets from some device(s) on the network and this caused problems in many forms. All master activity stopped so it looked like the device side wasn't working. You could ping the master but could not telnet to it or do anything else with it. The program port, however, was working. We will be looking into this in more detail to understand what could cause this situation. In the meantime, if you have multiple masters on the network lock up at the same time it is not going to be the issue this thread is about. It is more likely to be the issue I just explained. There is an easy solution: disable the zero config client in the master. This can be done with the following telnet command: "zeroconfig disable". This command requires a reboot to take effect. I hope this helps some of you out there that may be experiencing this type of problem after a system wide power up (when zero config devices start broadcasting packets on the network) or at random times when zero config devices are restarted or enter the network.

    Cheers.

    I'm getting ready to apply this hot fix in the hopes it also solves the problem I'm having. Serial and IR communication are fine. My MVP-9000i can connect to it, and I can even telnet to the master. I cannot connect to the master through NS3 though...until I reboot the master. Here's hoping this hotfix somehow manages to fix my problem as well.
  • Options
    GeschGesch Posts: 7
    Location of newer versions?

    Hello.

    We just finished our first semester in a new building with new technology... all controlled by AMX equipment. This issue being described here is exactly what I have been chasing. Great news for me if this solves the problem.

    My only question where are these versions located?

    " You will require master side firmware version 3.60.447 and device side firmware 1.30.4"

    The versions I see here...

    http://www.amx.com/techcenter/firmware.asp?Category=Central Controllers

    ...are from mid to late 2010.

    Thanks in Advance.
  • Options
    John NagyJohn Nagy Posts: 1,735
    Gresh, look again at the message you qoute...
    "I am pleased to report that the hot fix for this problem is now available from tech support. You can call and get it. You will require master side firmware version 3.60.447 and device side firmware 1.30.4."

    Your question is answered in the first and second lines. Available from tech support... you can call and get it.
  • Options
    GeschGesch Posts: 7
    A good speed reader I am not...

    Thanks.
  • Options
    I'm getting ready to apply this hot fix in the hopes it also solves the problem I'm having. Serial and IR communication are fine. My MVP-9000i can connect to it, and I can even telnet to the master. I cannot connect to the master through NS3 though...until I reboot the master. Here's hoping this hotfix somehow manages to fix my problem as well.

    Unfortunately, this fix did not fix my problem with not being able to connect to the master. On I go to find out what's going on here. I'll start a new thread to not hijack this one.
  • Options
    GeschGesch Posts: 7
    Device Firmware issues.

    Hey all. Awesome speed reader here.

    I was able to grab the hot fix files off of the FTP site. The master firmware file drops in fine but I receive a 'Transfer aborted - Firmware ID is invalid' when I try to update the device.

    This is a 4100 and I am attempting to send SW2105_NI_X100_Device_v1_30_4.kit. The device side is currently at 1.20.7.

    Suggestions?

    Thanks in advance.

    Gesch
  • Options
    ericmedleyericmedley Posts: 4,177
    Gesch wrote: »
    Hey all. Awesome speed reader here.

    I was able to grab the hot fix files off of the FTP site. The master firmware file drops in fine but I receive a 'Transfer aborted - Firmware ID is invalid' when I try to update the device.

    This is a 4100 and I am attempting to send SW2105_NI_X100_Device_v1_30_4.kit. The device side is currently at 1.20.7.

    Suggestions?

    Thanks in advance.

    Gesch

    Are you selecting device 5001 and not 0? it should be whatever the device number for the ports on your 4100. default is 5001.
  • Options
    GeschGesch Posts: 7
    Yes I am. Strange. I tried the update on a production 4100 and the master and device firmware went in fine.

    Still poking at it.
  • Options
    Any word on a hotfix for the NI-x000's? Just ran into this problem today on a NI-3000 and came prepared to solder in a new capacitor.. Same issue as this thread, I had an RS232 TX light stuck on, but none of the device ports were working. Reboot fixed it up.

    Kevin D.
  • Options
    GeschGesch Posts: 7
    I have been running this hot fix on 26 4100's/3100's for nearly two weeks and not one hang-up. That for the help and hand holding.

    Derek G
  • Options
    viningvining Posts: 4,368
    anybody got this hotfix firmware? Arrived at a site and most com ports aren't working. Reload firmwares, program, etc, etc. Called TS and left a message but.... Why dealers don't have a site so we can access hotfixs 24/7. Went to the ftp site they normally leave stuff but nothing.

    send to viningeleATmsnDOTcom

    With out the fix I'll probably leave soon. TIA.

    edit,
    Forget it! Got the calback from TS and the hotfix doesn't fix the cap problem.
  • Options
    viningvining Posts: 4,368
    The problem with the NI-3100 on yesterday's service call are as follows.

    TX lights would flash as the program would send commands.
    RX light would not flash in response making it seem all devices were dead.
    RX lights would flash for unsolicitated data being sent from devices.

    Only 1 device sent unsolicitated messages so it appeared everything else was dead and the 1 device that sent unsolicited responses did so, so frequently that it appeared the port was working when the program did TX to it. At first I thought that port was working but later realized I was being duped.

    Connected Hyper-Termial to a NI serial port and sent strings via control device to this port to display in HT but the while TX light flashed HT didn't RX anything.

    The unsolicited data sent from the 1 device that did so was received and processed normally. With HT still connected to the NI's serial port I could send strings to the master, the RX light would flash and I assume the RX data would be processed just as the data on the port with unsolicited feedback did. I didn't verify in debug though.

    So something it apparently is making the strings being sent out the serial ports disappear but still flashing the TX led. The ports will RX fine but since the devices never recieve commands they don't talk so unless they send unsolicited feedback you won't see flashing RX leds.

    I'm not sure if this is exactly the same thing that everyone else is seeing but I'm pretty sure this is what I'm seeing.

    Of course this was on a job that I've never been to before running programs I've never seen before and I went there thinking just the music server was down based on a phone call from the client. I did have the full client file from the original programmer/installer though.

    I did update the firmware to the most recent and reloaded the dev firmware and uploaded the program again but to no avail. I didn't use the Hot Fix firmware since it's supposedly for systems brought down by to much chatter, they'll go back to normal after a reboot for an unknown amount of time. Maybe they'll never fail again but they also may fail that day or the next week. That's another issue.

    This is the cap crap?
  • Options
    I can't recall anyone seeing the cap problem on the x100's, but that may just be the redesigned ones with flat LEDS.

    With the cap problem, RS485/422 will still work as they don't use the negative voltage you lose when the cap goes out.

    Kevin D.
  • Options
    viningvining Posts: 4,368
    I thought the cap problems were in the x100's but I'm could easily be remembering wrong. This is an older NI-x100 model so I guess will see. I hope it is though and not something else either wrong with these models or worse in my diagnosis of the problem. That would make me look quite the a$$. :)
  • Options
    Well, I started getting calls from a client that has a small system with a NI-2000 which was working fine and now looks up frequently. A reboot brings things back to normal, but after a couple days it locks up again. I went there last week and updated the firmware on the NI-2000 and the 2 MVP-7500s to the latest, changed the router to a Cisco/Linksys E3000, changed the encryption to WPA and when I left the site everything was working good again. Yesterday, just got a message saying that they had to reboot again. This is my second system needing to have constant reboots. Both systems run fine for over a year without any reboots. The other system a NI-3000 was sent back to repair since the RS-232 ports wouldn't work. But, this NI-2000, the serial ports still work, but the system keeps locking up. The running code hasn't been changed since over a year when the system was stable. I am getting very frustrated and not looking so good with the clients that I sold AMX controllers for their reliability, which now is questionable on this clients minds. I hope AMX can come up with a fix soon. Otherwise, my list of unhappy clients will grow since I have quite a few NIs out there.
  • Options
    viningvining Posts: 4,368
    That might be the problem the Hot Fix firmware discussed ealier in this thread is for. That problems goes away after a reboot and may go away for good or come back when ever. I would give TS a call monday Monday for the new Master & DEV firmware and give it a try.

    Worst case schedule daily reboots when after x amount of time with out any usage and around 3:30 am.
  • Options
    From what I read from this tread the Hot Fix is only for the NI-2100,3100, 4100 series. is there any hotfix for the NI-NI-2000, NI-3000, NI-4000 yet? I will call TEch Support on Monday. Thanks.
  • Options
    From what I read from this tread the Hot Fix is only for the NI-2100,3100, 4100 series. is there any hotfix for the NI-NI-2000, NI-3000, NI-4000 yet? I will call TEch Support on Monday. Thanks.

    Called when I posted on the 22nd... No hotfix for the x000's yet.

    Kevin D.
  • Options
    mushmush Posts: 287
    telnet wrote:
    (0140577557) CI2CMessageQueue Failed to send message, retry: 0, errno: 3d0004
    (0140580556) CI2CMessageQueue Failed to send message, retry: 1, errno: 3d0004
    (0140583555) CI2CMessageQueue Failed to send message, retry: 2, errno: 3d0004
    (0140586555) CI2CMessageQueue Failed to send message, retry: 0, errno: 3d0004
    (0140589554) CI2CMessageQueue Failed to send message, retry: 1, errno: 3d0004
    (0140592554) CI2CMessageQueue Failed to send message, retry: 2, errno: 3d0004
    (0140595553) CI2CMessageQueue Failed to send message, retry: 0, errno: 3d0004
    (0140598552) CI2CMessageQueue Failed to send message, retry: 1, errno: 3d0004
    (0140601551) CI2CMessageQueue Failed to send message, retry: 2, errno: 3d0004
    (0140604551) CI2CMessageQueue Failed to send message, retry: 0, errno: 3d0004
    (0140607436) Optimizing tCypherConnRx messages with 500 messages in queue.
    (0140607437) Optimization tCypherConnRx complete with 500 messages now in queue. 0 messages avoided.
    (0140607550) CI2CMessageQueue Failed to send message, retry: 1, errno: 3d0004
    (0140609436) CMessagePipe::Write tCypherConnRx (PipeReader) this=0x019AD118 has 500 Messages in queue (Deleting & Ignoring1)
    (0140610549) CI2CMessageQueue Failed to send message, retry: 2, errno: 3d0004
    (0140610550) CMessagePipe::Read tCypherConnRx (PipeReader) is now Reading again

    Are these the error messages you'd expect to see with this problem??
  • Options
    Jorde_VJorde_V Posts: 393
    mush wrote: »
    Are these the error messages you'd expect to see with this problem??

    Yes, I have seen those messages on the systems we were experiencing the issues with.
Sign In or Register to comment.