Home AMX User Forum AMXForums Archive Threads AMX Hardware

Card slots not coming online

a_riot42a_riot42 Posts: 1,624
At two different sites I have 4100s that once every few reboots won't allow the card slots back online. A reboot or two usually fixes this but its a major annoyance. Has anyone seen this behavior? I reported it to AMX but they want more debug which I can't give them as there are clients living in the locations where this is happening.
Thanks,
Paul
«1

Comments

  • Spire_JeffSpire_Jeff Posts: 1,917
    I have experienced this same problem. I believe it to be a problem with the ICSNet. I have experienced the problem on a 4100 in which the card slots don't come online, but I have also experienced it with a CF12 connected to an NI-3100. On the 3100, it was easier to narrow the problem down. I happened to have 2 of the 3100 both with ICSNet ports. I could unplug the frame from the problem processor and plug it into the second processor. As soon as I did that, the cards showed up on the second processor. Move it back to the problem processor and they never show up.

    With the 3100, it seemed like the only way to get the ICSNet back was to pull to power to the processor for a few seconds, altho once I had it work with a soft reboot. On the 4100, I have had it work with a reboot.

    Jeff

    P.S.
    I have reported it to tech support, but it has been so difficult to reliably reproduce that I have not expected an answer to be found quickly.
  • a_riot42a_riot42 Posts: 1,624
    Spire_Jeff wrote: »
    With the 3100, it seemed like the only way to get the ICSNet back was to pull to power to the processor for a few seconds, altho once I had it work with a soft reboot. On the 4100, I have had it work with a reboot.

    I have only seen it on the 4100s and a reboot usually solved it. I have watched the startup sequence and noticed no difference between a boot when they come online or not. Its a major bug though, since having no cards work disables the system until its rebooted.
    Paul
  • jjamesjjames Posts: 2,908
    a_riot42 wrote: »
    I have only seen it on the 4100s and a reboot usually solved it. I have watched the startup sequence and noticed no difference between a boot when they come online or not. Its a major bug though, since having no cards work disables the system until its rebooted.
    Paul

    Yup - we experience the same thing quite often with 4100s, but also with 3100s as well (Axlink bus strip is usually connected, and those won't come online.)

    I've learned to monitor the devices in the card frames and after 10 or 15 seconds (in DEFINE_START) check to make sure they're all online, if not - I send a reboot. If it reboots 5 times without them coming online, I put up a popup on the panels to have them call use immediately to remedy the situation. The latter hasn't happened yet, but just a bit of a CYA.
  • mpullinmpullin Posts: 949
    I've also had this problem recently. jj's approach is the right one, I believe.
    Another thing that helps is telnetting into the master and setting the Lontalk threshold to the max. For the 4100s it defaults to something really low like 50.
  • ColzieColzie Posts: 470
    I had a situation with an NI-4000 - you could watch diagnostics and see the cards going online/offline repeatedly. Had to send the unit back for repair, they said the ISCNet was bad.

    I wasn't involved from the start, but perhaps my situation started off like what you are seeing, Paul?
  • KennyKenny Posts: 209
    Same here, I had a 4100 that the cards wouldn't always come online after a reboot.
    It was pretty intermittent, I haven't seen it in the past month or so.
    A power cycle or another reboot always fixed it.
  • mpullinmpullin Posts: 949
    So 6 people in this thread alone have experienced cardframe issues with the NI-4x00s?
    I wonder if AMX has done anything to address this.
  • jjamesjjames Posts: 2,908
    mpullin wrote: »
    So 6 people in this thread alone have experienced cardframe issues with the NI-4x00s?
    I wonder if AMX has done anything to address this.

    I've been fighting this since July of 2007 (my first time seeing it was then.) I know I had reported it then and spoke to tech support at length about it. I've come to just accept it and check to make sure they're online before proceeding.
  • a_riot42a_riot42 Posts: 1,624
    jjames wrote: »
    Yup - we experience the same thing quite often with 4100s, but also with 3100s as well (Axlink bus strip is usually connected, and those won't come online.)

    I've learned to monitor the devices in the card frames and after 10 or 15 seconds (in DEFINE_START) check to make sure they're all online, if not - I send a reboot. If it reboots 5 times without them coming online, I put up a popup on the panels to have them call use immediately to remedy the situation. The latter hasn't happened yet, but just a bit of a CYA.

    That seems to the reasonable approach, but even needing something like that is rather lame. What's next, making sure the built in ports are online too?
    Paul
  • jjamesjjames Posts: 2,908
    a_riot42 wrote: »
    That seems to the reasonable approach, but even needing something like that is rather lame. What's next, making sure the built in ports are online too?
    Paul

    Agreed but . . . oh well.

    I'm still waiting for cruise control to actually take me to my destination. ;)
  • AuserAuser Posts: 506
    mpullin wrote: »
    So 6 people in this thread alone have experienced cardframe issues with the NI-4x00s?

    Make that seven.
  • ericmedleyericmedley Posts: 4,177
    Auser wrote: »
    Make that seven.

    The only issue I've ever had with the many 4100s I've put in was more of an overall ICSNet dropping offline. I was never able to isolate it to the card slot within the 4100 itself. So, I'll make it 7.5 people who've had trouble.
    e
  • ColzieColzie Posts: 470
    The card slots are just ICSNet expansion slots physically located in the same chassis as the NI. I doubt anyone has been having trouble with their card slots but having perfectly working ICSNet devices elsewhere in the system.

    Just my hunch.
  • bcirrisibcirrisi Posts: 148
    Make that 8!

    I've had the same problem, I ended up rolling back the firmware on a couple projects to make it work every time. Some one in tech supports does know about it, because they gave me the older firmware.
  • Any updates on this? Have a job that has a 3100 and two IRS4 cards... About one reboot a day as I'm testing results in the IRS4 cards not coming online. A soft reboot always fixes it.

    I'll implement James' track/reboot fix, but curious if AMX ever found an issue.All the latest firmware is installed.

    Kevin D.
  • jjamesjjames Posts: 2,908
    Nope. Dealing with a job that has a 4100 with the latest firmware . . . and I still see the problem every now and then.
  • Spire_JeffSpire_Jeff Posts: 1,917
    I thought they had verified the problem and I thought that there was a firmware released to address the problem.... but if you are running current firmware, I must be wrong. I have not had the problem recently, but I haven't been doing a lot of rebooting on the processors that would be affected.

    Jeff
  • the8thstthe8thst Posts: 470
    I have one job with two 3100's and two card frames that don't always come back online after fresh code is loaded. A soft reboot has always fixed it.
  • ericmedleyericmedley Posts: 4,177
    i can now officially say that I'm firmly in the camp of having a problem. So much so that I've written a work around in the coce to deal with it. I basically watch for the cards to come online. If they haven't done so after 60 seconds, I reboot the master.

    I reboot the master 3 times if still no workie. after 3 strikes, I get an email and page to my phone that trouble awaits. This is in two systems with 4100s.
  • bcirrisibcirrisi Posts: 148
    Still having the problem as well....
  • Spire_JeffSpire_Jeff Posts: 1,917
    Has anyone called tech support regarding this issue recently? I am fairly sure that there was a setting they had me change in the processor that helped the situation. I am sure I'm going to screw up my weekend, but in the last month or so, I have not run into the problem.

    Jeff
  • I thought there was a firmware update to address it also...

    I've had the problem in the past but haven't noticed it lately either. Usually they would come up within a few reboots.

    --John
  • jjamesjjames Posts: 2,908
    Correction - I am running the latest master firmware, not the latest device firmware for my NI-4100. I am currently running 1.13.7. It was supposedly addressed in that release (current being 1.20.7 and mentions nothing of AxLink lockups); 1.13.7 still locks up, however not as frequently. (I've seen it twice so far on this job.)
  • 3.50.430 with 1.20.7 device firmware here on the 3100 here.

    Kevin D.
  • Kenny wrote: »
    Same here, I had a 4100 that the cards wouldn't always come online after a reboot.
    It was pretty intermittent, I haven't seen it in the past month or so.
    A power cycle or another reboot always fixed it.

    I have seen this as well, where all 4 cards were not online. I have also experienced a situation where none of the Duet devices ever came online. The reboot fixes the problem, but it makes for a real flakey system when the homeowner only sees that his expensive system isn't working!
  • GSLogicGSLogic Posts: 562
    jjames wrote: »
    Nope. Dealing with a job that has a 4100 with the latest firmware . . . and I still see the problem every now and then.

    This will help in SOME situations.

    Force the master to wait before loading data_event online data
    TELNET: set device holdoff on
    query: get device holdoff
  • TechFreakTechFreak Posts: 39
    TechFreak wrote: »
    I have seen this as well, where all 4 cards were not online. I have also experienced a situation where none of the Duet devices ever came online. The reboot fixes the problem, but it makes for a real flakey system when the homeowner only sees that his expensive system isn't working!

    I have this problem with a 4100 as well, both the cards not online and the Duet virtuals not online. So basically, 3/4 of there system doesn't work. The reboot works to refresh, but it should just F-ing work the first time. This is like starting your car and none of the gauges work, and having to re-start your car to get the speedo to report your speed. I am getting tired of the AMX BS, with Vapor-Firmware that will someday fix the problem, constantly hearing from TS to update firmware, only to have other issues/problems introduced with the new firmware.

    I can't be the only one that feels like AMX quality control is out of control?!?
  • a_riot42a_riot42 Posts: 1,624
    GSLogic wrote: »
    This will help in SOME situations.

    Force the master to wait before loading data_event online data
    TELNET: set device holdoff on
    query: get device holdoff

    This has side effects as well though doesn't it? Doesn't it disable all messages during define_start?
    Paul
  • DHawthorneDHawthorne Posts: 4,584
    TechFreak wrote: »
    I have this problem with a 4100 as well, both the cards not online and the Duet virtuals not online. So basically, 3/4 of there system doesn't work. The reboot works to refresh, but it should just F-ing work the first time. This is like starting your car and none of the gauges work, and having to re-start your car to get the speedo to report your speed. I am getting tired of the AMX BS, with Vapor-Firmware that will someday fix the problem, constantly hearing from TS to update firmware, only to have other issues/problems introduced with the new firmware.

    I can't be the only one that feels like AMX quality control is out of control?!?

    Considering the breadth of their product line, problems like this are few and far between. 95% of the AMX products I have dealt with over the years were rock-solid ... of course, that remaining 5% stands out because they brings the whole system crashing down. The car analogy is apt there too ... the owner doesn't care if it's a minor issue or a major one that prevents his car from starting, and doesn't appreciate one bit how hard it might be to track down the problem. He just wants to use his car. I have a friend who ranted and raved about his Saturn (before GM ditched them) when the engine died on him ... how he would never buy another "piece of junk" Saturn again, etc., etc., what an unreliable car it was. Yet, I had one myself that was over 10 years old, was never in the shop for anything more than an oil change, and even still looked new. No matter how good your overall product line might be, if you have a few glaring issues, it those that people notice and remember.
Sign In or Register to comment.