Master Lost its Persistent Memory

TurnipTruck

Greeting,

I have a system with an NI-700 as the master. It has been up and running just fine for a year or so. Yesterday, during a power failure, it lost all values stored as persistent memory. I have since tried rebooting, loading code, power cycling and won't lose the values again. It seems to be a freak occurence.

Anyone else had something similar occur?

annuello

I've seen this before with NI-3100 masters. From what I determined, it looks like persistent variables are stored using the battery from the timekeeper (clock). When the battery goes flat it can no longer store persistent variables past a power-cycle. A few things that led me to this conclusion are:
1) The clock would not keep it's time, resetting to the year 2031.
2) Looking at the AMX documentation closely, it never claims that persistent variables are committed to FLASH. From a design perspective this makes sense, since PVs might be updated frequently.
This one is wrong. 3) I believe NLS stores persistent variables on the PC when re-dumping a program, then re-writes these back to the NI. This "hides" the flat battery issue when you are programming a unit.
4) Whilst my units were only 6 months old when the symptom appeared, they (or their timekeeper) may have been running for several years while sitting in shipping containers, in warehouses, etc. I am not guaranteed a fresh TK battery upon delivery.

When I first came across the issue you had to replace the entire timekeeper. I believe it is possible to open it up and replace the battery, but I have never bothered to do this. My way to cope with the issue is to save all "persistence" to XML files. I then load the files into volatile memory at boot time. (I know I'm contradicting point #2, but my PVs are basically set-and-forget.) I also time-sync against a time server. Prior to RMS I used i!-TimeManager which worked a treat. Post-RMS I let it handle all the time syncing, and just sync RMS against a time server. This works well for those of us that can not be bothered replacing all our timekeepers. 250+ and counting...

Roger McLean
Swinburne University

Marc Scheibein

Roger is right, and I can confirm points 1, 2 and 4.
Not sure how master/Studio are keeping the persistent values.

HARMAN_shane-a-roo

Point 3 is not correct. NLS plays no part in persistent values beyond the compilers naming of variables. Variable persistence is achieved in the master by matching the value of the variable with the same name from battery backed storage with the variable in the newly loaded program. Non-volatile is achieved by simply overlaying the variable layout of the current loaded program over the battery backed memory area from the previous execution, thereby restoring all non-volatile values.

In both cases a failure of the battery will cause a loss of the values in the battery backed ram.

TurnipTruck

I have never dissected an NI-series controller. Are the batteries easily replaced as they were in the Axcent 3?

John Gonzales

Replacing the battery isn't too difficult. There's a timekeeper chip that sits right below the CF slot. The battery is a yellow snaphat battery that clicks into the timekeeper chip. Here's a picture of the CMOS battery and a timekeeper chip (not from a Netlinx though): http://eu.st.com/stonline/products/families/memories/nvram/images/5917ph_2.jpg

--John

John Gonzales

Actually, here's a better picture that you can zoom in on and move around:
http://www.rapidonline.com/Electronic-Components/Integrated-Circuits/Real-Time-Clocks/SNAPHAT-Battery-and-crystal-modules-for-Timekeeper-memory-devices/72964/kw/

--John

HARMAN_shane-a-roo

Batteries on the NI series are not something that would be replaced in the field. It's a special lithium battery with pins that snaps on to a chip. The batteries are supposed to last for 10 yrs. So the question would be a) has your battery actually run out and if so b) why. To test a) you simply need to set some persistent/non-volatile values, pull the power from your master completely, restore power and see if the persistent values are still there. If they are, then it's not your battery that's the problem and something else occurred causing you to lose persistent values (ex. inadvertently downloading the wrong program). If you determine that it's actually the battery that is bad, I would suggest contacting Tech Support to find out what your options are.

John Gonzales

shane-a-roo wrote: »

Batteries on the NI series are not something that would be replaced in the field. It's a special lithium battery with pins that snaps on to a chip. The batteries are supposed to last for 10 yrs..

I have CFL light bulbs rated for 10,000 hours that I replace after a few months .


--John

DHawthorne

I've seen power surges and dirty momentary power interrupts cause this kind of thing. It's very rare, but I even had a master get it's running program scrambled; a re-load fixed it , and it never happened again. Even with a power conditioner and UPS, sometimes this stuff just happens. I wouldn't worry too much about it unless it starts repeating.

ericmedley

DHawthorne wrote: »

I've seen power surges and dirty momentary power interrupts cause this kind of thing. It's very rare, but I even had a master get it's running program scrambled; a re-load fixed it , and it never happened again. Even with a power conditioner and UPS, sometimes this stuff just happens. I wouldn't worry too much about it unless it starts repeating.

I have two masters out there that live in bad power land. they both have lost their programs twice over the years. A quick upload fixed the problem. It's obvioulsy not impossible for such a thing to happen. I have also seen one case of losing persistent variable values as well.

HARMAN_shane-a-roo

After further digging, apparently, I was in error. Replacement snaphat batteries can actually be order through AMX for repair in the field.

But you'd certainly want to verify that it's actually the battery that's failed and not some other RAM or Flash glitch.

annuello

shane-a-roo wrote: »

Point 3 is not correct. NLS plays no part in persistent values beyond the compilers naming of variables. Variable persistence is achieved in the master by matching the value of the variable with the same name from battery backed storage with the variable in the newly loaded program. Non-volatile is achieved by simply overlaying the variable layout of the current loaded program over the battery backed memory area from the previous execution, thereby restoring all non-volatile values.

In both cases a failure of the battery will cause a loss of the values in the battery backed ram.

Thanks for clarifying point #3, shane-a-roo. As I said, "I believe"... Now I believe otherwise! (If there were a BB Code for putting a line through #3 I'd do that straight away.) I was pondering this one overnight and was starting to doubt myself anyway. It's been so long since I've had to deal with the issue that my mind was getting a bit fuzzy. Thankfully I haven't had to worry about it since saving all PVs to flash via .xml files.

Roger McLean
Swinburne University

Thomas Hayes

I know that a power spike going in the negative direction ( I believe negative) will cause the unit to drop all memory. The issue I believe that the chipset used was not correctly spec'd by the manufacture and it could handle positive going spikes but not negative spike.