Size and depth of structure issue

fogled@mizzou

I've been revamping some of my code, and creating bigger nested structures to hold all my device information. At this point, I've got a structure with up to 4 nested sub-structure levels that adds about 200K to the TKN file on upload.

When I have this structure defined, no code will run on my controller. When I comment the structure out, code will run OK. I seem to be breaking some limit somewhere, but... I don't know what it is.

ericmedley

The max depth of an array is 5 dimension I beleive. Perhaps you've gone over that limit in your structures.

fogled@mizzou

I'm pretty sure I'm only 4 deep. Here's my actual structure code at the moment:

structure deviceip { 
    char name[32] 
    char ipaddr[20] 
    integer ipport
    integer iptype 
    dev devid
} 

structure msgs { 
    char msgdate[19]
    char msgtext[80]
}

structure lamps { 
    integer pwr
    char hr[5]
}

structure temps { 
    char val[5]
}

structure valmap { 
    char str[32]
}

structure paneldat { 
    char name[32] 
    integer tab 
    dev devid 
    char tpbuff[100]
    char tpinput[100]
}

structure displaysettings { 
    char name[32]
    integer pwr
    integer vmute
    integer feed
    integer follows
    integer status
    integer swtid //VGA Switch ID
    integer vport //VGA Port Number
    integer type
    integer commtype
    deviceip ipcomm[1] 
    char sercomm[64]
    dev serdevid[1]
    integer chan
    integer amute
    integer volcatv
    integer vol
    integer volset
    integer tmp
    integer projerr
    integer errorcount
    char text[32]
    char buffout[64] //holds buffer of integer commands as string data for fifo operations 
    char buffin[1024]
    integer lastcmd
    integer msgidx
    msgs currmsg[1]
    msgs prevmsg[128]
    lamps lamp[1] 
    temps temp[1] 
    integer lamplimlevel
}

structure audchans { 
    char name[20]
    char id[1]
    char chan[1]
    integer val
    integer mute
    integer volset
}

structure audiodata { 
    char name[32] 
    integer type 
    integer commtype 
    deviceip ipcomm[1]
    char sercomm[64]
    dev serdevid
    char buffin[256] //Extron Mp121
    char buffout[256] //Extron MP121
    audchans ch[20] 
}

structure lightchans { 
    char name[20]
    char chan[1]
    integer volset
    integer val
    integer isgang
}

structure lightingdata { 
    char name[32] 
    integer type 
    integer commtype 
    deviceip ipcomm[1]
    char sercomm[64]
    dev serdevid[1]
    char buffin[256] //Extron 
    char buffout[256] //Extron 
    lightchans ch[20] 
}

structure inputs { 
    char name[32]
    integer swtid //VGA Switch ID
    integer vport //VGA Switch Video Port (nornally Aud and Vid)
    integer aport //VGA Switch Audio Port (Use separate Aud/Vid for oddball stuff)
    integer follows 
}

structure switchdata { 
    char name[32] 
    integer type 
    char sercomm[64] 
    dev devid
    char buffin[512] 
    char buffout[512] 
} 

structure helpdata { 
    integer helpbtns[32] 
    char buffin[256] 
    char buffout[1024] 
}


structure room { 
    char name[32] 
    char dbid[12]
    integer status
    paneldat panels[1] 
    displaysettings d[8] 
    audiodata audio[1] 
    inputs in[8] 
    lightingdata lights[1] 
    switchdata swt[2] 
    helpdata help[1]
    // what else in the room? 
    char helpbuffin[256] //Help/Log gateway
    char helpbuffout[1024] //Help/Log Gateway
}

...and then in Define Start, I have: PERSISTENT roomdata rm[20]

Actually, you may be right because I'm using DEV way down in rm[x].d[x].ipcomm[x].devid - which is itself a DEV, but that's still just 4 deep, not 5 deep. I can't spot anything that goes more than 4 deep, can anyone else? Is there anything else in here that's a no-no?

Thanks,

fogled@mizzou

It looks like the total structure size was just too big. In the DisplaySettings structure, I had a "msgs" sub-structure with 128 slots, and that sub-structure had an 80-char container.

I backed the msgs structure call off to 10 instead of 128, and the system works OK.

DHawthorne

Telnet into your master and type in "show mem" . That will tell you how much memory of each type is being used. I'm willing to be you ran out of non-volatile memory. There is far less non-volatile available, and variable declarations default to it, so unless you specifically declared it volatile, it's a good bet it ate up all your non-volatile memory. If you need it to be persistent, write it to a file and read it on startup so it can remain volatile.

a_riot42

One thing I found with Netlinx that wasn't intuitive is that nested structures are sloooowwww. When I was writing a Netlinx Sonos module, I couldn't figure out why simply looping through a nested structure and retrieving data to send to a touch panel took so long. I converted it all to arrays, and was shocked at how the same code ran so much faster. I think that memory accesses to structures mustn't be optimized, and read/writes are much slower. So now I only use structures without nesting or very small nested structures. It would be great to be able to use them as objects of arbitrary size but it doesn't seem to work well in implementation.
Paul

fogled@mizzou

a_riot42 wrote: »

One thing I found with Netlinx that wasn't intuitive is that nested structures are sloooowwww.

So far, I haven't run into this. I'm walking an x[20].y[10].data nested structure (as big as it's going to get) with nested FOR loops , pulling data as needed, and it does the whole thing in a fraction of a second. I'll keep my eyes peeled for slowness, though.

I totally forgot I could telnet in and do a Show Mem though; thanks for the reminder. ;-)

jimmyw

if you are running through a LARGE struct array, the difference becomes clear, especially when looping many times for touchscreen feedback, its a factor of 4 times difference at least.

fogled@mizzou

jimmyw wrote: »

if you are running through a LARGE struct array...

Mine *WAS* large, until I put it on a diet. But I don't actually "walk" the structures anywhere but startup when I need to open all the IP ports the system makes connections to. Otherwise, everything is a direct call to a specific cell as needed. I'm not even using the message trapping part of my structure right now, it was just on the slate for implementation.

The "Program" part where all the feeback is, though... the system walks that all the time anyway, although the calls in the feedback are mostly only 2 levels deep, occasionally 3 (audio channel mute state). There should be a slowdown there, shouldn't there?

However, I doubt I'll see any noticeable slowdown, just because my structure isn't really *that* big.

Still interesting to know structures are much slower than arrays though. Not surprising, just... interesting.

fogled@mizzou

A side note here, I actually *can* detect that the nested structure is way slower than non-nested structures or arrays. But the only reason I could is because of a very poor command execution order in a block of code. So far, the slowness doesn't actually impact the system operation (once I fix my execution order).

catalino

I don't know how the AMX compiler handles structures, but if you have an array of these structures, it allocates ram for ALL fields, even if you are only using one or two values from each array. Also, if you have buttons/channels/etc declared in your system, the system allocates RAM to keep track of ALL channels up to the number you are declaring. So if you have
DEFINE_VARIABLE
INTEGER testChan = 255

and then

Button_Event[dvtpTestChan, testChan]
then your netlinx system allocates memory for all channels 1-254. in addition to 255.

Is there a reason for you to have such a monster struct? I generally like to have smaller structs, as they make the lookup process faster, and I can break them out into separate AXI files to make maintenance easier.

fogled@mizzou

The reason for the monster struct is because I'm a database developer too, and that's more or less the way the data ends up being normalized. I use one controller for multiple rooms - I've put an arbitrary limit of 20 just to be on the safe side - and each room has multiple devices, and each device can have multiple settings that need tracked. It ends up being a 3-4 level deep data structure, but it makes complete sense for the rest of my programming and has made it that much easier to do the rest of my programming.

I'm aware of the waste in memory allocation, it is what it is. I buy the high-end controllers with more memory so I can be lazy an more consistent about my system design and programming. The only reason I killed this one was because I had a defined a huge error message storage system to get around the fact that the controllers lose all their console messages on power loss.

But I don't really need that so much, because I've got the controllers reporting back to a central database system which stores the messages (kind of a homegrown RMS but without all the unnecessary crap). So, I just scaled back the size of that part, and got within the memory bounds.

RANT ON

At this point, I've started co-developing another system to replace AMX controllers, built on all open-source stuff: Postgre for data storage, Python for core logic processing, and PHP to serve UI's. Then, IP-serial devices for the crap that doesn't have IP-based communications. The design of my structures in AMX mirrors the structure of my real database I'm developing.

Quite frankly, I'm pissed at the way AMX has re-structured their technical support. I'm not a dealer, but I deal with dealer-sized deployments and equipment counts, and where a quick phone call used to get me what I needed, now it's a huge hassle to get anything out of AMX. As a result, I'm accelerating my drive to move away from vendor-specific solutions. I put myself at the mercy of AMX, and they cut me at the knees for it. I won't put up with that any longer than I have to.

And don't get me started on the AMX University bit. I tried to get one of my co-workers registered in that system; we kept filling out and submitting the forms, and we never hear anything back. I buy tons of AMX stuff, but it never gets credited to my account. From my perspective, it's nothing but a hokey PR disaster that's left a very bad taste in my mouth. I gave up on it.

I still need AMX right now, because I don't have anything better to take it's place yet, but AMX's days are numbered on my campus. In the meantime, I'll push the AMX controllers to their limit.

RANT OFF

a_riot42

fogled@mizzou wrote: »

RANT ON
Quite frankly, I'm pissed at the way AMX has re-structured their technical support.
RANT OFF

If you don't like AMX support, I doubt you will like PHP, Postgre, and Python support much either.
Paul

fogled@mizzou

The difference is, I know I'm on my own, but everything is out in the open and I can find out whatever I need. I've dealt with these open source tools for years; I can provide my own support for them more quickly and effectively than AMX can provide support for their proprietary products. Resolving AMX issues used to be easier and faster than researching open source, but for me, since the changes, not anymore.

The difference is, not being at any one person's or vendor's mercy. Our campus has dozens of very competent programmers who know these open source tools even better than I do. Our campus has only one certified AMX programmer - me - and only two in our entire town, and I can't even directly get help from AMX anymore.

The difference is, the open source software runs great on redundant virtual servers on our production floor, whose cost has already been completely amortized within our organization. AMX hardware very reliable, but is poor at redundancy, and quite expensive in comparison.

Sorry to sound so pissy about it. AMX can make their decisions, and I can make mine. Customer service is extremely important to me, and my perception is that AMX changed formerly great customer service into currently horrible customer service. I realize not everyone has experienced the same; I'm just one of those outliers that fell through the cracks. But as one of the few that has been cut off from good support, I have to react to the situation I find myself in. My reaction is to walk away in as orderly a fashion as I can.

jimmyw

fogled@mizzou wrote: »

At this point, I've started co-developing another system to replace AMX controllers, built on all open-source stuff: Postgre for data storage, Python for core logic processing, and PHP to serve UI's. Then, IP-serial devices for the crap that doesn't have IP-based communications. The design of my structures in AMX mirrors the structure of my real database I'm developing.

Let me say, AMX for the most part I am happy with, but I do understand your plight. Let me make a suggestion though.
I would either use
http://www.jr.com/startech/pe/TCH_PEX8S952/
or
http://www.jr.com/startech/pe/TCH_ICUSB23216F/
I can confirm both work quite well, instead of ip > serial, sometimes having an issue when the tcp/ip stack is waiting for more data to form a filled packet instead of sending with no delay.
I made a .net based system recently, but, decided to not pull the trigger. FWIW, look at the patent document and wireshark captures and watch the panel > master comms and you can make your panels play nice with you new system.

fogled@mizzou

I've been using MOXA N-port units, and haven't had any problems with them so far. From a deployment perspective, those seem a lot more modular than having to install a card in a PC, which changes fairly frequently and isn't anywhere near what I would consider control-system level reliable. Most room PC's on campus are targeted to be replaced by thin clients anyway, which don't have card slots at all.

In the best of worlds, I wouldn't even need to use serial. I don't drive serial to projectors anymore; all new installations are direct IP connections. All the higher-end audio DSP's are IP-based too. For really simple rooms where we just run audio control through the projector, I just drop in a touchpanel and a projector, run a network cable to each, enable them in the central controller, and I'm done. It's simple and relatively inexpensive, compared to controllers in each room and serial everywhere.

For the slightly more complex rooms, I drop a MOXA in to drive serial to a switch and a low-end audio control device. It's mostly just video switches, low-end audio controllers, and lighting systems that still require serial. Still modular and less expensive than dedicated controllers.

For the big rooms, I still buck up for a dedicated controller, but I'm getting really close to being able to drive even the biggest systems off a controller on my production floor with only IP-based connections. That's part of what my "monster" structure is all about. I'm hoping this summer will be the last time I wind up installing a dedicated controller for a single room of any size.

The idea with the server-based open source system is to completely ditch proprietary hardware, including touchpanels. Instead, I'll put the controls on smartphones and tablets for user convenience, and fixed touchscreen kiosk units in rooms. That's all commodity hardware, not vendor locked, and far less expensive.

This is what AMX is competing against. I can understand they need to cut costs, but cutting customer service to people like me is really just shooting themselves in the foot. My open-source system would still just be an idea if I hadn't been kicked out of being able to call their tech support directly, and take a month-long round-trip through a 3rd-party vendor just to get a piece of equipment repaired. But now, I've already got the project underway, and a simple test system in operation.

This is the last post I'm going to make on this thread. I've spoken my piece. Thanks for listening and understanding.