COMBINE_DEVICE vs. DEV array
DHawthorne
Posts: 4,584
In my never ending search for ways to cut back on uneccessary message queue loading, I have made the following observations. I have not really been able to take the time to properly test my conclusions, and I am wondering if anyone else has made the same observations, or can contribute an alternate reason for my conclusions.
Observed, but not quite verified:
1) SEND_LEVELS and CHANNEL feedback are always updated on virtual devices, but not on actual devices unless they actually change. So if I have a TIMELINE (or in mainline, not that I would ever do such a thing) feedback segment that says:
SEND_LEVEL vdvVirtual, 1, iVolume ;
[vdvVirtual, iMute] = (iVolume <= 5) ;
... both those updates go out every pass. Yet if I replace vdvVirtual with a real device, they only go out when iVolume changes.
2) Channel changes sent to a virtual update the virtual, but if an actual device is combined with it, and goes on line after the channel change, it does not always reflect the change even though the virtual does. I've noticed this mostly in the converse: a group of buttons combined with a virtual panel, when the real panel is rebooted, are ON, even thought the virtual channels are not. To sync them with the virtual, an ON, then an OFF needs to be sent to the virtual channels. This is by no means consistent - I had a combined virtual with a good 60 buttons on it, and about 20 of them were ON when the real panel rebooted. You would think it would be all or none. I had to abandon the approach, it was too wonky in a user environment.
3) Channel changes to a DEV array update when individual devices in the array go on and off line, without regard to when the change really occurred - as if a table of the channel states were kept in the master, and the array element devices were automatically kept up-to-date. This, by allrights, should happen with a combined virtual as well, but doesn't seem to, at least not consistently.
4) Channel changes (and levels) to a combined virtual seem to attempt to go out to the real device if it's online or not (I am real iffy on this one, I thought I saw this happening, but it's tough to verify in the conditions where I think I saw it).
These observations have led me to the conclusion that it is always better to use a DEV array without combining DEVs, or the array itself, to a virtual. Since most of these observations have taken place in rushed conditions and a high-pressure environment, I'd be curious if anyone else has noticed the same inconsistencies between combined virtuals and uncombined DEV arrays, or if my observations are flawed (like I saw something screwy once, jumped to a conclusion, and now have that idea wrongly in my head). I'd also like to hear if I have somehow missed something in my conclusion that combined DEVs = bad; the only time I am using them now is when I can do the combines dynamically and on demand (in which case, they are pretty much inactive most of the time). Right now I am just not seeing an up side to combining with a virtual on a continuing basis - they just don't seem to work quite right.
All of this points back to a large system (7 masters so far, and 22 panels; three area-centered controllers and four local systems that need to talk to the central ones) I have that was at one time bogging down into slow motion every other day due to a overloaded message queue. I have never been able to point to a single cause for this, but by cleaning up feedback methods, I have got it to where it will now go 5-6 weeks without a hang. I need to make it go forever, and I am looking at a few cases where I still have shared virtuals. The problem seems to be this: a device goes off line while one of the masters is sending it something. An offline event does not occur immediately, and the master resends messages, stacking them up in the queue. If it was a particulary busy set of updates, the queue becomes overloaded, impacting system performance. The offline event finally occurs, but the damage is done - the overloaded message queue has caused other devices to fall off line, starting another cycle of this. Sometimes it recovers on its own, but most I have to reboot the central master, it all depends on the timing. By cutting way back on the frquency of channel and level updates, I have cut back on this nasty cycle considerably. Using static IPs for all my devices helped a great deal as well - Modero panels don't seem to get along with all DHCP servers very well. I consider it a bug in the firmware that does not deal with disconnects cleanly, but that is something I have to live with until its fixed, and if I can eliminate the circumstances that trigger the bug, then I will be content with that. By converting combined virtuals to real devices in an array, I think I may be able to cut back a lot of unnecessary message traffic; a diagnostics session seems to point at my virtuals being the the source of my heaviest traffic.
Observed, but not quite verified:
1) SEND_LEVELS and CHANNEL feedback are always updated on virtual devices, but not on actual devices unless they actually change. So if I have a TIMELINE (or in mainline, not that I would ever do such a thing) feedback segment that says:
SEND_LEVEL vdvVirtual, 1, iVolume ;
[vdvVirtual, iMute] = (iVolume <= 5) ;
... both those updates go out every pass. Yet if I replace vdvVirtual with a real device, they only go out when iVolume changes.
2) Channel changes sent to a virtual update the virtual, but if an actual device is combined with it, and goes on line after the channel change, it does not always reflect the change even though the virtual does. I've noticed this mostly in the converse: a group of buttons combined with a virtual panel, when the real panel is rebooted, are ON, even thought the virtual channels are not. To sync them with the virtual, an ON, then an OFF needs to be sent to the virtual channels. This is by no means consistent - I had a combined virtual with a good 60 buttons on it, and about 20 of them were ON when the real panel rebooted. You would think it would be all or none. I had to abandon the approach, it was too wonky in a user environment.
3) Channel changes to a DEV array update when individual devices in the array go on and off line, without regard to when the change really occurred - as if a table of the channel states were kept in the master, and the array element devices were automatically kept up-to-date. This, by allrights, should happen with a combined virtual as well, but doesn't seem to, at least not consistently.
4) Channel changes (and levels) to a combined virtual seem to attempt to go out to the real device if it's online or not (I am real iffy on this one, I thought I saw this happening, but it's tough to verify in the conditions where I think I saw it).
These observations have led me to the conclusion that it is always better to use a DEV array without combining DEVs, or the array itself, to a virtual. Since most of these observations have taken place in rushed conditions and a high-pressure environment, I'd be curious if anyone else has noticed the same inconsistencies between combined virtuals and uncombined DEV arrays, or if my observations are flawed (like I saw something screwy once, jumped to a conclusion, and now have that idea wrongly in my head). I'd also like to hear if I have somehow missed something in my conclusion that combined DEVs = bad; the only time I am using them now is when I can do the combines dynamically and on demand (in which case, they are pretty much inactive most of the time). Right now I am just not seeing an up side to combining with a virtual on a continuing basis - they just don't seem to work quite right.
All of this points back to a large system (7 masters so far, and 22 panels; three area-centered controllers and four local systems that need to talk to the central ones) I have that was at one time bogging down into slow motion every other day due to a overloaded message queue. I have never been able to point to a single cause for this, but by cleaning up feedback methods, I have got it to where it will now go 5-6 weeks without a hang. I need to make it go forever, and I am looking at a few cases where I still have shared virtuals. The problem seems to be this: a device goes off line while one of the masters is sending it something. An offline event does not occur immediately, and the master resends messages, stacking them up in the queue. If it was a particulary busy set of updates, the queue becomes overloaded, impacting system performance. The offline event finally occurs, but the damage is done - the overloaded message queue has caused other devices to fall off line, starting another cycle of this. Sometimes it recovers on its own, but most I have to reboot the central master, it all depends on the timing. By cutting way back on the frquency of channel and level updates, I have cut back on this nasty cycle considerably. Using static IPs for all my devices helped a great deal as well - Modero panels don't seem to get along with all DHCP servers very well. I consider it a bug in the firmware that does not deal with disconnects cleanly, but that is something I have to live with until its fixed, and if I can eliminate the circumstances that trigger the bug, then I will be content with that. By converting combined virtuals to real devices in an array, I think I may be able to cut back a lot of unnecessary message traffic; a diagnostics session seems to point at my virtuals being the the source of my heaviest traffic.
0
Comments
1) "those updates go out every pass". Not sure what you mean by this. You would not get a Button event for Mute every pass, you'd get it only if the Mute state has changed.
For a real device on the network, the update goes out only if the value assigned is different from what Netlinx thinks the device value is, hence the problem on reboot of a panel.
What I have found is that the only reliable way of mamaging a panel coming online is to refresh it, that is, maintain a table of its state and resend all the feedback slowly after it is online. On ONLINE, wait a couple of seconds, send it 30 or so feedback states, wait a second, etc... Sending more at a time results in "missed" values. Sending too soon results in crashes. Basically I have my own panel management layer above the simple (or broken) one provided by Netlinx.
Fred
I am partial to DEV arrays for touch panels and only use virtuals for either G3 web panels or in the creation of modules.
When combining virtual panels to physical panels I always use:
DEFINE_COMBINE(vdvTP,dvTP) just after the DEFINE_DEVICE section
instead of COMBINE_DEVICES(vdvTP,dvTP) in DEFINE_START or elsewhere.
I believe the proper way to manage COMBINE_DEVICES is in the Online Data_Event for the physical device. Your first must uncombine and then combine for the feedback to sync.
DEFINE_DEVICE dvTP = 10001:1:0 vdvTP = 33001:1:0 // DEFINE_COMBINE(vdvTP,dvTP) // Optional combining method DATA_EVENT[dvTP] { ONLINE: { UNCOMBINE_DEVICES(vdvTP,dvTP) COMBINE_DEVICES(vdvTP,dvTP) } }How do you do your combining, Dave?
When I first programmed Netlinx UNCOMBINE_DEVICES didn't work at all so I delved and discovered using DEV sets which are far better.
Using DEV sets you can create one for each combination mode and you can still address the devices individually.
Virtual devices are only good for modules or as a touch panel which never goes offline. Keeping a virtual touch panel in a touch panel DEV set is good as a reference for refreshing all channels when panels come online.
We can read the internal message queue values, but can we flush the message queue from code?
I've found that creating a timeline with less than 100msec repeating a function (for emptying a serial stack) quickly fills the event queue and locks the system up unless the queue is increased.
I like to put feedback within a timeline which only triggers every half second to relieve the master of any work.
I also believe in making every processor a master thereby sharing the processing and causing the least problems when one goes down.
I'm waffling on now. I think I need to go to a meeting or better yet a conference.
I loop ever SEND_COMMAND and if feedback should be updated to a panel viewing that section, I send it. This way I never use the mainline for any feedback. This is really the same thing that COMBINE does except you have control over what each panel is receiving.
If you think about it, 95% of any given day a system is ideal, so why send commands.
I'd like to hear what others do!
The project that got me thinking about this is an ongoing one, and it has been ongoing for over three years. What I do now has little bearing on how I did some of the early segments of this jobs, but some of those methods are so tied into the structure of this monster program, it would be difficult to change them now, and more than a little time consuming.
I used two methods in this project. It was originally written for G3 panels (which have since gone by the wayside), so I had to conserve channel adresses, and the logical way seemed to be to create a virtual for every controlled device, then combine the real panels to those virtuals on-the-fly as needed and uncombine them when done. That part actually always worked pretty well, once I cobbled a routine to take care of it. The fact that I was uncombining them when done took them off the message handler, and they cause no issues.
For other portions of the program there were devices that that were always available to all panels at all times (like a weather station, a caller ID display, etc.). I thought it would be best to have a single "aux" virtual that was combined to the same port on all the panels (or, in the original, a single device number). So I made an array of these ports, and used DEFINE_COMBINE to attach them to the virtual, so they were always combined. This combine seemed to cause me the most grief, and I have since done away with it altogether. I just reference the array, no combining involved.
Nowadays, I avoid combining devices like the plague. I prefer to use a DEV array. On those occasions where a combine does make sense, I am using COMBINE_DEVICE, and, as suggested, in an ONLINE event. Persistent combines seem to just have issues that are not worth the trouble to deal with when there are other methods that work as well.
Also, as mentioned, I am slowly developing my own feedback layer for my designs. It's hardly a fully modular solution that can simply be plugged into any panel; more like a set of standard policies. For one thing, I don't see much need to update a touch panel more than once a second, so all my feedback goes into a one-second repeating timeline. All my button texts are being tracked in code, so the SEND_COMMAND to change a text feedback only goes out if the text has changed, with provisions to force an update when a page is first displayed so it reads correctly. I haven't gone as far as to track channel states or levels, as it seems if I use a DEV array with real devices, it's not necessary, but as far as I am concerned, the jury is out on that still. I am also generally tracking in code whether a page is being displayed at all, and only sending updates when the page is being used ... but this is sometimes problematic as well, tracking page flips and button presses to see if a page is displayed isn't always reliable if the panel goes offline. But I have not implemented all of these policies in this one old project; I am introducing them incrementally as time permits, as to not bankrupt my time resources on it. But that's what inspired this post - wondering if all of my policies were really necessary, or if I may have gottne a wrong idea somewhere along the line. I'd hate the effort to become wasted, and the great thing about a community like this is the ability to bounce such ideas off each other and refine them.
Dave 've been reading this post and have a question for all. May be slightly off topic, but...
I've got a combinable room - 2 rooms into one type of thing. I've been using a dev array of virtual panels, then combining and un-combining the real panels to the virtual panels as needed by room configuration.
The program alway references the array.
I borrowed the layout from somebody (thanks Brian). However, I notice that I am having a problem with channel feedback not responding on the touch panels. Only some of the time. If I take the panel offline then online the feedback updates.
Could that be because I am uncombining then re-combining the real to the virtual?
It only happens on some of the buttons, some of the time.
Any ideas?
If the panel is not too complex, perhaps you can force an update by refreshing the virtual when a combine takes place, so the real device gets updated at the same time.
That seems strange.
I was not using the SET_VIRTUAL_CHANNEL_COUNT command to set my channels for the virtual touch panels. Read this on another thread.
I'll have to try that the next time onsite.
The way I "fixed" it is to have the panel always go to a "wait" page, out of which the user can't go, and that mentions, well, "please wait" or something. I catch the ONLINE event from the program, do the updates as needed, and send the panel to the "real" main or current page.
Fred