Options
Program Bulletproofing....
sonny
Posts: 208
in AMX Hardware
Interested in concepts you might use in bullet proofing your code...i.e. ONERROR handling, time sync with all devices, status variable monitoring on touch panels, general processor health issues like memory usage, etc.
Thanks...Sonny
Thanks...Sonny
0
Comments
I keep an eye on overall memory (I'm aware of what's "normal" for a program), but am not doing anything more to track that.
I make very heavy use of syslog. See my code up on SouceForge; it's a module that makes it trivial to log messages to a Syslog server.
This allows me to generate messages that I can look at when my code finds unexpected values, or finds bad data, or gets into strange code paths, or whatever.
Beyond that, I test stuff. And that's about it for me.
I always Telnet in and turn on messages, then watch the master while I run it through operations. This frequently has shown me code errors that would have had very subtle effects on visible operations, yet might bog down the program or cause seemingly unrelated errors. My biggest thing personally seems to be getting array indexes mixed up and trying to reference a zero index, or one that is too high for the array size.
In diagnostics, turn on asyncronous notifications for all devices, and make sure nothing is sending undo message traffic back and forth. This causes the most insidious issues, because often things work just fine, but the master is running crippled from too much messaging. I even had a case where a runaway channel change was detrimenting a customer's entire wireless network.
Any thoughts??
...Sonny
What I have more trouble with is when a project grows on me, and I wasn't careful from the get-go in declaring variables with no greater persistency than necessary. Like declaring initialized variables as static, so they don't eat up your non-volitile memory. There is far less non-volitile memory in a NetLinx master than volitile, and the compiler does no check - in fact, it can't because there are so many different flavors of masters with different capacities. Some modules eat a ton of non-volitile memory as well - like the i!-Scheduler module...on an early processor, that can kill you. Ditto for WebControl pages.
Are you hosting the syslog server local to the AMX processor or is it running on a computer local to you that is receiving syslog messages from multiple processors?
Jeff
Jeff;s module is a Syslog client, so the assumption is there's a syslog server somewhere on the network. Most unix boxes (and Mac OS X) have a built in syslog server.
Fred
I think I was a little cloudy in what I was asking. I am wondering if anyone has experience sending all of the info to a server running in their main office. This means that all of your clients have Jeff-Aco's syslog client running and pointed to a central syslog server. If there is anyone doing this, I am wondering how much traffic is generated doing this and if there are any security concerns that may arise. I imagine that the easiest way to do this is to have your router route syslog traffic to a server (assumes static IP), but in doing this I believe everything is coming accross as clear text. Even so, is there any data that would be useful in letting a hacker gain access?
The reason I am thinking about this is having a central syslog configuration setup like this would allow you to easily monitor all clients from a single log file. Am I just trying to overcomplicate something without reason?
Jeff
Hi Jeff,
I've never thought of your application. That said, it would probably work.
Some basics on syslog servers:
1. Syslog servers come with every flavor of UNIX that I'm aware of.
2. Syslog servers are available on Windows as well. I've had quite good luck with Kiwi's Syslog Server, which can be found at: http://www.kiwisyslog.com/ This is free, but has commercial features. I did pay for it a while back, albiet only to support the author (I didn't actually need any of the advanced features).
3. Syslog, by default, works with UDP messages. These are generally transmitted across the Intenet without incident, but you'd most definitely need to expose this past your firewall. Kiwi Syslog supports TCP use of Syslog (although I don't believe this is part of the syslog "standard"); supporting this would take a trivial change to my module. TCP/IP is likely more reliable for this application (i.e. a temporary drop in connectivity would not yield lost messages). Having a retry queue is more "involved" work on the module, but still pretty easy. I'll do this if people express an interest, but please try UDP first. [Of course, the wonder of open source: You're free to make these changes yourself if you'd like; if you submit the changes to me, I'd likely roll that into my code base.]
4. Syslog has zero traffic unless you want to log a message. If you do log a message, it's a single packet.
5. The syslog specification states a format for the packet, that's it. The packet is quite small. Occational syslog packets would be totally unnoticed. Even frequent syslog messages would likely be totally insignificant in terms of bandwidth to download graphics or something.
6. In terms of security, the packets are in clear text. If you don't do something totally lame (uh, should I say "ill-advised") like send passwords in syslog messages, I don't see how Syslog could present a security concern.
7. Allowing syslog packets past your firewall allows someone to flood you with syslog packets. Left that way for an absurdly long time, I suppose this could yield a DOS (denial of service) attack if it actually fills your disk. To protect against this, you could trivially program your firewall to just allow syslog packets from your customers.
I use syslog all the time, and for me, it's quite nice to figure out when the "unexpected" comes up and resolve that in my code. I've had things "appear" okay but, after syslog messages, realized that there was actually a coding problem.
In a dealer/client scenero, I'm not sure this makes sense. Would you charge a monthly "maintenance fee" for this monitoring? If not, then would you fix a bug even if the customer never noticed it and complained?
I guess I'm just jaded by my dealer, before I started doing stuff myself. My dealer wouldn't change a single byte of code without a contract, and without a clear payment plan (i.e. what I'd pay for the work). If that's your model, then syslog probably doesn't make sense. But if your model is somehow more "quality" based rather than profit based, it might make sense.
It totally makes sense for stuff that you develop or care about on your own, or perhaps for shared code where you might want to improve it going forward (but not necessarily give the new version to the customer unless he asks - and pays).
Hope this clarifies,
-- Jeff
First, I appreciate the detailed response. Now, as far as the maintenance fee issue, I feel that being proactive in troubleshooting problems is a much wiser plan than being reactive. This being said, I would rather fix a bug before it becomes a problem and when I still have the ability to fix it on my time schedule as opposed to leaving a ticking time bomb that when it goes off I have to rearrange everything to deal with. Also, I am going to go out on a limb and say that even if the client does see the effects of a bug, they won't call if they can develop a workaround to it. This may or may not cause problems for me when they show off their system to friends (ie. Well, this function almost works, but you have to do this and this to make it work.). Our business is based heavily on word of mouth and protecting our reputation is extremely important (IMO). That doesn't mean that we don't get paid for our services, but it does occasionally mean taking responsibilty for a system we installed and charged for and ensuring that it is as we promised. I know you didn't ask for this sort of response, but I just wanted to let you know a little of my philosphy on the biz....
Thanks again for the info,
Jeff
Joe