Checksums in String Returns
Jeff
Posts: 374
I've got a plasma that requires checksums at the end of every string sent to it. I did a little research and found a checksum function written by one of the multitude of other Jeffs on the forum, and it is working fine. However, I've got a minor question about the return data from the Plasma.
Each return string from the plasma has a checksum at the end of it. I can't decide the most effective way to deal with that data. For the record, I can make this all work, but I have some free time right now and I'm trying to make the code elegant, pretty, and ideal.
1) I can use data.text in the data_event|string: handler. I can do something like
Unfortunately, I dont know what happens here if the string handler fires before the entire string shows up. Is this a problem?
2) I can use a buffer in the same string: handler. instead of being able to use a right_string to identify the checksum itself, I'll have to check every character against the previous string, and if that isn't the checksum, move one more character down the line. Haven't written code for this but I'm sure I could.
Here, I don't know what happens if I get two strings at once and the string handler only fires once. then my buffer still has stuff in it, but nothing is telling my code to process it.
3) I can use a buffer and put the same code I would be developing for option 2 in define_program. This would fire as long as my buffer isn't empty.
Downside here, I like to keep code out of define_program if at all possible. Maybe I could put this in a timeline, but how often do I really need to do this?
Maybe I'm overthinking all of this, and I should just stick with option 1. I'm still trying to learn the ideal way to do things. Still, I appreciate your help.
J
Each return string from the plasma has a checksum at the end of it. I can't decide the most effective way to deal with that data. For the record, I can make this all work, but I have some free time right now and I'm trying to make the code elegant, pretty, and ideal.
1) I can use data.text in the data_event|string: handler. I can do something like
if (calcchecksum(left_string(data.text,length_string(data.text)-1))=right_string(data.text,1))
Unfortunately, I dont know what happens here if the string handler fires before the entire string shows up. Is this a problem?
2) I can use a buffer in the same string: handler. instead of being able to use a right_string to identify the checksum itself, I'll have to check every character against the previous string, and if that isn't the checksum, move one more character down the line. Haven't written code for this but I'm sure I could.
Here, I don't know what happens if I get two strings at once and the string handler only fires once. then my buffer still has stuff in it, but nothing is telling my code to process it.
3) I can use a buffer and put the same code I would be developing for option 2 in define_program. This would fire as long as my buffer isn't empty.
Downside here, I like to keep code out of define_program if at all possible. Maybe I could put this in a timeline, but how often do I really need to do this?
Maybe I'm overthinking all of this, and I should just stick with option 1. I'm still trying to learn the ideal way to do things. Still, I appreciate your help.
J
0
Comments
Use CREATE_BUFFER and parse it yourself.
Me, I don't bother with checksums coming from a device unless the device is prone to errors. Most modern devices are not. I just ignore data I can't recognize.
I had a situation a while back with something where occasionally I would get two strings at once, and the data_event string handler would only fire once. It would parse that bit, and get to the end, and leave data in the buffer. How do I make it fire again if there is more in the buffer?
If there is no delimiter, you can just check the length of your buffer, and test if it's the size of or greater than the size of the expected data, then use LEFT_STRING to strip it out. My least favorite is the type that use variable lengths and a start character, but you can test for that as well, it's just not as straightforward (and varies widely with the protocol).
Read this TechNote : Buffer Parsing in Netlinx
Very helpful
I had never even considered using a while loop there. I'm not entirely sure I've ever used a while loop . . . ever. That should work perfectly
J
Make sure the while loop will end someday or get you master stuck!!
Yes, that's very important. When testing a string for a delimiter, it's pretty safe, since it's either there or not, and if your while ends when it's not, you are good. If you fail, however, to remove that delimiter when you parse a packet - well, you have created an endless loop. If the structure of the while loop is such that you cannot be sure if it's ever going to exit, put a timer on it. Don't let it run more than a few seconds (and even that is a lot, since nothing else will happen meanwhile).
Worst case scenario, I'm only actually handling three different responses from the plasma, I could just put
then put a switch|case inside it to handle each one, but I'd like to actually write something that will check for the checksum, not check for checksums that I've calculated.
J
I've got a for loop written that checks for a delimiter
The downside of this is, this only fires once. I could take this code (slightly modified to use global vars) and put it in my define_program, and probably be quite happy with it.
What I'd like to do, though, is find a way to combine this with a while loop such that it will run multiple times in the data_event, checking more than once for a delimiter once it has found one. I don't want to tie the while loop to the length of the string, just in case I get junk back from Plasma that doesn't end in a checksum, the while loop would get stuck . . . .
The other question I have, is what happens to my for loop once I've performed remove_string inside it . . . . the loop itself is dependant upon the length of the string but I will have changed the length of the string.
Thoughts?
J
Here's what I've got. Everything from the last post is the same, so I'm just going to recreate the data_event.
Basically I have a variable that says I'm parsing. The while loop runs as long as that variable is on. That variable gets turned off if the Buffer is ever empty OR the we've checked every character to see if its a checksum and failed. One of those two should always happen, so the loop shouldn't get stuck. I can add something like a timeline to cancel the while loop after 3 seconds and clear the buffer if necessary.
I've also got a variable that says we've found a checksum that stops the for loop. This way it restarts. If we don't restart the for loop from scratch, then a second checksum would never get found because it wouldn't be a checksum for the entirety of the message.
My reasoning for trying to do all of this in the data event is that I don't want it running 100 times a second when I'm just not dealing with that much data from the Plasma.
So do you guys think this will work, or am I just incredibly overcomplicating this? I'm beginning to consider DHawthorne's suggestion to ignore the checksum, but this also seemed like a good challenge to practice my programming skills on.
J
You are right, and this better for your career.
May be for this model of device, you won't need to process the CheckSum, but who knows, one day you are going to need it, so better get ready
would you please tell us the Model number so i can get the protocol , and practice together
I Love String Parsing
J
Try NS2 / Diagnostics / Emulate a Device
Another suggestion: you appear to be using some form of Hungarian Notation. If so, I suggest you use the form bFound for a flag value rather than nFound.
The eponymous Hungarian cosmonaut intended that the prefix indicate the usage of the variable, not its nominal type. Here are the prefixes that I use:
char sString[]
integer nInteger
long lLong
slong slSignedLong
sinteger siSignedInteger
structure tType (* The name of the type, not an instance *)
device dDevice
devlev dlDevLev
integer bBoolean
integer tpButtonNumber
integer pPopupNumber
integer mModuleNumber
I don't bother with a prefix for an instance of a custom struct unless it is commonly used.
I also generally indicate arrays with a plural variable name: sNames[]
Local variables like so: sMyName[]
Arguments passed into a routine: sArgName[]
Some folk use all caps for constants but I find them ugly and I don't find I need to distinguish constants from variables.
It all helps with code comprehension.
I've solved a few problems here, and found one that I think is fairly hard to get around.
The last piece of code I posted had
and was wrong. The +1 makes it not fire each time, it should just be
This original error came from a misunderstanding of how the loop structure worked (and partly not thinking things entirely through.
Once you fix that error, you can send a single string that ends with a checksum on it to the master, and it will parse it correctly. You can also send two strings that both end in checksums to the master, and it will parse both of them correctly.
However, I'm having two problems.
Problem number 1 I think is user error. When I'm using "Emulate a Device" I'm putting the following (including quotes) into the box
"$DF,$80,$60,$47,$01,$01,$08,$20,$81,$01,$60,$01,$00,$03"
I've checked the String Expressions checkbox.
In this instance, any hex character that appears after the $00 value just shows up as $00. The end of my cPlasmaBuffer, when I watch it in debug, shows up as $01,$00,$00, instead of $01,$00,$03. The actual protocol for my plasma includes several $00 characters . . . what am I doing wrong?
The second problem I'm running into, is what happens if there's a bad character or incomplete string sent to the master from the Plasma. If you get a single bad character in the buffer, then you've got serious issues, because none of the checksums will calculate.
I've got two possible solutions to that.
1) After some amount of time, clear the buffer. I don't really like this because it involves a timeline, or a wait, and something about that really doesn't seem like what I'm trying to do. Also, I'm going to lose data that way, although I figure you have to agree to occasionally lose data if you want to be able to get rid of junk.
2) Switch to using data.text instead of cPlasmaBuffer. Now my data.text will parse correctly every time, even if two strings get lumped together in data.text (which I've seen happen). However, if a string gets somehow broken in two different events (which I've not seen happen), I'd lose it completely.
I'm leaning towards #2. I've solved my original big problem with data.text, which was that a second string would always get lost. Can anyone think of a reason I shouldn't do this, and if so, a better solution?
Thanks for humoring me in this random quest of mine
J
Here is some food for thought on the transmit side. Don't worry about the variable naming convention, at the time I may have spent less than thirty minutes on it.
For debugging incoming serial data, this can be used to print incoming data.
Post up what you have for your incoming data parser. I will make some constructive comments on it.
Its going to take me some time to go through all of that. I haven't actually gotten to that point in my code yet, almost the entirety of my code is posted earlier in this thread. I'll pull your code into NS2 a little later today and go through it to see what all you're doing.
J
This is turning into something much more involved than I originally anticipated. However, I think I'm getting close. I don't have the perfect solution I once envisioned, but I guess you can't always have perfection. I do have one statement that I think is correct, but I'd love to have someone prove wrong.
There is no way to parse strings with checksums for delimiters in such a way that you are guaranteed never to miss a string.
I think the recovery process from having an incomplete string or bad data is inherently going to cause you to lose a string or two. I see this as incredibly unlikely to happen, but the possibility is out there. However, I believe that I have reached a point such that you should only ever lose one string, if that were to happen.
Here's what I've got.
Basically, I'm using the NetLinx buffer to capture the string from the plasma. If a string comes in, I turn on a flag that says nParsing, and I run a while loop for as long as nParsing is true. If at any point in time the buffer is empty, I turn nParsing off, and the while loop stops. If at any point in time I have checked every character in the string to see if it is a checksum, and none of them are, I turn nParsing off, and the while loop stops. There should be no instance in which one of those two events does not eventually come true. either I find a checksum and remove it from the string and it clears out the buffer, stopping the loop, or I get to the end and dont find a checksum, stopping the loop.
If I get a single full string from the device, it will find a checksum, remove it from the buffer, and the loop stops
If I get half of a string from the device, it will not find a checksum, and stop parsing. When the rest comes in, the loop will start again, and I will find the checksum the second time around.
If I get a string and a half from the device, It'll find the checksum, remove it from the buffer, keep going, not find a checksum, and stop. I'll wait for the second half, and find the checksum after it comes in.
If I get two whole strings from the device, it'll find the checksum, remove it from the buffer, keep going, find the checksum, remove it from the buffer, and stop.
Here's the tricky part. If I get a single character somewhere, then it throws off all of the math that makes the checksums work, and I'll never find another checksum, which means I wont get data again. Thats no good. However, I don't want to clear the buffer whenever I get through and don't find a checksum, in case I've just got half of a string and am waiting for the rest.
My solution, at least for the moment, was to check the protocol, and find that the longest possible string (including checksum) for my device, which was 7 characters long. I then set up an if statement after my for loop that says if we didn't find a checksum, and the buffer is longer than the longest string I could possibly get, that its time to clear the buffer and start over. Here is where I will likely lose some sort of response. On the other hand, if I'm getting bad data, or random characters, there's absolutely no way to avoid losing at least one response, so I think the fact that I may have it down to losing just one is a good thing.
Here is the only problem I have right now with all of this logic. There is a line in the code that says
This loop that I'm creating will keep running along the plasma buffer even after I've found a checksum and removed part of the buffer. I'd rather this not happen. I'd rather stop the for loop once a checksum is found, let the while loop get to the end, and start over because nParsing is still true (if indeed there are additional characters left in the buffer). I have a flag (nFound) that tells me whether or not I found something, so my first thought was that I could just change the for loop to read like this
In this case, the loop should stop if nLoop becomes longer than the buffer (which it already does), or if nFound becomes true. However, thats not what happens. somehow once i added in the "or !nFound", the first part stopped working and nLoop quickly got up into the thousands. Am I forgetting something basic from Programmer I here? it compiles perfectly and runs, the loop just doesn't stop.
Aside from that, can anyone think of something else I'm missing here? It seems like I've handled every eventuality except for that pesky for loop. If there's more that I'm missing, I'd love to hear it. Otherwise, I think this code could be lifted and reused repeatedly on anything that returns strings with checksums, as long as you changed the cutoff limit in the final if statement.
Also, I'm still curious as to why my buffer goes crazy whenever one of the hex characters it gets is $00. Every character after that always turns to $00, and I don't know why. Is this a limitation of Emulating a Device, or will this come back to bite me when I actually hook this up to my Plasma?
J
Every protocol with (or without) a checksum that I have encountered has either a fixed string length or a byte with a data count to tell you how long the string will be.
Which plasma is this?
There is never a guarantee that you receive a complete command, nor get them all. Something along the lines of what is the sound of a tree falling in a forest when you're not there?
If your code really did run slower than the device's "transmit inter-packet delay", you would lose a packet. So unless you are sorting a big list or something you should be safe.
Something to keep in mind some devices will respond to commands in bursts. This is no problem if you have a common end delimiter, but a bugger if the protocol doesn't. In my experiences with NEC product (15+ years) I have not seen any devices respond with broken up strings. So in the case of NEC it is safe to parse the data.text variable. Just keep in mind other non-NEC devices do behave differently, and you cannot rely on using data.text alone.
This hasn't been tested but should provide some ideas, and may even work.
Keep in mind when you are using a non-ASCII protocol through an AMX com port you need to check for send_string escape sequences, and break up accordingly.
This TV is an NEC PX-42VM5HA, the protocol is located here
The Power on command response (3FH 60H 80H 4EH 00H CKS) is 6 characters long. The Volume command response (7FH 60H 80H 7FH 02H DATA00 DATA01 CKS) is 8 characters long. The difference is any command with only one parameter (power on or off) is just 6 characters, anything with a variable (like which input to switch to) has more. From looking over things again, I'm noticed that I need to change my constant to 8 or I'll never get volume data. Ooops
I have no idea what you just said. Sorry about that. Any chance you could reword your answer so it is clearer to me? I'm sending hex strings through Emulate a Device . . . . . and if I send $36,$4F,$47,$00,$80,$47,$03 I get $36,$4F,$47,$00,$00,$00,$00.
And while in this particular instance I am working with an NEC plasma, my goal in all of this is to be able to write a universal checksum parsing routine that will work on anything. I think I'm there, with two minor exceptions. That goofy "or" statement I want to put in my for loop, which isn't working, and the fact that I need a constant for the longest possible correct string length from the plasma.
Again thanks for all of your help, I'm still playing with all of this . . . .
J
This doesn't sound right. Are you checking the hardware output back into your PC @ 9600,8O1?
What I was referring to prior was the RS232 escape sequences:
Excerpt from NISeries.WebConsole-ProgrammingGuide.pdf
If any of the 3 character combinations below are found anywhere within a SEND_STRING program instruction, they will be treated as a command and not the literal characters.
So what you have to do if search for any occurrence, if found, break the string apart and issue multiple sends.
It's a good exercise you're putting yourself through, just keep in mind checksum vary. I depends on the Engineer that implemented the protocol. Some may not even implement, some will with a way to defeat it, some do it to match all their other obtuse protocols, etc.
I can tell you have a good attitude and you don't get frustrated easily. Keep it up!
I wouldn't think I'm hitting hardware yet. I'm not connected to the plasma in any way. I'm using "Emulate a Device" within NetLinx studio to send random character strings to the master from 5001:1:0. I'm watching my buffer in debug mode when I send those strings, and anything after the $00 turns to $00.
For example, the correct response to a power on command would be 3FH 60H 80H 4EH 00H CKS. If I were to calculate this checksum and then send the entire string, to my master, the buffer would read 3F,60,80,4E,0,0. The checksum would get converted to a 0.
I haven't actually hooked any of this up to my Plasma yet because the Plasma is mounted on a wall in a 4th floor conference room and the NI-700 I ordered for the purpose has not yet arrived. I have an NI-3100 in my office hooked up to some stuff that I'm using for testing, but I can't easily rip it out, take it to the room, and test with the plasma, which is why I'm doing all of this theoretical work ahead of time.
J
If you have IP access to the unit, you might consider using IP to connect to the upstairs port in your program. This would allow you to test against the hardware, and would avoid the emulate issue you're seeing.
The 5th byte is a count of the data bytes to follow, except when there is only one, when it is that data byte. Great work there, NEC engineer!
The responses do not tell you what they are responding to. You have to know what was the last command sent in order to interpret them. This means that you will also know how long they will be.
It looks to me like when there is only one, the 5th byte is $00. Does that mean it doesn't actually send that byte?
It looks to me like they do tell you what they're responding to . . . . 3FH 60H 80H is the prefix for a command about power, 4EH says its powered on, 4FH says its powered off. Where are you getting the fact that it just sends an acknowledgement?
J
My reading of the protocol is that when there is only one byte of data, it is in the 5th byte.
I observed that many of the responses are identical in the first 4 bytes.