Serial Parsing and Queing in DEFINE_PROGRAM

TurnipTruck

Greetings,

Since Netlinx it has been convetion to keep DEFINE_PROGRAM reserved for feedback. However, I've been able to write more efficent parsing and queing routines for serial communications by placing them in the DEFINE_PROGRAM section.

Is anyone else parsing or queing this way? Just curious.

Thanks!

JohnMichnr

Yeah - I have started to do that - fill up a buffer and then in DEFINE_PROGRAM I will set teh test statements to look for data in the buffer then call a routine to parse it. I also put outgoing buffers down there if I need to buffer send data by repsonces back from the device.

GSLogic

Why wouldn't you just look for the data in the DATA_EVENT string return, this way Netlinx won't have the extra step of always looking in the DEFINE_PROGRAM for data?

TurnipTruck

Given how a DATA_EVENT may be triggered before a complete string has arrived in the buffer, I find it more simple and efficient to allow the mainline of the program to check for a CR or whatever makes a complete string each time around. This way, as soon as the complete string has arrived, it can be dealt with, rather than using waits to allow the string to arrive and possibly missing the next message.

My parsing commands are only executed if the buffer has a length greater than 0.

I use the DATA_EVENT for a CANCEL_WAIT / WAIT arrangement for an incomplete or corrupted message to timeout and dump the buffer.

This works especially well with UPB!

vining

You can do the same using an "IF" or a "WHILE" statement. If your looking for something that ends with "CR" just take up to the first "CR" and parse it or store it. If you don't get a "CR" don't do anything with the string until you do. If your device sends longs strings with several "CR"s get the "CR"s with a WHILE loop and pasre them or store them. If strings are in complete and a "CR" isn't sent leave it in the buffer until the remaining string arrives or at least a usable portion including the next "CR" which will create another string DATA_EVENT.

Even if your device sends one character at a time, one second apart just let it sit in the buffer until you get what you want. Just because you get data into a buffer doesn't mean you have to use it right away.

TurnipTruck

Don't WHILE statements in an event hold up the processing of events until complete?

GSLogic

TurnipTruck wrote:

Given how a DATA_EVENT may be triggered before a complete string has arrived in the buffer, I find it more simple and efficient to allow the mainline of the program to check for a CR
My parsing commands are only executed if the buffer has a length greater than 0.

The mainline is checking your IF statement around 1000 times a second, compared to the below code which will only check when there is an incoming STRING.

  STRING:
  {
	IF(FIND_STRING(sBuffer,"13",1))
	{
	  SEND_STRING 0, "'PARSE:<-- ', sBuffer";
	  PARSE_DATA();
	}
  }

The buffer will keep loading until it see's a CR and then trigger the PARSE_DATA(). Also, when using a BUFFER for large strings, create it using the CREAT_BUFFER in the DEFINE_START. I've done test where I loaded around [20000] in a VARIABLE (BUFFER) and it wouldn't get all the data sent, but when I used the CREAT_BUFFER it did receive all the data. I was told it is design to receive larger amounts of incoming data faster, because it ranks above a normal variable in the structure of the compliled program.

vining

The "WHILE" is used when a string is received with multiple "CR"'s for example. Use a"WHILE" loop to removed the string up to the first "CR" and do something with it. The "While" loop continues to the next "CR" so on until there are none left and it exits regardless of data remaining in the buffer that is as long as it doesn't still contain a "CR". It allows you to handle string data a little at a time depending on your needs.

So even if a string is incomplete the "WHILE" tests for the condition which in this case is FIND_STRING(Buffer,"'CR'",1) if there isn't one it exits. If there are "CR"s you get everything up to and including the first one (if that's what you want to do) and the "WHILE" loops goes around again and you get the next "CR". When the buffer no longer has a "CR" which is often the case when you get incomplete string the "WHILE" exits leaving the incomplete data in the buffer and you exit your DATA_EVENT. The remaining string comes in and initiates another DATA_EVENT and the "WHILE" repeats what it did before picking up where it left off. The new data is appended to the existing (remaining) data left in the buffer. That is of course if your buffer is large enough to handle the left over data and the new (delayed) data.

Your confusing a "WHILE" with a "WAIT_UNTIL".

Chip Moody

TurnipTruck wrote:

Since Netlinx it has been convetion to keep DEFINE_PROGRAM reserved for feedback.

I thought convention was to put feedback in a TIMELINE_EVENT...?

TurnipTruck wrote:

I've been able to write more efficent parsing and queing routines for serial communications by placing them in the DEFINE_PROGRAM section.

Is anyone else parsing or queing this way?

No - as already mentioned, I believe the most efficient is the DATA_EVENT -> STRING: -> WHILE(FIND()) method...

- Chip

Hedberg

See TN 616

TN 616 has an example of what a string handler might look like in the wild. In the example there, the buffer size is 100 which may not be large enough. Particularly for Clear1 stuff (which, it appears, the example writer may have had in mind) . My experience is that a couple of XAP 800s can spew out vast quantities of bytes, particularly when recalling presets.

Marc Scheibein

One thing to keep in mind with the WHILE is that it will never end automatically in NetLinx!!

In AXcess:
- the WHILE end automatically after 0.5 secs.
- the MEDIUM_WHILE doesn't end and doesn't track message buffers (PUSH/RELEASE, etc) for changes (!)
- the LONG_WHILE desn't end but checks message buffer for changes

In NetLinx:
any of the WHILE types doesn't end automatically!!! :eek: So it's "easy" to program and endless loop, and in most cases, you then need the magic "Runmode Disable" switch.

So what I mostly do is to program a counter into the WHILE and if the limit is reached, I manually set the WHILE condition to false.

The following is just a part of a code:

DATA_EVENT[dvMXP]
{
	STRING:
	{
		LOCAL_VAR CHAR sResponseLine[1000]
		LOCAL_VAR _sResponseStruct ResponseStruct
		LOCAL_VAR nWhileCOunt
		WHILE(FIND_STRING(sMXPBuffer,"$0D,$0A",1))
		{
			nWhileCount++
			IF(nWhileCOunt>500) // Endlos-WHILE verhindern
			{
				sMXPBuffer = ''
				nWhileCount = 0
			}

So if the check to "$0D,$0A" takes too long in any way, I reset the Buffer so the WHILE fails.

vining

I would probably add a 2nd condition to the "while" loop to actually exit the loop. Usually whith the Find_String you won't have a problem as long as you remember to remove what your trying to find so that condition doesn't exist any more.

There are lots of ways to ensure you don't get into an endless loop but with a While (find_string(buffer,......,1) followed by a remove_string or get_buffer_string etc you usually don't get into trouble.

DATA_EVENT[dvMXP]
{
	STRING:
	{
		LOCAL_VAR CHAR sResponseLine[1000]
		LOCAL_VAR _sResponseStruct ResponseStruct
		LOCAL_VAR nWhileCOunt
               
		WHILE(FIND_STRING(sMXPBuffer,"$0D,$0A",1) && nWhileCOunt < 501 )
		{
			nWhileCount++
			IF(nWhileCOunt>500) // Endlos-WHILE verhindern
			{
				sMXPBuffer = ''
				
			}
                {
                 nWhileCount = 0

TurnipTruck

Points well taken. However, I am still trying to find the down side to parsing the buffer in DEFINE_PROGRAM. An IF statement, checking for buffer length, does not seem to be a very processor intensive operation. If my understanding is correct, that extra step in DEFINE_PROGRAM will not slow down the events processing.

DHawthorne

TurnipTruck wrote:

Points well taken. However, I am still trying to find the down side to parsing the buffer in DEFINE_PROGRAM. An IF statement, checking for buffer length, does not seem to be a very processor intensive operation. If my understanding is correct, that extra step in DEFINE_PROGRAM will not slow down the events processing.

Often, that is all that I have in DEFINE_PROGRAM ... a few tests to see if my buffer needs processing, and fire the routines to do that if needed. I have found it to be more reliable and faster than using a STRING handler for serial devices. Perhaps I just haven't done my STRING handlers properly, but it seems to me sometimes I don't get a trigger when there is valid data in the buffer.

vining

Quite frankly the idea of parsing buffers from Define_Program never occurred to me. I guess doing a simple "IF (find_string(buff,"'Whatever'",1)" to check for particular data or any data for that matter each and every pass of the mainline wouldn't be so bad. As far as the Data_Event (String_Event) not triggering is something I haven't seen but it's also not something I've looked for. I believe the string_event itself is basically doing what you're doing, a system function called in the mainline that initiates a string_event function for our use.

GSLogic

TurnipTruck wrote:

I am still trying to find the down side to parsing the buffer in DEFINE_PROGRAM. An IF statement, checking for buffer length, does not seem to be a very processor intensive operation. If my understanding is correct, that extra step in DEFINE_PROGRAM will not slow down the events processing.

If you're only checking a couple devices in a single system, I know it's not a problem, but if you're talkng about a system that has many sub masters communicating back to a main master, you have to be aware of the processing power.

My weather module uses - IF(FIND_STRING(sBuffer,"'/HTML'",1)) to start the parsing and the buffer can receive up to [25000] before it triggers the parse function. Devices that need commands timed before sending - Lighting, Security, Thermostats, etc. I use the TIMELINE set to the proper time for each device.

DATA_EVENT[dvDEVICE]					
(* START TIMELINE ON INCOMING COMMAND *)
{
  STRING:
  {	
     IF(TIMELINE_ACTIVE(T_LINE) == 0) //IF KILLED CREATE TIMELINE TO QUE CMDS
     TIMELINE_CREATE(T_LINE,T_CMD_TIMER,1,TIMELINE_ABSOLUTE,TIMELINE_REPEAT);	
  }
}


TIMELINE_EVENT[T_LINE]				
(* TIMELINE WILL REPEAT EVERY XX TO SEND COMMANDS IN THE QUE *)
{
  IF(nBUSY == 0 AND LENGTH_STRING(cBuffer))
     fnSEND_device_CMD();
}


(* AT THE END IN THE PARSE() FUNCTION ADD *)
IF(LENGTH_STRING(cBuffer) == 0)
{
     TIMELINE_KILL(T_LINE);    //KILL COMMAND TIMER
}
ELSE
{
     nBUSY = 0;
}

Joe Hebert

Concatenation limitation

GSLogic wrote:

Also, when using a BUFFER for large strings, create it using the CREAT_BUFFER in the DEFINE_START. I've done test where I loaded around [20000] in a VARIABLE (BUFFER) and it wouldn't get all the data sent, but when I used the CREAT_BUFFER it did receive all the data. I was told it is design to receive larger amounts of incoming data faster, because it ranks above a normal variable in the structure of the compliled program.

Yes, you need to use CREATE_BUFFER for large incoming strings; however, it has nothing to do with the speed of the data coming in. The data can drip in 1 byte a second and it still won?t work without CREATE_BUFFER. It has to do with a string concatenation limitation. Even though we can work with 64K chunks of data, for some reason we can?t concatenate anything over 15999 bytes. Yet CREATE_BUFFER can. I don?t know why the limitation exists but here is code to prove it does:

DEFINE_DEVICE

dvTP	= 10001:1:0

DEFINE_VARIABLE

CHAR cArray1[15000]
CHAR cArray2[2000]
CHAR cBuffer[65000]

DEFINE_START

SET_LENGTH_ARRAY(cArray1,15000)
SET_LENGTH_ARRAY(cArray2,2000)

DEFINE_EVENT

BUTTON_EVENT[dvTP,1] {

   PUSH: {
      //we should get 17000 bytes with this concat but we only get 15999
      cBuffer = "cArray1,cArray2"
      SEND_STRING 0, "'Length of cBuffer = ',ITOA(LENGTH_ARRAY(cBuffer))"
   }
}

Here is the output when button 1 is pushed:
Line 1 :: Length of cBuffer = 15999 - 01:06:27

videoninja

Hi All,

Historically speaking, I have always parsed incoming data within the data_event string: handler. That is, until I needed to parse a massive amount of data coming back from several XAP800's. As I was watching the data come in, I would see the string: handler activate, but for some reason, much of my feedback and variables were not set properly. I moved the entire section into an if statement running in mainline and everything worked much more smoothly. If I have lots of data to parse, I'll usually pass it off to a function from the if statement. The only thing I do inside the string: handler at this point is append data.text to the existing rx_queue.

When sending strings, I generally queue them up and add a delimiter to the end of the tx data. I then pass this to a system call running in mainline which send the strings. The system call accepts the queue, the device to send to, and a delay to wait between transmissions. Not to different that running transmission from a timeline.

Of course, very few of my projects involve any master to master communication, so doing things this way seems to work well without any noticable downside.

TurnipTruck

I have become skilled enough as a programmer to write modules. Lately, and from here on out, I will write a comm module for every external piece of equipment I am controlling. Within that module. I put both the incoming and outgoing data buffers in DEFINE_PROGRAM. I have not yet found a way to do the buffering as an event without WHILEs and WAITs, both of which I don't like to use.

As the previous poster stated, I use a $0D as a delimiter in my outgoing queue and REMOVE_STRING looking for the $0D, until the queue is empty. The only WAIT that I will use is one that starts every time something is found in the buffer and canceled when it is sent. I use it in case I get junk in the buffer for some reason to allow it to timeout and be cleared.

setholle

String parsing in define_program

I personally never do parse serial information in define program. I find it very old fashioned.
Also on very large systems which I deal with often, this would be horrible. Your processor would be so slow it would be ridicolous. I have seen it happen.
If you run a counter in mainline you can track how different it is. Why not just parse everything in the string event for the device. It is much cleaner.
Also for that matter, on very large systems I use multiple staggered timelines for feedback instead of putting feedback in define program. When you have hundreds to thousands of buttons, having them in define program is nuts.
But even when I do small systems, i always use timelines it is just more efficient. For example use a timeline that only checks feedback every 10th or half a second. This is plenty fast for feedback.
If you check processor usage. YOu can see the difference.
It just seems better to always do it that way.
It will work doing in define_program. But why? It is using old access convention.

setholle

define_program part 2

Oh and just make a very efficient small function for sending out strings, so it is only accessed when you call the function.
Have trigger fire then call function.

Chip Moody

I'm with Seth - there are extremely few, if any - reasons to even have a DEFINE_PROGRAM section in a Netlinx program. Chances are if you do have code there, it can be made to run better outside of it.

- Chip

yuri

using timelines is certainly a good way of doing feedback.
Why would you want to check the state of a projector every 0.1 seconds?

TurnipTruck

Chip Moody wrote:

I'm with Seth - there are extremely few, if any - reasons to even have a DEFINE_PROGRAM section in a Netlinx program. Chances are if you do have code there, it can be made to run better outside of it.

- Chip

This is what I am learning. I got starterd in the Axcess days. DEFINE_PROGRAM was all there was!

Chip Moody

Me too... I just thought to look at my Tech II certificate - from the first AMX training program I went to - it's dated May 13th, 1997. Damn!

- Chip

TurnipTruck wrote:

I got starterd in the Axcess days. DEFINE_PROGRAM was all there was!

JasonS

I wrote a template for my serial comm modules that I have been refining for a long time, and am very pleased with how it works and its effeciency. I have a transmit buffer that is emptied from Define_Program based on length of the buffer and wether I am waiting for a response. Responses are parsed from the String Data_Event via a function. This function contains a While Loop that will handle multiple responses in the Rx buffer every time the Data Event happens. After a response is parsed the a flag is set that allows the Define_program section to send the next command in the TX Buffer. There is also a timeout that will re-enable the Tx buffer if no response is received. I have found this method to be much faster than an arbitrary delay or timeline for emptying the Tx Buffer.

As far as putting waits in the String event for data over 64 bytes or incomplete messages, it is completely unnecessary because the data_event will retrigger on additional bytes or the rest of an uncompleted message.

I read alot of posts on this forum where people are concerned about the processing capacity of Netlinx Masters, in my experience the amount of processing that can be done on a single master using effecient code would suprise you. (NO JAVA!)