String cat bug? What?s magical about $63F - $640?

Joe Hebert

Can someone please confirm this for me? Compile the following program as Netlinx:

(** start of program *)

DEFINE_VARIABLE
CHAR myarray[17000]
INTEGER x

DEFINE_START
FOR (x=1; x<=17000; x++) { myarray= "myarray,'a'" }
x = LENGTH_ARRAY(myarray)

(*** end of program ***)

After the program is loaded and running, start debug and watch the variables myarray and x. You will see that the current length of myarray is 15999 and not the expected 17000. This is confirmed by the value of variable x. If you inspect myarray[17000] you will see that ?a? is indeed there but for some reason the length fails to report correctly after string concatenation of more than 15999 bytes ($63F) ** Edit - After further review I realize that the previous sentence is not true. The data did not get set after 15999 bytes. **

What?s even wilder is if you replace the line:
FOR (x=1; x<=17000; x++) { myarray= "myarray,'a'" }

with this (which should perform exactly the same way):
FOR (x=1; x<=17000; x++) { myarray= "myarray,$61" }

You will see after you recompile and run that the length grows by one to 16000 ($640)

I ran across this problem while writing a program that grabs a web page and saves it to disc. As the web page arrives in chunks I concatenate the data and when the transfer is complete I save as a file. However, the file was always truncated in the same spot ? 15999 bytes.

NetLinx Studio V2.2 Build 2.2.0.83
NetLinx Compiler Build 2.2.0.108

Can anyone explain this? I searched Tech Notes but didn?t find anything.

TIA,
Joe

** EDIT **

Spire_Jeff

I can confirm that the same thing happens on an NI-3000 and an NXI with an NXC-ME. My gut feeling is that there is a limited amount of memory that the processor has direct access to. (I'm drawing a blank on proper terminology at the moment) And the way the code you submitted seems to work is the the processor puts contents of myarray into a temporary memory location, then adds the additional character and then writes the result to the myarray variable. The limit for this temporary memory location is probably near 16000 bytes (maybe 16384?) and this location might even hold currently executing code which could explain the difference between hex and ascii????. Now I have absolutely no documentation to support this claim, but from what I think I am recalling some of assembly language programming classes many many many years ago this sounds likes a feasible explaination. Now for a work around:

FOR (x=1; x<=17000; x++)
{
myarray[x]='a'
SET_LENGTH_ARRAY(myarray,x)
}

This seems to work and it seemed to work a LOT quicker. It should be rather easy to write a chunk of code to insert the text at the end of the array and adjust the size of the array. I'll try to give an example, but you will probably have to adjust it to fit your implementation:

DEFINE_DEVICE
dvDATA = 33001:1:0

DEFINE_VARIABLE
VOLATILE CHAR sMYARRAY[17000]

DATA_EVENT[dvDATA]
{
String:
{
STACK_VAR INTEGER x

WHILE(LENGTH_ARRAY(DATA.TEXT) && x<MAX_LENGTH_ARRAY(sMYARRAY))
{
x = LENGTH_ARRAY(sMYARRAY)
sMYARRAY[x+1]=GET_BUFFER_CHAR(DATA.TEXT)
SET_LENGTH_ARRAY(sMYARRAY,x+1)
}

send_command dvDATA,"' < ',ITOA(LENGTH_ARRAY(sMYARRAY)),' - ',RIGHT_STRING(sMYARRAY,5),' > '"

}
}



That should let you add everything you need to the array, then after you write your file, just set the array length to 0.

Hope this helps you get around the problem.

Jeff

Chip Moody

Joe, I think you've tripped across an undocumented memory limit imposed in NetLinx. While it's not what we would normally consider 16k of memory, that 15999 limit >is< just shy of decimal 16k - a number someone might pick if they had to set a limit.

Even if you declare the variable as VOLATILE, that limit is still imposed.

When we've got megabytes of volatile memory available, it's a shame that we can't create a memory structure that large.

- Chip

Spire_Jeff

Originally posted by Chip Moody


When we've got megabytes of volatile memory available, it's a shame that we can't create a memory structure that large.

- Chip

I don't think there is that limit to the array, just to the ability to deal with that much data all at once. I was able to get the array to fill up to 17000 with the above code. I am just guessing that there is either a hardware or performance issue behind the anomoly. Another thing I thought of is maybe the limit is due to some sort of processor cache size.... don't a lot of processors have something right around 16k of internal cache?

Ok, I'll stop speculating now and wait for an engineer or AMX employee to post the real reason.

Jeff

Joe Hebert

Jeff ?n? Chip - Thanks for confirming and posting.

I don?t think it?s a memory limitation nor the ability to deal with that much data at once since the data in the code I posted does get concatenated in the array properly. It?s just the length that doesn?t get set correctly. ** Edit - I was in error. The concatenation does not work. ** According to documentation we should be able to play around in 64K arrays.

I can write a work around for this but I?m hoping this gets fixed (if it?s indeed broken) so I don?t have to. It kind of defeats the whole purpose of LENGTH_ARRAY. If this is a bug it may be the result of another problem or may be causing another problem somewhere down the line that needs to be looked at.

What I?m intrigued by is if you use $61 instead of ?a? you can squeeze one more bite in. I wonder if that?s any type of clue.

Sleepless in Saint Charles?.

Joe

** EDIT **

kkettrey

In Netlinx, the act of string concatination is limited to 16K Bytes. This was confirmed by AMX support/engineering. Why? I don't know.

I found this out the hard way by trying to concatinate data from a website by using the string handler in my IP device's DATA_EVENT. (cBuffer="cBuffer,DATA.TEXT"). I had to change my code to utilize the old-school "create_buffer" keyword, since it enables concatination of incoming string up to 64K bytes (and is much faster since it does not have to do it's work through interpreted code).

So, one possible work-around is to use a virtual device in conjunction with a buffer/variable assigned to that virtual device with the "create_buffer" keyword. Send any character or strings that you need concatinated TO the virtual device. Then, the varible (buffer) that you assign to that virtual device will get that data appended to it. Yea, this might be slower, and your Output and Input lights may flash like crazy, but it should serve the purpose. An extra bonus is that you will have an event triggered any time more information is added to the variable.

Let me know if you need some sample code.

Kyle

Charles Law

Hi,

In the day of Axcent, there were time out in while loop(while, medium while, long while) in order to prevent endless loop.

I am not sure if it happen the same in for loop. Just a while guess!

Cheers,

Charles

Joe Hebert

Hi Kyle,

In Netlinx, the act of string concatination is limited to 16K Bytes. This was confirmed by AMX support/engineering. Why? I don't know. Well, that kind of bytes. I wonder why I couldn?t find a Tech Note mentioning that.

I found this out the hard way by trying to concatinate data from a website by using the string handler in my IP device's DATA_EVENT. (cBuffer="cBuffer,DATA.TEXT"). That?s exactly what I?m doing and how I found out. I was using the same code I had written before to grab some XML files from the web but never anything as large as what I need this time. I had to change my code to utilize the old-school "create_buffer" keyword, since it enables concatination of incoming string up to 64K bytes (and is much faster since it does not have to do it's work through interpreted code).

So, one possible work-around is to use a virtual device in conjunction with a buffer/variable assigned to that virtual device with the "create_buffer" keyword. Send any character or strings that you need concatinated TO the virtual device. Then, the varible (buffer) that you assign to that virtual device will get that data appended to it. Very cool, thanks! A vdvConcatenator was easy to implement and it works like a charm. The only downside is a slight timing issue. The STRING handler in my dvIP DATA_EVENT sends incoming data to vdvConcatenator. I use the OFFLINE handler of dvIP as a signal that all web data has arrived and that?s where I call my save to disc routine. If I don?t put a WAIT 1 before I call my save routine, the last chunk of incoming data doesn?t get appended in time to the buffer attached to vdvConcatenator and doesn?t get saved to disc. Yea, this might be slower, and your Output and Input lights may flash like crazy, but it should serve the purpose. An extra bonus is that you will have an event triggered any time more information is added to the variable.

Let me know if you need some sample code. Thanks again, I?m good now.

Kyle

NOTE: I need to back pedal a bit. I mentioned twice earlier that it was just the length of the array that was reporting in error after 15999 bytes and that the actual concatenation worked in the sample code I originally posted. I was wrong. Either my eyes or the debugger are playing tricks on me but when I checked this again I can see that the data did NOT get appended. My bad.

Still Sleepless in Saint Charles?

Joe

kkettrey

Oh, well since you are actually needing this for that reason (aquiring data from an IP device), you should just assign the buffer directly to your IP device. (You don't need the virtual device to do that). Then you will be able to imediately parse the data when the IP device goes offline.

The virtual device example that I gave you would be handy if you were having to assemple data from within your program (since you were doing exactly that in your FOR Loop example). Another good example might be building a large HTML file with a massive tables and tags that needs to be writen to the DOC.

Joe Hebert

Doh! Apparently I couldn?t see the forest through the trees. That?s about an easy a work around as there is. I did a CREATE_BUFFER with my original buffer and dvIP, commented out my one line concatenation, and that was that.

Thanks again, Kyle. I think this one has been put to bed.

Joe