Data.text or Buffer Problems

sridley

I am writing some code for a Media Server and am controling over IP. I am sending a request for a status update from the server every 3 seconds and I am capturing this into a buffer. My problem is that I have written the contents of this buffer out to Send_String 0 so I can check that this code is working. (I'm new to this so I am just trying to check every stage at this point). What I see in the telnet session however is an new string every 3 seconds but not the entire string. It is always missing the string.

The buffer is declared as a size of 1000 but the string is only coming through at about 120 charactors.

Any ideas what can be causing this?

DHawthorne

DATA.TEXT is not entirely reliable in some cases (mainly with physical devices), because it uses some internal timing parameters (which are not documented) to determine when the whole string has arrived. If there is any delay in sending it, the STRING event is triggered as complete, even when it's not.

Use CREATE_BUFFER in DEFINE_START, and test the buffer then for length or a delimiter in DEFINE_PROGRAM.

sridley

I have used Create_Buffer (cBuffer) in define start but I have a String Event for the device with cBuffer = "Data.Text".

If I send send_String 0, Data.Text then I see the whole string in the telnet session?

Marc Scheibein

sridley wrote:

I have used Create_Buffer (cBuffer) in define start but I have a String Event for the device with cBuffer = "Data.Text".

CREATE_BUFFER adds any incoming data of a device in cBuffer automatically (with some firmware internal advantages).
An operation like cBuffer = DATA.TEXT will overwrite cBuffer.
So depending on which instruction is operated first, you may loose your previous data, or you may get the new incoming data doubled in cBuffer.
To add new stuff manually, you may do it like sMyBuffer = "sMyBuffer,NewData".

mpullin

If there's a null character ($00) then nothing to the right of that character will show up in Notifications. It's a known bug in NetLinx Studio. If you must keep track of the strings coming back from the device, consider writing them to a text file instead.

sridley

Create_Buffer does not appear to be automatically getting the data from the device. Only if manually tell to to = data.text does anything appear in it.

AMXJeff

create_buffer does automatically get the data from the device, no need to use the equal operator. If you do not want to use create_buffer, follow Marc Scheibein advise, and use contactination. sMyBuffer = "sMyBuffer,Data.Text".

// proper create_buffer syntax.
CREATE_BUFFER dvDevice, cBuffer

sridley wrote:

Create_Buffer does not appear to be automatically getting the data from the device. Only if manually tell to to = data.text does anything appear in it.

sridley

Thanks, I found the error in my code and have now got the string automatically in the buffer.

Pretty neat now it's working!

Joe Hebert

AMXJeff wrote:

If you do not want to use create_buffer, follow Marc Scheibein advise, and use contactination. sMyBuffer = "sMyBuffer,Data.Text".

That is sound advice as long as the result of the concatenation is less than 16000 bytes. I?ve mentioned this before but it may be worth repeating. If you need concatenate anything more than 15999 bytes then you are forced to use CREATE_BUFFER. I?ve never seen this documented anywhere and I?ve never been able to get a reason why but I do know that this limitation exists. Here?s the code to prove it.

DEFINE_DEVICE

dvTP	= 10001:1:0

DEFINE_VARIABLE

CHAR cArray1[15000]
CHAR cArray2[2000]
CHAR cBuffer[65000]

DEFINE_START

SET_LENGTH_ARRAY(cArray1,15000)
SET_LENGTH_ARRAY(cArray2,2000)

DEFINE_EVENT

BUTTON_EVENT[dvTP,1] {

   PUSH: {
      //we should get 17000 bytes with this concatenation but we only get 15999
      cBuffer = "cArray1,cArray2"
      SEND_STRING 0, "'Length of cBuffer = ',ITOA(LENGTH_ARRAY(cBuffer))"
   }
}

Here is the output when button 1 is pushed:
Line 1 :: Length of cBuffer = 15999 - 10:12:27

AMXJeff

Alot of the string function have a limitation 16000 bytes. it is just where it ends because of the variable decleration. I guess they could create them bigger but, its not a high priority. Even as you suggest, there are ways around it. You can actually use VARIABLE_TO_STRING to go beyond 16000.

(**************************************)
(* Call Name: ConcatString 		  		  *)
(* Function: ConcatsString > 16K   	  *)
(* Params:   sConcatFinal, sConcatNew *)
(* Return:   n/a 				 					    *)
(**************************************)
DEFINE_FUNCTION ConcatString(CHAR sConcatFinal[],CHAR sConcatNew[])
{
		STACK_VAR CHAR sPieces[3];
		STACK_VAR CHAR sBuffer[MAX_DB_ENCODED_DATA_SIZE];
		STACK_VAR LONG lPos, lOldPos;
		
		lpos = 1;
		
		VARIABLE_TO_STRING(sConcatFinal,sBuffer,lPos);
		sPieces[1] = sBuffer[lPos-3];
		sPieces[2] = sBuffer[lPos-2];
		sPieces[3] = sBuffer[lPos-1];
		
		lOldPos = lpos;
		lpos = lpos - 3;
		
		VARIABLE_TO_STRING(sConcatNew,sBuffer,lPos);
		
		sBuffer[lOldPos-3] = sPieces[1];
		sBuffer[lOldPos-2] = sPieces[2];
		sBuffer[lOldPos-1] = sPieces[3];

		GET_BUFFER_STRING(sBuffer,3)
		
		sConcatFinal = sBuffer;
}	

Joe Hebert wrote:

That is sound advice as long as the result of the concatenation is less than 16000 bytes. I’ve mentioned this before but it may be worth repeating. If you need concatenate anything more than 15999 bytes then you are forced to use CREATE_BUFFER. I’ve never seen this documented anywhere and I’ve never been able to get a reason why but I do know that this limitation exists. Here’s the code to prove it.

DEFINE_DEVICE

dvTP	= 10001:1:0

DEFINE_VARIABLE

CHAR cArray1[15000]
CHAR cArray2[2000]
CHAR cBuffer[65000]

DEFINE_START

SET_LENGTH_ARRAY(cArray1,15000)
SET_LENGTH_ARRAY(cArray2,2000)

DEFINE_EVENT

BUTTON_EVENT[dvTP,1] {

   PUSH: {
      //we should get 17000 bytes with this concatenation but we only get 15999
      cBuffer = "cArray1,cArray2"
      SEND_STRING 0, "'Length of cBuffer = ',ITOA(LENGTH_ARRAY(cBuffer))"
   }
}

Here is the output when button 1 is pushed:
Line 1 :: Length of cBuffer = 15999 - 10:12:27

Joe Hebert

Very cool, Jeff. Works like a charm. Thanks for adding this arrow to my quiver.

AMXJeff wrote:

Alot of the string function have a limitation 16000 bytes. it is just where it ends because of the variable decleration. I guess they could create them bigger but, its not a high priority. Even as you suggest, there are ways around it. You can actually use VARIABLE_TO_STRING to go beyond 16000.

(**************************************)
(* Call Name: ConcatString 		  		  *)
(* Function: ConcatsString > 16K   	  *)
(* Params:   sConcatFinal, sConcatNew *)
(* Return:   n/a 				 					    *)
(**************************************)
DEFINE_FUNCTION ConcatString(CHAR sConcatFinal[],CHAR sConcatNew[])
{
		STACK_VAR CHAR sPieces[3];
		STACK_VAR CHAR sBuffer[MAX_DB_ENCODED_DATA_SIZE];
		STACK_VAR LONG lPos, lOldPos;
		
		lpos = 1;
		
		VARIABLE_TO_STRING(sConcatFinal,sBuffer,lPos);
		sPieces[1] = sBuffer[lPos-3];
		sPieces[2] = sBuffer[lPos-2];
		sPieces[3] = sBuffer[lPos-1];
		
		lOldPos = lpos;
		lpos = lpos - 3;
		
		VARIABLE_TO_STRING(sConcatNew,sBuffer,lPos);
		
		sBuffer[lOldPos-3] = sPieces[1];
		sBuffer[lOldPos-2] = sPieces[2];
		sBuffer[lOldPos-1] = sPieces[3];

		GET_BUFFER_STRING(sBuffer,3)
		
		sConcatFinal = sBuffer;
}	

a_riot42

Joe Hebert wrote: »

Very cool, Jeff. Works like a charm. Thanks for adding this arrow to my quiver.

I was using this routine but it appears to have a bug in it if you get a buffer past 65 536. After this limit is surpassed, the routine starts leaving binary data at the beginning of the buffer. I don't really understand how it works, but it appears that whatever binary data is being added to the buffer isn't being removed with the final get_buffer_string(sBuffer,3) as I would assume it is using more than 3 bytes to store that data. Anyone seen this bug or know how to fix it?
Thanks,
Paul

vining

This tech note might help:
http://www.amx.com/techsupport/techNote.asp?id=886

a_riot42

vining wrote: »

This tech note might help:
http://www.amx.com/techsupport/techNote.asp?id=886

Thanks I'll take a look. However, the function posted seems to work fine so I would rather just modify it to work past 65535 bytes. There isn't much point in a function designed to work around a 16000 byte limit only to break at 65535. I'm a little worried that a byte for byte for loop would steal the processor for too long with big files, and I don't want to write to the flash drive constantly either. There should be a built in concat function really.
Paul

Auser

AMXJeff wrote: »

DEFINE_FUNCTION ConcatString(CHAR sConcatFinal[],CHAR sConcatNew[])
{
  [...]
}	

Here's a revised version which should be more memory efficient and is virtually unlimited in the size of the resulting string.

The 63999 byte chunks come from a documented 64kB limitation of the variable_to_string stream marshaller. The documentation didn't specify what it classed as a kB, so the actual value can probably be increased to (2^16)-1, but I couldn't be orsed testing it.

// Name   : ==== String_Append ====
// Purpose: Append two string literals
// Params : 
// Returns: 
// Notes  : Works around a limitation in the maximum return size of
//          two string literals being appended.
//
define_function char String_Append(char _cString[], char _cTextToAppend[])
{
  stack_var    char      _cDestroyedData[3]
  stack_var    long      _lOriginalInjectionPoint
  stack_var    long      _lInjectionStartPos
  
  if(length_array(_cString) + length_array(_cTextToAppend) <= max_length_array(_cString))
  {
    _lOriginalInjectionPoint = length_array(_cString) - 2
    _lInjectionStartPos = _lOriginalInjectionPoint
    _cDestroyedData = right_string(_cString, 3)
    
    while(length_array(_cTextToAppend) > 63999)
    {
      variable_to_string(get_buffer_string(_cTextToAppend, 63999), _cString, _lInjectionStartPos)
      _cString[_lOriginalInjectionPoint]      = _cDestroyedData[1]
      _cString[_lOriginalInjectionPoint + 1]  = _cDestroyedData[2]
      _cString[_lOriginalInjectionPoint + 2]  = _cDestroyedData[3]
    
      _lOriginalInjectionPoint = length_array(_cString) - 2
      _lInjectionStartPos = _lOriginalInjectionPoint
      _cDestroyedData = right_string(_cString, 3)
    }
    if(length_array(_cTextToAppend))
    {
      variable_to_string(_cTextToAppend, _cString, _lInjectionStartPos)
      _cString[_lOriginalInjectionPoint]      = _cDestroyedData[1]
      _cString[_lOriginalInjectionPoint + 1]  = _cDestroyedData[2]
      _cString[_lOriginalInjectionPoint + 2]  = _cDestroyedData[3]
    }
    return TRUE
  }
  else
    return FALSE
}

I tested it by reading in a couple of ebooks from file, concatenating them, writing them back to another file and checking the result by diffing the final file with the source files to check for any anomalies. The resulting file size was around 4MB, so it should be pretty reliable.

PhreaK

Nice work. Don't suppose you'd be interested in contributing it to the NCL project string library?

Auser

I checked the NCL before I knocked that out to see if it had already been done and couldn't see a comparable function, hence why I posted it. Feel free to change it to be consistent with the rest of the libraries' content and add it in.

a_riot42

Auser wrote: »

I checked the NCL before I knocked that out to see if it had already been done and couldn't see a comparable function, hence why I posted it. Feel free to change it to be consistent with the rest of the libraries' content and add it in.

Does length_array on the concatenated string return the correct number after the function completes ?
Paul

Auser

PhreaK wrote: »

Nice work. Don't suppose you'd be interested in contributing it to the NCL project string library?

Actually, if you're going to use it for the NCL's, here is a more efficient version which negates a couple of issues that exist in the first version:

// Name   : ==== AppendString ====
// Purpose: Appends a string to a string variable
// Params : 
// Returns: True if successful, false otherwise
// Notes  : Works around a limitation in the maximum return size of two string
//          literals being appended. The 63999 byte chunks come from a
//          documented 64kB limitation of the variable_to_string stream
//          marshaller. The documentation didn't specify what it classed as a
//          kB, so the actual value can probably be increased to (2^16)-1, but
//          I couldn't be orsed testing it.
//
define_function char AppendString(char _cString[], char _cTextToAppend[])
{
  stack_var    long      _lSourceLength
  stack_var    long      _lResultingLength
  stack_var    char      _cPaddingLength
  stack_var    char      _cDestroyedData[3]
  stack_var    long      _lInjectionLength
  stack_var    long      _lInjectionSourceLength
  stack_var    long      _lInjectionStartPos
  
  _lSourceLength = length_array(_cString)
  _lInjectionLength = length_array(_cTextToAppend)
  _lResultingLength = _lSourceLength + _lInjectionLength
  if((_lResultingLength < 16000) && (_lResultingLength <= max_length_array(_cString)))
  {
    _cString = "_cString, _cTextToAppend"
    return TRUE
  }
  else
  {
    if(_lSourceLength < 3)
      _cPaddingLength = 3 - type_cast(_lSourceLength)
    
    if(_lResultingLength + _cPaddingLength <= max_length_array(_cString))
    {
      if(_lSourceLength)
        _cDestroyedData = right_string(_cString, 3 - _cPaddingLength)
      _lInjectionSourceLength = _lInjectionLength
      while(_lInjectionSourceLength > 63999)
      {
        _lInjectionSourceLength = _lInjectionSourceLength - 63996
        _lInjectionStartPos = _lSourceLength + _cPaddingLength + _lInjectionSourceLength - 2
        variable_to_string(mid_string(_cTextToAppend, _lInjectionSourceLength + 1, 63996), _cString, _lInjectionStartPos)
      }
      _lInjectionStartPos = _lSourceLength + _cPaddingLength - 2
      variable_to_string(left_string(_cTextToAppend, _lInjectionSourceLength), _cString, _lInjectionStartPos)
      
      if(_cPaddingLength)
      {
        for(_lInjectionStartPos = 1; _lInjectionStartPos <= 3 - _cPaddingLength; _lInjectionStartPos++)
          _cString[_lInjectionStartPos + _cPaddingLength] = _cDestroyedData[_lInjectionStartPos]
        set_length_array(_cString, _lResultingLength + _cPaddingLength)
        _cString = right_string(_cString, _lResultingLength)
      }
      else
      {
        _cString[_lSourceLength - 2]  = _cDestroyedData[1]
        _cString[_lSourceLength - 1]  = _cDestroyedData[2]
        _cString[_lSourceLength]      = _cDestroyedData[3]
        set_length_array(_cString, _lResultingLength)
      }
      return TRUE
    }
  }
  return FALSE
}

a_riot42 wrote: »

Does length_array on the concatenated string return the correct number after the function completes ?
Paul

I can't see any reason why it wouldn't, but haven't checked that explicitly. The test code I ran against it exported each result to a file and the file did not contain any garbage beyond the end of the result, so have to believe that this works correctly.

PhreaK

Awesome. I'll try to add it in when I get the chance (with credit where it's due)

Auser wrote: »

I can't see any reason why it wouldn't, but haven't checked that explicitly.

From what I've heard in the past about the way NetLinx manages arrays behind the scenes it keeps the array size in a separate chunk of memory (hence why length_array() etc are nice and fast). If this is only 2 bytes to match in with the variable serialization restrictions then there may be some interesting behaviour once you start playing with strings longer than (2^16)-1 characters.

Auser

PhreaK wrote: »

From what I've heard in the past about the way NetLinx manages arrays behind the scenes it keeps the array size in a separate chunk of memory (hence why length_array() etc are nice and fast). If this is only 2 bytes to match in with the variable serialization restrictions then there may be some interesting behaviour once you start playing with strings longer than (2^16)-1 characters.

Umm, I just got very confused after running a test. I looked at the reported length of the result and it was ~38,000 bytes short of what I expected after adding the file sizes reported by Windows for the two files which totalled ~2MB.

The reason was that the length reported by file_read() for each file was lower than that reported by Windows.

But here's where it got weird:

- The resulting file is an exact concatenation of the two source files, and the size reported by Windows is exactly the sum of both file sizes (as reported by Windows)
- The buffer containing the concatenated files (presumably) contains an exact concatenation of the two files and NetLinx reports its length as being exactly the sum of the reported file sizes when they're read in
- The byte counts in NetLinx land are short of the Windows byte count by exactly the number of lines with line breaks in the source files which are read in

What's going on I hear you ask?

Silly me forgot that the files were transferred between my laptop <-> NetLinx using an ASCII mode FTP transfer and the line endings were converted going in each direction.

So, the good news is that length_array() works perfectly above 2^16 bytes (as well as on the function above).

a_riot42

Auser wrote: »

So, the good news is that length_array() works perfectly above 2^16 bytes (as well as on the function above).

I looked it up and length_array returns a long so as long as you are concatenating files that are less than ~4GB it should return the correct value. I was more concerned about errant characters left after marshalling with large arrays. As as aside, I had forgotten that Netlinx has a double that is 64 bit. Can't think of when I would need that very often though.
Paul