ConcatString Function baggage?
vining
Posts: 4,368
It seems to me that using the concat string function adds bytes to the final string.
if I use it directly with data.text which means it fires a lot of times by the time I reach my end tag I get an extra 966 bytes.
LOCAL_VAR VOLATILE CHAR cTmpBuf[LUT_MAX_XML_STR];
nLUTRON_XML_RX_StrLen = (nLUTRON_XML_RX_StrLen + length_string(DATA.TEXT));
fnConcatString(cTmpBuf,DATA.TEXT);
if(find_String(DATA.TEXT,'</Project>',1))//use buffer? not likely to split across two DE's since at end but........
{
STACK_VAR INTEGER i;
STACK_VAR LONG nLen;
STACK_VAR _sLUT_XML sLUT_ClearXML;
STACK_VAR _sLUT_XML_Store sLUT_XML_Store;
nLen = length_string(cTmpBuf);
fnLUTRON_DeBug("'LUTRON_XML ###############################################:DEBUG<',ITOA(__LINE__),'>'");
fnLUTRON_DeBug("'LUTRON_XML '");
fnLUTRON_DeBug("'LUTRON_XML LENGHT_STR cTmpBuf-[ ',itoa(nLen),' ] :DEBUG<',ITOA(__LINE__),'>'");
fnLUTRON_DeBug("'LUTRON_XML '");
fnLUTRON_DeBug("'LUTRON_XML ###############################################:DEBUG<',ITOA(__LINE__),'>'");
If I change my code so I store up to a little more than 60,000 and then use that to concat with I end up with a closer match to what my in coming counter says I receive but still an extra 16 bytes.
LOCAL_VAR VOLATILE CHAR cTmpBuf[LUT_MAX_XML_STR];
LOCAL_VAR VOLATILE CHAR cTmpBuf2[65535];
LOCAL_VAR VOLATILE LONG nLenTmpBuf2;
nLUTRON_XML_RX_StrLen = (nLUTRON_XML_RX_StrLen + length_string(DATA.TEXT));
nLenTmpBuf2 = (nLenTmpBuf2 + length_string(DATA.TEXT));
cTmpBuf2 = "cTmpBuf2,DATA.TEXT";
if(nLenTmpBuf2 > 60000)
{
fnConcatString(cTmpBuf,cTmpBuf2);
cTmpBuf2 = '';
nLenTmpBuf2 = 0;
}
if(find_String(DATA.TEXT,'</Project>',1) && nLenTmpBuf2)//use buffer? not likely to split across two DE's since at end but........
{
fnConcatString(cTmpBuf,cTmpBuf2);
cTmpBuf2 = '';
nLenTmpBuf2 = 0;
}
if(find_String(DATA.TEXT,'</Project>',1))//use buffer? not likely to split across two DE's since at end but........
{
STACK_VAR INTEGER i;
STACK_VAR LONG nLen;
STACK_VAR _sLUT_XML sLUT_ClearXML;
STACK_VAR _sLUT_XML_Store sLUT_XML_Store;
nLen = length_string(cTmpBuf);
fnLUTRON_DeBug("'LUTRON_XML ###############################################:DEBUG<',ITOA(__LINE__),'>'");
fnLUTRON_DeBug("'LUTRON_XML '");
fnLUTRON_DeBug("'LUTRON_XML LENGHT_STR cTmpBuf-[ ',itoa(nLen),' ] :DEBUG<',ITOA(__LINE__),'>'");
with the first code snippet I get:
254 2019-02-25 (18:12:14):: LUTRON AXI DEBUG (1): LUTRON_XML LENGHT_STR cTmpBuf-[ 560186 ] :DEBUG<4431> Line 255 2019-02-25 (18:12:14):: LUTRON AXI DEBUG (1): LUTRON_XML Line 256 2019-02-25 (18:12:14):: LUTRON AXI DEBUG (1): LUTRON_XML ###############################################:DEBUG<4433> Line 257 2019-02-25 (18:12:14):: LUTRON AXI DEBUG (1): LUTRON_XML ###############################################:DEBUG<4532> Line 258 2019-02-25 (18:12:14):: Memory Available = 444432384 <380928> Line 259 2019-02-25 (18:12:14):: LUTRON AXI DEBUG (1): LUTRON_XML Line 260 2019-02-25 (18:12:14):: LUTRON AXI DEBUG (1): LUTRON_XML LENGTH_STRING GetXML_Str [ 322 ] Line 261 2019-02-25 (18:12:14):: LUTRON AXI DEBUG (1): LUTRON_XML Line 262 2019-02-25 (18:12:14):: LUTRON AXI DEBUG (1): LUTRON_XML XML FILE RETRIEVED COMPLETE [ 559220 ] Line 263 2019-02-25 (18:12:14):: LUTRON AXI DEBUG (1): LUTRON_XML TOTAL OF SEGMENTS STORED [ 560186 ] Line 264 2019-02-25 (18:12:14):: LUTRON AXI DEBUG (1): LUTRON_XML LENGTH_STRING cTmpBuf [ 560186 ]
and with the 2nd I get:
Line 7 2019-02-25 (19:02:20):: LUTRON AXI DEBUG (1): LUTRON_XML Line 8 2019-02-25 (19:02:20):: LUTRON AXI DEBUG (1): LUTRON_XML LENGHT_STR cTmpBuf-[ 559236 ] :DEBUG<4446> Line 9 2019-02-25 (19:02:20):: LUTRON AXI DEBUG (1): LUTRON_XML Line 10 2019-02-25 (19:02:20):: LUTRON AXI DEBUG (1): LUTRON_XML ###############################################:DEBUG<4448> Line 11 2019-02-25 (19:02:20):: LUTRON AXI DEBUG (1): LUTRON_XML ###############################################:DEBUG<4547> Line 12 2019-02-25 (19:02:20):: Memory Available = 444477440 <380928> Line 13 2019-02-25 (19:02:20):: LUTRON AXI DEBUG (1): LUTRON_XML Line 14 2019-02-25 (19:02:20):: LUTRON AXI DEBUG (1): LUTRON_XML LENGTH_STRING GetXML_Str [ 322 ] Line 15 2019-02-25 (19:02:20):: LUTRON AXI DEBUG (1): LUTRON_XML Line 16 2019-02-25 (19:02:20):: LUTRON AXI DEBUG (1): LUTRON_XML XML FILE RETRIEVED COMPLETE [ 559220 ] Line 17 2019-02-25 (19:02:20):: LUTRON AXI DEBUG (1): LUTRON_XML TOTAL OF SEGMENTS STORED [ 559236 ] Line 18 2019-02-25 (19:02:20):: LUTRON AXI DEBUG (1): LUTRON_XML LENGTH_STRING cTmpBuf [ 559236 ]
which tells me the concat function is adding something, somehow. It obviously adds more if I call it on every data.text received from my device and less if I store close to the max I can with a locally concatenated string. Has anyone ever notice using the function makes your string bigger and I'd assume if its bigger it has the potential to break parsing routine depending on where those added bytes are put.
This is the function I'm using:
DEFINE_FUNCTION fnConcatString(CHAR sConcatFinal[],CHAR sConcatNew[])
//modified from original //passing back by reference not return!!!!!
{
STACK_VAR CHAR sPieces[3];
STACK_VAR CHAR sBuffer[786432];
STACK_VAR LONG lPos;
STACK_VAR LONG lOldPos;
lpos = 1;
VARIABLE_TO_STRING(sConcatFinal,sBuffer,lPos);
sPieces[1] = sBuffer[lPos-3];
sPieces[2] = sBuffer[lPos-2];
sPieces[3] = sBuffer[lPos-1];
lOldPos = lpos;
lpos = lpos - 3;
VARIABLE_TO_STRING(sConcatNew,sBuffer,lPos);
sBuffer[lOldPos-3] = sPieces[1];
sBuffer[lOldPos-2] = sPieces[2];
sBuffer[lOldPos-1] = sPieces[3];
GET_BUFFER_STRING(sBuffer,3)
sConcatFinal = sBuffer;
}
0
Comments
My guess its the formatting being applied in variable_to_string.
A simple concatenation will not add any extra data:
sBuffer = "sBuffer,data.text";yeah but that stops working at 65535 which is why I started using the concat function. The Lutron XML I'm working with is currently 559220 bytes and could grow much bigger as more devices, scenes, etc are added. Right now I only have 100 dimmers, 24 T-Stats & 10 keypads so this file could easily double or triple in size on larger projects.
running this test with a 1500 byte string similar to a single MTU added each iteration of the for loop which runs 350 times. I should end up 525,000:
else if(nLUTRON_SendTest_Str == 97) { STACK_VAR INTEGER n; LOCAL_VAR CHAR cTmpBuf1[600000]; STACK_VAR CHAR cTmpBuf2[1500]; for(n=1;n<=350;n++) { cTmpBuf2 = "'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //1 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //2 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //3 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //4 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //5 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //6 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //7 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //8 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //9 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //10 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //11 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //12 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //13 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //14 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'"; //15 fnConcatString(cTmpBuf1,cTmpBuf2); fnLUTRON_DeBug("'LUTRON_XML Count-[ ',itoa(n),' ], length_cTmpBuf-[ ',itoa(length_string(cTmpBuf1)),' ]'"); } nLUTRON_SendTest_Str = 0;I end up with a length of:
That's an extra 612 bytes that can screw sh i t up.
it starts adding two bytes on each iteration after it reaches 66000
Now I gotta test to see which end those bytes are added and chopped them off consistently.
For the longest time I figured it was just something else not adding up and never bothered to try to get my counts to match, good thing I did cuz who knows what these extra bytes will break.
it was a lot worse when I was doing this concat function on every data.text which is similar to the test above.
Ok so after 66,000 the functions adds 2 bytes to the beginning each time it's call to append another string segment.
So it's not that big of a deal and shouldn't cause any problems in parsing since it's not being added anywhere in the middle of the final string.
I could just leave it but I decide to add a tag at the beginning which I use to remove and the extra bytes when my string is complete.
if(!length_string(cTmpBuf)) { cTmpBuf = '*BEGIN*'; } fnConcatString(cTmpBuf,cTmpBuf2); then later when complete: if(find_string(cTmpBuf,'*BEGIN*',1)) { REMOVE_STRING(cTmpBuf,'*BEGIN*',1); }So now my string lengths match and all is right with the world.
In further testing the extra bytes consistently added after the string reaches 66K which means these extra bytes are the added pointer bytes when we go above 65535, from 2 bytes pointers to 4 byte pointers.
I played with another method of removing the added bytes by doing it on each call of the function which works fine but probably adds a lot of overhead. I think I'll stick with regular concatenation up to 64000 and then call the concat function to append larger chucks instead doing it each data event.
I should probably modify the function too so that it remove the proper amount of bytes when going form 2 byte to 4 bytes.
{ STACK_VAR INTEGER n; STACK_VAR LONG nLen; STACK_VAR LONG nLenBefore; STACK_VAR INTEGER nLenAppend; LOCAL_VAR CHAR cTmpBuf1[600000]; STACK_VAR CHAR cTmpBuf2[1500]; LOCAL_VAR CHAR cTmpBuf3[2048]; for(n=1;n<=350;n++) { cTmpBuf2 = "'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //1 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //2 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //3 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //4 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //5 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //6 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //7 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //8 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //9 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //10 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //11 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //12 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //13 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890' //14 ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'"; //15 nLenBefore = length_string(cTmpBuf1); fnConcatString(cTmpBuf1,cTmpBuf2); nLen = length_string(cTmpBuf1); if(nLen > 65000)//only adds to the beginning after it reaches 66,000 bytes. { nLenAppend = length_string(cTmpBuf2); if(nLen - (nLenBefore+nLenAppend)) { STACK_VAR CHAR cGetBufStr[10]; cGetBufStr = GET_BUFFER_STRING(cTmpBuf1,nLen - (nLenBefore+nLenAppend)); fnLUTRON_DeBug("'Concat String nCnt-[ ',itoa(n),' ], length_string-[ ',itoa(length_string(cTmpBuf1)),' ], removed-[ ',cGetBufStr, ' ], left_string-[ ',left_string(cTmpBuf1,10),' ], right_string-[ ',right_string(cTmpBuf1,10),' ] :DEBUG<',ITOA(__LINE__),'>'"); } } else { fnLUTRON_DeBug("'Concat String nCnt-[ ',itoa(n),' ], length_string-[ ',itoa(nLen),' ], left_string-[ ',left_string(cTmpBuf1,10), ' ], right_string-[ ',right_string(cTmpBuf1,10),' ] :DEBUG<',ITOA(__LINE__),'>'"); } } nLUTRON_SendTest_Str = 0; }I modified the function to make a copy of the beginning of the string sent in and then remove anything that might be before it prior exiting. Seemed the easiest since I really don't know what the lPos and lOldPos are doing. Now it will work on string smaller that 65535 or bigger without the left over pointer bytes that occur when the pointers are over 65535.
DEFINE_FUNCTION fnConcatString(CHAR sConcatFinal[],CHAR sConcatNew[]) //modified from original //passing back by reference not return!!!!! { STACK_VAR CHAR sPieces[3]; STACK_VAR CHAR sBuffer[786432]; STACK_VAR LONG lPos; STACK_VAR LONG lOldPos; STACK_VAR CHAR cStrBegin[16];//could be smaller STACK_VAR INTEGER nFBS; lpos = 1; if(length_string(sConcatFinal) >= 16) { cStrBegin = left_string(sConcatFinal,16); } else if(length_string(sConcatFinal)) { cStrBegin = left_string(sConcatFinal,length_string(sConcatFinal)); } else if(length_string(sConcatNew) >= 16) { cStrBegin = left_string(sConcatNew,16); } else if(length_string(sConcatNew)) { cStrBegin = left_string(sConcatNew,length_string(sConcatNew)); } else { RETURN; } VARIABLE_TO_STRING(sConcatFinal,sBuffer,lPos); sPieces[1] = sBuffer[lPos-3]; sPieces[2] = sBuffer[lPos-2]; sPieces[3] = sBuffer[lPos-1]; lOldPos = lpos; lpos = lpos - 3; VARIABLE_TO_STRING(sConcatNew,sBuffer,lPos); sBuffer[lOldPos-3] = sPieces[1]; sBuffer[lOldPos-2] = sPieces[2]; sBuffer[lOldPos-1] = sPieces[3]; nFBS = find_string(sBuffer,cStrBegin,1); if(nFBS) { GET_BUFFER_STRING(sBuffer,nFBS-1); } sConcatFinal = sBuffer; }Looks better with a select active:
DEFINE_FUNCTION fnConcatString(CHAR sConcatFinal[],CHAR sConcatNew[]) //modified from original //passing back by reference not return!!!!! { STACK_VAR CHAR sPieces[3]; STACK_VAR CHAR sBuffer[786432]; STACK_VAR LONG lPos; STACK_VAR LONG lOldPos; STACK_VAR CHAR cStrBegin[16];//could be smaller STACK_VAR INTEGER nFBS; lpos = 1; SELECT { ACTIVE(length_string(sConcatFinal) >= 16):{cStrBegin = left_string(sConcatFinal,16);} ACTIVE(length_string(sConcatFinal)):{cStrBegin = left_string(sConcatFinal,length_string(sConcatFinal));} ACTIVE(length_string(sConcatNew) >= 16):{cStrBegin = left_string(sConcatNew,16);} ACTIVE(length_string(sConcatNew)):{cStrBegin = left_string(sConcatNew,length_string(sConcatNew));} ACTIVE(1):{RETURN;} } VARIABLE_TO_STRING(sConcatFinal,sBuffer,lPos); sPieces[1] = sBuffer[lPos-3]; sPieces[2] = sBuffer[lPos-2]; sPieces[3] = sBuffer[lPos-1]; lOldPos = lpos; lpos = lpos - 3; VARIABLE_TO_STRING(sConcatNew,sBuffer,lPos); sBuffer[lOldPos-3] = sPieces[1]; sBuffer[lOldPos-2] = sPieces[2]; sBuffer[lOldPos-1] = sPieces[3]; nFBS = find_string(sBuffer,cStrBegin,1); if(nFBS) { GET_BUFFER_STRING(sBuffer,nFBS-1); } sConcatFinal = sBuffer; }Here's another variation of the function that seems to work. I used for loops because I thought I'd need to dynamically increase the pieces copied from 3 to 5 but that wasn't the case so the original byte by byte copy would work just fine. The bytes added do still jump from 3 to 5 when it goes over 65535 but I'm still a bit confused was VST is doing that seems to add bytes at both the beginning and between the strings being joined.
DEFINE_FUNCTION fnConcatString(CHAR sConcatFinal[],CHAR sConcatNew[]) //modified from original //passing back by reference not return!!!!! { STACK_VAR CHAR sPieces[5]; STACK_VAR CHAR sBuffer[786432]; STACK_VAR LONG lPos; STACK_VAR LONG lOldPos; STACK_VAR LONG nLen[4]; STACK_VAR INTEGER i; STACK_VAR LONG n; lpos = 1; nLen[1] = length_string(sConcatFinal) + length_string(sBuffer); VARIABLE_TO_STRING(sConcatFinal,sBuffer,lPos); nLen[2] = length_string(sBuffer); nLen[3] = nLen[2] - nLen[1]; n = 3; for(i=1;i<=3;i++) { sPieces[i] = sBuffer[lPos-n]; n--; } lOldPos = lpos; lpos = lpos - 3; nLen[1] = ((length_string(sConcatNew) + length_string(sBuffer)) - nLen[3]); VARIABLE_TO_STRING(sConcatNew,sBuffer,lPos); nLen[2] = length_string(sBuffer); nLen[4] = (nLen[2] - nLen[1]); n = 3; for(i=1;i<=3;i++) { sBuffer[lOldPos-n] = sPieces[i]; n--; } get_buffer_string(sBuffer,nLen[4]); sConcatFinal = sBuffer; }Is this incoming string data? Why not use create_buffer instead of manually concatenating the string at the event.
As another option in the past I have piped some very large graphic data directly to a file on the master so it could be dynamically served on system touchpanels - URL's had the | character that reeked havoc on the G4 touchpanels, long story.
Probably cuz I didn't think of it
I'm so use to just using a local var and manually concatenating with that. It would be funny if a created buffer didn't have that 65535 limit but I'm afraid to try in case it doesn't. I'm gonna try it but if it works I might have to shoot myself.
Well that worked just fine. I guess I can't trust what I think I remember anymore cuz I wasted a lot of time for nothing. Oy vey!
Biggest danger with that manual buffer concatenation thing at the data_event is local_var and its default allocation out of non-volatile. Most folks don't realize that volatile can be specified so the system doesn't blow up when it runs out of memory.
I remember that from a recent post but prior I’m sure I didn’t know either and have never delcared a local var as volatile or persistent. With globals I rarely define them as anything but volatile. I don’t recall it ever being discussed at all in the last decade + years either but I can’t really trust my recollections can I.