Home AMX General Discussion

ConcatString Function baggage?

viningvining X MemberPosts: 4,365
edited February 2019 in AMX General Discussion

It seems to me that using the concat string function adds bytes to the final string.

if I use it directly with data.text which means it fires a lot of times by the time I reach my end tag I get an extra 966 bytes.

LOCAL_VAR VOLATILE CHAR cTmpBuf[LUT_MAX_XML_STR];

      nLUTRON_XML_RX_StrLen = (nLUTRON_XML_RX_StrLen + length_string(DATA.TEXT));
      fnConcatString(cTmpBuf,DATA.TEXT);

      if(find_String(DATA.TEXT,'</Project>',1))//use buffer?  not likely to split across two DE's since at end but........
           {
           STACK_VAR INTEGER i;
           STACK_VAR LONG nLen;
           STACK_VAR _sLUT_XML sLUT_ClearXML;
           STACK_VAR _sLUT_XML_Store sLUT_XML_Store;

           nLen = length_string(cTmpBuf);
           fnLUTRON_DeBug("'LUTRON_XML ###############################################:DEBUG<',ITOA(__LINE__),'>'");
           fnLUTRON_DeBug("'LUTRON_XML '");
           fnLUTRON_DeBug("'LUTRON_XML        LENGHT_STR cTmpBuf-[ ',itoa(nLen),' ]       :DEBUG<',ITOA(__LINE__),'>'");
           fnLUTRON_DeBug("'LUTRON_XML '");
           fnLUTRON_DeBug("'LUTRON_XML ###############################################:DEBUG<',ITOA(__LINE__),'>'");

If I change my code so I store up to a little more than 60,000 and then use that to concat with I end up with a closer match to what my in coming counter says I receive but still an extra 16 bytes.

          LOCAL_VAR VOLATILE CHAR cTmpBuf[LUT_MAX_XML_STR];
      LOCAL_VAR VOLATILE CHAR cTmpBuf2[65535];
      LOCAL_VAR VOLATILE LONG nLenTmpBuf2;

      nLUTRON_XML_RX_StrLen = (nLUTRON_XML_RX_StrLen + length_string(DATA.TEXT));
      nLenTmpBuf2 = (nLenTmpBuf2 + length_string(DATA.TEXT));
      cTmpBuf2 = "cTmpBuf2,DATA.TEXT";

      if(nLenTmpBuf2 > 60000)
           {
           fnConcatString(cTmpBuf,cTmpBuf2);
           cTmpBuf2 = '';
           nLenTmpBuf2 = 0;
           }
      if(find_String(DATA.TEXT,'</Project>',1) && nLenTmpBuf2)//use buffer?  not likely to split across two DE's since at end but........
           {
           fnConcatString(cTmpBuf,cTmpBuf2);
           cTmpBuf2 = '';
           nLenTmpBuf2 = 0;
           }            
      if(find_String(DATA.TEXT,'</Project>',1))//use buffer?  not likely to split across two DE's since at end but........
           {
           STACK_VAR INTEGER i;
           STACK_VAR LONG nLen;
           STACK_VAR _sLUT_XML sLUT_ClearXML;
           STACK_VAR _sLUT_XML_Store sLUT_XML_Store;

           nLen = length_string(cTmpBuf);
           fnLUTRON_DeBug("'LUTRON_XML ###############################################:DEBUG<',ITOA(__LINE__),'>'");
           fnLUTRON_DeBug("'LUTRON_XML '");
           fnLUTRON_DeBug("'LUTRON_XML        LENGHT_STR cTmpBuf-[ ',itoa(nLen),' ]       :DEBUG<',ITOA(__LINE__),'>'");

with the first code snippet I get:

254 2019-02-25 (18:12:14)::  LUTRON AXI DEBUG (1): LUTRON_XML        LENGHT_STR cTmpBuf-[ 560186 ]       :DEBUG<4431>
Line    255 2019-02-25 (18:12:14)::  LUTRON AXI DEBUG (1): LUTRON_XML 
Line    256 2019-02-25 (18:12:14)::  LUTRON AXI DEBUG (1): LUTRON_XML ###############################################:DEBUG<4433>
Line    257 2019-02-25 (18:12:14)::  LUTRON AXI DEBUG (1): LUTRON_XML ###############################################:DEBUG<4532>
Line    258 2019-02-25 (18:12:14)::  Memory Available = 444432384 <380928>
Line    259 2019-02-25 (18:12:14)::  LUTRON AXI DEBUG (1): LUTRON_XML 
Line    260 2019-02-25 (18:12:14)::  LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  GetXML_Str     [ 322 ]
Line    261 2019-02-25 (18:12:14)::  LUTRON AXI DEBUG (1): LUTRON_XML 
Line    262 2019-02-25 (18:12:14)::  LUTRON AXI DEBUG (1): LUTRON_XML      XML FILE RETRIEVED COMPLETE   [ 559220 ]
Line    263 2019-02-25 (18:12:14)::  LUTRON AXI DEBUG (1): LUTRON_XML      TOTAL OF SEGMENTS STORED      [ 560186 ]
Line    264 2019-02-25 (18:12:14)::  LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  cTmpBuf        [ 560186 ]

and with the 2nd I get:

Line      7 2019-02-25 (19:02:20)::  LUTRON AXI DEBUG (1): LUTRON_XML 
Line      8 2019-02-25 (19:02:20)::  LUTRON AXI DEBUG (1): LUTRON_XML        LENGHT_STR cTmpBuf-[ 559236 ]       :DEBUG<4446>
Line      9 2019-02-25 (19:02:20)::  LUTRON AXI DEBUG (1): LUTRON_XML 
Line     10 2019-02-25 (19:02:20)::  LUTRON AXI DEBUG (1): LUTRON_XML ###############################################:DEBUG<4448>
Line     11 2019-02-25 (19:02:20)::  LUTRON AXI DEBUG (1): LUTRON_XML ###############################################:DEBUG<4547>
Line     12 2019-02-25 (19:02:20)::  Memory Available = 444477440 <380928>
Line     13 2019-02-25 (19:02:20)::  LUTRON AXI DEBUG (1): LUTRON_XML 
Line     14 2019-02-25 (19:02:20)::  LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  GetXML_Str     [ 322 ]
Line     15 2019-02-25 (19:02:20)::  LUTRON AXI DEBUG (1): LUTRON_XML 
Line     16 2019-02-25 (19:02:20)::  LUTRON AXI DEBUG (1): LUTRON_XML      XML FILE RETRIEVED COMPLETE   [ 559220 ]
Line     17 2019-02-25 (19:02:20)::  LUTRON AXI DEBUG (1): LUTRON_XML      TOTAL OF SEGMENTS STORED      [ 559236 ]
Line     18 2019-02-25 (19:02:20)::  LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  cTmpBuf        [ 559236 ]

which tells me the concat function is adding something, somehow. It obviously adds more if I call it on every data.text received from my device and less if I store close to the max I can with a locally concatenated string. Has anyone ever notice using the function makes your string bigger and I'd assume if its bigger it has the potential to break parsing routine depending on where those added bytes are put.

This is the function I'm using:

DEFINE_FUNCTION fnConcatString(CHAR sConcatFinal[],CHAR sConcatNew[])
     //modified from original //passing back by reference not return!!!!!
     {
     STACK_VAR CHAR sPieces[3];
     STACK_VAR CHAR sBuffer[786432];
     STACK_VAR LONG lPos;
     STACK_VAR LONG lOldPos;

     lpos = 1;

     VARIABLE_TO_STRING(sConcatFinal,sBuffer,lPos);
     sPieces[1] = sBuffer[lPos-3];
     sPieces[2] = sBuffer[lPos-2];
     sPieces[3] = sBuffer[lPos-1];

     lOldPos = lpos;
     lpos = lpos - 3;

     VARIABLE_TO_STRING(sConcatNew,sBuffer,lPos);

     sBuffer[lOldPos-3] = sPieces[1];
     sBuffer[lOldPos-2] = sPieces[2];
     sBuffer[lOldPos-1] = sPieces[3];

     GET_BUFFER_STRING(sBuffer,3)

     sConcatFinal = sBuffer;
     }

Comments

  • HARMAN_icraigieHARMAN_icraigie Technical Trainer II, Harman Professional University Posts: 536

    My guess its the formatting being applied in variable_to_string.

    A simple concatenation will not add any extra data:

    sBuffer = "sBuffer,data.text";

  • viningvining X Member Posts: 4,365

    yeah but that stops working at 65535 which is why I started using the concat function. The Lutron XML I'm working with is currently 559220 bytes and could grow much bigger as more devices, scenes, etc are added. Right now I only have 100 dimmers, 24 T-Stats & 10 keypads so this file could easily double or triple in size on larger projects.

    running this test with a 1500 byte string similar to a single MTU added each iteration of the for loop which runs 350 times. I should end up 525,000:

    else if(nLUTRON_SendTest_Str == 97)
               {
               STACK_VAR INTEGER n;
               LOCAL_VAR CHAR cTmpBuf1[600000];
               STACK_VAR CHAR cTmpBuf2[1500];
    
               for(n=1;n<=350;n++)
                {
                cTmpBuf2 =  "'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'     //1
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'  //2
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'  //3
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'  //4
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'  //5
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'  //6
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'      //7
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'      //8
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'      //9
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'      //10
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'  //11
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'      //12
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'      //13
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'      //14
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'";    //15
                fnConcatString(cTmpBuf1,cTmpBuf2);
                fnLUTRON_DeBug("'LUTRON_XML Count-[ ',itoa(n),' ], length_cTmpBuf-[ ',itoa(length_string(cTmpBuf1)),' ]'");
                }
               nLUTRON_SendTest_Str = 0;
    

    I end up with a length of:

    LUTRON AXI DEBUG (1): LUTRON_XML Count-[ 350 ], length_cTmpBuf-[ 525612 ]
    

    That's an extra 612 bytes that can screw sh i t up.

    it starts adding two bytes on each iteration after it reaches 66000

    LUTRON AXI DEBUG (1): LUTRON_XML Count-[ 41 ], length_cTmpBuf-[ 61500 ]
    LUTRON AXI DEBUG (1): LUTRON_XML Count-[ 42 ], length_cTmpBuf-[ 63000 ]
    LUTRON AXI DEBUG (1): LUTRON_XML Count-[ 43 ], length_cTmpBuf-[ 64500 ]
    LUTRON AXI DEBUG (1): LUTRON_XML Count-[ 44 ], length_cTmpBuf-[ 66000 ]
    LUTRON AXI DEBUG (1): LUTRON_XML Count-[ 45 ], length_cTmpBuf-[ 67502 ]
    LUTRON AXI DEBUG (1): LUTRON_XML Count-[ 46 ], length_cTmpBuf-[ 69004 ]
    LUTRON AXI DEBUG (1): LUTRON_XML Count-[ 47 ], length_cTmpBuf-[ 70506 ]
    LUTRON AXI DEBUG (1): LUTRON_XML Count-[ 48 ], length_cTmpBuf-[ 72008 ]
    

    Now I gotta test to see which end those bytes are added and chopped them off consistently.

    For the longest time I figured it was just something else not adding up and never bothered to try to get my counts to match, good thing I did cuz who knows what these extra bytes will break.

    LUTRON AXI DEBUG (1): LUTRON_XML      XML FILE RETRIEVED COMPLETE   [ 559220 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      TOTAL OF SEGMENTS STORED      [ 559236 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  cTmpBuf        [ 559236 ]
    LUTRON AXI DEBUG (1): LUTRON_XML 
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTHS:
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  Head           [ 730 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  ProjName       [ 59 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  Dealer         [ 60 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  DealerInfo     [ 116 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  Latitude       [ 4 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  Longitude      [ 5 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  CopyRight      [ 63 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  GUID           [ 32 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  ProdType       [ 1 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  AppVer         [ 6 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  XMLVer         [ 3 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  DBExpDate      [ 10 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  DBExpTime      [ 8 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  IsConEnabled   [ 5 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  Global Prefs   [ 135 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  Areas          [ 319795 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  Occupancy Group[ 4453 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  Timeclocks     [ 2768 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  HVACs          [ 179699 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  GreenModes     [ 51048 ]
    

    it was a lot worse when I was doing this concat function on every data.text which is similar to the test above.

    LUTRON AXI DEBUG (1): LUTRON_XML      XML FILE RETRIEVED COMPLETE   [ 559220 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      TOTAL OF SEGMENTS STORED      [ 560186 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  cTmpBuf        [ 560186 ]
    
  • viningvining X Member Posts: 4,365

    Ok so after 66,000 the functions adds 2 bytes to the beginning each time it's call to append another string segment.

    LUTRON AXI DEBUG (1): Concat String LENGTH_STRING-[ 67502 ], left_string-[ $01$D012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678 ] :DEBUG<5168>
    LUTRON AXI DEBUG (1): Concat String LENGTH_STRING-[ 69004 ], left_string-[ $07$AE$01$D0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456 ] :DEBUG<5168>
    LUTRON AXI DEBUG (1): Concat String LENGTH_STRING-[ 70506 ], left_string-[ $0D$8C$07$AE$01$D01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234 ] :DEBUG<5168>
    LUTRON AXI DEBUG (1): Concat String LENGTH_STRING-[ 72008 ], left_string-[ $13j$0D$8C$07$AE$01$D012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012 ] :DEBUG<5168>
    

    So it's not that big of a deal and shouldn't cause any problems in parsing since it's not being added anywhere in the middle of the final string.

    I could just leave it but I decide to add a tag at the beginning which I use to remove and the extra bytes when my string is complete.

     if(!length_string(cTmpBuf))
                {
                cTmpBuf = '*BEGIN*';
                }
     fnConcatString(cTmpBuf,cTmpBuf2);
    
    then later when complete:
    
    if(find_string(cTmpBuf,'*BEGIN*',1))
                {
                REMOVE_STRING(cTmpBuf,'*BEGIN*',1);
                }
    

    So now my string lengths match and all is right with the world.

    LUTRON AXI DEBUG (1): LUTRON_XML      XML FILE RETRIEVED COMPLETE   [ 559220 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      TOTAL OF SEGMENTS STORED      [ 559220 ]
    LUTRON AXI DEBUG (1): LUTRON_XML      LENGTH_STRING  cTmpBuf        [ 559220 ]
    
  • viningvining X Member Posts: 4,365

    In further testing the extra bytes consistently added after the string reaches 66K which means these extra bytes are the added pointer bytes when we go above 65535, from 2 bytes pointers to 4 byte pointers.

    STACK_VAR LONG lPos;
    STACK_VAR LONG lOldPos;
    

    I played with another method of removing the added bytes by doing it on each call of the function which works fine but probably adds a lot of overhead. I think I'll stick with regular concatenation up to 64000 and then call the concat function to append larger chucks instead doing it each data event.

    I should probably modify the function too so that it remove the proper amount of bytes when going form 2 byte to 4 bytes.

    {
               STACK_VAR INTEGER n;
               STACK_VAR LONG nLen;
               STACK_VAR LONG nLenBefore;
               STACK_VAR INTEGER nLenAppend;
               LOCAL_VAR CHAR cTmpBuf1[600000];
               STACK_VAR CHAR cTmpBuf2[1500];
               LOCAL_VAR CHAR cTmpBuf3[2048];
    
               for(n=1;n<=350;n++)
                {
                cTmpBuf2 =  "'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'     //1
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'  //2
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'  //3
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'  //4
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'  //5
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'  //6
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'      //7
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'      //8
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'      //9
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'      //10
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'  //11
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'      //12
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'      //13
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'      //14
                       ,'1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890'";    //15
    
                nLenBefore = length_string(cTmpBuf1);
                fnConcatString(cTmpBuf1,cTmpBuf2);
                nLen = length_string(cTmpBuf1);
                if(nLen > 65000)//only adds to the beginning after it reaches 66,000 bytes.
                 {
                 nLenAppend = length_string(cTmpBuf2);
                 if(nLen - (nLenBefore+nLenAppend))
                      {
                      STACK_VAR CHAR cGetBufStr[10];
    
                      cGetBufStr = GET_BUFFER_STRING(cTmpBuf1,nLen - (nLenBefore+nLenAppend));
                      fnLUTRON_DeBug("'Concat String nCnt-[ ',itoa(n),' ], length_string-[ ',itoa(length_string(cTmpBuf1)),' ], removed-[ ',cGetBufStr,                      
                             ' ], left_string-[ ',left_string(cTmpBuf1,10),' ], right_string-[ ',right_string(cTmpBuf1,10),' ] :DEBUG<',ITOA(__LINE__),'>'");
                      }
                 }
                else
                 {
                 fnLUTRON_DeBug("'Concat String nCnt-[ ',itoa(n),' ], length_string-[ ',itoa(nLen),' ], left_string-[ ',left_string(cTmpBuf1,10),
                             ' ], right_string-[ ',right_string(cTmpBuf1,10),' ] :DEBUG<',ITOA(__LINE__),'>'");
                 }
                }
               nLUTRON_SendTest_Str = 0;
               }           
    
    LUTRON AXI DEBUG (1): Concat String nCnt-[ 39 ], length_string-[ 58500 ], left_string-[ 1234567890 ], right_string-[ 1234567890 ] :DEBUG<5200>
    LUTRON AXI DEBUG (1): Concat String nCnt-[ 40 ], length_string-[ 60000 ], left_string-[ 1234567890 ], right_string-[ 1234567890 ] :DEBUG<5200>
    LUTRON AXI DEBUG (1): Concat String nCnt-[ 41 ], length_string-[ 61500 ], left_string-[ 1234567890 ], right_string-[ 1234567890 ] :DEBUG<5200>
    LUTRON AXI DEBUG (1): Concat String nCnt-[ 42 ], length_string-[ 63000 ], left_string-[ 1234567890 ], right_string-[ 1234567890 ] :DEBUG<5200>
    LUTRON AXI DEBUG (1): Concat String nCnt-[ 43 ], length_string-[ 64500 ], left_string-[ 1234567890 ], right_string-[ 1234567890 ] :DEBUG<5200>
    LUTRON AXI DEBUG (1): Concat String nCnt-[ 45 ], length_string-[ 67500 ], removed-[ $01$D0 ], left_string-[ 1234567890 ], right_string-[ 1234567890 ] :DEBUG<5194>
    LUTRON AXI DEBUG (1): Concat String nCnt-[ 46 ], length_string-[ 69000 ], removed-[ $07$AC ], left_string-[ 1234567890 ], right_string-[ 1234567890 ] :DEBUG<5194>
    LUTRON AXI DEBUG (1): Concat String nCnt-[ 47 ], length_string-[ 70500 ], removed-[ $0D$88 ], left_string-[ 1234567890 ], right_string-[ 1234567890 ] :DEBUG<5194>
    LUTRON AXI DEBUG (1): Concat String nCnt-[ 48 ], length_string-[ 72000 ], removed-[ $13d ], left_string-[ 1234567890 ], right_string-[ 1234567890 ] :DEBUG<5194>
    ...........
    LUTRON AXI DEBUG (1): Concat String nCnt-[ 350 ], length_string-[ 525000 ], removed-[ $FC$EC ], left_string-[ 1234567890 ], right_string-[ 1234567890 ] :DEBUG<5194>
    
  • viningvining X Member Posts: 4,365
    edited February 2019

    I modified the function to make a copy of the beginning of the string sent in and then remove anything that might be before it prior exiting. Seemed the easiest since I really don't know what the lPos and lOldPos are doing. Now it will work on string smaller that 65535 or bigger without the left over pointer bytes that occur when the pointers are over 65535.

            DEFINE_FUNCTION fnConcatString(CHAR sConcatFinal[],CHAR sConcatNew[])
                 //modified from original //passing back by reference not return!!!!!
                 {
                 STACK_VAR CHAR sPieces[3];
                 STACK_VAR CHAR sBuffer[786432];
                 STACK_VAR LONG lPos;
                 STACK_VAR LONG lOldPos;
                 STACK_VAR CHAR cStrBegin[16];//could be smaller
                 STACK_VAR INTEGER nFBS;
    
                 lpos = 1;
    
                 if(length_string(sConcatFinal) >= 16)
                  {
                  cStrBegin = left_string(sConcatFinal,16);
                  }
                 else if(length_string(sConcatFinal))
                  {
                  cStrBegin = left_string(sConcatFinal,length_string(sConcatFinal));
                  }
                 else if(length_string(sConcatNew) >= 16)
                  {
                  cStrBegin = left_string(sConcatNew,16);
                  }
                 else if(length_string(sConcatNew))
                  {
                  cStrBegin = left_string(sConcatNew,length_string(sConcatNew));
                  }
                 else
                  {
                  RETURN;
                  }
                 VARIABLE_TO_STRING(sConcatFinal,sBuffer,lPos);
    
                 sPieces[1] = sBuffer[lPos-3];
                 sPieces[2] = sBuffer[lPos-2];
                 sPieces[3] = sBuffer[lPos-1];
    
                 lOldPos = lpos;
                 lpos = lpos - 3;
    
                 VARIABLE_TO_STRING(sConcatNew,sBuffer,lPos);
    
                 sBuffer[lOldPos-3] = sPieces[1];
                 sBuffer[lOldPos-2] = sPieces[2];
                 sBuffer[lOldPos-1] = sPieces[3];
    
                 nFBS = find_string(sBuffer,cStrBegin,1);
                 if(nFBS)
                  {
                  GET_BUFFER_STRING(sBuffer,nFBS-1);
                  }
    
                 sConcatFinal = sBuffer;
                 }
    
  • viningvining X Member Posts: 4,365
    edited February 2019

    Looks better with a select active:

    DEFINE_FUNCTION fnConcatString(CHAR sConcatFinal[],CHAR sConcatNew[])
         //modified from original //passing back by reference not return!!!!!
         {
         STACK_VAR CHAR sPieces[3];
         STACK_VAR CHAR sBuffer[786432];
         STACK_VAR LONG lPos;
         STACK_VAR LONG lOldPos;
         STACK_VAR CHAR cStrBegin[16];//could be smaller
         STACK_VAR INTEGER nFBS;
    
         lpos = 1;
    
         SELECT
          {
          ACTIVE(length_string(sConcatFinal) >= 16):{cStrBegin = left_string(sConcatFinal,16);}
          ACTIVE(length_string(sConcatFinal)):{cStrBegin = left_string(sConcatFinal,length_string(sConcatFinal));}
          ACTIVE(length_string(sConcatNew) >= 16):{cStrBegin = left_string(sConcatNew,16);}
          ACTIVE(length_string(sConcatNew)):{cStrBegin = left_string(sConcatNew,length_string(sConcatNew));}
          ACTIVE(1):{RETURN;}
          }
    
         VARIABLE_TO_STRING(sConcatFinal,sBuffer,lPos);
    
         sPieces[1] = sBuffer[lPos-3];
         sPieces[2] = sBuffer[lPos-2];
         sPieces[3] = sBuffer[lPos-1];
    
         lOldPos = lpos;
         lpos = lpos - 3;
    
         VARIABLE_TO_STRING(sConcatNew,sBuffer,lPos);
    
         sBuffer[lOldPos-3] = sPieces[1];
         sBuffer[lOldPos-2] = sPieces[2];
         sBuffer[lOldPos-1] = sPieces[3];
    
         nFBS = find_string(sBuffer,cStrBegin,1);
         if(nFBS)
          {
          GET_BUFFER_STRING(sBuffer,nFBS-1);
          }
    
         sConcatFinal = sBuffer;
         }
    
  • viningvining X Member Posts: 4,365

    Here's another variation of the function that seems to work. I used for loops because I thought I'd need to dynamically increase the pieces copied from 3 to 5 but that wasn't the case so the original byte by byte copy would work just fine. The bytes added do still jump from 3 to 5 when it goes over 65535 but I'm still a bit confused was VST is doing that seems to add bytes at both the beginning and between the strings being joined.

    DEFINE_FUNCTION fnConcatString(CHAR sConcatFinal[],CHAR sConcatNew[])
         //modified from original //passing back by reference not return!!!!!
         {
         STACK_VAR CHAR sPieces[5];
         STACK_VAR CHAR sBuffer[786432];
         STACK_VAR LONG lPos;
         STACK_VAR LONG lOldPos;
         STACK_VAR LONG nLen[4];
         STACK_VAR INTEGER i;
         STACK_VAR LONG n;
    
         lpos = 1;
         nLen[1] = length_string(sConcatFinal) + length_string(sBuffer);
         VARIABLE_TO_STRING(sConcatFinal,sBuffer,lPos);
    
         nLen[2] = length_string(sBuffer);
         nLen[3] = nLen[2] - nLen[1];
    
         n = 3;
         for(i=1;i<=3;i++)
          {
          sPieces[i] = sBuffer[lPos-n];
          n--;
          }
    
         lOldPos = lpos;
         lpos = lpos - 3;
    
         nLen[1] = ((length_string(sConcatNew) + length_string(sBuffer)) - nLen[3]);
         VARIABLE_TO_STRING(sConcatNew,sBuffer,lPos);
         nLen[2] = length_string(sBuffer);
         nLen[4] = (nLen[2] - nLen[1]);
    
         n = 3;
         for(i=1;i<=3;i++)
          {
          sBuffer[lOldPos-n] = sPieces[i];
          n--;
          }
         get_buffer_string(sBuffer,nLen[4]);
    
         sConcatFinal = sBuffer;
         }
    
  • HARMAN_icraigieHARMAN_icraigie Technical Trainer II, Harman Professional University Posts: 536

    @vining said:
    yeah but that stops working at 65535 which is why I started using the concat function. The Lutron XML I'm working with is currently 559220 bytes and could grow much bigger as more devices, scenes, etc are added. Right now I only have 100 dimmers, 24 T-Stats & 10 keypads so this file could easily double or triple in size on larger projects.

    Is this incoming string data? Why not use create_buffer instead of manually concatenating the string at the event.

    As another option in the past I have piped some very large graphic data directly to a file on the master so it could be dynamically served on system touchpanels - URL's had the | character that reeked havoc on the G4 touchpanels, long story.

  • viningvining X Member Posts: 4,365
    edited February 2019

    Is this incoming string data? Why not use create_buffer instead of manually concatenating the string at the event.

    Probably cuz I didn't think of it ;) I'm so use to just using a local var and manually concatenating with that. It would be funny if a created buffer didn't have that 65535 limit but I'm afraid to try in case it doesn't. I'm gonna try it but if it works I might have to shoot myself.

  • viningvining X Member Posts: 4,365

    Well that worked just fine. I guess I can't trust what I think I remember anymore cuz I wasted a lot of time for nothing. Oy vey!

  • HARMAN_icraigieHARMAN_icraigie Technical Trainer II, Harman Professional University Posts: 536

    Biggest danger with that manual buffer concatenation thing at the data_event is local_var and its default allocation out of non-volatile. Most folks don't realize that volatile can be specified so the system doesn't blow up when it runs out of memory.

    [LOCAL_VAR] [VOLATILE | PERSISTENT] [CONSTANT] [~type~] name [= ~value~]

  • viningvining X Member Posts: 4,365

    I remember that from a recent post but prior I’m sure I didn’t know either and have never delcared a local var as volatile or persistent. With globals I rarely define them as anything but volatile. I don’t recall it ever being discussed at all in the last decade + years either but I can’t really trust my recollections can I.

Sign In or Register to comment.