parsing multiple data out of 1 large piece of data

jisaac · November 2007

gentlemen,
please bear with me as im still relatively fresh out of prog II. i have stared endlessly at the following chunk of data:

ESCX200304200030010037Aventura ft Frankie J - Unknown Album00030010027Berlin - Top Gun Soundtrack00030010025Chris Isaak - Wicked Game00030100022D'Angelo - Brown Sugar00030010019DMX - Unknown Album00030010022Dominic Marte - Ven tu00030010022Drowning Pool - Sinner00030010029El Gran Combo - El Gran Combo00030010029El Gran Combo - Nuevo milenio00030010029El Gran Combo - Unknown Album00030010044El Gran Combo de Puerto Rico - Unknown Album00030010024George Michael - Unknown Album00030010042Gerald Levert, Eddie Levert - Private Line00030010016Godsmack - Awake00030010034Jagged Edge - Jagged Little Thrill00030010027Jagged Edge - Unknown Album00030010021Jay Z - Unknown Album00030010020Jay-Z - Kingdom Come00030010021Jay-Z - The Blueprint00030010021Jay-Z - Unknown Album00030010042Jessica Simpson - I Wanna Love You Forever

This is from an escient fireball (i know theres a module im using this as a learning tool)
i need to obviously get the artist/song data out of here (im hoping to display this as a list of available albums) i eventually want to organize these albums (theres many many more in the database im just showing this sample) into some sort of data structure/array to then populate the TP.

i dont know where to begin?? every string parsing idea i come up with falls short. Has someone come up with a slick way to handle large dynamic data like this & can share some pointers??

thanks again!!

vining · November 2007

Are you using the AMX or Escient FireBall module? The Escient module is open and allows you to see how they do the parsing while the AMX module the actual parsing is hidden in the comm module which then passes interpreted information to the UI mod. Which if you are learning does you absolutely no good at all.

jisaac · November 2007

thanks vining
yes i have the escient module, but there is so much going on in there (4000+ lines of code) that i cant clearly follow and or understand what they are doing. Thats why ive taken the approach to re-do some portions of the module small piece by small piece.
thanks

Spire_Jeff · November 2007

Have you downloaded the External Control Protocol document from Escient? I found it in the Manuals section of the website. The first step to dealing with string parsing is to figure out what the structure of the data is. If you don't have the luxury of a protocol document, you have to examine the data and try to figure out the structure (hopefully there is one) on your own. Fortunately in this case, you have some good documentation to help.

Escient Protocol Document wrote:

All external control commands are made up of the preamble (ESCX), command group
(01,02,10,20,50,70), sub command (specific task), specific data (# of data items, and a size of
data packet then the actual data packet repeated for the # of data items), and a carriage return
end marker.

The first step I would take is to grab a single complete command. Once I have the command, I would start to disassemble the command and handle it appropriately. For this instance, I would remove the preamble. Then I would grab the command group and sub command. Based on the command group and sub command, I would then handle the data as is needed according to the protocol document.

Hope this helps,
Jeff

jisaac · November 2007

Thanks jeff
i do have the protocol documentation and i am familiar with the command structure. I can parse 1 song/album and send it to a panel. Where i get stuck is parsing through a huge chunk of data and pulling out say 30 album titles to show a list of albums on a given media server.

thanks again.

Spire_Jeff · November 2007

In this case, I would use a while(find end marker) statement to keep looking at the incoming data one command at a time. Depending on how you are writing your code, you need to decide how to handle this information. Instead of just sending the data to your touch panel, I would probably populate an array with the data. After I have completed parsing all of the incoming data, I would then deal with populating the touch panel with the necessary data.

This happens to be one of the more involved protocols out there. I understand that you are trying to take only a little portion of the module and deal with just that piece, but as I recall, you have to deal with a lot of tracking to make the communication and control function properly.

Hope this helps a little,
Jeff

NMarkRoberts · November 2007

The length of the title is the last two digits of the number string. Who knows what happens if the title is less than 10 characters, or more than 99, or if the artist is 50 Cent, 10cc or 10,000 Maniacs. Maybe it's actually the last 3 or 4 digits. The rest of the number string is always 0003001?? so you can use that as a delimiter on everything but the last one. Here's some code that uses both pieces of information - untested:

program_name = 'Hello'

define_variable

integer nPosition
integer nLength
integer nTitleCount
char    sTitles[1000][100]
char    sBuffer[10000]

define_program

nTitleCount = 0
nPosition = 1

while (nPosition <> 0)
  {
  nPosition = find_string(sBuffer,'0003001',nPosition)
	
  if (nPosition <> 0)
    {
    nLength = atoi(mid_string(sBuffer,nPosition + 7,4))
    nTitleCount++
    sTitles[nTitleCount] = mid_string(sBuffer,nPosition + 11,nPosition + 10 + nLength)
    }
  }

DHawthorne · November 2007

NMarkRoberts wrote:

The length of the title is the last two digits of the number string. Who knows what happens if the title is less than 10 characters, or more than 99, or if the artist is 50 Cent, 10cc or 10,000 Maniacs. Maybe it's actually the last 3 or 4 digits. The rest of the number string is always 0003001?? so you can use that as a delimiter on everything but the last one.

We could only wish. Actually, the 0003001 means "the next piece of data is 3 digits long, and that data is 001." You can't guarantee it's always going to be that, though it often is consistent in a specific context. In this particular example, it is followed by the length of that one piece of data, the title, so the next four digits are the length of the title, followed by the title itself. Their entire protocol is like that. They always tell you how long the data segment will be just before sending the data itself, but it gets confusing when the "data" is the amount (not length or size, but amount) of data they will be sending ...

jisaac · November 2007

gentlemen,
the following is quite ugly but this sucker works-- at least for 1 go around, it will list the first 10 albums in the fireball, organized in a data structure/ array:

DATA_EVENT[0:dvFBPORT.PORT:0]
{
STRING:
{
SEND_STRING 0, "DATA.TEXT"
(* SET STRING *)
IF(FIND_STRING(DATA.TEXT,"13",1))
{
strRESPONSE = DATA.TEXT
TEMP_BUFF = GET_BUFFER_STRING(strRESPONSE,20)
TEMP_BUFF = ''
FOR (nFBLOOP=1;nFBLOOP <= 10;nFBLOOP++)
{
TEMP_BUFF = GET_BUFFER_STRING(strRESPONSE,9)
TEMP_BUFF = ''
nDATA_LENGTH = LEFT_STRING(strRESPONSE,2)
TEMP_BUFF = GET_BUFFER_STRING(strRESPONSE,2)
TEMP_BUFF = ''
ALBUM_DATA = GET_BUFFER_STRING(strRESPONSE,ATOI(nDATA_LENGTH))
FIREBALL_DATA[nFBLOOP].ALBUMS = ALBUM_DATA
ALBUM_DATA = ''
nDATA_LENGTH = ''
}

}
}
}

told you it was ugly--baby steps..
FYI- my code isnt THIS bad--the brackets got out of whack when i pasted the code??

vining · November 2007

If you look at the String_Event for the Escient from the Escient Module:

STRING:
	  {
	  (* SET STRING *)
	  IF(LENGTH_STRING(strRESPONSE))
	       {
	       strRESPONSE = "strRESPONSE,DATA.TEXT"
	       }
	  ELSE
	       {
	       strRESPONSE = DATA.TEXT
	       }
	  }

of course this works just as good cuz in this case who cares if strRESPONSE is empty.

 strRESPONSE = "strRESPONSE,DATA.TEXT"

They put the strings from the DATA.TEXT into a var or buffer because DATA.TEXT is limited to 254 chars or something to that affect. Now I don't use DATA.TEXT so I'm not 100% sure but if while you're processing your (FIND_STRING(DATA.TEXT,"13",1)) and more data comes in what ever was left in DATA.TEXT is lost and you'll end up with partial returns being processed. So what they did is no matter what comes into via DATA.TEXT it gets added to strRESPONSE and when more comes in it just keeps filling it up appending the newly recieved to the un processed remaining string so you can then do your (FIND_STRING(strRESPONSE,"13",1)) with a "while" and keep the routine going until there are no more complete returns to process and if you don't have a complete return left in the var (buffer) the string just sits there until the rest shows up.

jisaac · November 2007

Thanks Vining.
Im starting to work on a string & command que, which is what i believe you are referring to. I indeed want to
1. bring in data
2. make sure i have a complete strings
3. process that data
4. clear out buffers
5. repeat.

i have quite a long way till i have something even partially working, but im really learning this way. Thanks again.

vining · December 2007

I'd watch # 4. You might want to leave that out or at least see how that affect partial returns that may be waiting for the next segment to arrive.

GSLogic · December 2007

jisaac wrote:

strRESPONSE = DATA.TEXT
TEMP_BUFF = GET_BUFFER_STRING(strRESPONSE,20)
TEMP_BUFF = ''

If you are just clearing data from the strRESPONSE buffer, there is no need to send it to TEMP_BUFF and then clear TEMP_BUFF.
GET_BUFFER_STRING(strRESPONSE,20) by itself will remove the data from strRESPONSE.

Saves a few lines of code and less is always more.

parsing multiple data out of 1 large piece of data

Comments