Parsing Data from a Webpage

jweather · June 2008

Actually, http://inteliplex.com/wx and http://inteliplex.com/wx/ are two different URLs. You're requesting the former, it's redirecting you to the latter. Send GET /wx/ HTTP/1.1 and all should be well.

I believe it does the redirect so that if you have relative links on the /wx/index.html page (say to test.html), they redirect consistently to /wx/test.html in all cases rather than to /test.html if the user browsed to http://inteliplex.com/wx -- make sense?

Jeremy

mpullin · June 2008

jweather wrote: »

Actually, http://inteliplex.com/wx and http://inteliplex.com/wx/ are two different URLs. You're requesting the former, it's redirecting you to the latter. Send GET /wx/ HTTP/1.1 and all should be well.

I believe it does the redirect so that if you have relative links on the /wx/index.html page (say to test.html), they redirect consistently to /wx/test.html in all cases rather than to /test.html if the user browsed to http://inteliplex.com/wx -- make sense?

Jeremy

Interesting... thanks for pointing that out.

jjames · June 2008

When I send this (ZIP_CODE is 65065)

SEND_STRING 0:nLOCAL_PORT+1:0,"'GET /weather/map/',ZIP_CODE,' HTTP/1.1',13,10,
	'Host: www.weather.com',13,10,
	'User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0',13,10,
	'Accept: */*',13,10,
	'Accept-Language: en-us,en;q=0.5',13,10,13,10"

I get this . . .

HTTP/1.1 307 Temporary Redirect
Date: Fri, 20 Jun 2008 20:37:58 GMT
Server: Apache
Set-Cookie: partner=www; domain=(null); path=/; Version="1";
Set-Cookie: partner_url=(null); domain=(null); path=/; Version="1";
Location: [url]http://www.weather.com/weather/map/65065[/url]
Content-Length: 443
Keep-Alive: timeout=3, max=1
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>307 Temporary Redirect</title>
</head><body>
<h1>Temporary Redirect</h1>
<p>The document has moved <a href="http://www.weather.com/weather/map/65065">here</a>.</p>
<p>Additionally, a 307 Temporary Redirect
error was encountered while trying to use an ErrorDocument to handle the request.</p>
<hr>
<address>Apache Server at [url]www.weather.com[/url] Port 80</address>
</body></html>

Anyone? Please?

slip cougan · June 2008

i!-EquipmentMonitorOut.axi

Where can I find this file?

Thanks

gary

vining · June 2008

Try this URL:

http://www.weather.com/weather/tenday/USMO0665?cm_ven=LWO&cm_pla=city_page&cm_ite=10-day&site=city_page&cm_cat=rss&locid=USMO0665&par=LWO_rss

jjames · June 2008

Thanks, but I don't need the forecast, I'm trying to get the "local" radar for a specified zip code, and trying to scrape the image from returned html.

jjames · June 2008

slip cougan wrote: »

Where can I find this file?

Gary,
Log into AMX.com, then go to Tech Center -> Application Files -> integration! solutions and it should be there.

vining · June 2008

OK, try this site.

http://www.srh.noaa.gov/ridge/lite/

Click on a radar image type and then select an area from the list. Here's some snipets of code to help you figure out what your looking at.

char Rad_Image_Type [9][3]=

     {
     'N0R', // Base Reflectivity
     'N0S', // Storm Relative Motion
     'N0V', // Base Velocity
     'N1P', // One-Hour Precipitation
     'NCR', // Composite Reflectivity
     'NTP', // Storm Total Precipitation
     'N0Z', //Base Reflectivity //Long Range image (view out to 248 nmi)
     'NET', //
     'NVL'  //
     } 

 //MISSOURI 
     RadState.Missouri[1].Area = 'Kansas City' ;
     RadState.Missouri[2].Area = 'Springfield' ;
     RadState.Missouri[3].Area = 'St. Louis' ; 
     RadState.Missouri[1].Code = 'eax' ;  
     RadState.Missouri[2].Code = 'sgf' ;  
     RadState.Missouri[3].Code = 'lsx' ;

I never quite finished this module and I intended on making every radar site in the US available. I did finish the Satellite portion, I think?

jjames · June 2008

Thanks . . . I think I'm being too picky. LOL!

I can go ahead and scrape out the image I need if I can get to the source I see in my browser, I just think there's too much redirection going on. I want to stick with Weather.com since I think they're the most familiar and legible radar maps for the common client.

Thanks though!!!

jjames · June 2008

Quick update: I was finally able to find it. I had to include the "encoding" attribute, and I was then able to read the entire page and get the data I needed.

Chip Moody · June 2008

What a PIA! Glad ya' found it though.

jjames wrote: »

I had to include the "encoding" attribute

jjames · June 2008

Chip Moody wrote: »

What a PIA! Glad ya' found it though.

Thanks . . . actually, let me show you what I had to send:

SEND_STRING 0:nLOCAL_PORT+1:0,"'GET /forecast/',ZIP_CODE,'.html HTTP/1.1',13,10,
			'Host: weather.yahoo.com',13,10,
			'User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)',13,10,
			'Accept-Encoding: deflate;q=0',13,10,
			'TE: deflate;q=0',13,10,
			'expires: #now()#',13,10,
			'Keep-Alive: 300',13,10,
			'Cache-Control: no-cache, no-store, must-revalidate',13,10,
			'charset: utf-8',13,10,13,10"

It wasn't the "encoding" (my bad), it was the "charset". After a while Googl'ing, I came across this suggestion and it sure enough worked. It was a PITA . . . I've got it all working now, except I cannot load the Alaskan Doppler radar becase, well . . . there isn't one. [EDIT: There isn't a 600 mile radar, only the 2100 - though that might be good enough.]

So, now I need to adjust it . . . ugh. Oh well, the stuff you gotta go through to do something "cool" and probably won't be used much.

Parsing Data from a Webpage

Comments