#274347 - 14/01/2006 16:24
Did google maps change their output format?
|
carpal tunnel
Registered: 20/12/1999
Posts: 31594
Loc: Seattle, WA
|
Anyone using Parse_google.py besides me? I'm getting errors when I try to use it to parse a set of Google directions. Did google change the format of the web page so that it breaks the script?
|
|
Top
|
|
|
|
|
#274348 - 14/01/2006 18:04
Re: Did google maps change their output format?
[Re: tfabris]
|
carpal tunnel
Registered: 24/12/2001
Posts: 5528
|
You looked at the HTML? The errors you get would be kinda useful as well...
|
|
Top
|
|
|
|
|
#274349 - 15/01/2006 01:36
Re: Did google maps change their output format?
[Re: tman]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31594
Loc: Seattle, WA
|
I never tried to parse the HTML myself before, and (assuming a change is the cause of the problem) don't have a copy of the "old" html to compare against, so no, I didn't look at the HTML other than to glance at it and make sure it was a valid Google Maps file.
As far as the errors I'm getting, they are consistently the same no matter what route I try to parse, so I assumed that you would get the exact same error if you tried to parse a route of your own, and would have everything there to do the debugging with. So I skipped the step of copying the errors into the message.
If you're telling me that you created a fresh new route from Google Maps, parsed it, and did NOT get any errors, then that means it's not a change to their HTML and it's something else odd. So, anyway, here are the errors I'm getting:
Traceback (most recent call last):
File "c:\Progra~1\Directions\parse_google.py", line 115, in ?
wpoints = parse(inname)
File "c:\Progra~1\Directions\parse_google.py", line 99, in parse
xml.sax.parseString(content, h)
File "C:\progra~1\activepython\lib\xml\sax\__init__.py", line 49, in parseStri
ng
parser.parse(inpsrc)
File "C:\progra~1\activepython\lib\xml\sax\expatreader.py", line 90, in parse
xmlreader.IncrementalParser.parse(self, source)
File "C:\progra~1\activepython\lib\xml\sax\xmlreader.py", line 125, in parse
self.close()
File "C:\progra~1\activepython\lib\xml\sax\expatreader.py", line 154, in close
self.feed("", isFinal = 1)
File "C:\progra~1\activepython\lib\xml\sax\expatreader.py", line 148, in feed
self._err_handler.fatalError(exc)
File "C:\progra~1\activepython\lib\xml\sax\handler.py", line 38, in fatalError
raise exception
xml.sax._exceptions.SAXParseException: <unknown>:1:0: no element found
|
|
Top
|
|
|
|
|
#274350 - 15/01/2006 11:05
Re: Did google maps change their output format?
[Re: tfabris]
|
carpal tunnel
Registered: 24/12/2001
Posts: 5528
|
I've never used gpsapp or parse_google.py so the errors are helpful. The reason it doesn't work is because they've changed how the data is sent out. Before there was a small chunk of XML in the HTML which the parse_google.py script looked for but now the path is in a bunch of variables.
|
|
Top
|
|
|
|
|
#274351 - 15/01/2006 18:59
Re: Did google maps change their output format?
[Re: tman]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31594
Loc: Seattle, WA
|
Darn, OK, well that's what I was afraid of. Sigh. If I could code in python, I'd fix it myself.
|
|
Top
|
|
|
|
|
#274352 - 18/01/2006 16:32
Re: Did google maps change their output format?
[Re: tfabris]
|
old hand
Registered: 09/01/2002
Posts: 702
Loc: Tacoma,WA
|
The points are still there- are you clicking on "Link to this page"? Before when I did this that was the only way I could get the Points to appear in the html source. It looks like the XML may have changed to- the encoded points string appears after the string: "polylines: [{id: 'd',points:" in the source when you click "Link to this Page"
|
|
Top
|
|
|
|
|
#274353 - 18/01/2006 17:36
Re: Did google maps change their output format?
[Re: siberia37]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31594
Loc: Seattle, WA
|
Quote:
are you clicking on "Link to this page"?
Yes.
Quote:
It looks like the XML may have changed
Yes, my guess is that this is what's causing parse_google.py to fail.
Any idea what needs to be changed in the parser in order to make it work again?
|
|
Top
|
|
|
|
|
#274354 - 18/01/2006 19:14
Re: Did google maps change their output format?
[Re: tfabris]
|
old hand
Registered: 09/01/2002
Posts: 702
Loc: Tacoma,WA
|
It's not xml anymore, so it's not a trivial change- since you'd have to get rid of everything in the current parse_google that has to do with xml parsing. Instead it looks like you would need to do some good old string parsing. If I have time this week I'll try to dust off my old Delphi code that made a route file from StreetAtlas and modify that.
|
|
Top
|
|
|
|
|
#274355 - 18/01/2006 20:24
Re: Did google maps change their output format?
[Re: siberia37]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31594
Loc: Seattle, WA
|
Maybe there's a way to add a parameter to the Google route link so that it feeds the page back to us in the old format. Just a thought?
|
|
Top
|
|
|
|
|
#274356 - 21/01/2006 19:47
Re: Did google maps change their output format?
[Re: tfabris]
|
old hand
Registered: 09/01/2002
Posts: 702
Loc: Tacoma,WA
|
Try this out. I didn't actually test it with GPSApp- but the output looks reasonable. (2-part RAR file see next message for the rest of it).
Attachments
274929-from_gm.part1.rar (487 downloads)
|
|
Top
|
|
|
|
|
#274357 - 21/01/2006 19:47
Re: Did google maps change their output format?
[Re: siberia37]
|
old hand
Registered: 09/01/2002
Posts: 702
Loc: Tacoma,WA
|
Part 2
Attachments
274930-from_gm.part2.rar (508 downloads)
|
|
Top
|
|
|
|
|
#274358 - 22/01/2006 00:00
Re: Did google maps change their output format?
[Re: siberia37]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31594
Loc: Seattle, WA
|
YOU DA MAN. THANK YOU. The program works, I haven't tried actually DRIVING the resulting route file yet, I'll let you know if it works when I do. I should get the delphi development system so I can ask you for the sources to modify the code myself and add features. I can already think of features I want.  If you are having fun and feeling like adding features yourself, here's the kind of stuff I'd want: - After you've chosen the map file name, have it upload the resulting map file to <<specified FTP folder on the empeg>>. - Have it parse away the text, currently showing up on each of the turn directions, that says "Go 1.2 Miles". Or, if that's too much, just give it a post-process ability, so that after it generates the map file it can launch <<my batch file that I wrote>> passing it the first parameter of <<the path to the map file it just created>>. Then I could use my existing batch file to upload the stuff to the empeg.
|
|
Top
|
|
|
|
|
#274359 - 22/01/2006 00:03
Re: Did google maps change their output format?
[Re: tfabris]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31594
Loc: Seattle, WA
|
Oh, one thing it really needs is a way to remember across re-runs of the program, what's been typed in as options in those boxes. Otherwise I have to re-enter the path name each time.
Darn, I wish I had delphi.
|
|
Top
|
|
|
|
|
#274360 - 22/01/2006 21:00
Re: Did google maps change their output format?
[Re: tfabris]
|
old hand
Registered: 09/01/2002
Posts: 702
Loc: Tacoma,WA
|
No problem. I actually have code in there to save the last entered locations to the registry, but I commented it out and forget to re-enable it. I'll try to get to that soon and strip out the - go X miles thing.
|
|
Top
|
|
|
|
|
#274361 - 22/01/2006 21:02
Re: Did google maps change their output format?
[Re: tfabris]
|
old hand
Registered: 09/01/2002
Posts: 702
Loc: Tacoma,WA
|
Quote:
Darn, I wish I had delphi.
They used to let you download Delphi Personal Edition for free and you could use it for Non-Commercial purposes, but I can't find it anymore. You can still download Kylix for free (cross-platofrm Delphi)- but only for Linux not Windows!
|
|
Top
|
|
|
|
|
#274362 - 24/01/2006 04:54
Re: Did google maps change their output format?
[Re: tfabris]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31594
Loc: Seattle, WA
|
Quote:
I haven't tried actually DRIVING the resulting route file yet, I'll let you know if it works when I do.
Ah. Okay...
Remember when we were fiddling with Parse_Google.py and we had to change the calculation? Something about converting it from one coordinate system to the other. It was trying to do the conversion, but Google already did the conversion. So we had to comment out the bit where it did the conversion. Otherwise your car was off the track by about 100 feet or so. Remember that?
Well, the Delphi program needs the same thing done to it now. It's off by about 100 feet or so.
|
|
Top
|
|
|
|
|
#274363 - 24/01/2006 15:05
Re: Did google maps change their output format?
[Re: tfabris]
|
carpal tunnel
Registered: 19/01/2002
Posts: 3584
Loc: Columbus, OH
|
It was converting from NAD27toWGS84. grumble, grumble, search buttton, grumble grumble.
_________________________
~ John
|
|
Top
|
|
|
|
|
#274364 - 24/01/2006 15:23
Re: Did google maps change their output format?
[Re: JBjorgen]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31594
Loc: Seattle, WA
|
Yeah, what he said. The Delphi program seems to be doing that conversion, still.
|
|
Top
|
|
|
|
|
#274365 - 26/01/2006 00:25
Re: Did google maps change their output format?
[Re: tfabris]
|
old hand
Registered: 09/01/2002
Posts: 702
Loc: Tacoma,WA
|
I cleaned up the code - removed some uneccesary functions and made it strip out the - go XX miles part of the directions. I looked and looked but couldn't find anything in the code that was doing NAD27 to WGS84 conversion. The orginial code wouldn't have had that because it was From StreetAtlas which had it's coordinates in WGS84. So give it another shot- but it could it be that Google Maps changed their DATUM? Arggh- only a point by point comparsion from the raw Points output will tell us...
Part 1 of 2 (again)
Attachments
275171-from_gm.part1.rar (450 downloads)
|
|
Top
|
|
|
|
|
#274366 - 26/01/2006 00:26
Re: Did google maps change their output format?
[Re: siberia37]
|
old hand
Registered: 09/01/2002
Posts: 702
Loc: Tacoma,WA
|
Part 2 of 2 again
Attachments
275172-from_gm.part2.rar (467 downloads)
|
|
Top
|
|
|
|
|
#274367 - 26/01/2006 15:07
Re: Did google maps change their output format?
[Re: siberia37]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31594
Loc: Seattle, WA
|
Thanks for updating it! Boy, if it's not doing the conversion, then we need to get to the bottom of the problem. Cuz it's not very useful when you're talking tight city streets, it tends to make you miss turns when it's off by a factor of 100-200 feet. I tried taking a route I'd recenly done with parse_google.py and comparing it to the same route done with FromGM.exe, and they were different in some interesting ways, such as the number of points on the line being different. (FromGM had more points.) I'd assume Google changed the datum, but oddly, a some of those additional points were ones directly adjacent to a turn, and were exactly one digit off from the turn. It's not the double-points-at-a-turn thing rearing its ugly head again? Anyway, I'll send you the comparison of the two routes in a Private Message. PS: Also noticed that the text of the last data point no longer says "End At <address>" the way Parse_Google used to do. The last data point is now blank. Was that something specifically put into Parse_Google, or did the web site change the way it shows the instructions?
|
|
Top
|
|
|
|
|
#274368 - 26/01/2006 16:42
Re: Did google maps change their output format?
[Re: tfabris]
|
carpal tunnel
Registered: 19/01/2002
Posts: 3584
Loc: Columbus, OH
|
siberia: what version of Delphi, Indy, etc.. are you using? I'm able to open it with Delphi 5 Enterprise with Indy 9 installed, but some of the units in the uses clause seem to be from a later version of Delphi, namely Variants, StrUtils, XMLDoc, xmlintf. Variants are native to Delphi 5 and whatever is used in StrUtils can probably be easily worked around. I think the only one likely to cause much trouble is XMLDoc, so I'm working on an alternative there. With a little work, I think I can get it to compile.
Do you guys think it's possible that Google offset the datum by a certain amount to keep people from ripping their map data for free?
_________________________
~ John
|
|
Top
|
|
|
|
|
#274369 - 27/01/2006 23:12
Re: Did google maps change their output format?
[Re: JBjorgen]
|
old hand
Registered: 09/01/2002
Posts: 702
Loc: Tacoma,WA
|
Quote:
siberia: what version of Delphi, Indy, etc.. are you using? I'm able to open it with Delphi 5 Enterprise with Indy 9 installed, but some of the units in the uses clause seem to be from a later version of Delphi, namely Variants, StrUtils, XMLDoc, xmlintf. Variants are native to Delphi 5 and whatever is used in StrUtils can probably be easily worked around. I think the only one likely to cause much trouble is XMLDoc, so I'm working on an alternative there. With a little work, I think I can get it to compile.
Do you guys think it's possible that Google offset the datum by a certain amount to keep people from ripping their map data for free?
I'm using Delphi 7. I think most of the units have equivalents for old versions. I'm not sure Delphi 5 has an XML unit, we could change it to use OpenXML instead I suppose. I always hated the Delphi XML parser anyways- it has some strange quirks.
|
|
Top
|
|
|
|
|
#274370 - 27/01/2006 23:25
Re: Did google maps change their output format?
[Re: tfabris]
|
old hand
Registered: 09/01/2002
Posts: 702
Loc: Tacoma,WA
|
It looks like Google Maps is using NAD83. That seems to be the general consesus on the Google Maps forums anyways. Anyone want to contribute an example of NAD83->WGS84 conversion?
|
|
Top
|
|
|
|
|
#274371 - 27/01/2006 23:29
Re: Did google maps change their output format?
[Re: siberia37]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31594
Loc: Seattle, WA
|
Wow, so in addition to changing the XML output format, they also changed the coordinate system? Wow.
|
|
Top
|
|
|
|
|
#274372 - 28/01/2006 04:27
Re: Did google maps change their output format?
[Re: tfabris]
|
veteran
Registered: 21/01/2002
Posts: 1380
Loc: Erie, CO
|
Map data is provided by 3rd parties, maybe they are switching to a new vendor? Quote:
Geocoding data for map content in Google Local is provided under license by Navteq North America LLC ("NAVTEQ") and/or Tele Atlas North America, Inc. ("TANA") and/or other third parties, and subject to copyright protection and other intellectual property rights owned by or licensed to NAVTEQ, TANA and/or such other third parties.
From this page on Google Maps.
|
|
Top
|
|
|
|
|
#274373 - 30/01/2006 18:47
Re: Did google maps change their output format?
[Re: tfabris]
|
old hand
Registered: 09/01/2002
Posts: 702
Loc: Tacoma,WA
|
After doing some research now I'm not so sure.. NAD83 supposedly would only cause a 1 or 2 meter different between WGS84 so that's probably not the problem. The problem could be their maps aren't precise enough for this-- after all their maximum zoom is not really that small if you try it. I can still see about a 3-block radius on their tighest zoom, which means maybe they stopped paying for really precise maps?? I dunno- maybe someone should try to update the python script to make sure i'm not missing something. My code should be fairly easy to translate to python I hope.
|
|
Top
|
|
|
|
|
#274374 - 30/01/2006 19:12
Re: Did google maps change their output format?
[Re: siberia37]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31594
Loc: Seattle, WA
|
I had a very strange situation, too, and I dunno if it's the delphi program or if it's their data, but anyway...
I put in a route generated with the FromGM.exe program. The route was as accurate as expected (approx 100-300 feet off at any moment) until a certain point right in the middle of the route where it suddenly got off by (what seemed to be) a couple miles. I was still following the same path directionally, but my current-position icon was about 2 miles east of the path.
I checked to make sure it wasn't the GPS data by loading another route (an older one generated with parse_google.py) which used the same highway, and it was just fine.
|
|
Top
|
|
|
|
|
#274375 - 23/06/2006 15:49
Re: Did google maps change their output format?
[Re: siberia37]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31594
Loc: Seattle, WA
|
Digging up an old thread here... Quote:
The problem could be their maps aren't precise enough for this-- after all their maximum zoom is not really that small if you try it. I can still see about a 3-block radius on their tighest zoom, which means maybe they stopped paying for really precise maps?
I've been thinking about this, and I think the problem is that Google Maps changed the output format so that the line is drawn at whatever resolution is needed for that particular zoom level. So if you're zoomed out to see the entire route, the drawn line is gonna be pretty darn innacurate.
But...
Google Earth has a very nifty little, easy to parse, very accurate, XML format file when you use it to get directions. Just fire up Google Earth, get directions somewhere, and then right-click on the list of directions and hit either "copy" (to get the XML onto the clipboard) or "Save As" (to make a KML file of the same text). Note: If doing Save As, don't do KMZ as that's a zip file. Do KML.
Anyone wanna take a stab at updating "Parse_google.py" or "fromGM.exe" to read the Google Earth KML files?
Edit: Note: I'm also experimenting with what happens when I feed "FromGM.exe" a URL that's been pre-zoomed-in as far as it'll go.
|
|
Top
|
|
|
|
|
#274376 - 23/06/2006 16:48
Re: Did google maps change their output format?
[Re: tfabris]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31594
Loc: Seattle, WA
|
Quote:
Edit: Note: I'm also experimenting with what happens when I feed "FromGM.exe" a URL that's been pre-zoomed-in as far as it'll go.
Naw, doesn't work. When I feed it a long route (such as Seattle to Vancouver BC) it gives strange seemingly random lines that go off in odd directions half way through the trip.
|
|
Top
|
|
|
|
|
|
Previous Topic
Index