Jump to content

 
Photo

More OSM

- - - - -

  • Please log in to reply
14 replies to this topic

#1
frax

frax

    Hall of Fame

  • Associate Admin
  • PipPipPipPipPipPipPip
  • 2,299 posts
  • Gender:Male
  • Location:Stockholm, Sweden
  • Interests:music, hiking, friends, nature, photography, traveling. and maps!
  • Sweden

Hi,

Inspired by recent discussions on using OSM, I thought I would try to see if I could use Open Streetmap for a project I was working on just now. I have been working on a map over South Sudan, and now the client decided that they wanted major roads on there as well. It is just for background, so the accuracy is not crucial.

Anyhow - I suspect that the data I have handy, like the ArcGIS Data & Maps roads layer (from Delorme) is decent for my purposes, but I have hunch that the OSM data for this region might be even better. That also depends on how it is classified though, I have never worked with it, so it is part of the exploration!

I kind of assume that the data for this region is not so detailed (the OSM data for a decent sized Western city is probably more, in terms of size!?).

But - I have problems doing simple exports - e.g. XML or the SVG formats provided by the main OSM website.

And that site with shape file exports has only a few countries in Africa... Global Mapper only loads up pre-rendered tiles (no vector data).

I tried with Maperitive now, but all I get is timeouts for this area of the world... And downloading using the REST/API url tells me that the bbox is too large...

So is there some other smart/easy way? I am downloading the 12 GB planet file right now.

(this is just for fun, I delivered the map with the ESRI roads data)
Hugo Ahlenius
Nordpil - custom maps and GIS
http://nordpil.com/
Twitter

#2
Charles Syrett

Charles Syrett

    Ultimate Contributor

  • Validated Member
  • PipPipPipPipPipPip
  • 536 posts
  • Canada

If you're talking about CloudMade, I've had occasions where data is simply missing while they do maintenance, and then it reappears next day. Then again, I've never tried downloading African data. Have you tried Tracks4Africa?

Charles Syrett
Map Graphics
http://www.mapgraphics.com

#3
Nick H

Nick H

    Legendary Contributor

  • Validated Member
  • PipPipPipPipPip
  • 307 posts
  • Gender:Male
  • Location:Caversham, Reading, England.
  • United Kingdom

There is an OSM shapefile for Sudan here (2.2-meg download):

http://download.clou...om/africa/sudan

Regards, N.
Caversham, Reading, England.

#4
Hans van der Maarel

Hans van der Maarel

    CartoTalk Editor-in-Chief

  • Admin
  • PipPipPipPipPipPipPip
  • 3,834 posts
  • Gender:Male
  • Location:The Netherlands
  • Interests:Cartography, GIS, history, popular science, music.
  • Netherlands

I can do a custom extract out of the database for you, full data model, delivered as shapefiles. May not be the best option for very large areas, as it's a very slow process and the OSM servers are prone to lags. Contact me privately if you're interested.
Hans van der Maarel - Cartotalk Editor
Red Geographics
Email: hans@redgeographics.com / Twitter: @redgeographics

#5
Crischan

Crischan

    Master Contributor

  • Validated Member
  • PipPipPipPip
  • 199 posts
  • Gender:Male
  • Location:Eberswalde, Germany
  • Interests:Spatial Information Design
  • Germany

I have had trouble using the Cloudmade OSM XML data in the past where the data would be corrupted for some reason. Since then I am more than happy with the downloads (XML, Shape) from http://www.geofabrik...a/download.html (German company, English website)
They say the update their data daily using the OSM servers.

Cheers,
C
Crischan Wygoda
http://wygoda.net

#6
frax

frax

    Hall of Fame

  • Associate Admin
  • PipPipPipPipPipPipPip
  • 2,299 posts
  • Gender:Male
  • Location:Stockholm, Sweden
  • Interests:music, hiking, friends, nature, photography, traveling. and maps!
  • Sweden

Thanks for all the input! I had missed that Cloudmade site, I have downloaded the file to peek at the data. The Geofabrik site does not have data for Sudan (they have a whole Africa osm-file).

Hans, thanks for the offer, but I don't think it is needed right now. I assume that you have downloaded a planet.osm file (and converted it?).

Any tips on what I would use to do extracts from the giant planet.osm file?
Hugo Ahlenius
Nordpil - custom maps and GIS
http://nordpil.com/
Twitter

#7
Flanners

Flanners

    Newbie

  • Validated Member
  • Pip
  • 2 posts
  • Canada

I was just looking at this yesterday. I exported an osm XML file, then opened it in Quantum GIS- QGIS has a plugin (go to plugins menu/ manage plugins) that lets you load an osm file, which can be saved as a shape file.

That was just for a small area though, not a whole country...

#8
Nick H

Nick H

    Legendary Contributor

  • Validated Member
  • PipPipPipPipPip
  • 307 posts
  • Gender:Male
  • Location:Caversham, Reading, England.
  • United Kingdom

Any tips on what I would use to do extracts from the giant planet.osm file?

Hum, a twelve-gigabyte OSM file. The only practical way I can think of would be to load it into a PostGIS database using osm2pgsql. Once aboard the lugger, sorry, once in the database it would be possible to select the data required by extent and feature and bring it into GIS.

There are tools that are supposed to be able to subset OSM data, but I'm not sure how they would perform with a such a huge OSM file.

Regards, N.
Caversham, Reading, England.

#9
frax

frax

    Hall of Fame

  • Associate Admin
  • PipPipPipPipPipPipPip
  • 2,299 posts
  • Gender:Male
  • Location:Stockholm, Sweden
  • Interests:music, hiking, friends, nature, photography, traveling. and maps!
  • Sweden

I am trying the "OSM Loader" python script that I found on arcscripts - it takes a osm/bz2 archive and puts it into a file geodatabase. It bailed out on the full 13GB planet.osm though (some size limitation?) and I am also getting an error from bzip2.exe on that file (but the checksum is correct, so it shouldn't be corrupted!).

But that loader script seems to work fine on the 183MB africa.osm.bz2 file that I picked up... (still running though, so i haven't seen the result yet!)
Hugo Ahlenius
Nordpil - custom maps and GIS
http://nordpil.com/
Twitter

#10
Hans van der Maarel

Hans van der Maarel

    CartoTalk Editor-in-Chief

  • Admin
  • PipPipPipPipPipPipPip
  • 3,834 posts
  • Gender:Male
  • Location:The Netherlands
  • Interests:Cartography, GIS, history, popular science, music.
  • Netherlands

Hans, thanks for the offer, but I don't think it is needed right now. I assume that you have downloaded a planet.osm file (and converted it?).


I haven't dared doing that yet... No, my process queries the OSM database directly, which is why it's not exactly suited to doing large regions.

I might give the planet.osm-to-database a go. Once it's in there I can get by with just the diff files and I'd be able to get extracts faster since it'll be coming out of my local database.
Hans van der Maarel - Cartotalk Editor
Red Geographics
Email: hans@redgeographics.com / Twitter: @redgeographics

#11
frax

frax

    Hall of Fame

  • Associate Admin
  • PipPipPipPipPipPipPip
  • 2,299 posts
  • Gender:Male
  • Location:Stockholm, Sweden
  • Interests:music, hiking, friends, nature, photography, traveling. and maps!
  • Sweden

That OSM loader script seemed to work ok - running overnight for the entire africa planet file, so now I have that data in a file geodatabase for ArcGIS...
Hugo Ahlenius
Nordpil - custom maps and GIS
http://nordpil.com/
Twitter

#12
frax

frax

    Hall of Fame

  • Associate Admin
  • PipPipPipPipPipPipPip
  • 2,299 posts
  • Gender:Male
  • Location:Stockholm, Sweden
  • Interests:music, hiking, friends, nature, photography, traveling. and maps!
  • Sweden

I haven't dared doing that yet... No, my process queries the OSM database directly, which is why it's not exactly suited to doing large regions.


My understanding is that there is a maximum bbox of .25 degrees by .25 degrees for the REST api - so one would have to split into tiles and query for them separately, and then merge them...
Hugo Ahlenius
Nordpil - custom maps and GIS
http://nordpil.com/
Twitter

#13
Hans van der Maarel

Hans van der Maarel

    CartoTalk Editor-in-Chief

  • Admin
  • PipPipPipPipPipPipPip
  • 3,834 posts
  • Gender:Male
  • Location:The Netherlands
  • Interests:Cartography, GIS, history, popular science, music.
  • Netherlands

My understanding is that there is a maximum bbox of .25 degrees by .25 degrees for the REST api - so one would have to split into tiles and query for them separately, and then merge them...


I thought it was limited to a certain number of objects. A fixed size would be too easy...
Hans van der Maarel - Cartotalk Editor
Red Geographics
Email: hans@redgeographics.com / Twitter: @redgeographics

#14
frax

frax

    Hall of Fame

  • Associate Admin
  • PipPipPipPipPipPipPip
  • 2,299 posts
  • Gender:Male
  • Location:Stockholm, Sweden
  • Interests:music, hiking, friends, nature, photography, traveling. and maps!
  • Sweden

FYI - I am not able to work with the 13GB planet file in the OSM loader from arcscripts - I suspect the file is too large and that it reaches some limit in the bz2 lib for python...
Hugo Ahlenius
Nordpil - custom maps and GIS
http://nordpil.com/
Twitter

#15
redbeard

redbeard

    Contributor

  • Validated Member
  • PipPip
  • 26 posts
  • United States

And that site with shape file exports has only a few countries in Africa... Global Mapper only loads up pre-rendered tiles (no vector data).

So is there some other smart/easy way? I am downloading the 12 GB planet file right now.

(this is just for fun, I delivered the map with the ESRI roads data)


I know this thread is a little old...but

Actually globalmapper can read the OSM xml based file - just not through the menu that deals with online sources.

I will warn you, if you extract the osm file for some reason, it's huge. I suspect that the arc method might have run out of memory. I'm also interested in getting the entire planet file into a geodatabase. Has anyone since been able to do this?

Also, has anyone written a sqlite or spatialite importer for the Planet file? I ask as I wouldnt mind making my own tiles, but wish to symbolize using QGIS or ArcMap (or something similar). Editing the mapnik xml files gives me a headache.

Maybe I'll have to take a look at tilemill.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

-->