21 replies to this topic

[Peeks]

Hello everyone,

I've been struggling with sourcing some GIS data for months, now.

I've been asked to plot streets and administrative boundaries on a website, using online mapping controls like Google maps or MS Virtual Earth. (i know these online apps already show this info .. i need to manipulate it .. light highlight them and stuff).

I've struggled to find any data (free or commerical) that can help me here. We're using SQL 2008 as our database to store the spatial data. We've been requested to support USA, Canada, UK, Australia, NZ, and some European countries.

Problem is, I just can't find any data out there! if i do find it, it's hard to use or really simplified.

To start out, I've been trying to use the US Census TIGER LINE shapefiles and convert them into wgs84 and then import them into sql 2008. Now, there's polygon data for roads, states and counties .. but this data is pretty simplified. And when i find any country boundaries, it's REALLY simplified. As a good example, i found a boundary for USA which was around 3000 points only. So when i tried to find if some coastal lat/longs were part of the US mainland (or any US land), it failed because of the simplified boundary line was one long line and the points landed in the ocean.

I'm starting to get really frustrated because I'm sure there's data out there. I'm also scared that this data is ludicrously expensive

So can anyone help me?

Are there sources for pretty details administrative boundaries that I could put into our SQL Server.

Are there sources for streets?

I've looked at openstreetmap.org (or whatever that site is) but everyone there keeps saying the data is very incomplete. I also have no idea how to get any data out of there, even if i wanted to.

Please help - i'm really struggling with this I didn't know how much fun playing with maps could be (with the limited data i've had to play with) .. it's so powerful and exciting! It's killing me that i just don't have any good data to use!!!

please help!

Edited by Peeks, 29 April 2009 - 07:40 AM.

[Hans van der Maarel]

Hi Peeks, and welcome to Cartotalk.

First of all, if you're going to have to collect and maintain data for various countries, good luck! It can be difficult. The US is generally pretty good about making data available for free, other countries are not. Then, if you can actually get the data, you have to deal with different coordinate systems, different data models and so on.

One thing you may consider is commercial vendors (Tele Atlas, Navteq, AND), their datasets usually cover large sections of the earth. It comes at a price though.

I'm not a fan of OpenStreetmap. The data quality varies and it's quite difficult to get data out of it and into a GIS application. At least now that you can export XML directly from the site, and FME supports that XML, it's becoming easier, but it's still a lengthy process to get done for larger areas. There is somebody in Germany who is making shapefiles of individual countries once every few days though.

Trust me, we all love good data, but it's often hard to come by.

[MostlyHarmless]

One thing you may consider is commercial vendors (Tele Atlas, Navteq, AND), their datasets usually cover large sections of the earth. It comes at a price though.

I'm definitely spoiled because I work for NT and have their data at my disposal.

It's hard sometimes for sure...I used to work with TIGER files extensively for street data before I joined navteq. I've never really needed a whole country before, only local data which is very good in the Chicago area. You basically need street and administrative boundary data?

You can check out sources like GIS clearinghouses:

http://www.geography...wnloadable.html

http://data.geocomm....alog/index.html

[Peeks]

You basically need street and administrative boundary data?

You can check out sources like GIS clearinghouses:

http://www.geography...wnloadable.html

http://data.geocomm....alog/index.html

thanks for the warm reception guys

and yep! that's exactly what i'm after.
administrative boundaries for
a) country (admin 0?)
states (admin 1?)
c) counties (if applicable)
d) metro cities
e) streets

for USA/CANADA/UK/AUSTRALIA/NEW ZEALAND (just to start out with)

and of some sorta high resolution.

I envy you NCP if u have access to your NavTec data sources! lucky duck!

I checked out those two links and .. well .. it was all abit foreign to me. i wasn't sure exactly what i was after.

like for example..

USGS 24K Quad Grid (2 MB, SHP format

is that a Shapefile 2MB in size of .. US somethings in a 24 thousand size'd grid? .. how can that help.

I'm so confused and stressed

[Charles Syrett]

I think you may be confused and stressed simply because you're expecting far too much. Imagine wanting to become, say, a dentist and then finding out that all that cool looking equipment costs tons of money! GIS data is prepackaged information with a huge amount of potential value in it. Thousands of hours of human intelligence is poured into it. Why should it be free? Why should it even be cheap?

I suggest you do what the rest of us do -- find out what works for you (given costs etc) and make do with that. As you go about the task of improving and updating that TIGER data, you'll begin to get a better feeling for why good data costs good money.

Good luck with it, and enjoy making your maps!

Charles Syrett
Map Graphics
http://www.mapgraphics.com

[MostlyHarmless]

You just need to dig. You won't, unfortunately, find one giant site to meet all of your needs. For example, just looking through that first link I posted I found this: http://nationalatlas...ld/roadtrl.html

Looking further I found some boundary files: http://nationalatlas...tp.html#roadtrl


I know for sure you can get metro shapfiles from the census website as CSA, MSA, and urban areas. http://www.census.go.../bdy_files.html

I'm not real familiar with other countries and it may very well be harder to find...but you need to be a "GIS detective" as my old teach said.

[Hans van der Maarel]

like for example..

USGS 24K Quad Grid (2 MB, SHP format

is that a Shapefile 2MB in size of .. US somethings in a 24 thousand size'd grid? .. how can that help.

That would be a 2 MB shapefile containing an index grid for the USGS 1:24000 (hence the 24k) topographic maps (aka "Quads" for some reason I'm not entirely sure about)

[Charles Syrett]

(aka "Quads" for some reason I'm not entirely sure about)

"Quad" is short for "Quadrangle", = "four angles". Or four sides. Yeah, yeah, I know.... guess you could say that 99% of maps are quads, for that matter!

Charles Syrett
Map Graphics
http://www.mapgraphics.com

[MapMedia]

You should definitely check out ESRI's data.

If you want a Google Map look you will need to use raster tiles, not vector. Otherwise, you will need to purchase NT or TeleAtlas data for streets if you want them to resemble the quality of streets found on Google Maps, especially if you want to be consistent from US to Europe etc.

Otherwise, you will have to 'make do' with free data - you can get this from a few sources, including ESRI. Make a list of the layers you need (detailed streets, countries, administrative boundaries, etc.) and maybe we can point you in the right direction.

You can download Census 2000 TigerLine data here.

Lastly, since you are hosting thru SQL you should def. consider simplifying (removing vertices) to increase speed.

Good luck - Chris

[jrat]

If you only need to do simple stuff ( highlight lines ect.) and you like google earth; you can buy a version of google earth that will allow some manipulation/creation of the data. You can then use it as your platform. There is a fee but i don't think that it is that much.

[A. Fenix]

Welcome Peek!

Finding data is one of the most challenging and yet rewarding (when discovered) aspects of being a GIS Analyst and Cartographer. Indeed, all of us have to be a "GIS detective" as NoCoastPunk quoted! In my years of being a GIS analyst I have found ways to find data, but most of the time it's literally finding the right lead and/or google search string to uncover a buried data set. I work for a non-profit so rarely have the pleasure to purchase top notch data sets. So, I spend a lot of time hunting for the buried GIS data treasure, and editing/creating data sets. I've been able to uncover most data sets that I've hunted for, but rarely do I find the QUALITY of data that I actually need. But this is usually due to scale issues. For example, if you are mapping an entire state or country you won't need (nor want) very detailed data such as the USGS 24k quad data that you mentioned. However, if you're zooming into a city you'll definitely want and need as detailed of data as possible. This leads me to ask you some more specific questions; what is this mapping application supposed to do? How are your users going to interact with it? At what scale will most users be interested in? Finally, it seems that you are simply looking for background data for your online mapping application, such as streets and administrative boundaries. Google maps has all of this data already, which means by simply adding their api to your website you'd already be accessing all of their data sets. It seems that this is what you were asked to do in the first place, correct? There is also mapserver (http://mapserver.org/) as an OS option...

wishing you the best!

[eli]

good sources of Canada-wide data are geobase and the national topographic series (NTDB), which is available in both 50k and 250k.

i found some 1:250k data for Australia at Geoscience Australia.

Good luck!

[rudy]

good sources of Canada-wide data are geobase and the national topographic series (NTDB), which is available in both 50k and 250k.

i found some 1:250k data for Australia at Geoscience Australia.

Good luck!

Also geogratis for both smaller scale and larger scale topographic data. Also, refer to the Resources section of Cartotalk for some other data sources.

[Peeks]

Hello everyone again and thanks for the warm reception

First up, I do accept that you get what you pay for. So I do understand that we might need to purchase some commerical data. That said, we're a startup making a startup web app. So purchasing data that is 50K (and even more) is way out of our reach.

It's definately challenging being a GIS detective!

Currently we're utilising Google Maps on our web application. We show simple polygons overlayed on the Google Map. A good example of this is overlaying USA Zip Codes over the google map, based upon the bounding box of the current user's viewport. Nothing hard and dozens of sites are doing that.

What I'm struggling to do is find more polygons that represent
1) streets
2) suburbs or neighbourhoods
3) cities (eg. greater Los Angeles. Greater London, etc)
4) counties
5) states
6) countries

and with these poly's we wanted to overlay them on google maps with a massive amount of statistic data we have (which is also, nothing new).

For example, imagine i have a Co-Ordinate that is along the coast of the USA .. say .. this random point in NYC. : +40° 42' 36.53", -74° 0' 23.06" (40.710147, -74.006404)

With some free country boundary data i sourced (sorry, i forgot where i found it buried, on the net), when i asked if that point intersects the USA country shape, i got back a false result (no, it didn't). Checking out the shapefile, it was so simplified that staten island was a few sided poly only, that resulted in that point being pinned, in the Upper New York Bay.

So we're not trying to make our own tile server or custom tiles. We're trying to use the shapes that define the various administrative boundaries for a few countries, to be used in our spatial math (eg. crimes on this street or in this state) and then to visualise this.

does this make sense?

[MostlyHarmless]

What I'm struggling to do is find more polygons that represent
1) streets
2) suburbs or neighbourhoods
3) cities (eg. greater Los Angeles. Greater London, etc)
4) counties
5) states
6) countries

Street usually aren't polygons and should be the easiest to find. Suburbs or neighborhoods may be tricky and you may have to create them yourself. Cities and MSAs like I said can be gotten from the census website. I also have some good shapefiles for urban areas. States and countries should be easy if you look hard enough. I made this map just from searching the net.