Jump to content

 
Photo

Selection limit in ArcGIS?

- - - - -

  • Please log in to reply
16 replies to this topic

#1
cmdrico7812

cmdrico7812

    Key Contributor

  • Validated Member
  • PipPipPip
  • 68 posts
  • United States

I have a very large shapefile (1,000,000 + features). I'm trying to select those features in that layer where an attribute is greater than 0 (approximately 250,000 of them). Every time I try to do a Select by Attribute my ArcMap seemingly freezes. I left the program running over the weekend to see if it was just taking a long time, but come Monday it was still just sitting there. Does anyone know A) if I am trying to work with too big of a file for ArcMap, B) how to increase the amount of processor/RAM capacity dedicated to ArcMap, or C) know a better way to select such a large number of features from a very large shapefile? I should also mention that the shapefile is polyline that has a table join with a CSV file where the data is stored. Only 6 attribute fields in total though. Thanks so much for your help!

Eric

#2
Craig Jr

Craig Jr

    Contributor

  • Validated Member
  • PipPip
  • 15 posts
  • Gender:Male
  • Location:Cold Spring, NY
  • United States

You may want to try either putting a definition query on the layer ("Attribute" > 0) this will only display the data you are looking for. Another option would be to export the layer with the join to a .shp file. I have noticed quite a bit of lag time working with joined data in ArcGIS. Working with large datasets may be slow but it shouldn't be over a weekend slow.

#3
cmdrico7812

cmdrico7812

    Key Contributor

  • Validated Member
  • PipPipPip
  • 68 posts
  • United States

You may want to try either putting a definition query on the layer ("Attribute" > 0) this will only display the data you are looking for. Another option would be to export the layer with the join to a .shp file. I have noticed quite a bit of lag time working with joined data in ArcGIS. Working with large datasets may be slow but it shouldn't be over a weekend slow.


How do you do a definition query on the layer?

#4
cmdrico7812

cmdrico7812

    Key Contributor

  • Validated Member
  • PipPipPip
  • 68 posts
  • United States

You may want to try either putting a definition query on the layer ("Attribute" > 0) this will only display the data you are looking for. Another option would be to export the layer with the join to a .shp file. I have noticed quite a bit of lag time working with joined data in ArcGIS. Working with large datasets may be slow but it shouldn't be over a weekend slow.


How do you do a definition query on the layer?


Nevermind, I found it.

#5
James Hines

James Hines

    James Anthony Hines

  • Validated Member
  • PipPipPipPipPipPip
  • 543 posts
  • Gender:Male
  • Location:Centreville, Nova Scotia
  • Interests:Cartography, Philosophy, Politics, Psychology, Economics, Occultism, Spiritualism
  • Canada

Under Layers in ArcMap

> select layer
> right click & select properties
> There is literally a "Definition Query" Tab so select that
> Under the form select the "Query Builder" button
> Enlarge the attachment for an example:

Attached File  Query.jpg   57.71KB   15 downloads

"There is much beauty that we fail to see through our own eyes teeming with life forms that give us that perception of our reality.  Leaves on the trees blowing gently in the wind, or scarily, the waves pounding through high surf, or lightly on a warm summer’s day; that opportunity to sit or swim in the water on a white beach.   That comfort to shout, “The universal conscious do you hear me?  I am alive, guide me dear logos towards the path of rightnesses.”  Earned what has been kept, no longer to be absorbed into a life filled with cold damn winds and  that stubborn fog clouding  my vision with nothing but darkness."


#6
James Hines

James Hines

    James Anthony Hines

  • Validated Member
  • PipPipPipPipPipPip
  • 543 posts
  • Gender:Male
  • Location:Centreville, Nova Scotia
  • Interests:Cartography, Philosophy, Politics, Psychology, Economics, Occultism, Spiritualism
  • Canada

You may want to try either putting a definition query on the layer ("Attribute" > 0) this will only display the data you are looking for. Another option would be to export the layer with the join to a .shp file. I have noticed quite a bit of lag time working with joined data in ArcGIS. Working with large datasets may be slow but it shouldn't be over a weekend slow.


How do you do a definition query on the layer?


Nevermind, I found it.

Guess I didn't need to show that example after all.

"There is much beauty that we fail to see through our own eyes teeming with life forms that give us that perception of our reality.  Leaves on the trees blowing gently in the wind, or scarily, the waves pounding through high surf, or lightly on a warm summer’s day; that opportunity to sit or swim in the water on a white beach.   That comfort to shout, “The universal conscious do you hear me?  I am alive, guide me dear logos towards the path of rightnesses.”  Earned what has been kept, no longer to be absorbed into a life filled with cold damn winds and  that stubborn fog clouding  my vision with nothing but darkness."


#7
cmdrico7812

cmdrico7812

    Key Contributor

  • Validated Member
  • PipPipPip
  • 68 posts
  • United States

So it still freezes/takes forever when doing the definition query. Is there a way to join the attribtues from my table to the shapefile DBF outside of GIS and then open the shapefile in GIS and have them already be together? As oppossed to joining them in GIS and then exporting a new shapefile since that process is the one I ran all weekend and it still didn't finish. Thanks.

#8
Laura Miles

Laura Miles

    Master Contributor

  • Validated Member
  • PipPipPipPip
  • 116 posts
  • Canada

Did you say you are working in ArcMap? I find that doing things in ArcCatalog can speed them up a bit.
Laura

#9
cmdrico7812

cmdrico7812

    Key Contributor

  • Validated Member
  • PipPipPip
  • 68 posts
  • United States

Did you say you are working in ArcMap? I find that doing things in ArcCatalog can speed them up a bit.
Laura


I've never really used ArcCatalog much except for basic tasks. Can I do joins and such similar to ArcMap in it?

#10
cmdrico7812

cmdrico7812

    Key Contributor

  • Validated Member
  • PipPipPip
  • 68 posts
  • United States

Did you say you are working in ArcMap? I find that doing things in ArcCatalog can speed them up a bit.
Laura


I've never really used ArcCatalog much except for basic tasks. Can I do joins and such similar to ArcMap in it?


So I did Make Table View from my shapefile (since Creating a Join would only accept layer files), then I joined my table to it, but I don't know what the output is from these two functions nor the destination.

#11
radek

radek

    Contributor

  • Validated Member
  • PipPip
  • 39 posts
  • Gender:Male
  • Switzerland

try http://gis.stackexchange.com/

#12
Laura Miles

Laura Miles

    Master Contributor

  • Validated Member
  • PipPipPipPip
  • 116 posts
  • Canada

Try "Join Field", that should give you the ability to join a shapefile to your table...you didn't say what format that is in? Then you should be able to use "Select" to create a new feature class.
Laura

#13
Dustin Edge

Dustin Edge

    Contributor

  • Validated Member
  • PipPip
  • 31 posts
  • Australia

Why not move your data into a geodatabase and then create an Attribute Index (ArcToolbox - Data Mngt - Indexes). Those two steps would speed up your process....
Kind Regards,

Dustin


Dustin Edge
Snr GIS Analyst
AECOM Australia

#14
François Goulet

François Goulet

    Ultimate Contributor

  • Validated Member
  • PipPipPipPipPipPip
  • 688 posts
  • Gender:Male
  • Location:Mille-Isles, Qc
  • Interests:Cartography, History, Graphic Design and almost everything else...
  • Canada

Another option would be to export the layer with the join to a .shp file. I have noticed quite a bit of lag time working with joined data in ArcGIS.


I'd go with that too. If you export your data with a joined .csv file, the attribute of the .csv will be in the output shapefile. Should be easier and faster then.

It's not suppose to froze with "only" a couple hundred thousand features. I did it with millions. It was slow (5-10 minutes?) but it worked.

Creating an index would be a good idea too.

#15
cmdrico7812

cmdrico7812

    Key Contributor

  • Validated Member
  • PipPipPip
  • 68 posts
  • United States

Try "Join Field", that should give you the ability to join a shapefile to your table...you didn't say what format that is in? Then you should be able to use "Select" to create a new feature class.
Laura


My table is in CSV format. Would another format work better? I have Office 2010 so Excel can't save DBF files.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

-->