14 replies to this topic

[radek]

Hi All,

I am working with large national road network dataset [800k edges] build from TeleAtlas data.

I wanted to build OD Matrix for a set of 116k origins and 154k destinations. Each location should have info for 5k destinations eventually.

All datasets are stored in single file geodatabase.

Needless to say my attempt culminated in 'out of memory' msg :/

I followed the tips on esri forum but it didnt work out for me..

Is task like that possible to solve thru Network Analyst at all? Does anyone of you have experience/tips for solving such large problems on network? Are there any software alternatives?

Thanks a lot for help :]

[Geographic Techniques]

That's a lot of data for NA to handle. You'll need to process your data in much smaller subsets, then combine. A quad core processor with lots of memory may help.

I feel your pain, especially after the lengthy time it takes to load the data, then to find out it fails.

Good luck.

Doug

[MapMedia]

Have you tested the analysis settings on a subset? Maybe 100 destinations - so you can run a quick analysis, dial in your results? Then break down the larger job into small sections, maybe 10k, and see if that works, if not go smaller.

You can also set the memory limits of NA on your system to handle workloads in chunks. Network Analyst Memory Fix

[radek]

thanks for your feedback guyys :]

Doug,

i'm working on 8core machine with 8G of RAM and fast HD.. cant get anything more than that :/

Doug and MapMedia,

smaller subsection of origins would work fine.. is there any neat way of looping thru them automatically?

thanks again
r

[MapMedia]

smaller subsection of origins would work fine.. is there any neat way of looping thru them automatically?

One at a time I guess. I just ran a 'closest facility' analysis from center points of every US zip code, so broke the zip codes into batches of 3,000 and ran them sequentially, one after the other. I prefer this way for a few reasons, one being QA QC as I could interrogate the results before proceeding. The drive time/distances were largely equal to Google Map directions.

Bumping up virtual memory and making sure coordinate system of all layers will help.

Good luck.

[Geographic Techniques]

smaller subsection of origins would work fine.. is there any neat way of looping thru them automatically?

One at a time I guess. I just ran a 'closest facility' analysis from center points of every US zip code, so broke the zip codes into batches of 3,000 and ran them sequentially, one after the other. I prefer this way for a few reasons, one being QA QC as I could interrogate the results before proceeding. The drive time/distances were largely equal to Google Map directions.

Bumping up virtual memory and making sure coordinate system of all layers will help.

Good luck.

I agree...One session at time is the key. Unfortunately, NA can be quite memory consuming. You may want to close and re-start ArcGIS before each processing attempt if you continue to have memory issues. I would stick with up to 25K batches, although but you may do better with your power machine. I've been able to process up to 75K routes on my quad machine with 8 G's (point to point routes). Also, clean data is important...And eliminate any outliers that may induce extremely long distance processing (that will kill your memory).

Doug

[radek]

Thanks Doug :]

[dsl]

You might want to look into using ArcObjects with VBA to do this processing. I've done that before and it worked fine.

This is an example for 9.2 using VB6 which could be converted to VBA.

Python might also speed things up.

Cheers,
David

[radek]

Thanks David..
I'm trying to cook some python to do that.. so will be most likely back here for some more help..

[frax]

Does NA use more than one core at a time? By breaking up the subsets and running them through a script (e.g. python/VBA) you might be able to run things in parallel, but then the threads might fight over memory too...

[radek]

Hugo,

It doesn't seem for me like NA is optimized for multicore processing at the moment..

So it would be great if python could help here..

Thanks for reply :]

[MapMedia]

Hugo,

It doesn't seem for me like NA is optimized for multicore processing at the moment..

So it would be great if python could help here..

Thanks for reply :]

Yet you can tell ArcMap to break up the work in chunks by modifying the HSKEY parms. I use this when running complex closest facility or travel time queries. I haven't set it back so its default.

[frax]

Speaking of multicore processing, since ArcGIS is only a 32-bit application, it can't use more than the addressing limit (that is 2 GB, isn't it?) so that means that you should be able to process things in parallell without the threads fighting over memory. The bottleneck might be disk if it needs to read/write a lot of data in parallell.

MapMedia - what are the hskey params, never heard of that... Breaking up processing in chunks does not necessarily mean processing in parallel, by the way.

[radek]

MapMedia

HSKEY sounds interesting for me as well.. would you mind sharing some tips on that?

Thanks

[radek]

Hello All,

I am back with some findings and couple new problems problems :[

I managed to put the task into python script that adds origins one at a time. Tested it with small dataset and it works without a flaw.

When I run it on larger dataset though, I got some strange error messages in IDLE looking like that:

Error
Location "360323" in "Origins" is on a non-traversable network element.
Location "2876122" in "Destinations" is on a non-traversable network element.
< and several dozen simmilar lines about destinations...>

So I went back to do some manual checks suspecting errors in dataset..

However, when I loaded the same destinations with this one particular origin manually in ArcMap - the OD matrix worked without any problems and solved everything :/

I did noticed tho that all these problematic locations lie right next to the street network with strange attribute that look like:

Attribute | Along | Against
____________________________________

Oneway | Restricted | Restricted

So I suspect that this might be causing the drama :/

Is there any way to solve this issue?

Thanks again for your help..