Jump to content

 
Photo

Mosaics Taking too long

- - - - -

  • Please log in to reply
6 replies to this topic

#1
mhampson4

mhampson4

    Contributor

  • Validated Member
  • PipPip
  • 15 posts
  • United Kingdom

Hi

I am mosaicing a series of DEM files in ARCcatalogue (Load data - mosaic) since 10 AM yesterday (21st), I have pulled out the files I need to save processing time and started the process, It took around 12 hours to load the files and is now doing something (It doesnt say what). It is now 1.20 PM on the 22nd, I was wondering if anyone might have an idea how long I can expect to wait or if there is a way to see exactly what the program is doing? If it helps, ARCcatalogue has created 600Gb in files to create this raster (it stopped at 600Gb after loading the files).

I am creating large shapefiles through the feature to 3d function after mosaicing so if necessary I can merge the shapefiles after rather than this mosaicing, however it now seems a waste if this process is actually working, especially as I can leave this machine for several days if that is what it requires. It would take much longer to merge the individual shapefiles.

Thankyou
M.Hampson

#2
Hans van der Maarel

Hans van der Maarel

    CartoTalk Editor-in-Chief

  • Admin
  • PipPipPipPipPipPipPip
  • 3,834 posts
  • Gender:Male
  • Location:The Netherlands
  • Interests:Cartography, GIS, history, popular science, music.
  • Netherlands

Well, unfortunately processing a huge amount of data takes a huge amount of time.

How much source data are you trying to load? And where is it coming from (local drive or network drive)? Are you trying to reproject on the fly as well?

Have you tried other options? I've found that for example Global Mapper is great for working with huge volumes of data and if it's doing something that takes a lot of time it'll tell you when it'll be finished. Another alternative could be FME, it's latest version (released earlier this week) features enhanced support for point clouds and this would be a perfect testcase.
Hans van der Maarel - Cartotalk Editor
Red Geographics
Email: hans@redgeographics.com / Twitter: @redgeographics

#3
mhampson4

mhampson4

    Contributor

  • Validated Member
  • PipPip
  • 15 posts
  • United Kingdom

It is only 8Gb of source data from the local drive split into 4 already mosaiced files from origionally 739 DEM tiles (I found trying to mosaic all the tiles at once caused the machine to crash) , the mosaic however is on an external HDD as I didnt have the space.

I am trying to reproject, Iv read creating pyramids after would have been a good idea to speed it along but now that it has been running for so long restarting may take longer.

Unfortunately it is a university license of ARC and I cant get on any Global mapper machines for that long or get a copy for my own machine. I also cant get hold of FME.

Do you have any idea how long I may have to wait?

#4
Hans van der Maarel

Hans van der Maarel

    CartoTalk Editor-in-Chief

  • Admin
  • PipPipPipPipPipPipPip
  • 3,834 posts
  • Gender:Male
  • Location:The Netherlands
  • Interests:Cartography, GIS, history, popular science, music.
  • Netherlands

It is only 8Gb of source data from the local drive split into 4 already mosaiced files from origionally 739 DEM tiles (I found trying to mosaic all the tiles at once caused the machine to crash) , the mosaic however is on an external HDD as I didnt have the space.

I am trying to reproject, Iv read creating pyramids after would have been a good idea to speed it along but now that it has been running for so long restarting may take longer.

Unfortunately it is a university license of ARC and I cant get on any Global mapper machines for that long or get a copy for my own machine. I also cant get hold of FME.

Do you have any idea how long I may have to wait?


That shouldn't take that long really. You're at the moment just mosaicking, right? What format are your DEM files currently in?

I mosaicked a grand total of 15 Gb of aerial photos (Geotiff) and exported them to an ECW using Global Mapper and that took only about an hour and a half, with the data coming in from and going out to a network drive. So there's definately something wrong with your setup.
Hans van der Maarel - Cartotalk Editor
Red Geographics
Email: hans@redgeographics.com / Twitter: @redgeographics

#5
mhampson4

mhampson4

    Contributor

  • Validated Member
  • PipPip
  • 15 posts
  • United Kingdom

It is just mosaicing. They are .adf format, however I am mosaicing four previously created mosaics, also .adf.

The activity reading and writting to the disk is only at 1 to 2Mb/S, Usually this HDD is fine, it is connected through USB 2.0 and is a relatively new HDD with nothing else on it at the moment.

The problem may be the size of the area over which the DEM files represent (therefore requiring a huge amount of data points to be produced) if the above speed is typical and not blocked by anything this would be unavoidable, It is topography data at 10m intervals for the entire south coast of the UK.

Judging by the read write speed and the fact the program has created 400Gb of the files it is now manipulating, the mosaicing should be done in the next couple of days.

#6
CentralAmericaExpe

CentralAmericaExpe

    Contributor

  • Validated Member
  • PipPip
  • 16 posts
  • Gender:Male
  • Interests:"I do recognize risk, and having recognized risk, I spend a lot of time minimizing it." Tim Severin
  • Canada

It appears (inferring from progression of previous posts) that your hardware is woefully under spec'd for the raster job you are undertaking. I think this is not a software issue.

It has been my experience that most university GIS lab computers (I'm generalizing here based on my experience) I've worked on are poorly maintained, and even worse, are terribly under-powered for all but the most rudimentary course work.

I would approach this from a completely different angle: because your project appears to be research-related, I would simply find one of the several research-driven DEM products already created at super-continental level at various grains (SRTM comes to mind), then cut it down to size using some chop-DEM-to-polygon script of your choice (rather than stitching: a far more CPU cycle-intensive process, work from a super-continental pre-prepared DEM of the flavor of your choice and cut, cut, cut) ...

Good luck,
Derek-
"I do recognize risk, and having recognized risk, I spend a lot of time minimizing it." Tim Severin

#7
mhampson4

mhampson4

    Contributor

  • Validated Member
  • PipPip
  • 15 posts
  • United Kingdom

It appears (inferring from progression of previous posts) that your hardware is woefully under spec'd for the raster job you are undertaking. I think this is not a software issue.


I agree, I left it running and it finished after eight days, the next processing steps have been reasonably quick so all in all it was a long wait but worth it for the more detailed DEM files I could get.

Thankyou for your help




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

-->