Jump to content

 
Photo

Count Pixels

- - - - -

  • Please log in to reply
9 replies to this topic

#1
Unit Seven

Unit Seven

    Legendary Contributor

  • Moderator
  • PipPipPipPipPip
  • 265 posts
  • Gender:Male
  • Location:New Zealand
  • New Zealand

Hi all,

Been a while since I have posted on here but I have been lurking from time to time still. I have a problem that I am sure should be reasonably simple but need a push in the right direction. We have around 51,000 ortho imagery tiles which we want to get a distribution of pure black (0,0,0) pixels for. Ideally I would like out a list of tiles with a count of black pixels contained in that tile. The files are ECW's and due to the lossy nature of the format would prefer not to convert.

I know I can convert to point files and select only points with 0,0,0 and then count the number of points but is there a better way using spatial analyst, FME or GDAL or am I being too simplistic? Open ECW > count black pixels > output ECW Name and pixel count > Repeat.

Any pointers would be great?

Cheers,
Sam
S a m B r o w n

U N I T S E V E N
unit.seven@gmail.com

Miramar, Wellington
N E W Z E A L A N D

#2
Hans van der Maarel

Hans van der Maarel

    CartoTalk Editor-in-Chief

  • Admin
  • PipPipPipPipPipPipPip
  • 3,833 posts
  • Gender:Male
  • Location:The Netherlands
  • Interests:Cartography, GIS, history, popular science, music.
  • Netherlands

I know I can convert to point files and select only points with 0,0,0 and then count the number of points but is there a better way using spatial analyst, FME or GDAL or am I being too simplistic? Open ECW > count black pixels > output ECW Name and pixel count > Repeat.


FME could do it. I'll have to give it a shot to determine how fast it would be though.
Hans van der Maarel - Cartotalk Editor
Red Geographics
Email: hans@redgeographics.com / Twitter: @redgeographics

#3
Hans van der Maarel

Hans van der Maarel

    CartoTalk Editor-in-Chief

  • Admin
  • PipPipPipPipPipPipPip
  • 3,833 posts
  • Gender:Male
  • Location:The Netherlands
  • Interests:Cartography, GIS, history, popular science, music.
  • Netherlands

Okay, so it's possible, but sluggish.

The trick is to use a RasterCellCoercer to create a (center)point for every pixel, then overlay that with the original raster using a PointOnRasterValueExtractor. That will generate a band{}.value list holding the R, G and B pixel values. You can then use a TestFilter to test for all 0 values.

Took 17 minutes to run on a small sample.
Hans van der Maarel - Cartotalk Editor
Red Geographics
Email: hans@redgeographics.com / Twitter: @redgeographics

#4
Michael Schmeling

Michael Schmeling

    Master Contributor

  • Validated Member
  • PipPipPipPip
  • 204 posts
  • Gender:Male
  • Location:Kassel, Germany
  • Germany

GlobalMapper should be able to read ECW. You could write a GlobalMapper script that reads the tiles and exports them as jpeg.
Then use Imagemagick to create a histogram for each image.

I haven't tried this, but it may be faster than FME.
Michael Schmeling
Kassel, Germany
Arid Ocean Map Illustrations
http://maps.aridocean.com
Indie Cartographer
http://www.indiecartographer.com

#5
Michael Schmeling

Michael Schmeling

    Master Contributor

  • Validated Member
  • PipPipPipPip
  • 204 posts
  • Gender:Male
  • Location:Kassel, Germany
  • Germany

Oh, I think gdal_translate can read ECW, too. So you could use it to convert the tiles to jpeg and then run Imagemagick.
Michael Schmeling
Kassel, Germany
Arid Ocean Map Illustrations
http://maps.aridocean.com
Indie Cartographer
http://www.indiecartographer.com

#6
Unit Seven

Unit Seven

    Legendary Contributor

  • Moderator
  • PipPipPipPipPip
  • 265 posts
  • Gender:Male
  • Location:New Zealand
  • New Zealand

Was waiting on your FME reply Hans :D Thanks! It's always just a point in the direction to the right transformers—One day I'll make my fortune writing a GIS translation dictionary. Just a book that lists different processes and what they are called by each company. No wonder I couldn't find them though—FME 2006 doesn't have these. Time to get my upgrade I think.

Do you remember how small you test was? May be slow but we can spread this across numerous machines so think that may be the one. I'll let you know how it goes with the 176 gig we are looking at :(

Thanks Arid—ImageMagick came up in my searches and was an option with the conversion but was hoping to not duplicate the whole dataset.

A bit further down that track—how to bump all pure black 0,0,0 pixels to 1,1,1 in the ECW's. Again ideally without creating a duplicate dataset? ECW is lossy I believe so I might me expecting a bit much here.

Thanks again guys.
S a m B r o w n

U N I T S E V E N
unit.seven@gmail.com

Miramar, Wellington
N E W Z E A L A N D

#7
Hans van der Maarel

Hans van der Maarel

    CartoTalk Editor-in-Chief

  • Admin
  • PipPipPipPipPipPipPip
  • 3,833 posts
  • Gender:Male
  • Location:The Netherlands
  • Interests:Cartography, GIS, history, popular science, music.
  • Netherlands

Was waiting on your FME reply Hans :D Thanks! It's always just a point in the direction to the right transformers—One day I'll make my fortune writing a GIS translation dictionary. Just a book that lists different processes and what they are called by each company. No wonder I couldn't find them though—FME 2006 doesn't have these. Time to get my upgrade I think.

Do you remember how small you test was? May be slow but we can spread this across numerous machines so think that may be the one. I'll let you know how it goes with the 176 gig we are looking at :(


Definately get an upgrade, FME 2010 offers a lot more raster functionality than 2006, and is faster too. My test area was about 100 sq km of a 1:25k topographic map in GeoTIFF format.
Hans van der Maarel - Cartotalk Editor
Red Geographics
Email: hans@redgeographics.com / Twitter: @redgeographics

#8
Michael Schmeling

Michael Schmeling

    Master Contributor

  • Validated Member
  • PipPipPipPip
  • 204 posts
  • Gender:Male
  • Location:Kassel, Germany
  • Germany

Thanks Arid—ImageMagick came up in my searches and was an option with the conversion but was hoping to not duplicate the whole dataset.

A bit further down that track—how to bump all pure black 0,0,0 pixels to 1,1,1 in the ECW's. Again ideally without creating a duplicate dataset? ECW is lossy I believe so I might me expecting a bit much here.


You could run GDAL and Imagemagick on each tile separately, so you don't need to duplicate the whole dataset. Imagemagick could also be used to change pixel values.
Michael Schmeling
Kassel, Germany
Arid Ocean Map Illustrations
http://maps.aridocean.com
Indie Cartographer
http://www.indiecartographer.com

#9
Hans van der Maarel

Hans van der Maarel

    CartoTalk Editor-in-Chief

  • Admin
  • PipPipPipPipPipPipPip
  • 3,833 posts
  • Gender:Male
  • Location:The Netherlands
  • Interests:Cartography, GIS, history, popular science, music.
  • Netherlands

As it turns out, Safe is reading Cartotalk as well and their raster-guru Dmitri Bagh came up with this solution:

I generate a raster 5000*5000 pixels with random number (between 10K
and 100K) of randomly placed black pixels, and then with my favourite
RasterExpressionEvaluator, nodatasetter and another favourite
RasterCoercer I make vector from only black and calculate the area and
percentage. Depending on that random number of points I get in average
~30 seconds per run.

There is a small difference between the number of black pixels and the
area within a hundred or two (say, 75,200 and 75,350) that must be
rounding issue, I assume, but it is not reflected in the first digits
of the percentage number.


Attached File  BlackPercentage.fmw.zip   11.12KB   44 downloads
Hans van der Maarel - Cartotalk Editor
Red Geographics
Email: hans@redgeographics.com / Twitter: @redgeographics

#10
Unit Seven

Unit Seven

    Legendary Contributor

  • Moderator
  • PipPipPipPipPip
  • 265 posts
  • Gender:Male
  • Location:New Zealand
  • New Zealand

As it turns out, Safe is reading Cartotalk as well and their raster-guru Dmitri Bagh came up with this solution:

I generate a raster 5000*5000 pixels with random number (between 10K
and 100K) of randomly placed black pixels, and then with my favourite
RasterExpressionEvaluator, nodatasetter and another favourite
RasterCoercer I make vector from only black and calculate the area and
percentage. Depending on that random number of points I get in average
~30 seconds per run.

There is a small difference between the number of black pixels and the
area within a hundred or two (say, 75,200 and 75,350) that must be
rounding issue, I assume, but it is not reflected in the first digits
of the percentage number.


Attached File  BlackPercentage.fmw.zip   11.12KB   44 downloads


Thank-you Hans!!

I'll look into this shortly.
S a m B r o w n

U N I T S E V E N
unit.seven@gmail.com

Miramar, Wellington
N E W Z E A L A N D




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

-->