12 replies to this topic

[woneil]

Recently there was an extended discussion of the work published by the National Center for Environmental Analysis and Synthesis (NCEAS), with tome well-deserved praise for it and a few suggestions. See http://www.cartotalk...p...66&hl=NCEAS.

NCEAS makes their data on individual contributors to oceanic degradation available at http://www.nceas.ucs...lMarine/impacts. I have been trying to make use of their shipping data, here summarized in a severely downsampled overview plot:
 NCEAS_shipping.jpg   483.01K   103 downloads

To form their shipping tracks, NCEAS took a mass of mass of reported positions and connected them into coherent tracks using an algorithm. I have a lot of experience in trying to form tracks from scattered and sometimes corrupt position data and can well sympathize with the problems. It can be quite impossible to avoid false tracks.

Close examinaiton of the NCEAS shpping track plots shows much evidence of this. It is particularly easy to see in the region of the Bay of Bengal, Malay Peninsula, and S. Chna Sea, as shown here:
 Track_fan.jpg   206.64K   108 downloads
(I have transformed their plot into equirectangular lat-long coordinates and multiplied it to bring out the patterns despite downsampling.)

We see a fan of individual tracks originating in the Bay of Bengal to the west, near the point at which shipping turns to enter the approaches to the Strait of Malacca, and proceeding thence to the coast of the peninsula. Then on the other side of the peninsula we see tracks that appear to be direct continuations emerging from the coast and proceeding to the main shipping route up the S. China Sea. As a former ship's officer and shipping analyst I am absolutely certain that virturally all of these tracks must be compuational artifacts.

No doubt the NCEAS scientists concluded that suppressing these artifacts would be too laborious and too compromising of other objectives to be prudent in light of their objectives. But I would like to use the plots to illustrate the actual flow of shipping and the dense false tracks detract considerably from this. I don't need to suppress them totally, but I would like to find a way to de-emphasize them while preserving the main flow. Does anyone have any ideas?

[Dennis McClendon]

Hey, you think portaging a container ship is tough? Try carrying an oil tanker across that ridgeline! No matter how many local porters you hire, the load keeps shifting.

Sorry, no real ideas. I suppose if you had access to individual tracks, you could calculate which of those intersect coastlines more than once, and make them a background color little different from the sea.

[Hans van der Maarel]

Will,

I may have a way to do this, but only if it's vector data. Could you send me a small selection of sample data to play with?

Ideally this is something that should be fixed on the source data side of it and to be honest, I can't imagine how these artifacts can exist in the first place.

[BioGeoMan]

Ughhhh!!

I noted in the metadata for the Transformed Data of the Shipping Routes, the authors mentioned this: "Finally, we removed any tracks that crossed land (e.g. a single ship that records its location in the Atlantic and the Pacific would have a track connected across North America)."

It still looks to me like you are right about the source of the artifacts.

No solution...but still thinking.

[woneil]

Will,

I may have a way to do this, but only if it's vector data. Could you send me a small selection of sample data to play with?

Ideally this is something that should be fixed on the source data side of it and to be honest, I can't imagine how these artifacts can exist in the first place.

--------------------

Hans van der Maarel
Red Geographics

Hans,

Thank you.

NCEAS has published the data at
http://www.nceas.ucs...lMarine/impacts.

I know too little about geo data formats to be able to be sure whether any of the forms they make available are vector fomats. Their list includes
Compressed ESRI Grid, GeoTif, Compressed .ASC, FGDC Metadata (.XML)

Are any of these likely to meet the need?

[Hans van der Maarel]

I know too little about geo data formats to be able to be sure whether any of the forms they make available are vector fomats. Their list includes
Compressed ESRI Grid, GeoTif, Compressed .ASC, FGDC Metadata (.XML)

Are any of these likely to meet the need?

None I'm afraid, that's all vector.

But maybe, just maybe, it'd be possible to treat them as elevations and I could filter it out that way. It'd be less reliable that working with the original data.

[frax]

Hans means - that those are all raster formats.

Will, I think you might be layering things a bit too much on this - you are trying to model/derive something from an already model/derived layer without access to the source data. And you are saying that you have doubts about the quality of the first model/derivation...

Maybe you can filter out something out of this, but what will the quality be?

[Hans van der Maarel]

Hans means - that those are all raster formats.

Oops...

I really ought to have some coffee in the morning before posting here...

Yes, I meant to say they're all raster formats. Anyway, I've had a look at the data and there doesn't seem to be a whole lot that can be done about it, other than rejecting all values below a certain treshold. I've loaded the data (Geotiff) into Global Mapper, which sees it as having an elevation range of 0-1 meter. By playing with the 'water level' I can quickly show the effect of rejecting an elevation range.

But even then you've got some serious data quality problems. Take a look at the routes leading in and out of Antwerp for example. It's obvious there's quite some traffic going there, but it appears the ship track data is of much lower accuracy than the coastlines.

Also, and this is rather confusing me, there's places where I would expect a lot of traffic (Rotterdam, Bosporus and Dardanelles) which seem to be quite low compared to other places. But I'm not an expert on this matter

[woneil]

Thanks to everyone, and especially Hans. You've confirmed my suspicions.

I think perhaps I should just resort to the oldest and most selective of filters, the eye. I'll overlay the NCEAS data on the base map in Photoshop, trace the important routes with splines, and stroke the splines. I don't expect it to take a huge amount of work since I don't need great detail. In fact, I want the shipping lanes simple and smooth so they stand out and don't comfuse the reader.

A word regarding the data may help clarify some things. The basic data on ship positions comes from the World Meteorological Organization (WMO) Voluntary Observing Ships' (VOS) scheme. Ships which participate (on a voluntary basis) record and report local surface weather conditions at (nominally) six hour intervals from 0000Z. There are roughly 4,000 ships which participate (out of about 30,000 merchant ships worldwide), so there should nominally be something over 5 million reports each year, each of which among other matters specifies the ship's position. But actual reporting is much spottier -- NCEAS counted fewer than 1.2 million reports in 2004, the peak year of recent history.

The WMO naturally makes its strongest efforts to enlist ships which travel in the more deserted areas, so the sample is biased in this direction. Additionally in most cases reporting is manual and so the mate on watch is liable to skip the report when workload is high, as it is in places like the Stait of Malacca or the English Channel. Moreover, watchkeeping standards on some ships can be fairly lax and reports are not always terribly reliable.

After looking at the results I'm skeptical about whether there was much value in the whole exercise of trying to form tracks at all. While they synthesized some data they also generated a good deal of noise. I wonder whether they would not have been just as well off to take their 1.2 mllion points as a static data set and model effects either on a point diffusion basis or along a nearest-neighbor network.

[Hans van der Maarel]

I think perhaps I should just resort to the oldest and most selective of filters, the eye. I'll overlay the NCEAS data on the base map in Photoshop, trace the important routes with splines, and stroke the splines. I don't expect it to take a huge amount of work since I don't need great detail. In fact, I want the shipping lanes simple and smooth so they stand out and don't comfuse the reader.

One thing you can try is the filtering method I described, then bring the result into Illustrator and do a live trace (automatic raster to vector conversion), see what that does. I haven't tried this and I'm not sure the results will live up to the good old manual way though.

A word regarding the data may help clarify some things. The basic data on ship positions comes from the World Meteorological Organization (WMO) Voluntary Observing Ships' (VOS) scheme. Ships which participate (on a voluntary basis) record and report local surface weather conditions at (nominally) six hour intervals from 0000Z. There are roughly 4,000 ships which participate (out of about 30,000 merchant ships worldwide), so there should nominally be something over 5 million reports each year, each of which among other matters specifies the ship's position. But actual reporting is much spottier -- NCEAS counted fewer than 1.2 million reports in 2004, the peak year of recent history.

The WMO naturally makes its strongest efforts to enlist ships which travel in the more deserted areas, so the sample is biased in this direction. Additionally in most cases reporting is manual and so the mate on watch is liable to skip the report when workload is high, as it is in places like the Stait of Malacca or the English Channel. Moreover, watchkeeping standards on some ships can be fairly lax and reports are not always terribly reliable.

After looking at the results I'm skeptical about whether there was much value in the whole exercise of trying to form tracks at all. While they synthesized some data they also generated a good deal of noise. I wonder whether they would not have been just as well off to take their 1.2 mllion points as a static data set and model effects either on a point diffusion basis or along a nearest-neighbor network.

6 hours is a lot, if a ship is travelling at 15 knots it can cover 166 km in that time. So along coast lines this can really be a problem. Well, look at Antwerp, as I said. Add to that the spotty results and slightly skewed sample set and it would give me some very serious concerns about the usability of this dataset.

[woneil]

This is a draft of the fruits of my manual tracing efforts.
 Shipping_Japan.jpg   334.54K   90 downloads
And here, for comparison, is the original dataset (with country boundaries overlaid).
 Shipping_NCEAS.jpg   309.05K   78 downloads
I'm fully satisfied that my results are superior, at least for my purposes, to anything that could be gotten through manipulation of the NCEAS data. My reader has no need to be distracted by all of the many minor variations in routing, all the routes to countries other than Japan, and the mass of local traffic.

Many thanks to those who gave interesting and helpful advice.

Notwithstanding its problems the NCEAS dataset gives some important information not readily available elsewhere. I think this is a pretty good way to make effective use of it.

One thing you can try is the filtering method I described, then bring the result into Illustrator and do a live trace (automatic raster to vector conversion), see what that does. I haven't tried this and I'm not sure the results will live up to the good old manual way though.

I didn't try this, but I've worked enough with auto-tracing in various forms to feel confident that for something like this it is not nearly as good as manual tracing. It's hard for us to appreciate the incredible volume and comlexity of the calculations our eye-brain system goes through in doing a task like this, and how far beyond the capacity of present computers they truly are.

[Dennis McClendon]

Looks nice, but I can't help thinking that I'd gladly give up all that luscious--but utterly irrelevant--continental relief to find out the names of the ports where those magenta lines are ending up (in Australia and the Arabian Peninsula, for instance).

It's hard for us to appreciate the incredible volume and complexity of the calculations our eye-brain system goes through in doing a task like this, and how far beyond the capacity of present computers they truly are.

Amen, brother.

[woneil]

I'd gladly give up all that luscious--but utterly irrelevant--continental relief to find out the names of the ports where those magenta lines are ending up (in Australia and the Arabian Peninsula, for instance).

I understand the point but it really would not serve my needs in this case to give the names of specific ports because the routes in fact are meant to represent fairly broad classes of routes with a variety of closely-spaced termini. For instance, I deliberately do not want to distinguish between Tokyo and Osaka, or Sidney and Melbourne, let alone among the scores of oil terminals in the Persian Gulf. In any event, no effort has been made to show the tracks with anything approaching that precision.

I do need the relief in order to support my discussion and it doesn't seem sensible to make a separate map for it.