Jump to content

 
Photo

"Join Line" filter in MAPublisher VERY slow?

- - - - -

  • Please log in to reply
14 replies to this topic

#1
David Medeiros

David Medeiros

    Hall of Fame

  • Validated Member
  • PipPipPipPipPipPipPip
  • 1,082 posts
  • Gender:Male
  • Location:Redwood City CA
  • Interests:Cartography, wood working, wooden boats, fishing, camping, overland travel, exploring.
  • United States

When joining river segments by segment name I get intolerably long wait times and the end result usually has multiple layers of lines (new joined lines under segments). Does anybody here have any tips for using the MAPublsiher Join filter to speed it up and keep it clean?

Dave

MAPublisher 7.2 on Ill CS, joining 2566 segments.

GIS Reference and Instruction Specialist, Stanford Geospatial Center.

 

www.mapbliss.com

 


#2
Hans van der Maarel

Hans van der Maarel

    CartoTalk Editor-in-Chief

  • Admin
  • PipPipPipPipPipPipPip
  • 3,868 posts
  • Gender:Male
  • Location:The Netherlands
  • Interests:Cartography, GIS, history, popular science, music.
  • Netherlands

How long is intolerably long? Back in Ye Olde Days (MAPublisher 4) it was extremely slow, but I've found that in later versions it's really improved a lot. Any way you can pre-process your data before loading it into MAPublisher?

It should make 2 layers: the original one and a new one for the joined lines.
Hans van der Maarel - Cartotalk Editor
Red Geographics
Email: hans@redgeographics.com / Twitter: @redgeographics

#3
ELeFevre

ELeFevre

    Hall of Fame

  • Associate Admin
  • PipPipPipPipPipPipPip
  • 1,049 posts
  • Gender:Male
  • Location:Louisville, Colorado USA
  • Interests:Cartography, musical instruments, reading, hiking, craft beer
  • United States

In addition to Hans' comments, I would lighten up the layer as much as you can by removing unnecessary attributes and nodes (never hurts). Also, I would start by running a test on very small section of the layer just to get your settings right, and then go from there. As a last resort you could think about breaking the layer into a several sections. What scale are you working at? Hope this helps.



#4
David Medeiros

David Medeiros

    Hall of Fame

  • Validated Member
  • PipPipPipPipPipPipPip
  • 1,082 posts
  • Gender:Male
  • Location:Redwood City CA
  • Interests:Cartography, wood working, wooden boats, fishing, camping, overland travel, exploring.
  • United States

How long is intolerably long? Back in Ye Olde Days (MAPublisher 4) it was extremely slow, but I've found that in later versions it's really improved a lot. Any way you can pre-process your data before loading it into MAPublisher?

It should make 2 layers: the original one and a new one for the joined lines.


Well, I let it run overnight (about 10 hours) and when I got up it was less than 25% done! There are around 2500 segments to be joined by segment name. Its a river layer. I ejected as much of the line work as I could; segments out of frame and lake segments I'm not joining.

GIS Reference and Instruction Specialist, Stanford Geospatial Center.

 

www.mapbliss.com

 


#5
David Medeiros

David Medeiros

    Hall of Fame

  • Validated Member
  • PipPipPipPipPipPipPip
  • 1,082 posts
  • Gender:Male
  • Location:Redwood City CA
  • Interests:Cartography, wood working, wooden boats, fishing, camping, overland travel, exploring.
  • United States

In addition to Hans' comments, I would lighten up the layer as much as you can by removing unnecessary attributes and nodes (never hurts). Also, I would start by running a test on very small section of the layer just to get your settings right, and then go from there. As a last resort you could think about breaking the layer into a several sections. What scale are you working at? Hope this helps.


The scale is 1:925,000, page size is 8.5x11. What do you mean by remove unnecessary attributes and nodes? Should I delete extra columns in the attributes table? I'v run a simplify filter in MAPub to reduce the number of nodes, but it doesn't take away very many. I need to hold the integrity of the line shape since these are rivers in a watershed map.

GIS Reference and Instruction Specialist, Stanford Geospatial Center.

 

www.mapbliss.com

 


#6
pfyfield

pfyfield

    Master Contributor

  • Validated Member
  • PipPipPipPip
  • 100 posts
  • Gender:Male
  • Location:Portland, Oregon
  • United States

Do you have access to GIS software as well? I would first do a dissolve by stream name in Arc to do a lot of the heavy lifting. The line segments would still not carry through points where three or more meet when brought into Illustrator, but the MP filter should then take care of that.

I work with some huge stream datasets and that usually works for me. A few versions of MP ago it would bomb out, but now it works as it should.
Paul Fyfield
Cartographer, Bureau of Land Management
Oregon State Office
Portland, Oregon
pfyfield@blm.gov

#7
ELeFevre

ELeFevre

    Hall of Fame

  • Associate Admin
  • PipPipPipPipPipPipPip
  • 1,049 posts
  • Gender:Male
  • Location:Louisville, Colorado USA
  • Interests:Cartography, musical instruments, reading, hiking, craft beer
  • United States

In addition to Hans' comments, I would lighten up the layer as much as you can by removing unnecessary attributes and nodes (never hurts). Also, I would start by running a test on very small section of the layer just to get your settings right, and then go from there. As a last resort you could think about breaking the layer into a several sections. What scale are you working at? Hope this helps.


The scale is 1:925,000, page size is 8.5x11. What do you mean by remove unnecessary attributes and nodes? Should I delete extra columns in the attributes table? I'v run a simplify filter in MAPub to reduce the number of nodes, but it doesn't take away very many. I need to hold the integrity of the line shape since these are rivers in a watershed map.


Yes, I think in general it's beneficial to remove any unnecessary columns when you're working in Illustrator and MP. I don't know if this will speed up the join process per se, but you should see an overal improvement in Illustrator, or at least that's my experience. Just leave a column that you can rejoin to in case you need the columns at a later time.



#8
David Medeiros

David Medeiros

    Hall of Fame

  • Validated Member
  • PipPipPipPipPipPipPip
  • 1,082 posts
  • Gender:Male
  • Location:Redwood City CA
  • Interests:Cartography, wood working, wooden boats, fishing, camping, overland travel, exploring.
  • United States

I have access to a current version of ARC Map at school, what filters or actions should I use on the data set in ARC Map to clean it up and join the lines?

dave

GIS Reference and Instruction Specialist, Stanford Geospatial Center.

 

www.mapbliss.com

 


#9
pfyfield

pfyfield

    Master Contributor

  • Validated Member
  • PipPipPipPip
  • 100 posts
  • Gender:Male
  • Location:Portland, Oregon
  • United States

Dissolve on stream name. The tool, I believe, is under data management-generalization (I thankfully have a few days off and am not sitting in front of my work computer at the moment).

The tool will ask you for a layer and attribute to dissolve on. This will create a layer with only the streamname attribute and a single record for all arcs with that attribute value (depending on whether you chose the multipart option- play around with it).

Generally speaking, the MAPublisher tools cannot always handle the huge datasets produced by GIS. It's my favorite software ever, but it has limitations, mostly, I think, because of Illustrator. It's just not designed for that many nodes, arcs etc. It's a good practice to "preprocess" your data in GIS. Project, dissolve, clip, whatever, then import using MP.
Paul Fyfield
Cartographer, Bureau of Land Management
Oregon State Office
Portland, Oregon
pfyfield@blm.gov

#10
Andrew Patterson

Andrew Patterson

    Key Contributor

  • Validated Member
  • PipPipPip
  • 99 posts
  • Gender:Male
  • Location:Toronto, ON
  • Canada

Join Lines is a funny thing in MAPublisher. As Hans mentioned, it wasn't very well treated a few years ago but it's so useful that we've been trying with several releases to get it to a place where it not only does Things Right™, but does it Reasonably Fast™. We rewrote it in 7.0 and worked on it over a few point releases but decided that in 8.0 we had to go back to the drawing board. The algorithm we were using in the 7.x version was very clever, but not every well suited as it turns out to how Illustrator describes paths.

The imminent 8.0 release has another take on the Join Lines filter. We greatly simplified the algorithm and kept an eye on memory usage & speed. Hopefully it should do the job much faster than it's predecessor (it certainly seems to in our internal testing).

David, if you post the file & details on what you were running, we'll give it a shot in 8.0 and see how it performs. We're always interested to see how something like this works 'in the real world'.
Andrew Patterson
Lead Software Architect
Avenza Systems Inc.

email: andrew@avenza.com
phone: 416.487.5116

#11
David Medeiros

David Medeiros

    Hall of Fame

  • Validated Member
  • PipPipPipPipPipPipPip
  • 1,082 posts
  • Gender:Male
  • Location:Redwood City CA
  • Interests:Cartography, wood working, wooden boats, fishing, camping, overland travel, exploring.
  • United States

Thanks for the reply Andrew, sorry I didn't see it until Kimi PMd me a bit ago.

The data set is 25mb, I'll email it to you or Kimi if thats not too large?

It is a river line layer from the National Hydrography Data set (NHD Flowline). I cleaned out all of the lines with no names and was attempting to join the remaining segments by "GNIS_NAME".

I tried various settings, including [Proximity set to 0, and "Close Segment Gaps" unchecked] and [Proximity set to 2 meters, "Close Segment Gaps" checked]. I have run the Simplify filter on the lines and reduced the table contents to just the GNIS_NAME and FTYPE columns. In all cases the join began smoothly but then bogged down on longer lines. When I eventually cancel the operation and inspect what has been done so far, I find several lines joined properly but they usually also have duplicate segments pasted over them so that it would appear as if it had not been joined unless you deleted the segment to reveal the joined line beneath (same this has happened with other Joined layers I'm working with).

In a smaller file containing a Roads layer I successfully completed the Join function (after a long wait). The results were ok but showed the duplicate segments mentioned above and some segments that had huge gaps closed even though the "Close Gaps" was unchecked.

Let me know how to get you the file and if you need to know anything else.

David


PS, if your looking for any input on future iterations of the JOIN feature, I'd suggest that it actually join lines at their end point as opposed to just making them compound paths as seems to be the case in MAPub 7.2 (maybe this has already been changed).

GIS Reference and Instruction Specialist, Stanford Geospatial Center.

 

www.mapbliss.com

 


#12
Ted Florence

Ted Florence

    Master Contributor

  • Validated Member
  • PipPipPipPip
  • 193 posts
  • Gender:Male
  • Location:Toronto
  • Interests:Making great mapping software.
  • Canada

As an update to this thread on the MAPublisher Join Lines function please note the following:

We have ran some tests on the file that David sent and the results were even better than expected.

Here is what we found:

1. Using MAPublisher 7.2 with Illustrator CS we were more or less able to duplicate David's results. After about 2 hours it appeared that Illustrator was hung so we gave up. We tried again and this time we received an Adobe error message. Upon clicking OK in the error message dialog Illustrator did not crash or hang so we checked to see if any portion of the lines had been joined and found that only 364 segments were joined out of 35668 segments.

2. Using MAPublisher 7.6.3 (the latest released version) with Illustrator CS3 we had terrific results. Not only did the lines join successfully in only 90 seconds, the result was that there were originally 35668 segments and the dataset was reduced to 4055 segments after the Join Lines operation.

3. We then tried again using the upcoming MAPublisher 8.0 with Illustrator CS4 and achieved the same positive results as in #2, above, with MAPublisher 7.6.3 and CS3.

So clearly, whatever issues may have been present in the earlier 7.2 version and possibly prior have been addressed in subsequent releases and Join Lines now functions very well.

With all due respect, David, you are using a version of MAPublisher that is 2 years and 3 releases old and a version of Illustrator that is even older (5 + years) and while certainly you are right to have reasonable expectations of functional performance it is important to note that we have long since recognized that this function required improvement and responded to that with fixes in later releases.

So as it stands now, Join Lines is head and shoulders above what it used to be and performs excellently on your file.
Kimi will send you the joined file she created so you can use it accordingly.

And for all of you reading this, the thing to take away with you is that at Avenza we are always working hard to improve our products and respond to user requests and issues and this particular case is another example of why it is always a good idea to keep your MAPublisher maintenance in force and up to date.

Regards
Ted

Ted Florence

Avenza Systems Inc.

When Map Quality Matters ®

www.avenza.com

 

Cartographic and spatial imaging solutions for Adobe Creative Suite

Mobile mapping solutions for using, selling and distributing maps to mobile devices

 

 

 

mp_logo.gif    gi_logo.gifpdf-maps-icon.png
 


#13
Andrew Patterson

Andrew Patterson

    Key Contributor

  • Validated Member
  • PipPipPip
  • 99 posts
  • Gender:Male
  • Location:Toronto, ON
  • Canada

PS, if your looking for any input on future iterations of the JOIN feature, I'd suggest that it actually join lines at their end point as opposed to just making them compound paths as seems to be the case in MAPub 7.2 (maybe this has already been changed).


I believe this is how it works in 8.0 -- though that may already have been changed in 7.6 (I wasn't actually in on writing that filter). I think it it only resorts to compounds when it can't do a straight end-to-end join (i.e. when you tell it not to close gaps and there is one).
Andrew Patterson
Lead Software Architect
Avenza Systems Inc.

email: andrew@avenza.com
phone: 416.487.5116

#14
David Medeiros

David Medeiros

    Hall of Fame

  • Validated Member
  • PipPipPipPipPipPipPip
  • 1,082 posts
  • Gender:Male
  • Location:Redwood City CA
  • Interests:Cartography, wood working, wooden boats, fishing, camping, overland travel, exploring.
  • United States

Thanks again to the Avenza team for their help and generosity.

Ted, I would love to be on the most current version of MAPublisher and Illustrator. But CS is what I already had available to me when I took this contract job. The organization I'm working for agreed to buy me the MAPublisher plugin, but not an Illustrator upgrade, so this is what I'm stuck with for now. It's an amazing piece of software even at 2 years old, and you should know that prior to this I was using Ill 10 with an even older MAPub version so this is still a big improvement.

I also want to be sure you guys understand that I'm probably going to post more MAPub questions in the future, here and directly to Avenza. I hope you're not reading these as public criticisms of the product that you guys need to defend, I'm just looking for help from fellow cartographers on how to make the most of what I have.

David

GIS Reference and Instruction Specialist, Stanford Geospatial Center.

 

www.mapbliss.com

 


#15
Ted Florence

Ted Florence

    Master Contributor

  • Validated Member
  • PipPipPipPip
  • 193 posts
  • Gender:Male
  • Location:Toronto
  • Interests:Making great mapping software.
  • Canada

Thanks for the note, David, and for your understanding.
And no harm taken from your earlier posts.
Also, thank you for your kind comment about MAPublisher, "It's an amazing piece of software even at 2 years old".
Just keep in mind that we are constantly and continually working to make MAPublisher the best it can be so even if an when there is something that under-performs, you and all the other MAPublisher users out there can rest assured that we are on top of it.
Feel free to continue to post to Cartotalk as the vast populace of members do know their stuff but also keep in mind that as you do have maintenance you can also contact us directly with particular technical issues.
Regards
Ted


Thanks again to the Avenza team for their help and generosity.

Ted, I would love to be on the most current version of MAPublisher and Illustrator. But CS is what I already had available to me when I took this contract job. The organization I'm working for agreed to buy me the MAPublisher plugin, but not an Illustrator upgrade, so this is what I'm stuck with for now. It's an amazing piece of software even at 2 years old, and you should know that prior to this I was using Ill 10 with an even older MAPub version so this is still a big improvement.

I also want to be sure you guys understand that I'm probably going to post more MAPub questions in the future, here and directly to Avenza. I hope you're not reading these as public criticisms of the product that you guys need to defend, I'm just looking for help from fellow cartographers on how to make the most of what I have.

David


Ted Florence

Avenza Systems Inc.

When Map Quality Matters ®

www.avenza.com

 

Cartographic and spatial imaging solutions for Adobe Creative Suite

Mobile mapping solutions for using, selling and distributing maps to mobile devices

 

 

 

mp_logo.gif    gi_logo.gifpdf-maps-icon.png
 





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

-->