15 replies to this topic

[josie]

Hello everyone,

So, here at work more and more people are beginning to use GIS. Before it was only me and I had my data stored on my own computer in my own logical way . However, now the data must be shared and relocated to somewhere accessible for everyone. Thus, I got thinking, how do other people approach storage and arrangement of data? In GIS we often acquire a great deal of data and it is very important that it is stored correctly.

I myself keep a metadata sheet in every folder so that I know what is in there, where in the world it is, which projection, where I got it from etc. I am also working on a searchable database so that my fellow workers can search for data based on a number of criteria.

So, any advice, comments or general rules of practise? I would be glad to hear what people think about this.

Thanks

Josie

[josie]

I see noone is thus far enticed by the subject of data storage...

[rudy]

I see noone is thus far enticed by the subject of data storage...

Maybe because no one is quite awake yet . . . .

Any system that you use has to make sense to pretty much everyone . . . what makes sense to you might not make sense to anyone else. I've come across other people's data and files that to me seemed to have been stored in a haphazard manner. Even now, the system we have in place wouldn't be the way I would have set it up but it is understandable and it has been used for years by a number of people so I wouldn't go about changing it.

I really depends on what your data is. Some people store by subject, others by geography. One might make more sense than the other, depending on your data and how it is to be used. In any case, all users need to understand the data organization. You might also want to consider have only one administrator . . . perhaps you. If others start changing or deleting data without knowing the full story, it may lead to trouble.

[Jimbo]

Back in college I numbered the data by the step I was on, so as an example:

1_Source_Rd_Line
2_Clip_Rd_Line
3_Merge_Rd_Line
4_Projected_Rd_Line
5_Generalized_Rd_Line
6_Final_Rd_Line

All of the files were imported into a geodatabase. But when I think about it maybe it was not the best way of doing it, but at the time it was convient.

[Jimbo]

I see noone is thus far enticed by the subject of data storage...

Maybe because no one is quite awake yet . . .

To wake up get a coffee & play something loud & wild like Slip Knot. Now that will wake you, & just maybe your neighbors.

[jrat]

We use ESRI SDE for data storage. It is a validated versioned space that has one administrator. Editing privlages are tightly controled, and updates are verified by the administrator. Our data is organized by raster and vector, then the vector data is coded in the name to identify what it is. We only use one datum and projection so there is no need to worry about this. In ESRI software every feature class has its own metadata. The SDE works well, mutiple users/editors are supported. Our internal policies push me to develop data on my own computer then once it is complete our admin will add it to the SDE. As such I have alot of data on my computer stored in a very haphazard way. It is something I have alwayse wanted to take time an clean up but I don't feel like fixing all the dead links in my maps.

[josie]

Yeah, I know what you mean. I really don't want to move things around, I have kind of grown accustomed my haphazard style of arrangement

I agree with Rudy that other people changing data can become extremely troublesome. As GIS is new for everyone else here, they do not really understand how important it is to store it efficiently and not just dump it wherever there is some space. I think allocating an administrator is a really good move. I think one persons system is much better than a number of peoples all rolled in together.

[François Goulet]

Our data structure in mainly influence by our biggest client (Hydro-Quebec, Quebec's public electricity company). Since it's the biggest player in GIS in the province, almost everyone use a system close to them.

The dataset are sorted by thematic (Hydrography, Equipment, Hypsometry, ...) and the name are coded so every consultant could received data from another and know what it's about.

As an example, 3165_cm_vol_xxx_090811.shp means data created for the project 3165, created by Carto-Media (our company), containing linear feature for transportation (vo = "voie de communication" or roads... and "l" is for line ("p" for points and "s" for surfaces)), "xxx" should be a short name for the data and finally the version date.

It's sometime a pain in the *ss to follow all the norms, but it's easier after... even though I don't follow it at home... there, I'm my own boss

[Hans van der Maarel]

I've got all my general data sources stored in folders with names that refer to the kind of data it is. Project data goes in the project folder, using whatever naming system the customer has thought up.

Right now I'm not maintaining any metadata, but since I don't actually produce any data myself this is not a very big issue. Metadata is important though and if you use any 3rd party solutions for maintaining that, doublecheck whether you can convert the metadata between formats. By pure coincidence one of my clients asked me today to investigate whether their metadata could be converted from [one metadata add-on for a major GIS product] to [another metadata add-on for that same major GIS product]. The official suggestion from the vendor is: "probabely it's best to just re-enter all metadata".

[removes lengthy discussion on why storing metadata in XML is not necessarily 'open' or 'easy to use']

Anyway, iirc there was an interesting presentation at NACIS in Madison (2006) about metadata, that might be an interesting one to check out.

I've long been considering setting up a spatial database holding all my vector datasets (and again in my case these would be "general" datasets, such as VMAP). For large amounts of data they're often a smart choice. They'll also allow you to restrict access to the data, so that for example only the data manager is allowed to update or delete features.

[Robert2009]

If they don't backup their work it is time-consuming to do the work all over. What I do when I make some soil change daily I use Flash drive to backup them using the date like this

June_12_2009_STAFF.
June_13_2009_STAFF.
June_14_2009_STAFF.
so on...

Then from there, I delete June_12_2009_STAFF to keep 3 days backup...

There is one you can do is use the WINZIP and compress it into like DVD.
Here what you can do first.

Right click on the folder to "add to file.zip"
It will open the dialog box
Go to Split Zip File. There is a list that you can choose from. Select "sized for DVD-ROM"
I have had larger DVD file size and use it to compress it to burn them to my DVD. Really neat.

or Buy a external hard drive that can backup their work etc...

Good Luck

Robert

[frax]

I think you need to look at the organizational needs more closely, and maybe even talk to your ESRI sales rep about that (which does not necessarily mean you have to follow that).

I think a start might be to set up a file database on a network server, and establish naming conventions and metadata.

[Hans van der Maarel]

I think you need to look at the organizational needs more closely, and maybe even talk to your ESRI sales rep about that (which does not necessarily mean you have to follow that).

And what if you're not using ESRI software?

[frax]

Sorry I didn't mean to be a fanboy, I thought I remembered that Josie was living in ESRI-land (and they have various levels of workgroup/enterprise solutions). If you use another platform which sells enterprise/workgroup services/software, talk to the sales rep of the software! Or consider the alternatives, but that might be too much work, if you want things to grow organically...

[josie]

Yes, well (As Hans pointed out) I am infact a Manifold user, though it is interesting to learn about what resources people use for ESRI. I think to at least establish naming conventions (as Frax mentioned) would indeed be a good start. I think the main thing that people are agreeing on is that infact it really depends on what you do, who your clients are etc, there are no stone conventions. Just a sense of logic.

My metadata is stored as html documents (maybe this isn't the best way). I try and put a metadata sheet in every folder (when I remember to). It contains the extents of the data, type of data, source of data, location and a handful of other things.

I am hoping to build a spatial database for our storage so this discussion is very helpful when I think of how to approach it.

At home I also arrange things a bit different, but it is a real mess for anyone else to try and understand Maybe this little endeavour will give me some enthusiasm to change things around there too (though I doubt it)!

[Emily Martin]

I just completely overhauled my company's data storage - that was a job and a half! Not just GIS datasets either, it was the whole shebang. The most irritating and tedious part of it was updating links, sources and hyperlinks in everything GIS-related, which took almost as long as the workover. But when circumstances allow, such as reorganising datasets you have at home, there can be a better way...

The trick is to do as little work as possible. Decide on your new naming convention and filing system and document it, storing it at the root level or stuck on your noticeboard for ease of reference and to drum it into your head. Then change all the datasets and folders you can stomach (in my case about four ) to the new style. Then leave the rest.

Whenever you acquire a new dataset, store it in the new fashion. Whenever you update or replace an old dataset, store it in the new fashion. Over time your data will naturally migrate to the spiffy new system. It's the lazy, easy way to do it. Maybe I should call it the Sunday Mornin' method. The beauty of it is that you can start off with a big burst of organisational energy and then let it continue almost unaided when you get fed up. I think this is how a lot of people operate anyway, but it's surprisingly effective if you apply it with gusto and change the less frequently updated data first.

Emily