The discussion of the phantasm of the "paperless office" left me wondering exactly how much digital storage a surveyor would need to keep, say, twenty-five years worth of data readily accessible, digitally speaking. Yesterday's data acquisiton now occupies about 1 Gb on a USB drive. So, twenty-five years worth may well in the five or six Terrabyte (or more) category (without any laser scanner data, thanks)?
> The discussion of the phantasm of the "paperless office" left me wondering exactly how much digital storage a surveyor would need to keep, say, twenty-five years worth of data readily accessible, digitally speaking. Yesterday's data acquisiton now occupies about 1 Gb on a USB drive. So, twenty-five years worth may well in the five or six Terrabyte (or more) category (without any laser scanner data, thanks)?
Well I've had rolled drafting worksheets, 36 inches by 20 feet, with calculation notes on the margins. Final drawings ink on mylar either 24 x 36 inches or 8.5 x 14 inches. Project was buried utility easement acquisition over 5 miles long. Try scanning a project like that to digital and georeferencing. Yes it was done in 1927 datum SPC before GPS was common. I have no idea what size files a project like that would generate.
If I needed to recover that information today that is the path I would consider. Additional utilities were added about 10 years after first acquired and more may still be added.
Some of our projects here in the land survey team of London Underground are greater than 2tb over a few months.
Since I have been here for 5 years we have collected well over 50tb of data.
The storage, backup, processing, archiving and retieval of this data is all very important for us.
Every time I try to forecast the next few years data requirements I am proved wrong as we keep buying more laser scanners that collected more data faster than the last ones.
But disk storage is so cheap that this is almost irrelevant. Getting the process and data policies in place is the vital thing.
How did you manage to collect 1GB of data in a short time without a scanner?
Photographs?
I've got 1988-1998 on a single DVD for cad files and raw files.
My current work flow has me fitting a single year worth of projects on a single DVD.
Conventional surveying and GPS only. No scanning.
All files are purged of junk files, and if a file was archived last year it is ignored by the archive software this year.
> How did you manage to collect 1GB of data in a short time without a scanner?
>
> Photographs?
The data was in the form of high-resolution color scans of (a) mostly handwritten documents from the 19th century (with multiple notations from different hands in different colors of ink that B&W wouldn't capture very well) and (b) high-resolution color scans of maps that likewise wouldn't reproduce as well in B&W. It's the research part of the project file.
In the past, I was also trying to save all project data for the past 10 years or so. When my backup hard drive conked out on me several years ago, I realized that I did not need those data anyway. No client from 10 years ago came around & asked for a copy of his drawings. So now I just back up on a yearly basis. Files from 2-3 years old are overwritten or sometimes their hard drive just cannot be opened any longer.
My current policy is to submit to my clients a PDF as well as a DWG file via email that is also BCC to a Gmail account. Included in the work contract is that I can only hold their data for maximum of 6 months.
If they need their data from more than 1 year then I just charge them for another work.:-/
Storage space gets cheaper and cheaper by the day. You can get a 4 TB backup drive on Amazon for $200. I'm not overly concerned about the price of data storage considering that Moore's Law seems to be applicable for the foreseeable future.
> Storage space gets cheaper and cheaper by the day. You can get a 4 TB backup drive on Amazon for $200. I'm not overly concerned about the price of data storage considering that Moore's Law seems to be applicable for the foreseeable future.
Yes, the cost of maintaining the data in some accessible form is probably the real problem since both storage media and the software that uses some significant part of the data have a lifespan. While it isn't difficult to imagine that, say, a jpg, tif, or ascii file format won't be supported by some reader or viewer into the foreseeable future, data in proprietary format will effectively just evaporate for all practical purposes as the proprietary software necessary to use it disappears from the usual causes.
The life span of software to retrieve and view digital data isn't so much a question of storage space as it is of endless change in the commercial marketplace from which the software comes.
I recollect all those 5 inch floppies , then the modern two inch disks, the tape drives and the multitude of DIFFERENT DOS format back up programs. HM have faithfully backed up and now can not access that religiously stored data? today's computers unable to communicate with 1980s,90s and 2000. Fortunately computer storage grew so was able to retain basics on each each upgrade and attached hard drives for data.
The concern is how long before the owners of the cloud storage start ransoming huge fees to access the cloud stored data or a couple of strategic nuke wipe out the commercial world's financial data?
Kent, definitely suggest you retain your tried and proven 1950's technology of chalk and slate board...
RADU
A few years ago, I had a strange moment.
Stood in front of a dumpster, took a deep breath and threw away 95% of my paper records. Did not scan them, nothing just bye bye.
Was kinda weird looking at all that hard work go in the trash, but guess what have not needed it nor had any request for it.....just like I figured.
Not really a pack rat, if you can't eat it, screw it, or make money with it then honestly who needs it....lol
I keep all the data that I collect or create. Backed up to DVD's currently. A lot of the early GPS data I collected (late 80's-early to mid 90's) is on floppy disks. I still have the disks, but no floppy drive. I do have a few OLD computers laying around that I think maybe could be used to retrieve that data. Some data from the mid 90's I put on tape (the state of the art at the time). That is probably irretrievable.
It is true that I will probably not ever need a lot of that. But, I did at one time have a need for some GPS data from 1984 that was done for mapping of the City of Pittsburgh. It was supposedly the first GPS survey of an urban area. I wanted to be able to readjust the data on NAD83 (it was of course done on NAD27). But, the data was backed up using a VCR onto VHS tapes. No way at all to get it back, although I imagine if I wanted to spend a bunch of money someone somewhere could probably retrieve it. That was before PC hard drives were available. We had PC's with just floppy drives. I remember paying $1000 for a 10 MB (yes, MB) hard drive.
I still have lots of paper files, but I try to scan everything like field notes, log sheets, etc. Everything else is already digital.
Of course, to be usable it is important to have an indexing and/or search system setup. I can determine within seconds the job number and which DVD(s) the data is on by using a database I developed. I just type in any part of the project title, for example "Fort Irwin", or even just "Irwin" and it does a wildcard matching search
I have also begun scanning a lot of the old survey articles from journals that I have accumulated over the years. Makes it easier to find.
Part of the cost of storage needs to be migration as technology moves on. There are usually multiple storage media that are viable at a given time. You have to make sure you've migrated everything before you let the old technology go.
Use your slack moments to do the migration. I once copied all my 5.25" floppies to 3.5" floppies. Just last month I finished putting the 3.5 disks onto CD. About two shoeboxes of floppies fit on one CD. My machine is old and doesn't have DVD, but the next one will and I'll probably put my more important CD's on DVD.
My current backups are mostly to a pair of flash drives, but I occasionally burn copies also.
