Tech: Should you store treasured data on disks

Subject: RE: Tech: Should you store treasured data on disks
From: Nicholas Waller
Date: 03 Jun 10 - 02:37 AM

There's a digitally-restored print of the Powell and Pressburger film The Red Shoes out and about, and last autumn I saw a screening introduced by Thelma Schoonmaker, the Oscar-winning editor and widow of Michael Powell. She did a talk about the digitisation of the original three-strip Technicolor negative and the computer malarkey to colour-correct and line everything up.

Then she said that in this digital age actually the best way to store the movie for future generations was to print it back out to traditional film, as we know that can be stored for 100 years whereas digital storage media turn out to be flaky in themselves, not to mention subject to the obsolescence caused by rapid advances and changes in the technology. That's the first I had really heard of it as a serious long-term problem for archives.

Technically, I suppose, you could print the digital information for a movie or an album onto paper... but that would be thousands or millions of pages for each item and rather impractical...

Subject: RE: Tech: Should you store treasured data on disks
From: Jim Carroll
Date: 02 Jun 10 - 08:36 PM

Cautionary tale:
A publican we know had an elderly and extremely important traditional fiddle player as a regular at his pub in a town in south west Ireland; so he decided it was a good idea to record some of his playing for posterity, each time he came in. Realising the importance of what he was recording, he carefully placed the (reel-to-reel) tapes in metal biscuit tins and locked them in a built in cupboard in the public bar.
He did this for about ten years until the old man died. On the night of the funeral he took the precious horde of tapes out of the cupboard and placed them on his tape recorder to play them for the mourners. He watched in horror as the coating fell from the tapes as they went through his machine - he hadn't noticed that there was a hot water pipe going through the cupboard.
Make copies for yourself on discs, on hard discs, on cassettes, on reel-to-reel (apart from a few gluggers, our huge collection of tapes dating back 45 years still play).
If the material is that treasured (and of general interest) it should be in an established archive anyway.
Jim Carroll

Subject: RE: Tech: Should you store treasured data on disks
From: robomatic
Date: 02 Jun 10 - 04:32 PM

I agree with the cautionary tales regarding digital disks either CD or DVD. I think that magnetic media are good standbys. Hard drives are relatively inexpensive. It doesn't cost that much to have an external hard drive full of digitized family treasures stored in a secure part of the home or exchanged with a friend.
Of course, you should also keep the primary sources, documents, photographs, slides, etc. in a 'cool, dry, place'.

I transferred my family's audio tapes to digital media a few years ago. They were all in the range of forty years old. There was considerable quality differential between brands. DECCA tapes seemed as original. The London FFRR tapes of Gilbert and Sullivan required to be 'baked'.

As for 1 TB versus 500 GB hard drives, the preference goes to the more recently manufactured. The increased capacity may come from extra platters or extra density. What you care about with magnetic storage is not so much the density as it is the engineering and the quality of manufacture. Magnetic media involve a small magnetized area which is surrounded by a 'domain'. There are calculations made during design to assure that it can't be changed without a considerable outside force. When I buy a drive, I plug it in and leave it on for a long time, sometimes take it to work and just leave it there as an adjunct drive. This gives the little heating faults a chance to work themselves to the surface and fail in the first three months. If it still works, then I load it up with my precious data.

One nice thing about the 'portable drives, which are just hitting the 1 TB size- since they are powered off the USB, if you have any kind of power backup on your desktop, or if you are using a laptop with a decent battery, a power failure at the mains will not cause a powerfailure in the drive, so long as the computer is still running,whereas if it plugs into the wall, power failure there will upset that data transfer. The portable drives take up a lot less space, store better, and are easier to transport, as many of them will fit in a shirt pocket. They are no less reliable than the larger external drives. You pay a modest amount extra for the portability and they spin at a slower RPM, which means they don't transfer data as quickly.

Still it is scary. Having hundreds of gigabytes in one place. Avoid putting it through a procedure where you're holding it with one hand or regularly moving it about.

Subject: RE: Tech: Should you store treasured data on disks
From: JohnInKansas
Date: 02 Jun 10 - 03:12 PM

Billy W - Next time it happens, you can add a name to the GUEST,
as in GUEST,BWeeks. It does help people decide whether you're just trolling or actually are someone who wants help - or to be helpful.

And if you're consistent (if it happens more than once), you'll be able to find your guest posts instead of just "all of the anonymous guests' posts" should you wish to. There are 80,936 posts by "GUEST" as of today, which would make it tough to sort out yours, but "Guest, Shimrod" just above has only about 1600 to look through.

V Tam - I would think that gold or silver doesn't really matter much on recordable CDs or DVDs (and the silver is usually aluminum, just for the record). That layer is just a "reflector" and the only requirement is that it must provide a smooth, or smoothly varying, "background" for the "dots" in the photosensitive layer.

If the "burn" laser can change the photosensitive materials, then ambient light, heat, and chemicals can change them back or change them more, with the result that the data is lost no matter how bright and shiny the gold (which could be brass) is. Information on the really critical parts of the disks is always proprietary and there's really little published information on what various makers use, and possibly less information on how a specific disk construction can be expected to behave in storage.

John

Subject: RE: Tech: Should you store treasured data on disks
From: Rapparee
Date: 02 Jun 10 - 02:41 PM

Paint it on a cave wall and then seal up the cave.

No sweat.

Subject: RE: Tech: Should you store treasured data on disks
From: Billy Weeks
Date: 02 Jun 10 - 02:40 PM

Thanks, JiK. Joe sorted me out.

Subject: RE: Tech: Should you store treasured data on disks
From: VirginiaTam
Date: 02 Jun 10 - 02:28 PM

what about MAM-A (Mitsui) Gold Archive DVD-R with no logo?
Supposedly the 24 karat gold layer will not oxidise as silver does.

Also read somewhere that brand logo printed on cds and dvds compromises the integrity of the disc, making it decompose faster.

Subject: RE: Tech: Should you store treasured data on disks
From: GUEST,Shimrod
Date: 02 Jun 10 - 02:13 PM

I suppose that this isn't really a new problem: clay tablets got broken, papyrus scrolls and books got burned, wax cylinders became unplayable after they were played too much (or so I'm told), old film stock is flammable or was thrown away, the BBC threw away thousands of audio/video tapes and I heard a horror story a few years ago about some precious tapes of traditional singers which weren't stored correctly and all the oxide fell off. Meanwhile I can't play my collection of LPs because I haven't got a record deck any more (apart from the fact that some of them seem to have their grooves clogged with some 'cloggy stuff' - presumably because I didn't store them correctly). Entropy scrambles all information in the end.

Subject: RE: Tech: Should you store treasured data on disks
From: Haruo
Date: 02 Jun 10 - 01:19 PM

I copy everything onto mediaeval manuscripts, just in case. ;-)

And I console myself in the belief that it's all permanently backed up in heaven.

Haruo

Subject: RE: Tech: Should you store treasured data on disks
From: JohnInKansas
Date: 02 Jun 10 - 01:11 PM

GUEST (Anonymous)

If you clean out cookies, you sometimes loose the mudcat one. Disk Cleanup will do it sometimes. In IE, if mudcat.org is in your "always allow" list, usually the 'cat login cookie is safe during cleanup; but gets deleted if you don't have the site listed.

Other browsers may vary, and there are other mysterious causes for crumbles. It usually is something you did without full understanding of the consequences, like when one says "I do."

In this case, the unexpected isn't a big deal. Just go to the Quick Links box at the top of every page, click for the drop-down list, go to Log In and become a real personality again.

John

Subject: RE: Tech: Should you store treasured data on disks
From: GUEST
Date: 02 Jun 10 - 01:00 PM

I seem to have become a guest. Why do cookies crumble?

---

I can help if you need it. [e-mail]-

Subject: RE: Tech: Should you store treasured data on disks
From: GUEST,Billy Weeks
Date: 02 Jun 10 - 12:59 PM

When I put this question to a recording whizz some years ago, he suggested that the most reliable longlife recording medium was the late 19th/early 20th century cylinder. It was superior to the flat disc, in that the stylus moved at a constant speed in relation to the groove and all the information on a well-cared for cylinder could be retrieved a century after its manufacture, without serious loss. He had no views to offer about transferring CDs to cylinders, so I decided not to do it.

Subject: RE: Tech: Should you store treasured data on disks
From: peregrina
Date: 01 Jun 10 - 06:05 PM

papyrus (which is okay for vegetarian use) survives well in desert sand, but parchment needs to be indoors. And ferrous ink, well, there are better options! Longevity under the right conditions? A millennium or two.

Subject: RE: Tech: Should you store treasured data on disks
From: dick greenhaus
Date: 01 Jun 10 - 06:03 PM

And incised stone tablets work fine, if you keep them out of the weather.

Subject: RE: Tech: Should you store treasured data on disks
From: JohnInKansas
Date: 01 Jun 10 - 05:24 PM

For true archival purposes, even printing is somewhat less secure than might be assumed.

Possibly the best "archive" prints are those from laser printers, since the ink is heat-fused to the paper, and the black ink fades very little. I don't have experience with whether color laser prints retain color fidelity over long times. I would expect it to be fairly good, but haven't seen much comment, and even with the best color laser printers available now the initial color fidelity is a bit inferior to other methods.

Laser prints, however, when stored for long periods, even under good temperature/humidity conditions, do have a tendency to "fuse to each other" and I have recently gone through some records from ca. 1990 for which separating the pages of "laser" prints requires rather "delicate" handling. The printers of that era used fairly heavy ink applications, and newer printers that use finer inking may be less prone to this problem.

Ink jet printer makers claim long life for their prints, but the claims resemble quite closely the claims by writeable CD makers, and can't really be taken too seriously. For almost all available inks I've seen, fading is very rapid with any exposure to sunlight, and must be assumed to occur even in low light. Nearly all inkjet prints will "run" if exposed to even slight moisture, and images get fuzzy even in normal living area humidities.

"Waterproof" prints can be made on any inkjet printer by using one of the special "papers" available (Amazon Adventure Paper is the one most easily available in the US?) but the paper runs near 50 cents (US) per sheet for A size (8.5 x 11 inch) and about $1.25 per sheet for B (11 x 17 inch). Fading still happens with the waterproof prints.

Even fairly cheap "office grade" papers commonly sold here usually are reasonably close to "acid free" but you have to verify that quality to avoid fairly rapid embrittlement and crumbling for anything you intend to store for more than a year or two. (I have some horrible examples from ca 1994 and before.) Even being acid free, so that the ink color isn't destroyed, doesn't assure that the background page won't "yellow," which is almost as bad when you try to re-archive things that need it.

Hand writing using ferrous ink on animal skin or papyrus parchments has a reasonably verified longevity, if you're willing to make your own ink and parchment; but even for these meda the storage conditions best suited to long storage time are somewhat difficult to produce and ensure without personal ownership of a few very large sand dunes in a desert climate ... .

John

Subject: RE: Tech: Should you store treasured data on disks
From: treewind
Date: 01 Jun 10 - 08:33 AM

"I did have an interesting conversation with a technician in Maplins some months ago about the pros and cons of 500MB v.v. 1TB+ external drives. And he said that he would go for duplicate 500GB drives, since the new TB drives were just too fragile for archival use."

I wouldn't trust most Maplins staff with the technical knowledge to tell me what day of the week it was.

There are good reasons for using two drives instead of one, but in the context of this thread that's simply so you can duplicate your data and survive if one drive fails.

Subject: RE: Tech: Should you store treasured data on disks
From: Rob Naylor
Date: 31 May 10 - 07:32 PM

The seismic industry requires huge amounts of storage of valuable data.

My small survey department (30 people) was Sun Microsystems' first on-line Terabyte facility in the UK, back in 1987 (made up of dozens and dozens of 5 and 10 Gb hard drives).

But for long-term storage, only certain media have been approved. Initially it was 9-track tapes (I still occasionally need to read some. In that case we "bake" them to stabilise the oxide. I get about 90% recovery from 25-30 year old tapes, IF they were a good brand). Then it became 3480 cartridges, then 3590, DLT and 3590 cartridges for archive storage.

The industry has NEVER allowed optical media for long-term archiving...we've always known it was unreliabe...as are the more compact 8mm and 4 mm cassete-type magnetic media such as Exabyte.

Even with the most rigorously tested cartridge media, we still tend to re-copy the data every 10 years or so onto fresh media. We're talking thousands of terabytes here. A single marine seismic line, perhaps acquired in an hour, can generate 300 Mb of data used just to compute positions of the hydrophones alone, with perhaps another 3 Gb of data recorded from the hydrophones themselves.

So if organisations for whom these quantities of data are critical and extremely valuable don't use optical media for long-term storage, preferring instead to spend orders of magnitude more money on more stable cartirdge media, that says something about its reliability, I think.

Subject: RE: Tech: Should you store treasured data on disks
From: Rapparee
Date: 31 May 10 - 06:54 PM

If you REALLY want it to last, print it out. Then make copies of it.

Can you read the data on a 5" diskette? Does you computer even have a drive to read 3.5" diskettes? How about reel-to-reel tape? Eight inch disks? Drum memory? Punch cards? 8-track tape? Videodisks? Betamax?

Every single thing named above has been in use since 1971, which I first saw some of the first videocassettes -- which were brand new at an electronics fair in Tokyo.

The Library of Congress recommends copying to an external hard drive and then making a copy of that hard drive. Then copy the data only other hard drives as needed/every year.

Electronic and magnetic fields tend towards randomness -- entropy ALWAYS increases. Ink on paper ain't sexy, but it is permanent if cared for.

Subject: RE: Tech: Should you store treasured data on disks
From: oggie
Date: 31 May 10 - 06:36 PM

I use at least two formats at home and then also use two online storage facilities. So, for example, photos are on both photobucket and flickr and I have also made hard copies (ie printed) the most important ones.

Steve

Subject: RE: Tech: Should you store treasured data on disks
From: ChrisJBrady
Date: 31 May 10 - 05:54 PM

I have just been digitising some 100 year old 78s. Still playable - almost as good as new, and sounding great from an electric deck. Also remember that B&W films seem to last the ages - some being 100 years old. But think of the problems with colour film - with the dies fading. And then think of video tapes - with the oxide decomposing and clogging up the read heads. And then there's CDs and DVDs - the longevity and deterioration of which is largely a taboo subject because many artists would prefer that their customers might not appreciate paying big bucks for something likely to last a mere few years.

It seems that every advancement in recording technology and media formats give an ever shorter time in retention of recordings made with same.


I think that the great advantage of digital media is that recordings can be copied without loss of detail. But also that recordings have to be refreshed every few years - a big job but not an impossible one with today's external hard drives. And the s/w that drives them can have error correcting algorithms.

As an aside - Panasonic RAM discs also use error correcting algorithms. And I would trust them over and above CDs or DVDs. But they are rare and expensive.

However I did have an interesting conversation with a technician in Maplins some months ago about the pros and cons of 500MB v.v. 1TB+ external drives. And he said that he would go for duplicate 500GB drives, since the new TB drives were just too fragile for archival use.

CJB.

Subject: RE: Tech: Should you store treasured data on disks
From: Bernard
Date: 31 May 10 - 05:19 PM

What is equally as important as the data itself is the system with which to read the data.

It's not much use having backups if you don't have hardware and software with which to read it.

Iomega's Zip disks were once the industry standard backup, for example. There are computer users nowadays who have never even heard of them! I have working Zip100, Zip250 and Zip750 drives - the Zip750 is capable of reading (but not writing) Zip100 and Zip250. Only recently I had to extract data from some old Zip100 cartridges for a friend which, luckily, were still intact.

IDE drives (PATA) were superceded by SATA drives, so a lot of modern computers cannot read them - unless you can lay your hands on a USB / PATA drive caddy... but how long will USB be around?

So migrating your data to the most current media type is wise, but only if you've software capable of making use of the data...

My preference is to use Network Attached Storage with mirrored drives, and to keep multiple copies on other media as well. I also keep old hardware which often comes in handy when people find they have old media but not the hardware to read it!

Subject: RE: Tech: Should you store treasured data on disks
From: treewind
Date: 31 May 10 - 04:18 PM

"back them up back them up back them up"
Whilst I wouldn't for a moment wish to denigrate that good advice, it doesn't by itself address the issue of long term storage. In 20 years your backups are as likely to be unusable as your primary data.

Certainly keeping more than one copy of your archive data is a good idea, but it's also essential to copy it to new media from time to time - exactly the opposite of what's good for analogue data like tape recordings or print.

Subject: RE: Tech: Should you store treasured data on disks
From: olddude
Date: 31 May 10 - 04:03 PM

yup back them up back them up back them up ... I have lost too much even after 2 years .. if they get hot or cold .. they go and it makes no difference brand name or generic

Subject: RE: Tech: Should you store treasured data on disks
From: peregrina
Date: 31 May 10 - 03:58 PM

Even archives have to migrate their data at regular intervals.

The reading/retrieval software is just as ephemeral as the data on 'hard' storage materials--but because of development and planned obsolescence rather than decay.

In many ways data entrusted to electronic storage is far more vulnerable than, say, a medieval manuscript.

Subject: RE: Tech: Should you store treasured data on disks
From: JohnInKansas
Date: 31 May 10 - 03:30 PM

Of approximately 800 CDs used to back up data files (not music/video) after 5 to 7 years I went through the entire store due to excessive duplications of files and "too many disks" to store conveniently in my limited clean area. Even with a complete "index" of all the disks, the loss of useful search capabilities in Vista made it almost impossible to quickly find what I wanted.

Approximately 30% of the disks were "unmountable," and NO FILES were recoverable from them. A few disks had most files recoverable, but with some individual files "corrupted" and unreadable.

Since I had just copied folders from the hard drives, most files that had been on the hard drives for a while were duplicated on several CDs, so I didn't actually lose a significant amount of data; but had I had anything on a single CD as the only backup, I would have lost at least the 10% on disks that failed completely.

These disks were stored in "slim" jewel boxes in clean drawers (in the dark) in a temperature controlled (normal people temperature) in the end of our "office." [Note: storage drawers for 5" floppy disks are a very good fit for CDs, and I had a 12 drawer "set" - room for about 900 CDs - of drawers salvaged when I no longer had a drive to read the 5" floppies. I wasn't out of drawer space, but it was becoming difficult to find the right individual CD when I decided to change backup methods.]

I had a hard drive failure a few years ago on a machine in which I had an internal second hard drive with backups, and found that whatever had killed the main hard drive also terminated the backup drive. Since then, I've made regular data backups to two separate external USB hard drives that are connected only during backup.

I've had no loss of data in approximately 5 years of using external hard drives, although I have lost two "desktop external" drives during that time. I have not lost anything since switching to "portable external hard drives" better suited to occasional "moving around" on the desktop, or to storage (in padded individual cases) in side storage.

CDs used for distribution of software generally, I'm told, are "molded" or "pressed" with physical pits in the recording layer that should be stable enough to be trusted for reasonably long term archival storage. CDs "burned" on your computer rely on a chemical/crystalline structure change in a photosensitive layer that is also sensitive to heat, chemicals, simple aging, and to some extent to light in normal ambient wavelengths, and I don't believe they can be trusted for archiving. My experience with "burned" CDs suggests that they should be copied back to a hard drive and new copies burned about every three years or so, for what I might consider "acceptable risk."

CDs are great for snail-mailing large files, although most email services now have high enough file size limits to make even that use unnecessary in most cases.

I don't see any reason to expect "home burned" DVDs to be much different than CDs.

As long as the CD doesn't fail completely - the "unmountable ones" - a moderate number of "bits" lost in storage probably will be difficult to detect for music/video files; but I have used them only for "data files" which appear to be much more sensitive to even a few lost bits.

John

Subject: RE: Tech: Should you store treasured data on disks
From: GUEST,DWR
Date: 31 May 10 - 03:10 PM

That's what I was going to say -- back it up, backitup, BACKITUP! I do have single copies of some things, but anything in that category is mostly expendable or at least replaceable.

Things I value are in at least two different places. I do have a bunch of things on disks, but they are for the most part on a hard drive as well.

As far as the really valuable stuff, they always recommend that at least one copy is kept off site, in case of catastrophic destruction -- tornado, fire, etc.

Thanks for the reminders, Chris.

Dale

Subject: RE: Tech: Should you store treasured data on disks
From: ChrisJBrady
Date: 31 May 10 - 02:46 PM

More info here:

http://www.lne.fr/en/r_and_d/digital-optical-research/archiving-digital-gis-don.asp


http://spie.org/x648.html?product_id=848948

http://www.thedigitalnewsroom.com/en/News/2758/CD_DVD_and_Flash_Memory_are_unreliable_medias_.htm

http://www.thedigitalnewsroom.com/en/News/2758/CD_DVD_and_Flash_Memory_are_unreliable_medias_.htm

CD/DVD : a lifespan of 10 years ?

The Academy of Sciences and Technology will release on April 9th their report entitled "Longevity of Digital Information". First details confirm that the current storage medias are not reliable, including CD, DVD, but also Blu-Ray and Flash memory. "Some manufacturers claim that their media can retain electronic datas forever, but this is absolutely not the case" says Franck Laloë, one of the three analysts behind this report.

More density = more problems ?

He also mentions that "CD and DVD have a maximum lifespan of 15/20 years, but some of them are start to deteriorate after just a year. After 4 years, more than 8% of the CD/DVD tested are not working anymore. Time and scratches affect CD and DVD's datas of course, but even in protecting and keeping them in optimal conditions, these medias degrade. A huge quantity of personnal, medical, scientific, technic and public datas are in real danger. A periodic renewal of storage media could be a good, but expensive, solution." According to the report, hard drive disks and Flash media devices would also fail after 3/5 years, and increasing the density of media devices just keeps on increasing the problem. Blu-Ray are then more fragile than DVD, which are more fragile than CD...

Subject: Tech: Should you store treasured data on disks?
From: GUEST,ChrisJBrady
Date: 31 May 10 - 02:21 PM

Should you store treasured data [or music or videos] on disks?
By David Reid
Reporter, BBC Click

Mr Laloe said CDs meant to last a long time were only good for a few years.
http://news.bbc.co.uk/1/hi/programmes/click_online/8711747.stm
Preserving precious data, such as pictures and home videos, on CDs and DVDs could do more damage than good in the end.
That's what the French National Centre for Scientific Research has found out after testing the longevity of the portable media.
"We were surprised to see that the lifetime of discs, some of which were designed to last for centuries, actually rarely lasted longer than five to 10 years," said physicist Franck Laloe.
"In the most severe cases, which were happily quite rare, the data on some discs lasted just one year," he said.
Unlike audio and video tapes that wear out with play, discs are read with a laser so there is no physical contact.
However, the surface which contains more than seven miles of data is deteriorating with age.

Varied quality
French scientists calculated the likely life-span of a disc by artificially aging it with heat, water vapour and light.
They found CD quality varied even across the same brand so people willing to pay more for a known name won't necessarily get a better product.
"The brand alone doesn't tell you if you have something that is high or low quality," said Jean-Michel Lambert, from the French National Metrology Laboratory.
"Disc production varies. In the same brand we find discs produced by different manufacturers which means their quality and how long they last for is not necessarily the same," he added.
Also, manufacturers branding discs with their logos could be making the data on them more vulnerable to corruption.

No guarantees
Documentary-makers record about five hundred gigabytes of HD for a 52 minute documentary and need to have a way of storing footage.
Jerome Duc-Mauge, executive producer at Cocottes Minute, keeps rushes stored in a combination of hard drives and data cassettes.
But he still cannot be certain these storage solutions are the best options.
"This is a big drama, this issue of how long these pictures will last. We don't know. The manufacturer says to us, "Yeah, five years, 10 years, 15 years," he said.
"But we will see in 15 years, we'll see in 20 years if it is still here, or if it has just become a pile of dust at the bottom of the cassette."
There are a few precautions people can take to avoid losing their data. Mr Laloe said these meant being "vigilant".
"Every two or three years, you have to copy your archive onto fresh discs. And after that, because these new discs will last a bit longer, you will have to re-copy them after five or six years," he said.
Mr Lambert recommended spreading digital data rather than keeping it all archived in one place.
"You must have your information in two places at least - on a hard-disc, for example, and on another hard-disc or on a recordable DVD or CD."