Okay, so my hard drive is full of music and movies and TV shows and whatnot, and I need to find a place to put all that stuff so that I can download more, but I'm a little strapped on cash and I also have concerns about the storage quality of some of my
options here...
For instance, I can't buy a new hard drive, because that would be too much money, and also having like 250 gigs worth of stuff in one place seems like a risk to me. What if it fails?
So I've got CD-Rs too, that I can use. But these aren't the best either. I have about a hundred of these CDs containing things I've stored over the years, and about half of them don't work anymore, and I'm not sure why. Has this happened to anyone else? So basically, I'm wondering if anyone has any tips and tricks in terms of storing large amounts of data cheaply and easily, or any innovative ways.
What I've been doing lately is to put about 500mb of whatever on each CD and then fill the rest up with parity data so that if I
start getting bad sectors or whatever happens to CDs when they stop working like that, I can maybe recover. And I'm thinking about making two copies of everything and putting the second in a shoebox or something, and using the first one as a "general use" disc. But does that seem sort of like overkill? Is there a better way? I'm not TOO concerned about losing my stuff, but there's got to be some better way to do this...
Reliable backup, particularly for long durations, is still very much a black art. There are a lot of considerations, the two main ones being reliability of media, and its accessibility.
I personally would go with high quality DVDs, a good burner, PAR2 parity, and a decent place to store it. Treat your hard disk as some kind of LRU cache. Use the DVD C1/C2 counterpart to do a yearly scan and copy data off dying DVDs.
You may want to also burn copies of the necessary extraction tools onto the DVDs, like the archive program you used, necessary codecs, QuickPAR, etc.
Once DVDs start being replaced by a newer media, much like CDR's are now, move everything across.
I usually just distribute my data a little bit.
My music for instance, I have a copy of it on my file server ... sitting on a SCSI HDD (Yes, I got fed up of IDE's reliability years ago). I also have it sitting on a 20GB USB HDD. (I used to keep it on my laptop -- but due to space constraints, it's had to move). If I were to loose both though, I've still got the CDs and vinyl records the music came from. (well, a good 80% of it anyways -- never mind that vinyl's a pain to rip ;-)
I also have other stuff such as source code from Gentoo Linux. I keep this on two servers... normally on my webserver, but I often clean it out, throwing the old files on my file server for LANs. The disk drives on the webserver are quite reliable (and I'm running SCSI software RAID-1 anyways), and the ones in the file server have had a pretty good track record too. Since they're dissimilar drives (36GB IBM's in the webserver, 50GB Seagate's in my file server), they're highly unlikely to blow at the same time.
RAID might be a possibility. Cobble together enough smaller hard drives, and you can come up with a pretty reasonable RAID-5 array, which both provides decent storage, and redundancy. Linux, and I'll bet BSD as well, easily support software RAID, which whilst not as good as true hardware RAID, it does a reasonable job. I'm not sure about Windows though... I think it has it somewhere, but I wouldn't know where to look. (me == Windows newbie :-)
I spose, the real question is, what sort of data are we talking here? Is it stuff you can re-download easily? Is it irreplaceable work? Or bootlegged warez from LANs?
(Hrmm, post too long eh? Squeeks, what are you saying... Am I long-winded?)
While I'd love to have as much hard drive space at my disposal as you do, unfortunately I don't. I have my 40gb laptop hard drive (which is full), a 20gb mp3 player (which is full of other stuff) and a variety of 2 or 3 gig hard drives attached to old computers lying around. Like maybe 3 of them. As a result, using hard drives at all to continue storing my stuff is becoming a problem, much less allowing for redundancy. I also went and catalogged my old archive CDs I'd made over the years, and they total 47 discs worth of data. Hmm, maybe I should just throw some of this stuff away...
As for what I'm downloading, just pirated music and TV and movies, the general sorts of things that tend to take up a lot of space these days... it's all technically redownloadable, but this just seems like the easier path. It wouldn't kill me to lose it exactly, but it would be nice not to. Anything that was hard to find I actually DO make sure to distribute around to my devices. A few albums, for instance, that I have on both of my hard drives plus CDs.
>>2
What are C1/C2 errors and how would I measure them, anyway? Twenty minutes of searching on google failed to yield me a general answer for both of these questions... orz
> Am I long-winded?
Sometimes mate ;)
A 200GB hard drive is $80-$100 now...if you can't afford that, I wouldn't worry too much about robust backups of more than a few gigabytes.
I'm thinking about buying an external HD.
Would save me a lot of CDs to burn until I buy my eventual next computer...
Heh, I've got an external 20GB HDD, partitioned as 5GB FAT32 (for uni stuff), and 15GB XFS (for my private Linux stuff, and my music).
Works a treat. Apart from being a convenient backup medium, it fits in your pocket nicely, is fast (not as fast as flash though), and provides plenty of storage.
The cases are usually <$30 on eBay... and laptop drives vary between ~$10 for a 20GB right up to $200 for some 80GB drives.
I like the idea of an external hard drive and they sound cheap enough, my concern is how reliable they are. I mean I know CDs and DVDs go bad too, but with a hard drive I'd basically be backing up all the data I have into a single place, and if the hard drive were to fail after that, well, that'd be bad. Hard drives have always given off that vibe to me, though, of being kind of delicate and not very robust. Maybe I'm off about that.
They certainly seem inexpensive enough, though.
>>11
If you're that worried about your data, look into building your very own RAID system.
Well I'm not really opposed to it or anything, I just like to worry about things and since it's being discussed here...but yeah, raid would be nice. Unfortunately that does up the cost a little bit...
...What is this??? Am I actually going to have to spend more money for a better quality product? It's almost as if I can't have everything I want for free!
I've heard people mention the use of RAID before, and there's no doubt that such a system will work well, at least if you manage to ensure data is never accidentally erased. However, RAID also needs a lot more money and maintenance than other methods.
If safety is so important, I'd use the DVD method I mentioned above, and make two or three copies, not one. Store in different locations and forget about it. If you use decent DVD media, like Taido Yuden, you may not even need to check it except once a decade, by which time you probably will need to transfer the data to something other than DVD anyway.
>>14
Not necessarily.
Software RAID costs nothing other than the drives, is almost as reliable as Hardware RAID, isn't much slower, and is practically, set-and-forget.
Three servers I maintain:
All three have been a set-and-forget affair as far as RAID is concirned. The ASSN server did recently have disk troubles (as in, apparently two drives "failed" -- they seem okay in boxes here) -- all I did was grab a couple of spares, put them in the caddies, ripped out the "dead" drives, whacked the replacements in their place -- RAID reconstruction began immediately. Linux didn't even flinch.
I'm not talking about the card, I'm talking about the drives themselves. Power isn't free either.
While RAID is a viable means of data storage, I doubt it's more reliable than tossing a couple of DVDs in storage somewhere.
I think my problem is I tend to think of data storage in terms of decades. A decent DVD will be error-free in a decade, and data should be fully-extractable several decades down the road. That's just not possible with a RAID.
Heh, actually, that reminds me, anybody know a place which sells cheap MO media? I've got a Sony SMO-F544 5ΒΌ" 2.6GB MO drive that's been sitting on my SCSI bus doing nothing for years.
And a /home partition that's rapidly filling up with random crap.
I remember wondering as a kid how a magneto-optical drive works. Optics and magnetism just didn't seem to mix to my mind.
Are they using magnetic heads for writing, and optics to read?
To be honest, no idea... I've never bothered to pull the drive apart to find out. ;-)
Iomega makes a drive called "REV". Apparently it's somewhat geared twords long term backup; 50+ years. Since I'm also in search of a method of reliably storing my data for years, I'm thinking of getting that. The problems is it's around $280 for the drive and around $50 for the 35Gb/90Gb disks of which I will need quite a few. Up to now I've been backing my Anime and other stuff up on Taiyo Yuden CDRs, but I don't know how long they're going to last. I've got about 170 of them just sitting around in spindles waiting to go bad. How does DVD media compare? How reliable is it?
> Apparently it's somewhat geared twords long term backup; 50+ years.
Except that it's (apparently) only provided by one vendor, and isn't widely used.
The media could last 10,000 years, but if there's no way to extract the information, it's just a paperweight. Some rare proprietary system doesn't bode well for the future retrieval of the data.
If you want to back the stuff up, and don't need immediate access to it, you could use tape.
DDS2 tape drives aren't too expensive, and easily do 4GB. Or you could go DLT, they do over 20GB.
And they're standard.
>How does DVD media compare? How reliable is it?
It's good. I have had a few CDRs fail on me, and no failed DVD+R so far. It may be because CDR technology was at its limit, or getting low quality for profit, or the CDRs were from a bad batch, I dunno. Or maybe DVD technology is more mature, or error checking is better... whatever. Currently, DVDR > CDR.
The usual disclaimers:
a) Buy quality media
b) Use a quality burner
c) Store in a dark, dry place
I have about ~75 horridly cheap CDs, all different makes, from 1997-1999, yet they're all error-free. I suspect the cause is they've been in storage for the past six years. No light, low humidity, and nobody using them.
DVDs should do better than CDs, since they also have better error-correction. They have smaller pit sizes, so smaller abrasions can cause errors, but if you don't use them this isn't an issue. Just like my CDs.
>a) Buy quality media
My last CDR which failed was a TDK. (more expensive, brand name)
>b) Use a quality burner
It was a SONY. (brand name)
>c) Store in a dark, dry place
I found the CDR was failed just after the burn.
And Verify was even turned on that time, which I don't usually do.
I wasn't referring to your failed CDs (actually, I wasn't referring to your post at all), I was just providing some general guidelines for anyone curious. It's obvious to us, but maybe not to others.
I doubt you'll contest that if you used a poor burner, bad CDs, and left them in a damp place, you'd be having a lot more failures.
As for Sony... egh. I used to work in an electronics repair shop, and we didn't have a high opinion of Sony at all. They've been coasting on their reputation for years. Brand name means diddle shit if they don't carry through.
Same applies for TDK. They only fab some of the CDs they sell, so the batches vary in quality. Some of them are made by Taiyo Yuden, others are by SKC, and some by TDK themselves. The only way to know is to check the CD with software.
> Taiyo Yuden
Is damn good media imho. Never coastered on me once.
>>26
I know you were talking generally, I'm just sharing my experiences.
As for poor burners, you know when you have one. I once bought a burner that failed 5 CDs out of 4 out of the box, could not go to 701mb (max was around 690mb) and came with no burning software. I exchanged it for the SONY, more expensive but which performed flawlessly except for that one TDK failure.
Another story: years ago, I bought a burner that was occasionally failing on the burn (error message at the end). A cheap brand (Mitsumi) but normally reliable so I suspected the lens was dirty, despite that it was sold sealed. I cleaned the lens and it performed well afterwards.
Bad CDs: bad quality CDs fail at the end of/during the burn more frequently than quality CDs, so those are recognizable too.
The ones that are nasty are the ones that burn correctly and fail later. I had a couple of these, they were "no side" CDs (both side look shiny so one cannot tell which is the burnable side). Avoid.
>>21
The REV's compressed form is not cool. I couldn't get it to decompress on another computer, as the provided software kept crashing on me.
In uncompressed form, it's simply another filesystem, just like expecting a CD to work when you put it in. It's also good for incremental daily updates, which DVDs just don't work for. I think it's decent as a first level offsite backup, if your data fits on one or two tapes. If I can figure out how to get the compressed form working it will be a lot nicer.
> burn correctly and fail later
I wonder if a C1/C2 error check right after a burn would catch such CDs?
>>30 That's the general consensus amongst users, it seems.
"High C1 should be considered an indicator that the disc will eventually degrade and fail."
"One factor that has been demonstrated by many users, is that discs that do not burn well and show higher error-rates tend to degrade much faster. In worst cases, they can fail in as little as a few weeks, but months or years is more likely an accurate prediction for the less reliable of the media that's out there."
But then:
"As far as I know C1 values should not exceed 220 per second, and C2 values should not exceed 1 per hour(!).
I do have CDs however with a few thousand C1 errors per second and thousands of C2 errors per hour that are still readable without any major problem."
Dude, I have a pile of MO discs to get rid of. Seriously, you want 'em?
>>18
They are also known as WORM drives. Write Optical Read Magnetic. They use a laser to heat up the surface of the disc, making it writable magnetically. Without the laser, it can be read megnetically but cannot be written to.
Yes. The disc material has some sort of phase transition, where below a certain temperature the magnetic orientation is frozen, and above it, the magnetic orienation can be changed with an external field. So you heat up the bit you want to change, and apply an external magnetic field to flip it, and then cool it, and the magnetic orientation freezes.
I have no idea how this can be done fast enough to write huge amounts of data to the drive, but then again I really don't understand how any kind of magnetical storage can even work these days.
Hard drives have no problem doing it. But as I said, I have no idea how the hell they manage it.