AArdvark wrote: I bet the entire library of Congress could fit in ONE of those large storage thing-a-ma bobs, now I see aisles of them. That's a lotta ones and zeros.

Actually the History Channel had a show on the Library of Congress, and its attempts to digitize its collections. Just its collection on nitrate film takes up something like 150 long shelves in a large fire-controlled building in Virginia. They have a lot of stuff on a lot of servers at the LOC main buildings in Washington.

I don't know if Ben ever visited the LOC when he lived here in the Washington area, (I have, more than once) but even before they were doing digital collections they still used three buildings in downtown DC plus a large warehouse in Lanham, Maryland.

They're trying to digitize one item that's going to take time: the complete collection of newscasts for many years back in the 1980s, donated by CBS: 800,000 tapes. If you figure two gigabytes per hour of video, 2 hours per tape, and 4000 gigabytes now probably takes about 1 cubic foot, that alone is going to require 400 cubic feet of space, or a set of servers 40 feet long and ten feet high.

Some technical analyses of explosions, including atom smashers, can generate 10TB/sec of data. I think the estimate is "the world's total yearly production of print, film, optical, and magnetic content would require roughly 1.5 billion gigabytes of storage."

Whenever anything is copyrighted in the U.S., at least one, and often two, copies are supposed to be sent to the Copyright Office at the Library of Congress, most of which is deposited goes into the LOC collections. But there is a lot of stuff they don't get. Even so, there's a lot of data generated by a lot of things.

AArdvark wrote:What kind of data is worth someone physically breaking and entering into a house to steal? It's not like we are all high grade defense contractors or something.

I doubt most thieves want the data, they just want the computer so they can sell it or pawn it, and your data becomes an innocent victim. I'm sure there are a few cases where thieves have gone through computers in an attempt to find credit card numbers, checking information or maybe even blackmail material, but my guess is that Willie the Crack Head isn't going to those lengths. Incidently, TrueCrypt is a free (one of many) drive encryption utility that requires you to enter a password before you boot your computer. No password, no data, period.

AArdvark wrote:Speaking of data, whats on all those gigs of storage anyway? Not the specifics, I don't want Flack to violate and ND stuff, but what takes up so much space? I bet the entire library of Congress could fit in ONE of those large storage thing-a-ma bobs, now I see aisles of them. That's a lotta ones and zeros.

I work here, at the largest DOT complex outside of Washington DC. I work for Aviation Safety, and all of our web servers (both public and private) are hosted here -- plus all maintenance data, pilot information, safety data, etc. We also host Academy information, medical stuff, and so forth. In fact, we host lots of servers that the folks in DC access on a daily basis.

What kind of data is worth someone physically breaking and entering into a house to steal? It's not like we are all high grade defense contractors or something.

Speaking of data, whats on all those gigs of storage anyway? Not the specifics, I don't want Flack to violate any ND stuff, but what takes up so much space? I bet the entire library of Congress could fit in ONE of those large storage thing-a-ma bobs, now I see aisles of them. That's a lotta ones and zeros.



THE
HISTORY OF THE WORLD
AARDVARK

Couple of other notes:

Make Magazine (and several other sites) a couple of years ago documented hiding an external USB drive in a spare/old UPS. It's something a thief would be least likely to carry off -- they're almost worthless, and heavy.

The idea of sticking one in a freezer isn't a bad idea either. If you have a spare machine you're willing to dedicate to it, you could set it to Wake On Lan to save electricity.

Here's part of what we back up at work.

This is a rack mounted disk enclosure. This one is from EMC, but we have Dell ones too. Most of these are connected to our SAN and used for drive storage for virtual servers.

This particular one holds 15 drives. It's full, with 800 gig drives.



These racks hold 13 of those storage units. They're all filled.



Here are some new EMC racks combined with some older Dell racks. The Dell racks are also full, but with 400 gig drives.



In addition to those, we have added racks full of drives to pretty much the end of all 15 of our rows of racks. Not all of them are full, but some are.



This is essentially what we're backing up. I'm no longer responsible for the day-to-day operations of performing backups, but I used to be, and I helped designed our backup solution.

Ice Cream Jonsey wrote:(I could probably map a drive to a writable directly remote computer, and have robocopy back stuff up there, as well. Hmm. I have to find a secret place for a computer in my house. Flack, I like the idea you floated on the Gas Chamber about having a remote PC wirelessly connected in your arcade.)

Jonsey, can you get hold of a free used refrigerator or freezer that doesn't work? Something that looks ugly looking?

Now, get an old computer and either put a big hard drive in that or hook a USB external drive to it. Put a wireless USB adapter on it, and make sure it boots headless and starts up automatically. Put it inside the reefer and see that it can receive your signal.

Now you just put the refrigerator on the back porch, and, as is required by law, it's chained up to keep kids out. The cord for the surge suppressor runs out the back, and if it's an older computer, the noise it makes perfectly fits an old fridge or freezer (or it might not make noise). Sitting on the back porch outside "proves" it ain't worth nothing and a thief is in a hurry, they're not going to bother an old refrigerator.

And even if they do break in, if the computer is stuck in the ice box, they'll probably not know what it is like the condenser or converter and ignore it. Or even better if it's one of those really old gas refrigerators, sneak the computer under the coils at the top.

In that case, hide the cord rather than allowing it to show so it's not thought to be running. Have a program on it that has it reboot itself once a week, say, Sunday at 4AM so you don't have problems due to the clock overflowing.

Flack wrote:We run LTO-4 tapes and were planning on upgrading to LTO-5 ... I'm guessing we'll be running 4 for a bit longer. LTO-4 tapes hold 800 gig each (uncompressed) and cost $25 each. Average lifespan of a tape is 30 years ...
(LTO-5 tapes, by the way, hold 1.6 tb per tape and are even faster than the 4 series.).

Flack, I had no idea. My understanding was that the best you could do in tape backup were DC-3100 type tapes at about 60GB or something like that, and at the same price, around $25-30. I had no idea they have tapes that do 800GB at $25 a pop, that is a better solution and cheaper even than using Blu-Ray.

From what I've heard I'll bet they even have either speedloaders - jukebox-type systems that allow you to load them with 6 or 10 tapes (or more) and when a tape is full it automatically removes it and loads the next one - or supports a robotic tape loading system (they show one of these for optical disc in the Schwartzenegger movie Eraser).

With 800+ GB tapes at that low a price tape backup as either off-line or near-line storage does make a lot of sense for very large backup operations. For those not familiar, off-line means the tapes are stored, near-line means they are in a jukebox type system and it can be told to retrieve an item. Near-line storage takes a few minutes to mount a tape from the requestor, off-line you have shipped back from your storage facility.

I know about long-term requirements for government storage of documents. In Virginia, any case tried in a district court - including traffic tickets - can be appealed trial de novo to circuit court, provided you're willing to pay court costs for both trials if you lose. You can even get a jury trial, provided, again, you're willing to pay the $360 if you lose. If you have a trial in circuit court, all paperwork and exhibits must be archived for retrieval by any member of the public who wants to see it, for ten years after the case is decided. If some courthouses were very busy, imaging files and storing them on media, while putting the originals in boxes in storage facilities, would probably be cheaper than having every file in the court house, plus every document could be made available on the Internet.

Also, in both Virginia and Maryland - and I'd be surprised if other states don't offer it - you can send a copy of your will to the county clerk for safekeeping. The fee is $2 in Virginia and $5 in Maryland. So in my case I sent my will in a couple of years ago; my life expectancy is probably 20 years or less. For someone who was 25, their life expectancy is maybe 45 years. So these documents have to be kept for potentially a long time, decades, or until someone shows up with a death certificate.

I just looked it up, the drives are a bit steep, $3500 gets you an LTO-4 drive with an 8-tape autoloader, and you can go up to 16-tape autoloaders. In a business environment where data can be worth millions, or a government environment where not covering your ass can mean loss of career or worse, it makes sense.

Of course tape is slower, that's why your disk-disk-tape solution makes a lot of sense, you backup disk to disk, so you only need the extra disk space temporarily, long enough to make slower tape copies, then you keep reusing the backup disc space for making tape backups. If you have enough free space to support a week or two worth of files, then accidental deletions don't matter, since you're doing 4-hour replications, someone can get any file back from any 4-hour window over, say, the last week or two before even needing to go to the tape backups.

What does that mean though? That's bullshit.

If they can keep the ship afloat at half cost, why won't 8c on the dollar work? Why not -8c? Who gives a shit?

Flack wrote:We run LTO-4 tapes and were planning on upgrading to LTO-5 later this year, although now that the federal budget has been slashed like a Freddy Krueger victim

You think things are bad now, just wait. I don't remember what year they said, but soon, non-discretionary spending: social security, medicare and interest on the national debt, will shortly reach 92c of every dollar of federal revenue. The entire government will have to operate on 8c of each dollar.

Ice Cream Jonsey wrote: Also - I think I bought a pack of shitty CD-RWs at Best Buy.

I've never had too much trouble, I burn stuff and the test confirms it worked. I buy my disks at Staples or Micro Center or if they have a rare sale, Radio Shack. Like CVS in the drug business, RatShack is almost always the high-price leader.

Ice Cream Jonsey wrote: I've noticed errors on two audio discs that I have burned, and twice now I haven't been able to get Ubuntu burned to CD. I just want to run some fucking Ubuntu.

I found out my new machine is actually a 64-bit machine (I'm running 32-Bit Windows XP Professional with a COA for refurbished machines, so it is legal). So I decided to try it. I burned a copy of Ubuntu 10.10 Live CD 64-bit.

I am absolutely amazed at how far they've come. Not only that, for Internet connectivity I have an el cheapo $9 Tenda USB 802-11n chewing-gum sized adapter. I am writing this from Windows XP and the adapter works just fine after the driver was installed.

I booted up using the 64-bit live CD and once it started, I tried Firefox. It comes right up, figures out which network to connect to (I'm the only one running unsecured, everyone else around us requires a password) and went to Google. Not only that, X-Windows sized my monitor to the highest resolution the video card supported, and had a nice background image.

I did not have to set or tweak anything to get Linux and X-Windows and Wireless Internet to work "out of the box." The only problem, of course, was that since it was building a setup and running off a CD it was much slower, it probably took about 5 or 10 minutes to start vs less than a minute for XP off the hard drive.

Also I couldn't read the files off the other computer via windows networking; maybe I had to turn Samba on or something. But otherwise I was amazed. Oh, and it did recognize the NTFS file system that my hard drive is set up for.

What amazed me most is the installation recognized the cheap Chinese USB networking adapter without installing drivers.

ICJ, if you can't get it to work, give me your address and whether you want 32-bit or 64-bit, I'll mail you a disk that I've booted off of so I know it does work. It's a live CD and you can choose to install or run without installing.

bruce wrote:Bacula is cross platform and free.

As is Blackula.

We run LTO-4 tapes and were planning on upgrading to LTO-5 later this year, although now that the federal budget has been slashed like a Freddy Krueger victim, I'm guessing we'll be running 4 for a bit longer. LTO-4 tapes hold 800 gig each (uncompressed) and cost $25 each. Average lifespan of a tape is 30 years so they'll last way longer than any disc-based media, and they're a bit more resilient to being stored (and dropped) than a hard drive.

(LTO-5 tapes, by the way, hold 1.6 tb per tape and are even faster than the 4 series.)

They're slow to back up on and slower to restore from, but as far as long term data storage goes, they're still the best solution.

As for CD-Rs, I think they are getting cheaper and shittier all the time. I burn audio CDs at the slowest speed and I still get jitters and pops from time to time. The best brand of CD-R of all time (IMHO) were Verbatim Blues, man I loved those things.

Semantics aside, what it really boils down to is: how important is your data, how much do you want to spend (in time and money) backing it up, and what kind of disaster are you trying to protect your data from?

"That's a good question, because data is very over-rated, and data is very under-rated. " -- Denver Sports Talk Radio

Daily (constantly, actually) we perform DDT (disk-to-disk-to-tape) backups.

What is the appeal of tape in 2011? I honestly don't know, this isn't me being smug. This isn't me being the obnoxious Internet cretin that posts like this, like this - Flack, Flack, are you watching? Like this:

"Tape? What is tape?"

GYysdnNSDHs I hate that guy. So that's not me. Is tape considered more reliable than hard disk space, at the cost of speed? I seek knowledge.

Also - I think I bought a pack of shitty CD-RWs at Best Buy. I've noticed errors on two audio discs that I have burned, and twice now I haven't been able to get Ubuntu burned to CD. I just want to run some fucking Ubuntu.

Also, you are right -- robocopy is not a part of XP. It came with Windows NT 4.0 and has been available for download ever since then. There's also a front end you can download and install if you find the command line syntax confusing, although note that the old version (the NT one) and the new Vista/7 version are slightly different.

I installed robocopy on one of my XP machines. So it's pretty easy to get going - if you have any troubles, Commander, please let us know in this thread. We are learning things in this thread and it is GOOD.

Tdarcos wrote:

Flack wrote:If it were me, I'd just buy a second 2TB drive and either run RAID1 (a mirror that you would never have to worry about messing with), or set up a scheduled nightly backup to copy files from one drive to the other. You could either install it internally, or externally as a USB device.

I haven't found any good automatic backup software that is free and does the job simply. I like AMANDA but the server is only available for Linux. So I think I'd still be stuck with having to back up stuff manually.

Bacula is cross platform and free.

Not particularly simple, because it's a real backup program, but still.

Bruce

Semantics aside, what it really boils down to is: how important is your data, how much do you want to spend (in time and money) backing it up, and what kind of disaster are you trying to protect your data from?

Before we talk about my backups, or your backups, let me tell you about the backups I/we perform at work.

Daily (constantly, actually) we perform DDT (disk-to-disk-to-tape) backups. That means all of our servers get backed up to a giant collection of hard drives, which are then backed up at that point (instead of from the servers) to tape. This allows us to quickly restore files for users from disk rather than have to search catalogs of tapes to find a specific file. We also do it this way because, since we back up so many servers locally, that we literally could not finish our daily backups within a 24 hour period.

Our data center is connected to a second data center in the building next to ours via fiber. All of our critical servers are virital, and all of them are duplicated across the street and our users never know exactly which server they are hitting. Should my building explode, none of our external customers would notice as all network traffic would simply be routed across the street.

Additionally, all of our data gets replicated in real time "somewhere else in the country" with about a four hour lag. That means that if an F5 tornado hit our data center ... wait, that's a bad example. Our data center is in a basement that was designed as a nuclear fallout shelter during the cold war. If an F5 tornado leveled my building, we (the "celler dwellers") might not even hear it. But let's say an atomic bomb exploded and wiped Oklahoma off the map. Co-workers of mine could bring our entire infrastructure back online with four-hour old data, halfway across the country.

In addition to all of this, we have a disaster recovery binder that includes DVD copies of all of our software, license keys, and instructions on how to recreate our entire infrastructure from scratch. There are three duplicate copies of that binder: one is in a fireproof safe in our data center, one is at Iron Mountain (yeah, I know who they are; I have a swipe card in my wallet) and one is at our disaster recovery site. Our tape backups get rotated out through Iron Mountain as well on a weekly basis. Old ones sit in one of our safes.

The reason we have such a robust backup system (besides the obvious) is that we are required to maintain copies of all of our data for at least 7 years, and some of it for 30 years.

Now my guess is, you have neither the funding nor the desire nor the need to perform backups like that.

The first thing you need to determine is, are you trying to protect yourself from file deletion, hardware loss/failure, or both? Those are two different things that require different backup approaches. For example, to protect yourself against accidentally deleting files, you need many days/weeks/months worth of backups, and you need access to them. It's tough to recover a file you deleted tonight and created last week when your last blu-ray backup was performed a month ago. To protect against hardware failure, you need two current copies of your data. And to protect against loss of hardware (fire, theft, etc) you need that second copy to be located offsite.

So if you want to protect yourself against everything, you need both local backups, and offsite backups. And you need a system to keep them current -- typically, that means automation.

Burning a dual-layer Blu-Ray disc takes anywhere from 15 to 180 minutes, depending on the speed of your burner. I don't know about you, but I've got better things to do than sit around burning discs all week. Hard drive backups can be scheduled and automated. And don't kid yourself, there are dozens of free Windows-based backup solutions out there.

I can't dispute that disc-based backups are probably more resiliant than hard drives if you go around dropping them.On the other hand, no one really knows what the life span of a blu-ray disk really is. I have 25-year-old C64 floppy disks and hard drives from the mid-90s that are still working, and yet I have store bought audio CDs that have bit rot and no longer work. For a semi-long term solution, hard drives are better than burned discs. For real long term solutions, tape is better than hard drives.

And in the long run, disc-based backups will never be cheaper than hard drive-based backups, because you can reuse the hard drive again and again. Do enough backups, and the cost of the discs will exceed the cost of any hard drive. If you're just talking about making a single copy of some files and mailing them to someone else, well yeah, that's pretty cheap, although I am willing to bet I can buy a 45 gig hard drive cheaper than you can buy a blu-ray burner and 2 discs.

For your books, you should set up Dropbox. You can get a 2GB completely free. It appears as a folder on your desktop, so you could either copy (or just store) your files there. You can access the folder from any machine that has Dropbox installed, so you could work on your book anywhere (the library, a friend's house, etc). It's only 2GB, but it's free. If you need more space, you can expand it up to 100GB -- or, you can look into that Crashplan.com site, which is $5/month for unlimited space.

Also, you are right -- robocopy is not a part of XP. It came with Windows NT 4.0 and has been available for download ever since then. There's also a front end you can download and install if you find the command line syntax confusing, although note that the old version (the NT one) and the new Vista/7 version are slightly different.

Flack wrote:My backups are all batch files using robocopy command line copies.

Robocopy isn't part of XP, it apparently is available on Vista and 7. There is a freeware program ("Autobackup") to do automatic copies of files that change each day from one place to another, so I could just use the drive on my old computer, but have it back them up to duplicate space on the new computer by running the backup on the old computer as the equivalent of a cron job. Then backup files from the new computer on Blu-Ray and CD or DVD depending on frequency. I had to back up all my files related to the books I'm writing, since it was only about 400 meg, I backed it up on a CD. No sense wasting a DVD for that little.

Flack wrote:The cheapest blu-ray burners on TigerDirect are $69 for the drives

Uh, Flack, you have to read the description very carefully. Those are BD-ROM, i.e, a Blu-Ray reader. They can write DVDs but can only read Blu-Ray. There's a confusing one for $79 which you have to read the specs carefully to understand it does not write BD disks, it only reads Blu-Ray.

Blu-Ray writers are around $120 to $130. But I found one listed in the specs that does support writing BD disks, it's $109.

Ice Cream Jonsey wrote:I read something the other day that made me get a 16 GB flash drive.

I have one of those, I think I still have some of my music on it. Wish I had saved more there. But I think the cost of hard drives have dropped so much that a 1 TB is about the price of a 16GB jump drive these days. I checked; Amazon sells them for $30. A 1TB is around $70, and amazingly enough the next 1TB only cost me $20 more. I know what it was; when I purchased a 16GB jump drive a year or so ago it was around $69.

Ice Cream Jonsey wrote:(Actually... home PC, a flash drive, another PC and then a repository off my Dreamhost web host, using Subversion.)

I'm still trying to figure out how to set up an external Subversion host, I have one internal on the same computer for the files in my book, but I still want to figure out how on an external system. There's at least one free Subversion service available I could use to back up my files for my project but I can't figure out how to get it to work.

Ice Cream Jonsey wrote:(I had no idea terbyte drives were so cheap. I gotta get me one of those.)

Yeah, if you look carefully you can get 1TB as low as $59. With a 2TB Seagate with 5 year warranty for $89, it makes a lot more sense to spend the extra $30 and get twice as much space.

Some people think it's sleazy the way memory manufacturers call 1 GB 1024^3, but disk space manufacturers call 1 GB 1000^3 (and thus 1 TB is 1000^4) but I disagree for one simple reason: hard drive manufacturers have always transparently said so, and didn't try to hide it. Always clearly declared on the packages.

Flack wrote:If it were me, I'd just buy a second 2TB drive and either run RAID1 (a mirror that you would never have to worry about messing with), or set up a scheduled nightly backup to copy files from one drive to the other. You could either install it internally, or externally as a USB device.

I haven't found any good automatic backup software that is free and does the job simply. I like AMANDA but the server is only available for Linux. So I think I'd still be stuck with having to back up stuff manually.

Further, it's part of the same "all your eggs in one basket" situation that got me in trouble in the first place.

Flack wrote:The cheapest blu-ray burners on TigerDirect are $69 for the drives

Thanks for the tip, I'll check it out.

Flack wrote:and the have 15 packs of blu-ray discs (25 gig) for $30.

No, I got two 15-disc spindles for less than $29. A very good deal.

Flack wrote: That right there is around the same price as a new 2TB drive, and with a second hard drive the backups will run themselves, instead of you having to change out discs continually, plus continually buying more media.

I want my files backed up to disc so that I can have them outside of the computer, for the reason you run backups, so that your backup media is someplace other than in the computer. With files backed up on disc, I can also take them someplace else if I want to make a copy someplace else. Or I can mail a copy to someone else.

I mean, if I was, say, system administrator of a network, I'd be taking backups of our files and either putting them on external hard drives, or, more likely, backing them up on Blu-Ray disks and sending either the drive or the disc to Iron Mountain (a salt mine company in Pennsylvania that rents long term storage in a natural humidity and temperature stable environment, plus the salt keeps bugs away.)

If you have backup copies on disc, you can file them in a binder and put it away in a piece of furniture or a closet; if you drop it, chances are it won't matter. Drop a hard drive and you're in the situation I was in.

Plus it's much cheaper for file backup, I can back up 45 GB on two discs I can mail anywhere for about $6 including postage.

There's another reason, by getting a Blu-Ray drive I get my toes wet with what is right now bleeding edge technology, and I have the capacity to watch a Blu-Ray disc, an extra benefit I don't get with another hard drive.

Beyond which, the 2TB was really overkill for me, I could have bought 1 TB and it would have been far more than enough. I am, however, thinking about doing more video, and that eats a lot of space, and to make sure that's properly backed up I'd be better off with Blu-Ray.

I can back up everything from my old computer (120GB of the 160GB) on 4 BD disks, so the cost to me is about $4, and I can put the discs away in a binder and still the data is online and even a massive power surge destroying every piece of equipment here leaves me with a complete copy unaffected by power surges.

I mean, I see the point and if I strictly went by net costs, running two drives as pure mirrors might be a good idea, but I still think having backups which are not on-line is a good safety measure.

So yeah, that's pretty much my backup system.

I have what I call master batch files, and sub-batch-files.

For example, here are two of my "sub-batch-files":

home.bat:

robocopy d:\Home f:\Home /mir /r:3 /w:1

mp3.bat:

robocopy d:\mp3 f:\mp3 /mir /r:3 /w:1

Then I have a master batch file called "1TB_Backup_1":

call home.bat
call mp3.bat

The reason I do that is so if I rename a directory or change a drive letter or something, I only have to change it in one place, instead of hunting through all my backups. It's more of an enterprise approach and you can probably get away with just using a couple of dedicated batch files.

Also, note the switches I use with robocopy:

/r:x denotes the amount of retries. I set it to 3 because the default is one million (I am not kidding). So if your network connection drops or something it will literally retry a million times before moving to the next file. I do backups so frequently that I just try three times and move on. I'll get it tomorrow night.

/w:x is how long it waits between retries. I set it to one second, but the default is 30. This is basically for WAN copies, where in 30 seconds, something might go down and come back up. I'm doing local backups in my house, so if it's down, it's down.

Keep in mind with /MIR is that you will always mirror the destination to the source ... so if you delete the source and run the batch file, it will delete the destination as well.

If you have access to an FTP site, check out Fling FTP. I used to use that and it works great too. Basically you set up backup jobs and it FTP's them to a remote site for you. As long as the program is running, it handles the scheduling for you. For a 6 gig directory, I'm sure you could find a service or a friend that would let you back that up regularly and remotely. My problem is, my home directory is (just checked) 697 gig with 256,236 files. That's far past the "I need a favor" FTP limit.

Also, I looked up the one my friend is using: crashplan.com. $5/month per machine, unlimited space. It's not free, and I like free, but it's remote if you need that.