On-line backups ??

I am somewhat puzzled by the idea of on-line backups.

I think I must have a fundamental misunderstanding of something because I can't see how it can be done in any reasonable time fame.

Take my system, for example. 4 TB of data = 32 Tb of data, upload speed averages 1.5 mbps.

32x10^12 bits divided by 1.5x10^6 bits/sec = 21.33x10^6 seconds = 8 months, more or less. How could I conceivably spend the better part of a year, 24 hours a day seven days a week, doing a backup? It doesn't seem feasible to me.

Am I overlooking something here?

tanstaafl.

Your math looks right. However, 4TB of data is way over the average amount of data most people have. The current online backup solutions are geared more towards the average user who has backup needs in the range of 10s of gigabytes.

Beyond that, some of the backup providers do clever tricks to minimize data transfer. One of them out there that does a complete system backup won't actually transfer files over the wire if they see a duplicate already stored in their system from another user. This cuts down on a lot of bulk data from Windows and installed applications, leaving just the users specific data going over the wire. Others just ignore all system/application files and only back up what a user has in My Documents.

One final trick I know of is that some backup providers allow you to mail them a drive for the initial backup, this ensuring only differences from that backup have to go over the wire.

For me, I only backup about 30 gigs offsite, with most of it sitting on a Time Capsule at my grandparents house two states over. The rest of my data I'm not overly concerned with losing. I do have local backups in the house to a RAID in a NAS, and if a large enough disaster strikes to wipe that out, I have bigger problems to deal with.

Most people with multiple TB's of data don't need to send it all up to a backup site because the content is already backed up on hundreds or thousands of systems around the world and they just need to fire up a bittorrent client to download it again.

It took me weeks to send up just my digital photos as I've only got 640Kbit/s upload capacity. Yeah, by 2015 the CRTC says I should have 1Mbit, but it's not proposing that be passed into law or anything.

Even if I had the required transmission rate, I've also got a 120GB per month usage cap. 4TB would take a lot longer if I had to meter it out. Probably over 6 years if I was going to be doing any downloading at all. 2.8 years if I was using up the whole cap for only uploading. Yuk.

I've got probably 100-200 GB of critical data that I'd consider backing up online. That's in addition to the local and offsite backups though.

You mean 640Kbits/s, don't you?

That's pretty grim, even down here in the wilds of Mexico I've got double that on a bad day, and as much as 1.9 Mbit/sec on a really good day.

I have the fastest internet connection it is possible to have here around Lake Chapala, at 7 Mbps down, 1.8 Mbps up. That is one third the speed I had in California, and the service fails on average once every couple of days for periods ranging from a few minutes to a few days. As a bonus, I also get 75 TV channels to watch, but since I disconnected my TV set five months ago, I don't get much benefit from that. I could have just straight internet (without the cable TV) but for some strange reason that costs about 40% more than internet plus TV. Go figure.

tanstaafl.

Doug, your math is right, but as Tom said, most people aren't backing up 4TB of data.

What do you have that's taking up 4TB of data? If it's DVD/bluray rips, then that's also not the typical thing that most people are backing up. If it's all photos, then dang, you're a pretty prolific photographer Good for you!

Also as Tom said, very few of these backup companies are still offering unlimited storage. Some are, but they still bank on the assumption that most people are going to go with them because they think they'll need the unlimited space, but end up only using a few hundred GB at most, probably around 100GB on average (which might still be a little high).

Also figure that while your upload speed is typical (for the US at least, I don't know about Mexico), there are others with faster speeds. I have nearly 5Mbps, and it took me about a week for my initial upload of around 300GB. I do wish I had Bruno's 640Mbps connection, though Are you in Kansas City, Bruno?

After the initial backup, the differential backups are quick.

Online backup isn't feasible for everyone, but I'm a big fan of it because it works well for me.

I still think it's good to have a local backup as well...

Ha! I accidentally capitalized a lowercase "k" as an "M." Woops!

I'm in the same situation as Doug, but I've "only" got about 3TB of data. Most of it is indeed DVD rips I've made of my DVD collection. However, this took so much time, I really never want to do that ever again. I would love to back it up somewhere offsite, but as Doug says, it's just not feasible at current speeds.

Upon reflection, there's really only about 2.5 TB that would require backup, the rest is on-site backup of the original data and I obviously wouldn't be backing up my backups.

A great deal of my data is audio (my music collection, every single episode of NPR's Car Talk since February 2002, more than 3,000 audio books, plus photographs (mine and SWMBO's), games, income taxes, Rosetta Stone, etc. I have 6.5 TB of capacity, of which (including backups) about 5 TB is being used.

No doubt I could go through and get rid of at least one TB of data I'll never need nor look at again, but with hard drive storage so inexpensive ($80 for 2TB at NewEgg) why bother?

It looks like my best solution is to get a couple more 2 TB drives (WD20EARS, probably) to use in my external dock as backup drives and keep them at my neighbor's house.

tanstaafl.

I've been investigating and trying out Crashplan (I think the recommendation came from here). It kind of gives you the best of both worlds. You can back up locally on your LAN but also remotely to a friend or to the Crashplan servers. If backing up to a friend you can "seed" it doing a local backup and then transfer the backup somewhere else. Once seeded it only backs up differences. All of this can be done free and probably in return you could back up your friend if similarly inclined.

So far I'm pretty impressed and I've looked at quite a few "cloud" only solutions all with certain limitations that were frustrating.

Elephantdrive - slow uploads (they wanted me to break it into multiple uploads) and the software was using 600MB of memory/pagefile.
Sugarsync - just didn't get into it
Opendrive - limited versioning and backup type functions but economical
+ others

plus the many others that I reviewed but didn't actually install

Crashplan is definitely the new darling, easily beating out Carbonite.

Crashplan offers (for a not-insignificant amount of money) to send you a drive to back up to and mail back to them. I don't know if they do multiple hard drives, and if not, that still means only 2TB at a likely maximum, if not less.

When my Mozy plan runs out I'll be switching to Crashplan. I'm already recommending it to clients.

Crashplan seems like a super nice service and super cheap if you only have a reasonable amount of data to back up. Considering I have an unreasonable amount if I include everything, that wouldn't work out.

I think I'll stick to RSYNC from my ReadyNAS (which is the 2nd copy) to my web server for photos, but I might consider Crashplan for other document-type files which I also need to keep absolutely safe.

So, this brings me to the ULTIMATE question. Does anyone have a solution, commercial or otherwise of a truly completely idiot-proof photo backup procedure?

This is my ideal: A small box with an SD card slot connected to my parents' network. They pull an SD card from their camera and put it in the box. AUTOMATICALLY all the files are copied off the SD card, verified and then they remove the card. That box then uploads all the photos to a backup site. Which can be my web server or Crashplan or something like it. That box should probably have a drive in it to hold the copies. Optionally all the photos would also get sent up to some social site like Flickr or Facebook - but require privacy settings so we can restrict viewing access.

The bottom line is I'd love a way for them to be able to unload the images off their camera memory cards without interacting with a computer.

And if something like this doesn't exist, who wants to help me create this product and commercialize it?

Bruno, eye-fi does that, but wireless, so no box involved. Well mostly, it does not delete files.
In our case, photos are copied to local desktop, (if on home wireless, if not will do later) and Picasa (hidden by default),it can copy to other photo sites. From the desktop they could be backed to crashplan etc.

The camera does need to be on for an upload to occur, but if you have a load to do you can put the SD card into the USB reader and it will copy automatically from there.

edit

I second the Eye-Fi. I keep begging my mother to let me set her up with one, but she's resistant to technology change.

It's too bad, because for people like my mom, the Eye-Fi is magical. You simply bring the camera home, sit it on your desk, and power it on. All the photos are automatically transferred to the computer in a folder structure you define (I have it set for YYYY-MM-DD). Like Phoenix42 said, you then have the My Pictures directory (or wherever they're saved) set to backup and it's all automated. Much easier than having them move SD cards around.

I have one in my camera, but I'm probably going to move away from it because I want to shoot in RAW+JPEG, and RAW files take an eternity to transfer. JPEGs are pretty quick, though.

The other primary selling point of the Eye-Fi, though (at least the high-end ones) is the automatic geotagging. On the high-end cards, you get a lifetime geotagging service. It's not GPS, but it does an amazing job at estimating based on surrounding WiFi. Honestly, it's pretty impressive. It's fast, and it seems to get the location in even the most unlikely places. Anyway, it's a great feature for the parents.

I'm going to check exactly what model camera my mom has and the Eye-Fi (and me setting it al up for her) will hopefully be a nice mother's day gift. She'll be ecstatic at the prospect of more easily using her digital images.

Can you set up multiple backup destinations? For instance, can I send to FTP *AND* Costco *AND* Flickr *AND* xxxxx ?

I want to say yes, but I'm not certain, I'll verify with my card if I can configure an additional web target beyond what is currently configured above.

Even just two or three targets. I definitely want to send to FTP at a minimum, but also sending elsewhere would be great. Somewhere easy where I can teach her to do something with them. Perhaps Facebook if I can set the default album to be completely locked down and then give her the option of sharing individual images. I'm pretty sure she knows how to move around somewhat in Facebook (the new AOL)

At least with the version of the Eye-fi software currently on my machine, you're limited to uploading to one home computer and one online destination. FTP counts as an online destination, so if you upload to FTP, you can't also upload to Picasa, Flickr, and friends without some hackery. Some folks on the Eye-fi forums have suggested Pixelpipe as a multiplexer for the online services,

I almost always want to clean up my photos before uploading them, so I just have it deposit them on my file server via FTP and do my uploading to Picasa using the Picasa desktop app.

Hardware: A small little Atom-based system (eg. one of the Zotac HD-ID* boxes), or even a netbook --> say, a $200 Acer model with SD-slot.

Software: Ubuntu Linux (configured as "read-only"), and some simple shell scripts to monitor the SD slot and do the uploads.

That's what I would do, were it me and my folks.

Cheers

Mark, a small little tiny hand-sized system is what I was thinking. But I'd be looking for something that I could install and configure at most. I don't want to spend any time writing scripts nor inventing the technology if it's not something I'm going to turn around and sell. Mainly because it could take away a lot of time/energy from other things I'm already doing. Plus if my wife found out she'd kill me. Lots of projects around here to finish still.

How to set up PixelPipe with Eye-Fi - the instructions were easy to complete, even though it's unsupported by either Eye-Fi or Pixelpipe officially. There used to be posted on the Pixelpipe blog, but that link has apparently been deleted.

So far I've tested relaying from Pixelpipe to my photo-specific web site via FTP. Very slick. The Eye-Fi software has a pretty crappy UI, having been written in Adobe Air and not though out well enough, IMO. But it does get the job done. I'm surprised the company hasn't set up its own service to pipe photos to multiple sites at once.

This *should* work out well for my mom. So long as she doesn't lose her camera again. I'm having to go a month early on her birthday gift as well by getting her a new camera to use with the Eye-Fi I just got her for mother's day.

Wow, that is scary.

I've been looking at some cloud-based backups in the last days, and I remebered reading this thread a while back. So, Crashplan does look like the best out there, I think, but I am a bit concerned about security. I may be paranoyd, but I'd really NOT want Crashplan be able to look into my files, irrelevant as they may be. So, does anyone know how their service works at this regard? Clearly, file >transfer< is encrypted. But, then, once on their servers, I suppose they are accessible to their system admins? I'd love my stuff to be a bunch of meaningless bits to them, and nothing more.

Also, I too have 500GB of "valuable" files that I'd like to backup. I have a SHDSL, 2Mbps symmetric data link at home. It would still take forever to transfer all... But still, if the security part was convincing enough, I may be considering this.

I believe if you opt to apply the extra private password option then your data is supposed to be completely unaccessible to their admins. I think there is also an option to provide your own key. There is stuff in their FAQs that covers it

So very different to some of the other off line backup systems.

Taym, Crashplan encrypts your data before it leaves your computer to go to their servers or even to go to another computer in your home. This page has more info.

The nice part about Crashplan is that if you do have a very slow connection, at least it also includes the ability to back up locally as well. The application then detects which method will complete the fastest, and starts with that one. That way at least you're safe from hard drive failure and any environmental scares that would be localized to the area of the original data (burst water pipe over your computer, thief takes your computer but doesn't see system in another room, etc).

Thanks guys for pointing me to the right page. That's very interesting. I am seriously considering to set up some "reciprocal" backup with friends, to begin with.

Excellent idea, I'd forgotten that you could do that for free. Very good idea. Of course, you'll need to make sure they have connections that will accommodate this too.

There was an email to a podcast I listen to recently that said they'd backed up several hundred GB of data to Carbonite. His computer crashed, and when he went to restore the data, he realized he was VERY quickly reaching his bandwidth cap. He called Comcast and all they could tell him was to try downloading the data over the next few MONTHS. Ugh, caps are going to be a big problem very soon...

Yes, Matt, that concerns me a bit, I have to say.

Even assuming Crashplan has some recovery service where they ship a HDD or a set of BDs to you - which I don't think, or at least I have not yet read any such thing - still it would take time to get your data back, especially if you're not in the US. Your files at a friend's house is going to be available "immediately", if the whole thing is well planned.

I've been using CrashPlan for just over three weeks now. In the past I used Mozy and I've also for a long time used my own set of rsync scripts that backed up to some "unlimited" web space that I had access to.

Mozy never really worked for me, it took 5 months to do my initial backup and it just couldn't cope with very large files (my large virtual hard discs caused it to stall the backup for days). The "unlimited" web space provider recently spotted I was storing 100GB+ on their server and asked me to remove it. So I needed something new.

Over the last three weeks CrashPlan has backed up 60GB of my data to their cloud. That covers my photos, documents, source code, email and other important things. In comparison, it look 5 months to backup the same data to Mozy !

I still have 150GB of music and other assorted files to backup. After that I'll try backing up my larger virtual disc images (it has already coped with some 6GB ones that Mozy choked on).

I'm also using it to backup the whole of my Mac Book to my server, works well and is painless to use. It is also running within my main Windows virtual machine, continuously backing up all my local source code repositories (CrashPlan on that virtual machine is also simultaneously backing up to their cloud).

So it all seems to work pretty well. I am about to sign up for their $6 a month service (to cover more than one computer and various extra features).

It isn't perfect though, I have a few complaints.

The first one is fairly minor, it isn't very good at estimating the remaining time. It very naively assumes that if it has backed up lots of compressible stuff, that the rest of the backup set is similar. For example a few minutes ago it was claiming that I had 2.3 days left to backup 150GB...

Its bandwidth throttling is very inflexible. You don't get to vary the throttling based on time of day. Also the selection of speed limits are oddly restricted, for example you there is no option between 300kb and 1mb !? I'd really like to have it use very little upstream data during the day and have it fill most of the line over night.

It has, for me, a very serious bug/bad bit of design. There are a bunch of settings around whether you are backing up over the LAN (local network) or WAN (the Internet). Settings like speed for example. So you can tell it to use unlimited speed backing up to other local PCs and limiting the bandwidth used when backing up to their cloud.

All good, except they have screwed up their LAN/WAN detection. Rather than making use of netmasks, they decide that if they are backing up to a public routable IP address that it must be a WAN destination. So for example if I want my MacBook to backup to my local server, that will be seen as a WAN connection as they are both on public IP addresses. So I can't restrict the Internet bandwidth used by the Mac when backing up and also have it backup locally at full speed.

The final major negative point is the lack of account wide de-duplication of data. If you have two copies of a file on a machine, it will only back the data up once. If however you have the same file on two different machines and they are both backing up the cloud, that data will be sent twice. I believe they have plans to address this, which will be good.

I do wish that the local backups were stored as plain files, rather than stuffed into their own proprietary databases. But I understand why that is, as they actually backup blocks of data rather than files.

I have also had the backup stall a few times, for serval hours, seemingly because the server my client was talking to was having problems. Something I will have to keep an eye on.

You have to worry of course about whether it will stay unlimited, though they have at least been very clear to explain that they intend to stay unlimited and their reasons for thinking that they can when others have not. Only time will tell.

Someone pointed out to me that the $6/month family service is actually "as low as $6/month", but only when you pay for 4 years in advance.

You can enter a custom value of any number but I agree it's not the most flexible. I'm still trying to figure out a way to let it run full blast all the time and be throttled at my firewall/router.


Remember they are encrypted anyway. I kind of like the 4GB blob method. Hundreds of thousands of files can be unwieldy.


They do also say they will give a pro-rata refund if you cancel. Doesn't help if it's taken you 6 months to get everything up there though.


I paid the full 4 in advance for the cheap rate based on the pro-rata refund mentioned above.

There are some niggles I have with it but overall I'm happy. My upload speeds are fairly slow so it's taken me a few months to get to about 300GB "in the cloud". The real important stuff is done though.

I created separate jobs backing up the higher priority stuff first. One niggle there is I have a local and online copies. It does the local copy only first and then assumes that's enough until every thing else has at least one copy. I get the concept but would prefer the higher priority stuff always take priority for all locations.

Just to add my configuration.

I have a NAS running unRAID. It runs Crashplan and backs up to their online service. This is where the important stuff is (photos, MP3s, files etc).

My laptop, MythTV machine, work laptop etc backup to the unRAID NAS locally. Some parts also back up directly to the online service.

Crashplan can't backup its local backup store to anywhere else. Not really any point in that. Just backup directly.

Something I've found quite useful is the ability to restore through the webpage. Once or twice whilst away for work on a corporate LAN I've needed a file. I can't VPN out or anything so I just restore it via the webpage to the machine I'm on.

I did also find an issue when the linux NAS didn't have the right codepage causing Crashplan to skip a number of files e.g. "José González". Not really Crashplan's fault though I only found it by doing an audit of number of real files vs number it said it would restore.

Oh and another minor niggle. The backup page counts number of files + number of folders as "number of files"

Ah yes, hadn't spotted "Other" hidden down there at the bottom.

I want my bandwidth control in the software. I've also got bittorrent uploading data 24/7, controlling the two sharing bandwidth at the router would be much harder than having a scheduling option in the software (as uTorrent already has).

I really need them to fix the WAN/LAN detection. That is a completely brain dead design choice they've made there.

I could really do with the option to spot when a machine has moved from its home LAN too, so that you can ramp back the bandwidth usage (or disable it completely) when you take you laptop elsewhere.

I'm glad to hear that you're liking it overall, Andy. I'm a big fan of Crashplan and have installed it on several clients' computers.

I went with "$6/month" for three reasons: 1) it's the cheapest option, 2) I can put my entire family (in-laws too) on this one plan, and 3) I don't mind the four years in advance because that at least assures that the price isn't going to change on me for the next four years. If they change their plans tomorrow, presumably they'd honor the deals they've already made and grandfather me in. Then, in four years, if I think the new prices are horribly high, I'd wager that by then we'll have methods of backup available to us that we haven't even thought of yet


One thing I wish Crashplan could do is backup to a network drive, or backup files on a networked drive. This is a big irritation for me, as I don't have a server that I can install Crashplan onto.

How are you guys still finding crash plan?

I've just installed it as I'm looking for a decent backup solution, I'm primarily interested in backing up my source code off site.

There is something that I particularly dislike, they seem to have some sort of odd reseller system, some of the products can't be purchased in the UK (seriously, this is just data) because they have a UK reseller - although it would appear that the UK reseller uses their own storage, and of course it seems to be much worse value for money than the US offering.

Selecting backup folders is a bit nasty, no way of sorting stuff, so you have to scroll through the tree, I'd prefer a list of files/folders.

Regular expressions would be nice on a per folder basis too, would save me having to build the folder name into the regular expression.

Adrian

I'm still using CrashPlan and still reasonably happy.

I do wish they'd get on and fix some of the important design oddities. In particular I want them to fix their local/remote detection. Assuming that every machine hosted on a public IP address is remote is lazy and daft (and it a real *pain when it comes to restore time).

Having UK storage sounds like a good idea, to avoid the hop over the pond. And I'll bet that hosted storage in the UK costs more than in the US, so it is bound to cost a bit more.

When it comes down to it, it works much better that the other online solutions I've tried. All of which either don't cope with very high overall storage levels or can't cope with massive files (>40GB virtual hard disk files).

I use it to backup the massive virtual disk images on my laptops to my server (350GB of data backed up). I also use it to backup 285GB from my server to CrashPlan Central (and also a small amount direct from my laptops to CrashPlan Central, things like the local git repos for my current projects).

I like its flexibility compared to the others I've tried.

I have never yet made use of the backup to other users feature, as my 800k upstream is just too slow to host my servers and to start swapping large amounts of data with other people.

* to restore to my laptops from the server, I have to temporarily raise the server's send bandwidth limit, as the server is treated as being remote from the laptop on the same local network because I use public IP addresses

Cool, thanks! Sounds good.

Another annoyance is that you can't edit a regular expression, you can't even copy it to clipboard, so if you want to make a change, you have to enter the whole bloody thing in again, now that's irritating.

Still, it seems to be doing what I want and I take a lot of faith in peoples opinions here - and it seems to be that people think it's good.

We have bt infinity at work and I have 50Mbit here at home, so I might give the "friend backups" a go as well, can't hurt.

I've probably got a gigabyte or so of source code that needs backing up, other than that, there's not an awful lot that needs backing up.

Out of interest, when you backup your virtual machine images, do you stop/pause the VM? How does it cope with backing up files that are "in use" like a VM?

Thanks.

Adrian

I do have a tip if you are trying to use it to backup large virtual disk images.

Use snapshots to stop the vhd from being updated all the time. I keep a snapshot running, so that CP only has to scan the smaller snapshot file.

Every couple of weeks I merge the snapshot back into the main file, at which point CP has a few hours work to do to scan the vhd, but at least that means that on a daily basis I get to have a current backup of the image without CP spending hours each day processing a massive file.

I'll guess that I'm not the only person out there doing their Windows development in a 50GB VM on OSX

I thought someone might ask that

At the moment I'm not stopping/pausing them. I haven't had any problems yet and I have restored a couple of them.

My main dev VM is running pretty much 24/7, so it has lots of time when it is sat idle. I expect at that point all the guest OSes caches get flushed, so anything important gets written out.

I guess what I should be doing running script once a day to create a snapshot and then kick off and explicit CP run. But to be honest recovering the data in the lastest snapshot isn't really important, as long as I can restore a vaguely recent working copy of the VM so I don't have to rebuild it all when something goes wrong.

All my important data within the VM is backed up separately from CP running within the VM anyway And the only data I really care about on it is source code, so the whole open file issues don't really apply.

That's a good idea for minimising the amount of data that could be changed by the virtual machine, I hadn't thought of that trick.



You're not!

Although we use parallels, our actual "data" resides on the mac and is mounted by the VM, so the data gets backed up by the host and doesn't exist inside the VM, the virtual machine basically just exists as a build/test environment, so losing it isn't really much of an issue, it just means creating a new VM image with the necessary build tools and libraries installed - a pain, but not critical and as you say, it's the source code that is all I really care about too.

I also still recommend Crashplan. I've been using it since June of last year, and I've had one or two points where I've needed to do a recovery. It always works just as promised.

I've set up a couple of my clients with the "back up to friends" destination, and it works very well.

I've also set up a small real estate group with the "backup to another computer" destination. That's used mostly for those on the Family Unlimited+ plan, and you get a list of the other computers set up with Crashplan on your account. You're able to back up to any and all of the other computers. Recovery from these locations, if they're local like with this real estate group, are very quick.

So yes, I'm still very pleased with Crashplan.

I think the only thing I dislike is how difficult it is to work with mapped drives like NAS devices. How are you doing that, Andy? I've seen a couple solutions to this and was able to get one of them to work with one client of mine, but I didn't have a great deal of confidence in it.

One way I've seen is to change the process so that it runs as the local user instead of the system. That way it can get around some security feature in Windows that prevents accessing mapped drives or something.

Oh, wait, you're running this on OSX, aren't you? Nevermind then, it's easy for you

Still happy enough here.

Like Andy I found many other solutions did not seem to handle large numbers of files/data. That was even on more enterprise levels a few years back when I was looking for a solution for my old company.

Took about 6 months to get about 800GB uploaded to their backend. The deduplication works quite well too. Currently I'm at 959GB and the server reports 672GB, so there's about 30% compression/deduplication in there.

Originally I only was looking at about 2-300GB but with the unlimited plan I just added more stuff that doesn't change at a lower priority and eventually it was all up there. I do backup VMs but hadn't bothered doing anything particularly special.

In the time I've been using it (about 1 year now), there hasn't been a single update I don't think. There are a few quirks and inflexible things but really I find it's set and forget. It just works(tm). Most of the things I would prefer to change are more related to the initial backup stage. e.g. if you have two backup jobs and a local and remote server for the high priority one but only the remote server for the lower priority job, it will do the local high priority one first and then the lower priority job remotely. I realise it's trying to do make sure all the data is backed up to at least one place first, but that's not really what I wanted initially. I was able to work around it by fiddling with some priorities. I wanted the higher priority backed up remotely first (most important stuff like photos etc). This is still an issue if I have a large amount of new data in the lower priority backup. It doesn't do the high priority/remote server stuff first.

Every now and then I do an audit to make sure everything's there and apart from the initial issue with the code page (more my Linux box's faulty), it's all been good. Need to do another one actually.

One thing that does occasionally annoy me (particularly on my work machine) is that the CrashplanService.exe + CrashplanDesktop.exe (if open) processes seem to consume a slightly excessive amount of CPU. My work machine (a Dell Precision Workstation) is a few years old but not that underpowered. The corporate coreload does seem to have some wierdness though (e.g. a sometimes Remote Desktop session is unusable because it seems to be spread 1% CPU around a lot of processes. Go and log on locally and it comes good straight away).

A couple of times I've found my remote server e.g. atls5.crashplan.com has "disappeared" and it couldn't connect. At least twice that was due to network issues at their end. Whilst it is kind of sold as a cloud, it does seem that specific machines (not accounts) are assigned to specific backup servers. They do rotate around and I've seen at least three or four different servers assigned to my NAS backup account at various times.

I also can't find another company that does it for as little money. That's the only thing that worries me, that they're going to realize they're not charging enough and raise their prices. Oh well, I paid for 4 years up front, so presumably I'm safe for another 3.5 years unless they go out of business

To compare, Mozy would cost me $35 a month if I didn't add any more data to my backup. Carbonite initially looks like it has even better prices than Crashplan, but that $59/year doesn't let you back up ANY external hard drives! The higher priced plans only let you back up one! Also, their software client is the worst of these three.


Where do you see this information? I'd like to check that out too...


I've found it to be the same. That's exactly what I want from such a service. I have noticed some of those server-side issues, but they're very infrequent.

As an example of what I'd call a quirk - really it's a UI bug but something I found when doing my audits that is confusing.

On the backup tab it will show "'x' files" to be backed up. However this is actually files + folders.

When on the Restore tab it will show files and folders as separate line items. A bit confusing when you don't know.

Took me a while to figure that out and when I've mentioned it to support as a note alongside other issues, they haven't responded to that part. There may be a reason for it but I can't think of it.

Also where I have had real issues like the network problems, they were only responsive during the US business day. I can live with that I guess for $70/year.

I also paid the $280 for 4 years upfront.


I got the total data size (959GB) from the automated emails. I have it send one once a day and covers things like missed back ups etc. It also sends individual mails if a machine hasn't backed up for an extended period. My car PC is one that often does that because it's not online long enough in the garage to complete a backup. That one doesn't really need it though since the MP3s are rsynced from my NAS anyway but it's nice to have.

The compressed value you will see if you click on the "Crashplan Central" text from one of your backups on the Backup tab/screen.

I don't need to worry about about mapped drives on NASes, all my data is stored on really computers that can all run the CrashPlan service.

From reading their forums there isn't likely to be any software update any time soon. At the moment they are focussing all their effort on getting their enterprise version's features in line with the consumer one (the enterprise software doesn't support backup sets for example).

When you guys mention NAS, what kinds of stumbling blocks am I going to find in setting Crashplan up on Windows to back up specific folders, including the boot partition, to my NAS sitting on the same subnet?

This is really the only thing I need the software for, because honestly, all other backup software I've tried or read about just sounds like complete garbage. There just isn't any decent software available for Windows for this purpose. Nothing as nice as Time Machine in Mac OS at all.

My other option was just to set up a few scheduled backups via the ReadyNAS UI using rsync.

The inability to create a bootable 100% backup of a live Windows machine seems to be a real stumbling block.

Here, I do confess to having exactly one machine with WinXP-Pro on a hard drive. I've yet to figure out how to prevent having to do a full OS reinstall in the event of catastrophic drive failure.

The best idea I have so far, is to install a second identical drive, and boot from Linux/USB to bit-clone one drive to the other. Slow, and not terribly practical.

The second best idea is to do something similar, booting from Linux/USB, except do a file-by-file copy to the second drive (both drives ntfs). But for this scenario, I have no idea how to make the backup disk "bootable". Perhaps someone here knows a simple recipe for that?

Thanks

And along similar lines, I may someday want to move everything from the current mechanical drive to a SSD. Same problem. Hopefully with a simple solution.

Is there a reason why you can't just virtualise the WinXP machine ?

CrashPlan isn't designed for giving you a full machine recovery like Time Machine, it really is aimed at backing data up.

With more recent version of Windows the included backup software can in theory give you a full system backup. The problem is I've never managed to get it to run to completion (either across the network or to a USB attached drive) !

I believe the normally recommended "free" option is http://clonezilla.org/

I've never used it myself though. There are also various commerical apps, the most often mentioned one being http://www.acronis.co.uk/homecomputing/products/trueimage/

But again, I've never used that either

The only way I know of to make a full system, bootable windows backup is by using Windows Home Server. But something tells me that's not really up your alley.

Clonezilla appears to be a bit-cloning solution
(it's just a wrapper for the ntfsclone bit-cloner).
So not terribly useful other than that it can also
bit-clone to a self-compressed image file.

How do people move their Windows installs to SSDs?
There must be a few people on the planet who have done this
for WinXP. This normally requires partition realignment
to a 4KB or 1MB boundary, something that Clonezilla cannot handle.

??

Well, apart from the exact same issue (how to copy the existing f/s into a VM), I suppose that might be a good thing to do.

But it will screw up the software licences for WinXP (requires a telephone call to MS) and also for Adobe Photoshop on the same disk (requires a telephone call to Adobe, or perhaps just taking care to deregister before cloning, then reregister).

I wonder if the DVD copy utilities will still work from within the VM? Gotta try it, I guess.

The one Windows machine that I have virtualised, I used the VMWare migration tool, which is free. I then converted from VMWare's virtual disc format to VirtualBox's one using VBox's tools.

Everyone who has asked me that question about SSD conversion, I've recommended just doing a reinstall

For my HDD->SSD conversion, I used WinClone. Not a great answer though for those who don't have access to OS X.

http://bubba.org/winclone/

May be a starting point for finding the tools that work on the Linux side, as WinClone makes use of NTFSProgs.

Link

Yup, Window's will pickup on the hardware change, but when I last did this I was able to do it over the net. When it has required a call to MS it has been quick & painless, they didn't even ask why.



Copy? I was going to say no, but it looks like it is more of a maybe, and depends on the burner in question:
Configuring the Virtual Machine to Burn and Rip CDs, but this dates from 2009.
Random VMware forum tread, there are probably other similar threads.

I do CD ripping from within a VM. I've successfully done it with both VMWare and VirtualBox under Windows Vista, Windows 7 and OSX hosts (my ripping guest is a bare bones WinXP install).

Do make it work, you have to tell the VM software to operate the virtual drive in pass thru mode, rather than emulating a drive. I did have some messing about at some points to get it working.

I don't know wether it is as straight forward with DVD ripping.

After reading about what you guys go through to back up, I realize that my backup needs are not the same as yours.

To begin with, while much of my data is precious to me, very little of it is "mission critical." If it all went away tomorrow, I would be unhappy, but my day to day life would not be hugely disrupted.

So, my backup system is maybe once a week or so back up to external hard drive(s), and maybe four times a year back up to a pair of 2-TB drives that I keep at my neighbor's house. Good enough for me, but YMMV.

tanstaafl.

The vast bulk of my data is photos and FLAC rips of my CDs. The photos are irreplaceable of course (though obviously I wouldn't die if I lost them, I'd just be very sad for a bit). The FLACs represent a massive amount of invested time, it took me months to rip them all.

If I was just backing up my work data it would all be very easy and just about any online service would be fine for it, as it represents a fairly small amount of highly compressible data.

I don't use automated online backup because I can't bare to lose a day of data. I use it because if I took your approach then the backups simply wouldn't happen. I'm just not the sort of person who can reliably get into positive habits like that. I'd keep it up for a couple of months, after that I'd just stop doing the backups.

With an automated setup, it just happens without me having to do anything.

Like Andy said, my life won't be over if I lose this stuff, but I'd really rather not. The main reason I prefer services like these is simple: if my backup strategy weren't easy, I know I wouldn't be doing it at all. Your method works for you, Doug, but it's more bother than I'm willing to deal with, and isn't as convenient for me.

My risk is less than you suggest. My backups cover at least 10 years of data, perhaps more, but with my local external drives and my off-site external drives, I am only in danger of losing the most recent data, data added since the last backup, usually only about a week previous. Worst case scenario (house burns down taking my computer and backups with it) then I am at risk for the past two or three months of data, not pleasant but as you said, "my life won't be over..."

In any case, as I stood in the smoking ruins of what had been my house, my first thoughts probably wouldn't be about those books for my Kindle I downloaded last week.

tanstaafl.

I think if you don't want to spend money in software made by third parties, with such an old version of Windows your only option is Clonezilla. Ther could be other free apps around that I'm not aware of.
It worked fine with me when we tested it to replace the old Ghost we use at work, fearing it would not work well with Windows 7; as a side note, Ghost (for DOS) did work well, so we still use it (to clone hundreds of machines, including old laptops with win XP, not so old with Windows Vista, current ones with Windows 7).


Of course, there are several other applications for a price.

Probably not in XP (but maybe??) the standard Windows Backup and Restore Center, right there in the Control Panel, apparently will create a "...bootable 100% backup of a live Windows Machine". I played around with it last August, but went back to my regular, much simpler backup system (Karen's Replicator), although the capabilities of this seem impressive.

tanstaafl.

I don't know about everyone else, but I'm only interested in automated/scheduled solutions that can also handle deltas to reduce backup times. Ideally it would also handle versioning/snapshots and do so intelligently. But there really isn't anything around that seems to do this - no alternative to Time Machine in Mac OS.

There are a number of third-party commercial tools that can clone a Windows boot disk. That's useful to set up a base system to restore.

My wife was let go from her job two weeks ago. I told her right away. Look at the bright side, you won't have to use a POS Windows notebook anymore.

Yes, you're right, you can create a bootable backup with Vista. That's stuff you can do with 5-year old machines. But Mark is using XP, and I don't remember that to be available back then, even though I may not remember correctly.

I may be missing some feature, as I've played with Time Machine just a few times, but Rescue&Recovery (that comes with all Lenovo Laptops we distribute to users of all types at work, so I've ended up seeing that in action more, in the last years) works exactly like Time Machine. Snapshots in time, automatic if you wish, semi-automatic, manual, and they do delta. One single installation of R&R can be run from within the OS or before the OS, in its own pre-OS environment. Supports multiuser environments, as well as access levels to the backup sessions (based on NTFS, which is good). Handles disk space dynamically, or can be restrained to use a specific amt of it. And has a quite relevant set of advanced features. Not as user-friendly as time machine, as expected.
We have that running on thousands of laptops in the organization and it mostly works fine. Actually, I should say that in our experience it always did, as the only isues we had in years (we'e been using R&R since 2006) were caused by Lenovo (installing a corrupted version of our corporate disk image on 3 thousand laptops, making the base backup in R&R unreadable).

Restore points also finally work in a way I find useful (i Windows 7). But that's a different approach to "backup", if you wish to call it like that. I wouldn't. But, I was reading that R&R can now integrate with restore points to some degree, but I have not explored that much. I also seem to remember othe rsoftware that does the same, taking advantage of restore points, but I can't recall its name.

Still, in my own systems, I find both types of approaches useless. I never used restore points, and I deal with system disk (or any other disk that matters) crashes with mirror or other RAID. HDD are still cheap, I think. As I never had to reinstall Windows on my machines for any other reason, no matter how badly i treated them (and I did treat them badly at times), often migrating the same install from one machine to another when I feel too lazy, I would find R&R or similar (such as TM) very useless for me personally.

Of course, I happen to have laptops with data I backup daily, and system configuration not deserving a disk image (pretty standard). Should my work laptop crash, a reinstall would indeed take more time than drive re-image (not that much, today, when installing from pendrive), but it is such an unlikely event - and reinstall is such an easy process - that I prefer just not to bother with bootable drive image. Last 10 years proved I was right in taking that risk, even if my SSD should fail tonight I've saved a lot of time so far.

Just to share my experience.

I've just come across an interesting alternative for music backups. Free backups, only pay if you need to restore.

From the same guy who develops dBPowerAmp

http://forum.dbpoweramp.com/showthread.php?24268-Welcome-to-AudioSafe

Pricing for restores is not bad IMO, give that a typical MP3 album is ~100megs:

http://forum.dbpoweramp.com/showthread.php?24494-How-much-will-the-service-cost

Interesting they seem to be willing to also store some non-audio info

Though maybe that is to include space for cover art and misc external metadata files, rather than other files

In this case, it's not "data" that concerns me. I don't keep anything important at all on the Windows box. I am concerned about the software installation -- WinXP and all of the drivers and apps that took forever to get into the machine.

I can back all of that up easily, by booting Linux from USB and using rsync or mirrordir to replicate the filesystem onto another (ntfs) drive.

What I don't know how to do, is how to re-install the WinXP bootloader on that drive. I suspect this can be done by booting from a WinXP installation disc (CD), and then issuing some magic incantation to rewrite the boot sector etc..

Making a backup is trivial just using standard (free) Linux command-line tools. Maintaining the backup is similarly easy. But getting WinXP to boot from the backup image is my only dark area.

I suppose I could just bit-clone the drive once, to an identical drive, and from then on use rsync/mirrordir to refresh the backup periodically. That might work, but I really know almost nothing about booting requirements for MS-Win.

Thanks guys.

Maybe this article could be of some use to you then?

I'm afraid I don't know the gory details about what Windows needs to boot either. There are a number of guides including this one:

http://edoceo.com/exemplar/ntfsclone-transfer-windows

But like most guides of the sort it is incomplete. It shows the MBR and bootloaders being backed up, but never shows them being restored

Presumably it is a straight forward reversal with dd, but I've never actually tried it. Though I do actually have the need to do it at soon, maybe I'll give it a go

This guide confirms that it can be a relatively simple bit of work with ntfsclone and dd:

http://bahut.alma.ch/2005/04/cloning-xp-with-linux-and-ntfsclone_23.html

That method works only for bit-identical copies of the original media (sector-by-sector copies). Not useful here, but thanks.

Perfect! That's exactly what I was looking for.
I already knew about the old fdisk/mbr portion,
but not how to install the second-stage loader.
That link gives the magic incantations.

Boot from installation CD, open "Recovery Console", and do:
bootcfg /rebuild
fixmbr
fixboot

I'll try it out here later and report back.

Cheers

But again, that only works for bit-identical copies.

Cheers

In the comments there is a handy tip:

"Third, restoring a bootable windows to a different partition is slightly painful. Having a bootable XP/W2K cd will help a lot - fixboot and bootcfg but you can help yourself by pulling extra lines for partitions 1,2,3,4 in the boot.ini before starting!"

By which they mean, a normal boot.ini will look something like this:

[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /fastdetect

Before cloning, change it to:

[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /fastdetect
multi(0)disk(0)rdisk(0)partition(2)\WINDOWS="Microsoft Windows XP Professional 2" /fastdetect
multi(0)disk(0)rdisk(0)partition(3)\WINDOWS="Microsoft Windows XP Professional 3" /fastdetect
multi(0)disk(0)rdisk(0)partition(4)\WINDOWS="Microsoft Windows XP Professional 4" /fastdetect

Which means that if you restore to a different partition you can pick the correct one to boot from from the startup menu after you've cloned.

http://support.microsoft.com/kb/289022

Which bit in your case is not going to be bit-identical ? The partition table ?

The two tools for fixing it when it is screwed up that they referred to are fixboot and bootcfg. They can both be run from the WinXP recovery console (i.e. boot from the WinXP CD).

Fixboot can write a new MBR:

http://www.microsoft.com/resources/docum...t.mspx?mfr=true

Bootcfg (using "bootcfg /rebuild") can scan the disks for Windows partition and rebuild the boot.ini file:

http://support.microsoft.com/kb/291980

The drive/partition size, and/or starting sector on the drive, and/or logical geometry of the drive.

Cheers

Yup. Got all that from the other article link above by Archeon.
Thanks again, guys.

I managed to miss Archeon's earlier

I am late to this, and you got your answer. I can confirm that it works and it used to be the standrad way to install the bootloader. I suppose you may want to consider a way to do this w/o booting from CD if you want to keep using XP on newer machines with no optical drive. If so, you may consider BartPE by nu2(free version, and way more complete, of WinPE) to keep in one USB bootable medium the tools needed to make your HDD bootable and, as I used to back then, drive images and re-image tools.
http://www.nu2.nu/pebuilder/
http://www.nu2.nu/

Once you have a botable .iso of BartPE, I suppose you can easily use current Linux tools to port it to bootable USB. Should you want to do this from your Windows XP box for some reason, here's an how it used to be done:

http://users.telenet.be/Robvdb/BootPEUSB_UK.htm

I finally got around to following up on this.

Yesterday, I booted my (only) WinXP box from a Linux USB stick, and created an exact bit clone of the 1TB Win-C drive to a second mechanical drive. This was after first running defrag from within WinXP.

Next, I ran gparted on the clone, and told it to shrink the Win-C drive from 1TB down to about 100GB.

Then I installed a brand new 128GB A-Data XPG SX900 SSD, and bit-cloned the newly shrunk Win-C drive to the SSD.

Finally, re-ran gparted to expand the Win-C partition on the SSD to the full space available.

Removed the USB stick and the mechanical drives, and the system booted up into WinXP from the SSD without a hiccup, apart from the mandatory filesystem check that gparted tagged it for.

So now my one and only MS-Win box runs from an SSD, and the partition is small enough (120GB) that I can save exact bit-perfect replicas of it for disaster recovery.

I'm happy!

Interesting, I'm going to try and do that as well very soon, so your how-to comes quite handy.

What I'm wondering: does this method automatically align the partitions?

The quick way I did it maintains the original alignment, which in this case had the first (only) partition beginning at sector 64 (the 65th sector, 4kB aligned).

If the original partition is misaligned (non 4kB multiple), then I believe one would have to (a) properly align the copy, and (2) look back in this thread for help on how to "fix" the MS bootloader afterwards.

Cheers
-ml

So if the original partition is correctly aligned, which it should be since Windows Vista, then the target partition will also be correctly aligned. Ok, gotcha! This is comforting to know, thanks!