Distributed Backup Systems

[zourtney]

Occasionally, I get the feeling I should implement some sort of backup system between all my computers. Other than that 5 year old external hard drive sitting on my desk.

Of course, there are those paid, online ("cloud") storage solutions. And they're plentiful. But I was wondering if anyone was interested in setting up a sort of private peer-to-peer type backup system?

Ideal situation:

• 'puter on
• I select the directories on my system to back up
• One of more other people select my computer as the computer to back up data from
• Those computer(s) save a copy of my files
• Yay.

A quick check shows that there are several programs out there that do this.

• Crashplan
• Buddy Backup
• DataHaven.NET

Anyone interested? I could always pay some company $50+/yr but... 

[Nick]

How odd, I was just looking at a $15 NAS box that was linked on reddit, the pogoplug. (You can put debian and arch linux on it)

I looked at some backup stuff and cloud storage not too long ago. Anything that offered enough storage (500gb +) got expensive. I was thinking that could be a ok way for us to have communal picture storage without eating all of my bandwidth (though, if you mailed me a drive/stopped by then a USB port will work) But yeah.

For at-home backing up a little NAS box might work ok.

[zourtney]

I've been wanting a good at-home NAS box for a while now. I'll have to look into that one (Drobo and Synology are two others I've heard of, but aren't cheap). I think the combination of a NAS with off-site backup hardware would be awesome. We could both get one and let them talk to each other!

[zourtney]

I looked at someone's post about CrashPlan and Linux on the pogoplug. It's like 7 pages full of terminal commands  My favorite was the "cross fingers" part.

As such, I kinda like those Synology offerings. I want to want a DIY solution. I could build myself a little mini pc, toss some drives and FreeNAS on it...a few dozen late nights buried in a Linux command line prompt and voila! But it's ultimately not a lot cheaper. And it's certainly not easier. Maybe I'm lazy. Or inept. Or have other projects. Something like a Synology NAS [1 or 2] with pair of WD drives [1 or 2], and CrashPlan (free) installed. It could work.

I don't need any of that. We could just install CrashPlan (or another) and try this out. Until my drive's a'brimming. I'm going to go install it on two computers and see how it goes.

Edit: ok, via LAN at least, that was incredibly easy. So easy a caveman could do it. Anyone want me to back their stuff up? I'll donate some gigs (~10-20 is about all I'd want to spare on the desktop, for now)

Edit 2: of course, everything is encrypted, which is great. Except when it isn't. This is essentially useless for sharing pictures, etc... a simple rysnc wrapper might be enough for that...)

[zourtney]

How much disk space do we have available on the Randomland server? I was thinking of experimenting with rdiff-backup. In theory, it's basically the rysnc gimp backups I set up for work once upon a time...only built as a (hopefully usable) tool. The aforementioned duplicity will do the same, but in an encrypted tar file.

...Unless you all think it's silly (or invasive) to back up our pictures, media, etc to each other machines (encrypted or not). I do want a semi-regular, off-site backup of at least Randomland's databases.

Thoughts?

[Nick]

We have loads of disk space.
Filesystem      Size     Used   Avail    Use%
/dev/sda6       456G   67G    366G    16% /

There are lots of pic in /var/www already ( 17G     ./var )

Upload away! Just remember I have a bandwidth cap. Using Sneaker net (Mailing a flash drive) might be nice, or just do it in ~10gb bunches per month.  I am happy to give toward a communal thumbdrive Either a 64gb or a few 32gb ones, or a fist full of 16s would work.
(So cheap these days)

(Or .... If work does not mined, I can set up a linux box (I got one of those pogo plugs to play with) and you can transfer all you want (in the evening) to that via SSH on a forwarded port)

[zourtney]

So, I got myself a little NAS box. I stuck two 3tb hard drives in it, which are currently running in RAID 1. I've already loaded it up with all my pictures, music, and video projects -- all the stuff I care about.

I've still got plenty to figure out:

• A PC-to-NAS backup system. I'll probably start with robocopy / rsync scripts, or something simple
• A NAS-to-Randomland backup. For appropriate content, anyway. Pictures?
• A Randomland-Drupal-to-NAS backup. At least get a periodic dump of the database, or something.
• A Randomland-Anything-Else-to-NAS backup. For whatever you all want backed up remotely. If anything.
• A sweet codename. Like Randomland Yallpine Node or something.

As for the box itself, I like it. It's not (even close to) cheap, but I couldn't build something with a similar form factor for the price. It's nice and small; and it's quiet. More importantly to me at the moment is that it was really, really easy to set up. I had the thing full of drives, shared folders set up, and files being copied to it within a few hours of casual attention. It has impressive browser-based admin GUIs; a limited, but good package manager for the boilerplate stuff; and iOS, Android, and WindowsPhone(!) apps. I got music streaming to my phone and the PS3 in about 4 mouse clicks 

It runs some variant of Linux, so I can do pretty much anything I want with it. (The only kicker is that it's an ARM processor, so, I might have to learn to love to compile from source). One fun project could be using NodeJS or Python to make RESTful image API.

Anyway: cool toytool.

[Nick]

There are loads of packages for ARM, depending on what distro it's based off of. But compiling from source (usually) is not too hard.

How would the RL servers access your box for backups? At the very least it would be nice to stick a periodic ZIP of the database on there. SSH file transfer(SFTP)? Some other protocol? Virtual networking and a samba-type share?


The project sounds fun But what is a Yallpine?

[zourtney]

Well, I still need to poke some holes in my router firewall. Then I should be able to use some dynamic IP updater service to map it to some Randomland subdomain (or another domain entirely?). I'm not sure what protocol to use to transfer backups...guess I'd have to play with it. I'm sure it's doable (and probably relatively easy).Are you still backing up the database on a cron job?

[Nick]

Are you still backing up the database on a cron job?

Probably? If not it should be.

[zourtney]

Yos to all.

I set up my fancybox to be accessible from the outside world. I used the box's built-in utilities to tie it to a dynamic DNS hostname. Now I can access the admin GUI (or SSH in) from anywhere. Even better, I can view every picture I've ever taken through its mostly-tolerable web interface.

So now, I've got a couple muddy goals.

• Do some CNAME magic to expose my box as backup.randomland.net^*
• Follow this to pull backups of Randomland's databases, themes, and relevant files. Put it on a cron job, keep a few weeks' worth. And hopefully never need it.
• Make a similar backup system to sync pictures with Randomland. I'll push up my pictures; I'll pull down everyone else's. I just need to be cautious of bandwidth usage.

And maybe someday:

• Make a simple API to access these remote images. Or at least a way to make images displayable on Randomland without creating a separate copy on the server (since they'll already be local)
• Make Randomland awesome. Whatever it takes.

So yep. Once that's all set up, you all can upload pictures to Randomland, and have at least one other offsite copy. Yay 

* I might even break it up into two or more subdomains (one for each distinct purpose). I'm always a fan of naturey themes, so suggestions are welcome...or "yallpine".