[clue-tech] "rewindable" drive?

[Nate Duehr]

David L. Willson wrote:
> Thanks all!
> 
> Nate:    No, there's not a significant amount of politics involved, and for the little there is, we all like each other, and they trust me, so they generally do as I ask.  That said, there are some pretty stringent requirements, and the risks that come with a customized version of using a well-known PHP app.  The website must not go down.  If it goes down for any reason (hardware, software, human oops or human evil), it must come back up in 5 minutes, with the database at it's last consistent state.

Wow... a 5 minute SLA.  Those always lead to interesting solutions. 
Glad to hear it's not a political thing.  I "went there" just 'cause 
I've been there, done that.

It's harder in that environment, but definitely very interesting work...

> Angelo:  I'll be evaluating ZFS versus LVM's capabilities.  I can do quite a bit because it's an Ubuntu LAMP Server virtualized on a full Ubuntu Server, so the "getting underneath" it is there in spades.

That should be an interesting analysis.  I really like ZFS's ability to 
"just figure out the mess" when physical drives are put in the wrong 
place.  (On Sun servers, the real issue is often that the /dev tree gets 
rebuilt, and the disks are physically in the same locations, but their 
"location" in the /dev tree moved, which has been a Solaris/Sun pain in 
the butt for years.

They finally "dealt with it" by teaching ZFS to just handle it by 
meta-tagging the drives themselves.  Plug the drive into the box 
anywhere, if the OS can "see" it, it will know where it "belongs" in 
your multiple ZFS mirrors, RAID, whatever.  Kinda nice when you're in a 
hurry.

Sun has a video out on YouTube or somewhere from Sun Germany where the 
guys use USB sticks as disks to demo this.  Two four stick USB hubs 
stacked and zip-tied together makes up an "eight disk" array... they 
unmount and stop the ZFS filesystem, unplug all the sticks, scramble 
them on the table, plug them all in, and start the ZFS filesystem back 
up.  It recovers virtually instantly.  If I remember right, they also 
pull a "disk" from their RAID 5 "array" and then stream full rate video 
data from the remaining sticks WHILE the other one is re-syncing.

In general, it seems to work well, the little bit I've used it.  I'd 
like to see us use it more on our production systems here over Veritas, 
and that's saying a lot, because I've had a lot of experience dealing 
with and fixing VFS and VCS.  (Veritas Filesystem and Veritas Cluster 
Server).

> Dave:    Yes, I have a gold image.  Every night, I "pause" the VM and rsync it off to another box, which rsync's it off to an external drive, which is rotated monthly.  I'm looking for the fine-grained online backups, "an hour ago", "two hours ago", "a day ago", "two days ago", and "a week ago".  Currently, I can only offer "a day ago" and some further back at monthly levels of granularity.  My dream was to offer "anytime in the last 72 hours" or something like that.

Define "anytime".  Every hour?  Every minute?  :-)

I would think your "pause" trick could be used more than once a day, if 
users could handle it, sessions and things are recovering correctly over 
the pause, and the rsync isn't taking too long and timing things out. 
Of course on a day when the dataset changes DRASTICALLY, rsync would 
take too long and you'd have some downtime.  You'd just have to grab a 
stopwatch and see... how long a full sync takes, for example... that's a 
useable baseline...

I think not only rsync, but something layered over the top of it to get 
time-stamped rsync support (like rdiff-backup) might be good here too, 
if you're already down the rsync path.

The biggest pain I assume is that it's got an RDBMS under it.  (MySQL, 
Postgres, whatever?)  If you can "quiet" that DB, often-times a 
"snapshot" at that point is "adequate" for a quick and dirty recovery to 
that point in time.  Temp files and other "stuff" in the rest of the 
system can be problematic, depending on what application it is.  You 
didn't say.

> Mike:    Let me know if you come up with something.  I'm just sure this can be done with our current technology; I guess nobody's bothered to write it is all.  I believe that Microsoft's Volume Shadow Copy does very nearly what I want, but of course, only for Windows.  I'll say again what occurs to me pretty often.  Microsoft should be on our side.

Volume Shadow Copy is "interesting"... I'm kinda glad I don't really 
have to dig in and find out just HOW "interesting" though.  :-)

> In conclusion, I'm pretty sure that LVM, ZFS, and/or VMware Server 2.0 named snapshots will get my 90% of the way there, at a fraction of the cost in disk space.

Defintely.

> 
> Any DRBD experts out there?  Want to help me set some stuph up?

Ooh... is this also running on a cluster?  Wicked...

Nate