1 Cumulus: Efficient Filesystem Backup to the Cloud
7 - libuuid (sometimes part of e2fsprogs)
10 Building should be a simple matter of running "make". This will produce
11 an executable called "cumulus".
17 Two directories are needed for backups: one for storing the backup
18 snapshots themselves, and one for storing bookkeeping information to go
19 with the backups. In this example, the first will be "/cumulus", and
20 the second "/cumulus.db", but any directories will do. Only the first
21 directory, /cumulus, needs to be stored somewhere safe. The second is
22 only used when creating new snapshots, and is not needed when restoring.
24 1. Create the snapshot directory and the local database directory:
25 $ mkdir /cumulus /cumulus.db
27 2. Initialize the local database using the provided script schema.sql
29 $ sqlite3 /cumulus.db/localdb.sqlite
30 sqlite> .read schema.sql
33 3. If encrypting or signing backups with gpg, generate appropriate
34 keypairs. The keys can be kept in a user keyring or in a separate
35 keyring just for backups; this example does the latter.
36 $ mkdir /cumulus.db/gpg; chmod 700 /cumulus.db/gpg
37 $ gpg --homedir /cumulus.db/gpg --gen-key
38 (generate a keypair for encryption; enter a passphrase for
40 $ gpg --homedir /cumulus.db/gpg --gen-key
41 (generate a second keypair for signing; for automatic
42 signing do not use a passphrase to protect the secret key)
43 Be sure to store the secret key needed for decryption somewhere
44 safe, perhaps with the backup itself (the key protected with an
45 appropriate passphrase). The secret signing key need not be stored
46 with the backups (since in the event of data loss, it probably
47 isn't necessary to create future backups that are signed with the
50 To achieve better compression, the encryption key can be edited to
51 alter the preferred compression algorithms to list bzip2 before
53 $ gpg --homedir /cumulus.db/gpg --edit-key <encryption key>
55 (prints a terse listing of preferences associated with the
58 (allows preferences to be changed; copy the same preferences
59 list printed out by the previous command, but change the
60 order of the compression algorithms, which start with "Z",
61 to be "Z3 Z2 Z1" which stands for "BZIP2, ZLIB, ZIP")
64 Copy the provided encryption filter program, cumulus-filter-gpg,
65 somewhere it may be run from.
67 4. Create a script for launching the Cumulus backup process. A simple
71 export LBS_GPG_HOME=/cumulus.db/gpg
72 export LBS_GPG_ENC_KEY=<encryption key>
73 export LBS_GPG_SIGN_KEY=<signing key>
74 cumulus --dest=/cumulus --localdb=/cumulus.db --scheme=test \
75 --filter="cumulus-filter-gpg --encrypt" --filter-extension=.gpg \
76 --signature-filter="cumulus-filter-gpg --clearsign" \
77 /etc /home /other/paths/to/store
79 Make appropriate substitutions for the key IDs and any relevant
80 paths. Here "--scheme=test" gives a descriptive name ("test") to
81 this collection of snapshots. It is possible to store multiple sets
82 of backups in the same directory, using different scheme names to
83 distinguish them. The --scheme option can also be left out
90 Segment cleaning must periodically be done to identify backup segments
91 that are mostly unused, but are storing a small amount of useful data.
92 Data in these segments will be rewritten into new segments in future
93 backups to eliminate the dependence on the almost-empty old segments.
95 The provided cumulus-util tool can perform the necessary cleaning. Run
97 $ cumulus-util --localdb=/cumulus.db clean
98 Cleaning is still under development, and so may be improved in the
99 future, but this version is intended to be functional.
101 Old backup snapshots can be pruned from the snapshot directory
102 (/cumulus) to recover space. A snapshot which is still referenced by
103 the local database should not be deleted, however. Deleting an old
104 backup snapshot is a simple matter of deleting the appropriate snapshot
105 descriptor file (snapshot-*.lbs) and any associated checksums
106 (snapshot-*.sha1sums). Segments used by that snapshot, but not any
107 other snapshots, can be identified by running the clean-segments.pl
108 script from the /cumulus directory--this will perform a scan of the
109 current directory to identify unreferenced segments, and will print a
110 list to stdout. Assuming the list looks reasonable, the segments can be
112 $ rm `./clean-segments.pl`
113 A tool to make this easier will be implemented later.
115 The clean-segments.pl script will also print out a warning message if
116 any snapshots appear to depend upon segments which are not present; this
117 is a serious error which indicates that some of the data needed to
118 recover a snapshot appears to be lost.
121 Listing and Restoring Snapshots
122 -------------------------------
124 A listing of all currently-stored snapshots (and their sizes) can be
126 $ cumulus-util --store=/cumulus list-snapshot-sizes
128 If data from a snapshot needs to be restored, this can be done with
129 $ cumulus-util --store=/cumulus restore-snapshot \
130 test-20080101T121500 /dest/dir <files...>
131 Here, "test-20080101T121500" is the name of the snapshot (consisting of
132 the scheme name and a timestamp; this can be found from the output of
133 list-snapshot-sizes) and "/dest/dir" is the path under which files
134 should be restored (this directory should initially be empty).
135 "<files...>" is a list of files or directories to restore. If none are
136 specified, the entire snapshot is restored.
139 Alternate Restore Tool
140 ----------------------
142 The contrib/restore.pl script is a simple program for restoring the
143 contents of a Cumulus snapshot. It is not as full-featured as the
144 restore functionality in cumulus-util, but it is far more compact. It
145 could be stored with the backup files so a tool for restores is
146 available even if all other data is lost.
148 The restore.pl script does not know how to decompress segments, so this
149 step must be performed manually. Create a temporary directory for
150 holding all decompressed objects. Copy the snapshot descriptor file
151 (*.lbs) for the snapshot to be restored to this temporary directory.
152 The snapshot descriptor includes a list of all segments which are needed
153 for the snapshot. For each of these snapshots, decompress the segment
154 file (with gpg or the appropriate program based on whatever filter was
155 used), then pipe the resulting data through "tar -xf -" to extract. Do
156 this from the temporary directory; the temporary directory should be
157 filled with one directory for each segment decompressed.
159 Run restore.pl giving two arguments: the snapshot descriptor file
160 (*.lbs) in the temporary directory, and a directory where the restored
161 files should be written.