1 Backup Format Description
2 for Cumulus: Efficient Filesystem Backup to the Cloud
3 Version: "LBS Snapshot v0.6"
5 NOTE: This format specification is intended to be mostly stable, but is
6 still subject to change before the 1.0 release. The code may provide
7 additional useful documentation on the format.
9 NOTE2: The name of this project has changed from LBS to Cumulus.
10 However, to avoid introducing gratuitous changes into the format, in
11 most cases any references to "LBS" in the format description have been
12 left as-is. The name may be changed in the future if the format is
15 This document simply describes the snapshot format. It is described
16 from the point of view of a decompressor which wishes to restore the
17 files from a snapshot. It does not specify the exact behavior required
18 of the backup program writing the snapshot. For details of the current
19 backup program, see implementation.txt.
21 This document does not explain the rationale behind the format; for
28 In several places in the Cumulus format, a cryptographic checksum may be
29 used to allow data integrity to be verified. At the moment, only the
30 SHA-1 checksum is supported, but it is expected that other algorithms
31 will be supported in the future.
33 When a checksum is called for, the checksum is always stored in a text
34 format. The general format used is
35 <algorithm>=<hexdigits>
37 <algorithm> identifies the checksum algorithm used, and allows new
38 algorithms to be added later. At the moment, the only permissible value
39 is "sha1", indicating a SHA-1 checksum.
41 <hexdigits> is a sequence of hexadecimal digits which encode the
42 checksum value. For sha1, <hexdigits> should be precisely 40 digits
45 A sample checksum string is
46 sha1=67049e7931ad7db37b5c794d6ad146c82e5f3187
49 SEGMENTS & OBJECTS: STORAGE AND NAMING
50 ======================================
52 A Cumulus snapshot consists, at its base, of a collection of /objects/:
53 binary blobs of data, much like a file. Higher layers interpret the
54 contents of objects in various ways, but the lowest layer is simply
55 concerned with storing and naming these objects.
57 An object is a sequence of bytes (octets) of arbitrary length. An
58 object may contain as few as zero bytes (though such objects are not
59 very useful). Object sizes are potentially unbounded, but it is
60 recommended that the maximum size of objects produced be on the order of
61 megabytes. Files of essentially unlimited size can be stored in a
62 Cumulus snapshot using objects of modest size, so this should not cause
63 any real restrictions.
65 For storage purposes, objects are grouped together into /segments/.
66 Segments use the TAR format; each object within a segment is stored as a
67 separate file. Segments are named using UUIDs (Universally Unique
68 Identifiers), which are 128-bit numbers. The textual form of a UUID is
69 a sequence of lowercase hexadecimal digits with hyphens inserted at
70 fixed points; an example UUID is
71 a704eeae-97f2-4f30-91a4-d4473956366b
72 This segment could be stored in the filesystem as a file
73 a704eeae-97f2-4f30-91a4-d4473956366b.tar
74 The UUID used to name a segment is assigned when the segment is created.
76 Filters can be layered on top of the segment storage to provide
77 compression, encryption, or other features. For example, the example
78 segment above might be stored as
79 a704eeae-97f2-4f30-91a4-d4473956366b.tar.bz2
81 a704eeae-97f2-4f30-91a4-d4473956366b.tar.gpg
82 if the file data had been filtered through bzip2 or gpg, respectively,
83 before storage. Filtering of segment data is outside the scope of this
84 format specification, however; it is assumed that if filtering is used,
85 when decompressing the unfiltered data can be recovered (yielding data
88 Objects within a segment are numbered sequentially. This sequence
89 number is then formatted as an 8-digit (zero-padded) hexadecimal
90 (lowercase) value. The fully qualified name of an object consists of
91 the segment name, followed by a slash ("/"), followed by the object
92 sequence number. So, for example
93 a704eeae-97f2-4f30-91a4-d4473956366b/000001ad
96 Within the segment TAR file, the filename used for each object is its
97 fully-qualified name. Thus, when extracted using the standard tar
98 utility, a segment will produce a directory with the same name as the
99 segment itself, and that directory will contain a set of
100 sequentially-numbered files each storing the contents of a single
103 NOTE: When naming an object, the segment portion consists of the UUID
104 only. Any extensions appended to the segment when storing it as a file
105 in the filesystem (for example, .tar.bz2) are _not_ part of the name of
108 There are two additional components which may appear in an object name;
111 First, a checksum may be added to the object name to express an
112 integrity constraint: the referred-to data must match the checksum
113 given. A checksum is enclosed in parentheses and appended to the object
115 a704eeae-97f2-4f30-91a4-d4473956366b/000001ad(sha1=67049e7931ad7db37b5c794d6ad146c82e5f3187)
117 Secondly, an object may be /sliced/: a subset of the bytes actually
118 stored in the object may be selected to be returned. The slice syntax
121 where <start> is the first byte to return (as a decimal offset) and
122 <length> specifies the number of bytes to return (again in decimal). It
123 is invalid to select using the slice syntax a range of bytes that does
124 not fall within the original object. The slice specification should be
125 appended to an object name, for example:
126 a704eeae-97f2-4f30-91a4-d4473956366b/000001ad[264+1000]
127 selects only bytes 264..1263 from the original object. As an
128 abbreviation, the slice syntax
133 Both a checksum and a slice can be used. In this case, the checksum is
134 given first, followed by the slice. The checksum is computed over the
135 original object contents, before slicing.
140 In addition to the standard syntax for objects described above, the
141 special name "zero" may be used instead of segment/sequence number.
142 This represents an object consisting entirely of zeroes. The zero
143 object must have a slice specification appended to indicate the size of
144 the object. For example
146 represents a block consisting of 1024 null bytes. A checksum should not
147 be given. The slice syntax should use the abbreviated length-only form.
150 FILE METADATA LISTING
151 =====================
153 A snapshot stores two distinct types of data into the object store
154 described above: data and metadata. Data for a file may be stored as a
155 single object, or the data may be broken apart into blocks which are
156 stored as separate objects. The file /metadata/ log (which may be
157 spread across multiple objects) specifies the names of the files in a
158 snapshot, metadata about them such as ownership and timestamps, and
159 gives the list of objects that contain the data for the file.
161 The metadata log consists of a set of stanzas, each of which are
162 formatted somewhat like RFC 822 (email) headers. An example is:
165 checksum: sha1=11bd6ec140e4ec3110a91e1dd0f02b63b701421f
166 data: 2f46bce9-4554-4a60-a4a2-543637bd3989/000001f7
174 The meanings of all the fields are described later. A blank line
175 separates stanzas with information about different files. In addition
176 to regular stanzas, the metadata listing may contain a line containing
177 an object reference prefixed with "@". Such a line indicates that the
178 contents of the referenced object should be fetched and parsed as a
179 metadata listing at this point, prior to continuing to parse the current
182 Several common encodings are used for various fields. The encoding used
183 for each field is specified in the field listing that follows.
184 encoded string: An arbitrary string (octet sequence), with bytes
185 optionally escaped by replacing a byte with %xx, where "xx" is a
186 hexadecimal representation of the byte replaced. For example,
187 space can be replaced with "%20". This is the same escaping
188 mechanism as used in URLs.
189 integer: An integer, which may be written in decimal, octal, or
190 hexadecimal. Strings starting with 0 are interpreted as octal,
191 and those starting with 0x are intepreted as hexadecimal.
193 Common fields (required in all stanzas):
194 path [encoded string]: Full path of the file archived. Note: In
195 previous versions (<= 0.2) the name of this field was "name".
196 user [special]: The user ID of the file, as an integer, optionally
197 followed by a space and the corresponding username, as an
198 escaped string enclosed in parentheses.
199 group [special]: The group ID which owns the file. Encoding is the
200 same as for the user field: an integer, with an optional name in
201 parentheses following.
202 mode [integer]: Unix mode bits for the file.
203 type [special]: A single character which indicates the type of file.
204 The type indicators are meant to be consistent with the
205 characters used with the -type option to find(1), and the file
206 type checks in test(1):
214 Note that previous versions used '-' to indicate a regular file.
215 This character should not be generated in any new snapshots, but
216 may be encountered in old snapshots (those with a format version
218 mtime [integer]: Modification time of the file.
220 Optional common fields:
221 links [integer]: Number of hard links to this file, generally only
222 reported if greater than 1.
223 inode [string]: String specifying the inode number of this file when
224 it was dumped. If "links" is greater than 1, then searching for
225 other files that have an identical "inode" value can be used to
226 determine which files should be hard-linked together when
227 restoring. The inode field should be treated as an opaque
228 string and compared for equality as such; an implementation may
229 choose whatever representation is convenient. The format
230 produced by the standard tool is <major>/<minor>/<inode> (where
231 <major> and <minor> specify the device of the containing
232 filesystem and <inode> is the inode number of the file).
233 ctime [integer]: Change time for the inode.
235 Special fields used for regular files:
236 checksum [string]: Checksum of the file contents.
237 size [integer]: Size of the file, in bytes.
238 data [reference list]: Whitespace-separated list of object
239 references. The referenced data, when concatenated in the
240 listed order, will reconstruct the file data. Any reference
241 that begins with a "@" character is an indirect reference--the
242 given object includes a whitespace-separated list of object
243 references which should be parsed in the same manner as the data
246 Special fields used for symbolic links:
247 target[encoded string]: The target of the symlink, as returned by
248 readlink(2). Note: In old version of the format (<= 0.2), this
249 field was called "contents" instead of "target".
251 Special fields used for block and character device files:
252 device[special]: The major and minor number of the device. Encoded
253 as "major/minor", where major is the major device number encoded
254 into an integer, and minor is the minor device number.
260 The snapshot descriptor is a small file which describes a single
261 snapshot. It is one of the few files which is not stored as an object
262 in the segment store. It is stored as a separate file, in plain text,
263 but in the same directory as segments are stored.
265 The name of snapshot descriptor file is
266 snapshot-<scheme>-<timestamp>.lbs
267 <scheme> is a descriptive text which can be used to distinguish several
268 logically distinct sets of snapshots (such as snapshots for two
269 different directory trees) that are being stored in the same location.
270 <timestamp> gives the date and time the snapshot was taken; the format
271 is %Y%m%dT%H%M%S (20070806T092239 means 2007-08-06 09:22:39).
273 The contents of the descriptor are a set of RFC 822-style headers (much
274 like the metadata listing). The fields which are defined are:
275 Format: The string "LBS Snapshot v0.6" which identifies this file as
276 a Cumulus backup descriptor. The version number (v0.6) might
277 change if there are changes to the format. It is expected that
278 at some point, once the format is stabilized, the version
279 identifier will be changed to v1.0.
280 Producer: A informative string which identifies the program that
282 Date: The date the snapshot was produced. This matches the
283 timestamp encoded in the filename, but is written out in full.
284 A timezone is given. For example: "2007-08-06 09:22:39 -0700".
285 Scheme: The <scheme> field from the descriptor filename.
286 Segments: A whitespace-seprated list of segment names. Any segment
287 which is referenced by this snapshot must be included in the
288 list, since this list can be used in garbage-collecting old
289 segments, determining which segments need to be downloaded to
290 completely reconstruct a snapshot, etc.
291 Root: A single object reference which points to the metadata
292 listing for the snapshot.
293 Checksums: A checksum file may be produced (with the same name as
294 the snapshot descriptor file, but with extension .sha1sums
295 instead of .lbs) containing SHA-1 checksums of all segments.
296 This field contains a checksum of that file.
297 Intent: Informational; records the value of the --intent flag when
298 the snapshot was created, and can be used when determining which
299 snapshots to later delete.