Replace boost::scoped_ptr with std::unique_ptr.

[cumulus.git] / doc / format.txt

diff --git a/doc/format.txt b/doc/format.txt
index 2ab2696..2c40075 100644 (file)
--- a/doc/format.txt
+++ b/doc/format.txt
@@ -1,16 +1,13 @@
                        Backup Format Description
          for Cumulus: Efficient Filesystem Backup to the Cloud
                        Backup Format Description
          for Cumulus: Efficient Filesystem Backup to the Cloud

-                      Version: "LBS Snapshot v0.8"
+                   Version: "Cumulus Snapshot v0.11"

 
 NOTE: This format specification is intended to be mostly stable, but is
 still subject to change before the 1.0 release.  The code may provide
 additional useful documentation on the format.
 
 
 NOTE: This format specification is intended to be mostly stable, but is
 still subject to change before the 1.0 release.  The code may provide
 additional useful documentation on the format.
 

-NOTE2: The name of this project has changed from LBS to Cumulus.
-However, to avoid introducing gratuitous changes into the format, in
-most cases any references to "LBS" in the format description have been
-left as-is.  The name may be changed in the future if the format is
-updated.
+NOTE2: The name of this project has changed from LBS to Cumulus.  In
+some areas the name "LBS" is still used.

 
 This document simply describes the snapshot format.  It is described
 from the point of view of a decompressor which wishes to restore the
 
 This document simply describes the snapshot format.  It is described
 from the point of view of a decompressor which wishes to restore the


@@ -22,6 +19,30 @@ This document does not explain the rationale behind the format; for
 that, see design.txt.
 
 
 that, see design.txt.
 
 

+BACKUP REPOSITORY LAYOUT
+========================
+
+Cumulus backups are stored using a relatively simple layout.  Data files
+described below are written into one of several directories on the
+backup server, depending on their purpose:
+    snapshots/
+        Snapshot descriptor files, which quickly summarize each backup
+        snapshot stored.
+    segments0/
+    segments1/
+        Storage of the bulk of the backup data, in compressed/encrypted
+        form.  Technically any segment could be stored in either
+        directory (both directories will be searched when looking for a
+        segment).  However, data in segments0 might be faster to access
+        (but more expensive) depending on the storage backend.  The
+        intent is that segments0 can store filesystem tree metadata and
+        segments1 can store file contents.
+    meta/
+        Snapshot-specific metadata that is not core to the backup.  This
+        can include checksums of segments, some data for rebuilding
+        local database contents, etc.
+
+

 DATA CHECKSUMS
 ==============
 
 DATA CHECKSUMS
 ==============
 


@@ -35,8 +56,10 @@ format.  The general format used is
     <algorithm>=<hexdigits>
 
 <algorithm> identifies the checksum algorithm used, and allows new
     <algorithm>=<hexdigits>
 
 <algorithm> identifies the checksum algorithm used, and allows new

-algorithms to be added later.  At the moment, the only permissible value
-is "sha1", indicating a SHA-1 checksum.
+algorithms to be added later.  Permissible values are:
+    "sha1": SHA-1
+    "sha224": SHA-224 (added in version 0.11)
+    "sha256": SHA-256 (added in version 0.11)

 
 <hexdigits> is a sequence of hexadecimal digits which encode the
 checksum value.  For sha1, <hexdigits> should be precisely 40 digits
 
 <hexdigits> is a sequence of hexadecimal digits which encode the
 checksum value.  For sha1, <hexdigits> should be precisely 40 digits


@@ -72,6 +95,8 @@ fixed points; an example UUID is
 This segment could be stored in the filesystem as a file
     a704eeae-97f2-4f30-91a4-d4473956366b.tar
 The UUID used to name a segment is assigned when the segment is created.
 This segment could be stored in the filesystem as a file
     a704eeae-97f2-4f30-91a4-d4473956366b.tar
 The UUID used to name a segment is assigned when the segment is created.

+These files are stored in either the segments0 or segments1 directories
+on the backup server.

 
 Filters can be layered on top of the segment storage to provide
 compression, encryption, or other features.  For example, the example
 
 Filters can be layered on top of the segment storage to provide
 compression, encryption, or other features.  For example, the example


@@ -102,8 +127,8 @@ object.
 
 NOTE: When naming an object, the segment portion consists of the UUID
 only.  Any extensions appended to the segment when storing it as a file
 
 NOTE: When naming an object, the segment portion consists of the UUID
 only.  Any extensions appended to the segment when storing it as a file

-in the filesystem (for example, .tar.bz2) are _not_ part of the name of
-the object.
+in the filesystem (for example, .tar.bz2) and path information (for
+example, segments0) are _not_ part of the name of the object.

 
 There are two additional components which may appear in an object name;
 both are optional.
 
 There are two additional components which may appear in an object name;
 both are optional.


@@ -124,18 +149,20 @@ is invalid to select using the slice syntax a range of bytes that does
 not fall within the original object.  The slice specification should be
 appended to an object name, for example:
     a704eeae-97f2-4f30-91a4-d4473956366b/000001ad[264+1000]
 not fall within the original object.  The slice specification should be
 appended to an object name, for example:
     a704eeae-97f2-4f30-91a4-d4473956366b/000001ad[264+1000]

-selects only bytes 264..1263 from the original object.  As an
-abbreviation, the slice syntax
+selects only bytes 264..1263 from the original object.
+
+The slice syntax

     [<length>]
     [<length>]

-is shorthand for
-    [0+<length>]
-In place of a traditional slice, the annotation
-    [=<length>]
-may be used.  This is somewhat similar to specifying [<length>], but
+indicates that all bytes of the object are to be used, but

 additionally asserts that the referenced object is exactly <length>
 additionally asserts that the referenced object is exactly <length>

-bytes long--that is, this slice syntax does not change the bytes
-returned at all, but can be used to provide information about the
-underlying object store.
+bytes long.  Older versions of Cumulus can also use the syntax
+    [=<length>]
+as a synonym for length assertions, but this notation is deprecated.
+
+(In older versions of the format, the syntax [<length>] was a shorthand
+for [0+<length>]: that is, select the first <length> bytes of the object
+but make no assertions about the overall size.  The backup tool has not
+generated such slices since v0.8.)

 
 Both a checksum and a slice can be used.  In this case, the checksum is
 given first, followed by the slice.  The checksum is computed over the
 
 Both a checksum and a slice can be used.  In this case, the checksum is
 given first, followed by the slice.  The checksum is computed over the


@@ -151,7 +178,7 @@ object must have a slice specification appended to indicate the size of
 the object.  For example
     zero[1024]
 represents a block consisting of 1024 null bytes.  A checksum should not
 the object.  For example
     zero[1024]
 represents a block consisting of 1024 null bytes.  A checksum should not

-be given.  The slice syntax should use the abbreviated length-only form.
+be given.

 
 
 FILE METADATA LISTING
 
 
 FILE METADATA LISTING


@@ -275,20 +302,29 @@ The name of snapshot descriptor file is
 logically distinct sets of snapshots (such as snapshots for two
 different directory trees) that are being stored in the same location.
 <timestamp> gives the date and time the snapshot was taken; the format
 logically distinct sets of snapshots (such as snapshots for two
 different directory trees) that are being stored in the same location.
 <timestamp> gives the date and time the snapshot was taken; the format

-is %Y%m%dT%H%M%S (20070806T092239 means 2007-08-06 09:22:39).
+is %Y%m%dT%H%M%S (20070806T092239 means 2007-08-06 09:22:39).  It is
+recommended that the timestamp be given in UTC for consistent sorting
+even if the offset from UTC to local time changes, however the
+authoritative timestamp (including timezone) can be found in the Date
+field.  (In version v0.10 and earlier the timestamp is given in local
+time; in current versions UTC is used.)

 
 The contents of the descriptor are a set of RFC 822-style headers (much
 like the metadata listing).  The fields which are defined are:
 
 The contents of the descriptor are a set of RFC 822-style headers (much
 like the metadata listing).  The fields which are defined are:

-    Format: The string "LBS Snapshot v0.6" which identifies this file as
-        a Cumulus backup descriptor.  The version number (v0.6) might
-        change if there are changes to the format.  It is expected that
-        at some point, once the format is stabilized, the version
-        identifier will be changed to v1.0.
+    Format: The string "Cumulus Snapshot v0.11" which identifies this
+        file as a Cumulus backup descriptor.  The version number (v0.11)
+        might change if there are changes to the format.  It is expected
+        that at some point, once the format is stabilized, the version
+        identifier will be changed to v1.0.  (Earlier versions, format
+        v0.8 and earlier, used the string "LBS Snapshot" instead of
+        "Cumulus Snapshot", reflecting an earlier name for the project.
+        Consumers should be prepared for either name.)

     Producer: A informative string which identifies the program that
         produced the backup.
     Producer: A informative string which identifies the program that
         produced the backup.

-    Date: The date the snapshot was produced.  This matches the
-        timestamp encoded in the filename, but is written out in full.
-        A timezone is given.  For example: "2007-08-06 09:22:39 -0700".
+    Date: The date the snapshot was produced, in the local time zone.
+        This matches the timestamp encoded in the filename, but is
+        written out in full.  A timezone (offset from UTC) is given.
+        For example: "2007-08-06 02:22:39 -0700".

     Scheme: The <scheme> field from the descriptor filename.
     Segments: A whitespace-seprated list of segment names.  Any segment
         which is referenced by this snapshot must be included in the
     Scheme: The <scheme> field from the descriptor filename.
     Segments: A whitespace-seprated list of segment names.  Any segment
         which is referenced by this snapshot must be included in the