2 - Rework storage layer, to make it easier to support local files,
3 S3, and in the future more storage backends. cumulus-util now
4 permits URLS for specifying storage locations, including s3:// for
6 - The script-level interface between the main cumulus binary and
7 remote storage has changed. It should not yet be considered
9 - Add a tool (cumulus-sync) for copying snapshots between storage
11 - Add a (mostly proof-of-concept) FUSE interface for accessing
13 - Implement a basic garbage-collection command for deleting unused
14 segments. There still needs to be a way to select snapshots to
15 delete (perhaps based on some type of schedule).
16 - Assorted minor bugfixes.
19 - SNAPSHOT FORMAT CHANGE: The snapshot format has been extended to
20 allow the local database to be partially rebuilt from the contents
21 of the metadata log. This may be useful for disaster recovery and
22 other scenarios. The changes are small, but old tools will not
24 - Add the --rebuild-statcache option which will re-read all files
25 instead of depending on the statcache file to be correct. This
26 will additionally print warnings if a file has changed but would
27 not have been detected as changed based on the statcache, and will
28 add subfile incremental signatures for existing files that do not
32 - LOCAL DATABASE CHANGE: The addition of subfile incrementals has
33 requires an extension to the local database. The upgrade script
34 contrib/upgrade0.7-localdb.sql should be run prior to running
35 backups with this version.
36 - Name change: the system is now known as Cumulus (replacing the old
37 name of "LBS"). Some traces of the old name still remain.
38 - Initial support for direct backups to remote storage. A sample
39 script is provided for backing up to Amazon S3. Other scripts
40 should be simple to write. The interface should not yet be
41 considered stable, and may change in a future release.
42 - Support for efficient subfile incremental backups. Changes to a
43 small portion of a file do not require re-uploading substantially
44 all of the file, even in cases where there is an insertion or
45 deletion that shifts most of the file contents.
46 - Allow selected files/directories to be restored, not simply an
47 entire snapshot. Additionally, restore files in an order that
48 should optimize performance (restore files based on how they are
49 grouped into segments, instead of lexicographic order).
50 Currently, the implementation of these changes requires that all
51 metadata be loaded into memory when the restore tool runs, so it
52 is more memory-intensive than the old version. This may be fixed
53 in a future version; in the meantime, if the current restore tool
54 requires too much memory, try the old restore tool or the
56 - Add a verbose ("-v") flag. By default, files will not be listed
57 as they are backed up. The old behavior can be turned back on
61 - SNAPSHOT FORMAT CHANGE: A few minor tweaks have been made to the
62 snapshot format. There is nothing substantial--this is primarily
63 a cleaning up of the format before use becomes more widespread.
64 An efficient and explicit means for representing sparse files has
65 been added to the format. Updated tools will be needed to
66 properly understand the new snapshots. Tools will continue to be
67 able to read the old snapshot format.
68 - LOCAL DATABASE CHANGE: The local database has been changed so that
69 segment utilization data for past snapshots is stored more
70 efficiently. The script in contrib/upgrade0.6-localdb.sql must be
71 run on a database to upgrade it prior to running backups with the
73 - The statcache implementation has been reworked, and the format
74 changed. The first backup will run more slowly since the old
75 statcache data will be ignored. New statcache data will be
76 written to statcache2. The old statcache file can be deleted.
77 - Metadata is shared between snapshots where possible. The
78 --full-metadata option can be used to disable this (completely
79 writing out all metadata with the new snapshot).
80 - Add an "intent" field to snapshots, which specifies informally how
81 long a snapshot is intended to be kept. For example, 1 can be
82 used for daily snapshots, and 7 for weekly snapshots. Segment
83 cleaning is now partly guided by intent values, but tuning of the
84 cleaning algorithms is not yet finished.
87 - Have the lbs-util tool check the version number when reading a
88 snapshot, and signal an error if the format is not recognized.
89 - Include a sample script, contrib/parity-gen, for creating
90 RAID-like parity sets to recover from some segment corruption.
91 The par2 command (http://parchive.sourceforge.net/) is used to
92 actually generate the parity sets; the parity-gen script simply
93 automates maintaining the parity sets.
96 - Much improved Python interface for accessing and manipulating LBS
97 archives and local database information. The interface should not
98 yet be considered completely stable.
99 - Python implementation of lbs-util now includes most of the
100 features of the Perl implementation, plus some other new features.
101 The Perl library and utility are deprecated and have been removed
103 - Preliminary snapshot restore support in lbs-util. This is still
104 not yet extensively tested.
107 - Documentation improvements: a getting started README, and a
108 description of some of the implementation details.
109 - Include an example filter script, lbs-filter-gpg, for encrypting
110 segment data as it is written.
111 - Add support for signed snapshot descriptor files (via the
112 --signature-filter option). Due to the checksums contained in the
113 snapshot descriptor, the signature covers the entire contents of
115 - Preview release of lbs-util.py, a Python-based program for
116 managing LBS snapshots. It currently implements a very simple
117 form of automatic segment cleaning, though this isn't much tested.
118 - Improved segment repacking support; after cleaning segments,
119 objects can be written out in different groups based on age. This
120 may help with long-term segment cleaning efficiency, but probably
124 - LOCAL DATABASE CHANGE: A checksums file is now written out along
125 with the snapshot descriptor that contains checksums of segments
126 used in a snapshot. This will allow quick integrity checks of a
127 snapshot, without needing to decompress or decrypt the segment
128 files. The local database schema was modified to store the
129 checksums for segments.
130 - Build dependence on libtar has been dropped. All necessary
131 support for writing TAR files is now directly included with the
133 - The snapshot format is now documented, at least in part. See
135 - Snapshots now include link count and inode number for files with
136 multiple hard links, so that a restore program could determine
137 which files should be hard linked on restore. The reference
138 restore.pl script does not use this.
139 - Bugfix: Print a help message instead of crashing if no files are
141 - Bugfix: File descriptors of files being backed up were closed
142 twice. In some cases this might have led to an unrelated file
143 being closed (if the file descriptor was quickly re-used).
144 - Preview of a new lbs-util command for maintaining snapshots.
145 Functionality is currently limited.
148 - Update reference restore.pl script to handle octal/hexadecimal
149 (format change introduced in 0.2).
150 - Better support for mixing multiple backup schemes: the statcache
151 file is now per-scheme, so performance should be better in cases
152 where multiple different backups (of different source directories)
153 are made, all sharing the same local database directory.
156 - Bugfix: Do not print an error when only one directory is specified
160 - SNAPSHOT FORMAT CHANGE: Metadata logs in backups can now contain
161 octal and hexadecimal values. Old restore programs may not be
162 able to read these new backups. The reference restore.pl script
163 has not yet been updated.
164 - LOCAL DATABASE CHANGE: Backups may be assigned a name with the
165 --scheme= command-line option, to allow different backup sets to
166 be mixed in the same directory. The database schema for the local
168 - Improved selection (inclusion/exclusion) of files to be included
170 - Device major/minor numbers are dumped for block and character