cumulus.git
17 years agoContinue work on reference decoder.
Michael Vrable [Sat, 12 May 2007 06:18:43 +0000 (23:18 -0700)]
Continue work on reference decoder.

  - Add code to parse metadata sections (including following indirect
    pointers).
  - Add code to parse data block listings (including indirect pointers).
  - Verify checksums on all files.

Primarily left: actually write restored file contents out to the
filesystem.

17 years agoBegin work on a reference decoder for backups.
Michael Vrable [Sat, 12 May 2007 04:55:36 +0000 (21:55 -0700)]
Begin work on a reference decoder for backups.

The intent is that the reference decoder will eventually be a tool for
recovery, if need, before a better tool is written.  It should also help to
verify the format specification and backup tool.

The reference decoder can currently parse a single object reference and
extract the data for it.

17 years agoWrite backup descriptor to a file, not stdout.
Michael Vrable [Sat, 12 May 2007 03:54:39 +0000 (20:54 -0700)]
Write backup descriptor to a file, not stdout.

The backup descriptor names the object at the root of the snapshot, and
lists all the segments needed.  The filename used is currently fixed,
though, and should later be based on the current time.

17 years agoClean up error reporting.
Michael Vrable [Fri, 11 May 2007 21:49:50 +0000 (14:49 -0700)]
Clean up error reporting.

17 years agoUse larger metadata blocks, and don't output type field twice.
Michael Vrable [Fri, 11 May 2007 21:46:53 +0000 (14:46 -0700)]
Use larger metadata blocks, and don't output type field twice.

17 years agoDebugging output cleanup.
Michael Vrable [Fri, 11 May 2007 21:39:23 +0000 (14:39 -0700)]
Debugging output cleanup.

17 years agoAllow metadata to be written incrementally.
Michael Vrable [Fri, 11 May 2007 19:21:50 +0000 (12:21 -0700)]
Allow metadata to be written incrementally.

17 years agoRemove checksums and reference tracking; I think they are not needed.
Michael Vrable [Fri, 11 May 2007 17:37:29 +0000 (10:37 -0700)]
Remove checksums and reference tracking; I think they are not needed.

17 years agoA few minor adjustments to the ObjectReference interface.
Michael Vrable [Fri, 11 May 2007 17:32:00 +0000 (10:32 -0700)]
A few minor adjustments to the ObjectReference interface.

17 years agoAdd a new object-oriented wrapper for building object references.
Michael Vrable [Fri, 11 May 2007 05:10:26 +0000 (22:10 -0700)]
Add a new object-oriented wrapper for building object references.

17 years agoMiscellaneous changes.
Michael Vrable [Fri, 11 May 2007 00:43:00 +0000 (17:43 -0700)]
Miscellaneous changes.

  - Disable profiling output for now.  It can be turned on again later.
  - Allow destination directory for backup files to be specified on the
    command-line (but we should add proper command-line parsing later).
  - More work to track inter-object references, but not yet finished.

17 years agoStart work on a more object-oriented interface for creating objects.
Michael Vrable [Thu, 3 May 2007 22:56:04 +0000 (15:56 -0700)]
Start work on a more object-oriented interface for creating objects.

17 years agoRename tarstore -> store, since it is the only implementation now.
Michael Vrable [Thu, 3 May 2007 22:13:21 +0000 (15:13 -0700)]
Rename tarstore -> store, since it is the only implementation now.

17 years agoRemove old store implementation.
Michael Vrable [Thu, 3 May 2007 22:03:19 +0000 (15:03 -0700)]
Remove old store implementation.

17 years agoPartial support for tracker inter-object references.
Michael Vrable [Thu, 3 May 2007 21:23:21 +0000 (14:23 -0700)]
Partial support for tracker inter-object references.

17 years agoAdd indrect objects for listing contents of large files.
Michael Vrable [Wed, 25 Apr 2007 05:46:42 +0000 (22:46 -0700)]
Add indrect objects for listing contents of large files.

17 years agoRemove code using old segment writer.
Michael Vrable [Mon, 23 Apr 2007 04:41:25 +0000 (21:41 -0700)]
Remove code using old segment writer.

All writes should now happen to TAR-format segments, and data should be
serialized as text.

17 years agoAutomatically split segments to meet size targets.
Michael Vrable [Mon, 23 Apr 2007 04:09:11 +0000 (21:09 -0700)]
Automatically split segments to meet size targets.

17 years agoAdd URI-style escaping of filename characters.
Michael Vrable [Sat, 21 Apr 2007 18:05:28 +0000 (11:05 -0700)]
Add URI-style escaping of filename characters.

17 years agoRemove early test code for the tarfile storage backend.
Michael Vrable [Mon, 16 Apr 2007 04:56:33 +0000 (21:56 -0700)]
Remove early test code for the tarfile storage backend.

17 years agoReduce debugging output.
Michael Vrable [Mon, 16 Apr 2007 04:51:41 +0000 (21:51 -0700)]
Reduce debugging output.

17 years agoAdd checksums to tarstore, and partially migrate to using it for storage.
Michael Vrable [Mon, 16 Apr 2007 04:34:07 +0000 (21:34 -0700)]
Add checksums to tarstore, and partially migrate to using it for storage.

Each tarfile segment gets a file called "checksums" now which contains
SHA-1 hashes of all objects contained in it.  There is currently no
checksum computed for the checksum file itself.

Begin migrating scandir to using the tar-based storage for backups.
Backups in the old format are still written, and there is still some
other cleanup to be done, but most of the raw data is currently handled.

17 years agoBegin work on an alternate object store mechanism using the TAR format.
Michael Vrable [Sun, 15 Apr 2007 20:48:09 +0000 (13:48 -0700)]
Begin work on an alternate object store mechanism using the TAR format.

Each segment is placed in a separate file, and each object is a file stored
within that TAR archive.  libtar is used to write (and perhaps later read)
tar files.

17 years agoTag objects with a 4-byte type code.
Michael Vrable [Wed, 7 Feb 2007 21:45:30 +0000 (13:45 -0800)]
Tag objects with a 4-byte type code.

17 years agoReturn the names for allocated objects, and link file metaata to data.
Michael Vrable [Thu, 11 Jan 2007 22:47:41 +0000 (14:47 -0800)]
Return the names for allocated objects, and link file metaata to data.

Extend the interface for the new_object methods so that they can return the
segment UUID and object index for created objects.  Then, use this to write
out links from the file metadata to the data blocks making up the file, via
an indirect block.

17 years agoSupport for spreading objects across segments.
Michael Vrable [Sat, 30 Dec 2006 03:24:48 +0000 (19:24 -0800)]
Support for spreading objects across segments.

New segments are created automatically whenever an old segment becomes too
large (current limit is set at something over 1 MB, but this could be
adjusted).

17 years agoAppend checksums to segments to allow some verification.
Michael Vrable [Fri, 29 Dec 2006 19:49:20 +0000 (11:49 -0800)]
Append checksums to segments to allow some verification.

17 years agoAdd new interface for creating new segments.
Michael Vrable [Mon, 25 Dec 2006 06:28:59 +0000 (22:28 -0800)]
Add new interface for creating new segments.

17 years agoInitial support for objects encapsulated into segments.
Michael Vrable [Sun, 24 Dec 2006 04:15:31 +0000 (20:15 -0800)]
Initial support for objects encapsulated into segments.

17 years agoSome Makefile improvements:
Michael Vrable [Sun, 24 Dec 2006 01:21:10 +0000 (17:21 -0800)]
Some Makefile improvements:
  - (Mostly) automatic header dependency tracking.
  - Use pkg-config to pull in dependencies for libuuid.

17 years agoImprove comments, and track number of bytes written to a stream.
Michael Vrable [Sun, 24 Dec 2006 01:19:58 +0000 (17:19 -0800)]
Improve comments, and track number of bytes written to a stream.

17 years agoCompute SHA-1 checksums of regular files to be stored with index data.
Michael Vrable [Sat, 23 Dec 2006 20:46:19 +0000 (12:46 -0800)]
Compute SHA-1 checksums of regular files to be stored with index data.

17 years agoClean up output while scanning directories.
Michael Vrable [Sat, 23 Dec 2006 19:58:18 +0000 (11:58 -0800)]
Clean up output while scanning directories.

17 years agoExtend basic support for serializing simple data types.
Michael Vrable [Sat, 23 Dec 2006 06:53:06 +0000 (22:53 -0800)]
Extend basic support for serializing simple data types.

Add support for serializing integers, strings, and dictionaries.  Then,
extend the directory scanner to output a summary of stat information for
each file/directory.

17 years agoFill in a couple more details about a proposed file format.
Michael Vrable [Wed, 20 Dec 2006 02:55:15 +0000 (18:55 -0800)]
Fill in a couple more details about a proposed file format.

17 years agoBegin work to document a proposed file format for backups.
Michael Vrable [Sat, 16 Dec 2006 07:32:15 +0000 (23:32 -0800)]
Begin work to document a proposed file format for backups.

17 years agoAdd support for reading symlinks.
Michael Vrable [Sat, 16 Dec 2006 05:24:37 +0000 (21:24 -0800)]
Add support for reading symlinks.

17 years agoRead contents of all regular files processed.
Michael Vrable [Wed, 13 Dec 2006 16:44:21 +0000 (08:44 -0800)]
Read contents of all regular files processed.

17 years agoMore work on descending directory structure.
Michael Vrable [Tue, 12 Dec 2006 05:46:33 +0000 (21:46 -0800)]
More work on descending directory structure.

Improvements:
  - Directory listings are processed in sorted order.
  - Open regular files, but be very paranoid in doing so (try to avoid
    danger from race conditions).

17 years agoBuild with support for large files enabled.
Michael Vrable [Sun, 10 Dec 2006 19:45:38 +0000 (11:45 -0800)]
Build with support for large files enabled.

18 years agoInitial commit
Michael Vrable [Thu, 23 Nov 2006 06:33:10 +0000 (22:33 -0800)]
Initial commit