cumulus.git
2 years agoReplace boost::scoped_ptr with std::unique_ptr. master
Michael Vrable [Sat, 24 Nov 2018 20:55:34 +0000 (12:55 -0800)]
Replace boost::scoped_ptr with std::unique_ptr.

This is a direct replacement, skipping over the chance to clean up APIs at
this point.  It does allow us to drop the compile-time dependency on boost,
however.

2 years agoFix C++ compilation warnings/errors under g++ 8.2.
Michael Vrable [Sat, 24 Nov 2018 20:49:33 +0000 (12:49 -0800)]
Fix C++ compilation warnings/errors under g++ 8.2.

In metadata.cc:
    warning: In the GNU C Library, "major" is defined by
    <sys/sysmacros.h>.  For historical compatibility, it is currently
    defined by <sys/types.h> as well, but we plan to remove this soon.
    To use "major", include <sys/sysmacros.h> directly. If you did not
    intend to use a system-defined macro "major", you should undefine it
    after including <sys/types.h>.

In third_party/sha1.cc:
    error: types may not be defined within __builtin_offsetof

Alter the code to always assume that buffers are
unaligned, which should be correct at the cost of performance.  This
shouldn't matter much in practice since SHA-1 is mostly not being used
by Cumulus any longer.

In third_party/sha256.cc:
    warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use
    _DEFAULT_SOURCE"

7 years agoAlways perform checksum comparison (for disk corruption) when possible.
Michael Vrable [Wed, 10 Sep 2014 21:36:18 +0000 (14:36 -0700)]
Always perform checksum comparison (for disk corruption) when possible.

If the statcache indicates that a file should be unchanged, but we have
read the file contents anyway (generally, because this was needed for
segment cleaning), then comparing the old and new checksums is basically
free.  Do so, and report any discrepancies as they may indicate on-disk
corruption.

7 years agoDo not include object checksums in file contents listings.
Michael Vrable [Wed, 10 Sep 2014 21:16:28 +0000 (14:16 -0700)]
Do not include object checksums in file contents listings.

Do not list object-level checksums as these will make the metadata larger,
and the file-level checksum already ensures the file contents on restore
are correct.

7 years agoRework uri_encode/uri_decode to more cleanly work with bytes/strings.
Michael Vrable [Sun, 7 Sep 2014 23:37:43 +0000 (16:37 -0700)]
Rework uri_encode/uri_decode to more cleanly work with bytes/strings.

More clearly specify the intended behavior of uri_encode and uri_decode,
and make them work both for Python 2 and Python 3.

7 years agoImprove parsing performance.
Michael Vrable [Wed, 26 Feb 2014 01:18:30 +0000 (17:18 -0800)]
Improve parsing performance.

7 years agoImplement a new Python wrapper for the local database.
Michael Vrable [Tue, 18 Feb 2014 00:09:40 +0000 (16:09 -0800)]
Implement a new Python wrapper for the local database.

This provides a cleaned-up wrapper for the local Cumulus database, in a
separate module.  This also currently functions as a simple segment
cleaner (to be used until a more comprehensive cleaner is available).

7 years agoUse posixpath to join remote storage paths.
Michael Vrable [Wed, 5 Feb 2014 05:00:32 +0000 (21:00 -0800)]
Use posixpath to join remote storage paths.

7 years agoChange syntax for size-assertion slices. This is an incompatible change.
Michael Vrable [Wed, 5 Feb 2014 04:20:09 +0000 (20:20 -0800)]
Change syntax for size-assertion slices.  This is an incompatible change.

Simplify the syntax by replacing the [=nnn] syntax with just [nnn].  Slices
that do not specify a size are written [0+nnn]; the old backup tool has
been doing this already since about v0.8.  Before that point, [nnn] was a
shorthand for [0+nnn], and will be mis-interpreted by modern tools.

7 years agoUpdate Python dependencies.
Michael Vrable [Fri, 31 Jan 2014 16:09:12 +0000 (08:09 -0800)]
Update Python dependencies.

7 years agoUse the subprocess module, not os.popen2, for Python 3 compatibility.
Michael Vrable [Fri, 31 Jan 2014 16:03:58 +0000 (08:03 -0800)]
Use the subprocess module, not os.popen2, for Python 3 compatibility.

7 years agoUpdate tests to allow alternate Python versions (for py3 testing).
Michael Vrable [Fri, 31 Jan 2014 15:40:24 +0000 (07:40 -0800)]
Update tests to allow alternate Python versions (for py3 testing).

7 years agoByte/string handling fixes for Python 3.
Michael Vrable [Fri, 31 Jan 2014 00:06:53 +0000 (16:06 -0800)]
Byte/string handling fixes for Python 3.

7 years agoStart a cleanup on the storage backends.
Michael Vrable [Wed, 29 Jan 2014 05:34:22 +0000 (21:34 -0800)]
Start a cleanup on the storage backends.

The basic interface and file and s3 backends are updated.  Remaining
backends still need some work.

7 years agoManual 2to3 fixups.
Michael Vrable [Tue, 28 Jan 2014 16:28:22 +0000 (08:28 -0800)]
Manual 2to3 fixups.

7 years agoRun 2to3 over Python sources.
Michael Vrable [Tue, 28 Jan 2014 16:24:56 +0000 (08:24 -0800)]
Run 2to3 over Python sources.

7 years agoUpdate backend code and cumulus-sync for the new backup layout.
Michael Vrable [Wed, 23 Oct 2013 20:30:12 +0000 (13:30 -0700)]
Update backend code and cumulus-sync for the new backup layout.

7 years agoDelete a debugging message on hash algorithm registration.
Michael Vrable [Mon, 4 Nov 2013 21:51:00 +0000 (13:51 -0800)]
Delete a debugging message on hash algorithm registration.

7 years agoSegment expiration should affect metadata re-use too.
Michael Vrable [Fri, 25 Oct 2013 02:54:47 +0000 (19:54 -0700)]
Segment expiration should affect metadata re-use too.

7 years agoBugfix for database checksum reconstruction with zero blocks.
Michael Vrable [Mon, 21 Oct 2013 21:56:59 +0000 (14:56 -0700)]
Bugfix for database checksum reconstruction with zero blocks.

7 years agoSample code to reload the segment table into the local database.
Michael Vrable [Mon, 21 Oct 2013 21:39:43 +0000 (14:39 -0700)]
Sample code to reload the segment table into the local database.

7 years agoInclude timestamps in reconstructed segment metadata.
Michael Vrable [Mon, 21 Oct 2013 21:10:12 +0000 (14:10 -0700)]
Include timestamps in reconstructed segment metadata.

7 years agoUse a scoped_ptr to avoid FileFilter leaks.
Michael Vrable [Mon, 21 Oct 2013 17:53:16 +0000 (10:53 -0700)]
Use a scoped_ptr to avoid FileFilter leaks.

7 years agoFix output file size estimation.
Michael Vrable [Mon, 21 Oct 2013 17:02:17 +0000 (10:02 -0700)]
Fix output file size estimation.

When piping output through a filter (for compression/encryption), we need
to keep the underlying file descriptor open so we can check the output file
size.  Previously it was being closed too early so Tarfile::size_estimate
was failing.

7 years agoFix a linker error by linking against pthreads.
Michael Vrable [Mon, 21 Oct 2013 16:47:00 +0000 (09:47 -0700)]
Fix a linker error by linking against pthreads.

7 years agoUse scoped_ptr from boost.
Michael Vrable [Wed, 19 Jun 2013 17:27:04 +0000 (10:27 -0700)]
Use scoped_ptr from boost.

7 years agoUse valgrind to check for memory leaks in the tests.
Michael Vrable [Wed, 19 Jun 2013 17:26:42 +0000 (10:26 -0700)]
Use valgrind to check for memory leaks in the tests.

7 years agoEnsure metadata file is written prior to calculating checksum.
Michael Vrable [Wed, 19 Jun 2013 17:05:09 +0000 (10:05 -0700)]
Ensure metadata file is written prior to calculating checksum.

7 years agoRework filter process code to make the interface simpler.
Michael Vrable [Tue, 18 Jun 2013 16:55:51 +0000 (09:55 -0700)]
Rework filter process code to make the interface simpler.

7 years agoImplement a string_printf utility function.
Michael Vrable [Wed, 12 Jun 2013 17:53:42 +0000 (10:53 -0700)]
Implement a string_printf utility function.

7 years agoClean up the file hashing interface and segment metadata hashes.
Michael Vrable [Fri, 10 May 2013 04:53:07 +0000 (21:53 -0700)]
Clean up the file hashing interface and segment metadata hashes.

Switch the segment file hashes to the generic hashing code, and ensure that
the metadata checkpoint file also has a hash computed and stored.

7 years agoWrite out a generic segment metadata file with backups.
Michael Vrable [Thu, 2 May 2013 04:17:58 +0000 (21:17 -0700)]
Write out a generic segment metadata file with backups.

This can be used to rebuild local database state without redownloading all
segments.

7 years agoFactor out time formatting functions.
Michael Vrable [Thu, 2 May 2013 03:25:03 +0000 (20:25 -0700)]
Factor out time formatting functions.

7 years agoSwitch to generic hash algorithms for subfile incrementals.
Michael Vrable [Tue, 30 Apr 2013 15:32:57 +0000 (08:32 -0700)]
Switch to generic hash algorithms for subfile incrementals.

7 years agoClean up database, and timestamp handling in particular.
Michael Vrable [Mon, 29 Apr 2013 23:45:43 +0000 (16:45 -0700)]
Clean up database, and timestamp handling in particular.

7 years agoImprove database rebuilding.
Michael Vrable [Fri, 26 Apr 2013 19:50:30 +0000 (12:50 -0700)]
Improve database rebuilding.

  - Compute estimated sizes (lower bounds) for objects.
  - Do not rebuild subfile incrementals for very small objects.

7 years agoCleanup to used data tracking in localdb.
Michael Vrable [Thu, 25 Apr 2013 19:49:26 +0000 (12:49 -0700)]
Cleanup to used data tracking in localdb.

7 years agoAdd code to rebuild_database.py to recompute segment metadata.
Michael Vrable [Thu, 25 Apr 2013 03:52:15 +0000 (20:52 -0700)]
Add code to rebuild_database.py to recompute segment metadata.

This requires that the segments be available.

7 years agoDatabase rebuilder fixes.
Michael Vrable [Wed, 24 Apr 2013 19:57:32 +0000 (12:57 -0700)]
Database rebuilder fixes.

7 years ago.gitignore update
Michael Vrable [Wed, 24 Apr 2013 19:04:48 +0000 (12:04 -0700)]
.gitignore update

7 years agoConvert to sqlite3 module, from pysqlite2.
Michael Vrable [Wed, 24 Apr 2013 15:58:14 +0000 (08:58 -0700)]
Convert to sqlite3 module, from pysqlite2.

7 years agoReworking cleanup.
Michael Vrable [Mon, 18 Mar 2013 17:21:14 +0000 (10:21 -0700)]
Reworking cleanup.

7 years agoDrop intent field from the database.
Michael Vrable [Wed, 27 Feb 2013 18:12:36 +0000 (10:12 -0800)]
Drop intent field from the database.

7 years agoUpdate Python code to work with the new backup format.
Michael Vrable [Sat, 2 Feb 2013 16:40:36 +0000 (08:40 -0800)]
Update Python code to work with the new backup format.

7 years agoChanges to the Cumulus backup format and tools.
Michael Vrable [Fri, 14 Dec 2012 04:34:21 +0000 (20:34 -0800)]
Changes to the Cumulus backup format and tools.

  - Switch to a hierarchical file layout.
  - Remove old references to the "LBS" name.

7 years agoStart on a tool to rebuild the local database if it is lost.
Michael Vrable [Wed, 16 Jan 2013 20:15:22 +0000 (12:15 -0800)]
Start on a tool to rebuild the local database if it is lost.

This can also be used to reconstruct the local database in a new format, or
upgrade hash algorithms.

7 years agoStart work on tests for Cumulus.
Michael Vrable [Sat, 1 Dec 2012 05:12:53 +0000 (21:12 -0800)]
Start work on tests for Cumulus.

7 years agoImprove tracking of segments and segment utilization.
Michael Vrable [Fri, 21 Sep 2012 22:50:14 +0000 (15:50 -0700)]
Improve tracking of segments and segment utilization.

Update the local database and cumulus binary to keep better track of
segment utilization data, with an eventual goal of improving cleaning
algorithms.

7 years agoNEWS and TODO file updates.
Michael Vrable [Thu, 27 Sep 2012 04:24:49 +0000 (21:24 -0700)]
NEWS and TODO file updates.

7 years agoReword copyright notices on code in third_party for clarity.
Michael Vrable [Wed, 26 Sep 2012 15:23:57 +0000 (08:23 -0700)]
Reword copyright notices on code in third_party for clarity.

8 years agoUpdate copyright notices to use a central AUTHORS file.
Michael Vrable [Tue, 25 Sep 2012 23:09:25 +0000 (16:09 -0700)]
Update copyright notices to use a central AUTHORS file.

8 years agoRework hash implementations to provide additional algorithms.
Michael Vrable [Wed, 26 Sep 2012 14:54:25 +0000 (07:54 -0700)]
Rework hash implementations to provide additional algorithms.

Provide a generic hash interface, rework SHA-1 to use the interface, and
also add code for SHA-224/SHA-256.

8 years agoAdd sha224/sha256 as supported hash algorithms.
Michael Vrable [Thu, 27 Sep 2012 04:24:58 +0000 (21:24 -0700)]
Add sha224/sha256 as supported hash algorithms.

Document new hash algorithms in the format description, and include all
supported algorithms in the Python code.  The cumulus binary does not yet
support the new algorithms.

8 years agoCreate a third_party directory for files copied from other projects.
Michael Vrable [Thu, 30 Aug 2012 04:28:16 +0000 (21:28 -0700)]
Create a third_party directory for files copied from other projects.

8 years agoCollected bugfixes, improvements, and cleanups.
Michael Vrable [Mon, 29 Oct 2012 16:28:29 +0000 (09:28 -0700)]
Collected bugfixes, improvements, and cleanups.

 - Do not leak file descriptors when merging new include/exclude rules.
 - Improved error message when unable to open a directory.
 - Delete obsolete clean-segments.pl script (long since replaced by
   functionality in cumulus-util).

9 years agoUpdate usage text for new include/exclude filtering mechanism.
Michael Vrable [Tue, 25 Sep 2012 21:03:43 +0000 (14:03 -0700)]
Update usage text for new include/exclude filtering mechanism.

9 years agoFirst step towards a new, improved cumulus front-end.
Michael Vrable [Fri, 21 Sep 2012 00:41:02 +0000 (17:41 -0700)]
First step towards a new, improved cumulus front-end.

This commit adds several things:
  - Rules for selecting sets of backups for expiration, for managing old
    snapshots.
  - A configuration file format and parser for listing settings such as
    backup expiration policies.
  - Code in a small utility library that can expire old snapshots
    according to configuration settings.

Eventually this code should be part of a new cumulus front-end in Python
that can integrate snapshot and database management with backup runs.

9 years agoRefactor cumulus-util into a library plus small command front-end.
Michael Vrable [Thu, 20 Sep 2012 20:43:22 +0000 (13:43 -0700)]
Refactor cumulus-util into a library plus small command front-end.

There is still more cleanup work to be done here, but this first step
makes commands in cumulus-util importable as a Python module for re-use
elsewhere.

9 years agoSwitch to hashlib (Python >= 2.5) for hash algorithms.
Michael Vrable [Fri, 31 Aug 2012 03:15:27 +0000 (20:15 -0700)]
Switch to hashlib (Python >= 2.5) for hash algorithms.

The old sha module is deprecated, and hashlib gives access to newer hash
algorithms.  Use it to add SHA-256 support as well.

9 years agoReimplement the file include/exclude filtering mechanism.
Michael Vrable [Wed, 12 Sep 2012 03:30:32 +0000 (20:30 -0700)]
Reimplement the file include/exclude filtering mechanism.

Implement something more similar to that rsync does.  Accept some number of
starting points for the backup, and an ordered set of include/exclude
patterns.  Also permit rules to be merged in during the backup process, by
per-subtree rule files.

9 years agoAdd a cache around getpwuid/getgrgid to avoid repeated calls.
Michael Vrable [Sat, 16 Jun 2012 06:28:32 +0000 (23:28 -0700)]
Add a cache around getpwuid/getgrgid to avoid repeated calls.

9 years agoPut linker flags at end of compiler command-line.
Michael Vrable [Thu, 31 May 2012 03:18:27 +0000 (20:18 -0700)]
Put linker flags at end of compiler command-line.

Put $(LDFLAGS) after the sources, since otherwise the linker may be unable
to find all symbols.

9 years agoUpdate to support new snapshot version in Python tools.
Michael Vrable [Wed, 30 May 2012 04:47:50 +0000 (21:47 -0700)]
Update to support new snapshot version in Python tools.

Also, remove some older references to "lbs".

9 years agoSome updates to the backup format:
Michael Vrable [Wed, 30 May 2012 04:39:49 +0000 (21:39 -0700)]
Some updates to the backup format:

  - The name of the descriptor file is now based on the timestamp in UTC,
    not local time.  This should give better behavior in the case of
    frequent snapshots and daylight saving time changes.
  - Update the backup format to v0.11 (other changes may yet come before
    release); with this also include a name change from "LBS" to "Cumulus"
    in the format name.

9 years agoRename scandir.cc to main.cc.
Michael Vrable [Wed, 30 May 2012 04:15:55 +0000 (21:15 -0700)]
Rename scandir.cc to main.cc.

9 years agoPrepare 0.10 release.
Michael Vrable [Tue, 29 May 2012 15:07:21 +0000 (08:07 -0700)]
Prepare 0.10 release.

10 years agoEnable additional compiler warnings
Michael Vrable [Thu, 23 Jun 2011 18:51:01 +0000 (11:51 -0700)]
Enable additional compiler warnings

11 years agoApply fixes to s3:// URL parsing under Python 2.6.
Michael Vrable [Wed, 30 Jun 2010 19:29:49 +0000 (12:29 -0700)]
Apply fixes to s3:// URL parsing under Python 2.6.

The urlparse module starting in Python 2.6 appears to be compliant with RFC
3986, while previous versions were not.  This causes a change in the
behaviro of parsing s3:// URLs, however, resulting in breakage with Python
2.6.

Try to fix this by adjusting our code so that it works with either the old
or the new behavior.

11 years agoadd some further documentation on the sftp backend
Albert Dengg [Wed, 31 Mar 2010 20:33:03 +0000 (22:33 +0200)]
add some further documentation on the sftp backend

* document that password auth and passowrd protected keys are not supported
* document the configuration via ~/.ssh/config
* document that the host key already has to be in ~/.ssh/known_hosts

11 years agofix usage of wrong variable
Albert Dengg [Wed, 31 Mar 2010 20:33:02 +0000 (22:33 +0200)]
fix usage of wrong variable

* fix some leftover reference to the wrong variable (self.path instead
  of self.netloc)

11 years agoFormatting/spelling fix.
Michael Vrable [Tue, 30 Mar 2010 22:03:56 +0000 (15:03 -0700)]
Formatting/spelling fix.

11 years agoSome assorted fixes for the SFTP backend.
Michael Vrable [Tue, 30 Mar 2010 21:59:38 +0000 (14:59 -0700)]
Some assorted fixes for the SFTP backend.

  - Automatically call the close() method of a storage backend when the
    object is garbage collected.
  - Calling stat in the SFTP backend when the file doesn't exist will raise
    a Cumulus NotFoundError instead of a generic IOError.
  - Avoid the use of keyword arguments when calling SFTPClient methods (in
    my testing the first argument might be called 'path' instead of
    'filename', but avoid the problem altogether by just using positional
    arguments).

11 years agocode cleanup & regression fix
Albert Dengg [Fri, 26 Mar 2010 13:04:14 +0000 (14:04 +0100)]
code cleanup & regression fix

11 years agowork for python 2.5
Albert Dengg [Fri, 26 Mar 2010 13:04:13 +0000 (14:04 +0100)]
work for python 2.5

* import "with" statement from __future__ to be able to use it in 2.5

11 years ago* document requirements for sftp storage
Albert Dengg [Fri, 26 Mar 2010 13:04:12 +0000 (14:04 +0100)]
* document requirements for sftp storage

11 years agoimplement a basic sftp storage backend
Albert Dengg [Fri, 26 Mar 2010 13:04:11 +0000 (14:04 +0100)]
implement a basic sftp storage backend

* add an option to cumulus-store storage backends to explictly close the
  connection, needed because of the design of the ssh/sftp lib
* add a sftp storage backend based on paramiko

12 years agoHave FTP backend reconnect on timeout.
Michael Vrable [Wed, 30 Sep 2009 18:47:47 +0000 (11:47 -0700)]
Have FTP backend reconnect on timeout.

Additionally, ensure that transfers are done in binary mode.

Patch provided by Ralf Schlatterbeck <rsc@runtux.com>.

12 years agoThe map::at method does not always exist, so instead use map::find.
Michael Vrable [Sat, 26 Sep 2009 19:09:35 +0000 (12:09 -0700)]
The map::at method does not always exist, so instead use map::find.

Thanks to Achim J. Latz <achim.latz@qustodium.net>.

12 years agoEnsure printf format specifiers and types match (fixes compiler warning).
Michael Vrable [Wed, 23 Sep 2009 18:47:43 +0000 (11:47 -0700)]
Ensure printf format specifiers and types match (fixes compiler warning).

12 years agoImplement FTP backend and other code cleanups.
Ralf Schlatterbeck [Thu, 10 Sep 2009 21:42:20 +0000 (14:42 -0700)]
Implement FTP backend and other code cleanups.

Details:

- Fix a race condition in setup of RemoteStore: The backup_script was
  set *after* a thread was already started. Sometimes this lead to a
  segfault because the thread tested the backup script twice. During the
  first test the variable wasn't yet defined while in the second test it
  was -- this lead to a segfault due to an uninitialized output file.
  I've move the backup_script to the constructor as an optional
  parameter.
cumulus-util:
- put all command-documentation into docstrings of respective function
- add the docstrings to the usage text, so that we know which commands
  exist
- Fix hard-coded extension '.tar.gpg' for cmd_list_snapshot_sizes
- framework for automatically computing the right method to call for a
  given command
python/cumulus/store/file.py:
- fix constructor so that we can directly call it when it's not called
  via the factory with an URL as parameter
- for NotFoundError give type and filename
python/cumulus/store/ftp.py:
- new FTP backend

12 years agoAdd a few items to the TODO list.
Michael Vrable [Wed, 26 Aug 2009 18:45:19 +0000 (11:45 -0700)]
Add a few items to the TODO list.

12 years agoFix a segfault-causing bug when converting a numeric group to a name fails.
Michael Vrable [Sat, 8 Aug 2009 02:28:31 +0000 (19:28 -0700)]
Fix a segfault-causing bug when converting a numeric group to a name fails.

The original code had a copy-and-paste bug when converting a numeric group
id into a symbolic group name: rather than checking that getgrgid returned
a valid result, it checked the result of getpwuid.  If any files in the
backup snapshot belonged to a non-existent group, this resulted in a
segfault.

Problem found and patch provided by Chris Wilson <chris@aptivate.org>.

12 years agoAdd --exclude-name option.
Michael Vrable [Sat, 8 Aug 2009 02:22:42 +0000 (19:22 -0700)]
Add --exclude-name option.

The --exclude-name will exclude from a snapshot files or directories with
the specified name.  This contrasts with --exclude, which requires that the
full path be specified.

Initial patch by Chris Wilson <chris@aptivate.org>.  Committed with minor
modifications.

12 years agoUpdate for 0.9 release.
Michael Vrable [Tue, 28 Jul 2009 23:46:22 +0000 (16:46 -0700)]
Update for 0.9 release.

12 years agoUpdate README with information about remote storage.
Michael Vrable [Mon, 20 Jul 2009 01:02:38 +0000 (18:02 -0700)]
Update README with information about remote storage.

12 years agoInclude a missing header file.
Michael Vrable [Tue, 30 Jun 2009 18:25:30 +0000 (11:25 -0700)]
Include a missing header file.

This should fix compilation with newer versions of GCC.  Problem originally
reported by Robert Rebstock <rebstock@scienceworks.com>.

12 years agoImplement rudimentary garbage collection.
Michael Vrable [Sun, 31 May 2009 06:21:10 +0000 (23:21 -0700)]
Implement rudimentary garbage collection.

Implement a garbage collection method in cumulus-util which will search for
files not referenced by any current snapshots and delete them.  This still
doesn't let snapshots themselves be deleted automatically, but after
manually deleting a snapshot this will quickly delete all other old files.

12 years agoImplement metadata caching for S3 backend.
Michael Vrable [Sun, 31 May 2009 06:19:10 +0000 (23:19 -0700)]
Implement metadata caching for S3 backend.

Amazon S3 will return some limited object metadata when a list operation is
performed.  This is significantly cheaper than fetching the information for
objects one at a time.  In the S3 backend, implement a scan() method that
will list all objects and cache the metadata, then return cached results
when stat() is called.

12 years agolbs-filter-gpg has been renamed to cumulus-filter-gpg.
Michael Vrable [Thu, 26 Mar 2009 21:35:27 +0000 (14:35 -0700)]
lbs-filter-gpg has been renamed to cumulus-filter-gpg.

An instance of this was missed in the code.  Caught by Achim J. Latz
<achim.latz@qustodium.net>.

12 years agoInclude segment add/remove counts in list-snapshot-sizes.
Michael Vrable [Wed, 14 Jan 2009 22:06:31 +0000 (14:06 -0800)]
Include segment add/remove counts in list-snapshot-sizes.

12 years agoDetect decompression script needed for segments based on extension.
Michael Vrable [Mon, 15 Dec 2008 23:15:19 +0000 (15:15 -0800)]
Detect decompression script needed for segments based on extension.

12 years agocumulus-sync: Create a tool for copying snapshots between locations.
Michael Vrable [Mon, 15 Dec 2008 22:29:13 +0000 (14:29 -0800)]
cumulus-sync: Create a tool for copying snapshots between locations.

This will automatically find and copy all needed segments that are not
already present, and can handle both local filesystems and remote storage
with Amazon S3.

12 years agoDrop the use of exceptions for fatal error handling.
Michael Vrable [Thu, 20 Nov 2008 23:47:52 +0000 (15:47 -0800)]
Drop the use of exceptions for fatal error handling.

12 years agoImprove handling of file-not-found in remote storage layer.
Michael Vrable [Thu, 20 Nov 2008 23:38:39 +0000 (15:38 -0800)]
Improve handling of file-not-found in remote storage layer.

Implement a single NotFoundError which is thrown whenever a file does not
exist in any remote store, instead instance-specific error handling.

12 years agoDelete contrib/cumulus-store-s3.
Michael Vrable [Thu, 20 Nov 2008 20:11:41 +0000 (12:11 -0800)]
Delete contrib/cumulus-store-s3.

This has been replaced by cumulus-store, and the main cumulus executable
now uses a different interface so cumulus-store-s3 won't work any longer.

12 years agoRe-do cumulus side of upload script interface.
Michael Vrable [Thu, 20 Nov 2008 20:10:06 +0000 (12:10 -0800)]
Re-do cumulus side of upload script interface.

Update the cumulus executable so that the interface for the remote upload
script is compatible with the new cumulus-store script, allowing cumulus to
easily target different storage backends.

12 years agoIntroduce a script to provide access to remote repositories.
Michael Vrable [Thu, 20 Nov 2008 19:55:13 +0000 (11:55 -0800)]
Introduce a script to provide access to remote repositories.

cumulus-store is a Python script that uses the cumulus library code to
access various storage repositories (local file, S3, and extensible to
others).  It allows non-Python code to access these storage repositories
through a simple interface through stdin/stdout.

Additionally, make a few extensions and fixes to the cumulus Python
libraries.

12 years agoMakefile: Fix clean target.
Michael Vrable [Thu, 13 Nov 2008 22:38:44 +0000 (14:38 -0800)]
Makefile: Fix clean target.

12 years agocumulus-util: Automatically set Python search path.
Michael Vrable [Thu, 6 Nov 2008 18:47:56 +0000 (10:47 -0800)]
cumulus-util: Automatically set Python search path.

Attempt to set the Python library search path so that cumulus-util can find
the cumulus Python modules, without PYTHON_PATH having to be set
explicitly.

Modules are looked for in the "python" directory where the cumulus-util
binary resides; this is appropriate for running cumulus-util directly from
the source code directory, but may not be if the tools are installed
somewhere else.

12 years agocumulus-util: In list-snapshot-sizes output, display backup intent values.
Michael Vrable [Thu, 6 Nov 2008 18:47:18 +0000 (10:47 -0800)]
cumulus-util: In list-snapshot-sizes output, display backup intent values.