Michael Vrable [Fri, 10 Dec 2010 02:26:15 +0000 (18:26 -0800)]
Mostly implement inode merging for the in-proxy cleaner component.
This doesn't yet ensure modifications are serialized out again later.
Michael Vrable [Thu, 9 Dec 2010 20:12:04 +0000 (12:12 -0800)]
Fix to the cleaner when writing out a new inode map.
Michael Vrable [Thu, 9 Dec 2010 06:32:17 +0000 (22:32 -0800)]
Add code in the proxy cleaner component to iterate over new inodes.
For the moment we actually iterate over all inodes in the cleaned
checkpoint, but later can restrict to just those that are needed. Actually
merging inode changes also still needs to be implemented.
Michael Vrable [Tue, 7 Dec 2010 05:40:30 +0000 (21:40 -0800)]
In-progress commit of online cleaner.
This adds some code to check the cleaner log when the proxy is launched,
but doesn't yet merge any data.
Michael Vrable [Mon, 6 Dec 2010 04:46:49 +0000 (20:46 -0800)]
Rework the checkpoint record format to include a version vector.
This will be used for keeping track of whether we have incorporated changes
made by the cleaner or not, and is a first step (along with having the
clenaer write to the a separate set of logs) to making a functional online
cleaner.
Michael Vrable [Thu, 2 Dec 2010 19:39:56 +0000 (11:39 -0800)]
Try to avoid accessing profiling objects after they are freed.
Michael Vrable [Tue, 30 Nov 2010 04:59:14 +0000 (20:59 -0800)]
Write profile data if and only if requested.
Michael Vrable [Tue, 30 Nov 2010 04:57:27 +0000 (20:57 -0800)]
Allow profile results to be written to a file.
Michael Vrable [Mon, 29 Nov 2010 04:29:33 +0000 (20:29 -0800)]
Reduce debugging messages in non-verbose mode.
Michael Vrable [Mon, 29 Nov 2010 00:21:36 +0000 (16:21 -0800)]
More work on request profiling.
Michael Vrable [Tue, 23 Nov 2010 23:48:41 +0000 (15:48 -0800)]
Add locking, thread-ID tracking to profiling.
Michael Vrable [Tue, 23 Nov 2010 18:27:31 +0000 (10:27 -0800)]
Track requests that initiate inode fetches.
Michael Vrable [Tue, 23 Nov 2010 17:53:18 +0000 (09:53 -0800)]
Start to add request time profiling.
Allow a "profile" object to be created when a request is received, and
track the time at which various events occur when handling the request.
Michael Vrable [Thu, 18 Nov 2010 03:39:00 +0000 (19:39 -0800)]
Add a lookup_last method to the multi-store backend.
Michael Vrable [Wed, 17 Nov 2010 18:12:56 +0000 (10:12 -0800)]
Updates to the workload generator to perform a parameter sweep.
Michael Vrable [Fri, 12 Nov 2010 19:41:46 +0000 (11:41 -0800)]
Implement very basic grouped fetches of objects from the cloud.
Add an item prefetch call, which for now just collects together items that
have been prefetched and when one of them is actually fetched, a range
request is sent that will cover all the objects. Thus this doesn't improve
the time to fetch the first object (actually it will hurt), it does arrange
to have objects fetched in groups where possible which reduces the total
number of operations and the time to get subsequent objects.
Michael Vrable [Fri, 12 Nov 2010 03:12:47 +0000 (19:12 -0800)]
Add a prefetch method. At the moment it is a no-op.
Michael Vrable [Thu, 11 Nov 2010 23:34:42 +0000 (15:34 -0800)]
Move benchmark results elsewhere, out of the repository.
Michael Vrable [Tue, 9 Nov 2010 00:41:42 +0000 (16:41 -0800)]
Return failure code when an S3 put operation fails.
Previously, we were indicating success even when that wasn't true, leading
to errors later when we went to read data we hadn't written. Now, the
caller can retry the put operation if needed.
Michael Vrable [Thu, 4 Nov 2010 23:59:55 +0000 (16:59 -0700)]
Drop a verbose debugging message.
Michael Vrable [Thu, 4 Nov 2010 22:09:54 +0000 (15:09 -0700)]
Update disk cache usage tracking to handle sparse files.
Michael Vrable [Wed, 3 Nov 2010 23:38:03 +0000 (16:38 -0700)]
Bugfix in partial log fetching.
Do not bother mapping the file until the data is available, otherwise the
mapping may be deleted from under us while we are waiting.
Michael Vrable [Wed, 3 Nov 2010 21:35:07 +0000 (14:35 -0700)]
Implement fetching of cloud log items via range requests.
This still needs a bit more work for stability, checking for leaks, etc.,
but implements the basic functionality needed to selectively retrieve just
the needed byte ranges out of cloud log segments to download individual log
items.
Michael Vrable [Sun, 31 Oct 2010 21:53:05 +0000 (14:53 -0700)]
Convert rangeset implementation from a hashtable to GSequence.
This gives log(N) performance, but allows us to easily tell if a request
falls in the middle of an object and check for overlaps on inserts (though
that part isn't implemented yet).
Michael Vrable [Sat, 30 Oct 2010 00:40:14 +0000 (17:40 -0700)]
Implement a rangeset data type and use it to track items in log segments.
The range set can be used to track which byte ranges in a file correspond
to valid objects, and so can be used in the cloud log code to check that
1) objects are actually available if partial log fetching is implemented,
and 2) that objects being fetched have actually been authenticated (if
needed).
This isn't fully working yet. The range sets should be changed from a hash
table to a sequence, since some lookups in the cloud log code are into the
middle of an object (to skip the header and remap the data), and that needs
to be handled properly.
Michael Vrable [Tue, 26 Oct 2010 16:27:56 +0000 (09:27 -0700)]
Add support for byterange requests in the storage layer.
Michael Vrable [Tue, 26 Oct 2010 14:38:36 +0000 (07:38 -0700)]
Rename private structures to remove leading underscores.
This is a cosmetic change, but should make debugging and looking up
structure definitions easier.
Michael Vrable [Mon, 25 Oct 2010 05:43:24 +0000 (22:43 -0700)]
Merge git+ssh://c09-44.sysnet.ucsd.edu/scratch/bluesky
Michael Vrable [Mon, 25 Oct 2010 05:15:41 +0000 (22:15 -0700)]
Updates to the workload generator and some results from runs.
Michael Vrable [Thu, 21 Oct 2010 21:28:56 +0000 (14:28 -0700)]
Remove debugging messages when decrypting cloud log segments.
Michael Vrable [Thu, 21 Oct 2010 20:07:18 +0000 (13:07 -0700)]
Fix Amazon S3 store_lookup_last implementation.
Amazon will return directory listings in chunks. Be sure to repeat list
calls until we find the actual end of the directory listing.
Michael Vrable [Thu, 21 Oct 2010 05:34:45 +0000 (22:34 -0700)]
Fix a memory leak resulting from the in-memory inode map.
Actually release the reference to the old version of an inode when creating
a new one.
Michael Vrable [Tue, 19 Oct 2010 21:14:50 +0000 (14:14 -0700)]
Add a more aggressive mode of cleaning up disk space.
If enough space cannot be deleted in the disk cache, drop nearly all clean
data from memory so that most disk cache files are unreferenced, then make
another pass to delete log/cache files.
Michael Vrable [Tue, 19 Oct 2010 20:43:58 +0000 (13:43 -0700)]
Move disk cache cleanup code to cache.c from log.c
Michael Vrable [Tue, 19 Oct 2010 02:01:21 +0000 (19:01 -0700)]
A few more minor fixes cleaning up the cloud log state counting.
Michael Vrable [Tue, 19 Oct 2010 01:05:50 +0000 (18:05 -0700)]
Fix up cloud log state counting.
Counts of cloud log items in various states were not being kept up to date,
so that the counts were often wrong. Fix a few cases to make the counts
more accurate.
Michael Vrable [Tue, 19 Oct 2010 00:22:15 +0000 (17:22 -0700)]
Ignore lockmem executable.
Michael Vrable [Mon, 18 Oct 2010 21:48:57 +0000 (14:48 -0700)]
The signature on log items should not extend to the pointer locations.
This allows the cleaner to rewrite the pointers without invalidating the
signature.
Michael Vrable [Mon, 18 Oct 2010 21:48:47 +0000 (14:48 -0700)]
Add timeout on cleaner S3 retry.
Michael Vrable [Mon, 18 Oct 2010 21:41:44 +0000 (14:41 -0700)]
Add retries to the S3 backend in the cleaner.
Michael Vrable [Mon, 18 Oct 2010 20:03:15 +0000 (13:03 -0700)]
Small utility to use free memory in a system.
For benchmarking, to take memory away from the page cache.
Michael Vrable [Mon, 18 Oct 2010 04:40:44 +0000 (21:40 -0700)]
Support encrypted log items in the cleaner.
Michael Vrable [Sun, 17 Oct 2010 23:18:23 +0000 (16:18 -0700)]
Delete dead code.
Michael Vrable [Sun, 17 Oct 2010 23:17:40 +0000 (16:17 -0700)]
When decrypting a log item also clear out the IV field.
Not really needed, but this way the IV field being zero should be
synonymous with an unencrypted log item.
Michael Vrable [Sun, 17 Oct 2010 20:40:20 +0000 (13:40 -0700)]
Add per-item encryption/authentication to the cloud log storage.
We should generate encrypted data and decrypt it again on read, but we
don't yet enforce only reading data which passes the integrity check.
Michael Vrable [Fri, 15 Oct 2010 18:01:45 +0000 (11:01 -0700)]
Allow S3 bucket used for BlueSky storage to be specified.
Michael Vrable [Mon, 11 Oct 2010 18:35:17 +0000 (11:35 -0700)]
Work on a simple workload generator for benchmarking.
Michael Vrable [Fri, 8 Oct 2010 23:57:35 +0000 (16:57 -0700)]
Start adding in selective encryption of cloud log items.
Not fully hooked in, but some of the logic for encryption is written now.
Michael Vrable [Mon, 27 Sep 2010 23:30:13 +0000 (16:30 -0700)]
Another logging fix.
Michael Vrable [Mon, 27 Sep 2010 20:28:57 +0000 (13:28 -0700)]
Fix for journal committing.
Sometimes we could previously, under load, report that journal items were
committed when they were not. Try to track the uncommitted state more
carefully now.
Michael Vrable [Mon, 27 Sep 2010 17:57:59 +0000 (10:57 -0700)]
Implement handling of unstable data in WRITE/COMMIT nfs procedures.
Before all NFS operations were synchronous; now we support asynchronous
commits of file writes which might improve performance.
Michael Vrable [Mon, 27 Sep 2010 05:55:19 +0000 (22:55 -0700)]
Updated microbenchmarking script.
Michael Vrable [Wed, 22 Sep 2010 20:52:28 +0000 (13:52 -0700)]
Starting work on scripts to automate benchmarking.
Michael Vrable [Wed, 22 Sep 2010 18:47:17 +0000 (11:47 -0700)]
Improve cleaner performance.
When reading an object in, seek to and read just the needed bytes instead
of the entire log segment. Improves performance significantly.
Michael Vrable [Mon, 20 Sep 2010 22:09:12 +0000 (15:09 -0700)]
Remove obsolete file.
Michael Vrable [Mon, 20 Sep 2010 15:56:01 +0000 (08:56 -0700)]
Use a thread pool for inode fetches, and remove some debugging output.
Michael Vrable [Mon, 20 Sep 2010 15:55:03 +0000 (08:55 -0700)]
Remove an extraneous mutex unlock.
I'm surprised that this didn't cause trouble earlier; it seems that
unlocking an unlocked mutex raises no errors (but under heavy load, when
the mutex is locked by another thread then unlocking it can cause trouble).
Michael Vrable [Mon, 20 Sep 2010 03:07:27 +0000 (20:07 -0700)]
More fixes for memory management.
This should allow memory and cache space to be reclaimed by not keeping
items pinned in memory, finally. Still needs a bit more testing.
Michael Vrable [Sun, 19 Sep 2010 22:22:02 +0000 (15:22 -0700)]
Work on reducing memory pinned by the inode map.
Michael Vrable [Sun, 19 Sep 2010 18:31:15 +0000 (11:31 -0700)]
Add cleaner option to rewrite and compact all inodes (but not all data).
Michael Vrable [Sun, 19 Sep 2010 04:22:15 +0000 (21:22 -0700)]
Allow cloudlog items to be unreferenced in the background.
This is to avoid certain deadlocks, when we don't care if resources are
reclaimed immediately or not.
Michael Vrable [Sun, 19 Sep 2010 00:44:07 +0000 (17:44 -0700)]
Do not hold references to all inode data in inode map.
The inode map should not hold full refereces to all inode objects and all
corresponding data, since that will lock all such data in memory or the
disk cache. Do some initial work towards just holding weak references to
the data so that it can be expired from the cache.
Michael Vrable [Wed, 15 Sep 2010 20:29:42 +0000 (13:29 -0700)]
Restart journal sequence numbering properly.
Handle the case even where some old journal files have been deleted.
Michael Vrable [Tue, 14 Sep 2010 21:49:39 +0000 (14:49 -0700)]
Add in header fields for per-object encryption/authentication.
These aren't yet used.
Michael Vrable [Sat, 11 Sep 2010 00:11:05 +0000 (17:11 -0700)]
Add very basic caching to the cleaner S3 backend.
Michael Vrable [Fri, 10 Sep 2010 22:58:28 +0000 (15:58 -0700)]
Add S3 backend for the cleaner.
It does not yet cache files so performance is poor.
Michael Vrable [Fri, 10 Sep 2010 22:58:09 +0000 (15:58 -0700)]
Properly set the starting inode number for allocation after restarting.
Michael Vrable [Fri, 10 Sep 2010 20:53:00 +0000 (13:53 -0700)]
Fix for S3 list operation.
Michael Vrable [Fri, 10 Sep 2010 20:30:52 +0000 (13:30 -0700)]
Implement a list operation for the S3 storage backend.
Michael Vrable [Fri, 10 Sep 2010 00:59:03 +0000 (17:59 -0700)]
Drop encryption from the cloud storage backend.
Encryption should properly be provided at another layer, so in preparation
for that remove it from the storage layer.
Michael Vrable [Thu, 9 Sep 2010 20:30:39 +0000 (13:30 -0700)]
Allow cleaner to delete unused log segments.
It will not delete segments cleaned out in the current pass, just those
that were unreferenced at the start of the cleaning process.
Michael Vrable [Thu, 9 Sep 2010 19:09:52 +0000 (12:09 -0700)]
Improve segment cleaning.
Michael Vrable [Thu, 9 Sep 2010 06:44:49 +0000 (23:44 -0700)]
Improve cleaner: make sure new logs are written after existing ones.
Michael Vrable [Thu, 9 Sep 2010 05:07:17 +0000 (22:07 -0700)]
Extend cleaner with a simple policy for choosing segments to clean.
Michael Vrable [Thu, 9 Sep 2010 04:03:31 +0000 (21:03 -0700)]
Updates to the Python cleaner prototype.
This can now read in the old inode maps, rewrite inode data, and write out
an updated inode map/checkpoint.
Michael Vrable [Wed, 8 Sep 2010 20:24:42 +0000 (13:24 -0700)]
Begin work on a segment cleaner prototype.
Right now this can rebuild an inode map and compute segment utilization,
though it isn't very efficient.
Michael Vrable [Tue, 7 Sep 2010 21:10:50 +0000 (14:10 -0700)]
Add partial journal replay to filesystem recovery.
After loading a cloud checkpoint, replay just the last portion of the
journal that may not have been committed to the cloud.
Michael Vrable [Tue, 7 Sep 2010 20:28:42 +0000 (13:28 -0700)]
Include inode numbers in cloud log items.
Michael Vrable [Tue, 7 Sep 2010 20:26:10 +0000 (13:26 -0700)]
Finish up loading of checkpoints from cloud logs.
This is now working, at least with a very minimal test.
Michael Vrable [Tue, 7 Sep 2010 04:49:07 +0000 (21:49 -0700)]
In-progress work to implement inode map loading at server start.
This will be used to restore filesystem state from the cloud when the
program starts up again. It still needs more optimization, needs journal
replay to be run afterwards, and bugfixing.
Michael Vrable [Mon, 6 Sep 2010 05:27:34 +0000 (22:27 -0700)]
Improve object deserialization: properly parse object headers.
Michael Vrable [Thu, 2 Sep 2010 22:53:07 +0000 (15:53 -0700)]
Start at writing out inode maps to cloud storage.
Michael Vrable [Wed, 1 Sep 2010 18:15:29 +0000 (11:15 -0700)]
Fixes for journal replay, and drop the "superblock" cloud item.
That file in the cloud wasn't storing useful information any longer and
wasn't being used much; drop it. It's functionality will be replaced with
some form of commit log in the cloud journal.
Michael Vrable [Tue, 31 Aug 2010 22:00:09 +0000 (15:00 -0700)]
Fix some resource leaks in journal replay.
Michael Vrable [Tue, 31 Aug 2010 21:06:53 +0000 (14:06 -0700)]
Implement basic full log replay.
This still needs some checking over for bugs and minor fixes. It replays
the entire journal from the start to rebuild filesystem state. Still
needed: partial joural replay, starting from a checkpoint in the cloud.
Michael Vrable [Tue, 31 Aug 2010 20:02:38 +0000 (13:02 -0700)]
Update CRC-32 implementation.
Invert the result of the CRC computation at the end. This will catch extra
null bytes at the end of the buffer, but required updating the CRC
validation.
Michael Vrable [Fri, 27 Aug 2010 23:16:17 +0000 (16:16 -0700)]
Add in some support for journal replay.
This isn't all functional yet, but making it fully functional will require
updating the data fields that are written to the journal first...
Michael Vrable [Thu, 26 Aug 2010 00:28:26 +0000 (17:28 -0700)]
Start work on log replay for filesystem recovery.
Right now this implements scanning of one journal segment with consistency
checking to find items that were written out.
Also fix the checksum calculation on log entries so that they will validate
properly (we want to compute the checksum so that on validation, computing
the checksum of the entire object results in a value of zero).
Michael Vrable [Wed, 25 Aug 2010 20:24:34 +0000 (13:24 -0700)]
Add an inode map data structure to track the location of inodes in logs.
Michael Vrable [Tue, 24 Aug 2010 23:53:00 +0000 (16:53 -0700)]
Update logic for flushing data to cloud.
Do not force a commit of the most recent data, and instead just write out
whatever was last written to the journal.
This could be a win or a loss:
- We do not need to force a sync of all data to the journal when we
upload data to the cloud.
- But, we may end up writing out old data, which we'll then need to
overwrite a short time later.
Michael Vrable [Mon, 23 Aug 2010 17:21:37 +0000 (10:21 -0700)]
Make cache size run-time configurable.
Michael Vrable [Sun, 22 Aug 2010 05:42:35 +0000 (22:42 -0700)]
Implement new scheme for retaining needed journal segments.
Write full filesystem snapshots to the cloud, and keep track of the journal
position before the snapshot process starts. When it finishes, the journal
segments before that mark can be reclaimed (if needed).
This could be improved but should at least be safe.
Michael Vrable [Sun, 22 Aug 2010 04:48:50 +0000 (21:48 -0700)]
Fix a longstanding(?) memory-leak bug when truncating a file.
Michael Vrable [Fri, 20 Aug 2010 20:51:31 +0000 (13:51 -0700)]
Back out dirty reference tracking, as the design was flawed.
Objects can be written to the journal but not to the cloud--for example, if
a data block is written to the journal but overwritten before the file is
flushed to the cloud. This write-combining is good, but the old code for
tracking when a journal segment could be reclaimed couldn't handle this.
So, back out that dirty reference tracking code, in preparation for
replacing it with another approach.
Michael Vrable [Fri, 20 Aug 2010 00:16:53 +0000 (17:16 -0700)]
Make cloud storage more robust.
- Do not consider data committed until we get a reply from the cloud.
- Add retries on write and on read.
Michael Vrable [Thu, 19 Aug 2010 00:03:29 +0000 (17:03 -0700)]
Add a target size for the cache, and prune the cache when it gets larger.
Michael Vrable [Wed, 18 Aug 2010 20:45:40 +0000 (13:45 -0700)]
Track journal files which contain dirty data and which can be reclaimed.
Michael Vrable [Wed, 18 Aug 2010 01:46:58 +0000 (18:46 -0700)]
Implement a (dumb) cache garbage collector.
This is a proof of concept; it doesn't delete journal files and deletes
cache files nearly as soon as they are unused, so it needs better
algorithms for choosing when to delete files. But it does seem to work.
Michael Vrable [Tue, 17 Aug 2010 22:23:40 +0000 (15:23 -0700)]
Improve journal/cloud cache locking and add access time tracking.
Michael Vrable [Tue, 17 Aug 2010 18:02:56 +0000 (11:02 -0700)]
Debugging/refcount cleanups.