bluesky.git
14 years agoEnsure a reference to an async is held while it is locked.
Michael Vrable [Tue, 26 Jan 2010 22:58:06 +0000 (14:58 -0800)]
Ensure a reference to an async is held while it is locked.

Fixes a crash (assertion failure) with barriers where the barrier was
deallocated while it was still locked.

14 years agoClean up logging in storage layer.
Michael Vrable [Tue, 26 Jan 2010 22:57:52 +0000 (14:57 -0800)]
Clean up logging in storage layer.

14 years agoSome changes to support asynchronous fetching of inodes.
Michael Vrable [Tue, 26 Jan 2010 19:16:11 +0000 (11:16 -0800)]
Some changes to support asynchronous fetching of inodes.

Not 100% done yet, but almost there.

14 years agoImplement multi-threaded request processing and locking for NFS.
Michael Vrable [Tue, 26 Jan 2010 02:31:07 +0000 (18:31 -0800)]
Implement multi-threaded request processing and locking for NFS.

14 years agoInodes are clean immediately after being read from storage.
Michael Vrable [Mon, 25 Jan 2010 18:18:17 +0000 (10:18 -0800)]
Inodes are clean immediately after being read from storage.

14 years agoNFS cache writeback is now invoked periodically by a timer.
Michael Vrable [Mon, 25 Jan 2010 03:37:45 +0000 (19:37 -0800)]
NFS cache writeback is now invoked periodically by a timer.

Previously we didn't have timers set up and simply called the writeback
code after each operation (which could mean writeback wasn't invoked for
long stretches of time).

14 years agoBluesky core initializes a fresh filesystem, so no need to do so in nfsd.
Michael Vrable [Mon, 25 Jan 2010 03:22:38 +0000 (19:22 -0800)]
Bluesky core initializes a fresh filesystem, so no need to do so in nfsd.

14 years agoProperly decrement inode refcounts when finishing a request.
Michael Vrable [Mon, 25 Jan 2010 01:35:14 +0000 (17:35 -0800)]
Properly decrement inode refcounts when finishing a request.

Track each refcount increase made, and when the request is finally replied
to, drop those references.  In a few cases we manually drop references, but
this should be less error-prone if there are multiple possible return
paths.

14 years agoAdd a warning to catch possible refcounting bugs.
Michael Vrable [Fri, 22 Jan 2010 05:19:31 +0000 (21:19 -0800)]
Add a warning to catch possible refcounting bugs.

Not actually needed (was added to track down the corruption fixed in the
last changeset, which was unrelated), but probably worth keeping around.

14 years agoFix a nasty bug in the decryption code.
Michael Vrable [Fri, 22 Jan 2010 05:18:05 +0000 (21:18 -0800)]
Fix a nasty bug in the decryption code.

This was a typo but it led to various types of memory corruption; after
decrypting the returned string was marked as being larger than it should
have been--also meaning that the last bytes in the buffer were off in
unallocated space.

14 years agoTrack time to process each NFS RPC received.
Michael Vrable [Thu, 21 Jan 2010 00:11:34 +0000 (16:11 -0800)]
Track time to process each NFS RPC received.

14 years agoAdd support for dumping filesystem state for debugging purposes.
Michael Vrable [Wed, 20 Jan 2010 23:59:10 +0000 (15:59 -0800)]
Add support for dumping filesystem state for debugging purposes.

14 years agoRemove some debugging messages.
Michael Vrable [Thu, 14 Jan 2010 22:02:36 +0000 (14:02 -0800)]
Remove some debugging messages.

14 years agoEliminate last static buffer in async_rpc_send_reply.
Michael Vrable [Thu, 14 Jan 2010 00:12:30 +0000 (16:12 -0800)]
Eliminate last static buffer in async_rpc_send_reply.

14 years agoRemove most static variables in the NFS server functions.
Michael Vrable [Wed, 13 Jan 2010 19:53:01 +0000 (11:53 -0800)]
Remove most static variables in the NFS server functions.

14 years agoContinuing rework for async RPC.
Michael Vrable [Wed, 13 Jan 2010 19:35:23 +0000 (11:35 -0800)]
Continuing rework for async RPC.

Now NFS server functions are responsible for sending a reply to the client
instead of returning a result to the RPC dispatch routine.  This will
allows the server functions to handle memory cleanup (instead of requiring
static data) and allows returns to be decoupled in time from when the
request arrives.

There is still some static data to clean up, but this moves us much closer
to allowing multiple requests in progress at once.

14 years agoMore async RPC work.
Michael Vrable [Wed, 13 Jan 2010 19:24:31 +0000 (11:24 -0800)]
More async RPC work.

14 years agoMore cleanups to move RPC layer towards being asynchronous.
Michael Vrable [Wed, 13 Jan 2010 19:05:39 +0000 (11:05 -0800)]
More cleanups to move RPC layer towards being asynchronous.

14 years agoFix for a segfault when resizing files.
Michael Vrable [Wed, 13 Jan 2010 19:04:27 +0000 (11:04 -0800)]
Fix for a segfault when resizing files.

If we only have a reference to a file block but have not read it in, we
don't actually know the size.  We maybe ought to have a better fix for this
later.

14 years agoRPC header file cleanup.
Michael Vrable [Wed, 13 Jan 2010 04:23:54 +0000 (20:23 -0800)]
RPC header file cleanup.

Preparation for further reworking of the RPC system.

14 years agoRewrite NFS RPC dispatching.
Michael Vrable [Tue, 12 Jan 2010 21:44:02 +0000 (13:44 -0800)]
Rewrite NFS RPC dispatching.

The rpcgen-provided dispatching is inflexible, only allowing for a single
request to be handled at a time.  Rewrite it (add our own reading of RPC
packets off a TCP socket, calls to XDR decoding/encoding, etc.) so that we
can eventually support multi-threaded and asynchronous handling of
messages.

14 years agoQuick tool for benchmarking write latency.
Michael Vrable [Mon, 11 Jan 2010 22:50:10 +0000 (14:50 -0800)]
Quick tool for benchmarking write latency.

14 years agoAdd a new option to make caching writethrough instead of writeback.
Michael Vrable [Mon, 11 Jan 2010 22:29:00 +0000 (14:29 -0800)]
Add a new option to make caching writethrough instead of writeback.

14 years agoImplement fsstat.
Michael Vrable [Thu, 7 Jan 2010 00:29:16 +0000 (16:29 -0800)]
Implement fsstat.

We don't return values that are very useful since there is no real size of
the filesystem to report, but do implement the procedure.

14 years agoImprove NFS readdirplus to estimate response sizes.
Michael Vrable [Thu, 7 Jan 2010 00:05:25 +0000 (16:05 -0800)]
Improve NFS readdirplus to estimate response sizes.

It should now give a response closer to, but not larger than, the requested
size.

14 years agoWork on NFS rmdir operation.
Michael Vrable [Wed, 6 Jan 2010 19:03:15 +0000 (11:03 -0800)]
Work on NFS rmdir operation.

14 years agoMostly-working NFS rename support.
Michael Vrable [Wed, 6 Jan 2010 18:50:57 +0000 (10:50 -0800)]
Mostly-working NFS rename support.

14 years agoSilence some compiler warnings.
Michael Vrable [Wed, 6 Jan 2010 18:20:38 +0000 (10:20 -0800)]
Silence some compiler warnings.

14 years agoPreliminary NFS readdirplus support (not yet entirely to spec).
Michael Vrable [Tue, 29 Dec 2009 05:53:11 +0000 (21:53 -0800)]
Preliminary NFS readdirplus support (not yet entirely to spec).

14 years agoInclude post-opeation file attributes for the NFS write operation.
Michael Vrable [Tue, 29 Dec 2009 05:35:58 +0000 (21:35 -0800)]
Include post-opeation file attributes for the NFS write operation.

14 years agoCalculate size of symlinks.
Michael Vrable [Tue, 29 Dec 2009 04:56:17 +0000 (20:56 -0800)]
Calculate size of symlinks.

14 years agoFix up logic for handling short data blocks.
Michael Vrable [Mon, 28 Dec 2009 06:32:07 +0000 (22:32 -0800)]
Fix up logic for handling short data blocks.

This fixes a crash when there are writes that are not a multiple of the
block size.

14 years agoPreliminary symlink/readlink support.
Michael Vrable [Mon, 28 Dec 2009 04:49:53 +0000 (20:49 -0800)]
Preliminary symlink/readlink support.

Attributes on symlinks are not yet properly set.

14 years agoNFS link operation.
Michael Vrable [Mon, 28 Dec 2009 01:13:45 +0000 (17:13 -0800)]
NFS link operation.

14 years agoMinor cleanups.
Michael Vrable [Mon, 28 Dec 2009 00:23:33 +0000 (16:23 -0800)]
Minor cleanups.

14 years agoRe-enable file and memory storage backends.
Michael Vrable [Sat, 12 Dec 2009 23:41:08 +0000 (15:41 -0800)]
Re-enable file and memory storage backends.

14 years agoTrack when cached file data is written out.
Michael Vrable [Wed, 9 Dec 2009 19:34:11 +0000 (11:34 -0800)]
Track when cached file data is written out.

14 years agoMore work on writeback caching.
Michael Vrable [Mon, 7 Dec 2009 23:24:21 +0000 (15:24 -0800)]
More work on writeback caching.

14 years agoInitial work on proper writeback caching.
Michael Vrable [Sun, 6 Dec 2009 18:02:13 +0000 (10:02 -0800)]
Initial work on proper writeback caching.

14 years agoDrop some debugging messages.
Michael Vrable [Thu, 3 Dec 2009 21:41:15 +0000 (13:41 -0800)]
Drop some debugging messages.

14 years agoA few bugfixes to the storage barrier support.
Michael Vrable [Wed, 2 Dec 2009 23:21:10 +0000 (15:21 -0800)]
A few bugfixes to the storage barrier support.

14 years agoWork on a storage barrier operation.
Michael Vrable [Tue, 1 Dec 2009 21:39:42 +0000 (13:39 -0800)]
Work on a storage barrier operation.

The barrier completes when all of a set of operations has completed.

14 years agoSome initial work to support delayed flushing of data.
Michael Vrable [Sun, 29 Nov 2009 19:47:51 +0000 (11:47 -0800)]
Some initial work to support delayed flushing of data.

14 years agoMore work on synchronous/asynchronous operations.
Michael Vrable [Tue, 24 Nov 2009 21:18:50 +0000 (13:18 -0800)]
More work on synchronous/asynchronous operations.

Allow operations to be made synchronous for benchmarking, and add in a
storage-wide sync() call.

14 years agoMinor cleanup.
Michael Vrable [Mon, 23 Nov 2009 03:42:47 +0000 (19:42 -0800)]
Minor cleanup.

14 years agoAdd reference counting to asynchronous storage operations.
Michael Vrable [Sun, 22 Nov 2009 21:09:54 +0000 (13:09 -0800)]
Add reference counting to asynchronous storage operations.

14 years agoRework the storage interface so that operations are asynchronous.
Michael Vrable [Fri, 20 Nov 2009 04:33:55 +0000 (20:33 -0800)]
Rework the storage interface so that operations are asynchronous.

14 years agoS3Store cleanup.
Michael Vrable [Wed, 18 Nov 2009 19:53:01 +0000 (11:53 -0800)]
S3Store cleanup.

Just moving code around; no functional changes.

14 years agoBetter checks for non-existent inodes.
Michael Vrable [Wed, 18 Nov 2009 06:33:55 +0000 (22:33 -0800)]
Better checks for non-existent inodes.

14 years agoClean up code using bluesky_string_from_gstring.
Michael Vrable [Wed, 18 Nov 2009 03:19:13 +0000 (19:19 -0800)]
Clean up code using bluesky_string_from_gstring.

14 years agoFix the S3Store get operation to handle arbitrarily-sized objects.
Michael Vrable [Wed, 18 Nov 2009 01:16:26 +0000 (17:16 -0800)]
Fix the S3Store get operation to handle arbitrarily-sized objects.

14 years agoAllow storse to S3 to execute in the background, in parallel.
Michael Vrable [Wed, 18 Nov 2009 00:28:20 +0000 (16:28 -0800)]
Allow storse to S3 to execute in the background, in parallel.

14 years agoNew BlueSky interface for implementing readdir.
Michael Vrable [Fri, 13 Nov 2009 23:12:45 +0000 (15:12 -0800)]
New BlueSky interface for implementing readdir.

14 years agoExtra logging, begin defining BlueSky-specific error codes.
Michael Vrable [Thu, 12 Nov 2009 22:43:21 +0000 (14:43 -0800)]
Extra logging, begin defining BlueSky-specific error codes.

14 years agoDo not pad the final block of a file with zeroes.
Michael Vrable [Thu, 12 Nov 2009 02:33:48 +0000 (18:33 -0800)]
Do not pad the final block of a file with zeroes.

To accomplish this, a resize operation was added for reference-counted
strings.

15 years agoMove encryption into S3 backend.
Michael Vrable [Tue, 3 Nov 2009 00:11:34 +0000 (16:11 -0800)]
Move encryption into S3 backend.

15 years agoInclude file reorganization.
Michael Vrable [Sun, 1 Nov 2009 18:39:26 +0000 (10:39 -0800)]
Include file reorganization.

15 years agoSplit BlueSky include files into public and private parts.
Michael Vrable [Sat, 31 Oct 2009 17:56:08 +0000 (10:56 -0700)]
Split BlueSky include files into public and private parts.

Actually sorting out what goes where still needs to be done.

15 years agoStart work on a rename operation. Not yet complete.
Michael Vrable [Fri, 30 Oct 2009 22:48:14 +0000 (15:48 -0700)]
Start work on a rename operation.  Not yet complete.

15 years agoRemove some debugging log messages.
Michael Vrable [Fri, 30 Oct 2009 04:09:03 +0000 (21:09 -0700)]
Remove some debugging log messages.

15 years agoAdd filesystem superblock serialization.
Michael Vrable [Thu, 29 Oct 2009 22:41:14 +0000 (15:41 -0700)]
Add filesystem superblock serialization.

Also fix deserialization of case-folded directory entries.  Together, these
chagnes should allow a filesystem state to be reloaded properly.

15 years agoAdd high-resolution timekeeping functions (primarily for benchmarking).
Michael Vrable [Tue, 27 Oct 2009 19:29:28 +0000 (12:29 -0700)]
Add high-resolution timekeeping functions (primarily for benchmarking).

15 years agoAdd case-insensitive lookup support to BlueSky core.
Michael Vrable [Thu, 8 Oct 2009 22:04:28 +0000 (15:04 -0700)]
Add case-insensitive lookup support to BlueSky core.

15 years agoMake printf format specifiers 32/64-bit clean.
Michael Vrable [Wed, 23 Sep 2009 01:38:30 +0000 (18:38 -0700)]
Make printf format specifiers 32/64-bit clean.

15 years agoCache coherence thoughts.
Michael Vrable [Wed, 23 Sep 2009 00:29:32 +0000 (17:29 -0700)]
Cache coherence thoughts.

15 years agoMiscellaneous fixes.
Michael Vrable [Wed, 23 Sep 2009 00:29:09 +0000 (17:29 -0700)]
Miscellaneous fixes.

15 years agoMinor CMake fix (delete redundant line).
Michael Vrable [Wed, 16 Sep 2009 20:38:49 +0000 (13:38 -0700)]
Minor CMake fix (delete redundant line).

15 years agoUse and link to included libs3.
Michael Vrable [Wed, 16 Sep 2009 18:52:07 +0000 (11:52 -0700)]
Use and link to included libs3.

15 years agoInclude libs3 sources in the BlueSky tree.
Michael Vrable [Wed, 16 Sep 2009 18:30:44 +0000 (11:30 -0700)]
Include libs3 sources in the BlueSky tree.

This way we won't need to install it separately.  Not yet hooked into the
build system.

15 years agoAdd a little support for data writeback (not yet hooked up).
Michael Vrable [Wed, 16 Sep 2009 18:28:56 +0000 (11:28 -0700)]
Add a little support for data writeback (not yet hooked up).

15 years agoCommit unlinks to stable storage.
Michael Vrable [Mon, 14 Sep 2009 18:19:41 +0000 (11:19 -0700)]
Commit unlinks to stable storage.

15 years agoNFS file unlink operation.
Michael Vrable [Mon, 14 Sep 2009 17:44:10 +0000 (10:44 -0700)]
NFS file unlink operation.

15 years agoUpdate filesystem locking rules (in progress).
Michael Vrable [Mon, 14 Sep 2009 17:23:25 +0000 (10:23 -0700)]
Update filesystem locking rules (in progress).

15 years agoNFSv3 mkdir.
Michael Vrable [Fri, 11 Sep 2009 21:36:50 +0000 (14:36 -0700)]
NFSv3 mkdir.

15 years agoEnsure file size is properly serialized/deserialized.
Michael Vrable [Fri, 11 Sep 2009 21:02:53 +0000 (14:02 -0700)]
Ensure file size is properly serialized/deserialized.

15 years agoDirectory deserialization and loading of inodes from stable store.
Michael Vrable [Wed, 9 Sep 2009 23:58:28 +0000 (16:58 -0700)]
Directory deserialization and loading of inodes from stable store.

15 years agoProof-of-concept flushing of inodes to disk.
Michael Vrable [Fri, 4 Sep 2009 22:51:02 +0000 (15:51 -0700)]
Proof-of-concept flushing of inodes to disk.

15 years agoAllow crypto to be disabled (for local testing).
Michael Vrable [Fri, 4 Sep 2009 22:34:59 +0000 (15:34 -0700)]
Allow crypto to be disabled (for local testing).

15 years agoDrop unneeded HMAC implementation.
Michael Vrable [Fri, 4 Sep 2009 22:34:45 +0000 (15:34 -0700)]
Drop unneeded HMAC implementation.

15 years agoAdd pluggable support for multiple storage backends.
Michael Vrable [Fri, 4 Sep 2009 22:10:22 +0000 (15:10 -0700)]
Add pluggable support for multiple storage backends.

15 years agoSplit handling of file data out of inode.c.
Michael Vrable [Fri, 4 Sep 2009 04:47:29 +0000 (21:47 -0700)]
Split handling of file data out of inode.c.

15 years agoDirectory serialization (but not deserialization).
Michael Vrable [Thu, 3 Sep 2009 23:59:50 +0000 (16:59 -0700)]
Directory serialization (but not deserialization).

15 years agoWork to add inode serialization/deserialization routines.
Michael Vrable [Thu, 3 Sep 2009 23:49:13 +0000 (16:49 -0700)]
Work to add inode serialization/deserialization routines.

15 years agoUse GPLv3 license.
Michael Vrable [Wed, 2 Sep 2009 20:19:21 +0000 (13:19 -0700)]
Use GPLv3 license.

libs3 is GPLv3 licensed, as is Samba (if we use it for CIFS support).  For
compatibility, apply GPLv3 to our code as well.

15 years agoImplement data decryption as well.
Michael Vrable [Wed, 2 Sep 2009 04:23:26 +0000 (21:23 -0700)]
Implement data decryption as well.

15 years agoEncrypt data blocks being stored to S3.
Michael Vrable [Wed, 2 Sep 2009 03:35:55 +0000 (20:35 -0700)]
Encrypt data blocks being stored to S3.

15 years agoNFSv3 SETATTR operation.
Michael Vrable [Tue, 1 Sep 2009 21:04:33 +0000 (14:04 -0700)]
NFSv3 SETATTR operation.

15 years agoCMake build tweaks.
Michael Vrable [Tue, 1 Sep 2009 20:25:30 +0000 (13:25 -0700)]
CMake build tweaks.

15 years ago.gitignore updates for CMake.
Michael Vrable [Tue, 1 Sep 2009 18:28:12 +0000 (11:28 -0700)]
.gitignore updates for CMake.

15 years agoAdd missing file?
Michael Vrable [Tue, 1 Sep 2009 18:25:45 +0000 (11:25 -0700)]
Add missing file?

15 years agoCMake reorganization.
Michael Vrable [Tue, 1 Sep 2009 18:24:50 +0000 (11:24 -0700)]
CMake reorganization.

15 years agoAttempt at building with CMake.
Michael Vrable [Tue, 1 Sep 2009 05:25:45 +0000 (22:25 -0700)]
Attempt at building with CMake.

15 years ago.gitignore update: ignore binaries
Michael Vrable [Tue, 1 Sep 2009 03:14:27 +0000 (20:14 -0700)]
.gitignore update: ignore binaries

15 years agoSwitch to refcounted strings for storing cached file blocks.
Michael Vrable [Tue, 1 Sep 2009 03:13:52 +0000 (20:13 -0700)]
Switch to refcounted strings for storing cached file blocks.

15 years agoAdd fetching of blocks from S3.
Michael Vrable [Mon, 31 Aug 2009 19:49:26 +0000 (12:49 -0700)]
Add fetching of blocks from S3.

15 years agoImplement flushing of file blocks to Amazon S3.
Michael Vrable [Mon, 31 Aug 2009 19:09:14 +0000 (12:09 -0700)]
Implement flushing of file blocks to Amazon S3.

Right now many details are hardcoded.  Data can't be read back yet.  But it
implements some of the needed functionality.

15 years agoInclude libs3.h header file in our source.
Michael Vrable [Mon, 31 Aug 2009 18:09:21 +0000 (11:09 -0700)]
Include libs3.h header file in our source.

Later, we'll want to properly find it in an installed location.

15 years agoLink against libs3 for access to Amazon S3.
Michael Vrable [Mon, 31 Aug 2009 18:08:25 +0000 (11:08 -0700)]
Link against libs3 for access to Amazon S3.

libs3 is licensed as GPLv3.

15 years agoStart on a basic in-memory key/value store.
Michael Vrable [Mon, 31 Aug 2009 17:56:18 +0000 (10:56 -0700)]
Start on a basic in-memory key/value store.

15 years agoFile read and write operations.
Michael Vrable [Fri, 28 Aug 2009 04:30:23 +0000 (21:30 -0700)]
File read and write operations.