bluesky.git
14 years agoBetter track latency of operations issued to S3.
Michael Vrable [Mon, 22 Mar 2010 22:07:21 +0000 (15:07 -0700)]
Better track latency of operations issued to S3.

Track the time from when the operation is sent to S3 as well, not just from
when it was submitted locally, as there may be some local queueing
happening.

14 years agoAdd very simple write throttling when caches fill up.
Michael Vrable [Fri, 19 Mar 2010 22:59:52 +0000 (15:59 -0700)]
Add very simple write throttling when caches fill up.

We ought to actively flush data when needed, but this is enough for some
basic testing.

14 years agoMake dropping of inodes from cache more resistant to races.
Michael Vrable [Fri, 19 Mar 2010 22:59:00 +0000 (15:59 -0700)]
Make dropping of inodes from cache more resistant to races.

14 years agoQuick note on memory leak debugging under valgrind.
Michael Vrable [Wed, 17 Mar 2010 22:01:27 +0000 (15:01 -0700)]
Quick note on memory leak debugging under valgrind.

14 years agoInodes can now be dropped from the cache too, not just file blocks.
Michael Vrable [Wed, 17 Mar 2010 22:01:07 +0000 (15:01 -0700)]
Inodes can now be dropped from the cache too, not just file blocks.

14 years agoTrack memory usage statistics for cached data.
Michael Vrable [Wed, 17 Mar 2010 20:56:26 +0000 (13:56 -0700)]
Track memory usage statistics for cached data.

14 years agoProvide a simple configurable limit on the number of threads.
Michael Vrable [Tue, 16 Mar 2010 00:22:19 +0000 (17:22 -0700)]
Provide a simple configurable limit on the number of threads.

Primarily this is meant to make running under valgrind easier, since
valgrind can only handle so many threads.  By default leave the limit
unbounded (the old behavior).

14 years agoFix memory leak in bluesky_block_fetch.
Michael Vrable [Fri, 12 Mar 2010 00:56:26 +0000 (16:56 -0800)]
Fix memory leak in bluesky_block_fetch.

14 years agoReturn success code from NFSv3 commit operations.
Michael Vrable [Fri, 12 Mar 2010 00:22:02 +0000 (16:22 -0800)]
Return success code from NFSv3 commit operations.

14 years agoEliminate a warning about symlink serialization that was false.
Michael Vrable [Thu, 11 Mar 2010 22:24:12 +0000 (14:24 -0800)]
Eliminate a warning about symlink serialization that was false.

14 years agoRefine lock acquisition order for renames.
Michael Vrable [Wed, 10 Mar 2010 22:47:04 +0000 (14:47 -0800)]
Refine lock acquisition order for renames.

14 years agoAdd UDP transport support to the NFS server.
Michael Vrable [Wed, 10 Mar 2010 00:20:46 +0000 (16:20 -0800)]
Add UDP transport support to the NFS server.

SpecSFS seems to require basic UDP functionality.

14 years agospecSFS requires Java.
Michael Vrable [Wed, 10 Mar 2010 00:20:33 +0000 (16:20 -0800)]
specSFS requires Java.

14 years agoStart fetch of all blocks in a read at the start of the read operation.
Michael Vrable [Fri, 5 Mar 2010 22:25:29 +0000 (14:25 -0800)]
Start fetch of all blocks in a read at the start of the read operation.

14 years agoDo not signal barrier until all notifiers have run.
Michael Vrable [Fri, 5 Mar 2010 22:25:00 +0000 (14:25 -0800)]
Do not signal barrier until all notifiers have run.

14 years agoOnly treat an async storage operation as finished when notifiers finish.
Michael Vrable [Fri, 5 Mar 2010 00:16:52 +0000 (16:16 -0800)]
Only treat an async storage operation as finished when notifiers finish.

There's still a bit more work to be done with barriers, but this eliminates
some race conditions in some usages.

14 years agoPartial work on parallel data fetches from S3 for large reads.
Michael Vrable [Thu, 4 Mar 2010 23:20:35 +0000 (15:20 -0800)]
Partial work on parallel data fetches from S3 for large reads.

Currently causes crashes, but that might be related to a race condition
that needs fixing in the async storage layer..

14 years agoExplicitly mount NFS using version 3 in microbenchmarks.
Michael Vrable [Fri, 26 Feb 2010 01:19:26 +0000 (17:19 -0800)]
Explicitly mount NFS using version 3 in microbenchmarks.

We only support version 3, so ask for that when mounting.  This can prevent
problems in mounting in some cases, so we don't return the proper error
messages when a client asks for a version we don't support.

14 years agoUse s3cmd for cleaning up S3 storage accounts before benchmark runs.
Michael Vrable [Thu, 25 Feb 2010 21:03:11 +0000 (13:03 -0800)]
Use s3cmd for cleaning up S3 storage accounts before benchmark runs.

14 years agoImplement a synthetic benchmark for bulk reads of file data.
Michael Vrable [Wed, 24 Feb 2010 22:47:42 +0000 (14:47 -0800)]
Implement a synthetic benchmark for bulk reads of file data.

14 years agosynclient: Issue read requests for files after lookups return.
Michael Vrable [Tue, 23 Feb 2010 23:10:51 +0000 (15:10 -0800)]
synclient: Issue read requests for files after lookups return.

14 years agoFix a typo in SunRPC decoding.
Michael Vrable [Tue, 23 Feb 2010 21:01:25 +0000 (13:01 -0800)]
Fix a typo in SunRPC decoding.

The code accidentally used = instead of == in a conditional, leading to
decode errors in RPC packet framing.

14 years agoAllow larger reads and writes.
Michael Vrable [Tue, 23 Feb 2010 01:12:14 +0000 (17:12 -0800)]
Allow larger reads and writes.

Support reads up to 1 MB now, instead of just 32 kB.

14 years agoFix required package list (need git-core, not git).
Michael Vrable [Tue, 23 Feb 2010 01:11:56 +0000 (17:11 -0800)]
Fix required package list (need git-core, not git).

14 years agoDo not start the RPC services until the filesystem is ready.
Michael Vrable [Fri, 19 Feb 2010 00:28:11 +0000 (16:28 -0800)]
Do not start the RPC services until the filesystem is ready.

14 years agoFix for kvstore with default settings.
Michael Vrable [Fri, 19 Feb 2010 00:04:48 +0000 (16:04 -0800)]
Fix for kvstore with default settings.

14 years agoStore backends now consist of a type followed by extra information.
Michael Vrable [Thu, 18 Feb 2010 23:12:59 +0000 (15:12 -0800)]
Store backends now consist of a type followed by extra information.

For example: "kv:storagehost.com:9000".  The other backends don't use this
yet but should be made to do so.

14 years agoMake kvstore backend in BlueSky asynchronous.
Michael Vrable [Wed, 17 Feb 2010 22:53:23 +0000 (14:53 -0800)]
Make kvstore backend in BlueSky asynchronous.

Also, add synthetic delays to the kvstore server for latency testing.

14 years agoMore CMake tweaking.
Michael Vrable [Tue, 16 Feb 2010 23:29:08 +0000 (15:29 -0800)]
More CMake tweaking.

14 years agoBuild system cleanup.
Michael Vrable [Tue, 16 Feb 2010 22:58:25 +0000 (14:58 -0800)]
Build system cleanup.

14 years agoLink kvstore into main build system.
Michael Vrable [Tue, 16 Feb 2010 22:48:12 +0000 (14:48 -0800)]
Link kvstore into main build system.

14 years agoValues in the kvstore are raw byte arrays, not strings.
Michael Vrable [Tue, 16 Feb 2010 19:22:53 +0000 (11:22 -0800)]
Values in the kvstore are raw byte arrays, not strings.

14 years agoHook John's kvstore up to the BlueSky implementation.
Michael Vrable [Tue, 16 Feb 2010 19:17:45 +0000 (11:17 -0800)]
Hook John's kvstore up to the BlueSky implementation.

14 years ago.gitignore updates, and deleting old build system files.
Michael Vrable [Mon, 15 Feb 2010 23:53:28 +0000 (15:53 -0800)]
.gitignore updates, and deleting old build system files.

14 years agoReorganizing kvstore sources and switching build system to CMake.
Michael Vrable [Mon, 15 Feb 2010 23:50:09 +0000 (15:50 -0800)]
Reorganizing kvstore sources and switching build system to CMake.

14 years agoAdd John MucCullough's simple key/value storage server.
Michael Vrable [Sun, 14 Feb 2010 22:56:22 +0000 (14:56 -0800)]
Add John MucCullough's simple key/value storage server.

14 years agoDo not drop data from cache immediately after a write.
Michael Vrable [Fri, 5 Feb 2010 21:40:33 +0000 (13:40 -0800)]
Do not drop data from cache immediately after a write.

14 years agoAdd very rudimentary eviction data blocks from the cache.
Michael Vrable [Fri, 5 Feb 2010 21:23:06 +0000 (13:23 -0800)]
Add very rudimentary eviction data blocks from the cache.

14 years agoHook synthetic NFS client into test script.
Michael Vrable [Fri, 5 Feb 2010 01:14:12 +0000 (17:14 -0800)]
Hook synthetic NFS client into test script.

14 years agoFix for file storage backend.
Michael Vrable [Fri, 5 Feb 2010 00:34:50 +0000 (16:34 -0800)]
Fix for file storage backend.

14 years agoMake storage backend selectable via environment variable.
Michael Vrable [Fri, 5 Feb 2010 00:03:25 +0000 (16:03 -0800)]
Make storage backend selectable via environment variable.

14 years agoAllow NFS lookup requests to better proceed in parallel.
Michael Vrable [Thu, 4 Feb 2010 22:19:10 +0000 (14:19 -0800)]
Allow NFS lookup requests to better proceed in parallel.

After doing a directory lookup, drop the directory lock before fetching the
file inode.

14 years agoUpdate synthetic RPC client to record timing.
Michael Vrable [Thu, 4 Feb 2010 19:36:13 +0000 (11:36 -0800)]
Update synthetic RPC client to record timing.

For now, both it and readbench and effectively testing the time to look up
an inode, which should be a single fetch from S3.

14 years ago.gitignore fix.
Michael Vrable [Wed, 3 Feb 2010 23:17:34 +0000 (15:17 -0800)]
.gitignore fix.

14 years agoWork on a client for generating a synthetic stream of NFS operations.
Michael Vrable [Wed, 3 Feb 2010 22:01:19 +0000 (14:01 -0800)]
Work on a client for generating a synthetic stream of NFS operations.

This should provide another means of testing performance in addition to the
kernel NFS client.

14 years ago.gitignore updates.
Michael Vrable [Wed, 3 Feb 2010 00:01:43 +0000 (16:01 -0800)]
.gitignore updates.

14 years agoAdd microbenchmarks to build system.
Michael Vrable [Tue, 2 Feb 2010 22:23:18 +0000 (14:23 -0800)]
Add microbenchmarks to build system.

14 years agoAdd scripts for automatically testing reads at different settings.
Michael Vrable [Tue, 2 Feb 2010 21:04:52 +0000 (13:04 -0800)]
Add scripts for automatically testing reads at different settings.

14 years agoCreate a runtime option for sync/async frontend request handling.
Michael Vrable [Tue, 2 Feb 2010 20:29:20 +0000 (12:29 -0800)]
Create a runtime option for sync/async frontend request handling.

14 years agoSIGUSR1 causes nfsd to dump filesystem state now for debugging.
Michael Vrable [Mon, 1 Feb 2010 17:38:25 +0000 (09:38 -0800)]
SIGUSR1 causes nfsd to dump filesystem state now for debugging.

14 years agoWork on a parallel read benchmark.
Michael Vrable [Fri, 29 Jan 2010 22:20:41 +0000 (14:20 -0800)]
Work on a parallel read benchmark.

14 years agoLeak cleanup.
Michael Vrable [Fri, 29 Jan 2010 22:18:44 +0000 (14:18 -0800)]
Leak cleanup.

14 years agoFix a deadlock and a few memory leaks.
Michael Vrable [Thu, 28 Jan 2010 23:15:14 +0000 (15:15 -0800)]
Fix a deadlock and a few memory leaks.

14 years agoAdd inode prefetching support in NFS readdirplus.
Michael Vrable [Thu, 28 Jan 2010 20:53:16 +0000 (12:53 -0800)]
Add inode prefetching support in NFS readdirplus.

14 years agoFinish up asynchronous inode loading.
Michael Vrable [Thu, 28 Jan 2010 19:50:07 +0000 (11:50 -0800)]
Finish up asynchronous inode loading.

14 years agoLet storage tuning options be set via environment variables.
Michael Vrable [Thu, 28 Jan 2010 18:56:11 +0000 (10:56 -0800)]
Let storage tuning options be set via environment variables.

This should make benchmarking runs easier.

14 years agoSwitch to CTR instead of CBC mode for encryption.
Michael Vrable [Thu, 28 Jan 2010 01:06:04 +0000 (17:06 -0800)]
Switch to CTR instead of CBC mode for encryption.

This should handle arbitrarily messages better.  Later we ought to switch
to something like EAX, so consider this change temporary.

14 years agoSupport for sync operations on an inode.
Michael Vrable [Wed, 27 Jan 2010 22:13:43 +0000 (14:13 -0800)]
Support for sync operations on an inode.

14 years agoFlush file blocks to storage before the inode.
Michael Vrable [Wed, 27 Jan 2010 19:49:59 +0000 (11:49 -0800)]
Flush file blocks to storage before the inode.

Otherwise, some blocks might not have been assigned a name and so
seriaizing the inode will lose data.

14 years agoEnsure a reference to an async is held while it is locked.
Michael Vrable [Tue, 26 Jan 2010 22:58:06 +0000 (14:58 -0800)]
Ensure a reference to an async is held while it is locked.

Fixes a crash (assertion failure) with barriers where the barrier was
deallocated while it was still locked.

14 years agoClean up logging in storage layer.
Michael Vrable [Tue, 26 Jan 2010 22:57:52 +0000 (14:57 -0800)]
Clean up logging in storage layer.

14 years agoSome changes to support asynchronous fetching of inodes.
Michael Vrable [Tue, 26 Jan 2010 19:16:11 +0000 (11:16 -0800)]
Some changes to support asynchronous fetching of inodes.

Not 100% done yet, but almost there.

14 years agoImplement multi-threaded request processing and locking for NFS.
Michael Vrable [Tue, 26 Jan 2010 02:31:07 +0000 (18:31 -0800)]
Implement multi-threaded request processing and locking for NFS.

14 years agoInodes are clean immediately after being read from storage.
Michael Vrable [Mon, 25 Jan 2010 18:18:17 +0000 (10:18 -0800)]
Inodes are clean immediately after being read from storage.

14 years agoNFS cache writeback is now invoked periodically by a timer.
Michael Vrable [Mon, 25 Jan 2010 03:37:45 +0000 (19:37 -0800)]
NFS cache writeback is now invoked periodically by a timer.

Previously we didn't have timers set up and simply called the writeback
code after each operation (which could mean writeback wasn't invoked for
long stretches of time).

14 years agoBluesky core initializes a fresh filesystem, so no need to do so in nfsd.
Michael Vrable [Mon, 25 Jan 2010 03:22:38 +0000 (19:22 -0800)]
Bluesky core initializes a fresh filesystem, so no need to do so in nfsd.

14 years agoProperly decrement inode refcounts when finishing a request.
Michael Vrable [Mon, 25 Jan 2010 01:35:14 +0000 (17:35 -0800)]
Properly decrement inode refcounts when finishing a request.

Track each refcount increase made, and when the request is finally replied
to, drop those references.  In a few cases we manually drop references, but
this should be less error-prone if there are multiple possible return
paths.

14 years agoAdd a warning to catch possible refcounting bugs.
Michael Vrable [Fri, 22 Jan 2010 05:19:31 +0000 (21:19 -0800)]
Add a warning to catch possible refcounting bugs.

Not actually needed (was added to track down the corruption fixed in the
last changeset, which was unrelated), but probably worth keeping around.

14 years agoFix a nasty bug in the decryption code.
Michael Vrable [Fri, 22 Jan 2010 05:18:05 +0000 (21:18 -0800)]
Fix a nasty bug in the decryption code.

This was a typo but it led to various types of memory corruption; after
decrypting the returned string was marked as being larger than it should
have been--also meaning that the last bytes in the buffer were off in
unallocated space.

14 years agoTrack time to process each NFS RPC received.
Michael Vrable [Thu, 21 Jan 2010 00:11:34 +0000 (16:11 -0800)]
Track time to process each NFS RPC received.

14 years agoAdd support for dumping filesystem state for debugging purposes.
Michael Vrable [Wed, 20 Jan 2010 23:59:10 +0000 (15:59 -0800)]
Add support for dumping filesystem state for debugging purposes.

14 years agoRemove some debugging messages.
Michael Vrable [Thu, 14 Jan 2010 22:02:36 +0000 (14:02 -0800)]
Remove some debugging messages.

14 years agoEliminate last static buffer in async_rpc_send_reply.
Michael Vrable [Thu, 14 Jan 2010 00:12:30 +0000 (16:12 -0800)]
Eliminate last static buffer in async_rpc_send_reply.

14 years agoRemove most static variables in the NFS server functions.
Michael Vrable [Wed, 13 Jan 2010 19:53:01 +0000 (11:53 -0800)]
Remove most static variables in the NFS server functions.

14 years agoContinuing rework for async RPC.
Michael Vrable [Wed, 13 Jan 2010 19:35:23 +0000 (11:35 -0800)]
Continuing rework for async RPC.

Now NFS server functions are responsible for sending a reply to the client
instead of returning a result to the RPC dispatch routine.  This will
allows the server functions to handle memory cleanup (instead of requiring
static data) and allows returns to be decoupled in time from when the
request arrives.

There is still some static data to clean up, but this moves us much closer
to allowing multiple requests in progress at once.

14 years agoMore async RPC work.
Michael Vrable [Wed, 13 Jan 2010 19:24:31 +0000 (11:24 -0800)]
More async RPC work.

14 years agoMore cleanups to move RPC layer towards being asynchronous.
Michael Vrable [Wed, 13 Jan 2010 19:05:39 +0000 (11:05 -0800)]
More cleanups to move RPC layer towards being asynchronous.

14 years agoFix for a segfault when resizing files.
Michael Vrable [Wed, 13 Jan 2010 19:04:27 +0000 (11:04 -0800)]
Fix for a segfault when resizing files.

If we only have a reference to a file block but have not read it in, we
don't actually know the size.  We maybe ought to have a better fix for this
later.

14 years agoRPC header file cleanup.
Michael Vrable [Wed, 13 Jan 2010 04:23:54 +0000 (20:23 -0800)]
RPC header file cleanup.

Preparation for further reworking of the RPC system.

14 years agoRewrite NFS RPC dispatching.
Michael Vrable [Tue, 12 Jan 2010 21:44:02 +0000 (13:44 -0800)]
Rewrite NFS RPC dispatching.

The rpcgen-provided dispatching is inflexible, only allowing for a single
request to be handled at a time.  Rewrite it (add our own reading of RPC
packets off a TCP socket, calls to XDR decoding/encoding, etc.) so that we
can eventually support multi-threaded and asynchronous handling of
messages.

14 years agoQuick tool for benchmarking write latency.
Michael Vrable [Mon, 11 Jan 2010 22:50:10 +0000 (14:50 -0800)]
Quick tool for benchmarking write latency.

14 years agoAdd a new option to make caching writethrough instead of writeback.
Michael Vrable [Mon, 11 Jan 2010 22:29:00 +0000 (14:29 -0800)]
Add a new option to make caching writethrough instead of writeback.

14 years agoImplement fsstat.
Michael Vrable [Thu, 7 Jan 2010 00:29:16 +0000 (16:29 -0800)]
Implement fsstat.

We don't return values that are very useful since there is no real size of
the filesystem to report, but do implement the procedure.

14 years agoImprove NFS readdirplus to estimate response sizes.
Michael Vrable [Thu, 7 Jan 2010 00:05:25 +0000 (16:05 -0800)]
Improve NFS readdirplus to estimate response sizes.

It should now give a response closer to, but not larger than, the requested
size.

14 years agoWork on NFS rmdir operation.
Michael Vrable [Wed, 6 Jan 2010 19:03:15 +0000 (11:03 -0800)]
Work on NFS rmdir operation.

14 years agoMostly-working NFS rename support.
Michael Vrable [Wed, 6 Jan 2010 18:50:57 +0000 (10:50 -0800)]
Mostly-working NFS rename support.

14 years agoSilence some compiler warnings.
Michael Vrable [Wed, 6 Jan 2010 18:20:38 +0000 (10:20 -0800)]
Silence some compiler warnings.

14 years agoPreliminary NFS readdirplus support (not yet entirely to spec).
Michael Vrable [Tue, 29 Dec 2009 05:53:11 +0000 (21:53 -0800)]
Preliminary NFS readdirplus support (not yet entirely to spec).

14 years agoInclude post-opeation file attributes for the NFS write operation.
Michael Vrable [Tue, 29 Dec 2009 05:35:58 +0000 (21:35 -0800)]
Include post-opeation file attributes for the NFS write operation.

14 years agoCalculate size of symlinks.
Michael Vrable [Tue, 29 Dec 2009 04:56:17 +0000 (20:56 -0800)]
Calculate size of symlinks.

14 years agoFix up logic for handling short data blocks.
Michael Vrable [Mon, 28 Dec 2009 06:32:07 +0000 (22:32 -0800)]
Fix up logic for handling short data blocks.

This fixes a crash when there are writes that are not a multiple of the
block size.

14 years agoPreliminary symlink/readlink support.
Michael Vrable [Mon, 28 Dec 2009 04:49:53 +0000 (20:49 -0800)]
Preliminary symlink/readlink support.

Attributes on symlinks are not yet properly set.

14 years agoNFS link operation.
Michael Vrable [Mon, 28 Dec 2009 01:13:45 +0000 (17:13 -0800)]
NFS link operation.

14 years agoMinor cleanups.
Michael Vrable [Mon, 28 Dec 2009 00:23:33 +0000 (16:23 -0800)]
Minor cleanups.

14 years agoRe-enable file and memory storage backends.
Michael Vrable [Sat, 12 Dec 2009 23:41:08 +0000 (15:41 -0800)]
Re-enable file and memory storage backends.

14 years agoTrack when cached file data is written out.
Michael Vrable [Wed, 9 Dec 2009 19:34:11 +0000 (11:34 -0800)]
Track when cached file data is written out.

14 years agoMore work on writeback caching.
Michael Vrable [Mon, 7 Dec 2009 23:24:21 +0000 (15:24 -0800)]
More work on writeback caching.

14 years agoInitial work on proper writeback caching.
Michael Vrable [Sun, 6 Dec 2009 18:02:13 +0000 (10:02 -0800)]
Initial work on proper writeback caching.

14 years agoDrop some debugging messages.
Michael Vrable [Thu, 3 Dec 2009 21:41:15 +0000 (13:41 -0800)]
Drop some debugging messages.

14 years agoA few bugfixes to the storage barrier support.
Michael Vrable [Wed, 2 Dec 2009 23:21:10 +0000 (15:21 -0800)]
A few bugfixes to the storage barrier support.