bluesky.git
14 years agoAnother update to the Azure test code.
Michael Vrable [Mon, 3 May 2010 04:51:22 +0000 (21:51 -0700)]
Another update to the Azure test code.

14 years agoStart work on a test program to communicate with Windows Azure.
Michael Vrable [Mon, 3 May 2010 02:10:02 +0000 (19:10 -0700)]
Start work on a test program to communicate with Windows Azure.

This could provide another backend for BlueSky, and it might be nice to at
least make a few benchmark measurements of Azure to complement the S3 ones.

14 years agoSimple tool for analyzing a dump of a single S3 connection.
Michael Vrable [Thu, 29 Apr 2010 23:07:21 +0000 (16:07 -0700)]
Simple tool for analyzing a dump of a single S3 connection.

Will try to extract information about any delays in the transfers.

14 years agoMore playing with parsing of packet traces.
Michael Vrable [Wed, 28 Apr 2010 22:06:50 +0000 (15:06 -0700)]
More playing with parsing of packet traces.

Extract window size values (and handle TCP window scaling).

14 years agoMore updates to S3 pipelining test.
Michael Vrable [Wed, 28 Apr 2010 21:30:14 +0000 (14:30 -0700)]
More updates to S3 pipelining test.

It seems there is a bug in S3 with many pipelined requests that we are
triggering in testing:
http://developer.amazonwebservices.com/connect/thread.jspa?messageID=39907

14 years agoCreate a simple Python script for sending pipelined GET requests to S3.
Michael Vrable [Wed, 28 Apr 2010 18:04:12 +0000 (11:04 -0700)]
Create a simple Python script for sending pipelined GET requests to S3.

S3 doesn't seem to respond properly to these yet.

14 years agoPrint flow identification in TCP parsing output.
Michael Vrable [Tue, 27 Apr 2010 21:13:45 +0000 (14:13 -0700)]
Print flow identification in TCP parsing output.

Also fix a division-by-zero bug in computing bandwidth if no data is
transferred.

14 years agoSome initial results of running the NFS replay code.
Michael Vrable [Mon, 26 Apr 2010 22:36:45 +0000 (15:36 -0700)]
Some initial results of running the NFS replay code.

14 years agoFix a memory leak in the NFS-over-UDP code.
Michael Vrable [Mon, 26 Apr 2010 05:38:27 +0000 (22:38 -0700)]
Fix a memory leak in the NFS-over-UDP code.

The TCP code still has a per-connection leak, but at least this should help
with the UDP per-request leak.

14 years agoEnable real-time trace replay instead of maximum-speed replay mode.
Michael Vrable [Sun, 25 Apr 2010 00:34:09 +0000 (17:34 -0700)]
Enable real-time trace replay instead of maximum-speed replay mode.

14 years agoAllow longer server names; fixes (temporarily) a buffer overflow.
Michael Vrable [Sat, 24 Apr 2010 02:46:13 +0000 (19:46 -0700)]
Allow longer server names; fixes (temporarily) a buffer overflow.

14 years agoCode fixes for TBBT.
Michael Vrable [Fri, 23 Apr 2010 20:30:25 +0000 (13:30 -0700)]
Code fixes for TBBT.

14 years agoAdd two missing files from the TBBT import.
Michael Vrable [Fri, 23 Apr 2010 19:50:15 +0000 (12:50 -0700)]
Add two missing files from the TBBT import.

14 years agoImport TBBT (NFS trace replay).
Michael Vrable [Fri, 23 Apr 2010 19:48:14 +0000 (12:48 -0700)]
Import TBBT (NFS trace replay).

Code downloaded from http://www.ecsl.cs.sunysb.edu/TBBT/.

14 years agoMerge git+ssh://root@c09-44.sysnet.ucsd.edu/scratch/bluesky
Michael Vrable [Fri, 23 Apr 2010 19:43:57 +0000 (12:43 -0700)]
Merge git+ssh://root@c09-44.sysnet.ucsd.edu/scratch/bluesky

14 years agoAdd pcap dump parser for extracting S3 performance measurements.
Michael Vrable [Wed, 21 Apr 2010 23:13:58 +0000 (16:13 -0700)]
Add pcap dump parser for extracting S3 performance measurements.

14 years agoMore measurement results of S3 latency.
Michael Vrable [Wed, 21 Apr 2010 23:13:39 +0000 (16:13 -0700)]
More measurement results of S3 latency.

14 years agoAdd new results.
Michael Vrable [Tue, 20 Apr 2010 03:10:02 +0000 (20:10 -0700)]
Add new results.

14 years agoCommit some benchmark results of standard Samba.
Michael Vrable [Tue, 13 Apr 2010 18:10:49 +0000 (11:10 -0700)]
Commit some benchmark results of standard Samba.

14 years agoUpdate needed dependencies for building Samba.
Michael Vrable [Fri, 9 Apr 2010 18:40:50 +0000 (11:40 -0700)]
Update needed dependencies for building Samba.

14 years agoCleanups while workign on Samba code.
Michael Vrable [Wed, 7 Apr 2010 22:36:58 +0000 (15:36 -0700)]
Cleanups while workign on Samba code.

14 years agoAdd some measurements of speed reading data from S3.
Michael Vrable [Tue, 6 Apr 2010 23:41:22 +0000 (16:41 -0700)]
Add some measurements of speed reading data from S3.

14 years agoInitial work to use Berkeley DB for locally logging FS changes.
Michael Vrable [Tue, 6 Apr 2010 03:59:41 +0000 (20:59 -0700)]
Initial work to use Berkeley DB for locally logging FS changes.

Not yet hooked up to anything else, but commit this now so the file doesn't
get lost.

14 years agoPull in updated tool for measuring S3 read performance.
Michael Vrable [Tue, 6 Apr 2010 00:08:53 +0000 (17:08 -0700)]
Pull in updated tool for measuring S3 read performance.

14 years agoImprove statistics gathered.
Michael Vrable [Fri, 2 Apr 2010 20:59:41 +0000 (13:59 -0700)]
Improve statistics gathered.

14 years agoAdd general statistics-gathering infrastructure.
Michael Vrable [Fri, 2 Apr 2010 19:20:21 +0000 (12:20 -0700)]
Add general statistics-gathering infrastructure.

14 years agoBenchmark results to S3.
Michael Vrable [Fri, 2 Apr 2010 05:00:45 +0000 (22:00 -0700)]
Benchmark results to S3.

14 years agoAdd some more results.
Michael Vrable [Thu, 1 Apr 2010 18:56:36 +0000 (11:56 -0700)]
Add some more results.

14 years agoFix filesystem locking when creating a new inode.
Michael Vrable [Thu, 1 Apr 2010 01:24:49 +0000 (18:24 -0700)]
Fix filesystem locking when creating a new inode.

14 years agoAdd warnings for failed storage operations.
Michael Vrable [Thu, 1 Apr 2010 00:30:17 +0000 (17:30 -0700)]
Add warnings for failed storage operations.

14 years agoDisable most debugging print messages; should help with performance.
Michael Vrable [Wed, 31 Mar 2010 21:53:54 +0000 (14:53 -0700)]
Disable most debugging print messages; should help with performance.

14 years agoTry to clean up the locking for LRU lists.
Michael Vrable [Wed, 31 Mar 2010 21:01:59 +0000 (14:01 -0700)]
Try to clean up the locking for LRU lists.

14 years agoA few minor fixes.
Michael Vrable [Wed, 31 Mar 2010 17:48:22 +0000 (10:48 -0700)]
A few minor fixes.

14 years agoCopy in a few more benchmark results.
Michael Vrable [Tue, 30 Mar 2010 23:09:24 +0000 (16:09 -0700)]
Copy in a few more benchmark results.

14 years agoStore benchmark results for native Linux NFS server.
Michael Vrable [Tue, 30 Mar 2010 18:14:34 +0000 (11:14 -0700)]
Store benchmark results for native Linux NFS server.

14 years agoTrack number of unanswered RPC requests, for debugging.
Michael Vrable [Tue, 30 Mar 2010 03:56:00 +0000 (20:56 -0700)]
Track number of unanswered RPC requests, for debugging.

14 years agoDo not fill in a non-zero inode change_count until the inode is loaded.
Michael Vrable [Tue, 30 Mar 2010 03:36:03 +0000 (20:36 -0700)]
Do not fill in a non-zero inode change_count until the inode is loaded.

14 years agoPossible null pointer dereference fix.
Michael Vrable [Tue, 30 Mar 2010 03:30:09 +0000 (20:30 -0700)]
Possible null pointer dereference fix.

14 years agoRework cache flushing logic--this version should work much better.
Michael Vrable [Mon, 29 Mar 2010 19:48:03 +0000 (12:48 -0700)]
Rework cache flushing logic--this version should work much better.

14 years agoIn-progress work on better cache flushing.
Michael Vrable [Fri, 26 Mar 2010 16:48:05 +0000 (09:48 -0700)]
In-progress work on better cache flushing.

14 years agoAdd LRU lists for tracking inodes that are dirty/accessed.
Michael Vrable [Thu, 25 Mar 2010 23:34:22 +0000 (16:34 -0700)]
Add LRU lists for tracking inodes that are dirty/accessed.

14 years agoMinor bugfix to pending store operation counts.
Michael Vrable [Wed, 24 Mar 2010 21:57:23 +0000 (14:57 -0700)]
Minor bugfix to pending store operation counts.

14 years agoFix a bug in committing of dirty inodes to stable storage.
Michael Vrable [Mon, 22 Mar 2010 23:38:53 +0000 (16:38 -0700)]
Fix a bug in committing of dirty inodes to stable storage.

Previously, if an inode changed while it was being committed, we would
stall all future commits.  Now allow those commits to proceed once the
first commit is done.

14 years agoBetter track latency of operations issued to S3.
Michael Vrable [Mon, 22 Mar 2010 22:07:21 +0000 (15:07 -0700)]
Better track latency of operations issued to S3.

Track the time from when the operation is sent to S3 as well, not just from
when it was submitted locally, as there may be some local queueing
happening.

14 years agoAdd very simple write throttling when caches fill up.
Michael Vrable [Fri, 19 Mar 2010 22:59:52 +0000 (15:59 -0700)]
Add very simple write throttling when caches fill up.

We ought to actively flush data when needed, but this is enough for some
basic testing.

14 years agoMake dropping of inodes from cache more resistant to races.
Michael Vrable [Fri, 19 Mar 2010 22:59:00 +0000 (15:59 -0700)]
Make dropping of inodes from cache more resistant to races.

14 years agoQuick note on memory leak debugging under valgrind.
Michael Vrable [Wed, 17 Mar 2010 22:01:27 +0000 (15:01 -0700)]
Quick note on memory leak debugging under valgrind.

14 years agoInodes can now be dropped from the cache too, not just file blocks.
Michael Vrable [Wed, 17 Mar 2010 22:01:07 +0000 (15:01 -0700)]
Inodes can now be dropped from the cache too, not just file blocks.

14 years agoTrack memory usage statistics for cached data.
Michael Vrable [Wed, 17 Mar 2010 20:56:26 +0000 (13:56 -0700)]
Track memory usage statistics for cached data.

14 years agoProvide a simple configurable limit on the number of threads.
Michael Vrable [Tue, 16 Mar 2010 00:22:19 +0000 (17:22 -0700)]
Provide a simple configurable limit on the number of threads.

Primarily this is meant to make running under valgrind easier, since
valgrind can only handle so many threads.  By default leave the limit
unbounded (the old behavior).

14 years agoFix memory leak in bluesky_block_fetch.
Michael Vrable [Fri, 12 Mar 2010 00:56:26 +0000 (16:56 -0800)]
Fix memory leak in bluesky_block_fetch.

14 years agoReturn success code from NFSv3 commit operations.
Michael Vrable [Fri, 12 Mar 2010 00:22:02 +0000 (16:22 -0800)]
Return success code from NFSv3 commit operations.

14 years agoEliminate a warning about symlink serialization that was false.
Michael Vrable [Thu, 11 Mar 2010 22:24:12 +0000 (14:24 -0800)]
Eliminate a warning about symlink serialization that was false.

14 years agoRefine lock acquisition order for renames.
Michael Vrable [Wed, 10 Mar 2010 22:47:04 +0000 (14:47 -0800)]
Refine lock acquisition order for renames.

14 years agoAdd UDP transport support to the NFS server.
Michael Vrable [Wed, 10 Mar 2010 00:20:46 +0000 (16:20 -0800)]
Add UDP transport support to the NFS server.

SpecSFS seems to require basic UDP functionality.

14 years agospecSFS requires Java.
Michael Vrable [Wed, 10 Mar 2010 00:20:33 +0000 (16:20 -0800)]
specSFS requires Java.

14 years agoStart fetch of all blocks in a read at the start of the read operation.
Michael Vrable [Fri, 5 Mar 2010 22:25:29 +0000 (14:25 -0800)]
Start fetch of all blocks in a read at the start of the read operation.

14 years agoDo not signal barrier until all notifiers have run.
Michael Vrable [Fri, 5 Mar 2010 22:25:00 +0000 (14:25 -0800)]
Do not signal barrier until all notifiers have run.

14 years agoOnly treat an async storage operation as finished when notifiers finish.
Michael Vrable [Fri, 5 Mar 2010 00:16:52 +0000 (16:16 -0800)]
Only treat an async storage operation as finished when notifiers finish.

There's still a bit more work to be done with barriers, but this eliminates
some race conditions in some usages.

14 years agoPartial work on parallel data fetches from S3 for large reads.
Michael Vrable [Thu, 4 Mar 2010 23:20:35 +0000 (15:20 -0800)]
Partial work on parallel data fetches from S3 for large reads.

Currently causes crashes, but that might be related to a race condition
that needs fixing in the async storage layer..

14 years agoExplicitly mount NFS using version 3 in microbenchmarks.
Michael Vrable [Fri, 26 Feb 2010 01:19:26 +0000 (17:19 -0800)]
Explicitly mount NFS using version 3 in microbenchmarks.

We only support version 3, so ask for that when mounting.  This can prevent
problems in mounting in some cases, so we don't return the proper error
messages when a client asks for a version we don't support.

14 years agoUse s3cmd for cleaning up S3 storage accounts before benchmark runs.
Michael Vrable [Thu, 25 Feb 2010 21:03:11 +0000 (13:03 -0800)]
Use s3cmd for cleaning up S3 storage accounts before benchmark runs.

14 years agoImplement a synthetic benchmark for bulk reads of file data.
Michael Vrable [Wed, 24 Feb 2010 22:47:42 +0000 (14:47 -0800)]
Implement a synthetic benchmark for bulk reads of file data.

14 years agosynclient: Issue read requests for files after lookups return.
Michael Vrable [Tue, 23 Feb 2010 23:10:51 +0000 (15:10 -0800)]
synclient: Issue read requests for files after lookups return.

14 years agoFix a typo in SunRPC decoding.
Michael Vrable [Tue, 23 Feb 2010 21:01:25 +0000 (13:01 -0800)]
Fix a typo in SunRPC decoding.

The code accidentally used = instead of == in a conditional, leading to
decode errors in RPC packet framing.

14 years agoAllow larger reads and writes.
Michael Vrable [Tue, 23 Feb 2010 01:12:14 +0000 (17:12 -0800)]
Allow larger reads and writes.

Support reads up to 1 MB now, instead of just 32 kB.

14 years agoFix required package list (need git-core, not git).
Michael Vrable [Tue, 23 Feb 2010 01:11:56 +0000 (17:11 -0800)]
Fix required package list (need git-core, not git).

14 years agoDo not start the RPC services until the filesystem is ready.
Michael Vrable [Fri, 19 Feb 2010 00:28:11 +0000 (16:28 -0800)]
Do not start the RPC services until the filesystem is ready.

14 years agoFix for kvstore with default settings.
Michael Vrable [Fri, 19 Feb 2010 00:04:48 +0000 (16:04 -0800)]
Fix for kvstore with default settings.

14 years agoStore backends now consist of a type followed by extra information.
Michael Vrable [Thu, 18 Feb 2010 23:12:59 +0000 (15:12 -0800)]
Store backends now consist of a type followed by extra information.

For example: "kv:storagehost.com:9000".  The other backends don't use this
yet but should be made to do so.

14 years agoMake kvstore backend in BlueSky asynchronous.
Michael Vrable [Wed, 17 Feb 2010 22:53:23 +0000 (14:53 -0800)]
Make kvstore backend in BlueSky asynchronous.

Also, add synthetic delays to the kvstore server for latency testing.

14 years agoMore CMake tweaking.
Michael Vrable [Tue, 16 Feb 2010 23:29:08 +0000 (15:29 -0800)]
More CMake tweaking.

14 years agoBuild system cleanup.
Michael Vrable [Tue, 16 Feb 2010 22:58:25 +0000 (14:58 -0800)]
Build system cleanup.

14 years agoLink kvstore into main build system.
Michael Vrable [Tue, 16 Feb 2010 22:48:12 +0000 (14:48 -0800)]
Link kvstore into main build system.

14 years agoValues in the kvstore are raw byte arrays, not strings.
Michael Vrable [Tue, 16 Feb 2010 19:22:53 +0000 (11:22 -0800)]
Values in the kvstore are raw byte arrays, not strings.

14 years agoHook John's kvstore up to the BlueSky implementation.
Michael Vrable [Tue, 16 Feb 2010 19:17:45 +0000 (11:17 -0800)]
Hook John's kvstore up to the BlueSky implementation.

14 years ago.gitignore updates, and deleting old build system files.
Michael Vrable [Mon, 15 Feb 2010 23:53:28 +0000 (15:53 -0800)]
.gitignore updates, and deleting old build system files.

14 years agoReorganizing kvstore sources and switching build system to CMake.
Michael Vrable [Mon, 15 Feb 2010 23:50:09 +0000 (15:50 -0800)]
Reorganizing kvstore sources and switching build system to CMake.

14 years agoAdd John MucCullough's simple key/value storage server.
Michael Vrable [Sun, 14 Feb 2010 22:56:22 +0000 (14:56 -0800)]
Add John MucCullough's simple key/value storage server.

14 years agoDo not drop data from cache immediately after a write.
Michael Vrable [Fri, 5 Feb 2010 21:40:33 +0000 (13:40 -0800)]
Do not drop data from cache immediately after a write.

14 years agoAdd very rudimentary eviction data blocks from the cache.
Michael Vrable [Fri, 5 Feb 2010 21:23:06 +0000 (13:23 -0800)]
Add very rudimentary eviction data blocks from the cache.

14 years agoHook synthetic NFS client into test script.
Michael Vrable [Fri, 5 Feb 2010 01:14:12 +0000 (17:14 -0800)]
Hook synthetic NFS client into test script.

14 years agoFix for file storage backend.
Michael Vrable [Fri, 5 Feb 2010 00:34:50 +0000 (16:34 -0800)]
Fix for file storage backend.

14 years agoMake storage backend selectable via environment variable.
Michael Vrable [Fri, 5 Feb 2010 00:03:25 +0000 (16:03 -0800)]
Make storage backend selectable via environment variable.

14 years agoAllow NFS lookup requests to better proceed in parallel.
Michael Vrable [Thu, 4 Feb 2010 22:19:10 +0000 (14:19 -0800)]
Allow NFS lookup requests to better proceed in parallel.

After doing a directory lookup, drop the directory lock before fetching the
file inode.

14 years agoUpdate synthetic RPC client to record timing.
Michael Vrable [Thu, 4 Feb 2010 19:36:13 +0000 (11:36 -0800)]
Update synthetic RPC client to record timing.

For now, both it and readbench and effectively testing the time to look up
an inode, which should be a single fetch from S3.

14 years ago.gitignore fix.
Michael Vrable [Wed, 3 Feb 2010 23:17:34 +0000 (15:17 -0800)]
.gitignore fix.

14 years agoWork on a client for generating a synthetic stream of NFS operations.
Michael Vrable [Wed, 3 Feb 2010 22:01:19 +0000 (14:01 -0800)]
Work on a client for generating a synthetic stream of NFS operations.

This should provide another means of testing performance in addition to the
kernel NFS client.

14 years ago.gitignore updates.
Michael Vrable [Wed, 3 Feb 2010 00:01:43 +0000 (16:01 -0800)]
.gitignore updates.

14 years agoAdd microbenchmarks to build system.
Michael Vrable [Tue, 2 Feb 2010 22:23:18 +0000 (14:23 -0800)]
Add microbenchmarks to build system.

14 years agoAdd scripts for automatically testing reads at different settings.
Michael Vrable [Tue, 2 Feb 2010 21:04:52 +0000 (13:04 -0800)]
Add scripts for automatically testing reads at different settings.

14 years agoCreate a runtime option for sync/async frontend request handling.
Michael Vrable [Tue, 2 Feb 2010 20:29:20 +0000 (12:29 -0800)]
Create a runtime option for sync/async frontend request handling.

14 years agoSIGUSR1 causes nfsd to dump filesystem state now for debugging.
Michael Vrable [Mon, 1 Feb 2010 17:38:25 +0000 (09:38 -0800)]
SIGUSR1 causes nfsd to dump filesystem state now for debugging.

14 years agoWork on a parallel read benchmark.
Michael Vrable [Fri, 29 Jan 2010 22:20:41 +0000 (14:20 -0800)]
Work on a parallel read benchmark.

14 years agoLeak cleanup.
Michael Vrable [Fri, 29 Jan 2010 22:18:44 +0000 (14:18 -0800)]
Leak cleanup.

14 years agoFix a deadlock and a few memory leaks.
Michael Vrable [Thu, 28 Jan 2010 23:15:14 +0000 (15:15 -0800)]
Fix a deadlock and a few memory leaks.

14 years agoAdd inode prefetching support in NFS readdirplus.
Michael Vrable [Thu, 28 Jan 2010 20:53:16 +0000 (12:53 -0800)]
Add inode prefetching support in NFS readdirplus.

14 years agoFinish up asynchronous inode loading.
Michael Vrable [Thu, 28 Jan 2010 19:50:07 +0000 (11:50 -0800)]
Finish up asynchronous inode loading.

14 years agoLet storage tuning options be set via environment variables.
Michael Vrable [Thu, 28 Jan 2010 18:56:11 +0000 (10:56 -0800)]
Let storage tuning options be set via environment variables.

This should make benchmarking runs easier.

14 years agoSwitch to CTR instead of CBC mode for encryption.
Michael Vrable [Thu, 28 Jan 2010 01:06:04 +0000 (17:06 -0800)]
Switch to CTR instead of CBC mode for encryption.

This should handle arbitrarily messages better.  Later we ought to switch
to something like EAX, so consider this change temporary.