Michael Vrable [Mon, 19 Jul 2010 19:46:30 +0000 (12:46 -0700)]
Allow batched log writes when writing dirty inodes.
Michael Vrable [Sun, 18 Jul 2010 04:35:54 +0000 (21:35 -0700)]
Basic filesystem journaling.
Infrastructure for writing log entries synchronously (though the log format
is not yet finished and isn't yet useful), and a partial hook into the
BlueSky filesystem.
Michael Vrable [Thu, 15 Jul 2010 22:54:52 +0000 (15:54 -0700)]
Add synchronous inode logging in the NFS server.
This is in preparation for adding inode logging for data durability.
Michael Vrable [Thu, 15 Jul 2010 22:54:05 +0000 (15:54 -0700)]
Barriers did not handle requests that finished too quickly.
Michael Vrable [Wed, 14 Jul 2010 23:32:08 +0000 (16:32 -0700)]
Commit a few log benchmark results.
Michael Vrable [Wed, 14 Jul 2010 22:25:51 +0000 (15:25 -0700)]
Bugfix.
Michael Vrable [Wed, 14 Jul 2010 22:23:17 +0000 (15:23 -0700)]
Update commit log benchmarks.
Michael Vrable [Wed, 14 Jul 2010 05:25:39 +0000 (22:25 -0700)]
Make the log benchmark configurable and make a parameter sweep script.
Michael Vrable [Wed, 14 Jul 2010 00:30:50 +0000 (17:30 -0700)]
A new microbenchmark tool to figure out what format to use for logs.
We want to log filesystem operations to disk so they are persistent across
proxy crashes, but should do so in a manner that is relatively high
performance... Try to figure out what that should be.
Michael Vrable [Tue, 13 Jul 2010 00:26:21 +0000 (17:26 -0700)]
Attempt to batch together database writes for performance.
Doesn't seem to help all that much though...
Michael Vrable [Mon, 12 Jul 2010 19:29:15 +0000 (12:29 -0700)]
Switch to an explicit BDB operations queue instead of thread pool.
Create a dedicated thread for handling BDB operations, and pass gets/puts
to it via a queue. This is in preparation for batching operations
together in transactions to see if we can improve performance that way.
Michael Vrable [Wed, 7 Jul 2010 19:05:04 +0000 (12:05 -0700)]
Remove localstore.c; for now BDB work will be done in store-bdb.c.
Michael Vrable [Wed, 7 Jul 2010 19:03:40 +0000 (12:03 -0700)]
Some test work with using Berkeley DB for a local disk cache.
Implement this at first by simply making it a new storage backend. Later
we will have to make it a separate layer so we can stack a Berkeley DB
cache/write log with a remote storage option.
Michael Vrable [Wed, 30 Jun 2010 23:59:14 +0000 (16:59 -0700)]
Some new format design notes.
Michael Vrable [Wed, 30 Jun 2010 19:25:14 +0000 (12:25 -0700)]
Add another S3 benchmark tool.
Michael Vrable [Wed, 30 Jun 2010 19:24:50 +0000 (12:24 -0700)]
More S3 benchmark work.
Michael Vrable [Fri, 18 Jun 2010 01:11:15 +0000 (18:11 -0700)]
Add an S3 test script for range requests.
Michael Vrable [Wed, 16 Jun 2010 22:26:29 +0000 (15:26 -0700)]
Add results from a simple test run of the multi:s3 backend.
Michael Vrable [Wed, 16 Jun 2010 20:50:29 +0000 (13:50 -0700)]
Testing S3 with more object sizes.
Michael Vrable [Tue, 15 Jun 2010 18:29:29 +0000 (11:29 -0700)]
More Amazon S3 test script updates.
Michael Vrable [Fri, 4 Jun 2010 21:33:54 +0000 (14:33 -0700)]
Fix a possibel deadlock from synchronizing the superblock.
Michael Vrable [Fri, 4 Jun 2010 17:47:25 +0000 (10:47 -0700)]
Add a "multi" storage backend which doubles all GET requests.
This can be used to test whether the performance with S3 improves when
making parallel requests to reduce latency.
Michael Vrable [Thu, 3 Jun 2010 22:25:31 +0000 (15:25 -0700)]
Updates to script for testing multiple parallel fetches.
Michael Vrable [Fri, 28 May 2010 19:08:59 +0000 (12:08 -0700)]
Testing of multiple requests in parallel.
Michael Vrable [Wed, 26 May 2010 21:17:41 +0000 (14:17 -0700)]
Add a few more test scripts.
Michael Vrable [Thu, 13 May 2010 20:32:24 +0000 (13:32 -0700)]
More trace analysis scripts.
Michael Vrable [Sat, 8 May 2010 02:12:51 +0000 (19:12 -0700)]
Improve analysis of S3 packet traces.
Michael Vrable [Thu, 6 May 2010 23:32:45 +0000 (16:32 -0700)]
Keep IP addresses with network measurements.
Michael Vrable [Wed, 5 May 2010 20:46:44 +0000 (13:46 -0700)]
More S3 performance evaluation scripts.
Michael Vrable [Wed, 5 May 2010 00:24:32 +0000 (17:24 -0700)]
Compile fix for directory renaming; use virtual host access method for S3.
Michael Vrable [Tue, 4 May 2010 23:53:14 +0000 (16:53 -0700)]
More cloud storage performance-measurement scripts.
Michael Vrable [Tue, 4 May 2010 19:22:51 +0000 (12:22 -0700)]
Directory reorganization.
Michael Vrable [Tue, 4 May 2010 19:19:42 +0000 (12:19 -0700)]
Work on more tools for automating cloud storage performance measurement.
Michael Vrable [Tue, 4 May 2010 04:46:50 +0000 (21:46 -0700)]
A few more Azure/analysis updates.
Michael Vrable [Mon, 3 May 2010 04:51:22 +0000 (21:51 -0700)]
Another update to the Azure test code.
Michael Vrable [Mon, 3 May 2010 02:10:02 +0000 (19:10 -0700)]
Start work on a test program to communicate with Windows Azure.
This could provide another backend for BlueSky, and it might be nice to at
least make a few benchmark measurements of Azure to complement the S3 ones.
Michael Vrable [Thu, 29 Apr 2010 23:07:21 +0000 (16:07 -0700)]
Simple tool for analyzing a dump of a single S3 connection.
Will try to extract information about any delays in the transfers.
Michael Vrable [Wed, 28 Apr 2010 22:06:50 +0000 (15:06 -0700)]
More playing with parsing of packet traces.
Extract window size values (and handle TCP window scaling).
Michael Vrable [Wed, 28 Apr 2010 21:30:14 +0000 (14:30 -0700)]
More updates to S3 pipelining test.
It seems there is a bug in S3 with many pipelined requests that we are
triggering in testing:
http://developer.amazonwebservices.com/connect/thread.jspa?messageID=39907
Michael Vrable [Wed, 28 Apr 2010 18:04:12 +0000 (11:04 -0700)]
Create a simple Python script for sending pipelined GET requests to S3.
S3 doesn't seem to respond properly to these yet.
Michael Vrable [Tue, 27 Apr 2010 21:13:45 +0000 (14:13 -0700)]
Print flow identification in TCP parsing output.
Also fix a division-by-zero bug in computing bandwidth if no data is
transferred.
Michael Vrable [Mon, 26 Apr 2010 22:36:45 +0000 (15:36 -0700)]
Some initial results of running the NFS replay code.
Michael Vrable [Mon, 26 Apr 2010 05:38:27 +0000 (22:38 -0700)]
Fix a memory leak in the NFS-over-UDP code.
The TCP code still has a per-connection leak, but at least this should help
with the UDP per-request leak.
Michael Vrable [Sun, 25 Apr 2010 00:34:09 +0000 (17:34 -0700)]
Enable real-time trace replay instead of maximum-speed replay mode.
Michael Vrable [Sat, 24 Apr 2010 02:46:13 +0000 (19:46 -0700)]
Allow longer server names; fixes (temporarily) a buffer overflow.
Michael Vrable [Fri, 23 Apr 2010 20:30:25 +0000 (13:30 -0700)]
Code fixes for TBBT.
Michael Vrable [Fri, 23 Apr 2010 19:50:15 +0000 (12:50 -0700)]
Add two missing files from the TBBT import.
Michael Vrable [Fri, 23 Apr 2010 19:48:14 +0000 (12:48 -0700)]
Import TBBT (NFS trace replay).
Code downloaded from http://www.ecsl.cs.sunysb.edu/TBBT/.
Michael Vrable [Fri, 23 Apr 2010 19:43:57 +0000 (12:43 -0700)]
Merge git+ssh://root@c09-44.sysnet.ucsd.edu/scratch/bluesky
Michael Vrable [Wed, 21 Apr 2010 23:13:58 +0000 (16:13 -0700)]
Add pcap dump parser for extracting S3 performance measurements.
Michael Vrable [Wed, 21 Apr 2010 23:13:39 +0000 (16:13 -0700)]
More measurement results of S3 latency.
Michael Vrable [Tue, 20 Apr 2010 03:10:02 +0000 (20:10 -0700)]
Add new results.
Michael Vrable [Tue, 13 Apr 2010 18:10:49 +0000 (11:10 -0700)]
Commit some benchmark results of standard Samba.
Michael Vrable [Fri, 9 Apr 2010 18:40:50 +0000 (11:40 -0700)]
Update needed dependencies for building Samba.
Michael Vrable [Wed, 7 Apr 2010 22:36:58 +0000 (15:36 -0700)]
Cleanups while workign on Samba code.
Michael Vrable [Tue, 6 Apr 2010 23:41:22 +0000 (16:41 -0700)]
Add some measurements of speed reading data from S3.
Michael Vrable [Tue, 6 Apr 2010 03:59:41 +0000 (20:59 -0700)]
Initial work to use Berkeley DB for locally logging FS changes.
Not yet hooked up to anything else, but commit this now so the file doesn't
get lost.
Michael Vrable [Tue, 6 Apr 2010 00:08:53 +0000 (17:08 -0700)]
Pull in updated tool for measuring S3 read performance.
Michael Vrable [Fri, 2 Apr 2010 20:59:41 +0000 (13:59 -0700)]
Improve statistics gathered.
Michael Vrable [Fri, 2 Apr 2010 19:20:21 +0000 (12:20 -0700)]
Add general statistics-gathering infrastructure.
Michael Vrable [Fri, 2 Apr 2010 05:00:45 +0000 (22:00 -0700)]
Benchmark results to S3.
Michael Vrable [Thu, 1 Apr 2010 18:56:36 +0000 (11:56 -0700)]
Add some more results.
Michael Vrable [Thu, 1 Apr 2010 01:24:49 +0000 (18:24 -0700)]
Fix filesystem locking when creating a new inode.
Michael Vrable [Thu, 1 Apr 2010 00:30:17 +0000 (17:30 -0700)]
Add warnings for failed storage operations.
Michael Vrable [Wed, 31 Mar 2010 21:53:54 +0000 (14:53 -0700)]
Disable most debugging print messages; should help with performance.
Michael Vrable [Wed, 31 Mar 2010 21:01:59 +0000 (14:01 -0700)]
Try to clean up the locking for LRU lists.
Michael Vrable [Wed, 31 Mar 2010 17:48:22 +0000 (10:48 -0700)]
A few minor fixes.
Michael Vrable [Tue, 30 Mar 2010 23:09:24 +0000 (16:09 -0700)]
Copy in a few more benchmark results.
Michael Vrable [Tue, 30 Mar 2010 18:14:34 +0000 (11:14 -0700)]
Store benchmark results for native Linux NFS server.
Michael Vrable [Tue, 30 Mar 2010 03:56:00 +0000 (20:56 -0700)]
Track number of unanswered RPC requests, for debugging.
Michael Vrable [Tue, 30 Mar 2010 03:36:03 +0000 (20:36 -0700)]
Do not fill in a non-zero inode change_count until the inode is loaded.
Michael Vrable [Tue, 30 Mar 2010 03:30:09 +0000 (20:30 -0700)]
Possible null pointer dereference fix.
Michael Vrable [Mon, 29 Mar 2010 19:48:03 +0000 (12:48 -0700)]
Rework cache flushing logic--this version should work much better.
Michael Vrable [Fri, 26 Mar 2010 16:48:05 +0000 (09:48 -0700)]
In-progress work on better cache flushing.
Michael Vrable [Thu, 25 Mar 2010 23:34:22 +0000 (16:34 -0700)]
Add LRU lists for tracking inodes that are dirty/accessed.
Michael Vrable [Wed, 24 Mar 2010 21:57:23 +0000 (14:57 -0700)]
Minor bugfix to pending store operation counts.
Michael Vrable [Mon, 22 Mar 2010 23:38:53 +0000 (16:38 -0700)]
Fix a bug in committing of dirty inodes to stable storage.
Previously, if an inode changed while it was being committed, we would
stall all future commits. Now allow those commits to proceed once the
first commit is done.
Michael Vrable [Mon, 22 Mar 2010 22:07:21 +0000 (15:07 -0700)]
Better track latency of operations issued to S3.
Track the time from when the operation is sent to S3 as well, not just from
when it was submitted locally, as there may be some local queueing
happening.
Michael Vrable [Fri, 19 Mar 2010 22:59:52 +0000 (15:59 -0700)]
Add very simple write throttling when caches fill up.
We ought to actively flush data when needed, but this is enough for some
basic testing.
Michael Vrable [Fri, 19 Mar 2010 22:59:00 +0000 (15:59 -0700)]
Make dropping of inodes from cache more resistant to races.
Michael Vrable [Wed, 17 Mar 2010 22:01:27 +0000 (15:01 -0700)]
Quick note on memory leak debugging under valgrind.
Michael Vrable [Wed, 17 Mar 2010 22:01:07 +0000 (15:01 -0700)]
Inodes can now be dropped from the cache too, not just file blocks.
Michael Vrable [Wed, 17 Mar 2010 20:56:26 +0000 (13:56 -0700)]
Track memory usage statistics for cached data.
Michael Vrable [Tue, 16 Mar 2010 00:22:19 +0000 (17:22 -0700)]
Provide a simple configurable limit on the number of threads.
Primarily this is meant to make running under valgrind easier, since
valgrind can only handle so many threads. By default leave the limit
unbounded (the old behavior).
Michael Vrable [Fri, 12 Mar 2010 00:56:26 +0000 (16:56 -0800)]
Fix memory leak in bluesky_block_fetch.
Michael Vrable [Fri, 12 Mar 2010 00:22:02 +0000 (16:22 -0800)]
Return success code from NFSv3 commit operations.
Michael Vrable [Thu, 11 Mar 2010 22:24:12 +0000 (14:24 -0800)]
Eliminate a warning about symlink serialization that was false.
Michael Vrable [Wed, 10 Mar 2010 22:47:04 +0000 (14:47 -0800)]
Refine lock acquisition order for renames.
Michael Vrable [Wed, 10 Mar 2010 00:20:46 +0000 (16:20 -0800)]
Add UDP transport support to the NFS server.
SpecSFS seems to require basic UDP functionality.
Michael Vrable [Wed, 10 Mar 2010 00:20:33 +0000 (16:20 -0800)]
specSFS requires Java.
Michael Vrable [Fri, 5 Mar 2010 22:25:29 +0000 (14:25 -0800)]
Start fetch of all blocks in a read at the start of the read operation.
Michael Vrable [Fri, 5 Mar 2010 22:25:00 +0000 (14:25 -0800)]
Do not signal barrier until all notifiers have run.
Michael Vrable [Fri, 5 Mar 2010 00:16:52 +0000 (16:16 -0800)]
Only treat an async storage operation as finished when notifiers finish.
There's still a bit more work to be done with barriers, but this eliminates
some race conditions in some usages.
Michael Vrable [Thu, 4 Mar 2010 23:20:35 +0000 (15:20 -0800)]
Partial work on parallel data fetches from S3 for large reads.
Currently causes crashes, but that might be related to a race condition
that needs fixing in the async storage layer..
Michael Vrable [Fri, 26 Feb 2010 01:19:26 +0000 (17:19 -0800)]
Explicitly mount NFS using version 3 in microbenchmarks.
We only support version 3, so ask for that when mounting. This can prevent
problems in mounting in some cases, so we don't return the proper error
messages when a client asks for a version we don't support.
Michael Vrable [Thu, 25 Feb 2010 21:03:11 +0000 (13:03 -0800)]
Use s3cmd for cleaning up S3 storage accounts before benchmark runs.
Michael Vrable [Wed, 24 Feb 2010 22:47:42 +0000 (14:47 -0800)]
Implement a synthetic benchmark for bulk reads of file data.
Michael Vrable [Tue, 23 Feb 2010 23:10:51 +0000 (15:10 -0800)]
synclient: Issue read requests for files after lookups return.
Michael Vrable [Tue, 23 Feb 2010 21:01:25 +0000 (13:01 -0800)]
Fix a typo in SunRPC decoding.
The code accidentally used = instead of == in a conditional, leading to
decode errors in RPC packet framing.
Michael Vrable [Tue, 23 Feb 2010 01:12:14 +0000 (17:12 -0800)]
Allow larger reads and writes.
Support reads up to 1 MB now, instead of just 32 kB.