bluesky.git
13 years agoUpdate a results README file
Michael Vrable [Sun, 6 Mar 2011 23:29:07 +0000 (15:29 -0800)]
Update a results README file

13 years agoScript for parsing sfsres files to extract more detailed latency values
Michael Vrable [Sun, 6 Mar 2011 23:27:51 +0000 (15:27 -0800)]
Script for parsing sfsres files to extract more detailed latency values

13 years agoBenchmark results without range requests.
Michael Vrable [Sun, 6 Mar 2011 04:52:49 +0000 (20:52 -0800)]
Benchmark results without range requests.

13 years agoReduce verbosity of debugging of disk cache contents
root [Fri, 4 Mar 2011 22:20:25 +0000 (14:20 -0800)]
Reduce verbosity of debugging of disk cache contents

13 years agoUse virtual hosts for specifying S3 buckets
root [Fri, 4 Mar 2011 22:19:30 +0000 (14:19 -0800)]
Use virtual hosts for specifying S3 buckets

This is needed to get requests directed to the right place when using
different locations (US-West vs. US-California, for example).

13 years agoAdd more benchmark results
Michael Vrable [Fri, 4 Mar 2011 01:36:48 +0000 (17:36 -0800)]
Add more benchmark results

13 years agoMore memory usage stats
Michael Vrable [Fri, 4 Mar 2011 00:10:11 +0000 (16:10 -0800)]
More memory usage stats

13 years agoMeasure effect of read latencies vs. connection idleness
Michael Vrable [Fri, 4 Mar 2011 00:07:42 +0000 (16:07 -0800)]
Measure effect of read latencies vs. connection idleness

13 years agoAdd more memory overhead data
Michael Vrable [Thu, 3 Mar 2011 21:17:37 +0000 (13:17 -0800)]
Add more memory overhead data

13 years agoUpdate memory usage plot script
Michael Vrable [Thu, 3 Mar 2011 21:15:39 +0000 (13:15 -0800)]
Update memory usage plot script

13 years agoMemory usage stats from another benchmark run (on the real proxy)
Michael Vrable [Thu, 3 Mar 2011 16:37:41 +0000 (08:37 -0800)]
Memory usage stats from another benchmark run (on the real proxy)

13 years agoStatistics on memory usage during a test benchmark run
Michael Vrable [Thu, 3 Mar 2011 16:35:50 +0000 (08:35 -0800)]
Statistics on memory usage during a test benchmark run

13 years agoInclude S3-US-West results in graphs
Michael Vrable [Thu, 3 Mar 2011 00:02:45 +0000 (16:02 -0800)]
Include S3-US-West results in graphs

13 years agoLimit the number of concurrent log uploads to the cloud
Michael Vrable [Wed, 2 Mar 2011 22:20:46 +0000 (14:20 -0800)]
Limit the number of concurrent log uploads to the cloud

This is mostly aimed at limiting memory usage, since each cloud log segment
is stored entirely in memory before it is uploaded, so a very large number
of parallel uploads could require allocating a large amount of memory.

13 years agoMove encryption of cloud log segments into background threads.
Michael Vrable [Wed, 2 Mar 2011 01:09:29 +0000 (17:09 -0800)]
Move encryption of cloud log segments into background threads.

Crypto operations looked to be a CPU bottleneck, but using separate threads
will at least allow multiple log segments to be encrypted in parallel on
separate CPU cores.

13 years agoUpdates to run microbenchmarks against S3 us-west.
Michael Vrable [Tue, 1 Mar 2011 07:23:39 +0000 (23:23 -0800)]
Updates to run microbenchmarks against S3 us-west.

13 years agoMerge git+ssh://c09-44.sysnet.ucsd.edu/scratch/bluesky
Michael Vrable [Tue, 1 Mar 2011 04:26:24 +0000 (20:26 -0800)]
Merge git+ssh://c09-44.sysnet.ucsd.edu/scratch/bluesky

13 years agoCheck in results using S3 US-west location.
Michael Vrable [Tue, 1 Mar 2011 04:25:59 +0000 (20:25 -0800)]
Check in results using S3 US-west location.

13 years agoMerge git+ssh://idril.vpn.vrable.net/home/mvrable/local/bluesky
Michael Vrable [Tue, 1 Mar 2011 00:28:12 +0000 (16:28 -0800)]
Merge git+ssh://idril.vpn.vrable.net/home/mvrable/local/bluesky

13 years agoPrint out Azure objects as they are deleted.
Michael Vrable [Tue, 1 Mar 2011 00:27:46 +0000 (16:27 -0800)]
Print out Azure objects as they are deleted.

13 years agoConfig file for SPECsfs CIFS testing.
Michael Vrable [Mon, 28 Feb 2011 19:34:24 +0000 (11:34 -0800)]
Config file for SPECsfs CIFS testing.

13 years agoResults from SPECsfs runs against Windows Azure.
Michael Vrable [Mon, 28 Feb 2011 19:33:09 +0000 (11:33 -0800)]
Results from SPECsfs runs against Windows Azure.

13 years agoPlots for the CIFS SPECsfs results.
Michael Vrable [Mon, 28 Feb 2011 16:45:26 +0000 (08:45 -0800)]
Plots for the CIFS SPECsfs results.

13 years agoMerge git+ssh://c09-45.sysnet.ucsd.edu/scratch/bluesky
Michael Vrable [Mon, 28 Feb 2011 16:32:41 +0000 (08:32 -0800)]
Merge git+ssh://c09-45.sysnet.ucsd.edu/scratch/bluesky

13 years agoMake the azure test python script clean up files from BlueSky runs.
Michael Vrable [Mon, 28 Feb 2011 16:31:51 +0000 (08:31 -0800)]
Make the azure test python script clean up files from BlueSky runs.

13 years agoMerge git+ssh://c09-44.sysnet.ucsd.edu/scratch/bluesky
Michael Vrable [Mon, 28 Feb 2011 16:25:32 +0000 (08:25 -0800)]
Merge git+ssh://c09-44.sysnet.ucsd.edu/scratch/bluesky

13 years agoCheck in some SPECsfs runs for CIFS.
Michael Vrable [Mon, 28 Feb 2011 16:25:13 +0000 (08:25 -0800)]
Check in some SPECsfs runs for CIFS.

13 years agoTurn on profiling in microbenchmarking.
root [Mon, 28 Feb 2011 13:33:13 +0000 (05:33 -0800)]
Turn on profiling in microbenchmarking.

13 years agoCheck in README and sample plots for some of the new data.
Michael Vrable [Mon, 28 Feb 2011 06:39:43 +0000 (22:39 -0800)]
Check in README and sample plots for some of the new data.

13 years agoChecking in some results with a smaller, slower disk.
Michael Vrable [Mon, 28 Feb 2011 06:19:38 +0000 (22:19 -0800)]
Checking in some results with a smaller, slower disk.

13 years agoParameters for investigating latency anomalies, and results.
Michael Vrable [Mon, 28 Feb 2011 04:39:55 +0000 (20:39 -0800)]
Parameters for investigating latency anomalies, and results.

13 years agoCheck in some detailed timing results to investigate strange latency vs.
Michael Vrable [Mon, 28 Feb 2011 04:37:58 +0000 (20:37 -0800)]
Check in some detailed timing results to investigate strange latency vs.
requested load figures.

13 years agoMerge branch 'master' of git+ssh://niniel.sysnet.ucsd.edu/home/mvrable/local/bluesky
root [Sun, 27 Feb 2011 22:26:07 +0000 (14:26 -0800)]
Merge branch 'master' of git+ssh://niniel.sysnet.ucsd.edu/home/mvrable/local/bluesky

13 years agoAdd some proxy statistics.
Michael Vrable [Sun, 27 Feb 2011 22:25:41 +0000 (14:25 -0800)]
Add some proxy statistics.

13 years agoSwitch disk used for storing data during benchmarks.
root [Sun, 27 Feb 2011 21:50:53 +0000 (13:50 -0800)]
Switch disk used for storing data during benchmarks.

13 years agoCheck in some recent SPECsfs benchmark results.
Michael Vrable [Sun, 27 Feb 2011 21:35:36 +0000 (13:35 -0800)]
Check in some recent SPECsfs benchmark results.

13 years agoMake NFS rsize/wsize configurable in the microbenchmark runs.
Michael Vrable [Sun, 27 Feb 2011 20:46:31 +0000 (12:46 -0800)]
Make NFS rsize/wsize configurable in the microbenchmark runs.

13 years agoFix byte range request syntax.
Michael Vrable [Fri, 25 Feb 2011 01:07:19 +0000 (17:07 -0800)]
Fix byte range request syntax.

13 years agoDelete dead code.
Michael Vrable [Fri, 25 Feb 2011 00:53:24 +0000 (16:53 -0800)]
Delete dead code.

13 years agoAdd a CURL connection pool to the Azure backend.
Michael Vrable [Fri, 25 Feb 2011 00:52:46 +0000 (16:52 -0800)]
Add a CURL connection pool to the Azure backend.

This will allow HTTP connections to be re-used.

13 years agoAdd byte range support to the Azure backend.
Michael Vrable [Fri, 25 Feb 2011 00:40:06 +0000 (16:40 -0800)]
Add byte range support to the Azure backend.

13 years agoAdd Windows Azure support to BlueSky.
Michael Vrable [Thu, 24 Feb 2011 23:54:18 +0000 (15:54 -0800)]
Add Windows Azure support to BlueSky.

This is not much tested but seems to work at a basic level.

13 years agoCheck in some minor changes to microbenchmark runs.
Michael Vrable [Wed, 23 Feb 2011 23:36:28 +0000 (15:36 -0800)]
Check in some minor changes to microbenchmark runs.

13 years agoBuild a quick storage library for accessing Windows Azure.
Michael Vrable [Wed, 23 Feb 2011 23:25:48 +0000 (15:25 -0800)]
Build a quick storage library for accessing Windows Azure.

This will be used by the Python cleaner, but that is not yet hooked up.

13 years agoRun both read, write, and mixed workloads.
Michael Vrable [Mon, 21 Feb 2011 04:41:52 +0000 (20:41 -0800)]
Run both read, write, and mixed workloads.

13 years agoSet up for benchmark runs with small reads/writes within larger files.
Michael Vrable [Mon, 21 Feb 2011 04:40:21 +0000 (20:40 -0800)]
Set up for benchmark runs with small reads/writes within larger files.

13 years agoMore reworking of microbenchmark handling of reads/writes and file layout.
Michael Vrable [Sun, 20 Feb 2011 04:28:46 +0000 (20:28 -0800)]
More reworking of microbenchmark handling of reads/writes and file layout.

13 years agoChange how mixed read/write workloads are done in mixedbench
Michael Vrable [Sun, 20 Feb 2011 03:30:49 +0000 (19:30 -0800)]
Change how mixed read/write workloads are done in mixedbench

Split reads and writes into separate threads, instead of mixing operations
within a single thread.  This does not allow arbitrary mixes any longer,
since the ratios are quantized based on the number of total threads.

13 years agoCommit script to parse microbenchmark results.
Michael Vrable [Mon, 14 Feb 2011 23:06:03 +0000 (15:06 -0800)]
Commit script to parse microbenchmark results.

13 years agoDelete old benchmark execution script. It is obsolete.
Michael Vrable [Mon, 14 Feb 2011 19:15:36 +0000 (11:15 -0800)]
Delete old benchmark execution script.  It is obsolete.

13 years agoUpdate benchmark script to run a few other cases.
Michael Vrable [Mon, 14 Feb 2011 07:41:10 +0000 (23:41 -0800)]
Update benchmark script to run a few other cases.

13 years agoAdd a "native" target to run the kernel NFS server.
Michael Vrable [Mon, 14 Feb 2011 05:57:38 +0000 (21:57 -0800)]
Add a "native" target to run the kernel NFS server.

13 years agoFix standard deviation estimation.
Michael Vrable [Mon, 14 Feb 2011 05:39:56 +0000 (21:39 -0800)]
Fix standard deviation estimation.

Properly reset statistics counters between intervals.

13 years agoMore benchmark changes.
Michael Vrable [Mon, 14 Feb 2011 05:35:36 +0000 (21:35 -0800)]
More benchmark changes.

13 years agoBenchmark fixups.
Michael Vrable [Mon, 14 Feb 2011 05:30:35 +0000 (21:30 -0800)]
Benchmark fixups.

13 years agoSpread test files for mixedbench across multiple subdirectories.
Michael Vrable [Mon, 14 Feb 2011 05:12:52 +0000 (21:12 -0800)]
Spread test files for mixedbench across multiple subdirectories.

13 years agoReduce effective memory size on benchmark client.
Michael Vrable [Mon, 14 Feb 2011 03:42:38 +0000 (19:42 -0800)]
Reduce effective memory size on benchmark client.

Lock memory to reduce the amount available for the NFS file cache, so
that requests will go the proxy.

13 years agoSplit benchmark file system setup and execution into separate steps.
Michael Vrable [Mon, 14 Feb 2011 03:01:14 +0000 (19:01 -0800)]
Split benchmark file system setup and execution into separate steps.

13 years agoAllow interval at which benchmark results are printed to be controlled.
Michael Vrable [Sun, 13 Feb 2011 20:57:13 +0000 (12:57 -0800)]
Allow interval at which benchmark results are printed to be controlled.

13 years agoWork on automatically running benchmarks.
Michael Vrable [Sun, 13 Feb 2011 20:34:00 +0000 (12:34 -0800)]
Work on automatically running benchmarks.

13 years agoMinor fix to benchmark running script.
Michael Vrable [Thu, 10 Feb 2011 18:40:47 +0000 (10:40 -0800)]
Minor fix to benchmark running script.

13 years agoAdd script to clean up files from a benchmarking run with nfsproxy.
root [Thu, 10 Feb 2011 00:49:49 +0000 (16:49 -0800)]
Add script to clean up files from a benchmarking run with nfsproxy.

13 years agoAdd script to delete state from S3 for benchmarking cleanup.
Michael Vrable [Thu, 10 Feb 2011 00:31:51 +0000 (16:31 -0800)]
Add script to delete state from S3 for benchmarking cleanup.

13 years agoAdd script for stopping a running nfsproxy
root [Thu, 10 Feb 2011 00:27:53 +0000 (16:27 -0800)]
Add script for stopping a running nfsproxy

13 years agoAdd script to run a simple microbenchmark.
Michael Vrable [Wed, 9 Feb 2011 23:26:54 +0000 (15:26 -0800)]
Add script to run a simple microbenchmark.

13 years agoAdd setup script for starting the proxy for benchmarking.
root [Wed, 9 Feb 2011 23:23:36 +0000 (15:23 -0800)]
Add setup script for starting the proxy for benchmarking.

13 years agoA few more updates to the mixed read/write microbenchmark.
Michael Vrable [Mon, 7 Feb 2011 18:02:04 +0000 (10:02 -0800)]
A few more updates to the mixed read/write microbenchmark.

13 years agoChanges to output formattng of mixedbench.
Michael Vrable [Mon, 7 Feb 2011 15:22:44 +0000 (07:22 -0800)]
Changes to output formattng of mixedbench.

13 years agoAdd a new more general-purpose microbenchmark tool in C.
Michael Vrable [Mon, 7 Feb 2011 13:58:01 +0000 (05:58 -0800)]
Add a new more general-purpose microbenchmark tool in C.

13 years agoFix a potential race between creating and destroying mmaped strings
Michael Vrable [Thu, 3 Feb 2011 00:07:58 +0000 (16:07 -0800)]
Fix a potential race between creating and destroying mmaped strings

Potentially, we might try to create a new reference to a memory-mapped
region while another thread unreferences and frees that region.  Ensure
when freeing the mapping that there really are no mappings (doube-check
after taking the lock).

13 years agoFix a memory leak.
Michael Vrable [Wed, 2 Feb 2011 18:01:00 +0000 (10:01 -0800)]
Fix a memory leak.

13 years agoFix use-after-free in network address lookups.
Michael Vrable [Wed, 2 Feb 2011 17:30:47 +0000 (09:30 -0800)]
Fix use-after-free in network address lookups.

13 years agoUpdate simplestore to use a standard port number.
Michael Vrable [Fri, 28 Jan 2011 23:48:46 +0000 (15:48 -0800)]
Update simplestore to use a standard port number.

13 years agoAdd simplestore server to build.
Michael Vrable [Fri, 28 Jan 2011 23:36:35 +0000 (15:36 -0800)]
Add simplestore server to build.

13 years agoAdd a connection pool to the simplesever client
Michael Vrable [Mon, 24 Jan 2011 01:12:01 +0000 (17:12 -0800)]
Add a connection pool to the simplesever client

Re-use TCP connections across requests instead of closing the connection
each time.

13 years agoStart work on a new cleaner backend
Michael Vrable [Mon, 24 Jan 2011 00:52:52 +0000 (16:52 -0800)]
Start work on a new cleaner backend

Begin adding support for the simplestore backend to the cleaner code.  This
is not yet tested and so probably needs a bit more work.

13 years agoRename filestore -> simplestore
Michael Vrable [Fri, 21 Jan 2011 06:12:53 +0000 (22:12 -0800)]
Rename filestore -> simplestore

This is more consistent with the naming in the rest of the BlueSky code.

13 years agoEnable range requests in the simple storage backend
Michael Vrable [Fri, 21 Jan 2011 01:55:31 +0000 (17:55 -0800)]
Enable range requests in the simple storage backend

13 years agoFix off-by-one error in the BlueSky simple storage backend.
Michael Vrable [Fri, 21 Jan 2011 00:34:59 +0000 (16:34 -0800)]
Fix off-by-one error in the BlueSky simple storage backend.

One extra byte of data was being included into the data returned, leading
to data corruption.  Fix this.

13 years agoCheck in a new very simple client/server storage protocol implementation.
Michael Vrable [Tue, 18 Jan 2011 05:35:38 +0000 (21:35 -0800)]
Check in a new very simple client/server storage protocol implementation.

This is meant for local testing; the server writes data to a local disk
without synchronization so data isn't guaranteed to be durable but this
should provide better performance for benchmarking (where we care about the
BlueSky performance and not the storage sever itself).

13 years agoCommit a basic but functional online cleaner implementation.
Michael Vrable [Fri, 10 Dec 2010 23:02:15 +0000 (15:02 -0800)]
Commit a basic but functional online cleaner implementation.

The cleaner needn't be aware of parallelism; the proxy will handle merging
of data.

13 years agoFixes and debugging assertions.
Michael Vrable [Fri, 10 Dec 2010 19:58:25 +0000 (11:58 -0800)]
Fixes and debugging assertions.

13 years agoFix a race condition in bluesky_mmap_unref.
Michael Vrable [Fri, 10 Dec 2010 17:12:50 +0000 (09:12 -0800)]
Fix a race condition in bluesky_mmap_unref.

It turned out that the old code had a race--between decrementing and
testing the reference count and acquiring the lock, another thread could
potentially acquire a reference, release the reference, and then unmap the
data.  This still left the reference count as zero, so a second unmap was
attempted.  Now, use mmap->addr to detect if the unmap needs to be done or
not which should eliminate this race.

13 years agoMostly implement inode merging for the in-proxy cleaner component.
Michael Vrable [Fri, 10 Dec 2010 02:26:15 +0000 (18:26 -0800)]
Mostly implement inode merging for the in-proxy cleaner component.

This doesn't yet ensure modifications are serialized out again later.

13 years agoFix to the cleaner when writing out a new inode map.
Michael Vrable [Thu, 9 Dec 2010 20:12:04 +0000 (12:12 -0800)]
Fix to the cleaner when writing out a new inode map.

13 years agoAdd code in the proxy cleaner component to iterate over new inodes.
Michael Vrable [Thu, 9 Dec 2010 06:32:17 +0000 (22:32 -0800)]
Add code in the proxy cleaner component to iterate over new inodes.

For the moment we actually iterate over all inodes in the cleaned
checkpoint, but later can restrict to just those that are needed.  Actually
merging inode changes also still needs to be implemented.

13 years agoIn-progress commit of online cleaner.
Michael Vrable [Tue, 7 Dec 2010 05:40:30 +0000 (21:40 -0800)]
In-progress commit of online cleaner.

This adds some code to check the cleaner log when the proxy is launched,
but doesn't yet merge any data.

13 years agoRework the checkpoint record format to include a version vector.
Michael Vrable [Mon, 6 Dec 2010 04:46:49 +0000 (20:46 -0800)]
Rework the checkpoint record format to include a version vector.

This will be used for keeping track of whether we have incorporated changes
made by the cleaner or not, and is a first step (along with having the
clenaer write to the a separate set of logs) to making a functional online
cleaner.

13 years agoTry to avoid accessing profiling objects after they are freed.
Michael Vrable [Thu, 2 Dec 2010 19:39:56 +0000 (11:39 -0800)]
Try to avoid accessing profiling objects after they are freed.

13 years agoWrite profile data if and only if requested.
Michael Vrable [Tue, 30 Nov 2010 04:59:14 +0000 (20:59 -0800)]
Write profile data if and only if requested.

13 years agoAllow profile results to be written to a file.
Michael Vrable [Tue, 30 Nov 2010 04:57:27 +0000 (20:57 -0800)]
Allow profile results to be written to a file.

13 years agoReduce debugging messages in non-verbose mode.
Michael Vrable [Mon, 29 Nov 2010 04:29:33 +0000 (20:29 -0800)]
Reduce debugging messages in non-verbose mode.

13 years agoMore work on request profiling.
Michael Vrable [Mon, 29 Nov 2010 00:21:36 +0000 (16:21 -0800)]
More work on request profiling.

13 years agoAdd locking, thread-ID tracking to profiling.
Michael Vrable [Tue, 23 Nov 2010 23:48:41 +0000 (15:48 -0800)]
Add locking, thread-ID tracking to profiling.

13 years agoTrack requests that initiate inode fetches.
Michael Vrable [Tue, 23 Nov 2010 18:27:31 +0000 (10:27 -0800)]
Track requests that initiate inode fetches.

13 years agoStart to add request time profiling.
Michael Vrable [Tue, 23 Nov 2010 17:53:18 +0000 (09:53 -0800)]
Start to add request time profiling.

Allow a "profile" object to be created when a request is received, and
track the time at which various events occur when handling the request.

13 years agoAdd a lookup_last method to the multi-store backend.
Michael Vrable [Thu, 18 Nov 2010 03:39:00 +0000 (19:39 -0800)]
Add a lookup_last method to the multi-store backend.

13 years agoUpdates to the workload generator to perform a parameter sweep.
Michael Vrable [Wed, 17 Nov 2010 18:12:56 +0000 (10:12 -0800)]
Updates to the workload generator to perform a parameter sweep.

13 years agoImplement very basic grouped fetches of objects from the cloud.
Michael Vrable [Fri, 12 Nov 2010 19:41:46 +0000 (11:41 -0800)]
Implement very basic grouped fetches of objects from the cloud.

Add an item prefetch call, which for now just collects together items that
have been prefetched and when one of them is actually fetched, a range
request is sent that will cover all the objects.  Thus this doesn't improve
the time to fetch the first object (actually it will hurt), it does arrange
to have objects fetched in groups where possible which reduces the total
number of operations and the time to get subsequent objects.

13 years agoAdd a prefetch method. At the moment it is a no-op.
Michael Vrable [Fri, 12 Nov 2010 03:12:47 +0000 (19:12 -0800)]
Add a prefetch method.  At the moment it is a no-op.