bluesky.git
13 years agoWork on a tool for a synthetic read benchmark
Michael Vrable [Wed, 16 Mar 2011 08:32:49 +0000 (01:32 -0700)]
Work on a tool for a synthetic read benchmark

13 years agoAdd an option to disable aggregating reads in the proxy
Michael Vrable [Wed, 16 Mar 2011 02:50:36 +0000 (19:50 -0700)]
Add an option to disable aggregating reads in the proxy

This is for benchmarking purposes.  It uses range requests but does not
delay submitting the reads (to wait for batching opportunities).

13 years agoMinor changes to graphs and benchmark control scripts
Michael Vrable [Wed, 16 Mar 2011 00:15:15 +0000 (17:15 -0700)]
Minor changes to graphs and benchmark control scripts

13 years agoExtract wall clock timestamp of each run when parsing sfsres files
Michael Vrable [Wed, 16 Mar 2011 00:14:38 +0000 (17:14 -0700)]
Extract wall clock timestamp of each run when parsing sfsres files

13 years agoBenchmark results using more concurrent requests and no crypto
Michael Vrable [Tue, 15 Mar 2011 20:13:33 +0000 (13:13 -0700)]
Benchmark results using more concurrent requests and no crypto

13 years agoSPECsfs results from repeating the Azure measurements
Michael Vrable [Tue, 15 Mar 2011 04:37:23 +0000 (21:37 -0700)]
SPECsfs results from repeating the Azure measurements

13 years agoAdd an option to disable cryptographic operations
Michael Vrable [Tue, 15 Mar 2011 04:24:25 +0000 (21:24 -0700)]
Add an option to disable cryptographic operations

This is intended for benchmarking, to measure the overhead of just the
non-cryptographic operations.

13 years agoSpeed up the deletion of all files in an Azure container
Michael Vrable [Tue, 15 Mar 2011 03:56:08 +0000 (20:56 -0700)]
Speed up the deletion of all files in an Azure container

13 years agoNFS trace replay results
Michael Vrable [Tue, 15 Mar 2011 00:35:30 +0000 (17:35 -0700)]
NFS trace replay results

13 years agoMerge git+ssh://vrable1.sysnet.ucsd.edu/scratch/bluesky
Michael Vrable [Mon, 14 Mar 2011 22:00:13 +0000 (15:00 -0700)]
Merge git+ssh://vrable1.sysnet.ucsd.edu/scratch/bluesky

13 years agoRevert "Enable real-time trace replay instead of maximum-speed replay mode."
Michael Vrable [Mon, 14 Mar 2011 21:40:44 +0000 (14:40 -0700)]
Revert "Enable real-time trace replay instead of maximum-speed replay mode."

This reverts commit c0cab09e97b2c3fce55a743a978e4ea81b5ffbdb.

13 years agoCheck in new benchmark results
Michael Vrable [Mon, 14 Mar 2011 19:42:15 +0000 (12:42 -0700)]
Check in new benchmark results

13 years agoPlot updates
Michael Vrable [Mon, 14 Mar 2011 16:48:14 +0000 (09:48 -0700)]
Plot updates

13 years agoCheck in new Azure benchmark results
Michael Vrable [Mon, 14 Mar 2011 03:10:28 +0000 (20:10 -0700)]
Check in new Azure benchmark results

13 years agoSPECsfs results for the non-log-structured case
Michael Vrable [Sun, 13 Mar 2011 17:04:03 +0000 (10:04 -0700)]
SPECsfs results for the non-log-structured case

13 years agoCheck in some of the recent SPECsfs results
Michael Vrable [Sun, 13 Mar 2011 04:34:56 +0000 (20:34 -0800)]
Check in some of the recent SPECsfs results

13 years agoTry to improve support for no-aggregation mode
Michael Vrable [Sun, 13 Mar 2011 04:32:55 +0000 (20:32 -0800)]
Try to improve support for no-aggregation mode

Make it at least a bit more performance-competitive by fetching objects
when prefetching is requested.

13 years agoSPEC benchmark setup using no range requests
Michael Vrable [Sat, 12 Mar 2011 08:47:36 +0000 (00:47 -0800)]
SPEC benchmark setup using no range requests

Also a few graph updates.

13 years agoResults from a SPECsfs run with 8GB of memory
Michael Vrable [Sat, 12 Mar 2011 02:39:56 +0000 (18:39 -0800)]
Results from a SPECsfs run with 8GB of memory

13 years agoOptimize overwrites of the last block of a file
Michael Vrable [Fri, 11 Mar 2011 22:31:26 +0000 (14:31 -0800)]
Optimize overwrites of the last block of a file

13 years agoDrop another debugging message in non-verbose mode
Michael Vrable [Fri, 11 Mar 2011 20:11:58 +0000 (12:11 -0800)]
Drop another debugging message in non-verbose mode

13 years agoReally use the US-West region for the SPECsfs benchmark
Michael Vrable [Fri, 11 Mar 2011 19:42:14 +0000 (11:42 -0800)]
Really use the US-West region for the SPECsfs benchmark

13 years agoUpdate to a warning message to provide more debugging info
Michael Vrable [Fri, 11 Mar 2011 19:34:04 +0000 (11:34 -0800)]
Update to a warning message to provide more debugging info

13 years agoUpdates to scripts for a mixedbench microbenchmarking run
Michael Vrable [Fri, 11 Mar 2011 19:29:59 +0000 (11:29 -0800)]
Updates to scripts for a mixedbench microbenchmarking run

13 years agoCheck in some old data regarding S3 performance
Michael Vrable [Fri, 11 Mar 2011 09:33:43 +0000 (01:33 -0800)]
Check in some old data regarding S3 performance

These experiments ought to be re-run, but use the old data as a placeholder
for now.

13 years agoServer rename for benchmarks
Michael Vrable [Fri, 11 Mar 2011 05:12:29 +0000 (21:12 -0800)]
Server rename for benchmarks

13 years agoTweaks to code that handles read-modify-write on data blocks
Michael Vrable [Fri, 11 Mar 2011 04:58:10 +0000 (20:58 -0800)]
Tweaks to code that handles read-modify-write on data blocks

13 years agoPrepare for microbench re-run with latest code
Michael Vrable [Fri, 11 Mar 2011 04:23:47 +0000 (20:23 -0800)]
Prepare for microbench re-run with latest code

13 years agoReduce verbosity
Michael Vrable [Fri, 11 Mar 2011 01:58:23 +0000 (17:58 -0800)]
Reduce verbosity

13 years agoAdd a script to parse the S3 and NFS traffic statistics from a run
Michael Vrable [Fri, 11 Mar 2011 00:40:58 +0000 (16:40 -0800)]
Add a script to parse the S3 and NFS traffic statistics from a run

13 years agoUpdates to automatic SPECsfs-launching script
Michael Vrable [Fri, 11 Mar 2011 00:40:44 +0000 (16:40 -0800)]
Updates to automatic SPECsfs-launching script

13 years agoReorganize benchmark scripts to support running SPECsfs as well
Michael Vrable [Thu, 10 Mar 2011 23:25:34 +0000 (15:25 -0800)]
Reorganize benchmark scripts to support running SPECsfs as well

13 years agoSome configuration files for possible Postmark benchmark runs
Michael Vrable [Thu, 10 Mar 2011 22:52:44 +0000 (14:52 -0800)]
Some configuration files for possible Postmark benchmark runs

13 years agoScript update
Michael Vrable [Thu, 10 Mar 2011 22:42:19 +0000 (14:42 -0800)]
Script update

13 years agoUpdate some benchmark-running scripts
Michael Vrable [Thu, 10 Mar 2011 22:38:44 +0000 (14:38 -0800)]
Update some benchmark-running scripts

13 years agoAdd runtime options to disable some of the log-structured optimizations
Michael Vrable [Thu, 10 Mar 2011 19:36:44 +0000 (11:36 -0800)]
Add runtime options to disable some of the log-structured optimizations

13 years agoAdd a debug option to dump statistics counters every ten seconds
Michael Vrable [Thu, 10 Mar 2011 19:17:33 +0000 (11:17 -0800)]
Add a debug option to dump statistics counters every ten seconds

Data is appended to a file, so that it can be analyzed later.  Timestamps
are incldued to align results with benchmarks.

13 years agoParse and reorganize some microbenchmark results
Michael Vrable [Thu, 10 Mar 2011 18:45:41 +0000 (10:45 -0800)]
Parse and reorganize some microbenchmark results

13 years agoTrack inter-request delay in TCP analysis
Michael Vrable [Thu, 10 Mar 2011 06:15:41 +0000 (22:15 -0800)]
Track inter-request delay in TCP analysis

When analyzing TCP connections, track the time the connection goes idle
(between the end of one request and start of the next).

13 years agoDisable profiling collection entirely when no data is requested
root [Thu, 10 Mar 2011 01:10:40 +0000 (17:10 -0800)]
Disable profiling collection entirely when no data is requested

13 years agoNote down more kernel compile benchmark results
Michael Vrable [Thu, 10 Mar 2011 00:48:28 +0000 (16:48 -0800)]
Note down more kernel compile benchmark results

13 years agoTweak inode prefetching
root [Wed, 9 Mar 2011 06:12:21 +0000 (22:12 -0800)]
Tweak inode prefetching

13 years agoRework inode prefetching.
Michael Vrable [Wed, 9 Mar 2011 04:57:15 +0000 (20:57 -0800)]
Rework inode prefetching.

Attempt to coalesce range requests for inodes where possible.

13 years agoStart keeping notes on tarball unpack, checksum, kernel compile, etc.
Michael Vrable [Wed, 9 Mar 2011 04:41:06 +0000 (20:41 -0800)]
Start keeping notes on tarball unpack, checksum, kernel compile, etc.

13 years agoPostmark results summary
Michael Vrable [Tue, 8 Mar 2011 23:35:31 +0000 (15:35 -0800)]
Postmark results summary

13 years agoMinor benchmark additions/changes
Michael Vrable [Tue, 8 Mar 2011 23:34:22 +0000 (15:34 -0800)]
Minor benchmark additions/changes

13 years agoInfrastructure to generate PDF figures for the paper
Michael Vrable [Tue, 8 Mar 2011 23:28:49 +0000 (15:28 -0800)]
Infrastructure to generate PDF figures for the paper

13 years agoCommit SPECsfs results for an NFS server running in EC2
Michael Vrable [Tue, 8 Mar 2011 20:40:46 +0000 (12:40 -0800)]
Commit SPECsfs results for an NFS server running in EC2

13 years agoRe-run some results with newer BlueSky code
Michael Vrable [Mon, 7 Mar 2011 18:28:08 +0000 (10:28 -0800)]
Re-run some results with newer BlueSky code

13 years agoOptimize complete overwrites of file system blocks
Michael Vrable [Mon, 7 Mar 2011 00:22:30 +0000 (16:22 -0800)]
Optimize complete overwrites of file system blocks

Do not fetch the old data in this case, since it doesn't matter what it
was.

13 years agoCheck in some more benchmark results for BlueSky/S3-west
Michael Vrable [Sun, 6 Mar 2011 23:31:03 +0000 (15:31 -0800)]
Check in some more benchmark results for BlueSky/S3-west

13 years agoUpdate a results README file
Michael Vrable [Sun, 6 Mar 2011 23:29:07 +0000 (15:29 -0800)]
Update a results README file

13 years agoScript for parsing sfsres files to extract more detailed latency values
Michael Vrable [Sun, 6 Mar 2011 23:27:51 +0000 (15:27 -0800)]
Script for parsing sfsres files to extract more detailed latency values

13 years agoBenchmark results without range requests.
Michael Vrable [Sun, 6 Mar 2011 04:52:49 +0000 (20:52 -0800)]
Benchmark results without range requests.

13 years agoReduce verbosity of debugging of disk cache contents
root [Fri, 4 Mar 2011 22:20:25 +0000 (14:20 -0800)]
Reduce verbosity of debugging of disk cache contents

13 years agoUse virtual hosts for specifying S3 buckets
root [Fri, 4 Mar 2011 22:19:30 +0000 (14:19 -0800)]
Use virtual hosts for specifying S3 buckets

This is needed to get requests directed to the right place when using
different locations (US-West vs. US-California, for example).

13 years agoAdd more benchmark results
Michael Vrable [Fri, 4 Mar 2011 01:36:48 +0000 (17:36 -0800)]
Add more benchmark results

13 years agoMore memory usage stats
Michael Vrable [Fri, 4 Mar 2011 00:10:11 +0000 (16:10 -0800)]
More memory usage stats

13 years agoMeasure effect of read latencies vs. connection idleness
Michael Vrable [Fri, 4 Mar 2011 00:07:42 +0000 (16:07 -0800)]
Measure effect of read latencies vs. connection idleness

13 years agoAdd more memory overhead data
Michael Vrable [Thu, 3 Mar 2011 21:17:37 +0000 (13:17 -0800)]
Add more memory overhead data

13 years agoUpdate memory usage plot script
Michael Vrable [Thu, 3 Mar 2011 21:15:39 +0000 (13:15 -0800)]
Update memory usage plot script

13 years agoMemory usage stats from another benchmark run (on the real proxy)
Michael Vrable [Thu, 3 Mar 2011 16:37:41 +0000 (08:37 -0800)]
Memory usage stats from another benchmark run (on the real proxy)

13 years agoStatistics on memory usage during a test benchmark run
Michael Vrable [Thu, 3 Mar 2011 16:35:50 +0000 (08:35 -0800)]
Statistics on memory usage during a test benchmark run

13 years agoInclude S3-US-West results in graphs
Michael Vrable [Thu, 3 Mar 2011 00:02:45 +0000 (16:02 -0800)]
Include S3-US-West results in graphs

13 years agoLimit the number of concurrent log uploads to the cloud
Michael Vrable [Wed, 2 Mar 2011 22:20:46 +0000 (14:20 -0800)]
Limit the number of concurrent log uploads to the cloud

This is mostly aimed at limiting memory usage, since each cloud log segment
is stored entirely in memory before it is uploaded, so a very large number
of parallel uploads could require allocating a large amount of memory.

13 years agoMove encryption of cloud log segments into background threads.
Michael Vrable [Wed, 2 Mar 2011 01:09:29 +0000 (17:09 -0800)]
Move encryption of cloud log segments into background threads.

Crypto operations looked to be a CPU bottleneck, but using separate threads
will at least allow multiple log segments to be encrypted in parallel on
separate CPU cores.

13 years agoUpdates to run microbenchmarks against S3 us-west.
Michael Vrable [Tue, 1 Mar 2011 07:23:39 +0000 (23:23 -0800)]
Updates to run microbenchmarks against S3 us-west.

13 years agoMerge git+ssh://c09-44.sysnet.ucsd.edu/scratch/bluesky
Michael Vrable [Tue, 1 Mar 2011 04:26:24 +0000 (20:26 -0800)]
Merge git+ssh://c09-44.sysnet.ucsd.edu/scratch/bluesky

13 years agoCheck in results using S3 US-west location.
Michael Vrable [Tue, 1 Mar 2011 04:25:59 +0000 (20:25 -0800)]
Check in results using S3 US-west location.

13 years agoMerge git+ssh://idril.vpn.vrable.net/home/mvrable/local/bluesky
Michael Vrable [Tue, 1 Mar 2011 00:28:12 +0000 (16:28 -0800)]
Merge git+ssh://idril.vpn.vrable.net/home/mvrable/local/bluesky

13 years agoPrint out Azure objects as they are deleted.
Michael Vrable [Tue, 1 Mar 2011 00:27:46 +0000 (16:27 -0800)]
Print out Azure objects as they are deleted.

13 years agoConfig file for SPECsfs CIFS testing.
Michael Vrable [Mon, 28 Feb 2011 19:34:24 +0000 (11:34 -0800)]
Config file for SPECsfs CIFS testing.

13 years agoResults from SPECsfs runs against Windows Azure.
Michael Vrable [Mon, 28 Feb 2011 19:33:09 +0000 (11:33 -0800)]
Results from SPECsfs runs against Windows Azure.

13 years agoPlots for the CIFS SPECsfs results.
Michael Vrable [Mon, 28 Feb 2011 16:45:26 +0000 (08:45 -0800)]
Plots for the CIFS SPECsfs results.

13 years agoMerge git+ssh://c09-45.sysnet.ucsd.edu/scratch/bluesky
Michael Vrable [Mon, 28 Feb 2011 16:32:41 +0000 (08:32 -0800)]
Merge git+ssh://c09-45.sysnet.ucsd.edu/scratch/bluesky

13 years agoMake the azure test python script clean up files from BlueSky runs.
Michael Vrable [Mon, 28 Feb 2011 16:31:51 +0000 (08:31 -0800)]
Make the azure test python script clean up files from BlueSky runs.

13 years agoMerge git+ssh://c09-44.sysnet.ucsd.edu/scratch/bluesky
Michael Vrable [Mon, 28 Feb 2011 16:25:32 +0000 (08:25 -0800)]
Merge git+ssh://c09-44.sysnet.ucsd.edu/scratch/bluesky

13 years agoCheck in some SPECsfs runs for CIFS.
Michael Vrable [Mon, 28 Feb 2011 16:25:13 +0000 (08:25 -0800)]
Check in some SPECsfs runs for CIFS.

13 years agoTurn on profiling in microbenchmarking.
root [Mon, 28 Feb 2011 13:33:13 +0000 (05:33 -0800)]
Turn on profiling in microbenchmarking.

13 years agoCheck in README and sample plots for some of the new data.
Michael Vrable [Mon, 28 Feb 2011 06:39:43 +0000 (22:39 -0800)]
Check in README and sample plots for some of the new data.

13 years agoChecking in some results with a smaller, slower disk.
Michael Vrable [Mon, 28 Feb 2011 06:19:38 +0000 (22:19 -0800)]
Checking in some results with a smaller, slower disk.

13 years agoParameters for investigating latency anomalies, and results.
Michael Vrable [Mon, 28 Feb 2011 04:39:55 +0000 (20:39 -0800)]
Parameters for investigating latency anomalies, and results.

13 years agoCheck in some detailed timing results to investigate strange latency vs.
Michael Vrable [Mon, 28 Feb 2011 04:37:58 +0000 (20:37 -0800)]
Check in some detailed timing results to investigate strange latency vs.
requested load figures.

13 years agoMerge branch 'master' of git+ssh://niniel.sysnet.ucsd.edu/home/mvrable/local/bluesky
root [Sun, 27 Feb 2011 22:26:07 +0000 (14:26 -0800)]
Merge branch 'master' of git+ssh://niniel.sysnet.ucsd.edu/home/mvrable/local/bluesky

13 years agoAdd some proxy statistics.
Michael Vrable [Sun, 27 Feb 2011 22:25:41 +0000 (14:25 -0800)]
Add some proxy statistics.

13 years agoSwitch disk used for storing data during benchmarks.
root [Sun, 27 Feb 2011 21:50:53 +0000 (13:50 -0800)]
Switch disk used for storing data during benchmarks.

13 years agoCheck in some recent SPECsfs benchmark results.
Michael Vrable [Sun, 27 Feb 2011 21:35:36 +0000 (13:35 -0800)]
Check in some recent SPECsfs benchmark results.

13 years agoMake NFS rsize/wsize configurable in the microbenchmark runs.
Michael Vrable [Sun, 27 Feb 2011 20:46:31 +0000 (12:46 -0800)]
Make NFS rsize/wsize configurable in the microbenchmark runs.

13 years agoFix byte range request syntax.
Michael Vrable [Fri, 25 Feb 2011 01:07:19 +0000 (17:07 -0800)]
Fix byte range request syntax.

13 years agoDelete dead code.
Michael Vrable [Fri, 25 Feb 2011 00:53:24 +0000 (16:53 -0800)]
Delete dead code.

13 years agoAdd a CURL connection pool to the Azure backend.
Michael Vrable [Fri, 25 Feb 2011 00:52:46 +0000 (16:52 -0800)]
Add a CURL connection pool to the Azure backend.

This will allow HTTP connections to be re-used.

13 years agoAdd byte range support to the Azure backend.
Michael Vrable [Fri, 25 Feb 2011 00:40:06 +0000 (16:40 -0800)]
Add byte range support to the Azure backend.

13 years agoAdd Windows Azure support to BlueSky.
Michael Vrable [Thu, 24 Feb 2011 23:54:18 +0000 (15:54 -0800)]
Add Windows Azure support to BlueSky.

This is not much tested but seems to work at a basic level.

13 years agoCheck in some minor changes to microbenchmark runs.
Michael Vrable [Wed, 23 Feb 2011 23:36:28 +0000 (15:36 -0800)]
Check in some minor changes to microbenchmark runs.

13 years agoBuild a quick storage library for accessing Windows Azure.
Michael Vrable [Wed, 23 Feb 2011 23:25:48 +0000 (15:25 -0800)]
Build a quick storage library for accessing Windows Azure.

This will be used by the Python cleaner, but that is not yet hooked up.

13 years agoRun both read, write, and mixed workloads.
Michael Vrable [Mon, 21 Feb 2011 04:41:52 +0000 (20:41 -0800)]
Run both read, write, and mixed workloads.

13 years agoSet up for benchmark runs with small reads/writes within larger files.
Michael Vrable [Mon, 21 Feb 2011 04:40:21 +0000 (20:40 -0800)]
Set up for benchmark runs with small reads/writes within larger files.

13 years agoMore reworking of microbenchmark handling of reads/writes and file layout.
Michael Vrable [Sun, 20 Feb 2011 04:28:46 +0000 (20:28 -0800)]
More reworking of microbenchmark handling of reads/writes and file layout.

13 years agoChange how mixed read/write workloads are done in mixedbench
Michael Vrable [Sun, 20 Feb 2011 03:30:49 +0000 (19:30 -0800)]
Change how mixed read/write workloads are done in mixedbench

Split reads and writes into separate threads, instead of mixing operations
within a single thread.  This does not allow arbitrary mixes any longer,
since the ratios are quantized based on the number of total threads.

13 years agoCommit script to parse microbenchmark results.
Michael Vrable [Mon, 14 Feb 2011 23:06:03 +0000 (15:06 -0800)]
Commit script to parse microbenchmark results.