Initial commit of YCSB.

Creation of git repository and inital commit of YCSB.
adally · Apr 19, 2010 · 24efaff · 24efaff
commit 24efaff
Show file tree

Hide file tree

Showing 106 changed files with 24,172 additions and 0 deletions.
diff --git a/LICENSE.txt b/LICENSE.txt
@@ -0,0 +1,163 @@
+Apache License
+Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+"License" shall mean the terms and conditions for use, reproduction,
+and distribution as defined by Sections 1 through 9 of this document.
+
+"Licensor" shall mean the copyright owner or entity authorized by the
+copyright owner that is granting the License.
+
+"Legal Entity" shall mean the union of the acting entity and all other
+entities that control, are controlled by, or are under common control
+with that entity. For the purposes of this definition, "control" means
+(i) the power, direct or indirect, to cause the direction or
+management of such entity, whether by contract or otherwise, or (ii)
+ownership of fifty percent (50%) or more of the outstanding shares, or
+(iii) beneficial ownership of such entity.
+
+"You" (or "Your") shall mean an individual or Legal Entity exercising
+permissions granted by this License.
+
+"Source" form shall mean the preferred form for making modifications,
+including but not limited to software source code, documentation
+source, and configuration files.
+
+"Object" form shall mean any form resulting from mechanical
+transformation or translation of a Source form, including but not
+limited to compiled object code, generated documentation, and
+conversions to other media types.
+
+"Work" shall mean the work of authorship, whether in Source or Object
+form, made available under the License, as indicated by a copyright
+notice that is included in or attached to the work (an example is
+provided in the Appendix below).
+
+"Derivative Works" shall mean any work, whether in Source or Object
+form, that is based on (or derived from) the Work and for which the
+editorial revisions, annotations, elaborations, or other modifications
+represent, as a whole, an original work of authorship. For the
+purposes of this License, Derivative Works shall not include works
+that remain separable from, or merely link (or bind by name) to the
+interfaces of, the Work and Derivative Works thereof.
+
+"Contribution" shall mean any work of authorship, including the
+original version of the Work and any modifications or additions to
+that Work or Derivative Works thereof, that is intentionally submitted
+to Licensor for inclusion in the Work by the copyright owner or by an
+individual or Legal Entity authorized to submit on behalf of the
+copyright owner. For the purposes of this definition, "submitted"
+means any form of electronic, verbal, or written communication sent to
+the Licensor or its representatives, including but not limited to
+communication on electronic mailing lists, source code control
+systems, and issue tracking systems that are managed by, or on behalf
+of, the Licensor for the purpose of discussing and improving the Work,
+but excluding communication that is conspicuously marked or otherwise
+designated in writing by the copyright owner as "Not a Contribution."
+
+"Contributor" shall mean Licensor and any individual or Legal Entity
+on behalf of whom a Contribution has been received by Licensor and
+subsequently incorporated within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of
+   this License, each Contributor hereby grants to You a perpetual,
+   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+   copyright license to reproduce, prepare Derivative Works of,
+   publicly display, publicly perform, sublicense, and distribute the
+   Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of
+   this License, each Contributor hereby grants to You a perpetual,
+   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+   (except as stated in this section) patent license to make, have
+   made, use, offer to sell, sell, import, and otherwise transfer the
+   Work, where such license applies only to those patent claims
+   licensable by such Contributor that are necessarily infringed by
+   their Contribution(s) alone or by combination of their
+   Contribution(s) with the Work to which such Contribution(s) was
+   submitted. If You institute patent litigation against any entity
+   (including a cross-claim or counterclaim in a lawsuit) alleging
+   that the Work or a Contribution incorporated within the Work
+   constitutes direct or contributory patent infringement, then any
+   patent licenses granted to You under this License for that Work
+   shall terminate as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the Work
+   or Derivative Works thereof in any medium, with or without
+   modifications, and in Source or Object form, provided that You meet
+   the following conditions:
+
+(a) You must give any other recipients of the Work or Derivative Works
+a copy of this License; and
+
+(b) You must cause any modified files to carry prominent notices
+stating that You changed the files; and
+
+(c) You must retain, in the Source form of any Derivative Works that
+You distribute, all copyright, patent, trademark, and attribution
+notices from the Source form of the Work, excluding those notices that
+do not pertain to any part of the Derivative Works; and
+
+(d) If the Work includes a "NOTICE" text file as part of its
+distribution, then any Derivative Works that You distribute must
+include a readable copy of the attribution notices contained within
+such NOTICE file, excluding those notices that do not pertain to any
+part of the Derivative Works, in at least one of the following places:
+within a NOTICE text file distributed as part of the Derivative Works;
+within the Source form or documentation, if provided along with the
+Derivative Works; or, within a display generated by the Derivative
+Works, if and wherever such third-party notices normally appear. The
+contents of the NOTICE file are for informational purposes only and do
+not modify the License. You may add Your own attribution notices
+within Derivative Works that You distribute, alongside or as an
+addendum to the NOTICE text from the Work, provided that such
+additional attribution notices cannot be construed as modifying the
+License.
+
+You may add Your own copyright statement to Your modifications and may
+provide additional or different license terms and conditions for use,
+reproduction, or distribution of Your modifications, or for any such
+Derivative Works as a whole, provided Your use, reproduction, and
+distribution of the Work otherwise complies with the conditions stated
+in this License.
+
+5. Submission of Contributions. Unless You explicitly state otherwise,
+   any Contribution intentionally submitted for inclusion in the Work
+   by You to the Licensor shall be under the terms and conditions of
+   this License, without any additional terms or
+   conditions. Notwithstanding the above, nothing herein shall
+   supersede or modify the terms of any separate license agreement you
+   may have executed with Licensor regarding such Contributions.
+
+6. Trademarks. This License does nr work.
+
+To apply the Apache License to your work, attach the following
+boilerplate notice, with the fields enclosed by brackets "[]" replaced
+with your own identifying information. (Don't include the brackets!)
+The text should be enclosed in the appropriate comment syntax for the
+file format. We also recommend that a file or class name and
+description of purpose be included on the same "printed page" as the
+copyright notice for easier identification within third-party
+archives.
+
+Copyright [yyyy] [name of copyright owner]
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+implied.
+
+See the License for the specific language governing permissions and
+limitations under the License.
+
+
diff --git a/NOTICE.txt b/NOTICE.txt
@@ -0,0 +1,9 @@
+=========================================================================
+NOTICE file for use with, and corresponding to Section 4 of,            
+the Apache License, Version 2.0,                                  
+in this case for the YCSB project.                     
+=========================================================================
+
+   This product includes software developed by
+   Yahoo! Inc. (www.yahoo.com)
+   Copyright (c) 2010 Yahoo! Inc.  All rights reserved.
diff --git a/README b/README
@@ -0,0 +1,38 @@
+Yahoo! Cloud System Benchmark (YCSB)
+
+Overview
+========
+
+It is difficult to decide which system is right for your application,
+partially because the features differ between systems, and partially
+because there is not an easy way to compare the performance of one
+system versus another.
+
+The goal of the YCSB project is to develop a framework and common set
+of workloads for evaluating the performance of different "key-value"
+and "cloud" serving stores. The project comprises two things:
+
+* The YCSB Client, an extensible workload generator
+
+* The Core workloads, a set of workload scenarios to be executed by
+  the generator
+
+Although the core workloads provide a well rounded picture of a
+system's performance, the Client is extensible so that you can define
+new and different workloads to examine system aspects, or application
+scenarios, not adequately covered by the core workload. Similarly, the
+Client is extensible to support benchmarking different
+databases. Although we include sample code for benchmarking HBase and
+Cassandra, it is straightforward to write a new interface layer to
+benchmark your favorite database.
+
+A common use of the tool is to benchmark multiple systems and compare
+them. For example, you can install multiple systems on the same
+hardward configuration, and run the same workloads against each
+system. Then you can plot the performance of each system (for example,
+as latency versus throughput curves) to see when one system does
+better than another.
+
+Detailed information about using the benchmark is available in the
+doc/index.html file.
+
diff --git a/build.xml b/build.xml
@@ -0,0 +1,51 @@
+<?xml version="1.0" encoding="UTF-8"?>
+   <project name="ycsb" default="compile" basedir=".">
+
+    <property name="src.dir"     value="src"/>
+    <property name="doc.dir"     value="doc"/>
+    <property name="dbsrc.dir"     value="${src.dir}/com/yahoo/ycsb/db"/>
+    <property name="dblib.dir"   value="${dbsrc.dir}/lib"/>
+
+    <property name="build.dir"   value="build"/>
+    <property name="classes.dir" value="${build.dir}/classes"/>
+    <property name="jar.dir"     value="${build.dir}/jar"/>
+
+
+    <path id="classpath">
+        <fileset dir="${dblib.dir}" includes="**/*.jar"/>
+    </path>
+
+
+      <target name="compile">
+        <mkdir dir="${classes.dir}"/>
+        <javac srcdir="${src.dir}" destdir="${classes.dir}" excludes="com/yahoo/ycsb/db/**" deprecation="on">
+          <compilerarg value="-Xlint:unchecked"/>
+        </javac>
+        <antcall target="makejar"/>
+    </target>
+
+      <target name="dbcompile">
+        <mkdir dir="${classes.dir}"/>
+        <javac srcdir="${dbsrc.dir}" destdir="${classes.dir}" classpathref="classpath" deprecation="on">
+          <compilerarg value="-Xlint:unchecked"/>
+        </javac>
+        <antcall target="makejar"/>
+    </target>
+
+   	<target name ="makejar" description="Create a jar for the YCSB project">
+
+   	         <jar jarfile="build/ycsb.jar" includes="**/*.class" basedir="${classes.dir}"/>
+
+   	      </target>
+    <target name="clean">
+       <delete includeemptydirs="true">
+          <fileset dir="build" includes="**/*"/>
+       </delete>
+    </target>
+
+   <target name="doc">
+      <javadoc sourcepath="${src.dir}" destdir="${doc.dir}/javadoc" packagenames="com.yahoo.ycsb,com.yahoo.ycsb.workloads"/> 
+   </target>
+
+</project>
+
diff --git a/doc/coreproperties.html b/doc/coreproperties.html
@@ -0,0 +1,30 @@
+<HTML>
+<HEAD>
+<TITLE>YCSB - Core workload package properties</TITLE>
+</HEAD>
+<BODY>
+<H1><img src="images/ycsblogo-small.png"> Yahoo! Cloud Serving Benchmark</H1>
+<H3>Version 0.1</H3>
+<HR>
+<A HREF="index.html">Home</A> - <A href="coreworkloads.html">Core workloads</A> - <a href="tipsfaq.html">Tips and FAQ</A>
+<HR>
+<H2>Core workload package properties</h2>
+The property files used with the core workload generator can specify values for the following properties:<p>
+<UL>
+<LI><b>fieldcount</b>: the number of fields in a record (default: 10) 
+<LI><b>fieldlength</b>: the size of each field (default: 100) 
+<LI><b>readallfields</b>: should reads read all fields (true) or just one (false) (default: true) 
+<LI><b>readproportion</b>: what proportion of operations should be reads (default: 0.95) 
+<LI><b>updateproportion</b>: what proportion of operations should be updates (default: 0.05) 
+<LI><b>insertproportion</b>: what proportion of operations should be inserts (default: 0) 
+<LI><b>scanproportion</b>: what proportion of operations should be scans (default: 0) 
+<LI><b>readmodifywriteproportion</b>: what proportion of operations should be read a record, modify it, write it back (default: 0) 
+<LI><b>requestdistribution</b>: what distribution should be used to select the records to operate on - uniform, zipfian or latest (default: uniform) 
+<LI><b>maxscanlength</b>: for scans, what is the maximum number of records to scan (default: 1000) 
+<LI><b>scanlengthdistribution</b>: for scans, what distribution should be used to choose the number of records to scan, for each scan, between 1 and maxscanlength (default: uniform) 
+<LI><b>insertorder</b>: should records be inserted in order by key ("ordered"), or in hashed order ("hashed") (default: hashed) 
+</UL>
+<HR>
+YCSB - Yahoo! Research - Contact [email protected].
+</BODY>
+</HTML>
diff --git a/doc/coreworkloads.html b/doc/coreworkloads.html
@@ -0,0 +1,59 @@
+<HTML>
+<HEAD>
+<TITLE>YCSB - Core workloads</TITLE>
+</HEAD>
+<BODY>
+<H1><img src="images/ycsblogo-small.png"> Yahoo! Cloud Serving Benchmark</H1>
+<H3>Version 0.1</H3>
+<HR>
+<A HREF="index.html">Home</A> - <A href="coreworkloads.html">Core workloads</A> - <a href="tipsfaq.html">Tips and FAQ</A>
+<HR>
+<H2>Core workloads</h2>
+YCSB includes a set of core workloads that define a basic benchmark for cloud systems. Of course, you can define your own workloads, as described <a href="workload.html">here</A>. However,
+the core workloads are a useful first step, and obtaining these benchmark numbers for a variety of different systems would allow you to understand the performance
+tradeoffs of different systems.
+<P>
+The core workloads consist of six different workloads:
+<P>
+<B>Workload A: Update heavy workload</B>
+<P>
+This workload has a mix of 50/50 reads and writes. An application example is a session store recording recent actions.
+<P>
+<B>Workload B: Read mostly workload</B>
+<P>
+This workload has a 95/5 reads/write mix. Application example: photo tagging; add a tag is an update, but most operations are to read tags.
+<P>
+<B>Workload C: Read only</B>
+<P>
+This workload is 100% read. Application example: user profile cache, where profiles are constructed elsewhere (e.g., Hadoop).
+<P>
+<B>Workload D: Read latest workload</B>
+<P>
+In this workload, new records are inserted, and the most recently inserted records are the most popular. Application example: user status updates; people want to read the latest.
+<P>
+<B>Workload E: Short ranges</B>
+<P>
+In this workload, short ranges of records are queried, instead of individual records. Application example: threaded conversations, where each scan is for the posts in a given thread (assumed to be clustered by thread id).
+<P>
+<B>Workload F: Read-modify-write</B>
+<P>
+In this workload, the client will read a record, modify it, and write back the changes. Application example: user database, where user records are read and modified by the user or to record user activity.
+
+<HR>
+<H2>Running the workloads</H2>
+All six workloads have a data set which is similar. Workloads D and E insert records during the test run. Thus, to keep the database size consistent, we recommend the following sequence:
+<OL>
+<LI>Load the database, using workload A's parameter file (workloads/workloada) and the "-load" switch to the client.
+<LI>Run workload A (using workloads/workloada and "-t") for a variety of throughputs.
+<LI>Run workload B (using workloads/workloadb and "-t") for a variety of throughputs.
+<LI>Run workload C (using workloads/workloadc and "-t") for a variety of throughputs. 
+<LI>Run workload F (using workloads/workloadf and "-t") for a variety of throughputs.
+<LI>Run workload D (using workloads/workloadd and "-t") for a variety of throughputs. This workload inserts records, increasing the size of the database.
+<LI>Delete the data in the database.
+<LI>Reload the database, using workload E's parameter file (workloads/workloade) and the "-load switch to the client.
+<LI>Run workload E (using workloads/workloadd and "-t") for a variety of throughputs. This workload inserts records, increasing the size of the database.
+</OL>
+<HR>
+YCSB - Yahoo! Research - Contact [email protected].
+</BODY>
+</HTML>