View on GitHub

Quelea

A Haskell library for declarative programming over eventually consistent store (Cassandra).

Download this project as a .zip file Download this project as a tar.gz file

LWW Register Transactions Benchmark

The aim of this benchmark to compare the performance of various ANSI transaction isolation levels implemented in Quelea (Namely, Read Committed (RC), Monotonic Atomic View (MAV), and Repeatable Read (RR)). For this purpose, we use a register data type with least-writer-wins semantics for writes. We compose transactions with 10 operations - 5 EC writes and 5 EC reads, execute them under different isolation levels in different experiments (one experiment per isolation level).

Building the benchmark

Since the isolation level of transactions in an experiment is fixed, and all transactions are composed of EC reads and writes, contract classification does nothing interesting in this benchmark. It is therefore not necessary to build this benchmark; existing binaries can be used. Nevertheless, if you want to build, navigate to ~/git/quelea/tests/LWW, and run make LWW_txn. Successful build generates one binary - LWW_txn.

Running the Benchmark

First, we run a basic experiment to measure latency and throughput, when a single client sends 1000 successive transation requests (with an inter-request delay of 1ms (1000 μs)). We take the measurements for each of the NoTxn, RC, MAV, and RR cases. The commands to execute, and output generated in our sample runs are shown below. As with the BankAccount benchmark, experiments can be terminated either manually (via CTRL+C), or by setting a hard time limit via --terminateAfter option, which accepts number of seconds as argument.

NoTxn

./LWW_txn --kind Daemon --numRounds 1000 --numThreads 1 --delayReq 1000 --measureLatency --txnKind NoTxn

lww-txn-notxn1

Once you have aggregate latency and throughput data, terminate the experiment by pressing CTRL-C. If you have encountered an error, or if you suspect that execution is not making progress, please refer to the troubleshooting guide for quick fix.

RC

./LWW_txn --kind Daemon --numRounds 1000 --numThreads 1 --delayReq 1000 --measureLatency --txnKind RC

lww-txn-rc1

MAV

./LWW_txn --kind Daemon --numRounds 1000 --numThreads 1 --delayReq 1000 --measureLatency --txnKind MAV

lww-txn-mav1

RR

./LWW_txn --kind Daemon --numRounds 1000 --numThreads 1 --delayReq 1000 --measureLatency --txnKind RR

lww-txn-rr1

The experiment can now be repeated varying the number of client threads (--numThreads) from 2 to 4. Screenshots capturing measurements for sample runs when --numThreads is 4 are shown below:

NoTxn

lww-txn-notxn2

RC

lww-txn-rc2

MAV

lww-txn-mav2

RR

lww-txn-rr2

Observations

As demonstrated in sample runs, experimental results should indicate that throughput depends on the strength of the isolation level of transactions:

  • Througput is highest when no transactional guarantees (including atomicity) are required.
  • Transactions requiring the weakest isolation level (RC) yeild highest throughput.
  • Transactions requiring strongest isolation level (RR) yeild lowest throughput.