Benchmark

Five passes against a server that is already running. Each pass rebuilds the database benchmark. The page shows the mean time.

Run it

Start the listener first. Security stays off, which is the default. Then:

make benchmark

The port defaults to 27018. BENCH_ROWS defaults to 1000 orders. BENCH_LOOKUPS defaults to 100 point reads. The script then asks whether to repeat the same steps on a MongoDB already listening on port 27017. Answering no runs Kova only. Answering yes drops and recreates only the database named benchmark on that server. BENCH_MONGO=1 answers yes. MONGO_PORT chooses the MongoDB port.

make benchmark KOVA_PORT=27019 BENCH_ROWS=1000 BENCH_LOOKUPS=100

The run creates the database benchmark. It does not need that database to exist already. If a previous run left one, that database is removed first. Other databases stay. A release binary is the number worth keeping. Stop the listener that is already up, then start that binary with make run-release. make release only builds it. A debug binary still runs, and the page names which binary answered.

What it measures

Each order is sku, dept, job, sal, qty, and note. job cycles through clerk, analyst, manager, and salesman. note contains that word, so a text search for clerk hits a quarter of the rows. bonus is set on every 10th order. Four departments sit in dept. After the indexes exist, sku 0 is deleted so the point lookups start at 1.

MQL inserts the rows, builds a unique index, a range index, a hash index, a compound index, and a sparse index, then updates, deletes, numbers a field, commits one transaction, and aborts another. It reads by index, scans an unindexed field, groups, looks up dept, writes high_pay with $out, and searches the text view orders_search, including a misspelling.

SQL reads those same rows: a point lookup, a range, a count, a group, an inner join, text search, a fuzzy match, and groups. CREATE TABLE, INSERT, UPDATE, and DELETE are sent as SQL and written to the pages. One update and one delete change a single row. Another update and another delete match every order whose qty is 7.

The second point-lookup pass runs after the first pass has already read the pages. Index, scan, hit, miss, and WAL are the change in serverStatus during that step. Hit, miss, and WAL are Kova only.

How the times are compared

The benchmark runs the whole pass five times. BENCH_REPEATS changes that number. Each pass drops and recreates the database benchmark. The time in the table is the mean of those passes. Rows, index, scan, hit, miss, and WAL are means too. The smaller line under a time is the sample standard deviation: the square root of the sum of squared distances from the mean, after that sum is divided by one less than the number of passes.

When MongoDB ran the same step, each difference is Kova minus Mongo on that pass. If the absolute value of the mean difference is less than or equal to the standard deviation of those differences, the step is similar. A similar step is not a win for either side, and the row is not tinted. A tinted row means Kova's mean was faster by more than that standard deviation. The line under the MQL table counts Kova's wins, MongoDB's wins, the similar steps, and the steps that could not be compared because MongoDB does not have that feature. Those last steps are listed at the end of the MQL table. The SQL table lists Kova only.

This run

The inserted documents are benchmark-data.json. The timings and the process figures are benchmark-result.json.

Process memory and disk

Times

This run repeated each step 5 times. Each pass drops and recreates the database benchmark. The time is the mean of those passes. The smaller line is the sample standard deviation: the square root of the sum of squared distances from the mean, after that sum is divided by one less than the number of passes. A step is similar when MongoDB also finished it and the absolute difference of the two means is less than or equal to the standard deviation of the paired differences, Kova minus Mongo on the same pass. A similar step is not a win. A tinted row is a step Kova won by more than that.

Each paired cell names Kova and Mongo. The last line is Kova minus Mongo in milliseconds, then how many times Kova's mean is Mongo's. An oxide mark sits beside the faster time. The whole row is tinted when Kova was faster by more than the standard deviation. The last line says similar when the difference of the means stayed within the standard deviation of the paired differences, and neither side wins. Under the MQL table, the count is how many steps Kova won, how many MongoDB won, how many were similar, and how many could not be compared because MongoDB does not have that feature. Those steps are listed last. The SQL table lists Kova only. N/A means not available: MongoDB did not run that step, or did not report that counter. Index on Kova is how many times the step chose an index. One range is 1. Index on Mongo is how many index keys the step walked. Scan is how many collection scans the step started, and one walk of the collection is 1. Hit is Kova only: pages read from the buffer pool. A scan of 1 can show many hits, one per page.

MQL

Kova was faster in 9 steps. MongoDB was faster in 11 steps. 4 steps were similar: the difference stayed within the standard deviation. 7 steps could not be compared, because MongoDB does not have that feature.

SQL

An operation name opens the command. A question mark explains that column. A second point lookup runs after the first has already read the pages.