Text search

Find documents by the words in them. Typos, field weights, and recency live on the same query.

Every string field is tokenized when the document is written. The index updates on insert, update, delete, and drop. On startup it is rebuilt by scanning pages, unless you created a view with an on-disk index. The BSON stays in the pages. The text index stores tokens.

The analyzer is fixed in kova.toml under [search]. default splits on punctuation and whitespace, then lowercases. en_stem also stems English words, which finds clerk and clerks together. stopwords lists extra words to drop at index time and at query time. Changing the analyzer later does not rewrite tokens already stored. Create the index with the analyzer you want.

Find with $text

After the sample is loaded, database db has the view emp_search over every employee, with stemming and an index on disk.

db.emp_search.find({
  $text: { $search: "clerk" }
})

A filter can combine $text with ordinary predicates. The text part is scored. The rest of the filter still has to match.

db.emp.find({
  deptno: 20,
  $text: { $search: "analyst" }
})

Wildcards use * for a run of characters and ? for one character: cl*, cl?rk.

A misspelling

$fuzzy is a fraction of each word's length, from 0 to 1. 0.3 lets a short word miss a character.

db.emp_search.find({
  $text: { $search: "clrek", $fuzzy: 0.3 }
})

The same search in SQL, which is what a command-line SQL client prints:

db.runCommand({
  sql: "SELECT ename, job, sal FROM emp_search WHERE TEXT 'clrek' FUZZY 0.3"
})

Without FUZZY, the word has to match a token:

SELECT ename, job, sal FROM emp_search WHERE TEXT 'clerk'

Weights and recency

$boost multiplies the score of a field. $decay pulls newer dates forward. Both are Kova extensions. With warn_on_non_mongo_queries left on, the server logs a warning when a query uses them. The query still runs.

db.articles.find({
  $text: {
    $search: "kiln",
    $boost: { title: 3, body: 1 },
    $decay: { field: "published", halfLifeSeconds: 604800 }
  }
})

halfLifeSeconds is the age at which the date contribution drops by half. The field is a BSON date. $analyzer inside a query is ignored: the pipeline was chosen when the index was built.

Facets

SQL can turn the same search into buckets. Each row is a facet name, a bucket, and a count.

db.runCommand({
  sql: "SELECT facet, bucket, doc_count FROM emp_search WHERE TEXT 'clerk' FACETS job TERMS, sal RANGES (1500), (1500, 3000), (3000)"
})

TERMS groups by the distinct values of a field. RANGES groups a number. (1500) is everything up to 1500. (1500, 3000) is the span between them. (3000) at the end is everything above 3000.

An index that stays on disk

The default text index is in memory and rebuilt from pages on startup. A view with kova.inverted: true writes a Tantivy index under data/partition-N/search/<view>/. It survives a restart, and writes to the source collection keep it current. Field types are learned from the documents that pass the pipeline. The first type seen for a field stays. How to create that view is on Views.