Skip to main content

MDX extraction and stemming

Two indexing/matching improvements help Docsense find the pages readers expect.

Improved MDX extraction​

What it is for​

Many Docusaurus pages mix Markdown with MDX components:

<Tip title="Install the plugin first">
Complete setup before you customize the SearchBar.
</Tip>

A naive stripper that only removes tags would drop Install the plugin first. Docsense pulls textish attributes (such as title, description, alt, aria-label, label, cta) and keeps surrounding prose so those words stay searchable.

It also drops import / export lines and MDX comments so they do not pollute the index.

What still may not index​

ContentIndexed?
Markdown headings and proseYes
Fenced code bodiesYes (as code)
String attributes like title="…"Yes
Text only inside complex runtime components with no string propsOften no. Prefer real Markdown headings for critical terms
tip

If a phrase must be findable, put it in a heading or normal paragraph, not only inside a custom component that renders text from data files.

English stemming​

What it is for​

Readers type different word forms than your docs use:

QueryDocs sayWithout stemmingWith stemming
installing pluginInstall the pluginMay missMatches
streaming audioStream live audioWeakerStronger

Docsense runs a compact Porter stemmer on English tokens during matching. Exact matches still score highest; stem matches score just below exact and above fuzzy typos.

What is not stemmed​

  • Very short tokens
  • Tokens that look like API paths or identifiers (v1/sessions, values with /, #, ., +, _)
  • Non-English / CJK text (not supported yet)

Next steps​