The 30 second answer
Analysis turns text into the terms that are indexed and searched: character filters clean the text, a tokenizer splits it, and token filters lowercase, stem, remove stop words or add synonyms. A query only matches if it produces the same terms as the document did, so the analysis at index time and search time must be compatible. Stemming and synonyms raise recall but can hurt precision.
The pipeline. Elasticsearch documents an analyzer as three building blocks: zero or more character filters, exactly one tokenizer, and zero or more token filters. Example for "Women's Running-Shoes":
- Character filter: strip HTML or map characters.
- Tokenizer: split into
Women's,Running,Shoes. - Token filters: lowercase, remove possessives, stem to
runandshoe, maybe add synonyms such assneaker.
Index-time and search-time. The same analyzer usually runs on both sides. If the index stems shoes to shoe but the query does not, shoes never matches. A search analyzer can differ on purpose, for example adding synonyms only at query time so you can change them without reindexing.
Trade-offs.
- Stemming increases recall, but over-stemming merges unrelated words.
- Stop words shrink the index, but removing them breaks phrase queries and short queries.
- Synonyms fix vocabulary gaps but can drift: "jumper" means a sweater in British English and a pinafore dress in American English.
- Languages: analyzers are language-specific, and compound languages such as German need decompounding. Zalando's job ad mentions consolidating multilingual search indices; per-language fields or indices are the usual answer.
Field types. Map free text as text, analysed, for matching, and exact values such as brand, SKU or size as keyword, not analysed, for filters, sorting and aggregations. Multi-fields index the same source both ways.
Debugging. The _analyze API shows exactly which tokens a string produces with a given analyzer, the first thing to run when "this should match but doesn't".