elasticsearch

Elasticsearch

Model the index for the queries. Mapping is a contract; text vs keyword and analyzers define what search can do well. RAG (vector + memory): rag.

Stack and scope

Item	Role
Index / mapping	Field schema, analyzers, norms
Query DSL	bool, match, term, range, filters
Aggregations	Facets, metrics, buckets
Bulk API	Batch ingestion
Client	Official client for the project's language (JS/Java/Go/Python/etc.)

OpenSearch shares much of the DSL — preserve the product already adopted. Do not mix APIs from incompatible versions without checking.

Non-negotiable rules

• Define explicit mapping for critical fields; do not rely only on dynamic mapping in production.
• Filters (filter/must_not in bool) for exact criteria — cacheable and without score.
• Full-text queries in text; IDs, status, tags in keyword (or .keyword).
• Deep pagination with search_after / point-in-time; avoid from high on large indices.
• Bulk with conscious size; idempotent retries when the domain allows it.
• Never expose the cluster to the internet without auth; queries coming from the client must be built on the server.

Mapping

{
  "mappings": {
    "properties": {
      "title": { "type": "text", "fields": { "raw": { "type": "keyword" } } },
      "status": { "type": "keyword" },
      "created_at": { "type": "date" },
      "tenant_id": { "type": "keyword" }
    }
  }
}

• Multi-fields (text + keyword) when sorting/aggregating and searching full-text on the same field.
• Nested / join only when really needed — query and memory cost.
• Aliases for blue/green reindex; apps point to the alias, not the raw versioned name.

Query DSL

{
  "query": {
    "bool": {
      "must": [{ "match": { "title": "rust async" } }],
      "filter": [
        { "term": { "tenant_id": "org_123" } },
        { "term": { "status": "published" } }
      ]
    }
  },
  "sort": [{ "created_at": "desc" }, { "_id": "asc" }],
  "size": 20
}

• Multi-tenant: always filter tenant_id (or equivalent) on the server.
• Prefer bool composed to ad hoc scripts on the hot path.
• minimum_should_match, fuzziness, and boosts only with measured relevance criteria.
• Source filtering (_source) — do not bring huge fields unnecessarily.

Ingestion and updates

• Bulk: batches (e.g., a few MB / N docs), periodic flush, handle items[].error.
• Partial updates vs reindex: immutable documents + new version when the model allows it.
• Ingest pipelines to normalize/enrich at the cluster edge.
• Reindex + alias swap for breaking mapping changes.

Aggregations and analytics

• Aggs on keyword / numeric / date fields with an explicit calendar.
• Limit size of terms aggs; use composite to paginate large buckets.
• Do not use aggs as a warehouse substitute without measuring heap/CPU cost.

Performance and ops (app-facing)

• Timeouts and track_total_hits conscious (avoid true always on huge lists).
• Circuit breakers and rejection: handle 429/503 with backoff.
• Monitor p99 of search/index, refresh lag, rejected bulk.
• Shards: start simple; oversharding worsens latency — adjust with evidence.

Anti-patterns

• Free dynamic mapping in production without review
• query_string raw with user input
• from/size for deep pagination
• Leading wildcard (*foo) in hot path
• A “does everything” index without aliases/rollover strategy when volume requires it
• Ignoring multi-tenant isolation in the query

Completion criteria

• Mapping aligned with access patterns
• Queries with tenancy/authz filter
• Safe pagination for the expected volume
• Bulk/ingestion errors handled
• Acceptable relevance/latency on critical paths
• Client and credentials only on the server