Skip to content

Embed

txi embed runs the embedding pipeline against conversations already in the index. Sync runs this automatically for new and changed conversations; use txi embed to backfill, recover from an interrupted run, or recompute after changing chunking config.

txi embed

By default, only conversations that have no chunks yet are processed. Conversations that already have chunks are skipped.

How it works

For each targeted conversation, the pipeline:

  1. Loads turns and groups them into windows via the active chunking strategy (see Configuration).
  2. Inserts chunk rows into chunks, keyed by content hash.
  3. Reuses any existing embedding whose text_hash matches (cache hit).
  4. Embeds the remaining chunks in batches through the active embedding provider.
  5. Stores vectors in chunk_embeddings and records a row in embedding_runs.

The provider/model is taken from the active config; PydanticAI handles batching, retries, and rate limiting for the supported providers (voyageai, openai, cohere, sentence-transformers).

Options

  • --only PATH — restrict to specific transcript files (matches by conversations.source_path).
  • --force — delete existing chunks and embeddings for targeted conversations before re-running. Use this after editing [chunking] config.
  • --config PATH — override the default config file.

Skipping embedding during sync

Use txi sync --no-embed to ingest structural rows only. This is handy for quick re-syncs when you don't need fresh vectors, or when iterating on a parser.

Switching providers

The embedding provider and model are config-driven. To benchmark a different provider:

  1. Stop and ensure the corpus is clean. The DB is dimension-locked at creation; if the new model has different dimensions you must point at a fresh DB path (paths.db).
  2. Edit config.toml [embedding] (provider, model, dimensions, api_key_env).
  3. Run txi sync (fresh DB) or txi embed --force (existing DB if dimensions match).

A migration command to recompute embeddings in place across dimension/model changes is planned but not yet implemented; the dimension-lock error message points at the future txi reembed command.