memory-os/SKILL.md

---
name: memory-os
description: "Qdrant-based RAG pipeline for wiki/note vault ingestion — sync, parse, chunk, embed (dense: bge-m3 + sparse: BM25), store in Qdrant (1024d COSINE), and auto-inject retrieved context into Hermes responses."
version: 1.8.0
author: Hermes Agent
metadata:
  hermes:
    tags: [rag, qdrant, vector-search, wiki-ingest, memory, embedding, bm25, ollama, bge-m3, multilingual]
    category: research
    related_skills: [llm-wiki, obsidian, llama-cpp]
    sources:
      - references/chunking-implementation.md
      - references/dimension-mismatch-s3fs-blockers.md
      - references/ingest-debug-session.md
      - references/verify-threshold-bug.md
      - references/webdav-migration.md
      - references/search-api-payload-fields.md
---

# Memory OS — Wiki RAG Pipeline

A production RAG pipeline that continuously syncs an Obsidian vault (or any markdown wiki) into Qdrant for semantic + BM25 retrieval, and auto-injects relevant context into every Hermes response.

**Architecture:**

```
Obsidian vault (WebDAV: /mnt/yandex-disk/obsidian/mozg/ — Yandex Disk davfs2)
    ↓ (sync_obsidian_to_wiki.py — every 30 min via Hermes cron, job_id: 2346a68b601d)
wiki-raw/ directory (80 .md files, 73 ingested + 7 empty skipped)
    ↓ (wiki_continuous_ingest.py — ARQ worker)
Parse → Chunk → Embed (dense: bge-m3 1024d, sparse: BM25)
    ↓
Qdrant collection "knowledge_base" (dense 1024d COSINE + sparse "sparse")
    ↓
context_enhancer.py (CLI search tool — 4-level fallback cascade)
    ↓
icarus/hooks.py (auto-inject into Hermes responses)
```

## When This Skill Activates

When the user:
- Asks about Memory OS setup, debugging, or configuration
- Reports that sync_obsidian_to_wiki.py or the ingress pipeline isn't running
- Asks about alternative Obsidian vault mounts (WebDAV vs s3fs) for the pipeline
- Reports Qdrant returning no results or empty searches
- Asks to reingest, reset, or rebuild the search index
- Reports embedding mismatches or search quality issues
- Deploys or updates the ingest pipeline
- Asks about the wiki ingestion cron job

## Obsidian Vault Source Path

The pipeline picks up notes from an Obsidian vault. The canonical source is:

| Mount | Path | Status |
|---|---|---|
| **WebDAV (Yandex Disk)** | `/mnt/yandex-disk/obsidian/mozg/` | ✅ Active — davfs2, file_mode=600 |
| s3fs (Garage) | `/opt/hermes/obsidian-vault/` | ❌ Defunct — bucket empty at /etc/fstab entry |

**READ-ONLY guarantee:** sync_obsidian_to_wiki.py copies FROM vault TO wiki target. It NEVER modifies or deletes source files. The vault is mounted via davfs2 which is inherently read-only by filesystem mode, and the code only calls `copy2()` and `unlink()` on the target path.

**s3fs/obsidian-vault defunct — do NOT use `/opt/hermes/obsidian-vault/`.** The old s3fs mount is empty. The canonical vault is WebDAV at `/mnt/yandex-disk/obsidian/mozg/`. Any cron job or script referencing `/opt/hermes/obsidian-vault/` is stale — redirect to the WebDAV path. The stale mount has 8 root-owned leftover files; they are NOT the real vault.

**WebDAV switch-over (2026-07-16):** The s3fs mount at `s3.nixg.ru` was empty. Changed `OBSIDIAN_VAULT` from `/opt/hermes/obsidian-vault` to `/mnt/yandex-disk/obsidian/mozg/`. First sync: 74 files in 44s, 0 errors.

## Hermes Cron Setup

The sync is a managed Hermes cron job:

| Property | Value |
|---|---|
| Job ID | `2346a68b601d` |
| Schedule | `*/30 * * * *` |
| Runner | Wrapper script at `~/.hermes/scripts/obsidian-sync.sh` |
| no_agent | `true` — pure shell execution, no LLM |
| Script | `cd /opt/hermes/memory-os && .venv/bin/python3 scripts/sync_obsidian_to_wiki.py` |

The wrapper exists because Hermes cron's script field requires a path under `~/.hermes/scripts/`. It calls `.venv/bin/python3 scripts/sync_obsidian_to_wiki.py`.

**sync → ingest chain:** `sync_obsidian_to_wiki.py` internally calls `wiki_continuous_ingest.py` when files change. Must use `.venv/bin/python3` (system python3 lacks `arq`). Fixed 2026-07-16.

**no_agent optimization:** The cron job uses `no_agent=true` — pure script execution, no LLM reasoning needed. Avoids wasting tokens on every 30-min tick. Verified 2026-07-19.

## WebDAV vs s3fs Comparison

| Aspect | WebDAV (davfs2) | s3fs (Garage) |
|---|---|---|
| Works | ✅ Active | ❌ Empty mount |
| POSIX | ⚠️ `file_mode=600` | ✅ Normal perms |
| Safety | ✅ Read-only by fuse | ✅ Script guard |
| Config | `webdav.yandex.ru` at `/mnt/yandex-disk` | `s3.nixg.ru` at `/opt/hermes/obsidian-vault` |

| Component | Path | Role |
|---|---|---|
| State tracker | `/opt/hermes/email/state/wiki_ingest_state.json` | Tracks which files were queued |
| Failures log | `/opt/hermes/email/state/wiki_ingest_failures.json` | Records ingest errors |
| Sync state | `/opt/hermes/email/state/obsidian_sync_state.json` | Tracks Obsidian vault sync state (mtime/size per file) |
| Sync script | `scripts/sync_obsidian_to_wiki.py` | Syncs Obsidian vault → wiki-raw/ |
| Ingest worker | `scripts/wiki_continuous_ingest.py` | ARQ worker: parse → chunk → embed → Qdrant |
| **Bulk ingest** | `scripts/bulk_wiki_ingest_ollama.py` | One-shot re-index: reads all .md, chunks (headings→paragraphs→words), embeds via Ollama bge-m3, stores in Qdrant. Replaces manual pipeline reset. |
| Search CLI | `scripts/context_enhancer.py` | 4-level fallback search (hybrid → dense → lexical → SQLite) |
| Hermes hook | `icarus/hooks.py` | Auto-injects Qdrant context into responses |
| Docker env | `docker/.env` | Settings for Ollama, Qdrant, Redis |

## Key Environment Variables

```bash
# Qdrant
QDRANT_URL=http://localhost:6333
QDRANT_COLLECTION=knowledge_base

# Embedding (dense — local Ollama bge-m3 1024d)
OLLAMA_EMBEDDING_URL=http://localhost:11434
OLLAMA_EMBEDDING_MODEL=bge-m3:latest

# Embedding dimension (must match collection)
EMBEDDING_DIMS=1024

# Embedding (sparse — BM25 via FastEmbed)
# Uses subprocess to ai-lab venv at _FASTEMBED_PYTHON path in context_enhancer.py

# Redis (for ARQ queue)
REDIS_HOST=127.0.0.1
REDIS_PORT=6379
REDIS_PASSWORD=<password>
```

## Key Commands

```bash
# Reset everything (re-ingest from scratch)
cd /opt/hermes/memory-os

# 1. Clear state so all files are considered new
echo '{}' > /opt/hermes/email/state/wiki_ingest_state.json

# 2. Drop and recreate Qdrant collection (1024d COSINE + sparse)
python3 -c "
from qdrant_client import QdrantClient, models
c = QdrantClient('http://localhost:6333')
c.delete_collection('knowledge_base')
c.create_collection(
    collection_name='knowledge_base',
    vectors_config=models.VectorParams(size=1024, distance=models.Distance.COSINE),
    sparse_vectors_config={'sparse': models.SparseVectorParams(
        index=models.SparseIndexParams(on_disk=False, full_scan_threshold=10000)
    )}
)
print('Created 1024d COSINE + sparse')
"

# 3. Bulk re-index all files (with chunking)
/opt/hermes/memory-os/.venv/bin/python3 scripts/bulk_wiki_ingest_ollama.py

# 4. Search
python3 scripts/context_enhancer.py "your query" --top-k 5 --format markdown
python3 scripts/context_enhancer.py "your query" --top-k 10 --threshold 0.35 --format compact
```

## Structure of Qdrant Collection

The collection `knowledge_base` must have:
- **dense** vectors: **1024 dimensions**, COSINE distance (bge-m3 via Ollama)
- **sparse** vectors: BM25 (named `sparse`, NOT `bm25`)

```python
models.VectorParams(size=1024, distance=models.Distance.COSINE)
models.SparseVectorParams(
    index=models.SparseIndexParams(on_disk=False, full_scan_threshold=10000)
)
```

The sparse vector name MUST be `"sparse"` — this matches what the ingest worker sends. Creating it as `"bm25"` causes a 400 Bad Request error on every write.

**Historical note:** This was previously 768d (nomic-embed-text). Migrated to 1024d (bge-m3) on 2026-07-16 for multilingual support. See `references/chunking-implementation.md` for the current chunking approach.

## Debugging Checklist (when search returns nothing or poor results)

### 1. Check Qdrant has points
```bash
python3 -c "
from qdrant_client import QdrantClient
c = QdrantClient('http://localhost:6333')
info = c.get_collection('knowledge_base')
print(f'Points: {info.points_count}')
print(f'Status: {info.status}')
"
```
If points_count = 0, the collection was never ingested or was dropped.

### 2. Check sparse vector name matches
```bash
python3 -c "
from qdrant_client import QdrantClient
c = QdrantClient('http://localhost:6333')
info = c.get_collection('knowledge_base')
print('Sparse vectors:', info.config.params.sparse_vectors)
"
```
If sparse vectors key is `"bm25"` but the worker sends `"sparse"`, you get 400 errors. Recreate the collection with the correct name.

### 3. Check dense embedding model matches
The `context_enhancer.py`'s `embed_query()` must use the SAME model that was used during ingest. If OpenRouter was used for search but local Ollama for ingest (or vice versa), dense vectors won't match and search degrades to lexical fallback.

Check what the search client uses vs what was used during ingest:
```bash
# Search client config
grep -n "OLLAMA_EMBEDDING_MODEL\|EMBEDDING_MODEL" scripts/context_enhancer.py

# Ingest config (docker/.env or wiki_continuous_ingest.py)
grep -n "embedding\|model" docker/.env
grep -rn "embed\|nomic" scripts/wiki_continuous_ingest.py
```

### 4. Check FastEmbed BM25 works (sparse engine)
```bash
python3 -c "
from fastembed.sparse import SparseTextEmbedding
model = SparseTextEmbedding(model_name='Qdrant/bm25')
sparse = list(model.embed(['test query']))[0]
print(f'indices: {len(sparse.indices)}, values: {len(sparse.values)}')
"
```
If this fails, the FastEmbed subprocess in context_enhancer.py needs the `FASTEMBED_SITEPKGS` env var.

### 5. Check worker logs
```bash
docker logs docker-worker-1 --tail 50
```
Look for 400 errors (bad sparse name), DNS failures (ollama unreachable), or timeouts.

### 6. Check Docker networking
After every `docker compose up -d`, the worker container loses secondary networks:
```bash
docker network connect ollama_default docker-worker-1
docker inspect docker-worker-1 | jq '.[].NetworkSettings.Networks | keys'
```

### 7. Check ARQ queue isn't stale
```bash
redis-cli -a <password> keys 'arq:*' | head -20
redis-cli -a <password> llen 'arq:queue:health-check'
```
If there are stale jobs in the queue, clear them with:
```bash
redis-cli -a <password> del 'arq:queue:health-check'
```

### 8. Test dense embedding directly
```bash
curl -s http://localhost:11434/api/embeddings \
  -d '{"model":"bge-m3:latest","prompt":"test query"}' | \
  python3 -c "import sys, json; d=json.load(sys.stdin); print(f'Dims: {len(d[\"embedding\"])}')"
# Should print "Dims: 1024"
```

### 9. Check collection dimensions match `.env` embedding settings (400 Bad Request)
When you get a `400 Client Error: Bad Request for url: http://localhost:6333/collections/knowledge_base/points/query`, the most likely cause is the collection was created with one embedding dimension (e.g. 768d for nomic-embed-text) but `.env` now points to a different model with different dimensions (e.g. 1024d for bge-m3).

Cross-check:
```bash
# What the collection expects:
python3 -c "
from qdrant_client import QdrantClient
c = QdrantClient('http://localhost:6333')
info = c.get_collection('knowledge_base')
print(f'Collection dims: {info.config.params.vectors.size}')
print(f'Distance: {info.config.params.vectors.distance}')
"

# What .env says:
grep -E 'EMBEDDING_DIMS|EMBEDDING_MODEL' /opt/hermes/memory-os/.env
```

If they don't match, you must either:
- (a) **Recreate the collection** with the correct dimension (destructive — lose all points). Use the current model's dimension.
- (b) **Change `.env`** to match the collection (switch back to the old model).
- (c) **Create a second collection** for the new dimension, keep the old one.

> **Current config (2026-07-16):** bge-m3 1024d COSINE. See `references/chunking-implementation.md` for the chunking approach.

> See also `references/dimension-mismatch-s3fs-blockers.md` for full reproduction transcript.

## Pitfalls

- **All state files live under `/opt/hermes/email/state/`, NOT `~/.hermes/`.** Three scripts reference state files — sync_obsidian_to_wiki.py, wiki_continuous_ingest.py, dlq_manager.py. All use `/opt/hermes/email/state/`. If any script still uses `Path.home() / ".hermes"` or `os.path.expanduser("~/.hermes")` for state paths, patch it. The old paths at `~/.hermes/obsidian_sync_state.json`, `~/.hermes/wiki_ingest_state.json`, and `~/.hermes/wiki_ingest_failures.json` are stale.
- **Missing wrapper script blocks cron execution silently.** The Hermes cron job `obsidian-sync` (job_id `2346a68b601d`) runs `~/.hermes/scripts/obsidian-sync.sh`. If this file is missing, the cron tick produces no output and no error — the job just does nothing. On 2026-07-19 the script was documented in the skill but never created on disk. **Fix:** `mkdir -p ~/.hermes/scripts`; write the wrapper; `chmod +x`. Then switch the job to `no_agent=true` so pure script execution doesn't burn LLM tokens.
- **`~` expansion differs between Python and the shell when running from `/opt/hermes/memory-os/`.
- **`~` expansion differs between Python and the shell when running from `/opt/hermes/memory-os/`.** `context_enhancer.py` uses `os.path.expanduser("~/.hermes/state.db")` for `LINEAGE_DB`. When the script runs via `.venv/bin/python` from `/opt/hermes/memory-os/`, `~` resolves to `/opt/hermes/` — NOT `/home/estorozhenko/`. This means `LINEAGE_DB` becomes `/opt/hermes/.hermes/state.db` — a different Hermes state DB that doesn't have the `lineage` table. The `register_lineage()` call then silently fails with `[LINEAGE-WARNING] Failed to register lineage: no such table: lineage`. **Fix:** create the `lineage` table in `/opt/hermes/.hermes/state.db` too, or set `STATE_DB_PATH=/home/estorozhenko/.hermes/state.db` in `.env` or process environment. Symptom: search results appear but `[LINEAGE-WARNING]` is printed to stderr.
- **Fabric directory (`~/fabric/`) does not exist by default.** The Icarus plugin writes fabric entries to `FABRIC_DIR = Path.home() / "fabric"` (line 19 of `state.py`). This directory is never auto-created before `write_entry()` is called (it calls `mkdir(parents=True, exist_ok=True)` in `write_entry()` itself, so writes succeed, but `read_recent()` and `read_cross_agent()` return empty if `FABRIC_DIR.exists()` is False). **Fix:** `mkdir -p ~/fabric/` to ensure the directory exists before any Icarus session starts. Without this, the first session after enabling Icarus will have no `[fabric]` context injected.
- **`HERMES_AGENT_NAME` is not set.** If `HERMES_AGENT_NAME` is missing from `.env` or environment, `state.AGENT_NAME` is empty string, and the logger warns: `"icarus: HERMES_AGENT_NAME not set — fabric entries will use agent=\"agent\""`. This means all fabric entries are tagged with `agent: agent` instead of a meaningful name. **Fix:** add `HERMES_AGENT_NAME=hermes` (or another name) to `/opt/hermes/.hermes/config.yaml` or `/opt/hermes/.hermes/.env`. The name is used in fabric filenames (`agent-entry_type-slug-id.md`) and in multi-agent deployments.
- **Fabric entries are NOT injected until mid-session — they only appear starting from turn 2+.** `pre_llm_call` in `hooks.py` calls `state.recall()` which reads fabric entries. On the first turn of a session, `is_first_turn=True` additionally triggers `_search_facts()` for durable facts. Fabric entries from a previous session are injected as `[fabric]` blocks. If you started a new session and see no `[fabric]` block, the directory may not exist, or no entries were written by the previous session (because `on_session_end` writes them, and if the session was closed abnormally, the hook never fired).
- **`memory_store.db` (durable facts) is empty by default.** The `facts` table in `~/.hermes/memory_store.db` starts with 0 rows. It's populated only by explicit `memory` tool calls. A `_search_facts()` call always returns empty until the first fact is saved. This is normal — no fix needed.
- **Session history FTS5 (`_search_sessions`) needs a session_id exclusion to avoid self-referencing.** The `_search_sessions()` function in `hooks.py` (line 331) filters out the current session by `session_id` to avoid injecting the current conversation's own messages. If this filter breaks, the agent will see its own replies from the same session and recursively inject them. The filter uses `WHERE session_id != ?` in the SQL.
- **Session history injection produces `[sessions]` blocks, NOT `[fabric]` or `[qdrant]`.** The `pre_llm_call` hook injects four separate blocks: `[fabric]` (from fabric entries), `[qdrant]` (from Qdrant vault search), `[sessions]` (from FTS5 session history), and `[facts]` (from durable facts). Each has its own dedup set (`_injected_fabric`, `_injected_qdrant`, `_injected_sessions`, `_injected_facts`). If you see a `[sessions]` block, it came from FTS5, not Qdrant.

- **CRITICAL: icarus/hooks.py threshold must be 0.30, not 0.55.** `_search_qdrant()` passes `threshold=0.55` to `search_with_fallback()`. RRF fusion (hybrid dense+sparse) returns scores 0.33-0.50 even for good matches — the old gate silently filters EVERYTHING. This is fundamentally different from dense-only search which returns 0.90+ for the same queries. A threshold of 0.35 still blocks some RRF results (0.33 scores). The safe floor is 0.30. Verified 2026-07-16: hybrid search with threshold=0.55 returned 0 results for every query tested; 0.30 returned 4 results. Without this fix, `search_with_fallback` hits `level="none"`, cascades into SQLite fallback (`[CE-FALLBACK] SQLite search failed: no such table: lineage`), pollutes logs, and Icarus injects nothing on every turn — the injection code runs, finds nothing, and returns silently.
- **`FASTEMBED_VENV` must be set in `.env` or the sparse BM25 subprocess uses system python without fastembed.** If `context_enhancer.py` doesn't find `FASTEMBED_VENV` in env, it falls back to `sys.executable` (system python). The `.env` must contain `FASTEMBED_VENV=/opt/hermes/memory-os/.venv/bin/python3` and `FASTEMBED_SITEPKGS=/opt/hermes/memory-os/.venv/lib/python3.12/site-packages`. Also `context_enhancer.py` needs `load_dotenv()` to read `.env` (added 2026-07-16).
- **`python-dotenv` must be installed system-wide** if any script uses `load_dotenv()`. Install with `pip3 install --break-system-packages python-dotenv`.
- **Sparse vector name mismatch is the most common ingest failure.** The collection must name the sparse config `"sparse"`, not `"bm25"` or anything else. The worker always sends to `"sparse"`.
- **Dense model mismatch kills semantic search.** The search client must use the exact same embedding model as the ingest pipeline. Mixing local Ollama with OpenRouter embeddings (or different models) produces near-zero semantic similarity scores.
- **Docker compose up -d disconnects secondary networks.** After any `docker compose up -d`, the worker loses connection to `ollama_default`. Always re-run `docker network connect`.
- **FASTEMBED_SITEPKGS must be in subprocess env.** The sparse embedding runs in a subprocess; the env var must be explicitly passed via `env=_env` with `_env.setdefault()`.
- **Subprocess python path must be the venv, not system python.** If `_FASTEMBED_PYTHON` points to `/usr/bin/python3` but `fastembed` is installed in `/opt/ai-lab/.venv/bin/python3`, the subprocess silently fails (empty stdout → JSON parse error). Always verify which python can import fastembed, and set the path accordingly.
- **`python-dotenv` may be missing in cron/system environment.** The `wiki_continuous_ingest.py` script imports `from dotenv import load_dotenv`. If the script runs via cron or systemd (not the ai-lab venv), it crashes with `ModuleNotFoundError`. Install system-wide: `pip install python-dotenv`, or patch the script to avoid the dependency.
- **Cron entry for ingest must use .venv python, not system python.** The cron job `0 * * * * cd /opt/hermes/memory-os && python3 scripts/wiki_continuous_ingest.py` fails with `ModuleNotFoundError` because `dotenv` and `arq` are only in the project's `.venv/`. Fix: use `/opt/hermes/memory-os/.venv/bin/python3` instead of `python3`. Verified 2026-07-16: after the fix, the script runs cleanly (`⏭️ Nada novo. 4 arquivos rastreados, 4 inalterados.`).
- **icarus/hooks.py is dead code until registered as a Hermes user plugin.** The plugin lives at `/opt/hermes/.hermes/plugins/icarus/` (NOT in the memory-os project dir). Register with `hermes plugins enable icarus` — NOT by editing config.yaml manually. The plugin has its own `plugin.yaml` (v0.3.0) with 16 tools and 4 hooks (on_session_start, pre_llm_call, post_llm_call, on_session_end). Without `hermes plugins enable`, the file is never executed.
- **icarus hooks.py needs a PYTHONPATH fix before Qdrant search works.** The `_search_qdrant()` function at line 285 does `from scripts.context_enhancer import ...` but `/opt/hermes/memory-os` is not on `sys.path`. Fix: add `import sys` at the top of hooks.py, then insert `sys.path.insert(0, '/opt/hermes/memory-os')` inside `_search_qdrant()` before the import. Without this, `_search_qdrant()` always raises `ModuleNotFoundError` and returns an empty list silently (fail-open).
- **bge-m3 scores are higher than nomic-embed-text.** With nomic-embed-text (768d), hybrid scores were 0.33-0.50. With bge-m3 (1024d), scores are 0.48-0.65 for Russian queries. The Icarus threshold was raised from 0.30 to 0.40. CLI default stays at 0.35 (fine for dense-only).
- **bge-m3 context limit is ~6000 characters.** The Ollama /api/embeddings endpoint returns 500 for texts longer than ~6000 chars. The `bulk_wiki_ingest_ollama.py` script chunks files via `chunk_text()` — recursive split by ##/###/#### headings → paragraphs → words (last resort). `MAX_CHUNK_SIZE=5000`, `CHUNK_OVERLAP=300`. Each chunk becomes a separate Qdrant point with `chunk_index` and `chunk_total` in payload.
- **Switching embedding models requires a full re-index.** When changing from nomic-embed-text (768d) to bge-m3 (1024d): delete old Qdrant collection, recreate with correct dims, re-index all files. Both .env and context_enhancer.py must be updated.
- **Score threshold has two separate regimes.** Dense-only search (via search_knowledge_base) returns COSINE scores 0.90+ for good matches — threshold 0.35 works fine. Hybrid search (via search_with_fallback with sparse vector, used by Icarus) uses RRF fusion which normalises to 0.33-0.50. The Icarus threshold in hooks.py must be set to 0.30, not 0.35 or 0.55. If you see level="none" and SQLite fallback errors, the threshold is too high for RRF.
- **Dimension mismatch between Qdrant collection and .env embedding config is invisible until search time.** If `.env` was changed to a different embedding model with different dimensions (e.g. nomic-embed-text 768d -> bge-m3 1024d) but the Qdrant collection still has the old dimensions, `context_enhancer.py` returns `400 Bad Request` at query time. Cross-check: `collection.config.params.vectors.size` vs `EMBEDDING_DIMS` in `.env`.
- **sync_obsidian_to_wiki.py has a safety guard against empty vaults.** If the WebDAV vault mount is empty but the state file has entries, the script skips deletion (`VAULT ПУСТ... пропускаю удаление`). This prevents data loss when the mount is temporarily disconnected. The canonical vault is now WebDAV at `/mnt/yandex-disk/obsidian/mozg/` — s3fs is defunct.
- **sync → ingest chain needs .venv python.** Inside `sync_obsidian_to_wiki.py`, the post-sync call to `wiki_continuous_ingest.py` uses `os.system(f"python3 ...")`. System python lacks `arq`. Must use `.venv/bin/python3`. Fixed 2026-07-16.
- **State file tracks queue time, not ingest completion.** After resetting the state file and enqueuing, you must wait for the ARQ worker to process before Qdrant has points. Check `wiki_ingest_failures.json` for processing errors.
- **Failures.json is append-only** — it accumulates errors across runs. To get a clean view, read it with jq or truncate it when resetting.
- **search-api `file` and `source` fields depend on which ingest path stored the data.** The Docker worker stores `file_path` (`/wiki/homelab/foo.md`) and `source` (`wiki-homelab`). The bulk ingest script stores `path` (full filesystem path) and `filename` (basename), but no `source`. The search-api was looking for `payload.get("file")` which neither path stores — always returned null. Fixed by cascading through `file_path` → `path` → `filename`. For `source`, if missing or "unknown", the fix extracts the `wiki-` directory component from `file_path`. See `references/search-api-payload-fields.md`. After changing search-api code, rebuild and restart: `docker compose build search-api && docker compose up -d search-api`.