---
name: news-aggregation-pipeline
description: "Use when building news/RSS aggregation pipelines."
---

# News Aggregation Pipeline

Build domain news aggregators / news DBs with custom crawlers (RSS + HTML),
LLM classification, and vector search. Covers the /opt/news project (IT +
hospitality IT news) and any similar future domain pipeline.

## Current project (source of truth)

- PRD: `/opt/news/PRD.md` — architecture, DB schema, taxonomy, phases 0–3,
  Definition of Done, open questions. READ IT FIRST when working on /opt/news.
- Stack decisions: Python 3.12 venv, feedparser/httpx/selectolax/trafilatura,
  SQLite canonical DB, Qdrant collection `news`, Ollama (bge-m3 embeddings,
  qwen3:8b classifier), Hermes cron, Telegram for digests.
- Status: phase 0 (PRD done; next: sources.yaml ~30 sources, validate_sources.py,
  keywords.yaml, git init on gitea).

## Architecture principles

- **Canonical DB = SQLite**, vector index (Qdrant) = derived layer. SQLite holds
  articles, dedup keys, crawler state (etag/modified/last_error/status).
- **Source registry = single YAML in git** (`sources.yaml`): slug, name, url,
  feed_url (null → html_crawler), crawler, domain, lang, priority (P0/P1/P2 →
  cadence 15min/1h/4h), tags, scrape_selector, enabled.
- **Never physically delete sources** — disable with `enabled: false`
  (user rule: no deleting data). Registry is the only source of truth; SQLite
  and statuses are derived.
- **Feed health**: every fetch writes last_fetch/last_error/status; after N=5
  consecutive errors mark `dead`; watchdog alerts ONLY on alive↔dead transitions
  (user hates routine noise).
- **Domain taxonomy with crossover**: `it_general`, `it_hospitality`, `other`
  (dropped or relevance=low); IT news with clear hospitality application gets
  both domains (e.g. cyber breach of hotel chain).

## Crawler patterns

### rss_crawler (~80% of volume)
1. Read sources.yaml, group by priority.
2. GET with `If-None-Match`/`If-Modified-Since` from stored etag/modified →
   304 = skip (polite + cheap).
3. Normalize URL (strip UTM params, anchors) → `canonical_url` = dedup key.
4. If feed has summary-only entries, fetch full text via html fetch.
5. Dedup: `sha256(canonical_url)` unique in SQLite; existing → skip.
6. hashlib sha256, python-dateutil for published_at.

### html_crawler (~15%)
- httpx + selectolax (parsing) + trafilatura (text extraction).
- `scrape_selector` per source (e.g. `article h2 a`), auto-link discovery fallback.
- Rate-limit 1–2s, honest User-Agent (`news-crawler/0.1 (+https://nixg.ru)`),
  simple robots.txt cache; robots denies → skip source.

### api_crawler (phase 2)
- Telegram channels via MTProto gateway (slidge/tg), HN/Reddit APIs.

## LLM classification (qwen3:8b via Ollama)

- Fast keyword pre-filter (keywords.yaml, hard words per domain; ≥1 hard word →
  candidate), then LLM for final domain/topics/relevance/summary.
- Prompt: title + first ~800 chars; output strict JSON:
  `domain, topics, keywords, relevance (critical/high/low), summary`.
- **PITFALL: qwen3:8b requires `extra_body {"think": false}`** — otherwise it
  emits ~3778 tokens of thinking with empty content (known quirk, applies to
  all text aux calls on this box).
- critical/high → Qdrant index; low → SQLite only.

## Indexing

- Embeddings via Ollama: bge-m3 (1024d, already wired in memory-os search-api)
  or nomic-embed-text (768d). Verify model present in Ollama before first run.
- Qdrant collection `news` — SEPARATE from memory-os `knowledge_base`.
- Dense + sparse BM25 payload; payload: article_id, url, title, published_at,
  domain, topics, source_slug.
- Semantic dedup (phase 2): cosine > 0.95 against existing point → duplicate.

## Infra reuse on this box

- Qdrant docker-qdrant-1 :6333, Redis docker-redis-1 :6379, ARQ worker,
  Ollama :11434 (localhost), Hermes cron, Telegram gateway, gitea.nixg.ru.
- If sharing Redis with memory-os, use separate namespace `news:*`.
- `host.docker.internal` does NOT work on Linux — use service names on shared
  networks or `172.17.0.1`.

## Scheduling (user's watchdog preferences)

- Mechanical crawler runs: plain scripts via Hermes cron (no LLM).
- Watchdogs: `no_agent: true`, `deliver: local`, empty stdout = silent;
  script must stay quiet unless a real threshold/state change fires (e.g.
  check_icq_cert.sh pattern: only speaks at <=45 days).
- Daily digest: LLM-driven cron, delivery to Telegram.

## Definition of Done (from PRD)

- ≥100 articles/day with zero dupes (count(sha256) == count(id)).
- Hand-check 20 random articles: ≥90% correct domain.
- Qdrant returns sensible hits for 10 control queries.
- Watchdog silent for 3 consecutive days when healthy.