openspec-lab/openspec/changes/archive/2026-09-06-local-extractor/specs/web-extract-local/spec.md

Delta for web-extract-local

ADDED Requirements

Requirement: Local Extraction Mode

The system MUST support extracting page content fully locally (no cloud API) from the same HTTP endpoint used by the Tavily provider.

Scenario: Extract page locally

• GIVEN tavily_extract_proxy.py runs with --local (or --local-socks)
• WHEN a POST /extract with {"urls":["https://..."]} is sent
• THEN the server fetches each URL, converts to markdown with trafilatura
• AND returns Tavily-shaped JSON: {"results":[{"url","title","raw_content","metadata"}], "failed_results":[], "failed_urls":[]}

Scenario: Page fails to fetch

• GIVEN a URL is unreachable or returns HTTP >= 400
• WHEN extraction is attempted
• THEN the URL lands in failed_results with an error message
• AND other URLs still process normally

Requirement: SOCKS Support for Local Fetches

The local extractor MUST be able to fetch through the SOCKS5 tunnel (--local-socks 127.0.0.1:1080) for sites blocked from RU IPs.

Scenario: Fetch through tunnel

• GIVEN --local-socks 127.0.0.1:1080 is set
• WHEN page fetch happens
• THEN the request is routed through the SOCKS5 tunnel

Requirement: Backward Compatible Interface

The local mode MUST NOT break the existing Tavily forward mode: the same port/endpoint keeps working for both.

Scenario: Forward still works

• GIVEN the service runs with --socks (forward mode)
• WHEN POST /extract is sent
• THEN the request is forwarded to api.tavily.com as before