OpenSpec: исследование + tavily-proxy-setup, local-extractor (spec-driven для инфраструктуры)

This commit is contained in:
estorozhenko
2026-09-06 12:12:33 +00:00
commit d1a19ba6cd
26 changed files with 1669 additions and 0 deletions
@@ -0,0 +1,34 @@
# Delta for web-extract-local
## ADDED Requirements
### Requirement: Local Extraction Mode
The system MUST support extracting page content fully locally (no cloud API) from the same HTTP endpoint used by the Tavily provider.
#### Scenario: Extract page locally
- GIVEN tavily_extract_proxy.py runs with --local (or --local-socks)
- WHEN a POST /extract with {"urls":["https://..."]} is sent
- THEN the server fetches each URL, converts to markdown with trafilatura
- AND returns Tavily-shaped JSON: {"results":[{"url","title","raw_content","metadata"}], "failed_results":[], "failed_urls":[]}
#### Scenario: Page fails to fetch
- GIVEN a URL is unreachable or returns HTTP >= 400
- WHEN extraction is attempted
- THEN the URL lands in failed_results with an error message
- AND other URLs still process normally
### Requirement: SOCKS Support for Local Fetches
The local extractor MUST be able to fetch through the SOCKS5 tunnel (--local-socks 127.0.0.1:1080) for sites blocked from RU IPs.
#### Scenario: Fetch through tunnel
- GIVEN --local-socks 127.0.0.1:1080 is set
- WHEN page fetch happens
- THEN the request is routed through the SOCKS5 tunnel
### Requirement: Backward Compatible Interface
The local mode MUST NOT break the existing Tavily forward mode: the same port/endpoint keeps working for both.
#### Scenario: Forward still works
- GIVEN the service runs with --socks (forward mode)
- WHEN POST /extract is sent
- THEN the request is forwarded to api.tavily.com as before