mirror of
https://gitverse.ru/kpa39l/openspec-lab.git
synced 2026-09-29 21:25:04 +00:00
estorozhenko
38 lines
1.5 KiB
Markdown
38 lines
1.5 KiB
Markdown
# web-extract-local Specification
|
|
|
|
## Purpose
|
|
TBD - created by archiving change local-extractor. Update Purpose after archive.
|
|
|
|
## Requirements
|
|
|
|
### Requirement: Local Extraction Mode
|
|
The system MUST support extracting page content fully locally (no cloud API) from the same HTTP endpoint used by the Tavily provider.
|
|
|
|
#### Scenario: Extract page locally
|
|
- GIVEN tavily_extract_proxy.py runs with --local (or --local-socks)
|
|
- WHEN a POST /extract with {"urls":["https://..."]} is sent
|
|
- THEN the server fetches each URL, converts to markdown with trafilatura
|
|
- AND returns Tavily-shaped JSON: {"results":[{"url","title","raw_content","metadata"}], "failed_results":[], "failed_urls":[]}
|
|
|
|
#### Scenario: Page fails to fetch
|
|
- GIVEN a URL is unreachable or returns HTTP >= 400
|
|
- WHEN extraction is attempted
|
|
- THEN the URL lands in failed_results with an error message
|
|
- AND other URLs still process normally
|
|
|
|
### Requirement: SOCKS Support for Local Fetches
|
|
The local extractor MUST be able to fetch through the SOCKS5 tunnel (--local-socks 127.0.0.1:1080) for sites blocked from RU IPs.
|
|
|
|
#### Scenario: Fetch through tunnel
|
|
- GIVEN --local-socks 127.0.0.1:1080 is set
|
|
- WHEN page fetch happens
|
|
- THEN the request is routed through the SOCKS5 tunnel
|
|
|
|
### Requirement: Backward Compatible Interface
|
|
The local mode MUST NOT break the existing Tavily forward mode: the same port/endpoint keeps working for both.
|
|
|
|
#### Scenario: Forward still works
|
|
- GIVEN the service runs with --socks (forward mode)
|
|
- WHEN POST /extract is sent
|
|
- THEN the request is forwarded to api.tavily.com as before
|