Public document
Low-Cost Internet Research Tooling
research/low-cost-research-stack/Low-Cost-Internet-Research-Tooling.md
Low-Cost Internet Research Tooling
Last updated: 2026-08-16. This cumulative monthly report tracks tools and techniques that reduce search cost, rate limiting, extraction failures, and LLM token use.
Executive Summary
No monthly research runs have completed yet. The installed baseline is:
- Self-hosted SearXNG for primary discovery.
- DDGS/DuckDuckGo for lightweight fallback discovery.
- trafilatura with readability-lxml for clean page extraction.
- Playwright for difficult JavaScript-rendered pages.
- Brave or Tavily free tiers only as last-resort search fallbacks.
- SQLite, DuckDB, or filesystem Markdown for research caching and citations.
- Batched summarization through Ollama or an inexpensive API model.
Research Statistics
- Completed monthly runs: 0
- Non-duplicate sources retained: 0
- Pages successfully extracted: 0
- New tools evaluated: 0
- Existing-tool updates evaluated: 0
- Recommendations adopted: 0
- Recommendations rejected or deferred: 0
Current Stack and Update Assessment
Search and discovery
Scroll horizontally to view all columns.
| Component | Role | Current assessment | Latest checked version/date | Recommendation |
|---|---|---|---|---|
| SearXNG | Primary self-hosted metasearch | Installed and loopback-only | Pending first monthly run | Keep |
| DDGS/DuckDuckGo | Free lightweight fallback | Installed and smoke-tested | Pending first monthly run | Keep |
| Brave Search | Last-resort free-tier API | Credential-dependent | Pending first monthly run | Keep as fallback |
| Tavily | Last-resort research API | Client installed; credential-dependent | Pending first monthly run | Keep as fallback |
Extraction and difficult pages
Scroll horizontally to view all columns.
| Component | Role | Current assessment | Latest checked version/date | Recommendation |
|---|---|---|---|---|
| trafilatura | Primary clean-text extraction | Installed and smoke-tested | Pending first monthly run | Keep |
| readability-lxml | Complementary extraction | Installed and smoke-tested | Pending first monthly run | Keep |
| Playwright + Chromium | JavaScript/difficult-page extraction | Installed and smoke-tested | Pending first monthly run | Keep only for escalation |
Research memory and citations
Scroll horizontally to view all columns.
| Component | Role | Current assessment | Latest checked version/date | Recommendation |
|---|---|---|---|---|
| SQLite | Small durable URL/page/citation cache | Available through Python and smoke-tested | Pending first monthly run | Keep |
| DuckDB | Analytical cache and source statistics | Installed and smoke-tested | Pending first monthly run | Keep where analytics justify it |
| Filesystem Markdown | Human-readable evidence and synthesis | Active | Pending first monthly run | Keep |
Summarization
Scroll horizontally to view all columns.
| Component | Role | Current assessment | Latest checked version/date | Recommendation |
|---|---|---|---|---|
| Ollama | Local batch synthesis | Mac endpoint available through approved LocalSage workflow | Pending first monthly run | Prefer when available |
| Inexpensive API model | Batch-synthesis fallback | Provider-dependent | Pending first monthly run | Use one batch pass, not per result |
New and Emerging Tools
No tools evaluated yet.
Ranked Recommendations
No recommendations yet. Monthly research must distinguish “interesting” from a measurable reduction in cost, tokens, latency, maintenance, or failure rate.
Evaluation Criteria
Score candidates from 0–10 for:
- Search or extraction quality
- Token reduction
- Monetary cost
- Rate-limit resilience
- Local/self-hosted operation
- Maintenance burden
- Security and privacy
- Integration effort
Require a clear migration or trial advantage over the current stack. Do not recommend churn for novelty.
Monthly Run History
No completed runs yet.
Appendix: Sources
No retained sources yet.
Source: Created manually by Sage for cron-driven research: Monthly low-cost internet research tooling review (db79e6d436a9), using research-stack-smoke.py.