dsh-tool-web-enhanced

English | 中文

A drop-in replacement for the stock web_search tool that is modular by sections: the native search results stay as the first section, and you attach additional search modules — each contributing its own section — such as a local SearXNG instance and RAG databases (local markdown sources). The native behaviour is unchanged; everything else is optional.

What it is

dsh-tool-web-enhanced is a drop-in replacement for DeepSeek Harness' stock web_search tool. When no modules are configured, web_search behaves exactly like stock: the native results are the only section. Turn on a module and it contributes its own section to the same search response:

• the native DeepSeek search results remain the first section, unchanged;
• you can attach additional search modules, each rendered as its own section — a local SearXNG instance, RAG databases (local markdown sources), and more;
• the extension point is a clean module interface (SearchSection) plus a config surface (sections:), so adding a new section type is a small, documented, code-level step (fork or PR the repo).

Everything is optional: with no modules configured, web_search is exactly stock.

Features

• Modular per-section architecture — each search source is a SearchSection registered under sections:. Native results stay first; every additional module renders as its own section.
• Built-in modules — a SearXNG section (rendered as SearXNG results), a RAG section over local markdown databases (one RAG — <dbName> block per database), and a Parallel section (Parallel Web Systems Search API, rendered as Parallel results).
• Alternative web_fetch provider — an opt-in parallel-extract fetch provider (Parallel Web Systems Extract API) that returns a URL's full document as markdown. Registered into ctx.web; selected by the deployment profile's fetchProvider: 'parallel-extract'.
• Optional topic and sources parameters — topic forwards a vertical hint to modules that support it; sources picks any combination of native / SearXNG / RAG / Parallel (native, searxng, rag, parallel, or all).
• Silent degradation — a module that is absent, disabled, or unreachable is simply omitted, never an error; results degrade to the remaining sections.
• Self-contained drop-in — the bundle registers the enhanced tools and disables the stock tool-web row automatically on install.

Install

npm install dsh-tool-web-enhanced

This is a DSH bundle: package.json carries dsh.bundle.patch = ./cordis.patch.yml, which inserts the enhanced plugin row and disables the stock tool-web row in one install. Installing the package is the whole swap for CLI profiles — no manual profile edit required. For preset-realm web surfaces, the preset still disables its own tool-web row.

# cordis.patch.yml (bundled with this package)
- insert:
    - id: tool-web-enhanced
      name: dsh-tool-web-enhanced
      config:
        search: true
        fetch: true
        sections:
          searxng:
            enabled: true
            url: 'http://127.0.0.1:8080'
          parallel:
            enabled: true
            apiKeyEnv: PARALLEL_API_KEY
            apiKey: ''
          rag:
            enabled: true
            storePath: ''
            embeddings:
              provider: auto
              apiKeyEnv: EMBEDDING_API_KEY
              apiKey: ''
            databases: []
          # Parallel Extract fetch provider — OPT-IN (enabled: false by default).
          parallelExtract:
            enabled: false
            apiKeyEnv: PARALLEL_API_KEY
            apiKey: ''
            extractMode: full
            timeoutMs: 60000

- id: tool-web
  disabled: true

Installing self-disables the stock tool-web row, so this package is the entire web-search swap.

Configuration

The enhanced behaviour lives under one unified sections: container. Keys are neutral parameter names. Stock search / fetch keys keep their existing names and defaults.

Key	Type	Default	Description
search	boolean	true	Register web_search.
fetch	boolean	true	Register web_fetch (unchanged).
sections.searxng.enabled	boolean	true	Enable the SearXNG section.
sections.searxng.url	string	http://127.0.0.1:8080	Base URL of the local SearXNG JSON API.
sections.parallel.enabled	boolean	true	Enable the Parallel (Parallel Web Systems Search API) section.
sections.parallel.apiKeyEnv	string	PARALLEL_API_KEY	Env var holding the Parallel API key.
sections.parallel.apiKey	string	''	Literal Parallel API key (wins over apiKeyEnv).
sections.parallel.mode	string	fast	Parallel search mode: turbo / fast / basic / advanced.
sections.parallel.maxResults	number	10	Max results returned by the section (≤10, no pagination).
sections.rag.enabled	boolean	true	Enable the RAG section + rag_index tool.
sections.rag.storePath	string	'' (auto)	Search-index store path; empty → a default under the data home.
sections.rag.embeddings.provider	string	auto	Embedding selection: auto / local / remote. auto → remote when a key is set, else local.
sections.rag.embeddings.apiKeyEnv	string	EMBEDDING_API_KEY	Env var holding the remote provider's key.
sections.rag.embeddings.apiKey	string	''	Literal remote provider key (wins over apiKeyEnv).
sections.rag.embeddings.model	string	(a multilingual embedding model)	Remote embedding model.
sections.rag.embeddings.baseURL	string	(your embeddings endpoint)	Remote embeddings API base URL (embeddings-API-compatible).
sections.rag.embeddings.localModel	string	(a small local embedding model)	Local embedding model (downloaded on first use).
sections.rag.databases[].name	string	—	Database (section) name.
sections.rag.databases[].path	string	—	Directory of markdown files to index.
sections.rag.databases[].topK	number	5	Results returned per database.
parallelExtract.enabled	boolean	false	Register the Parallel Extract fetch provider (ctx.web). OPT-IN.
parallelExtract.apiKeyEnv	string	PARALLEL_API_KEY	Env var holding the Parallel API key (same key as sections.parallel).
parallelExtract.apiKey	string	''	Literal Parallel API key (wins over apiKeyEnv).
parallelExtract.extractMode	string	full	full → the complete markdown document; snippets → excerpts only.
parallelExtract.timeoutMs	number	60000	Per-call timeout (ms); the Extract API is slow (1–20s).

The stock search / fetch keys are kept unchanged for drop-in compatibility.

Usage

web_search accepts the stock query plus two optional parameters:

Param	Required	Description
query	yes	The search query.
topic	no	Vertical hint, forwarded to sections that support it (e.g. SearXNG categories): general, news, science, it, files, social media, images, videos, map, music.
sources	no	Comma-separated tokens — native plus each enabled section id. Default all. Examples: native,searxng, searxng,rag, or searxng,parallel.

The output shape carries the native results plus a sections array — one entry per module that returned results:

{
  "content": "...",                 // optional native answer
  "sources": [ { "url": "...", "title": "...", "snippet": "..." } ],  // native
  "truncated": false,
  "sections": [
    {
      "name": "SearXNG results",
      "sources": [ { "url": "...", "title": "...", "snippet": "...", "score": 0.9 } ]
    },
    {
      "name": "RAG — my-docs",
      "sources": [ { "url": "...", "title": "...", "path": "...", "score": 0.72 } ]
    },
    {
      "name": "Parallel results",
      "sources": [ { "url": "...", "title": "...", "snippet": "..." } ]
    }
  ]
}

Connecting SearXNG

The SearXNG section is optional, and the plugin only talks to a SearXNG instance over its local JSON API (format=json). Point sections.searxng.url at the base URL of any instance that exposes JSON output:

GET {sections.searxng.url}/search?q=<query>&format=json[&categories=<topic>]

The simplest way to stand one up is a Docker Compose service exposing the JSON API on a local port. Having no running instance is fine: the SearXNG section is silently omitted when it is disabled, unreachable, or empty.

Guarantee: when a module is absent, disabled, or unreachable, web_search never errors — the section is simply omitted and results degrade to whatever remains (down to native-only, exactly stock).

Parallel section

The Parallel section queries the Parallel Web Systems Search API (a declarative-semantic web search built for AI agents) and renders the sources as a Parallel results block under the native results. It calls POST https://api.parallel.ai/v1/search with an x-api-key header (not a bearer token) and a body of { objective, search_queries, mode }:

POST {https://api.parallel.ai/v1/search}
Headers: x-api-key: <key>
Body: { "objective": "<query>", "search_queries": ["<query>"], "mode": "fast" }

The section needs a key to do anything — set sections.parallel.apiKeyEnv to an env var (default PARALLEL_API_KEY) or sections.parallel.apiKey to a literal key. With no resolvable key the section is silently inert (returns undefined and never calls the API). It is thus entirely opt-in: shipping the default config enables it, but nothing is fetched or sent until a key is present in the environment. The key is never committed to any repo file.

By default it requests the fast (mode: fast) tier and caps results at sections.parallel.maxResults (default 10, the API's per-call maximum — the API has no pagination). Failures (network, timeout, non-2xx, malformed response) degrade silently to undefined, exactly like the SearXNG section.

Parallel Extract fetch provider

The web_fetch tool retrieves a URL through a provider selected by the web seam's fetchProvider config (default: the stock HTTP provider). This package registers an opt-in alternative: parallel-extract, backed by the Parallel Web Systems Extract API. It calls POST https://api.parallel.ai/v1/extract with an x-api-key header and a body of { urls: [<url>], advanced_settings: { full_content: <bool> } }, and maps the returned document to the fetch result's markdown text body.

POST https://api.parallel.ai/v1/extract
Headers: x-api-key: <key>, Content-Type: application/json
Body: { "urls": ["<url>"], "advanced_settings": { "full_content": true } }

It is fully opt-in and inert by default: parallelExtract.enabled defaults to false, so the provider is never registered and the stock web_fetch is never displaced. To use it:

1. Enable the provider: parallelExtract.enabled: true (with apiKeyEnv defaulting to PARALLEL_API_KEY, or a literal apiKey).
2. Pin the web seam to it in the deployment profile (this package does not, and must not, set the seam config): fetchProvider: 'parallel-extract' (or $DSH_WEB_FETCH_PROVIDER=parallel-extract).

Without a resolvable key the provider reports itself unavailable (its available() is false) and a direct call fails cleanly with a structured WebError. Failures (non-2xx, malformed response, no result / errors[], timeout) also surface as clean WebErrors following the other fetch providers' contract — never a misleading result.

parallelExtract.extractMode controls what comes back:

• full (default): requests advanced_settings.full_content = true and returns the complete markdown document (results[].full_content), falling back to the joined excerpts when the API returns null.
• snippets: leaves full_content off and returns the joined results[].excerpts — cheaper and faster if you only need fragments.

The API accepts up to 20 URLs per request and charges $1 per 1000 URLs; the provider sends one URL per web_fetch call, enforcing the per-request cap in buildParallelExtractBody.

RAG section

The RAG module indexes local markdown databases into an on-machine store and, on every search, retrieves the most similar chunks per database — one RAG — <dbName> section per configured database.

The embedding step is used in two places: to index each chunk, and to embed the query on every search. With the local path (no key configured) indexing and query data stay on the machine; a remote provider is used only if you configure one — nothing is sent unless a provider is configured.

When RAG is enabled with at least one database, a rag_index tool is registered. It rebuilds the local RAG index for all configured databases and returns the number of chunks indexed per database. The index is also built automatically (async, non-blocking) on startup.

Ingestion filters (secrets hygiene). The sections.rag config accepts three optional keys controlling what enters the index:

• excludePaths: string[] — glob patterns (POSIX, relative to each database root) of paths to skip during the walk. They are merged with the built-in defensive defaults, which always apply: **/.env, **/*.conf, **/.credentials.yaml (today these match no *.md — a no-op safety net if the walk ever broadens).
• ignoreDotfiles: boolean (default false) — skip dotfiles and dot-directories (.env.md, .git/, …). Off by default so the current walk behaviour is preserved.
• denyContent: string[] — regex sources; any chunk whose text matches a pattern is dropped before embedding (the exact text that would otherwise go to the embedder). Built-in defaults always apply on top of the configured patterns: sk-[A-Za-z0-9_-]{15,} (OpenAI/Anthropic-style API keys) and secret environment assignments (DEEPSEEK_API_KEY=…, OPENCODE_GO_KEY_n=…, DEEPINFRA_TOKEN=…, PARALLEL_API_KEY=… with a non-trivial value). Prose that merely names these variables (e.g. "the DEEPINFRA_TOKEN config") is not matched, so legitimate technical discussion stays searchable.

Example:

rag:
  enabled: true
  excludePaths:
    - '**/secrets/**'
    - 'journals/sessions/**'
  ignoreDotfiles: true
  denyContent:
    - 'AKIA[0-9A-Z]{16}'       # AWS access keys, on top of the built-ins
  databases: [ … ]

The filters only affect NEW ingestion. Because unchanged .md files are skipped by their mtime, chunks already stored keep serving until a forced re-chunk: delete rag.db (or the files/chunks rows) and re-run rag_index, or trigger the automatic dimension-change rebuild. Path-level excludes (excludePaths) additionally self-clean: files that disappear from the walk have their rows removed on the next ensureIndex.

Adding your own section

The whole point of this package is that web_search is modular by sections. To add a new search source you write a small, self-contained module — no changes to the core tool:

1. Define a SearchSection — give it an id (used as a sources token), an enabled flag, and a run(query, ctx) method that returns the section's result blocks (SectionBlock[]).
2. Add its config slice under sections: in cordis.patch.yml — any parameters the module needs.
3. Wire it into buildSections — register the new module alongside the built-in ones so it is instantiated when enabled.

That's it — roughly fifteen lines. The module contract lives in src/modules.ts (the SearchSection interface and buildSections composition point). Because modules are an isolated list, the package is fork/PR-friendly: a new section type is a small, documented, code-level addition that composes with the native-first output shape and the sources selection.

Output shape

See Usage above: web_search returns the canonical stock fields (content, sources for native, truncated) plus a sections[] array — one entry per module that returned results, each with a name and its own sources[]. A module with no results is omitted entirely.

Development

• npm run build — compiles src/ to lib/ with tsc (NodeNext).

• node --test — runs the unit tests in test/ against the built lib/.

• Smoke-test in a DSH profile — install the local checkout into an isolated development profile, then inspect the composed configuration:

  dsh plugin --profile dev add /path/to/dsh-tool-web-enhanced
  dsh --profile dev --dump-config
  

  The dumped tree must show the tool-web-enhanced row plus the disabled tool-web row. Exercise web_search end-to-end in that profile afterward.

License

MIT