Capabilities
Web Crawl
AnyCap web crawl gives agents a direct URL-to-Markdown extraction path through one CLI. Instead of wiring a separate crawling, scraping, or content extraction API, the agent can fetch a known page, normalize the content, and move it into the rest of the workflow through the same capability runtime used for search, generation, and analysis.
CLI usage
Read a known URL
anycap crawl https://example.com
Get the page title only
anycap crawl https://go.dev | jq -r '.data.title'
Extract Markdown content
anycap crawl https://docs.go.dev/ref/spec | jq -r '.data.markdown'
When agents need web crawl
Convert known URLs into clean Markdown for downstream summarization or analysis.
Extract documentation pages without maintaining a separate scraping stack.
Fetch content directly when search has already identified the right source page.
Normalize web page content before feeding it into a research, writing, or coding workflow.
Related pages
Related capability
Web Search
Search helps discover candidate pages before crawl extracts the content of the chosen URL.
Related capability
Grounded Web Search
Use grounded search when the workflow needs a synthesized answer with citations.
Pricing
Pricing
See crawl pricing alongside search and the rest of the capability stack.
FAQ
What does AnyCap web crawl let agents do?
It lets agents take a known URL and convert the page into clean Markdown through one CLI command. That makes web crawl a practical content extraction layer for documentation, articles, and reference pages.
Why not just call this scraping?
Scraping, crawl API, web data API, and content extraction API are all adjacent market terms. Web crawl is the capability label, but the page should also serve the broader content extraction intent because that is how demand usually appears.
When should teams use crawl instead of search?
Use crawl when the target URL is already known and the agent needs the page content itself. Use search when the workflow still needs discovery or source selection.
Why is Markdown conversion useful for AI agents?
Because Markdown is easier for downstream tools and models to process consistently. It reduces the amount of HTML cleanup or extraction logic teams need to maintain on their own.