We count page visits with Google Analytics to see which pages are read. Nothing is stored until you accept, and we never store search queries. Privacy.

wellknown
SearchCapabilitiesReachabilityReliabilityJournalBlogSourcesPricingDocs
/mcpAdd to your agentSign in
wellknown

A live, machine-queryable index of AI agents, MCP servers and tools. Every record separates what publishers declare from what we observed.

Product
  • About
  • Search
  • Capabilities
  • Reachability
  • Reliability report
  • Journal
  • Blog
  • Sources & freshness
  • Network status
  • Pricing
  • Sponsor
Developers
  • Documentation
  • HTTP API
  • MCP server
  • Claim an agent
  • Dashboard
For machines
  • llms.txt
  • openapi.json
  • agent-card.json
  • sitemap.xml
  • About our crawler
Records are indexed from public sources and attributed to them. Observed data is ours; declared data is theirs.PrivacyTermsv0.1 · early network
Capabilities/Data & Search/data.web-scraping

Web Scraping

Fetch and extract web page content.

3.1k records2433 liveRefine in search →
Agent Discovery PlacementMCP serverMCP
by sadri-dridi
Live

Labeled placement in agent discovery indexes, measured with daily HMAC nicknames.

Browser Automation~100%Agent Discovery~100%People & Company Research~58%Web Scraping~54%Prompt Management~53%
https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-discovery-placement/mcpok 25 h ago
Agents Find Your ProductMCP serverMCP
by sadri-dridi
Live

Make a product discoverable to agents: catalog, llms.txt, agent card, hops, MCP door.

Browser Automation~100%E-commerce Operations~91%Workflow Automation~83%People & Company Research~58%Web Scraping~54%
https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agents-find-your-product/mcpok 25 h ago
Agent Crawl ReadinessMCP serverMCP
by sadri-dridi
Live

Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.

Web Scraping~100%Browser Automation~100%People & Company Research~58%Prompt Management~53%Weather~51%
https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcpok 27 h ago
mcp-webextratorMCP serverMCP
by AceDataCloud
Live

MCP server for web extraction and rendering via AceDataCloud WebExtrator

Data Extraction~100%Web Scraping
https://webextrator.mcp.acedata.cloud/mcppublished 7 d agook 2 h ago
DeckofcardsMCP serverMCP
by pipeworx-io
Live

Deck of Cards MCP — wraps deckofcardsapi.com (free, no auth)

Weather~100%Web Scraping~99%Agent Memory~99%Packages & Dependencies~96%News & Feeds~93%
https://gateway.pipeworx.io/deckofcards/mcpok 28 h ago
UTC ClockMCP serverMCP
by sadri-dridi
Live

Current UTC timestamp for autonomous clients.

Browser Automation~100%People & Company Research~58%Web Scraping~54%Prompt Management~53%Weather~51%
https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/utc-clock/mcpok 28 h ago
Agent Discovery IndexMCP serverMCP
by sadri-dridi
Live

Where AI agents discover products, and how to be listed there with measured results.

E-commerce Operations~100%Browser Automation~100%Agent Discovery~100%People & Company Research~58%Web Scraping~54%
https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-discovery-index/mcpok 28 h ago
AI Crawler Checker — Robots.txt Policy for Six BotsMCP serverMCP
by kriton-dev
Live

Check declared robots.txt permissions for six AI crawlers at URL paths

Web Scraping~100%
https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-checkok 17 h ago
AirqualityMCP serverMCP
by pipeworx-io
Live

Air Quality MCP — wraps air-quality-api.open-meteo.com (free, no auth)

Weather~100%Web Scraping~99%Agent Memory~99%Packages & Dependencies~96%News & Feeds~93%
https://gateway.pipeworx.io/airquality/mcpok 32 h ago
NationalizeMCP serverMCP
by pipeworx-io
Live

Nationalize MCP — nationality prediction from first name (nationalize.io, free, no auth)

Weather~100%Web Scraping~99%Agent Memory~99%Packages & Dependencies~96%News & Feeds~93%
https://gateway.pipeworx.io/nationalize/mcpok 33 h ago
Fintech IntelMCP serverMCP
by pipeworx-io
Live

FinTech Intel MCP — Compound tools that chain SEC, CFPB, FDIC,

Weather~100%Web Scraping~99%Agent Memory~99%Banking~98%Packages & Dependencies~96%
https://gateway.pipeworx.io/fintech-intel/mcpok 34 h ago
Open Agent Utility IndexMCP serverMCP
by sadri-dridi
Live

Transparent public utilities for agents. Requests are counted without storing identifiers.

Browser Automation~100%People & Company Research~58%Web Scraping~54%Prompt Management~53%Weather~51%
https://agent-observatory-sensor.nolimit-observatory.workers.dev/mcpok 35 h ago
AgifyMCP serverMCP
by pipeworx-io
Live

Agify MCP — age prediction from first name (agify.io, free, no auth)

Weather~100%Web Scraping~99%Agent Memory~99%Packages & Dependencies~96%News & Feeds~93%
https://gateway.pipeworx.io/agify/mcpok 35 h ago
A.U.G.U.R ForesightAgentA2A
by MAD (maddegen.art) — operated by Alf Agent
Live

Tool exchange for AI agents run by Alf Agent: 79 tools — crypto markets and DeFi on every chain (prices, TVL, yields, stablecoins, perps funding, order books, gas, EVM contracts and token security), Solana and pump.fun intelligence from our own live recording of every bonding-curve trade, launch bu…

Deep Web ResearchWeb ScrapingWeatherPayments~100%Markets & Trading
https://agent.maddegen.art/hub/api/v1ok 2 h ago
402registryAgentA2A
by 402registry
Live

A paid check of whether AI crawlers can actually see a website: a plain HTTP fetch with no JavaScript, the way GPTBot, ClaudeBot and PerplexityBot fetch pages. A check costs $0.01 over x402 (USDC); a passing check earns a dated badge and a registry listing. Over A2A this agent is a receptionist: ev…

Web Scraping~100%SEO
https://402registry.com/a2aok 3 h ago
Naukri Jobs Scraper (Apify) - Naukri.com jobs with salary in LPAMCP serverMCP
by datadmon
Live

Naukri jobs as JSON: salary in LPA, skills, applicants, full JD, whole-board daily feed. $0.80/1k

Web Scraping~73%
https://mcp.apify.com/?tools=data_daemon/naukri-jobs-scraperok 6 h ago
IIMJobs Jobs Scraper (Apify) - India MBA, finance & management jobsMCP serverMCP
by datadmon
Live

IIMJobs MBA/finance jobs JSON, full JD. $2/1k + $0.001/run. Apify Bearer token required.

Web Scraping~73%
https://mcp.apify.com/?tools=data_daemon/iimjobs-jobs-scraperok 8 h ago
JSON CheckMCP serverMCP
by sadri-dridi
Live

Check whether text is valid JSON. The body is discarded.

Browser Automation~100%People & Company Research~58%Web Scraping~54%Prompt Management~53%Weather~51%
https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/json-check/mcpok 2 d ago
Robots InspectMCP serverMCP
by sadri-dridi
Live

Fetch a public robots.txt and return group counts only.

Browser Automation~100%People & Company Research~58%Web Scraping~54%Prompt Management~53%Weather~51%
https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/robots-inspect/mcpok 2 d ago
HTTP StatusMCP serverMCP
by sadri-dridi
Live

Short catalog of common HTTP status codes.

Browser Automation~100%E-commerce Operations~91%People & Company Research~58%Web Scraping~54%Prompt Management~53%
https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/http-status/mcpok 2 d ago
← Previouspage 111 of 154Next →
Machine query
GET /api/v1/search?capability=data.web-scraping&status=live
Aliases

scraping, scraper, crawl, crawler, fetch url, web fetch, firecrawl, extract from website

Also in Data & Search
  • Web Search
  • Databases
  • Vector Search
  • API Access
  • News & Feeds
  • Weather