Web search
Give your agents access to real-time information from the web
Overview
Web search tools allow your agents to access real-time information from the internet. While agents have extensive knowledge from their training data, that knowledge has a cutoff date. Web search bridges the gap, enabling agents to:
- Find current information about companies, people, and events
- Retrieve the latest news and trends
- Verify facts against up-to-date sources
- Research topics not covered in training data
Web search is most effective when combined with agent reasoning. The agent decides what to search for, interprets results, and synthesizes information from multiple sources.
How it works
Web search in MagOneAI is provided by the web-search MCP server, not a proprietary search engine. It is multi-provider: it works out of the box with no API key, and can be given keys for higher-quality or higher-volume providers. The providers wired up today are:
- SearXNG — a self-hosted metasearch engine that aggregates Google, Bing, DuckDuckGo and more. Free, and needs no API key.
- Serper — Google results via the Serper API. Paid, fast, and returns rich results including the knowledge graph and "people also ask".
- Perplexity — the Perplexity Search API. Paid, and returns ranked results with pre-extracted content snippets, which often removes the need for a follow-up fetch.
Because search is MCP-backed, the same tooling model applies as for any other tool: agents discover the search tools, call them during reasoning, and the platform handles execution. Each keyed provider has its own quotas and rate limits, set by your plan and key.
Capabilities
The web-search MCP server exposes five tools to agents.
search_web
Search the web using the self-hosted SearXNG metasearch engine, which aggregates Google, Bing, DuckDuckGo and others. No API key needed, so this is the default choice.
Parameters: query (required), num_results, language, region, time_range, safe_search, engines, categories
Example use case: "Find the latest news about artificial intelligence regulation in the EU."
serper_search
Search Google via the Serper API. Paid, fast, and reliable. Returns organic results plus the knowledge graph and "people also ask" blocks. Requires a Serper API key.
Parameters: query (required), search_type, num_results, country, language, location
perplexity_search
Search via Perplexity's Search API. Returns ranked results with rich pre-extracted content snippets, which often means you do not need a follow-up scrape_url call. Supports batch queries (up to 5 at once), domain allow and deny lists, and date-range filters. Requires a Perplexity API key.
Parameters: query (required), max_results, max_tokens_per_page, country, search_language_filter, search_domain_filter, search_recency_filter, search_after_date_filter, search_before_date_filter
scrape_url
Fetch a URL and extract its full text content. Use it after a search to read a specific result in full. Pass return_full=true for knowledge-base-style ingestion, which returns full markdown, the link graph, title, and status.
Parameters: url (required), max_chars, return_full, extra_args
kb_resync_urls
Re-crawl the URL documents in one or more knowledge bases to keep them fresh, in a single call. Build it as START to this tool to END, then put the use case on a schedule — no ForEach or child use case needed.
Parameters: kb_ids, kb_id
On a Tool node you must pass kb_ids yourself. The platform only fills the target in automatically when an agent calls this tool, taking it from that agent's attached knowledge bases. A Tool node has no attached knowledge bases, so whatever you bind is what gets re-crawled, and binding nothing means nothing is targeted.
There's no dedicated "structured extraction" tool. To pull specific fields out of a page — contact details, product specs, article metadata — the agent fetches the page with scrape_url, then reasons over that text to extract what it needs. You guide the extraction through the agent's instructions (for example, "from the fetched page, return the company name, headquarters location, CEO name, and employee count as JSON").
Configuration
Web search is set up like any external MCP integration: an operator connects the web-search MCP server and configures it with a search provider and API key.
Start with SearXNG, add keys only if you need them
search_web runs on SearXNG and needs no configuration at all. Both API keys are optional. Add a Serper key (from serper.dev) for Google results, or a Perplexity key (from perplexity.ai/settings/api) for results with pre-extracted snippets. Keys are stored in HashiCorp Vault and injected into the MCP server's requests at runtime.
Connect the web-search MCP server
Wire up the web-search MCP server with the provider's API key. Once connected, its search and fetch tools appear in the tool picker. See External MCP servers for how MCP connections work.
Attach to agents
In agent configuration, enable the web search tools. Agents can now search and fetch pages during reasoning.
search_web is free, but serper_search and perplexity_search consume your provider's API quota. Each enforces its own rate limits based on your plan and key — monitor usage on the provider's dashboard to stay within budget. Credentials can be scoped per user, per project, or per organization, so one team's usage doesn't draw down another's.
Use cases
Web search unlocks powerful capabilities across many domains. Here are common patterns and use cases.
Sales intelligence
Research companies and contacts before meetings.
Example workflow:
Agent system prompt:
You are a sales research assistant. Before meetings, you research
the prospect's company, recent news, and key contacts.
For each company, find:
1. Recent press releases or news articles
2. Company size, funding, and growth trajectory
3. Key decision makers in the target department
4. Recent product launches or initiatives
5. Potential pain points we can address
Synthesize this into a pre-meeting brief.Agent actions:
- Search for company news: "Acme Corp press releases 2024"
- Search for funding information: "Acme Corp Series B funding"
- Search for decision makers: "Acme Corp VP Engineering"
- Retrieve and parse relevant articles
- Synthesize findings into a structured brief
Market analysis
Monitor market trends, competitors, and industry developments.
Example workflow:
- Agent searches for industry trend reports
- Retrieves competitor product pages and pricing
- Finds analyst opinions and market forecasts
- Synthesizes information into a market intelligence report
Competitive intelligence
Track competitor activities and announcements.
Example workflow:
- Agent monitors competitor websites for changes
- Searches for competitor mentions in news
- Fetches competitor pages and reads off product features
- Compares against your product offerings
- Generates competitive analysis report
Fact checking
Verify claims against current, authoritative sources.
Example workflow:
- Agent receives a claim to verify
- Searches for authoritative sources on the topic
- Retrieves full articles from credible publications
- Cross-references information across multiple sources
- Returns verification result with source citations
When an agent uses results from the web in a chat response, the sources surface as citations so users can trace each claim back to the page it came from.
Combining with other tools
Web search becomes more powerful when combined with other MagOneAI capabilities.
Web search + RAG
Use web search to supplement internal knowledge bases.
Pattern:
- Agent searches internal documents via RAG (Retrieval Augmented Generation)
- If information is missing or outdated, agent searches the web
- Agent cross-references external information with internal policies
- Agent synthesizes both sources into a comprehensive answer
Example: "What's our company policy on AI tool usage?" → search internal docs → policy found but mentions specific tools → search web for current tool capabilities → compare with policy → provide informed answer.
Web search + Email
Research topics and send summary reports.
Pattern:
- Agent receives a research request via email or workflow trigger
- Agent searches the web for relevant information
- Agent retrieves and analyzes source pages
- Agent composes email summary with key findings and citations
- Agent sends report to stakeholders
Example: Daily competitor monitoring workflow that emails a digest of competitor news each morning.
Web search + Database
Enrich database records with external information.
Pattern:
- Agent queries database for records needing enrichment
- For each record, agent searches the web for additional data
- Agent fetches the relevant pages and reads the information it needs from them
- Agent updates database with enriched data
Example: CRM enrichment workflow that finds company size, industry, and key contacts for new leads.
Best practices
Generic instructions like "search the web" lead to poor results. Be specific:
✅ Good: "Search for recent press releases from [Company] about product launches in the last 3 months"
❌ Bad: "Find information about [Company]"
Specificity in agent instructions leads to more relevant search queries and better results.
Search results are raw data. Agents need instructions on how to use them:
When you receive search results:
1. Identify the most relevant and recent results
2. Prioritize authoritative sources (company websites, major news outlets)
3. Retrieve full content from top 3-5 results
4. Cross-reference information across sources
5. Synthesize findings, noting any conflicting informationNot all web information is accurate. Teach agents to validate:
- Check source credibility
- Look for multiple corroborating sources
- Note publication dates (prefer recent information)
- Flag conflicting information for human review
- Cite sources in outputs for transparency
Better queries yield better results. Agent should construct queries that:
- Use specific keywords and phrases
- Include date ranges when recency matters
- Use site-specific searches when appropriate (e.g., "site:company.com")
- Employ Boolean operators for complex queries
- Avoid overly broad or ambiguous terms
Web search APIs have rate limits. Design workflows that:
- Batch research tasks instead of searching in real-time for every query
- Cache search results for a reasonable time period
- Use search judiciously — only when internal data is insufficient
- Monitor usage on your provider's dashboard to avoid hitting quota
Search can fail (rate limits, network issues, no results). Agents should:
- Retry with alternative query phrasing if no results found
- Degrade gracefully to internal knowledge if search unavailable
- Inform users when searches fail and why
- Log failures for monitoring and debugging
Web search is most effective when combined with specific agent instructions. Tell the agent exactly what to search for, how to interpret results, and how to synthesize information from multiple sources.
Privacy and compliance
Web search accesses public internet data. Consider these privacy and compliance aspects:
Data handling
- Search queries are sent to the search API provider
- Retrieved content is processed by the agent (LLM provider sees this content)
- Search results are not permanently stored by MagOneAI (unless you explicitly log them)
Sensitive information
Never include sensitive information in search queries. Search queries may be logged by search API providers. Avoid searching for:
- Personally identifiable information (PII)
- Confidential business data
- Authentication credentials
- Internal system details
Compliance considerations
- GDPR: Web search retrieves publicly available data, but be mindful of how you use personal information found online
- Copyright: Retrieved content is subject to copyright. Don't republish scraped content without permission
- Terms of Service: Respect the terms of service of websites you scrape or retrieve content from
Rate limiting and robots.txt
MagOneAI's web retrieval respects:
- Standard rate limits to avoid overwhelming target websites
robots.txtdirectives where appropriate- HTTP status codes and retry-after headers
Troubleshooting
Symptoms: Search returns empty results or very few results.
Solutions:
- Query may be too specific or contain typos
- Try broader search terms or alternative phrasings
- Check if safe search settings are too restrictive
- Verify regional/language settings match content availability
Symptoms: Results don't match the agent's intent or user's needs.
Solutions:
- Refine agent instructions to construct more specific queries
- Use Boolean operators (AND, OR, NOT) in queries
- Add date constraints for time-sensitive information
- Implement result filtering logic in agent reasoning
Symptoms: URLs from search results can't be retrieved.
Solutions:
- Some sites block automated access
- Page may require JavaScript rendering (not supported by basic retrieval)
- Retry after a delay (temporary issue)
- Some sites require authentication (not accessible via web search)
Symptoms: Tools return "rate limit exceeded" errors.
Solutions:
- Reduce search frequency in workflows
- Implement caching for repeated searches
- Upgrade your search provider plan or raise its quota on the provider's dashboard
- Batch search tasks instead of real-time searches