VebAPI
AI SEO Crawler Check API | AI SEO Audit Tool - VebAPI
GET Version 1.0 1 Credit

AI SEO crawler check

Checks whether a website allows AI bots (e.g., GPTBot, Google-Extended, PerplexityBot)

Ready to query this endpoint?

Sign up to get a free API key instantly with free trial credits. No credit card required.

Get Free API Key Sign In

Request Details

GET /seo/aiseochecker
Parameter Type Requirement Description
website string Required enter the website url you want to check in
X-API-KEY string Required Your vebapi api key

What it does 

Checks whether a website allows AI bots (e.g., GPTBot, Google-Extended, PerplexityBot) to crawl and use its content. It reads the site’s robots.txt, evaluates access rules for major AI/LLM crawlers, and returns an allow/deny matrix plus practical suggestions (e.g., what to add/change in robots.txt) if access is blocked.

Why it’s useful (benefits)

  • Know your AI exposure: Quickly see if LLMs can crawl or train on your content.

  • Compliance & policy control: Verify that your robots.txt reflects your intended AI policy.

  • Easy fixes: Get specific suggestions to enable/limit AI crawling.

  • Bulk auditing: Integrate into your CI/SEO pipelines to monitor many domains.

Base URL

https://vebapi.com

Endpoint

GET /api/seo/aiseochecker

Authentication

Send your API key in the header: X-API-KEY: YOUR_API_KEY

Input Parameters

Name In Type Required Description Example
website query string Yes Domain or URL to check. Subdomains are treated as provided (robots.txt is fetched from that host). codeconia.com or https://codeconia.com

Notes
• The service fetches https://<host>/robots.txt (or http if https is unavailable).
• If website is a full URL, the host component is used for robots.txt.

Output (Response Body)

Field Type Description Example
url string The normalized host or site you asked to check. "codeconia.com"
robots_found boolean Whether a robots.txt file was found and parsed successfully. true
ai_access object Key/value map of major AI/LLM crawler user-agents to boolean access. true = allowed, false = blocked. See example on your page
ai_bots_allowed boolean Overall flag: true if all tracked AI bots are allowed, false if one or more are blocked. true
suggestions string[] Human-readable suggestions based on the analysis (how to enable or improve AI crawling/compliance). See example on your page

ai_access keys (typical)

GPTBot, ChatGPT-User, Google-Extended, AnthropicBot, ClaudeBot, PerplexityBot, CCBot, Amazonbot, Bytespider, facebookexternalhit, cohere-ai, YouBot, NeevaBot, ai-crawler, Applebot, Baiduspider, Sogou, YandexBot, PhindBot, DuckDuckBot, Yeti, 360Spider, ias-va

Meaning: For each listed bot, true = allowed by current robots.txt rules; false = blocked (via Disallow for that UA or wildcard rules that match it).

Example

You already display an example request and response on the page, so they’re not duplicated here.

Status Codes (typical)

  • 200 OK – Analysis succeeded.

  • 400 Bad Request – Missing/invalid website parameter.

  • 401 Unauthorized – Missing/invalid X-API-KEY.

  • 404 Not Found – Host reachable but no robots.txt and no fallback (you’ll still get robots_found: false if host exists).

  • 500 Server Error – Unexpected error while fetching/parsing.

Implementation Notes & Best Practices

  • To allow GPTBot explicitly, add for example:

    User-agent: GPTBot
Allow: /

  • To opt out of certain AI bots, specify:

    User-agent: GPTBot
Disallow: /

  • Prefer host-specific rules if you have multiple subdomains.

  • Re-check after changes; CDN caches can delay robots.txt propagation.

Request Example (cURL)
Expected JSON Response 
{
    "url": "codeconia.com",
    "robots_found": true,
    "ai_access": {
        "GPTBot": true,
        "ChatGPT-User": true,
        "Google-Extended": true,
        "AnthropicBot": true,
        "ClaudeBot": true,
        "PerplexityBot": true,
        "CCBot": true,
        "Amazonbot": true,
        "Bytespider": true,
        "facebookexternalhit": true,
        "cohere-ai": true,
        "YouBot": true,
        "NeevaBot": true,
        "ai-crawler": true,
        "Applebot": true,
        "Baiduspider": true,
        "Sogou": true,
        "YandexBot": true,
        "PhindBot": true,
        "DuckDuckBot": true,
        "Yeti": true,
        "360Spider": true,
        "ias-va": true
    },
    "ai_bots_allowed": true,
    "suggestions": [
        "Your site is currently open for AI bots. You're AI-friendly!",
        "You can still improve by providing structured data (schema.org) for better AI comprehension.",
        "Add clear content usage terms if you want to allow or limit AI training use."
    ]
}

Flexible Developer Pricing

Select the plan that fits your execution volume. Standard plans come with full access to all endpoints, high concurrency limits, and active support.

🔥 Special Offer Pricing

Simple, Transparent Pricing

Choose a plan now, scale anytime. One API key for all endpoints.

Monthly billing Yearly billing 2 months free
Loading pricing…
Enterprise & High Volume

Need more than 1 million requests a month?

Get volume-based pricing, custom rate limits, dedicated support, and tailored SLAs. Our team will build a plan around your scale.

  • ✓ Custom credit volume
  • ✓ Higher rate limits
  • ✓ Priority support & SLA
  • ✓ Flexible billing
Contact Support

Typical reply within 24 hours