{"result":[{"_id":"legal-privacy","_type":"legalPage","_updatedAt":"2026-10-05T08:04:43Z","body":null,"category":null,"excerpt":null,"longDescription":null,"publishedAt":null,"readTime":null,"sections":[{"body":[{"_key":"k00005","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"IndexLinks (indexlinks.app) is operated by Vynara Technologies. This policy explains what we collect when you use the IndexLinks website, app, API and AI connector, why, and who else processes it. Questions: "},{"_key":"k00003","_type":"span","marks":["k00002"],"text":"hello@indexlinks.app"},{"_key":"k00004","_type":"span","marks":[],"text":"."}],"markDefs":[{"_key":"k00002","_type":"link","href":"mailto:hello@indexlinks.app"}],"style":"normal"}],"heading":"Who we are"},{"body":[{"_key":"k00003","_type":"block","children":[{"_key":"k00001","_type":"span","marks":["strong"],"text":"Account details."},{"_key":"k00002","_type":"span","marks":[],"text":" Your email address, your name if you give it, and a salted hash of your password. If you sign in with Google, we receive your Google account ID, email and name. We never receive your Google password."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00006","_type":"block","children":[{"_key":"k00004","_type":"span","marks":["strong"],"text":"Websites and pages."},{"_key":"k00005","_type":"span","marks":[],"text":" The websites you add, the page URLs and sitemaps you submit or that we read from your sitemap, and what we record about each page: its title and description as fetched, its delivery status for each search engine, and suggestions about the page."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00009","_type":"block","children":[{"_key":"k00007","_type":"span","marks":["strong"],"text":"Crawler visits."},{"_key":"k00008","_type":"span","marks":[],"text":" If you connect a host such as Cloudflare or Vercel so we can show crawl receipts, we record search engine and AI crawler visits to your pages: the time, the page, the crawler's user agent and IP address, and whether it was verified. We use this only to show you which crawlers visited."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00012","_type":"block","children":[{"_key":"k00010","_type":"span","marks":["strong"],"text":"Connected services."},{"_key":"k00011","_type":"span","marks":[],"text":" If you connect Google Search Console, we store an encrypted token with read-only access to show your pages' Google status. If you connect Cloudflare or Vercel, we store an encrypted token for the access you approve. You can disconnect any of them in the app."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00015","_type":"block","children":[{"_key":"k00013","_type":"span","marks":["strong"],"text":"API keys and AI connectors."},{"_key":"k00014","_type":"span","marks":[],"text":" We store a hash of each API key, its name and when it was last used. When you connect IndexLinks to an AI assistant such as Claude or ChatGPT, we create a key named after that assistant. We receive only the requests the assistant makes through the IndexLinks tools you use; we never see your conversations with it."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00018","_type":"block","children":[{"_key":"k00016","_type":"span","marks":["strong"],"text":"Billing."},{"_key":"k00017","_type":"span","marks":[],"text":" Payments are handled by Dodo Payments. We receive your plan, subscription status and billing period, not your full card details."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00021","_type":"block","children":[{"_key":"k00019","_type":"span","marks":["strong"],"text":"Free checks."},{"_key":"k00020","_type":"span","marks":[],"text":" When you run a free site check, we fetch the public page you enter and keep the report for a short time so you can share its link."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00024","_type":"block","children":[{"_key":"k00022","_type":"span","marks":["strong"],"text":"Usage."},{"_key":"k00023","_type":"span","marks":[],"text":" We collect basic product usage to understand how IndexLinks is used and to fix problems."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"}],"heading":"What we collect"},{"body":[{"_key":"k00002","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"We use your data to run the service you signed up for: send your pages to search engines and AI crawlers, show their status and crawl receipts, bill your plan, send the emails you ask for, answer support requests, and keep the service secure, for example with rate limits and abuse prevention. We do not sell your personal data and we do not use it for advertising."}],"markDefs":[],"style":"normal"}],"heading":"How we use it"},{"body":[{"_key":"k00003","_type":"block","children":[{"_key":"k00001","_type":"span","marks":["strong"],"text":"Search engines and crawler networks"},{"_key":"k00002","_type":"span","marks":[],"text":", which receive the page URLs you submit: Bing, Yandex, Naver, Seznam, Yep and others through IndexNow, Baidu if you configure it, and Google through your own Search Console. Submitted page URLs can also appear on our public discovery pages so AI crawlers can find them."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00006","_type":"block","children":[{"_key":"k00004","_type":"span","marks":["strong"],"text":"Service providers"},{"_key":"k00005","_type":"span","marks":[],"text":" that process data for us: Cloudflare (hosting and database), Dodo Payments (payments), SendHQ (email) and Google (sign-in and Search Console)."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00009","_type":"block","children":[{"_key":"k00007","_type":"span","marks":["strong"],"text":"The AI assistant you connect"},{"_key":"k00008","_type":"span","marks":[],"text":", which receives the results of the tools it runs for you."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00012","_type":"block","children":[{"_key":"k00010","_type":"span","marks":["strong"],"text":"Authorities"},{"_key":"k00011","_type":"span","marks":[],"text":", when the law requires it."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"}],"heading":"Who we share it with"},{"body":[{"_key":"k00002","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"We keep data only as long as we need it. Crawler visit records are deleted after 90 days. Free check reports are deleted after 14 days, and anonymous AI readiness reports after 30 days. Account details, websites and page records are kept while your account is open; when you ask us to close your account, we delete them within 30 days, except billing records we must keep by law."}],"markDefs":[],"style":"normal"}],"heading":"How long we keep it"},{"body":[{"_key":"k00005","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"You can disconnect Search Console, Cloudflare, Vercel and AI connectors, and delete API keys and websites, in the app at any time. To access, correct, export or delete your data, or to close your account, email "},{"_key":"k00003","_type":"span","marks":["k00002"],"text":"hello@indexlinks.app"},{"_key":"k00004","_type":"span","marks":[],"text":". Depending on where you live, for example the EU, UK or California, you may have further rights, including the right to object and to complain to a data protection authority."}],"markDefs":[{"_key":"k00002","_type":"link","href":"mailto:hello@indexlinks.app"}],"style":"normal"}],"heading":"Your choices and rights"},{"body":[{"_key":"k00002","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"Passwords are stored as salted hashes, connection tokens are encrypted, and API keys are stored as hashes. All traffic uses HTTPS."}],"markDefs":[],"style":"normal"}],"heading":"Security"},{"body":[{"_key":"k00002","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"IndexLinks is not meant for children under 16, and we do not knowingly collect their data."}],"markDefs":[],"style":"normal"}],"heading":"Children"},{"body":[{"_key":"k00002","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"We will update this page when this policy changes. For significant changes we will email account holders."}],"markDefs":[],"style":"normal"}],"heading":"Changes"},{"body":[{"_key":"k00005","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"Vynara Technologies, "},{"_key":"k00003","_type":"span","marks":["k00002"],"text":"hello@indexlinks.app"},{"_key":"k00004","_type":"span","marks":[],"text":"."}],"markDefs":[{"_key":"k00002","_type":"link","href":"mailto:hello@indexlinks.app"}],"style":"normal"}],"heading":"Contact"}],"seoDescription":null,"seoTitle":null,"shortDescription":"How IndexLinks collects, uses and protects your data.","slug":"privacy","status":"published","title":"Privacy policy","toolType":null},{"_id":"legal-terms","_type":"legalPage","_updatedAt":"2026-10-05T08:04:43Z","body":null,"category":null,"excerpt":null,"longDescription":null,"publishedAt":null,"readTime":null,"sections":[{"body":[{"_key":"k00002","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"These terms are an agreement between you and Vynara Technologies, which operates IndexLinks (indexlinks.app). By creating an account or using the IndexLinks website, app, API or AI connector, you accept them. If you use IndexLinks for a company, you accept them on its behalf."}],"markDefs":[],"style":"normal"}],"heading":"Agreement"},{"body":[{"_key":"k00002","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"IndexLinks sends your pages to search engines and AI crawlers through their official channels (IndexNow, your Google Search Console, Baidu's push API and our discovery pages), checks how crawlers see your site, and shows the delivery status and verified crawler visits for each page. Search engines decide on their own whether and when to crawl or index a page. IndexLinks does not guarantee that any page will be indexed or ranked."}],"markDefs":[],"style":"normal"}],"heading":"What IndexLinks does"},{"body":[{"_key":"k00002","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"Keep your login and API keys secure; you are responsible for what happens under your account, including actions taken by AI assistants you connect. Give us accurate information and tell us if you think your account has been misused."}],"markDefs":[],"style":"normal"}],"heading":"Your account"},{"body":[{"_key":"k00002","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"Only add websites you own or are authorised to manage."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00004","_type":"block","children":[{"_key":"k00003","_type":"span","marks":[],"text":"Do not use IndexLinks to submit spam, links on sites you do not control, link schemes or private blog networks, or content that is illegal or infringes others' rights."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00006","_type":"block","children":[{"_key":"k00005","_type":"span","marks":[],"text":"Do not try to break, overload, scrape or get around the limits or security of the service."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00008","_type":"block","children":[{"_key":"k00007","_type":"span","marks":[],"text":"We may suspend submissions or accounts that break these rules."}],"markDefs":[],"style":"normal"}],"heading":"Acceptable use"},{"body":[{"_key":"k00005","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"Paid plans are billed monthly or yearly in advance through Dodo Payments, at the prices shown on the "},{"_key":"k00003","_type":"span","marks":["k00002"],"text":"pricing page"},{"_key":"k00004","_type":"span","marks":[],"text":". Each plan includes a monthly page allowance; extra pages are billed as shown there. A submitted page that does not get a verified crawler visit within 24 hours does not count toward your plan. You can cancel at any time; your plan stays active until the end of the period you paid for. We will give notice before changing the price of your plan."}],"markDefs":[{"_key":"k00002","_type":"link","href":"https://indexlinks.app/pricing"}],"style":"normal"}],"heading":"Plans and billing"},{"body":[{"_key":"k00002","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"You can use the IndexLinks API and AI connector within your plan's limits. Actions an AI assistant takes through the connector, such as sending pages, count as your actions and use your allowance. Do not share your API keys."}],"markDefs":[],"style":"normal"}],"heading":"API and AI connector"},{"body":[{"_key":"k00002","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"IndexLinks works with services we do not control, including search engines, Google Search Console, Cloudflare, Vercel and AI assistants. Their own terms apply to your use of them, and we are not responsible for their availability or decisions."}],"markDefs":[],"style":"normal"}],"heading":"Third-party services"},{"body":[{"_key":"k00005","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"You keep all rights to your websites and data. You allow us to use them only to provide the service, as described in our "},{"_key":"k00003","_type":"span","marks":["k00002"],"text":"privacy policy"},{"_key":"k00004","_type":"span","marks":[],"text":"."}],"markDefs":[{"_key":"k00002","_type":"link","href":"https://indexlinks.app/privacy"}],"style":"normal"}],"heading":"Your content"},{"body":[{"_key":"k00005","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"You can stop using IndexLinks and ask us to close your account at any time by emailing "},{"_key":"k00003","_type":"span","marks":["k00002"],"text":"hello@indexlinks.app"},{"_key":"k00004","_type":"span","marks":[],"text":". We may suspend or close accounts that break these terms, or end the service with reasonable notice."}],"markDefs":[{"_key":"k00002","_type":"link","href":"mailto:hello@indexlinks.app"}],"style":"normal"}],"heading":"Ending the service"},{"body":[{"_key":"k00002","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"IndexLinks is provided as is. We work to keep it reliable, but we do not promise it will be uninterrupted or error free, or that search engines will act on your submissions. To the extent the law allows, we are not liable for indirect or consequential losses, or for lost profits, revenue or rankings, and our total liability is limited to the fees you paid us in the 12 months before the claim."}],"markDefs":[],"style":"normal"}],"heading":"Disclaimers and liability"},{"body":[{"_key":"k00002","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"We may update these terms. We will post the new version here and, for significant changes, email account holders before they take effect."}],"markDefs":[],"style":"normal"}],"heading":"Changes"},{"body":[{"_key":"k00005","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"Vynara Technologies, "},{"_key":"k00003","_type":"span","marks":["k00002"],"text":"hello@indexlinks.app"},{"_key":"k00004","_type":"span","marks":[],"text":"."}],"markDefs":[{"_key":"k00002","_type":"link","href":"mailto:hello@indexlinks.app"}],"style":"normal"}],"heading":"Contact"}],"seoDescription":null,"seoTitle":null,"shortDescription":"The terms for using IndexLinks.","slug":"terms","status":"published","title":"Terms of service","toolType":null},{"_id":"post-ai-crawlers-blocked-cloudflare-robots-txt-misses","_type":"post","_updatedAt":"2026-10-13T11:01:36Z","body":[{"_key":"k00002","_type":"block","children":[{"_key":"k00001","_type":"span","marks":[],"text":"A post going around said Cloudflare blocks Common Crawl's CCBot without asking the site owner. We run about 15 domains on Cloudflare, so we checked every one of them: the Cloudflare settings, the live robots.txt files, and a few days of request logs filtered by crawler user agent."}],"markDefs":[],"style":"normal"},{"_key":"k00004","_type":"block","children":[{"_key":"k00003","_type":"span","marks":[],"text":"Cloudflare had been refusing verified AI crawlers on two of our domains until recently. Most of what we found, though, we had written ourselves. Below are the six misses, why each one happens, and how to check your own site in a few minutes."}],"markDefs":[],"style":"normal"},{"_key":"k00006","_type":"block","children":[{"_key":"k00005","_type":"span","marks":[],"text":"Why robots.txt is only half the answer"}],"markDefs":[],"style":"h2"},{"_key":"k00010","_type":"block","children":[{"_key":"k00007","_type":"span","marks":[],"text":"Most guides about AI crawlers stop at robots.txt. robots.txt is a request to well-behaved crawlers, and it is only read if the crawler gets through. A CDN or firewall can refuse a crawler before robots.txt comes into it. In that case your robots.txt can say "},{"_key":"k00008","_type":"span","marks":["code"],"text":"Allow: /"},{"_key":"k00009","_type":"span","marks":[],"text":" while GPTBot gets a 403 on every page, and nothing in your CMS or your repository will show it."}],"markDefs":[],"style":"normal"},{"_key":"k00012","_type":"block","children":[{"_key":"k00011","_type":"span","marks":[],"text":"So there are two places to look: what your robots.txt says, and what your edge actually answers when a crawler asks for a page."}],"markDefs":[],"style":"normal"},{"_key":"k00014","_type":"block","children":[{"_key":"k00013","_type":"span","marks":[],"text":"Miss 1: Cloudflare's AI bot blocking was switched on"}],"markDefs":[],"style":"h2"},{"_key":"k00022","_type":"block","children":[{"_key":"k00015","_type":"span","marks":[],"text":"On July 1, 2025 Cloudflare announced it was "},{"_key":"k00017","_type":"span","marks":["k00016"],"text":"changing the default to block AI crawlers"},{"_key":"k00018","_type":"span","marks":[],"text":" unless they pay. On July 1, 2026 it split AI traffic into three groups, Search, Agent and Training, each of which can be allowed, blocked, or blocked only on pages that show ads. According to the "},{"_key":"k00020","_type":"span","marks":["k00019"],"text":"Cloudflare bots changelog"},{"_key":"k00021","_type":"span","marks":[],"text":", since September 15, 2026 new domains get Training and Agent bots blocked on pages that display ads, and crawlers that do both search and training are caught by the Training block."}],"markDefs":[{"_key":"k00016","_type":"link","href":"https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/"},{"_key":"k00019","_type":"link","href":"https://developers.cloudflare.com/changelog/product/bots/"}],"style":"normal"},{"_key":"k00024","_type":"block","children":[{"_key":"k00023","_type":"span","marks":[],"text":"On two of our domains the older \"Block AI bots\" setting was on. Verified AI crawlers were turned away there until late September on one and early October on the other. Nothing on the sites themselves showed it. We found it only by filtering the logs by crawler user agent."}],"markDefs":[],"style":"normal"},{"_key":"k00027","_type":"block","children":[{"_key":"k00025","_type":"span","marks":["strong"],"text":"How to check:"},{"_key":"k00026","_type":"span","marks":[],"text":" in the Cloudflare dashboard, open Security settings and look at the AI bot policies, AI Crawl Control and Bot Fight Mode. Then check that each crawler you want is set to Allow."}],"markDefs":[],"style":"normal"},{"_key":"k00029","_type":"block","children":[{"_key":"k00028","_type":"span","marks":[],"text":"Miss 2: our own robots.txt blocked training crawlers"}],"markDefs":[],"style":"h2"},{"_key":"k00041","_type":"block","children":[{"_key":"k00030","_type":"span","marks":[],"text":"Four of our sites had "},{"_key":"k00031","_type":"span","marks":["code"],"text":"User-agent: CCBot"},{"_key":"k00032","_type":"span","marks":[],"text":" and "},{"_key":"k00033","_type":"span","marks":["code"],"text":"User-agent: Bytespider"},{"_key":"k00034","_type":"span","marks":[],"text":" groups with "},{"_key":"k00035","_type":"span","marks":["code"],"text":"Disallow: /"},{"_key":"k00036","_type":"span","marks":[],"text":", and a "},{"_key":"k00037","_type":"span","marks":["code"],"text":"Content-Signal"},{"_key":"k00038","_type":"span","marks":[],"text":" line with "},{"_key":"k00039","_type":"span","marks":["code"],"text":"ai-train=no"},{"_key":"k00040","_type":"span","marks":[],"text":". They were added during an earlier round of changes meant to make the sites ready for AI agents, and nobody had looked at them since."}],"markDefs":[],"style":"normal"},{"_key":"k00043","_type":"block","children":[{"_key":"k00042","_type":"span","marks":[],"text":"Training crawlers matter even if you only care about being cited. Common Crawl's archive feeds many training datasets, and what a model knows about your brand, product and prices comes partly from crawls like these. Block them and the model answers about you from older or second-hand sources. You can block them on purpose. Just make sure it is a decision someone made."}],"markDefs":[],"style":"normal"},{"_key":"k00052","_type":"block","children":[{"_key":"k00044","_type":"span","marks":[],"text":"Cloudflare can also add these lines for you. Its "},{"_key":"k00046","_type":"span","marks":["k00045"],"text":"managed robots.txt"},{"_key":"k00047","_type":"span","marks":[],"text":" puts "},{"_key":"k00048","_type":"span","marks":["code"],"text":"Content-signal: search=yes, ai-train=no"},{"_key":"k00049","_type":"span","marks":[],"text":" and "},{"_key":"k00050","_type":"span","marks":["code"],"text":"Disallow: /"},{"_key":"k00051","_type":"span","marks":[],"text":" groups for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent in front of your own file, and the two are served as one response. With that setting on, a site that has no robots.txt of its own gets one created with those rules."}],"markDefs":[{"_key":"k00045","_type":"link","href":"https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/"}],"style":"normal"},{"_key":"k00057","_type":"block","children":[{"_key":"k00053","_type":"span","marks":["strong"],"text":"How to check:"},{"_key":"k00054","_type":"span","marks":[],"text":" open "},{"_key":"k00055","_type":"span","marks":["code"],"text":"https://yourdomain.com/robots.txt"},{"_key":"k00056","_type":"span","marks":[],"text":" in a browser and read every group. Read the live file, not the copy in your repository, because the copy in your repository may not be what crawlers get."}],"markDefs":[],"style":"normal"},{"_key":"k00059","_type":"block","children":[{"_key":"k00058","_type":"span","marks":[],"text":"Miss 3: render files disallowed"}],"markDefs":[],"style":"h2"},{"_key":"k00070","_type":"block","children":[{"_key":"k00060","_type":"span","marks":[],"text":"Two of our sites disallowed "},{"_key":"k00061","_type":"span","marks":["code"],"text":"/_next/"},{"_key":"k00062","_type":"span","marks":[],"text":", "},{"_key":"k00063","_type":"span","marks":["code"],"text":"/static/"},{"_key":"k00064","_type":"span","marks":[],"text":" and "},{"_key":"k00065","_type":"span","marks":["code"],"text":"*.json"},{"_key":"k00066","_type":"span","marks":[],"text":". Those paths hold the JavaScript, styles and data that a crawler needs to render the page. Google's "},{"_key":"k00068","_type":"span","marks":["k00067"],"text":"robots.txt introduction"},{"_key":"k00069","_type":"span","marks":[],"text":" is direct about this: if the absence of these resources makes the page harder for Google's crawler to understand, don't block them. AI crawlers that render pages are in the same position."}],"markDefs":[{"_key":"k00067","_type":"link","href":"https://developers.google.com/search/docs/crawling-indexing/robots/intro"}],"style":"normal"},{"_key":"k00091","_type":"block","children":[{"_key":"k00071","_type":"span","marks":["strong"],"text":"How to check:"},{"_key":"k00072","_type":"span","marks":[],"text":" search your robots.txt for build folders such as "},{"_key":"k00073","_type":"span","marks":["code"],"text":"/_next/"},{"_key":"k00074","_type":"span","marks":[],"text":", "},{"_key":"k00075","_type":"span","marks":["code"],"text":"/static/"},{"_key":"k00076","_type":"span","marks":[],"text":", "},{"_key":"k00077","_type":"span","marks":["code"],"text":"/assets/"},{"_key":"k00078","_type":"span","marks":[],"text":" or "},{"_key":"k00079","_type":"span","marks":["code"],"text":"/build/"},{"_key":"k00080","_type":"span","marks":[],"text":", and for file patterns like "},{"_key":"k00081","_type":"span","marks":["code"],"text":"*.js"},{"_key":"k00082","_type":"span","marks":[],"text":", "},{"_key":"k00083","_type":"span","marks":["code"],"text":"*.css"},{"_key":"k00084","_type":"span","marks":[],"text":" or "},{"_key":"k00085","_type":"span","marks":["code"],"text":"*.json"},{"_key":"k00086","_type":"span","marks":[],"text":". Keep disallows for private paths such as "},{"_key":"k00087","_type":"span","marks":["code"],"text":"/api/"},{"_key":"k00088","_type":"span","marks":[],"text":" or "},{"_key":"k00089","_type":"span","marks":["code"],"text":"/admin/"},{"_key":"k00090","_type":"span","marks":[],"text":", and remove the ones that cover render files."}],"markDefs":[],"style":"normal"},{"_key":"k00093","_type":"block","children":[{"_key":"k00092","_type":"span","marks":[],"text":"Miss 4: Crawl-delay lines"}],"markDefs":[],"style":"h2"},{"_key":"k00100","_type":"block","children":[{"_key":"k00094","_type":"span","marks":[],"text":"The same two sites had "},{"_key":"k00095","_type":"span","marks":["code"],"text":"Crawl-delay"},{"_key":"k00096","_type":"span","marks":[],"text":" lines. Google's "},{"_key":"k00098","_type":"span","marks":["k00097"],"text":"robots.txt documentation"},{"_key":"k00099","_type":"span","marks":[],"text":" says fields such as crawl-delay aren't supported, so Google ignores them. Some other crawlers do honor them, so the only effect was to slow down everyone except Google. Unless a crawler is actually overloading your server, remove them."}],"markDefs":[{"_key":"k00097","_type":"link","href":"https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt"}],"style":"normal"},{"_key":"k00102","_type":"block","children":[{"_key":"k00101","_type":"span","marks":[],"text":"Miss 5: Browser Integrity Check and error 1010"}],"markDefs":[],"style":"h2"},{"_key":"k00107","_type":"block","children":[{"_key":"k00103","_type":"span","marks":[],"text":"Cloudflare's Browser Integrity Check was on for all our zones. It returns "},{"_key":"k00105","_type":"span","marks":["k00104"],"text":"error 1010"},{"_key":"k00106","_type":"span","marks":[],"text":", \"The owner of this website has banned your access based on your browser's signature\", to clients whose request looks wrong to it. In our tests it refused requests with Python and Java HTTP library user agents and some malformed ones. The major crawler user agents passed."}],"markDefs":[{"_key":"k00104","_type":"link","href":"https://developers.cloudflare.com/support/troubleshooting/http-status-codes/cloudflare-1xxx-errors/error-1010/"}],"style":"normal"},{"_key":"k00111","_type":"block","children":[{"_key":"k00108","_type":"span","marks":[],"text":"That still matters. Many fetch tools, feed readers and AI agents that pull a page during a conversation use plain HTTP libraries. We turned the check off. One of our sites is on Cloudflare Pages, and there the "},{"_key":"k00109","_type":"span","marks":["code"],"text":"pages.dev"},{"_key":"k00110","_type":"span","marks":[],"text":" hostname itself returned 1010 to the same clients, which no zone setting changed."}],"markDefs":[],"style":"normal"},{"_key":"k00113","_type":"block","children":[{"_key":"k00112","_type":"span","marks":[],"text":"Miss 6: refusals that are not settings at all"}],"markDefs":[],"style":"h2"},{"_key":"k00115","_type":"block","children":[{"_key":"k00114","_type":"span","marks":[],"text":"Not every refusal comes from a setting. In the same logs one section of a site returned 522 timeouts to GPTBot, and another site's preview images returned thousands of 500 errors in three days. The crawler sees an error either way, and no checklist of settings will catch it."}],"markDefs":[],"style":"normal"},{"_key":"k00118","_type":"block","children":[{"_key":"k00116","_type":"span","marks":["strong"],"text":"How to check:"},{"_key":"k00117","_type":"span","marks":[],"text":" in your CDN or server logs, group responses by user agent and status code for the crawlers you care about, and look at every 4xx and 5xx."}],"markDefs":[],"style":"normal"},{"_key":"k00120","_type":"block","children":[{"_key":"k00119","_type":"span","marks":[],"text":"How to test your own site"}],"markDefs":[],"style":"h2"},{"_key":"k00122","_type":"block","children":[{"_key":"k00121","_type":"span","marks":[],"text":"Read your live robots.txt from top to bottom, including anything your CDN adds in front of it."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00126","_type":"block","children":[{"_key":"k00123","_type":"span","marks":[],"text":"Request one of your pages with a browser user agent, then again with each crawler's user agent, and compare the status codes. A quick way is "},{"_key":"k00124","_type":"span","marks":["code"],"text":"curl -s -o /dev/null -w \"%{http_code}\" -A \"CCBot/2.0 (https://commoncrawl.org/faq/)\" https://yourdomain.com/"},{"_key":"k00125","_type":"span","marks":[],"text":". A 403, a challenge page, or a 402 (Cloudflare's pay per crawl) means that crawler doesn't get the page."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00128","_type":"block","children":[{"_key":"k00127","_type":"span","marks":[],"text":"In Cloudflare, check AI bot policies, Bot Fight Mode, Browser Integrity Check, managed robots.txt, and any WAF custom rules that match user agents."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00130","_type":"block","children":[{"_key":"k00129","_type":"span","marks":[],"text":"Look at your logs for errors returned to crawlers."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00132","_type":"block","children":[{"_key":"k00131","_type":"span","marks":[],"text":"One caveat about step 2: your test comes from your own IP address, not the crawler's published ranges. A verified crawler can sometimes get through where your test is refused, and the reverse can happen with rules based on IP addresses. User-agent rules and AI bot blocking usually catch both, so a refusal in this test is worth checking."}],"markDefs":[],"style":"normal"},{"_key":"k00134","_type":"block","children":[{"_key":"k00133","_type":"span","marks":[],"text":"What we changed in the IndexLinks check"}],"markDefs":[],"style":"h2"},{"_key":"k00139","_type":"block","children":[{"_key":"k00135","_type":"span","marks":[],"text":"We added this test to the free "},{"_key":"k00137","_type":"span","marks":["k00136"],"text":"AI Search Readiness check"},{"_key":"k00138","_type":"span","marks":[],"text":". It now requests your page as 10 AI crawlers with their real user agents:"}],"markDefs":[{"_key":"k00136","_type":"link","href":"https://indexlinks.app/ai-search-readiness"}],"style":"normal"},{"_key":"k00141","_type":"block","children":[{"_key":"k00140","_type":"span","marks":[],"text":"Answer crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Amazonbot."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00143","_type":"block","children":[{"_key":"k00142","_type":"span","marks":[],"text":"Training crawlers: GPTBot, ClaudeBot, CCBot, meta-externalagent and Bytespider."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00145","_type":"block","children":[{"_key":"k00144","_type":"span","marks":[],"text":"Each answer is compared with what a browser gets, and the report names any crawler that was refused, with the status code and the Cloudflare error code when there is one. Training-crawler blocks, in robots.txt or at the edge, now fail the check instead of showing as a note. The homepage site check runs the same test. Customers who turn on the weekly re-check get an email that names any crawler that has become blocked since the last run, so a settings change or a new default doesn't go unnoticed for weeks the way ours did."}],"markDefs":[],"style":"normal"},{"_key":"k00147","_type":"block","children":[{"_key":"k00146","_type":"span","marks":[],"text":"None of this makes a page appear in an AI answer. It only tells you whether the crawlers can reach the page at all, which is the first thing to rule out."}],"markDefs":[],"style":"normal"}],"category":"Crawlers","excerpt":"Cloudflare settings, our own robots.txt and a bot check were turning AI crawlers like GPTBot and CCBot away. Six misses from an audit of our own sites, and how to test yours.","longDescription":null,"publishedAt":"2026-10-13T11:01:35Z","readTime":"7 min read","sections":null,"seoDescription":"Cloudflare settings, robots.txt lines and bot checks can turn GPTBot, ClaudeBot and CCBot away. Six misses from our own sites, and how to test yours.","seoTitle":"Is Cloudflare Blocking AI Crawlers? 6 Misses to Check","shortDescription":null,"slug":"ai-crawlers-blocked-cloudflare-robots-txt-misses","status":"published","title":"AI Crawlers Blocked Without Anyone Deciding: 6 Misses We Found on Our Own Sites","toolType":null},{"_id":"post-crawled-currently-not-indexed","_type":"post","_updatedAt":"2026-10-10T09:37:27Z","body":[{"_key":"k00003","_type":"block","children":[{"_key":"k00001","_type":"span","marks":["strong"],"text":"TL;DR:"},{"_key":"k00002","_type":"span","marks":[],"text":" \"Crawled - currently not indexed\" is a Google Search Console status meaning Googlebot fetched the page but Google chose not to add it to the index. It is a judgment about the page, not a crawl error, so resubmitting the URL rarely helps. Fix it by confirming the page should be indexed, then making it clearly unique, useful and well linked, and validating the fix in the Page indexing report."}],"markDefs":[],"style":"normal"},{"_key":"k00008","_type":"block","children":[{"_key":"k00004","_type":"span","marks":[],"text":"Crawled currently not indexed is the status Google gives a URL that Googlebot has already visited and processed, but that Google decided not to store in its index for now. Google's own definition in the "},{"_key":"k00006","_type":"span","marks":["k00005"],"text":"Page indexing report documentation"},{"_key":"k00007","_type":"span","marks":[],"text":" is short: \"The page was crawled by Google but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.\" Everything below follows from that sentence: the crawl already happened, so the fix is never \"get it crawled again\". The fix is giving Google a reason to keep the page."}],"markDefs":[{"_key":"k00005","_type":"link","href":"https://support.google.com/webmasters/answer/7440203"}],"style":"normal"},{"_key":"k00010","_type":"block","children":[{"_key":"k00009","_type":"span","marks":[],"text":"What does \"Crawled - currently not indexed\" mean in Google Search Console?"}],"markDefs":[],"style":"h2"},{"_key":"k00012","_type":"block","children":[{"_key":"k00011","_type":"span","marks":[],"text":"\"Crawled - currently not indexed\" means Google reached the URL, received a response, rendered the content, and then left it out of the index. You find it in Search Console under Indexing > Pages, in the \"Why pages aren't indexed\" table, with the source listed as Google systems rather than Website."}],"markDefs":[],"style":"normal"},{"_key":"k00014","_type":"block","children":[{"_key":"k00013","_type":"span","marks":[],"text":"That source label matters. Statuses with source Website (for example \"Excluded by 'noindex' tag\" or \"Blocked by robots.txt\") are caused by something on your site that you can switch off. Crawled currently not indexed has no single switch. Google has seen the page and decided, at least for now, that it does not add enough to be worth indexing."}],"markDefs":[],"style":"normal"},{"_key":"k00016","_type":"block","children":[{"_key":"k00015","_type":"span","marks":[],"text":"Compare it with its neighbour:"}],"markDefs":[],"style":"normal"},{"_key":"k00022","_type":"block","children":[{"_key":"k00017","_type":"span","marks":["strong"],"text":"Discovered - currently not indexed:"},{"_key":"k00018","_type":"span","marks":[],"text":" Google knows the URL but has not crawled it yet. The last crawl date is empty. That is a crawl scheduling problem. See our guide to "},{"_key":"k00020","_type":"span","marks":["k00019"],"text":"discovered currently not indexed"},{"_key":"k00021","_type":"span","marks":[],"text":"."}],"level":1,"listItem":"bullet","markDefs":[{"_key":"k00019","_type":"link","href":"https://indexlinks.app/blog/discovered-currently-not-indexed"}],"style":"normal"},{"_key":"k00025","_type":"block","children":[{"_key":"k00023","_type":"span","marks":["strong"],"text":"Crawled - currently not indexed:"},{"_key":"k00024","_type":"span","marks":[],"text":" Google has crawled it, and the last crawl date is filled in. That is an indexing decision."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00027","_type":"block","children":[{"_key":"k00026","_type":"span","marks":[],"text":"Is this status always a problem?"}],"markDefs":[],"style":"h2"},{"_key":"k00029","_type":"block","children":[{"_key":"k00028","_type":"span","marks":[],"text":"No. Google states in the same report documentation that \"Non-indexed URLs can be fine\" and that most sites have some unindexed pages. Before fixing anything, sort the affected URLs into two groups:"}],"markDefs":[],"style":"normal"},{"_key":"k00040","_type":"block","children":[{"_key":"k00030","_type":"span","marks":["strong"],"text":"URLs that should not be indexed anyway."},{"_key":"k00031","_type":"span","marks":[],"text":" Typical examples: tag and author archives, "},{"_key":"k00032","_type":"span","marks":["code"],"text":"?page=2"},{"_key":"k00033","_type":"span","marks":[],"text":" pagination, internal search results, filter and sort parameters such as "},{"_key":"k00034","_type":"span","marks":["code"],"text":"?color=red&sort=price"},{"_key":"k00035","_type":"span","marks":[],"text":", feed URLs like "},{"_key":"k00036","_type":"span","marks":["code"],"text":"/feed/"},{"_key":"k00037","_type":"span","marks":[],"text":", thin attachment pages in WordPress, staging or print versions. For these, the right move is usually to stop generating or linking them, add "},{"_key":"k00038","_type":"span","marks":["code"],"text":"noindex"},{"_key":"k00039","_type":"span","marks":[],"text":", or canonicalize them, not to push them into the index."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00043","_type":"block","children":[{"_key":"k00041","_type":"span","marks":["strong"],"text":"URLs you need in Google."},{"_key":"k00042","_type":"span","marks":[],"text":" Product pages, articles, landing pages, docs. These are the ones worth work."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00047","_type":"block","children":[{"_key":"k00044","_type":"span","marks":[],"text":"Google also notes that sites with fewer than 500 pages probably do not need to study this report at all and can use "},{"_key":"k00045","_type":"span","marks":["code"],"text":"site:"},{"_key":"k00046","_type":"span","marks":[],"text":" searches for their key pages instead. If your homepage and main pages are indexed and a handful of archive URLs sit in this bucket, there is nothing to fix."}],"markDefs":[],"style":"normal"},{"_key":"k00049","_type":"block","children":[{"_key":"k00048","_type":"span","marks":[],"text":"What should you check first when pages show up there?"}],"markDefs":[],"style":"h2"},{"_key":"k00051","_type":"block","children":[{"_key":"k00050","_type":"span","marks":[],"text":"Start by confirming the issue still exists, because the report lags behind reality."}],"markDefs":[],"style":"normal"},{"_key":"k00054","_type":"block","children":[{"_key":"k00052","_type":"span","marks":["strong"],"text":"Open the issue in Indexing > Pages"},{"_key":"k00053","_type":"span","marks":[],"text":" and export the example URLs. The list is capped at 1,000 rows and is not guaranteed to be complete, so for large sites also filter the report by a sitemap that contains only the pages you care about."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00057","_type":"block","children":[{"_key":"k00055","_type":"span","marks":["strong"],"text":"Run each important URL through the [URL Inspection tool](https://support.google.com/webmasters/answer/9012289)."},{"_key":"k00056","_type":"span","marks":[],"text":" Check whether it now says \"URL is on Google\". Pages often get indexed days after the report was refreshed."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00060","_type":"block","children":[{"_key":"k00058","_type":"span","marks":["strong"],"text":"Check the last crawl date."},{"_key":"k00059","_type":"span","marks":[],"text":" If the crawl was months ago and the page has changed a lot since, the decision was made on an old version."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00065","_type":"block","children":[{"_key":"k00061","_type":"span","marks":["strong"],"text":"Run a live test"},{"_key":"k00062","_type":"span","marks":[],"text":" (\"Test live URL\"). Confirm the page returns HTTP 200, is not blocked, has no "},{"_key":"k00063","_type":"span","marks":["code"],"text":"noindex"},{"_key":"k00064","_type":"span","marks":[],"text":", and that the rendered screenshot shows the main content. If the rendered HTML is an empty JavaScript shell, Google indexed nothing because it saw nothing."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00068","_type":"block","children":[{"_key":"k00066","_type":"span","marks":["strong"],"text":"Compare the user-declared and Google-selected canonical."},{"_key":"k00067","_type":"span","marks":[],"text":" If Google picked a different URL as canonical, you are looking at a duplication problem, not a quality one."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00079","_type":"block","children":[{"_key":"k00069","_type":"span","marks":[],"text":"Our free "},{"_key":"k00071","_type":"span","marks":["k00070"],"text":"URL health checker"},{"_key":"k00072","_type":"span","marks":[],"text":", "},{"_key":"k00074","_type":"span","marks":["k00073"],"text":"noindex checker"},{"_key":"k00075","_type":"span","marks":[],"text":" and "},{"_key":"k00077","_type":"span","marks":["k00076"],"text":"canonical URL checker"},{"_key":"k00078","_type":"span","marks":[],"text":" run the same basic checks on any public URL without a Search Console login."}],"markDefs":[{"_key":"k00070","_type":"link","href":"https://indexlinks.app/tools/url-health-checker"},{"_key":"k00073","_type":"link","href":"https://indexlinks.app/tools/noindex-checker"},{"_key":"k00076","_type":"link","href":"https://indexlinks.app/tools/canonical-url-checker"}],"style":"normal"},{"_key":"k00081","_type":"block","children":[{"_key":"k00080","_type":"span","marks":[],"text":"Why does Google crawl a page and then not index it?"}],"markDefs":[],"style":"h2"},{"_key":"k00083","_type":"block","children":[{"_key":"k00082","_type":"span","marks":[],"text":"Google does not publish a per-URL reason, so you diagnose by elimination. These are the causes that come up again and again, roughly in order of how often they explain the status:"}],"markDefs":[],"style":"normal"},{"_key":"k00089","_type":"block","children":[{"_key":"k00084","_type":"span","marks":["strong"],"text":"Thin or near-duplicate content."},{"_key":"k00085","_type":"span","marks":[],"text":" Programmatic pages that differ only by a city name, product variants with identical descriptions, or short posts that restate what dozens of other pages already say. Google's systems, in its own words on the "},{"_key":"k00087","_type":"span","marks":["k00086"],"text":"recrawl documentation"},{"_key":"k00088","_type":"span","marks":[],"text":", \"prioritize the fast inclusion of high quality, useful content\"."}],"level":1,"listItem":"bullet","markDefs":[{"_key":"k00086","_type":"link","href":"https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl"}],"style":"normal"},{"_key":"k00092","_type":"block","children":[{"_key":"k00090","_type":"span","marks":["strong"],"text":"Too many low value URLs on the site."},{"_key":"k00091","_type":"span","marks":[],"text":" If thousands of parameter or archive URLs are crawlable, the important ones compete with them. This is a site level signal, not just a page level one."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00095","_type":"block","children":[{"_key":"k00093","_type":"span","marks":["strong"],"text":"Weak internal linking."},{"_key":"k00094","_type":"span","marks":[],"text":" A page linked only from the XML sitemap and nowhere in the navigation looks unimportant. Orphan pages are a frequent pattern in this bucket."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00098","_type":"block","children":[{"_key":"k00096","_type":"span","marks":["strong"],"text":"Rendering problems."},{"_key":"k00097","_type":"span","marks":[],"text":" Content injected late by JavaScript, content behind a click, or a blocked API call can leave the rendered HTML nearly empty."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00101","_type":"block","children":[{"_key":"k00099","_type":"span","marks":["strong"],"text":"Soft 404 behaviour."},{"_key":"k00100","_type":"span","marks":[],"text":" Empty category pages, \"no results found\" pages, or out of stock pages with no content often land here or in Soft 404."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00104","_type":"block","children":[{"_key":"k00102","_type":"span","marks":["strong"],"text":"The page is simply new."},{"_key":"k00103","_type":"span","marks":[],"text":" Google says in the Page indexing report docs that it can take \"a week or so\" to crawl and index a new page. Some URLs sit in this status for a few days before moving to Indexed with no action at all."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00106","_type":"block","children":[{"_key":"k00105","_type":"span","marks":[],"text":"How do you fix it?"}],"markDefs":[],"style":"h2"},{"_key":"k00108","_type":"block","children":[{"_key":"k00107","_type":"span","marks":[],"text":"The fix is to change what Google sees when it crawls the page next. Work through these in order and stop when the important URLs are indexed."}],"markDefs":[],"style":"normal"},{"_key":"k00110","_type":"block","children":[{"_key":"k00109","_type":"span","marks":[],"text":"1. Make each page worth indexing on its own"}],"markDefs":[],"style":"h3"},{"_key":"k00112","_type":"block","children":[{"_key":"k00111","_type":"span","marks":[],"text":"Ask one question per page: what does this page give a searcher that the pages already ranking do not? Concrete fixes:"}],"markDefs":[],"style":"normal"},{"_key":"k00114","_type":"block","children":[{"_key":"k00113","_type":"span","marks":[],"text":"Merge thin pages that target the same intent into one stronger page and 301 redirect the rest."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00116","_type":"block","children":[{"_key":"k00115","_type":"span","marks":[],"text":"Add the specific detail that is missing: specs, prices, steps, screenshots, original data, a comparison table."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00118","_type":"block","children":[{"_key":"k00117","_type":"span","marks":[],"text":"For programmatic pages, make sure each page has unique data. If 80% of the text is a template, Google sees 80% duplication."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00122","_type":"block","children":[{"_key":"k00119","_type":"span","marks":[],"text":"Remove or "},{"_key":"k00120","_type":"span","marks":["code"],"text":"noindex"},{"_key":"k00121","_type":"span","marks":[],"text":" pages that exist only for navigation and have no standalone value."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00124","_type":"block","children":[{"_key":"k00123","_type":"span","marks":[],"text":"2. Cut the low value URLs Google keeps finding"}],"markDefs":[],"style":"h3"},{"_key":"k00129","_type":"block","children":[{"_key":"k00125","_type":"span","marks":[],"text":"Reduce the noise so the important URLs stand out. Return 404 or 410 for pages you deleted. Google's "},{"_key":"k00127","_type":"span","marks":["k00126"],"text":"crawl budget guide"},{"_key":"k00128","_type":"span","marks":[],"text":" notes that a 404 is \"a strong signal not to crawl that URL again\", while soft 404 pages \"will continue to be crawled, and waste your budget\". Remove faceted and parameter URLs from internal links, and keep them out of the sitemap."}],"markDefs":[{"_key":"k00126","_type":"link","href":"https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget"}],"style":"normal"},{"_key":"k00131","_type":"block","children":[{"_key":"k00130","_type":"span","marks":[],"text":"3. Strengthen internal links to the pages you want indexed"}],"markDefs":[],"style":"h3"},{"_key":"k00133","_type":"block","children":[{"_key":"k00132","_type":"span","marks":[],"text":"Link to each important URL from at least one relevant, already indexed page with descriptive anchor text, not only from the footer or the sitemap. A link from the homepage or a top category page is the strongest signal you control."}],"markDefs":[],"style":"normal"},{"_key":"k00135","_type":"block","children":[{"_key":"k00134","_type":"span","marks":[],"text":"4. Make sure the content renders"}],"markDefs":[],"style":"h3"},{"_key":"k00137","_type":"block","children":[{"_key":"k00136","_type":"span","marks":[],"text":"Check the live test screenshot and the \"View tested page > HTML\" output. The main text, headings and links should be in the rendered HTML. If they are not, move to server-side rendering or static generation for that template."}],"markDefs":[],"style":"normal"},{"_key":"k00139","_type":"block","children":[{"_key":"k00138","_type":"span","marks":[],"text":"5. Keep the sitemap honest"}],"markDefs":[],"style":"h3"},{"_key":"k00155","_type":"block","children":[{"_key":"k00140","_type":"span","marks":[],"text":"List only canonical, indexable URLs that return 200. Keep "},{"_key":"k00141","_type":"span","marks":["code"],"text":"<lastmod>"},{"_key":"k00142","_type":"span","marks":[],"text":" accurate: Google's "},{"_key":"k00144","_type":"span","marks":["k00143"],"text":"sitemap documentation"},{"_key":"k00145","_type":"span","marks":[],"text":" says it uses "},{"_key":"k00146","_type":"span","marks":["code"],"text":"lastmod"},{"_key":"k00147","_type":"span","marks":[],"text":" only when it is \"consistently and verifiably\" accurate, and it ignores "},{"_key":"k00148","_type":"span","marks":["code"],"text":"<priority>"},{"_key":"k00149","_type":"span","marks":[],"text":" and "},{"_key":"k00150","_type":"span","marks":["code"],"text":"<changefreq>"},{"_key":"k00151","_type":"span","marks":[],"text":" entirely. A single sitemap file is limited to 50,000 URLs or 50MB uncompressed. Our "},{"_key":"k00153","_type":"span","marks":["k00152"],"text":"sitemap validator"},{"_key":"k00154","_type":"span","marks":[],"text":" flags non-200 and non-canonical URLs inside a sitemap."}],"markDefs":[{"_key":"k00143","_type":"link","href":"https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap"},{"_key":"k00152","_type":"link","href":"https://indexlinks.app/tools/sitemap-validator"}],"style":"normal"},{"_key":"k00157","_type":"block","children":[{"_key":"k00156","_type":"span","marks":[],"text":"6. Then request indexing, once"}],"markDefs":[],"style":"h3"},{"_key":"k00159","_type":"block","children":[{"_key":"k00158","_type":"span","marks":[],"text":"After a real change, use \"Request indexing\" in the URL Inspection tool for your most important URLs. There is a daily quota per property, and Google is explicit that requesting a recrawl several times for the same URL will not get it crawled any faster. Requesting indexing on an unchanged page asks Google to make the same decision again."}],"markDefs":[],"style":"normal"},{"_key":"k00161","_type":"block","children":[{"_key":"k00160","_type":"span","marks":[],"text":"How do you run validation in Search Console?"}],"markDefs":[],"style":"h2"},{"_key":"k00165","_type":"block","children":[{"_key":"k00162","_type":"span","marks":[],"text":"After fixing all instances you care about, open the issue in the Page indexing report and click "},{"_key":"k00163","_type":"span","marks":["strong"],"text":"Validate fix"},{"_key":"k00164","_type":"span","marks":[],"text":". Google's documented process:"}],"markDefs":[],"style":"normal"},{"_key":"k00167","_type":"block","children":[{"_key":"k00166","_type":"span","marks":[],"text":"Search Console immediately checks a few pages. If the issue is still present on them, validation stops."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00169","_type":"block","children":[{"_key":"k00168","_type":"span","marks":[],"text":"Otherwise it queues only the affected URLs for recrawl, not the whole site."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00171","_type":"block","children":[{"_key":"k00170","_type":"span","marks":[],"text":"Validation \"typically takes up to about two weeks, but in some cases can take much longer\"."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00173","_type":"block","children":[{"_key":"k00172","_type":"span","marks":[],"text":"A single remaining instance fails the validation. Do not click Validate fix again until the current cycle has passed or failed."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00175","_type":"block","children":[{"_key":"k00174","_type":"span","marks":[],"text":"Google's own tip: create a sitemap with only your most important pages, filter the report by it, and validate against that subset. It finishes faster than validating every affected URL. An issue disappears from the table 90 days after its last instance is gone."}],"markDefs":[],"style":"normal"},{"_key":"k00177","_type":"block","children":[{"_key":"k00176","_type":"span","marks":[],"text":"For this status specifically, validation is less meaningful than for errors like a 5xx or a redirect loop, because the \"fix\" is a quality judgment. Many SEOs skip validation here and simply track the indexed count for the sitemap of important pages."}],"markDefs":[],"style":"normal"},{"_key":"k00179","_type":"block","children":[{"_key":"k00178","_type":"span","marks":[],"text":"How long does it take for a crawled page to be indexed?"}],"markDefs":[],"style":"h2"},{"_key":"k00184","_type":"block","children":[{"_key":"k00180","_type":"span","marks":[],"text":"There is no fixed time. Google's "},{"_key":"k00182","_type":"span","marks":["k00181"],"text":"ask Google to recrawl"},{"_key":"k00183","_type":"span","marks":[],"text":" page says crawling \"can take anywhere from a few days to a few weeks\", and that requesting a crawl \"does not guarantee that inclusion in search results will happen instantly or even at all\". For a page that was improved, expect at least one more crawl before the status changes. Pages that stay in this bucket for more than a month after real changes usually need consolidation or stronger internal links, not more requests."}],"markDefs":[{"_key":"k00181","_type":"link","href":"https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl"}],"style":"normal"},{"_key":"k00186","_type":"block","children":[{"_key":"k00185","_type":"span","marks":[],"text":"Which popular fixes do not work?"}],"markDefs":[],"style":"h2"},{"_key":"k00188","_type":"block","children":[{"_key":"k00187","_type":"span","marks":[],"text":"Some popular advice wastes time or carries risk:"}],"markDefs":[],"style":"normal"},{"_key":"k00191","_type":"block","children":[{"_key":"k00189","_type":"span","marks":["strong"],"text":"Hitting Request indexing every day."},{"_key":"k00190","_type":"span","marks":[],"text":" Google's documentation says it does not speed things up, and it burns the daily quota."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00204","_type":"block","children":[{"_key":"k00192","_type":"span","marks":["strong"],"text":"Using the Google Indexing API for normal pages."},{"_key":"k00193","_type":"span","marks":[],"text":" Google restricts the "},{"_key":"k00195","_type":"span","marks":["k00194"],"text":"Indexing API"},{"_key":"k00196","_type":"span","marks":[],"text":" to pages with "},{"_key":"k00197","_type":"span","marks":["code"],"text":"JobPosting"},{"_key":"k00198","_type":"span","marks":[],"text":" or "},{"_key":"k00199","_type":"span","marks":["code"],"text":"BroadcastEvent"},{"_key":"k00200","_type":"span","marks":[],"text":" structured data, and its spam policies apply to anything submitted through it. See our guide to the "},{"_key":"k00202","_type":"span","marks":["k00201"],"text":"Google Indexing API"},{"_key":"k00203","_type":"span","marks":[],"text":"."}],"level":1,"listItem":"bullet","markDefs":[{"_key":"k00194","_type":"link","href":"https://developers.google.com/search/apis/indexing-api/v3/using-api"},{"_key":"k00201","_type":"link","href":"https://indexlinks.app/blog/google-indexing-api"}],"style":"normal"},{"_key":"k00207","_type":"block","children":[{"_key":"k00205","_type":"span","marks":["strong"],"text":"Buying links to the page or using \"indexer\" services."},{"_key":"k00206","_type":"span","marks":[],"text":" These do not change the page Google already judged."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00210","_type":"block","children":[{"_key":"k00208","_type":"span","marks":["strong"],"text":"Changing the URL and republishing the same content."},{"_key":"k00209","_type":"span","marks":[],"text":" You get a new URL with the same content and often a new duplicate."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00212","_type":"block","children":[{"_key":"k00211","_type":"span","marks":[],"text":"How can you get other crawlers to see the page faster?"}],"markDefs":[],"style":"h2"},{"_key":"k00220","_type":"block","children":[{"_key":"k00213","_type":"span","marks":[],"text":"Google's decision is one search engine. Bing powers ChatGPT search, and Bing, Yandex, Naver, Seznam and Yep accept "},{"_key":"k00215","_type":"span","marks":["k00214"],"text":"IndexNow"},{"_key":"k00216","_type":"span","marks":[],"text":" pings, up to 10,000 URLs per request. Notifying them when you publish or update a page means the improved version is crawled outside Google quickly, regardless of how long Googlebot takes. Read "},{"_key":"k00218","_type":"span","marks":["k00217"],"text":"what IndexNow can and cannot do"},{"_key":"k00219","_type":"span","marks":[],"text":" before relying on it: it notifies, it does not force indexing."}],"markDefs":[{"_key":"k00214","_type":"link","href":"https://www.indexnow.org/documentation"},{"_key":"k00217","_type":"link","href":"https://indexlinks.app/blog/what-indexnow-can-and-cannot-do"}],"style":"normal"},{"_key":"k00225","_type":"block","children":[{"_key":"k00221","_type":"span","marks":[],"text":"IndexLinks sends each new or updated page you own to those channels, to your Search Console, and to the AI crawlers, and shows a receipt for every verified crawl. A crawl is what it guarantees, within 24 hours; indexing remains each engine's decision. To see which signals on your site might block indexing, run the free site check on the "},{"_key":"k00223","_type":"span","marks":["k00222"],"text":"IndexLinks homepage"},{"_key":"k00224","_type":"span","marks":[],"text":"."}],"markDefs":[{"_key":"k00222","_type":"link","href":"https://indexlinks.app/"}],"style":"normal"},{"_key":"k00227","_type":"block","children":[{"_key":"k00226","_type":"span","marks":[],"text":"FAQ"}],"markDefs":[],"style":"h2"},{"_key":"k00230","_type":"block","children":[{"_key":"k00228","_type":"span","marks":["strong"],"text":"Should I resubmit URLs that show \"Crawled - currently not indexed\"?"},{"_key":"k00229","_type":"span","marks":[],"text":" Not without changing them. Google's report documentation says there is \"no need to resubmit this URL for crawling\". Improve the page first, then request indexing once."}],"markDefs":[],"style":"normal"},{"_key":"k00233","_type":"block","children":[{"_key":"k00231","_type":"span","marks":["strong"],"text":"Can \"Crawled - currently not indexed\" hurt my rankings?"},{"_key":"k00232","_type":"span","marks":[],"text":" The status itself is not a penalty. Pages in it simply do not rank. A large share of the site in this bucket can, however, be a sign of thin or duplicate templates that are worth fixing site wide."}],"markDefs":[],"style":"normal"},{"_key":"k00236","_type":"block","children":[{"_key":"k00234","_type":"span","marks":["strong"],"text":"Why did an indexed page move to \"Crawled - currently not indexed\"?"},{"_key":"k00235","_type":"span","marks":[],"text":" Google re-evaluates pages on each crawl. Pages can drop out after a content change, after similar pages were published elsewhere on the site, or as Google reassesses quality across the web."}],"markDefs":[],"style":"normal"},{"_key":"k00239","_type":"block","children":[{"_key":"k00237","_type":"span","marks":["strong"],"text":"What is the difference between crawled and discovered currently not indexed?"},{"_key":"k00238","_type":"span","marks":[],"text":" Discovered means Google knows the URL but has not fetched it, so the last crawl date is empty. Crawled means it fetched the page and declined to index it."}],"markDefs":[],"style":"normal"},{"_key":"k00244","_type":"block","children":[{"_key":"k00240","_type":"span","marks":["strong"],"text":"Does a sitemap fix this status?"},{"_key":"k00241","_type":"span","marks":[],"text":" No. A sitemap helps discovery. This page has already been discovered and crawled; the sitemap only helps by keeping the list of important URLs clean and the "},{"_key":"k00242","_type":"span","marks":["code"],"text":"lastmod"},{"_key":"k00243","_type":"span","marks":[],"text":" dates accurate."}],"markDefs":[],"style":"normal"}],"category":"Search Console","excerpt":"Google crawled the page and chose not to index it. How to tell if that matters, why it happens, and the fixes that change Google's decision.","longDescription":null,"publishedAt":"2026-10-10T09:37:27Z","readTime":"10 min read","sections":null,"seoDescription":"Crawled currently not indexed means Google fetched the page but left it out. Learn the real causes, what to check first, and the fixes that work.","seoTitle":"Crawled - Currently Not Indexed: Causes and Fixes","shortDescription":null,"slug":"crawled-currently-not-indexed","status":"published","title":"Crawled - Currently Not Indexed: What It Means and How to Fix It","toolType":null},{"_id":"post-discovered-currently-not-indexed","_type":"post","_updatedAt":"2026-10-10T09:38:31Z","body":[{"_key":"k00003","_type":"block","children":[{"_key":"k00001","_type":"span","marks":["strong"],"text":"TL;DR:"},{"_key":"k00002","_type":"span","marks":[],"text":" \"Discovered - currently not indexed\" is a Google Search Console status meaning Google knows the URL exists but has not crawled it yet. Google's own explanation is that crawling was expected to overload your site, so the crawl was rescheduled. Fix it by making your server fast and error free for Googlebot, cutting the low value URLs that compete for crawls, and linking important pages from pages Google already crawls."}],"markDefs":[],"style":"normal"},{"_key":"k00008","_type":"block","children":[{"_key":"k00004","_type":"span","marks":[],"text":"Discovered currently not indexed is the status for a URL that Google has found, through a link or a sitemap, but has not fetched. Google's "},{"_key":"k00006","_type":"span","marks":["k00005"],"text":"Page indexing report documentation"},{"_key":"k00007","_type":"span","marks":[],"text":" defines it this way: \"The page was found by Google, but not crawled yet. Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl. This is why the last crawl date is empty on the report.\" Because Google has never seen the content, content quality cannot be the direct reason. The status is about crawl scheduling: how much Google is willing and able to crawl on your site, and how much it wants this particular URL."}],"markDefs":[{"_key":"k00005","_type":"link","href":"https://support.google.com/webmasters/answer/7440203"}],"style":"normal"},{"_key":"k00010","_type":"block","children":[{"_key":"k00009","_type":"span","marks":[],"text":"What does \"Discovered - currently not indexed\" mean in Google Search Console?"}],"markDefs":[],"style":"h2"},{"_key":"k00015","_type":"block","children":[{"_key":"k00011","_type":"span","marks":[],"text":"It means the URL is in Google's crawl queue but has not reached the front of it. In Search Console you see it under Indexing > Pages, in the \"Why pages aren't indexed\" table, with source Google systems. Inspect one of the URLs with the "},{"_key":"k00013","_type":"span","marks":["k00012"],"text":"URL Inspection tool"},{"_key":"k00014","_type":"span","marks":[],"text":" and the \"Last crawl\" field reads N/A."}],"markDefs":[{"_key":"k00012","_type":"link","href":"https://support.google.com/webmasters/answer/9012289"}],"style":"normal"},{"_key":"k00017","_type":"block","children":[{"_key":"k00016","_type":"span","marks":[],"text":"How it differs from the related statuses:"}],"markDefs":[],"style":"normal"},{"_key":"k00023","_type":"block","children":[{"_key":"k00018","_type":"span","marks":["strong"],"text":"Crawled - currently not indexed:"},{"_key":"k00019","_type":"span","marks":[],"text":" Google fetched the page and then chose not to index it. That is a judgment on the page. See our guide to "},{"_key":"k00021","_type":"span","marks":["k00020"],"text":"crawled currently not indexed"},{"_key":"k00022","_type":"span","marks":[],"text":"."}],"level":1,"listItem":"bullet","markDefs":[{"_key":"k00020","_type":"link","href":"https://indexlinks.app/blog/crawled-currently-not-indexed"}],"style":"normal"},{"_key":"k00026","_type":"block","children":[{"_key":"k00024","_type":"span","marks":["strong"],"text":"Discovered - currently not indexed:"},{"_key":"k00025","_type":"span","marks":[],"text":" Google has not fetched the page. That is a crawl capacity or crawl demand problem."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00029","_type":"block","children":[{"_key":"k00027","_type":"span","marks":["strong"],"text":"URL is unknown to Google:"},{"_key":"k00028","_type":"span","marks":[],"text":" Google has not found the URL at all. That is a discovery problem: no links, no sitemap entry."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00031","_type":"block","children":[{"_key":"k00030","_type":"span","marks":[],"text":"For a brand new site or page, a few days in this status is normal. Google says a new page or site can take \"a week or so\" to be crawled and indexed."}],"markDefs":[],"style":"normal"},{"_key":"k00033","_type":"block","children":[{"_key":"k00032","_type":"span","marks":[],"text":"Why does Google discover a page but not crawl it?"}],"markDefs":[],"style":"h2"},{"_key":"k00038","_type":"block","children":[{"_key":"k00034","_type":"span","marks":[],"text":"Google's "},{"_key":"k00036","_type":"span","marks":["k00035"],"text":"crawl budget guide"},{"_key":"k00037","_type":"span","marks":[],"text":" explains the two inputs that decide how much of your site is crawled:"}],"markDefs":[{"_key":"k00035","_type":"link","href":"https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget"}],"style":"normal"},{"_key":"k00041","_type":"block","children":[{"_key":"k00039","_type":"span","marks":["strong"],"text":"Crawl capacity limit."},{"_key":"k00040","_type":"span","marks":[],"text":" How many parallel connections Googlebot uses and how long it waits between fetches. Every site starts with the same conservative limit. It rises when the site responds quickly and consistently, and falls when \"the site slows down\", or \"responds with server errors (5xx HTTP status codes) or rate-limiting signals (such as HTTP 429)\"."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00044","_type":"block","children":[{"_key":"k00042","_type":"span","marks":["strong"],"text":"Crawl demand."},{"_key":"k00043","_type":"span","marks":[],"text":" How much Google wants to crawl your URLs, based on perceived inventory, popularity, and staleness. Duplicate and low value URLs dilute it."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00046","_type":"block","children":[{"_key":"k00045","_type":"span","marks":[],"text":"Google states that its crawl budget guide is aimed at sites with 1 million or more pages that change weekly, sites with 10,000 or more pages that change daily, and \"sites with a large portion of their total URLs classified by Search Console as Discovered - currently not indexed\". That last group is the one this status belongs to, at any size."}],"markDefs":[],"style":"normal"},{"_key":"k00048","_type":"block","children":[{"_key":"k00047","_type":"span","marks":[],"text":"In practice the usual causes are:"}],"markDefs":[],"style":"normal"},{"_key":"k00051","_type":"block","children":[{"_key":"k00049","_type":"span","marks":["strong"],"text":"A slow or overloaded server."},{"_key":"k00050","_type":"span","marks":[],"text":" Long response times or 5xx errors make Google back off. Shared hosting, cold serverless starts and heavy uncached pages are common culprits."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00054","_type":"block","children":[{"_key":"k00052","_type":"span","marks":["strong"],"text":"Too many URLs for the site's crawl demand."},{"_key":"k00053","_type":"span","marks":[],"text":" Faceted navigation, calendar pages, session IDs, sort parameters and infinite pagination can produce many times more URLs than real pages."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00057","_type":"block","children":[{"_key":"k00055","_type":"span","marks":["strong"],"text":"Publishing in large batches."},{"_key":"k00056","_type":"span","marks":[],"text":" Thousands of new programmatic or product URLs released at once on a site Google crawls lightly."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00060","_type":"block","children":[{"_key":"k00058","_type":"span","marks":["strong"],"text":"Weak internal linking."},{"_key":"k00059","_type":"span","marks":[],"text":" URLs found only in a sitemap, with no links from crawled pages, get low priority."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00063","_type":"block","children":[{"_key":"k00061","_type":"span","marks":["strong"],"text":"A site that Google does not value much yet."},{"_key":"k00062","_type":"span","marks":[],"text":" New domains and sites with mostly thin content get less crawl demand overall."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00065","_type":"block","children":[{"_key":"k00064","_type":"span","marks":[],"text":"How do you check your site's crawl health?"}],"markDefs":[],"style":"h2"},{"_key":"k00070","_type":"block","children":[{"_key":"k00066","_type":"span","marks":[],"text":"Open the Crawl Stats report: in Search Console go to Settings > Crawl stats. The "},{"_key":"k00068","_type":"span","marks":["k00067"],"text":"Crawl Stats report help"},{"_key":"k00069","_type":"span","marks":[],"text":" lists what to read:"}],"markDefs":[{"_key":"k00067","_type":"link","href":"https://support.google.com/webmasters/answer/9679690"}],"style":"normal"},{"_key":"k00073","_type":"block","children":[{"_key":"k00071","_type":"span","marks":["strong"],"text":"Host status."},{"_key":"k00072","_type":"span","marks":[],"text":" Should be green. If it is red, open the details for robots.txt availability, DNS resolution and host connectivity failures."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00076","_type":"block","children":[{"_key":"k00074","_type":"span","marks":["strong"],"text":"Average response time."},{"_key":"k00075","_type":"span","marks":[],"text":" Each resource counts as a separate response. A rising line here usually comes before a drop in total crawl requests."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00079","_type":"block","children":[{"_key":"k00077","_type":"span","marks":["strong"],"text":"By response."},{"_key":"k00078","_type":"span","marks":[],"text":" A visible share of 5xx, 429 or timeouts is the clearest reason Google reschedules crawls."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00082","_type":"block","children":[{"_key":"k00080","_type":"span","marks":["strong"],"text":"By purpose."},{"_key":"k00081","_type":"span","marks":[],"text":" \"Discovery\" is crawls of new URLs, \"Refresh\" is recrawls of known ones. If discovery is a tiny share while you publish lots of new pages, new URLs are waiting in the queue."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00087","_type":"block","children":[{"_key":"k00083","_type":"span","marks":[],"text":"Our free "},{"_key":"k00085","_type":"span","marks":["k00084"],"text":"URL health checker"},{"_key":"k00086","_type":"span","marks":[],"text":" gives a quick read on status code, redirects and response time for any URL."}],"markDefs":[{"_key":"k00084","_type":"link","href":"https://indexlinks.app/tools/url-health-checker"}],"style":"normal"},{"_key":"k00089","_type":"block","children":[{"_key":"k00088","_type":"span","marks":[],"text":"How do you get the queued pages crawled?"}],"markDefs":[],"style":"h2"},{"_key":"k00091","_type":"block","children":[{"_key":"k00090","_type":"span","marks":[],"text":"Work from the server outwards. Each step either raises the crawl capacity limit or increases demand for the URLs that matter."}],"markDefs":[],"style":"normal"},{"_key":"k00093","_type":"block","children":[{"_key":"k00092","_type":"span","marks":[],"text":"Make the server fast and reliable for Googlebot"}],"markDefs":[],"style":"h3"},{"_key":"k00097","_type":"block","children":[{"_key":"k00094","_type":"span","marks":[],"text":"Get typical server response time down and keep 5xx and 429 responses near zero. Cache HTML at the edge, warm serverless functions, and do not rate limit verified Googlebot. Support conditional requests: Google's guide recommends returning "},{"_key":"k00095","_type":"span","marks":["code"],"text":"304 Not Modified"},{"_key":"k00096","_type":"span","marks":[],"text":" for unchanged pages, which saves crawls for new ones."}],"markDefs":[],"style":"normal"},{"_key":"k00099","_type":"block","children":[{"_key":"k00098","_type":"span","marks":[],"text":"Cut the URLs that waste crawls"}],"markDefs":[],"style":"h3"},{"_key":"k00105","_type":"block","children":[{"_key":"k00100","_type":"span","marks":[],"text":"Return "},{"_key":"k00101","_type":"span","marks":["code"],"text":"404"},{"_key":"k00102","_type":"span","marks":[],"text":" or "},{"_key":"k00103","_type":"span","marks":["code"],"text":"410"},{"_key":"k00104","_type":"span","marks":[],"text":" for removed pages. Google calls a 404 \"a strong signal not to crawl that URL again\"."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00107","_type":"block","children":[{"_key":"k00106","_type":"span","marks":[],"text":"Fix soft 404s, which Google says \"will continue to be crawled, and waste your budget\"."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00111","_type":"block","children":[{"_key":"k00108","_type":"span","marks":[],"text":"Block faceted and parameter URLs you never want crawled in robots.txt. Google advises against using "},{"_key":"k00109","_type":"span","marks":["code"],"text":"noindex"},{"_key":"k00110","_type":"span","marks":[],"text":" for this, because Google \"will still request, but then drop the page\", which still costs a crawl."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00113","_type":"block","children":[{"_key":"k00112","_type":"span","marks":[],"text":"Remove long redirect chains, which Google says have \"a negative effect on crawling\"."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00118","_type":"block","children":[{"_key":"k00114","_type":"span","marks":[],"text":"Our "},{"_key":"k00116","_type":"span","marks":["k00115"],"text":"robots.txt checker"},{"_key":"k00117","_type":"span","marks":[],"text":" shows which paths a given rule blocks before you deploy it."}],"markDefs":[{"_key":"k00115","_type":"link","href":"https://indexlinks.app/tools/robots-txt-checker"}],"style":"normal"},{"_key":"k00120","_type":"block","children":[{"_key":"k00119","_type":"span","marks":[],"text":"Link to the waiting pages from pages Google already crawls"}],"markDefs":[],"style":"h3"},{"_key":"k00122","_type":"block","children":[{"_key":"k00121","_type":"span","marks":[],"text":"Add links from the homepage, category hubs, and recent posts to the URLs stuck in the queue. A link from a page Google recrawls daily is the cheapest way to raise demand for a new URL."}],"markDefs":[],"style":"normal"},{"_key":"k00124","_type":"block","children":[{"_key":"k00123","_type":"span","marks":[],"text":"Keep sitemaps clean and accurate"}],"markDefs":[],"style":"h3"},{"_key":"k00140","_type":"block","children":[{"_key":"k00125","_type":"span","marks":[],"text":"List only canonical URLs that return 200. Keep "},{"_key":"k00126","_type":"span","marks":["code"],"text":"<lastmod>"},{"_key":"k00127","_type":"span","marks":[],"text":" honest: Google's "},{"_key":"k00129","_type":"span","marks":["k00128"],"text":"sitemap documentation"},{"_key":"k00130","_type":"span","marks":[],"text":" says it uses "},{"_key":"k00131","_type":"span","marks":["code"],"text":"lastmod"},{"_key":"k00132","_type":"span","marks":[],"text":" only when it is \"consistently and verifiably\" accurate and ignores "},{"_key":"k00133","_type":"span","marks":["code"],"text":"<priority>"},{"_key":"k00134","_type":"span","marks":[],"text":" and "},{"_key":"k00135","_type":"span","marks":["code"],"text":"<changefreq>"},{"_key":"k00136","_type":"span","marks":[],"text":". Split large sitemaps by section so you can filter the Page indexing report per sitemap; each file can hold up to 50,000 URLs or 50MB uncompressed. Our "},{"_key":"k00138","_type":"span","marks":["k00137"],"text":"sitemap validator"},{"_key":"k00139","_type":"span","marks":[],"text":" checks every URL inside a sitemap."}],"markDefs":[{"_key":"k00128","_type":"link","href":"https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap"},{"_key":"k00137","_type":"link","href":"https://indexlinks.app/tools/sitemap-validator"}],"style":"normal"},{"_key":"k00142","_type":"block","children":[{"_key":"k00141","_type":"span","marks":[],"text":"Release large batches gradually"}],"markDefs":[],"style":"h3"},{"_key":"k00144","_type":"block","children":[{"_key":"k00143","_type":"span","marks":[],"text":"If you generate thousands of pages at once, publish them in waves over days or weeks and link each wave from crawled hubs. This matches the crawl capacity Google has assigned rather than asking for a jump."}],"markDefs":[],"style":"normal"},{"_key":"k00146","_type":"block","children":[{"_key":"k00145","_type":"span","marks":[],"text":"Request indexing for a few key pages"}],"markDefs":[],"style":"h3"},{"_key":"k00151","_type":"block","children":[{"_key":"k00147","_type":"span","marks":[],"text":"For your most important URLs, use \"Request indexing\" in the URL Inspection tool. Google's "},{"_key":"k00149","_type":"span","marks":["k00148"],"text":"recrawl documentation"},{"_key":"k00150","_type":"span","marks":[],"text":" notes a quota per property and says repeated requests for the same URL do not speed it up. It is a nudge for a handful of pages, not a fix for thousands."}],"markDefs":[{"_key":"k00148","_type":"link","href":"https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl"}],"style":"normal"},{"_key":"k00153","_type":"block","children":[{"_key":"k00152","_type":"span","marks":[],"text":"How long does a URL stay in this state?"}],"markDefs":[],"style":"h2"},{"_key":"k00155","_type":"block","children":[{"_key":"k00154","_type":"span","marks":[],"text":"There is no fixed duration. Google says crawling \"can take anywhere from a few days to a few weeks\". After server and URL hygiene fixes, watch the Crawl Stats report: total crawl requests should rise and the Discovery share should grow within a few weeks. If URLs stay discovered for months while crawl health is green, crawl demand is the limit, and internal links plus fewer low value URLs matter more than anything else."}],"markDefs":[],"style":"normal"},{"_key":"k00157","_type":"block","children":[{"_key":"k00156","_type":"span","marks":[],"text":"What about Bing, ChatGPT search and other crawlers?"}],"markDefs":[],"style":"h2"},{"_key":"k00165","_type":"block","children":[{"_key":"k00158","_type":"span","marks":[],"text":"Google's queue is only one queue. Bing, which ChatGPT search draws on, and Yandex, Naver, Seznam and Yep accept "},{"_key":"k00160","_type":"span","marks":["k00159"],"text":"IndexNow"},{"_key":"k00161","_type":"span","marks":[],"text":" notifications of up to 10,000 URLs per request, so new pages can be crawled there without waiting for Googlebot. Read "},{"_key":"k00163","_type":"span","marks":["k00162"],"text":"what IndexNow can and cannot do"},{"_key":"k00164","_type":"span","marks":[],"text":" first: Google does not participate."}],"markDefs":[{"_key":"k00159","_type":"link","href":"https://www.indexnow.org/documentation"},{"_key":"k00162","_type":"link","href":"https://indexlinks.app/blog/what-indexnow-can-and-cannot-do"}],"style":"normal"},{"_key":"k00170","_type":"block","children":[{"_key":"k00166","_type":"span","marks":[],"text":"IndexLinks sends each new or updated page on your own site to those engines, to your Search Console, and to the AI crawlers, and records a receipt for every verified crawl within 24 hours. It guarantees the crawl, not indexing. To spot the signals that keep pages out of the index, run the free site check on the "},{"_key":"k00168","_type":"span","marks":["k00167"],"text":"IndexLinks homepage"},{"_key":"k00169","_type":"span","marks":[],"text":"."}],"markDefs":[{"_key":"k00167","_type":"link","href":"https://indexlinks.app/"}],"style":"normal"},{"_key":"k00172","_type":"block","children":[{"_key":"k00171","_type":"span","marks":[],"text":"FAQ"}],"markDefs":[],"style":"h2"},{"_key":"k00175","_type":"block","children":[{"_key":"k00173","_type":"span","marks":["strong"],"text":"Is \"Discovered - currently not indexed\" a penalty?"},{"_key":"k00174","_type":"span","marks":[],"text":" No. It is a scheduling state. Google has not looked at the content yet."}],"markDefs":[],"style":"normal"},{"_key":"k00188","_type":"block","children":[{"_key":"k00176","_type":"span","marks":["strong"],"text":"Should I use the Google Indexing API to get these pages crawled?"},{"_key":"k00177","_type":"span","marks":[],"text":" No. Google limits the "},{"_key":"k00179","_type":"span","marks":["k00178"],"text":"Indexing API"},{"_key":"k00180","_type":"span","marks":[],"text":" to pages with "},{"_key":"k00181","_type":"span","marks":["code"],"text":"JobPosting"},{"_key":"k00182","_type":"span","marks":[],"text":" or "},{"_key":"k00183","_type":"span","marks":["code"],"text":"BroadcastEvent"},{"_key":"k00184","_type":"span","marks":[],"text":" structured data, and spam policies apply to what you submit. See our guide to the "},{"_key":"k00186","_type":"span","marks":["k00185"],"text":"Google Indexing API"},{"_key":"k00187","_type":"span","marks":[],"text":"."}],"markDefs":[{"_key":"k00178","_type":"link","href":"https://developers.google.com/search/apis/indexing-api/v3/using-api"},{"_key":"k00185","_type":"link","href":"https://indexlinks.app/blog/google-indexing-api"}],"style":"normal"},{"_key":"k00191","_type":"block","children":[{"_key":"k00189","_type":"span","marks":["strong"],"text":"Will backlinks fix it?"},{"_key":"k00190","_type":"span","marks":[],"text":" External links can raise crawl demand for a URL, but server health and internal links are faster and fully in your control."}],"markDefs":[],"style":"normal"},{"_key":"k00194","_type":"block","children":[{"_key":"k00192","_type":"span","marks":["strong"],"text":"Does deleting and republishing the page help?"},{"_key":"k00193","_type":"span","marks":[],"text":" Changing the URL creates a new URL that enters the same queue. It can also leave a 404 or redirect behind. Fix the queue, not the URL."}],"markDefs":[],"style":"normal"},{"_key":"k00197","_type":"block","children":[{"_key":"k00195","_type":"span","marks":["strong"],"text":"Can a small site have this status?"},{"_key":"k00196","_type":"span","marks":[],"text":" Yes. On small sites it most often points to a slow host, URLs found only through the sitemap, or a new domain with little crawl demand."}],"markDefs":[],"style":"normal"}],"category":"Search Console","excerpt":"Google found the URL but has not crawled it. Read crawl health in Search Console, cut wasted URLs, and get queued pages crawled.","longDescription":null,"publishedAt":"2026-10-10T09:38:31Z","readTime":"8 min read","sections":null,"seoDescription":"Discovered currently not indexed means Google knows the URL but has not crawled it. Check crawl stats, fix server limits and get pages crawled.","seoTitle":"Discovered - Currently Not Indexed: How to Fix It","shortDescription":null,"slug":"discovered-currently-not-indexed","status":"published","title":"Discovered - Currently Not Indexed: Why Google Waits and How to Fix It","toolType":null},{"_id":"post-google-indexing-api","_type":"post","_updatedAt":"2026-10-10T09:44:09Z","body":[{"_key":"k00007","_type":"block","children":[{"_key":"k00001","_type":"span","marks":["strong"],"text":"TL;DR:"},{"_key":"k00002","_type":"span","marks":[],"text":" The Google Indexing API is a free Google endpoint that lets a site owner notify Google to crawl or remove a URL, but Google only supports it for pages with "},{"_key":"k00003","_type":"span","marks":["code"],"text":"JobPosting"},{"_key":"k00004","_type":"span","marks":[],"text":" or "},{"_key":"k00005","_type":"span","marks":["code"],"text":"BroadcastEvent"},{"_key":"k00006","_type":"span","marks":[],"text":" (livestream) structured data. The default quota is 200 publish requests per day per project. For ordinary pages such as blog posts, products and landing pages, use a sitemap and Search Console's Request indexing instead."}],"markDefs":[],"style":"normal"},{"_key":"k00018","_type":"block","children":[{"_key":"k00008","_type":"span","marks":[],"text":"The Google Indexing API is the "},{"_key":"k00009","_type":"span","marks":["code"],"text":"https://indexing.googleapis.com/v3/urlNotifications:publish"},{"_key":"k00010","_type":"span","marks":[],"text":" endpoint, which accepts a URL and a notification type ("},{"_key":"k00011","_type":"span","marks":["code"],"text":"URL_UPDATED"},{"_key":"k00012","_type":"span","marks":[],"text":" or "},{"_key":"k00013","_type":"span","marks":["code"],"text":"URL_DELETED"},{"_key":"k00014","_type":"span","marks":[],"text":") and tells Google it may recrawl or drop that page. Google's "},{"_key":"k00016","_type":"span","marks":["k00015"],"text":"Indexing API documentation"},{"_key":"k00017","_type":"span","marks":[],"text":" states the limit in its first paragraph: \"The Indexing API can only be used to crawl pages with either JobPosting or BroadcastEvent embedded in a VideoObject.\" It also says that Google's spam policies apply to content submitted through it and that you must not \"circumvent our submission limits, such as by using multiple accounts.\" Many popular tutorials and WordPress plugins still present it as a way to get any page indexed fast. That is not what Google supports, and the quota page, last updated in July 2026, still says the same."}],"markDefs":[{"_key":"k00015","_type":"link","href":"https://developers.google.com/search/apis/indexing-api/v3/using-api"}],"style":"normal"},{"_key":"k00020","_type":"block","children":[{"_key":"k00019","_type":"span","marks":[],"text":"Who should use the Google Indexing API?"}],"markDefs":[],"style":"h2"},{"_key":"k00022","_type":"block","children":[{"_key":"k00021","_type":"span","marks":[],"text":"Use the Indexing API when your pages are short lived and carry the supported markup:"}],"markDefs":[],"style":"normal"},{"_key":"k00027","_type":"block","children":[{"_key":"k00023","_type":"span","marks":["strong"],"text":"Job boards and careers pages"},{"_key":"k00024","_type":"span","marks":[],"text":" with "},{"_key":"k00025","_type":"span","marks":["code"],"text":"JobPosting"},{"_key":"k00026","_type":"span","marks":[],"text":" structured data, where a listing must appear while it is open and disappear when it closes."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00034","_type":"block","children":[{"_key":"k00028","_type":"span","marks":["strong"],"text":"Livestream pages"},{"_key":"k00029","_type":"span","marks":[],"text":" with "},{"_key":"k00030","_type":"span","marks":["code"],"text":"BroadcastEvent"},{"_key":"k00031","_type":"span","marks":[],"text":" embedded in a "},{"_key":"k00032","_type":"span","marks":["code"],"text":"VideoObject"},{"_key":"k00033","_type":"span","marks":[],"text":", where the event matters only around its start time."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00042","_type":"block","children":[{"_key":"k00035","_type":"span","marks":[],"text":"Do not use it for blog posts, ecommerce products, programmatic landing pages, or documentation. For those, Google's "},{"_key":"k00037","_type":"span","marks":["k00036"],"text":"ask Google to recrawl"},{"_key":"k00038","_type":"span","marks":[],"text":" page names exactly two methods: the URL Inspection tool for a few URLs and a sitemap for many. Our guide on how to "},{"_key":"k00040","_type":"span","marks":["k00039"],"text":"request indexing in Google"},{"_key":"k00041","_type":"span","marks":[],"text":" covers both."}],"markDefs":[{"_key":"k00036","_type":"link","href":"https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl"},{"_key":"k00039","_type":"link","href":"https://indexlinks.app/blog/google-request-indexing"}],"style":"normal"},{"_key":"k00044","_type":"block","children":[{"_key":"k00043","_type":"span","marks":[],"text":"What are the Indexing API quotas and limits?"}],"markDefs":[],"style":"h2"},{"_key":"k00049","_type":"block","children":[{"_key":"k00045","_type":"span","marks":[],"text":"Google's "},{"_key":"k00047","_type":"span","marks":["k00046"],"text":"quota and pricing page"},{"_key":"k00048","_type":"span","marks":[],"text":" sets these defaults per Google Cloud project:"}],"markDefs":[{"_key":"k00046","_type":"link","href":"https://developers.google.com/search/apis/indexing-api/v3/quota-pricing"}],"style":"normal"},{"_key":"k00056","_type":"block","children":[{"_key":"k00050","_type":"span","marks":["strong"],"text":"Publish requests:"},{"_key":"k00051","_type":"span","marks":[],"text":" 200 per day, counting both "},{"_key":"k00052","_type":"span","marks":["code"],"text":"URL_UPDATED"},{"_key":"k00053","_type":"span","marks":[],"text":" and "},{"_key":"k00054","_type":"span","marks":["code"],"text":"URL_DELETED"},{"_key":"k00055","_type":"span","marks":[],"text":". The day resets at midnight Pacific Time."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00059","_type":"block","children":[{"_key":"k00057","_type":"span","marks":["strong"],"text":"Metadata (read only) requests:"},{"_key":"k00058","_type":"span","marks":[],"text":" 180 per minute."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00062","_type":"block","children":[{"_key":"k00060","_type":"span","marks":["strong"],"text":"All endpoints combined:"},{"_key":"k00061","_type":"span","marks":[],"text":" 380 requests per minute."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00067","_type":"block","children":[{"_key":"k00063","_type":"span","marks":["strong"],"text":"Batches:"},{"_key":"k00064","_type":"span","marks":[],"text":" up to 100 calls in one HTTP request to "},{"_key":"k00065","_type":"span","marks":["code"],"text":"https://indexing.googleapis.com/batch"},{"_key":"k00066","_type":"span","marks":[],"text":", each part no larger than 1MB. Quota is still counted per URL, so a batch of 10 uses 10."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00070","_type":"block","children":[{"_key":"k00068","_type":"span","marks":["strong"],"text":"Price:"},{"_key":"k00069","_type":"span","marks":[],"text":" free."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00072","_type":"block","children":[{"_key":"k00071","_type":"span","marks":[],"text":"The default quota is \"for initial API onboarding and testing\". To go beyond it, and to get approval for JobPosting or BroadcastEvent pages, you submit Google's quota request form, and Google notes the quota \"may increase or decrease based on the document quality.\""}],"markDefs":[],"style":"normal"},{"_key":"k00074","_type":"block","children":[{"_key":"k00073","_type":"span","marks":[],"text":"How do you set up the Indexing API?"}],"markDefs":[],"style":"h2"},{"_key":"k00079","_type":"block","children":[{"_key":"k00075","_type":"span","marks":[],"text":"Setup follows Google's "},{"_key":"k00077","_type":"span","marks":["k00076"],"text":"prerequisites guide"},{"_key":"k00078","_type":"span","marks":[],"text":" and takes about 15 minutes:"}],"markDefs":[{"_key":"k00076","_type":"link","href":"https://developers.google.com/search/apis/indexing-api/v3/prereqs"}],"style":"normal"},{"_key":"k00082","_type":"block","children":[{"_key":"k00080","_type":"span","marks":["strong"],"text":"Create or pick a project"},{"_key":"k00081","_type":"span","marks":[],"text":" in the Google Cloud console."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00085","_type":"block","children":[{"_key":"k00083","_type":"span","marks":["strong"],"text":"Enable the Web Search Indexing API"},{"_key":"k00084","_type":"span","marks":[],"text":" (listed as \"Indexing API\") for that project."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00088","_type":"block","children":[{"_key":"k00086","_type":"span","marks":["strong"],"text":"Create a service account"},{"_key":"k00087","_type":"span","marks":[],"text":" under IAM & Admin > Service accounts. Skip the optional roles step."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00091","_type":"block","children":[{"_key":"k00089","_type":"span","marks":["strong"],"text":"Create a key"},{"_key":"k00090","_type":"span","marks":[],"text":" for the service account and choose JSON. Store the file as a secret; anyone with it can use your quota."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00096","_type":"block","children":[{"_key":"k00092","_type":"span","marks":["strong"],"text":"Add the service account as an Owner in Search Console."},{"_key":"k00093","_type":"span","marks":[],"text":" Open the verified property, go to Settings > Users and permissions, and add the service account's email (it ends in "},{"_key":"k00094","_type":"span","marks":["code"],"text":"iam.gserviceaccount.com"},{"_key":"k00095","_type":"span","marks":[],"text":") with Owner permission. A Full user is not enough for this API."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00101","_type":"block","children":[{"_key":"k00097","_type":"span","marks":["strong"],"text":"Authenticate with the scope"},{"_key":"k00098","_type":"span","marks":[],"text":" "},{"_key":"k00099","_type":"span","marks":["code"],"text":"https://www.googleapis.com/auth/indexing"},{"_key":"k00100","_type":"span","marks":[],"text":"."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00103","_type":"block","children":[{"_key":"k00102","_type":"span","marks":[],"text":"How do you send an Indexing API request?"}],"markDefs":[],"style":"h2"},{"_key":"k00113","_type":"block","children":[{"_key":"k00104","_type":"span","marks":[],"text":"A request is an authenticated "},{"_key":"k00105","_type":"span","marks":["code"],"text":"POST"},{"_key":"k00106","_type":"span","marks":[],"text":" with "},{"_key":"k00107","_type":"span","marks":["code"],"text":"Content-Type: application/json"},{"_key":"k00108","_type":"span","marks":[],"text":" and a body naming one URL. In Python, with the official "},{"_key":"k00109","_type":"span","marks":["code"],"text":"google-auth"},{"_key":"k00110","_type":"span","marks":[],"text":" library installed ("},{"_key":"k00111","_type":"span","marks":["code"],"text":"pip install google-auth requests"},{"_key":"k00112","_type":"span","marks":[],"text":"), the whole call is six lines:"}],"markDefs":[],"style":"normal"},{"_key":"k00115","_type":"block","children":[{"_key":"k00114","_type":"span","marks":["code"],"text":"from google.oauth2 import service_account"}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00117","_type":"block","children":[{"_key":"k00116","_type":"span","marks":["code"],"text":"from google.auth.transport.requests import AuthorizedSession"}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00119","_type":"block","children":[{"_key":"k00118","_type":"span","marks":["code"],"text":"creds = service_account.Credentials.from_service_account_file(\"key.json\", scopes=[\"https://www.googleapis.com/auth/indexing\"])"}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00121","_type":"block","children":[{"_key":"k00120","_type":"span","marks":["code"],"text":"session = AuthorizedSession(creds)"}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00123","_type":"block","children":[{"_key":"k00122","_type":"span","marks":["code"],"text":"r = session.post(\"https://indexing.googleapis.com/v3/urlNotifications:publish\", json={\"url\": \"https://example.com/jobs/42\", \"type\": \"URL_UPDATED\"})"}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00125","_type":"block","children":[{"_key":"k00124","_type":"span","marks":["code"],"text":"print(r.status_code, r.json())"}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00129","_type":"block","children":[{"_key":"k00126","_type":"span","marks":[],"text":"An HTTP 200 response returns a "},{"_key":"k00127","_type":"span","marks":["code"],"text":"UrlNotificationMetadata"},{"_key":"k00128","_type":"span","marks":[],"text":" object. Google's wording is careful: a 200 \"means that Google may try to recrawl this URL soon.\" It is not a promise of a crawl or of indexing."}],"markDefs":[],"style":"normal"},{"_key":"k00135","_type":"block","children":[{"_key":"k00130","_type":"span","marks":[],"text":"To remove a page, send the same request with "},{"_key":"k00131","_type":"span","marks":["code"],"text":"\"type\": \"URL_DELETED\""},{"_key":"k00132","_type":"span","marks":[],"text":". Google requires the URL to return 404 or 410, or carry "},{"_key":"k00133","_type":"span","marks":["code"],"text":"<meta name=\"robots\" content=\"noindex\">"},{"_key":"k00134","_type":"span","marks":[],"text":", before you send it."}],"markDefs":[],"style":"normal"},{"_key":"k00145","_type":"block","children":[{"_key":"k00136","_type":"span","marks":[],"text":"To check what you sent, call "},{"_key":"k00137","_type":"span","marks":["code"],"text":"GET https://indexing.googleapis.com/v3/urlNotifications/metadata?url="},{"_key":"k00138","_type":"span","marks":[],"text":" followed by the URL encoded address. The response shows "},{"_key":"k00139","_type":"span","marks":["code"],"text":"latest_update"},{"_key":"k00140","_type":"span","marks":[],"text":" and "},{"_key":"k00141","_type":"span","marks":["code"],"text":"latest_remove"},{"_key":"k00142","_type":"span","marks":[],"text":" with a "},{"_key":"k00143","_type":"span","marks":["code"],"text":"notify_time"},{"_key":"k00144","_type":"span","marks":[],"text":". Google states this \"doesn't tell you when Google indexes or removes a URL; it only returns whether you successfully submitted a request.\""}],"markDefs":[],"style":"normal"},{"_key":"k00147","_type":"block","children":[{"_key":"k00146","_type":"span","marks":[],"text":"What do common Indexing API errors mean?"}],"markDefs":[],"style":"h2"},{"_key":"k00156","_type":"block","children":[{"_key":"k00148","_type":"span","marks":["strong"],"text":"403 \"Permission denied. Failed to verify the URL ownership.\""},{"_key":"k00149","_type":"span","marks":[],"text":" The service account is not an Owner of the Search Console property that contains the URL, or the URL belongs to a different property (for example "},{"_key":"k00150","_type":"span","marks":["code"],"text":"http://"},{"_key":"k00151","_type":"span","marks":[],"text":" vs "},{"_key":"k00152","_type":"span","marks":["code"],"text":"https://"},{"_key":"k00153","_type":"span","marks":[],"text":", or "},{"_key":"k00154","_type":"span","marks":["code"],"text":"www"},{"_key":"k00155","_type":"span","marks":[],"text":" vs non-www on a URL prefix property)."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00159","_type":"block","children":[{"_key":"k00157","_type":"span","marks":["strong"],"text":"403 \"Indexing API has not been used in project ... or it is disabled.\""},{"_key":"k00158","_type":"span","marks":[],"text":" Enable the API in the same Cloud project as the key."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00162","_type":"block","children":[{"_key":"k00160","_type":"span","marks":["strong"],"text":"429 \"Quota exceeded\"."},{"_key":"k00161","_type":"span","marks":[],"text":" You passed 200 publish requests for the day or a per minute limit. Wait for the midnight Pacific reset or request more quota."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00167","_type":"block","children":[{"_key":"k00163","_type":"span","marks":["strong"],"text":"400 Invalid argument."},{"_key":"k00164","_type":"span","marks":[],"text":" Usually a missing "},{"_key":"k00165","_type":"span","marks":["code"],"text":"type"},{"_key":"k00166","_type":"span","marks":[],"text":", a relative URL, or the wrong content type."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00169","_type":"block","children":[{"_key":"k00168","_type":"span","marks":[],"text":"Is it safe to use the Indexing API for normal pages?"}],"markDefs":[],"style":"h2"},{"_key":"k00177","_type":"block","children":[{"_key":"k00170","_type":"span","marks":[],"text":"Google's documentation makes three points that matter here. The API \"can only be used\" for the two supported page types. Spam policies apply to anything submitted. And quota depends on document quality. Sending blog or product URLs through it is outside what Google supports, and tools that rotate many service accounts to raise the 200 per day limit are doing the exact thing Google tells you not to do. Even when such pages get crawled, a crawl is not indexing: the page still has to pass Google's quality bar. If pages are stuck, the fixes are in our guides to "},{"_key":"k00172","_type":"span","marks":["k00171"],"text":"crawled currently not indexed"},{"_key":"k00173","_type":"span","marks":[],"text":" and "},{"_key":"k00175","_type":"span","marks":["k00174"],"text":"discovered currently not indexed"},{"_key":"k00176","_type":"span","marks":[],"text":"."}],"markDefs":[{"_key":"k00171","_type":"link","href":"https://indexlinks.app/blog/crawled-currently-not-indexed"},{"_key":"k00174","_type":"link","href":"https://indexlinks.app/blog/discovered-currently-not-indexed"}],"style":"normal"},{"_key":"k00179","_type":"block","children":[{"_key":"k00178","_type":"span","marks":[],"text":"What should you use instead for regular pages?"}],"markDefs":[],"style":"h2"},{"_key":"k00181","_type":"block","children":[{"_key":"k00180","_type":"span","marks":[],"text":"Use these supported channels:"}],"markDefs":[],"style":"normal"},{"_key":"k00198","_type":"block","children":[{"_key":"k00182","_type":"span","marks":["strong"],"text":"XML sitemap"},{"_key":"k00183","_type":"span","marks":[],"text":" with accurate "},{"_key":"k00184","_type":"span","marks":["code"],"text":"<lastmod>"},{"_key":"k00185","_type":"span","marks":[],"text":", submitted in Search Console. Google's "},{"_key":"k00187","_type":"span","marks":["k00186"],"text":"sitemap documentation"},{"_key":"k00188","_type":"span","marks":[],"text":" says it uses "},{"_key":"k00189","_type":"span","marks":["code"],"text":"lastmod"},{"_key":"k00190","_type":"span","marks":[],"text":" when it is consistently accurate and ignores "},{"_key":"k00191","_type":"span","marks":["code"],"text":"<priority>"},{"_key":"k00192","_type":"span","marks":[],"text":" and "},{"_key":"k00193","_type":"span","marks":["code"],"text":"<changefreq>"},{"_key":"k00194","_type":"span","marks":[],"text":". Check yours with our "},{"_key":"k00196","_type":"span","marks":["k00195"],"text":"sitemap validator"},{"_key":"k00197","_type":"span","marks":[],"text":"."}],"level":1,"listItem":"bullet","markDefs":[{"_key":"k00186","_type":"link","href":"https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap"},{"_key":"k00195","_type":"link","href":"https://indexlinks.app/tools/sitemap-validator"}],"style":"normal"},{"_key":"k00201","_type":"block","children":[{"_key":"k00199","_type":"span","marks":["strong"],"text":"URL Inspection > Request indexing"},{"_key":"k00200","_type":"span","marks":[],"text":" for a handful of important URLs, within the daily quota per property."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00204","_type":"block","children":[{"_key":"k00202","_type":"span","marks":["strong"],"text":"Internal links"},{"_key":"k00203","_type":"span","marks":[],"text":" from pages Google already crawls often."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00213","_type":"block","children":[{"_key":"k00205","_type":"span","marks":["strong"],"text":"IndexNow"},{"_key":"k00206","_type":"span","marks":[],"text":" for Bing, Yandex, Naver, Seznam and Yep: up to 10,000 URLs per request per the "},{"_key":"k00208","_type":"span","marks":["k00207"],"text":"IndexNow documentation"},{"_key":"k00209","_type":"span","marks":[],"text":". Google does not take part. See "},{"_key":"k00211","_type":"span","marks":["k00210"],"text":"what IndexNow can and cannot do"},{"_key":"k00212","_type":"span","marks":[],"text":"."}],"level":1,"listItem":"bullet","markDefs":[{"_key":"k00207","_type":"link","href":"https://www.indexnow.org/documentation"},{"_key":"k00210","_type":"link","href":"https://indexlinks.app/blog/what-indexnow-can-and-cannot-do"}],"style":"normal"},{"_key":"k00218","_type":"block","children":[{"_key":"k00214","_type":"span","marks":[],"text":"IndexLinks uses only these official channels for sites you own: IndexNow, your Search Console, Baidu push, and the AI crawlers through its Discover hub, with a receipt for every verified crawl within 24 hours. It guarantees a crawl, not indexing, and it does not send normal pages through the Indexing API. Run the free site check on the "},{"_key":"k00216","_type":"span","marks":["k00215"],"text":"IndexLinks homepage"},{"_key":"k00217","_type":"span","marks":[],"text":" to see what might be blocking yours."}],"markDefs":[{"_key":"k00215","_type":"link","href":"https://indexlinks.app/"}],"style":"normal"},{"_key":"k00220","_type":"block","children":[{"_key":"k00219","_type":"span","marks":[],"text":"FAQ"}],"markDefs":[],"style":"h2"},{"_key":"k00223","_type":"block","children":[{"_key":"k00221","_type":"span","marks":["strong"],"text":"Is the Google Indexing API free?"},{"_key":"k00222","_type":"span","marks":[],"text":" Yes. Google states that \"all use of the Indexing API is available without payment.\""}],"markDefs":[],"style":"normal"},{"_key":"k00226","_type":"block","children":[{"_key":"k00224","_type":"span","marks":["strong"],"text":"How many URLs can I submit per day?"},{"_key":"k00225","_type":"span","marks":[],"text":" 200 publish requests per project per day by default. Higher limits require Google's approval."}],"markDefs":[],"style":"normal"},{"_key":"k00229","_type":"block","children":[{"_key":"k00227","_type":"span","marks":["strong"],"text":"Does the Indexing API work for WordPress posts?"},{"_key":"k00228","_type":"span","marks":[],"text":" Plugins can send the request, but Google only supports the API for JobPosting and livestream pages. Use a sitemap for posts."}],"markDefs":[],"style":"normal"},{"_key":"k00232","_type":"block","children":[{"_key":"k00230","_type":"span","marks":["strong"],"text":"Does a 200 response mean my page is indexed?"},{"_key":"k00231","_type":"span","marks":[],"text":" No. It means Google received the notification and \"may try to recrawl this URL soon.\" Check the result with the URL Inspection tool."}],"markDefs":[],"style":"normal"},{"_key":"k00239","_type":"block","children":[{"_key":"k00233","_type":"span","marks":["strong"],"text":"Can I use the Indexing API to remove pages?"},{"_key":"k00234","_type":"span","marks":[],"text":" Yes, with "},{"_key":"k00235","_type":"span","marks":["code"],"text":"URL_DELETED"},{"_key":"k00236","_type":"span","marks":[],"text":", after the page returns 404 or 410 or carries a "},{"_key":"k00237","_type":"span","marks":["code"],"text":"noindex"},{"_key":"k00238","_type":"span","marks":[],"text":" tag. It applies to the supported page types."}],"markDefs":[],"style":"normal"}],"category":"Indexing","excerpt":"What the Google Indexing API supports (JobPosting and livestream pages), the 200 per day quota, setup, a working Python request, and errors.","longDescription":null,"publishedAt":"2026-10-10T09:44:08Z","readTime":"7 min read","sections":null,"seoDescription":"The Google Indexing API only supports JobPosting and livestream pages. Quotas, service account setup, a working request, and errors explained.","seoTitle":"Google Indexing API: Quotas, Setup and Limits","shortDescription":null,"slug":"google-indexing-api","status":"published","title":"Google Indexing API: Who Can Use It, Quotas and Setup","toolType":null},{"_id":"post-google-request-indexing","_type":"post","_updatedAt":"2026-10-10T09:39:04Z","body":[{"_key":"k00003","_type":"block","children":[{"_key":"k00001","_type":"span","marks":["strong"],"text":"TL;DR:"},{"_key":"k00002","_type":"span","marks":[],"text":" To request indexing in Google, open Google Search Console, paste the full URL into the URL Inspection bar at the top, wait for the result, and click \"Request indexing\". You must be an owner or full user of the property, there is a daily quota per property, and a request only queues a crawl: it does not guarantee the page will be indexed. For many URLs, submit a sitemap instead."}],"markDefs":[],"style":"normal"},{"_key":"k00008","_type":"block","children":[{"_key":"k00004","_type":"span","marks":[],"text":"Google request indexing is the \"Request indexing\" button in Search Console's URL Inspection tool, which asks Google to crawl one specific URL on a site you manage. Google's "},{"_key":"k00006","_type":"span","marks":["k00005"],"text":"ask Google to recrawl documentation"},{"_key":"k00007","_type":"span","marks":[],"text":" lists exactly two methods: the URL Inspection tool for \"just a few URLs\" and a sitemap for \"many URLs at once\". It also sets expectations plainly: \"Crawling can take anywhere from a few days to a few weeks\" and \"requesting a crawl does not guarantee that inclusion in search results will happen instantly or even at all.\""}],"markDefs":[{"_key":"k00005","_type":"link","href":"https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl"}],"style":"normal"},{"_key":"k00010","_type":"block","children":[{"_key":"k00009","_type":"span","marks":[],"text":"How do you request indexing in Google Search Console?"}],"markDefs":[],"style":"h2"},{"_key":"k00012","_type":"block","children":[{"_key":"k00011","_type":"span","marks":[],"text":"Requesting indexing takes five steps:"}],"markDefs":[],"style":"normal"},{"_key":"k00017","_type":"block","children":[{"_key":"k00013","_type":"span","marks":[],"text":"Sign in to "},{"_key":"k00015","_type":"span","marks":["k00014"],"text":"Google Search Console"},{"_key":"k00016","_type":"span","marks":[],"text":" and select the property that contains the URL. You must be a verified owner or a full user; restricted users cannot request indexing."}],"level":1,"listItem":"number","markDefs":[{"_key":"k00014","_type":"link","href":"https://search.google.com/search-console/about"}],"style":"normal"},{"_key":"k00021","_type":"block","children":[{"_key":"k00018","_type":"span","marks":[],"text":"Paste the complete URL, including "},{"_key":"k00019","_type":"span","marks":["code"],"text":"https://"},{"_key":"k00020","_type":"span","marks":[],"text":", into the \"Inspect any URL\" bar at the top of the page and press Enter."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00026","_type":"block","children":[{"_key":"k00022","_type":"span","marks":[],"text":"Wait for the "},{"_key":"k00024","_type":"span","marks":["k00023"],"text":"URL Inspection tool"},{"_key":"k00025","_type":"span","marks":[],"text":" to fetch Google's indexed data. It shows \"URL is on Google\" or \"URL is not on Google\" with a reason."}],"level":1,"listItem":"number","markDefs":[{"_key":"k00023","_type":"link","href":"https://support.google.com/webmasters/answer/9012289"}],"style":"normal"},{"_key":"k00030","_type":"block","children":[{"_key":"k00027","_type":"span","marks":[],"text":"Optional but worth it: click \"Test live URL\" to check the current version returns HTTP 200, is not blocked by robots.txt, and has no "},{"_key":"k00028","_type":"span","marks":["code"],"text":"noindex"},{"_key":"k00029","_type":"span","marks":[],"text":"."}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00032","_type":"block","children":[{"_key":"k00031","_type":"span","marks":[],"text":"Click \"Request indexing\". Search Console runs a quick live check and then confirms \"URL was added to a priority crawl queue.\""}],"level":1,"listItem":"number","markDefs":[],"style":"normal"},{"_key":"k00042","_type":"block","children":[{"_key":"k00033","_type":"span","marks":[],"text":"If the live check finds a blocking problem, such as a "},{"_key":"k00034","_type":"span","marks":["code"],"text":"noindex"},{"_key":"k00035","_type":"span","marks":[],"text":" tag or a 404, the request fails and Search Console tells you why. Fix that first; our free "},{"_key":"k00037","_type":"span","marks":["k00036"],"text":"noindex checker"},{"_key":"k00038","_type":"span","marks":[],"text":" and "},{"_key":"k00040","_type":"span","marks":["k00039"],"text":"URL health checker"},{"_key":"k00041","_type":"span","marks":[],"text":" catch the common blockers without logging in."}],"markDefs":[{"_key":"k00036","_type":"link","href":"https://indexlinks.app/tools/noindex-checker"},{"_key":"k00039","_type":"link","href":"https://indexlinks.app/tools/url-health-checker"}],"style":"normal"},{"_key":"k00044","_type":"block","children":[{"_key":"k00043","_type":"span","marks":[],"text":"What is the Request indexing quota?"}],"markDefs":[],"style":"h2"},{"_key":"k00046","_type":"block","children":[{"_key":"k00045","_type":"span","marks":[],"text":"Google documents that there is \"a daily limit to how many index requests you can submit\" per property, and a separate per-property daily limit for live tests. Google does not publish the exact number in the help center, and it can change. When you hit it, Search Console shows \"Quota exceeded\" and you try again the next day."}],"markDefs":[],"style":"normal"},{"_key":"k00048","_type":"block","children":[{"_key":"k00047","_type":"span","marks":[],"text":"Two related limits are documented precisely:"}],"markDefs":[],"style":"normal"},{"_key":"k00055","_type":"block","children":[{"_key":"k00049","_type":"span","marks":[],"text":"The "},{"_key":"k00050","_type":"span","marks":["strong"],"text":"URL Inspection API"},{"_key":"k00051","_type":"span","marks":[],"text":", which reads index status but cannot request indexing, allows 2,000 queries per day and 600 per minute per site, according to Google's "},{"_key":"k00053","_type":"span","marks":["k00052"],"text":"Search Console API usage limits"},{"_key":"k00054","_type":"span","marks":[],"text":"."}],"level":1,"listItem":"bullet","markDefs":[{"_key":"k00052","_type":"link","href":"https://developers.google.com/webmaster-tools/limits"}],"style":"normal"},{"_key":"k00066","_type":"block","children":[{"_key":"k00056","_type":"span","marks":[],"text":"The "},{"_key":"k00057","_type":"span","marks":["strong"],"text":"Indexing API"},{"_key":"k00058","_type":"span","marks":[],"text":" defaults to 200 publish requests per day per project, but only for pages with "},{"_key":"k00059","_type":"span","marks":["code"],"text":"JobPosting"},{"_key":"k00060","_type":"span","marks":[],"text":" or "},{"_key":"k00061","_type":"span","marks":["code"],"text":"BroadcastEvent"},{"_key":"k00062","_type":"span","marks":[],"text":" markup. See our guide to the "},{"_key":"k00064","_type":"span","marks":["k00063"],"text":"Google Indexing API"},{"_key":"k00065","_type":"span","marks":[],"text":"."}],"level":1,"listItem":"bullet","markDefs":[{"_key":"k00063","_type":"link","href":"https://indexlinks.app/blog/google-indexing-api"}],"style":"normal"},{"_key":"k00068","_type":"block","children":[{"_key":"k00067","_type":"span","marks":[],"text":"Because the quota is small, spend it on pages where speed matters: a new homepage, a launch page, a corrected price, a page you just fixed after an indexing error."}],"markDefs":[],"style":"normal"},{"_key":"k00070","_type":"block","children":[{"_key":"k00069","_type":"span","marks":[],"text":"Does requesting indexing again make it faster?"}],"markDefs":[],"style":"h2"},{"_key":"k00072","_type":"block","children":[{"_key":"k00071","_type":"span","marks":[],"text":"No. Google states that \"requesting a recrawl multiple times for the same URL won't get it crawled any faster.\" One request puts the URL in the priority queue; further requests for the same URL only spend quota. Wait, then inspect the URL again to check the \"Last crawl\" date."}],"markDefs":[],"style":"normal"},{"_key":"k00074","_type":"block","children":[{"_key":"k00073","_type":"span","marks":[],"text":"When should you submit a sitemap instead?"}],"markDefs":[],"style":"h2"},{"_key":"k00078","_type":"block","children":[{"_key":"k00075","_type":"span","marks":[],"text":"Use a sitemap when you have more URLs than you can sensibly submit by hand: a new site, a site migration, a batch of new products, or a programmatic section. In Search Console go to Indexing > Sitemaps, enter the sitemap URL, and click Submit. You can also reference it in robots.txt with a "},{"_key":"k00076","_type":"span","marks":["code"],"text":"Sitemap:"},{"_key":"k00077","_type":"span","marks":[],"text":" line."}],"markDefs":[],"style":"normal"},{"_key":"k00083","_type":"block","children":[{"_key":"k00079","_type":"span","marks":[],"text":"Rules from Google's "},{"_key":"k00081","_type":"span","marks":["k00080"],"text":"sitemap documentation"},{"_key":"k00082","_type":"span","marks":[],"text":":"}],"markDefs":[{"_key":"k00080","_type":"link","href":"https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap"}],"style":"normal"},{"_key":"k00085","_type":"block","children":[{"_key":"k00084","_type":"span","marks":[],"text":"One sitemap file holds up to 50,000 URLs or 50MB uncompressed. Use a sitemap index for more."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00093","_type":"block","children":[{"_key":"k00086","_type":"span","marks":[],"text":"Google uses "},{"_key":"k00087","_type":"span","marks":["code"],"text":"<lastmod>"},{"_key":"k00088","_type":"span","marks":[],"text":" when it is consistently accurate and ignores "},{"_key":"k00089","_type":"span","marks":["code"],"text":"<priority>"},{"_key":"k00090","_type":"span","marks":[],"text":" and "},{"_key":"k00091","_type":"span","marks":["code"],"text":"<changefreq>"},{"_key":"k00092","_type":"span","marks":[],"text":"."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00095","_type":"block","children":[{"_key":"k00094","_type":"span","marks":[],"text":"List only canonical URLs that return 200."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00100","_type":"block","children":[{"_key":"k00096","_type":"span","marks":[],"text":"Hosted platforms often handle this for you. Google notes that platforms such as Blogger, WordPress, Wix and Squarespace usually submit new content to search engines automatically. Check yours with our "},{"_key":"k00098","_type":"span","marks":["k00097"],"text":"sitemap validator"},{"_key":"k00099","_type":"span","marks":[],"text":"."}],"markDefs":[{"_key":"k00097","_type":"link","href":"https://indexlinks.app/tools/sitemap-validator"}],"style":"normal"},{"_key":"k00102","_type":"block","children":[{"_key":"k00101","_type":"span","marks":[],"text":"Why did Request indexing not work?"}],"markDefs":[],"style":"h2"},{"_key":"k00104","_type":"block","children":[{"_key":"k00103","_type":"span","marks":[],"text":"If a URL is still not indexed days or weeks after a request, the inspection result usually says why:"}],"markDefs":[],"style":"normal"},{"_key":"k00110","_type":"block","children":[{"_key":"k00105","_type":"span","marks":["strong"],"text":"Discovered - currently not indexed:"},{"_key":"k00106","_type":"span","marks":[],"text":" Google has not crawled it yet, usually because of crawl capacity or demand. See "},{"_key":"k00108","_type":"span","marks":["k00107"],"text":"discovered currently not indexed"},{"_key":"k00109","_type":"span","marks":[],"text":"."}],"level":1,"listItem":"bullet","markDefs":[{"_key":"k00107","_type":"link","href":"https://indexlinks.app/blog/discovered-currently-not-indexed"}],"style":"normal"},{"_key":"k00116","_type":"block","children":[{"_key":"k00111","_type":"span","marks":["strong"],"text":"Crawled - currently not indexed:"},{"_key":"k00112","_type":"span","marks":[],"text":" Google crawled it and chose not to index it. Requesting again changes nothing until the page changes. See "},{"_key":"k00114","_type":"span","marks":["k00113"],"text":"crawled currently not indexed"},{"_key":"k00115","_type":"span","marks":[],"text":"."}],"level":1,"listItem":"bullet","markDefs":[{"_key":"k00113","_type":"link","href":"https://indexlinks.app/blog/crawled-currently-not-indexed"}],"style":"normal"},{"_key":"k00122","_type":"block","children":[{"_key":"k00117","_type":"span","marks":["strong"],"text":"Duplicate, Google chose different canonical than user:"},{"_key":"k00118","_type":"span","marks":[],"text":" Google indexed another URL in its place. Check with our "},{"_key":"k00120","_type":"span","marks":["k00119"],"text":"canonical URL checker"},{"_key":"k00121","_type":"span","marks":[],"text":"."}],"level":1,"listItem":"bullet","markDefs":[{"_key":"k00119","_type":"link","href":"https://indexlinks.app/tools/canonical-url-checker"}],"style":"normal"},{"_key":"k00127","_type":"block","children":[{"_key":"k00123","_type":"span","marks":["strong"],"text":"Excluded by 'noindex' tag"},{"_key":"k00124","_type":"span","marks":[],"text":" or "},{"_key":"k00125","_type":"span","marks":["strong"],"text":"Blocked by robots.txt:"},{"_key":"k00126","_type":"span","marks":[],"text":" a site setting is stopping it."}],"level":1,"listItem":"bullet","markDefs":[],"style":"normal"},{"_key":"k00129","_type":"block","children":[{"_key":"k00128","_type":"span","marks":[],"text":"Also remember that \"URL is on Google\" does not guarantee the page ranks for anything. Google's URL Inspection documentation says to search for the exact URL to confirm it appears."}],"markDefs":[],"style":"normal"},{"_key":"k00131","_type":"block","children":[{"_key":"k00130","_type":"span","marks":[],"text":"How do you get other search engines to crawl the page?"}],"markDefs":[],"style":"h2"},{"_key":"k00139","_type":"block","children":[{"_key":"k00132","_type":"span","marks":[],"text":"Request indexing only reaches Google. Bing, which ChatGPT search relies on, and Yandex, Naver, Seznam and Yep accept "},{"_key":"k00134","_type":"span","marks":["k00133"],"text":"IndexNow"},{"_key":"k00135","_type":"span","marks":[],"text":" pings of up to 10,000 URLs per request with no daily button quota. Bing Webmaster Tools also has its own URL submission tool. Our article on "},{"_key":"k00137","_type":"span","marks":["k00136"],"text":"what IndexNow can and cannot do"},{"_key":"k00138","_type":"span","marks":[],"text":" covers the limits."}],"markDefs":[{"_key":"k00133","_type":"link","href":"https://www.indexnow.org/documentation"},{"_key":"k00136","_type":"link","href":"https://indexlinks.app/blog/what-indexnow-can-and-cannot-do"}],"style":"normal"},{"_key":"k00144","_type":"block","children":[{"_key":"k00140","_type":"span","marks":[],"text":"IndexLinks automates this for sites you own: each new or updated page goes to IndexNow engines, your Search Console, Baidu and the AI crawlers, and you get a receipt for every verified crawl within 24 hours. It guarantees a crawl, not indexing. Start with the free site check on the "},{"_key":"k00142","_type":"span","marks":["k00141"],"text":"IndexLinks homepage"},{"_key":"k00143","_type":"span","marks":[],"text":"."}],"markDefs":[{"_key":"k00141","_type":"link","href":"https://indexlinks.app/"}],"style":"normal"},{"_key":"k00146","_type":"block","children":[{"_key":"k00145","_type":"span","marks":[],"text":"FAQ"}],"markDefs":[],"style":"h2"},{"_key":"k00149","_type":"block","children":[{"_key":"k00147","_type":"span","marks":["strong"],"text":"How long after Request indexing does Google crawl the page?"},{"_key":"k00148","_type":"span","marks":[],"text":" Often within a day or two, but Google only commits to \"a few days to a few weeks\". Check the \"Last crawl\" date in URL Inspection."}],"markDefs":[],"style":"normal"},{"_key":"k00152","_type":"block","children":[{"_key":"k00150","_type":"span","marks":["strong"],"text":"Can I request indexing for a page I do not own?"},{"_key":"k00151","_type":"span","marks":[],"text":" No. Google says you \"can't request indexing for URLs that you don't manage\". You need owner or full user access to the property."}],"markDefs":[],"style":"normal"},{"_key":"k00155","_type":"block","children":[{"_key":"k00153","_type":"span","marks":["strong"],"text":"Is there a bulk Request indexing option?"},{"_key":"k00154","_type":"span","marks":[],"text":" Not in Search Console. For bulk, submit or resubmit a sitemap. The Indexing API is limited to job posting and livestream pages."}],"markDefs":[],"style":"normal"},{"_key":"k00158","_type":"block","children":[{"_key":"k00156","_type":"span","marks":["strong"],"text":"Should I request indexing for every new blog post?"},{"_key":"k00157","_type":"span","marks":[],"text":" Not if your sitemap updates automatically and the post is linked from your blog index. Save the quota for urgent or important pages."}],"markDefs":[],"style":"normal"}],"category":"Search Console","excerpt":"Step by step Request indexing in Search Console, the daily quota, when to use a sitemap instead, and why a request may not work.","longDescription":null,"publishedAt":"2026-10-10T09:39:04Z","readTime":"6 min read","sections":null,"seoDescription":"How to request indexing in Google Search Console, the daily quota, when a sitemap is better, and what to do when a request does not get indexed.","seoTitle":"Google Request Indexing: How to Use It in Search Console","shortDescription":null,"slug":"google-request-indexing","status":"published","title":"How to Request Indexing in Google Search Console","toolType":null}]}