3 stories
Discovered – currently not indexed means Google has not fetched the page. Crawled means it fetched it and declined. How to diagnose and fix each status.
A 429 means Too Many Requests. Serve one to Googlebot and crawling slows across your whole host within seconds, and the recovery is measured in weeks.
Shopify's generated robots.txt is right for most stores. Here is what it contains today, the six cases that justify robots.txt.liquid, and how to add rules.
Showing 3 of 3 stories
You're all caught up.
Get the top AI news and tools straight to your inbox every week.
Weekly · AI news
Join 15,000+ AI enthusiasts
