← back Monday, July 28, 2025

Boosted SEO Crawl Efficiency by 85% with Improved HTML Crawling

While reviewing crawl reports across DentalPlans.com properties, I noticed an alarming split: only 45.4% of crawl activity was spent on HTML, while 40.6% was consumed by JSON API endpoints. Since the marketing site is primarily server-rendered and optimized for SEO, this misallocation was a clear drain on crawl budget and search engine visibility.

I traced the cause to session-based visitor tracking calls and other non-indexable requests that were triggering for every bot visit. By blocking these endpoints from crawlers, crawling immediately shifted toward more valuable content.

After just three months, JSON crawl requests dropped to just 1.8% while HTML crawling surged to 83.9%. This is a dramatic gain that significantly improves how efficiently search engines discover and index our marketing pages.