Missing URLs in Staging? Fix Your Crawl Issues

Got missing URLs on your staging site – even after reconfiguring Screaming Frog and Ahrefs? In this Helpdesk episode, we answer Beck Park’s question from Foresight Digital about missing “Careers” and “Why Choose Us” pages during crawls.

We break down what might be happening:

• Are the pages orphaned?
• Is your JavaScript-only menu hiding them?
• How can HTML-first navigation fix the problem?

Cara walks through the diagnosis and a simple two-step solution to improve your crawl coverage and indexing success.

Video Transcript

0:08 Today we have a question from Beck Park at Foresight Digital. She says, “Our crawlers, Screaming Frog and Ahrefs, are missing several URLs on our staging site despite multiple crawl reconfigurations. The missing URLs include key pages like ‘Careers’ and ‘Why Choose Us’ content. Is this a crawl configuration issue or is a problem with our headless CMS?”

0:29 Thanks for your question. So, it sounds like you’re missing some key URLs from your staging site despite even tweaking your crawling tools. And you’re wondering, is this a crawl issue or a headless CMS issue or something else? Don’t worry, we hear this stuff all the time and we can definitely help.

0:44 First, let’s talk about orphan pages. Think of your website like a map. Search engine crawlers follow the roads, which are your internal links, to all your destinations and pages. Now, if your page has no roads or internal links leading to it, we call this an orphan page. The crawlers simply have no way to find it.

1:02 You mentioned missing pages like the ‘Careers’ page or the ‘Why Choose Us’ page. If you’re seeing your main pages but not the subpages, there’s a good chance that those subpages are orphan pages. And the fix here is pretty straightforward: just make sure that every page on your site has at least one internal link leading to it. It’s like building those roads for crawlers to find the pages.

1:23 Now, let’s look at your main menu. When we took a deeper look at your website, we noticed that it was built entirely on JavaScript. JavaScript makes for those sleek, interactive menus that we love, but the crawlers primarily depend on HTML. They’re getting better at JavaScript, but they do prefer the basics.

1:39 So, right now, if your menu is only JavaScript, the crawlers might not actually see some of those pages at all. They’re really missing that crucial navigation path. This is especially important even as AI crawlers become more prevalent, because many of them struggle with JavaScript as well.

1:56 So the solution here is to implement your main menu in static HTML first and then enhance it with JavaScript. This way the crawlers still can see the links in HTML and the users can get that beautiful JavaScript experience.

2:08 So to recap: make sure you link to those orphan pages and build your navigation menu using HTML. These two fixes should significantly improve your crawl coverage and ensure that all of your important pages can be found.

2:20 If you’re looking to boost your SEO knowledge and get your website performing at its best, make sure to join our free partner club. Get access to free SEO resources, our expert help desk where you can ask questions like this one and get answers to all of your burning SEO questions, and even free SEO audits for your clients. You’ll find the link below. Thanks for watching.