Google search index frustrations. Out of ideas and looking for advice

Marketing & promotion
1

Hi all. I posted this on the Google Search Console community as well but they haven’t been a huge help in the past so I’m hoping someone here might have ideas.

I know tons of people ask about the “Crawled - currently not indexed” issue and trust me, I’ve not only seen all the common advice but spent months and months reading up on this and trying to solve it, including reducing myself to using Gemini. Nothing’s helped and to me, something either feels off or I’m just missing some key element.

Short history: I have a hobby blog that I’ve been running since 2012. It’s had hiatuses and has been through various hosts and platforms over the year. I re-launched it using self-hosted Ghost last Summer. Unfortunately, I wasn’t paying attention to SEO for much of its life and I’m sure it wasn’t being indexed properly for a long time and has a poor reputation with Google as a result.

Initially, there were a couple of elements of my Ghost configuration that could have been preventing proper crawling so I’ve corrected those. I can see in GSC that the site is being regularly crawled (though it doesn’t seem to have been for over a week now for some reason) but Google keeps refusing to index the pages.

To avoid making this post too long, here are my observations that are confusing me:

  • I’ve added my sitemaps and RSS feed but it never reads them again after initially adding them.
  • I’ve verified through the Screenshots of the Live Test that pages are rendering in Googlebot properly.
  • The only thing Google has indexed it the site’s home page.
  • I have the posts set to show recent and featured posts in the side bar, creating internal linkbacks.
  • I do put new posts on LinkedIn and Bluesky, my Discord and I’m also in a handful of blog roll indexes. I don’t use much social media and try to link my posts elsewhere when I can and they’re relevant but as many know, most communities get upset at self-promotion, even when it’s the only kind of promotion you have.
  • Every time I look up a URL in GSC, it says there is no referring sitemap and no referring page, both of which are not true. It also shows N/A for all the crawled stats, even if I’ve manually had it do so multiple times or if it shows up in the Crawl Stats report.
  • Every URL read attempt in the Crawl Stats report shows an OK status.
  • I’ve tried “Validate Fix” three times now. Every time, it sits processing for months and nothing changes when it completes.
  • I have been waiting literally 6+ months since I corrected every issue I was told about and despite seeing constant crawling, nothing is being indexed beyond the home page.

And before anyone makes the quality argument, I’m sorry but I just don’t buy that. My posts are in adherence to all of Google’s guidelines, they are 100% human written, link to relevant content and with all the linkfarm slop I see at the top of most search results now, I just don’t accept that my posts aren’t good enough. They’re not going to win a Pulitzer but are easily on par with stuff that is getting indexed. Multiple other search engines also have no issue with them. I’ve also talked to a couple of SEO professionals on LinkedIn who offered to take a free look at things for me and both of them said they don’t understand why this is happening.

What’s frustrating here is that GSC just appears to have such bad mojo towards my site that it refuses to update sitemaps after they’re added and straight up lies to me about the stats for crawls. Its own logs show that it is checking the site but is just ignoring everything it finds and not telling me the truth about what it’s doing. This feels to me almost like something’s bugged as opposed to a real issue but since there is basically no support for this, I can’t even confirm if that’s a real issue.

Surely after fixing the issues months ago and with months of successful crawling, GSC knows the site is reliable now, right? If not, does anyone know how long it can be stuck in limbo like this before it finally gets trusted again? And what about the stats that are incorrect? I have several larger posts that I’ve held off publishing because I don’t want to put in all that effort just to be lost in search engine limbo but at this point, it looks like it could be stuck here indefinitely.

This is a hobby blog that I don’t monetize and which costs me money to run. I’m not looking to turn it into a career, I just want people to be able to find what I say so those interested can read it and discuss it. But I’m out of ideas now and even Google’s own AI is telling me “You’ve done everything right, just keep waiting.”

I’m really hoping someone might have some insight here because I’m rather demoralized at this point and feel like I’ve tried everything.

Thanks.

3

I think I found it. Your old post URLs don’t resolve. Two examples pulled from the Wayback Machine:

https://geekbravado.com/2016/10/12/announcing-my-2016-extra-life-campaign-for-cheo-lan-party-edition/ returns a 404.

https://geekbravado.com/2018/06/15/so-i-found-out-some-interesting-brain-stuff-this-week/ 301s to /2018/06/15/so-i-found-out-some-interesting-brain-stuff/ and that 404s too.

So there are redirects in place, but they point at a date-based URL structure that Ghost doesn’t use. Ghost serves posts at /slug/, no year and month. Every old backlink and social link from the WordPress years ends in a 404, some via a 301 first. That is the worst version of this for Google: the site looks brand new, and the crawler keeps hitting broken chains.

If the Ghost slugs match the old WordPress slugs, one regex line in redirects.yaml (Settings > Labs > Redirects) fixes the whole archive:

  • from: ^/\d{4}/\d{2}/\d{2}/(.*)$
    to: /$1
    permanent: true

Worth checking a handful of old URLs first to see if the slugs line up. Where did the existing 301s come from, an old WordPress redirect plugin or something at the host level? That decides whether they need to go.

4

Hey Richard!

Thank you for this information, it’s very helpful! That’s weird to me because I thought I had Ghost configured to redirect any URLs it didn’t recognize to the home page. It clearly isn’t though as I just checked the links you posted.

The thing is that I can’t just do 1:1 redirects for every old post because I didn’t carry many of those posts over when I moved to Medium and then to Ghost as they were either posts I didn’t like anymore or like the Extra Life one, were for events that had long passed. I’m not sure what’s up with the “brain stuff” post as that should be redirecting via my Ghost redirects file.I checked and indeed, the URL you referenced should be redirecting to the new one but it’s applying the date to the front for some reason. I put in a second redirect and now it’s going to the right place. That’s odd. It’s strange though because GSC isn’t showing a lot of failed redirections. It showed some initially, which I manually fixed in my Ghost redirects file and they diminished. So it’s either not counting them all or something else weird is going on. GSC says the only 404 errors in the last 90 days are for ads.txt, which I’ve read doesn’t impact your search ranking.

Also, a Google engineer actually responded to my post I put in their forums and dug into my account on the back end. Though he was limited in what he was allowed to tell me, he did say to my great surprise that a forum I haven’t used in 12 years but that I had linked to my blog in the signature of back when I launched it had backlinked to the site I’m not kidding you, 149,000 times and only 4,000 of those has the nofollow tag applied. None of that appears in my GSC but he said it’s absolutely been happening and could be causing Google to think I’m faking backlinks. I have no idea how or why this site would be doing that but it’s a small forum that I think is still Drupal based so who knows how it’s configured. I recovered my account there and emptied out my signature. I emailed them asking if that removed the signature from prior posts and I also asked to just have my account deleted but they haven’t responded yet. I’m not counting on them doing anything. The engineer did suggest disavowing backlinks from that domain but GSC won’t let me do it because my site is added as a domain property and not a URL property. I’m seeing if I can change that without breaking things.

Is the best practice in this scenario just to redirect any unknown links to the home page or will that also upset Google? If so, I can try to find all my old links and redirect to new posts where I can. However, this post seems to indicate it can be better to let posts with no replacement 404 rather than redirecting all unknown links to the home page.

Thanks again!

5

Good news, most of this is small stuff.

The date-prefix thing is a redirects.yaml bug, not a Ghost quirk. If a to value has no leading slash, the browser treats it as a relative path and resolves it against the current URL, so /2018/06/15/ gets glued on. Check the file for entries like

to: so-i-found-out-some-interesting-brain-stuff/

and make them

to: /so-i-found-out-some-interesting-brain-stuff/

Chances are the “fixed” ones you did by hand earlier were the same issue.

On the catch-all: don’t redirect unknown URLs to the home page. Google treats that as a soft 404 and it looks worse than a plain 404. Posts with no replacement can just 404, or 410 if you want to be explicit. Only redirect where there’s a real new post, and start with the old URLs that actually had traffic or backlinks, which you can pull from the Links report in GSC or the Wayback Machine.

The disavow problem is easy too. Add geekbravado.com as a URL-prefix property next to the domain property. Both can exist at the same time, nothing breaks, and the disavow tool works on the prefix one.

The 149k links are less scary than they sound. A signature shows on every post, so a forum with 149k pages gives you 149k links. Google has handled that pattern for twenty years. Clearing the signature was the right move; I wouldn’t expect much from the disavow.

1 Like
6

OK, I fixed the issue with the “brain stuff” post by adding another redirect. I also dug back through the GSC history and found another couple of posts to put in redirects for, though they haven’t been hit often or recently.

I added the forum site to the disavowed list. Question though: How does a link in my signature that appears many times on the site count as a backlink to Google? If those links aren’t clicked on, there’s no traffic to the site so how does it know I’ve been backlinked? Or if the site uses Google Analytics, is it just counting all those links, even if they aren’t clicked? This is another case of my SEO ignorance. :slight_smile:

I also did discover something yesterday that blew my mind but I don’t know if it’s relevant. So the first incarnation of my blog was hosted on WordPress.com, their own hosting platform. The geekbravado.com domain was attached to it back then but has long since been removed. However, my account there was apparently still active and geekbravado.wordpress.com was still active and being indexed by Google. That’s probably where some of those old backlinks with dates in the slugs were coming from. I swear I deleted that site but I guess not. I deleted both the site and closed my WordPress.com account. Not sure if that will have any impact but maybe?

If these two things don’t help (and I know I’ll have to wait potentially weeks to see if they do), I’m pretty much out of ideas.

7

Backlinks have nothing to do with clicks or analytics. Google crawls the forum, sees your URL in the HTML of every page, and counts it. That’s all a backlink is.

The WordPress.com thing is not a maybe, I think that’s your actual problem. If geekbravado.wordpress.com was still live with the full original archive, Google had an older, more linked version of every post, with canonicals pointing at itself. Your Ghost site was the duplicate. The home page is the only page that’s really different, so that’s the only page that got indexed. It also explains the date-based URLs I ran into earlier: when I fetched one of them I got WordPress.com markup back, not Ghost, and I couldn’t figure out why.

Deleting it works, it’s just the slow route. A redirect from the subdomain would have been faster, but once the site is gone that’s off the table. Now Google has to drop the wordpress.com URLs one by one before it picks your Ghost versions as the originals. A few weeks is realistic, could be longer.

Two things to speed it up: use URL Inspection in GSC to request indexing on your ten best posts, and check that the Ghost posts have a canonical pointing at geekbravado.com and not at anything else. Ghost does that by default, but worth a look since this site has been through a few migrations.

I wouldn’t touch anything else for now. Everything you’ve fixed this week is real, give it time to land.