Indexing Hell

Aw, yeah, baby! Google’s indexing algorithm still hasn’t gotten its shit together, so I’m going to be applying some of the only fixes I’ve confirmed to be effective. First of all, a recap: Google does not index many of my pages, meaning they can’t show up in search results. For those fortunate enough to have not worked with Google indexing, you may be surprised to know that Google is more than willing to throw its hands up in the air and exclaim, “we don’t know why this is happening.”

A disclaimer: I am stupid and don’t know anything. However, I do try not to be. If hours of my research into this problem hasn’t yielded satisfactory explanations, solutions, or justifications, I’m left to assume that my resources just… suck. Ignorance is my fault to an extent, but I can’t help it if the documentation I’m reading isn’t helping me learn. In that sense, maybe some fault lies with how these concepts are communicated, and maybe this post will convince someone within this sphere to change their language.

Anyway, when a page is “discovered but currently not indexed,” Google links the domain owner to a resource or two containing irrelevant suggestions and practices you’ve likely already implemented:

“Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl.” — No, it wasn’t. My site has been pinged hundreds of times in one day by various site-management tools.

“The page was crawled by Google but not indexed. It may or may not be indexed in the future.” Wow! Very helpful. Unfortunately, “may not be” is not an acceptable outcome, so I’d like to know how to achieve the other one. Oh, you don’t have anything for that. “God works in mysterious ways.”

Trying to find non-first-party resources is met with similar faith-based declarations, deference to the Google totem, or other forms of magical thinking. A common mantra is “low-quality content,” which should read as hysterical to anyone who’s navigated the mélange of garbage Google makes host to every single day.

When this is pointed out, you may get a reply along the lines of, “well, it’s not referring to whether or not the page is actually quality,” which, to me, just reinforces the idea that all of this is being communicated in vibe-based buzzwords that don’t actually mean what words normally mean. Most people don’t seem to know what’s going on or why, but they see other people they recognize as authoritative use vague phrases and terminology, so they adapt those for themselves. It’s coded language that doesn’t serve to be deciphered, but merely works to signal to others that you’re in the same algorithmic, faith-based cult.

I’m tired of running in circles and receiving contradictory advice. I don’t want to pretend there’s consistency to any reasons Google indexes some of my pages and not others, because there isn’t, evident by the fact that any suggestion can be met with exceptions. It’s an amorphous blob of adaptive algorithms that we work for, instead of computers working for us.

Oh, and if you’re naive enough to think you can make a support query, don’t bother. Despite being one of the wealthiest companies on Earth, Google can’t spare a penny to hire enough people to actually have a human look at your message. I haven’t even received the dignity of a stock reply; they just ignore your tickets now.

The closest I’ve gotten to uncovering any kind of explanation for this bullshittery is “backlinks.” To my understanding, this refers to, well, links back to the page in question. Many Google followers explain this process as Google attempting to determine if your content has “value” before they index it. And, if I may: what the fuck sense does that make? How can people determine the value of my content if they can’t fucking see it?

Isn’t that circular logic? For small, contained communities, such as the sciences, I can see this being a functional approach. For heavily-saturated topics, I don’t understand how this would work (and apparently, it doesn’t). I could relentlessly self-promote my work across other sites, but then I would just be getting traffic via attrition, not quality. Wouldn’t that not be a good metric for quality? It seems to me that returning visits, as well as time spent on a page or site, should be the main arbiter of quality. But again, that still assumes they can get there in the first place.

It seems to me that Google’s standards would just lead to low-quality content whose URLs are spammed by bots topping their search results.

…

So, anyway…

I first really butted heads with this issue when I published my review of the Yu-Gi-Oh! Early Days Collection. And, y’know, I don’t ever think any of my work is perfect. However, I’m not modest. I know when I’ve worked hard on something, and I know when it bears results, and I know the standards of my contemporaries. That was a good review. Anyone who might say it’s below Google’s general “quality” of search results would not be saying so genuinely. It’d just be an insane position, even if you didn’t like the article, because we’re grading it on a curve. I will not take seriously any suggestion that the article was “low quality.” Come on.

I worked tirelessly to complete that review close to the game’s launch in order to maximize traffic. This intent was made meaningless by Google, who, again with no explanation, did not index the page. No one could find it. Anyone who wanted details on this new game were unable to find one of the best critical sources available for it at the time, despite my efforts. I tweaked various things and tried daily manual indexing requests, but nothing worked.

That page is now indexed, albeit at least a month later than it should have been, and as such, I lost who knows how much traffic in the interim. So, what changed? You’re not going to believe it if you haven’t heard: I made a blog post where I linked to that review twenty-some times after I had first learned of the “backlinks” explanation. By the next day, my review was indexed.

Amazing. All the bullshit lip service about how these esoteric practices were necessary to combat low-quality content, and all it took for me to flip the visibility switch was to link spam on my own goddamn site. If that doesn’t demonstrate how pointless and destructive these barriers are, then I don’t know what does, frankly.

To me, I feel like there ought to be some kind of “good faith” system in Google’s indexing algorithm. If my site has been vetted for sincere content in the past, it should override any indexing barriers and index every new page by default, until I otherwise demonstrate my site is no longer reliable. That makes perfect sense to me. Am I saying I know more than Google’s staff of programmers with master’s degrees? I’ll grant you that’s not very likely, but Herman Cain of all people was a brain surgeon.

So, back to present day. I’ve recently begun writing video game guides. I did so because I thought my advice wasn’t commonly found. The bulk of these pages have not been indexed. I can’t be very helpful to people then, can I? Not to mention, it just feels bad knowing that your hard work only exists in a black hole.

I think my hand is forced. I will have to regularly make blog posts containing link spam to my own content so that Google will index it. I don’t like to do this, because it looks like shit, but that’s the fix I’ve been given.

This won’t be for all of my pages; mostly because that would be far too much work. But for my more important content, yeah, they’ll get the treatment until it stops working. I want to reiterate that there is no consistency with which pages get indexed. Some pages do, and some don’t. Some have backlinks, but many don’t. There is no behavior I can change on my end to make pages index by default, because that implies there’s consistency in the first place. Oh, and if it needs to be mentioned, my site natively backlinks due to my taxonomy pages and sitemap. There’s no reason I should have to do more.

So, game on, Google. You want me to be annoying? I’ll show you annoying!

…

Okay, look. Now that I‘ve calmed down a bit, I know some more knowledgeable people may read this article and find my tone aggravating, if not downright luddite. That’s understandable. Just please, try and put yourself in my shoes. From my perspective, I’ve tried really hard to figure this shit out, and I’ve had no follow-up when my attempts were met with failure. I just want the content I put so much time, care, and sincerity into to be treated with the same amount of respect I see Google treat web pages that… don’t have those things. Being told you’re not good enough with no explanation as to why is one of the worst feelings in the world. I think anyone reading this knows that.

If the link spam there doesn’t get those pages indexed, I’ll return to add some more until it does. If that method no longer works, which wouldn’t be shocking, I’ll… crash out again, I dunno.

Other Shit

It hasn’t been that long since my last post, so nothing’s changed that much. I finished my playthrough of Alpha Gaiden, and it has an in-progress review. I intended to play through at least the other main route, but my attention has been grabbed by my replay of Super Robot Wars J. I’ve started my review for that one already, in the hopes that it’ll be released on better time when my enthusiasm for talking about the game doesn’t slowly wane upon completing it.

Starting the guides has been really therapeutic, because it gives me a place to talk about some of these games without needing something in a complete narrative structure (i.e., a review). Guides can be created in pieces, which is liberating. I think it might also cut down on the density of my reviews in some areas, as writing surrounding a game’s meta can be corralled into its own page. I do think this will be for the best.

Well, I’ve got some work to do. Thank you for reading my rant, and I hope you can empathize with me, even when I fly off the handle somewhat.