In the past few hours, Google apparently updated its database and moved blogs way, way down the list of search results. First Atrios mentioned it and thought it strange, and then Sisyphus Shrugged found it had happened to her. That prompted me to google my own site, and it's true for me, too: this blog was the first search result for "Fables of the reconstruction" and the third (I recall) for "Mithras" until now. Now, it is the 53rd result for "Fables of the reconstruction" and does not appear at all in the first 200 results for "Mithras" (although, strangely, my profiles on other websites do show up in those results.)
Strangely, this blog still has a google pagerank of 5 (out of 10), which is google's numerical way of denoting importance.
We'll see how this shakes out. But if it persists, this is very bad news for blogs. I have written elsewhere that blogs are only an important phenomenon (important other than for the intrinsic satisfaction of writing) to the extent that search engines prioritize blogs. There are, by my estimate, only about 500,000 people who actively search out political and current events blogs. That number is not likely to grow much; all of the people who are likely to take an active interest have had a chance to join us by now. While not nothing, such a number is quite small in cultural terms. What makes blogging potentially culturally relevant is being indexed by search engines. Until today, if you googled any topic of current interest, the first page of results included blogs. That made blogs relevant to general internet users, but much more importantly, it made blogs vital for journalists researching stories. The prominent placement of blogs in google search results meant that bloggers, rightly or wrongly, were paid attention to. It also meant that blogs occupied valuable internet real estate that attracts advertising revenue.
Take search engines away, and the entire enterprise dries up. No google hits, no blogs. It's that simple.
As I said, we'll see how this shakes out.
Update 2/16/05: Moved up from comments:
Before this change, googling "Fables of the reconstruction" returned this blog as the first result. When I wrote this post, it had fallen to 53. Well, now it's not in the first 200 results, which is where I stop looking. Links to this blog and comments I have left on other blogs rank higher than the blog itself.
I don't know what google did, but I have to say, it seems fucked up.
Update 2/18/05: Based on a completely unscientific, nonrandom sample, it appears that this affects blogs that use their blogging software's domain, such as typepad.com, blogs.com and blogspot.com. Blogs with their own domains do not appear to be affected. What this means, I can only speculate, and I am down with the flu today so I am not going to research it.
I think there's a case to be made that blogging has just enough cachet to make it past a move like this; technorati, for example, is already way better than google for those who know to use it.
However, even if John and Jane Smith don't figure out Technorati, I still don't think all is lost. From The Register in 2003:
Google is to create a search tool specifically for weblogs, most likely giving material generated by the self-publishing tools its own tab.
It isn't clear if weblogs will be removed from the main search results, but precedent suggests they will be. After Google acquired Usenet groups from Deja.com, it developed a unique user interface and a refined search engine, and removed the groups from the main index. After a sticky start, Usenet veterans welcomed the new interface. Google recently acquired Blogger, and sources suggest this is the most likely option.
With Google's penchant (is that two french words in one comment?) for rolling out beta tests with little to no fanfare, I wonder if we're just seeing the first step.
Posted by: Auguste | February 15, 2005 at 05:28 AM
Sure, Google has a bunch of specialized searches. They are obscure, however. Maybe google will stick blogs in news results. I'm not sure how those work; my perception is they are indexed to favor most recent over most linked, which inverts the current situation.
No, either way, if this persists, blogs have lost significant ground in one step.
Posted by: Mithras | February 15, 2005 at 08:25 AM
If bloggers yell about this enough, I don't think Google will hold the line on this. And if this is a long term Google change, bloggers are going to yell.
Secondly, I've been wondering when Google is going to buy either Technorati or Feedster and plug it into their search structure. It just seems like destiny.
Posted by: PSoTD | February 15, 2005 at 10:04 AM
Man, I can't think of a better thing to have happen. First, the search traffic is really trash anyway. Maybe - just maybe - there's people who find out about a blog via a Google search. But I can't really believe it's that big of a deal. Secondly, the blogs really were skewing searches - I can't count the number of times I had to click through 2 or 3 pages of search results to get to the actual stuff I was looking for and not just some blog index that just happened to have all the words I was searching for mashed together in their weekly archive. Third, maybe this will take some air out of these Jackanapes' blog ads. I know, it'll be pain all around, but I don't think the people we care about will be hurting - just my feeling.
Posted by: Hal | February 15, 2005 at 10:32 AM
Well, I think that in some cases, blogs do work that no-one else is doing, and downranking them is inappropriate. For example, that whole thing with Dred Scott still pulls a lot of google traffic for me, and though I don't really care that *I* get the traffic, there isn't too much other information out there about that sort of thing.
Maybe a better solution would be for Google to manage the index pages better; there's only a few tools out there, and the way they handle archiving should be easy enough to cope with.
Posted by: paperwight | February 15, 2005 at 11:53 AM
I was wondering what I'd done to piss off so many readers -- traffic was down maybe 10% or so all of a sudden. And there was a drop-off in weird google searches. And I'm no longer #1 on google for "tentacle sex"!
But really, I can't get too worked up about it. So somebody looking for tentacle sex will get his jollies on some other site...no prob. So a lot of casual readers who used to click through looking for something will never even peek in. I don't see it as a loss. It just means we'll have a slightly smaller but more serious and involved readership.
Posted by: PZ Myers | February 15, 2005 at 01:59 PM
I'm not concerned about it. Most of my traffic comes from Google, but it is rare one of those Googlers leaves a comment. The traffic I am concerned about are repeat visitors who come to my blog from other related blogs.
Posted by: Kim Siever | February 15, 2005 at 03:01 PM
If bloggers yell about this enough, I don't think Google will hold the line on this.
Um, why?
I know, it'll be pain all around, but I don't think the people we care about will be hurting - just my feeling.
It's not pain I am concerned about, it's actual cultural relevance.
Maybe a better solution would be for Google to manage the index pages better; there's only a few tools out there, and the way they handle archiving should be easy enough to cope with.
Paperwight, I don't understand what you're talking about when you mention the "index pages." I get the impression that search results are dependent on a formula, and that "is blog/is not blog" is one of the terms of that formula, and the contribution of the "is blog" result just changed. (Okay, I have a caveman's understanding of how this stuff works.)
Hal, PZ and Kim-
See, in my view, if blogs have cultural relevance, it is as an explanatory device. If that explanatory function is limited to those who purposely seek out blogs, then its usefulness is very limited. It will reduce the power of distributed thinking to a cool but small university, rather than a widening conversation. (Jesus, I normally hate using that phrase.) If I google "Valerie Plame" and don't get blogs (although I still do now), then I get someone else's spin, someone most likely within existing power structures.
But hey, maybe I am blowing this out of proportion. Like I said, let's wait and see. I wonder if google will reveal what it's done, though.
Posted by: Mithras | February 15, 2005 at 03:22 PM
Paperwight, I don't understand what you're talking about when you mention the "index pages." I get the impression that search results are dependent on a formula, and that "is blog/is not blog" is one of the terms of that formula, and the contribution of the "is blog" result just changed.
What I mean (and I think I'm explaining it badly) is that blogs archive posts differently. For example, my blog archives by week and by category (each of those a page, regardless of the number of posts), and of course, each individual post can come up as a separate page. There might be a peculiar combination of words in each archive which triggers an inapposite Google result.
If those pages could be ignored, while individual posts were crawled, I think much of the Google problem could be solved.
Posted by: paperwight | February 15, 2005 at 04:22 PM
If bloggers yell about this enough, I don't think Google will hold the line on this.
Um, why?
Haven't you been watching the news? We bloggers - we're powerful. ;)
Honestly, word of mouth is important to any Internet enterprise, even one as large as Google. If bloggers complain, complain, complain about a perceived mistreatment, that's not going to be good for Google - it's image, and it could be usage as well. If bloggers find that another search tool is more useful for them, both for purposes of search and for traffic, that will be regularly announced on blogs.
To be honest, I think Google better get moving on what they play to do with Blogger and blogs in general. Ask Jeeves bought bloglines last week. Is Google going to let bloggers start singing hosannas for Teoma? I don't think so - like someone else posted, I think there's something else up their sleeve, and it's probably going to excite bloggers in a good way, based on their past development history.
Posted by: PSoTD | February 15, 2005 at 04:33 PM
It seems like Google is also cracking down on bloggers in its offices.
Posted by: BVSC | February 15, 2005 at 04:42 PM
Weird, but kind of makes sense if Google is aiming to provide "authoritative" information for people's searches, no?
Anyway, wouldn't you blog even w/out google hits?
I realize these are insubstantive comments.
Posted by: bitchphd | February 15, 2005 at 05:22 PM
I think we'll just have to wait and see how it pans out. It's true that it operationally places a lower value on blog entries vs. other kinds of pages, but whether that is inappropriate or not is hard to determine.
Posted by: PZ Myers | February 15, 2005 at 05:52 PM
I did not use Google to find *any* of the blogs I read. I found almost all of them through blogrolls or links provided by other blogs. I suspect this is true for most people who use blogs.
As others have suggested, I'd guess the big losers in this case will be the people who have blog ads.
Posted by: david | February 15, 2005 at 10:54 PM
david - yeah, that wasn't my point. People who seek out blogs purposely will still find them. It's everyone else who won't.
PZ - My first question for google is, what did they do and why? It might be nice to get an explanation.
Dr. Bitch - I don't think moving blogs down the list will cause results to be more authoritative, because the algorithm doesn't make sure what moves up is better in quality. And yes, I would blog without search engines, but yes, that's not what I was posting about. I didn't mean literally that blogging would stop.
Bella - I have read other things about the Jen story that make it sound like he was stupid. I don't think google has it in for us.
PSoTD - Hope so.
Paperwight - okay, now I get it. Funny, I never thought an "index page" result was a problem, so long as I could CTRL+F to the search term I was looking for.
Posted by: Mithras | February 15, 2005 at 11:13 PM
I googled myself and Blog Day Afternoon came in number one on google. I host my own site, so I thought perhaps google was whacking blog engines, i.e., blogs.com, blogspot.com, etc. But a query for "mithras" on google, yielded this site in the top 10: mithrastheprophet.blogspot.com/
Posted by: Jeff | February 16, 2005 at 08:44 AM
I dunno -- I did a search for "how bad is Gonzales" and got the Kos entry first and a number of other blog entries and discussions in the first page of results. That seems promising on the "get the opinions out there" angle. I can imagine that the proliferation of blog-links and cross-referencing makes handling those pages different than handling regular web pages, but the scrolling-off-the-front-page thing should make it self-correcting as well. who knows....
Posted by: acm | February 16, 2005 at 09:57 AM
If Duncan Black's blog isn't the top result for "Atrios," then Google's algorithm is defective on its own terms. The goal is to produce maximally relevant results, relative to the likely interests of the user. The vast majority of users who Google "Atrios" or "Eschaton" are probably looking for his blog.
Posted by: Lindsay Beyerstein | February 16, 2005 at 10:24 AM
I'm a nutritionist, and a friend of mine thought it would be a great idea for me to have a webpage. I didn't want to fuss with programming, so when I saw Google's Blogger, we thought it would be perfect for the content I wanted to post. And it has been. My blog isn't so much a journal as a collection of brief articles and recipes.
I can't tell you how many people have found me by Googling "non-hydrogenated vegetable shortening". It would be a shame if all they get now is Crisco's site!
So if what you say is true, and even fewer people will read my posts, maybe I'll need to leave Google's Blogger altogether.
Posted by: fanatic | February 16, 2005 at 10:57 AM
Before this change, googling "Fables of the reconstruction" returned this blog as the first result. When I wrote this post, it had fallen to 53. Well, now it's not in the first 200 results, which is where I stop looking. Links to this blog and comments I have left on other blogs rank higher than the blog itself.
I don't know what google did, but I have to say, it seems fucked up.
Posted by: Mithras | February 16, 2005 at 11:19 AM
I'm now coming up _higher_ than before in google. I'm the number one result for both "Cyn" and "pink haired girl", and before I was at least halfway down the page for "Cyn."
Maybe it's because I have my own domain?
Posted by: Cyn | February 16, 2005 at 12:48 PM
I'm still the #1 hit for "paperwight", and while I have my own domain name, it's purely forwarding -- fairshot.typepad.com is what comes up when you search for me (or in any searches that turn up my writing). A search for "fairshot.org" only turns up the main page of the blog.
Maybe Google hasn't gotten around to me yet, but Mithras is right, something is pooched.
Posted by: paperwight | February 16, 2005 at 02:07 PM
I'm still first for "pharyngula" (hah, big surprise there), but Lindsay's right: if you google for "Atrios", there's no direct link to his site on the first page, although there are a couple of other bloggers and you can get to it indirectly. That is kind of screwy.
Posted by: PZ Myers | February 16, 2005 at 03:01 PM
Well I got to thinking about the Atrios thing. Alot of people link back to comments or articles on his site witout using the main page or labeling it clearly as Atrios or Eschaton.
Now this mean he gets hits to his articles and comments and trackbacks but they don't return to search engine results. I do realize alot of folks have his blog blogrolled. But how many have it as Atrios, or Eschaton, or even Duncan Black. So what we end up with is alot of hits and links to his site that don't reflect the main page but the side pages so to speak.
Now like the old Google Bombs of old you get results by combining a word with a link. If everyone placed a few Atrios links on their page that said Atrios and linked to the main page, well possibly we could see a change within a day or two.
If not then Google does indeed have some explaining to do. As in why the number one relevenat link is not showing up. What changes to the algorythm have they implemented that banish the number one listing below the 200 mark. And sadly we just may have to accept that Google has gone the way of the dodo and find a new search engine, blogger approved, and advertise it on blogs until it becomes the most popular or Google changes their ways.
Posted by: IXLNXS | February 16, 2005 at 06:21 PM
I agree -- Atrios should be #1 for the search-term "atrios". But Google is generally screwy. One of my posts is number #1 for
realism ethics definition
moral realism definition
And I'm #3 for
shafer-landau moral realism
I certainly should not be #3 for "shafer-landau moral realism." So apparently Google ranks some blogs inordinately high on some searches, as well as ranking others inordinately low on others.
Posted by: david | February 16, 2005 at 07:26 PM
OK, I'll bite. Who should be #1 for "shafer-landau moral realism"?
Posted by: PZ Myers | February 16, 2005 at 09:46 PM
*rolls eyes* Oh, come on, PZ. Don't you know that?
Posted by: Mithras | February 16, 2005 at 09:47 PM
"shafer-landau moral realism"?? WTF? Ummm... wow, I think I just felt my IQ drop several points. I'll just go slink off to a corner and drool for a while...
Posted by: Ruth | February 16, 2005 at 09:57 PM
I guess this should be #1:
http://www.oup.com/us/catalog/general/subject/Philosophy/EthicsMoralPhilosophy/?view=usa&ci=0199280207
But if it had to be a blog post, I'd say this one would be a better choice than mine:
http://peasoup.typepad.com/peasoup/2004/08/shaferlandaus_m.html
Posted by: david | February 16, 2005 at 09:58 PM
Shafer-Landau thinks of his view as a kind of non-naturalism, but not of the substance dualism sort, according to which there is some sort of non-natural moral stuff in the world. Rather, he’s a (materialist) substance monist. But he’s also a property dualist: in addition to physical properties, there are irreducibly moral properties as well. This theory is ontologically akin to non-reductive materialism in the philosophy of mind, but Shafer-Landau considers his view epistemologically non-naturalist, since there is no science of ethics.
Er, yes. Obviously. Hey Ruth, make room for me over in that corner.
Posted by: Mithras | February 16, 2005 at 10:02 PM
Just to clarify my point: My blog is irrelevant to most people's interests (it gets something like 50 visits a day, and I'm guessing a good number of those are robots). So, my point was just that if Google were perfect, my blog wouldn't be coming up #1 on any but extremely obscure searches.
If you prefer a less hoity-toity-sounding example, my blog is also highly ranked for the search "girl child bbs."
Posted by: david | February 16, 2005 at 10:04 PM
I get your point, David; we're just teasing. I guess the issue is not whether or not google results are "perfect" (whatever that means in this context), but that google engineers have apparently made some sort of judgment about blogs which affect their place in the culture.
Posted by: Mithras | February 16, 2005 at 10:12 PM
I don't think I even want to KNOW what "girl child bbs" is, let alone why it gets links to your blog, David. Is it, like, the philosophy of child porn or something?
Mithras, join me in the corner!
Posted by: Ruth | February 16, 2005 at 10:13 PM
I think part of the problem may be the penchant of many bloggers for charming and fanciful names. "Fables of the Reconstruction" is a lovely blog name. It was also a major album that sold a lot of copies, an article in the Village Voice and other newsish places, and a few other things. How high ought the blog of that name rank in an exact search? I dunno, honestly.
Posted by: Robert | February 17, 2005 at 01:08 AM
That's a good point, Robert. Most people searching for "Fables of the reconstruction" alone are not looking for this blog. But how about "'Fables of the reconstruction' Mithras"? I'm 113th in that search.
Posted by: Mithras | February 17, 2005 at 01:23 AM
This is disturbing:
The conservative blog Powerline is still #1 for its google search: http://www.google.com/search?hl=en&lr=&safe=off&c2coff=1&q=powerline
Could google be removing only certain blogs on a case-by-case basis? I'm not saying that they are; I'm saying I don't understand the technical details and am genuinely wondering.
Posted by: C | February 17, 2005 at 07:26 PM
And also:
http://www.google.com/search?hl=en&q=instapundit
http://www.google.com/search?hl=en&lr=&safe=off&c2coff=1&q=kim+du+toit
http://www.google.com/search?hl=en&lr=&safe=off&c2coff=1&q=volokh
http://www.google.com/search?hl=en&q=lileks
http://www.google.com/search?hl=en&q=free+republic
But, this soothes my liberal suspicions a bit:
http://www.google.com/search?hl=en&q=daily+kos
http://www.google.com/search?hl=en&q=boing+boing
Sorry these aren't hotlinks - I tried doing that but I can't get it to work.
Posted by: C | February 17, 2005 at 07:35 PM
Links to this blog and comments I have left on other blogs rank higher than the blog itself.
Maybe they have some kind of whine detection algorithm going on?
Sorry, couldn't resist, does it *really* matter so much? Surely if you wish to write, and write interesting stuff, you will be found and picked up on and followed by people who matter to *you* (if you tell them about the blog that is). Does being #1 on google for ' Fables of the Reconstruction + Mithras' really mean anything at all? Does anyone apart from you ever type in such an obtuse search?
Posted by: K | February 17, 2005 at 09:32 PM
K - Obtuse? Now, that hurts.
Anyway, climb down. I think you're missing the point of the post.
C - All of those have their own domains. It might as simple as known blog places like LJ, Typepad and Blogger were downgraded en masse.
Posted by: Mithras | February 17, 2005 at 09:39 PM
Hmmmm, this issue didn't seem to effect my blog's ranking at all...
Googling "Cursed by a Classical Education" still comes up with me as numero uno!
A glitch perhaps?
See you on the high ground!
MajorDad1984
Posted by: MajorDad1984 | February 17, 2005 at 11:51 PM
Hmmmm, this issue didn't seem to effect my blog's ranking at all...
Seems not to affect those with their own domain.
Posted by: Mithras | February 18, 2005 at 12:59 AM
just somehow get your blog as part of a por site and it will be back on tp
obviously i know nothing of this technology thing you call computers
Posted by: violent femme | February 18, 2005 at 01:15 AM
It doesn't affect you if you have your own domain?
Of course, registering a domain costs money. Therefore, you can buy your way up Google's searches, but all us poor folk better hush up and stay obscure.
My, how wholesome and, um, democratic.
Posted by: akeeyu | February 18, 2005 at 12:44 PM
Looks like all you guys are mystified about how this happened. Must be you missed the big controversy over the new no-follow tag developed by Google and implemented by all the major search engines and blogging hosts.
This was presented by Google as an effort to fight comment spam. It was soon demonstrated, however, that if it did succeed in lessening spam, this would be the least of its effects; its primary effect would be to lower the pagerank of blogs, and change the nature of the blogosphere, from that of a cooperative effort, to one that is more atomized and less influential. Clearly, the effects of the new tag are already being felt. Hence your discussion here.
Here's a bit from Danny Sullivan at SEW: Link
and, a further sample of the no-follow controversy: Link.
Posted by: Doug Thacker | February 18, 2005 at 01:03 PM
Doug - I am under the weather today, so I'm not going to read your links for the time being. But in the interim, riddle me this: if the problem is nofollow tags, then why were MT blogs with their own domains not affected? If nofollow is the culprit, then shouldn't we expect to see the same effect on those blogs with their own domains?
akeeyu - don't let's jump the gun before we know what happened and why.
Posted by: Mithras | February 18, 2005 at 01:37 PM
Mithras, to answer your question to Doug:
Yes.
Over time this will probably happen. This is the beginning of the unintended circumstance of the 'no-follow' tag. (Actually, it's not a tag, it's an attribute.) Expect Yahoo! and MSN results to follow the same decline in ranking blogs.
I predicted this in my podcast on Jan 21st:
http://www.grandforkscity.com/miller-report/audio/dave-winer-shoots-kills-blogs.mp3
I'm going to do a follow-up podcast on this topic this weekend and explain it better. I will be using this blog to answer some of the questions and offer my insight on this. It will be available at podcastingworld.
Posted by: Scott | February 18, 2005 at 05:37 PM
I don't know if I buy that idea that it's only blogs within the service domains. I have my own domain name, and I saw a noticeable and abrupt drop in traffic, and a distinct decline in google searches.
Posted by: PZ Myers | February 18, 2005 at 07:51 PM
PZ - Damn, I thought I had figured something out. There must be some other factor I am missing.
Posted by: Mithras | February 18, 2005 at 10:08 PM
Traffic for me has been up slightly and the page rankings are consistently high. Type in City Record (no quotes needed) and I'm 12th, add boston to the search and I end up number 1. I don't have my own domain--the url is bostonhistory.typepad.com I do get a lot of misguided Google searches--so many on certain subjects I just went ahead and wrote entries to give people the information they're looking for.
Posted by: Charles | February 18, 2005 at 10:40 PM
I also have my own domain and have dropped significantly -- I used to be the number one hit for Paul Goyette and now I'm not in the first 100.
I don't think this can be the result of just the nofollow tag -- it has to be a change on Google's end. I've added the nofollow tag to my comments section, but not to other sections of my blog -- so it should only reduce the Google rank for links out of my comments section, right? And the same shopuld be true elsewhere...
Posted by: paul | February 19, 2005 at 07:03 PM