Google’s Existential Crisis
Google’s success was founded on the execution[^1] of two innovations: 1. Leveraging the hyperlink to create the best index (i.e. crowd-sourcing); and 2. Targeting advertising to search results to monetize that index. Apparent diversification into new areas like content (YouTube) and operating systems (Android and Chrome OS) can be understood to exist in service to the two founding innovations. These driving principles are Google’s DNA, and as successful as the company has been, if there are fundamental problems with these two principles, it could mean trouble for the company, and trouble for the web at large, to the extent that Google is good for it.
The Algorithm
The search engine’s ranking algorithm, PageRank, measures ‘relevancy.’ It began with the simple insight that the more links there are to a particular page, the more important it is, but the algorithm very quickly became increasingly sophisticated, and it now considers “more than 500 million variables and 2 billion terms.”
Google’s relationship with the web is symbiotic. The web became useable in no small part because Google made it possible to find relevant things. As the web became more important, so did Google. And as Google became more important, it also became a bigger target. Search engine optimization became a prominent and diverse industry that ranges from qualified professionals exhorting legitimate and sanctioned best practices to cargo cultists offering opaque rituals to increase search result rankings. Orthogonally, companies like Demand Media began to mine real-time search data and hire cheap labor to create abundant shallow content, exploiting holes in the web to make money on ads. Recently, it even seemed like such ‘content farms’ could rise to become an existential threat to Google by degrading search results into meaninglessness.
This shows Google’s symbiosis with the web to be profoundly weird in a Heisenbergian way: Google cannot observe the web without changing it. Search engine optimizers and content farms both respond to Google’s algorithm. SEO changes the way publishers create and present data, as well as what content they produce. Content farms form a more directly parasitic relationship, because as search traffic data exposes demand for content, they produce it. But as Google’s relevancy algorithm increases the measure for quality, the quality of content changes to fit those criteria. This puts Google in the strange position of being able to invoke content by defining what is of high quality. Which presents a problem analogous to the one of standardized testing: it shifts incentives to ‘teach to the test’ instead of to teach what’s important. The simple solution is that if you can make a test good enough, the incentives will align properly.[^2]
As the test-maker, Google becomes in no small way the arbiter of truth. The tension that this forms between Google and the content farm industry describes a sort of Turing arms race: the growing ability of content farms to cheaply produce content in an increasingly automatic fashion pushes against the escalating precision of Google’s quality algorithm. The inevitable conclusion of this process is automatically generated, demand-invoked content that’s indistinguishable or better than anything people can provide.
Not only that, but in December 2009, Google began to personalize search results according to users’ locations, search histories, social graphs, etc. As the search profile becomes more extensive, the personal search algorithm becomes more intensely personal, and an ‘objective,’ One True Search ceases to exist.
As I’ve discussed before, Google’s incentives are such that it wants to increase use of the web, broadly. This is good, as long as the web is good, however you define that. But as its power to create the web (or at least influence it) waxes, so does its responsibility. It’s not hard to imagine a dystopian future where the entire web devolves into a broad, shallowly addictive ad-ridden hell (this already describes some sectors of the web). Google would have to have an extremely long-term outlook to decide to serve as superego instead of as id to the web. But maybe that’s not so hard to imagine either.
Ads
Google famously makes the vast majority of its money on advertising, specifically with its AdWords program, which auctions keywords to advertisers. The key to the program’s success is its highly targeted nature, i.e. the relevance of the advertisements.[^3] The weird tension here is that the quality of the search results is in conflict with the quality of search ads. If the search results are perfect, and exactly what the user wants every time, then there’s no need for her to ever bother with advertisements. Conversely, if the product being advertised is maximally relevant, then it should be her first search result anyway. Advertisements only offer value to a user if their relevance has parity with search’s. But they only offer value to an advertiser if they are exposed where they wouldn’t be organically, i.e. where they are irrelevant, or distracting.
The other tension is one that stems from something I briefly touched on in another post: Google wants to encourage quality in the web and provide a good user experience so that people continue to use the web and its services. Simultaneously, Google wants to extract the most value from the user by exposing her to the most ads, i.e. distracting her. Minimally distracting ads would be nonexistent ones and maximally useful services would be ad-free. Conversely, maximally value-extracting services would be all ads all the time. The two incentives of quality and distraction are directly opposed. The result of this cognitive dissonance is what we see when we use Google products: generally tasteful, useful, and understated ads.
So when Google places relevant ads, advertisers are in some sense paying Google simply to do its job. But when Google places irrelevant ads, it undermines the quality of the web, its search results, and its user experience. And maybe the scariest part of this to me is that the quality incentive is one conceptual remove farther than the distraction one.
Maybe it’s no more perverse than any company’s tension between making the highest quality product in order to invoke desire and spending the least amount of money possible in its production. And maybe Google is culturally constituted to thinking in uniquely long-term ways that ultimately make everyone better off. The recent battle with SEO-driven content farms that resulted in the ‘Panda’ update to the search algorithm would seem to indicate so.[^4] But I imagine it’s easy to think long-term when you’re succeeding. We’ll see what happens when the success inevitably slows.
Notes
[^1]: Google’s ranking algorithm was based on earlier ideas: citation analysis and Hyperlink-Induced Topic Search (HITS).
[^2]: I’m not competent to speak to whether that’s possible in either case.
[^3]: The auction part, which helps Google maximize its profits, is also really important, but I would say secondarily so.
[^4]: More cynically, you could call it a response to growing criticism about search quality.