Column

Findability: SEO Insight

RC_How

How Search Engines Work

by: | July 11, 2011

Before beginning to optimize the pages of your website, it’s important to understand what search engines are looking for and how they decide where your pages rank for a given search. Armed with this basic information, you can make better, more informed choices. And, as I noted in the first SEO Basics post, being an informed consumer is always a good idea!

A (Very) Brief Explanation of Search Engines
What Is a Search Engine? (an article on bbc.co.uk) says:

“A search engine makes it possible to find a specific bit of information in the huge mass of data stored on the web.”

There are literally hundreds of search engines, but most of us are primarily familiar with the three largest in the U.S. – Google, Bing and Yahoo. Many of the smaller engines operate in niches or offer specific functionality. Blekko, for instance, is a newcomer that claims to ‘slash out’ spam and malware by including only quality sites. DuckDuckGo is a search engine that pledges not to track your search activity. In other countries, there are options that are certainly not household names here, as Search Engine Land shows in its 10 Alternative Search Engines to Look at Internationally post.

Google is by far the most popular search engine in the United States. Bing (which also powers Yahoo search) is a pretty distant second, though it has recently gained a little in popularity.

The Complicated Business of Retrieving Information Online

The BBC’s simple definition of a search engine belies the inherent complexity behind the process of retrieving information online. Think about the issues involved with first finding all the content that’s on the Internet. In 2008, Google announced that it had identified over 1 trillion unique URLs on the web, and that number is undoubtedly much higher now. The complexity of not only finding all those pages, but also ‘reading’ them and understanding the content, is daunting.

Search engines like Google and Bing – known as crawler-based engines – generally operate in a similar way.

  • Spiders (also known as bots or crawlers) find the content on the web by following links or by being directed to a page in some other way.
  • That information is then indexed in a database to allow for future retrieval when someone does a search.
  • When someone goes to the search engine and types a query into the search box, an algorithm attempts to identify the most relevant results from the index.

What Criteria Do Search Engines Use In Deciding Rankings?
Each search engine has its own unique criteria, but in general terms  they’re looking to see if there’s a relationship between your page and a searcher’s query. If it’s confident that your page is the best alternative for a search, then your site will show up near the top of the search results.

But, out of all the pages in their indexes, how do search engines know what’s the ‘best alternative’ for a given search? They use a complex mathematical formula called an algorithm, which contains many different factors. Google has publicly said that there are over 200 different variables in its algorithm. Each variable is weighted by importance in its ability to help the search engine deliver the best possible results for every search.

Search engines don’t share the details of their algorithms (that’s their ‘secret sauce’), though they do provide some general guidance. Google offers a PDF called search engine optimization starter guide, and Matt Cutts, the head of Google’s webspam team, frequently answers SEO-related questions on his blog. Likewise, Bing has a post on its blog called Search Engine Optimization for Bing.

Through research, testing and experimentation, search engine optimizers have developed ideas about other factors that the search engines haven’t explained publicly. However, the sheer complexity of hundreds of factors (which themselves may be complicated formulas), and their variable weights makes it difficult to know for certain how much – or even if – any factor is truly relevant. Further complications arise because the search engines make frequent changes; Google, for instance, averages at least one change per day to the algorithm.

You can see why SEO is not a cut-and-dried science.Get In Media

In the next post in this series, I’ll examine some of the factors that almost certainly play a role in rankings, starting with on-page variables.

Posted in: Content, Findability: SEO Insight, Lead Generation, Optimization, Paid Search (PPC)

About the Writer:

Rob Croll is Program Director for the Internet Marketing Bachelor and Master degrees at Full Sail University. He also owns a consulting firm that works with small businesses and nonprofits. Rob has been included on Social Media Marketing Magazine's list of Top 100 Marketing Professors on Twitter and has had numerous articles published both online and in print. He has a BS in Finance and an MFA in Media Design. Follow Rob on Twitter @rcroll.

View Full Bio »

More Columns