If you have been on the internet for a while or are geeky (both apply to me!) , then you probably know how search engines work (and should rather be out enjoying the sun than reading this post). And yet , many of the small business owners who are looking to take their businesses online , don’t have a real understanding of how the various search engines actually work.
From my conversations with small business owners and others who are looking to bring their businesses online , or give their businesses more online exposure, they need to understand how the search engines actually work before they can even start contemplating highly technical things like Search Engine Optimisations.
Hopefully this blog can help as a primer to help those people understand how search engines work.
What kind of search engines are there:
Search engines generally fall into three categories:
- Human-edited directories : These search engines are either entirely constructed by hand (Yahoo was like this in the beginning) or they are constructed via a Web crawl or or the site owner submits and categorizes them. They are then ranked by hand. Some examples are the Open Directory Project (dmoz.org), about.com and Mahalo
- Automated search engines : These search engines are built by a computer programmed algorithm via web crawlers and the algorithm catalogs and ranks the sites as well. Sites like Google and Bing are automated search engines and are the most used by searchers today. You usually want your site/blog to perform well on these search engines.
- Meta search engines (aggregators) : These search engines generally take the results of other search engines and present them in a different way either by presenting the results from multiple engines together. Or they present the other search engines results with a different look.
How search engines work:
The search engines that account for the majority market share today are the automated types (Google, Yahoo! and Bing). Google and Bing (and other engines in your region of the world) have similar overall infrastructure with regards to how they work:
- Web crawlers crawl the web. These crawlers follow links to discover pages (posts you write etc) on the web.
- The search engines use a extraction process to gather information from the pages that they crawl (such as text content, links etc.)
- The content from the crawled web pages are stored in Indices. The search engines then store the content using word-based keys. This is very similar to the index in a book. When you look at a word in the index of your book, you learn the page number(s) that the word is on. In the same way, the search engine can look up a word that someone is searching for and find out all the web pages associated with that word. (When you search for the phrase “how search engines work” the search engines will find this post and list it in the search results)
- The search engines then score the results to determine what pages are most relevant for each search.
So basically, when someone does a search (which the engines call a query) and the engine then checks the index for all the web pages associated with that search, the engine needs a way to rank the pages it finds in a order that is useful to the searcher.
Google and Bing amongst others use a number of factors to help them score the results , and these factors are adjusted all of the time based on new algorithms, tests and other criteria that the companies develop. Google and Microsoft keep the details of these scoring factors secret.
Finally once the search engine compiles and ranks the pages that are relevant for your search query, they display them in a list , that is known as a organic search result.