Google API - Yahoo API - MSN API

Scanning or 'reading' the result listings of the search engines via a program or bot can be done via different methods. This document will discuss the two most common ways to scan the listing of a search engine.

Method 1 : API

The major search engines offer a 'search API' for developers. This API consists out of a set of functions and objects so that you can pull specific data from the database of the search engine. The results are often returned in xml-format. This method allows developers to pull search results for keyphrases at very high speed. Connecting through API requires identification ; some search engines even monitor traffic per API-key and limit the number of queries.

Method 2 : Get contents

Another method is to execute a query on a search engine via http-protocol, load the contents of each resultpage and filter out the listed url's. This method is slower because the page needs to be loaded entirely every time you execute a search-query and so, you receive much more data then is actually needed. This is the method used by search engines to index a website (they read the contents of each individual page an filter out what they are looking for). As the result pages are loaded in the way a normal user would do, the results are identical to what a normal user sees.

API versus Get contents

Many organisations did a research on the differences between the actual result listings (what end-users see) of the search engines and their API-method. The conclusion of these researches are that all search engines have differences between their real listings and the results returned via their API.

More information on API

What method uses SERP Scanner ?

SERP Scanner is based on the 'get contents' method as these are identical to the real results of the search engines. We are monitoring resultlists with both methods, in order to determine the difference-ratio between both.