Analyzing Search Engine URL Patterns

This week I’ve been comparing the URLs generated by a multitude of different search engines and finding patterns within them. 

We have some custom code that logs information about visits to any particular website on which we install the code.  You may be interested in studying this data for several reasons:

1. See how much traffic your site has accumulated and from where (i.e. natural search engine, pay per click, etc.)

2. See what keyword terms users are typing in to reach your site

3. See where users drop off on your site.

4. Gain a better understanding of what those weird characters mean in your URL.

You may argue that simply installing Google Analytics will report items 1-3 that I mentioned above to you in a nice and neatly laid out report.  However, you may want to see the originating data itself for proof, rather than trust what the GA report tells you.  I’m not saying that GA is inaccurate data, but I am telling you that it is interesting to examine the raw data yourself.  Look at the patterns below from each search engine.  The text that is in bold point out important aspects of the search query. 

• For instance Google follows the pattern %://%.google.%/%?q=% or %://%.google.%/%&q=%.  The search terms are delimited by plus signs. 
• For a Google Pay Per Click, the pattern is the same except it would contain the &gclid= parameter.  This gclid is an ID that is automatically generated by Google Analytics.  Another popular variation for a Google search is %://%.google.%/search?q=%.
• For Yahoo, it generally follows the pattern %://%.yahoo.com/%?p%.  There are of course many variations of it as well, such as %://%.yahoo.com/%&p% and %://%.yahoo.com/%?k%.
  
• Yahoo Pay Per Click is similar but it contains the term OVRAW= or source=overture.
  
• Bing typcially follows the pattern %://%.bing.com/%?q=% and it’s search terms are delimited by plus signs.
• AOL typically follows the pattern %://%.aol.com/%?q=%

Google

• HTTP_REFERER=http://www.google.com/search?q=sql+server+books+online&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a;REQUESTED_URL=/index.aspx

Google Pay Per Click

• HTTP_REFERER=http://www.google.com/search?hl=en&source=hp&q=sql+server+books+online&aq=1&aqi=g10&aql=&oq=consumer+credit+&gs_rfai=;REQUESTED_URL=/Default.asp?referrer=google&cccs&gclid=CLvWld294aECFV195Qod51BVJQ

Yahoo

• HTTP_REFERER=http://search.yahoo.com/search?fr=yfp-t-701-s&toggle=1&cop=mss&ei=UTF8&rd=r2&p=sql%20server%20books%20online;REQUESTED_URL=/index.asp

Yahoo Pay Per Click

• HTTP_REFERER=http://search.yahoo.com/search;_ylt=A0geu7_.3j1LmkEBrBVXNyoA?p=sql+server+books+online&fr2=sb-bot&fr=slv8-;REQUESTED_URL=/index.aspx?source=overture&OVRAW=sql%20books&OVKEY=sql%20books&OVMTC=advanced&OVADID=1112587012&OVKWID=8031160012

Bing

• HTTP_REFERER=http://www.bing.com/search?q=sql+server+books+online&form=DLCDF8&pc=MDDC&src=IE-SearchBox;REQUESTED_URL=/index.aspx

AOL

• HTTP_REFERER=http://search.aol.com/aol/search?s_it=wscreen-searchboxhtml&q=www.sqlserverbooksonline.org;REQUESTED_URL=/get_started/get_started.aspx

Of course, as I mentioned above, there are variations of the general patterns I listed.  However, I have seen that over the course of 5 years, the pattern change only very slightly if at all!