Search Engine Traffic Reports

The Search Engine Traffic reports have been developed by Chirp over the last year with a view to assisting our clients with Search Engine Optimisation (SEO). This is the practice of adding new content or making changes to existing content in order to improve the website's performance with relation search engines. It can also involve building a network of inbound links from authoritative websites, but that aspect is now less important than it was just one or two years ago.

So what are these reports, where do you find them and how can you make use of the information?

/pagestats/

The reports for your website can be found by going to your Site Administration page and clicking on Search Engine Traffic. You can also access them by adding '/pagestats/' to your homepage URL (i.e. www.example.org/pagestats/). They are designed to complement the ubiquitous Webalizer visitor traffic reports which simply aggregate the information recorded in server log files.

searches

The first screen (or page) of the Search Engine Traffic report contains some important information about your website with relation to search engines (Google, Yahoo!, MSN, Live Search, etc,).

On the left hand side you're presented with a list of URLs from your website with a number next to them indicating how many times they've been 'found' using a search engine in the last 30 days. You can click on a URL in the list to view specific details on that page (more on this later).

On the right hand side you should see two boxes. The first titled Most popular words shows the most common one- and two-word combinations entered into search engines that resulted in someone clicking through to your website. You can click on them to get more information on the exact phrases used and which pages they directed people to (see "referers" below).

The second box titled Search engines shows you which search engines have been used to find your website. The percentage indicates the proportion of searches that came through each search engine. Typically "google" will appear at the top of this list as it's currently the most popular search engine. In this case "google" refers to all the google website from google.com to google.co.za.

You can view a graph of this by clicking tje "show graph" link;. You can also click on the search engine names to view a table of all searches that found your website made using that search engine.

Returning to the list on the left hand side, clicking on a URL in the list allows you to examine the traffic to a specific page. On this screen you're presented with a list of the search phrases that have found this particular page.

Again on the right you have a number of boxes.

The first shows the most popular one- and two-word combinations that have been used to find this page.

The second box titled 'Last indexed' shows you when each of the major search engines last 'read' the page. The names can be a bit confusing as they are the names of the 'spiders' (automated web crawling applications) that work for each of the search engines in creating and updating their index. The main spiders you should be aware of are: Googlebot (Google); Slurp (Yahoo!); and msnbot (MSN or Live Search).

The third box again shows which search engines have been used to find the page in question and in what proportion, and you can view this in graphical form.

referers

At the top-right of the page you should seen links to "searches", "referers" and "spiders". The "searches" link covers the pages we've discussed already, so let's look at "referers".

A "referral" is when someone clicks on a link from another website that brings them to your website. In this case "referrers" is deliberately misspelt in order to conform to the official specifications.

On this page you can either select a date from the drop-down list or enter a key word or phrase. In each case you'll be presented with a table showing you the specific searches that found your website, which search engine was used, and which page of the website they were directed to. This information can be useful in tracking the response to your publicity campaigns (a radio spot for example) or to check whether the words that you want to find your website are actually working, and in which search engines.

spiders

Finally the "spiders" link. As mentioned earlier, these are automated 'agents' used by search engines to read your website and add it to the search engine index. They're also referred to sometimes as 'robots'.

The first page you see in this section displays a graph showing how many pages each recognised spider (we limit this to the top dozen or so known agents) has visited in the last 30 days. The totals for the major search engines should all be about the same, though in some cases there can be reasons for them to diverge. If you see an agent "htdig" then that indicates that our server is indexing for the 'site search' feature on your website.

On the left hand side of this page you can see a list of spiders and an indication of how recently they have indexed different pages of the website. A website that is doing well with search engines could expect to have 80-90% of content re-indexed each week - at least by Googlebot. The other spiders can be less regular.

This is important because when you change the content of your homepage for example, it takes some time for the search engine results pages (SERPs) to be updated. First you have to wait for their spiders to read the page and then you have to wait for the actual index to be updated which should be very quick but sometimes takes as long as a couple of weeks for smaller websites. One of the goals of Search Engine Optimisation is to reduce this waiting period.

Search Engine Optimistion (SEO)

Firstly, we should be clear on what the Search Engine Traffic reports are actually measuring. There are a number of constraints:

• data on search engines and search traffic is currently only kept for 30 days;
• the system only records cases where someone has clicked directly on a link on a search engine results page (SERP) and not when they copy and paste the address into their browser;
• no information is available as to where in the results list your website appeared, or whether the user clicked through to other websites before coming to yours; and
• we can only record this event when the target is an HTML page or a file that is delivered through a script (not links directly to PDF and Word files).

Despite these limitations you can still extract a lot of useful information on how the search engines view your website and monitor trends over the longer term. By looking at the same time at the Visitor Statistics report you can get an idea of exactly how important the search engines are to your website compared to other means such as inbound links and advertising.

The first thing you should check is how many pages are being indexed and how regularly. This gives you an indication of how long it will take for a change you make to be reflected in the search results and, eventually, in your traffic reports. Then you should look at which pages are the most popular and try to work out why. Is the reason something you can build upon, or is it showing that the search engines have the wrong idea about your website's content? In other words, are things moving in the right direction and just need a push, or do we need to make some changes to influence the search engines?

Depending on the answer to these questions you may want to look at the structure and content of your website. Does each page of your website have a unique TITLE (the TITLE is displayed at the top of the browser window) and do most or all pages have a unique META description tag (you need to view the Page Source to see this). We can help answer these questions if it's getting too technical.

Overall what you'd like to see is people finding your website and finding what they're looking for. There's no point having hundreds of visitors coming to your website if all they look at is a page that's not related to your business. To that end it's important to monitor all activity on your website to see whether it's working as expected.

This is not always a simple task and the terminology can be confusing for beginners. If you have any questions about these reports or about your website don't hesitate to contact us.