feddy.ca

Google introduced two improvements to the Google Webmaster Console this week. First, Top Search Queries now has historical data and other improvements. In addition to data going back up to six months, the data is updated more frequently, sometimes daily. Secondly, sites which display SiteLinks in Google search results can now manage SiteLinks with the new SiteLinks tool in the console.

I am always impressed with the speed of the tools in the Webmaster Console. Adding historical data does not seem to be a problem for Google, even considering the number of sites they must need to aggregate this for. Google’s systems seem more than capable of keeping up with what ever data the Webmaster tools team wants to throw at them to serve to their users.

In a previous post I explained the registration and verification process for Google’s Webmaster tools. Now that Google has updated the Webmaster tools interface, it seems like a good time to examine the features found in the Google Webmaster Console.

In preparing this post, I found I was beat to the punch by Eric Lander’s Comprehensive Guide, so check that guide out for more information on the Webmaster Console.

First, the Google Webmaster Central page is worth noting as a launch page into Google’s Webmaster tools and other information. Webmaster tools, Submit Content, Webmaster Central blog, FAQ, and developer tools links provide a central location for locating Google resources for website owners. From this page you can click the Webmaster Tools link and login to the Webmaster Console.

Authenticated, you will arrive at your Dashboard page which consists of a list of your managed sites, a Message Center, and a list of links to Console Tools.

Dashboard

The Dashboard provides a central summary of the sites you have registered, displaying the site verification and sitemap status. The message center provides a method for Google to communicate with site owners of sites that are verified. This is the preferred method rather than email due to email spoofing occurring in the past. The types of messages that will appear initially in Message Center are for search quality issues and penalty notifications. I suspect (speculation) that malware/badware notifications will be added at a future date. Do not worry about the lack of messages appearing here; that is a good thing.

Also on the Dashboard page is a list of links to tools that hopefully you will not need to use often; reporting spam, reporting paid links and requesting reconsideration over any penalties that may be applied to a domain due to violation of Google’s Webmaster Guidelines.

The primary tasks performed from the Dashboard will be to registering a new site, as explained previously, or manage a previously registered site. To manage a registered site just click the site link to go to the site’s Overview page.

Site Overview

The Site Overview consists of six sections: Overview, Diagnostic, Statistics, Links, Sitemaps and Tools. These sections can be navigated using the left-hand navigation menu.

Overview

The Overview section currently provides the last crawl date by Googlebot and the sites status in the Google index. There are quick links to top search queries and index stats as well as to help center topics that may be of interest.

Diagnostics

The Diagnostics section consists of two sections, Web Crawl and Mobile Web Crawl. These items will display any URLs from your site that Googlebot had issues crawling. Issues such as 404 errors (pages or content not available) and content excluded from indexing by robots.txt will appear here. Checking this information regularly will help you identify and correct problems that may be preventing Google from indexing content on your site.

Statistics

The Statistics section currently provides five detailed reports useful to website owners.

• The Top search queries report displays the search terms used on Google that returned your site and the average position of your site for those terms in the last seven days. This can be useful in analyzing incoming search patterns for your site.
• The Crawl stats report displays the distribution of your site’s pages PageRank and a history of the page with the highest monthly page rank.
• The Subscriber stats report displays the number of subscribers to your site’s feeds as seen by Google products like iGoogle or Google Reader.
• The What Googlebot sees report gives an overview of Phrases, Keywords and Content of your site. This report seems to have replaced the Page analysis report that was available until the recent revamp. The Phrases section lists the phrases found in the anchor text of external links to your site. The Keywords section lists the top keywords found in your site’s content and external links. The Content section displays a breakdown of the content type and encodings of your site’s pages. This information can be useful to administrators of sites, but has little bearing on a website owner’s day-to-day information gathering.
• The Index stats report provides links to the Google Advanced queries to list the pages indexed by Google, incoming external links to your home page, the current cache of your site, the info Google has on your site (site description) and a list of related pages.

Links

The Links section contains two reports: Pages that link to yours and Pages with internal links. The first report lists the sites linking to yours as known by Google. The second report lists the pages and link counts for all the internal links on your site.

Sitemaps

The Sitemaps section allows you to notify Google of a sitemap for your site. A sitemap is a file describing all the pages of your site in a standard format that Google can read. Using a sitemap file allows you to notify Google of changes on your site without having to wait for the next Googlebot crawl of your site.

If you have a sitemap created for your site, you can add it to your site’s sitemap page here and see information as to its status, last download date and number of URLs submitted. Click the Add Sitemap link and follow the instructions on screen.

Tools

Google provides some excellent tools for webmasters to manage their site with Google. From robots.txt analysis to removing content, the tools to manage a site’s visibility and content are found in this section.

All site owners should verify their robots.txt file, especially if they have specific disallow statements. Google provides a tool to report on the status of the Googlebot’s last check of your site’s robots.txt file, parsing results and a tool to test changes to the file and view the results. For more information on the robots.txt syntax, see robotstxt.org. More information can also be found in this post on the Official Google Webmaster Central blog.

The Enhanced Image Search tool allows you to control whether Google uses the Google Image Labeler to associate labels with your images to improve search quality. This may be of interest to sites which host images as a prime feature and expect to get substantial traffic from image searches. You can opt-in and out from this tool’s page; the default setting is opted-out when a site is registered. Currently, I would classify Enhanced Image Search as an experimental feature that would not be of interest to most small site owners.

Manage site verification allows you to check all verified site owners, and re-verify the site if needed. The Set crawl rate tool displays statistics on Googlebot visits to your site over the last 90 days and allows you to change the Googlebot crawl frequency. There are three settings — Slower, Normal and Faster — with Faster only available if Googlebot’s crawl rate becomes a factor in your site’s crawl frequency.

The Set preferred domain tool allows you to tell Google which domain you wish to use in their index if you use domain addresses with and without the www prefix. All sites should use a redirect on one of these addresses to your primary domain name so there is no ambiguousness with traffic arriving at your site. This is done at the site level which will affect all search engines and visitors to your site. Read the Redirect section of this post for more information.

The Remove URLs tool allows you to remove pages on your site from Google’s index. Follow the instructions on the page and submit a page for removal. The page will be removed with Googlebot’s next crawl. More information can be found in this post about removing content from Google’s index.

Conclusion

Claiming your site on Google is a quick and easy way to establish your ownership of a site and have management tools available to you at your disposal. You may use the tools infrequently but as Google continues to expand the service the Webmaster Console’s usefulness will only increase. Most uses of the tools are within the reach of the average individual site owner, however, a website administrator can help with advice and use of the more advanced tools.

In Improve snippets with a meta description makeover on the Google Webmaster Central blog, an explanation of how Google provides page description information and uses meta descriptions provides valuable information in controlling the appearance of your site’s pages in Google search results. It is encouraging to see this thorough explanation as I have been confused as to the benefit of meta descriptions and the priority one should give to customizing them for each page.

The meta description is one place Google (or any search engine) can get your page snippet — the short text preview Google displays for each web result — from. This is preferable as it gives you control of what is displayed as the page description for search results rather than an extract from your page or other sources.

To summarize, it seems to me the key points for meta descriptions are:

• Quality meta descriptions without keyword stuffing are more likely to be used as a page description.
• Provide a different description for each page. Use site level descriptions for the main page and other aggregation pages and page specific descriptions for other pages.
• Descriptions should be human readable. Use a sentence structure when appropriate, or clearly tagged and separated information for permalink pages such as blog post or product pages.
• While not addressed in the Google article, it is touched on in the comments. Keep your descriptions short so they are not cut off. It appears descriptions are limited to between 140 and 160 characters. Keep the important information at the beginning of the description if you must exceed the character limit (approximately two lines in Google’s search results). Tweak your descriptions when you see them truncated on Google’s search results to provide the best results.

Considering this information, I need to look at an automated way to insert appropriate meta descriptions on every page. My previous standard was a site level meta description for every page, or just the home page. If I cannot find a satisfactory plugin myself I will add that to the list of development tasks I have. Once developed, this is something I can easily provide to clients to maximize the appropriateness of their meta descriptions in an automated way. This is the type of value-add I like to offer my clients. If only I had more time to focus on these tasks.

All website owners should register with Google Webmaster Tools. The benefits of registering your site(s) with Google’s webmaster console are simple but valuable to any site owner. You will receive information on Google’s view of your website — including configuration problems you may otherwise not know about, information on how your site is performing, set index and crawl settings and you will receive communication from Google on search quality issues. In addition, more features are added regularly. Google is the current leader in providing access to tools and information for site owners in this way, but any other search engine’s or aggregator’s tools (Bloglines offers Publisher Tools to manage your site’s feed characteristics as they appear on Bloglines) should be utilized as well.

Registering

To participate you require a Google account. If you already have a GMail account or an account with Google for one of their other services, simply sign on at the Google Webmaster tools page. If not, use the Create Google Account link on the page to setup a Google account.

Add a Site

Once you sign in, you need to add a site to your account. You can add more than one site, so if you have multiple sites you can manage them all under a single Google account.

Google does not allow anyone to add a site to his or her account. You must claim the site and prove ownership by being able to modify the site in a manner Google provides. To get started, enter the URL (http://www.mysite.com/) of your site in the text field and click the Add Site button. The site will be added but not verified, which means very limited information on the site will be provided until verification is complete. To verify the site, click the “Verify your site” link.

Verify a Site

On the verification page, you are presented with two choices to verify the site: Add a Meta tag or Upload a HTML file. I find for most users uploading a file is the easier method. After selecting an option, the steps required for verification are displayed. For the HTML file method, copy the filename and create a blank or dummy text file on your local PC. Then using your upload method, copy the file to the root of your web site so the file can be accessed with the URL found in the instructions. If you are using the Meta tag method, edit your site templates or files to insert the Meta tag required. Both the file and the Meta tag must stay in place to maintain verification. This allows a domain to change hands and the new owner to revoke verification from the old owner by removing the file or Meta tag.

To complete the verification process, click Verify. If successful, you will return to the site dashboard page with a successful verification message. If there is a problem, a message describing the problem will appear. It may ask that you wait for the verification process to complete due to connectivity problems with Google’s verification agent. Otherwise, check the file name and Meta tag are correct and accessible anonymously from your browser.

Once you have verified the site you can explore the tools and statistics available on your site. There is a lot of information that is very useful, including query stats that display your search query result position for recent search queries. I will post more on using the Google Webmaster console in a future post.

Something I have learned the past few years while talking to friends and clients regarding new or existing web sites is the lack of understanding of how search engines index your site’s content and how that content shows up in search results. Of course that is the basis of the Search Engine Optimization (SEO) industry, of which I am not a big fan (that explanation is for a later post). A recent post by Matt Cutts of Google on the changes in the frequency of index updates is very interesting and displays how far the industry has come in just the last 7 years.

Matt states that in 2000 when he joined Google there was a 3-4 month period where they did not update their index at all and another search engine went for over a year without updating their index (perhaps one of the casualties of the search engine shake up). This would mean that no new sites or new content from existing sites would show up in searches until the index update. It was mid-2000 when Google started regular monthly index updates, driving the search engine industry to provide accurate and fresh results for searchers.

Since then Google has been improving their index updates to the point where things can appear in the index only minutes after being posted. Of course, other search engines have had to follow suit. This is where I appreciate Google’s focus on their search customers (although content owners love fresh results as well).

Changes in technology have helped Google reach these new levels of freshness. Instead of Google spiders having to crawl each site daily (which is impossible for them to do when they are indexing billions of sites) sites can ping Google when they have updates. This is possible now with the rise of RSS and sitemaps. Sites do not have to be a traditional blog to utilize these techniques either.

In the past, clients would bring new projects and expect a site to be created and launched in two months, as well as indexed by all the major search engines with a high result on key search terms on launch day. When I explained that sites had to be submitted for crawling by the search engines, and then there was a waiting period before they would be added to the index and available in search results, for a total wait time of 4-6 months, it often opened up their eyes to the search engine industry. Many wanted to pay to be included in the index and listed as the #1 result but after some explanation, they would understand the reality of the web. As all were small organizations or individuals, I stressed the importance of focusing on their content and doing what they could with the search engines but not obsessing over their initial rankings. Some dropped their site project with this news; others went forward and discovered their wait for a crawl and to show up in the index was not as detrimental as they thought it would be. Now it seems very easy to set up a site with feeds and sitemap pinging capabilities and you can be discovered and indexed in days or hours. Then you can immediately work on building content and incoming links from valuable resources (not link exchanges) to increase your visibility.

Some more good general advice is provided in a Google Webmaster Central post on getting indexed (English at bottom) for the Portuguese market, but it is relevant to every site. The top two points are critical – Be a subject authority (write good content people are looking for) and keep the search engines informed of your site updates, which are hopefully frequent. If you are not checking off these two points, then all the other optimization will do little to gain and maintain visitors, no matter how high you get your site to rank.