Getting a Site Indexed

Something I have learned the past few years while talking to friends and clients regarding new or existing web sites is the lack of understanding of how search engines index your site’s content and how that content shows up in search results. Of course that is the basis of the Search Engine Optimization (SEO) industry, of which I am not a big fan (that explanation is for a later post). A recent post by Matt Cutts of Google on the changes in the frequency of index updates is very interesting and displays how far the industry has come in just the last 7 years.

Matt states that in 2000 when he joined Google there was a 3-4 month period where they did not update their index at all and another search engine went for over a year without updating their index (perhaps one of the casualties of the search engine shake up). This would mean that no new sites or new content from existing sites would show up in searches until the index update. It was mid-2000 when Google started regular monthly index updates, driving the search engine industry to provide accurate and fresh results for searchers.

Since then Google has been improving their index updates to the point where things can appear in the index only minutes after being posted. Of course, other search engines have had to follow suit. This is where I appreciate Google’s focus on their search customers (although content owners love fresh results as well).

Changes in technology have helped Google reach these new levels of freshness. Instead of Google spiders having to crawl each site daily (which is impossible for them to do when they are indexing billions of sites) sites can ping Google when they have updates. This is possible now with the rise of RSS and sitemaps. Sites do not have to be a traditional blog to utilize these techniques either.

In the past, clients would bring new projects and expect a site to be created and launched in two months, as well as indexed by all the major search engines with a high result on key search terms on launch day. When I explained that sites had to be submitted for crawling by the search engines, and then there was a waiting period before they would be added to the index and available in search results, for a total wait time of 4-6 months, it often opened up their eyes to the search engine industry. Many wanted to pay to be included in the index and listed as the #1 result but after some explanation, they would understand the reality of the web. As all were small organizations or individuals, I stressed the importance of focusing on their content and doing what they could with the search engines but not obsessing over their initial rankings. Some dropped their site project with this news; others went forward and discovered their wait for a crawl and to show up in the index was not as detrimental as they thought it would be. Now it seems very easy to set up a site with feeds and sitemap pinging capabilities and you can be discovered and indexed in days or hours. Then you can immediately work on building content and incoming links from valuable resources (not link exchanges) to increase your visibility.

Some more good general advice is provided in a Google Webmaster Central post on getting indexed (English at bottom) for the Portuguese market, but it is relevant to every site. The top two points are critical – Be a subject authority (write good content people are looking for) and keep the search engines informed of your site updates, which are hopefully frequent. If you are not checking off these two points, then all the other optimization will do little to gain and maintain visitors, no matter how high you get your site to rank.