Crawling Dead Ends

Search engine dead ends are something you have to remind site owners about, as they do not think about the web in terms of how a search engine navigates a site, just how people do, particularly themselves. If they can navigate to said page, then everybody can is their thinking.

Google recently announced they are experimenting crawling through web forms on a small, select group of high quality web sites. Is this notice that soon any site owner will be able to hide content behind web forms and expect it to be crawled? No, I don’t think so. First, there are other search engines to think about; Google is not the other engine. Second, the coverage from Googlebot crawling through web forms certainly will not be as extensive as link crawling.

While this search advancement is welcome, I don’t think it should change how good, accessible sites are designed. The advice that Matt gives in his fictional site review still stands. If you need to place a selection of some kind in front of content that you need crawled (like your whole site), it is best to do this with links rather than a web form. There are other reasons for maximizing the content on the site’s main page rather than it being a simple portal with an image and region dropdown.

2 Thoughts.

  1. It seems to me that Google may use this to bring a site ranking down as well as up. Further, how will this stop people from spamming forums? It may encourage it, or encourage the creation of bots to spam forums for you.

    I am still confused about this since I see forum results in google searches already.

    -Joe
    Do-It-Yourself Pest Control

  2. I don’t see your point Joe. How will crawling new pages as a result of filling out web forms cause web rank to drop, especially considering the following from Google’s announcement?

    The web pages we discover in our enhanced crawl do not come at the expense of regular web pages that are already part of the crawl, so this change doesn’t reduce PageRank for your other pages.

    What do forums have to do with crawling through web forms? Most forums are very accessible to search engines without using a web form, that is why you see plenty of forum results in searches.

    The real issue is creating sites that works for people and are findable by search engines. Google doesn’t need to find content through web forms if sites are designed right. Spending time wondering what Google is conspiring against your site is wasted, they don’t have time to single any site out unless you break the webmaster guidelines.

Comments are closed.