Webmaster tips & trix   contact | privacylink partnersfree website content
Home » Articles » Search engine optimization
Articles

» Affiliate programs
» SE optimization
» Miscellaneous
» Newsletters & E-zines
» Promotion
» Web design
» Usability
» Domain names
» Create income
» Web writing
» Hosting
» Guerrilla marketing
» Accessibility
» Web credibility
» Data recovery

» Archive

 


Special articles

» Google tools/services
» Yahoo! tools/services


Search engine robots - How they work, what they do (Part 1)

Daria Goetsch
Search Innovation
May 1, 2003

Why Isn't My Website In The Search Engine?

If your site isn't found in the search engines, it is probably because the robots couldn't deal with it. It could be something as simple as not being able to find the site, or it may be more complicated issues involving the robot's not being able to crawl the site or figure out what your pages are all about.

Submitting your site to the major search engines: that will help with the "can't find it" problem. Even having links pointing back to your site can be enough to attract the search engine robots. Google, for example, suggests that you may not have to submit your pages; they will find your site if you have a link pointing back to it from at least one other site on the web.

If the robots can find your site but can't make sense of it, then you may need to look at the content and technology used on your pages. Frames, Flash, dynamically generated pages, and invalid HTML source code can cause problems when the search engine robot tries to access your web pages. While some search engines are beginning to be able to index dynamically generated pages and Flash (e.g. Google and AllTheWeb), use of some of these technologies can hinder your ability to be indexed by the search engine robots.

Text in images cannot be read by the search engine robots. Using ALT image text is an important way to help the robots "read" your images. Websites with extensive images rely heavily on ALT text to present their content.

How Do I Get The Most Out Of Indexing?

If you know what to "feed" the spidering robots you will help yourself with search engine ranking.

Having a website full of good content is the major factor. Search engines exist to serve their visitors, not to rank your website. You need to be sure to present yourself in your site in the way that will be most useful to the search engine visitor. Each search engine has its own idea of what is important in a page, but they all value text highly. Making sure that the text on your pages includes your most important keyword phrases will help the search engine evaluate the content of those pages.

Making sure that you have good title and meta tags will further assist the search engines in understanding what your page is about. If the text on the page is about widgets, the title is about widgets, and the meta tags are about widgets, the search engine will have a pretty good idea that you are all about widgets. When their visitors search for widgets, the search engines know to list your site in the results.

A sitemap page is a very good way of giving the search engine robot every opportunity to reach your website pages. Since robots click through the links of your web pages, make sure that at least your most important pages are included in the sitemap; you may even want to include all your pages there, depending on the size of your site. Be sure to add a link to the sitemap page from each page on your site.

Another important consideration is that of keeping all of your pages within a small number of "clicks" from your top page. Many robots will not follow links more than two or three levels deep, so if your "widgets" page can only be reached from your home page by following multiple links (e.g. home page >> about us page >> products page >> widgets page), the robot may not crawl deep enough to get to the widgets page.

Testing Your Website For Search Engine Robot Accessibility

To get an idea just what the search engine robot "sees" on your page, you can look at the Sim Spider tool. You may be surprised at how different your site looks to the robot. You can find this tool at http://www.searchengineworld.com/cgi-bin/sim_spider.cgi

You will see text and ALT image text show up in the results. If your entire website is built in Flash, you will see nothing at all because robots don't understand Flash movies.

The Bottom Line

When it comes to search engine robots, think simply. Lots of good content and text, hyperlinks the robots can follow, optimization of your pages, topical links pointing back to your site and a sitemap will help insure the best results when the robots come visiting.

Resources

SpiderSpotting - Search Engine Watch
http://searchenginewatch.com/webmasters/spiders.html

Robotstxt.org
List of robots and protocols for setting up a robots.txt file.
http://www.robotstxt.org/

Spider-Food
Tutorials, forums and articles about Search Engine spiders and Search Engine Marketing.
http://spider-food.net/

Spiderhunter.com
Articles and resources about tracking Search Engine spiders.
http://www.spiderhunter.com/

Sim Spider Search Engine Robot Simulator
Search Engine World has a spider that simulates what the Search Engine robots read from your website.
http://www.searchengineworld.com/cgi-bin/sim_spider.cgi

About the author
Daria Goetsch is the founder and Search Engine Marketing Consultant for Search Innovation Marketing (http://www.searchinnovation.com), a Search Engine Promotion company serving small businesses. She has specialized in search engine optimization since 1998, including three years as the Search Engine Specialist for O'Reilly & Associates, a technical book publishing company.

Copyright © 2002-2004 Search Innovation Marketing. http://www.searchinnovation.com All Rights Reserved.


Latest articles

» Change your mind about an eBay bid?
We have all made choices in life that two seconds later we know we should take back. Especially when there is money involved this can become a problem.

» A simple way to create 7 effective autoresponder messages
Email is the Net's most powerful marketing tool. And autoresponders are the best idea yet for marketing with email.

» 7 ways to drive laser-targeted traffic to your website
Getting people who matter to see one’s website is a difficult undertaking if he tries to consider the fact that there are rivals everywhere waiting to pin him down.

» Website valuation: Why standard website pricing methods will emerge
The market of buying and selling developed websites is becoming more and more liquid each day.

» One way links are better than reciprocal links
You probably know by now that where your website ranks in the search engine rankings dramatically affects how many visitors you have to your site. Did you also know that you can change where your site is ranked by being proactive and getting as many one way links to your site as possible?

» How to make visitors stay at your website
The very first thing which you should provide the visitors with is some free interesting reading material.

» How to make your visitors click your ads
Here is a simple solution; Convert your banner advertisement to look like a text advertisement!

» Offline advertising should be a part of your online strategy
Day by day, online business has become more & more complicated and competitive.

» How to sell traffic
Selling the traffic arriving at your site is a good method to increase profits from your portal.

» Make money from online auctions
Online auctions have the best benefit of a vast platform. Your product is viewed by loads of people & hence there is more possibility of finding a suitable bidder.

» Groupware explained in easy terms
Groupware is a term used frequently to describe collaborative software. Groupware is application software that integrates work on a single project by several concurrent users at separated random workstations.

» Timely back up can save you from disasters
Few things which people often back up are e-mail addresses, bank records, photographs, personal records, software’s, music etc.

» Why should one go for autoresponders
Autoresponders are programs which get automatically executed in particular situations.

» Become your own boss - Start your own online business today
A survey conducted by SBA states that two third of new business survives at least two year and about forty four percent survives at least four year.

» Express your thoughts - Creating your own blog!
What exactly is a blog? Technically speaking it is a journal or a newsletter which is regularly updated and can be used by any one.

» Pop-up ads - To be or not to be?
According to a study conducted by the Bunnyfoot University, “The Efficacy of Pop-ups and the Resulting Effect on Brands” Internet users feel harangues and harassed by pop-up ads.

» Why content is king on the Internet
The advantages that Internet holds over the rest of the other communication mediums should not wasted because of the inability to find a comprehensive plan that will bind all these faculties together.

» 10 niche marketing tips
In our increasingly driven consumer economies, the average customer is bombarded by choices. With increased saturation of the market, companies look towards niche marketing to search new, ever-evolving and sophisticated consumers.

» Using free traffic exchange
These days internet has emerged as both, a market and hub for marketing. Unlike the ‘brick and mortar’ world where large manufacturers manage to squeeze out the market bases of smaller companies, the internet provides haven like the free traffic exchange.

» Ten ways to drive traffic to your website
Developing a web site and then letting it grow is like planting a tree and then nurturing it.

» Marketing through keyword articles
One of the most effective tools of Internet marketing is the use of keyword articles.

» Want to make money online? Market a service to businesses
Don walked across the street from his house to mine to announce he had finally retired. "But I'm not ready for the golf course," he said. "I want to make a living on the Internet. What can I sell?"

» Web site design mistakes - Database parameters in URLs
Creating a web site takes thought, planning and execution. Unfortunately, many designs are dead in the water before they are even published as far as search engine optimization is concerned. Whatever you do, avoid these critical mistakes.

» Alexa Toolbar - The ultimate internet tool
There are numerous tools available on the Internet to assist online businesses. A valuable tool that you should use is the Alexa Toolbar. Even better, this tool is free.

» Web site design mistakes
Some wise human once said "Learn from the mistakes of others. There isn't nearly enough time to make them all yourself." Hence this article. Here are five of the most annoying and common web design mistakes.

» Abandonment - Why visitors don't turn into customers
Every good Internet business understands the value of conversions versus hits received. Far too often, businesses become fixated on the hits they are receiving instead of monitoring their hit to sale conversion rate.

» Creative search engine optimization - A case study
Search engine optimization this and search engine optimization that. You read and hear about it all day, but what about your site?


Tools & services to enhance your online business

» Site Build It!
Over 100,000 small businesses of all kinds outperform larger, well-financed competitors. Read about this all-in-one site-building-hosting-marketing system of tools that delivers results.

» Secrets To Their Success
Take a private tour of two "Mom & Pop" web sites every month that earn $100,000+ a year... and discover the exact step-by-step strategies they have personally used to generate these massive profits.