What is a
First of all, you need to understand what a search
engine actually searches. When a potential visitor does a
search in a search engine, such as Google or AllTheWeb/FAST,
she is not really searching the web; rather, she is looking
at a database compiled by that search engine. This database
consists of the text and links from the web pages that have
been visited by the search engine's robot.
How is a search engine database compiled?
Search engines compile these databases automatically
using software programs called "robots" or "spiders".
These automatic programs visit pages on the World Wide Web,
much as humans visit web pages using browsers, by starting
at some arbitrary location and following links. When a website
owner "submits" a page to a search engine, in most
cases he/she is supplying the search engine's robot with a
starting point for their automatic journey. Starting in that
location, the robot then follows links and thus "discovers"
other pages in your website or visits other sites to which
your site is linked. (This, by the way, is how search engines
can find individual pages or whole sites that have never been
submitted to them--if there is a link to one site from another
site, chances are good that eventually a search engine robot
is going to find that link and follow it.)
Even though robots visit pages like human visitors do, what
they can do with what they "see" is quite different.
When a human visitor uses a browser to view a web page, that
visitor can read the text on the page, look at images, play
movies, listen to sounds, submit information in forms, follow
hyperlinks, and any number of other tasks. The human visitor
really interacts with the site. The search engine robot, on
the other hand, can only do a few of these things. It is this
difference that can keep your dynamic page from being included
in the search engine database.
What does a robot do?
Search engine robots are very simple creatures. They can "read"
text, and they can follow links. That's it. Robots cannot
view a Flash movie, they cannot fill in a form, and they cannot
click a "submit" button. What that means is that
no matter how much great information your web page may contain,
if a visitor has to select it from a list, or type a password,
or submit a form full of information to get there, no robot
will ever visit that page.
The origins of dynamic pages
Most dynamic web pages are generated in response to
queries run against databases. Behind your widget website
there is a large database of widgets. When a visitor comes
to your site and looks for left-handed blue widgets, it is
this database that supplies the response. The database provides
that information to the visitor. Typically the visitor checks
a box or selects from a list or even types text onto the page
and presses a "submit" button. Once he/she jumps
through those hoops, your visitor gets her page full of left-handed
I can't see you
Unfortunately, when a search engine robot visits this
page, it cannot check that box, it cannot select from that
list, and it cannot click the "submit" button. Put
simply, the robot cannot get to page of widgets. If the robot
can't get there, the page will not be included in the search
engine database. If it's not in the database, searchers cannot