Notes from 46th International STC Conference
Cincinnati, Ohio, May 16-19, 1999

What Web Authors and Designers Should Know about Search Engines

Gay Marie Logsdon
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN

Session Description: Use principles based on search engine technology to increase the number of hits your Web sites receive and make information more accessible. This presentation offers an overview of the technology underlying prominent search engines and their implications for Web site design, from the standpoint of both the writer and the artist.

Handout: The presenter provided a valuable handout, listing URLs for numerous search engines, specific tips for Web writing and design to maximize search engine exposure, suggestions for site registration, and a bibliography of online and paper resources. For a copy, contact Gay Marie Logsdon or Dan Voss.

• Presentation covered three major elements: (1) search engine technology, (2) tips for Web writing and design, and (3) site registration.
• Search engine technology
   
• AltaVista "spider" is known as Scooter; it scans documents and collects key words and phrases, mostly nouns... good engine for technical research.
• Excite: Architect Spider... very up to date; 50-60 million documents
• Lycos: Latin for wolf spider... catalogues 50-50 million sites
• HotBot: Slurp is the spider (software "robot"). 50-60 million sites; good for technical searches.
• InfoSeek: fast and accurate, but small database. Good for general, broad searches; not as good for highly specific. Spider's name is Sidewinder.
• Spiders are sent out every six months or so to scour the Web, bringing back a vast database. It is this database that is combed during individual searches; i.e., the spider does not go out and crawl the Web on each separate search... appealing a metaphor as that may be!
• Metacrawlers... not as extensive in their range and depth as the ones listed above, but still useful
   
• MetaCrawler: this one searches several of the other search engines and displays number of hits on each.
• Inference Find: good for obscure topics
• SavvySearch
• Picking a search engine
   
• Coverage
• Search properties
• Delivery
• Results
• Ease of use
• How to search effectively
   
• Be aware of stopwords
• Put the word in a context
• Use " " for exact quotes
• Exclude words
• Read the HELP menu!
• Picking a precise phrase as a descriptor is extremely important. For example, consider this search on AltaVista with progressively more specific descriptors, and the number of hits each search produced. Note that the first two searches were Simple and the second were advanced, using the Boolean term "AND" to narrow the search.
  
  

  Descriptor	Number of Hits
  Americans	1,230,320
  Native Americans	84,183
  Native Americans AND Art	18,938
  Native Americans AND Petroglyphs	287
  Native Americans AND Petroglyphs AND Dinosaur National Monument	3

• Search engines use proprietary formulas that quantify "relevancy" as they search, thus establishing the ranking of the hits that are displayed.
• A good sources for more information on search engines: Search Engine Watch.
• Tips for writing
   
• The key premise is: If the spider can't see it, it doesn't exist.
• Spider can't see:
   
• Text in images
• Text in frames
• Stopwords
• Sometimes in tables
• Spider can see:
   
• Title tags
• Heading tags (1st thru 6th... with weight differentiation in the algorithm for prioritizing for relevancy)
• Body text (top-ranked sites repeat key buzzword phrases)
• File names (often ignored, but effective use, including proper hyphenation, can be a key discriminator in ranking)
• Hyperlinks (especially useful to embed key terms in these, especially for internal search engines)
• Keyword metatags... put most important words as far forward in the meta-tags as possible. Specifically, include:
   
• Acronyms and long forms
• Single words and phrases
• Common variants
• Nicknames
• International variants
• Common misspellings
• Description metatags (covers several descriptors... everything after the equal sign... most search engines are set to dig down only 20 words or 200 characters)
• Comment tags
• Do spiders bite? Yes.
  
  
  
  Here are some "fatal faults," guaranteed to get your site blacklisted:
   
• Spamming... deliberately repeating key words huge amounts of times to get more hits
• Embedding keywords subliminally in backgrounds, watermarks, etc., to "trick" the spider
• Using stopwords
• Bottom line: spiders are smarter than you think, and they don't like to be conned.
• Important point: If site's home page is mostly graphic, it becomes critical to make effective use of the title tag and metatags to attract the search engines.
• Rule of Thumb: If your site is not in the first 25 in the list, it doesn't count.
• The "Triple A Aardvark Principle" for being first in the phone book does not apply to search engines.
• What, then, does determine top ranking?
   
• Placement of key words
• Proportion of key words (repeat them!)
• Popularity of site (visit counts)
• Miscellaneous other
  
  Example: Use of "free words"... the ORNL site listed a bunch of otherwise unconnected "NASA-Speak" keywords at the end of its site, to trigger the NASA search engines. It worked.
• Book reference: Achieving Top Ten Rankings in Internet Search Engines... has good, specific suggestions. To obtain a copy, for a limited time you may order at a discount via the presenter, Gay Marie Logsdon. Thereafter, you may contact the author, Fredrick Marckini, directly.
• How to gain points with search engines:
   
• Consider search engine technology
• Use keywords at the beginning of the first paragraph and early in the metatags
• Avoid frames
• Use a clear, descriptive title tag with key words
• Include keywords in internal links
• For IMG SRC tags, with ALT = next to it, use keywords in the ALT = text. Many search engines hit these, which increases the keyword count
• Place metatags early in the site
• Also use comment tags to catch search engines that don't search for metatags (including the biggest one, AltaVista)
• Update your site regularly, or the spider will consider you dead during its six-month "crawl," and you will be purged from the rankings.
• Do not use the competition's name as a ruse to lure their customers via search engines; you will be banned from the search engines.
• Bottom line: be honest and ethical, yet strategic -- and the spider will come to you!
• Site registration
   
• Pick the most relevant search engines for your purpose
• Submit your URLs directly to those search engines... categorize
• Resubmit when changed.
• The fastest sites in picking up registrations and adding them to the index are AltaVista and HotBot (about 2 days)...others range up to several weeks.
• Register your domain name to fit the keyword
• Create individual doorway pages... hosted under separate URLs (but don't go overboard, creating thousands of them... or you will be banned, just as for other "fatal faults," above)
• Tips on specific search engines
   
• AltaVista: Will consider every page separately; title can have up to 1,024 characters
• InfoSeek: Will consider every page; keyword metatag has up to 1,000 characters
• Yahoo: Can submit only one page per site; difficult to get revised listing; keyword repetition is important
• HotBot: Can submit only the front page; will consider metatags, comment tags, etc.
• Web resources on search engines
   
• Web Position Gold: will trouble-shoot your site for effectiveness relative to specific search engines, and make specific suggestions for improvement.
• E Street Web Partners: not only offers for-fee consultation on tailoring Web sites for maximum search engine exposure, it has also posted a valuable -- and free! --o nline tutorial that offers concise coverage of search engine ranking criteria, use of keywords, Meta tagging, "doorway" pages, and other useful techniques to "set bait for the spiders."