How do you make a Domino site searchable by web search engines

Most web search engines like Excite or AltaVista so not follow links with "?" because they think those are CGI links; Notes 5 allows you to replace the "?" with "!". If you are not using Notes 5, you can do the following:

Create a view named "Pages" that selects all documents that you want searchable, and first sorted column of the view contain each documents key value named "Key". The "Key" value can be @Text(@DocumentUniqueID).

If this is a 4.6 view, use the "treat as HTML" feature. In one of the columns, set the formula to something like this:
"<a href=\"Pages/" + Key + "\"">Page " + Key + " from MySite.com</a>"
If it is 4.5, you'll need to create the same <a>..</a> tag in a column, and surround it with "[" and "]" to make it passthru HTML.

Submit this view (the url "http://server/path/database/Pages") to the web search engine. The engine will traverse the view, follow all the links, and index your pages. However, if the database contains 1000 documents, but the web server is only set to display 50 at a time in views, only the first 50 will get indexed.

Here's a better way to do this from Niels Harremoes (niels.u.harremoes@dk.pwcglobal.com): use an agent named e.g. GenLinks to print a page with links to all documents and then point the search engine to http://server/path/database.nsf/GenLinks

If you already have a view containing all the documents you want indexed, you could also just use that view. This also has the advantage of not cluttering your database with another view:

Sub Initialize
  Const viewname = "(CrawlIndex)" 'A view displaying all the documents to be indexed - use a view alias if needed
  Dim ses As New NotesSession
  Dim view As NotesView
  Set view = ses.CurrentDatabase.GetView(viewname)
  Dim d As NotesDocument
  Set d = view.GetFirstDocument
  While Not d Is Nothing
    Print {<A HREF="} & viewname & {/} & d.UniversalID & {">} & d.Title(0) & {</A><BR>} & Chr(13)
    Set d = view.GetNextDocument(d)
  Wend
End Sub

Of course, generating the links in this way is a lot less efficient than having Domino generate a view - however, since the agent will only be called by the search engine once upon visiting the site, this shouldn't be a problem.

Other things to be aware of: Text inside collapsed sections won't get indexed, since the search engine won't go click the "expand" link. Attachments won't get indexed either. A workaround for that would be to expand the agent above to generate links to all attachments as well as to the document itself.

Applies to Notes Versions: 4 4.5 4.6 5