Avoid returning menu items in SharePoint Search results

SharePoint Search results can contain hits on menu items, both Top Nav and Quick Launch. Although this might seem strange at first, this is the default behaviour. The reason for this, is that the SharePoint Crawler is actually not much different from a browser requesting a page. The resulting HTML is then crawled, without executing any client side code of course. In SharePoint 2010, this behaviour is a little less annoying because the results get a good low rank and end up below all other hits. But what if you want to avoid these hits alltogether?

As I mentioned earlier, the SharePoint Crawler behaves almost like a normal browser. One of the differences is, that it identifies itself with a custom User Agent string: “MS Search 5.0 Robot” (the version number changes for different versions of SharePoint). We can use this value to determine if the request is made by a crawler or a user. There are a couple of ways to do this, but the easiest is to do it with some lines of code in the master page. Of course you should do this in your custom master page, but you can also use the v4.master (or default.master in case of SharePoint 2007).

Just add the following lines of code, just after the closing </html> tag:

 <script language=C# runat="server"> 
    protected void Page_Load(object sender, EventArgs e)
     string userAgent = Request.ServerVariables["HTTP_USER_AGENT"];
        if (userAgent != null &amp;&amp; userAgent.Contains("MS Search"))
             V4QuickLaunchMenu.Visible = false;

In this case, we stop the Quick Launch from rendering, but everything with an ID can be excluded from the output this way.

You will also need to allow code blocks to be executed, so add the following in the web.config (within the PageParserPath):

<PageParserPath VirtualPath="/_catalogs/masterpage/*" CompilationMode="Always" AllowServerSideScript="true" IncludeSubFolders="true" />

All we need now, is a way to test this. To force Internet Explorer to send a different string, setup the following registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\User Agent