We've been having a few good (meaning "interesting") discussions lately about custom claims providers and search. As it turns out, there are instances when you need to install your custom claim provider on a search box ("box" being something I'll define down below) in order to get security trimming working correctly in your search results. When this applies though is different depending on whether you are using FAST Search 2010 or SharePoint Search 2010.
First a little explanation is probably useful here. Every time a user makes a query, the claims in the user token are decoded. However the SiteData Web Service, which is what the crawler uses to retrieve security information about content, always returns encoded claims. So how do we reconcile that?
In FAST Search 2010, the claims are always decoded before they are stored by the FAST Search 2010 indexer. We do this because the FAST Query servers don't have the SharePoint bits installed, so they can't encode the claims. Remember the user claims come over decoded, so in order to match the encoded claims the SiteData Web Service returns, the user claims would have to be encoded to do a comparison. Since we can't install a custom claim provider on a FAST Query server we have to decode the claims that we get from the SiteData Web Service, and in order to do that any custom claims provider you are using must be installed in the FAST Content SSA. Doing so allows us to decode the claims, store the claims decoded, and then when the decoded claims for a user are presented we can do a comparison. So in the case of FAST, we are concerned with using any custom claims providers at crawl time.
For SharePoint Search 2010 its kind of the opposite problem. SharePoint expects that it will be installed everywhere, so it works on the premise that at query time, it can encode the claims from the user so that a comparison can be made to the ACLs that are stored for the content. Where you will find this breaks down though is in the scenario where you have not deployed the custom claims provider to servers that are running the query processor, also known as the Query and Site Settings Service. In most cases, you install your custom claims provider on all servers in your farm - WFEs as well as application servers. The query processor needs this custom claims provider installed in order to encode the claims. So if everything is running in a single farm and you install the custom claims provider on all servers, you should be good. The situation that came up recently (and that prompted this post) is a scenario where you have a separate services farm and you are consuming SharePoint Search services from that farm. In that case you need to make sure that any custom claims providers are installed on every server in the services farm that is running the Query and Site Settings Service. If you do not do that you will find that your users' custom claims cannot be evaluated, and as a result they will typically see no search results returned.
This was an interesting scenario and took some great troubleshooting by a cast of characters here...especially calling out my brilliant little brother Luca for helping us get over the final hurdle here, as well as Sanjeev and Michael P. Thanks all for helping us understand this better.