In this post, I would like to talk about a TMG safe search problem which I dealt with recently. The problem was that the TMG safesearch feature wasn’t working properly even though the all the requirements were met. We decided to collect dynamic logs while reproducing the problem (TMG data packager + client side logs). Then I started analyzing the logs:

 

- The Safesearch related rule was already enabled:

 

SafeSearch Enabled

   Action Allow

Applies Always true

Default false

Description The SafeSearch rule allows access to the "Search Engines" URL category, and filters out adult content from search results of supported search engines.

ID 

Logging Enabled

Source Port Limits Disabled

Type Access Rule

From

  Network Internal, Included, Array scope

Applies to all content

Protocols Specified Protocols

 HTTP, Included, Array scope

 HTTPS, Included, Array scope

UrlCategory Search Engines, Included, Enterprise Scope

Users All Users, Included, Array scope

 

- It already included "All users"

- TMG server version was the latest (SP2 + rollup2) 

- Safesearch XML file was in place with the correct settings:

 

<Configuration>

                <provider domainPattern=".google." safeSearchSuffix="&amp;safe=strict">

                               <searchQuery pattern="/search?"/>

                               <searchQuery pattern="/images?"/>

                </provider>

                <provider domainPattern=".yahoo.com" safeSearchSuffix="&amp;vm=r">

                               <searchQuery pattern="/search?"/>

                               <searchQuery pattern="/search;"/>

                               <searchQuery pattern="/search/images?"/>

                               <searchQuery pattern="/search/images;"/>

                               <searchQuery pattern="/search/video?"/>

                               <searchQuery pattern="/search/video;"/>

                </provider>

                <provider domainPattern="www.bing.com" safeSearchSuffix="&amp;adlt=strict">

                               <searchQuery pattern="/search?"/>

                </provider>

</Configuration>

 

- It's the same as the one provided in ISA blog:

 

http://blogs.technet.com/b/isablog/archive/2010/09/21/new-in-forefront-tmg-update-1-safesearch-enforcement.aspx

 

<provider domainPattern="www.bing.com" safeSearchSuffix="&amp;adlt=strict" >

         <searchQuery pattern="/search?" />

 </provider>

 

=> Even though all configuration was correct, TMG server was still not forwarding the correct search URL towards Bing (which would help Bing filter search results)

 

- We see the search requests sent by the client on the client side network trace. Some examples:

 

2857      12:46:22.524738               50.167033           0.000000              10.96.96.6           10.110.0.121      HTTP     GET http://www.bing.com/search?q=xxx&qs=n&form=QBLH&pq=xxx&sc=0-0&sp=-1&sk= HTTP/1.1        

3642      12:46:44.659253               72.301548           0.054843              10.96.96.6           10.110.0.121      HTTP     GET http://www.bing.com/images/search?q=xxx&FORM=HDRSC2 HTTP/1.1          

3832      12:46:48.339288               75.981583           3.680035              10.96.96.6           10.110.0.121      HTTP     GET http://www.bing.com/search?q=xxx&FORM=HDRSC1 HTTP/1.1           

 

=> Relevant session in TMG ETL Logs: (collected as a result of TMG data packager)

 

...

 [0]197c.1cc8 10/03/2012-10:46:48.780 [1a2c7b91 1a2c7d03] [WP proxyext...] Info: WPPISAPUBLIC:Context property:WPPISAPUBLIC:HTTP URL = http://www.bing.com/search?q=xxx&FORM=HDRSC1

... 

 

=> Safesearch filter decides that there's no need for safesearch analysis:

 

[0]197c.1cc8 10/03/2012-10:46:48.781 [1a2c7b91 1a2c7d03] [HTTPFLT...] Entering CHttpFilterSafeSearchEnforcer::IsSafeSearchAnalysisRequired

...

[0]197c.1cc8 10/03/2012-10:46:48.781 [1a2c7b91 1a2c7d03] [HTTPFLT...] Info:SafeSearch analysis is not required, exiting

 

=> Network trace collected on the external interface of TMG server:

 

- We can see that the request is being sent as is without any modifications towards the Bing server:

 

256368  06:46:49.2700830             875.6370830                      10.110.7.54         212.252.126.59         HTTP     HTTP:Request, GET /search, Query:q=xxx&FORM=HDRSC1               {HTTP:3723, TCP:3721, IPv4:3651}

256370  06:46:49.3080860             875.6750860                      212.252.126.59         10.110.7.54         TCP        TCP:Flags=...A...., SrcPort=HTTP(80), DstPort=58199, PayloadLen=0, Seq=3768470858, Ack=3665294676, Win=25515 (scale factor 0x0) = 25515    {TCP:3721, IPv4:3651}

256372  06:46:49.3600890             875.7270890                      212.252.126.59         10.110.7.54         HTTP     HTTP:Response, HTTP/1.1, Status: Ok, URL: /search         {HTTP:3723, TCP:3721, IPv4:3651}

 

http://www.bing.com/search?q=sex&FORM=HDRSC1

 

=> Normally it should have been sent something like below so that adult content wouldn't have been returned by Bing:

 

http://www.bing.com/search?q=sex&FORM=HDRSC1&adlt=strict

 

At this point everything seemed to be correctly configured but for some reason the safesearch filter wasn’t kicking in. After some more research and with the help of an escalation engineer from TMG team, we found out that in order for safesearch filter to kick in, the request itself also should be matching the Safesearch rule which is automatically created when Safesearch is enabled.

 

In my customer scenario, problem was that the access request sent by the user was hitting an enterprise level access rule and hence the safesearch rule wasn’t hit and the filter wasn’t activated as a result. After my customer re-arranged the enterprise level rules so that search engine related requests don’t hit any enterprise level access rule but hits the array level “Safesearch” rule created automatically, the problem was resolved.

 

Hope this helps

 

Thanks,

Murat