My Learnings

Search Tips and Tricks

2009-03-05T10:16:00Z

I was troubleshooting an issue where some users have added links like http://www.microsoft.comdefault.aspx and there are errors in the crawl logs as there is a ‘/’ missing in the above URL. We had around 4 links like this and it is very difficult to find it by looking at the UI as the Portal is really huge with lots of site collections /sub sites. But it can be found easily from the SSP_Search database.

If you have a URL like HTTP://, it will be crawled as Anchor Text and there should be an entry for the same in the MSSAnchortext table. Then ran the following query to find more about the URL.

select sourcedocid From mssanchortext where link like 'http://www.microsoft.com/%'

The above query returned the sourcedoicid for the URL. Now we need to look into the MSSCrawlURL table to see who is referrer or source. In the MSSCrawlURL table sourcedocid is mapped to DocId. So the following query will give you the source where the incorrect link is stored.

select * from msscrawlurl where docid = 1415

If you combine both the queries it will look like,

select * from msscrawlurl where docid in (select sourcedocid From mssanchortext where link like 'http://www.microsoft.com/%')

How-to enable stemming when you search using the scope 'This Site' or 'This List'

2009-03-04T11:03:00Z

When we search on a SharePoint site using the ‘This Site’ or ‘This List’ scope, the results are shows in the Osssearchresults.aspx page. This page is not similar to the results.aspx which is shown when we use the ‘All Sites’ scope. The results.aspx page is coming from the Search Center template and this page contains a bunch of web parts like Search Summary, Search Statistics, Search Best Bets, Search Paging, Search High Confidence results and Search Core results. You can enable stemming through the Search Core results web part properties.

But you cannot edit the Osssearchresults.aspx in the similar fashion to enable stemming as this page is stored in the _layouts folder. So we need to edit this page to enable stemming and use this feature in the ‘This Site or This List’ scope.

Keep in mind that modifying a page within the _Layouts folder may be overwritten during a patch / Cumulative Update / Service pack installation of MOSS 2007.

Steps

Open the OSSSearchresults.aspx from the 12\template\layouts folder using SharePoint designer / Visual studio or even using Notepad.

Search for SearchWC:CoreResultsWebPart.

Add this - “StemmingEnabled=true” to the following section of the OssSearchresults.aspx page and do an IIS Reset.

===================================

===================================

How-To Rollup two lists in two site collections on a page

2009-02-19T22:00:23Z

We can use the content by Query web part for rolling up of lists in a single place. But this only supports a single site collection. If you want to show the data from two lists residing in two different site collections, then we can use SharePoint Designer. I just explored with the help of my colleague and it works just fine.

In my example I am using three site collections; root, sites/first and sites/second. I have the default document library; Shared Documents in all site collections.

1. Open the root site collection using SharePoint Designer.

2. Create a new aspx page in the root or in the pages library depending on the requirement.

3. Click on the Task Pane and enable the ‘Data source Library’ and now you can see the same on the right side of the screen.

4. Click on the plus sign on the XML Web Services and click on Connect to a web service.

5. Click on Connect to a web service inside the XML Web Services section.

6. Type the URL for the lists.asmx web service within the first site collection. In our example, provide http://server/sites/first/_vti_bin/lists.asmx and click on the Connect Now button.

7. This will provide the following screen. In operation drop-down, select GetListItems.

8. Double click on the listname from the Parameters section and type the name of the document library / list. In my case, I typed Shared Documents and click ok.

9. Repeat the steps from 5 to 8 for the different list in a different site collection.

10. Now we have two connections to two different lists in two different site collections.

11. The last and final step is to link these two sources through Linked Sources.

12. Click on the plus sign in the Linked sources and click on ‘Create a new linked source’.

13. This will bring a new window and click on ‘Configure Linked Source’

14. In the resulting window, select the two XML web services that we have created and added to the list of Selected data sources and click on Finish.

15. Now click the new linked data source and click on Show Data.

16. The above step will bring the following screen. The last step is to insert the z:row to the page that we have created in step 2.

17. Now it’s your turn to add some beautification to the list.

Some interesting facts about SharePoint 2007 Search

2008-11-14T21:21:00Z

Can we search in any language other than English? Do we need language pack for the same?

Language Pack has nothing to do with search in languages other than English or the language in which SharePoint is depoyed. Out of the box, MOSS already shipped with the major wordbreakers/stemmers, although with very bad quality for some of the languages such as Chinese and Deutsch

Despite the quality of the word breakers, by design you may encounter two problems,

1. In index time, the ifilters should emit a correct LCID for that language. However, this is not possible with some of the file types. For example, when processing file types like TXT, XLS(XLSX) and RTF, The ifilters will return 1033(en-us) instead of the correct ones. So what will happen? You may get nothing when you search for any long word, only single character for Japanese/Chinese in these files. Other language may have the same problem, but not as obvious like that.

2. In query time, when user submit the keyword through the browser, MOSS will detect the browser language setting that the user is using. And it will use the value to call the corresponding wordbreaker. If this wordbreaker does not match with the one used in index time, you will be in trouble again. For example if you use a English client to search for something in Chinese, without modifying browser language setting to chinese, even the files indexed in the right language, you will still get no result for a word.

The space needed for the index on the query machine is approximatley 2.8 times of the size of the actual index. What is the logic behind this?

Lets say the index size is X.
During crawls, we accumulate more shadow indexes because of items that are indexed. When these shadow indexes cover about 0.1 times X (10% of X), we do a master merge.

A master merge takes the 1.1X (X + 0.1X) and creates a new index with that in the same location as the old index. The size of the new index is roughly 1.1X. So before the old index is deleted, the requirement is for at least 2.2X (for both indexes).

However, since query servers are expected to be online at all times, the master merge should have minimal impact on query latencies. To achieve this, we use more than the 1.1X space by creating temp files during the master merge.

This leads to the worst case number of 2.8X so that master merges can succeed while not impacting query latencies.

Then we will delete the old index on both, the indexer and query machines immediately after the master merge is complete.

How does the duplicate document is identified when we do a search?

Document similarity for purposes of identifying duplicates is based only on a hash of the content of the document. No File properties (e.g. file name, type, author, create and modify dates) are input to this hash. The MSSDuplicateHashes table in the SSP’s search database holds, for each document, all the 64bit hashes necessary to determine if one document is a near-duplicate of another. This is read while doing a search if duplicate collapsing is enabled.

What are discovered definitions and how does search find those?

Discovered Definitions are a feature in MOSS that can be enabled/disabled in the properties of the SearchCoreResults webpart. When enabled, the results web part will display not only document matches for a term, but also any definitions it has discovered for that word during crawling.

Definition extraction feature in MOSS 2007 is a feature that extracts meaning of definition from indexed text.

Definition Extraction is done during the crawl. The crawler looks for couple verbs like ‘is a’ or ‘is the’ and then, when a nebulous threshold is reached, it extracts the definition of the related word for later use in search results display with the words “What people are saying about <term>”.

At query time passed search token is compared with existing entry in definitions database. If a match is found the definitions link is populated at the bottom of the search results page. Collapsing the link shows number of definitions.

SQL tables which contains data which is not so useful to keep for a long time in a SharePoint Farm

2008-10-20T16:22:00Z

In my experience I have seen tables which are part of the content db's and also SSP db's which are actually not useful. But these tables grow really fast and fill the hard disk space. In this post, I would be talking about four of those tables; two from the content db and two from the SSP database and also the supported mechanism to clear those.

Tables from the content db

1. EventCache

2. AuditData

EventCache table contains all the changes that happens in the particular web application. This is used by different features; alerts, search, content deployment etc. Alert actually triggers the email based on the change which gets added to this table. There is a setting related to Change Log in the Central admin-->Application Management-->Web application General Settings (http://CentralAdminSite/_admin/vsgeneralsettings.aspx). The default value is 15 days and you can set to Never as well. It is good to have the default value and the data would get cleared automatically.

AuditData table contains all the auditing related information once it is enabled in the web application. There is no UI to control the data as such and if you enable it across all the objects in a web application, the table will become huge in a day or two depending on the number of users accessing the site. I have seen situations where the table grows to 17 Gigs and this creates major performance impact. At the same time, lot of companies need to keep the data for some kind of compliance purposes. The best way is to move the data from the table to another SQL Database and then delete the entries in the table using the STSADM -o trimauditlog command. This command is available only after installing the Infrastructure updates. The other supported way to play around this table is thorugh Object Model. Check this blog for more info.

Tables in the SSP Database

1. UserProfileEventLog

This is similar to the EventCache table of a content db. This table contains all the changes that happens to the user profile store. If you add a user profile, this will add this entry and if you delete a profile a new row is added. Imagine if you delete 40K users from the profile store and reimport in a troubleshooting scenario. The table will contain 80K rows and can impact your performance and also eats up the disk space where the MDF and LDF is stored. I have seen many customers running into this scenario. There is an STSADM comand (ProfileChangeLog) to confiugre the number of days for which the data to be present in the table. But if the table already contains huge data, then there may be issues when the timer job runs to clear this table. You may need to install this hotfix which actually solves the problem if the size of the table is huge. The timer job only runs at 10:00 PM and there is no way to control this.

2. AspStateTempSessions

This table is used to store the Asp.Net session state. InfoPath Forms Services, Usage analysis, Project server, workflow etc uses this table. In addition, it’s possible that custom developed content/pages are created that rely on ASP.Net Session State. By default, there should be a SQL job created namely - <SSPdatabaseName>_Job_DeleteExpiredSessions which will run event one minute to clear the expired sessions from the table. But I have seen customers where this job doesn't exist at all and the table grows really big. Also the SQL Agent service should be running to trigger this job.

Find the template that is applied to a site collection or a subsite

2008-06-11T14:38:00Z

We can find the template ID that is chosen while creating a site collection or a site from the database. Its easy to find the template once we have the ID. The ID is stored in the Webs table of the content db. The ID is just a numberic value like 3, 4, 50 etc. Now we need to convert the ID to the actual name of the template. This mapping is stored in couple of XML files based on whether the template is part of WSS or MOSS. The XML files are stored here - C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\TEMPLATE\1033\XML. The WSS templates are referenced in the WEBTEMP.XML and for MOSS, it is in the WEBTEMPSPS.XML. If there is a custom site definition, you can modify any of these two files and also can create a new one.

Also check this blog for the Out of the box templates that is present in MOSS - http://blogs.msdn.com/dwinter/archive/2006/07/07/659613.aspx

Check-out feature of SharePoint 2007

2008-05-28T14:47:00Z

We have 2 kinds of Check outs, Short term and long term. If you click the drop-down of a document in a Doc lib and choose “check out”, that is a long term. When Word opens the document in “Read-Write” mode, it places a short term checkout on the document. In Office 2003 the short term checkouts were 10 minutes; in 2007 it is changed to 60 minutes. The short term check-out refreshed automatically as long as Word is open. There is no way to force the expiration of a short term checkout.

The discard check-out option available in the drop-down menu of a document library is used only for the long term checkout. If a user opens a document from a document library in the Read-Write mode, then that creates a short-term lockout and no other user can edit the document till the first user closes it. You can find the expiration time in the 'CheckoutExpires' column of the AllDocs table. It becomes NULL only when the first user close the document.

The interesting part of this behaviour is this. If a user does a short check-out and keeps the client application ‘Word’, open with the document and goes away for the weekend, then no other user can modify this document till that user returns and closes ‘Word’. Not even the administrator has the control to get this document back to read-write mode.

Windows XP with Office 2003 / Office 2007:

=====================================

Default short term checkout Expire time is 10 minutes.

If the opened document is closed, the lock will be released immediately.

If the application (Winword.exe) crashed then we have to wait for 10 minutes and the lock will expire in 10 minutes

Windows VISTA with Office 2007:

==============================

Default short term checkout Expire time is 60 minutes.

If the open document is closed, the lock will be released immediately.

If the application (Winword.exe) crashed then the lock will be released immediately.

As per KB http://support.microsoft.com/kb/899709/en-us the suggested workaround is "wait 10 minutes before you click Edit in ProgramName to open the document again."

Functionality of 'Site Use Confirmation and Deletion' in Sharepoint

2008-05-25T10:52:00Z

This feature works the same way as SPS 2003 / WSS V2. When you configure this through Central admin site-->App Management-->Site Use Confirmation and Deletion with a number of days and some number of notices, Sharepoint will only wait for those number of days and will start sending the notices. It is not actually checking whether the site is being used or not. Once it reaches the number of days, SharePoint will send an email to the site owner to check whether the site is being used or not. If the owner clicks on the link and ensures that the site is in use, then the count will reset to the value which was actually set when we configured the Site use confirmation and deletion. CertificationDate and DeadWebNotifyCount are the two fields in the Sites table of the content database where the information is stored. Please keep in mind that modifying the sharepoint databases are not supported

Troubleshooting Alerts

2008-05-25T09:33:00Z

Question: How to Troubleshoot Alerts in WSS 3.0 / MOSS

Answer:

The most common issue in alert is the user will get the initial email, but will not get when changes are made to the list where he configured the alert. Check the following settings in order and ensure that it is correct.

1. Is it an upgrade from V2 to V3? If yes, check the upgrade section at the bottom of this page.

2. If this is a new install of V3 (WSS or MOSS), then do the following

a. Create a new web app and see if the alerts work there. If yes, you can move the content db of the non-working site to this one. If this is not possible, then try the following.

b. Open the command prompt and go to the 12\Bin folder. Run this command and see whether alerts are enabled for the web application.

Stsadm.exe-o getproperty -url http://problemsite -pn alerts-enabled

The expected output is <Property Exist="Yes" Value="yes" />. If you don’t get this, run the following command to change the value.

stsadm.exe -o setproperty -pn alerts-enabled -pv "true" -url http://problemsite

If the property is Yes and still the alerts are not sent, toggle the property from Yes to NO

and then from No to Yes. This may delete all the existing alerts

c. Check the property job-immediate-alerts schedule through command prompt. Run this command from the 12\bin folder. Even though you have properties like job-daily-alerts and job-weekly-alerts, the only timer job that exists is the job-immediate-alerts.

stsadm.exe -o getproperty -url http://ProblemSite -pn job-immediate-alerts

The expected output is

<Property Exist="Yes" Value="every 5 minutes between 0 and 59"/>. If you don’t get this, run the following command to change the value.

stsadm.exe -o setproperty -pn job-immediate-alerts -pv "every 5 minutes

between 0 and 59" -url http://ProblemSite

d. Confirm the above step through the UI. Central adminàOperationsàTimer Job Definitions and ensure that a job named Immediate Alerts is present for the web application.

e. Configure the alert for a user in a list. Instead of typing the domain name\ user name, type the email address of the user and see if he gets initial email. Then make a change to the list and see whether he gets the mail related to the change.

f. If he gets the alert as expected, then create a new alert and this time select the user from the people picker or type the domain name\user name.

g. If the alert is not working after doing the above step, then check the ImmedSubscriptions / SchedSubscriptions table of the content db of the web application depending on the type of alert and see a new record is added and ensure that the email field for the user is populated correctly. If not, check whether the email address is present in the user’s profile through the SSP admin page.

h. The initial alert is not security trimmed. So irrespective of whether the user has privileges on the list or not, the initial alert will be sent. If the user is not getting alert for any changes, then check whether the user has Read permission on the list.

i. This also happen if there is some issue in their mail provider (third party email providers. In this case, capture the ULS log with all information events in the verbose mode and check for lines which says Alert has been sent.

j. Email enabled security groups can also be used for configuring alerts. If the changes are not notified to the members of the group, check whether the group is added to the list with minimum Read permission. Also some email providers block email groups if the number of members are more than certain number.

k. Open SQL Query Analyzer; connect to the content database of problematic site. Run the following query.

Select * from timerlock

This server is responsible for processing the timer service. You can follow this KB –

http://support.microsoft.com/kb/934838 to sync the accounts and password across the

Farm. Each content db can have a different server present in the timerlock table.

The internal working mechanism of how an alert should work in v3:

The EventCache table records the SQL level events as they occur and the EventData and ACL columns are not NULL for an alert event. There is an alerts timer job that runs periodically to process the records in the EventCache. After the alerts timer job runs, it nulls out the EventData and ACL columns. Then, it will log an event into the EventLog table. So check the following in SQL.

select * from eventcache where EventData is not null

This will output all of the subscriptions which have not been processed yet. We can see if there are some alerts which are not processed.

select * from eventlog where ListID = 'xxx'

You can get the ListID from the EventCache table by running

Select * from EventCache and check the documents which correspond to the problematic list.

If you cannot find any record, perform the following tests:

Run filemon on the MOSS server which is responsible for the timer service and check if the Timer

service picks up the alert template during the whole process.

Upload a new document to the document library which is supposed to have the alerts. Begin running

the filemon and analyze the logs.

p. select * from eventcache order by EventTime DESC

Check if the latest log is the one corresponds to your uploaded document. Make sure the EventData and ACL columns are not NULL.

After 5 minutes or more minutes, check the EventCache table again to see if the EventData and ACL columns are NULLed.

If so, stop filemon after the EventData and ACL columns are NULLed . Review log.

If its a scheduled alert (Dialy, Weekly), then you need to see the entries in the EventSubsmatches table as well.

Upgrade from V2 to V3

1. Check this KB - http://support.microsoft.com/kb/936759/en-us

2. If no alerts are getting triggered even for a new web app, then run the Psconfig Wizard without changing any options.

3. If the alert works on new web apps, then you can move the site collections from the upgraded web app to the newly created web app through stsadm –o backup / restore or stsadm –o export / import. This action will be useful only if you have fewer site collections and this is only a workaround.

Some interesting behaviours

The alert was working for one web app and not for any other web applications. The timerlock table in the content db of the working web app has the serverX and all the other content db has ServerY. Shutdown the ServerY and wait for the timerlock table for the non-working content db's to take ServerX. Once that is done, restart the ServerY and now the alerts started working. This is actually a workaround and not a fix. Now you need to find the actual issue by looking into the logs of ServerY.

All the elements of alert was checked and still the alerts dont fire. Then found that the time on the servers in the farm were different and the difference was only 2 mins. Once we did the time sync with the domain time server using the Net Time command, the alerts started flowing.