Vedant Kulshreshtha

I have no special talents. I am only passionately curious.

FAST ESP: Architecture

FAST ESP: Architecture

  • Comments 2
  • Likes

FAST ESP is built on a distributed system architecture that enables any type of information to be accessed through core search and filter engines. Incoming content is fed through a number of processors that automatically extract information, generate document summaries, and utilize metadata information to provide rapid and relevant search and alert results.

The core platform has been designed to allow for modification and customization by content owners and rapid integration with external data sources. At a high-level, FAST ESP does following tasks:

  • retrieves or accepts content from different content sources
  • transforms all content into an internal document representation
  • analyzes and processes these documents to allow for enhanced relevancy
  • indexes the documents and makes them searchable
  • processes search queries against these documents
  • applies algorithms or business rule-based ranking to the results
  • presents the results along with navigation options

System Architecture

FAST ESP accepts more than 370+ data formats from disparate sources within the Enterprise, creating a single interface available for searching and alerting of information. Data is either added to the system:

  • using push APIs or
  • retrieved from external sources using predefined and/or custom data connectors. T

The flexible query and document-processing framework enables powerful relevance enhancements to be applied to data and queries, ensuring that the end-user receives highly relevant results. Flexible APIs on the query side enable easy integration with customer applications.

 

The core solution is built on two proven, high-capacity core engines, the FAST Search Engine and the FAST Filter Engine, which provide search and filter/alert matching. Pre- and post-processing modules provide easily employable document feeding and retrieval.

Data Flow Diagram

The data flow through the FAST ESP system consists of the following basic steps:

1). Submitting Content - Content is submitted using either,

  • one of the Content Connectors that are included with FAST ESP
  • one of the FAST Content Connectors, available as separate software packages
  • the FAST Content API to push content directly to FAST ESP

2). Analyzing and Processing Documents – Each submitted entity is converted to a document, that complies with the FAST ESP internal document format, and goes through a set of document processing steps performed by the FAST Document Processing Engine. This helps to extract additional information, such as the language of the content, and to add additional information to the document to improve the search relevancy.

Processed documents are then passed on to the FAST Search Engine.

3). Matching Documents and Search Queries - The end-user or external query application submits search queries through a search front end (SFE) or directly to the FAST Query API. The Query API sends the query to the FAST Query & Result Server, which pre-processes the search queries, to improve the relevancy of the results, and sends it to the FAST Search Engine. The engine matches them against its indices and returns a list of resulting documents along with result set navigation options which let the user further refine the search. The FAST Query & Result Server can then perform post-processing on the result list, such as category result grouping, sorting, or adding navigators for dynamic drill-down.

Finally, the result list is returned through the FAST Query API back to the Search Front End (SFE) or the external query application

4). Managing and Tuning - The FAST Administrator Interface (also referred to as the Admin GUI) allows you to easily manage and monitor your FAST ESP implementation. FAST Home and Search Business Center applications are used for setting up and tuning search sites, creating users and user access, and accessing other FAST interfaces

SharePoint 2007 - Search Architecture

If you are SharePoint 2007 developer/architect, you can read more about its enterprise architecture in the SharePoint SDK. Online version is here: Enterprise Search Architecture.

SharePoint 2007 delivers search capabilities tightly integrated with other core infrastructure workloads, including portals, business intelligence, enterprise content management, and collaboration. SharePoint search can be used to find people and expertise, access structured data in line-of-business systems, index a wide variety of content sources, and federate to other search systems.

SharePoint’s search architecture might look similar but some components and features like Content API, advanced linguistics processing, advanced results processing, flexible relevancy tuning are not there or as advanced. But as I mentioned earlier, its important to remember that SharePoint 2007 and FAST ESP both target a specific but different set of customer needs and you need to evaluate which one would be the most relevant tool for your scenario.

I would cover the differences in more detail in another blog post. As per the recent Enterprise Search roadmap update announcement, customers would be able to leverage many of FAST ESP features within future SharePoint release :-)

Comments
  • PingBack from http://iworker.cz/2009/05/03/fast-and-sharepoint-2007.html

  • Nice Article Vedant!

    But a small suggestion in the above article and in section content flow content being pushed in index and retrieved from Index is not quite clear and this may be a point of confusion for a newbee to understand.

    It will be excellent if that view comes out

    Regards,

    Chirag Shah

    talk2cshah.blogspot.com

Your comment has been posted.   Close
Thank you, your comment requires moderation so it may take a while to appear.   Close
Leave a Comment