Lucene - Introduction and Indexing

Subscribe Send me a message home page tags


In this post, we will present a brief introduction to Lucene core components and document indexing.

This is a reading note of Lucene In Action - Second Edition so some of the notions may not be up-to-date but the general idea should still be valid.

Introduction

Lucene is a high-performance, scalable information retrieval (IR) library. It concerns itself with text indexing and searching.

The figure below presents the overall process of a search application:

overall-process.png

As we can see in the figure, there are two directions of flow:

The first flow is the searching part of a search application and the second flow is the indexing part. In this post, we will focus on the indexing part of a search application.

Indexing is a process that transforms the original data into a highly efficient cross-reference lookup in order to facilitate rapid searching.

The indexing workflow is illustrated in the figure below:

indexing-workflow.png

To build an index, we first need to collect our data, which is the raw content of the index. Then we need to transform the row content into documents. A document is Lucene's atomic unit of data used for indexing and searching. It's a container that holds one or more fields, which in turn contain the "real" content. Creating documents can be challenging because we need to deal with many different formats and need to figure out a unified representation of all the data. Note that this step is application-specific and it's not Lucene's responsibility.

Not all documents are equal. Some documents are more important than others. The importance of documents is represented by the boosting factor. There are two types of boosting:

documents.png

The transformation from documents to tokens is done by analyzers in Lucene. We will cover this in later posts.

Lucene provides the following core classes to support indexing and searching

Indexing

Click here or the image to get PDF version of the mind map.

Lucene_Indexing.png

----- END -----

Welcome to join reddit self-learning community.
Send me a message Subscribe to blog updates

Want some fun stuff?

/static/shopping_demo.png