System Design - The Google Cluster Architecture

Subscribe Send me a message home page tags


This post is a reading note of paper Web Search for a Planet: The Google Cluster Architecture. Most of the content are copied directly from the paper.

Introduction

The work done by search engines is computationally intensive. Fortunately, search engines can benefit from request-level parallelism. The application is essentially stateless thus requires little synchronization. This allows the system to process requests independently by allocating the work to different servers.

The paper highlighted two basic insights:

The following general strategy is used in many places in the architecture. This strategy is well-known now.

Serving a Google Query

  1. When a user enters a query to Google, the user's browser first performs a domain name system (DNS) lookup to map www.google.com to a particular IP address.
  2. The user's browser then sends a HTTP request to one of the clusters, and thereafter, the processing of that query is entirely local to that cluster.

This process also follows a general pattern: split, process and combine.

Architecture Diagrams

Overall Architecture

There are three main components in this architecture

The main activity in the index server consists of

The main activity in the document servers is fetching documents from disk.

Note that index servers typically have less disk space than document servers because the former have a more CPU-intensive workload.

overall_architecture.png
Index Related Architecture

The overall index is partitioned so that a single query can use multiple processors. Index is divided into multiple shards and each shard has a pool of machines as the support. A request will be routed to one of the machines in the pool.

index_architecture.png

----- END -----

Welcome to join reddit self-learning community.
Send me a message Subscribe to blog updates

Want some fun stuff?

/static/shopping_demo.png