Sharding & Horizontal Scale

When to shard

Sharding splits data across multiple servers (shards) so no single machine holds the entire data set. Use sharding when vertical scaling (bigger RAM, faster disk) is no longer enough — typically collections exceeding hundreds of GB or write throughput beyond one replica set.

Most apps never need sharding. A well-indexed replica set on modern hardware handles terabytes for many workloads. Shard when metrics prove you have outgrown a single replica set.

Real-life example: Sharding is opening multiple checkout counters at a supermarket — each counter (shard) handles a portion of customers so no single line blocks everyone.

Sharded cluster components

• mongos — query router. Your application connects to mongos, not individual shards. mongos routes queries to the correct shard(s).
• Config servers — store cluster metadata: which chunk lives on which shard, shard key ranges, balancer state. Deploy as a replica set (CSRS).
• Shards — each shard is a replica set holding a subset of data. Minimum production setup: 3 config servers + 3 shards × 3 nodes each.
• Chunks — contiguous ranges of shard key values. Balancer migrates chunks between shards to keep data evenly distributed.

Shard keys — the most important decision

The shard key determines how data is distributed. Choose a field (or compound fields) with high cardinality and even distribution. A bad shard key causes hot shards — one server gets all writes while others sit idle.

Real-life example: Sharding by customerId spreads orders across shards evenly. Sharding by orderDate alone sends all today's orders to one shard — a traffic jam on one lane.

sh.enableSharding("rishtaara")

// Hashed shard key — good even distribution for ObjectId-like fields
sh.shardCollection("rishtaara.events", { userId: "hashed" })

// Compound ranged shard key — tenantId + createdAt for multi-tenant SaaS
sh.shardCollection("rishtaara.logs", { tenantId: 1, createdAt: 1 })

Hashed vs ranged sharding

• Hashed sharding — MongoDB hashes the shard key value. Even distribution even if keys are sequential (timestamps, auto-increments). Queries must include the full shard key or mongos broadcasts to all shards (scatter-gather).
• Ranged sharding — data partitioned by key ranges. Efficient range queries (date ranges, alphabetical) if the query includes the shard key prefix. Risk of hot spots with monotonically increasing keys.
• Compound shard keys — combine a high-cardinality prefix (tenantId, userId) with a range suffix (createdAt) for SaaS and time-series patterns.
• Immutable shard keys — you cannot change a document's shard key value after insert (MongoDB 5.0+ allows with certain restrictions — still avoid changing shard keys in design).

// TARGETED — includes shard key userId; hits one shard
db.events.find({ userId: ObjectId("665a..."), type: "click" })

// SCATTER-GATHER — no shard key; mongos queries ALL shards
db.events.find({ type: "click", createdAt: { $gte: ISODate("2024-06-01") } })

// Always design queries to include the shard key when possible

Deployment architecture notes

Run at least one mongos per application tier or use a load balancer in front of multiple mongos instances. Config servers need low-latency connectivity to all shards.

Atlas Global Clusters automate geo-sharding — data for Indian users lives in Mumbai, EU users in Frankfurt, with zone-aware shard keys.

Real-life example: Production sharded clusters are airports with multiple terminals (shards), a central flight board (config servers), and information desks (mongos) that direct passengers to the right gate.

• Test shard key choice with simulated load before committing — resharding is possible but disruptive.
• Avoid sharding collections smaller than 100 GB — overhead exceeds benefit.
• Monitor chunk migration and balancer activity — uneven chunk counts signal a bad key.
• Use zone sharding to pin data to specific regions for data residency (GDPR, India DPDP).