Reliability
What does it mean for a message to be durable versus delivered?
Workshop recap
BUILD AT SCALE — August 2026
Corgi Cafe · Shoreditch, London
We took a realtime chat app, ran multiple copies of it, deliberately broke live messaging, and used the failure to understand why distributed systems need different architectural choices.
Workshop
Build it. Scale it. Break it.
We started with a small realtime room-based chat application, then changed one assumption at a time.
- 01One server
- 02Multiple servers
- 03Shared database
- 04Local WebSocket connections
- 05Broken cross-instance realtime delivery
- 06Shared event distribution
The point was not to memorise Kubernetes or Kafka. It was to see a system fail, form a hypothesis about why, make an architectural choice and test the same failure again.
What we built
The live workshop network.
Attendees worked with a realtime chat product and followed it from a simple single-service system to a multi-instance architecture.
- React + TypeScript
- Node.js + TypeScript
- WebSockets
- PostgreSQL
- Docker
- Kafka
- Load testing with k6
💬 Join the live workshop chat
We’ll use the shared BUILD AT SCALE chat throughout the session for questions, experiments and results.
The experiment
We scaled the system and broke the product.
With one chat-service process, realtime messaging worked.
Then we ran two chat-service instances with Kafka disabled. Alice could connect to Chat A while Bob connected to Chat B. Alice’s message was stored in shared PostgreSQL, but Bob might not receive it live. After refreshing, the durable message could appear in history.
Alice
Chat A
Chat A
PostgreSQL
Shared history
Shared history
Bob
Chat B
Chat B
How can a message be stored successfully but still fail to arrive live?
PostgreSQL was shared, but active WebSocket connections belonged to individual application processes.
Durable does not mean delivered live.
The architecture choice
Now we had a reason to introduce Kafka.
Not because it is “the thing scalable apps use”, but because each chat instance owned different local WebSocket connections and every instance needed to see the event.
- POST message
- PostgreSQL persistence
- Kafka · chat.messages
- Chat A + Chat B consume
- Each broadcasts to its browsers
3 partitionswithin the topic
Keyed by roomIdfor room-level routing
Distinct groupsone per chat instance
What we were actually learning
The trade-offs behind data-intensive systems.
Scalability
What workload are we scaling: users, connections, messages or rooms?
Distribution
What changes when state that lived in one process is split across several?
Ordering
Kafka orders within a partition; roomId gives us useful room-level routing.
Measurement
“Scalable” means little without load parameters and latency and error measurements.
The workshop’s questions are informed by the systems thinking in Martin Kleppmann’s Designing Data-Intensive Applications.
Keep building
Run the experiment yourself.
The workshop guide walks through the system from the simple version to the multi-instance failure and the architecture choices that follow.