System design is the practice of defining the architecture, components, and interactions of a software system to meet specific requirements. It's what happens between "we should build this" and "let's start coding" — the thinking that determines whether a system holds together under real load or falls apart when it matters.
For developers earlier in their career, system design can feel intimidating. It involves concepts that aren't taught in most programming courses: how databases scale, when to use a cache, what a load balancer actually does, why distributed systems are hard.
This guide demystifies the core concepts. You don't need to be a staff engineer at Google to think in these terms — you just need the vocabulary and the mental models.
What System Design Is Actually About
System design is fundamentally about making deliberate choices between trade-offs. Almost every system design decision is a trade-off:
- Consistency vs. availability
- Read performance vs. write performance
- Simplicity vs. flexibility
- Cost vs. capability
There's rarely a single right answer. The skill is knowing what trade-offs each choice involves, and making the choice that fits your specific requirements.
Core Concept 1: Scalability
Scalability is a system's ability to handle increased load — more users, more data, more requests — without a degradation in performance.
There are two fundamental approaches to scaling:
Vertical scaling (scaling up): Give your existing server more resources — more CPU, more RAM, more storage. Simple, requires no architectural changes. Limits: there's a maximum size for a single machine, and it's expensive.
Horizontal scaling (scaling out): Add more servers. Distribute the load across multiple machines. Can scale nearly indefinitely, but requires the system to be stateless (no server stores session data locally) and needs a load balancer to distribute traffic.
Most systems start with vertical scaling (it's simpler) and move to horizontal scaling when vertical has been maxed out or becomes cost-prohibitive.
Core Concept 2: Load Balancing
A load balancer sits in front of a group of servers and distributes incoming requests across them. Without a load balancer, all traffic would hit a single server — that server becomes both a performance bottleneck and a single point of failure.
Common load balancing strategies:
- Round robin: Requests are distributed to servers in rotation (server 1, server 2, server 3, server 1, ...). Simple and effective when servers have similar capacity.
- Least connections: Requests go to the server with the fewest active connections. Better when requests vary significantly in processing time.
- IP hash: The client's IP address determines which server handles its requests. Useful when session affinity (always hitting the same server) is required.
Examples: Nginx, AWS Elastic Load Balancer, Cloudflare, HAProxy.
Core Concept 3: Databases and How They Scale
Databases are typically the first bottleneck in a growing system. Understanding how they scale is essential.
Read replicas: Most web apps have far more reads than writes. A read replica is a copy of the database that can serve read queries, freeing the primary database to handle writes. Many managed database providers (RDS, Supabase) make this straightforward to set up.
Database sharding: Splitting data across multiple databases horizontally. Each "shard" holds a subset of the data (e.g., users A–M on shard 1, N–Z on shard 2). Dramatically increases write capacity. Also dramatically increases complexity. Only worth it at very large scale.
Connection pooling: Each database connection consumes resources. A connection pool maintains a set of reusable connections, reducing the overhead of creating new connections on every request. PgBouncer is the standard PostgreSQL connection pooler.
Indexing: Indexes dramatically speed up read queries by allowing the database to find rows without scanning the entire table. The downside is that indexes slow down writes and consume disk space. Index columns that appear in WHERE clauses, JOIN conditions, and ORDER BY clauses.
Core Concept 4: Caching
Caching stores the results of expensive operations so they can be served quickly without repeating the work.
Why caching works: Many requests ask for the same data. If 10,000 users per minute are all requesting the same homepage content, serving it from a cache (which returns in milliseconds) instead of recomputing it from the database every time reduces database load by orders of magnitude.
Levels of caching:
- Database query cache: Cache the result of expensive database queries. Redis is the standard tool. Set a TTL (time-to-live) that matches how often the data changes.
- Application-level cache: Cache computed results in memory within your application server. Fast but limited to a single server instance.
- CDN (Content Delivery Network): Cache static assets (JavaScript, CSS, images) and sometimes API responses at edge servers geographically close to users. Cloudflare, CloudFront, Fastly.
- Browser cache: HTTP cache headers instruct browsers to store assets locally and not re-request them on subsequent page loads.
Cache invalidation is one of the genuinely hard problems in distributed systems. When cached data becomes stale, the cache must be updated or cleared. Common strategies: TTL expiration (let it expire naturally), explicit invalidation (clear the cache when the underlying data changes), and cache-aside (application checks cache, fetches from DB on miss, populates cache).
Core Concept 5: Message Queues and Asynchronous Processing
Not every operation needs to happen synchronously — in the same request-response cycle that triggered it.
Sending an email when a user signs up doesn't need to happen before the user gets their response. Processing an uploaded video into multiple resolutions could take minutes. Syncing data to a third-party system can tolerate a few seconds of delay.
Message queues (also called job queues) let you defer this work:
- Request comes in (user signs up)
- Application enqueues a job ("send welcome email to user@example.com")
- Application immediately returns a response to the user
- A background worker picks up the job and processes it (sends the email)
This pattern improves response times, increases resilience (if the email service is down, jobs queue up and retry), and allows work to be distributed across multiple workers.
Common tools: BullMQ (Node.js, Redis-backed), Celery (Python, Redis or RabbitMQ), Sidekiq (Ruby), Temporal (any language, more powerful).
Core Concept 6: APIs and Service Communication
How different parts of a system communicate is a fundamental design decision.
REST (Representational State Transfer): The dominant pattern for web APIs. Uses HTTP methods (GET, POST, PUT, DELETE) on resource-oriented URLs. Stateless, cacheable, well-understood. The right default for most new systems.
GraphQL: Clients request exactly the data they need. Reduces over-fetching and under-fetching. More complex to implement server-side. Worth it when you have multiple clients (mobile, web) with very different data needs.
gRPC: High-performance, binary protocol. Used for service-to-service communication where latency matters. Common in microservices architectures at companies like Netflix and Uber.
WebSockets: For real-time, bidirectional communication. Use when you need the server to push updates to the client (chat applications, live notifications, collaborative tools).
Core Concept 7: Data Consistency and the CAP Theorem
The CAP theorem states that a distributed system can guarantee at most two of these three properties simultaneously:
- Consistency: Every read sees the most recent write
- Availability: Every request receives a response (even if it might not have the latest data)
- Partition tolerance: The system continues operating even if network partitions separate nodes
Since network partitions happen in real systems, the practical choice is usually between Consistency and Availability.
In practice: most web applications choose availability over strict consistency, accepting "eventual consistency" — the data will become consistent, but there may be a brief window where different users see different data. This is acceptable for most use cases (a user's feed doesn't need to be perfectly consistent millisecond-to-millisecond) and wrong for others (financial transactions must be consistent).
Core Concept 8: Microservices vs. Monolith
Monolith: One application with all functionality together. Simpler to develop, test, and deploy. The right starting point for almost all products.
Microservices: Each major capability is a separate, independently deployable service. Allows different teams to develop and scale services independently. Introduces distributed system complexity: service discovery, inter-service communication, distributed tracing, data consistency across services.
The pragmatic advice: start with a monolith. Break out services only when you have a specific, proven reason — a service that needs to be scaled independently, a team boundary that makes code sharing painful, or a technology difference (e.g., an ML service that needs Python while the rest of the app uses Node.js).
Most companies that adopted microservices early have spent years consolidating back into more manageable structures. Amazon, Netflix, and Google did build microservices — after years of operating a monolith at scale.
Putting It Together: A System Design Checklist
When designing any new system, work through these questions:
- What are the scale requirements? How many users, requests per second, GB of data?
- What's the read/write ratio? Mostly reads (add caching, read replicas)? Mostly writes (different architecture)?
- What are the consistency requirements? Must all users always see the same data, or is eventual consistency acceptable?
- What operations are performance-critical? These drive your caching and indexing strategy.
- What work can be done asynchronously? These operations become background jobs.
- What are the single points of failure? Add redundancy to eliminate them.
- How will you know when something breaks? Logging, error tracking, and alerting are not optional.
Blueprint AI generates a complete system architecture document for any product — covering exactly these questions, tailored to your specific app type and requirements. Instead of working through this from scratch, you get a concrete starting point to refine with your team.