Routing & Load Balancing — Mixture-of-Experts at Scale

Every networked system begins with a simple question: when someone asks for something, where should the request go? Routing answers that question by choosing a path. Load balancing answers the next question: when several places could serve the request, which one should receive it now? These two ideas sit behind websites, mobile apps, payment systems, internal services, and cloud platforms. They are easy to name, but the details shape reliability, cost, speed, and failure behavior.

Start with a request

Imagine a person opens an app and taps a button. The app needs data from a server. It sends a request across a network, which is just a set of connected machines that agree how to exchange messages. The request does not magically know its destination. It carries addressing information, and many systems along the way make small forwarding decisions.

At the human level, the destination may look like api.example.com. A service called the Domain Name System (DNS) turns that name into an Internet Protocol (IP) address, which is a numeric address used by computers on a network. From there, routers and switches move packets toward that address. A packet is a small piece of a message, with a header that says where it is going and a body that contains part of the data.

When the packet reaches the provider's environment, it may still not be at the final application. It may first hit an edge router, a firewall, a load balancer, an application gateway, or a service mesh proxy. Each layer narrows the question: is this allowed, where should it go, and which healthy target should handle it?

A request is resolved by DNS, forwarded by routers, and finally handed to the service.

Routing is path selection

Routing is the process of deciding the next hop for traffic. A hop is one step from one network device or system to another. The full path from user to service may contain many hops, but each router usually only decides the next one. This local decision-making is what lets very large networks work without every device knowing everything.

Routers use routing tables. A routing table is a set of rules that say, in effect, traffic for this range of addresses should go there. Some routes are static, meaning an operator configured them directly. Others are dynamic, meaning routers learn them from one another through routing protocols. A routing protocol is a rule system that helps routers share what they know about reachable networks.

Routing is not only an internet backbone concern. Inside a cloud or data center, routing decides how traffic moves between virtual networks, subnets, containers, and services. In software systems, the word route can also mean an application rule, such as send requests for /payments to the payments service. The level changes, but the core idea is the same: match a destination or condition, then choose where to send the traffic next.

📌

One word, many layers. A route can mean a network path, a cloud network rule, or an application rule, so always ask which layer is being discussed.

Load balancing is target selection

Load balancing begins when more than one target can serve the same kind of request. A target might be a physical server, a virtual machine, a container, a function, or another load balancer. The load balancer receives traffic on behalf of the service, then forwards each request or connection to one of the available targets.

The simplest reason to load balance is capacity. Ten servers can often handle more work than one. But capacity is only part of the story. Load balancing also improves availability because one failed target does not have to take down the whole service. It can improve maintenance because engineers can remove one target from rotation, update it, test it, and add it back without stopping everything.

A load balancer is not a magic performance machine. It does not make slow code fast, fix an overloaded database, or remove the need to understand bottlenecks. It distributes incoming work according to rules and observations. If all targets are unhealthy or the shared dependency behind them is failing, the load balancer has nowhere good to send traffic.

⚠️

Not a cure-all. A load balancer can avoid a bad target, but it cannot rescue a system when every target depends on the same failing component.

Different layers see different facts

Networks are often described in layers because each layer has a different job and a different view of the traffic. At a lower layer, a device may see addresses and ports but not the meaning of the request. At a higher layer, a gateway may understand that a request is asking for /checkout or carries a particular host name.

Layer 4 load balancing works at the transport layer, commonly using Transmission Control Protocol (TCP) or User Datagram Protocol (UDP). TCP is a protocol that creates a reliable connection between two endpoints. UDP is a protocol that sends messages without setting up that same reliable connection. A Layer 4 load balancer usually chooses a target based on facts such as source address, destination address, port, and connection state.

Layer 7 load balancing works at the application layer. It can make decisions from application data, such as Hypertext Transfer Protocol (HTTP) method, path, headers, cookies, or host name. HTTP is the request and response protocol used by the web and many application programming interfaces, often called APIs. An API is a defined way for software to ask other software to do something.

Layer 4 is often simpler and faster because it inspects less. Layer 7 is often more flexible because it understands more. Neither is universally better. The right layer depends on what decision you need to make and how much complexity you are willing to operate.

Use lower-layer balancing when you mainly need fast connection distribution and do not need to inspect the request content.
Use higher-layer balancing when routing depends on host names, paths, headers, authentication context, or versioned APIs.
Be cautious when terminating Transport Layer Security (TLS), the encryption protocol used to protect data in transit, because the load balancer can then read and must protect sensitive traffic.
Keep mental separation between forwarding a connection, routing a request, and authorizing a user; they are related but not the same control.

Layer 4 balances on addresses and ports; Layer 7 can decide from application data like the URL.

Algorithms are trade-offs, not trivia

A load-balancing algorithm is the rule used to pick a target. Round robin sends traffic to targets in turn. Least connections chooses the target with the fewest active connections. Weighted rules give some targets more traffic than others, often because they are larger or newer. Hash-based rules use a repeatable calculation on some input, such as a client address or session identifier, so similar traffic tends to land on the same target.

The best algorithm depends on the shape of the work. If every request costs about the same and every target is equal, round robin may be fine. If some requests stay open for a long time, least connections may do better. If one target has twice the capacity of another, weights can express that. If a user's session is stored in memory on one target, session affinity may be needed, though it creates its own risk.

Session affinity, sometimes called sticky sessions, means the same client is repeatedly sent to the same target. This can make stateful applications easier to run, but it weakens balancing. One popular customer, one chatty device, or one long-lived session can overload a single target while others sit idle. A more resilient design stores shared session state outside any one target, but that adds another dependency that must be operated well.

⚠️

Stickiness has a price. Session affinity can simplify application design, but it couples users to individual targets and can make failures more visible.

When requests cost the same, round robin is fine; when they vary, least connections balances better.

Health checks decide what is safe

A load balancer needs a way to know whether a target should receive traffic. A health check is a small test sent on a schedule. It might ask whether a server process is listening, whether an HTTP endpoint returns success, or whether the application can reach a key dependency. If the check fails enough times, the target is removed from rotation. If it later passes enough times, it can be added back.

Health checks sound simple, but they are design choices. A shallow check asks whether the machine is alive. A deep check asks whether the service can do useful work. Shallow checks are less likely to create extra load or false alarms, but they may miss real failures. Deep checks catch more problems, but they may depend on databases, caches, identity systems, or third-party services, which can cause large parts of the fleet to be marked unhealthy at once.

Good systems also use draining. Draining means a target stops receiving new work while existing work is allowed to finish. This matters during deployments, scaling events, and maintenance. Without draining, a load balancer may cut off users in the middle of requests or close long-lived connections abruptly.

A load balancer probes each target on a schedule, removing failures and re-adding recovered ones.

Decisions leaders and practitioners actually face

Routing and load balancing are not just implementation details. They are where architecture becomes operational reality. A design that looks clean on a diagram may be expensive to run, hard to debug, or fragile during regional failures. A design that is boring and explicit may be easier to recover at 3 a.m.

One recurring decision is centralization versus locality. A central load balancer or gateway gives teams one place to apply policy, observe traffic, and manage certificates. Local balancing, such as a sidecar proxy near each service instance, can reduce hops and give finer control. Central systems can become bottlenecks or high-value failure points. Local systems can multiply configuration and make behavior harder to reason about.

Another decision is how much intelligence belongs in the network. Smart routing can shift traffic away from bad regions, send premium customers to larger pools, or test a new version with a small percentage of users. But every clever rule is another thing to document, monitor, and verify. When traffic does something surprising, the first question will be which rule made that decision.

Choose the simplest routing layer that can make the decision safely.
Design for failure before choosing products: ask what happens when a target, zone, region, dependency, or load balancer fails.
Make health checks represent user-visible usefulness, but avoid checks that collapse the whole fleet during a shared dependency outage.
Prefer stateless service instances when possible, so traffic can move freely during scaling and recovery.
Record the reason for every traffic rule; undocumented routing logic becomes hidden architecture.

What to observe when things go wrong

When users report slowness or errors, routing and load balancing can either reveal the problem or hide it. A single average latency number is not enough. Latency is the time a request takes to complete, and averages can hide a small group of users having a terrible experience. Percentiles, such as the 95th or 99th percentile, show the slow edge of experience more clearly.

Useful signals include request rate, error rate, latency, active connections, target health, queue depth, retries, timeouts, and saturation. Saturation means a resource is near or beyond its useful limit, such as central processing unit, memory, network bandwidth, file descriptors, or database connections. The goal is not to collect every possible metric. The goal is to connect a user symptom to the place where a decision or bottleneck occurred.

Logs and traces add context that metrics cannot. A log is a recorded event. A trace follows one request across several services. Together, they help answer whether traffic went to the intended target, whether it was retried, whether it crossed regions, and where time was spent. Without this visibility, teams often change routing rules by guesswork, which can make an incident worse.

✅

Make paths visible. A system is easier to operate when every request can be tied to the routing rules and targets that handled it.

Key takeaways

Routing chooses a path; load balancing chooses among eligible targets.
Layer 4 balancing is usually simpler and sees connection details, while Layer 7 balancing is more flexible because it understands application requests.
Load-balancing algorithms encode trade-offs about fairness, capacity, state, and failure behavior.
Health checks and draining are as important as traffic distribution because they decide what is safe to use.
Centralized traffic control improves consistency but can create bottlenecks; distributed control improves locality but adds operational complexity.
Good observability turns routing from hidden magic into explainable system behavior.

Glossary

Route: A rule or chosen path that tells traffic where to go next.
Router: A network device or software component that forwards traffic toward its destination.
Load balancer: A system that receives traffic for a service and distributes it across multiple eligible targets.
Domain Name System: The naming system that translates human-readable names such as api.example.com into network addresses.
Internet Protocol address: A numeric address used to identify a device or service endpoint on an Internet Protocol network.
Health check: A repeated test used to decide whether a target should receive traffic.
Session affinity: A load-balancing behavior that sends the same client or session to the same target repeatedly.
Transport Layer Security: A protocol that encrypts data in transit so other parties on the network cannot easily read or alter it.
Latency: The time it takes for a request to travel through the system and receive a response.