DesignFlix 12: What Heartbeats and Health Checks Actually Tell You

The process is running. The health endpoint returns 200. Users still can't complete a request.

That isn't necessarily a contradiction. A signal can tell us one part of the system is responding without proving the feature works. Heartbeats and health checks are useful when we know what each one actually measures.

Let's compare them through a worker and a web service, then decide what should happen when their signals stop.

A heartbeat reports back periodically

Suppose an export worker sends a heartbeat to a collector every few seconds. The collector records when it last heard from the worker. If no fresh signal arrives within the chosen interval, it flags a problem.

That tells us the collector hasn't heard from the worker. It doesn't tell us whether the process died, the network dropped the message, or the worker paused for longer than expected.

A heartbeat can also carry progress, such as the last completed job or current lease. Then we can distinguish "still reporting" from "still getting work done." Don't claim progress unless the signal actually measures it.

Kubernetes node heartbeats are one concrete use of periodic status reporting. The broader idea isn't limited to a single central monitor or one particular message format.

A health check asks a particular question

A checker may connect to a web service and call /ready. That is a pull-style check: the observer asks, and the service responds.

But what does /ready test? It might confirm that the process accepts HTTP requests, or it might also verify a dependency needed to serve them. A TCP connection, an HTTP response and a successfully completed business request provide different evidence.

Write the meaning of each check down. "Can this instance accept new requests?" is a useful readiness question. "Did one customer successfully generate a report?" is a different end-to-end question, usually measured elsewhere.

All izz well... allegedly.

Readiness and liveness trigger different actions

In Kubernetes, readiness helps determine whether a container should receive service traffic. Liveness can cause the container to restart after a configured failure threshold. A startup probe can protect slow-starting applications from premature liveness checks. The probe documentation explains those actions.

Consider a web service whose database is temporarily unavailable. It may be unable to serve database-dependent requests, so a readiness failure can be appropriate. Restarting every application instance may not fix the database and can make recovery harder.

Liveness should identify a condition for which restarting this container is a useful response. Readiness should describe its ability to take the traffic assigned to it. They can share some implementation, but shouldn't share a meaning merely because both return HTTP statuses.

A healthy reply can miss a broken path

The worker's heartbeat loop might keep running while job processing is stuck. The web service's health route might return 200 while its main request handler cannot obtain a database connection.

What work could be broken while this particular check still passes?

Use that question to improve the signal, or pair it with another measurement. Job age, completion rate, request failures and synthetic checks can reveal problems a shallow process check won't catch.

Avoid making every probe execute an expensive full workflow, though. Monitoring traffic can become load of its own. Choose a cheap local signal for routing or restart decisions, and use separate end-to-end checks where their cost and side effects are controlled.

Choose how much missed evidence is enough

One late heartbeat or slow response can be a temporary pause. Acting immediately on every miss can create repeated removals and restarts in an otherwise recoverable system.

Choose intervals, timeouts and consecutive-failure thresholds around the expected pauses and the consequence of waiting. Too slow leaves failures undetected longer. Too aggressive creates false alarms and unnecessary recovery actions.

Watch the collector and checker too. If the monitoring path fails, many healthy workers may appear absent together. Compare the signal with other evidence before turning a shared observer failure into a fleet-wide restart.

Match the signal to the action

Heartbeats are useful for periodic presence and progress reporting. Health checks are useful for asking specific questions from an observer's position. Either can carry more or less detail depending on its design; push versus pull isn't a universal ranking of accuracy or cost.

For each signal, define what it proves, what it can miss, and what action a failure should trigger. Then test a stuck worker, a broken dependency and a failed monitoring path. A health design is doing useful work when those different failures lead to appropriate responses.