What is a database health check?
A database health check is a defined test or evidence set used to decide whether a database service can safely perform an intended workload operation now. The right check may test process liveness, fresh connection readiness, authentication, a minimal query, transaction behavior, replica freshness, storage, capacity or recovery. A single Boolean cannot explain every layer, so publish scoped results and stage-specific evidence.
Health is different from availability. Health signals describe current internal or end-to-end conditions; availability measures whether valid demand receives the required outcome over time. Use health checks to support routing and diagnosis, but use workload SLIs to measure service impact.
Database health check levels: liveness, readiness and diagnostics
Choose the cheapest check that supports the decision. Running a complex diagnostic query for every load balancer probe can overload the database; using a process check to route traffic can send users to a service that cannot authenticate or query.
| Level | Decision supported | Recommended scope |
|---|---|---|
| Liveness | Should the process or container be restarted? | Local process event loop or engine response; no external dependency when possible. |
| Readiness | Should this instance receive new traffic? | Fresh dependency access needed for the workload, with tight budget and clear failure policy. |
| End-to-end synthetic | Can a representative identity complete a minimal business-safe operation? | DNS through query result from representative network paths. |
| Deep diagnostic | Why is service degraded and what action is safe? | Capacity, locks, storage, replication, pools, plans and change evidence. |
Database liveness versus readiness
Liveness asks whether the checked component is responsive enough to remain running. Readiness asks whether it should receive new workload traffic. If readiness failure automatically restarts a healthy process during a downstream database outage, restart storms can make recovery worse. Define separate handlers, thresholds and actions, and document which dependencies each check includes.
Why a port check is not a complete database health check
A TCP connection proves only network-layer acceptance. It does not prove TLS trust, authentication, database selection, permissions, query execution, replication freshness, read/write state, storage capacity or acceptable latency. Keep the port result as one stage, then use a protocol-aware client and least-privilege health identity for deeper validation.
Design a safe database health check query
What should a database health check query do?
A minimal readiness query should verify the required connection path and enough engine behavior to support routing. Use a deterministic, low-cost, read-only operation with a strict timeout and a dedicated least-privilege identity. If the workload requires a particular database, schema, replica freshness or transaction mode, the check must validate that requirement rather than querying an unrelated system catalog.
Avoid health check queries that create load or lock contention
Do not scan large tables, sort unbounded data, update shared rows, hold transactions, require unavailable indexes or run at high frequency from every replica. Estimate concurrency: a cheap query multiplied by hundreds of instances and short intervals can become material load. Add jitter, caching only where decision semantics allow, strict cancellation, rate limits and monitoring of health-check resource use.
Should a database health check perform a write?
A write probe can validate the writer path, transactions and durable storage but creates data, locks and failure complexity. Use it only when write readiness cannot be inferred safely, in a dedicated bounded object with idempotent cleanup and explicit retention. Never use customer tables or a broad account. Monitor duplicate or orphaned records after timeouts because the write may commit even when the client does not receive confirmation.
Database health check metrics and diagnostic signals
| Area | Leading signals | User-impact signal |
|---|---|---|
| Connections and pools | Active/idle count, waiters, churn, lifetime and acquisition time. | Fresh login failure, pool timeout, rejected connection. |
| Queries and locks | Queue depth, long transactions, blocked sessions and plan changes. | Error, timeout, wrong result or latency SLO miss. |
| Storage and logs | Free space, IOPS, latency, log growth and checkpoint pressure. | Read/write failure, commit delay or read-only transition. |
| Replication | Lag, queue, replay state, quorum and replica health. | Stale read, unsafe failover or unavailable writer. |
| Capacity | CPU, memory, cache, workers, saturation and headroom. | Tail latency, throttling, admission rejection or outage. |
Use trends and baselines instead of universal thresholds
A connection count, cache ratio or replication delay can be healthy for one workload and dangerous for another. Define thresholds from capacity tests, workload patterns, failover requirements and business deadlines. Use rate and duration conditions, segment by role and environment, and review thresholds after architecture or traffic changes. Pair leading indicators with user-impact evidence to reduce noise.
Replication health must include data freshness
A replica process can be connected and streaming while replay is delayed beyond the workload contract. Measure the relevant lag dimension—time, bytes, log position or business watermark—and distinguish receive, apply and query visibility. A read endpoint should be marked degraded or removed from traffic when freshness or consistency requirements are not met.
What a database health check report should include
Database health check report template
Start with scope and decision: engine, environment, topology, endpoint roles, workload, time window, evidence sources and limitations. Summarize status by layer, observed impact, trend and confidence. For each finding, distinguish observation from inference; include supporting measurements, risk, recommended action, owner, priority, rollback or safety condition and revalidation step.
- Executive result: healthy, degraded, unavailable or unknown for a defined operation.
- Evidence: time, source, endpoint, query, metrics, logs and comparison baseline.
- Findings: severity, affected workload, causal confidence and alternative explanations.
- Action: owner, change, safety checks, expected effect, validation and deadline.
Avoid declaring the database healthy from one green check
State exactly what was tested: “fresh TLS-authenticated read from workload A to reader endpoint B completed within the threshold” is useful; “database healthy” is too broad. Include untested layers, data freshness, writer state, capacity, redundancy, time window and sources. Health can change immediately after the observation, so attach time and recurrence.
Automate database health checks without creating outages
Use least-privilege health identities and safe secret delivery
Give synthetic checks a dedicated, attributable identity with only the database, schema and operation needed. Store and deliver credentials through the approved secret or workload identity system, rotate them, and alert on unexpected use. A shared administrator account makes a probe dangerous and prevents attribution. Do not log the connection string or query data.
Prevent retries and probes from amplifying failure
Set strict timeouts, bounded retries with backoff and jitter, concurrency limits and circuit behavior. Coordinate probe frequency across replicas and regions. During an outage, thousands of health checks and reconnects can consume connection slots, CPU and logs needed for recovery. Monitoring should degrade gracefully and preserve evidence without becoming the dominant workload.
Do not couple every health signal to an automatic restart
A restart is appropriate for some local liveness failures but can worsen downstream database, DNS, identity or network incidents. Map each signal to a safe action: remove from traffic, reduce load, open an incident, refresh a dependency, fail over under controlled criteria, or restart only the unhealthy component. Test automation under partial and dependency failures.
Troubleshoot a failed database health check
Health check fails but application traffic works
The probe can use a different identity, source network, DNS resolver, endpoint, database, query, TLS trust store or timeout. It may require a privilege users do not need, or it may open fresh sessions while applications reuse a pool. Compare effective context and avoid weakening production controls to satisfy a broken probe. Fix the check so it represents the decision.
Health check passes but users report database errors
The check may be too shallow, query another database, miss a tenant or shard, ignore tail latency, use an administrator identity, read a fresh replica while writers fail, or run from a privileged network. Segment user errors, replay a sanitized representative operation, compare endpoints and identities, and expand the check only when it remains safe and decision-relevant.
Database health check timeout
Identify whether time is spent in DNS, TCP, TLS, authentication, pool acquisition, query execution, lock wait or result processing. Capture stage timing and cancellation behavior. A larger timeout may reduce false alarms but delay traffic removal and consume resources; tune it against normal and failure distributions, not one slow sample.
A repeatable database health check workflow
Design from the decision backward, validate in a safe environment, then monitor both the check and its resource cost.
- Define the decision and scopeName the workload, endpoint role, operation, freshness, latency, environment and action driven by the result.
- Choose the check levelSeparate liveness, readiness, end-to-end synthetic validation and deep diagnostics.
- Create a safe identity and queryUse least privilege, approved secret delivery, deterministic low-cost operations and no customer data.
- Set budgets and failure behaviorDefine timeout, retries, jitter, concurrency, thresholds, state transitions and safe automatic actions.
- Test positive and negative casesValidate normal service, blocked network, TLS failure, invalid identity, replica lag, saturation and recovery.
- Report evidence and maintain the checkRecord scope, time, result, stage, cost and limitations; review after topology, driver, schema or workload changes.
Prepare a database health check for compatibility review
Prepare sanitized engine, topology, endpoint role, driver, source, TLS and authentication method, check level, query purpose, timeout, pool behavior, replica freshness, observed metrics, exact failing stage and recent changes. Do not include credentials, full connection strings or customer data.
Review database health check compatibility
Use the InfiniSynapse DB Compatibility Checker to organize endpoint, driver, TLS and engine questions from sanitized evidence. Treat the output as guidance, then validate the check's real decision, workload path, resource cost and failure behavior.
Open DB Compatibility CheckerDatabase health check FAQ
What is a database health check?
A database health check is a defined test or evidence set used to decide whether a database service can safely perform an intended workload operation now.
What is the difference between database liveness and readiness?
Liveness asks whether a component should remain running. Readiness asks whether it should receive new workload traffic. They need different scope, thresholds and actions.
What query should a database health check use?
Use a deterministic, low-cost, read-only operation with strict timeout and a dedicated least-privilege identity that validates the database and behavior required by the workload.
Should a database health check perform writes?
Only when write readiness cannot be validated safely another way. Use a dedicated bounded object, least privilege, idempotent cleanup, strict timeout and monitoring for duplicate or orphaned results.
Why can a health check pass while users see database errors?
The check may be too shallow or use another identity, endpoint, database, network, query, shard, replica role or latency threshold. Compare it with representative user operations.
What should a database health check report include?
Include scope, decision, topology, time window, evidence sources, result by layer, impact, observations, inferences, limitations, findings, priority, owner, safe action, validation and deadline.