It Is Always DNS: Why the Joke Keeps Being Right

The joke survives because DNS sits in front of everything, fails in ways that do not look like DNS failures, and remembers its mistakes for as long as the TTL says. It is not that DNS is unreliable. It is that DNS is the first dependency of every request, and a caching layer that fails softly.

Every connection begins with turning a name into an address. Before TLS, before HTTP, before your application code, something resolved a hostname. When that step is slow, wrong, or serving something stale, the symptom appears wherever the request eventually fails — in a timeout, a certificate error, a connection refused — and the resolution step is the last place anyone looks.

Table of contents

• Reason one: it is upstream of everything
• Reason two: it fails as a delay, not an error
• Reason three: caching makes mistakes persistent
• Reason four: nobody owns it
• The checks that resolve it fastest
• It is not always DNS
• How this fits the rest of the stack
• FAQ

Reason one: it is upstream of everything

There is no request that skips it. Your application calling a database by hostname, your load balancer health-checking a backend, your CI pulling a package, your monitoring reaching an endpoint — every one starts with a lookup.

That makes DNS the only component that can plausibly be implicated in every incident. When six unrelated services degrade at once, the shared dependency is almost never all six services having independent problems. It is something underneath them, and DNS is underneath all of them.

It also means the blast radius of a DNS problem does not match its cause. One wrong record, one expired domain, one resolver misbehaving, and the visible failure is spread across everything that happened to look up that name — which is why the initial reports describe a dozen different symptoms.

Reason two: it fails as a delay, not an error

This is the property that makes DNS problems hard to recognise. A failed lookup does not usually return an error immediately. It times out, retries against the next configured resolver, times out again, and eventually gives up.

Default resolver timeouts are measured in seconds, and there are typically several attempts. So a DNS problem presents as everything is slow, which is the least specific symptom in operations and sends people to check CPU, memory, database load, and network throughput — all of which look fine, because they are.

It gets worse in connection-per-request architectures. A service resolving a hostname on every outbound call, with a five second resolution delay, has a five second penalty on every single request. Latency graphs show a cliff, error rates stay near zero, and nothing in the application changed.

# time the resolution specifically, apart from the request
time dig +short api.example.com
time getent hosts api.example.com

# compare against connecting by address directly
curl -o /dev/null -s -w 'dns=%{time_namelookup} connect=%{time_connect} total=%{time_total}\n' \
  https://api.example.com

That last command is the one to keep. time_namelookup isolates the resolution portion of a request. If it is 5 seconds and time_connect is 5.02, you have your answer in one command and you did not have to guess.

Reason three: caching makes mistakes persistent

DNS caches at every layer — the application, the operating system, the local resolver, the recursive resolver, and the browser. Each holds an answer for the TTL that came with it.

The consequences are the ones people describe as propagation:

• A record you fixed is still being served from a cache with an hour left on its TTL.
• It works from your machine and not your colleague’s, because you have different caches in different states.
• It works in the browser and not from the server, or the reverse.
• A negative answer gets cached too — NXDOMAIN has its own TTL, so a name that did not exist when a resolver first asked will keep not existing for a while after you create it.

That last one catches people during a launch. You create the record, test immediately, get NXDOMAIN, create it again, and conclude something is broken. Nothing is broken; a resolver cached the negative answer from your first premature test.

The fix is procedural and it happens before the change: lower the TTL well in advance. Drop it to 300 seconds a day or two before a planned move, make the change, confirm it, and raise it afterwards. One extra edit, and the entire waiting problem disappears.

Reason four: nobody owns it

The opinion, and the real explanation for why it keeps happening.

DNS records are usually spread across systems with different owners. The registrar account was created by whoever set up the company. The nameservers might be at a different provider. Some records are managed in a web console, some in Terraform, some by a certificate tool that adds and removes validation records automatically.

There is frequently no single place to see all of it and no review process for changes. Someone edits a record in a console at four in the afternoon, and there is no diff, no approval, no history beyond whatever the provider logs.

Meanwhile domain and certificate renewals are attached to a credit card and an email address that may both predate half the current team. An expired domain is not a technical failure — it is an ownership failure that presents as one.

The fixes are organisational rather than technical: put records in version control with the rest of your infrastructure, restrict who can edit them in the console, monitor domain and certificate expiry with alerts that go to a team rather than a person, and write down who owns the registrar account.

The checks that resolve it fastest

1. Query an authoritative nameserver directly. dig @ns1.provider.com example.com. This is ground truth with no cache in the way. If the authoritative answer is right and a public resolver is wrong, you are waiting on a TTL. If the authoritative answer is wrong, the change did not save.
2. Compare several public resolvers. dig @1.1.1.1, @8.8.8.8, @9.9.9.9. Disagreement is a propagation window; agreement on the wrong answer is a record problem.
3. Check the TTL on the answer. It tells you how long the wrong value will persist.
4. Trace the delegation. dig +trace example.com walks from the root and shows exactly where a broken delegation is.
5. Check the domain’s expiry. whois. Free, instant, and occasionally the whole answer.
6. Check locally. Your own OS and browser cache separately from the resolver, so flush them before blaming anything upstream.
7. Time the lookup. curl -w 'dns=%{time_namelookup}' to confirm whether resolution is the slow part.

Steps one and seven are the highest-value. The first separates my change is wrong from my change is waiting, which are entirely different problems. The last separates DNS is slow from something else is slow, which is the ambiguity that makes these incidents long.

It is not always DNS

The counterpoint deserves a hearing. DNS gets blamed disproportionately partly because it is a satisfying answer — it explains a broad, vague failure with a single cause, and there is a shared joke ready to endorse it.

The failures that get misattributed most often:

• Certificate expiry. Resolution is fine; the connection is refused at the TLS layer. Different fix entirely.
• Connection pool exhaustion, which also presents as everything is slow.
• A resolver’s limits rather than DNS itself — rate limiting or a misconfigured search domain causing every lookup to try several suffixes first.
• Service discovery in a cluster, which uses DNS as a transport but whose real failure is the registry behind it.

So the discipline is to check DNS early because it is cheap to check, and then move on if it is clean. Two commands rule it in or out. The mistake is not suspecting DNS; it is staying on DNS after dig gave you a correct answer in twenty milliseconds.

How this fits the rest of the stack

The recurring theme is ownership: expired domains, records nobody can find, and certificates renewed by whoever remembered. Consolidating that helps more than any tooling — on RunxBuild, adding a custom domain to a site or service tells you exactly which record to create, and the certificate is issued and renewed once the record points at us, so one of the two recurring expiry problems stops being yours. Custom domains and certificates on RunxBuild covers what to add and in what order. When you are planning a migration and want the DNS cutover to be the only variable, the RunxBuild hosting calculator prices the static site, the service, the database, and bandwidth separately so the rest of the move is already settled.

Useful related references:

• Does Microsoft Edge Use Its Own DNS Server? What Secure DNS Actually Changes
• DNS Resolution Error: 6 Causes and the Right Fix
• What Does Flushing DNS Do? Clearing the Local Cache of Stale Answers
• Custom domains and certificates on RunxBuild

FAQ

Why is it always DNS?

Because DNS is the first dependency of every request, so it can plausibly affect everything at once. It also fails as a timeout rather than an error, which presents as general slowness, and it caches its answers so mistakes persist for the length of the TTL after being fixed.

How do I tell if DNS is actually the problem?

Time the lookup separately: curl -o /dev/null -s -w 'dns=%{time_namelookup} connect=%{time_connect} total=%{time_total}\n' https://example.com. If time_namelookup accounts for most of the total, it is DNS. If it is a few milliseconds, look elsewhere.

What is DNS propagation really?

Not propagation at all — it is many independent caches expiring at different times. Each resolver holds a record for the TTL published when it looked it up. Lowering the TTL to around 300 seconds a day or two before a planned change is what shortens the window.

Why does a new DNS record still return NXDOMAIN?

A negative answer was cached. NXDOMAIN responses have their own TTL, so if a resolver was asked for the name before you created the record, it will keep returning NXDOMAIN until that cached negative expires. Testing a name before creating it is how people cause this.

What should I check first in a suspected DNS incident?

Query an authoritative nameserver directly with dig @ns1.provider.com example.com. That bypasses every cache and tells you whether your change saved correctly. If authoritative is right and public resolvers are wrong, you are simply waiting on TTLs.