On this page

DENIC .de DNSSEC Outage (May 2026)

On the evening of 5 May 2026, Germany's country-code top-level domain, .de, broke for every user behind a DNSSEC-validating resolver. Google Public DNS, Cloudflare 1.1.1.1 and many ISP resolvers started answering SERVFAIL for .de names, and stayed that way for roughly three hours. The registry, DENIC eG, ran a routine, scheduled DNSSEC key rollover. A bug in in-house code produced three different key pairs on three HSMs where there should have been one key copied to all three. All three HSMs signed the zone, but only one public key, published under key tag 33834, appeared in the .de DNSKEY set. So about two thirds of the signatures in the zone could not be validated (DENIC analysis and addendum, DENIC final report).

It was not an attack. DENIC's final report rules out compromise, a Knot DNS malfunction, an HSM malfunction and a classic key-tag collision. Three continuously running validation tools did detect the problem, but, in DENIC's words, "the notifications were not processed correctly, meaning that no timely intervention took place."

This case study is about the gap between "we have monitoring" and "monitoring stops a bad change". It also shows how DNSSEC failures at the top of the tree spread, and how resolver operators around the world limited the damage within an hour.

TL;DR

  • What broke: The .de zone was published with RRSIG signatures from three different private keys. Only one matching public key (ZSK, key tag 33834) was in the DNSKEY RRset. About one signature in three validated (DENIC).
  • Why everything broke, not just signed domains: The signatures over the zone's NSEC3 records were also bad. Validators need those records to prove that an unsigned delegation is legitimately unsigned. Without that proof they marked delegations bogus, so "even second-level domains for which DNSSEC is not used at all could not be resolved" (DENIC final report).
  • Root cause: A bug in DENIC's in-house rollover agent, which generates key material and loads it into all connected HSMs. It generated a separate key pair per HSM instead of one shared pair. The test environment had one HSM at one site, so the multi-HSM code path never ran in testing or in the "cold" parallel run.
  • Detection failure: Three validation tools flagged the anomaly, but the alerts were not acted on in time.
  • Mitigation came from outside: Large resolver operators, Cloudflare among them, turned off DNSSEC validation for .de with negative trust anchors or equivalent overrides. Cloudflare's override went live at 22:17 UTC (Cloudflare). DENIC began distributing a correct zone at 00:08 and was fully restored by 01:15 local time on 6 May (DENIC press release).
  • What would have prevented it: Verify every signature against the published DNSKEY set before the zone leaves the signer. Test the multi-HSM path, even if only with SoftHSM. Route validation alerts through a dead-man's switch so they cannot be silently dropped. Keep a known-good signed zone ready to switch back to.

The Alert

The first public signal came from users, not from DENIC. heise online reports getting the first reader reports at about 21:50 German time. Anyone querying through a validating resolver got failures for .de names. heise.de itself stayed reachable for some readers only because its address was still in their resolvers' caches (heise).

Operators who ran dig saw the shape of the problem straight away. heise quotes the extended DNS error returned by a validating resolver:

RRSIG with malformed signature found for de/soa (keytag=33834)

DNSViz also showed an invalid signature on the .de SOA record. Cloudflare's write-up quotes the same kind of error, returned as EDE 6 (DNSSEC Bogus) by other resolvers:

EDE: 6 (DNSSEC Bogus): RRSIG with malformed signature found for example.de/nsec3 (keytag=33834)

That second line explains the whole incident. The failing record is not example.de's own data. It is an NSEC3 record in the .de zone, signed by the .de ZSK with tag 33834.

The confusing part

The key tag in the error, 33834, was in the DNSKEY RRset. Operators saw a key that existed, with a tag that matched, and a signature that would not verify. That pattern looks like corruption, not like a missing key. It turned out that three different private keys were all signing under the same advertised tag.

Shortly after 22:55 German time, DENIC posted a "Partial Service Disruption" notice on status.denic.de. At 23:28 it published a statement saying "all DNSSEC-signed .de domains are currently affected in their reachability" and that the cause was not yet known (heise, Born's IT blog). heise pointed out on the night that unsigned .de domains were failing too. DENIC's later analysis confirmed this and explained it through NSEC3.

Impact

DimensionWhat is knownSource
Zone affected.de, the second-largest ccTLD after .cn, with about 17.9 million registrations at the end of Q1 2026Domain Name Wire
Who failedUsers behind validating resolvers, including Google 8.8.8.8, Cloudflare 1.1.1.1 and many ISP resolvers. Non-validating resolvers were unaffectedDENIC final report, heise
Which domainsEffectively all .de names for those users, signed and unsigned, because NSEC3 signatures were invalidDENIC
Duration"approximately three hours" per DENIC. Exact onset is disputed (see the timeline)DENIC
Valid signatures"only about a third of all signatures"DENIC final report
Number of failed queries, users or domainsNot published. Neither DENIC nor Cloudflare gave figures. Cloudflare noted that "the SERVFAIL rate looks more alarming than the actual user impact" because of retries and stale servingCloudflare
Security impactNone. DENIC found no signs of compromise of the signing system or other infrastructureDENIC final report

Numbers you will see elsewhere

Claims of "18 million domains down" or of government domains being singled out are unsourced; CTI Pilot removed both. Every .de name was unresolvable for users of validating resolvers, and nobody has published how many users that was.

Impact was uneven and confusing: the same name could fail on one resolver and work on another, or work until a cached answer expired.

Timeline

DENIC's own reports give no UTC timestamps. The press release gives times with no timezone label, presumably German local time (CEST, UTC+2), and the final report gives none at all. Miek Gieben's critique of the post-mortem calls this out first: "No timeline." The table below shows each time as the source gives it, plus a UTC conversion where the source's timezone is clear or very likely.

sequenceDiagram
    participant Agent as DENIC rollover agent
    participant HSM as Three HSMs at two sites
    participant Zone as .de zone served by Knot
    participant Mon as DENIC validation tools
    participant Res as Validating resolvers
    participant Users as Users

    Note over Agent,HSM: April 2026 — third-generation signer goes live
    Note over Agent,Zone: 5 May — routine ZSK rollover
    Agent->>HSM: Generate key material
    HSM-->>Agent: Three DIFFERENT key pairs, one per HSM
    Agent->>Zone: Publish ONE DNSKEY (tag 33834)
    HSM->>Zone: All three HSMs sign RRsets
    Mon->>Mon: Detects invalid signatures
    Note over Mon: Notifications not processed correctly
    Zone->>Res: About two thirds of RRSIGs fail validation
    Res->>Users: SERVFAIL for .de (as caches expire)
    Note over Res: ~19:30 UTC — Cloudflare sees SERVFAIL rise
    Res->>Res: 22:17 UTC — Cloudflare marks .de insecure
    Zone->>Res: 00:08 local — correct zone distribution begins
    Note over Zone,Users: 01:15 local — normal operation fully restored
Time as publishedLikely UTCEventSource
2011—DNSSEC introduced for .deDENIC final report
April 2026—Third-generation signing system goes into operationDENIC final report
~19:30 UTC, 5 May19:30DENIC "started publishing incorrect DNSSEC signatures" as seen from 1.1.1.1, and SERVFAILs spikeCloudflare
~21:50 (German time), 5 May~19:50heise receives first reader reportsheise
21:57 (no tz label), 5 May19:57 if CESTDENIC's stated start: "a technical incident occurred in the DNS for the .de domain extension from 21:57 onwards"DENIC press release
Shortly after 22:55 (German time)~20:55"Partial Service Disruption" notice on status.denic.deheise
22:17 UTC22:17Cloudflare deploys an override marking .de insecure, which "marked the end of impact for 1.1.1.1"Cloudflare
23:28 (German time)21:28DENIC status statement: cause "not yet fully identified"heise
00:08 (no tz label), 6 May22:08 if CEST"DENIC initiated the distribution of a correct DNS zone"DENIC press release
01:15 (no tz label), 6 May23:15 if CESTOperational status before the outage "fully restored"DENIC press release
14:30, 6 May12:30DENIC update: cause linked to a "routine, scheduled key rollover". Future rollovers suspendedDENIC press release
17:00 UTC, 6 May17:00Cloudflare publishes its resolver-side analysisCloudflare
8 May—DENIC initial analysis: three key pairs, one key tagDENIC analysis
11 May—Addendum: "There was no keytag collision involved." One pair per HSM instead of one pair copied to threeDENIC analysis
11 June—Final report: in-house rollover agent bug, single-HSM test environment, alert handling failureDENIC final report
13 June—Miek Gieben publishes a critique of the post-mortemmiek.nl

Disputed: when did it start?

The sources give at least three different start times.

  • Cloudflare: about 19:30 UTC (21:30 CEST), based on what 1.1.1.1 observed.
  • heise, cited by CTI Pilot: first disruption about 21:43 German time (19:43 UTC). heise's own article says its first reports came in at about 21:50.
  • DENIC: "from 21:57 onwards", with no timezone. Read as CEST, that is 19:57 UTC.

If DENIC's time is CEST, which is the only reading consistent with its own 22:55 status notice and 23:28 statement, Cloudflare observed failures about 27 minutes before DENIC's official start. DENIC has not explained the gap. It might be when the bad signatures first reached the authoritative servers, or when DENIC judged the impact significant, or something else. Duration figures of "three hours" and "three and a half hours" both come from picking different start points.

A second-hand claim, attributed to the firm Blackfort in it-administrator.de, says DENIC fixed the zone at 20:15 UTC with a signing run under a new key tag (32911). That does not match DENIC's own 00:08 or 01:15 timeline and is not confirmed by any DENIC document. Treat it as unverified.

Root Cause

Background: how a TLD signs its zone

The KSK signs only the DNSKEY RRset and is anchored by a DS record in the parent (the root zone). The ZSK signs everything else: SOA, the DS records of signed children, and the NSEC3 records that prove a name or DS record does not exist. A validator reads the RRSIG's key tag, finds the DNSKEY with that tag, and verifies. The key tag is a 16-bit checksum over the DNSKEY RDATA (RFC 4034, Appendix B), a lookup hint rather than an identifier. If no key verifies, the RRset is bogus. Refresher: DNS Deep Dive: DNSSEC.

DENIC signs with Knot DNS, in-house tools and HSMs in two data centres separated geographically and at the network level (DENIC final report). To sign from several HSMs, they all need the same key material. That is the rollover agent's job.

The bug: one key per HSM instead of one key for all HSMs

DENIC's final report describes the agent as in-house software that "generates key material and loads it into all connected HSMs". During the 5 May ZSK rollover it did not generate one key pair and distribute it. Instead, "a separate key pair was generated for each connected HSM and loaded into exactly one of them" (DENIC final report).

flowchart TB
    subgraph intended ["Intended: one key, three copies"]
        A1["Rollover agent"] --> K1["Generate ONE key pair"]
        K1 --> H1["HSM 1: key K"]
        K1 --> H2["HSM 2: key K"]
        K1 --> H3["HSM 3: key K"]
        K1 --> D1["DNSKEY K published (tag 33834)"]
    end
    subgraph actual ["Actual on 5 May 2026"]
        A2["Rollover agent (bug)"] --> G1["Generate key pair A"]
        A2 --> G2["Generate key pair B"]
        A2 --> G3["Generate key pair C"]
        G1 --> J1["HSM 1: key A"]
        G2 --> J2["HSM 2: key B"]
        G3 --> J3["HSM 3: key C"]
        G1 --> D2["Only DNSKEY A published (tag 33834)"]
    end
    style D2 fill:#ff6b6b,color:#fff
    style J2 fill:#cc5500,color:#fff
    style J3 fill:#cc5500,color:#fff

The three key pairs had different key material but the same metadata, including key tag 33834. The initial analysis on 8 May described this as "three different key pairs were generated" for one key tag. The 11 May addendum and final report then corrected the framing: "This was therefore not a classic key tag collision." One of the three was published as the ZSK, on the assumption that all three were identical.

Why would three different keys share one key tag?

A key tag is computed from the public key's RDATA, so three genuinely different public keys would almost always produce three different tags. DENIC says the keys shared metadata including the tag. It has not published code or explained how that happened.

Miek Gieben asks this directly in his critique: different keys "would be definition have different key tags… But not here?" He suggests the tag "was calculated once and reused?" (miek.nl). That remains a hypothesis; DENIC's reports do not confirm it. Either way, every signature carried tag 33834, a validator found one candidate DNSKEY, and only signatures from the HSM holding the matching private key verified.

Why about one signature in three verified

"All three HSMs were used in signing, but only one had a key with a corresponding DNSKEY RR" (DENIC addendum). DENIC does not describe how signing work is split between HSMs, but the result was that roughly a third of the RRSIGs came from the HSM with the published key. The final report says "only about a third of all signatures" were valid. Gieben objects to the report's vaguer word "majority" for the invalid ones: "Majority? How many?"

The .de zone is updated incrementally, not regenerated as a whole. Every zone change increments the SOA serial, so the SOA record is re-signed with every update. Each re-signing could land on any HSM, so the SOA signature flipped between valid and invalid during the incident: "it was at times validatable and at times not validatable" (DENIC), so the same query could give different answers minutes apart.

Why unsigned domains failed too: NSEC3

When a validating resolver follows a referral from .de to example.de, it needs to know whether example.de is signed. If .de holds a DS record for it, the child is signed. If not, .de must prove that no DS exists, and that proof is an NSEC3 record signed by the .de ZSK. Without a valid proof, the resolver cannot tell "this child is unsigned" apart from "an attacker stripped the DS record". DNSSEC's rule is to fail closed.

flowchart TD
    Q["Resolver asks for www.example.de"] --> R["Root: .de is signed (DS for .de)"]
    R --> DE["Query .de servers"]
    DE --> REF["Referral to example.de NS"]
    REF --> DS{"Does .de have a DS for example.de?"}
    DS -->|"Yes: DS RRset + RRSIG"| V1{"RRSIG valid?"}
    DS -->|"No: NSEC3 proof + RRSIG"| V2{"NSEC3 RRSIG valid?"}
    V1 -->|"~1/3 of the time"| SEC["Continue to child (secure)"]
    V1 -->|"~2/3 of the time"| BOGUS["BOGUS → SERVFAIL"]
    V2 -->|"~1/3 of the time"| INS["Child is provably insecure → resolve normally"]
    V2 -->|"~2/3 of the time"| BOGUS
    style BOGUS fill:#ff6b6b,color:#fff

DENIC states this directly: signatures over NSEC3 records could not be validated, so validating resolvers marked delegations as bogus, and "even second-level domains for which DNSSEC is not used at all could not be resolved" (DENIC final report). This is why DENIC's first status message, which said signed domains were affected, was wrong, and why heise was right to contradict it on the night.

Generalise this

A parent zone's signatures protect its children's absence of signatures as well as their presence. Turning DNSSEC on for a TLD makes every delegation under it depend on the TLD signer working, whether or not the registrant ever heard of DNSSEC. The same "parent signs the proof of non-existence" design means a bad signer at the root would break every TLD.

Why testing did not catch it

"The test environment consists of a single HSM at a single location." The bug needs more than one HSM, so it appeared neither in testing nor in the "cold" parallel run (DENIC final report). The system had been externally audited; Born's IT blog, paraphrasing DENIC, says the faulty code was added after testing and audit (borncity.com), which DENIC's English report does not state as clearly. Gieben's response: multiple HSMs "can be faked with SoftHSM", or "You're a TLD, buy another HSM" (miek.nl).

Why it was published anyway

Two things let a broken zone reach the internet.

  1. No full-zone validation gate before publication. Because the zone is updated incrementally "due to its size", complete zone files are not produced for every version (DENIC final report). Validating each version in full before release is harder than in a design that writes a fresh signed zone file each time. One of DENIC's new measures is "partial validation of the zone before deployment". Gieben questions whether incremental updates are still justified at today's zone sizes and hardware.
  2. Alerts that fired into the void. DENIC runs three continuously running checking and validation tools against the production zone. "These systems detected the issues as intended, but the notifications were not processed correctly" (DENIC). DENIC has not said what "not processed correctly" means: a mis-routed page, a suppressed alert, a broken integration, or an alert nobody was on call for. Gieben's reaction ("Holy shit, 'not processed correctly'") is the right one. This is the single most important unanswered question in the post-mortem.
flowchart LR
    BUG["Rollover agent bug"] --> KEYS["3 key pairs, 1 published"]
    KEYS --> SIGN["Zone signed by 3 HSMs"]
    SIGN --> GATE{"Pre-publication validation?"}
    GATE -->|"Not in place"| PUB["Bad zone published to authoritative servers"]
    PUB --> MON{"3 validation tools"}
    MON -->|"Detected"| ALERT["Notifications"]
    ALERT -->|"Not processed correctly"| NONE["No timely intervention"]
    NONE --> OUT["~3h outage for validating resolvers"]
    TEST{"Multi-HSM test?"} -->|"Single HSM only"| BUG
    style GATE fill:#cc5500,color:#fff
    style ALERT fill:#cc5500,color:#fff
    style OUT fill:#ff6b6b,color:#fff

Four layers could each very likely have stopped this or cut it to minutes: unit tests, a multi-HSM test, pre-publication validation and alert response. All four failed or were missing.

Detection

Inside DENIC

DENIC's monitoring worked as a sensor and failed as a system. Three validators saw the bad signatures, but nobody acted on what they saw. DENIC's first status notice (shortly after 22:55 German time) came about an hour after heise's first reader reports and roughly an hour and a half after Cloudflare saw SERVFAILs rise.

At the resolvers

Resolver operators saw the incident as a SERVFAIL spike limited to one TLD. Cloudflare's post adds several details useful to anyone running resolvers:

  • Failures ramped up instead of jumping. Failures "climbed steadily over the following three hours as cached records slowly started expiring." Each name broke only when its cached answer expired and a fresh lookup hit the bad signatures (Cloudflare).
  • Serve-stale hid most of the damage. 1.1.1.1 implements RFC 8767 and kept serving expired .de answers when fresh lookups failed. Overall NOERROR rates stayed fairly stable. Excluding stale answers, NOERROR fell steadily from 19:30 UTC.
  • Retries inflated the volume. Query volume rose noticeably because clients retried failed lookups, often three or more times.
  • Their error code was wrong. Other validators returned EDE 6 (DNSSEC Bogus). 1.1.1.1 returned EDE 22 (No Reachable Authority), which "on the surface suggests a connectivity problem with the upstream nameservers rather than a DNSSEC validation failure." A bug meant the trust-chain verifier generated the Bogus code but never attached it to the response. Cloudflare listed fixing this as its own action item.

Error codes are part of your incident response

An operator who saw EDE 22 from 1.1.1.1 might have spent time chasing network reachability to the .de nameservers. Extended DNS Errors (RFC 8914) exist so that a SERVFAIL can carry its reason. Check that your resolver actually emits the right codes, and that your dashboards break SERVFAILs down by EDE code.

Coordination happened in the DNS-OARC Mattermost, where operators compared observations. Cloudflare says resolver operators across the internet "independently applied Negative Trust Anchors within an hour."

Response

Resolver operators: negative trust anchors

A negative trust anchor (NTA), defined in RFC 7646, tells a validating resolver to treat a zone as unsigned for a limited time. Cloudflare notes that "RFC 7646 explicitly names TLD misconfiguration as the primary use case for NTAs."

  • Cloudflare 1.1.1.1: Its resolver, internally called Big Pineapple, has no native NTA feature. Engineers used an existing override rule to mark .de as an insecure zone. It works like an NTA but is not the RFC mechanism. It went live at 22:17 UTC and ended impact for 1.1.1.1 users. A similar override on Cloudflare's separate internal resolver restored origin connectivity for CDN customers with .de origins (Cloudflare). CTO Dane Knecht announced on X that validation for .de was turned off (heise).
  • Other large operators: DENIC thanked "some operators of large resolvers" who temporarily turned off validation for .de. Which operators, and when, has not been published as a complete list.

The trade-off is explicit. With an NTA in place, .de answers are not DNSSEC-protected, so a spoofing attack against .de names during that window would not be detected. Cloudflare judged this acceptable because the failure was widespread, publicly confirmed and affected all validators equally. In one engineer's words: "There is no user of 1.1.1.1 resolving a .de name right now who would prefer a SERVFAIL over an unvalidated response."

heise suggested that end users switch to non-validating resolvers (Level 3's 4.2.2.1, Quad9's 9.9.9.10). That is an NTA for everything with no expiry, so change it back afterwards.

DENIC

DENIC began distributing a corrected zone at 00:08 and was fully back to normal by 01:15 local time on 6 May (DENIC press release). DENIC has not described how the zone was fixed. It does not say whether it re-signed everything with the one published key, rolled back to the previous ZSK, or published the missing DNSKEYs. The next afternoon it suspended all future rollovers until the cause was found.

Correction to a common framing

Some write-ups call this a KSK rollover or blame "signing with an old key". DENIC's reports describe a ZSK rollover and a new-key generation bug.

Remediation (DENIC's Stated Measures)

From the final report:

MeasureStatus per DENICGieben's view (miek.nl)
Improved code reviewAlready in place—
Additional alerts and metrics, including for the validation toolsShort-termToo vague. Is it only checking after publication?
Faster switchover to a valid backup zoneShort-term"the worst possible action" unless it is tested and exercised regularly
Partial validation of the zone before deploymentShort-term"a sane thing to do"
No further ZSK rollovers until secure development practices and test coverage improveShort-termUnderstandable, but for how long?
External security and process analysisPlannedThe code had already been audited: "'nuff said"
Review incident response process and outage communicationsUnder reviewWanted a real timeline in the report

What Would Have Prevented It

These are concrete controls for anyone who runs a signer: a registry, a large DNS host, or an enterprise with signed zones. Most of them also apply to any system that signs things with keys held in more than one HSM or KMS region, such as code signing, JWT issuers or certificate authorities.

1. Sign, then verify against the published key set

The signer should never emit a signature it has not verified itself against the public DNSKEY set it is about to publish. Not against the key handle it just used, and not against its own metadata. That one check catches this bug no matter what caused it.

# verify_signed_rrsets.py — run on every RRset the signer produces, before it is released.
# Needs: pip install dnspython cryptography
import sys
import dns.dnssec, dns.name, dns.rdataclass, dns.rdatatype, dns.zone, dns.rrset

def verify_zone(zone: dns.zone.Zone, origin: dns.name.Name) -> list[str]:
    """Validate every RRSIG in the zone against the zone's own DNSKEY RRset."""
    apex = zone.get_node(origin)
    dnskeys = apex.get_rdataset(dns.rdataclass.IN, dns.rdatatype.DNSKEY)
    keys = {origin: dnskeys}
    published_tags = {dns.dnssec.key_id(k) for k in dnskeys}
    errors = []
    for name, node in zone.nodes.items():
        abs_name = name.derelativize(origin)
        for rds in node.rdatasets:
            if rds.rdtype == dns.rdatatype.RRSIG:
                continue
            sigs = node.get_rdataset(dns.rdataclass.IN, dns.rdatatype.RRSIG, rds.rdtype)
            if sigs is None:
                continue  # unsigned glue / delegation NS is expected
            for sig in sigs:
                if sig.key_tag not in published_tags:
                    errors.append(f"{abs_name}/{dns.rdatatype.to_text(rds.rdtype)}: tag {sig.key_tag} not in DNSKEY")
            rrset = dns.rrset.from_rdata_list(abs_name, rds.ttl, list(rds))
            sigset = dns.rrset.from_rdata_list(abs_name, sigs.ttl, list(sigs))
            try:
                dns.dnssec.validate(rrset, sigset, keys, origin)
            except dns.dnssec.ValidationFailure as e:
                errors.append(f"{abs_name}/{dns.rdatatype.to_text(rds.rdtype)}: {e}")
    return errors

if __name__ == "__main__":
    origin = dns.name.from_text(sys.argv[2])
    z = dns.zone.from_file(sys.argv[1], origin=origin, relativize=True)
    errs = verify_zone(z, origin)
    for e in errs[:50]:
        print("BOGUS", e)
    print(f"{len(errs)} invalid RRsets")
    sys.exit(1 if errs else 0)  # non-zero exit blocks publication

For an incrementally updated zone, run the same check on every RRSIG in each diff (IXFR) before it goes to the authoritative servers, so the cost scales with the change rather than the zone, plus a periodic full pass. If even that is too slow, sample from every HSM on every release: a one-in-three failure rate shows up within a handful of samples.

2. Use the standard zone verifiers as a release gate

For zones that can be written out in full, the usual tools already do this and are maintained by people who think about DNSSEC all day:

# BIND's verifier: checks every RRset is signed by a key in the DNSKEY set
# and that the chain from KSK to DNSKEY is intact. Exit code != 0 blocks the release.
dnssec-verify -o example.de signed/example.de.zone

# ldns: also checks NSEC/NSEC3 chains and signature validity periods
ldns-verify-zone -V 3 signed/example.de.zone

# Knot DNS's own checker, with DNSSEC checks turned on
kzonecheck --dnssec on -o example.de signed/example.de.zone

Wire one of them into the pipeline so that publishing is impossible without a passing result, not merely recommended.

3. Assert key consistency across HSMs at generation time

The bug was in the step that puts key material on more than one HSM. After that step, check that every HSM holds the same public key, and that the key tag computed from that public key matches the one about to be published.

#!/usr/bin/env bash
# check_hsm_key_consistency.sh LABEL — fail if HSMs disagree on the public key for LABEL.
set -euo pipefail
LABEL="$1"
declare -A FP
for MODULE_SLOT in "/usr/lib/hsm-a.so:0" "/usr/lib/hsm-b.so:0" "/usr/lib/hsm-c.so:0"; do
  MOD="${MODULE_SLOT%%:*}"; SLOT="${MODULE_SLOT##*:}"
  pkcs11-tool --module "$MOD" --slot "$SLOT" --read-object --type pubkey \
              --label "$LABEL" -o /tmp/pub.der
  FP["$MODULE_SLOT"]=$(sha256sum /tmp/pub.der | cut -d' ' -f1)
done
UNIQUE=$(printf '%s\n' "${FP[@]}" | sort -u | wc -l)
if [ "$UNIQUE" -ne 1 ]; then
  echo "FATAL: $UNIQUE distinct public keys for label $LABEL across HSMs"
  printf '%s %s\n' "${!FP[@]}" "${FP[@]}"
  exit 1
fi
echo "OK: all HSMs hold the same public key for $LABEL"

Then compute the key tag from the DNSKEY you are about to publish. Never copy it from metadata:

import dns.dnssec, dns.rdata, dns.rdataclass, dns.rdatatype
dnskey = dns.rdata.from_text(dns.rdataclass.IN, dns.rdatatype.DNSKEY, published_dnskey_text)
assert dns.dnssec.key_id(dnskey) == expected_tag, "key tag does not match published key material"

4. Test the topology you run in production

A single-HSM test environment cannot exercise multi-HSM code. Real HSMs are expensive, but PKCS#11 is a standard interface, so the logic can be tested against software tokens:

# Create three independent software "HSMs" for integration tests
export SOFTHSM2_CONF=$PWD/softhsm2.conf
mkdir -p tokens && echo "directories.tokendir = $PWD/tokens" > softhsm2.conf
for i in 1 2 3; do
  softhsm2-util --init-token --free --label "hsm$i" --so-pin 1234 --pin 1234
done
# Run the rollover agent against all three, then the consistency check from step 3
./rollover-agent --pkcs11-module /usr/lib/softhsm/libsofthsm2.so --tokens hsm1,hsm2,hsm3
./check_hsm_key_consistency.sh zsk-next

This is Gieben's point: make "number of HSMs" a test parameter and run with 1, 2 and N. And a "cold" parallel run only helps if something validates its output.

5. Make validation alerts impossible to lose

DENIC's detectors worked. The failure was between the detector and a human. Two patterns address that.

Page on "zone is bogus" with no deduplication into a low-priority channel, and add a dead-man's switch so that a broken alerting pipeline is itself an alert:

# Prometheus alert rules (illustrative)
groups:
- name: dnssec
  rules:
  - alert: ZoneSignaturesBogus
    expr: dnssec_invalid_rrsig_total{zone="example.de"} > 0
    for: 1m
    labels: { severity: page, team: dns }
    annotations:
      summary: "Invalid RRSIGs published for {{ $labels.zone }}"
      runbook: "https://runbooks.internal/dns/bogus-zone"
  - alert: DnssecValidatorSilent
    # Each validator exports a heartbeat. If any stops reporting, page.
    expr: time() - dnssec_validator_last_run_timestamp_seconds > 300
    labels: { severity: page, team: dns }
  - alert: Watchdog
    # Always firing. An external service pages if it STOPS receiving this.
    expr: vector(1)
    labels: { severity: none }
# Alertmanager: route the watchdog to an external dead-man's-switch service
route:
  receiver: dns-pager
  routes:
  - matchers:
      - 'alertname = "Watchdog"'
    receiver: deadmans-switch
    repeat_interval: 1m
receivers:
- name: dns-pager
  pagerduty_configs: [ { routing_key: "<secret>" } ]
- name: deadmans-switch
  webhook_configs: [ { url: "https://deadman.example.net/ping/dns" } ]

Then test the path end to end. Publish a deliberately broken signature into a staging zone on a schedule and check that a human gets paged. An alert that has never fired in a drill is an assumption, not a control. See Alert Design for routing and severity practice.

6. External validation canaries

Check the zone the way the world sees it, from outside your network, through real validating resolvers:

#!/usr/bin/env bash
# canary-resolve.sh ZONE — query names through validating resolvers, alert on SERVFAIL/bogus.
ZONE="$1"
NAMES=("$ZONE" "nic.$ZONE" "unsigned-canary.$ZONE" "signed-canary.$ZONE")
FAIL=0
for r in 1.1.1.1 8.8.8.8 9.9.9.9 127.0.0.1; do
  for n in "${NAMES[@]}"; do
    # +cd is NOT set, so the resolver validates. Look for SERVFAIL.
    status=$(dig +dnssec +time=2 +tries=1 @"$r" "$n" SOA | awk '/status:/{print $6}' | tr -d ,)
    if [ "$status" = "SERVFAIL" ]; then
      echo "SERVFAIL $n via $r"; FAIL=1
    fi
  done
done
# Local validation with full diagnostics (BIND's delv)
delv @127.0.0.1 "$ZONE" SOA +rtrace >/dev/null 2>&1 || { echo "delv validation failed"; FAIL=1; }
exit $FAIL

Include both a signed and an unsigned canary delegation: the unsigned one is what exercises the NSEC3 proofs that broke here. Public resolvers' caches hide a failure for up to the TTL, so also query uncached (nonexistent) names through your own validating resolver with a small cache.

7. Keep a known-good zone and a rehearsed rollback

The pre-publish ZSK rollover method (RFC 6781 and RFC 7583 cover the timing) keeps the old ZSK in the DNSKEY RRset for a while. As long as the old key is published, re-signing with it is a valid rollback. Make that a single, rehearsed command:

# Illustrative runbook step: re-sign with the previous ZSK and publish the last good zone
rollback-signer --zone example.de --use-key "$PREVIOUS_ZSK_ID" --verify --publish

DENIC's "faster switchover to a valid backup zone" is the same idea. Gieben's warning applies: a switchover that has never been exercised will fail when needed. Run it in drills, on a schedule, against production-sized data.

8. Roll out changes to signing in stages

A new signer generation, or a new rollover agent, is a deployment like any other. Apply canary deployment thinking:

  • Run the new signer first on a low-traffic zone the registry controls, for example a test TLD in a lab root, or an internal zone, through multiple full rollovers, with the production HSM topology.
  • Put a soak period between "new signer live" and "first real rollover on the production zone", with the validators from step 5 watching.
  • Make the first production rollover on the new system a scheduled event with people watching, not routine automation.

9. If you operate a resolver: be ready to add an NTA in minutes

Everyone running a validating resolver should have a tested NTA runbook. The commands differ by product:

# Runtime, no restart. Removed with insecure_remove.
unbound-control insecure_add de
unbound-control flush_zone de
unbound-control flush_bogus
# Permanent config equivalent (unbound.conf) — remember to remove it afterwards:
#   server:
#     domain-insecure: "de"
# Negative trust anchor with an automatic expiry (default 1h, max 1 week)
rndc nta -lifetime 2h de
rndc nta -dump          # list active NTAs
rndc flushtree de
rec_control add-nta de "DENIC bogus signatures 2026-05-05"
rec_control get-ntas
rec_control wipe-cache de$
-- In kresd's control socket or config
trust_anchors.set_insecure({'de'})
cache.clear('de.')

Also turn on serve-stale, which carried much of 1.1.1.1 through this incident:

server:
  serve-expired: yes
  serve-expired-ttl: 86400          # serve answers up to 1 day past expiry
  serve-expired-client-timeout: 1800  # ms to wait for a fresh answer first
options {
    stale-cache-enable yes;
    stale-answer-enable yes;
    max-stale-ttl 86400;
};

NTAs must expire

An NTA turns off a security control for a whole zone. Always use a lifetime (BIND does this by default), log who added it and why, and have a check that alerts on NTAs older than a few hours. RFC 7646 says NTAs should be removed as soon as the zone validates again.

Lessons for System Design

1. Detection without response is not detection

"The notifications were not processed correctly" is the most expensive sentence in the report. Monitoring is a path: sensor, rule, routing, a human, and the authority to act. Test the whole path with injected faults, and watch the path itself with a dead-man's switch.

2. Validate output the way consumers will

The signer trusted its own metadata (tag 33834); the internet trusted only the published DNSKEY set. Release gates should check output the way the consumer will. The same lesson appears in config pushes such as CrowdStrike 2024 and Cloudflare 2019.

3. Test environments must reproduce cardinality

One HSM at one site said nothing about three HSMs at two sites. If code loops over N things (HSMs, regions, replicas, shards), test with N greater than 1.

4. A parent's signer is a hard dependency for every child

DNSSEC puts part of every child domain's availability on the parent's signer, including children that never opted in. Anything where one signed document vouches for many others (certificate chains, registry indexes, image manifests) has the same blast radius shape.

5. Fail-closed systems need a sanctioned fail-open lever

DNSSEC correctly fails closed. The NTA is the designed, scoped, expiring override for when the signer rather than an attacker is the problem. Build the same into your own fail-closed controls, or operators will improvise a broader, permanent bypass at 3 a.m.

6. Caches buy time, and they also hide the problem

Serve-stale turned a cliff into a slope, which also made early dashboards understate the outage. During an incident, ask how much of the "good" traffic is stale data about to expire.

7. Rare operations rot, and so do vague post-mortems

Freezing rollovers makes a rare code path rarer still. The lasting fix makes rollovers frequent and boring, so bugs show up in drills. And use Gieben's critique as a post-mortem checklist: a UTC timeline, the actual mechanism, why the tests missed it, and verifiable action items (see Postmortem Framework).

What Would You Do?

Scenario 1: You run the resolver fleet for a large ISP. At 21:40 local time SERVFAILs for one ccTLD start rising. dig shows "RRSIG with malformed signature" for the TLD's NSEC3 records, and DNS-OARC chat says other operators see the same. The registry has said nothing yet. Do you (A) wait for the registry, (B) add an NTA for the TLD now with a 2-hour lifetime, or (C) turn off DNSSEC validation globally?

(B). The evidence is that the parent signer is broken, and several independent operators confirm it. RFC 7646 describes exactly this case. A time-limited NTA restores service for that one TLD while keeping validation for everything else. Waiting (A) costs your users hours of outage for a problem you cannot fix. Turning validation off globally (C) removes protection from every zone to fix a problem in one. Log the change, set an expiry, and remove it once the zone validates again.

Scenario 2: You are building a new signing pipeline that uses three HSMs across two sites. Your test lab has one HSM. Your manager says a second HSM is not in the budget this year. What do you do?

Test with software tokens. Use SoftHSM (or another PKCS#11 emulator) to create several independent tokens and run the full rollover against N greater than 1. Add an invariant check after key distribution that every HSM holds the same public key. Add a sign-then-verify step against the published DNSKEY set in production. None of this needs new hardware. If the budget question comes back, the DENIC outage is a good argument for buying the second HSM anyway.

Key Lessons

Key Lessons

  • Sign, then verify against what you publish. A signer that checks its own RRSIGs against the published DNSKEY set catches this whole class of bug.
  • Compute key tags from key material, and assert that every HSM holds the same public key after distribution.
  • Test with production cardinality. Single-HSM tests could not find a multi-HSM bug. SoftHSM makes N tokens cheap.
  • Alerts are a pipeline. Test the path from detector to human, and add a dead-man's switch.
  • NSEC3 signatures protect unsigned children too. A bad TLD signer breaks every delegation, signed or not.
  • Resolver operators need an NTA runbook and serve-stale. Those two controls are what limited the damage on the night.
  • Freezing a risky operation is a stopgap. Make it safe to do often.

Quiz

Quiz

Q1: Why did .de domains that do not use DNSSEC fail to resolve? A validating resolver following a referral to an unsigned child needs a signed NSEC3 proof from .de that no DS record exists. Most NSEC3 signatures were invalid, so the resolver could not prove the delegation was legitimately insecure and marked it bogus.

Q2: Why did roughly one third of signatures validate? Three HSMs each signed with their own different private key, but only one corresponding public key (tag 33834) was published. Only signatures from the HSM with the published key verified.

Q3: Why did the SOA signature flip between valid and invalid? The zone is updated incrementally and each change bumps the serial, so the SOA is re-signed with every update. Each re-signing could be done by any of the three HSMs.

Q4: DENIC's monitoring detected the problem. Why did the outage last about three hours? The notifications from the three validation tools "were not processed correctly", so nobody acted on them. DENIC has not published what went wrong in alert handling.

Q5: What is a negative trust anchor, and what is the risk of using one? An RFC 7646 mechanism that tells a validating resolver to treat a zone as unsigned for a limited time. While it is active, answers for that zone are not DNSSEC-protected, so spoofing would not be detected. It should be scoped to one zone, time-limited and removed as soon as the zone validates.

One-Liner Summary

A key-rollover bug gave three HSMs three different private keys under one published key tag, so two thirds of .de's signatures were bogus. Validators failed every .de name, signed or not, for about three hours. DENIC's own validators saw it, but nobody acted on their alerts.

Sources