Watching an IP die

Every network has them: IP addresses your source of truth swears are in use, but that nothing on the wire has answered for in months. Someone allocated .47 for a server three years ago. The server was decommissioned. Nobody updated IPAM. Now .47 is a ghost — marked active, occupying space, and one day it’ll get handed to a new host and collide with something, or just sit there forever, quietly wrong.

Finding those is harder than it looks. And the naive version is dangerous. This is the story of building a tool that does it carefully — and the bugs I caught along the way that would have made it lie.

The tool is ReclaimNet, the first thing out of Nimentus. It’s open source. This is how it came to be.

The trap in “is this host alive?”

The obvious approach: ping every address, whatever doesn’t answer is dead, reclaim it.

This is wrong, and it’s the kind of wrong that causes outages. Plenty of live, important hosts don’t answer ping — firewalled ICMP is the default on half the boxes in any enterprise. A server can be perfectly alive and utterly silent to a probe. Mark it dead, hand its address to something else, and you’ve just taken down production with a tool that was “only cleaning up.”

So the real problem isn’t detecting life. It’s this asymmetry:

• Presence is proof. If something answers, it’s alive. Certain.
• Absence is not proof. Silence could mean dead, or firewalled, or busy — or you weren’t really listening.

You can never prove a host is dead from a single look. You can only accumulate failure to find life, across multiple independent methods, over time — and get progressively more confident. That word, confident, turned out to be the whole design.

Confidence, not a yes/no

ReclaimNet never declares an IP dead. It maintains a liveness score per address, and a separate confidence in that score.

The score runs 0–100 with bands: alive-chatty (seen constantly), alive-silent (was alive, gone quiet), half-dead, and dead. A host that’s actively talking saturates near 100. When it goes silent, a small penalty walks the score down over time — through the bands — so where it sits tells you how long it’s been quiet. The bands aren’t categories; they’re stages of decay.

The penalty is small and slow while a sighting jumps the score back up fast. The system is quick to trust life and reluctant to declare death — exactly the asymmetry the problem demands. But the score alone isn’t enough, which is where most of the hard thinking went.

The bug that taught me everything: confidence has to mean “did we look,” not “did we see”

Here’s the subtle part. What should confidence be based on?

My first instinct: the more times we’ve observed a host, the more confident we are. Reasonable, and completely backwards — and it took watching a real host die to see why.

I powered off a lab VM and watched its score decay correctly, minute by minute, toward dead. But its confidence collapsed to near zero, and the tool refused to commit to the verdict. Why? Because the moment the host went dark, it stopped being observed. Its observation count froze. And since confidence was tied to observation count, the host most likely to be dead had the least confidence to say so. A dead host can never accumulate the evidence that would let you call it dead. The detection mechanism strangled itself.

The fix reframes the entire idea. Confidence isn’t about how many times you saw the host. It’s about how thoroughly you looked — how many times a working sensor swept that host’s segment, whether or not anything answered. A successful sweep that finds nothing is strong evidence about the host: we looked, competently, and nobody was home. Stack up enough of those and you become highly confident it’s dead — the exact opposite of the broken version.

Coverage, not observation. That one distinction is the difference between a tool you can trust to touch your source of truth and one that quietly lies. It was subtle enough to survive a full passing test suite — because none of the tests checked it. The bug lived exactly where I wasn’t looking.

The lesson that wasn’t in the code: where you stand changes what’s true

The second hard lesson had nothing to do with logic. I was running the prober from my laptop, which had a mesh VPN running. A powered-off host kept coming back as alive — because the VPN’s network stack was answering on its behalf, from somewhere else entirely.

The tool wasn’t wrong. It was faithfully reporting what it could reach. The problem was where it was standing. A liveness tool is only as honest as its vantage point — run it behind a VPN, a NAT, an overlay, and it reports the plumbing, not the host. Moving the prober onto a jumphost with real, direct adjacency to the segment fixed it instantly.

That’s now a deployment principle, learned empirically: the sensor has to sit where it can see the truth, not a translation of it.

Two witnesses are better than one

ReclaimNet is modular by design — collectors plug into a stable core. It started with one source: ARP tables pulled via SNMP from the segment’s gateway, a passive, strong signal (if a host recently resolved at layer 2, something’s there). Then I added an active prober: ICMP plus TCP probes to common ports, because a host that ignores ping will often answer a connection — and a refused connection still proves the host is up.

The two corroborate. Where they agree, confidence climbs faster than either could alone. A host has to go silent on both — ARP aging out and the prober finding nothing — before it falls. One witness can be fooled; two agreeing is much harder to argue with. The architecture means a third and fourth witness — NetFlow, cloud APIs, DHCP — slot in later without touching the scoring core. The core is the gold.

Closing the loop: reconciling against the source of truth

The point of all this isn’t to scan a subnet — it’s to answer a sharper question: does reality match what you documented?

ReclaimNet reads its targets from NetBox — the IPs you’ve actually claimed exist — and reconciles them against what it observes. That produces the two findings that matter:

• Documented, but dead. NetBox says active; ReclaimNet has watched, thoroughly, and found nothing. These are your reclaim candidates.
• Alive, but undocumented. A host answering on the wire that NetBox has no record of. Shadow IPs — the opposite problem, and a security one.

And then it closes the loop. ReclaimNet writes its verdict back into NetBox — as a tag and custom fields on the IP object — so the answer lives where engineers already work, not buried in another tool’s dashboard.

The write is deliberate, never automatic. Given that a contaminated input once made a dead host look alive, letting an unattended process rewrite the source of truth on its own judgment is exactly the mistake the whole design exists to avoid. A human stays in the loop. The tool informs; it doesn’t decide.

Where it is now

ReclaimNet runs as a service — collecting continuously, scoring over time, reconciling against NetBox, ready to write verdicts back when asked. It found a dead-but-documented host in my lab on the first real run, on the first prefix I pointed it at.

It also does something the enterprise discovery tools mostly don’t: it doesn’t just snapshot now. It accumulates days of evidence and tells you how sure it is — and that temporal confidence is the part I think is genuinely worth building.

It’s open source, built in public, and deliberately small — one problem, solved completely. That’s the whole idea behind Nimentus: the unglamorous problems the big platforms leave behind, handled properly, one at a time.

Fino alla fine.