Designing for the day the house goes dark

The lab spans two buildings — the main house and the apartment — and that physical fact is the most useful design constraint we have. Entry № 003 ended on an honest weakness: if the whole main house goes dark, the machine-team that runs most of the lab’s services pauses. The design answer wasn’t to fix that. It was to make sure nothing you’d need during an outage depends on that team at all.

The two services that must not blink

The phone book. Everything in the lab finds everything else by name, through a service called DNS — effectively the network’s phone book. If the phone book dies, a perfectly healthy network feels dead: nothing can find anything, including the dashboards that would tell you what’s wrong. So the lab keeps two phone books — one in the main house, one in the apartment — automatically kept identical, each able to answer on its own. Either building, alone, can still look up every name.

The database. One database holds the memory for a growing pile of services. The original lives in the main house — and a live copy runs in the apartment, continuously fed every change, never more than seconds behind, on entirely separate power and hardware. If the main house goes dark, recovery is telling the copy “you’re the original now” — minutes of work, instead of an archaeology dig through Sunday’s backups.

Placement is architecture

The deeper habit this built: for every new service, the first question isn’t how do we run it? — it’s which building, and what happens when the other one is dark? Replaceable services ride the machine-team and get resilience for free. Irreplaceable data gets a deliberate street address, and the critical pair — the phone book and the database — always straddles the gap.

Home labs love to cosplay the cloud’s terminology; this is the part worth actually stealing. Availability zones aren’t a billing feature. Ours are just two buildings and a conduit — but the reasoning is identical, and it works exactly the same way when a breaker trips.