router bork (true)
we rebooted the router and it also reset the router to entirely default settings. this was an error in the router, and now we're gonna move to our own router
timeline: no
summary:
router came up, all dhcp allocated static ips were gone. this itself didn't make the servers unreachable because their dhcp leases had not yet expired. however, the public /25 configuration had been erased, and we had to reverse engineer it from DNS records and add it back to the router config.
once we did this, everything immediately started working again. we've since self-assigned all static IPs on all servers so that athey are not relying on DHCP to receive requests.
if this same issue with the router happens again, the outage will still happen until we re-add the /25 configuration. however that config is documented this time and does not need to be reverse engineered