Challenge
Operating a distributed edge platform means keeping compute, storage, and networking healthy across locations. This environment covered more than 50 global points of presence (POPs) and edge data centers.
My contribution
I maintained the infrastructure and investigated critical failures affecting service availability. My work spanned Linux systems, virtualization, storage, networking, and the automation and monitoring tools used to operate them.
System overview
KVM and libvirt supported virtualization, while Ceph provided distributed storage. HAProxy and Keepalived supported traffic handling and failover. BGP and LACP were part of the networking stack, with SaltStack for configuration automation and Prometheus for monitoring.
These layers are connected operationally: an application symptom may originate in a host, storage path, or network dependency. Understanding those dependencies matters when diagnosing failures across multiple sites.
Outcomes
Supported operations across 50+ global POPs and edge data centers, helping sustain 99.99%+ service uptime.
Engineering considerations
Failover and monitoring need to be considered together. A component can be running while a dependent service is unavailable, so useful diagnosis requires correlating signals across compute, storage, and networking. A valuable next improvement is to connect those signals to documented recovery procedures.
Supporting examples
Related learning material, separate from this production environment: