Senior VoIP Operations & Reliability Engineer (Carrier-Class Voice Platform)
Harden the current production system (immediate priority)
Take ownership of the reliability of our live Kamailio and Asterisk production system from day one, while the next-generation platform is still in development.
Assess the current system end to end and find its weak points: single points of failure, brittle failover, missing redundancy, capacity headroom, and the failure scenarios it does not yet handle gracefully.
Close those gaps incrementally and safely, without disrupting live customer traffic: add redundancy and failover, tighten configuration, and remove fragility.
Add the observability the current system is missing so problems are caught before customers feel them, and stand up alerting, dashboards, and SIP capture against the live fleet.
Stabilize day-to-day operations: triage and resolve recurring issues, document the system as it actually runs, and write the runbooks that do not exist yet.
Work hand in hand with the development team
- Pair with the programmers throughout development as the operational voice in the room: review designs, challenge assumptions, and find the failure modes that code reviews miss.
- Push for the logging, metrics, health checks, graceful shutdown, retry behavior, and failure handling that the team needs to add for the platform to survive production.
- Transfer operational knowledge to the team: help developers understand how their code behaves under load and failure, and raise the whole group's instinct for production reality.
Map the full failure surface of the platform
Map the full failure surface of the platform (node failure, data-center loss, upstream carrier outage, registration storms, partial network partitions, resource exhaustion) and make sure every scenario has a defined, tested behavior.
Design and run a rigorous test program
- Functional, load, stress, soak, and failover testing, with realistic call models (concurrent calls, BHCA, registration churn).
- Build fault-injection and chaos testing into the pipeline so failure handling is proven, not assumed.
- Validate the high-availability and scalability design under real conditions: active-active and active-passive topologies, geographic redundancy, graceful degradation, automated failover with measured recovery times, and capacity limits.
Keep it up (day-to-day reliability engineering)
- Own platform uptime as a daily responsibility, not a quarterly goal. Customers should experience no downtime.
- Build and own the observability stack: SIP capture (HEP/Homer), CDR and quality pipelines, metrics, dashboards, and alerting that catches problems before customers do.
- Define SLOs and SLIs for signaling, media, and registration, and hold the platform to them.
- Run incident response: detect, triage, mitigate, and resolve, then drive blameless postmortems and make sure the same failure cannot recur.
- Write and maintain runbooks, and lead disaster-recovery and failover drills so the team can execute under pressure.
Participate in (and help design) a sustainable on-call rotation
- Tune and operate the production fleet: Asterisk, Kamailio, OpenSIPS, and the supporting network layer, under live carrier traffic.