Senior Systems Engineer

Description Role Purpose The Senior Systems Engineer builds, operates and supports Vumatel's integrations and payment integrations — the systems that connect our business platforms to the underlying fibre network infrastructure and to our external payment providers. These systems run 24/7 and are business-critical: an integration failure stops customer activations, service changes and payment collection, with immediate operational and revenue consequences. The role is therefore equally weighted between engineering new capability and keeping existing integrations reliable, observable, scalable and recoverable under load. Reporting line and team structure Reports to: Principal OSS Solutions Architect. Works alongside: Systems Engineers on the Payments and NMS teams, the QA and BSS teams, Active Network team, together with the Platform/DevOps and Site Reliability functions who own the shared AWS and Kubernetes environments. Key Responsibilities Building and maintaining network integrations Write, test and review C# services that translate business-system events (new order, upgrade, downgrade, suspension, cease) into provisioning instructions issued to network element management and OSS platforms, and that feed the resulting state back to the originating system.Implement and maintain the API and messaging contracts between Vumatel systems and ISP partner systems, including request validation, retry and back-off behaviour, idempotency handling and duplicate suppression so that a repeated instruction never double-provisions a customer.Diagnose provisioning failures reported by the NOC or Service Desk: trace the transaction through application logs and message queues, identify whether the fault sits with the request, the integration or the downstream network platform, and either correct the data, replay the transaction or raise the defect with the responsible party.Monitor the integration platforms for exceptions and failed transactions, investigate the underlying fault, and drive each to resolution — whether by correcting data, replaying the transaction, patching the integration or escalating to the responsible system owner — working the outstanding exception backlog down on a scheduled basis rather than allowing it to accumulate. Building and maintaining payments Develop and support the C# services that initiate payment instructions to payment service providers, receive and verify provider callbacks and webhooks, and record payment outcomes against customer accounts.Build and maintain reconciliation processes that compare Vumatel's records against provider settlement and statement files, and investigate variances with Finance until each is explained or corrected.Prepare the integrations for scheduled high-volume events, in particular month-end and debit order collection runs: confirm capacity headroom in advance, monitor throughput and rejection rates during the run, and remain available for issues arising from it.Implement and maintain secure handling of payment credentials and sensitive data, including secrets stored in AWS Secrets Manager, key rotation, encryption in transit and at rest, and restriction of sensitive values from application logs. Data and persistence Design, review and evolve PostgreSQL schemas supporting the integration platforms, including indexing strategy, partitioning of high-volume transaction tables and archival of historical records.Write and tune SQL, review query execution plans for slow or lock-contending queries identified through monitoring, and remediate them before they affect transaction throughput.Author and apply versioned database migrations, and verify each migration against production-like data volumes in a lower environment before release. Deployment, infrastructure and scalability Containerise services using Docker and maintain the Kubernetes deployment configuration for them, including resource requests and limits, readiness and liveness probes, horizontal pod autoscaling rules and rolling update strategies that allow deployment without dropping in-flight transactions.Perform load and soak testing against pre-production environments ahead of significant releases or expected volume growth, document the results, and adjust scaling configuration and connection pooling accordingly.Review capacity, error rate, latency and queue depth trends on a weekly basis and raise remediation work into the team backlog before thresholds are breached. Reliability, monitoring and incident response Build and maintain dashboards, metrics, structured logging and alert rules for every integration the incumbent owns, ensuring each production failure mode produces an actionable alert routed to the correct responder.Respond to production alerts within the agreed response times, and lead technical triage during integration incidents affecting provisioning or payments, including after hours.Act as the technical point of contact during major incidents, providing regular status updates to the NOC, business stakeholders and management until service is restored.Conduct and document post-incident reviews, produce written root cause analyses, and implement the resulting corrective actions to completion rather than leaving them as recommendations.Write and keep current the runbooks, escalation paths and troubleshooting guides used by the NOC and by other engineers on call. Testing, quality and change control Write unit, integration and contract tests for all new and amended code, and maintain the automated test suites and mock or sandbox provider environments that allow integrations to be tested without touching production systems.Maintain and improve the CI/CD pipelines for the services owned by the team, including automated test gates, container image builds and deployment promotion between environments.Review colleagues' pull requests, with attention to failure handling, backwards compatibility of integration contracts, observability and rollback safety.Prepare and submit change records for production releases, define and test the rollback plan for each, and execute changes during approved maintenance windows, which are frequently scheduled outside of business hours to protect customer-facing availability. Collaboration, documentation and knowledge sharing Work with the OSS Solutions Architect to refine integration designs, and produce sequence diagrams, interface specifications and architecture decision records for the integrations built.Attend daily stand-ups, sprint planning, refinement and retrospective sessions, and provide realistic estimates and status on committed work.Mentor junior and intermediate engineers through pairing, code review and structured handover of on-call knowledge.Maintain accurate technical documentation in the team's knowledge base, and record all work against the appropriate tickets in the team's tracking system. Requirements Minimum Requirements(essential) Experience At least eight (8) years' experience in a comparable systems, software or integration engineering role.Demonstrable experience building and supporting production integrations between business systems and external third-party platforms via APIs and messaging.Demonstrable experience operating high-availability systems that run 24/7, including production incident response and root cause analysis.Experience designing for scale and resilience: retry and back-off strategies, idempotency, graceful degradation, circuit breaking and recovery from partial failure. Technical C# and the .NET ecosystem, to a level of independently designing, building and supporting production services.PostgreSQL, including schema design, indexing, query tuning and migration management.Sound understanding of REST APIs, webhooks, message queueing and asynchronous processing patterns.Practical experience with monitoring, alerting, structured logging and distributed tracing.Version control (Git), code review practice and CI/CD pipelines.Automated testing across unit, integration and contract levels.Elasticsearch, for centralised log aggregation, search and operational analytics.Ruby/Sinatra for tooling, automation, data remediation and supporting legacy services.Experience mentoring engineers or acting as a technical lead on a delivery workstream. Other Strong written communication in English, sufficient to produce runbooks, incident reports and interface specifications used by other teams.Willingness and ability to work in the Bryanston office on a full-time basis.Willingness to participate in a standby rota and to perform scheduled maintenance work outside of normal business hours.Clear criminal and credit record, given the role's access to payment systems and customer data. Advantageous Skills And Experience Python for tooling, automation, data remediation and supporting legacy services.MSSQL, particularly where integrating with or migrating from existing MSSQL-backed business systems.Docker and Kubernetes, including containerising services and maintaining their deployment configuration, scaling and health checks.AWS, including deploying and operating production workloads and the associated networking, security and monitoring services.Background in telecommunications or a telco-adjacent industry, including exposure to OSS/BSS platforms, provisioning and activation flows, fibre network concepts (OLT, ONT, GPON, splice and splitter topology) or TM Forum Open APIs.Experience integrating with South African payment service providers, banking systems, debit order collection processes (including DebiCheck), card acquiring or EFT reconciliation.Awareness of PCI DSS obligations and POPIA requirements as they apply to payment and customer data.Infrastructure-as-code experience (for example Terraform), and experience with event streaming or brokered messaging platforms (for example Kafka or RabbitMQ).Experience mentoring engineers or acting as a technical lead on a delivery workstream. Qualifications Essential: A completed Bachelor's degree or a recognised diploma (NQF Level 6 or above) in Computer Science, Software Engineering, Information Systems, Electrical or Computer Engineering, or a related technical discipline.Relevant industry certifications — for example AWS Solutions Architect or Developer Associate, or Certified Kubernetes Administrator — are advantageous but not a substitute for the above. Work Level Senior Job Type Permanent Salary Market Related EE Position No Location JHB North