DirectJobHub
Jobs · Engineering · Maryland

Principal Metrics & Observability Engineer

Nyla Technology Solutions · Annapolis Junction, MD · 1 wk ago
On-siteEngineering$217k–$257k/yrFull-time

Active security clearance at the TS/SCI polygraph level is required.

About the role

We are seeking a high-performing Software Engineer with deep expertise in Splunk to join an agile engineering team responsible for designing, developing, and maintaining a comprehensive distributed Metrics and Monitoring solution for High-Performance Computing (HPC) systems. In this role, you will be instrumental in ensuring the reliability, performance, and scalability of mission-critical HPC infrastructure by turning complex system telemetry into meaningful dashboards, correlation models, and automated alerting pipelines.

Responsibilities

  • Design, develop, and maintain a distributed Metrics and Monitoring solution for HPC systems.
  • Create dynamic Splunk dashboards, reports, and data models.
  • Develop Splunk alerting mechanisms, event correlation models, and incident management workflows.
  • Automate processes using Python, Shell, or Perl scripts.
  • Interface with log management utilities (Syslog-ng, rsyslog) and message brokers (Apache Kafka).
  • Apply Infrastructure as Code (IaC) principles using tools like Ansible.
  • Utilize CI/CD methodologies (GitLab CI) and source control with Git.

Requirements

  • Bachelor’s Degree in Data Analytics, Intelligence Studies, Computer Science, Finance/Economics, or a related discipline, plus 12+ years of specialized data analytics or financial intelligence experience.
  • OR Master’s Degree in a related field, plus 10+ years of specialized analytical experience.
  • OR High School Diploma / GED, plus 14+ years of hands-on intelligence analysis, financial tracking, and data discovery experience in lieu of a degree.

Skills

  • Splunk Expertise: Deep, hands-on experience using Splunk SPL, creating dynamic dashboards, building reports, data onboarding, defining field extractions, data modeling, and performance tuning.
  • Alerting & Event Management: Proven experience developing Splunk alerting mechanisms, event correlation models, and incident management workflows.
  • Software Development & Scripting: Recent software development experience using Python, along with practical proficiency developing Shell or Perl scripts for process automation.
  • Log Ingestion & Messaging: Operational familiarity interfacing with log management utilities (Syslog-ng, rsyslog) and message brokers (Apache Kafka).
  • Infrastructure as Code (IaC): Experience applying IaC principles using automation tools like Ansible.
  • Linux Environment: Strong command of the Linux CLI and system administration tools.
  • CI/CD & Version Control: Practical experience with CI/CD methodologies (GitLab CI) and source control with Git.

Desired Skills

  • Practical experience using the Atlassian Tool Suite (Jira, Confluence) for workflow tracking and technical documentation.

About Nyla Technology Solutions

Nyla Technology Solutions delivers exceptional Artificial Intelligence (AI), Data Science, and Software Engineering services for the U.S. Government. Nyla embraces a forward-thinking and bold approach at every turn, earning us a solid reputation as technical trendsetters within the industry. We have a passion for developing solutions that have a quick and immediate impact on mission. Headquartered in Columbia, Maryland, our customers love how we tackle their most challenging problems and get things done.

Pay

The annual base salary range for this role is $217,000-$257,000 (USD), which does not include discretionary bonus compensation or our comprehensive benefits package. Actual compensation offered to the successful candidate may vary from the posted hiring range based on geographic location, work experience, education, and/or skill level.

Benefits

  • Nyla FLEX Program: Custom-tailored benefits to fit your lifestyle, focusing on pay, leave, and schedule flexibility.
  • Pay Flexibility: Option to cash out annual leave hours, opt out of other benefits, or arrange for additional contract hours (over 40 hrs/week). Earn 1.3 times your hourly rate after working over 1880 hours on contract.
  • Leave: 4 weeks of Annual Leave plus 11 holidays, with an additional day of Annual Leave for each year of service. Option to buy up to 6 weeks of annual leave (240 hours total plus holidays). Includes paid anniversary leave.
  • Schedule: Freedom to scale down to 30-32 hours per week while retaining full benefits.
  • Healthcare: 100% company-covered medical, dental, and vision.
  • Retirement: 10% 401k matching with full vesting from day one.
  • Professional Development: $5,000 per year for fees, tuition, or time off for continued growth.
  • Student Loan Repayment: Available as part of our benefits package.
  • Volunteering: 8 hours of paid volunteering annually to support your community.

Similar jobs