Lead Pyspark / SQL Engineer - Threat Detection & DataBricks

Job Summary

We are seeking a skilled PySpark / SQL Engineer to support our Threat Detection team in building and migrating security analytics pipelines using Databricks.

Requirements

• 5-10+ years of experience as a PySpark/SQL Engineer, with a strong focus on data engineering and analytics.
• Prior experience in building threat detection or log analytics pipelines using PySpark, SQL, and Databricks.
• Hands-on experience with Terraform for deploying data infrastructure.
• Proficient in PySpark for large-scale data processing and transformation.
• Familiarity with cloud platforms such as AWS or Azure is preferred.
• Strong analytical skills and attention to detail when working with complex datasets.
• Proven ability to work effectively in collaborative, cross-functional teams.
• Excellent verbal and written communication skills in English.

Responsibilities

• Design and build threat detection pipelines using PySpark, SQL, and Databricks.
• Support the migration of detection rules and content from a legacy platform to Databricks.
• Create and maintain PySpark log pipelines and associated rule configuration files.
• Write unit tests to ensure pipeline accuracy and stability.
• Perform data validation checks to ensure data integrity.
• Deploy pipelines and infrastructure using Terraform.
• Optimize existing data workflows and queries for performance and scalability.
• Collaborate with cross-functional teams to understand data requirements and ensure alignment with detection objectives.