Elastic Pipeline Engineer/Architect

Key Responsibilities

• Participate in discovery sessions alongside the Lead Architect and engineering and compliance stakeholders to map the current evidence lifecycle, identify manual touchpoints and custody gaps, and confirm the redaction ruleset
• Review the existing Elastic deployment to understand current index structure, ingestion patterns, access control configuration, and data retention policies, and evaluate candidate integration points at the Elastic ingest layer
• Develop and configure the automated redaction and evidence pipeline within the Elastic ingest layer in accordance with the approved solution design
• Implement field-level redaction rules dropping, masking, or hashing traceability fields per the data classification policy and unit-test them against representative pipeline output samples across all classified field types
• Validate that tamper-evidence metadata is correctly stamped at execution and preserved accurately within Elasticsearch to maintain an immutable chain of custody
• Configure Elasticsearch index templates, field mappings, and Index Lifecycle Management (ILM) policies aligned to audit and data retention requirements
• Configure auditor-facing search and retrieval views, implementing the role-based access model that distinguishes auditor views from operational engineering views
• Conduct integration and acceptance testing for evidence retrieval and data redaction against agreed acceptance criteria, documenting test results and remediation actions prior to handover
• Support UAT, capturing and resolving defects identified in the auditor retrieval workflow prior to engagement closure
• Contribute to pipeline architecture documentation, operational runbooks, and knowledge transfer sessions with the engineering and operations teams
• Participate in agile ceremonies daily stand-ups, sprint demos, and retrospectives

Required Skills & Minimum Experience

• Elastic Stack: Elasticsearch, ingest pipelines, index templates & ILM - 5 years
• Log pipeline / data transformation tooling (Logstash, ingest processors, Beats) - 5 years
• Python for data / integration automation - 4 years
• Data pipeline / ETL-style development - 3 years
• Structured & semi-structured data (JSON, schemas, field mappings) - 3 years
• Git / GitOps workflow - 3 years
• Data redaction, masking, or hashing techniques for sensitive fields - 2 years
• Role-based access control & search/retrieval configuration in Elastic (Kibana) - 2 years
• Data privacy / PII handling & data classification - 2 years
• CICD pipeline development (TeamCity, GitLab CI, Jenkins, or equivalent) - 2 years
• YAML / JSON data modelling - 2 years
• Linux / Bash - 2 years