Sr. Databricks Engineer - Hybrid NYC

About the role

Job Summary: We are seeking an experienced Senior Data Engineer to design, develop, and optimize scalable data platforms that power enterprise analytics, machine learning, and AI-driven solutions. The ideal candidate will bring deep expertise in modern data engineering practices, cloud-native architectures, Lakehouse platforms, and distributed data processing technologies. This role will play a critical part in building reliable, high-performance data ecosystems leveraging Databricks, Spark, Delta Lake, Snowflake, Kafka, and AWS, while also contributing to the adoption of Generative AI, Large Language Models (LLMs), Retrieval Augmented Generation (RAG), and AI-assisted engineering solutions. Ideal Candidate Profile: A senior-level data engineer with extensive experience in Databricks, Spark, AWS, Kafka, Snowflake, and Lakehouse architectures, who is equally passionate about modern AI technologies and building intelligent data platforms for the future. Location: New York, New York. Job type: Contract. Education: Bachelors.

Responsibilities

• Data Platform Engineering: Design, build, and maintain scalable batch and real-time data pipelines. Develop and optimize data ingestion, transformation, and processing frameworks for structured, semi-structured, and unstructured datasets. Implement modern Lakehouse architectures utilizing Databricks, Delta Lake, and Medallion (Bronze, Silver, Gold) design patterns. Build data solutions that support enterprise analytics, reporting, and machine learning initiatives. Ensure data quality, governance, lineage, security, and compliance across data ecosystems.
• Big Data & Streaming Solutions: Develop distributed data processing applications using PySpark and Spark SQL. Build and maintain streaming pipelines using Kafka, Spark Structured Streaming, and AWS Kinesis. Design fault-tolerant, scalable systems capable of processing large data volumes with low latency. Optimize workload performance through partitioning strategies, clustering, caching, and query tuning.
• Cloud & Lakehouse Architecture: Architect and implement cloud-based data solutions on AWS. Utilize AWS services including S3