Databricks Architect
Zenith Services Inc. · Miami, FL · 3 mo ago
HybridInformation TechnologyFull-time
Key Responsibilities
- Design, develop, and maintain data pipelines using Databricks, PySpark, and Spark SQL.
- Build and optimize ETL/ELT processes for large-scale data processing.
- Develop data ingestion frameworks from various sources including databases, APIs, and cloud storage.
- Implement Delta Lake solutions to ensure data quality, reliability, and performance.
- Collaborate with data analysts, data scientists, and business stakeholders to deliver data solutions.
- Optimize Spark jobs and Databricks workloads for performance and cost efficiency.
- Integrate Databricks with cloud services such as AWS, Azure, or GCP.
- Monitor, troubleshoot, and resolve data pipeline and platform issues.
- Participate in code reviews, testing, deployment, and documentation activities.
Required Skills
- Strong experience with Databricks, Apache Spark, PySpark, and Spark SQL.
- Proficiency in Python and SQL.
- Experience with Delta Lake, Data Lake, and Lakehouse architectures.
- Knowledge of data modeling, ETL/ELT processes, and data warehousing concepts.
- Experience with cloud platforms (Azure, AWS, or GCP).
- Familiarity with CI/CD pipelines, Git, and DevOps practices.
- Strong analytical and problem-solving skills.