Sr. Data Architect (Databricks)

About Us

Techwave is a leading global IT and engineering services and solutions company revolutionizing digital transformations. Founded in 2004 and headquartered in Houston, TX, USA, we enable clients to maximize their potential through a wide array of technology services, including Enterprise Resource Planning, Application Development, Analytics, Digital, and the Internet of Things (IoT). We foster a culture of growth and inclusivity, challenging our team at every step to excel and pushing the boundaries of what's possible.

Responsibilities

• Solution Architecture
  • Design end-to-end Lakehouse architectures using Databricks.
  • Define enterprise data architecture aligned with business and technology strategies.
  • Lead cloud data modernization and migration initiatives.
  • Develop scalable, secure, and highly available data platforms.
• Data Engineering
  • Design ETL/ELT frameworks using Apache Spark and Databricks.
  • Architect batch and real-time streaming data pipelines.
  • Design Medallion Architecture (Bronze, Silver, Gold).
  • Define data ingestion strategies for structured, semi-structured, and unstructured data.
  • Implement Change Data Capture (CDC) and incremental processing.
• Data Modeling
  • Design enterprise data models using Dimensional Modeling, Star Schema, Snowflake Schema, and Data Vault.
  • Build semantic models for reporting and analytics.
• Databricks Platform Architecture
  • Design and optimize Delta Lake, Unity Catalog, Databricks Workflows, Delta Live Tables (DLT), Databricks SQL, MLflow, and Lakeflow.
  • Establish workspace standards and architecture patterns.
  • Define notebook, job, and code organization strategies.
• Performance & Cost Optimization
  • Optimize Spark jobs and SQL workloads.
  • Improve partitioning, clustering, and file layouts.
  • Reduce cloud infrastructure costs through autoscaling and efficient cluster sizing.
  • Implement monitoring for compute utilization and query performance.
• Security & Governance
  • Implement Unity Catalog for centralized governance.
  • Design role-based access control (RBAC).
  • Configure row-level and column-level security.
  • Establish metadata management, lineage, and auditing.
  • Ensure compliance with enterprise security and regulatory standards.
• AI & Advanced Analytics
  • Architect AI/ML platforms using Databricks.
  • Enable ML lifecycle management with MLflow.
  • Design data pipelines for machine learning workloads.
  • Support Generative AI and Retrieval-Augmented Generation (RAG) solutions by integrating Databricks with vector databases and AI services.
• DevOps & Automation
  • Define CI/CD strategies using Azure DevOps or GitHub Actions.
  • Implement Infrastructure as Code (Terraform preferred).
  • Automate deployment of notebooks, jobs, workflows, and platform configurations.
  • Establish version control and branching strategies.
• Leadership & Governance
  • Lead architecture reviews and design workshops.
  • Define technical standards and best practices.
  • Mentor Data Engineers and Technical Leads.
  • Collaborate with business stakeholders, project managers, and cloud architects.
  • Provide technical leadership during project delivery.
• Typical Project Responsibilities
  • Design enterprise Lakehouse architectures.
  • Migrate legacy data warehouses to Databricks.
  • Build scalable ETL/ELT pipelines using Spark.
  • Implement Delta Lake and Unity Catalog.
  • Establish governance, security, and metadata management.
  • Optimize performance and cloud costs.
  • Deliver curated datasets for BI, AI/ML, and analytics.
  • Define DevOps, CI/CD, and operational best practices.
  • Lead architecture reviews and guide technical teams throughout the project lifecycle.

Requirements

• Technical Skills
  • Databricks: Databricks Lakehouse Platform, Apache Spark (PySpark and Spark SQL), Delta Lake, Unity Catalog, Delta Live Tables, Databricks Workflows, Databricks SQL, MLflow, Structured Streaming.
  • Cloud Platforms: Microsoft Azure (preferred), including Azure Data Lake Storage Gen2, Azure Data Factory, Azure Synapse Analytics, Azure Key Vault, Microsoft Entra ID. AWS or Google Cloud experience is also valuable.
  • Data Engineering: ETL/ELT Architecture, Data Lakes, Data Warehousing, Data Modeling, CDC, Streaming Architecture, Metadata Management.
  • Programming: Python, PySpark, SQL, Scala (preferred).
  • Databases: SQL Server, Oracle, PostgreSQL, Snowflake.
  • BI & Analytics: Power BI, Tableau, Semantic Modeling.
  • DevOps: Git, Azure DevOps, GitHub, Terraform, CI/CD Pipelines.
• Experience
  • 10–15+ years of IT experience.
  • 5+ years of hands-on Databricks architecture and implementation experience.
  • Experience designing enterprise data platforms on Azure, AWS, or GCP.
  • Proven experience in large-scale data migration and cloud modernization projects.
  • Experience integrating enterprise systems such as SAP, Oracle, Salesforce, Workday, or other business applications.
• Qualifications
  • Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
  • Master's degree is preferred.
• Preferred Certifications
  • Databricks Certified Data Engineer Professional
  • Databricks Certified Data Engineer Associate
  • Databricks Certified Machine Learning Professional (preferred)
  • Microsoft Certified: Azure Data Engineer Associate
  • Microsoft Certified: Azure Solutions Architect Expert
  • Azure Fundamentals (AZ-900)
• Key Competencies
  • Enterprise architecture and solution design
  • Distributed data processing expertise
  • Data governance and security
  • Performance tuning and optimization
  • Cost optimization
  • Stakeholder management
  • Leadership and mentoring
  • Problem-solving and analytical thinking
  • Communication and presentation skills
  • Strategic planning