Data Engineer

Responsibilities

• Design and implement scalable ETL and ELT pipelines on the Databricks platform
• Create PySpark and Spark SQL transformations
• Build Bronze, Silver, and Gold data layers using the Medallion Architecture
• Deploy Delta Live Tables, Workflows, and Unity Catalog
• Tune Spark jobs for scalability and performance
• Model analytical workloads with dimensional data schemas
• Introduce data quality validations and monitoring mechanisms
• Collaborate with analytics, engineering, and architecture teams
• Develop reusable data frameworks and automation
• Support CI/CD deployment processes and Git-based development

Requirements

• 7–10+ years of Data Engineering experience
• Extensive hands-on Databricks experience
• Strong PySpark programming capabilities
• Advanced Spark SQL skills
• Experience with Delta Lake
• Experience implementing Medallion Architecture
• Hands-on Delta Live Tables experience
• Unity Catalog experience
• Strong SQL optimization and performance tuning
• Experience with AWS, Azure, or GCP
• CI/CD and Git experience
• Solid understanding of distributed data systems
• Experience in designing enterprise data models

Technologies

• Databricks
• PySpark
• Spark SQL
• Delta Lake
• Medallion Architecture
• Delta Live Tables
• Unity Catalog
• AWS
• Azure
• GCP
• Git
• CI/CD

Preferred Qualifications

• Databricks certification
• Experience with streaming data
• Experience building reusable data frameworks
• Experience with enterprise governance
• Exposure to AI-assisted development tools including Databricks AI Genie