Senior Data Engineer

Responsibilities

• Engage regularly with engagement managers, project teams, and stakeholders across functional and technical groups, escalating issues that require leadership review
• Design, build, and optimize ETL and ELT pipelines using Azure Data Factory and Databricks
• Develop and tune PySpark and Spark SQL notebooks for large-scale data transformations
• Architect comprehensive data solutions spanning development, UAT, and production environments with Unity Catalog
• Lead design discussions with client architects and counterpart teams
• Collaborate on data contracts and schema agreements across diverse teams
• Oversee the design and optimization of high volume data pipelines
• Define and enforce data engineering standards including naming conventions, partitioning strategies, cluster configurations, and Spark tuning
• Drive performance optimization through AQE tuning, liquid clustering, broadcast joins, and shuffle partition management
• Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies
• Conduct root cause analysis on production incidents and implement durable fixes
• Mentor junior and mid-level engineers via code reviews and paired programming
• Evaluate new technologies and advise on adoption, such as DABs, DLT, Auto Loader, Serverless Compute, and Event Hubs

Requirements

• Proficiency in Python, PySpark, Spark SQL, and SQL Server
• Experience with Azure services including ADF, ADLS Gen2, Key Vault, and Azure Monitor
• Databricks expertise with Delta Lake, Unity Catalog, and Workflows
• Experience with Apache Airflow
• G ratify version control and CI/CD workflows with Git or Azure DevOps
• Deep understanding of Spark internals such as DAG optimization, spill analysis, and skew handling
• Delta Lake advanced features including time travel, deletion vectors, and predictive I/O
• Unity Catalog governance covering row and column security, external locations, and system tables
• Infrastructure as Code experience with Terraform and Azure ARM templates
• Bachelor's degree in Computer Science, Information Technology, Computer Engineering, or a related IT discipline, or equivalent experience
• Limited immigration sponsorship may be available
• Ability to travel about 10 percent on average depending on client engagements

Technologies

• Python, PySpark, Spark SQL, SQL Server
• Azure Data Factory, Azure Data Lake Storage Gen2, Key Vault, Azure Monitor
• Databricks, Delta Lake, Unity Catalog, Workflows
• Apache Airflow
• Git, Azure DevOps
• Terraform, Azure ARM templates
• DABs, Delta Live Tables (DLT), Auto Loader, Serverless Compute, Event Hubs