Senior Azure Data Engineer
Senior
Azure
Azure Data Engineer
Azure Data Factory
Azure Data Lake
Big Data
Bigdata
Cloud
Cloud Data Engineering
Cloud Platform
Data Architecture
Data Engineer
Data Engineering
Data Factory
Data Factory Azure
Data Governance
Data Integration
Data Lakehouse
Data Pipeline
Data Platform
Data Processing
Data Security
Databricks
ETL
Microsoft Azure
Spark
SQL
Job Description
This Senior Azure Data Engineer role supports an enterprise-scale lakehouse data platform built on Azure Databricks and a Delta Lake environment. The position focuses on designing, building, and optimizing secure, high-performing ETL/ELT pipelines while providing technical leadership across data engineering and governance practices.
Location and Work Model
- Location: Plano, TX (onsite)
- Schedule: Hybrid schedule in Plano, TX
- Eligibility: Candidates must currently reside in the Dallas–Fort Worth (DFW) area
Compensation
- Salary: USD 105,000 - 150,000 per year
- Minimum rate: $105,000.00
- Maximum rate: $150,000.00
Responsibilities
- Design and build scalable ETL/ELT pipelines using Azure Databricks with PySpark and Spark SQL
- Develop ingestion workflows from multiple source systems (including Event Hub, APIs, Azure SQL, and Blob Storage) into Bronze/Silver/Gold layers
- Apply best practices for Delta Lake and Unity Catalog, including schema evolution, versioning, and governance
- Optimize pipeline performance and cost using partitioning, caching, clustering, and Spark tuning
- Integrate Databricks with ADLS Gen2, Azure Data Factory (ADF), Azure Synapse, Event Hub, Key Vault, and Azure DevOps
- Implement secure access controls using managed identities, RBAC, and data protection practices such as masking
- Partner with analytics and business teams to deliver trusted datasets, metrics, and KPIs
- Provide technical leadership through mentoring, code reviews, and architectural guidance
- Coordinate with enterprise architecture and security teams to align with data privacy and security policies
- Implement data quality checks, alerting, and lineage tracking using Great Expectations, Unity Catalog, and data observability tools
- Build monitoring dashboards, alerts, and recovery mechanisms for critical pipelines
Required Qualifications
- Education: Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience)
- Experience: 5+ years of data engineering, including 3+ years with Azure Databricks and Apache Spark (PySpark)
- Strong experience with Delta Lake, Unity Catalog, Databricks notebooks, and SQL-based warehousing patterns
- Hands-on experience with ADLS Gen2, Event Hub, ADF, Key Vault, and Synapse
- Strong coding skills in Python and SQL; familiarity with Scala is a plus
- Solid understanding of data modeling for OLTP/OLAP, including star and snowflake schemas, and performance tuning
- Experience implementing CI/CD for Databricks using Databricks CLI, Terraform, and/or Azure DevOps
- Certification: Databricks certification (e.g., Databricks Certified Data Engineer Professional)
- Experience with streaming and real-time pipelines, including Structured Streaming and Event Hub (Kafka familiarity is included)
- Exposure to ML platform tooling such as MLflow or support for ML pipelines
- Experience with reporting and consumption layers such as Power BI, Purview, or Microsoft Fabric
- Knowledge of governance and compliance standards such as GDPR, HIPAA, and SOC2
Technologies
- Azure Databricks, PySpark, Spark SQL
- Delta Lake, Unity Catalog
- Python, SQL, Scala
- ADLS Gen2, Azure Data Factory (ADF), Azure Synapse
- Event Hub, Key Vault, Azure DevOps
- Managed identities, RBAC, masking
- Great Expectations, data observability platforms
- Databricks notebooks, Databricks CLI, Terraform
- Structured Streaming, Kafka
- MLflow, Power BI, Purview, Microsoft Fabric
Benefits
- Medical, Prescription, Dental & Vision Benefits (for employees working 20+ hours per week)
- Health Savings Account (HSA) (for employees working 20+ hours per week)
- Life & Disability Insurance (for employees working 20+ hours per week)
- MetLife Voluntary Benefits
- Employee Assistance Program (EAP)
- 401K Retirement Savings Plan
- Direct Deposit & weekly epayroll
- Referral Bonus Programs
- Certification and training opportunities
Additional Candidate Requirements
- Relocation is not available
- Candidates must be authorized to work in the United States without sponsorship now or in the future