DeveloperJobs.io
← Back to all jobs

Job Description

This Senior Azure Data Engineer role supports an enterprise-scale lakehouse data platform built on Azure Databricks and a Delta Lake environment. The position focuses on designing, building, and optimizing secure, high-performing ETL/ELT pipelines while providing technical leadership across data engineering and governance practices.

Location and Work Model

  • Location: Plano, TX (onsite)
  • Schedule: Hybrid schedule in Plano, TX
  • Eligibility: Candidates must currently reside in the Dallas–Fort Worth (DFW) area

Compensation

  • Salary: USD 105,000 - 150,000 per year
  • Minimum rate: $105,000.00
  • Maximum rate: $150,000.00

Responsibilities

  • Design and build scalable ETL/ELT pipelines using Azure Databricks with PySpark and Spark SQL
  • Develop ingestion workflows from multiple source systems (including Event Hub, APIs, Azure SQL, and Blob Storage) into Bronze/Silver/Gold layers
  • Apply best practices for Delta Lake and Unity Catalog, including schema evolution, versioning, and governance
  • Optimize pipeline performance and cost using partitioning, caching, clustering, and Spark tuning
  • Integrate Databricks with ADLS Gen2, Azure Data Factory (ADF), Azure Synapse, Event Hub, Key Vault, and Azure DevOps
  • Implement secure access controls using managed identities, RBAC, and data protection practices such as masking
  • Partner with analytics and business teams to deliver trusted datasets, metrics, and KPIs
  • Provide technical leadership through mentoring, code reviews, and architectural guidance
  • Coordinate with enterprise architecture and security teams to align with data privacy and security policies
  • Implement data quality checks, alerting, and lineage tracking using Great Expectations, Unity Catalog, and data observability tools
  • Build monitoring dashboards, alerts, and recovery mechanisms for critical pipelines

Required Qualifications

  • Education: Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience)
  • Experience: 5+ years of data engineering, including 3+ years with Azure Databricks and Apache Spark (PySpark)
  • Strong experience with Delta Lake, Unity Catalog, Databricks notebooks, and SQL-based warehousing patterns
  • Hands-on experience with ADLS Gen2, Event Hub, ADF, Key Vault, and Synapse
  • Strong coding skills in Python and SQL; familiarity with Scala is a plus
  • Solid understanding of data modeling for OLTP/OLAP, including star and snowflake schemas, and performance tuning
  • Experience implementing CI/CD for Databricks using Databricks CLI, Terraform, and/or Azure DevOps
  • Certification: Databricks certification (e.g., Databricks Certified Data Engineer Professional)
  • Experience with streaming and real-time pipelines, including Structured Streaming and Event Hub (Kafka familiarity is included)
  • Exposure to ML platform tooling such as MLflow or support for ML pipelines
  • Experience with reporting and consumption layers such as Power BI, Purview, or Microsoft Fabric
  • Knowledge of governance and compliance standards such as GDPR, HIPAA, and SOC2

Technologies

  • Azure Databricks, PySpark, Spark SQL
  • Delta Lake, Unity Catalog
  • Python, SQL, Scala
  • ADLS Gen2, Azure Data Factory (ADF), Azure Synapse
  • Event Hub, Key Vault, Azure DevOps
  • Managed identities, RBAC, masking
  • Great Expectations, data observability platforms
  • Databricks notebooks, Databricks CLI, Terraform
  • Structured Streaming, Kafka
  • MLflow, Power BI, Purview, Microsoft Fabric

Benefits

  • Medical, Prescription, Dental & Vision Benefits (for employees working 20+ hours per week)
  • Health Savings Account (HSA) (for employees working 20+ hours per week)
  • Life & Disability Insurance (for employees working 20+ hours per week)
  • MetLife Voluntary Benefits
  • Employee Assistance Program (EAP)
  • 401K Retirement Savings Plan
  • Direct Deposit & weekly epayroll
  • Referral Bonus Programs
  • Certification and training opportunities

Additional Candidate Requirements

  • Relocation is not available
  • Candidates must be authorized to work in the United States without sponsorship now or in the future

Similar Jobs