Senior Data Engineer (Azure Data Factory Databricks, PostgreSQL, Python)

Career Developers is seeking a Senior Data Engineer to design and maintain scalable production data pipelines in Azure environments, leveraging Data Factory and Databricks with PostgreSQL and Python. This onsite role is based in Reston, Virginia.

Responsibilities

• Architect and maintain scalable production data pipelines that support enterprise data platforms.
• Conduct exploratory data analysis, statistical assessment, and data quality checks to enable analytics and reporting.
• Develop and optimize ETL and ELT workflows, including job design, orchestration, and performance tuning.
• Create and optimize relational database schemas, SQL queries, and stored procedures.
• Operate within Azure data environments such as Data Factory, Databricks, and Data Lake Storage.
• Collaborate with analytics, BI, and engineering teams to deliver reliable data products.
• Implement comprehensive monitoring, logging, and observability for data pipelines.
• Maintain documentation, data lineage, and governance standards for data assets.
• Participate in architecture discussions and contribute to the long-term data platform strategy.

Requirements

• Bachelor's degree in Computer Science, Statistics, Mathematics, Information Science, Data Engineering, Finance, or related field; Master’s preferred.
• 5+ years of experience building production data pipelines as a data engineer.
• 5+ years of applied data analysis and statistical work.
• 3+ years of ETL/ELT development experience.
• 3+ years working with relational databases and performance tuning.
• Experience with Azure data services such as Data Factory, Databricks, Data Lake, Synapse, Delta Lake, or similar.
• Strong programming skills in Python, PySpark, SQL, and Bash.
• Experience with Kafka, SAP Data Services, or similar integration tools preferred.
• Experience with PostgreSQL, SAP ASE, or comparable databases.
• Experience with Power BI or BusinessObjects is a plus.
• Experience with CI/CD pipelines, Git, Kubernetes, and infrastructure-as-code.
• Knowledge of data governance, cataloging, and access control.
• Experience with monitoring tools such as Datadog, Grafana, or Prometheus.
• Ability to work independently while managing multiple priorities.
• Strong communication and documentation skills.
• Must be eligible to work in the United States.

Technologies

• Azure Data Factory
• Databricks
• Data Lake Storage
• Azure Synapse
• Delta Lake
• Spark
• PostgreSQL
• SAP Data Services
• SAP ASE
• Power BI
• BusinessObjects
• Python
• PySpark
• SQL
• Bash
• Kafka
• Kubernetes
• Git
• CI/CD
• Datadog
• Grafana
• Prometheus