Databricks Data Engineer
Job Description
Robotics Technologies is seeking a Databricks Data Engineer to support the Retail Spine initiative in Phoenix, AZ. This onsite role focuses on designing, building, and optimizing scalable data solutions in a fast-paced, Azure-based environment, with hands-on work across Databricks and PySpark ETL/ELT. You will also work with CDC-driven incremental processing and Kafka streaming to support real-time or near real-time data needs.
In this position, you will help develop new pipelines and improve existing workflows, quickly ramping on current codebases to deliver measurable enhancements. The work spans multiple data formats and enterprise platforms, with an emphasis on reliability, performance, scalability, and monitoring.
Responsibilities
- Design, build, and optimize end-to-end ETL/ELT pipelines using Databricks and Apache Spark (PySpark).
- Develop and maintain scalable data solutions on Microsoft Azure, leveraging Databricks, Delta Lake, and Kafka streaming.
- Build robust pipelines with Change Data Capture (CDC) for incremental processing and real-time or near real-time requirements.
- Support both new pipeline development and enhancement of existing pipelines, including quickly understanding current codebases to deliver improvements.
- Integrate and process data from diverse sources including Flat files, XML, JSON, SQL Server and JDBC, Kafka topics, APIs, MQ, and Unity Catalog volumes.
- Implement data quality frameworks by writing technical data quality rules and quarantining error records.
- Apply canonical data modeling techniques including SCD Type 1 and SCD Type 2 implementations.
- Ensure high levels of data reliability, performance, scalability, and monitoring across workflows.
- Automate workflows using Databricks Workflows and CI/CD pipelines (including GitHub), with focus on reuse, efficiency, and standardization.
- Collaborate with developers, architects, and product owners, engaging with stakeholders, source teams, and downstream consumers as needed.
- Participate in Agile/Scrum ceremonies and deliver high-quality solutions within defined timelines.
Requirements
- Extensive hands-on experience with Databricks, with the ability to contribute from day one.
- Proven experience building end-to-end data engineering pipelines with CDC capabilities.
- Strong proficiency in PySpark and SQL (required); Python experience is a plus.
- Experience working in Microsoft Azure cloud environments.
Tech Stack
- Databricks, Apache Spark, PySpark
- Microsoft Azure, Delta Lake
- Kafka, Change Data Capture (CDC)
- SQL Server, JDBC, Unity Catalog
- Databricks Workflows, CI/CD, GitHub
- SCD Type 1, SCD Type 2, Agile/Scrum
- XML, JSON, Flat files, APIs, MQ, SQL
Job ID: J55801
Location: Phoenix, AZ (onsite)
Duration: 9 Months + Extension
Employment Type: W-2, 1099, C2C
Salary/Bill Rate: USD 71 - 71 per hourly
Bill Rate: $71/hour (C2C)
Work Authorization: US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other (USC, GC, H-1B, OPT-EAD, GC-EAD, H4-EAD listed)
Domain Exposure: Consumer Staples
Client: To Be Discussed Later