Data Engineer
Job Description
Robotics Technologies is hiring a Data Engineer for an onsite role in Quincy, MA. This position is set for a 12-month engagement with the possibility of extension and focuses on building and optimizing Databricks-driven data pipelines.
Role Overview
In this Data Engineer role, you will deliver end-to-end data engineering work using Databricks. The work includes ETL/ELT development, SQL-based processing, and hands-on implementation using Scala and Spark. You will also support performance tuning and enhancement of data workflows to ensure reliable, efficient processing across Spark jobs and streaming workloads.
Key Responsibilities
- Build and maintain Databricks-based data pipelines using ETL/ELT patterns.
- Develop and optimize Spark jobs, data transformations, and data processing workflows within Databricks.
- Implement solutions using Scala and Spark, including work with data frames and notebooks and SQL.
- Perform performance tuning and enhancements across Scala/PySpark and SQL components.
- Apply and support Databricks Lakehouse concepts, including delta lake storage and relevant architecture patterns.
- Support streaming data processing using Spark structured streaming.
- Work with Databricks orchestration to manage and run data processing jobs.
Required Experience
- Minimum 8+ years of extensive data engineering experience in Databricks, including ETL/ELT using data pipelines.
- Strong SQL and processor-related experience.
- 8+ years hands-on development experience implementing Databricks solutions using Scala and Spark, including data frames and notebooks/SQL.
- 8+ years hands-on development and performance tuning/enhancement experience using Scala/PySpark and SQL.
- Strong Databricks experience developing and optimizing spark jobs, data transformations, and data processing workflows.
Databricks and Data Platform Expertise
- Understanding of Delta Lake storage.
- Experience with Unity catalog.
- Experience with Autoloader.
- Knowledge of Lakehouse architecture and Medallion architecture.
- Familiarity with Databricks serverless options.
- Working knowledge of Dataset and data frame concepts.
- Experience with Databricks clusters configuration and sizing.
- Knowledge of Databricks job orchestration.
- Experience with Spark structured streaming.
Location and Engagement Details
- Location: Quincy, MA (onsite)
- Duration: 12 Months + Extension
Compensation
- Salary: USD 86 per hour