DeveloperJobs.io
← Back to all jobs

Job Description

Work onsite in Foster City, CA while building data foundations that support manufacturing intelligence use cases. This 12-month contract role focuses on designing and operating pipelines and models that power a Manufacturing Intelligence data lakehouse using Databricks and Delta Lake.

  • Pre-tax commuter benefits
  • Employer-subsidized healthcare through HireArt (eligibility begins the first of the month following 60 days of service)
  • Flexible Spending Account for healthcare-related costs
  • HireArt covers short- and long-term disability and life insurance
  • 401k package

What you’ll do

  • Build and maintain batch and streaming ingestion pipelines into Delta Lake using Auto Loader, Lakeflow Declarative Pipelines, Lakeflow Connect, or Structured Streaming, including CDC feeds from AWS DMS and SAP SLT
  • Design and maintain bronze, silver, and gold layers using MOPS naming conventions and data dictionary standards
  • Develop incremental dbt models on Databricks using MERGE, upsert, and SCD Type 2 patterns, including handling late or out-of-order records
  • Model manufacturing data to support analytics and reporting, including unit-level traceability (SFC), work orders, nonconformances, station results, and first-pass yield
  • Process large-scale and semi-structured datasets with PySpark and Databricks SQL, optimizing performance and cost via clustering, partitioning, OPTIMIZE, VACUUM, file sizing, and compute configuration
  • Maintain data quality and pipeline reliability using dbt tests, Lakeflow expectations, Lakehouse Monitoring, freshness SLAs, and automated alerts
  • Orchestrate workloads with Databricks Workflows or Airflow, and support Redshift-to-Databricks migration with SQL conversion, validation, reconciliation, and cutover
  • Partner with analysts and manufacturing teams to define requirements and document lineage, data contracts, and table definitions in Unity Catalog
  • Implement CI/CD using Databricks Asset Bundles and GitHub Actions across development, staging, and production
  • Manage MOPS-specific Databricks configurations including Unity Catalog permissions, compute resources, SQL warehouses, service principals, secrets, and OAuth authentication aligned with central governance
  • Configure and troubleshoot AWS ingestion infrastructure including DMS, Kinesis, S3, and IAM, coordinating Terraform-based provisioning with infrastructure teams
  • Monitor pipeline health and Databricks Unit (DBU) costs, maintain operational documentation, and support incident response and root cause analysis

What you bring

  • Bachelor’s degree in computer science, information systems, engineering, or a related discipline (or equivalent practical experience)
  • 6+ years of data engineering experience
  • Strong proficiency in SQL, Python, and PySpark
  • Hands-on production experience with Databricks, including Delta Lake, Unity Catalog permissions, Workflows, and Spark performance tuning
  • Production experience with dbt, including incremental modeling and testing
  • Strong understanding of dimensional modeling, medallion architecture, CDC, and incremental data loading
  • Experience deploying data code via Git-based workflows and CI/CD tools such as GitHub Actions or Databricks Asset Bundles
  • Working knowledge of AWS services (S3, IAM, DMS, or Kinesis) and infrastructure-as-code tools such as Terraform

Preferred

  • Databricks Certified Data Engineer Associate or Professional certification; AWS Solutions Architect Associate is a plus
  • Experience preparing data for Databricks AI/BI dashboards, Genie, machine learning, or large language model applications
  • Experience with manufacturing or industrial data, including quality, yield, traceability, or MES
  • Understanding of data governance and compliance requirements for data pipelines

Details

  • Location: Foster City, CA (onsite)
  • Salary: USD 68 per hour
  • Employment type: Contract
  • Duration: Full-time, 12-month contract staffed via HireArt
  • Local requirement: Onsite role available to candidates local to the Foster City, CA area

Similar Jobs