DeveloperJobs.io
← Back to all jobs

Job Description

Build manufacturing-operations data infrastructure, from CDC ingestion to a Medallion lakehouse and semantic layers for analytics.

Responsibilities

  • Design and build batch and streaming Change Data Capture (CDC) pipelines from scratch for manufacturing-operations data
  • Ingest data from SAP sources (MES, HANA, SLT), on-premises databases, AWS DMS, and industrial sources such as PLCs
  • Create and maintain Medallion lakehouse layers: Bronze, Silver, and Gold, aligned to manufacturing operations
  • Model dimensional data for manufacturing use cases including unit traceability, work orders, non-conformance reporting, assembly line station results, logistics, and first-pass yield
  • Develop incremental SQL and dbt models, including MERGE/upsert, SCD Type 2, and late-arriving data handling
  • Process semi-structured test and diagnostic payloads using Python and PySpark
  • Monitor pipeline health, data quality, and freshness SLAs with AWS CloudWatch alarms, log monitoring, and Grafana dashboards
  • Implement automated data quality tests to validate ingested and transformed datasets
  • Orchestrate production workflows with Apache Airflow
  • Manage AWS ingestion infrastructure including S3, IAM, DMS, and Kinesis, using Git-based CI/CD
  • Partner with cross-functional teams to deliver semantic layers that support BI dashboards and downstream analytical applications

Requirements

  • 5 to 10 years of hands-on data engineering experience
  • Proven experience building data pipelines in manufacturing, industrial, automotive, or semiconductor operations
  • Direct, hands-on experience with SAP (ERP, MES, HANA, or SLT) and strong understanding of factory-floor concepts such as non-conformances, production testing/diagnostics, yield, work orders, or supply chain logistics
  • Deep, hands-on experience building CDC pipelines from scratch across relational databases and operational/industrial sources
  • Strong proficiency in dimensional modeling, Medallion (Bronze/Silver/Gold) architecture, and building semantic layers for analytics
  • Advanced programming skills in Python, PySpark, and SQL
  • Production experience in AWS: S3, DMS, Kinesis, CloudWatch, IAM
  • Workflow orchestration experience with Airflow (or similar)

Technologies

  • Change Data Capture (CDC)
  • SAP, MES, HANA, SLT
  • AWS DMS, AWS CloudWatch, Grafana
  • Apache Airflow, S3, IAM, Kinesis
  • Git-based CI/CD
  • Python, PySpark, SQL, dbt
  • MERGE, SCD Type 2

Bonus Points (Preferred)

  • Production experience with dbt, Amazon Redshift, Databricks, or Grafana
  • Experience structuring lakehouse data to support AI and agentic applications

Other Requirements

  • Applicants must be authorized to work for ANY employer in the U.S. Role cannot sponsor or take over sponsorship of an employment Visa at this time
  • #LI-Onsite

Location: Foster City, CA (onsite) | Compensation: USD 69 to 76 per hourly

Similar Jobs