Data Engineer [214448]
Apache Airflow
AWS
Aws Cloudwatch
Aws Dms
Business Intelligence
Cdc
Change Data Capture
Dashboards
Data Architecture
Data Engineer
Data Engineering
Data Integration
Data Modeling
Data Pipeline
Data Pipelines
Data Platform
Data Processing
Data Transforms
Data Warehouse
Databases
Grafana
Integration
SQL
Streaming Data
Workflow Orchestration
Job Description
Build manufacturing-operations data infrastructure, from CDC ingestion to a Medallion lakehouse and semantic layers for analytics.
Responsibilities
- Design and build batch and streaming Change Data Capture (CDC) pipelines from scratch for manufacturing-operations data
- Ingest data from SAP sources (MES, HANA, SLT), on-premises databases, AWS DMS, and industrial sources such as PLCs
- Create and maintain Medallion lakehouse layers: Bronze, Silver, and Gold, aligned to manufacturing operations
- Model dimensional data for manufacturing use cases including unit traceability, work orders, non-conformance reporting, assembly line station results, logistics, and first-pass yield
- Develop incremental SQL and dbt models, including MERGE/upsert, SCD Type 2, and late-arriving data handling
- Process semi-structured test and diagnostic payloads using Python and PySpark
- Monitor pipeline health, data quality, and freshness SLAs with AWS CloudWatch alarms, log monitoring, and Grafana dashboards
- Implement automated data quality tests to validate ingested and transformed datasets
- Orchestrate production workflows with Apache Airflow
- Manage AWS ingestion infrastructure including S3, IAM, DMS, and Kinesis, using Git-based CI/CD
- Partner with cross-functional teams to deliver semantic layers that support BI dashboards and downstream analytical applications
Requirements
- 5 to 10 years of hands-on data engineering experience
- Proven experience building data pipelines in manufacturing, industrial, automotive, or semiconductor operations
- Direct, hands-on experience with SAP (ERP, MES, HANA, or SLT) and strong understanding of factory-floor concepts such as non-conformances, production testing/diagnostics, yield, work orders, or supply chain logistics
- Deep, hands-on experience building CDC pipelines from scratch across relational databases and operational/industrial sources
- Strong proficiency in dimensional modeling, Medallion (Bronze/Silver/Gold) architecture, and building semantic layers for analytics
- Advanced programming skills in Python, PySpark, and SQL
- Production experience in AWS: S3, DMS, Kinesis, CloudWatch, IAM
- Workflow orchestration experience with Airflow (or similar)
Technologies
- Change Data Capture (CDC)
- SAP, MES, HANA, SLT
- AWS DMS, AWS CloudWatch, Grafana
- Apache Airflow, S3, IAM, Kinesis
- Git-based CI/CD
- Python, PySpark, SQL, dbt
- MERGE, SCD Type 2
Bonus Points (Preferred)
- Production experience with dbt, Amazon Redshift, Databricks, or Grafana
- Experience structuring lakehouse data to support AI and agentic applications
Other Requirements
- Applicants must be authorized to work for ANY employer in the U.S. Role cannot sponsor or take over sponsorship of an employment Visa at this time
- #LI-Onsite
Location: Foster City, CA (onsite) | Compensation: USD 69 to 76 per hourly