Senior Data Engineer
Job Description
Gatik AI builds driverless technology, and this role is central to turning real-world fleet activity and simulation outputs into datasets teams can trust. As a Senior Data Engineer (on-site in Santa Clara, CA), you will own the data platform that powers training-ready, searchable, and versioned data for driverless operations.
What you’ll be doing
You will design, build, and run production-grade data pipelines that ingest, decode, transform, and store multi-sensor logs from Gatik’s autonomous truck fleet and simulation environments. This includes integrating structured and unstructured data so teams can label, curate, index, and catalog drives, events, and scenarios quickly.
- Develop internal tools and services for data labeling, curation, indexing, and cataloging to speed up discovery of drives, events, and scenarios.
- Partner with ML researchers, autonomy engineers, and data scientists to define schemas and APIs used for training, evaluation, and debugging.
- Build and maintain feature stores, metadata systems, and dataset versioning for both event-based signals and unstructured sensor data (camera, lidar, radar).
- Combine synthetic and simulated data with real-world logs to support hybrid training, scenario replay, and simulation workflows.
- Create scenario-mining and event-tagging pipelines that surface rare and safety-relevant situations such as unprotected turns, dock approaches, and adverse weather.
- Optimize for cost, latency, and traceability, maintaining full lineage from raw log to trained model and evaluation result.
- Work with simulation, cloud platform, and safety teams to automate closed-loop testing, regression analysis, and fleet performance analytics.
- Set standards for data quality and observability, and mentor engineers through design reviews and code reviews.
What you bring
- Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field, or equivalent practical experience.
- 8+ years building data-intensive software systems, ideally in autonomous driving, robotics, or large-scale ML environments.
- Strong proficiency in Python and SQL, with working familiarity with C++.
- Hands-on ETL/ELT pipeline experience using modern frameworks such as Apache Spark, Flyte, Union, Ray, or Airflow.
- Cloud-native architecture experience on AWS (S3, EC2, EKS, Glue, Athena) or equivalent Google Cloud services.
- Familiarity with sensor data (camera, lidar, radar, GPS/IMU) and serialization formats such as Protobuf, ROS/ROS 2 bags, and MCAP.
- Deep understanding of data quality, observability, and lineage in high-volume systems.
- A track record of building reliable, performant infrastructure supporting both ad-hoc exploration and repeatable production workflows.
Technologies you’ll use
Python, SQL, C++, Apache Spark, Flyte, Union, Ray, Airflow, AWS (S3, EC2, EKS, Glue, Athena), Google Cloud, Protobuf, ROS/ROS 2 bags, MCAP
Nice to have
- Experience in AD/ADAS, autonomous trucking, or robotics, especially with perception, prediction, or planning datasets.
- Familiarity with ML orchestration platforms such as Kubeflow, SageMaker, or Vertex AI.
- Experience with temporal and spatial data, including geospatial indexing (H3, S2, PostGIS), HD maps, and multi-sensor time-series alignment.
- Exposure to synthetic data generation, simulation logging, or scenario replay pipelines.
- Experience with lakehouse and table formats such as Delta Lake, Apache Iceberg, or Parquet at petabyte scale.
- Familiarity with safety-critical development practices and data traceability for safety cases or regulatory requirements.
- Strong software engineering fundamentals: CI/CD, testing, code review, and service deployment best practices.
- Experience working with cross-functional, distributed teams across research and production organizations.
Salary range: $180,000 - $260,000 per year.
E-Verify notice: By submitting your application, you acknowledge that Gatik participates in the U.S. Department of Homeland Security E-Verify program.