Sr. Data Engineer
Job Description
Arlo Technologies, Inc. is hiring a Senior Staff Data Engineer to design and scale its data platform on Databricks.
Responsibilities
- Define and evolve data architecture on the Databricks Lakehouse using medallion patterns (Bronze/Silver/Gold) for batch and streaming workloads.
- Design, build, and maintain scalable ETL/ELT pipelines using Python, PySpark, and SQL.
- Implement and manage Unity Catalog, including catalogs and schemas, access controls, data lineage, auditing, and secure data sharing.
- Define and enforce data governance practices across data quality, metadata management, data classification, retention, privacy, and regulatory compliance.
- Build and maintain CI/CD pipelines for data workflows using Git and GitHub Actions, Azure DevOps, or Jenkins, including Databricks Asset Bundles with automated testing and deployment.
- Implement data quality checks, plus monitoring and alerting to support reliable, trusted data.
- Monitor and optimize Databricks jobs for performance, reliability, and cost.
- Mentor engineers, lead design reviews, and contribute to the data roadmap.
Requirements
- 5–6 years of experience in data engineering, building production data pipelines and architectures.
- Hands-on expertise in Databricks (including Spark, Delta Lake, and Workflows).
- Hands-on experience with Unity Catalog (access control, lineage, catalog and schema design).
- Strong understanding of data governance concepts: data quality, cataloguing, lineage, privacy, and compliance (including GDPR and CCPA).
- Advanced skills in Python (PySpark) and SQL.
- Experience with CI/CD and DataOps practices.
- Experience with data modelling and a major cloud platform: AWS, Azure, or GCP.
- Strong communication and cross-functional collaboration.
- Bachelor’s degree in Computer Science, Engineering, or a related field.
Technologies
- Databricks
- Spark
- Delta Lake
- Workflows
- Unity Catalog
- Python
- PySpark
- SQL
- Git
- GitHub Actions
- Azure DevOps
- Jenkins
- Databricks Asset Bundles
- AWS
- Azure
- GCP
Location
- Milpitas, CA (onsite)
Compensation
- USD 140,000 - 160,000 per year
- The total compensation package may also include other elements, including bonus, equity, and a full range of benefits.