Senior PySpark Developer
Job Description
This Senior PySpark Developer role is focused on developing and optimizing distributed data pipelines and cloud-native lakehouse components.
Responsibilities
- Build and optimize PySpark pipelines for large-scale data ingestion and transformation.
- Develop Iceberg-based data models and manage cloud-native data lakes.
- Design and orchestrate Airflow workflows to implement ETL/ELT solutions.
- Troubleshoot distributed systems and provide support for CI/CD and cloud migration initiatives.
Requirements
- 8+ years of IT experience.
- 6+ years in Data Engineering.
- Experience with Airflow workflow design.
- Proficiency in SQL and data modeling.
- Strong AWS background including: S3, Glue, Lambda, ECS/EKS, EMR, RDS, IAM, CloudWatch.
- Hands-on experience with cloud-native data lake and Lakehouse development.
- Experience implementing Kafka and event-driven pipelines.
- Knowledge of data quality and governance frameworks.
- Domain experience in Healthcare or Insurance.
- Location: Remote.
- Visa: USC/GC.
Technologies
- PySpark
- Iceberg
- Airflow
- ETL/ELT
- AWS: S3, Glue, Lambda, ECS/EKS, EMR, RDS, IAM, CloudWatch
- Kafka
Compensation and Schedule
- $60.00 per hour
- Expected hours: 40.0 hours per week
- Work location: Remote