DeveloperJobs.io
← Back to all jobs

Job Description

This Senior PySpark Developer role is focused on developing and optimizing distributed data pipelines and cloud-native lakehouse components.

Responsibilities

  • Build and optimize PySpark pipelines for large-scale data ingestion and transformation.
  • Develop Iceberg-based data models and manage cloud-native data lakes.
  • Design and orchestrate Airflow workflows to implement ETL/ELT solutions.
  • Troubleshoot distributed systems and provide support for CI/CD and cloud migration initiatives.

Requirements

  • 8+ years of IT experience.
  • 6+ years in Data Engineering.
  • Experience with Airflow workflow design.
  • Proficiency in SQL and data modeling.
  • Strong AWS background including: S3, Glue, Lambda, ECS/EKS, EMR, RDS, IAM, CloudWatch.
  • Hands-on experience with cloud-native data lake and Lakehouse development.
  • Experience implementing Kafka and event-driven pipelines.
  • Knowledge of data quality and governance frameworks.
  • Domain experience in Healthcare or Insurance.
  • Location: Remote.
  • Visa: USC/GC.

Technologies

  • PySpark
  • Iceberg
  • Airflow
  • ETL/ELT
  • AWS: S3, Glue, Lambda, ECS/EKS, EMR, RDS, IAM, CloudWatch
  • Kafka

Compensation and Schedule

  • $60.00 per hour
  • Expected hours: 40.0 hours per week
  • Work location: Remote

Similar Jobs