Senior AWS Python Developer / PySpark Data Engineer

System One is hiring a Senior Python and PySpark Data Engineer to support hands-on AWS big data and data processing initiatives. This hybrid role spans Lafayette, LA, Knoxville, TN, Columbia, SC, and Birmingham, AL, where you will build scalable data pipelines and strengthen production performance across large datasets.

The work centers on scalable Python and Spark-based processing on AWS, with a strong emphasis on SQL and operational reliability. You will partner with technical and business teams while contributing CI/CD automation and troubleshooting across application and data processing workflows.

What you’ll do

• Develop scalable Python applications and data processing solutions
• Integrate APIs to connect services and data sources
• Write and optimize complex SQL queries, including data transformations
• Process large datasets using Spark and AWS services
• Troubleshoot application and data processing performance issues
• Contribute to CI/CD automation
• Work closely with technical and business teams across the organization

What you bring

• 8+ years of hands-on Python development, including data-intensive and big data applications
• Strong Python fundamentals with experience building scalable, production-grade applications
• 5+ years working with Apache Spark / PySpark, including troubleshooting and optimizing Spark jobs
• 5+ years hands-on AWS experience, including Amazon EMR, AWS Lambda, EventBridge, Step Functions, Redshift, S3, and Glue
• Advanced SQL experience, including complex queries, data transformations, joins, and recursive CTEs
• Experience developing and consuming REST APIs and integrating with external services and data sources
• A strong background working with large datasets, including manipulation, transformation, analysis, and processing
• Experience with big data technologies such as Hadoop, EMR, Spark, and Hive
• 4+ years of experience with AI-related technologies or applications
• Hands-on CI/CD and infrastructure automation experience using GitLab and Terraform
• Strong debugging, performance tuning, analytical, and problem-solving skills
• Ability to collaborate effectively across development teams and in a matrixed organization
• A Bachelor’s degree in Computer Science, Information Systems, or a related field

Technologies

• Python, PySpark, Apache Spark
• SQL
• Amazon EMR, AWS Lambda, Amazon EventBridge, AWS Step Functions
• Amazon Redshift, Amazon S3, AWS Glue
• Hadoop, Hive
• REST APIs
• GitLab, Terraform, CI/CD, AWS services

Type: Direct Hire
Work model: Hybrid
Minimum experience: 4 years
Location: Lafayette, LA · Knoxville, TN · Columbia, SC · Birmingham, AL