Data Engineer

Shift4 is looking for a hands-on Data Engineer to build and improve SQL- and Python-based batch jobs that transform raw payments data in Snowflake into datasets and reports used by internal teams and merchant-facing consumers. This onsite role in Center Valley, PA focuses on correctness, reconciliation controls, Snowflake cost awareness, and secure handling of data in a payments environment.

What you will do

• Design, write, and maintain Python jobs that execute SQL over Snowflake to produce transaction, settlement, fee, and reconciliation datasets for internal and merchant-facing consumers.
• Write performant, readable SQL using window functions, CTEs, incremental logic, and semi-structured handling for JSON/VARIANT on large transaction tables.
• Package jobs as Docker containers and run them on the existing scheduler, ensuring each job is idempotent, rerunnable, parameterized, and observable with logging and alerting for failures or bad output.
• Implement data-quality checks and reconciliation controls so data is accurate and totals consistently align across sources and reporting periods.
• Manage Snowflake costs through warehouse sizing, clustering, query profiling, and avoiding wasteful scans.
• Partner with finance, operations, and product stakeholders to translate payments questions into correct, maintainable queries.
• Apply security controls for a payments environment, including secrets handling, least-privilege access, PII masking, and audit trails.
• Document the job library and contribute to code review, testing, and CI practices for the team.
• Explore and apply AI tools to improve efficiency across the development lifecycle, including query authoring, testing, and documentation.

Requirements

• 1 to 3 years of experience in data engineering, analytics engineering, or backend roles building production SQL-heavy data workloads.
• Strong SQL skills, including window functions, CTEs, aggregation and joins over large tables, and the ability to tune queries by reading execution plans.
• Hands-on Snowflake experience, including warehouse and credit/cost awareness, stages and COPY INTO, streams and tasks, time travel, and VARIANT/semi-structured data; plus experience with Snowpark or the Snowflake Python connector.
• Solid Python for data work, including pandas or Polars, parameterized SQL, configuration and secrets management, structured logging, error handling, and unit tests.
• Docker fundamentals, including building images, running scheduled batch jobs in containers, and debugging container failures.
• Basic Git and CI/CD habits, with comfort in code review and writing testable, rerunnable jobs.
• Strong attention to numerical correctness, with persistence in investigating when totals do not tie out.

Technologies

• Python, SQL, Snowflake
• Docker, pandas, Polars
• Git, Snowpark, Snowflake Python connector
• VARIANT, JSON, CTEs, window functions
• Generic GitHub-style code review and CI/CD

Nice to have

• Payments domain knowledge covering authorization, capture, settlement, refunds, chargebacks, interchange and scheme fees, plus merchant and acquirer data models and reconciliation.
• Experience with orchestration and transformation tooling such as Airflow, Prefect, Dagster, or dbt.
• AWS exposure from the developer side, including S3, IAM, Secrets Manager, and ECS/Fargate scheduled tasks.
• Data-quality frameworks and pipeline monitoring/alerting tools such as Great Expectations, dbt tests, or Soda.
• Experience in a PCI or otherwise regulated environment.
• Exposure to BI tools consuming outputs, such as Sigma, Tableau, Power BI, or Looker.