Lead Data Engineer
Job Description
Prove’s Data Science team builds infrastructure that turns complex identity inputs into verified digital trust. As a Lead Data Engineer, you’ll design and operate high-throughput, DAG-based pipelines that support both real-time and batch inference used in revenue-generating products focused on securing transactions and reducing fraud.
This hybrid role is based in New York, NY and reports to the Director of Informatics (FLSA: Exempt). With $175,000 - 200,000 per year plus company bonus for Metro 2 areas, the scope is hands-on engineering, cross-functional collaboration, and continuous improvement of data systems.
What you’ll do
- Architect, build, and maintain resilient data pipelines that support real-time and batch inference, transforming raw intelligence into verifiable scientific statements.
- Convert multidisciplinary domain logic into efficient, maintainable, modular DAGs and execution workflows.
- Implement and advocate for data governance, metadata management, and semantic data ontologies.
- Work with Data Science, informatics, and product engineering teams to deliver high-integrity data extracts for revenue-generating products.
- Continuously evaluate and adopt modern data engineering tools, frameworks, and storage paradigms.
- Optimize distributed data processing pipelines for high availability, computational efficiency, and cost governance.
- Drive engineering excellence through code reviews, architectural discussions, automated testing, and CI/CD practices.
- Promote Prove’s cultural values of humility, passion, inclusion, and leadership.
What you bring
- 7+ years of professional experience in data engineering or building complex, data-driven software products.
- Deep, first-principles understanding of data systems, distributed compute engines, data modeling, and modern pipeline architecture.
- Proven ability to collaborate across technical and scientific functions, translating cross-disciplinary logic into concrete data systems.
- High intellectual curiosity and a proactive approach to experimentation and technological innovation.
- Hands-on experience with data orchestration tools such as Airflow, Dagster, or Prefect, plus large-scale distributed frameworks.
- Strong interest in learning about Prove’s products and markets through in-house and external training.
Technologies
- Airflow, Dagster, Prefect
- Linux/Unix, shell scripting
- CI/CD
Benefits
- Competitive salaries & a Bonus Plan (for eligible roles) and an Equity Plan.
- Modern Health for financial, mental, and physical wellness.
- 401(k) Retirement Plan & Match (US Offices) and Local Country Pension (International Offices).
- Unlimited Vacation and flexible hours.
- Comprehensive medical benefits for you and your family.
- Emotional & Physical Wellness: EAP and Prove Well-Being Reimbursement.
- Bottomless snacks & beverages for certain office locations.
- Daily GrubHub stipend for lunch when coming into the office (US Offices).
- Opportunity to work and connect with other talented Provers.
Preferred qualifications
- Advanced degree (Master’s or Ph.D.) in Computer Science, Mathematics, Statistics, Physics, or a related quantitative field.
- Experience architecting solutions for identity resolution, graph data structures, or fraud prevention systems.
- Fluency in Linux/Unix environments, command-line utilities, and shell scripting.
- Experience in high-growth or pre-IPO technology companies.
Compensation details (Metro 2)
Prove defines Metro 2 as the NYC metro area, Seattle metro area, Los Angeles metro area, and the Miami metro area. The anticipated salary pay range for this role is $175,000 - 200,000 plus company bonus.