Lead Data Engineer - Pipelines, Spark Streaming and Spark Offline
Job Description
Responsibilities
- Collaborate with JPMorgan's lines of business and functions to deliver software solutions
- Architect, develop, and productionize efficient data pipelines, data services, and data platforms that contribute to business outcomes
- Design and implement scalable, efficient, and reliable data processing pipelines; analyze insights to drive and optimize results
- Design and develop features and entities for machine learning and rule-based use cases using Spark or other big data environments
- Act on opportunities to converge physical, IT, and data security architectures to manage access
- Apply reuse-first and AI-assisted practices within delivery and operations, ensuring traceability, auditability, and alignment with resiliency and security expectations
Requirements
- Formal training or certification in data engineering concepts with 5+ years of applied experience
- Proven experience leveraging enterprise-authorized AI capabilities to support data engineering workflows, with strong validation habits and awareness of data sensitivity
- Ability to review and validate AI-assisted outputs before use, escalating when uncertain and adhering to data handling requirements
- Proficient programming skills in Python and PySpark
- Experience across the data lifecycle, building data frameworks and working with data lakes
- Experience with batch and real-time data processing using Spark or Flink and feature engineering in both batch and real-time contexts
- Working knowledge of AWS Glue and EMR for data processing and real-time data processing, including features with Flink, Databricks live tables, or Spark streaming
- Experience with Databricks and Databricks live tables
- Experience building services with Glue, Lamida, EMR or Flask, and deploying them on AWS EKS or Kubernetes
- Experience with both relational and NoSQL databases
- Experience in ETL data pipelines for both batch and real-time processing, data warehousing, and NoSQL databases
Technologies
- Python
- PySpark
- Spark
- Spark Streaming
- Flink
- Hadoop
- Kafka
- Databricks
- Databricks live tables
- AWS Glue
- AWS EMR
- Flask
- Lamida
- AWS EKS
- Kubernetes
- Docker
Benefits
- Base salary
- Commission-based pay
- Discretionary incentive compensation (cash and/or forfeitable equity)
- Comprehensive health care coverage
- On-site health and wellness centers
- Retirement savings plan
- Backup childcare
- Tuition reimbursement
- Mental health support
- Financial coaching
About Us
JPMorganChase is among the oldest financial institutions, delivering innovative solutions to millions of consumers, small businesses, and a wide range of corporate, institutional, and government clients under the J.P. Morgan and Chase brands. With a history spanning more than 200 years, we are a leader across investment banking, consumer and small business banking, commercial banking, financial transaction processing, and asset management.
We offer a competitive total rewards package that reflects role, experience, skill set, and location. Eligible employees may receive commission-based pay and discretionary incentive compensation, paid in cash and/or forfeitable equity, based on individual achievements. A range of benefits and programs is available to support employee needs, subject to eligibility.
Our people are our strength, and the diverse talents they bring to our global workforce are central to our success. We are an equal opportunity employer and value diversity and inclusion. We do not discriminate on any protected attribute including race, religion, color, national origin, gender, sexual orientation, gender identity, gender expression, age, marital or veteran status, pregnancy or disability.
JPMorgan Chase & Co. is an Equal Opportunity Employer, including Disability/Veterans.
About the Team
J.P. Morgan's Commercial & Investment Bank operates as a global leader across banking, markets, securities services, and payments. Corporations, governments, and institutions worldwide entrust us with their business in more than 100 countries. The Commercial & Investment Bank provides strategic advisory, capital formation, risk management, and liquidity solutions on a global scale.