Lead Data Engineer role within JPMorganChase Corporate Technology in Jersey City, NJ, focusing on maintaining data pipelines and architectures to support business objectives.
Responsibilities
- Over seven years in data engineering, including more than two years guiding delivery and architecture across multi‑team data platforms.
- Hands‑on experience building and operating an AWS‑hosted Databricks Lakehouse.
- Extensive Delta Lake expertise, including ACID tables and partitioning.
- Proven Spark on Databricks experience, covering performance tuning, cluster sizing, skew mitigation, joins, caching, and file sizing.
- Experience designing both streaming and batch pipelines, including Structured Streaming, incremental processing, backfills, and late arriving data.
- Strong AWS fundamentals for data platforms, including S3, IAM, KMS, networking basics (VPC/security groups), and logging/auditing.
- Experience implementing data governance and security controls in Databricks, such as Unity Catalog, table/column permissions, and credential passthrough patterns as applicable.
- Ownership of reliability through monitoring/alerting, incident response, root cause analysis, and SLO/SLA management.
Requirements
- Architect the lake house by designing bronze, silver, and gold layers (or equivalents) and domain data products.
- Deliver ingestion at scale from AWS sources into Databricks (batch and streaming), including CDC where needed.
- Build maintainable pipelines using Delta Live Tables or standard Jobs with modular structure, testing, and documentation.
- Operational excellence through productionizing workloads via Databricks Workflows/Jobs, robust retries, checkpointing, idempotency, and safe re-runs.
- Governance by design with least privilege enforcement, data classification (PII), auditing, lineage/metadata, and controlled sharing/consumption.
- Performance and cost management by tuning Spark/Delta workloads, right‑sizing clusters, optimizing storage layout, and managing job/warehouse spend.
- Lead and mentor by setting engineering standards, conducting design reviews, improving code quality, and upskilling engineers in Spark/Databricks best practices.
- Cross‑functional delivery translating stakeholder needs into technical plans, communicating tradeoffs, and aligning with security and platform teams.
- CI/CD and IaC using Terraform for Databricks and AWS resources, with promotions across environments.
- Testing including unit and integration tests for transformations, data quality checks, contract testing, and replay/backfill procedures.
- Strong version control and code review discipline, with clear documentation and runbooks.
Technologies
- Databricks
- Delta Lake
- Spark
- Delta Live Tables (DLT)
- Unity Catalog
- Databricks Workflows/Jobs
- Terraform
- Amazon S3
- AWS IAM
- AWS KMS
- VPC
Benefits
- Health care coverage
- On‑site health and wellness centers
- Retirement savings plan
- Backup childcare
- Tuition reimbursement
- Mental health support
- Financial coaching
- Commission‑based pay
- Discretionary incentive compensation (cash and/or forfeitable equity)
About JPMorganChase
JPMorganChase is one of the oldest financial institutions, delivering innovative financial solutions to millions of consumers, small businesses and many of the world’s prominent corporate, institutional and government clients under the J.P. Morgan and Chase brands. Our history spans over 200 years and today we are a leader in investment banking, consumer and small business banking, commercial banking, financial transaction processing and asset management.
We offer a competitive total rewards package including base salary determined based on the role, experience, skill set and location. Those in eligible roles may receive commission‑based pay and/or discretionary incentive compensation, paid in the form of cash and/or forfeitable equity, awarded in recognition of individual achievements and contributions. These benefits include comprehensive health care.
We recognize that our people are our strength and the diverse talents they bring to our global workforce are directly linked to our success. We are an equal opportunity employer and place a high value on diversity and inclusion at our company. We do not discriminate on the basis of any protected attribute, including race, religion, color, national origin, gender, sexual orientation, gender identity, gender expression, age, marital or veteran status, pregnancy or disability, or any other basis.
About the Team
Our corporate functions cover finance and risk as well as human resources and marketing, forming an essential support structure that helps ensure our businesses, clients, customers and employees succeed.
