The Lead Data Engineer at Navanta, LLC owns the data foundation that supports clean, modeled, reconcilable lakehouse data for public Call Report information and secure ingestion from bank-core systems as new client environments come online. This role partners closely with AI/ML, Security, and platform teams to deliver governed, queryable data products for Navanta’s AI platforms.
Key Responsibilities
- Design and implement the lakehouse architecture using Apache Iceberg (or similar) on object storage, including a catalog for table management and per-bank isolation, dbt models, and a query engine.
- Build secure, least-privilege ingestion from bank systems, using log-based CDC where permitted, with query-based and batch/SFTP fallbacks, plus an in-bank collector pattern.
- Own data modeling for the semantic and metric layer, covering deposits, concentration, uninsured exposure, asset quality, and peer groups.
- Manage schema drift, data quality, and reconciliation processes, ensuring ingestion is observable and recoverable.
- Collaborate with AI/ML teams on the structured-query path and with Security on PII classification at landing, aligned with regulatory data-handling requirements.
- Document data lineage, transformation logic, and access controls to support audit and exam readiness.
- Define and enforce data contracts, quality thresholds, and alerting for pipeline failures.
Required Qualifications
- 8–12+ years of data engineering experience with end-to-end ownership from ingestion through serving, including 2+ years in a lead or senior role.
- Strong Python and expert SQL, plus rigorous data modeling practices for analytics.
- Hands-on lakehouse experience with Iceberg/Delta/Hudi (or equivalent) and modern transformation tooling.
- Proven ability to build reliable pipelines from messy operational and transactional source systems.
- Comfort with CDC mechanics and the realities of sourcing from databases not directly controlled.
- Bachelor’s degree in computer science, mathematics, information systems, or a related field, or equivalent hands-on experience.
- Experience in financial services or a regulated data environment is strongly preferred.
Technologies
Python, SQL, Apache Iceberg, Polaris, Nessie, Lakekeeper, dbt, Trino, Presto, DuckDB, Debezium, Kafka, Redpanda, Dagster, Airflow, S3, MinIO, SQL Server, PostgreSQL, pgvector, Delta, Hudi.
Core Competencies
- End-to-end ownership of ingestion-through-serving pipelines with an emphasis on reliability and observability.
- Rigorous data modeling for analytics, including semantic layers, metric definitions, and reconciliable outputs.
- Security and compliance mindset, including PII handling, least-privilege access, and data governance aligned to regulatory guidance.
- Cross-functional collaboration with AI/ML and platform engineering to deliver governed, queryable data products.
Key Performance Indicators (KPIs)
- Data freshness and pipeline reliability, including meeting SLAs for data ingestion and bank-core feeds.
- Data quality score across key metrics versus source reconciliation.
- Time required to onboard a new bank’s data environment, from kickoff to a queryable lakehouse.
- PII classification coverage at landing and zero unauthorized data-access incidents.
- Semantic layer adoption, measured as the percentage of assistant queries resolved via governed metrics versus ad hoc SQL.
Work Structure & Expectations
This is a full-time role combining ongoing pipeline operations with initiative-based lakehouse build-out and new bank onboarding. Collaboration with AI/ML, platform engineering, and security teams is required, and the role includes an on-call rotation covering data pipeline reliability.
Location and Work Environment
- Location: Alpharetta, GA (onsite)
- Typical office environment
- Travel: up to 20% may be required
Physical Demands
- Regularly required to sit and use hands to finger, handle, or touch objects, tools, or controls.
- Frequently required to talk or hear.
- Occasionally required to stand; walk; and stoop, kneel, crouch, or crawl.
- Occasionally required to lift and/or move up to 10 pounds, usually waist high, up to 50 feet away.
- Specific vision abilities include close vision and the ability to adjust focus.
