Lead Data Engineer, Consumer

Lead hands-on Data Product Engineering to deliver trusted, scalable, reusable Silver-layer data products from source systems through the enterprise data platform at Versant Media.

Responsibilities

• Lead delivery of source-to-Silver data products
• Lead and develop a distributed/offshore Data Engineering team delivering from source ingestion through curated Silver-layer data products
• Translate product roadmaps and requirements into engineering plans, including milestones, estimates, dependencies, and delivery commitments
• Design and oversee ingestion, transformation, standardization, orchestration, and publication across operational, SaaS, file, streaming, API, and partner sources
• Ensure Silver-layer data products are validated, standardized, documented, reusable, performant, secure, and ready for Gold-layer analytics and AI consumption
• Establish repeatable patterns for batch, incremental, change-data-capture, and event-driven processing
• Build scalable engineering foundations using reusable frameworks, templates, libraries, and pipeline patterns to reduce repeated effort across products and domains
• Apply modern engineering practices: source control, peer review, automated testing, CI/CD, observability, release management, and incident remediation
• Define and maintain standards for naming, partitioning, schema evolution, error handling, replay/recovery, performance, cost management, and documentation
• Partner with the Data Platform team to leverage approved workspace, compute, storage, security, and deployment patterns
• Identify technical debt and drive pragmatic improvements to increase delivery speed, reliability, and maintainability
• Coordinate with Data Modelers to implement canonical entities, conformed dimensions, data contracts, source-to-target mappings, and enterprise modeling standards
• Partner with Product Managers and domain leaders to clarify intended outcomes, source-system realities, priority use cases, and acceptance criteria
• Coordinate dependencies with source-system owners, platform teams, analytics teams, and external partners
• Establish trusted, governed data practices: profiling, reconciliation, quality testing, freshness monitoring, lineage, and alerting for each delivered data product
• Ensure source-to-Silver traceability, including authoritative source identification, transformation logic, ownership, metadata, and data-quality expectations
• Implement access controls and sensitive-data controls, including classification, masking, retention, and regional/data-residency requirements
• Drive resolution of data defects, schema changes, pipeline failures, and quality issues through clear ownership and service-level expectations
• Set priorities, technical direction, delivery expectations, and quality standards for the engineering team
• Coach engineers in data engineering, cloud development, testing, observability, and product-oriented delivery practices
• Create an onshore/offshore delivery model with defined handoffs, overlap hours, documentation standards, ceremonies, and escalation paths
• Communicate delivery progress, risks, tradeoffs, and decisions to technical and business stakeholders
• Build a culture of ownership, continuous improvement, and reliable execution

Requirements

• 8+ years of data engineering, software engineering, or data-platform experience, including 2+ years leading engineers, technical workstreams, or large-scale delivery
• Proven experience designing and delivering enterprise-scale ingestion and transformation pipelines
• Strong hands-on expertise in SQL, Python, Spark/PySpark, and modern ELT/ETL patterns
• Experience with Databricks, Delta Lake, or comparable lakehouse platforms; experience with cloud storage, orchestration, and CI/CD
• Strong understanding of Bronze/Silver/Gold (or equivalent layered data-platform) patterns
• Experience implementing data quality, observability, lineage, metadata, and production support practices
• Ability to partner with architects, data modelers, product leaders, and domain stakeholders
• Experience leading distributed teams and creating delivery practices across time zones
• Strong communication skills, including explaining technical choices, risks, and tradeoffs to non-technical stakeholders

Technology

• SQL, Python, Spark, PySpark
• ELT, ETL
• Databricks, Delta Lake
• CI/CD
• Bronze/Silver/Gold
• change-data-capture

Preferred Qualifications

• Experience with Unity Catalog or comparable governance, catalog, and access-control capabilities
• Experience with data contracts, schema evolution, change data capture, APIs, event streaming, and large-volume data processing
• Experience supporting multiple data domains, global data products, or regional data-residency requirements
• Experience with infrastructure-as-code, DataOps, and automated environment provisioning
• Experience in a regulated, high-scale, or operationally sensitive environment

Location

• New York, NY (onsite)

Compensation

• USD 170,000 - 190,000 per yearly

Additional Information

• External candidates may be required to attend an in-person interview with a VERSANT Media employee at one of its locations prior to a hiring decision
• Reasonable accommodations are available for qualified individuals with disabilities or disabled veterans during the application and/or recruitment process ([email protected])
• Fair and equitable compensation practices commitment
• Versant Media is not accepting unsolicited assistance from search firms for this opportunity