The Data Engineering Lead serves as the technical owner of Builders’ Azure Databricks lakehouse platform for property and casualty insurance. This role leads the architecture, modernization, governance alignment, and team leadership needed to deliver a scalable, secure, reliable, high-performing data platform.
Location
Atlanta, GA (onsite)
Summary
The Data Engineering Lead is responsible for designing, developing, and evolving the Azure Databricks lakehouse platform. The position covers lakehouse architecture, phased migration from Guidewire and legacy systems, data modeling for property and casualty workflows, and operational excellence across pipelines. Governance and security are reinforced through Unity Catalog, including implementation of access controls and support for lineage and data quality rules.
Responsibilities
- Design the Databricks lakehouse architecture using Delta Lake bronze/silver/gold medallion layers for property and casualty policy, rating, claims, billing, reinsurance, and customer data.
- Plan and lead phased data migration from Guidewire (PolicyCenter, ClaimCenter, BillingCenter) and legacy systems into Azure Databricks, including historical loads, incremental/CDC loads, reconciliation, and decommissioning support.
- Model property and casualty insurance data including accounts, policies, insured risks and exposures, coverages, endorsements, premiums, claims, reserves, payments, and recoveries.
- Deliver data for Actuarial and Finance, including written/earned/unearned premium, loss triangles, IBNR inputs, loss ratios, Schedule P, and NAIC statutory reporting.
- Integrate third-party sources such as geospatial and weather data.
- Define Unity Catalog design including metastore, catalogs, schemas, development/test/production environments, naming conventions, permissions, row/column-level security, and lineage.
- Manage pipeline orchestration and operations in Databricks, including job scheduling and dependencies with Databricks Workflows (Lakeflow Jobs), retries, alerting, SLA monitoring, and production failure handling with runbooks.
- Improve performance and cost through Spark tuning, partitioning and liquid clustering, Photon utilization, and appropriate cluster and Databricks SQL warehouse sizing.
- Partner with IT (Information Technology) and the Platform Engineer on storage, networking, security, identity, IaC, and CI/CD.
- Set engineering standards for pipelines, testing, code review, and documentation; lead, mentor, and review Data Engineers and junior contractors.
- Partner with the data governance lead to implement access controls, PII masking, lineage, and data quality rules.
- Perform other duties as assigned.
Requirements
- Associate degree or an equivalent combination of education and experience.
- Ten or more years of experience in data engineering, including six or more years of hands-on experience with Databricks (PySpark, Spark SQL, Delta Lake), plus experience leading or mentoring engineering teams.
- Databricks Data Engineer Professional certification; Azure data or solutions architect certification is preferred.
- Knowledge of Unity Catalog governance model, permissions, row/column-level security, lineage, and system tables.
- Experience leading large data migrations from core insurance and legacy systems into a lakehouse or data warehouse, including property and casualty insurance data and understanding of the policy and claims lifecycle and core P&C KPIs.
- Knowledge of data modeling including dimensional modeling, data vault, SCD patterns, and insurance data models.
- Technical proficiency in Spark internals and performance tuning, streaming (structured streaming, Auto Loader), CDC patterns, MERGE, and Lakeflow/Delta Live Tables.
- Expert-level skill in SQL and Python, with software engineering practices such as testing, code review, and modular design.
- Knowledge of data infrastructure including ADLS Gen2, Azure Data Factory, Event Hubs, Key Vault, Entra ID, VNets, and Private Link.
- Experience scheduling and monitoring production pipelines with Databricks Workflows and/or Azure Data Factory, including dependencies, parameterization, and monitoring using Databricks system tables.
- Experience with Git-based version control (GitHub or Bitbucket), including branching strategies and pull request reviews, plus Databricks Git folders.
- Experience with IaC and CI/CD for data, including Terraform, Databricks Asset Bundles, Azure DevOps, GitHub Actions, or Bitbucket Pipelines.
- Experience with Guidewire data (Guidewire DataHub/InfoCenter, Cloud Data Access); supporting actuarial reserving and pricing for property and casualty lines is preferred, as is familiarity with API and integration tools such as MuleSoft.
- Knowledge of insurance data standards such as ACORD is preferred.
Technologies
Azure Databricks, Delta Lake, Unity Catalog, Lakeflow, Databricks SQL, Databricks Workflows, PySpark, Spark SQL, Spark, Delta Live Tables, structured streaming, Auto Loader, MERGE, ADLS Gen2, Azure Data Factory, Event Hubs, Key Vault, Entra ID, VNets, Private Link, GitHub, Bitbucket, Terraform, Databricks Asset Bundles, Azure DevOps, GitHub Actions, Bitbucket Pipelines, Guidewire DataHub/InfoCenter, Cloud Data Access, MuleSoft, ACORD, Python, SQL, IaC, CI/CD.
Benefits
- Competitive Salary
- Bonus Structure
- Profit Sharing
- Medical, Dental, Vision Insurance
- Employer Paid Short Term Disability
- Employer Paid Long Term Disability
- Employer Paid Life Insurance
- Voluntary Life Insurance
- 401K with Company Match
- PTO
About Builders
Builders is a mid-sized mutual with remarkable strengths. Rated A by AM Best, Builders has forged rock-solid financial strength and a reputation for reliability and fairness in fulfilling promises to customers. Kind, collaborative, and customer-centric teams foster a rewarding atmosphere of excellence, trust, and mutual respect, and Builders has received the “Culture Excellence” honors from Top Workplaces.
Builders leverages supportive partnerships with knowledgeable independent agencies to drive better services and protection for policyholders. Its financial excellence, people, and partnerships are intended to build outstanding outcomes and peace of mind for agents and clients, aligning with “Insurance Built Strong®.”
Equal Opportunity Employer
Builders Insurance Group is an equal opportunity employer. Applicants are welcome from all walks of life, and the company does not discriminate based on any protected status.
