Senior Data Engineer

NTT Ltd is seeking a Senior Data Engineer in Charlotte, North Carolina to design and optimize scalable data pipelines using Databricks and related technologies. This role supports AI-driven document processing and ontology-based data modeling, including building ingestion and extraction workflows and delivering standardized JSON outputs for downstream underwriting processes.

Responsibilities

• Design, develop, and optimize scalable data pipelines using Databricks, Python, PySpark, and SQL.
• Build document ingestion, classification, extraction, validation, and transformation pipelines for PDFs, emails, attachments, Excel/XLSX files, insurance applications, Statements of Values (SOVs), and other business documents.
• Integrate LLMs and AI models into data-processing workflows to extract structured information from unstructured and semi-structured documents.
• Troubleshoot false positives, incorrect metadata, and inaccurate field values produced by LLM-based extraction pipelines.
• Implement document classification, source-specific extraction rules, validation and reconciliation procedures, and quality-control mechanisms.
• Transform extracted information into standardized JSON structures for downstream applications and underwriting workflows.
• Design and implement ontology-based data models to represent insurance entities, business concepts, and relationships.
• Map extracted document attributes to standardized ontology concepts and metadata structures.
• Optimize processing of large and complex Excel/XLSX and SOV files, including performance, scalability, and concurrency considerations.
• Implement reliable data persistence using PostgreSQL, Lakebase, or related database technologies.
• Develop and maintain APIs and data products to support downstream application integration.
• Collaborate with brokerage, underwriting, AI, and data engineering teams to align extraction outputs with business requirements.
• Monitor production pipelines, troubleshoot data-processing issues, and continuously improve reliability, performance, and data quality.

Requirements

• 7+ years of experience in data engineering, software development, or production data pipeline development.
• 4+ years hands-on experience with Databricks and scalable data-processing pipelines.
• 5+ years experience with Python, PySpark, and SQL.
• 3+ years experience processing unstructured and semi-structured data, including PDFs, emails, Excel files, JSON, and business documents.
• 2+ years experience integrating LLMs or AI models into document-processing or data-extraction pipelines.
• 3+ years experience implementing structured data extraction, document classification, validation, reconciliation, and data-quality controls.
• Strong understanding of data modeling, schemas, metadata, entity relationships, and ontology or semantic modeling concepts.
• 2+ years experience designing JSON outputs, APIs, and data products for downstream application consumption.
• 2+ years experience with PostgreSQL, Lakebase, or similar relational database technologies.
• 2+ years experience troubleshooting production performance, scalability, concurrency, and data-processing issues.
• Strong understanding of cloud-based data architectures, distributed processing, and pipeline optimization.
• Strong analytical, problem-solving, communication, and cross-functional collaboration skills.

Technologies

• Databricks
• Python
• PySpark
• SQL
• LLMs
• AI models
• PostgreSQL
• Lakebase
• JSON
• APIs
• Excel/XLSX
• PDFs
• Emails

Location and Travel

• Location: Charlotte, North Carolina, United States (onsite)
• Travel: NA

Nice to Have

• Experience in insurance, commercial insurance, brokerage, underwriting, or risk management.
• Experience processing insurance applications, Statements of Values (SOVs), policy documents, and underwriting submissions.
• Experience with LLM frameworks, prompt engineering, structured outputs, and AI evaluation techniques.
• Experience building custom ontologies, knowledge graphs, or semantic data layers.
• Experience with document AI, OCR, information extraction, and document intelligence platforms.
• Experience with Delta Lake, Lakehouse architecture, Databricks Workflows, and Delta Live Tables.
• Experience with cloud platforms such as AWS, Azure, or Google Cloud.
• Experience optimizing large-scale document ingestion and distributed processing workloads.

Education

Bachelor’s degree in Computer Science, Information Technology, Data Engineering, Engineering, or a related field, or equ.