Senior Data Engineer
Senior
Ai/ml Data Engineering
Change Data Capture
CI/CD
Data Architecture
Data Engineer
Data Engineering
Data Governance
Data Integration
Data Lakehouse
Data Pipeline
Data Pipelines
Data Platform
Data Processing
Databricks
Databricks Lakehouse
Databricks Workflows
Delta Lake
Lakehouse
Medallion Framework
Spark
SQL
Job Description
Hillwood Development Company LLC is building an enterprise Data Lakehouse that supports analytics and AI across the organization, and its Corporate IT team in Dallas is looking for a Senior Data Engineer to help design, evolve, and operationalize the platform. In this onsite role, you will deliver secure, scalable, governed data pipelines on the Databricks ecosystem, while keeping data accurate, cost-efficient, and ready for AI use cases.
What you’ll do
- Design, develop, and maintain scalable data pipelines in Databricks to integrate data from enterprise business systems, including JD Edwards, OneStream, Yardi, Entrata, Cecilian, Quickbase, UKG, Medius, and other applications.
- Develop and maintain data models across Bronze, Silver, and Gold layers using Hillwood’s enterprise data architecture and medallion framework.
- Implement incremental processing and change data capture (CDC) to keep datasets current, accurate, and efficient.
- Administer and maintain Unity Catalog, including schema management, data governance, permissions, and lineage tracking.
- Build production-ready pipelines with automated scheduling, monitoring, alerting, and recovery capabilities.
- Optimize workflows, jobs, and compute resources to improve performance, scalability, and cost efficiency.
- Establish engineering best practices, including version control, code reviews, automated testing, and CI/CD.
- Create and maintain technical documentation, standards, and operational procedures to support long-term sustainability.
- Implement data quality frameworks, validation rules, reconciliation processes, and monitoring to ensure accuracy and reliability.
- Investigate and resolve data quality issues by identifying root causes.
- Partner with business stakeholders and data owners to define standards, ownership, business definitions, and quality expectations.
- Ensure certified and business-critical datasets align with Hillwood’s data governance framework and quality standards.
- Use AI-assisted development tools, including coding assistants and platform-native AI capabilities, to accelerate development and improve quality.
- Support standards for the responsible use of AI-assisted software development and continuously evaluate emerging AI technologies.
- Prepare and optimize lakehouse data assets for AI, machine learning, and natural language access use cases.
- Design and build data foundations for advanced analytics and AI initiatives, including feature stores, vector search, retrieval architectures, and document-based data pipelines.
- Collaborate with AI engineers, developers, and stakeholders to integrate structured and unstructured data within the Databricks ecosystem.
- Support AI workloads and platform capabilities such as model serving, agent frameworks, and emerging GenAI use cases.
- Identify and onboard new structured, unstructured, and third-party data sources to expand enterprise platform value.
- Work directly with finance and business unit stakeholders to understand business processes, data requirements, and reporting needs.
- Validate data outputs and business metrics with data owners before datasets are certified for business use.
- Communicate technical concepts, risks, and trade-offs effectively to both technical and non-technical audiences.
- Partner with external engineering resources, review solution quality, and assume long-term ownership of delivered data pipelines and platform components.
- Serve as an internal subject matter expert for the architecture, operation, and ongoing evolution of Hillwood’s Enterprise Data Lakehouse platform.
What you’ll bring
- Strong proficiency in SQL, Python, PySpark, and modern data engineering development practices.
- Strong understanding of lakehouse architectures, data modeling, and enterprise data management concepts.
- Knowledge of data governance, data quality, metadata management, and data lineage best practices.
- Familiarity with source control, testing, code reviews, and CI/CD processes.
- Strong analytical and problem-solving skills, including ability to identify root causes and implement effective fixes.
- Ability to balance performance, scalability, maintainability, and cost when designing data solutions.
- Effective written and verbal communication skills, including explaining technical concepts to non-technical audiences.
- Strong organizational skills and attention to detail, with the ability to manage multiple priorities in a fast-paced environment.
- Ability to work independently while collaborating with business stakeholders, technical teams, and external partners.
- Curiosity and adaptability to evaluate emerging technologies, AI capabilities, and evolving data engineering best practices.
- Customer-focused mindset and commitment to delivering high-quality, business-driven solutions.
- Bachelor’s degree in Computer Science, Data Engineering, Information Systems, Software Engineering, or a related field.
- Minimum five years of data engineering experience, including at least two years building and supporting production solutions on the Databricks platform using PySpark, Spark SQL, Delta Lake, and Databricks Workflows.
- Experience designing, building, and maintaining enterprise data pipelines, integrations, and lakehouse architectures in a cloud environment.
- Hands-on experience administering and utilizing Unity Catalog, including metadata management, data lineage, security, and access controls.
- Experience operating production data platforms, including monitoring, alerting, incident response, troubleshooting, and performance optimization.
- Experience implementing data quality controls, reconciliation processes, and data governance standards.
- Experience integrating data from ERP, financial, operational, or other enterprise systems.
- Experience with Microsoft Azure and Git-based development and deployment tools, including Azure DevOps and/or GitHub.
- Experience leveraging AI-assisted development tools as part of a professional software engineering workflow.
- Experience supporting AI and machine learning initiatives through data engineering, feature engineering, vector search, retrieval-augmented generation (RAG), model-serving pipelines, or related technologies.
- Demonstrated success partnering with business stakeholders to understand requirements, validate data, and deliver business-focused solutions.
Technologies you’ll work with
- Databricks, SQL, Python, PySpark, Spark SQL, Delta Lake, Databricks Workflows
- Unity Catalog, Bronze, Silver, Gold, medallion framework, change data capture (CDC)
- CI/CD, Git, Azure DevOps, GitHub, Microsoft Azure
- JD Edwards, OneStream, Yardi, Entrata, Cecilian, Quickbase, UKG, Medius
- feature stores, vector search, retrieval architectures, document-based data pipelines
- model serving, agent frameworks, GenAI, retrieval-augmented generation (RAG)
Benefits
- Medical Insurance
- Dental Insurance
- Vision Insurance
- Flexible Spending Accounts (FSA)
- Health Savings Accounts (HSA)
- Vacation time
- Sick time
- Holidays
- Paid family caregiver leave
- Mental health and wellness resources
- Life Insurance
- Disability Insurance
- 401(k) with company match
- Tuition assistance
- Additional programs to support Associates and their families
Preferred experience
- Databricks Data Engineer certification or equivalent Databricks credentials.
- Experience supporting JD Edwards, OneStream, Yardi, Entrata, UKG, Medius, or similar enterprise platforms.
- Experience working with data in real estate, construction, energy, financial services, or related industries.
- Experience transitioning, stabilizing, and optimizing data solutions built by third-party consultants or implementation partners.
- Experience supporting LLM, generative AI, or agent-based solution development.
- Experience designing solutions that combine structured and unstructured data to support AI-enabled business applications.
Similar Jobs
L