Lead Data Engineer – Physical AI Platform, Data Engineering

Caterpillar is building the next generation of data capabilities for its Physical AI Platform, with a focus on delivering dependable data pipelines, microservices, and cloud-based data platforms. In this Lead Data Engineer role in Chicago, you will help shape data architecture and operational excellence using AWS and Python to support both business and engineering teams.

What you’ll do

• Collaborate with Principal Software Engineers and Data Architects to define solution architecture and drive technical alignment.
• Lead solution design and optimization of scalable Python data pipelines and microservices for both real-time and batch processing across enterprise platforms.
• Develop cloud-native data ingestion and streaming solutions using AWS services including Kinesis, S3, DynamoDB, and EventBridge.
• Own the design, implementation, and operational excellence of data integration frameworks and source data pipelines supporting CI Autonomy initiatives.
• Partner with business, product, and engineering stakeholders to translate complex needs into scalable data architectures, workflows, mappings, and system designs.
• Establish automated testing, data quality controls, and validation frameworks to support integrity, reliability, and compliance across distributed data ecosystems.
• Lead operational monitoring, performance tuning, and root-cause analysis for production data platforms using observability tools such as CloudWatch.

Core requirements

• Ability to lead analysis and resolution of complex issues within distributed data platforms, designing scalable and resilient solutions.
• Effective communication across teams, including constructive feedback and documentation that helps others understand and support data systems and processes.
• Experience leading the design and development of backend systems and data pipelines using Python, Java, and modern frameworks.
• Experience guiding data engineering delivery in an Agile environment through the full software development lifecycle.
• Capability to lead the design and integration of APIs, data pipelines, streaming platforms, and databases to enable reliable data exchange across enterprise systems.
• Expertise designing scalable, event-driven data systems and architectures that are reliable, maintainable, and aligned with business needs.
• Strong working knowledge of AWS services and data engineering tools to support testing and deployment, troubleshoot issues, and operate data solutions across environments.
• Ability to define and implement testing strategies including functional, performance, and data quality testing for reliable, high-performing outcomes.

Technologies you’ll work with

• Python, Java, AWS, Kinesis, S3, DynamoDB, EventBridge, CloudWatch
• CI/CD, Azure DevOps, Jira, Jenkins, SQL
• Relational databases, NoSQL databases, microservices, APIs
• CI Autonomy, Helios Data Platform

What top candidates typically bring

• Bachelor’s degree in Computer Science, Computer Engineering, or a related field.
• 8+ years of experience in data engineering or related disciplines with increasing responsibility.
• Extensive experience with modern, large-scale, complex Caterpillar data platforms such as Helios Data Platform.
• Strong foundation building and deploying Python solutions in production.
• Experience leading teams to build high-throughput, scalable data pipelines.
• Hands-on AWS data services experience (including Kinesis, S3, DynamoDB, EventBridge) at scale.
• Strong SQL skills, including data quality and validation practices.
• Experience deploying software using CI/CD tools such as Azure DevOps, Jira, Jenkins, etc.
• Experience building microservices that support real-time data ingestion.
• Experience working with relational and NoSQL databases.
• Ability to ensure data integrity across distributed and streaming systems.
• Experience with monitoring, testing, and automation in large-scale data environments.

Location and working model

• Chicago, IL (onsite)
• Full-time role working onsite five days a week
• Domestic relocation assistance is available
• Visa sponsorship is available for eligible applicants

Compensation

Salary range: USD 128,470 - 208,770 per year.

Benefits

• Medical, dental, and vision benefits
• Paid time off plan (Vacation, Holidays, Volunteer, etc.)
• 401(k) savings plans
• Health Savings Account (HSA)
• Flexible Spending Accounts (FSAs)
• Health Lifestyle Programs
• Employee Assistance Program
• Voluntary Benefits and Employee Discounts
• Career Development
• Incentive bonus
• Disability benefits
• Life Insurance
• Parental leave
• Adoption benefits
• Tuition Reimbursement
• These benefits also apply to part-time employees