Databricks Data Engineer - Manager - Tech Cons - Open Location
Manager
Artificial Intelligence
Big Data
Bigdata
Cloud Data Engineering
Cloud Data Platform
Cloud Platform
Cloud Platforms
Data Analysis
Data Analytics
Data Architecture
Data Engineer
Data Engineering
Data Engineering Lead
Data Integration
Data Integration Tools
Data Lakehouse
Data Pipeline
Data Platform
Data Processing
Data Warehouse
Database
Databases
Databricks
ETL
Informatica
Information Technology (IT)
Lead Data Engineering
Software Development
Software Engineering
Spark
SQL
Job Description
EY is hiring a Manager of Data Engineering to lead delivery of complex cloud analytics solutions with a strong focus on Databricks.
Responsibilities
- Review and analyze business requirements, then translate them into technical requirements.
- Design, build, and run scalable data architecture and modeling solutions.
- Stay current on trends and emerging technologies to maintain a competitive edge.
- Lead data engineering workstream delivery and ensure quality across processes.
- Engage with clients daily through working sessions; identify opportunities for additional services.
- Create resource plans and budgets and manage engagement economics.
- Lead design and development of scalable data engineering solutions using Databricks on cloud platforms such as AWS, Azure, and GCP.
- Oversee architecture for complex cloud analytics solutions and ensure alignment with business objectives and best practices.
- Manage and mentor data engineers; promote innovation, collaboration, and continuous improvement.
- Collaborate with clients to understand analytics needs and deliver tailored solutions for business value.
- Maintain data quality, integrity, and security across the data lifecycle using data governance best practices.
- Drive end-to-end data pipeline development including ingestion, transformation, and storage using Databricks and other cloud services.
- Communicate effectively with stakeholders (technical and non-technical) about complex data concepts and project progress.
- Manage client relationships and expectations to sustain satisfaction and engagement.
- Produce comprehensive documentation and reports to communicate progress and outcomes.
Requirements
- Bachelor’s degree in computer science, Engineering, or a related field required; Master’s degree preferred.
- Typically 4 - 6 years of relevant experience in data engineering with a focus on cloud data solutions and analytics.
- Proven expertise in Databricks and experience with Spark for big data processing.
- Hands-on experience with AI-native data engineering practices, including generative AI and agentic workflows to accelerate pipeline development, testing, documentation, and optimization.
- Strong background in data architecture and design for complex cloud analytics solutions.
- Experience leading and managing teams with an emphasis on mentoring and talent development.
- Strong programming skills in Python, Scala, or SQL.
- Familiarity with AI-powered IDEs and coding assistants such as Cursor, GitHub Copilot, and Databricks Assistant for day-to-day development.
- Excellent problem-solving skills; able to work independently and collaboratively.
- Strong communication and interpersonal skills with a client management focus.
- Strategic leadership to align data engineering initiatives with organizational goals and drive strategic vision.
- Project management experience managing multiple projects and teams with delivery and scope responsibility.
- Stakeholder engagement experience, including executives, to understand needs and present solutions.
- Change management skills for client change related to data transformation and technology adoption.
- Risk management ability to identify risks in data projects and develop mitigation strategies.
- Technical leadership experience leading technical discussions and making architectural decisions.
Technologies
- Databricks, AWS, Azure, GCP
- Spark
- Generative AI, agentic workflows
- Python, Scala, SQL
- Cursor, GitHub Copilot, Databricks Assistant
- ETL
- AI-powered IDEs
Benefits
- Medical and dental coverage
- Pension and 401(k) plans
- Paid time off options
Location
- Chicago, IL (onsite)
- Anywhere in Country
Salary
- USD 125,500 - 261,600 per year
- Competitive
- NYC, Boston, Washington DC Metro, Washington State, Southern California offices: $150,700 - $251,200
- Bay Area California offices: $157,100 - $261,600
- All other US office locations (including Sacramento): $125,500 - $230,200
- Individual salaries depend on factors including education, experience, knowledge, skills, and geography.
Large-Scale Implementation Programs
- Enterprise Data Lake Implementation: Designed and deployed a cloud-based data lake for a Fortune 500 retail client, integrating multiple sources (ERPs, POS systems, e-commerce platforms) to support advanced analytics and reporting.
- Real-Time Analytics Platform: Managed development of a real-time analytics platform using Databricks for financial services, enabling real-time fraud detection and risk assessment via streaming ingestion and processing.
- Data Warehouse Modernization: Oversaw modernization of a legacy data warehouse to a cloud-native architecture for a healthcare provider, implementing ETL using Databricks and improving analytics and reporting accessibility.
Ideally, You’ll Also Have
- Experience with advanced data analytics tools and techniques.
- Familiarity with machine learning concepts and applications.
- Knowledge of industry trends and best practices in data engineering.
- Familiarity with cloud platforms (AWS, Azure, GCP) and their data services.
- Knowledge of data governance and compliance standards.
- Experience with machine learning frameworks and tools.
Travel
- Travel may be required regularly based on client needs.