Data Lake

Data Lake Developer Jobs

82 jobs found

Today

Lead Software Engineer - Python, AWS, Data, Agentic AI

JPMorganChase Houston, TX

Houston, TX

Today

Full time Backend Developer Python Amazon Emr

Apply Now

Job Description

Lead Software Engineer at JPMorganChase in Houston, TX, focused on delivering secure, scalable technology across multiple business functions with emphasis on Python, AWS, data-intensive workflows, and agentic AI.

Responsibilities

Architect and implement creative software solutions, handling development and debugging to address non-standard problems and decompose complex technical challenges.
Write secure, production-grade code and perform code reviews and debugging for teammates.
Proactively identify recurring issues and automate remediation to boost stability across applications and systems.
Lead evaluation sessions with external vendors, startups, and internal teams to assess architectures, credentials, and fit within the existing information architecture.
Foster communities of practice among software engineers to promote awareness and adoption of cutting-edge technologies.
Contribute to an inclusive, diverse, and respectful team culture.

Requirements

Formal training or certification in software engineering and application development with at least seven years of hands-on experience.
Hands-on expertise in Python and/or Java with a track record of delivering secure, production-quality code.
Extensive experience building cloud-native solutions on AWS covering compute, storage, security/IAM, networking, and observability.
Experience with LLMs and agentic systems, including orchestration patterns, tool calls, retrieval, structured outputs, evaluations, and safety/guardrails.
Background building data-intensive applications and workflows with Databricks (Spark, Delta, jobs/workflows).
Proficiency in JavaScript/TypeScript and React for modern web applications.
Experience with automation and continuous delivery practices; strong SDLC discipline including CI/CD, testing, code quality, and release controls.
Advanced understanding of agile delivery, application resiliency, and secure engineering practices.
Strong system design skills with the ability to make pragmatic trade-offs and align stakeholders.

Technologies

Python
Java
JavaScript
TypeScript
React
AWS
EC2
EKS
RDS
S3
Lambda
EMR
Glue
Databricks
Spark
Delta Lake
PySpark
Hadoop
Snowflake
Airflow

Benefits

Comprehensive health care coverage
On-site health and wellness centers
A retirement savings plan
Backup childcare
Tuition reimbursement
Mental health support
Financial coaching

View full job page →

Today

Senior Software Engineer II

RWE Americas, LLC Austin, TX

Austin, TX

Today

Full time Senior Automation Azure

Apply Now

Job Description

The Senior Software Engineer II will advance the Smart Data Pipeline platform on Microsoft Azure, combining hands-on development with architectural leadership to automate ingestion, cleaning, analysis, forecasting, alerting, and equipment lifecycle management for wind assets, with potential expansion to solar and energy storage systems in the future. This role blends technical delivery with cross-functional ownership and mentoring to shape technical strategy across the team.

Responsibilities

Architect and own scalable, cloud-native data pipeline solutions that integrate data systems across the SDP platform.
Define system architecture, data flows, and service boundaries for ingestion, pre-analysis, analysis, alerting, and monitoring pipelines.
Create and maintain architecture diagrams, blueprints, and reusable design patterns to guide development teams.
Drive technology decisions, platform standards, and architectural guidelines; lead architecture reviews and technical decisions.
Lead design and implementation of data collection, pre-analysis, analysis, forecasting, alerting, and VMM pipelines, ensuring data quality, reliability, and validation.
Own solution delivery on Azure based platforms including Azure Functions, pipeline scheduling, cloud storage, and data lake integration across DEV, UAT, and PROD environments.
Define and enforce best practices for CI/CD pipelines, infrastructure automation, and observability frameworks (logging, alerting, monitoring dashboards).
Lead integration of measurement systems (lidar, met mast, sensor networks) with enterprise IT platforms, ensuring data consistency, reliability, and traceability.
Lead end-to-end feature delivery (Design Build Deploy Operate); review and approve code across multiple services, enforcing security, performance, and coding standards.
Define and enforce engineering standards, coding guidelines, and architectural patterns; ensure compliance with enterprise architecture and data governance policies.
Act as the primary technical liaison between product owners, business stakeholders, and development teams, translating business requirements into technical architectures and delivery plans.
Mentor engineers across multiple levels; lead knowledge sharing, documentation, and capability-building initiatives that foster engineering excellence and continuous improvement.

Requirements

Bachelor’s degree in Computer Science, Software Engineering, or a related field, or equivalent industry certification or experience.
One or more Azure certifications (eg, AZ-900, AZ-204, AZ-305) or equivalent cloud platform credential is strongly preferred.
Minimum 7 years of relevant software engineering or data platform experience, including hands-on development and technical leadership roles.
Proven experience designing and managing complex, multi-stage data pipelines (ingestion, analysis, forecasting, alerting) with strong data cleaning, transformation, and validation workflows.
Experience architecting and delivering solutions on Azure based platforms, including Kubernetes (AKS), containers, Azure Functions, data lake integration, and multi-environment deployment management.
Experience with distributed systems, microservices, messaging systems (RabbitMQ), event-driven architecture, and API design and integration.
Experience designing access control for applications, including human-to-system and system-to-system access schemes.
Experience leading enterprise-scale IT implementations with coordination of multiple workstreams and stakeholder communications, using Agile or hybrid methodologies.
Strong frontend development background with React and TypeScript, backend development with C#, Python, Terraform, or equivalent IaC, plus SQL and experience with large datasets and data formats such as Parquet and CSV.
Experience with Linux and Windows servers, the ability to own Dockerfiles for both platforms, and ownership of SDP's third-party integrations (such as Bullfrog).
Experience working with global teams across Asia, Europe, and the Americas.
This position is an office-based role with some travel and visits to other RWE Americas office and field locations.
Must be able to sit, walk, or stand for long durations of time.
Additional experience in one or more of the following is a plus: Dapr, measurement systems (lidar, met mast, SCADA, sensors), time-series ML pipelines, climate or wind resource data processing, Azure Databricks and Delta Lake with medallion architecture, Delta Live Tables, Lakeflow, Apache Airflow, and cross-domain problem solving.
Strong diagnostics skills and the ability to design cost-effective systems for suboptimal deployments; excellent collaboration and communication abilities; and a commitment to mentoring and engineering excellence.

Technologies

Azure
Azure Functions
Azure Kubernetes Service (AKS)
Kubernetes
Containers
Data Lake
Azure Databricks
Delta Lake
Delta Live Tables
Lakeflow
Bullfrog
React
Typescript
C#
Python
Terraform
SQL
Parquet
CSV
Linux
Windows
Docker
Dapr
Apache Airflow
RabbitMQ
SCADA
Azure DevOps
Jira
Confluence
GitHub

Benefits

Medical coverage
Dental insurance
Vision coverage
Life insurance
Short-term disability
Long-term disability
401(k) with company match
Flexible Spending Accounts
Employee Assistance Program
Education assistance
Parental leave
Paid time off and holidays
Short-term incentive program

Location

Onsite in Austin, Texas, USA (78701). This role requires presence in the office with occasional travel to other RWE Americas offices and field locations as needed.

View full job page →

1d ago

Lead Data Engineer

INSPYR Solutions Doraville, GA 🇺🇸 Remote

Doraville, GA

1d ago

$60 - $70/hr Contract Artificial Intelligence Big Data Bigdata

Apply Now

Job Description

Lead Data Engineer at INSPYR Solutions will architect and optimize distributed data pipelines, establish Lakehouse architecture, enable ML workflows, ensure data quality, and improve production reliability.

Responsibilities

Architect, build, and optimize distributed data pipelines using Apache Spark in a high-volume, mission-critical environment.
Design and maintain enterprise Lakehouse infrastructure with Delta Lake, ensuring ACID compliance, lineage, auditability, and governance.
Develop automated ingestion frameworks (batch, streaming, and event-driven) across multiple cloud services and integration points.
Prepare feature-ready datasets and establish reproducible machine learning deployment patterns to enable ML workflows.
Lead platform-wide data quality, access control, and cataloging frameworks.
Implement cost optimization, cluster tuning, and performance engineering strategies.
Collaborate with Finance, BI, Operations, and ML teams to translate complex business needs into scalable data solutions.
Own production reliability, troubleshooting, and root cause analysis for data and ML pipelines.

Requirements

7+ years of experience in advanced data engineering with distributed compute technologies.
Expert Spark engineering experience including performance tuning, cluster configuration, partition strategies, and large dataset optimization.
Hands-on experience with Lakehouse architectures featuring ACID transactions, schema evolution, and governance frameworks.
Strong Python and SQL skills for large-scale data transformations.
Experience supporting machine learning pipelines or model operationalization.
Proven track record architecting cloud-native data platforms on Azure, AWS, or GCP.
Experience integrating diverse, complex data sources at enterprise scale.
Demonstrated ability to own mission-critical production systems.
Experience with distributed streaming frameworks such as Kafka or Event Hubs.
Experience building or supporting ML platforms, feature stores, or experiment tracking systems.
Background in data security, compliance controls, or audit-ready governance.
Experience automating data operations with CI/CD and infrastructure as code.

Technologies

Databricks, Python, SQL
Apache Spark, Delta Lake
MLflow, Notebooks
Hugging Face Transformers, LangChain, LlamaIndex
LLMs: Anthropic Claude, Meta LLaMA, Google Gemini
Kafka, Event Hubs, ADLS, S3, GCS
Git, GitHub, GitLab, Azure Repos, Databricks Repos
GitHub Actions, Azure DevOps
Unity Catalog, RBAC, REST APIs

Benefits

Comprehensive medical benefits
Competitive pay
401(k) retirement plan

Core Tools

Databricks (Spark, Delta Lake, MLflow, Notebooks)
Python & SQL
Apache Spark via Databricks
Delta Lake for lakehouse architecture

Cloud Platforms

Azure, AWS, or GCP
Cloud storage options: ADLS, S3, GCS

Data Integration

Kafka or Event Hubs for streaming data
Auto Loader for Databricks file ingestion
REST APIs
AI/ML and MLflow for model tracking and deployment
Hugging Face Transformers, LangChain, LlamaIndex for LLM integration
LLMs: Anthropic Claude, Meta LLaMA, Google Gemini

DevOps

Git-based version control (GitHub, GitLab, Azure Repos)
Databricks Repos for notebooks and code
CI/CD: GitHub Actions, Azure DevOps

Security & Governance

Unity Catalog for data governance
RBAC to manage access controls

View full job page →

1d ago

Data Engineer Hadoop, HIVE & Python

DenkenSolutions Inc Birmingham, AL

Birmingham, AL

1d ago

$58 - $58/hr Full time Python Analytics Big Data

Apply Now

Job Description

DenkenSolutions Inc is seeking a Data Engineer to design, develop, and support data engineering and analytics solutions using both on-premises tools and cloud services. This onsite role in Birmingham, AL offers a competitive rate of USD 58 per hour and the opportunity to work with a robust data tech stack centered on Hadoop, Hive, and Python within a collaborative, solutions-focused environment.

Responsibilities

Design, develop, and maintain data engineering and analytics solutions using Hadoop, Hive, and Python.
Build end-to-end data pipelines and analytics capabilities across on-premises tools and cloud services.
Create functional and technical designs for data engineering and analytics projects.
Model data across diverse schemas and integrate data from multiple sources.
Implement data quality tooling to ensure data reliability.
Develop and support big data solutions with Hadoop, Hive, and Spark.
Develop and support AI and ML models using R and Python.
Collaborate with on-premises tools such as MSBI (SSIS/SSAS), Informatica, Oracle GoldenGate, SQL, Oracle, and SQL Server.
Leverage Microsoft cloud services including Azure Data Lake, Azure Data Factory, Azure Databricks, Azure Synapse, Azure Key Vault, and Power BI.
Apply containerization techniques with Docker and OpenShift.
Follow Agile practices and DevOps CI/CD processes.
Design and develop data sourcing, enrichment, and delivery solutions using APIs and web services.

Requirements

Experience with batch and real-time (streaming) data processing frameworks.
Experience with data modelling, data access patterns, schemas, and data storage techniques.
Experience with data quality tooling.
Ability to create functional and technical designs for data engineering and analytics solutions.
Experience implementing data models across diverse schemas and integrating multiple data source types.
Hands-on development with big data technologies such as Hadoop, Hive, and Spark.
Hands-on with statistical models and AI/ML solutions using R and/or Python.
5+ years of hands-on experience designing, developing, testing, deploying, and supporting data engineering and analytics solutions using on-premises tools such as MSBI (SSIS/SSAS), Informatica, Oracle GoldenGate, SQL, Oracle, and SQL Server.
3+ years of hands-on experience designing, developing, testing, deploying, and supporting data engineering and analytics solutions using Microsoft cloud-based tools such as Azure Data Lake, Azure Data Factory, Azure Databricks, Python, Azure Synapse, Azure Key Vault, and Power BI.
Experience with containerization methodologies Docker, OpenShift, etc.
Experience with Agile as well as DevOps, CI/CD methodologies.
Hands-on experience designing and developing solutions involving data sourcing, enrichment and delivery using APIs & Web Services.

Technologies

Hadoop
Hive
Spark
R
Python
MSBI (SSIS/SSAS)
Informatica
Oracle GoldenGate
SQL
Oracle
SQL Server
Azure Data Lake
Azure Data Factory
Azure Databricks
Azure Synapse
Azure Key Vault
Power BI
Docker
OpenShift
APIs and Web Services

View full job page →

2d ago

Senior Data Engineer

firstPROPhiladelphia Philadelphia, PA

Philadelphia, PA

2d ago

$60 - $70/hr Contract Senior Analytics Automation

Apply Now

Job Description

The contract-to-hire Senior Data Engineer focuses on real-time streaming data pipelines for connected medical devices, operating in a hybrid environment in Philadelphia. The role leverages Azure Databricks, Azure services, and ML/AI data workflows. Compensation ranges from USD 60 to 70 per hour.

Responsibilities

Design and implement batch and streaming data pipelines on Azure and Databricks, with emphasis on real-time IoT telemetry within a Medallion architecture.
Create ETL/ELT workflows to ingest, transform, and validate large volumes of structured and unstructured data.
Develop and maintain data services, APIs, and microservices for application, analytics, and ML/AI teams.
Deploy real-time streaming solutions using Azure Event Hubs, Azure Stream Analytics, and related Azure integration patterns, focusing on cost-effective throughput, proper partitioning, and downstream delivery to Databricks.
Optimize production Databricks pipelines with PySpark, Spark SQL, and Delta Lake, including tuning for performance, reliability, and cost.
Troubleshoot and resolve complex pipeline issues across Databricks, Azure, and on-premises systems, including root cause analysis and corrective actions.
Collaborate with data analysts, software engineers, ML engineers, and business stakeholders to translate requirements into technical designs and delivery priorities.
Apply data quality, validation, and privacy-first practices, delivering reliable pipelines through software engineering standards, documentation, testing, and CI/CD.

Requirements

Bachelor's degree in Computer Science, Mathematics, Engineering, or a related technical field with 6+ years of professional experience in data engineering, analytics, or warehousing; or Master's degree in a related field with 3+ years of experience.
5+ years designing, building, and operating big data and real-time streaming pipelines across cloud and on-premises environments.
5+ years applying DevOps and CI/CD practices to data and analytics workloads.
Production experience building data services, APIs, or microservices for downstream data consumption.
Strong hands-on experience designing and building production-grade data pipelines on Databricks.
Demonstrated experience with Spark optimization and tuning in Databricks, including performance analysis, partitioning strategies, caching, shuffle optimization, and cost-aware pipeline design.
Strong experience with Azure cloud services for data engineering and streaming workloads.
Data quality, validation, and privacy-aware handling for regulated or sensitive data.

Technologies

Databricks
Spark
PySpark
Spark SQL
Delta Lake
Azure
Azure Event Hubs
Azure Stream Analytics
Azure Databricks
Medallion architecture

Outcomes

Onboard to the Azure Databricks environment and contribute to troubleshooting, stabilization, and optimization of existing batch and streaming data pipelines.
Stand up Azure streaming ingestion for telemetry data and deliver production-ready pipelines integrated with Databricks to support API, ML, and downstream analytics consumption within the Medallion architecture.
Design and deliver data services or consumption patterns that enable business and ML teams to access near real-time telemetry data reliably, securely, and at scale.

Work Hours and Travel Requirements

Willingness to assist in troubleshooting and analysis during off-hours production issues as needed.
The IT team operates in a hybrid environment requiring a minimum of two days per week in the downtown Philadelphia office.

View full job page →

2d ago

Senior Data Engineer, Engineering Data Analytics

NVIDIA Santa Clara, CA

Santa Clara, CA

2d ago

$168k - $270k/yr Full time Senior Analytics Big Data

Apply Now

Job Description

Based in Santa Clara, CA onsite, NVIDIA seeks a Senior Data Engineer to design and scale cloud-based data platforms and analytics for engineering data analytics, focusing on data models, pipelines, and AI enabled insights.

Responsibilities

Build and evolve trusted engineering analytics datasets, data models, and data products for semiconductor product, manufacturing, and test data.
Translate complex domain concepts into reliable data structures, metric logic, validation rules, and reusable analytics layers.
Own and improve curated data layers, including prep/fact tables, silver/gold datasets, semantic views, and analytics-ready outputs.
Partner with product engineering, UI, and data engineering teams to turn ambiguous engineering questions into scalable data solutions.
Define data quality checks, acceptance criteria, and validation frameworks for production analytics data.
Provide technical direction by defining standards, reviewing designs, and ensuring long-term maintainability.
Help guide the evolution of data architecture across modern warehouse, data lake, and lakehouse technologies such as Redshift, S3/Athena, and Databricks.
Support AI-enabled analytics by building well-governed, semantically clear datasets for AI-based exploration, anomaly detection, prediction, and recommendations.
Optimize data pipelines and analytics datasets for correctness, performance, scalability, reliability, and cost.

Requirements

Strong SQL skills, including advanced concepts such as window functions, CTEs, complex joins, aggregation patterns, query optimization, and analytical query design.
Strong Python skills, or equivalent experience building data-intensive software systems.
Experience designing data models, analytics datasets, data products, or application data layers.
Experience building or owning production data pipelines, data platforms, or analytics systems.
Solid understanding of data correctness, table grain, lineage, metric definitions, validation rules, and data quality standards.
Ability to learn complex technical domains and identify when data outputs are technically valid but semantically wrong.
Ability to work cross-functionally with domain experts, engineers, product/UI teams, and data engineering teams while providing technical ownership and judgment.
Interest in applied AI/ML and how trusted data foundations enable AI-based exploration, anomaly detection, predictive analytics, and recommendations.
Bachelor’s or Master’s degree in Computer Science or Computer Engineering or Electrical Engineering (or equivalent experience) and 8+ years of relevant experience.

Technologies

Python, SQL, Redshift, S3, Athena, Glue, EMR, Spark, Databricks, Delta Lake

Benefits

Equity
Benefits

Ways to Stand Out

Experience with semiconductor product engineering, test engineering, yield analytics, manufacturing analytics, quality, reliability, or hardware engineering data is a strong plus.
Experience with modern cloud data platforms and lakehouse technologies such as S3, Athena, Glue, Redshift, EMR, Spark, Databricks, Delta Lake, or similar technologies.
Experience with AI/ML enabled analytics, including LLMs, RAG, AI-based data exploration, natural-language-to-SQL, feature engineering, anomaly detection, prediction, or recommendation systems.
Experience building engineering analytics platforms, internal data products, or decision-support tools for technical users.

View full job page →

2d ago

Senior Data Engineer

E Logic Baltimore, MD

Baltimore, MD

2d ago

$120k+/yr Full time Senior Big Data Bigdata

Apply Now

Job Description

Senior Data Engineer role enabling IRS project delivery by leading data management, governance, and administration within Databricks and cloud native environments.

Responsibilities

Design, scale, and maintain data pipelines for intake, governance, and tenant services on the Enterprise Data Platform, ensuring reliability and scalability.
Optimize Databricks clusters, oversee workspaces, and implement platform policies to enforce standards and security.
Collaborate closely with cloud infrastructure and security teams to establish secure guardrails for data ingestion, access control, and cross team workflows.

Location

Baltimore, Maryland, onsite

Requirements

Minimum 8+ years of dedicated cloud technology experience, with AWS or Azure preferred.
3 to 5 years hands-on experience deploying, governing, and managing Databricks environments.
Strong proficiency in PySpark, SQL, Python, Delta Lake, and building robust ETL/ELT pipelines.
Ability to pass a federal government background check or Public Trust clearance.

Technologies

Databricks
PySpark
SQL
Python
Delta Lake
AWS
Azure

Compensation

Salary: USD 120,000 per year

Important Notice

This role is part of a proposal for Internal Revenue Services. Hiring is contingent upon the selection of the consultant. Selected candidates will be included in the proposal and must authorize the use of their resume for submission.