Hadoop

Hadoop Developer Jobs

4 jobs found

Today

Senior Python Developer with Spark

CGI Group, Inc. Reston, VA

Reston, VA

Today

$99k - $154k/yr Full time Python Senior Apache Spark

Apply Now

Job Description

Senior Python Developer with Spark to design, build, and optimize large-scale data processing systems in a cloud-native AWS environment, based in Reston, VA with hybrid work options.

Responsibilities

Design, implement, and optimize expansive data processing pipelines within a cloud-native AWS framework.
Build high-performance data pipelines using Spark and Python while handling complex relational datasets.
Identify bottlenecks in data workflows, optimize Spark jobs, and author advanced SQL to support hierarchical and analytical use cases.
Collaborate with cross-functional teams to deliver scalable, reliable data solutions that support critical business functions in a financial services setting.

Requirements

8+ years of hands-on Python for data engineering, including building and maintaining data pipelines.
Deep expertise in Apache Spark, including partitioning, caching, broadcast joins, shuffle optimization, understanding of DAGs/stages/tasks, and memory/resource management.
Extensive experience with big data ecosystems such as Hadoop, Hive, and EMR.
Advanced proficiency in SQL, including recursive CTEs for hierarchical data, query optimization, indexing strategies, and execution plan analysis.
Strong experience with AWS services including EMR, Lambda, Step Functions, EventBridge, Redshift, S3, and Glue.
Experience building and consuming APIs, along with data transformation and ingestion workflows.
Proven ability to work with large-scale datasets, performing data analysis and extracting actionable insights.
Familiarity with data modeling concepts (normalized/denormalized structures, handling hierarchical data).
Hands-on experience with CI/CD pipelines and tools such as GitLab and Terraform.
Strong understanding of performance troubleshooting in distributed systems and identifying bottlenecks.
Ability to clearly explain technical decisions, especially around Spark optimization and SQL logic.
Strong analytical thinking, problem-solving skills, and attention to detail.
Effective collaboration skills in cross-functional, matrixed environments.
Education: Bachelor's degree in Computer Science, Information Systems, or a related field.
Desired: Experience in financial services or regulated environments.
Desired: Exposure to data visualization tools.
Desired: Familiarity with event-driven architectures on AWS.

Technologies

Python
Apache Spark
Hadoop
Hive
EMR
AWS
Lambda
Step Functions
EventBridge
Redshift
S3
Glue
SQL
GitLab
Terraform
APIs

Benefits

Competitive compensation
Comprehensive insurance options
Matching contributions through the 401(k) plan and the share purchase plan
Paid time off for vacation, holidays, and sick time
Paid parental leave
Learning opportunities and tuition assistance
Wellness and Well-being program

Skills

Amazon Web Services Cloud
Detail-oriented
GitHub
Problem Solving
Python
SQL
Terraform

View full job page →

2d ago

Hadoop and Spark Developer

Infosys Charlotte, NC

Charlotte, NC

2d ago

Full time Big Data Bigdata Data Engineer

Apply Now

Job Description

The Hadoop and Spark Developer role at Infosys is based onsite in Charlotte, NC. The position emphasizes requirements capture, design discussions, coding, testing, integration, monitoring, and documentation, with a strong focus on the software development lifecycle and ongoing learning.

Responsibilities

Assist with targeted interviews and workshops to elicit user needs and ensure precise documentation of stakeholder requirements.
Participate in design discussions to define software specifications and contribute to the development of process documentation.
Develop, refine, and document code for new software applications in alignment with detailed design specifications to satisfy user needs.
Execute test plans, analyze results, and report defects to the development team to support software performance improvements.
Support the integration of new features to ensure smooth transitions within the current system infrastructure.
Monitor and troubleshoot live applications, promptly resolving issues and documenting maintenance procedures for ongoing support.
Systematically document coding practices, testing methodologies, and technical solutions to enrich the team's knowledge base.

Assigned job role area of responsibility (Technology Consultant 1)

Assist in conducting focused interviews and workshops to gather user requirements and ensure precise stakeholder documentation.
Participate in design discussions to capture software specifications and support the creation of process documentation.
Develop, refine, and document code for new software applications in line with detailed design specifications to meet user requirements.
Execute test plans, analyze results, and report defects to the development team to drive software performance enhancements.
Support the integration of new software features, enabling smooth transitions within the current system landscape.
Monitor and troubleshoot live applications, delivering prompt issue resolution and maintaining comprehensive maintenance documentation.
Systematically document coding practices, testing methodologies, and technical solutions to enrich the team's knowledge base.

Your contribution to the team

An agile mindset paired with clear communication and strong problem-solving abilities.
Ability to contribute across multiple areas of the software development lifecycle.
Commitment to developing innovative solutions that advance progress and address future challenges.
Proactive in monitoring and resolving production issues to prevent escalation.
Commitment to ongoing learning, knowledge sharing, and contributing to team development.

Requirements

Agile mindset with strong communication and problem-solving abilities.
Capability to work across multiple stages of the SDLC.
Focus on innovating solutions that drive progress and address future challenges.
Proactive monitoring and resolution of production issues to prevent escalation.
Commitment to continuous learning, knowledge sharing, and fostering team development.

View full job page →

2d ago

Power BI Developer - Freelance

Reapplix Inc Remote 🌍 Remote

Remote

2d ago

$100 - $200/hr Freelance Analytics API Azure

Apply Now

Job Description

Remote freelance Power BI Developer role focused on data visualization, analytics, and cloud-based data platforms; responsible for designing dashboards, integrating data sources, and building ETL data pipelines.

Responsibilities

Create and refine Power BI reports and dashboards that clearly convey essential metrics and trends
Consolidate data from AWS, Azure Data Lake, Oracle, Microsoft SQL Server, and Hadoop ecosystems to produce unified analytical views
Build ETL workflows using Talend, Informatica, and custom scripts in Python and Shell to ready data for analysis
Collaborate with cross-functional teams to capture business requirements and translate them into technical solutions
Create and maintain RESTful APIs to enable seamless data exchange across platforms
Perform analyses with advanced techniques including model training, big data processing with Spark and Apache Hive, and linked data integration
Maintain data quality and consistency through solid database design, analytical rigor, and adherence to data warehousing best practices
Participate in agile processes, including sprint planning and reviews, to drive continuous improvement

Requirements

Proven track record as a Power BI Developer or related role with a solid portfolio of dashboards and reports
Strong SQL skills (Microsoft SQL Server, Oracle), plus Python, VBA, Bash, and Shell Scripting for automation and data manipulation
Hands-on experience with cloud platforms including AWS and Azure Data Lake for scalable storage and processing
Familiarity with big data technologies such as Hadoop, Spark, Apache Hive, and related ecosystems
Experience with ETL tools like Talend or Informatica for building data pipelines
Understanding of database design principles for data warehouses and linked data architectures
Experience working in Agile development environments with a focus on analytics projects
Strong analytical abilities to interpret complex datasets and derive actionable insights
Knowledge of Looker or similar BI tools is advantageous; familiarity with model training techniques is a plus

Technologies

Power BI
AWS
Azure Data Lake
Oracle
Microsoft SQL Server
Hadoop
Spark
Apache Hive
Talend
Informatica
Python
VBA
Bash
Unix shell
Shell Scripting
RESTful APIs
Looker

View full job page →

2d ago

Lead Software Engineer - Agentic

JPMorganChase Houston, TX

Houston, TX

2d ago

Full time Backend Developer Apache Airflow Apache Spark

Apply Now

Job Description

The Lead Software Engineer - Agentic role at JPMorganChase, within the Corporate Sector, focuses on delivering secure, production-grade code and guiding architectural decisions across agentic and related domains to support the firm’s strategic objectives.

Responsibilities

Develops innovative software solutions, participates in design, implementation, and debugging, and applies approaches that go beyond routine methods to resolve complex technical challenges
Writes secure, production-ready code and conducts reviews and debugging of colleagues' work to maintain quality
Identifies opportunities to automate repetitive remediation tasks to improve overall stability of software applications and systems
Leads evaluation sessions with external vendors, startups, and internal teams to rigorously assess architectural designs, technical credentials, and suitability for integration with existing information architecture
Champions communities of practice across Software Engineering to promote awareness and adoption of new and cutting-edge technologies
Fosters an inclusive team culture that values diversity, opportunity, and respectful collaboration

Requirements

Formal training or certification in Software Engineering and application development, plus 7+ years of hands-on experience
Strong hands-on expertise in Python and/or Java with a track record of delivering secure, production-grade software
Extensive experience building cloud-native solutions on AWS, including compute, storage, security/IAM, networking, and observability
Solid background with LLMs and agentic systems, covering orchestration patterns, tool and function calling, retrieval, structured outputs, evaluations, and safety guardrails
Experience developing data-intensive applications and workflows using Databricks (Spark, Delta, jobs/workflows)
Proficiency in JavaScript/TypeScript and React for modern web application development
Strong automation and continuous delivery expertise with disciplined SDLC practices (CI/CD, testing, code quality, release controls)
Advanced understanding of agile delivery, application resiliency, and secure engineering practices
Robust system design skills with the ability to make pragmatic trade-offs and secure stakeholder alignment

Technologies

Python
Java
AWS (EC2, EKS, RDS, S3, Lambda, EMR, Glue)
Databricks, Spark, Delta
Snowflake
PySpark, Hadoop, Airflow
SQL
JavaScript, TypeScript, React
LLMs, agentic systems

Benefits

Comprehensive health care coverage
On-site health and wellness centers
Retirement savings plan
Backup childcare
Tuition reimbursement
Mental health support
Financial coaching
Commission-based pay and discretionary incentive compensation, paid in cash and/or forfeitable equity

View full job page →

Hadoop Developer Jobs

Senior Python Developer with Spark

Hadoop and Spark Developer

Power BI Developer - Freelance

Lead Software Engineer - Agentic

Get Job Alerts