Pyspark

Pyspark Developer Jobs

41 jobs found

1d ago

Senior Data Engineer

Sgn Global Technologies Remote 🌍 Remote

Remote

1d ago

$50 - $55/hr Full time Senior Azure Machine Learning Azure Synapse Analytics

Apply Now

Job Description

At Sgn Global Technologies, a remote W2 Senior Data Engineer is sought to design and sustain data infrastructure that powers analytics across cloud-based environments. The role centers on building and maintaining SQL databases and ELT/ETL pipelines, with hands-on work in Python, PySpark or Polars, Azure Synapse Analytics, and Azure Machine Learning. This position emphasizes delivering modular, maintainable code within a modern data stack.

Location: Remote

Compensation: USD 50 - 55 per hourly

Responsibilities

Maintaining SQL databases and performing advanced SQL and T-SQL operations.
Designing, implementing, and maintaining ELT/ETL pipelines in cloud-based data analytics environments.
Working with Azure Synapse Analytics and Azure Machine Learning using the modern data stack. DP-203 (or equivalent) certification is preferred.
Data manipulation using Python (Pandas required).
Experience with PySpark and/or Polars is preferred.
Developing reusable, modular, and maintainable code is preferred.

Requirements

Bachelor's degree in Data Engineering, Computer Science, Data Science, Machine Learning, Mathematics, or a related field
OR 5+ years of relevant applied work experience
5+ years of hands-on experience in maintaining SQL databases and performing advanced SQL and T-SQL operations.
5+ years of hands-on experience in designing, implementing, and maintaining ELT/ETL pipelines in cloud-based data analytics environments.
3+ years of hands-on experience in working with Azure Synapse Analytics and Azure Machine Learning using the modern data stack. DP-203 (or equivalent) certification is preferred.
Data manipulation using Python (Pandas required).
Experience with PySpark and/or Polars is preferred.

Technologies

SQL
T-SQL
Azure Synapse Analytics
Azure Machine Learning
Python
Pandas
PySpark
Polars

View full job page →

3d ago

Machine Learning Engineer

WireScreen New York, NY

New York, NY

3d ago

$150k - $195k/yr Full time Big Data Bigdata Data Architecture

Apply Now

Job Description

WireScreen is seeking a Machine Learning Engineer to drive entity resolution and the evolution of a knowledge graph. Based in New York, NY with a hybrid work setup, you will scale data ingestion, deploy ML models across millions of records, and work closely with cross-functional teams under the direction of the VP of Engineering.

Responsibilities

Refine our existing entity resolution algorithms to reveal hidden connections between people and organizations across China
Expand the knowledge graph by incorporating alternative data to map the power structure of China
Train, validate, and deploy ML models that operate on tens of millions of records daily
Partner with Product to design and implement evaluation harnesses for classical ML and agentic systems
Integrate agent workflows into internal tools to enhance the scale and speed of the Research team

Requirements

4+ years of experience tackling clustering-type ML problems, ideally in the domain of knowledge graphs or entity resolution; other domains may include recommendation engines, cohort analysis, or outlier/anomaly detection
End-to-end machine learning model experience in production, including experimentation, training, testing, tuning, deployment, and ongoing operation; model families may include clustering, classification/regression, dimensionality reduction and embeddings, nearest-neighbor/similarity methods (e.g. KNN, SVM), ensembles, NLP, and deep learning
Significant experience with Python programming and SQL

Technologies

Python
SQL
PySpark
Temporal
FastAPI
Scikit-learn
NumPy
Docker
Terraform
Kubernetes

Benefits

Competitive compensation including salary, equity, and rapid growth potential
100% company-paid Medical, Dental, and Vision coverage for employees
FSA, HSA, and 401(k) options to help you plan for healthcare expenses and retirement
Generous paid time off plus company-wide holidays to help you rest and recharge
Pre-tax commuter benefits to help you save on transit and parking
Hybrid office schedule designed to give you flexibility while staying connected with your team

Nice to Have

Experience working with frontier or state-of-the-art models and/or fine-tuning your own LLMs for specific tasks
Experience solving problems across large, heterogeneous unstructured datasets and/or with semantic search, computer vision (OCR), or linear optimization problems
Experience with any of the following technologies: PySpark, Temporal, FastAPI, Scikit-learn, NumPy, Docker, Terraform, Kubernetes
Early-stage startup experience (Series B or earlier)
B2B SaaS experience
#LI-LG1

View full job page →

4d ago

Data Engineer I

Murphy USA El Dorado, AR

El Dorado, AR

4d ago

Full time Big Data Bigdata Cloud

Apply Now

Job Description

Murphy USA is seeking a Data Engineer I for an onsite role in El Dorado, AR. You will contribute to developing and supporting data products on a cloud-based data lake, build scalable data pipelines, and empower enterprise analytics by partnering with business units to drive analytics execution. This position emphasizes hands-on development, cross-functional collaboration, and ongoing improvements to data quality and governance across the organization.

Responsibilities

Develop and support data products on a modern cloud-based data lake, applying cross-domain technology knowledge to deliver a robust, scalable data engineering platform.
Provide data services for enterprise analytics, leveraging automated pipelines at scale and streamlining data transformations for priority use cases.
Contribute directly to code development and collaborate with business units and peer technology teams to enable analytics execution.
Partner across the organization to architect, implement, deploy, and maintain data driven system solutions.
Build and simplify enterprise data pipelines with a focus on security, reusability, and data quality.
Debug issues, perform root cause analysis, and participate in proactive maintenance activities to prevent problems.
Contribute to reducing technical debt and adhere to clean code practices.
Bring development experience in one or more languages such as Python, SQL, or PySpark.
Experience with Infrastructure as Code tools, including Databricks Asset Bundled and Terraform.
Engage with the data governance council to define, develop, and review standards, guidelines, and data models to promote data quality and best practices.
Utilize DevOps tools to automate deployment and validation, including test automation of data pipelines.
Write and execute comprehensive testing plans, protocols, and documentation for assigned components of the data system.
Identify defects and implement solutions for issues with code and data system integration.
Effectively plan and collaborate with technical and non-technical teams on business initiatives.

Requirements

Bachelor's degree required.
Minimum 1 year of relevant experience.
Technical knowledge of BI systems design.
Understanding of big data architecture and the technology landscape.
Experience with ETL orchestration.
Strong communication skills.
Comfort working in a fast-paced, multi-task environment.

Technologies

Python
SQL
PySpark
Databricks Asset Bundled
Terraform

View full job page →

4d ago

Cybersecurity Engineering Development Program (CSEDP) Software Engineer

National Security Agency Fort Meade, MD

Fort Meade, MD

4d ago

$87k - $123k/yr Full time Cloud Platforms Data Security Encryption

Apply Now

Job Description

The Cybersecurity Engineering Development Program (CSEDP) at the National Security Agency offers an entry level software engineering track focused on cryptographic software, providing hands-on experience across the cryptographic development lifecycle.

Responsibilities

The Cybersecurity Engineering Development Program is a U.S. Government initiative to develop participants in the evolving field of cybersecurity, while building leadership and communication skills to meet current and emerging cryptographic mission demands.
Designed for entry level developers, the program delivers in-depth training and practical experience across the full cryptographic lifecycle, including design and development, key and code production, vulnerability discovery, and applying cryptography in an operational environment.
Participants typically complete 3 to 4 tours over a 3-year development period across Crypto Engineering categories such as End Cryptographic Unit Design/Development, Crypto Key Design/Development, Crypto Assessment and Certification, and Crypto Application.

Requirements

Bachelor's degree in Computer Science or Software Engineering.
Related fields may be considered if the program includes a concentration of CS courses in foundational areas such as algorithms, computer architecture (excluding network architecture), programming methodologies and languages, data structures, logic and computation, and advanced mathematics (e.g., calculus, discrete mathematics).
Relevant experience must pertain to the software development process (requirements analysis, software design, implementation, testing, integration, deployment/installation, and maintenance) and programming activity.
Entry level with a Bachelor's degree and no prior experience.
An Associate's degree with 2 years of relevant experience may be considered for individuals with substantial related experience.

Technologies

C++, C, Java, Python, PIG Analytics, PySpark
Angular, React
CI/CD, Docker, Kubernetes, AWS, cloud computing
Linux, VMware, ElasticSearch
Rust, microservices, Ansible
Databases (relational and non-relational), SIEM, IDS, vulnerability assessment tools
TLS/SSL, HSMs, cryptographic libraries, APIs

Competencies

Strong background in advanced mathematics, including number theory, abstract algebra, linear algebra, probability theory, and discrete mathematics.
Solid understanding of data structures, algorithms, operating systems, and computer architecture.
Proficiency in C/C++, Python, and Java.
Knowledge of symmetric and asymmetric cryptography, including hash functions, encryption, digital signatures, and key exchange.
Familiarity with TLS/SSL and network security principles for secure communications.
Understanding of hardware security modules (HSMs) and other secure computing (cloud) platforms.
Experience with Security Information and Event Management (SIEM), intrusion detection systems (IDS), and vulnerability assessment tools.
Familiarity with cryptographic libraries, APIs, and adherence to security standards (NIST, ISO 27001, OWASP).

Benefits

Comprehensive benefits package.

Pay, Benefits, and Work Schedule

Salary Range: $87,362 - $123,388 (Entry/Developmental).
Location: Fort Meade, MD onsite.
Work Schedule: Full-time, Monday through Friday, with an 8-hour day between 6:00 a.m. and 6:00 p.m. (flexible).
Training will be provided based on the selectee's needs and experience.
DCIPS trial period: two years for most selections; one year for veterans' preference eligible employees, running concurrently with the position commitment.
After the trial period, NSA determines whether continued employment serves the public interest, based on performance, agency needs, and organizational goals. If certification is not granted, a termination notice will be provided prior to the end of the trial period.

How to Apply

Apply as soon as possible, noting that postings may close earlier than the stated end date due to changing requirements.
Review the minimum qualifications; only applicants meeting the required qualifications will be contacted to proceed with the employment process.
Complete the resume tool to highlight relevant work experience and education related to the position, and answer applicable screening questions.
Information collected will be used to determine eligibility; inaccuracies may lead to disqualification.
A confirmation email will be sent after submitting the first application and after any future updates to submitted applications.
Ensure spam filters accept emails from noreply@intelligencecareers.gov; for vacancies with testing requirements, also include @uwe.nsa.gov, @nsa.gov, and @pearson.com.
U.S. Citizenship is required for all applicants.
The four narrative questions provide an opportunity to demonstrate dedication to public service; provide responses of 200 words or less for each question.
You will be asked to certify that your responses are in your own words and not generated by a consultant or AI tool.

View full job page →

5d ago

Lead Data Engineer

Deloitte Stamford, CT

Stamford, CT

5d ago

$135k - $265k/yr Full time Manager Big Data Bigdata

Apply Now

Job Description

Based in Stamford, CT on site, this Lead Data Engineer role centers on Databricks, directing the design, construction, and delivery of modern data and analytics solutions for clients, while leading delivery teams and advising stakeholders. Compensation ranges from USD 134,500 to 265,100 per year.

Responsibilities

Direct the design and deployment of Databricks based data and analytics solutions for client engagements.
Oversee project teams and workstreams across requirements gathering, solution design, development, testing, and deployment activities.
Translate business and technical requirements into scalable data pipelines, data models, and platform capabilities.
Advise client stakeholders on architecture, implementation approaches, and operating model considerations for modern data platforms.
Monitor solution quality, resolve issues, and track delivery progress while supporting project economics and timelines.

Requirements

Bachelor's degree
6+ years of experience delivering data engineering, analytics, or cloud data platform solutions
3+ years of experience with Databricks and Apache Spark
3+ years of experience with Python and SQL
2+ years of experience leading project teams or workstreams
Ability to travel approximately 50%, depending on client work
Limited immigration sponsorship may be available

Technologies

Databricks
Apache Spark
Python
SQL
Delta Lake
Unity Catalog
MLflow
AWS
Microsoft Azure
Google Cloud Platform (GCP)

The Team

Our team supports clients as they modernize their data environments and build scalable platforms for analytics, reporting, and decision making. Professionals in this area work across strategy, architecture, implementation, and deployment to help organizations improve how they manage and use data. The team frequently collaborates with business and technology stakeholders to deliver cloud enabled, enterprise scale solutions.

Information for Applicants with a Need for Accommodation

https://www2.deloitte.com/us/en/pages/careers/articles/join-deloitte-assistance-for-disabled-applic...

View full job page →

7d ago

Senior Data Engineer, Underwriting Technical Lead

Travelers Hartford, CT

Hartford, CT

7d ago

$139k - $230k/yr Full time Senior Amazon Web Services AWS

Apply Now

Job Description

Responsibilities

Develop and deploy sophisticated data pipelines and analytics solutions, troubleshoot issues, apply transformations, and advocate for data cleansing and quality improvements.
Architect comprehensive data architectures, integrate new data sources, and ensure design consistency across initiatives aligned with the data strategy.
Assess diverse data sources to determine value and applicability, recommending data for inclusion in analytics workflows.
Embed core data management practices including governance, security, and data quality controls.
Serve as a data and technology subject matter expert within business lines, supporting delivery and training end users on data products and the analytics environment.
Conduct data and system analysis, diagnose defects and high complexity incidents, and implement appropriate resolutions.
Collaborate with cross functional teams to enable delivery and educate end users on complex data products and analytics environment.
Perform other duties as assigned.

Requirements

Bachelor’s degree in a STEM field or equivalent.
Ten years of related experience.
Proven expertise designing, building, and sustaining scalable data pipelines on AWS, with hands on work across S3, Glue, Redshift, Lambda, and EMR.
Strong Databricks experience, including developing and optimizing data workflows and leveraging Lakehouse architecture for large scale processing.
Solid understanding of AI and ML concepts, with ability to collaborate with data science teams and support deployment or operation of models within the data ecosystem.
Hands on experience with dbt for data modeling, transformation, and documentation, applying robust software engineering practices to analytics workflows.
Advanced PySpark skills for distributed processing, capable of producing production grade code for large scale batch and streaming workloads.
Candidates from Azure environments (Azure Data Factory, Synapse Analytics, Azure Databricks) will be considered if they demonstrate the ability to operate effectively within an AWS based stack.
Willingness to assume a technical leadership role, mentoring engineers, guiding architectural decisions, and partnering with stakeholders across engineering and the business.
Bachelor’s degree in computer science or related STEM field, or equivalent education/experience.
6 additional years of data engineering experience.

Technologies

AWS ecosystem and services: S3, Glue, Redshift, Lambda, EMR
Databricks and Lakehouse tooling
dbt for modeling and documentation
PySpark for distributed processing
Azure Data Factory, Synapse Analytics, Azure Databricks

Benefits

Health coverage
Retirement savings plan
Paid time off
Wellness program
Encouragement of volunteering

Employment Practices

Travelers is an equal opportunity employer. We value the unique abilities and talents each individual brings to our organization and recognize that we benefit in numerous ways from our differences.

View full job page →

8d ago

Python Data Engineer

Glint Tech Solutions LLC Plano, TX

Plano, TX

8d ago

Full time Python Amazon Web Services Apache Airflow

Apply Now

Job Description

Glint Tech Solutions LLC is hiring a Python Data Engineer to support a telecommunications technology client from its Plano, Texas on-site office. The role centers on building scalable data pipelines and data models, with a strong emphasis on Palantir Foundry and Gotham to enable enterprise-scale data engineering initiatives. Collaboration with cross-functional teams helps ensure data quality, governance, and performance across large data workloads.

Responsibilities

Design and optimize scalable ETL and ELT data pipelines
Create data models and orchestrate data workflows
Maintain data quality, validation processes, and performance tuning
Ingest and integrate data from multiple enterprise sources

Requirements

Proven Python development experience focusing on ETL, automation, and data processing
Hands-on experience with Palantir Foundry and Gotham
Proficiency with PySpark, Pandas, and NumPy
SQL skills with PostgreSQL or MySQL and experience with MongoDB
Experience with Airflow or similar workflow orchestration tools
Cloud platform experience on AWS, Azure, or GCP
Understanding of Spark, Hadoop, S3, HDFS, and data warehousing concepts