DeveloperJobs.io
← Back to all jobs

Job Description

Shamrock Trading Corporation is seeking a Data Engineer to source, transform, and analyze data across internal and external systems. The role designs a cloud-focused analytics and BI data platform, builds reliable pipelines for analysts, and implements streaming and data governance practices on modern AWS-based platforms.

Responsibilities

  • Build, optimize, and maintain ETL and ELT pipelines using Databricks, Apache Spark, and Delta Lake.
  • Apply performance tuning approaches to improve Spark-based workloads.
  • Use AWS services such as S3, ECS, and Lambda, along with Glue, to implement robust data architectures.
  • Develop streaming solutions using Kafka and Databricks Structured Streaming.
  • Implement data validation, observability, and governance best practices using Unity Catalog or comparable tooling.
  • Collaborate with analysts, data scientists, and application engineers to support business data needs.
  • Deliver infrastructure-as-code and CI/CD practices for data pipelines, using tools such as Terraform and GitHub Actions.

Requirements

  • Bachelor’s degree in computer science, data science, or a related technical field (or equivalent practical experience).
  • 2-4 years of experience in data engineering, with a focus on cloud-based platforms.
  • Strong hands-on experience with Databricks, including Spark, Delta Lake, and MLflow.
  • Experience building and maintaining AWS-based data pipelines, currently using AWS Lambda, Docker / ECS, MSK, Airflow, Databricks, and Unity Catalog.
  • Working knowledge of Git workflows using GitHub, GitLab, or similar systems.
  • Proficiency in Python.
  • Experience with Databricks components such as PySpark, SQL, Spark Declarative Pipelines/DLT, and Databricks Asset Bundles.
  • Experience with Kafka and the Kafka Connect ecosystem, including Schema Registry and Avro.
  • Familiarity with CI/CD for data pipelines and infrastructure as code using GitHub Actions and Terraform.
  • Strong SQL skills for transformation and performance tuning.
  • Understanding of data security and governance best practices.
  • Interest in working directly with customer teams, including business units and internal IT.

Technologies

Databricks, Apache Spark, Delta Lake, AWS, S3, ECS, Lambda, Glue, Kafka, Databricks Structured Streaming, Unity Catalog, Terraform, GitHub Actions, MLflow, Docker, MSK, Airflow, GitHub, GitLab, Python, PySpark, SQL, Spark Declarative Pipelines, DLT, Databricks Asset Bundles, Kafka Connect, Schema Registry, Avro, FastAPI, AWS API Gateway, OpenAPI Specifications, Postgres, Microsoft SQL Server, MongoDB, Databricks Dashboards, Grafana, PowerBI, Excel, Kafka Connect ecosystem, API architectures, Infrastructure-as-code (IaC).

Benefits

  • Fully paid healthcare, dental and vision premiums for employees and eligible dependents, plus gym benefits.
  • Generous company 401(k) contributions and employee stock ownership after one year.
  • Competitive PTO and fully paid time off opportunities to volunteer within your community.
  • Ongoing training and professional development opportunities.

Onsite Attendance

  • Regular onsite attendance is an essential function of this position.
  • The role requires direct collaboration with cross-functional teams, real-time problem-solving, and timely responses to customer and operational needs.
  • Onsite presence supports efficient communication with colleagues and subject matter experts, improving issue resolution and service delivery.
  • The role also involves team meetings, operational processes, and occasional onsite trainings to meet customer service objectives.

What Will Set You Apart

  • Proven experience with relational and NoSQL databases such as Postgres, Microsoft SQL Server, and MongoDB.
  • Familiarity with data lakehouse architectures and optimization strategies.
  • Familiarity with API architectures including FastAPI, AWS API Gateways, and OpenAPI Specifications.
  • Experience with data visualization using tools such as Databricks Dashboards, Grafana, PowerBI, and Excel.

Similar Jobs