Senior Data Engineer

IvoryCloud seeks a Senior Data Engineer to design and optimize data pipelines and analytics platforms using Databricks, supporting large-scale data engineering and analytics workloads for a Department of Energy client. This hybrid role is based in Rockville, Maryland, with a practical mix of in-office and remote work.

Responsibilities

• Create and sustain scalable data pipelines leveraging Azure Databricks and Apache Spark.
• Develop and refine batch and streaming data processing solutions.
• Build robust ETL and ELT frameworks for structured and semi-structured data from SQL, APIs, SaaS apps, and Data Live Lake Storage.
• Implement transformation logic using PySpark, SQL, and Delta Lake within a Lakehouse Medallion architecture.
• Integrate and model data from SQL, SaaS systems, and cloud storage into unified analytics-ready and AI-ready structures.
• Collaborate with analytics, BI, and AI/ML teams to deliver trusted datasets, semantic layers, and AI-ready data (features, vector-ready patterns, embeddings).
• Enhance data quality, observability, lineage, governance, and security in alignment with Azure and enterprise standards.
• Optimize data performance, storage costs, and processing efficiency in cloud environments.
• Contribute to architecture discussions and advance the evolution of the organization’s data platform and engineering best practices.

Requirements

• U.S. citizenship is required and non-negotiable.
• Work in the Rockville, MD office two days per week is required and non-negotiable.
• Must pass multiple background and employment checks, including federal and local checks.
• Bachelor’s degree in Technology, Business, or a related field.
• Minimum of five years of relevant experience.
• Databricks experience of at least four years is required.

Technical Environment

• Databricks, Apache Spark, PySpark
• SQL, Delta Lake, Data Live Lake Storage
• Python, dbt, Airflow
• Azure Data Factory, Kafka, Snowflake
• Git, CI/CD Pipelines
• Cloud platforms: Azure, AWS, GCP
• Unity Catalog, Delta Live Tables, Workflows, Lakeflow
• SQL Warehouse, Mosaic AI, Synapse pipelines, Dagster
• Azure OpenAI, Event Hubs, Kinesis, Redshift, Synapse

Benefits

• 401(k)
• Health insurance
• Paid time off
• Vision insurance

What Success Looks Like

• Deliver reliable, production-ready data pipelines that support critical business use cases.
• Improve data processing performance and system scalability.
• Establish reusable engineering patterns and best practices across teams.
• Provide analytics and AI teams with high-quality, trusted datasets.
• Contribute to the modernization and growth of IvoryCloud's data platform capabilities.

Technical Environment

• Databricks
• Apache Spark
• PySpark
• SQL
• Delta Lake
• Python
• dbt
• Airflow
• Azure Data Factory
• Kafka
• Snowflake
• Git
• CI/CD Pipelines
• Cloud Platforms: Azure, AWS, or GCP

Education & Certifications

• B.S. in a Business or Technology-related field

Legal

IvoryCloud is an Equal Opportunity Employer (EEO).

Pay

USD 155,000.00 – 200,000.00 per year

Location and Work Arrangement

Location: Rockville, MD. Work arrangement: hybrid. In-person presence is required at the Rockville office two days per week.

Experience

• Databricks: 4 years (Required)

Application Questions

• This role requires U.S. Citizenship. Do you meet this requirement?
• Can you meet the hybrid requirement of two days a week in office in Rockville, Maryland?
• Are you able to come into the office for in-person interviews?