Consultant, Data Engineer

Build reliable data pipelines for client teams using Databricks and modern lakehouse patterns. At Infinitive Inc, this hybrid consulting role lets you deliver production-ready ETL, partnering with senior engineers, architects, and business analysts while working on projects for teams in McLean, Virginia.

You will design, build, and maintain batch and incremental data workflows on Databricks, moving data into Delta Lake tables using Python, PySpark, and Spark SQL. You will also document your work so client teams can run pipelines, understand source-to-target mappings, and follow runbooks.

What you’ll do

• Develop and maintain batch and incremental ETL pipelines in Databricks using Python, PySpark, and Spark SQL.
• Ingest data from relational databases, APIs, flat files, and cloud storage into Delta Lake (including Amazon S3 and Azure Data Lake Storage).
• Implement bronze, silver, and gold layers using medallion architecture.
• Write and tune SQL for transformations, data validation, and reporting tables.
• Add data quality checks, manage schema changes, and troubleshoot failed jobs.
• Schedule and monitor workloads with Databricks Workflows using Unity Catalog permissions.
• Tune Spark jobs for performance and cost, including partitioning, file sizing, and cluster configuration.
• Use Git for version control, participate in code reviews, and contribute to CI/CD pipelines.
• Document pipelines, source-to-target mappings, and runbooks so client teams can operate what you build.
• Join client meetings to clarify requirements, report status, and walk through your work.

What you’ll bring

• 1 to 3 years of professional experience in data engineering, software engineering, or a related technical role.
• Bachelor’s degree in computer science, information systems, engineering, mathematics, or a related field (or equivalent experience).
• Hands-on experience with Databricks from a professional setting or a substantial academic project.
• Working proficiency in Python for data processing.
• Experience building transformations with Apache Spark and PySpark (DataFrames and Spark SQL).
• Strong SQL, including joins, window functions, CTEs, and aggregations.
• Understanding of ETL concepts such as incremental loads, slowly changing dimensions, dimensional modeling, and data quality checks.
• Familiarity with at least one cloud platform: AWS, Azure, or Google Cloud.
• Clear written and verbal communication, including comfort explaining technical work to non-technical stakeholders.
• Authorization to work in the United States (confirm sponsorship policy).

Nice to have

• Databricks Certified Data Engineer Associate certification.
• Experience with Delta Lake features such as MERGE and time travel, or Lakeflow Declarative Pipelines (formerly Delta Live Tables).
• Exposure to Unity Catalog, dbt, Apache Airflow, or Azure Data Factory.
• Experience with Terraform or CI/CD tools such as GitHub Actions or Azure DevOps.
• Prior consulting or client-facing experience.
• Background in financial services, healthcare, or media data.

Location: McLean, VA (hybrid). Compensation: USD 75,000 - 120,000 per year.

Technologies you’ll use: Databricks, Python, PySpark, Spark SQL, Delta Lake, medallion architecture, Amazon S3, Azure Data Lake Storage, Databricks Workflows, Unity Catalog, Apache Spark, SQL, Git, CI/CD, DataFrames, ETL, AWS, Azure, Google Cloud.