DeveloperJobs.io
← Back to all jobs

Job Description

Better Debt Solutions is building and operating a Microsoft Fabric-based enterprise data platform, and they need an experienced Data Engineer to help design, deliver, and optimize data products that power operational reporting and AI-driven insights. This onsite role in Irvine focuses on lakehouse and warehouse architecture, reliable ingestion and transformations, performance tuning, and data reliability through observability and monitoring.

In this position, you will work across the Fabric and Azure ecosystem to structure data in OneLake using a medallion approach, modernize ingestion with incremental loading and CDC, and ensure pipelines run efficiently and predictably at peak demand.

Responsibilities

  • Design and implement scalable data warehouse and Lakehouse architectures using Microsoft Fabric and Azure.
  • Architect and organize data structures within OneLake, Fabric Warehouse, and Lakehouse using a medallion (bronze, silver, gold) approach.
  • Develop and maintain ingestion and transformation pipelines using Data Factory in Fabric (pipelines and Copy jobs) plus Spark/Python notebooks.
  • Ingest data from REST APIs and relational databases including MySQL/MariaDB, PostgreSQL/Redshift, and SQL Server, including sources behind on-premises data gateways.
  • Create dimensional models and star schemas with SCD Type 2 history and conformed dimensions for cross-functional reporting.
  • Centralize business logic in curated gold-layer tables to keep semantic models consistent and to feed near-real-time operational dashboards.
  • Profile and tune Spark and SQL workloads to reduce transformation run times using partitioning, file sizing, join strategies, and caching.
  • Redesign legacy full-reload processes into incremental and change-data-capture (CDC) pipelines to minimize compute overhead.
  • Monitor and manage Fabric capacity (CU) consumption using the Fabric Capacity Metrics app to prevent compute throttling during peak workloads.
  • Maintain Delta tables with OPTIMIZE, VACUUM, and V-Order to support fast reads in Direct Lake and warehouse queries.
  • Audit pipelines, notebooks, and dataflows to eliminate redundant processing and consolidate duplicated logic.
  • Implement automated freshness, volume, and completeness monitoring with proactive alerting for all production pipelines.
  • Reconcile loaded datasets against source systems to identify and resolve ingestion gaps and anomalies.
  • Perform safe historical corrections and backfills using dry runs and validation before production deployment.
  • Collaborate with vendor partners, internal business users, and BI and analytics teams to deliver cost-efficient, scalable data solutions.

Requirements

  • Bachelor’s degree in Computer Science, Data Science, Information Systems, or related field (or equivalent professional experience).
  • 7+ years hands-on experience in Data Engineering or Data Platform Development, including direct production experience with Microsoft Fabric and Azure.
  • Strong proficiency in Python, PySpark, and advanced SQL.
  • Ability to identify, decompose, and solve complex data challenges independently and as part of a team.
  • Proven track record reducing run times and compute costs through workload optimization, incremental loading, and dimensional modeling (star schema, SCD Type 2).
  • Practical experience with Git-based development and deployment promotion workflows using Azure DevOps or GitHub.
  • Excellent communication skills, including the ability to bridge conversations across executive leadership (CEO, COO), vendor partners, and non-technical stakeholders.

Technologies

  • Microsoft Fabric, Azure, OneLake, Fabric Warehouse
  • Data Factory in Fabric, Spark, Python, PySpark
  • REST APIs
  • MySQL, MariaDB, PostgreSQL, Redshift, SQL Server
  • On-premises data gateways
  • SCD Type 2, star schema
  • Delta tables, OPTIMIZE, VACUUM, V-Order, Direct Lake
  • Fabric Capacity Metrics app, Fabric capacity (CU)
  • Azure DevOps, GitHub, Git-based development
  • SQL

Benefits

  • Comprehensive medical, dental, and vision insurance coverage options.
  • Paid vacation and paid company holidays throughout the year.
  • Lunch is on us every Friday!
  • Vibrant team culture with regular quarterly social events, team outings, and annual holiday parties.
  • 401(k) retirement savings plan options.
  • Direct visibility and reporting to executive leadership (CEO & COO), with autonomy to architect and scale the enterprise data footprint.

Bonus Points

  • Microsoft Fabric Data Engineer Associate (DP-700) certification.
  • Familiarity with Azure AI Foundry or Azure Machine Learning.
  • Experience using AI-assisted coding tools to accelerate pipeline development.
  • Familiarity with secondary cloud ecosystems such as Google Cloud Platform (GCP).
  • Prior database administration (DBA) experience managing data security, redundancy, and high availability.

Location and Compensation

  • Irvine, CA (Fully In-Office)
  • $140,000 - $155,000 per year
  • Full-Time (Exempt/Salary)

In this role, priorities can shift based on system optimization needs, business analytics demands, and strategic platform initiatives.

Similar Jobs