Data Engineer

Realign LLC is hiring a Data Engineer for an onsite, full-time permanent role in Charlotte, NC.

Responsibilities

• Ensure the design and engineering approach for complex data solutions stays consistent across multiple data flows and systems.
• Perform research, design, prototyping, and exploration activities, including evaluating new toolsets.
• Design and develop client requirements using multiple architectural components.
• Maintain, improve, clean, and manipulate data for operational and analytics data systems.
• Continuously improve technical problem solving and solution design, including challenging existing approaches.
• Collaborate with development teams to understand data requirements and confirm the feasibility of the data architecture for accurate implementation.
• Assemble large, complex datasets aligned to functional and non-functional requirements.
• Build processes for data transformation, data structures, metadata, data quality controls, and dependency and workload management.
• Define and implement data pipelines that enable faster, better, data-informed business decision-making.
• Support existing test suites (integration, regression, performance), review test results, identify test issues, and triage root causes.
• Document and communicate deployment, maintenance, support, and business functionality information.
• Follow team delivery and release process, including code deployment and release cadence.
• Identify gaps in adherence to data management standards and partner with stakeholders to close gaps.
• Lead concept and experimentation testing and synthesize findings to validate and improve analytical solutions.
• Define key performance indicators and internal controls.
• Mentor more junior Data Engineers and coach team members through delivery/release CI-CD activities, including branching timelines, pull requests, issue triage, merge and conflict resolution, and release notes.

Requirements

• Minimum 8+ years hands-on data engineer and/or software development experience, including at least 5 years onsite.
• Experience with big data processing systems such as Hadoop and Cloudera.
• Programming experience with Python, including modular Python programming.
• Strong SQL and UNIX experience.
• Hands-on experience with Hadoop ecosystem tools, including Hive, Spark, and Pispark.
• Proven ability to provide technology leadership, guiding junior engineers to complete projects and deliver results.
• Cyber security domain experience, preferably vulnerability data management.
• Ability to switch between technology leadership and hands-on coding when needed.
• Experience with Cloud Services such as AWS and Azure.
• Experience with data stores including HIVE, HBase, and Elastic.
• Experience with code versioning systems such as Bitbucket.
• Experience building and working with CI/CD pipelines.
• Strong knowledge of data engineering principles and the SDLC process.
• Ability to identify, communicate, and clarify requirements gaps and challenges early with business partners.

Technologies

• Python
• Hadoop, Cloudera
• Hadoop Hive, Spark, Pispark
• SQL
• UNIX
• HIVE, HBase, Elastic
• AWS, Azure
• Bitbucket
• CI-CD, CICD

Job Details

• Location: Charlotte, NC (onsite)
• Salary: USD 119,000 per year
• Job type: Full Time
• Category: IT
• Role: Data Engineer
• Employment: Full-time permanent

Engagement & Interactions

• Primary level of engagement: Works as a team member under supervision from a more senior domain expert.
• Primary interactions: Business partners and stakeholders, Product Owner, Scrum Master, Development Team, Feature Lead, Architect, Domain Community of Practice, Data Scientists.

Role Context

• Objective of role: Knowledge in the cyber security domain.