DeveloperJobs.io
← Back to all jobs

Job Description

Cobalt Benefits Group LLC is seeking an AI Developer to design, build, test, and deploy governed AI solutions onsite in Boston.

Responsibilities

  • Design, prototype, build, test, and deploy production AI applications and agents; choose between Copilot Studio, workflow, agent, custom-service, vendor, or non-AI options using Build-vs-Buy guidance.
  • Build retrieval-augmented generation (RAG) systems, including document ingestion, chunking, indexing, metadata, retrieval tuning, and grounded responses with source citations.
  • Integrate with internal and third-party systems via authenticated REST and GraphQL APIs; collaborate on data pipelines for documents and structured data.
  • Apply least-privilege permissions, single sign-on, secrets management, and PHI/PII protections; implement output validation plus logging and retention controls with IT Security.
  • Create evaluation harnesses using golden test sets and subject-matter expert rubrics; run production sampling and measure accuracy, citation support, completeness, latency, and transaction cost (including results by plan type).
  • Implement telemetry for prompts, retrieval, tool calls, and outputs; maintain runbooks, hand-off documentation, cost controls, and operational monitoring.
  • Advance work through the CBG AI Solution Delivery Playbook decision gates and required artifacts (risk assessments, requirements, evaluation plans, and transition documentation).
  • Collaborate with the AI Risk Manager, Compliance, Data Protection Officer, Security, and the AI Ethics Committee as appropriate; share reusable components, templates, and approved prompt patterns.

Requirements

  • Bachelor’s degree in Computer Science, Information Systems, related field, or equivalent practical experience; 3+ years of professional software development experience.
  • Hands-on generative AI development and evaluation experience, including RAG, context engineering, tool/function calling, and at least one production or production-pilot solution.
  • Strong Python and/or C#/.NET development skills; experience building production APIs, integrating with REST, using OAuth (or similar authorization), and securely managing secrets and configuration.
  • Ability to review, test, debug, and maintain AI-generated code; working knowledge of SQL, data modeling, and data pipelines.
  • Experience handling HIPAA-regulated information, PHI masking, and access controls.
  • Experience with AI regression testing, monitoring, model version management, release controls, cost optimization, and guarded multi-step agents.
  • Ability to document technical decisions, follow governance and change control, run business requirements sessions, and recommend delaying or stopping solutions that fail evaluation thresholds.

Technologies

  • Copilot Studio, Power Automate, Power Platform, Azure OpenAI, Azure AI Search, Microsoft 365 Copilot, Power Apps
  • Azure AI Foundry, Azure API Management, Entra ID, Key Vault
  • Azure SQL, Cosmos DB, Azure Data Lake Storage, Blob Storage, Dataverse
  • GitHub, CI/CD pipelines
  • REST, GraphQL, OAuth, Python, C#, .NET, SQL

Benefits

  • Fantastic medical, dental, and vision insurance
  • Twice annual employer HSA contributions covering 50% of the HDHP plan’s annual deductible
  • Company provided Basic Life and AD&D
  • Company paid Short-Term and Long-Term Disability
  • Flexible Spending Accounts
  • 401(k) Retirement Plan with up to a 6% employer-match (100% fully vested after 3 years)
  • 10+ paid holidays; fully paid half day Summer Fridays
  • Generous paid vacation and sick time
  • Annual Paid Volunteer Day
  • Annual Tuition Reimbursement; Annual Health and Wellness Reimbursement
  • Lots of fun company events

Preferred

  • Experience with Microsoft Copilot Studio, Power Automate, Power Platform, Azure OpenAI, and Azure AI Search.
  • Demonstrated improvement of retrieval or extraction systems using before-and-after evaluation results.
  • Experience with AI coding agents and Model Context Protocol (MCP).
  • Familiarity with healthcare and employee benefits domains including claims, eligibility, plan documents, or EDI.

Personal Attributes

  • Collaborative communicator who translates business processes into technical requirements and partners across business, engineering, security, and compliance.
  • Analytical, detail-oriented problem solver focused on measurable quality, documentation, and continuous improvement.
  • Sound judgment and accountability in privacy, fairness, human oversight, and responsible AI delivery.
  • Adaptable, hands-on builder comfortable selecting between low-code and custom engineering approaches.

Measures of Success

  • First 90 days: complete AI governance onboarding, ship an enhancement through Production Readiness Review, establish a golden-test-set evaluation harness.
  • First 6 months: release a new approved AI solution, publish reusable components adopted by another project, enable monitoring dashboards.
  • First 12 months: meet approved business KPIs, demonstrate reusable delivery patterns, and maintain current inventory, risk classification, evaluation, and monitoring records.

Compliance and Responsible AI

  • Solutions must meet CBG governance and technical security requirements, with privacy, security, and human oversight proportionate to risk.
  • Automated claim denials without human review and use of PHI in public generative AI models are prohibited.
  • Work aligns with HIPAA, HITRUST CSF, NIST AI Risk Management Framework, and applicable OWASP guidance.

Location and Salary

  • Location: Boston, MA (onsite)
  • Salary: USD 120,000 - 155,000 per year
  • Minimum experience: 3 years

Similar Jobs