Senior Software Engineer II Applied AI and Evaluations
Source: remoteok
About the Role
For over 20 years, Smartsheet has helped people and teams achieve anything. From seamless work management to smart, scalable solutions, they've always worked with flow. They're building tools that empower teams to automate the manual, uncover insights, and scale smarter. They are creating space to think big, take action, and unlock the kind of work that truly matters. Because when challenge meets purpose, and passion turns into progress, that's magic at work, and it's what they show up for everyday.
Smartsheet is building the next generation of AI-powered work management through SmartAssist, their intelligent agent platform. As they scale from early demos to production-grade agents, quality is the critical frontier and they are looking for a Agent Quality Engineer to own it.
This is not a QA role. It's a deeply technical, high-autonomy position at the intersection of LLM evaluation, prompt and context engineering, and retrieval-augmented generation. You will diagnose why their agents fail, design the systems that catch regressions, and drive measurable improvements across their orchestrator and subagent fleet.
You will work closely with their Agent Engineering and AI Platform teams, embedded in a team that has already shipped evaluation infrastructure on Databricks/MLflow and is building toward a mature Agent Development Lifecycle (ADLC).
Responsibilities
You Will:
- Own agent quality end-to-end: diagnosis, improvement, and validation across SmartAssist's orchestrator and subagents
- Identify failure modes across quality dimensions factual accuracy, completeness, tone, actionability, and latency and prioritize what to fix
- Drive quality improvements through prompt engineering