AI agent transparency for high-stakes software pipelines
A speculative AI security agent that scans dependencies and blocks merges when it finds violations. The design question is not how to show results. It is how a person knows when to trust the agent, when to override it, and how to leave a record that holds up when they are not in the room anymore.
At Eleos Health I design AI tools for clinicians. The thing I keep learning is that the model is rarely the problem. The failure lives in the interface around it: how reasoning gets shown, what the person is expected to do with it, and what happens when they disagree.
Sentinel is a speculative project built to test whether that holds in a different domain. The setup: an AI agent that runs security checks across a codebase before every merge. It scans dependencies, runs policy checks, and blocks merges when it finds violations.
The design question is not "how do you show results." It is: how does a person know when to trust the agent, when to override it, and how to leave a record that holds up when they are not in the room anymore?
4:40pm Thursday. The team ships Friday.
Priya pushes her branch and Sentinel blocks it. A dependency she added has a known CVE. What Sentinel does not know: the dependency runs in an isolated environment where the attack path the CVE describes is impossible. She has seen this exact false positive before. The release is eighteen hours away.
This scenario puts every design problem under pressure simultaneously. The interface has to help Priya understand what the agent found and why, give her a path to provide context it was missing, and create a record that her security officer can reconstruct three weeks later without asking anyone.
The override flow, playable
Everything below runs in the browser, built in code, not a static mockup. Read the agent's reasoning, override the block, and watch the audit record form as you type.
Limited public exploit data exists for this CVE in non-public-facing configurations.
Overridden on payments-service · 2024-11-02
Reason: isolated network, no external request routing.
The button sits at the bottom on purpose. You read before you act.
override CVE-2024-28849 by priya@company.com
reason: (waiting for a substantive reason)
consistent with override on payments-service (2024-11-02)
surfaced automatically if a related incident is filed against this repo
Status: structure over duration
A progress bar tells you how much time has passed. That is the wrong information. Priya does not need to know she is 60% through a run. She needs to know what the system is doing and whether any of it has gone wrong yet.
The track shows four check categories, each resolving as it completes. Green for pass, red for block. If a segment turns red, the rest of the run keeps going. The track is a navigation surface, not a summary.
Confidence: qualitative, not numeric
Confidence is moderate because limited public exploit data exists for this CVE in non-public-facing configurations.
CVE-2024-28849 overridden on payments-service. Reason: isolated network, no external request routing.
No percentage. Three pips, one word, and the single sentence that changes the decision, with org precedent surfaced automatically.
"73% confident" implies precision that does not exist for most AI security judgments. It invites calibration theater: Priya trying to decide whether 73% is high enough rather than evaluating the actual reasoning.
Three pips. A word. One sentence explaining what drives it. "Confidence is moderate because limited public exploit data exists for this CVE in non-public-facing configurations." That sentence is what changes her decision.
The prior precedent note sits below it. If this CVE has been overridden elsewhere in the org, that surfaces automatically: "CVE-2024-28849 was overridden on payments-service. Reason: isolated network, no external request routing." Priya should not have to go looking for the fact that her team already solved this. The interface knows it.
Override: friction that earns its place
The override button is at the bottom of the reasoning panel. After the CVE description, after the confidence signal, after the precedent note, after the policy explanation.
If the button is at the top, the reasoning becomes a justification for a decision already made. At the bottom, it becomes the basis for a decision not yet made. Same button, different position, different cognitive outcome.
The modal asks one question: "Why does CVE-2024-28849 not apply here?" Not a dropdown. A specific text prompt. The specificity of the prompt shapes the specificity of the answer, and the answer is what makes the audit record useful.
The audit preview: accountability before the action
override CVE-2024-28849 by priya@company.com
reason: Dependency runs in an isolated container with no external network access, so the attack path in the CVE is not reachable.
consistent with override on payments-service (2024-11-02)
The record forms as she types; its left border turns green once the reason is substantive. A form to fill out becomes a statement someone will read.
Inside the override modal, before Priya submits, she can see exactly what the audit record will say. Her reason updates the preview in real time as she types. She sees what her security officer will read, three weeks later, reviewing this without any of the original context.
The record's left border is gray until the reason field is substantive. Then it turns green. That is the only animation in the prototype, and it is doing specific work: it tells Priya the system has registered her input as meaningful before she commits.
The shift from "I need to add a reason to unlock this button" to "I am creating a record someone will read" is not subtle. It is the entire point.
The principle: Accountability works best when it is visible before the action, not after. Accountability as consequence shapes behavior at the margins. Accountability as awareness shapes judgment at the moment of decision.
I built this same logic into LQA at Eleos. Compliance review that happens months after documentation is useless because nothing can be changed. Moving the check to the moment of writing created a feedback loop that changed behavior, not just outcomes. The audit preview is the same argument in a different context.
After the override
When Priya submits, the CVE row updates to show what was recorded. Her reason appears back to her, in a monospaced block, inside the check row.
Reason: Dependency runs in an isolated container with no external network access. Attack path described in the CVE is not reachable.
Consistent with the override on payments-service (2024-11-02).
Sentinel will surface this record automatically if a related incident is filed against this repo.
The decision and its reasoning live inside the check itself, legible to a security officer three weeks later with no original context.
That last sentence is not for Priya. It is there to make her security officer's scenario tangible at the moment Priya makes her decision. The two users of this system, separated by three weeks, are connected through one interface moment.
What failure looks like
Since this is speculative, the useful question is not "did it work" but "what would break it."
What I would build next
The healthcare parallel
In LQA I replaced a compliance score with factor-level reasoning. Scores tell clinicians they passed or failed without telling them what to fix. They stopped reading scores. The flag became a number to clear.
CVE flags without reasoning become the same thing. A gate to route around rather than a signal to evaluate.
The timing argument runs deeper. Both LQA and Sentinel move the accountability moment to the point of decision rather than a review cycle weeks later. That is not a UX improvement. In both domains it is the difference between a system that changes how people think and one that generates paperwork after the fact.