AGENT CONTROL GUIDE
By AnyCap Team
Human-in-the-loop AI agents
Human-in-the-loop AI places an explicit review, correction, or approval checkpoint inside an otherwise automated workflow. The human is not there to repeat work the agent can already do. They intervene where judgment, accountability, or visual context matters, and their response becomes structured state for the next agent step.
THE EXECUTION LOOP
A review checkpoint should change what the agent does next
01 / Produce
The agent creates a reviewable artifact
A design, screenshot, generated image, video, or diagram reaches a point where human judgment matters more than another automatic retry.
02 / Pause
The harness opens a review checkpoint
Execution pauses before an irreversible or high-cost action. The checkpoint names what the reviewer must decide, instead of asking for generic feedback.
03 / Ground
The human points to the evidence
Rectangles, arrows, points, freehand marks, labels, or a narrated recording tie each comment to the relevant visual region or moment.
04 / Continue
The agent receives structured feedback
The returned annotations become workflow state. The agent can revise the artifact, request another review, or continue after explicit approval.
Three different reasons to involve a human
Approval
May the workflow continue?
Publishing, spending credits, sending an external message, or accepting a final artifact.
Correction
What exactly must change?
UI review, image revision, video feedback, or any task where spatial context prevents ambiguity.
Escalation
Is the agent outside its safe boundary?
Conflicting evidence, missing permissions, uncertain policy, repeated failures, or a decision that requires accountable human judgment.
VERIFIED ANYCAP WORKFLOW
Structured visual feedback instead of a vague approval prompt
AnyCap can open a non-blocking annotation session for an image, URL, video, or audio target. The reviewer marks the artifact and clicks Done. The agent polls the session, reads the annotations or recording, applies the requested change, and can open a fresh review round.
anycap annotate hero-banner.png --no-wait -o hero-banner-annotated.pnganycap annotate poll --session <session_id>URL review uses a narrated screen recording as its primary artifact; image, video, and audio sessions can support collaborative visual review. A simple yes/no decision does not need annotation and should stay a lightweight harness approval.
Failure boundaries
Do not ask for human review after every harmless step. Constant approval destroys the speed benefit of an agentic workflow.
Do not treat a vague comment such as “make it better” as structured feedback. The checkpoint should collect a decision, a marked region, or a concrete requested change.
Do not let the model silently infer approval. The harness must preserve an explicit stop condition and record the human response before continuation.
Do not confuse AnyCap with the whole harness. AnyCap supplies the review and media capability; scheduling, permissions, retries, and continuation policy remain the harness's responsibility.