Skip to content

Exit confidence threshold (≥40) causes false-positive completions on documentation keywords #224

@Shatla-tech

Description

@Shatla-tech

Problem

In lib/response_analyzer.sh, the heuristic exit condition triggers at a very low confidence score:

```bash

line ~641

if [[ $confidence_score -ge 40 || "$has_completion_signal" == "true" ]]; then
exit_signal=true
fi
```

A score of 40 can be reached from just 4 keyword matches (+10 each for "done", "complete", etc.). This means:

  • Writing a CHANGELOG entry ("implementation complete")
  • Updating a README ("setup is done")
  • Any documentation mentioning completion words

...can all trigger premature exit mid-task.

Impact

Ralph exits before the task is actually finished whenever Claude writes documentation or comments containing common completion words. This is a silent failure — the loop ends, no error is shown, but work is incomplete.

Suggested Fix

Raise the threshold significantly and require explicit EXIT_SIGNAL when in JSON mode (which is the default):

```bash

For JSON output mode (default), require explicit EXIT_SIGNAL

if [[ "$CLAUDE_OUTPUT_FORMAT" == "json" ]]; then
# JSON mode: only exit on explicit EXIT_SIGNAL from RALPH_STATUS block
# Never use heuristic confidence scoring — Claude controls exit explicitly
: # exit_signal already set from JSON parse above
else
# Text mode fallback: raise threshold to 70 (was 40)
if [[ $confidence_score -ge 70 && "$has_completion_signal" == "true" ]]; then
exit_signal=true
fi
fi
```

This respects the existing JSON mode design intent (Claude sends explicit EXIT_SIGNAL) and makes text mode heuristics less fragile.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions