Skip to content

feat(cli): graph + vectors + installer index progress (catch-up: PR-2/3/4)#330

Merged
HumanBean17 merged 12 commits into
masterfrom
feat/index-installer-progress
Jun 15, 2026
Merged

feat(cli): graph + vectors + installer index progress (catch-up: PR-2/3/4)#330
HumanBean17 merged 12 commits into
masterfrom
feat/index-installer-progress

Conversation

@HumanBean17

Copy link
Copy Markdown
Owner

What (catch-up)

This is the catch-up for the index-output-rework stack. The four stacked PRs (#326#329) each merged into their base branch (the previous stack branch), not master — so master got only PR-1 (#326). This PR brings the remaining PR-2 (graph) + PR-3 (vectors/optimize/retire) + PR-4 (installer) onto master.

Since master already has PR-1's tree (via the #326 squash), the Files Changed here are PR-2 + PR-3 + PR-4 only. See propose/active/INDEX-OUTPUT-REWORK-PROPOSE.md + plans/active/PLAN-INDEX-OUTPUT-REWORK.md for the full design (each PR was already spec-reviewed and code-quality-reviewed individually).

Changes

  • PR-2build_ast_graph.py emits JCIRAG_PROGRESS kind=graph (count-first exact pass-1 total + pass 2–6 steps); sync+async drains route progress via ProgressRelay/make_relay; operator commands render a determinate graph bar.
  • PR-3java_index_flow_lancedb.py emits kind=vectors (approximate total + throttled per-file ticks; parent emits the terminal event); lance_optimize emits kind=optimize in-process (both call sites); Spinner/emit_vectors_* retired; full Vectors → Optimize → Graph list.
  • PR-4install/update render the unified progress on stderr (wizard stdout preserved); --quiet/--verbose wired through update, --verbose through install.

Merge reconciliation

master advanced since the stack branched (#322 installer, #323 config, #325 version, #326 PR-1). This branch merges origin/master in:

  • Resolved (add/add, feature branch is the superset): java_codebase_rag/progress.py, tests/test_progress.py (master had the PR-1 state; the branch has PR-1 + CallbackRenderer/make_relay/build_index_progress_context + later tests).
  • Auto-merged cleanly: installer.py (feat(installer): default cross_service_resolution to brownfield_only on install #322 cross_service_resolution + PR-4), pyproject.toml (version 0.6.2 + rich>=14,<15), tests/test_installer.py.

Ontology / reindex

None (stays 17; pure output/UX; no re-index). New dep rich>=14,<15 landed in #326.

Tests

.venv/bin/ruff check . clean; .venv/bin/python -m pytest tests -q833 passed, 13 skipped (heavy cocoindex/torch tests gated; no segfault). End-to-end java-codebase-rag init renders vectors → optimize → graph on stderr with clean JSON stdout.

Recommend: squash-merge

The branch carries the full stack's commits (PR-1's are already on master via the #326 squash). A squash merge lands PR-2/3/4 as one clean commit on master without duplicating PR-1's history.

🤖 Generated with Claude Code

HumanBean17 and others added 12 commits June 14, 2026 23:15
Propose (propose/active/INDEX-OUTPUT-REWORK-PROPOSE.md), plan
(plans/active/PLAN-INDEX-OUTPUT-REWORK.md), and per-PR agent prompts
(plans/active/AGENT-PROMPTS-INDEX-OUTPUT-REWORK.md) for unifying the
index-build output across init/increment/install/reprocess/update and
adding a real progress bar (rich + JCIRAG_PROGRESS protocol).

Gating spike passed 2026-06-14: a print(file=sys.stderr) inside a
CocoIndex flow function reaches the parent's captured stderr (130/130
lines relayed; pre-walk divergence 0 on the fixture). PR-1 unblocked.

Planning only; no production code changed.

Co-Authored-By: Claude <[email protected]>
rich>=14,<15 (cocoindex[lancedb] requires rich>=14). New
java_codebase_rag/progress.py: ProgressEvent, parse_progress_line,
IndexProgressRenderer (rich Live + non-TTY fallback), ProgressRelay
(parse-first, single-writer). 13 light unit tests. No production
caller yet; PR-2+ wire it in.

Co-Authored-By: Claude <[email protected]>
cocoindex[lancedb]>=1.0.0a43 transitively requires rich>=14, so the
originally-suggested rich>=13.7,<14 cap is unsatisfiable (pip install -e .
fails). Corrected in propose Open Q5 + risk row, plan Resolved decisions +
PR-1 step + DoD, and the PR-1 agent prompt. Verified compatible on rich
14.3.4.

Co-Authored-By: Claude <[email protected]>
…ne, carry-forward totals

- ProgressRelay now mirrors _LineFilter's noise-continuation suppression
  (_suppress_next) + new test_progress_relay_suppresses_noise_continuation
- remove dead _is_partial param from _route_line
- clamp completed→total on status=done both directions (over-count safe)
- monotonic completed (max with current) so a stray smaller done can't rewind
- non-TTY done concise line carries forward last-seen total/elapsed_s

Minor #6 (apply/stop atomicity) deferred to PR-2 (drain-thread wiring).

Co-Authored-By: Claude <[email protected]>
PR-1 code-quality review Minor #6: apply() issues several locked rich
calls not atomic vs. a concurrent stop() from the main thread. Only
matters once PR-2 wires the drain thread. Added as cross-PR risk #10
with the PR-2 mitigation (gate apply on _started, or hold the Live
lock across the sequence).

Co-Authored-By: Claude <[email protected]>
build_ast_graph emits JCIRAG_PROGRESS kind=graph under --verbose
(count-first exact total in pass 1; pass 2-6 step lines). The sync
and async subprocess drains route progress events to an on_progress
callback via ProgressRelay (parse-first, single-writer). init/increment/
reprocess render a determinate graph-phase bar in default TTY mode
(running only after the builder spawns); --quiet silent; --verbose raw.
Renderer.apply() is a no-op after stop() (drain-thread safety).

Co-Authored-By: Claude <[email protected]>
…lay factory

- ProgressRelay guards renderer.apply() so a render-chain exception
  can't silently kill the drain thread (try/except + stderr note)
- incremental pass-1 total excludes removed files (done no longer
  undercounts then clamps)
- make_relay() factory centralizes the sync+async drain wiring
- drop unreachable console fallback in ProgressRelay._route_line

erase unused-renderer (review Minor #3) and failing-builder tests
(Minor #6) deferred.

Co-Authored-By: Claude <[email protected]>
…ize phase

process_*_file emit JCIRAG_PROGRESS kind=vectors (approximate total from
app_main pre-walk + throttled per-file ticks); the parent emits the
terminal vectors event on cocoindex exit (drives clamp-on-completion).
optimize_lance_tables emits kind=optimize in-process (covers both call
sites). server.run_refresh_pipeline wires the async drain + renderer.
Retire Spinner + emit_vectors_start/_finish. Operator commands render
Vectors -> Optimize -> Graph.

Co-Authored-By: Claude <[email protected]>
_approximate_vectors_total's YAML predicate ("/application" in fn)
was always False for a bare filename, so the pre-walk under-counted
application YAML and the divergence went negative (gap=-4). Use
fn.startswith("application"). Move the vectors tick emission under
the lock to match its comment. Drop a dead type-ignore. Tighten the
divergence test to assert gap==0 on the fixture.

Co-Authored-By: Claude <[email protected]>
install/update render the unified Vectors->Optimize->Graph progress on
stderr during their indexing sub-step (no longer stdout chatter / silent
quiet=True); wizard conversational stdout is preserved. _cmd_update
forwards --quiet/--verbose (both ignored today); _cmd_install forwards
--verbose. CLI docs + README updated.

Co-Authored-By: Claude <[email protected]>
run_update's graph catch-up is best-effort again (graph failure -> exit 0
+ Warning, not exit 1) — matches the original intent and the output/UX-
only scope. Both installer brackets catch BaseException to set the failed
footer marker before re-raising. Footer uses perf_counter like the
operator footer.

Co-Authored-By: Claude <[email protected]>
Catch-up: master advanced (#322 installer cross_service_resolution,
#323 config embedding.model resolution, #325 version 0.6.2, #326 PR-1
progress.py) while the index-output-rework stack was based on #320.
This merges those in so the catch-up PR (#330) carries only PR-2/3/4.

Conflicts resolved (both add/add, feature branch is the superset):
- java_codebase_rag/progress.py  (master had PR-1 state; branch has
  PR-1 + CallbackRenderer/make_relay/build_index_progress_context)
- tests/test_progress.py         (master had PR-1's 14 tests; branch
  adds PR-2/3/4 tests)

Auto-merged cleanly: installer.py (#322 + PR-4), pyproject.toml
(version 0.6.2 + rich>=14,<15), tests/test_installer.py.

Verified: ruff clean; full suite 833 passed, 13 skipped (heavy-gated).

Co-Authored-By: Claude <[email protected]>
@HumanBean17

Copy link
Copy Markdown
Owner Author
  • Non-TTY graph phase emits 6 graph · done terminal lines (one per pass); could collapse passes 2–6 into one summary line.
    • erase spins up an unused renderer (blank Live region, no progress to render) — needs a small header/footer-vs-renderer refactor.
    • A couple of footer/separator consistencies between the operator and installer footers ((exit=N) suffix, _PIPELINE_SEP literal).

@HumanBean17 HumanBean17 merged commit a5427f9 into master Jun 15, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant