feat(cli): graph + vectors + installer index progress (catch-up: PR-2/3/4)#330
Merged
Conversation
Propose (propose/active/INDEX-OUTPUT-REWORK-PROPOSE.md), plan (plans/active/PLAN-INDEX-OUTPUT-REWORK.md), and per-PR agent prompts (plans/active/AGENT-PROMPTS-INDEX-OUTPUT-REWORK.md) for unifying the index-build output across init/increment/install/reprocess/update and adding a real progress bar (rich + JCIRAG_PROGRESS protocol). Gating spike passed 2026-06-14: a print(file=sys.stderr) inside a CocoIndex flow function reaches the parent's captured stderr (130/130 lines relayed; pre-walk divergence 0 on the fixture). PR-1 unblocked. Planning only; no production code changed. Co-Authored-By: Claude <[email protected]>
rich>=14,<15 (cocoindex[lancedb] requires rich>=14). New java_codebase_rag/progress.py: ProgressEvent, parse_progress_line, IndexProgressRenderer (rich Live + non-TTY fallback), ProgressRelay (parse-first, single-writer). 13 light unit tests. No production caller yet; PR-2+ wire it in. Co-Authored-By: Claude <[email protected]>
cocoindex[lancedb]>=1.0.0a43 transitively requires rich>=14, so the originally-suggested rich>=13.7,<14 cap is unsatisfiable (pip install -e . fails). Corrected in propose Open Q5 + risk row, plan Resolved decisions + PR-1 step + DoD, and the PR-1 agent prompt. Verified compatible on rich 14.3.4. Co-Authored-By: Claude <[email protected]>
…ne, carry-forward totals - ProgressRelay now mirrors _LineFilter's noise-continuation suppression (_suppress_next) + new test_progress_relay_suppresses_noise_continuation - remove dead _is_partial param from _route_line - clamp completed→total on status=done both directions (over-count safe) - monotonic completed (max with current) so a stray smaller done can't rewind - non-TTY done concise line carries forward last-seen total/elapsed_s Minor #6 (apply/stop atomicity) deferred to PR-2 (drain-thread wiring). Co-Authored-By: Claude <[email protected]>
PR-1 code-quality review Minor #6: apply() issues several locked rich calls not atomic vs. a concurrent stop() from the main thread. Only matters once PR-2 wires the drain thread. Added as cross-PR risk #10 with the PR-2 mitigation (gate apply on _started, or hold the Live lock across the sequence). Co-Authored-By: Claude <[email protected]>
build_ast_graph emits JCIRAG_PROGRESS kind=graph under --verbose (count-first exact total in pass 1; pass 2-6 step lines). The sync and async subprocess drains route progress events to an on_progress callback via ProgressRelay (parse-first, single-writer). init/increment/ reprocess render a determinate graph-phase bar in default TTY mode (running only after the builder spawns); --quiet silent; --verbose raw. Renderer.apply() is a no-op after stop() (drain-thread safety). Co-Authored-By: Claude <[email protected]>
…lay factory - ProgressRelay guards renderer.apply() so a render-chain exception can't silently kill the drain thread (try/except + stderr note) - incremental pass-1 total excludes removed files (done no longer undercounts then clamps) - make_relay() factory centralizes the sync+async drain wiring - drop unreachable console fallback in ProgressRelay._route_line erase unused-renderer (review Minor #3) and failing-builder tests (Minor #6) deferred. Co-Authored-By: Claude <[email protected]>
…ize phase process_*_file emit JCIRAG_PROGRESS kind=vectors (approximate total from app_main pre-walk + throttled per-file ticks); the parent emits the terminal vectors event on cocoindex exit (drives clamp-on-completion). optimize_lance_tables emits kind=optimize in-process (covers both call sites). server.run_refresh_pipeline wires the async drain + renderer. Retire Spinner + emit_vectors_start/_finish. Operator commands render Vectors -> Optimize -> Graph. Co-Authored-By: Claude <[email protected]>
_approximate_vectors_total's YAML predicate ("/application" in fn)
was always False for a bare filename, so the pre-walk under-counted
application YAML and the divergence went negative (gap=-4). Use
fn.startswith("application"). Move the vectors tick emission under
the lock to match its comment. Drop a dead type-ignore. Tighten the
divergence test to assert gap==0 on the fixture.
Co-Authored-By: Claude <[email protected]>
install/update render the unified Vectors->Optimize->Graph progress on stderr during their indexing sub-step (no longer stdout chatter / silent quiet=True); wizard conversational stdout is preserved. _cmd_update forwards --quiet/--verbose (both ignored today); _cmd_install forwards --verbose. CLI docs + README updated. Co-Authored-By: Claude <[email protected]>
run_update's graph catch-up is best-effort again (graph failure -> exit 0 + Warning, not exit 1) — matches the original intent and the output/UX- only scope. Both installer brackets catch BaseException to set the failed footer marker before re-raising. Footer uses perf_counter like the operator footer. Co-Authored-By: Claude <[email protected]>
Catch-up: master advanced (#322 installer cross_service_resolution, #323 config embedding.model resolution, #325 version 0.6.2, #326 PR-1 progress.py) while the index-output-rework stack was based on #320. This merges those in so the catch-up PR (#330) carries only PR-2/3/4. Conflicts resolved (both add/add, feature branch is the superset): - java_codebase_rag/progress.py (master had PR-1 state; branch has PR-1 + CallbackRenderer/make_relay/build_index_progress_context) - tests/test_progress.py (master had PR-1's 14 tests; branch adds PR-2/3/4 tests) Auto-merged cleanly: installer.py (#322 + PR-4), pyproject.toml (version 0.6.2 + rich>=14,<15), tests/test_installer.py. Verified: ruff clean; full suite 833 passed, 13 skipped (heavy-gated). Co-Authored-By: Claude <[email protected]>
Owner
Author
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What (catch-up)
This is the catch-up for the index-output-rework stack. The four stacked PRs (#326–#329) each merged into their base branch (the previous stack branch), not
master— somastergot only PR-1 (#326). This PR brings the remaining PR-2 (graph) + PR-3 (vectors/optimize/retire) + PR-4 (installer) ontomaster.Since
masteralready has PR-1's tree (via the #326 squash), the Files Changed here are PR-2 + PR-3 + PR-4 only. Seepropose/active/INDEX-OUTPUT-REWORK-PROPOSE.md+plans/active/PLAN-INDEX-OUTPUT-REWORK.mdfor the full design (each PR was already spec-reviewed and code-quality-reviewed individually).Changes
build_ast_graph.pyemitsJCIRAG_PROGRESS kind=graph(count-first exact pass-1 total + pass 2–6 steps); sync+async drains route progress viaProgressRelay/make_relay; operator commands render a determinate graph bar.java_index_flow_lancedb.pyemitskind=vectors(approximate total + throttled per-file ticks; parent emits the terminal event);lance_optimizeemitskind=optimizein-process (both call sites);Spinner/emit_vectors_*retired; fullVectors → Optimize → Graphlist.install/updaterender the unified progress on stderr (wizard stdout preserved);--quiet/--verbosewired throughupdate,--verbosethroughinstall.Merge reconciliation
masteradvanced since the stack branched (#322 installer, #323 config, #325 version, #326 PR-1). This branch mergesorigin/masterin:java_codebase_rag/progress.py,tests/test_progress.py(master had the PR-1 state; the branch has PR-1 +CallbackRenderer/make_relay/build_index_progress_context+ later tests).installer.py(feat(installer): default cross_service_resolution to brownfield_only on install #322cross_service_resolution+ PR-4),pyproject.toml(version0.6.2+rich>=14,<15),tests/test_installer.py.Ontology / reindex
None (stays 17; pure output/UX; no re-index). New dep
rich>=14,<15landed in #326.Tests
.venv/bin/ruff check .clean;.venv/bin/python -m pytest tests -q→ 833 passed, 13 skipped (heavy cocoindex/torch tests gated; no segfault). End-to-endjava-codebase-rag initrendersvectors → optimize → graphon stderr with clean JSON stdout.Recommend: squash-merge
The branch carries the full stack's commits (PR-1's are already on
mastervia the #326 squash). A squash merge lands PR-2/3/4 as one clean commit onmasterwithout duplicating PR-1's history.🤖 Generated with Claude Code