Skip to content

Commit 09dc36d

Browse files
committed
[Wiring] §01: unified install pipeline (layers 1-15, update.sh) + three-tier workspace separation (DEC-040 migration)
1 parent 2a24bd9 commit 09dc36d

1 file changed

Lines changed: 67 additions & 10 deletions

File tree

docs/wiring/exocortex_wiring_and_logic.html

Lines changed: 67 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -587,8 +587,13 @@ <h3>Filesystem layout</h3>
587587
</tr>
588588
<tr>
589589
<td class="path-col">/a0/usr/Exocortex/</td>
590-
<td>Exocortex working data (wiki, journals, sleep reports, office state, procedural memory)</td>
591-
<td>Per-container, survives restart</td>
590+
<td>Exocortex repo clone (git, origin = Stranglehold/Exocortex) + deployed code + <strong>curated</strong> artifacts (specs, design notes, letters, essays, <code>prompts/idle_activation.md</code>). <strong>As of 2026-05-25 the agent's runtime OUTPUT was migrated OUT</strong> (it was polluting the git clone + blocking <code>update.sh</code> pulls) — see the workspace row.</td>
591+
<td>Per-container; updated via <code>update.sh</code> (git pull). Was at risk on image rebuild before; runtime now lives in the workspace.</td>
592+
</tr>
593+
<tr>
594+
<td class="path-col">/a0/usr/workdir/workspace/</td>
595+
<td><strong>Agent sovereign workspace (DEC-040).</strong> The in-A0 agent's living output, separated from the repo: <code>wiki/</code>, <code>self-improvement/journal.jsonl</code>, <code>field-reports/</code>, <code>office/</code> engine state, operational <code>config.json</code>, governing docs (<code>program.md</code>, <code>interests.md</code>), and an empty, agent-authored <code>identity.md</code> (DEC-040 — agent-only write).</td>
596+
<td>Survives <strong>both</strong> A0 image rebuild <em>and</em> Exocortex <code>update.sh</code> pulls — it is under <code>/a0/usr/</code> (persistent) yet outside the repo (untouched by git).</td>
592597
</tr>
593598
<tr>
594599
<td class="path-col">/a0/usr/memory/</td>
@@ -724,6 +729,24 @@ <h3>Deploy targets summary</h3>
724729
</tbody>
725730
</table>
726731

732+
<h3>Install pipeline (unified) &amp; three-tier separation</h3>
733+
<p><strong>One entry point:</strong> <code>update.sh</code><code>git pull</code><code>install_all.sh</code>. Re-running after any A0 update re-applies everything, including base-path patches. As of 2026-05-24 the master <code>install_all.sh</code> covers <strong>layers 1–15</strong> (was 1–10): the core hardening stack <em>plus</em> the OSS + SWARMFISH V2 plugins, the idle engine (now including the <code>idle_watch.py</code> daemon + its <code>[program:idle_watch]</code> supervisord entry — previously an orphan deployed by no installer), the theme system, metacognitive injection, epistemic integrity, write guard, and artifact system.</p>
734+
<ul>
735+
<li><strong>Run mode:</strong> intended to run <em>inside</em> the container from <code>/a0/usr/Exocortex/</code>. A docker shim in <code>install_all.sh</code> converts <code>docker cp</code>/<code>docker exec</code> into local ops, so the container name is moot in-container; <code>export CONTAINER=${CONTAINER:-exocortex_v16}</code> covers host-side runs (env-aware child scripts inherit it). Host-mode also hits Git-Bash path translation + Windows-python-can't-open-<code>/d/</code> issues — in-container is the supported path.</li>
736+
<li><strong>V1 vs V2:</strong> OSS/SWARMFISH <strong>V2</strong> (in-A0 plugins) are canonical and in the one-shot. The V1 standalone Docker services (<code>services/oss</code>, <code>services/swarmfish</code>) are preserved as an independent project but deliberately NOT wired into <code>install_all.sh</code>.</li>
737+
<li><strong>Excluded from the one-shot:</strong> <code>install_htn_plans</code> (superseded by the graph engine), <code>install_phase3_profiles</code> (its base <code>_50</code> is dead/shadowed), <code>install_artifact_browser</code> (not live), and host-side eval tooling.</li>
738+
</ul>
739+
<p><strong>Three-tier persistence model</strong> — the reason the 2026-05-25 workspace migration matters:</p>
740+
<table>
741+
<thead><tr><th>Tier</th><th>Location</th><th>Survives</th><th>Holds</th></tr></thead>
742+
<tbody>
743+
<tr><td>A0 base</td><td class="path-col">/a0/ (non-usr)</td><td>nothing — wiped on image rebuild</td><td>the framework</td></tr>
744+
<tr><td>Exocortex repo</td><td class="path-col">/a0/usr/Exocortex/</td><td>A0 rebuild; <em>changed</em> by <code>update.sh</code> (git)</td><td>code + curated artifacts</td></tr>
745+
<tr><td>Agent workspace</td><td class="path-col">/a0/usr/workdir/workspace/</td><td><strong>both</strong> A0 rebuild and <code>update.sh</code></td><td>the agent's living output (DEC-040)</td></tr>
746+
</tbody>
747+
</table>
748+
<p>Runtime paths (office / wiki / self-improvement / field-reports) were repointed from <code>/a0/usr/Exocortex/</code> to the workspace on 2026-05-25; code, prompt, and module-import paths stay in Exocortex. Verified live: idle cycle <strong>#552 wrote engine state to the workspace</strong>, leaving the old Exocortex location stale (cycle_count 551). The destructive cleanup (git rm the tracked wiki, remove old runtime, gitignore) is gated until the new home has baked over several cycles.</p>
749+
727750
<h3>See also</h3>
728751
<ul>
729752
<li><a href="#sec-02">§02 Extension Load Path Resolution</a> — the path-with-<code>python</code>-segment finding for extension files</li>
@@ -2769,6 +2792,10 @@ <h2><span class="num">09</span>Reasoning Persistence &amp; PACE <a class="anchor
27692792
<strong>Status: ACTIVE INERT</strong> — extensions exist, fire on schedule, do the right computation, and their output never reaches the model. Verified 2026-05-16 via grep against both repo and live container code. Fix design is approved (Opus 2026-05-16 response). Implementation drafted but held for deployment after empirical observation of the heartbeat fix in isolation.
27702793
</div>
27712794

2795+
<div class="callout info">
2796+
<strong>Relationship to model-side thinking (2026-05-16 amendment):</strong> The reasoning-state and PACE injections are designed to deliver <em>compressed diagnostic state</em> across turn boundaries — what the model already tried, what worked, what the current strategy is. They are not a substitute for the model's own real-time reasoning. Per <code>enable_thinking_correction_20260516.md</code>, the model is now configured to think freely (<code>enable_thinking: true</code> at the request level; <code>--reasoning off</code> at the server only suppresses empty template-tag injection). The model thinks during a turn; the injectors carry compressed traces of past thinking across turns. Both are load-bearing for agentic capability. Quality over speed.
2797+
</div>
2798+
27722799
<h3>Wiring — The Broken Chain</h3>
27732800
<div class="diagram">
27742801
<div class="mermaid">
@@ -3183,6 +3210,24 @@ <h4>Seam #19 — Reasoning state &amp; PACE injection chain inert <span class="s
31833210
<p><strong>Resolution path:</strong> Step 1 — deploy heartbeat fix, observe one cycle. Step 2 — if loops persist (separate from premature stale-detect), deploy injectors. Step 3 — observe another cycle for preamble repetition disappearing.</p>
31843211
</div>
31853212

3213+
<h4>Seam #20 — Qwen mid-response token-repetition degeneration <span class="status warn">RARE</span></h4>
3214+
<div class="subsys">
3215+
<p><strong>Where it lives:</strong> The Qwen3.6-27B generation path. Surfaces inside the model's <code>thoughts</code> array as repeated short tokens (<code>"EXECUTING."</code>, <code>"NOW."</code>, <code>"GO."</code>, <code>"Step count: 8."</code>) that never produce a closing JSON brace. The agent's outer loop waits indefinitely. Web-UI Stop is the only intervention.</p>
3216+
<p><strong>What goes wrong:</strong> Observed once during cycle 17 (MAINTAIN, pre-heartbeat-fix). The model entered a degenerate generation mode mid-response. <strong>Cycles 18-61 ran cleanly on the same model and same config</strong>, so this is intermittent under the current production conditions, not categorical. Two remaining hypotheses:</p>
3217+
<ol>
3218+
<li><strong>Context-length degradation</strong> at whatever depth cycle 17 reached. Possible but cycles 18-61 covered similar depth.</li>
3219+
<li><strong>Quantization sensitivity</strong> — Q4_K_S behaving differently from Q4_K_M / Q4_K_XL at the generation boundary. The active model is <code>Qwen3.6-27B-Q4_K_S</code>; the consolidated brief (2026-05-16) directs the upstream MTP build to <code>Qwen3.6-27B-Q4_K_M-mtp.gguf</code>.</li>
3220+
</ol>
3221+
<p><strong>Withdrawn hypothesis:</strong> Earlier wiring drafts named <code>enable_thinking: false</code> not being sent as the strongest candidate. <strong>Jake's 2026-05-16 decision reversed that direction</strong> per <code>enable_thinking_correction_20260516.md</code>: thinking is load-bearing for agent capability and should be enabled. v16's <code>chat_model.kwargs.enable_thinking: true</code> is now correct. The <code>--reasoning off</code> server flag suppresses only the empty template-tag injection; genuine reasoning is preserved. Quality over speed. MTP accelerates response tokens and leaves thinking tokens alone, which is the accepted tradeoff.</p>
3222+
<p><strong>How to detect (live):</strong></p>
3223+
<ul>
3224+
<li>Watch the web-UI <em>Office</em> tab during a cycle — degenerate token streams are visible character-by-character via the response_stream_chunk channel</li>
3225+
<li>Container log scan: <code>docker logs exocortex_v16 --since=10m | grep -E '"EXECUTING\\.|"NOW\\.|"GO\\."'</code></li>
3226+
<li>Behavioral signal: cycle runs past expected duration without closing; supervisor's existing loop detection fires at the message level, blind to within-generation degeneration</li>
3227+
</ul>
3228+
<p><strong>Resolution path:</strong> Watch and wait. The upstream MTP build (consolidated brief Priority 1) brings in a new server (ggml-org main with <code>--spec-type draft-mtp</code>) on a different quantization (Q4_K_M). If the symptom never recurs under that backend, the cause was likely either Q4_K_S quirks or a transient stimulus tied to cycle 17 specifically. If it does recur on the new backend, attention shifts to context-length sensitivity. No code-level fix is currently planned; the existing supervisor loop-detection + web-UI Stop are sufficient handling for a single observed occurrence.</p>
3229+
</div>
3230+
31863231
<h4>Seam #8 — Wrong-path extension deploy <span class="status err">ACTIVE</span></h4>
31873232
<div class="subsys">
31883233
<p><strong>Where it lives:</strong> Any manual <code>docker cp</code> that constructs the destination path by hand. The required path includes the <code>python/</code> segment: <code>/a0/usr/agents/agent0/extensions/python/&lt;hook&gt;/</code>. The natural-feeling path that omits it (<code>/a0/usr/agents/agent0/extensions/&lt;hook&gt;/</code>) is invisible to A0's loader.</p>
@@ -3250,6 +3295,9 @@ <h2><span class="num">14</span>Deployment &amp; Install Pattern <a class="anchor
32503295
<p class="section-summary">
32513296
27 install scripts in <code>scripts/</code> totaling 2420 lines. They are the canonical path for getting Exocortex components into a running container. Audit reveals: <strong>only 4 of 27 have a function abstraction</strong> for the install-file primitive; <strong>0 of 27 verify md5</strong> after deploy. The verification gap is the empirical cause of tonight's mis-deploy: every install script trusts <code>docker cp</code> to land the right bytes at the right path, and none of them re-read to confirm. Path discipline is correct (<code>EXT_DEST</code> in <code>install_idle_engine.sh</code> points to the proper <code>extensions/python/&lt;hook&gt;/</code> form); the bug was that I bypassed the install script entirely and did a manual <code>docker cp</code> to the wrong path.
32523297
</p>
3298+
<p class="section-summary">
3299+
<strong>2026-05-16 update:</strong> the verification gap now has a runnable instrument. <code>scripts/audit_extensions.py</code> walks every extension hook on a live container, computes md5 across all four extension-host paths, and classifies each file (<em>OK</em>, <em>DUP same</em>, <em>DUP DIVERGENT</em>, <em>DUP CANONICAL DIVERGENT</em>, <em>WRONG-PATH DEAD</em>). First run on v16 found <strong>7 dead extensions</strong> (silently not loaded) and <strong>43 divergent copies</strong> (stale wrong-path duplicates behind canonical versions). v17 has its own drift (10 dead, 19 divergent). The tool is intended as the empirical pre-commit / post-deploy check the shared-library refactor (below) will codify; it can be run standalone today: <code>python3 scripts/audit_extensions.py exocortex_v16</code>.
3300+
</p>
32533301

32543302
<h3>Inventory</h3>
32553303
<p>Each script targets one component or layer. Most have grown over time and inline the deploy primitives directly. Note the line-count distribution: a few large scripts (200+ lines) deploy multi-file features; many small scripts (20-60 lines) deploy single components.</p>
@@ -3446,6 +3494,16 @@ <h2><span class="num">??</span>Open Questions <a class="anchor" href="#sec-open-
34463494

34473495
<h3>Active</h3>
34483496

3497+
<div class="subsys">
3498+
<div class="subsys-header">
3499+
<h3>Q8 — <code>/api/api_message</code> request discarded on client connection close <span class="status warn">TRACKED — return to diagnose</span></h3>
3500+
</div>
3501+
<p><strong>Surfaced:</strong> 2026-05-17, STRESS_TEST_014 harness bring-up. A plain <code>curl -m &lt;timeout&gt;</code> POST to <code>/api/api_message</code> that closes the socket at timeout (before A0 finishes the multi-minute turn) results in: the message lands, the full Exocortex before-main stack fires (MEM-ENHANCE → BST → REASON-INJ → PACE → KARP all logged), then processing orphans — no LLM completion, inference server idle, no error, HTTP 000 client-side, chat left with an incomplete/greeting message.</p>
3502+
<p><strong>Likely root cause (not an agent bug):</strong> documented inline in <code>services/idle_watch.py _fire_fresh_cycle</code>: <em>"Raw-socket close caused Uvicorn to discard requests; keeping the connection alive ensures delivery."</em> The endpoint requires a long-lived connection held open for the entire turn (minutes). The idle engine works because it posts via <code>http.client</code> with <code>timeout=3600</code> in a daemon thread, waits only ~3s for a quick 4xx, then reads results <em>asynchronously from state files</em> — it never depends on the HTTP response body. Closing early = Uvicorn discards the in-flight request.</p>
3503+
<p><strong>Why tracked, not fixed now:</strong> per Jake 2026-05-17, the STRESS_TEST_014 harness pivots to the proven idle-engine invocation pattern (long-lived connection + poll chat.json) so injection-chain validation isn't blocked. The api_message connection-lifetime behavior is a real, separate defect/gotcha in a documented endpoint and warrants its own diagnosis: is it Uvicorn config, an A0 request-handling path, or expected-but-undocumented? Related to but distinct from <code>memory/feedback_api_message_health_check.md</code> (don't use as readiness probe) and the prior ping-loop incident.</p>
3504+
<p><strong>Next step when revisited:</strong> confirm the long-connection pattern delivers reliably (the harness pivot will be the first data point), then determine whether the early-close discard is fixable server-side or must be documented as a hard client contract.</p>
3505+
</div>
3506+
34493507
<div class="subsys">
34503508
<div class="subsys-header">
34513509
<h3>Q1 — Qwen mid-response token-repetition pattern <span class="status err">ACTIVE</span></h3>
@@ -3459,19 +3517,18 @@ <h3>Q1 — Qwen mid-response token-repetition pattern <span class="status err">A
34593517
<li>Quantization mismatch — LM Studio reports <code>Qwen3.6-27B-Q4_K_S.gguf</code> loaded; Opus's letter referenced production using <code>Qwen3.6-27B-Q4_K_XL</code>. Different quantization may have different output stability properties.</li>
34603518
<li>Some interaction with the integrity-check Phase 0 specifically — cycle 17 was investigating a real status-mismatch finding in the wiki index and may have hit a tool/result pattern that primed degenerate output.</li>
34613519
</ul>
3462-
<p><strong>Next investigation:</strong> Check the actual HTTP request body A0 sends to LM Studio. Verify <code>enable_thinking</code> field. If missing, that's a one-line fix in the LiteLLM config or A0's chat model adapter. Could also try forcing the Q4_K_XL model and rerunning the same cycle conditions.</p>
3463-
<p><strong>Status:</strong> Not blocking overnight observation (v16 cycles since 17 have completed without reproducing it). Worth investigating in the morning when context is fresh.</p>
3520+
<p><strong>2026-05-16 ~17:30 UTC update — Jake reversed hypothesis 1:</strong> per <code>enable_thinking_correction_20260516.md</code>, thinking is load-bearing for agentic capability (the <code>&lt;think&gt;</code> block is where the model reasons about tool selection, argument construction, multi-step planning). v16's <code>enable_thinking: true</code> is now <em>correct</em>. Hardcoded <code>"enable_thinking": False</code> was removed from <code>extensions/before_main_llm_call/_71_cache_warmer.py</code> and <code>inference/warm_cache.py</code>. Server-side <code>--reasoning off</code> remains in place — it suppresses only empty template-tag injection, not genuine reasoning. <strong>Hypothesis 1 is withdrawn.</strong> Remaining: context-length degradation, Q4_K_S quantization sensitivity.</p>
3521+
<p><strong>Status:</strong> Demoted to RARE pending recurrence. Upstream MTP build (consolidated brief Priority 1) moves to Q4_K_M, which is the next opportunity to bisect quantization sensitivity.</p>
34643522
</div>
34653523

34663524
<div class="subsys">
34673525
<div class="subsys-header">
3468-
<h3>Q5 — <code>steps_used: 0</code> reporting bug across recent cycles <span class="status warn">MINOR</span></h3>
3526+
<h3>Q5 — <code>steps_used: 0</code> reporting bug across recent cycles <span class="status ok">FIXED 2026-05-16</span></h3>
34693527
</div>
3470-
<p><strong>Surfaced:</strong> 2026-05-16 ~07:50 UTC while resolving Q2. Every cycle in <code>feed.jsonl</code> from 2026-05-16 (cycle 14 EXPLORE, cycles 18-21 BUILD) reports <code>"steps_used": 0</code> despite each cycle producing detailed productive output (wiki page deepened, memories saved, page marked STABLE). The May 14 entries show real step counts (cycle 30: 14 steps, cycle 31: 20 steps). Between then and now the step tracking stopped working.</p>
3471-
<p><strong>What it doesn't break:</strong> Cycles still complete productively. Office UI still shows the right activity strings. Memory saves still happen. This is a metrics/reporting issue, not a behavioral one.</p>
3472-
<p><strong>What it might affect:</strong> <code>_select_cycle_type</code> doesn't use <code>steps_used</code> directly, but downstream metrics tracking does (cycle efficiency calculations, model evaluation against step budget, etc.). The longer this runs the more historical metrics get the wrong count.</p>
3473-
<p><strong>Next investigation:</strong> Trace <code>steps_used</code> back through <code>cycle_close.py</code> to find where the value is being computed or passed. Likely a regression in a recent edit. Probably a 10-minute fix once the regression is located.</p>
3474-
<p><strong>Status:</strong> Not blocking anything. Worth fixing for clean metrics going forward.</p>
3528+
<p><strong>Surfaced:</strong> 2026-05-16 ~07:50 UTC while resolving Q2.</p>
3529+
<p><strong>Root cause:</strong> The agent's activation prompt template (<code>prompts/idle_activation.md</code>) showed the agent the <code>cycle_close.py</code> invocation it must use to close a cycle. The example omitted <code>--steps-used</code>. <code>cycle_close.py</code> supports the flag (defaults to 0); the agent simply never knew it should pass it. So every cycle reported 0 regardless of actual work.</p>
3530+
<p><strong>Fix:</strong> Added <code>--steps-used &lt;N&gt;</code> to the cycle_close.py invocation block in <code>prompts/idle_activation.md</code>, plus a one-line instruction explaining that the agent should pass an approximate tool-call count. Deployed to both containers (md5 <code>4c59ddcceaea975baf99fda8dc6e20a6</code> on v16, v17, repo). No daemon restart needed — <code>_build_activation_prompt()</code> in <code>idle_watch.py</code> re-reads the template each fire.</p>
3531+
<p><strong>Verification:</strong> Pending. Next BUILD/EXPLORE/MAINTAIN cycle's <code>feed.jsonl</code> entry should show a non-zero <code>steps_used</code> if the agent honors the new instruction. If 0 persists for the next 3+ cycles, the instruction needs to be more explicit.</p>
34753532
</div>
34763533

34773534
<div class="subsys">

0 commit comments

Comments
 (0)