You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<td><strong>Agent sovereign workspace (DEC-040).</strong> The in-A0 agent's living output, separated from the repo: <code>wiki/</code>, <code>self-improvement/journal.jsonl</code>, <code>field-reports/</code>, <code>office/</code> engine state, operational <code>config.json</code>, governing docs (<code>program.md</code>, <code>interests.md</code>), and an empty, agent-authored <code>identity.md</code> (DEC-040 — agent-only write).</td>
596
+
<td>Survives <strong>both</strong> A0 image rebuild <em>and</em> Exocortex <code>update.sh</code> pulls — it is under <code>/a0/usr/</code> (persistent) yet outside the repo (untouched by git).</td>
<h3>Install pipeline (unified) & three-tier separation</h3>
733
+
<p><strong>One entry point:</strong><code>update.sh</code> → <code>git pull</code> → <code>install_all.sh</code>. Re-running after any A0 update re-applies everything, including base-path patches. As of 2026-05-24 the master <code>install_all.sh</code> covers <strong>layers 1–15</strong> (was 1–10): the core hardening stack <em>plus</em> the OSS + SWARMFISH V2 plugins, the idle engine (now including the <code>idle_watch.py</code> daemon + its <code>[program:idle_watch]</code> supervisord entry — previously an orphan deployed by no installer), the theme system, metacognitive injection, epistemic integrity, write guard, and artifact system.</p>
734
+
<ul>
735
+
<li><strong>Run mode:</strong> intended to run <em>inside</em> the container from <code>/a0/usr/Exocortex/</code>. A docker shim in <code>install_all.sh</code> converts <code>docker cp</code>/<code>docker exec</code> into local ops, so the container name is moot in-container; <code>export CONTAINER=${CONTAINER:-exocortex_v16}</code> covers host-side runs (env-aware child scripts inherit it). Host-mode also hits Git-Bash path translation + Windows-python-can't-open-<code>/d/</code> issues — in-container is the supported path.</li>
736
+
<li><strong>V1 vs V2:</strong> OSS/SWARMFISH <strong>V2</strong> (in-A0 plugins) are canonical and in the one-shot. The V1 standalone Docker services (<code>services/oss</code>, <code>services/swarmfish</code>) are preserved as an independent project but deliberately NOT wired into <code>install_all.sh</code>.</li>
737
+
<li><strong>Excluded from the one-shot:</strong><code>install_htn_plans</code> (superseded by the graph engine), <code>install_phase3_profiles</code> (its base <code>_50</code> is dead/shadowed), <code>install_artifact_browser</code> (not live), and host-side eval tooling.</li>
738
+
</ul>
739
+
<p><strong>Three-tier persistence model</strong> — the reason the 2026-05-25 workspace migration matters:</p>
<tr><td>A0 base</td><tdclass="path-col">/a0/ (non-usr)</td><td>nothing — wiped on image rebuild</td><td>the framework</td></tr>
744
+
<tr><td>Exocortex repo</td><tdclass="path-col">/a0/usr/Exocortex/</td><td>A0 rebuild; <em>changed</em> by <code>update.sh</code> (git)</td><td>code + curated artifacts</td></tr>
745
+
<tr><td>Agent workspace</td><tdclass="path-col">/a0/usr/workdir/workspace/</td><td><strong>both</strong> A0 rebuild and <code>update.sh</code></td><td>the agent's living output (DEC-040)</td></tr>
746
+
</tbody>
747
+
</table>
748
+
<p>Runtime paths (office / wiki / self-improvement / field-reports) were repointed from <code>/a0/usr/Exocortex/</code> to the workspace on 2026-05-25; code, prompt, and module-import paths stay in Exocortex. Verified live: idle cycle <strong>#552 wrote engine state to the workspace</strong>, leaving the old Exocortex location stale (cycle_count 551). The destructive cleanup (git rm the tracked wiki, remove old runtime, gitignore) is gated until the new home has baked over several cycles.</p>
749
+
727
750
<h3>See also</h3>
728
751
<ul>
729
752
<li><ahref="#sec-02">§02 Extension Load Path Resolution</a> — the path-with-<code>python</code>-segment finding for extension files</li>
@@ -2769,6 +2792,10 @@ <h2><span class="num">09</span>Reasoning Persistence & PACE <a class="anchor
2769
2792
<strong>Status: ACTIVE INERT</strong> — extensions exist, fire on schedule, do the right computation, and their output never reaches the model. Verified 2026-05-16 via grep against both repo and live container code. Fix design is approved (Opus 2026-05-16 response). Implementation drafted but held for deployment after empirical observation of the heartbeat fix in isolation.
2770
2793
</div>
2771
2794
2795
+
<divclass="callout info">
2796
+
<strong>Relationship to model-side thinking (2026-05-16 amendment):</strong> The reasoning-state and PACE injections are designed to deliver <em>compressed diagnostic state</em> across turn boundaries — what the model already tried, what worked, what the current strategy is. They are not a substitute for the model's own real-time reasoning. Per <code>enable_thinking_correction_20260516.md</code>, the model is now configured to think freely (<code>enable_thinking: true</code> at the request level; <code>--reasoning off</code> at the server only suppresses empty template-tag injection). The model thinks during a turn; the injectors carry compressed traces of past thinking across turns. Both are load-bearing for agentic capability. Quality over speed.
2797
+
</div>
2798
+
2772
2799
<h3>Wiring — The Broken Chain</h3>
2773
2800
<divclass="diagram">
2774
2801
<divclass="mermaid">
@@ -3183,6 +3210,24 @@ <h4>Seam #19 — Reasoning state & PACE injection chain inert <span class="s
3183
3210
<p><strong>Resolution path:</strong> Step 1 — deploy heartbeat fix, observe one cycle. Step 2 — if loops persist (separate from premature stale-detect), deploy injectors. Step 3 — observe another cycle for preamble repetition disappearing.</p>
<p><strong>Where it lives:</strong> The Qwen3.6-27B generation path. Surfaces inside the model's <code>thoughts</code> array as repeated short tokens (<code>"EXECUTING."</code>, <code>"NOW."</code>, <code>"GO."</code>, <code>"Step count: 8."</code>) that never produce a closing JSON brace. The agent's outer loop waits indefinitely. Web-UI Stop is the only intervention.</p>
3216
+
<p><strong>What goes wrong:</strong> Observed once during cycle 17 (MAINTAIN, pre-heartbeat-fix). The model entered a degenerate generation mode mid-response. <strong>Cycles 18-61 ran cleanly on the same model and same config</strong>, so this is intermittent under the current production conditions, not categorical. Two remaining hypotheses:</p>
3217
+
<ol>
3218
+
<li><strong>Context-length degradation</strong> at whatever depth cycle 17 reached. Possible but cycles 18-61 covered similar depth.</li>
3219
+
<li><strong>Quantization sensitivity</strong> — Q4_K_S behaving differently from Q4_K_M / Q4_K_XL at the generation boundary. The active model is <code>Qwen3.6-27B-Q4_K_S</code>; the consolidated brief (2026-05-16) directs the upstream MTP build to <code>Qwen3.6-27B-Q4_K_M-mtp.gguf</code>.</li>
3220
+
</ol>
3221
+
<p><strong>Withdrawn hypothesis:</strong> Earlier wiring drafts named <code>enable_thinking: false</code> not being sent as the strongest candidate. <strong>Jake's 2026-05-16 decision reversed that direction</strong> per <code>enable_thinking_correction_20260516.md</code>: thinking is load-bearing for agent capability and should be enabled. v16's <code>chat_model.kwargs.enable_thinking: true</code> is now correct. The <code>--reasoning off</code> server flag suppresses only the empty template-tag injection; genuine reasoning is preserved. Quality over speed. MTP accelerates response tokens and leaves thinking tokens alone, which is the accepted tradeoff.</p>
3222
+
<p><strong>How to detect (live):</strong></p>
3223
+
<ul>
3224
+
<li>Watch the web-UI <em>Office</em> tab during a cycle — degenerate token streams are visible character-by-character via the response_stream_chunk channel</li>
<li>Behavioral signal: cycle runs past expected duration without closing; supervisor's existing loop detection fires at the message level, blind to within-generation degeneration</li>
3227
+
</ul>
3228
+
<p><strong>Resolution path:</strong> Watch and wait. The upstream MTP build (consolidated brief Priority 1) brings in a new server (ggml-org main with <code>--spec-type draft-mtp</code>) on a different quantization (Q4_K_M). If the symptom never recurs under that backend, the cause was likely either Q4_K_S quirks or a transient stimulus tied to cycle 17 specifically. If it does recur on the new backend, attention shifts to context-length sensitivity. No code-level fix is currently planned; the existing supervisor loop-detection + web-UI Stop are sufficient handling for a single observed occurrence.</p>
<p><strong>Where it lives:</strong> Any manual <code>docker cp</code> that constructs the destination path by hand. The required path includes the <code>python/</code> segment: <code>/a0/usr/agents/agent0/extensions/python/<hook>/</code>. The natural-feeling path that omits it (<code>/a0/usr/agents/agent0/extensions/<hook>/</code>) is invisible to A0's loader.</p>
27 install scripts in <code>scripts/</code> totaling 2420 lines. They are the canonical path for getting Exocortex components into a running container. Audit reveals: <strong>only 4 of 27 have a function abstraction</strong> for the install-file primitive; <strong>0 of 27 verify md5</strong> after deploy. The verification gap is the empirical cause of tonight's mis-deploy: every install script trusts <code>docker cp</code> to land the right bytes at the right path, and none of them re-read to confirm. Path discipline is correct (<code>EXT_DEST</code> in <code>install_idle_engine.sh</code> points to the proper <code>extensions/python/<hook>/</code> form); the bug was that I bypassed the install script entirely and did a manual <code>docker cp</code> to the wrong path.
3252
3297
</p>
3298
+
<pclass="section-summary">
3299
+
<strong>2026-05-16 update:</strong> the verification gap now has a runnable instrument. <code>scripts/audit_extensions.py</code> walks every extension hook on a live container, computes md5 across all four extension-host paths, and classifies each file (<em>OK</em>, <em>DUP same</em>, <em>DUP DIVERGENT</em>, <em>DUP CANONICAL DIVERGENT</em>, <em>WRONG-PATH DEAD</em>). First run on v16 found <strong>7 dead extensions</strong> (silently not loaded) and <strong>43 divergent copies</strong> (stale wrong-path duplicates behind canonical versions). v17 has its own drift (10 dead, 19 divergent). The tool is intended as the empirical pre-commit / post-deploy check the shared-library refactor (below) will codify; it can be run standalone today: <code>python3 scripts/audit_extensions.py exocortex_v16</code>.
3300
+
</p>
3253
3301
3254
3302
<h3>Inventory</h3>
3255
3303
<p>Each script targets one component or layer. Most have grown over time and inline the deploy primitives directly. Note the line-count distribution: a few large scripts (200+ lines) deploy multi-file features; many small scripts (20-60 lines) deploy single components.</p>
<h3>Q8 — <code>/api/api_message</code> request discarded on client connection close <spanclass="status warn">TRACKED — return to diagnose</span></h3>
3500
+
</div>
3501
+
<p><strong>Surfaced:</strong> 2026-05-17, STRESS_TEST_014 harness bring-up. A plain <code>curl -m <timeout></code> POST to <code>/api/api_message</code> that closes the socket at timeout (before A0 finishes the multi-minute turn) results in: the message lands, the full Exocortex before-main stack fires (MEM-ENHANCE → BST → REASON-INJ → PACE → KARP all logged), then processing orphans — no LLM completion, inference server idle, no error, HTTP 000 client-side, chat left with an incomplete/greeting message.</p>
3502
+
<p><strong>Likely root cause (not an agent bug):</strong> documented inline in <code>services/idle_watch.py _fire_fresh_cycle</code>: <em>"Raw-socket close caused Uvicorn to discard requests; keeping the connection alive ensures delivery."</em> The endpoint requires a long-lived connection held open for the entire turn (minutes). The idle engine works because it posts via <code>http.client</code> with <code>timeout=3600</code> in a daemon thread, waits only ~3s for a quick 4xx, then reads results <em>asynchronously from state files</em> — it never depends on the HTTP response body. Closing early = Uvicorn discards the in-flight request.</p>
3503
+
<p><strong>Why tracked, not fixed now:</strong> per Jake 2026-05-17, the STRESS_TEST_014 harness pivots to the proven idle-engine invocation pattern (long-lived connection + poll chat.json) so injection-chain validation isn't blocked. The api_message connection-lifetime behavior is a real, separate defect/gotcha in a documented endpoint and warrants its own diagnosis: is it Uvicorn config, an A0 request-handling path, or expected-but-undocumented? Related to but distinct from <code>memory/feedback_api_message_health_check.md</code> (don't use as readiness probe) and the prior ping-loop incident.</p>
3504
+
<p><strong>Next step when revisited:</strong> confirm the long-connection pattern delivers reliably (the harness pivot will be the first data point), then determine whether the early-close discard is fixable server-side or must be documented as a hard client contract.</p>
<li>Quantization mismatch — LM Studio reports <code>Qwen3.6-27B-Q4_K_S.gguf</code> loaded; Opus's letter referenced production using <code>Qwen3.6-27B-Q4_K_XL</code>. Different quantization may have different output stability properties.</li>
3460
3518
<li>Some interaction with the integrity-check Phase 0 specifically — cycle 17 was investigating a real status-mismatch finding in the wiki index and may have hit a tool/result pattern that primed degenerate output.</li>
3461
3519
</ul>
3462
-
<p><strong>Next investigation:</strong>Check the actual HTTP request body A0 sends to LM Studio. Verify<code>enable_thinking</code>field. If missing, that's a one-line fix in the LiteLLM config or A0's chat model adapter. Could also try forcing the Q4_K_XL model and rerunning the same cycle conditions.</p>
3463
-
<p><strong>Status:</strong>Not blocking overnight observation (v16 cycles since 17 have completed without reproducing it). Worth investigating in the morning when context is fresh.</p>
3520
+
<p><strong>2026-05-16 ~17:30 UTC update — Jake reversed hypothesis 1:</strong>per <code>enable_thinking_correction_20260516.md</code>, thinking is load-bearing for agentic capability (the <code><think></code> block is where the model reasons about tool selection, argument construction, multi-step planning). v16's<code>enable_thinking: true</code>is now <em>correct</em>. Hardcoded <code>"enable_thinking": False</code> was removed from <code>extensions/before_main_llm_call/_71_cache_warmer.py</code> and <code>inference/warm_cache.py</code>. Server-side <code>--reasoning off</code> remains in place — it suppresses only empty template-tag injection, not genuine reasoning. <strong>Hypothesis 1 is withdrawn.</strong> Remaining: context-length degradation, Q4_K_S quantization sensitivity.</p>
3521
+
<p><strong>Status:</strong>Demoted to RARE pending recurrence. Upstream MTP build (consolidated brief Priority 1) moves to Q4_K_M, which is the next opportunity to bisect quantization sensitivity.</p>
<p><strong>Surfaced:</strong> 2026-05-16 ~07:50 UTC while resolving Q2. Every cycle in <code>feed.jsonl</code> from 2026-05-16 (cycle 14 EXPLORE, cycles 18-21 BUILD) reports <code>"steps_used": 0</code> despite each cycle producing detailed productive output (wiki page deepened, memories saved, page marked STABLE). The May 14 entries show real step counts (cycle 30: 14 steps, cycle 31: 20 steps). Between then and now the step tracking stopped working.</p>
3471
-
<p><strong>What it doesn't break:</strong> Cycles still complete productively. Office UI still shows the right activity strings. Memory saves still happen. This is a metrics/reporting issue, not a behavioral one.</p>
3472
-
<p><strong>What it might affect:</strong><code>_select_cycle_type</code> doesn't use <code>steps_used</code> directly, but downstream metrics tracking does (cycle efficiency calculations, model evaluation against step budget, etc.). The longer this runs the more historical metrics get the wrong count.</p>
3473
-
<p><strong>Next investigation:</strong> Trace <code>steps_used</code> back through <code>cycle_close.py</code> to find where the value is being computed or passed. Likely a regression in a recent edit. Probably a 10-minute fix once the regression is located.</p>
3474
-
<p><strong>Status:</strong> Not blocking anything. Worth fixing for clean metrics going forward.</p>
3528
+
<p><strong>Surfaced:</strong> 2026-05-16 ~07:50 UTC while resolving Q2.</p>
3529
+
<p><strong>Root cause:</strong> The agent's activation prompt template (<code>prompts/idle_activation.md</code>) showed the agent the <code>cycle_close.py</code> invocation it must use to close a cycle. The example omitted <code>--steps-used</code>. <code>cycle_close.py</code> supports the flag (defaults to 0); the agent simply never knew it should pass it. So every cycle reported 0 regardless of actual work.</p>
3530
+
<p><strong>Fix:</strong> Added <code>--steps-used <N></code> to the cycle_close.py invocation block in <code>prompts/idle_activation.md</code>, plus a one-line instruction explaining that the agent should pass an approximate tool-call count. Deployed to both containers (md5 <code>4c59ddcceaea975baf99fda8dc6e20a6</code> on v16, v17, repo). No daemon restart needed — <code>_build_activation_prompt()</code> in <code>idle_watch.py</code> re-reads the template each fire.</p>
3531
+
<p><strong>Verification:</strong> Pending. Next BUILD/EXPLORE/MAINTAIN cycle's <code>feed.jsonl</code> entry should show a non-zero <code>steps_used</code> if the agent honors the new instruction. If 0 persists for the next 3+ cycles, the instruction needs to be more explicit.</p>
0 commit comments