Skip to content

Concurrent state writes drop jobs and delete live job/log files (state.json has no locking; prune reaps running jobs) #428

Description

@HoneyTyagii

Summary

The per-workspace state.json is shared by every plugin process (foreground commands, detached --background workers, and hooks), but saveState()/upsertJob() do an unsynchronized read-modify-write. Concurrent processes clobber each other's job entries, and because saveState() also deletes the job/log files of any job missing from the set it writes, a lost entry takes its on-disk artifacts with it. The same delete-on-write happens when the 50-job cap prunes a job that is still running.

Confirmed by reproduction on main (details below).

Affected code

plugins/codex/scripts/lib/state.mjs

export function saveState(cwd, state) {
  const previousJobs = loadState(cwd).jobs;   // read
  ...
  const nextJobs = pruneJobs(state.jobs ?? []); // keeps top-50 by updatedAt
  ...
  const retainedIds = new Set(nextJobs.map((job) => job.id));
  for (const job of previousJobs) {
    if (retainedIds.has(job.id)) continue;
    removeJobFile(resolveJobFile(cwd, job.id)); // <-- destructive
    removeFileIfExists(job.logFile);            // <-- destructive
  }
  fs.writeFileSync(...);                         // write
}

updateState() (and therefore upsertJob, setConfig) is loadState → mutate → saveState, with no lock or atomic swap.

Reproduction

Run against main (isolates state under a temp CLAUDE_PLUGIN_DATA so nothing real is touched):

import fs from "node:fs";
import os from "node:os";
import path from "node:path";

process.env.CLAUDE_PLUGIN_DATA = fs.mkdtempSync(path.join(os.tmpdir(), "cxc-repro-"));
const state = await import("./plugins/codex/scripts/lib/state.mjs");
const cwd = process.cwd();

// --- Concurrent read-modify-write clobber ---
const a = state.loadState(cwd);   // process A sees empty jobs
const b = state.loadState(cwd);   // process B sees empty jobs
a.jobs.push({ id: "job-A", updatedAt: new Date().toISOString() });
state.saveState(cwd, a);          // A persists job-A
b.jobs.push({ id: "job-B", updatedAt: new Date().toISOString() });
state.saveState(cwd, b);          // B, with a stale snapshot, clobbers A
console.log(state.listJobs(cwd).map((j) => j.id));                 // ['job-B']
console.log(fs.existsSync(state.resolveJobFile(cwd, "job-A")));     // false  <-- A's file deleted

// --- Prune deletes a live job's log past the 50 cap ---
const log = state.resolveJobLogFile(cwd, "job-running");
fs.writeFileSync(log, "live worker output\n");
state.upsertJob(cwd, { id: "job-running", status: "running", logFile: log, updatedAt: "2000-01-01T00:00:00.000Z" });
for (let i = 0; i < 55; i++) state.upsertJob(cwd, { id: `filler-${i}`, status: "completed", updatedAt: `2026-07-0${(i%9)+1}T00:0${i%6}:00.000Z` });
console.log(state.listJobs(cwd).some((j) => j.id === "job-running")); // false
console.log(fs.existsSync(log));                                      // false  <-- live log deleted

Observed output:

[ 'job-B' ]
job-A survived: false
job-A file still on disk: false

running job still tracked: false
running job's live log still exists: false

Impact

  • A --background task started while a foreground command (or hook) is also writing state can silently vanish from /codex:status, and its job/log files are deleted mid-run.
  • A long-running job that hasn't bumped updatedAt recently can be pruned past the 50-job cap while still running, deleting its live log out from under the worker.
  • Because prune ordering is purely updatedAt-recency, "still running" is not protected from deletion.

Environment

  • Plugin: openai-codex/codex 1.0.5
  • Node v25.1.0 (also reproduces logically on any version — no version-specific APIs involved)
  • Windows 11 (state layer is platform-agnostic; not Windows-specific)

Suggested direction (non-prescriptive)

  • Serialize state mutations with an atomic write (write temp file + rename) plus a lock file or a re-read-and-merge step inside saveState, so a stale in-memory snapshot can't drop another process's jobs.
  • Never delete artifacts for jobs whose status is queued/running during prune; only reap terminal jobs.
  • Consider merging previousJobs with the incoming set by id (last-writer-wins per job) rather than replacing the whole list.

Happy to open a PR if a direction is preferred.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions