Skip to content

Commit ac52fd7

Browse files
colbymchenryclaude
colbymchenry
and
claude
authored
Self-contained distribution: bundle Node + node:sqlite, drop better-sqlite3/wasm (closes colbymchenry#238) (colbymchenry#282)
* fix(db): eliminate concurrent-read "database is locked"; add node:sqlite backend (colbymchenry#238) WAL + busy_timeout were already enabled, so the issue's suggested fix was a no-op. The real causes, addressed here: - busy_timeout is now set first (before journal_mode) and lowered 120s -> 5s, so open-time pragmas wait out a lock instead of hanging for two minutes. - getCodeGraph no longer opens a second connection to the default project when a tool passes its own projectPath (the in-process lock amplifier). - The wasm fallback (no WAL) gets a bounded read-retry on SQLITE_BUSY. - New: node:sqlite backend, preferred over wasm, so installs whose native better-sqlite3 build fails land on a real-WAL backend instead of no-WAL wasm. - codegraph status / codegraph_status now report the effective journal mode, so a lock report is triageable (wal vs delete). - CLI hard-blocks Node < 20 to actually enforce the engines floor. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * refactor(db)!: node:sqlite is the sole backend; drop better-sqlite3 + wasm Now that distribution will bundle a Node 24 runtime, node:sqlite (real SQLite with WAL + FTS5) is always available. Collapse the three-backend adapter to node:sqlite only and remove the machinery the other two needed: - Remove better-sqlite3 (optionalDependency) and node-sqlite3-wasm (dependency). - Remove WasmDatabaseAdapter, the named->positional param translation, the SQLITE_BUSY read-retry, the wasm fallback banner, the backend env override, and the native/node-sqlite/wasm selection chain. - createDatabase now opens node:sqlite directly, with a clear error pointing at the bundled release / Node 22.5+ when the module is absent. - NodeSqliteAdapter.close() is idempotent and pragma() supports { simple }, to match the better-sqlite3 behavior callers relied on. - status (CLI + MCP) reports the single node:sqlite backend; journal-mode diagnostics and the getCodeGraph single-connection fix are retained. - Tests repointed off better-sqlite3 onto node:sqlite. Net -1044 lines. Running from source now requires Node 22.5+. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat(dist): self-contained bundle prototype (vendored Node + install channels) Phase 3 of the node:sqlite migration: ship a vendored Node runtime so CodeGraph runs with no system Node and no native build (node:sqlite is built in). - scripts/build-bundle.sh: build a per-platform archive (official Node + dist + prod deps + launcher). Same recipe per platform; pins Node v24.16.0. - install.sh: curl|sh installer (no Node required) — detects os/arch, pulls the archive from Releases, symlinks onto PATH; re-run to upgrade, --uninstall to remove. The VPS/SSH path. - scripts/npm-shim.js: thin launcher for the npm channel — resolves the per-platform optionalDependency bundle and execs it, so `npm i -g` keeps working and the real work runs on the bundled Node regardless of the user's. - BUNDLING.md: distribution design + release-pipeline TODO (CI matrix, platform packages, code signing, brew, retiring the Node-version gate). Validated end-to-end: darwin-arm64 and linux-x64 bundles both run init + index + status (Backend: node:sqlite, Journal: wal) + FTS query with NO system Node — linux-x64 verified in a clean ubuntu:24.04 amd64 container. Release archives are gitignored; CI will produce and upload them. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat(dist): add Windows PowerShell installer (install.ps1) The `irm … | iex` one-liner for Windows, mirroring install.sh: detect arch, pull the matching bundle from Releases, extract to %LOCALAPPDATA%\codegraph, add it to user PATH. Re-run to upgrade. (Windows bundle production in build-bundle.sh is still TODO.) Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat(dist): release workflow + npm packaging; README/CHANGELOG for bundled distro - .github/workflows/release.yml: manually-triggered (workflow_dispatch) release matrix. Builds a self-contained bundle per platform on its own runner (darwin-arm64/x64, linux-x64/arm64), publishes a GitHub Release with all archives, and publishes the npm thin-installer (shim + per-platform packages). Windows targets are TODO (build-bundle.sh is unix-only). - scripts/pack-npm.sh: assemble the npm packages from built bundles — per-platform packages tagged os/cpu + the main shim package with them as optionalDependencies (esbuild pattern). Proven locally: npm-install the tarballs, run via the shim, resolves the bundle and runs on the bundled Node 24 (node:sqlite / WAL). - README: install section now leads with the no-Node one-liners (curl|sh, irm|iex) then npm/npx; "bundled · none required" badge. - CHANGELOG: standout headline for the self-contained release, plus Added/Changed/ Removed for the install channels, node:sqlite backend, and dropped deps. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * feat(dist): Windows bundles + single-trigger release workflow - build-bundle.sh: add win32-x64 / win32-arm64 targets — download Node's Windows zip, bundle node.exe + a .cmd launcher, output a .zip. Verified structurally (PE32+ node.exe, CRLF .cmd, portable node_modules). Since there are no native addons, any target builds on any OS, so the whole matrix builds on one runner. - pack-npm.sh: handle .zip bundles and win32 packages (os: win32, node.exe). - release.yml: simplified to your spec — manual trigger reads the version from package.json, builds all platform bundles, creates the GitHub Release with notes pulled from CHANGELOG.md, and publishes the npm shim + platform packages. - BUNDLING.md: Windows + build-anywhere notes; release pipeline documented. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
1 parent b8aec39 commit ac52fd7

24 files changed

Lines changed: 1054 additions & 794 deletions

‎.github/workflows/release.yml‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
name: Release
2+
3+
# Manually triggered ("Run workflow"). On trigger it:
4+
# 1. reads the version from package.json,
5+
# 2. builds a self-contained bundle for every platform (one runner — there's no
6+
# native compilation, so cross-packaging is fine),
7+
# 3. creates the GitHub Release (tag v<version>) with all archives, using the
8+
# release notes from CHANGELOG.md,
9+
# 4. publishes the npm thin-installer (shim + per-platform packages).
10+
#
11+
# Before triggering: bump package.json and make sure CHANGELOG.md has the matching
12+
# section (## [<version>], or ## [Unreleased]). Set the NPM_TOKEN repo secret.
13+
on:
14+
workflow_dispatch: {}
15+
16+
permissions:
17+
contents: write # create the GitHub Release + tag
18+
19+
jobs:
20+
release:
21+
runs-on: ubuntu-latest
22+
steps:
23+
- uses: actions/checkout@v4
24+
- uses: actions/setup-node@v4
25+
with:
26+
node-version: 22
27+
registry-url: https://registry.npmjs.org
28+
- run: npm ci
29+
- name: Ensure zip/unzip
30+
run: sudo apt-get update -qq && sudo apt-get install -y -qq zip unzip
31+
32+
- name: Build all platform bundles
33+
run: |
34+
for t in darwin-arm64 darwin-x64 linux-x64 linux-arm64 win32-x64 win32-arm64; do
35+
bash scripts/build-bundle.sh "$t"
36+
done
37+
ls -lh release
38+
39+
- name: Resolve version
40+
id: ver
41+
run: echo "version=$(node -p "require('./package.json').version")" >> "$GITHUB_OUTPUT"
42+
43+
- name: Release notes from CHANGELOG.md
44+
run: |
45+
V="${{ steps.ver.outputs.version }}"
46+
node scripts/extract-release-notes.mjs "$V" > notes.md 2>/dev/null \
47+
|| node scripts/extract-release-notes.mjs Unreleased > notes.md 2>/dev/null || true
48+
if [ ! -s notes.md ]; then
49+
echo "::error::No release notes in CHANGELOG.md for [$V] or [Unreleased]."
50+
exit 1
51+
fi
52+
echo "----- release notes -----"; cat notes.md
53+
54+
- name: Create GitHub Release
55+
env:
56+
GH_TOKEN: ${{ github.token }}
57+
run: |
58+
gh release create "v${{ steps.ver.outputs.version }}" \
59+
release/codegraph-* \
60+
--title "v${{ steps.ver.outputs.version }}" \
61+
--notes-file notes.md
62+
63+
- name: Publish to npm
64+
env:
65+
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
66+
run: |
67+
bash scripts/pack-npm.sh "${{ steps.ver.outputs.version }}"
68+
# Platform packages first, then the main shim (which depends on them).
69+
for dir in release/npm/codegraph-* release/npm/main; do
70+
echo "publishing $dir"
71+
( cd "$dir" && npm publish --access public )
72+
done

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,3 +49,4 @@ test_frameworks
4949
test-languages/
5050

5151
nul
52+
release/

‎BUNDLING.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Distribution: self-contained bundles
2+
3+
CodeGraph ships a **vendored Node runtime** alongside the app. Because Node 22.5+
4+
has a built-in real SQLite (`node:sqlite`, with WAL + FTS5), bundling Node means:
5+
6+
- **No native build** — `better-sqlite3` is gone, so there are zero native addons
7+
to compile or rebuild.
8+
- **No wasm fallback** — and therefore no more `database is locked` (issue #238).
9+
- **No Node-version dependence** — the app always runs on the bundled Node,
10+
whatever the user has (or doesn't have) installed.
11+
12+
## What's in a bundle
13+
14+
Built by [`scripts/build-bundle.sh`](scripts/build-bundle.sh) — one archive per
15+
platform, identical recipe (only the Node download differs):
16+
17+
```
18+
codegraph-<target>/
19+
node | node.exe # official Node runtime for <target>
20+
lib/
21+
dist/ # compiled app (+ tree-sitter .wasm grammars, schema.sql)
22+
node_modules/ # production deps only (pure JS / wasm — portable)
23+
bin/
24+
codegraph | codegraph.cmd # launcher → runs the bundled Node with the app
25+
```
26+
27+
Targets: `darwin-arm64`, `darwin-x64`, `linux-x64`, `linux-arm64`, `win32-x64`,
28+
`win32-arm64`. Unix targets produce `.tar.gz` (shell launcher); Windows produces
29+
`.zip` (`node.exe` + a `.cmd` launcher).
30+
31+
```bash
32+
scripts/build-bundle.sh linux-x64 # -> release/codegraph-linux-x64.tar.gz
33+
scripts/build-bundle.sh win32-x64 # -> release/codegraph-win32-x64.zip
34+
```
35+
36+
Because dropping better-sqlite3 left **zero native addons**, building a bundle is
37+
pure file-packaging — **any** target builds on **any** OS (the whole matrix builds
38+
on one Linux runner). Cross-compilation isn't a concern; only *run-testing* a
39+
bundle needs the target platform (or emulation, e.g. `docker run --platform
40+
linux/amd64`).
41+
42+
## Install channels (all deliver the same bundle)
43+
44+
1. **`curl | sh`** ([`install.sh`](install.sh)) — no Node required; ideal for a
45+
fresh Linux VPS over SSH. Detects os/arch, pulls the archive from GitHub
46+
Releases, symlinks `codegraph` onto PATH. Re-run to upgrade; `--uninstall` to
47+
remove.
48+
2. **npm** ([`scripts/npm-shim.js`](scripts/npm-shim.js)) — preserves
49+
`npm i -g @colbymchenry/codegraph`. The main package is a tiny shim; the
50+
bundles ship as per-platform `optionalDependencies`
51+
(`@colbymchenry/codegraph-<target>` with `os`/`cpu`), so npm installs only the
52+
matching one. The shim — run by the user's Node — execs the bundle, so the
53+
real work runs on the bundled Node 24. Works even on old Node.
54+
3. **Windows** ([`install.ps1`](install.ps1)) — `irm … | iex`; same flow as
55+
install.sh (detect arch, pull the `.zip` from Releases, add to PATH).
56+
4. **Homebrew / Scoop** — TODO (tap + cask pointing at the Release archives).
57+
58+
## Release pipeline
59+
60+
[`.github/workflows/release.yml`](.github/workflows/release.yml) — manually
61+
triggered. Reads the version from `package.json`, builds every platform bundle on
62+
one runner, creates the GitHub Release (notes from `CHANGELOG.md`), and publishes
63+
the npm shim + per-platform packages. Requires the `NPM_TOKEN` repo secret.
64+
65+
Still TODO:
66+
- **Code signing** — the main gap for "download & run": macOS Gatekeeper needs a
67+
Developer ID + notarization; Windows needs Authenticode. Homebrew softens the
68+
macOS case (handles quarantine).
69+
- Retire the now-vestigial Node-version gate in `src/bin/codegraph.ts` — the
70+
bundle always runs Node 24, and the npm shim does no tree-sitter work.
71+
- Re-wire `npm uninstall` cleanup (the agent-config `preuninstall`) through the
72+
shim — the generated main package doesn't carry it.

‎CHANGELOG.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,37 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
99

1010
## [Unreleased]
1111

12+
### 🎉 Self-contained: CodeGraph bundles its own runtime — install anywhere, on any Node (or none)
13+
14+
**No more `database is locked`. No more native build failures. No more "WASM fallback active."**
15+
16+
CodeGraph used to need `better-sqlite3`, a native module compiled against your exact
17+
Node version. When that build failed (common on Windows and locked-down machines) it
18+
silently dropped to a slow WASM SQLite build with **no WAL** — the root cause of the
19+
intermittent `database is locked` errors on concurrent MCP tool calls
20+
([#238](https://github.com/colbymchenry/codegraph/issues/238)). That entire class of
21+
problem is **gone**: CodeGraph now ships a self-contained Node runtime and uses Node's
22+
built-in `node:sqlite` (real SQLite, full WAL + FTS5).
23+
24+
- ✅ **Zero native compilation** — nothing to build, ever; nothing to rebuild when Node changes.
25+
- ✅ **Runs on any Node version — or with no Node at all.** Install via the standalone installers with no Node present, or keep using `npm`/`npx` on any version (your Node only launches the bundled runtime).
26+
- ✅ **`database is locked` fixed at the root** — real WAL means readers never block on a writer.
27+
- ⚡ **5–10× faster** than the old WASM fallback for anyone who was stuck on it.
28+
29+
```bash
30+
# macOS / Linux — no Node required
31+
curl -fsSL https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.sh | sh
32+
# Windows (PowerShell) — no Node required
33+
irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex
34+
# or, if you have Node (any version):
35+
npm i -g @colbymchenry/codegraph
36+
```
37+
1238
### Added
39+
- **Standalone installers** — one-line install with no Node.js required:
40+
`curl -fsSL .../install.sh | sh` (macOS/Linux) and `irm .../install.ps1 | iex`
41+
(Windows). They fetch the matching self-contained bundle from GitHub Releases
42+
and put `codegraph` on your PATH.
1343
- **Lua**: CodeGraph now indexes Lua (`.lua`) — functions, methods (table `t.f`
1444
and `t:m` definitions become methods with a `t::f` receiver-qualified name),
1545
local variables, `require(...)` imports, and the call edges between them.
@@ -21,6 +51,23 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
2151
signatures, generics, and Roblox instance-path `require(script.Parent.X)`
2252
imports.
2353

54+
### Changed
55+
- **SQLite backend is now Node's built-in `node:sqlite`** (real SQLite, WAL +
56+
FTS5), shipped inside a bundled Node runtime. This fixes the concurrent-read
57+
`database is locked` errors ([#238](https://github.com/colbymchenry/codegraph/issues/238))
58+
at the root and removes the native build step entirely.
59+
- **`npm i -g` / `npx` now install a self-contained bundle.** The main package is
60+
a tiny shim; the runtime ships as per-platform `optionalDependencies`, so the
61+
install works on any Node version (your Node only launches the bundle).
62+
- **`codegraph status`** now reports the effective journal mode (`wal` vs not),
63+
so a `database is locked` report is triageable at a glance.
64+
65+
### Removed
66+
- **`better-sqlite3`** (optional native dependency) and **`node-sqlite3-wasm`**
67+
(WASM fallback) — along with the native-build banner, the WASM fallback path,
68+
and the no-WAL lock retries they required. The dependency tree now has zero
69+
native addons.
70+
2471
### Fixed
2572
- **Installer**: re-running `codegraph install` now removes the broken
2673
auto-sync hooks that pre-0.8 versions wrote to Claude Code's

‎README.md‎

Lines changed: 20 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88

99
[![npm version](https://img.shields.io/npm/v/@colbymchenry/codegraph.svg)](https://www.npmjs.com/package/@colbymchenry/codegraph)
1010
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
11-
[![Node.js](https://img.shields.io/badge/Node.js-20--24-green.svg)](https://nodejs.org/)
11+
[![Self-contained](https://img.shields.io/badge/Node.js-bundled%20%C2%B7%20none%20required-brightgreen.svg)](https://nodejs.org/)
1212

1313
[![Windows](https://img.shields.io/badge/Windows-supported-blue.svg)](#)
1414
[![macOS](https://img.shields.io/badge/macOS-supported-blue.svg)](#)
@@ -23,11 +23,24 @@
2323

2424
### Get Started
2525

26+
**No Node.js required** — one command grabs the right build for your OS:
27+
2628
```bash
27-
npx @colbymchenry/codegraph
29+
# macOS / Linux
30+
curl -fsSL https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.sh | sh
31+
32+
# Windows (PowerShell)
33+
irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex
34+
```
35+
36+
Already have Node? Use npm instead (works on any version):
37+
38+
```bash
39+
npx @colbymchenry/codegraph # zero-install, or:
40+
npm i -g @colbymchenry/codegraph
2841
```
2942

30-
<sub>Interactive installer auto-configures your agent(s) — Claude Code, Cursor, Codex CLI, opencode</sub>
43+
<sub>CodeGraph bundles its own runtime — nothing to compile, no native build, works the same everywhere.<br />The interactive installer auto-configures your agent(s) — Claude Code, Cursor, Codex CLI, opencode.</sub>
3144

3245
#### Initialize Projects
3346

@@ -456,10 +469,10 @@ The `.codegraph/config.json` file controls indexing:
456469

457470
**Indexing is slow** — Check that `node_modules` and other large directories are excluded. Use `--quiet` to reduce output overhead.
458471

459-
**Indexing is slow / MCP `database is locked` / WASM fallback active** — `codegraph` ships with a WASM SQLite fallback for environments where `better-sqlite3` (a native module, declared as `optionalDependencies`) can't install. The fallback is 5-10x slower than the native backend and uses a journal mode that lets writers block readers, so MCP queries can also hit `database is locked` while indexing runs. Run `codegraph status` and look at the `Backend:` line:
472+
**Indexing is slow, or MCP hits `database is locked`** — both trace to the SQLite backend. `codegraph` picks the best available, in order: native `better-sqlite3` (fastest; an `optionalDependencies` native module), then Node's built-in `node:sqlite` (Node ≥ 22.5), then a bundled WASM build. Run `codegraph status` and read the **`Backend:`** and **`Journal:`** lines:
460473

461-
- `Backend: native` — you're on the fast path, nothing to do.
462-
- `Backend: wasm` — you're on the slow fallback. Common causes: missing C build tools, prebuilt binary unavailable for your Node version, or your Node version changed after install. Fix:
474+
- `Backend: native` or `node:sqlite` with `Journal: wal` — fast path with lock-free concurrent reads; nothing to do.
475+
- `Backend: wasm` — the native module didn't load *and* `node:sqlite` is unavailable (Node < 22.5). WASM is 5-10x slower and has no WAL, so heavy concurrent use can briefly hit `database is locked`. The simplest fix is Node ≥ 22.5 (you get `node:sqlite` automatically); otherwise restore the native backend:
463476

464477
```bash
465478
# macOS
@@ -479,6 +492,7 @@ The `.codegraph/config.json` file controls indexing:
479492
```
480493

481494
After the fix, `codegraph status` should show `Backend: native`.
495+
- `Journal:` shows anything other than `wal` on a `native` / `node:sqlite` backend — WAL couldn't be enabled on this filesystem (common on network shares and WSL2 `/mnt`), so reads can block on writes. Move the project (with its `.codegraph/` folder) onto a local disk.
482496

483497
**MCP server not connecting** — Ensure the project is initialized/indexed, verify the path in your MCP config, and check that `codegraph serve --mcp` works from the command line.
484498

0 commit comments

Comments
 (0)