Skip to content

Commit a2c2d16

Browse files
thomashoneymanf-ffsoikin
authored
Require all packages to solve / compile and include all valid compilers in their metadata (#669)
Co-authored-by: Fabrizio Ferrai <[email protected]> Co-authored-by: Fyodor Soikin <[email protected]>
1 parent d189d0d commit a2c2d16

File tree

145 files changed

+11997
-3634
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

145 files changed

+11997
-3634
lines changed

‎.env.example‎

Lines changed: 30 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,38 +1,44 @@
1-
# =====
2-
# Dev Configuration
3-
# The devShell reads this file to set defaults, so changing values here
4-
# affects local development.
5-
# =====
1+
# -----------------------------------------------------------------------------
2+
# Server Configuration (dev defaults, required in all environments)
3+
# -----------------------------------------------------------------------------
64

7-
# Server port - used by both the server and E2E tests
5+
# Port the registry server listens on
6+
# - Dev/Test: 9000 (from this file)
7+
# - Prod: Set in deployment config
88
SERVER_PORT=9000
99

1010
# SQLite database path (relative to working directory)
11+
# - Dev: Uses local ./db directory
12+
# - Test: Overridden to use temp state directory
13+
# - Prod: Set to production database path
1114
DATABASE_URL="sqlite:db/registry.sqlite3"
1215

13-
# =====
14-
# Dev Secrets
15-
# these must be set in .env when running scripts like legacy-importer
16-
# =====
16+
# -----------------------------------------------------------------------------
17+
# Secrets (required for production, use dummy values for local dev)
18+
# -----------------------------------------------------------------------------
19+
# IMPORTANT: Never commit real secrets. The values below are dummies for testing.
1720

18-
# GitHub personal access token for API requests when running scripts
19-
GITHUB_TOKEN="ghp_your_personal_access_token"
20-
21-
# =====
22-
# Prod Secrets
23-
# these must be set in .env to run the production server and some scripts
24-
# =====
25-
26-
# DigitalOcean Spaces credentials for S3-compatible storage
27-
SPACES_KEY="digitalocean_spaces_key"
28-
SPACES_SECRET="digitalocean_spaces_secret"
29-
30-
# Pacchettibotti bot account credentials
31-
# Used for automated registry operations (commits, releases, etc.)
21+
# GitHub personal access token for pacchettibotti bot
22+
# Used for: commits to registry repos, issue management
3223
PACCHETTIBOTTI_TOKEN="ghp_pacchettibotti_token"
3324

3425
# Pacchettibotti SSH keys (base64-encoded)
26+
# Used for: signing authenticated operations (unpublish, transfer)
3527
# Generate with: ssh-keygen -t ed25519 -C "[email protected]"
3628
# Encode with: cat key | base64 | tr -d '\n'
3729
PACCHETTIBOTTI_ED25519_PUB="c3NoLWVkMjU1MTkgYWJjeHl6IHBhY2NoZXR0aWJvdHRpQHB1cmVzY3JpcHQub3Jn"
3830
PACCHETTIBOTTI_ED25519="YWJjeHl6"
31+
32+
# DigitalOcean Spaces credentials for S3-compatible storage
33+
# Used for: uploading/downloading package tarballs
34+
SPACES_KEY="digitalocean_spaces_key"
35+
SPACES_SECRET="digitalocean_spaces_secret"
36+
37+
38+
# -----------------------------------------------------------------------------
39+
# Script-only Secrets (not used by server, used by scripts like legacy-importer)
40+
# -----------------------------------------------------------------------------
41+
42+
# Personal GitHub token for API requests when running scripts
43+
# This is YOUR token, not pacchettibotti's
44+
GITHUB_TOKEN="ghp_your_personal_access_token"

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
/output-es
44
/scratch
55
/.vscode
6+
/scripts/analysis
67

78
result*
89

@@ -15,6 +16,8 @@ result*
1516
*.sqlite3
1617
*.sqlite3-wal
1718
*.sqlite3-shm
19+
20+
TODO.md
1821
.spec-results
1922

2023
# Keep it secret, keep it safe.

‎AGENTS.md‎

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
# AGENTS.md
2+
3+
The PureScript Registry implements a package registry for PureScript.
4+
5+
- @SPEC.md contains the registry specification. Use this to understand the core data types and operations of the registry.
6+
- @CONTRIBUTING.md describes the structure of the registry codebase, what the various packages in the monorepo represent, and how to build and test the registry code.
7+
8+
## Development Environment
9+
10+
We use Nix with direnv. Expect to be in a Nix shell automatically, but if you are not, you can enter one:
11+
12+
```sh
13+
nix develop
14+
```
15+
16+
### Nix Quirks
17+
18+
- If Nix tries to fetch from git during a build and fails, then most likely `spago.yaml` files have been changed but the lockfiles were not updated. Update them with `spago build`.
19+
- If a Nix build appears to be stale, then most likely files were modified but not tracked by Git. Nix flakes only consider Git-tracked files. Add modified files with `git add` and retry.
20+
21+
### Build & Test Commands
22+
23+
The registry is implemented in PureScript. Use spago to build it.
24+
25+
```sh
26+
spago build # Build all PureScript code
27+
```
28+
29+
The registry contains unit tests, end-to-end tests, and nix flake checks.
30+
31+
- Run unit tests when you complete a change with `spago test` or `spago test -p <package-name>`.
32+
- Run end-to-end tests when working on the registry server in `app`.
33+
- On Linux systems you can run all flake checks (the tests run in CI) using `nix flake check -L`.
34+
35+
### End-to-End Tests
36+
37+
The end-to-end (integration) tests are in `app-e2e`. They can be run via Nix on Linux:
38+
39+
```sh
40+
nix build .#checks.x86_64-linux.integration
41+
```
42+
43+
Alternately, they can be run on macOS or for more iterative development of tests using two terminals: one to start the test env, and one to execute the tests.
44+
45+
```sh
46+
# Terminal 1: Start test environment (wiremock mocks + registry server on port 9000)
47+
nix run .#test-env
48+
49+
# Terminal 2: Run E2E tests once server is ready
50+
spago-test-e2e
51+
```
52+
53+
Options: `nix run .#test-env -- --tui` for interactive TUI, `-- --detached` for background mode to use a single terminal.
54+
55+
State is stored in `/tmp/registry-test-env` and cleaned up on each `nix run .#test-env`. To examine state after a test run (for debugging), stop the test-env but don't restart it. This is useful, for example, to read the logs of the most recent run. For example:
56+
57+
```sh
58+
# after a test run, see the logs (log name is today's date)
59+
cat /tmp/registry-test-env/scratch/logs/*.log
60+
```
61+
62+
#### Smoke Test (Linux only)
63+
64+
The smoke test verifies that the server comes up properly and tests deployment. Only run this test if you are making changes which could break the deployment of the server.
65+
66+
```sh
67+
nix build .#checks.x86_64-linux.smoke -L
68+
```
69+
70+
## Formatting
71+
72+
```sh
73+
# Format PureScript
74+
purs-tidy format-in-place app app-e2e foreign lib scripts
75+
purs-tidy check app app-e2e foreign lib scripts
76+
77+
# Format Nix files
78+
nixfmt *.nix nix/**/*.nix
79+
```
80+
81+
## Project Structure
82+
83+
- `app/` — Registry server implementation.
84+
- `app-e2e/` — E2E tests for the server API.
85+
- `lib/` — **Public library** for consumers (Spago, Pursuit, etc.). Only types and functions useful to external tools belong here. Avoid implementation-specific code.
86+
- `foreign/` — FFI bindings to JavaScript libraries.
87+
- `scripts/` — Runnable modules for registry tasks (LegacyImporter, PackageTransferrer, PackageSetUpdater, etc.). Run via `nix run .#legacy-importer`, etc.
88+
- `test-utils/` — Shared test utilities.
89+
- `db/` — SQLite schemas and migrations (use `dbmate up` to initialize).
90+
- `types/` — Dhall type specifications.
91+
- `nix/` — Nix build and deployment configuration.
92+
93+
## Scratch Directory & Caching
94+
95+
The `scratch/` directory (gitignored) is used by scripts for:
96+
- `.cache/` — Cached API responses, downloaded packages, etc.
97+
- `logs/` — Log files
98+
- `registry/`, `registry-index/` — Local clones for testing, also modified and optionally committed to by scripts
99+
100+
Caching is critical for the legacy importer due to the expense of downloading packages. The `Registry.App.Effect.Cache` module handles caching.
101+
102+
## PureScript Conventions
103+
104+
### Custom Prelude
105+
106+
Always use `Registry.App.Prelude` in `app/` and `app-e2e/` directories:
107+
108+
```purescript
109+
import Registry.App.Prelude
110+
```
111+
112+
### Effects via Run
113+
114+
Use the `run` library for extensible effects. Do NOT perform HTTP calls, console logs, or other effects directly in `Aff`. Check for existing effects in `app/src/App/Effect/` or consider adding one.
115+
116+
### Import Style
117+
118+
Import types unqualified, values qualified. Use shortened module names:
119+
120+
```purescript
121+
import Registry.App.Prelude
122+
123+
import Data.Array as Array
124+
import Data.String as String
125+
import Node.FS.Aff as FS.Aff
126+
import Parsing (Parser)
127+
import Parsing as Parsing
128+
import Parsing.Combinators as Parsing.Combinators
129+
import Registry.Operation (AuthenticatedData)
130+
import Registry.SSH as SSH
131+
```
132+
133+
### Syntax
134+
135+
Never use `let/in` syntax unless in an `ado` block. Always `do/let`.
136+
137+
```purs
138+
func =
139+
-- NEVER use let/in syntax
140+
let x = 1
141+
in x + x
142+
143+
func = do
144+
-- ALWAYS use do/let syntax
145+
let x = 1
146+
x + x
147+
```

‎CLAUDE.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
AGENTS.md

‎CONTRIBUTING.md‎

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -72,20 +72,29 @@ nix build .#checks.x86_64-linux.smoke -L
7272

7373
### Integration Test
7474

75+
You can run the integration tests with the following on Linux:
76+
77+
```sh
78+
nix build .#checks.x86_64-linux.integration -L
79+
```
80+
81+
On macOS or for iterative development, you can instead start the test environment and run the tests separately.
82+
7583
```sh
7684
# Terminal 1: Start the test environment (wiremock mocks + registry server)
7785
nix run .#test-env
7886

79-
# Terminal 2: Once the server is ready, run the E2E tests
80-
spago run -p registry-app-e2e
87+
# Terminal 2: Run E2E tests once server is ready
88+
spago-test-e2e
8189
```
8290

8391
The test environment:
8492
- Starts wiremock services mocking GitHub, S3, Pursuit, etc.
85-
- Starts the registry server on port 9000 with a temporary SQLite database
93+
- Starts the registry server with a temporary SQLite database
8694
- Uses fixture data from `app/fixtures/`
95+
- State is stored in `/tmp/registry-test-env` and cleaned up on each `nix run .#test-env`
8796

88-
Press `Ctrl+C` in Terminal 1 to stop all services. State is cleaned up automatically.
97+
Press `Ctrl+C` in Terminal 1 to stop all services.
8998

9099
All arguments after `--` are passed directly to process-compose:
91100

@@ -101,7 +110,11 @@ process-compose attach # Attach TUI
101110
process-compose down # Stop all services
102111
```
103112

104-
You can also set `STATE_DIR` to use a persistent state directory instead of a temp dir.
113+
To examine state after a test run (e.g., for debugging), stop the test-env but don't restart it. The state remains in `/tmp/registry-test-env`:
114+
- `db/registry.sqlite3` — SQLite database
115+
- `scratch/registry/` — Local registry clone with metadata
116+
- `scratch/registry-index/` — Local manifest index clone
117+
- `repo-fixtures/` — Git fixture repositories
105118

106119
## Available Nix Commands
107120

‎SPEC.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -387,6 +387,7 @@ All packages in the registry contain a `purs.json` manifest file in their root d
387387
- `version`: a valid [`Version`](#version)
388388
- `license`: a valid [`License`](#license)
389389
- `location`: a valid [`Location`](#location)
390+
- `ref`: a `string` representing the reference (e.g., a Git commit or Git tag) at the `location` that was used to fetch this version's source code
390391
- `owners` (optional): a non-empty array of [`Owner`](#owner)
391392
- `description` (optional): a description of your library as a plain text string, not markdown, up to 300 characters
392393
- `includeFiles` (optional): a non-empty array of globs, where globs are used to match file paths (in addition to the `src` directory and other [always-included files](#always-included-files)) that you want included in your package tarball
@@ -397,7 +398,7 @@ Note:
397398

398399
- Globs you provide at the `includeFiles` and `excludeFiles` keys must contain only `*`, `**`, `/`, `.`, `..`, and characters for Linux file paths. It is not possible to negate a glob (ie. the `!` character), and globs cannot represent a path out of the package source directory.
399400
- When packaging your project source, the registry will first "include" your `src` directory and always-included files such as your `purs.json` file. Then it will include files which match globs indicated by the `includeFiles` key ([always-ignored files](#always-ignored-files) cannot be included). Finally, it will apply the excluding globs indicated by the `excludeFiles` key to the included files ([always-included files](#always-included-files) cannot be excluded).
400-
- Dependencies you provide at the `dependencies` key must exist in the registry, and the dependency ranges must be solvable (ie. it must be possible to produce a single version of each dependency that satisfies the provided version bounds, including any transitive dependencies).
401+
- Dependencies you provide at the `dependencies` key must exist in the registry, the dependency ranges must be solvable (ie. it must be possible to produce a single version of each dependency that satisfies the provided version bounds, including any transitive dependencies), and transitive dependencies are not allowed (ie. any modules you import in your code must come from packages listed in your dependencies).
401402

402403
For example:
403404

@@ -411,6 +412,7 @@ For example:
411412
"githubOwner": "purescript",
412413
"githubRepo": "purescript-control"
413414
},
415+
"ref": "v4.2.0",
414416
"include": ["test/**/*.purs"],
415417
"exclude": ["test/graphs"],
416418
"dependencies": { "newtype": ">=3.0.0 <4.0.0", "prelude": ">=4.0.0 <5.0.0" }
@@ -424,11 +426,12 @@ For example:
424426

425427
All packages in the registry have an associated metadata file, which is located in the `metadata` directory of the `registry` repository under the package name. For example, the metadata for the `aff` package is located at: https://github.com/purescript/registry/blob/main/metadata/aff.json. Metadata files are the source of truth on all published and unpublished versions for a particular package for what there content is and where the package is located. Metadata files are produced by the registry, not by package authors, though they take some information from package manifests.
426428

427-
Each published version of a package records three fields:
429+
Each published version of a package records the following fields:
428430

429431
- `hash`: a [`Sha256`](#Sha256) of the compressed archive fetched by the registry for the given version
430432
- `bytes`: the size of the tarball in bytes
431433
- `publishedTime`: the time the package was published as an `ISO8601` string
434+
- `compilers`: compiler versions this package is known to work with. This field can be in one of two states: a single version indicates that the package worked with a specific compiler on upload but has not yet been tested with all compilers, whereas a non-empty array of versions indicates the package has been tested with all compilers the registry supports.
432435

433436
Each unpublished version of a package records three fields:
434437

‎app-e2e/spago.yaml‎

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,16 +5,29 @@ package:
55
dependencies:
66
- aff
77
- arrays
8+
- codec-json
89
- console
910
- datetime
10-
- effect
11-
- either
12-
- maybe
13-
- prelude
11+
- exceptions
12+
- fetch
13+
- integers
14+
- json
15+
- node-child-process
16+
- node-execa
17+
- node-fs
18+
- node-path
19+
- node-process
20+
- ordered-collections
21+
- registry-app
22+
- registry-foreign
1423
- registry-lib
24+
- registry-scripts
1525
- registry-test-utils
26+
- routing-duplex
27+
- run
1628
- spec
1729
- spec-node
1830
- strings
31+
- transformers
1932
run:
2033
main: Test.E2E.Main

0 commit comments

Comments
 (0)