Skip to content

Commit 3a5a069

Browse files
filipdaniskofilipdanisko
committed
Add configurable validation reports
1 parent 6317ed2 commit 3a5a069

12 files changed

Lines changed: 693 additions & 44 deletions

File tree

‎README.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -198,6 +198,9 @@ changes.
198198
| `openknowledge spec 0.1` | Print a specific embedded spec version. |
199199
| `openknowledge validate [key-or-path]` | Validate a bundle against the latest spec. |
200200
| `openknowledge validate --spec 0.1 [key-or-path]` | Validate against a specific spec version. |
201+
| `openknowledge validate --format json [key-or-path]` | Print a machine-readable validation report. |
202+
| `openknowledge validate --format json --out <file> [key-or-path]` | Write a machine-readable validation report to a file. |
203+
| `openknowledge validate --rule <rule=off\|warn\|error> [key-or-path]` | Override one validation rule severity for the run. |
201204
| `openknowledge list [key-or-path]` | Print a bundle tree with inline validation issues. |
202205
| `openknowledge list --spec 0.1 [key-or-path]` | List while validating against a specific spec version. |
203206
| `openknowledge list --json [key-or-path]` | Print machine-readable inventory output. |
@@ -223,6 +226,12 @@ It does not fail on optional fields, unknown concept types, unknown frontmatter
223226
keys, broken local links, non-blocking Markdown syntax warnings, or missing
224227
index files.
225228

229+
For CI and editor integrations, `openknowledge validate --format json` emits a
230+
machine-readable report with summary counts, checks, active severity policy, and
231+
combined/separate issue arrays. Bundle-local `openknowledge.toml` can configure
232+
lint severities under `[validation.rules]`, and repeatable `--rule` flags can
233+
override them per run.
234+
226235

227236
## License and attribution
228237

‎Wiki/changelog/cli.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,24 @@ cut.
2323
Candidate V4 changes are commits after the `v0.3.0` release tag. Keep new
2424
entries in this section until they are released.
2525

26+
### 2026-07-03 - Validation JSON reports and rule severities
27+
28+
* `openknowledge validate` now supports `--format json`, `--json`, and
29+
`--format json --out <file>` for machine-readable validation reports with
30+
summary counts, policy metadata, check statuses, combined issues, errors, and
31+
warnings.
32+
* Validation rule severities can be configured with `[validation.rules]` in
33+
`openknowledge.toml` and overridden per run with repeatable
34+
`--rule rule=off|warn|error` flags.
35+
* Default validation behavior remains unchanged when no severity overrides are
36+
configured.
37+
* Source anchors: `packages/cli/internal/okf/validation_policy.go`,
38+
`packages/cli/internal/okf/validation_types.go`,
39+
`packages/cli/internal/okf/ast_validate.go`,
40+
`packages/cli/cmd/openknowledge/main.go`.
41+
* Docs updated: `README.md`, `Wiki/features/commands/validate.md`,
42+
`Wiki/features/commands/index.md`, `Wiki/changelog/cli.md`.
43+
2644
### 2026-07-03 - LLM wiki landing and README positioning
2745

2846
* Reworded the landing page title, heading, metadata, prompt copy, and product

‎Wiki/features/commands/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ anchors. Keep this index as the quick map.
1212

1313
## Validate Bundles
1414

15-
* [Validate](validate.md) - validate a bundle against OKF.
15+
* [Validate](validate.md) - validate a bundle against OKF with configurable rule severities and optional JSON reports.
1616

1717
## Connect And Resolve Bundles
1818

‎Wiki/features/commands/validate.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,9 @@ that cannot be parsed is an error.
1919
```sh
2020
openknowledge validate [key-or-path]
2121
openknowledge validate --spec <version> [key-or-path]
22+
openknowledge validate --format json [key-or-path]
23+
openknowledge validate --format json --out <file> [key-or-path]
24+
openknowledge validate --rule <rule=off|warn|error> [key-or-path]
2225
openknowledge validate --quiet [key-or-path]
2326
openknowledge validate --help
2427
```
@@ -29,6 +32,10 @@ openknowledge validate --help
2932
| --- | --- | --- |
3033
| `key-or-path` | argument | Registry key or knowledge base root. Defaults to the current directory. |
3134
| `--spec` | flag | OKF spec version. Defaults to latest. |
35+
| `--format` | flag | Output format: `text` or `json`. Defaults to `text`. |
36+
| `--json` | flag | Alias for `--format json`. |
37+
| `--out` | flag | Write the JSON validation report to a file. Requires JSON output. |
38+
| `--rule` | flag | Override one validation rule severity as `rule=off`, `rule=warn`, or `rule=error`. May be repeated. |
3239
| `--quiet` | flag | Print only errors and exit with status. |
3340

3441
## Validation Checks
@@ -58,6 +65,65 @@ The report currently includes these checks:
5865
Errors make `openknowledge validate` exit with status `1`. Warnings are printed
5966
but still exit with status `0`.
6067

68+
## Rule Severity Configuration
69+
70+
Default validation preserves the OKF v0.1 conformance behavior described above:
71+
hard rules are errors, and non-blocking hygiene checks are warnings. For lint
72+
workflows that need stricter or looser behavior, bundles can configure rule
73+
severities in `openknowledge.toml`:
74+
75+
```toml
76+
[validation.rules]
77+
link-target = "error"
78+
markdown-syntax = "off"
79+
frontmatter-format = "warn"
80+
```
81+
82+
The same rules can be overridden for a single run:
83+
84+
```sh
85+
openknowledge validate --rule link-target=error --rule markdown-syntax=off Wiki
86+
```
87+
88+
Supported severities are `off`, `warn`, and `error`. CLI `--rule` values
89+
override `openknowledge.toml`. Unknown rule names or severities fail with a
90+
usage error so typos do not silently weaken validation.
91+
92+
Current rule names are:
93+
94+
| Rule | Default | Covered behavior |
95+
| --- | --- | --- |
96+
| `bundle-read` | error | A Markdown file could not be read. |
97+
| `utf-8` | error | Markdown content is not valid UTF-8. |
98+
| `frontmatter` | error | Frontmatter starts but cannot be parsed. |
99+
| `concept-frontmatter` | error | A concept document is missing YAML frontmatter. |
100+
| `concept-type` | error | A concept document has an empty or missing `type`. |
101+
| `index-frontmatter` | error | A non-root `index.md` uses disallowed frontmatter. |
102+
| `log-frontmatter` | error | `log.md` uses concept frontmatter. |
103+
| `log-date` | error | A `log.md` `##` heading is not `YYYY-MM-DD`. |
104+
| `frontmatter-format` | warning | Frontmatter is parseable but not cleanly formatted. |
105+
| `markdown-syntax` | warning | Markdown body syntax looks malformed. |
106+
| `okf-version` | warning | Root `okf_version` differs from the selected spec. |
107+
| `link-target` | warning | A local Markdown link is missing or escapes the root. |
108+
109+
## JSON Reports
110+
111+
`--format json` prints the full machine-readable validation report to stdout.
112+
`--format json --out <file>` writes the same report to disk:
113+
114+
```sh
115+
openknowledge validate --format json --out okf-report.json Wiki
116+
```
117+
118+
The JSON report includes:
119+
120+
* bundle counts and selected spec version
121+
* `summary.status`, `errorCount`, `warningCount`, and `issueCount`
122+
* active policy metadata, including config path and severity overrides
123+
* check statuses after configured severities are applied
124+
* a combined `issues` array plus separate `errors` and `warnings` arrays
125+
* each issue's path, line, rule, severity, and message
126+
61127
### ❌ Current errors
62128

63129
* The target cannot be read as a bundle directory.
@@ -102,6 +168,16 @@ reserved file instead of a concept document.
102168
* Distinguish fatal frontmatter parse errors from non-blocking Markdown and
103169
frontmatter formatting warnings.
104170
* Surface broken local links without blocking partially written knowledge.
171+
* Generate JSON reports for CI, editor integrations, or external lint tooling.
172+
* Escalate or suppress rule severities for project-specific lint policies.
173+
174+
## Command Change History
175+
176+
### 2026-07-03
177+
178+
`openknowledge validate` added JSON reports with `--format json`, `--json`, and
179+
`--out`, plus configurable rule severities through `[validation.rules]` in
180+
`openknowledge.toml` and repeatable `--rule rule=off|warn|error` overrides.
105181

106182
---
107183

@@ -110,6 +186,8 @@ reserved file instead of a concept document.
110186
> **Source anchors**
111187
>
112188
> * `packages/cli/internal/okf/validate.go`
189+
> * `packages/cli/internal/okf/validation_policy.go`
190+
> * `packages/cli/internal/okf/validation_types.go`
113191
> * `packages/cli/cmd/openknowledge/main.go`
114192
>
115193
> **Update notes**

‎packages/cli/cmd/openknowledge/main.go‎

Lines changed: 90 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1458,9 +1458,33 @@ func runValidate(args []string) int {
14581458
fs.SetOutput(os.Stderr)
14591459
quiet := fs.Bool("quiet", false, "print only errors")
14601460
specVersion := fs.String("spec", "latest", "OKF spec version")
1461+
format := fs.String("format", "text", "output format: text or json")
1462+
out := fs.String("out", "", "write a machine-readable JSON report to this file")
1463+
asJSON := fs.Bool("json", false, "print the machine-readable JSON report")
1464+
ruleOverrides := stringListFlag{}
1465+
fs.Var(&ruleOverrides, "rule", "override validation rule severity as rule=off|warn|error; may be repeated")
14611466
if err := fs.Parse(args); err != nil {
14621467
return 2
14631468
}
1469+
if *asJSON {
1470+
*format = "json"
1471+
}
1472+
*format = strings.TrimSpace(strings.ToLower(*format))
1473+
if *format == "" {
1474+
*format = "text"
1475+
}
1476+
if *format != "text" && *format != "json" {
1477+
fmt.Fprintf(os.Stderr, "unsupported validate format: %s\n", *format)
1478+
return 2
1479+
}
1480+
if *quiet && *format == "json" {
1481+
fmt.Fprintln(os.Stderr, "--quiet cannot be combined with JSON validation output")
1482+
return 2
1483+
}
1484+
if strings.TrimSpace(*out) != "" && *format != "json" {
1485+
fmt.Fprintln(os.Stderr, "--out requires --format json or --json")
1486+
return 2
1487+
}
14641488

14651489
root := "."
14661490
if fs.NArg() > 1 {
@@ -1477,12 +1501,42 @@ func runValidate(args []string) int {
14771501
return 2
14781502
}
14791503

1480-
result, err := okf.ValidateWithVersion(root, *specVersion)
1504+
validationOptions, err := okf.LoadValidationOptions(root)
1505+
if err != nil {
1506+
fmt.Fprintln(os.Stderr, err)
1507+
return 2
1508+
}
1509+
cliOptions := okf.ValidationOptions{}
1510+
for _, override := range ruleOverrides {
1511+
rule, severity, err := okf.ParseValidationRuleOverride(override)
1512+
if err != nil {
1513+
fmt.Fprintln(os.Stderr, err)
1514+
return 2
1515+
}
1516+
if err := okf.SetValidationRuleSeverity(&cliOptions, rule, severity); err != nil {
1517+
fmt.Fprintln(os.Stderr, err)
1518+
return 2
1519+
}
1520+
}
1521+
validationOptions = okf.MergeValidationOptions(validationOptions, cliOptions)
1522+
1523+
result, err := okf.ValidateWithVersionAndOptions(root, *specVersion, validationOptions)
14811524
if err != nil {
14821525
fmt.Fprintln(os.Stderr, err)
14831526
return 2
14841527
}
14851528

1529+
if *format == "json" {
1530+
if err := printValidationJSONResult(result, strings.TrimSpace(*out)); err != nil {
1531+
fmt.Fprintln(os.Stderr, err)
1532+
return 1
1533+
}
1534+
if len(result.Errors) > 0 {
1535+
return 1
1536+
}
1537+
return 0
1538+
}
1539+
14861540
if *quiet {
14871541
for _, issue := range result.Errors {
14881542
fmt.Fprintln(os.Stderr, issue)
@@ -1500,6 +1554,25 @@ func runValidate(args []string) int {
15001554
return 0
15011555
}
15021556

1557+
func printValidationJSONResult(result okf.Result, out string) error {
1558+
data, err := json.MarshalIndent(result, "", " ")
1559+
if err != nil {
1560+
return err
1561+
}
1562+
data = append(data, '\n')
1563+
if out == "" {
1564+
fmt.Print(string(data))
1565+
return nil
1566+
}
1567+
if err := os.WriteFile(out, data, 0644); err != nil {
1568+
return err
1569+
}
1570+
terminal.success("Wrote validation report")
1571+
fmt.Printf("%s %s\n", terminal.muted("root"), terminal.path(result.Root))
1572+
fmt.Printf("%s %s\n", terminal.muted("out"), terminal.path(out))
1573+
return nil
1574+
}
1575+
15031576
func printValidationResult(result okf.Result) {
15041577
terminal.title("Open Knowledge Validate", "against Open Knowledge Format v"+result.SpecVersion)
15051578

@@ -2005,6 +2078,8 @@ Usage:
20052078
openknowledge spec latest|<version>
20062079
openknowledge validate [key-or-path]
20072080
openknowledge validate --spec <version> [key-or-path]
2081+
openknowledge validate --format json [key-or-path]
2082+
openknowledge validate --rule <rule=off|warn|error> [key-or-path]
20082083
openknowledge list [key-or-path]
20092084
openknowledge list --spec <version> [key-or-path]
20102085
openknowledge list --json [key-or-path]
@@ -2460,6 +2535,9 @@ Validate a bundle against an Open Knowledge Format spec.
24602535
Usage:
24612536
openknowledge validate [key-or-path]
24622537
openknowledge validate --spec <version> [key-or-path]
2538+
openknowledge validate --format json [key-or-path]
2539+
openknowledge validate --format json --out <file> [key-or-path]
2540+
openknowledge validate --rule <rule=off|warn|error> [key-or-path]
24632541
openknowledge validate --quiet [key-or-path]
24642542
openknowledge validate --help
24652543
@@ -2468,14 +2546,24 @@ Arguments:
24682546
24692547
Flags:
24702548
--spec OKF spec version. Defaults to latest.
2549+
--format Output format: text or json. Defaults to text.
2550+
--json Alias for --format json.
2551+
--out Write a JSON validation report to a file. Requires JSON output.
2552+
--rule Override one validation rule severity as rule=off|warn|error.
2553+
May be repeated and overrides [validation.rules] config.
24712554
--quiet Print only validation errors.
24722555
2556+
Config:
2557+
openknowledge.toml may define [validation.rules] with rule severities:
2558+
link-target = "error"
2559+
markdown-syntax = "off"
2560+
24732561
Versions:
24742562
%s
24752563
24762564
Exit codes:
24772565
0 Validation passed, with or without warnings.
2478-
1 Validation found errors.
2566+
1 Validation found errors after configured severity overrides.
24792567
2 Usage or setup error.
24802568
`, supportedSpecVersionsText())
24812569
}

‎packages/cli/cmd/openknowledge/main_test.go‎

Lines changed: 56 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,8 @@ func TestHelpTextIncludesCommandsFlagsAndExamples(t *testing.T) {
4141
"openknowledge to tar --out <file> [path]",
4242
"openknowledge to graph --out <file> [path]",
4343
"openknowledge validate --spec <version> [key-or-path]",
44+
"openknowledge validate --format json [key-or-path]",
45+
"openknowledge validate --rule <rule=off|warn|error> [key-or-path]",
4446
"openknowledge list --spec <version> [key-or-path]",
4547
"openknowledge list --json [key-or-path]",
4648
"Commands:",
@@ -247,8 +249,11 @@ func TestCommandHelpTextIncludesCommandSpecificDetails(t *testing.T) {
247249
help: validateHelpText(),
248250
required: []string{
249251
"openknowledge validate --quiet [key-or-path]",
252+
"openknowledge validate --format json --out <file> [key-or-path]",
253+
"--rule",
254+
"[validation.rules]",
250255
"Exit codes:",
251-
"Validation found errors.",
256+
"Validation found errors after configured severity overrides.",
252257
},
253258
},
254259
"list": {
@@ -359,6 +364,56 @@ func TestRunValidateAcceptsRegistryKey(t *testing.T) {
359364
}
360365
}
361366

367+
func TestRunValidatePrintsJSONReportWithConfiguredRules(t *testing.T) {
368+
root := t.TempDir()
369+
writeMainTestFile(t, root, "index.md", "# Bundle\n\n[Missing](missing.md)\n")
370+
writeMainTestFile(t, root, "openknowledge.toml", "[validation.rules]\nlink-target = \"error\"\n")
371+
372+
output, code := captureMainStdout(t, func() int {
373+
return runValidate([]string{"--json", root})
374+
})
375+
if code != 1 {
376+
t.Fatalf("expected configured validation error exit code, got %d\n%s", code, output)
377+
}
378+
var report okf.Result
379+
if err := json.Unmarshal([]byte(output), &report); err != nil {
380+
t.Fatalf("expected JSON validation report: %v\n%s", err, output)
381+
}
382+
if report.Summary.Status != "fail" || report.Summary.ErrorCount != 1 {
383+
t.Fatalf("unexpected JSON validation summary: %#v", report.Summary)
384+
}
385+
if len(report.Errors) != 1 || report.Errors[0].Rule != "link-target" || report.Errors[0].Severity != okf.ValidationSeverityError {
386+
t.Fatalf("expected escalated link-target error, got %#v", report.Errors)
387+
}
388+
if report.Policy.Overrides["link-target"] != okf.ValidationSeverityError || !strings.HasSuffix(report.Policy.ConfigPath, "openknowledge.toml") {
389+
t.Fatalf("expected policy metadata in report, got %#v", report.Policy)
390+
}
391+
}
392+
393+
func TestRunValidateWritesJSONReport(t *testing.T) {
394+
root := t.TempDir()
395+
writeMainTestFile(t, root, "index.md", "# Bundle\n")
396+
out := filepath.Join(t.TempDir(), "okf-report.json")
397+
398+
_, code := captureMainStdout(t, func() int {
399+
return runValidate([]string{"--format", "json", "--out", out, root})
400+
})
401+
if code != 0 {
402+
t.Fatalf("expected JSON report write to succeed, got exit code %d", code)
403+
}
404+
data, err := os.ReadFile(out)
405+
if err != nil {
406+
t.Fatal(err)
407+
}
408+
var report okf.Result
409+
if err := json.Unmarshal(data, &report); err != nil {
410+
t.Fatalf("expected written JSON validation report: %v\n%s", err, string(data))
411+
}
412+
if report.Summary.Status != "pass" || report.Root == "" {
413+
t.Fatalf("unexpected written validation report: %#v", report)
414+
}
415+
}
416+
362417
func TestRunListAcceptsRegistryKey(t *testing.T) {
363418
root := t.TempDir()
364419
writeMainTestFile(t, root, "index.md", "# Bundle\n")

0 commit comments

Comments
 (0)