Release 1.0.0-beta.14 benchmark suite catalog
This commit is contained in:
parent
acbe58f70e
commit
d3e628553f
69
.llm/BETA_14_BENCHMARK_SUITE_CATALOG_AND_METADATA_GATE.md
Normal file
69
.llm/BETA_14_BENCHMARK_SUITE_CATALOG_AND_METADATA_GATE.md
Normal file
@ -0,0 +1,69 @@
|
|||||||
|
# 1.0.0-beta.14 Benchmark Suite Catalog And Metadata Gate
|
||||||
|
|
||||||
|
Status: release scope for `1.0.0-beta.14`.
|
||||||
|
|
||||||
|
`1.0.0-beta.14` is a docs/tooling metadata slice for the existing benchmark
|
||||||
|
suite. It keeps the `1.0.0-beta` source language, typed core, runtime,
|
||||||
|
standard library, API surface, compiler diagnostics, diagnostic output shape,
|
||||||
|
ABI/layout behavior, and compiler-known runtime names unchanged.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
- Add `benchmarks/README.md` as the top-level benchmark suite catalog.
|
||||||
|
- Document `python3 benchmarks/runner.py --suite-list` as the non-JSON suite
|
||||||
|
inventory command.
|
||||||
|
- Document `python3 benchmarks/runner.py --suite-list --json` as the beta
|
||||||
|
tooling metadata form for the same inventory.
|
||||||
|
- Record the current suite inventory:
|
||||||
|
`math-loop`, `branch-loop`, `parse-loop`, `array-index-loop`,
|
||||||
|
`string-eq-loop`, `array-struct-field-loop`,
|
||||||
|
`enum-struct-payload-loop`, `vec-i32-index-loop`,
|
||||||
|
`vec-string-eq-loop`, and `json-quote-loop`.
|
||||||
|
- Document that benchmark timings are local-machine evidence only.
|
||||||
|
- Update README, language roadmap, language spec, release notes, post-beta
|
||||||
|
roadmap, and this release contract to introduce beta14.
|
||||||
|
|
||||||
|
## Acceptance
|
||||||
|
|
||||||
|
- Public docs name current release/stage `1.0.0-beta.14`.
|
||||||
|
- `benchmarks/README.md` explains both root suite-list commands:
|
||||||
|
`python3 benchmarks/runner.py --suite-list` and
|
||||||
|
`python3 benchmarks/runner.py --suite-list --json`.
|
||||||
|
- The suite catalog lists the current benchmark suite inventory without adding
|
||||||
|
or removing benchmark kernels.
|
||||||
|
- The suite catalog verifies required scaffold files for each suite:
|
||||||
|
`benchmark.json`, `run.py`, `slovo.toml`, and `src/main.slo`.
|
||||||
|
- The docs say timing output is local-machine evidence only and do not publish
|
||||||
|
timing numbers.
|
||||||
|
- The JSON suite listing is described as beta tooling metadata, not a stable
|
||||||
|
public schema.
|
||||||
|
- Explicit exclusions include no new benchmark kernels, no timing publication,
|
||||||
|
no performance thresholds, no stable JSON schema, no source-language change,
|
||||||
|
no runtime change, no stdlib/API change, no diagnostic-output change, and no
|
||||||
|
ABI/layout change.
|
||||||
|
|
||||||
|
## Explicit Non-Scope
|
||||||
|
|
||||||
|
- no new benchmark kernels or implementation language slots
|
||||||
|
- no benchmark timing publication
|
||||||
|
- no release performance threshold
|
||||||
|
- no stable JSON schema or stable benchmark metadata compatibility promise
|
||||||
|
- no source-language syntax, typed-core, lowering, runtime, stdlib, or API
|
||||||
|
change
|
||||||
|
- no compiler diagnostic or diagnostic-output shape change
|
||||||
|
- no ABI/layout or optimizer guarantee
|
||||||
|
- no worker-owned release publication work before controller review and gates
|
||||||
|
|
||||||
|
## Expected Controller Verification
|
||||||
|
|
||||||
|
- Run lightweight docs checks:
|
||||||
|
- `git diff --check -- README.md benchmarks/README.md docs/POST_BETA_ROADMAP.md docs/language/SPEC-v1.md docs/language/ROADMAP.md docs/language/RELEASE_NOTES.md .llm/BETA_14_BENCHMARK_SUITE_CATALOG_AND_METADATA_GATE.md`
|
||||||
|
- run an `rg` check for stale beta13-only current-stage phrasing across the
|
||||||
|
same touched docs
|
||||||
|
- Smoke the metadata commands:
|
||||||
|
- `python3 benchmarks/runner.py --suite-list`
|
||||||
|
- `python3 benchmarks/runner.py --suite-list --json`
|
||||||
|
- Run focused compiler/tooling checks:
|
||||||
|
- `cargo test --test benchmark_suite_catalog_beta14`
|
||||||
|
- `cargo test --test benchmark_math_loop_scaffold`
|
||||||
|
- Do not commit, tag, push, or run release publication from this worker scope.
|
||||||
76
.llm/reviews/BETA_14_RELEASE_REVIEW.md
Normal file
76
.llm/reviews/BETA_14_RELEASE_REVIEW.md
Normal file
@ -0,0 +1,76 @@
|
|||||||
|
# 1.0.0-beta.14 Release Review
|
||||||
|
|
||||||
|
Status: ready for publication after controller release gate.
|
||||||
|
|
||||||
|
## Verdict
|
||||||
|
|
||||||
|
No blocking findings found for the benchmark suite catalog and metadata gate.
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
No blocking findings.
|
||||||
|
|
||||||
|
Non-blocking note: `compiler/tests/benchmark_suite_catalog_beta14.rs` verifies
|
||||||
|
byte-stable JSON, the current 10 benchmark names/directories, top-level counts,
|
||||||
|
required scaffold-file status, missing-file lists, checksum metadata presence,
|
||||||
|
timing mode strings, and implementation slot names. The current runner output
|
||||||
|
also includes `run_args`, `source_stem`, and per-implementation source paths.
|
||||||
|
If those fields become contractual in a later release, the focused gate should
|
||||||
|
parse the JSON and assert them per benchmark.
|
||||||
|
|
||||||
|
## Scope Checked
|
||||||
|
|
||||||
|
- Public release docs in scope: `README.md`, `benchmarks/README.md`,
|
||||||
|
`docs/POST_BETA_ROADMAP.md`, `docs/language/SPEC-v1.md`,
|
||||||
|
`docs/language/ROADMAP.md`, `docs/language/RELEASE_NOTES.md`,
|
||||||
|
`docs/compiler/ROADMAP.md`, `docs/compiler/RELEASE_NOTES.md`, and
|
||||||
|
`.llm/BETA_14_BENCHMARK_SUITE_CATALOG_AND_METADATA_GATE.md`.
|
||||||
|
- Tooling and tests in scope: `benchmarks/runner.py`,
|
||||||
|
`compiler/tests/benchmark_suite_catalog_beta14.rs`,
|
||||||
|
`compiler/tests/benchmark_math_loop_scaffold.rs`,
|
||||||
|
`scripts/release-gate.sh`, `compiler/Cargo.toml`, and
|
||||||
|
`compiler/Cargo.lock`.
|
||||||
|
- Release-facing current-stage wording points at `1.0.0-beta.14`. Remaining
|
||||||
|
beta13 references are historical beta13 diagnostics/release notes, the
|
||||||
|
existing `diagnostics_schema_beta13` gate, or older review/contract files.
|
||||||
|
- Benchmark docs and runner output consistently keep timing local-machine-only,
|
||||||
|
do not publish timing numbers, do not define performance thresholds, and do
|
||||||
|
not claim a stable benchmark JSON schema.
|
||||||
|
- The top-level catalog documents the current 10 suites and all suite-list
|
||||||
|
commands. Relative links to `benchmarks/README.md` resolve from the files
|
||||||
|
that introduce them.
|
||||||
|
- The runner preserves per-suite `run.py --list --json` behavior and adds root
|
||||||
|
`benchmarks/runner.py --suite-list` and `--suite-list --json` metadata.
|
||||||
|
- `scripts/release-gate.sh` now runs `cargo test --test
|
||||||
|
benchmark_suite_catalog_beta14` before the full compiler test suite.
|
||||||
|
- Cargo package version and lockfile version both read `1.0.0-beta.14`.
|
||||||
|
- Focused private/local publication text scan found no machine-local paths,
|
||||||
|
private checkout/user names, private remotes, or local IP text in the reviewed
|
||||||
|
release surface.
|
||||||
|
|
||||||
|
## Verification Commands And Results
|
||||||
|
|
||||||
|
- `python3 benchmarks/runner.py --suite-list --json`: passed. Output reported
|
||||||
|
`benchmark_count: 10`, `implementation_slots: 60`, status `ok`, cold/hot
|
||||||
|
timing modes, required scaffold-file status with no missing files, checksum
|
||||||
|
metadata, runtime args, implementation source paths, and the local-only
|
||||||
|
timing disclaimer.
|
||||||
|
- `python3 benchmarks/runner.py --suite-list`: passed.
|
||||||
|
- `python3 benchmarks/math-loop/run.py --list --json`: passed, confirming the
|
||||||
|
existing per-benchmark list mode still works through the shared runner.
|
||||||
|
- `python3 benchmarks/math-loop/run.py --suite-list --json`: passed, confirming
|
||||||
|
suite listing also resolves correctly through a per-benchmark wrapper.
|
||||||
|
- `cargo test --test benchmark_suite_catalog_beta14`: passed, 1 test.
|
||||||
|
- `cargo test --test benchmark_math_loop_scaffold`: passed, 1 test.
|
||||||
|
- `git diff --check`: passed.
|
||||||
|
- `git diff --check -- README.md benchmarks/README.md docs/POST_BETA_ROADMAP.md docs/language/SPEC-v1.md docs/language/ROADMAP.md docs/language/RELEASE_NOTES.md .llm/BETA_14_BENCHMARK_SUITE_CATALOG_AND_METADATA_GATE.md`:
|
||||||
|
passed.
|
||||||
|
- `bash -n scripts/release-gate.sh`: passed.
|
||||||
|
- `cargo fmt --check`: passed.
|
||||||
|
- Focused `rg` stale-current-stage scan for beta13 current release/stage wording
|
||||||
|
in the touched release docs produced no matches.
|
||||||
|
- Focused `rg` private/local publication text scan over README, docs,
|
||||||
|
benchmarks, `.llm`, compiler tests, and scripts produced no matches.
|
||||||
|
|
||||||
|
The full `./scripts/release-gate.sh` was not run during this review; the
|
||||||
|
focused beta14 gate coverage and requested lightweight checks passed.
|
||||||
58
README.md
58
README.md
@ -6,7 +6,7 @@ This repository is the canonical public monorepo for the language design,
|
|||||||
standard library source, compiler, runtime, examples, benchmarks, and technical
|
standard library source, compiler, runtime, examples, benchmarks, and technical
|
||||||
documents.
|
documents.
|
||||||
|
|
||||||
Current release: `1.0.0-beta.13`.
|
Current release: `1.0.0-beta.14`.
|
||||||
|
|
||||||
## Repository Layout
|
## Repository Layout
|
||||||
|
|
||||||
@ -24,7 +24,7 @@ scripts/ local release and document tooling
|
|||||||
|
|
||||||
## Beta Scope
|
## Beta Scope
|
||||||
|
|
||||||
`1.0.0-beta.13` keeps the `1.0.0-beta` language baseline, includes the
|
`1.0.0-beta.14` keeps the `1.0.0-beta` language baseline, includes the
|
||||||
`1.0.0-beta.1` tooling/install hardening slice, the `1.0.0-beta.2`
|
`1.0.0-beta.1` tooling/install hardening slice, the `1.0.0-beta.2`
|
||||||
runtime/resource foundation bundle, the `1.0.0-beta.3` standard-library
|
runtime/resource foundation bundle, the `1.0.0-beta.3` standard-library
|
||||||
stabilization bundle, the `1.0.0-beta.4` language-usability diagnostics
|
stabilization bundle, the `1.0.0-beta.4` language-usability diagnostics
|
||||||
@ -35,7 +35,8 @@ alias foundation, the `1.0.0-beta.9` collection alias unification and
|
|||||||
generic reservation slice, the `1.0.0-beta.10` developer-experience API
|
generic reservation slice, the `1.0.0-beta.10` developer-experience API
|
||||||
discovery slice, and the `1.0.0-beta.11` local package API documentation
|
discovery slice, and the `1.0.0-beta.11` local package API documentation
|
||||||
slice, plus the `1.0.0-beta.12` concrete vector query and prefix parity
|
slice, plus the `1.0.0-beta.12` concrete vector query and prefix parity
|
||||||
slice, and the `1.0.0-beta.13` diagnostic catalog and schema policy slice.
|
slice, the `1.0.0-beta.13` diagnostic catalog and schema policy slice, and
|
||||||
|
the `1.0.0-beta.14` benchmark suite catalog and metadata gate.
|
||||||
The language baseline supports practical local command-line, file, and
|
The language baseline supports practical local command-line, file, and
|
||||||
loopback-network programs with:
|
loopback-network programs with:
|
||||||
|
|
||||||
@ -73,23 +74,31 @@ S-expression/JSON relationship, required and optional fields, JSON-line
|
|||||||
discipline, source-less diagnostics, manifest diagnostic metadata,
|
discipline, source-less diagnostics, manifest diagnostic metadata,
|
||||||
compatibility and migration classes, and the current golden diagnostic code
|
compatibility and migration classes, and the current golden diagnostic code
|
||||||
catalog.
|
catalog.
|
||||||
|
The `1.0.0-beta.14` benchmark metadata slice documents the existing benchmark
|
||||||
|
suite catalog in [`benchmarks/README.md`](benchmarks/README.md). It explains
|
||||||
|
`python3 benchmarks/runner.py --suite-list` for the human-readable suite
|
||||||
|
inventory and `python3 benchmarks/runner.py --suite-list --json` for beta
|
||||||
|
tooling metadata, with required scaffold-file verification for each current
|
||||||
|
suite. Benchmark timings remain local-machine evidence only; the JSON field
|
||||||
|
set is not a stable public schema.
|
||||||
|
|
||||||
Still deferred before stable: executable generics, maps/sets, broad package
|
Still deferred before stable: executable generics, maps/sets, broad package
|
||||||
registry semantics, stable Markdown schema, stable stdlib/API compatibility
|
registry semantics, stable Markdown schema, stable stdlib/API compatibility
|
||||||
freeze, DNS/TLS/async networking, LSP/watch guarantees, SARIF and daemon
|
freeze, DNS/TLS/async networking, LSP/watch guarantees, SARIF and daemon
|
||||||
protocols, stable `1.0.0` diagnostics freeze, re-exports/globs/hierarchical
|
protocols, stable `1.0.0` diagnostics freeze, re-exports/globs/hierarchical
|
||||||
modules, mutable vectors, slice/view APIs, iterators, new compiler-known
|
modules, mutable vectors, slice/view APIs, iterators, new compiler-known
|
||||||
runtime names, stable ABI and layout, performance claims, and runtime changes
|
runtime names, stable ABI and layout, performance claims, stable benchmark
|
||||||
for generic collections.
|
JSON metadata schema, and runtime changes for generic collections.
|
||||||
|
|
||||||
The next likely language slice after `1.0.0-beta.13` should continue from the
|
The next likely language slice after `1.0.0-beta.14` should continue from the
|
||||||
developer-experience, package, and reserved generic/collection lanes without
|
developer-experience, package, benchmark metadata, and reserved
|
||||||
claiming executable generics, maps, sets, traits, inference, monomorphization,
|
generic/collection lanes without claiming executable generics, maps, sets,
|
||||||
iterators, ABI stability, runtime changes, LSP/watch protocols, SARIF/daemon
|
traits, inference, monomorphization, iterators, ABI stability, runtime changes,
|
||||||
protocols, registry semantics, stable Markdown schema, a stable `1.0.0`
|
LSP/watch protocols, SARIF/daemon protocols, registry semantics, stable
|
||||||
diagnostics freeze, standard-library/API compatibility freeze, mutable
|
Markdown schema, stable benchmark JSON schema, a stable `1.0.0` diagnostics
|
||||||
vectors, slice/view APIs, new runtime names, or performance claims until the
|
freeze, standard-library/API compatibility freeze, mutable vectors, slice/view
|
||||||
contract and gates are explicit.
|
APIs, new runtime names, or performance claims until the contract and gates
|
||||||
|
are explicit.
|
||||||
|
|
||||||
## Build And Test
|
## Build And Test
|
||||||
|
|
||||||
@ -332,6 +341,28 @@ This release does not change the source language, runtime, stdlib API,
|
|||||||
diagnostic output shape, compiler CLI, LSP/watch behavior, SARIF/daemon
|
diagnostic output shape, compiler CLI, LSP/watch behavior, SARIF/daemon
|
||||||
protocols, stable Markdown schema, or stable `1.0.0` diagnostics freeze.
|
protocols, stable Markdown schema, or stable `1.0.0` diagnostics freeze.
|
||||||
|
|
||||||
|
## 1.0.0-beta.14 Benchmark Suite Catalog And Metadata Gate
|
||||||
|
|
||||||
|
The `1.0.0-beta.14` release documents the existing benchmark suite catalog as
|
||||||
|
beta-scoped metadata tooling. It adds
|
||||||
|
[`benchmarks/README.md`](benchmarks/README.md) with the current ten-suite
|
||||||
|
inventory, local evidence policy, suite-list commands, and exclusions.
|
||||||
|
|
||||||
|
The root suite catalog commands are:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 benchmarks/runner.py --suite-list
|
||||||
|
python3 benchmarks/runner.py --suite-list --json
|
||||||
|
```
|
||||||
|
|
||||||
|
The non-JSON listing is for local review. The JSON listing is beta tooling
|
||||||
|
metadata for local gates and adapters, not a stable public schema.
|
||||||
|
|
||||||
|
This release does not add benchmark kernels, publish timing numbers, define
|
||||||
|
performance thresholds, change the source language, runtime, stdlib/API
|
||||||
|
surface, diagnostic output, compiler ABI/layout behavior, or make cross-machine
|
||||||
|
performance claims.
|
||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
- [Language Manifest](docs/language/MANIFEST.md)
|
- [Language Manifest](docs/language/MANIFEST.md)
|
||||||
@ -339,6 +370,7 @@ protocols, stable Markdown schema, or stable `1.0.0` diagnostics freeze.
|
|||||||
- [Diagnostics Policy](docs/language/DIAGNOSTICS.md)
|
- [Diagnostics Policy](docs/language/DIAGNOSTICS.md)
|
||||||
- [Local Package And Workspace Guide](docs/language/PACKAGES.md)
|
- [Local Package And Workspace Guide](docs/language/PACKAGES.md)
|
||||||
- [Standard Library API Catalog](docs/language/STDLIB_API.md)
|
- [Standard Library API Catalog](docs/language/STDLIB_API.md)
|
||||||
|
- [Benchmark Suite Catalog](benchmarks/README.md)
|
||||||
- [Compiler Manifest](docs/compiler/GLAGOL_COMPILER_MANIFEST.md)
|
- [Compiler Manifest](docs/compiler/GLAGOL_COMPILER_MANIFEST.md)
|
||||||
- [Post-Beta Roadmap](docs/POST_BETA_ROADMAP.md)
|
- [Post-Beta Roadmap](docs/POST_BETA_ROADMAP.md)
|
||||||
- [Slovo Whitepaper](docs/papers/SLOVO_WHITEPAPER.md)
|
- [Slovo Whitepaper](docs/papers/SLOVO_WHITEPAPER.md)
|
||||||
|
|||||||
83
benchmarks/README.md
Normal file
83
benchmarks/README.md
Normal file
@ -0,0 +1,83 @@
|
|||||||
|
# Slovo Benchmark Suite Catalog
|
||||||
|
|
||||||
|
Release stage: `1.0.0-beta.14`.
|
||||||
|
|
||||||
|
The benchmark suite is beta-scoped local tooling. It catalogs deterministic
|
||||||
|
same-machine comparison scaffolds for Slovo and sibling implementations. It
|
||||||
|
does not publish benchmark results, set performance thresholds, define optimizer
|
||||||
|
claims, or create cross-machine comparisons.
|
||||||
|
|
||||||
|
## Suite Listing
|
||||||
|
|
||||||
|
From the repository root, list the suite catalog for humans:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 benchmarks/runner.py --suite-list
|
||||||
|
```
|
||||||
|
|
||||||
|
List the same catalog as beta tooling metadata:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 benchmarks/runner.py --suite-list --json
|
||||||
|
```
|
||||||
|
|
||||||
|
The non-JSON listing is for local review. The JSON listing is for local gates
|
||||||
|
and tooling adapters that need the current suite inventory. The JSON field set
|
||||||
|
is not a stable public schema; it may change during beta releases.
|
||||||
|
Both forms verify the required scaffold files for each suite:
|
||||||
|
`benchmark.json`, `run.py`, `slovo.toml`, and `src/main.slo`.
|
||||||
|
|
||||||
|
Each suite still owns its local metadata and run commands:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 benchmarks/<suite>/run.py --list
|
||||||
|
python3 benchmarks/<suite>/run.py --list --json
|
||||||
|
python3 benchmarks/<suite>/run.py --dry-run
|
||||||
|
```
|
||||||
|
|
||||||
|
## Current Suites
|
||||||
|
|
||||||
|
All current suites provide Slovo, C, Rust, Python, Clojure, and Common
|
||||||
|
Lisp/SBCL source slots. Missing local toolchains are skipped by the per-suite
|
||||||
|
runner where possible.
|
||||||
|
|
||||||
|
| Suite | Focus | Base checksum | Hot-loop checksum | Runtime args |
|
||||||
|
| --- | --- | --- | --- | --- |
|
||||||
|
| `math-loop` | arithmetic and scalar accumulation | `5000001` | `50000001` | none |
|
||||||
|
| `branch-loop` | deterministic branch-heavy integer loop | `1185071` | `220775` | none |
|
||||||
|
| `parse-loop` | repeated signed decimal `i32` parsing | `345000001` | `450000001` | `12345` |
|
||||||
|
| `array-index-loop` | immutable fixed-array indexing | `3875007` | `38750007` | none |
|
||||||
|
| `string-eq-loop` | fixed string lookup and equality | `4600001` | `46000001` | `omega` |
|
||||||
|
| `array-struct-field-loop` | fixed-array access through an immutable struct field | `3875011` | `38750011` | none |
|
||||||
|
| `enum-struct-payload-loop` | enum payload matching with struct and array access | `3500013` | `35000013` | none |
|
||||||
|
| `vec-i32-index-loop` | runtime-owned `(vec i32)` indexing | `3875007` | `38750007` | none |
|
||||||
|
| `vec-string-eq-loop` | runtime-owned `(vec string)` lookup and equality | `4600001` | `46000001` | `omega` |
|
||||||
|
| `json-quote-loop` | compact JSON string quoting | `15000001` | `150000001` | `slo"vo\path` |
|
||||||
|
|
||||||
|
The base loop count is `1000000` for every current suite. The hot-loop count is
|
||||||
|
`10000000` for every current suite. The runner supplies loop counts and runtime
|
||||||
|
arguments at execution time so native compilers cannot fold the work into a
|
||||||
|
constant answer.
|
||||||
|
|
||||||
|
## Local Evidence Only
|
||||||
|
|
||||||
|
Benchmark output is local-machine evidence only:
|
||||||
|
|
||||||
|
- cold-process mode measures execution after each implementation has been built
|
||||||
|
once; it does not include compile time
|
||||||
|
- hot-loop mode is startup-amortized local evidence and reports total time plus
|
||||||
|
normalized timing for the base loop count
|
||||||
|
- Clojure timings include JVM and Clojure startup
|
||||||
|
- Common Lisp timings include SBCL script startup
|
||||||
|
- reported timings depend on the local CPU, OS, compiler versions, toolchain
|
||||||
|
availability, thermal/load state, and runner configuration
|
||||||
|
|
||||||
|
This catalog intentionally publishes no timing numbers.
|
||||||
|
|
||||||
|
## Exclusions
|
||||||
|
|
||||||
|
`1.0.0-beta.14` does not add benchmark kernels, publish timings, define
|
||||||
|
performance thresholds, define a stable JSON schema, change the Slovo source
|
||||||
|
language, change runtime behavior, change standard-library or API contracts,
|
||||||
|
change diagnostic output, change ABI/layout behavior, or make cross-machine
|
||||||
|
performance claims.
|
||||||
@ -18,6 +18,12 @@ from typing import Callable
|
|||||||
|
|
||||||
TIMING_SCOPE = "local-machine comparison only"
|
TIMING_SCOPE = "local-machine comparison only"
|
||||||
TIMING_MODES = ["cold-process", "hot-loop"]
|
TIMING_MODES = ["cold-process", "hot-loop"]
|
||||||
|
SUITE_NAME = "glagol-local-benchmark-suite"
|
||||||
|
LOCAL_TIMING_DISCLAIMER = (
|
||||||
|
"Local timing comparison only; not a published benchmark result and not a cross-machine performance claim."
|
||||||
|
)
|
||||||
|
REQUIRED_BENCHMARK_FILES = ["benchmark.json", "run.py", "slovo.toml", "src/main.slo"]
|
||||||
|
EXPECTED_IMPLEMENTATION_NAMES = ["slovo", "c", "rust", "python", "clojure", "common_lisp"]
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
@ -52,6 +58,14 @@ class Implementation:
|
|||||||
|
|
||||||
|
|
||||||
def main(root: Path, argv: list[str]) -> int:
|
def main(root: Path, argv: list[str]) -> int:
|
||||||
|
if any(arg == "--suite-list" for arg in argv):
|
||||||
|
return main_suite(root, argv)
|
||||||
|
if not (root / "benchmark.json").is_file():
|
||||||
|
parser = argparse.ArgumentParser(description="Shared local Glagol benchmark runner.")
|
||||||
|
parser.add_argument("--suite-list", action="store_true", help="list suite metadata and exit")
|
||||||
|
parser.add_argument("--json", action="store_true", help="emit JSON for --suite-list")
|
||||||
|
parser.error("run from a benchmark run.py, or pass --suite-list at the benchmark suite root")
|
||||||
|
|
||||||
spec = read_spec(root)
|
spec = read_spec(root)
|
||||||
implementations = available_implementations(root, spec)
|
implementations = available_implementations(root, spec)
|
||||||
parser = argparse.ArgumentParser(description=f"Run local {spec.name} timing comparisons.")
|
parser = argparse.ArgumentParser(description=f"Run local {spec.name} timing comparisons.")
|
||||||
@ -89,6 +103,25 @@ def main(root: Path, argv: list[str]) -> int:
|
|||||||
return 1 if any(result["status"] == "failed" for result in results) else 0
|
return 1 if any(result["status"] == "failed" for result in results) else 0
|
||||||
|
|
||||||
|
|
||||||
|
def main_suite(root: Path, argv: list[str]) -> int:
|
||||||
|
parser = argparse.ArgumentParser(description="List local Glagol benchmark suite metadata.")
|
||||||
|
parser.add_argument("--suite-list", action="store_true", help="list suite metadata and exit")
|
||||||
|
parser.add_argument("--json", action="store_true", help="emit JSON for suite metadata")
|
||||||
|
args = parser.parse_args(argv)
|
||||||
|
if not args.suite_list:
|
||||||
|
parser.error("pass --suite-list to list benchmark suite metadata")
|
||||||
|
|
||||||
|
suite_root = resolve_suite_root(root)
|
||||||
|
emit_suite_list(build_suite_catalog(suite_root), args.json)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_suite_root(root: Path) -> Path:
|
||||||
|
if (root / "benchmark.json").is_file():
|
||||||
|
return root.parent
|
||||||
|
return root
|
||||||
|
|
||||||
|
|
||||||
def read_spec(root: Path) -> BenchmarkSpec:
|
def read_spec(root: Path) -> BenchmarkSpec:
|
||||||
data = json.loads((root / "benchmark.json").read_text(encoding="utf-8"))
|
data = json.loads((root / "benchmark.json").read_text(encoding="utf-8"))
|
||||||
loop_count = int(data["loop_count"])
|
loop_count = int(data["loop_count"])
|
||||||
@ -278,6 +311,143 @@ def select_implementations(implementations: list[Implementation], names: list[st
|
|||||||
return [impl for impl in implementations if impl.name in selected_names]
|
return [impl for impl in implementations if impl.name in selected_names]
|
||||||
|
|
||||||
|
|
||||||
|
def build_suite_catalog(suite_root: Path) -> dict[str, object]:
|
||||||
|
benchmarks: list[dict[str, object]] = []
|
||||||
|
implementation_slot_count = 0
|
||||||
|
missing_required_files: list[str] = []
|
||||||
|
missing_implementation_slots: list[str] = []
|
||||||
|
|
||||||
|
for root in suite_benchmark_roots(suite_root):
|
||||||
|
spec = read_spec(root)
|
||||||
|
implementations = available_implementations(root, spec)
|
||||||
|
implementation_slot_count += len(implementations)
|
||||||
|
benchmark = suite_benchmark_metadata(suite_root, root, spec, implementations)
|
||||||
|
benchmarks.append(benchmark)
|
||||||
|
|
||||||
|
directory = str(benchmark["directory"])
|
||||||
|
for required_file in benchmark["required_files"]:
|
||||||
|
assert isinstance(required_file, dict)
|
||||||
|
if required_file["status"] != "present":
|
||||||
|
missing_required_files.append(f"{directory}/{required_file['path']}")
|
||||||
|
|
||||||
|
present_implementations = {
|
||||||
|
str(implementation["name"])
|
||||||
|
for implementation in benchmark["implementation_slots"]
|
||||||
|
if isinstance(implementation, dict)
|
||||||
|
}
|
||||||
|
for expected in EXPECTED_IMPLEMENTATION_NAMES:
|
||||||
|
if expected not in present_implementations:
|
||||||
|
missing_implementation_slots.append(f"{directory}:{expected}")
|
||||||
|
|
||||||
|
return {
|
||||||
|
"suite": SUITE_NAME,
|
||||||
|
"timing_scope": TIMING_SCOPE,
|
||||||
|
"timing_modes": TIMING_MODES,
|
||||||
|
"timing_disclaimer": LOCAL_TIMING_DISCLAIMER,
|
||||||
|
"benchmark_count": len(benchmarks),
|
||||||
|
"benchmarks": benchmarks,
|
||||||
|
"verification": {
|
||||||
|
"status": "ok" if not missing_required_files and not missing_implementation_slots else "incomplete",
|
||||||
|
"benchmark_metadata_files": len(benchmarks),
|
||||||
|
"required_files": len(benchmarks) * len(REQUIRED_BENCHMARK_FILES),
|
||||||
|
"missing_required_files": missing_required_files,
|
||||||
|
"implementation_slots": implementation_slot_count,
|
||||||
|
"expected_implementation_slots": len(benchmarks) * len(EXPECTED_IMPLEMENTATION_NAMES),
|
||||||
|
"missing_implementation_slots": missing_implementation_slots,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def suite_benchmark_roots(suite_root: Path) -> list[Path]:
|
||||||
|
return sorted(
|
||||||
|
[path for path in suite_root.iterdir() if path.is_dir() and (path / "benchmark.json").is_file()],
|
||||||
|
key=lambda path: path.name,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def suite_benchmark_metadata(
|
||||||
|
suite_root: Path,
|
||||||
|
root: Path,
|
||||||
|
spec: BenchmarkSpec,
|
||||||
|
implementations: list[Implementation],
|
||||||
|
) -> dict[str, object]:
|
||||||
|
return {
|
||||||
|
"name": spec.name,
|
||||||
|
"directory": str(root.relative_to(suite_root)),
|
||||||
|
"source_stem": spec.source_stem,
|
||||||
|
"timing_modes": TIMING_MODES,
|
||||||
|
"loop_count_source": "stdin",
|
||||||
|
"loop_count": spec.loop_count,
|
||||||
|
"hot_loop_count": spec.hot_loop_count,
|
||||||
|
"expected_checksum": spec.expected_checksum,
|
||||||
|
"hot_expected_checksum": spec.hot_expected_checksum,
|
||||||
|
"required_files": [
|
||||||
|
{
|
||||||
|
"path": relative,
|
||||||
|
"status": "present" if (root / relative).is_file() else "missing",
|
||||||
|
}
|
||||||
|
for relative in REQUIRED_BENCHMARK_FILES
|
||||||
|
],
|
||||||
|
"checksum_metadata": {
|
||||||
|
"cold_process": {
|
||||||
|
"expected_checksum": spec.expected_checksum,
|
||||||
|
"stdin": spec.stdin_text,
|
||||||
|
},
|
||||||
|
"hot_loop": {
|
||||||
|
"expected_checksum": spec.hot_expected_checksum,
|
||||||
|
"stdin": spec.hot_stdin_text,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"run_args": spec.run_args,
|
||||||
|
"implementation_slots": [
|
||||||
|
{
|
||||||
|
"name": impl.name,
|
||||||
|
"language": impl.language,
|
||||||
|
"source": str(impl.source.relative_to(suite_root)),
|
||||||
|
}
|
||||||
|
for impl in implementations
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def emit_suite_list(metadata: dict[str, object], as_json: bool) -> None:
|
||||||
|
if as_json:
|
||||||
|
print(json.dumps(metadata, indent=2, sort_keys=True))
|
||||||
|
return
|
||||||
|
|
||||||
|
print(f"{metadata['suite']}: {metadata['timing_scope']}")
|
||||||
|
print(str(metadata["timing_disclaimer"]))
|
||||||
|
print(f"benchmark_count={metadata['benchmark_count']}")
|
||||||
|
print(f"timing_modes={','.join(TIMING_MODES)}")
|
||||||
|
verification = metadata["verification"]
|
||||||
|
assert isinstance(verification, dict)
|
||||||
|
print(f"verification_status={verification['status']}")
|
||||||
|
print(f"required_files={verification['required_files']}")
|
||||||
|
print(f"implementation_slots={verification['implementation_slots']}")
|
||||||
|
print("benchmarks:")
|
||||||
|
for benchmark in metadata["benchmarks"]:
|
||||||
|
assert isinstance(benchmark, dict)
|
||||||
|
print(
|
||||||
|
" {name} ({directory}): loop_count={loop_count} hot_loop_count={hot_loop_count} "
|
||||||
|
"expected_checksum={expected_checksum} hot_expected_checksum={hot_expected_checksum}".format(
|
||||||
|
name=benchmark["name"],
|
||||||
|
directory=benchmark["directory"],
|
||||||
|
loop_count=benchmark["loop_count"],
|
||||||
|
hot_loop_count=benchmark["hot_loop_count"],
|
||||||
|
expected_checksum=benchmark["expected_checksum"],
|
||||||
|
hot_expected_checksum=benchmark["hot_expected_checksum"],
|
||||||
|
)
|
||||||
|
)
|
||||||
|
print(" required_files:")
|
||||||
|
for required_file in benchmark["required_files"]:
|
||||||
|
assert isinstance(required_file, dict)
|
||||||
|
print(f" {required_file['path']}: {required_file['status']}")
|
||||||
|
print(" implementations:")
|
||||||
|
for implementation in benchmark["implementation_slots"]:
|
||||||
|
assert isinstance(implementation, dict)
|
||||||
|
print(f" {implementation['name']}: {implementation['language']} ({implementation['source']})")
|
||||||
|
|
||||||
|
|
||||||
def emit_list(root: Path, spec: BenchmarkSpec, implementations: list[Implementation], as_json: bool) -> None:
|
def emit_list(root: Path, spec: BenchmarkSpec, implementations: list[Implementation], as_json: bool) -> None:
|
||||||
metadata = {
|
metadata = {
|
||||||
"benchmark": spec.name,
|
"benchmark": spec.name,
|
||||||
@ -520,3 +690,7 @@ def shlex_quote(value: str) -> str:
|
|||||||
if value and all(char.isalnum() or char in "/._:-" for char in value):
|
if value and all(char.isalnum() or char in "/._:-" for char in value):
|
||||||
return value
|
return value
|
||||||
return "'" + value.replace("'", "'\"'\"'") + "'"
|
return "'" + value.replace("'", "'\"'\"'") + "'"
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main(Path(__file__).resolve().parent, sys.argv[1:]))
|
||||||
|
|||||||
2
compiler/Cargo.lock
generated
2
compiler/Cargo.lock
generated
@ -4,4 +4,4 @@ version = 3
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "glagol"
|
name = "glagol"
|
||||||
version = "1.0.0-beta.13"
|
version = "1.0.0-beta.14"
|
||||||
|
|||||||
@ -1,6 +1,6 @@
|
|||||||
[package]
|
[package]
|
||||||
name = "glagol"
|
name = "glagol"
|
||||||
version = "1.0.0-beta.13"
|
version = "1.0.0-beta.14"
|
||||||
edition = "2021"
|
edition = "2021"
|
||||||
description = "Glagol, the first compiler for the Slovo language"
|
description = "Glagol, the first compiler for the Slovo language"
|
||||||
license = "MIT OR Apache-2.0"
|
license = "MIT OR Apache-2.0"
|
||||||
|
|||||||
134
compiler/tests/benchmark_suite_catalog_beta14.rs
Normal file
134
compiler/tests/benchmark_suite_catalog_beta14.rs
Normal file
@ -0,0 +1,134 @@
|
|||||||
|
use std::{
|
||||||
|
env,
|
||||||
|
path::Path,
|
||||||
|
process::{Command, Output},
|
||||||
|
};
|
||||||
|
|
||||||
|
const BENCHMARKS: &[&str] = &[
|
||||||
|
"array-index-loop",
|
||||||
|
"array-struct-field-loop",
|
||||||
|
"branch-loop",
|
||||||
|
"enum-struct-payload-loop",
|
||||||
|
"json-quote-loop",
|
||||||
|
"math-loop",
|
||||||
|
"parse-loop",
|
||||||
|
"string-eq-loop",
|
||||||
|
"vec-i32-index-loop",
|
||||||
|
"vec-string-eq-loop",
|
||||||
|
];
|
||||||
|
|
||||||
|
const IMPLEMENTATIONS: &[&str] = &["slovo", "c", "rust", "python", "clojure", "common_lisp"];
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn suite_catalog_is_byte_stable_and_lists_current_benchmarks() {
|
||||||
|
let repo = Path::new(env!("CARGO_MANIFEST_DIR")).join("..");
|
||||||
|
let python = python_command();
|
||||||
|
|
||||||
|
let first = run_suite_catalog(&repo, &python);
|
||||||
|
let second = run_suite_catalog(&repo, &python);
|
||||||
|
|
||||||
|
assert_success("first suite catalog run", &first);
|
||||||
|
assert_success("second suite catalog run", &second);
|
||||||
|
assert_eq!(
|
||||||
|
first.stdout, second.stdout,
|
||||||
|
"suite catalog JSON must be byte-stable across runs"
|
||||||
|
);
|
||||||
|
|
||||||
|
let stdout = String::from_utf8_lossy(&first.stdout);
|
||||||
|
for needle in [
|
||||||
|
r#""suite": "glagol-local-benchmark-suite""#,
|
||||||
|
r#""benchmark_count": 10"#,
|
||||||
|
r#""benchmark_metadata_files": 10"#,
|
||||||
|
r#""required_files": 40"#,
|
||||||
|
r#""missing_required_files": []"#,
|
||||||
|
r#""implementation_slots": 60"#,
|
||||||
|
r#""expected_implementation_slots": 60"#,
|
||||||
|
r#""missing_implementation_slots": []"#,
|
||||||
|
r#""status": "ok""#,
|
||||||
|
r#""timing_scope": "local-machine comparison only""#,
|
||||||
|
r#""timing_disclaimer": "Local timing comparison only; not a published benchmark result and not a cross-machine performance claim.""#,
|
||||||
|
r#""cold-process""#,
|
||||||
|
r#""hot-loop""#,
|
||||||
|
r#""loop_count": 1000000"#,
|
||||||
|
r#""hot_loop_count": 10000000"#,
|
||||||
|
r#""checksum_metadata""#,
|
||||||
|
r#""expected_checksum""#,
|
||||||
|
r#""hot_expected_checksum""#,
|
||||||
|
r#""required_files""#,
|
||||||
|
r#""path": "benchmark.json""#,
|
||||||
|
r#""path": "run.py""#,
|
||||||
|
r#""path": "slovo.toml""#,
|
||||||
|
r#""path": "src/main.slo""#,
|
||||||
|
r#""status": "present""#,
|
||||||
|
r#""implementation_slots""#,
|
||||||
|
r#""loop_count_source": "stdin""#,
|
||||||
|
] {
|
||||||
|
assert_contains(&stdout, needle);
|
||||||
|
}
|
||||||
|
|
||||||
|
for benchmark in BENCHMARKS {
|
||||||
|
assert_contains(&stdout, &format!(r#""name": "{}""#, benchmark));
|
||||||
|
assert_contains(&stdout, &format!(r#""directory": "{}""#, benchmark));
|
||||||
|
}
|
||||||
|
|
||||||
|
for implementation in IMPLEMENTATIONS {
|
||||||
|
assert_contains(&stdout, &format!(r#""name": "{}""#, implementation));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn run_suite_catalog(repo: &Path, python: &str) -> Output {
|
||||||
|
Command::new(python)
|
||||||
|
.arg("benchmarks/runner.py")
|
||||||
|
.arg("--suite-list")
|
||||||
|
.arg("--json")
|
||||||
|
.current_dir(repo)
|
||||||
|
.output()
|
||||||
|
.unwrap_or_else(|err| {
|
||||||
|
panic!(
|
||||||
|
"run `{}` benchmarks/runner.py --suite-list --json: {}",
|
||||||
|
python, err
|
||||||
|
)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn python_command() -> String {
|
||||||
|
if let Some(python) = env::var_os("PYTHON") {
|
||||||
|
return python.to_string_lossy().into_owned();
|
||||||
|
}
|
||||||
|
|
||||||
|
for candidate in ["python3", "python"] {
|
||||||
|
if Command::new(candidate)
|
||||||
|
.arg("--version")
|
||||||
|
.output()
|
||||||
|
.map(|output| output.status.success())
|
||||||
|
.unwrap_or(false)
|
||||||
|
{
|
||||||
|
return candidate.to_string();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
panic!("benchmark suite catalog test requires python3 or python")
|
||||||
|
}
|
||||||
|
|
||||||
|
fn assert_success(context: &str, output: &Output) {
|
||||||
|
let stdout = String::from_utf8_lossy(&output.stdout);
|
||||||
|
let stderr = String::from_utf8_lossy(&output.stderr);
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
output.status.success(),
|
||||||
|
"{} failed\nstdout:\n{}\nstderr:\n{}",
|
||||||
|
context,
|
||||||
|
stdout,
|
||||||
|
stderr
|
||||||
|
);
|
||||||
|
assert!(stderr.is_empty(), "{} wrote stderr:\n{}", context, stderr);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn assert_contains(haystack: &str, needle: &str) {
|
||||||
|
assert!(
|
||||||
|
haystack.contains(needle),
|
||||||
|
"suite catalog output missing `{}`\nstdout:\n{}",
|
||||||
|
needle,
|
||||||
|
haystack
|
||||||
|
);
|
||||||
|
}
|
||||||
@ -379,6 +379,38 @@ Why thirteenth: diagnostics already have a machine schema and broad golden
|
|||||||
coverage, but the compatibility policy and current code inventory need one
|
coverage, but the compatibility policy and current code inventory need one
|
||||||
central reference before future tooling or migration work builds on them.
|
central reference before future tooling or migration work builds on them.
|
||||||
|
|
||||||
|
### 14. Benchmark Suite Catalog And Metadata Gate
|
||||||
|
|
||||||
|
Goal: document the existing benchmark suite inventory and metadata listing
|
||||||
|
commands before benchmark tooling grows additional consumers.
|
||||||
|
|
||||||
|
Work:
|
||||||
|
|
||||||
|
- add [`benchmarks/README.md`](../benchmarks/README.md) as the top-level
|
||||||
|
benchmark suite catalog
|
||||||
|
- document `python3 benchmarks/runner.py --suite-list` for the non-JSON suite
|
||||||
|
inventory
|
||||||
|
- document `python3 benchmarks/runner.py --suite-list --json` for beta tooling
|
||||||
|
metadata
|
||||||
|
- list the current suite inventory without adding new benchmark kernels
|
||||||
|
- state that benchmark timings are local-machine evidence only
|
||||||
|
- keep suite-list JSON beta-scoped rather than a stable public schema
|
||||||
|
- keep timing publication, performance thresholds, source-language/runtime/
|
||||||
|
stdlib/API/diagnostic-output changes, and ABI/layout changes out of scope
|
||||||
|
|
||||||
|
Released in `1.0.0-beta.14`:
|
||||||
|
[`benchmarks/README.md`](../benchmarks/README.md) now catalogs the current
|
||||||
|
benchmark suite, documents the root suite-list commands, records local-machine
|
||||||
|
evidence policy, and names the explicit exclusions. The release is
|
||||||
|
documentation/tooling metadata only; it does not add kernels, publish timing
|
||||||
|
numbers, define performance thresholds, define a stable JSON schema, or change
|
||||||
|
the source language, runtime, stdlib/API surface, diagnostics, or ABI/layout
|
||||||
|
behavior.
|
||||||
|
|
||||||
|
Why fourteenth: the benchmark suite is already part of the public monorepo, but
|
||||||
|
its suite-level inventory and metadata boundary need one central reference
|
||||||
|
before future tooling can rely on it.
|
||||||
|
|
||||||
## Stable `1.0.0` Gate
|
## Stable `1.0.0` Gate
|
||||||
|
|
||||||
Slovo should not become stable until all of these are true:
|
Slovo should not become stable until all of these are true:
|
||||||
@ -390,6 +422,7 @@ Slovo should not become stable until all of these are true:
|
|||||||
- release gates are reproducible on a clean checkout
|
- release gates are reproducible on a clean checkout
|
||||||
- diagnostics and formatter output are stable for promoted features
|
- diagnostics and formatter output are stable for promoted features
|
||||||
- performance publications are repeatable and labeled as local-machine evidence
|
- performance publications are repeatable and labeled as local-machine evidence
|
||||||
|
- benchmark metadata promoted to stable, if any, has an explicit schema policy
|
||||||
- the language can build useful local CLI tools, libraries, file-processing
|
- the language can build useful local CLI tools, libraries, file-processing
|
||||||
programs, and basic host-interaction programs without undocumented behavior
|
programs, and basic host-interaction programs without undocumented behavior
|
||||||
|
|
||||||
|
|||||||
@ -10,11 +10,43 @@ integration/readiness release, not the first real beta.
|
|||||||
|
|
||||||
## Unreleased
|
## Unreleased
|
||||||
|
|
||||||
Next scoped Glagol work is expected to continue after the `1.0.0-beta.13`
|
Next scoped Glagol work is expected to continue after the `1.0.0-beta.14`
|
||||||
diagnostic catalog and schema policy hardening update.
|
benchmark suite catalog and metadata gate.
|
||||||
|
|
||||||
No unreleased compiler scope is committed here yet.
|
No unreleased compiler scope is committed here yet.
|
||||||
|
|
||||||
|
## 1.0.0-beta.14
|
||||||
|
|
||||||
|
Release label: `1.0.0-beta.14`
|
||||||
|
|
||||||
|
Release date: 2026-05-22
|
||||||
|
|
||||||
|
Release state: benchmark suite catalog and metadata gate
|
||||||
|
|
||||||
|
### Summary
|
||||||
|
|
||||||
|
The beta.14 compiler-side contract is tooling-only benchmark catalog hardening.
|
||||||
|
It keeps benchmark execution local and unchanged while adding deterministic
|
||||||
|
suite-level metadata for release verification.
|
||||||
|
|
||||||
|
- Bump the `glagol` compiler package version to `1.0.0-beta.14`.
|
||||||
|
- Add `python3 benchmarks/runner.py --suite-list --json` for deterministic
|
||||||
|
suite-level benchmark listing and verification while preserving normal
|
||||||
|
per-benchmark `run.py` execution through the shared runner.
|
||||||
|
- Emit all 10 current benchmark names and directories, timing modes, cold/hot
|
||||||
|
loop counts, checksum metadata, required scaffold-file status,
|
||||||
|
implementation slots, and the local-only timing disclaimer.
|
||||||
|
- Add focused `benchmark_suite_catalog_beta14` coverage requiring byte-stable
|
||||||
|
suite catalog output across two runs and the current 10-benchmark inventory.
|
||||||
|
- Run the focused beta.14 suite catalog test in `scripts/release-gate.sh`
|
||||||
|
before the full compiler test suite.
|
||||||
|
|
||||||
|
### Explicit Deferrals
|
||||||
|
|
||||||
|
This release does not implement `glagol bench`, timing-result publication, a
|
||||||
|
new benchmark kernel, runtime changes, source-language changes, a stable JSON
|
||||||
|
schema claim, benchmark thresholds, or cross-machine performance claims.
|
||||||
|
|
||||||
## 1.0.0-beta.13
|
## 1.0.0-beta.13
|
||||||
|
|
||||||
Release label: `1.0.0-beta.13`
|
Release label: `1.0.0-beta.13`
|
||||||
|
|||||||
@ -22,8 +22,8 @@ general-purpose beta release.
|
|||||||
|
|
||||||
A Glagol feature is done only when it has parser/lowerer support, checker behavior, diagnostics for invalid forms, backend behavior or explicit unsupported diagnostics, and tests.
|
A Glagol feature is done only when it has parser/lowerer support, checker behavior, diagnostics for invalid forms, backend behavior or explicit unsupported diagnostics, and tests.
|
||||||
|
|
||||||
Current stage: `1.0.0-beta.13`, released on 2026-05-22 as a diagnostic
|
Current stage: `1.0.0-beta.14`, released on 2026-05-22 as a benchmark suite
|
||||||
catalog and schema policy hardening update. It keeps the `1.0.0-beta`
|
catalog and metadata gate update. It keeps the `1.0.0-beta`
|
||||||
language/compiler support baseline and includes the
|
language/compiler support baseline and includes the
|
||||||
`1.0.0-beta.1` tooling hardening release, the `1.0.0-beta.2` runtime/resource
|
`1.0.0-beta.1` tooling hardening release, the `1.0.0-beta.2` runtime/resource
|
||||||
foundation release, the `1.0.0-beta.3` standard-library stabilization release,
|
foundation release, the `1.0.0-beta.3` standard-library stabilization release,
|
||||||
@ -55,9 +55,14 @@ source-language runtime behavior. The beta.13 tooling/docs-only diagnostics
|
|||||||
slice centralizes the `slovo.diagnostic` schema name/version constants and
|
slice centralizes the `slovo.diagnostic` schema name/version constants and
|
||||||
adds structural gates for S-expression diagnostics, `--json-diagnostics`, and
|
adds structural gates for S-expression diagnostics, `--json-diagnostics`, and
|
||||||
artifact-manifest diagnostics metadata without changing emitted machine
|
artifact-manifest diagnostics metadata without changing emitted machine
|
||||||
diagnostic shape.
|
diagnostic shape. The beta.14 tooling-only benchmark slice adds deterministic
|
||||||
|
suite-level local benchmark listing and release-gate coverage for
|
||||||
|
`python3 benchmarks/runner.py --suite-list --json`, including all current
|
||||||
|
benchmark directories, timing modes, cold/hot loop counts, checksum metadata,
|
||||||
|
required scaffold-file status, implementation slots, and a local-only timing
|
||||||
|
disclaimer without publishing timing results or claiming a stable JSON schema.
|
||||||
|
|
||||||
Next stage target: post-`1.0.0-beta.13` developer-experience, package, and
|
Next stage target: post-`1.0.0-beta.14` developer-experience, package, and
|
||||||
collection/generic planning. Generic vectors, generic collections, maps, sets,
|
collection/generic planning. Generic vectors, generic collections, maps, sets,
|
||||||
generic stdlib dispatch, runtime collection changes, collection unification,
|
generic stdlib dispatch, runtime collection changes, collection unification,
|
||||||
stable human diagnostic text, stable Markdown schema, LSP/watch protocols,
|
stable human diagnostic text, stable Markdown schema, LSP/watch protocols,
|
||||||
|
|||||||
@ -8,7 +8,7 @@ Historical `exp-*` releases listed here are experimental maturity milestones.
|
|||||||
The pushed tag `v2.0.0-beta.1` is historical. It is now documented as an
|
The pushed tag `v2.0.0-beta.1` is historical. It is now documented as an
|
||||||
experimental integration/readiness release, not as a beta maturity claim.
|
experimental integration/readiness release, not as a beta maturity claim.
|
||||||
|
|
||||||
The current release is `1.0.0-beta.13`, published on 2026-05-22. It keeps the
|
The current release is `1.0.0-beta.14`, published on 2026-05-22. It keeps the
|
||||||
`1.0.0-beta` language surface, includes the first post-beta tooling/install
|
`1.0.0-beta` language surface, includes the first post-beta tooling/install
|
||||||
hardening bundle from `1.0.0-beta.1`, and adds the first runtime/resource
|
hardening bundle from `1.0.0-beta.1`, and adds the first runtime/resource
|
||||||
foundation bundle from `1.0.0-beta.2` plus the first standard-library
|
foundation bundle from `1.0.0-beta.2` plus the first standard-library
|
||||||
@ -23,12 +23,49 @@ collection alias unification and generic reservation slice from
|
|||||||
`1.0.0-beta.10`, and the local package API documentation extension from
|
`1.0.0-beta.10`, and the local package API documentation extension from
|
||||||
`1.0.0-beta.11`, plus the concrete vector query and prefix parity slice from
|
`1.0.0-beta.11`, plus the concrete vector query and prefix parity slice from
|
||||||
`1.0.0-beta.12`, and the diagnostic catalog and schema policy slice from
|
`1.0.0-beta.12`, and the diagnostic catalog and schema policy slice from
|
||||||
`1.0.0-beta.13`.
|
`1.0.0-beta.13`, plus the benchmark suite catalog and metadata gate from
|
||||||
|
`1.0.0-beta.14`.
|
||||||
|
|
||||||
## Unreleased
|
## Unreleased
|
||||||
|
|
||||||
No unreleased language scope is committed here yet.
|
No unreleased language scope is committed here yet.
|
||||||
|
|
||||||
|
## 1.0.0-beta.14
|
||||||
|
|
||||||
|
Release label: `1.0.0-beta.14`
|
||||||
|
|
||||||
|
Release name: Benchmark Suite Catalog And Metadata Gate
|
||||||
|
|
||||||
|
Release date: 2026-05-22
|
||||||
|
|
||||||
|
Status: released beta documentation/tooling metadata update on the
|
||||||
|
`1.0.0-beta` language baseline.
|
||||||
|
|
||||||
|
`1.0.0-beta.14` documents the existing benchmark suite catalog and metadata
|
||||||
|
gate without changing the source language, runtime, standard library, API
|
||||||
|
surface, diagnostic output, compiler-known runtime names, or ABI/layout
|
||||||
|
behavior:
|
||||||
|
|
||||||
|
- Adds [`benchmarks/README.md`](../../benchmarks/README.md) as the top-level
|
||||||
|
catalog for the current ten benchmark suites:
|
||||||
|
`math-loop`, `branch-loop`, `parse-loop`, `array-index-loop`,
|
||||||
|
`string-eq-loop`, `array-struct-field-loop`,
|
||||||
|
`enum-struct-payload-loop`, `vec-i32-index-loop`,
|
||||||
|
`vec-string-eq-loop`, and `json-quote-loop`.
|
||||||
|
- Documents `python3 benchmarks/runner.py --suite-list` as the non-JSON suite
|
||||||
|
inventory command.
|
||||||
|
- Documents `python3 benchmarks/runner.py --suite-list --json` as the beta
|
||||||
|
tooling metadata form for local gates and adapters.
|
||||||
|
- States that benchmark timings are local-machine evidence only and publishes
|
||||||
|
no timing numbers.
|
||||||
|
- Records that suite-list JSON is beta tooling metadata, not a stable public
|
||||||
|
schema.
|
||||||
|
|
||||||
|
This release does not add benchmark kernels, publish timings, define
|
||||||
|
performance thresholds, define a stable JSON schema, change source-language,
|
||||||
|
runtime, stdlib/API, diagnostic-output, compiler ABI/layout, or optimizer
|
||||||
|
behavior, or make cross-machine performance claims.
|
||||||
|
|
||||||
## 1.0.0-beta.13
|
## 1.0.0-beta.13
|
||||||
|
|
||||||
Release label: `1.0.0-beta.13`
|
Release label: `1.0.0-beta.13`
|
||||||
|
|||||||
@ -10,46 +10,53 @@ Long-horizon planning lives in
|
|||||||
release train from the historical `v2.0.0-beta.1` tag toward and beyond the
|
release train from the historical `v2.0.0-beta.1` tag toward and beyond the
|
||||||
first real general-purpose beta Slovo contract.
|
first real general-purpose beta Slovo contract.
|
||||||
|
|
||||||
Current stage: `1.0.0-beta.13`, released on 2026-05-22 as a post-beta
|
Current stage: `1.0.0-beta.14`, released on 2026-05-22 as a post-beta
|
||||||
diagnostic catalog and schema policy update. It keeps the `1.0.0-beta`
|
benchmark suite catalog and metadata gate. It keeps the `1.0.0-beta` language
|
||||||
language contract and includes the `1.0.0-beta.1` tooling hardening release,
|
contract and includes the `1.0.0-beta.1` tooling hardening release, the
|
||||||
the `1.0.0-beta.2`
|
`1.0.0-beta.2` runtime/resource foundation release, the `1.0.0-beta.3`
|
||||||
runtime/resource foundation release, the `1.0.0-beta.3` standard-library
|
standard-library stabilization release, the `1.0.0-beta.4`
|
||||||
stabilization release, the `1.0.0-beta.4` language-usability diagnostics
|
language-usability diagnostics release, the `1.0.0-beta.5` package/workspace
|
||||||
release, the `1.0.0-beta.5` package/workspace discipline release, and a narrow
|
discipline release, and a narrow `std.net` source facade for blocking loopback
|
||||||
`std.net` source facade for blocking loopback TCP client/server primitives over
|
TCP client/server primitives over opaque `i32` handles and concrete `result`
|
||||||
opaque `i32` handles and concrete `result` families, plus a narrow `std.json`
|
families, plus a narrow `std.json` source facade for compact JSON text
|
||||||
source facade for compact JSON text construction, plus top-level module-local
|
construction, plus top-level module-local transparent aliases for supported
|
||||||
transparent aliases for supported concrete target types, plus local alias use
|
concrete target types, plus local alias use inside the current concrete vector,
|
||||||
inside the current concrete vector, option, and result facades, plus a
|
option, and result facades, plus a generated standard-library API catalog that
|
||||||
generated standard-library API catalog that lists exact exported helper
|
lists exact exported helper signatures with module-local aliases normalized to
|
||||||
signatures with module-local aliases normalized to concrete public types, plus
|
concrete public types, plus `glagol symbols` deterministic source metadata for
|
||||||
`glagol symbols` deterministic source metadata for files, projects, and
|
files, projects, and workspaces, plus
|
||||||
workspaces, plus `glagol doc <file|project|workspace> -o <dir>` public API
|
`glagol doc <file|project|workspace> -o <dir>` public API sections for local
|
||||||
sections for local packages/modules with exact exported function signatures,
|
packages/modules with exact exported function signatures, exported struct
|
||||||
exported struct fields, exported enum variants/payload types, non-export
|
fields, exported enum variants/payload types, non-export filtering, and
|
||||||
filtering, and module-local alias normalization, plus source-authored concrete
|
module-local alias normalization, plus source-authored concrete vector helper
|
||||||
vector helper parity: `std.vec_i64` gains `count_of`, `starts_with`,
|
parity: `std.vec_i64` gains `count_of`, `starts_with`, `without_prefix`,
|
||||||
`without_prefix`, `ends_with`, and `without_suffix`, and `std.vec_f64` gains
|
`ends_with`, and `without_suffix`, and `std.vec_f64` gains `count_of`, plus
|
||||||
`count_of`, plus [`docs/language/DIAGNOSTICS.md`](DIAGNOSTICS.md) as the beta
|
[`docs/language/DIAGNOSTICS.md`](DIAGNOSTICS.md) as the beta
|
||||||
`slovo.diagnostic` version `1` policy and current golden diagnostic code
|
`slovo.diagnostic` version `1` policy and current golden diagnostic code
|
||||||
catalog. JSON parsing, recursive JSON values, executable generics, generic
|
catalog, plus [`../../benchmarks/README.md`](../../benchmarks/README.md) as
|
||||||
aliases, parameterized aliases, cross-module alias visibility, maps/sets,
|
the beta-scoped benchmark suite catalog documenting the current suite
|
||||||
traits, inference, monomorphization, iterators, runtime changes for generic
|
inventory, `python3 benchmarks/runner.py --suite-list`, and
|
||||||
collections, DNS, TLS, UDP, async IO, non-loopback binding, HTTP frameworks,
|
`python3 benchmarks/runner.py --suite-list --json`. Benchmark timings remain
|
||||||
rich host-error ADTs, stable ABI/layout, stable Markdown schema, stable
|
local-machine evidence only, and suite-list JSON is beta tooling metadata
|
||||||
stdlib/API compatibility freeze, LSP/watch, SARIF/daemon protocols,
|
rather than a stable public schema. JSON parsing, recursive JSON values,
|
||||||
stable `1.0.0` diagnostics freeze, re-exports/globs/hierarchical modules,
|
executable generics, generic aliases, parameterized aliases, cross-module alias
|
||||||
mutable vectors, slice/view APIs, new runtime names, performance claims, and
|
visibility, maps/sets, traits, inference, monomorphization, iterators, runtime
|
||||||
package registry semantics remain deferred.
|
changes for generic collections, DNS, TLS, UDP, async IO, non-loopback binding,
|
||||||
|
HTTP frameworks, rich host-error ADTs, stable ABI/layout, stable Markdown
|
||||||
|
schema, stable stdlib/API compatibility freeze, LSP/watch, SARIF/daemon
|
||||||
|
protocols, stable `1.0.0` diagnostics freeze, re-exports/globs/hierarchical
|
||||||
|
modules, mutable vectors, slice/view APIs, new runtime names, timing
|
||||||
|
publication, performance claims, stable benchmark JSON schema, and package
|
||||||
|
registry semantics remain deferred.
|
||||||
|
|
||||||
Next stage target: continue after `1.0.0-beta.13` from developer-experience,
|
Next stage target: continue after `1.0.0-beta.14` from developer-experience,
|
||||||
package, and reserved generic/map/set planning without claiming executable
|
package, benchmark metadata, and reserved generic/map/set planning without
|
||||||
generics, an LSP/watch protocol, SARIF/daemon protocol, stable Markdown
|
claiming executable generics, an LSP/watch protocol, SARIF/daemon protocol,
|
||||||
schema, registry semantics, stable `1.0.0` diagnostics freeze, or stable
|
stable Markdown schema, registry semantics, stable benchmark JSON schema,
|
||||||
standard-library/API compatibility freeze, mutable vectors, slice/view APIs,
|
stable `1.0.0` diagnostics freeze, or stable standard-library/API
|
||||||
new runtime names, performance claims, maps/sets, or iterators until the exact
|
compatibility freeze, mutable vectors, slice/view APIs, new runtime names,
|
||||||
scope is frozen from the manifest and roadmap.
|
performance claims, maps/sets, or iterators until the exact scope is frozen
|
||||||
|
from the manifest and roadmap.
|
||||||
|
|
||||||
The final experimental precursor scope is `exp-125`, defined in
|
The final experimental precursor scope is `exp-125`, defined in
|
||||||
`.llm/EXP_125_UNSIGNED_U32_U64_NUMERIC_AND_STDLIB_BREADTH_ALPHA.md`. Its
|
`.llm/EXP_125_UNSIGNED_U32_U64_NUMERIC_AND_STDLIB_BREADTH_ALPHA.md`. Its
|
||||||
|
|||||||
@ -5,7 +5,8 @@ Status: living beta contract for `1.0.0-beta`, with the post-beta
|
|||||||
foundation update, `1.0.0-beta.10` developer-experience API discovery update,
|
foundation update, `1.0.0-beta.10` developer-experience API discovery update,
|
||||||
`1.0.0-beta.11` local package API documentation update, and `1.0.0-beta.12`
|
`1.0.0-beta.11` local package API documentation update, and `1.0.0-beta.12`
|
||||||
concrete vector query and prefix parity update, and `1.0.0-beta.13`
|
concrete vector query and prefix parity update, and `1.0.0-beta.13`
|
||||||
diagnostic catalog and schema policy update. The language contract integrates
|
diagnostic catalog and schema policy update, and `1.0.0-beta.14` benchmark
|
||||||
|
suite catalog and metadata gate. The language contract integrates
|
||||||
promoted language slices through `exp-125` and the historical publication
|
promoted language slices through `exp-125` and the historical publication
|
||||||
baseline through `exp-123`. `1.0.0-beta` is the first real general-purpose
|
baseline through `exp-123`. `1.0.0-beta` is the first real general-purpose
|
||||||
beta release. `exp-125` completed the unsigned numeric and stdlib breadth
|
beta release. `exp-125` completed the unsigned numeric and stdlib breadth
|
||||||
@ -157,6 +158,16 @@ Current v1 release surface and explicit experimental targets:
|
|||||||
current golden diagnostic code catalog; this is docs/tooling policy only, not
|
current golden diagnostic code catalog; this is docs/tooling policy only, not
|
||||||
a source-language/runtime change, LSP/watch contract, SARIF/daemon protocol,
|
a source-language/runtime change, LSP/watch contract, SARIF/daemon protocol,
|
||||||
stable Markdown schema, or stable `1.0.0` diagnostics freeze
|
stable Markdown schema, or stable `1.0.0` diagnostics freeze
|
||||||
|
- `1.0.0-beta.14` benchmark suite catalog and metadata target:
|
||||||
|
[`benchmarks/README.md`](../../benchmarks/README.md) documents the existing
|
||||||
|
benchmark suite inventory and root suite-list commands:
|
||||||
|
`python3 benchmarks/runner.py --suite-list` and
|
||||||
|
`python3 benchmarks/runner.py --suite-list --json`; benchmark timing remains
|
||||||
|
local-machine evidence only, and suite-list JSON is beta tooling metadata
|
||||||
|
only, not a stable public schema; this is docs/tooling metadata only, not new
|
||||||
|
benchmark kernels, timing publication, performance thresholds,
|
||||||
|
source-language/runtime/stdlib/API changes, diagnostic-output changes, or
|
||||||
|
ABI/layout changes
|
||||||
- `exp-1` owned runtime strings: compiler-known `std.string.concat` accepts two
|
- `exp-1` owned runtime strings: compiler-known `std.string.concat` accepts two
|
||||||
`string` values and returns an immutable runtime-owned `string`; existing
|
`string` values and returns an immutable runtime-owned `string`; existing
|
||||||
string equality, length, printing, locals, parameters, returns, and calls work
|
string equality, length, printing, locals, parameters, returns, and calls work
|
||||||
@ -1379,6 +1390,39 @@ protocols, stable Markdown schema, stable `1.0.0` diagnostics freeze,
|
|||||||
source-map/debug-metadata contracts, localized diagnostic text, automated
|
source-map/debug-metadata contracts, localized diagnostic text, automated
|
||||||
machine fix-its, or a compiler-emitted diagnostic catalog artifact.
|
machine fix-its, or a compiler-emitted diagnostic catalog artifact.
|
||||||
|
|
||||||
|
### 4.4.11 Post-Beta Benchmark Suite Catalog And Metadata Gate
|
||||||
|
|
||||||
|
Status: released in `1.0.0-beta.14` as a docs/tooling metadata update for the
|
||||||
|
existing benchmark suite.
|
||||||
|
|
||||||
|
`1.0.0-beta.14` adds
|
||||||
|
[`benchmarks/README.md`](../../benchmarks/README.md) as the top-level
|
||||||
|
benchmark suite catalog. The catalog records the current suite inventory,
|
||||||
|
base/hot-loop checksum metadata, local evidence policy, and explicit
|
||||||
|
exclusions.
|
||||||
|
|
||||||
|
The root suite-list commands are documented as:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 benchmarks/runner.py --suite-list
|
||||||
|
python3 benchmarks/runner.py --suite-list --json
|
||||||
|
```
|
||||||
|
|
||||||
|
The non-JSON listing is local review output. The JSON listing is beta tooling
|
||||||
|
metadata for local gates and adapters. The field set is intentionally not a
|
||||||
|
stable public schema in this release.
|
||||||
|
|
||||||
|
The source language, typed core, runtime, standard library, public API surface,
|
||||||
|
compiler diagnostics, diagnostic output shape, compiler-known runtime names,
|
||||||
|
ABI/layout behavior, and optimizer contract are unchanged by this target.
|
||||||
|
Benchmark timings remain local-machine evidence only and no timing numbers are
|
||||||
|
published by the release contract.
|
||||||
|
|
||||||
|
This target explicitly does not add benchmark kernels, implementation language
|
||||||
|
slots, timing publication, performance thresholds, cross-machine performance
|
||||||
|
claims, a stable benchmark JSON schema, source-language/runtime/stdlib/API
|
||||||
|
changes, diagnostic-output changes, or ABI/layout guarantees.
|
||||||
|
|
||||||
## 4.5 v2.0.0-beta.1 Experimental Integration Readiness
|
## 4.5 v2.0.0-beta.1 Experimental Integration Readiness
|
||||||
|
|
||||||
Status: current experimental Slovo-side release contract, released 2026-05-17.
|
Status: current experimental Slovo-side release contract, released 2026-05-17.
|
||||||
|
|||||||
@ -6,7 +6,7 @@ Do not edit this file by hand.
|
|||||||
## Stability Tiers
|
## Stability Tiers
|
||||||
|
|
||||||
- `beta-supported`: exported from `lib/std` and covered by source-search, promotion, or facade gates in the current beta line.
|
- `beta-supported`: exported from `lib/std` and covered by source-search, promotion, or facade gates in the current beta line.
|
||||||
- `experimental`: not used for exported `lib/std` helpers in `1.0.0-beta.13`; future releases may mark new helpers this way before they graduate.
|
- `experimental`: not used for exported `lib/std` helpers in `1.0.0-beta.14`; future releases may mark new helpers this way before they graduate.
|
||||||
- `internal`: helper names that are not exported from their module; they are intentionally omitted from this catalog.
|
- `internal`: helper names that are not exported from their module; they are intentionally omitted from this catalog.
|
||||||
|
|
||||||
The catalog is a beta API discovery aid, not a stable `1.0.0` standard-library freeze.
|
The catalog is a beta API discovery aid, not a stable `1.0.0` standard-library freeze.
|
||||||
|
|||||||
@ -64,6 +64,7 @@ fi
|
|||||||
cd "${compiler_dir}"
|
cd "${compiler_dir}"
|
||||||
cargo fmt --check
|
cargo fmt --check
|
||||||
cargo test --test diagnostics_schema_beta13
|
cargo test --test diagnostics_schema_beta13
|
||||||
|
cargo test --test benchmark_suite_catalog_beta14
|
||||||
# Full cargo test includes unignored integration gates such as dx_v1_7,
|
# Full cargo test includes unignored integration gates such as dx_v1_7,
|
||||||
# beta_v2_0_0_beta_1, and beta_1_0_0.
|
# beta_v2_0_0_beta_1, and beta_1_0_0.
|
||||||
cargo test
|
cargo test
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user