Release 1.0.0-beta.14 benchmark suite catalog
This commit is contained in:
sanjin
parent
acbe58f70e
commit
d3e628553f
69
.llm/BETA_14_BENCHMARK_SUITE_CATALOG_AND_METADATA_GATE.md
Normal file
69
.llm/BETA_14_BENCHMARK_SUITE_CATALOG_AND_METADATA_GATE.md
Normal file
@ -0,0 +1,69 @@
|
||||
# 1.0.0-beta.14 Benchmark Suite Catalog And Metadata Gate
|
||||
|
||||
Status: release scope for `1.0.0-beta.14`.
|
||||
|
||||
`1.0.0-beta.14` is a docs/tooling metadata slice for the existing benchmark
|
||||
suite. It keeps the `1.0.0-beta` source language, typed core, runtime,
|
||||
standard library, API surface, compiler diagnostics, diagnostic output shape,
|
||||
ABI/layout behavior, and compiler-known runtime names unchanged.
|
||||
|
||||
## Scope
|
||||
|
||||
- Add `benchmarks/README.md` as the top-level benchmark suite catalog.
|
||||
- Document `python3 benchmarks/runner.py --suite-list` as the non-JSON suite
|
||||
inventory command.
|
||||
- Document `python3 benchmarks/runner.py --suite-list --json` as the beta
|
||||
tooling metadata form for the same inventory.
|
||||
- Record the current suite inventory:
|
||||
`math-loop`, `branch-loop`, `parse-loop`, `array-index-loop`,
|
||||
`string-eq-loop`, `array-struct-field-loop`,
|
||||
`enum-struct-payload-loop`, `vec-i32-index-loop`,
|
||||
`vec-string-eq-loop`, and `json-quote-loop`.
|
||||
- Document that benchmark timings are local-machine evidence only.
|
||||
- Update README, language roadmap, language spec, release notes, post-beta
|
||||
roadmap, and this release contract to introduce beta14.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Public docs name current release/stage `1.0.0-beta.14`.
|
||||
- `benchmarks/README.md` explains both root suite-list commands:
|
||||
`python3 benchmarks/runner.py --suite-list` and
|
||||
`python3 benchmarks/runner.py --suite-list --json`.
|
||||
- The suite catalog lists the current benchmark suite inventory without adding
|
||||
or removing benchmark kernels.
|
||||
- The suite catalog verifies required scaffold files for each suite:
|
||||
`benchmark.json`, `run.py`, `slovo.toml`, and `src/main.slo`.
|
||||
- The docs say timing output is local-machine evidence only and do not publish
|
||||
timing numbers.
|
||||
- The JSON suite listing is described as beta tooling metadata, not a stable
|
||||
public schema.
|
||||
- Explicit exclusions include no new benchmark kernels, no timing publication,
|
||||
no performance thresholds, no stable JSON schema, no source-language change,
|
||||
no runtime change, no stdlib/API change, no diagnostic-output change, and no
|
||||
ABI/layout change.
|
||||
|
||||
## Explicit Non-Scope
|
||||
|
||||
- no new benchmark kernels or implementation language slots
|
||||
- no benchmark timing publication
|
||||
- no release performance threshold
|
||||
- no stable JSON schema or stable benchmark metadata compatibility promise
|
||||
- no source-language syntax, typed-core, lowering, runtime, stdlib, or API
|
||||
change
|
||||
- no compiler diagnostic or diagnostic-output shape change
|
||||
- no ABI/layout or optimizer guarantee
|
||||
- no worker-owned release publication work before controller review and gates
|
||||
|
||||
## Expected Controller Verification
|
||||
|
||||
- Run lightweight docs checks:
|
||||
- `git diff --check -- README.md benchmarks/README.md docs/POST_BETA_ROADMAP.md docs/language/SPEC-v1.md docs/language/ROADMAP.md docs/language/RELEASE_NOTES.md .llm/BETA_14_BENCHMARK_SUITE_CATALOG_AND_METADATA_GATE.md`
|
||||
- run an `rg` check for stale beta13-only current-stage phrasing across the
|
||||
same touched docs
|
||||
- Smoke the metadata commands:
|
||||
- `python3 benchmarks/runner.py --suite-list`
|
||||
- `python3 benchmarks/runner.py --suite-list --json`
|
||||
- Run focused compiler/tooling checks:
|
||||
- `cargo test --test benchmark_suite_catalog_beta14`
|
||||
- `cargo test --test benchmark_math_loop_scaffold`
|
||||
- Do not commit, tag, push, or run release publication from this worker scope.
|
||||
76
.llm/reviews/BETA_14_RELEASE_REVIEW.md
Normal file
76
.llm/reviews/BETA_14_RELEASE_REVIEW.md
Normal file
@ -0,0 +1,76 @@
|
||||
# 1.0.0-beta.14 Release Review
|
||||
|
||||
Status: ready for publication after controller release gate.
|
||||
|
||||
## Verdict
|
||||
|
||||
No blocking findings found for the benchmark suite catalog and metadata gate.
|
||||
|
||||
## Findings
|
||||
|
||||
No blocking findings.
|
||||
|
||||
Non-blocking note: `compiler/tests/benchmark_suite_catalog_beta14.rs` verifies
|
||||
byte-stable JSON, the current 10 benchmark names/directories, top-level counts,
|
||||
required scaffold-file status, missing-file lists, checksum metadata presence,
|
||||
timing mode strings, and implementation slot names. The current runner output
|
||||
also includes `run_args`, `source_stem`, and per-implementation source paths.
|
||||
If those fields become contractual in a later release, the focused gate should
|
||||
parse the JSON and assert them per benchmark.
|
||||
|
||||
## Scope Checked
|
||||
|
||||
- Public release docs in scope: `README.md`, `benchmarks/README.md`,
|
||||
`docs/POST_BETA_ROADMAP.md`, `docs/language/SPEC-v1.md`,
|
||||
`docs/language/ROADMAP.md`, `docs/language/RELEASE_NOTES.md`,
|
||||
`docs/compiler/ROADMAP.md`, `docs/compiler/RELEASE_NOTES.md`, and
|
||||
`.llm/BETA_14_BENCHMARK_SUITE_CATALOG_AND_METADATA_GATE.md`.
|
||||
- Tooling and tests in scope: `benchmarks/runner.py`,
|
||||
`compiler/tests/benchmark_suite_catalog_beta14.rs`,
|
||||
`compiler/tests/benchmark_math_loop_scaffold.rs`,
|
||||
`scripts/release-gate.sh`, `compiler/Cargo.toml`, and
|
||||
`compiler/Cargo.lock`.
|
||||
- Release-facing current-stage wording points at `1.0.0-beta.14`. Remaining
|
||||
beta13 references are historical beta13 diagnostics/release notes, the
|
||||
existing `diagnostics_schema_beta13` gate, or older review/contract files.
|
||||
- Benchmark docs and runner output consistently keep timing local-machine-only,
|
||||
do not publish timing numbers, do not define performance thresholds, and do
|
||||
not claim a stable benchmark JSON schema.
|
||||
- The top-level catalog documents the current 10 suites and all suite-list
|
||||
commands. Relative links to `benchmarks/README.md` resolve from the files
|
||||
that introduce them.
|
||||
- The runner preserves per-suite `run.py --list --json` behavior and adds root
|
||||
`benchmarks/runner.py --suite-list` and `--suite-list --json` metadata.
|
||||
- `scripts/release-gate.sh` now runs `cargo test --test
|
||||
benchmark_suite_catalog_beta14` before the full compiler test suite.
|
||||
- Cargo package version and lockfile version both read `1.0.0-beta.14`.
|
||||
- Focused private/local publication text scan found no machine-local paths,
|
||||
private checkout/user names, private remotes, or local IP text in the reviewed
|
||||
release surface.
|
||||
|
||||
## Verification Commands And Results
|
||||
|
||||
- `python3 benchmarks/runner.py --suite-list --json`: passed. Output reported
|
||||
`benchmark_count: 10`, `implementation_slots: 60`, status `ok`, cold/hot
|
||||
timing modes, required scaffold-file status with no missing files, checksum
|
||||
metadata, runtime args, implementation source paths, and the local-only
|
||||
timing disclaimer.
|
||||
- `python3 benchmarks/runner.py --suite-list`: passed.
|
||||
- `python3 benchmarks/math-loop/run.py --list --json`: passed, confirming the
|
||||
existing per-benchmark list mode still works through the shared runner.
|
||||
- `python3 benchmarks/math-loop/run.py --suite-list --json`: passed, confirming
|
||||
suite listing also resolves correctly through a per-benchmark wrapper.
|
||||
- `cargo test --test benchmark_suite_catalog_beta14`: passed, 1 test.
|
||||
- `cargo test --test benchmark_math_loop_scaffold`: passed, 1 test.
|
||||
- `git diff --check`: passed.
|
||||
- `git diff --check -- README.md benchmarks/README.md docs/POST_BETA_ROADMAP.md docs/language/SPEC-v1.md docs/language/ROADMAP.md docs/language/RELEASE_NOTES.md .llm/BETA_14_BENCHMARK_SUITE_CATALOG_AND_METADATA_GATE.md`:
|
||||
passed.
|
||||
- `bash -n scripts/release-gate.sh`: passed.
|
||||
- `cargo fmt --check`: passed.
|
||||
- Focused `rg` stale-current-stage scan for beta13 current release/stage wording
|
||||
in the touched release docs produced no matches.
|
||||
- Focused `rg` private/local publication text scan over README, docs,
|
||||
benchmarks, `.llm`, compiler tests, and scripts produced no matches.
|
||||
|
||||
The full `./scripts/release-gate.sh` was not run during this review; the
|
||||
focused beta14 gate coverage and requested lightweight checks passed.
|
||||
58
README.md
58
README.md
@ -6,7 +6,7 @@ This repository is the canonical public monorepo for the language design,
|
||||
standard library source, compiler, runtime, examples, benchmarks, and technical
|
||||
documents.
|
||||
|
||||
Current release: `1.0.0-beta.13`.
|
||||
Current release: `1.0.0-beta.14`.
|
||||
|
||||
## Repository Layout
|
||||
|
||||
@ -24,7 +24,7 @@ scripts/ local release and document tooling
|
||||
|
||||
## Beta Scope
|
||||
|
||||
`1.0.0-beta.13` keeps the `1.0.0-beta` language baseline, includes the
|
||||
`1.0.0-beta.14` keeps the `1.0.0-beta` language baseline, includes the
|
||||
`1.0.0-beta.1` tooling/install hardening slice, the `1.0.0-beta.2`
|
||||
runtime/resource foundation bundle, the `1.0.0-beta.3` standard-library
|
||||
stabilization bundle, the `1.0.0-beta.4` language-usability diagnostics
|
||||
@ -35,7 +35,8 @@ alias foundation, the `1.0.0-beta.9` collection alias unification and
|
||||
generic reservation slice, the `1.0.0-beta.10` developer-experience API
|
||||
discovery slice, and the `1.0.0-beta.11` local package API documentation
|
||||
slice, plus the `1.0.0-beta.12` concrete vector query and prefix parity
|
||||
slice, and the `1.0.0-beta.13` diagnostic catalog and schema policy slice.
|
||||
slice, the `1.0.0-beta.13` diagnostic catalog and schema policy slice, and
|
||||
the `1.0.0-beta.14` benchmark suite catalog and metadata gate.
|
||||
The language baseline supports practical local command-line, file, and
|
||||
loopback-network programs with:
|
||||
|
||||
@ -73,23 +74,31 @@ S-expression/JSON relationship, required and optional fields, JSON-line
|
||||
discipline, source-less diagnostics, manifest diagnostic metadata,
|
||||
compatibility and migration classes, and the current golden diagnostic code
|
||||
catalog.
|
||||
The `1.0.0-beta.14` benchmark metadata slice documents the existing benchmark
|
||||
suite catalog in [`benchmarks/README.md`](benchmarks/README.md). It explains
|
||||
`python3 benchmarks/runner.py --suite-list` for the human-readable suite
|
||||
inventory and `python3 benchmarks/runner.py --suite-list --json` for beta
|
||||
tooling metadata, with required scaffold-file verification for each current
|
||||
suite. Benchmark timings remain local-machine evidence only; the JSON field
|
||||
set is not a stable public schema.
|
||||
|
||||
Still deferred before stable: executable generics, maps/sets, broad package
|
||||
registry semantics, stable Markdown schema, stable stdlib/API compatibility
|
||||
freeze, DNS/TLS/async networking, LSP/watch guarantees, SARIF and daemon
|
||||
protocols, stable `1.0.0` diagnostics freeze, re-exports/globs/hierarchical
|
||||
modules, mutable vectors, slice/view APIs, iterators, new compiler-known
|
||||
runtime names, stable ABI and layout, performance claims, and runtime changes
|
||||
for generic collections.
|
||||
runtime names, stable ABI and layout, performance claims, stable benchmark
|
||||
JSON metadata schema, and runtime changes for generic collections.
|
||||
|
||||
The next likely language slice after `1.0.0-beta.13` should continue from the
|
||||
developer-experience, package, and reserved generic/collection lanes without
|
||||
claiming executable generics, maps, sets, traits, inference, monomorphization,
|
||||
iterators, ABI stability, runtime changes, LSP/watch protocols, SARIF/daemon
|
||||
protocols, registry semantics, stable Markdown schema, a stable `1.0.0`
|
||||
diagnostics freeze, standard-library/API compatibility freeze, mutable
|
||||
vectors, slice/view APIs, new runtime names, or performance claims until the
|
||||
contract and gates are explicit.
|
||||
The next likely language slice after `1.0.0-beta.14` should continue from the
|
||||
developer-experience, package, benchmark metadata, and reserved
|
||||
generic/collection lanes without claiming executable generics, maps, sets,
|
||||
traits, inference, monomorphization, iterators, ABI stability, runtime changes,
|
||||
LSP/watch protocols, SARIF/daemon protocols, registry semantics, stable
|
||||
Markdown schema, stable benchmark JSON schema, a stable `1.0.0` diagnostics
|
||||
freeze, standard-library/API compatibility freeze, mutable vectors, slice/view
|
||||
APIs, new runtime names, or performance claims until the contract and gates
|
||||
are explicit.
|
||||
|
||||
## Build And Test
|
||||
|
||||
@ -332,6 +341,28 @@ This release does not change the source language, runtime, stdlib API,
|
||||
diagnostic output shape, compiler CLI, LSP/watch behavior, SARIF/daemon
|
||||
protocols, stable Markdown schema, or stable `1.0.0` diagnostics freeze.
|
||||
|
||||
## 1.0.0-beta.14 Benchmark Suite Catalog And Metadata Gate
|
||||
|
||||
The `1.0.0-beta.14` release documents the existing benchmark suite catalog as
|
||||
beta-scoped metadata tooling. It adds
|
||||
[`benchmarks/README.md`](benchmarks/README.md) with the current ten-suite
|
||||
inventory, local evidence policy, suite-list commands, and exclusions.
|
||||
|
||||
The root suite catalog commands are:
|
||||
|
||||
```bash
|
||||
python3 benchmarks/runner.py --suite-list
|
||||
python3 benchmarks/runner.py --suite-list --json
|
||||
```
|
||||
|
||||
The non-JSON listing is for local review. The JSON listing is beta tooling
|
||||
metadata for local gates and adapters, not a stable public schema.
|
||||
|
||||
This release does not add benchmark kernels, publish timing numbers, define
|
||||
performance thresholds, change the source language, runtime, stdlib/API
|
||||
surface, diagnostic output, compiler ABI/layout behavior, or make cross-machine
|
||||
performance claims.
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Language Manifest](docs/language/MANIFEST.md)
|
||||
@ -339,6 +370,7 @@ protocols, stable Markdown schema, or stable `1.0.0` diagnostics freeze.
|
||||
- [Diagnostics Policy](docs/language/DIAGNOSTICS.md)
|
||||
- [Local Package And Workspace Guide](docs/language/PACKAGES.md)
|
||||
- [Standard Library API Catalog](docs/language/STDLIB_API.md)
|
||||
- [Benchmark Suite Catalog](benchmarks/README.md)
|
||||
- [Compiler Manifest](docs/compiler/GLAGOL_COMPILER_MANIFEST.md)
|
||||
- [Post-Beta Roadmap](docs/POST_BETA_ROADMAP.md)
|
||||
- [Slovo Whitepaper](docs/papers/SLOVO_WHITEPAPER.md)
|
||||
|
||||
83
benchmarks/README.md
Normal file
83
benchmarks/README.md
Normal file
@ -0,0 +1,83 @@
|
||||
# Slovo Benchmark Suite Catalog
|
||||
|
||||
Release stage: `1.0.0-beta.14`.
|
||||
|
||||
The benchmark suite is beta-scoped local tooling. It catalogs deterministic
|
||||
same-machine comparison scaffolds for Slovo and sibling implementations. It
|
||||
does not publish benchmark results, set performance thresholds, define optimizer
|
||||
claims, or create cross-machine comparisons.
|
||||
|
||||
## Suite Listing
|
||||
|
||||
From the repository root, list the suite catalog for humans:
|
||||
|
||||
```bash
|
||||
python3 benchmarks/runner.py --suite-list
|
||||
```
|
||||
|
||||
List the same catalog as beta tooling metadata:
|
||||
|
||||
```bash
|
||||
python3 benchmarks/runner.py --suite-list --json
|
||||
```
|
||||
|
||||
The non-JSON listing is for local review. The JSON listing is for local gates
|
||||
and tooling adapters that need the current suite inventory. The JSON field set
|
||||
is not a stable public schema; it may change during beta releases.
|
||||
Both forms verify the required scaffold files for each suite:
|
||||
`benchmark.json`, `run.py`, `slovo.toml`, and `src/main.slo`.
|
||||
|
||||
Each suite still owns its local metadata and run commands:
|
||||
|
||||
```bash
|
||||
python3 benchmarks/<suite>/run.py --list
|
||||
python3 benchmarks/<suite>/run.py --list --json
|
||||
python3 benchmarks/<suite>/run.py --dry-run
|
||||
```
|
||||
|
||||
## Current Suites
|
||||
|
||||
All current suites provide Slovo, C, Rust, Python, Clojure, and Common
|
||||
Lisp/SBCL source slots. Missing local toolchains are skipped by the per-suite
|
||||
runner where possible.
|
||||
|
||||
| Suite | Focus | Base checksum | Hot-loop checksum | Runtime args |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `math-loop` | arithmetic and scalar accumulation | `5000001` | `50000001` | none |
|
||||
| `branch-loop` | deterministic branch-heavy integer loop | `1185071` | `220775` | none |
|
||||
| `parse-loop` | repeated signed decimal `i32` parsing | `345000001` | `450000001` | `12345` |
|
||||
| `array-index-loop` | immutable fixed-array indexing | `3875007` | `38750007` | none |
|
||||
| `string-eq-loop` | fixed string lookup and equality | `4600001` | `46000001` | `omega` |
|
||||
| `array-struct-field-loop` | fixed-array access through an immutable struct field | `3875011` | `38750011` | none |
|
||||
| `enum-struct-payload-loop` | enum payload matching with struct and array access | `3500013` | `35000013` | none |
|
||||
| `vec-i32-index-loop` | runtime-owned `(vec i32)` indexing | `3875007` | `38750007` | none |
|
||||
| `vec-string-eq-loop` | runtime-owned `(vec string)` lookup and equality | `4600001` | `46000001` | `omega` |
|
||||
| `json-quote-loop` | compact JSON string quoting | `15000001` | `150000001` | `slo"vo\path` |
|
||||
|
||||
The base loop count is `1000000` for every current suite. The hot-loop count is
|
||||
`10000000` for every current suite. The runner supplies loop counts and runtime
|
||||
arguments at execution time so native compilers cannot fold the work into a
|
||||
constant answer.
|
||||
|
||||
## Local Evidence Only
|
||||
|
||||
Benchmark output is local-machine evidence only:
|
||||
|
||||
- cold-process mode measures execution after each implementation has been built
|
||||
once; it does not include compile time
|
||||
- hot-loop mode is startup-amortized local evidence and reports total time plus
|
||||
normalized timing for the base loop count
|
||||
- Clojure timings include JVM and Clojure startup
|
||||
- Common Lisp timings include SBCL script startup
|
||||
- reported timings depend on the local CPU, OS, compiler versions, toolchain
|
||||
availability, thermal/load state, and runner configuration
|
||||
|
||||
This catalog intentionally publishes no timing numbers.
|
||||
|
||||
## Exclusions
|
||||
|
||||
`1.0.0-beta.14` does not add benchmark kernels, publish timings, define
|
||||
performance thresholds, define a stable JSON schema, change the Slovo source
|
||||
language, change runtime behavior, change standard-library or API contracts,
|
||||
change diagnostic output, change ABI/layout behavior, or make cross-machine
|
||||
performance claims.
|
||||
@ -18,6 +18,12 @@ from typing import Callable
|
||||
|
||||
TIMING_SCOPE = "local-machine comparison only"
|
||||
TIMING_MODES = ["cold-process", "hot-loop"]
|
||||
SUITE_NAME = "glagol-local-benchmark-suite"
|
||||
LOCAL_TIMING_DISCLAIMER = (
|
||||
"Local timing comparison only; not a published benchmark result and not a cross-machine performance claim."
|
||||
)
|
||||
REQUIRED_BENCHMARK_FILES = ["benchmark.json", "run.py", "slovo.toml", "src/main.slo"]
|
||||
EXPECTED_IMPLEMENTATION_NAMES = ["slovo", "c", "rust", "python", "clojure", "common_lisp"]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@ -52,6 +58,14 @@ class Implementation:
|
||||
|
||||
|
||||
def main(root: Path, argv: list[str]) -> int:
|
||||
if any(arg == "--suite-list" for arg in argv):
|
||||
return main_suite(root, argv)
|
||||
if not (root / "benchmark.json").is_file():
|
||||
parser = argparse.ArgumentParser(description="Shared local Glagol benchmark runner.")
|
||||
parser.add_argument("--suite-list", action="store_true", help="list suite metadata and exit")
|
||||
parser.add_argument("--json", action="store_true", help="emit JSON for --suite-list")
|
||||
parser.error("run from a benchmark run.py, or pass --suite-list at the benchmark suite root")
|
||||
|
||||
spec = read_spec(root)
|
||||
implementations = available_implementations(root, spec)
|
||||
parser = argparse.ArgumentParser(description=f"Run local {spec.name} timing comparisons.")
|
||||
@ -89,6 +103,25 @@ def main(root: Path, argv: list[str]) -> int:
|
||||
return 1 if any(result["status"] == "failed" for result in results) else 0
|
||||
|
||||
|
||||
def main_suite(root: Path, argv: list[str]) -> int:
|
||||
parser = argparse.ArgumentParser(description="List local Glagol benchmark suite metadata.")
|
||||
parser.add_argument("--suite-list", action="store_true", help="list suite metadata and exit")
|
||||
parser.add_argument("--json", action="store_true", help="emit JSON for suite metadata")
|
||||
args = parser.parse_args(argv)
|
||||
if not args.suite_list:
|
||||
parser.error("pass --suite-list to list benchmark suite metadata")
|
||||
|
||||
suite_root = resolve_suite_root(root)
|
||||
emit_suite_list(build_suite_catalog(suite_root), args.json)
|
||||
return 0
|
||||
|
||||
|
||||
def resolve_suite_root(root: Path) -> Path:
|
||||
if (root / "benchmark.json").is_file():
|
||||
return root.parent
|
||||
return root
|
||||
|
||||
|
||||
def read_spec(root: Path) -> BenchmarkSpec:
|
||||
data = json.loads((root / "benchmark.json").read_text(encoding="utf-8"))
|
||||
loop_count = int(data["loop_count"])
|
||||
@ -278,6 +311,143 @@ def select_implementations(implementations: list[Implementation], names: list[st
|
||||
return [impl for impl in implementations if impl.name in selected_names]
|
||||
|
||||
|
||||
def build_suite_catalog(suite_root: Path) -> dict[str, object]:
|
||||
benchmarks: list[dict[str, object]] = []
|
||||
implementation_slot_count = 0
|
||||
missing_required_files: list[str] = []
|
||||
missing_implementation_slots: list[str] = []
|
||||
|
||||
for root in suite_benchmark_roots(suite_root):
|
||||
spec = read_spec(root)
|
||||
implementations = available_implementations(root, spec)
|
||||
implementation_slot_count += len(implementations)
|
||||
benchmark = suite_benchmark_metadata(suite_root, root, spec, implementations)
|
||||
benchmarks.append(benchmark)
|
||||
|
||||
directory = str(benchmark["directory"])
|
||||
for required_file in benchmark["required_files"]:
|
||||
assert isinstance(required_file, dict)
|
||||
if required_file["status"] != "present":
|
||||
missing_required_files.append(f"{directory}/{required_file['path']}")
|
||||
|
||||
present_implementations = {
|
||||
str(implementation["name"])
|
||||
for implementation in benchmark["implementation_slots"]
|
||||
if isinstance(implementation, dict)
|
||||
}
|
||||
for expected in EXPECTED_IMPLEMENTATION_NAMES:
|
||||
if expected not in present_implementations:
|
||||
missing_implementation_slots.append(f"{directory}:{expected}")
|
||||
|
||||
return {
|
||||
"suite": SUITE_NAME,
|
||||
"timing_scope": TIMING_SCOPE,
|
||||
"timing_modes": TIMING_MODES,
|
||||
"timing_disclaimer": LOCAL_TIMING_DISCLAIMER,
|
||||
"benchmark_count": len(benchmarks),
|
||||
"benchmarks": benchmarks,
|
||||
"verification": {
|
||||
"status": "ok" if not missing_required_files and not missing_implementation_slots else "incomplete",
|
||||
"benchmark_metadata_files": len(benchmarks),
|
||||
"required_files": len(benchmarks) * len(REQUIRED_BENCHMARK_FILES),
|
||||
"missing_required_files": missing_required_files,
|
||||
"implementation_slots": implementation_slot_count,
|
||||
"expected_implementation_slots": len(benchmarks) * len(EXPECTED_IMPLEMENTATION_NAMES),
|
||||
"missing_implementation_slots": missing_implementation_slots,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def suite_benchmark_roots(suite_root: Path) -> list[Path]:
|
||||
return sorted(
|
||||
[path for path in suite_root.iterdir() if path.is_dir() and (path / "benchmark.json").is_file()],
|
||||
key=lambda path: path.name,
|
||||
)
|
||||
|
||||
|
||||
def suite_benchmark_metadata(
|
||||
suite_root: Path,
|
||||
root: Path,
|
||||
spec: BenchmarkSpec,
|
||||
implementations: list[Implementation],
|
||||
) -> dict[str, object]:
|
||||
return {
|
||||
"name": spec.name,
|
||||
"directory": str(root.relative_to(suite_root)),
|
||||
"source_stem": spec.source_stem,
|
||||
"timing_modes": TIMING_MODES,
|
||||
"loop_count_source": "stdin",
|
||||
"loop_count": spec.loop_count,
|
||||
"hot_loop_count": spec.hot_loop_count,
|
||||
"expected_checksum": spec.expected_checksum,
|
||||
"hot_expected_checksum": spec.hot_expected_checksum,
|
||||
"required_files": [
|
||||
{
|
||||
"path": relative,
|
||||
"status": "present" if (root / relative).is_file() else "missing",
|
||||
}
|
||||
for relative in REQUIRED_BENCHMARK_FILES
|
||||
],
|
||||
"checksum_metadata": {
|
||||
"cold_process": {
|
||||
"expected_checksum": spec.expected_checksum,
|
||||
"stdin": spec.stdin_text,
|
||||
},
|
||||
"hot_loop": {
|
||||
"expected_checksum": spec.hot_expected_checksum,
|
||||
"stdin": spec.hot_stdin_text,
|
||||
},
|
||||
},
|
||||
"run_args": spec.run_args,
|
||||
"implementation_slots": [
|
||||
{
|
||||
"name": impl.name,
|
||||
"language": impl.language,
|
||||
"source": str(impl.source.relative_to(suite_root)),
|
||||
}
|
||||
for impl in implementations
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def emit_suite_list(metadata: dict[str, object], as_json: bool) -> None:
|
||||
if as_json:
|
||||
print(json.dumps(metadata, indent=2, sort_keys=True))
|
||||
return
|
||||
|
||||
print(f"{metadata['suite']}: {metadata['timing_scope']}")
|
||||
print(str(metadata["timing_disclaimer"]))
|
||||
print(f"benchmark_count={metadata['benchmark_count']}")
|
||||
print(f"timing_modes={','.join(TIMING_MODES)}")
|
||||
verification = metadata["verification"]
|
||||
assert isinstance(verification, dict)
|
||||
print(f"verification_status={verification['status']}")
|
||||
print(f"required_files={verification['required_files']}")
|
||||
print(f"implementation_slots={verification['implementation_slots']}")
|
||||
print("benchmarks:")
|
||||
for benchmark in metadata["benchmarks"]:
|
||||
assert isinstance(benchmark, dict)
|
||||
print(
|
||||
" {name} ({directory}): loop_count={loop_count} hot_loop_count={hot_loop_count} "
|
||||
"expected_checksum={expected_checksum} hot_expected_checksum={hot_expected_checksum}".format(
|
||||
name=benchmark["name"],
|
||||
directory=benchmark["directory"],
|
||||
loop_count=benchmark["loop_count"],
|
||||
hot_loop_count=benchmark["hot_loop_count"],
|
||||
expected_checksum=benchmark["expected_checksum"],
|
||||
hot_expected_checksum=benchmark["hot_expected_checksum"],
|
||||
)
|
||||
)
|
||||
print(" required_files:")
|
||||
for required_file in benchmark["required_files"]:
|
||||
assert isinstance(required_file, dict)
|
||||
print(f" {required_file['path']}: {required_file['status']}")
|
||||
print(" implementations:")
|
||||
for implementation in benchmark["implementation_slots"]:
|
||||
assert isinstance(implementation, dict)
|
||||
print(f" {implementation['name']}: {implementation['language']} ({implementation['source']})")
|
||||
|
||||
|
||||
def emit_list(root: Path, spec: BenchmarkSpec, implementations: list[Implementation], as_json: bool) -> None:
|
||||
metadata = {
|
||||
"benchmark": spec.name,
|
||||
@ -520,3 +690,7 @@ def shlex_quote(value: str) -> str:
|
||||
if value and all(char.isalnum() or char in "/._:-" for char in value):
|
||||
return value
|
||||
return "'" + value.replace("'", "'\"'\"'") + "'"
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(Path(__file__).resolve().parent, sys.argv[1:]))
|
||||
|
||||
2
compiler/Cargo.lock
generated
2
compiler/Cargo.lock
generated
@ -4,4 +4,4 @@ version = 3
|
||||
|
||||
[[package]]
|
||||
name = "glagol"
|
||||
version = "1.0.0-beta.13"
|
||||
version = "1.0.0-beta.14"
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "glagol"
|
||||
version = "1.0.0-beta.13"
|
||||
version = "1.0.0-beta.14"
|
||||
edition = "2021"
|
||||
description = "Glagol, the first compiler for the Slovo language"
|
||||
license = "MIT OR Apache-2.0"
|
||||
|
||||
134
compiler/tests/benchmark_suite_catalog_beta14.rs
Normal file
134
compiler/tests/benchmark_suite_catalog_beta14.rs
Normal file
@ -0,0 +1,134 @@
|
||||
use std::{
|
||||
env,
|
||||
path::Path,
|
||||
process::{Command, Output},
|
||||
};
|
||||
|
||||
const BENCHMARKS: &[&str] = &[
|
||||
"array-index-loop",
|
||||
"array-struct-field-loop",
|
||||
"branch-loop",
|
||||
"enum-struct-payload-loop",
|
||||
"json-quote-loop",
|
||||
"math-loop",
|
||||
"parse-loop",
|
||||
"string-eq-loop",
|
||||
"vec-i32-index-loop",
|
||||
"vec-string-eq-loop",
|
||||
];
|
||||
|
||||
const IMPLEMENTATIONS: &[&str] = &["slovo", "c", "rust", "python", "clojure", "common_lisp"];
|
||||
|
||||
#[test]
|
||||
fn suite_catalog_is_byte_stable_and_lists_current_benchmarks() {
|
||||
let repo = Path::new(env!("CARGO_MANIFEST_DIR")).join("..");
|
||||
let python = python_command();
|
||||
|
||||
let first = run_suite_catalog(&repo, &python);
|
||||
let second = run_suite_catalog(&repo, &python);
|
||||
|
||||
assert_success("first suite catalog run", &first);
|
||||
assert_success("second suite catalog run", &second);
|
||||
assert_eq!(
|
||||
first.stdout, second.stdout,
|
||||
"suite catalog JSON must be byte-stable across runs"
|
||||
);
|
||||
|
||||
let stdout = String::from_utf8_lossy(&first.stdout);
|
||||
for needle in [
|
||||
r#""suite": "glagol-local-benchmark-suite""#,
|
||||
r#""benchmark_count": 10"#,
|
||||
r#""benchmark_metadata_files": 10"#,
|
||||
r#""required_files": 40"#,
|
||||
r#""missing_required_files": []"#,
|
||||
r#""implementation_slots": 60"#,
|
||||
r#""expected_implementation_slots": 60"#,
|
||||
r#""missing_implementation_slots": []"#,
|
||||
r#""status": "ok""#,
|
||||
r#""timing_scope": "local-machine comparison only""#,
|
||||
r#""timing_disclaimer": "Local timing comparison only; not a published benchmark result and not a cross-machine performance claim.""#,
|
||||
r#""cold-process""#,
|
||||
r#""hot-loop""#,
|
||||
r#""loop_count": 1000000"#,
|
||||
r#""hot_loop_count": 10000000"#,
|
||||
r#""checksum_metadata""#,
|
||||
r#""expected_checksum""#,
|
||||
r#""hot_expected_checksum""#,
|
||||
r#""required_files""#,
|
||||
r#""path": "benchmark.json""#,
|
||||
r#""path": "run.py""#,
|
||||
r#""path": "slovo.toml""#,
|
||||
r#""path": "src/main.slo""#,
|
||||
r#""status": "present""#,
|
||||
r#""implementation_slots""#,
|
||||
r#""loop_count_source": "stdin""#,
|
||||
] {
|
||||
assert_contains(&stdout, needle);
|
||||
}
|
||||
|
||||
for benchmark in BENCHMARKS {
|
||||
assert_contains(&stdout, &format!(r#""name": "{}""#, benchmark));
|
||||
assert_contains(&stdout, &format!(r#""directory": "{}""#, benchmark));
|
||||
}
|
||||
|
||||
for implementation in IMPLEMENTATIONS {
|
||||
assert_contains(&stdout, &format!(r#""name": "{}""#, implementation));
|
||||
}
|
||||
}
|
||||
|
||||
fn run_suite_catalog(repo: &Path, python: &str) -> Output {
|
||||
Command::new(python)
|
||||
.arg("benchmarks/runner.py")
|
||||
.arg("--suite-list")
|
||||
.arg("--json")
|
||||
.current_dir(repo)
|
||||
.output()
|
||||
.unwrap_or_else(|err| {
|
||||
panic!(
|
||||
"run `{}` benchmarks/runner.py --suite-list --json: {}",
|
||||
python, err
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
fn python_command() -> String {
|
||||
if let Some(python) = env::var_os("PYTHON") {
|
||||
return python.to_string_lossy().into_owned();
|
||||
}
|
||||
|
||||
for candidate in ["python3", "python"] {
|
||||
if Command::new(candidate)
|
||||
.arg("--version")
|
||||
.output()
|
||||
.map(|output| output.status.success())
|
||||
.unwrap_or(false)
|
||||
{
|
||||
return candidate.to_string();
|
||||
}
|
||||
}
|
||||
|
||||
panic!("benchmark suite catalog test requires python3 or python")
|
||||
}
|
||||
|
||||
fn assert_success(context: &str, output: &Output) {
|
||||
let stdout = String::from_utf8_lossy(&output.stdout);
|
||||
let stderr = String::from_utf8_lossy(&output.stderr);
|
||||
|
||||
assert!(
|
||||
output.status.success(),
|
||||
"{} failed\nstdout:\n{}\nstderr:\n{}",
|
||||
context,
|
||||
stdout,
|
||||
stderr
|
||||
);
|
||||
assert!(stderr.is_empty(), "{} wrote stderr:\n{}", context, stderr);
|
||||
}
|
||||
|
||||
fn assert_contains(haystack: &str, needle: &str) {
|
||||
assert!(
|
||||
haystack.contains(needle),
|
||||
"suite catalog output missing `{}`\nstdout:\n{}",
|
||||
needle,
|
||||
haystack
|
||||
);
|
||||
}
|
||||
@ -379,6 +379,38 @@ Why thirteenth: diagnostics already have a machine schema and broad golden
|
||||
coverage, but the compatibility policy and current code inventory need one
|
||||
central reference before future tooling or migration work builds on them.
|
||||
|
||||
### 14. Benchmark Suite Catalog And Metadata Gate
|
||||
|
||||
Goal: document the existing benchmark suite inventory and metadata listing
|
||||
commands before benchmark tooling grows additional consumers.
|
||||
|
||||
Work:
|
||||
|
||||
- add [`benchmarks/README.md`](../benchmarks/README.md) as the top-level
|
||||
benchmark suite catalog
|
||||
- document `python3 benchmarks/runner.py --suite-list` for the non-JSON suite
|
||||
inventory
|
||||
- document `python3 benchmarks/runner.py --suite-list --json` for beta tooling
|
||||
metadata
|
||||
- list the current suite inventory without adding new benchmark kernels
|
||||
- state that benchmark timings are local-machine evidence only
|
||||
- keep suite-list JSON beta-scoped rather than a stable public schema
|
||||
- keep timing publication, performance thresholds, source-language/runtime/
|
||||
stdlib/API/diagnostic-output changes, and ABI/layout changes out of scope
|
||||
|
||||
Released in `1.0.0-beta.14`:
|
||||
[`benchmarks/README.md`](../benchmarks/README.md) now catalogs the current
|
||||
benchmark suite, documents the root suite-list commands, records local-machine
|
||||
evidence policy, and names the explicit exclusions. The release is
|
||||
documentation/tooling metadata only; it does not add kernels, publish timing
|
||||
numbers, define performance thresholds, define a stable JSON schema, or change
|
||||
the source language, runtime, stdlib/API surface, diagnostics, or ABI/layout
|
||||
behavior.
|
||||
|
||||
Why fourteenth: the benchmark suite is already part of the public monorepo, but
|
||||
its suite-level inventory and metadata boundary need one central reference
|
||||
before future tooling can rely on it.
|
||||
|
||||
## Stable `1.0.0` Gate
|
||||
|
||||
Slovo should not become stable until all of these are true:
|
||||
@ -390,6 +422,7 @@ Slovo should not become stable until all of these are true:
|
||||
- release gates are reproducible on a clean checkout
|
||||
- diagnostics and formatter output are stable for promoted features
|
||||
- performance publications are repeatable and labeled as local-machine evidence
|
||||
- benchmark metadata promoted to stable, if any, has an explicit schema policy
|
||||
- the language can build useful local CLI tools, libraries, file-processing
|
||||
programs, and basic host-interaction programs without undocumented behavior
|
||||
|
||||
|
||||
@ -10,11 +10,43 @@ integration/readiness release, not the first real beta.
|
||||
|
||||
## Unreleased
|
||||
|
||||
Next scoped Glagol work is expected to continue after the `1.0.0-beta.13`
|
||||
diagnostic catalog and schema policy hardening update.
|
||||
Next scoped Glagol work is expected to continue after the `1.0.0-beta.14`
|
||||
benchmark suite catalog and metadata gate.
|
||||
|
||||
No unreleased compiler scope is committed here yet.
|
||||
|
||||
## 1.0.0-beta.14
|
||||
|
||||
Release label: `1.0.0-beta.14`
|
||||
|
||||
Release date: 2026-05-22
|
||||
|
||||
Release state: benchmark suite catalog and metadata gate
|
||||
|
||||
### Summary
|
||||
|
||||
The beta.14 compiler-side contract is tooling-only benchmark catalog hardening.
|
||||
It keeps benchmark execution local and unchanged while adding deterministic
|
||||
suite-level metadata for release verification.
|
||||
|
||||
- Bump the `glagol` compiler package version to `1.0.0-beta.14`.
|
||||
- Add `python3 benchmarks/runner.py --suite-list --json` for deterministic
|
||||
suite-level benchmark listing and verification while preserving normal
|
||||
per-benchmark `run.py` execution through the shared runner.
|
||||
- Emit all 10 current benchmark names and directories, timing modes, cold/hot
|
||||
loop counts, checksum metadata, required scaffold-file status,
|
||||
implementation slots, and the local-only timing disclaimer.
|
||||
- Add focused `benchmark_suite_catalog_beta14` coverage requiring byte-stable
|
||||
suite catalog output across two runs and the current 10-benchmark inventory.
|
||||
- Run the focused beta.14 suite catalog test in `scripts/release-gate.sh`
|
||||
before the full compiler test suite.
|
||||
|
||||
### Explicit Deferrals
|
||||
|
||||
This release does not implement `glagol bench`, timing-result publication, a
|
||||
new benchmark kernel, runtime changes, source-language changes, a stable JSON
|
||||
schema claim, benchmark thresholds, or cross-machine performance claims.
|
||||
|
||||
## 1.0.0-beta.13
|
||||
|
||||
Release label: `1.0.0-beta.13`
|
||||
|
||||
@ -22,8 +22,8 @@ general-purpose beta release.
|
||||
|
||||
A Glagol feature is done only when it has parser/lowerer support, checker behavior, diagnostics for invalid forms, backend behavior or explicit unsupported diagnostics, and tests.
|
||||
|
||||
Current stage: `1.0.0-beta.13`, released on 2026-05-22 as a diagnostic
|
||||
catalog and schema policy hardening update. It keeps the `1.0.0-beta`
|
||||
Current stage: `1.0.0-beta.14`, released on 2026-05-22 as a benchmark suite
|
||||
catalog and metadata gate update. It keeps the `1.0.0-beta`
|
||||
language/compiler support baseline and includes the
|
||||
`1.0.0-beta.1` tooling hardening release, the `1.0.0-beta.2` runtime/resource
|
||||
foundation release, the `1.0.0-beta.3` standard-library stabilization release,
|
||||
@ -55,9 +55,14 @@ source-language runtime behavior. The beta.13 tooling/docs-only diagnostics
|
||||
slice centralizes the `slovo.diagnostic` schema name/version constants and
|
||||
adds structural gates for S-expression diagnostics, `--json-diagnostics`, and
|
||||
artifact-manifest diagnostics metadata without changing emitted machine
|
||||
diagnostic shape.
|
||||
diagnostic shape. The beta.14 tooling-only benchmark slice adds deterministic
|
||||
suite-level local benchmark listing and release-gate coverage for
|
||||
`python3 benchmarks/runner.py --suite-list --json`, including all current
|
||||
benchmark directories, timing modes, cold/hot loop counts, checksum metadata,
|
||||
required scaffold-file status, implementation slots, and a local-only timing
|
||||
disclaimer without publishing timing results or claiming a stable JSON schema.
|
||||
|
||||
Next stage target: post-`1.0.0-beta.13` developer-experience, package, and
|
||||
Next stage target: post-`1.0.0-beta.14` developer-experience, package, and
|
||||
collection/generic planning. Generic vectors, generic collections, maps, sets,
|
||||
generic stdlib dispatch, runtime collection changes, collection unification,
|
||||
stable human diagnostic text, stable Markdown schema, LSP/watch protocols,
|
||||
|
||||
@ -8,7 +8,7 @@ Historical `exp-*` releases listed here are experimental maturity milestones.
|
||||
The pushed tag `v2.0.0-beta.1` is historical. It is now documented as an
|
||||
experimental integration/readiness release, not as a beta maturity claim.
|
||||
|
||||
The current release is `1.0.0-beta.13`, published on 2026-05-22. It keeps the
|
||||
The current release is `1.0.0-beta.14`, published on 2026-05-22. It keeps the
|
||||
`1.0.0-beta` language surface, includes the first post-beta tooling/install
|
||||
hardening bundle from `1.0.0-beta.1`, and adds the first runtime/resource
|
||||
foundation bundle from `1.0.0-beta.2` plus the first standard-library
|
||||
@ -23,12 +23,49 @@ collection alias unification and generic reservation slice from
|
||||
`1.0.0-beta.10`, and the local package API documentation extension from
|
||||
`1.0.0-beta.11`, plus the concrete vector query and prefix parity slice from
|
||||
`1.0.0-beta.12`, and the diagnostic catalog and schema policy slice from
|
||||
`1.0.0-beta.13`.
|
||||
`1.0.0-beta.13`, plus the benchmark suite catalog and metadata gate from
|
||||
`1.0.0-beta.14`.
|
||||
|
||||
## Unreleased
|
||||
|
||||
No unreleased language scope is committed here yet.
|
||||
|
||||
## 1.0.0-beta.14
|
||||
|
||||
Release label: `1.0.0-beta.14`
|
||||
|
||||
Release name: Benchmark Suite Catalog And Metadata Gate
|
||||
|
||||
Release date: 2026-05-22
|
||||
|
||||
Status: released beta documentation/tooling metadata update on the
|
||||
`1.0.0-beta` language baseline.
|
||||
|
||||
`1.0.0-beta.14` documents the existing benchmark suite catalog and metadata
|
||||
gate without changing the source language, runtime, standard library, API
|
||||
surface, diagnostic output, compiler-known runtime names, or ABI/layout
|
||||
behavior:
|
||||
|
||||
- Adds [`benchmarks/README.md`](../../benchmarks/README.md) as the top-level
|
||||
catalog for the current ten benchmark suites:
|
||||
`math-loop`, `branch-loop`, `parse-loop`, `array-index-loop`,
|
||||
`string-eq-loop`, `array-struct-field-loop`,
|
||||
`enum-struct-payload-loop`, `vec-i32-index-loop`,
|
||||
`vec-string-eq-loop`, and `json-quote-loop`.
|
||||
- Documents `python3 benchmarks/runner.py --suite-list` as the non-JSON suite
|
||||
inventory command.
|
||||
- Documents `python3 benchmarks/runner.py --suite-list --json` as the beta
|
||||
tooling metadata form for local gates and adapters.
|
||||
- States that benchmark timings are local-machine evidence only and publishes
|
||||
no timing numbers.
|
||||
- Records that suite-list JSON is beta tooling metadata, not a stable public
|
||||
schema.
|
||||
|
||||
This release does not add benchmark kernels, publish timings, define
|
||||
performance thresholds, define a stable JSON schema, change source-language,
|
||||
runtime, stdlib/API, diagnostic-output, compiler ABI/layout, or optimizer
|
||||
behavior, or make cross-machine performance claims.
|
||||
|
||||
## 1.0.0-beta.13
|
||||
|
||||
Release label: `1.0.0-beta.13`
|
||||
|
||||
@ -10,46 +10,53 @@ Long-horizon planning lives in
|
||||
release train from the historical `v2.0.0-beta.1` tag toward and beyond the
|
||||
first real general-purpose beta Slovo contract.
|
||||
|
||||
Current stage: `1.0.0-beta.13`, released on 2026-05-22 as a post-beta
|
||||
diagnostic catalog and schema policy update. It keeps the `1.0.0-beta`
|
||||
language contract and includes the `1.0.0-beta.1` tooling hardening release,
|
||||
the `1.0.0-beta.2`
|
||||
runtime/resource foundation release, the `1.0.0-beta.3` standard-library
|
||||
stabilization release, the `1.0.0-beta.4` language-usability diagnostics
|
||||
release, the `1.0.0-beta.5` package/workspace discipline release, and a narrow
|
||||
`std.net` source facade for blocking loopback TCP client/server primitives over
|
||||
opaque `i32` handles and concrete `result` families, plus a narrow `std.json`
|
||||
source facade for compact JSON text construction, plus top-level module-local
|
||||
transparent aliases for supported concrete target types, plus local alias use
|
||||
inside the current concrete vector, option, and result facades, plus a
|
||||
generated standard-library API catalog that lists exact exported helper
|
||||
signatures with module-local aliases normalized to concrete public types, plus
|
||||
`glagol symbols` deterministic source metadata for files, projects, and
|
||||
workspaces, plus `glagol doc <file|project|workspace> -o <dir>` public API
|
||||
sections for local packages/modules with exact exported function signatures,
|
||||
exported struct fields, exported enum variants/payload types, non-export
|
||||
filtering, and module-local alias normalization, plus source-authored concrete
|
||||
vector helper parity: `std.vec_i64` gains `count_of`, `starts_with`,
|
||||
`without_prefix`, `ends_with`, and `without_suffix`, and `std.vec_f64` gains
|
||||
`count_of`, plus [`docs/language/DIAGNOSTICS.md`](DIAGNOSTICS.md) as the beta
|
||||
Current stage: `1.0.0-beta.14`, released on 2026-05-22 as a post-beta
|
||||
benchmark suite catalog and metadata gate. It keeps the `1.0.0-beta` language
|
||||
contract and includes the `1.0.0-beta.1` tooling hardening release, the
|
||||
`1.0.0-beta.2` runtime/resource foundation release, the `1.0.0-beta.3`
|
||||
standard-library stabilization release, the `1.0.0-beta.4`
|
||||
language-usability diagnostics release, the `1.0.0-beta.5` package/workspace
|
||||
discipline release, and a narrow `std.net` source facade for blocking loopback
|
||||
TCP client/server primitives over opaque `i32` handles and concrete `result`
|
||||
families, plus a narrow `std.json` source facade for compact JSON text
|
||||
construction, plus top-level module-local transparent aliases for supported
|
||||
concrete target types, plus local alias use inside the current concrete vector,
|
||||
option, and result facades, plus a generated standard-library API catalog that
|
||||
lists exact exported helper signatures with module-local aliases normalized to
|
||||
concrete public types, plus `glagol symbols` deterministic source metadata for
|
||||
files, projects, and workspaces, plus
|
||||
`glagol doc <file|project|workspace> -o <dir>` public API sections for local
|
||||
packages/modules with exact exported function signatures, exported struct
|
||||
fields, exported enum variants/payload types, non-export filtering, and
|
||||
module-local alias normalization, plus source-authored concrete vector helper
|
||||
parity: `std.vec_i64` gains `count_of`, `starts_with`, `without_prefix`,
|
||||
`ends_with`, and `without_suffix`, and `std.vec_f64` gains `count_of`, plus
|
||||
[`docs/language/DIAGNOSTICS.md`](DIAGNOSTICS.md) as the beta
|
||||
`slovo.diagnostic` version `1` policy and current golden diagnostic code
|
||||
catalog. JSON parsing, recursive JSON values, executable generics, generic
|
||||
aliases, parameterized aliases, cross-module alias visibility, maps/sets,
|
||||
traits, inference, monomorphization, iterators, runtime changes for generic
|
||||
collections, DNS, TLS, UDP, async IO, non-loopback binding, HTTP frameworks,
|
||||
rich host-error ADTs, stable ABI/layout, stable Markdown schema, stable
|
||||
stdlib/API compatibility freeze, LSP/watch, SARIF/daemon protocols,
|
||||
stable `1.0.0` diagnostics freeze, re-exports/globs/hierarchical modules,
|
||||
mutable vectors, slice/view APIs, new runtime names, performance claims, and
|
||||
package registry semantics remain deferred.
|
||||
catalog, plus [`../../benchmarks/README.md`](../../benchmarks/README.md) as
|
||||
the beta-scoped benchmark suite catalog documenting the current suite
|
||||
inventory, `python3 benchmarks/runner.py --suite-list`, and
|
||||
`python3 benchmarks/runner.py --suite-list --json`. Benchmark timings remain
|
||||
local-machine evidence only, and suite-list JSON is beta tooling metadata
|
||||
rather than a stable public schema. JSON parsing, recursive JSON values,
|
||||
executable generics, generic aliases, parameterized aliases, cross-module alias
|
||||
visibility, maps/sets, traits, inference, monomorphization, iterators, runtime
|
||||
changes for generic collections, DNS, TLS, UDP, async IO, non-loopback binding,
|
||||
HTTP frameworks, rich host-error ADTs, stable ABI/layout, stable Markdown
|
||||
schema, stable stdlib/API compatibility freeze, LSP/watch, SARIF/daemon
|
||||
protocols, stable `1.0.0` diagnostics freeze, re-exports/globs/hierarchical
|
||||
modules, mutable vectors, slice/view APIs, new runtime names, timing
|
||||
publication, performance claims, stable benchmark JSON schema, and package
|
||||
registry semantics remain deferred.
|
||||
|
||||
Next stage target: continue after `1.0.0-beta.13` from developer-experience,
|
||||
package, and reserved generic/map/set planning without claiming executable
|
||||
generics, an LSP/watch protocol, SARIF/daemon protocol, stable Markdown
|
||||
schema, registry semantics, stable `1.0.0` diagnostics freeze, or stable
|
||||
standard-library/API compatibility freeze, mutable vectors, slice/view APIs,
|
||||
new runtime names, performance claims, maps/sets, or iterators until the exact
|
||||
scope is frozen from the manifest and roadmap.
|
||||
Next stage target: continue after `1.0.0-beta.14` from developer-experience,
|
||||
package, benchmark metadata, and reserved generic/map/set planning without
|
||||
claiming executable generics, an LSP/watch protocol, SARIF/daemon protocol,
|
||||
stable Markdown schema, registry semantics, stable benchmark JSON schema,
|
||||
stable `1.0.0` diagnostics freeze, or stable standard-library/API
|
||||
compatibility freeze, mutable vectors, slice/view APIs, new runtime names,
|
||||
performance claims, maps/sets, or iterators until the exact scope is frozen
|
||||
from the manifest and roadmap.
|
||||
|
||||
The final experimental precursor scope is `exp-125`, defined in
|
||||
`.llm/EXP_125_UNSIGNED_U32_U64_NUMERIC_AND_STDLIB_BREADTH_ALPHA.md`. Its
|
||||
|
||||
@ -5,7 +5,8 @@ Status: living beta contract for `1.0.0-beta`, with the post-beta
|
||||
foundation update, `1.0.0-beta.10` developer-experience API discovery update,
|
||||
`1.0.0-beta.11` local package API documentation update, and `1.0.0-beta.12`
|
||||
concrete vector query and prefix parity update, and `1.0.0-beta.13`
|
||||
diagnostic catalog and schema policy update. The language contract integrates
|
||||
diagnostic catalog and schema policy update, and `1.0.0-beta.14` benchmark
|
||||
suite catalog and metadata gate. The language contract integrates
|
||||
promoted language slices through `exp-125` and the historical publication
|
||||
baseline through `exp-123`. `1.0.0-beta` is the first real general-purpose
|
||||
beta release. `exp-125` completed the unsigned numeric and stdlib breadth
|
||||
@ -157,6 +158,16 @@ Current v1 release surface and explicit experimental targets:
|
||||
current golden diagnostic code catalog; this is docs/tooling policy only, not
|
||||
a source-language/runtime change, LSP/watch contract, SARIF/daemon protocol,
|
||||
stable Markdown schema, or stable `1.0.0` diagnostics freeze
|
||||
- `1.0.0-beta.14` benchmark suite catalog and metadata target:
|
||||
[`benchmarks/README.md`](../../benchmarks/README.md) documents the existing
|
||||
benchmark suite inventory and root suite-list commands:
|
||||
`python3 benchmarks/runner.py --suite-list` and
|
||||
`python3 benchmarks/runner.py --suite-list --json`; benchmark timing remains
|
||||
local-machine evidence only, and suite-list JSON is beta tooling metadata
|
||||
only, not a stable public schema; this is docs/tooling metadata only, not new
|
||||
benchmark kernels, timing publication, performance thresholds,
|
||||
source-language/runtime/stdlib/API changes, diagnostic-output changes, or
|
||||
ABI/layout changes
|
||||
- `exp-1` owned runtime strings: compiler-known `std.string.concat` accepts two
|
||||
`string` values and returns an immutable runtime-owned `string`; existing
|
||||
string equality, length, printing, locals, parameters, returns, and calls work
|
||||
@ -1379,6 +1390,39 @@ protocols, stable Markdown schema, stable `1.0.0` diagnostics freeze,
|
||||
source-map/debug-metadata contracts, localized diagnostic text, automated
|
||||
machine fix-its, or a compiler-emitted diagnostic catalog artifact.
|
||||
|
||||
### 4.4.11 Post-Beta Benchmark Suite Catalog And Metadata Gate
|
||||
|
||||
Status: released in `1.0.0-beta.14` as a docs/tooling metadata update for the
|
||||
existing benchmark suite.
|
||||
|
||||
`1.0.0-beta.14` adds
|
||||
[`benchmarks/README.md`](../../benchmarks/README.md) as the top-level
|
||||
benchmark suite catalog. The catalog records the current suite inventory,
|
||||
base/hot-loop checksum metadata, local evidence policy, and explicit
|
||||
exclusions.
|
||||
|
||||
The root suite-list commands are documented as:
|
||||
|
||||
```bash
|
||||
python3 benchmarks/runner.py --suite-list
|
||||
python3 benchmarks/runner.py --suite-list --json
|
||||
```
|
||||
|
||||
The non-JSON listing is local review output. The JSON listing is beta tooling
|
||||
metadata for local gates and adapters. The field set is intentionally not a
|
||||
stable public schema in this release.
|
||||
|
||||
The source language, typed core, runtime, standard library, public API surface,
|
||||
compiler diagnostics, diagnostic output shape, compiler-known runtime names,
|
||||
ABI/layout behavior, and optimizer contract are unchanged by this target.
|
||||
Benchmark timings remain local-machine evidence only and no timing numbers are
|
||||
published by the release contract.
|
||||
|
||||
This target explicitly does not add benchmark kernels, implementation language
|
||||
slots, timing publication, performance thresholds, cross-machine performance
|
||||
claims, a stable benchmark JSON schema, source-language/runtime/stdlib/API
|
||||
changes, diagnostic-output changes, or ABI/layout guarantees.
|
||||
|
||||
## 4.5 v2.0.0-beta.1 Experimental Integration Readiness
|
||||
|
||||
Status: current experimental Slovo-side release contract, released 2026-05-17.
|
||||
|
||||
@ -6,7 +6,7 @@ Do not edit this file by hand.
|
||||
## Stability Tiers
|
||||
|
||||
- `beta-supported`: exported from `lib/std` and covered by source-search, promotion, or facade gates in the current beta line.
|
||||
- `experimental`: not used for exported `lib/std` helpers in `1.0.0-beta.13`; future releases may mark new helpers this way before they graduate.
|
||||
- `experimental`: not used for exported `lib/std` helpers in `1.0.0-beta.14`; future releases may mark new helpers this way before they graduate.
|
||||
- `internal`: helper names that are not exported from their module; they are intentionally omitted from this catalog.
|
||||
|
||||
The catalog is a beta API discovery aid, not a stable `1.0.0` standard-library freeze.
|
||||
|
||||
@ -64,6 +64,7 @@ fi
|
||||
cd "${compiler_dir}"
|
||||
cargo fmt --check
|
||||
cargo test --test diagnostics_schema_beta13
|
||||
cargo test --test benchmark_suite_catalog_beta14
|
||||
# Full cargo test includes unignored integration gates such as dx_v1_7,
|
||||
# beta_v2_0_0_beta_1, and beta_1_0_0.
|
||||
cargo test
|
||||
|
||||
Loading…
Reference in New Issue
Block a user