84 lines
3.5 KiB
Markdown
84 lines
3.5 KiB
Markdown
# Slovo Benchmark Suite Catalog
|
|
|
|
Release stage: `1.0.0-beta.14`.
|
|
|
|
The benchmark suite is beta-scoped local tooling. It catalogs deterministic
|
|
same-machine comparison scaffolds for Slovo and sibling implementations. It
|
|
does not publish benchmark results, set performance thresholds, define optimizer
|
|
claims, or create cross-machine comparisons.
|
|
|
|
## Suite Listing
|
|
|
|
From the repository root, list the suite catalog for humans:
|
|
|
|
```bash
|
|
python3 benchmarks/runner.py --suite-list
|
|
```
|
|
|
|
List the same catalog as beta tooling metadata:
|
|
|
|
```bash
|
|
python3 benchmarks/runner.py --suite-list --json
|
|
```
|
|
|
|
The non-JSON listing is for local review. The JSON listing is for local gates
|
|
and tooling adapters that need the current suite inventory. The JSON field set
|
|
is not a stable public schema; it may change during beta releases.
|
|
Both forms verify the required scaffold files for each suite:
|
|
`benchmark.json`, `run.py`, `slovo.toml`, and `src/main.slo`.
|
|
|
|
Each suite still owns its local metadata and run commands:
|
|
|
|
```bash
|
|
python3 benchmarks/<suite>/run.py --list
|
|
python3 benchmarks/<suite>/run.py --list --json
|
|
python3 benchmarks/<suite>/run.py --dry-run
|
|
```
|
|
|
|
## Current Suites
|
|
|
|
All current suites provide Slovo, C, Rust, Python, Clojure, and Common
|
|
Lisp/SBCL source slots. Missing local toolchains are skipped by the per-suite
|
|
runner where possible.
|
|
|
|
| Suite | Focus | Base checksum | Hot-loop checksum | Runtime args |
|
|
| --- | --- | --- | --- | --- |
|
|
| `math-loop` | arithmetic and scalar accumulation | `5000001` | `50000001` | none |
|
|
| `branch-loop` | deterministic branch-heavy integer loop | `1185071` | `220775` | none |
|
|
| `parse-loop` | repeated signed decimal `i32` parsing | `345000001` | `450000001` | `12345` |
|
|
| `array-index-loop` | immutable fixed-array indexing | `3875007` | `38750007` | none |
|
|
| `string-eq-loop` | fixed string lookup and equality | `4600001` | `46000001` | `omega` |
|
|
| `array-struct-field-loop` | fixed-array access through an immutable struct field | `3875011` | `38750011` | none |
|
|
| `enum-struct-payload-loop` | enum payload matching with struct and array access | `3500013` | `35000013` | none |
|
|
| `vec-i32-index-loop` | runtime-owned `(vec i32)` indexing | `3875007` | `38750007` | none |
|
|
| `vec-string-eq-loop` | runtime-owned `(vec string)` lookup and equality | `4600001` | `46000001` | `omega` |
|
|
| `json-quote-loop` | compact JSON string quoting | `15000001` | `150000001` | `slo"vo\path` |
|
|
|
|
The base loop count is `1000000` for every current suite. The hot-loop count is
|
|
`10000000` for every current suite. The runner supplies loop counts and runtime
|
|
arguments at execution time so native compilers cannot fold the work into a
|
|
constant answer.
|
|
|
|
## Local Evidence Only
|
|
|
|
Benchmark output is local-machine evidence only:
|
|
|
|
- cold-process mode measures execution after each implementation has been built
|
|
once; it does not include compile time
|
|
- hot-loop mode is startup-amortized local evidence and reports total time plus
|
|
normalized timing for the base loop count
|
|
- Clojure timings include JVM and Clojure startup
|
|
- Common Lisp timings include SBCL script startup
|
|
- reported timings depend on the local CPU, OS, compiler versions, toolchain
|
|
availability, thermal/load state, and runner configuration
|
|
|
|
This catalog intentionally publishes no timing numbers.
|
|
|
|
## Exclusions
|
|
|
|
`1.0.0-beta.14` does not add benchmark kernels, publish timings, define
|
|
performance thresholds, define a stable JSON schema, change the Slovo source
|
|
language, change runtime behavior, change standard-library or API contracts,
|
|
change diagnostic output, change ABI/layout behavior, or make cross-machine
|
|
performance claims.
|