slobodanmargetic988/weeyuga-benchmarks-public
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

B3 staging seed — 21 runs + catalogue v1.0-draft + methodology + README

Browse Source 
Initial population of the weeyuga-benchmarks-public archive (PRIVATE
staging visibility — flips public after Miljan + Stevan security audit
sign-off per Sloba's 17:34Z dispatch).

Contents:
- README.md       — public-facing intro (warns staging state, schema overview, citation pattern, license split)
- LICENSE         — CC-BY-4.0 default (auto-init from Gitea)
- catalogue.json  — schema_version=1.0-draft (locked once Tomas ratifies); 21 benchmarks indexed, 13 complete + 8 meta-only
- methodology.md  — mirror of WeeyugaWeb docs/BENCHMARKS/HARNESS.md (canonical methodology)
- runs/<id>/run.jsonl|run.log|run.md|metadata.json — packaged copies of every run in WeeyugaWeb docs/BENCHMARKS/runs/*

Run set covers:
- Mission 1 (2026-04-28/29): pavilion-weeyuga-v1 + reconstructed v3 (96 calls, 16 models routed via weeyuga :11435)
- Predator trio (2026-05-04): granite-4.1-8B + gemma-4-E4B-it + qwen3.5-9B
- Predator qwen rerun (2026-05-04): qwen3.5-9B think500/nothink + qwen3-14B feasibility
- A3B campaign (2026-05-04/05): pavilion-a3b + predator-a3b NGL matrix + ctx sweep + NGL+ctx 2D + NGL=6 deep dive
- VPS50 CPU matrix + gemma-e4b CPU lane (2026-05-04/05)

Visibility GATE: this repo stays private until Miljan G1-G4 audit and
Stevan G3 credential audit both green. After sign-off, single API call
flips visibility=public, anonymous read on, push-protection requires
auth, issues moderate by default.

No raw IPs, no SSH user@host strings, no /Users/ paths, no whisper
transcripts in any of these files. Hardware names (pavilion, predator,
vps50) are intentional and fine to share.

Builder: WeeyugaWeb/scripts/benchmarks/build_catalogue.py (deterministic,
idempotent, ~5s wall on 21 runs).
Publish flow: WeeyugaWeb/scripts/benchmarks/publish_bench_run.py
(builds packaged dirs, regenerates catalogue, optional --push to mirror
into this repo, optional --deploy stub for cicd rsync).

Owner: mac/benchmark-tester-ben (Ben).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Slobodan Margetic
2026-05-05 19:46:01 +02:00
parent 5c726cf585
commit a18db6a3da
70 changed files with 16023 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
runs/23066b38-ea9c-4dd3-b2f5-32912a67fce4/run.jsonl Normal file
View File

@@ -0,0 +1,15 @@
{"type": "meta", "benchmark_run_id": "23066b38-ea9c-4dd3-b2f5-32912a67fce4", "harness_version": "pavilion-a3b-1", "started_at_utc": "2026-05-04T22:11:43Z", "host": "Slobodans-MacBook-Air.local", "node": "pavilion", "engine": "llamacpp", "pavilion_target_url": "http://10.8.0.3:11437", "purpose": "Sloba 2026-05-04: Qwen3-30B-A3B (MoE) on weaker Pavilion hardware (GTX 1050 4GB / 16GB RAM) for cross-machine A3B comparison", "models": {"qwen3-30b-a3b-pavilion": {"cell_id": "pavilion:llamacpp:qwen3:30b-a3b-iq2xxs-think500", "gguf": "Qwen3-30B-A3B-UD-IQ2_XXS.gguf", "reasoning_budget": 500, "ngl": 12, "ctx": 2048, "note": "30B total / 3B active MoE; 9.9GB IQ2_XXS; partial CPU offload heavy"}}}
{"type": "vram_snapshot", "model_key": "qwen3-30b-a3b-pavilion", "cell_id": "pavilion:llamacpp:qwen3:30b-a3b-iq2xxs-think500", "raw": "3842, 163, 4096"}
{"type": "llama_bench", "model_key": "qwen3-30b-a3b-pavilion", "cell_id": "pavilion:llamacpp:qwen3:30b-a3b-iq2xxs-think500", "raw": "llama-bench.exe : ggml_cuda_init: found 1 CUDA devices (Total VRAM: 4095 MiB):\r\nAt line:1 char:1\r\n+ & 'D:\\WeeyugaTools\\llama.cpp\\llama-bench.exe' -m 'D:\\WeeyugaModels\\ll ...\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n + CategoryInfo : NotSpecified: (ggml_cuda_init:...RAM: 4095 MiB)::String) [], RemoteException\r\n + FullyQualifiedErrorId : NativeCommandError\r\n \r\n Device 0: NVIDIA GeForce GTX 1050, compute capability 6.1, VMM: yes, VRAM: 4095 MiB\r\nload_backend: loaded CUDA backend from D:\\WeeyugaTools\\llama.cpp\\ggml-cuda.dll\r\nload_backend: loaded RPC backend from D:\\WeeyugaTools\\llama.cpp\\ggml-rpc.dll\r\nload_backend: loaded CPU backend from D:\\WeeyugaTools\\llama.cpp\\ggml-cpu-haswell.dll\r\n| model | size | params | backend | ngl | test | t/s |\r\n| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |\r\n| qwen3moe 30B.A3B IQ2_XXS - 2.0625 bpw | 9.65 GiB | 30.53 B | CUDA | 12 | pp256 | 6.27 \u00b1 2.87 |\r\n| qwen3moe 30B.A3B IQ2_XXS - 2.0625 bpw | 9.65 GiB | 30.53 B | CUDA | 12 | tg64 | 4.90 \u00b1 0.67 |\r\n\r\nbuild: 665abc609 (8951)\r\n\r"}
{"type": "call", "cell_id": "pavilion:llamacpp:qwen3:30b-a3b-iq2xxs-think500", "model_key": "qwen3-30b-a3b-pavilion", "prompt_id": "hello", "run_idx": 0, "phase": "cold", "total_duration_ms": 226760, "prompt_tokens": 9, "completion_tokens": 64, "tokens_per_sec": 0.28223672605397776, "finish_reason": "length", "reasoning_chars": 273, "content_chars": 0, "error": null}
{"type": "call", "cell_id": "pavilion:llamacpp:qwen3:30b-a3b-iq2xxs-think500", "model_key": "qwen3-30b-a3b-pavilion", "prompt_id": "hello", "run_idx": 1, "phase": "warm", "total_duration_ms": 15935, "prompt_tokens": 9, "completion_tokens": 64, "tokens_per_sec": 4.016316284907436, "finish_reason": "length", "reasoning_chars": 273, "content_chars": 0, "error": null}
{"type": "call", "cell_id": "pavilion:llamacpp:qwen3:30b-a3b-iq2xxs-think500", "model_key": "qwen3-30b-a3b-pavilion", "prompt_id": "hello", "run_idx": 2, "phase": "warm", "total_duration_ms": 16991, "prompt_tokens": 9, "completion_tokens": 64, "tokens_per_sec": 3.7667000176564063, "finish_reason": "length", "reasoning_chars": 266, "content_chars": 0, "error": null}
{"type": "call", "cell_id": "pavilion:llamacpp:qwen3:30b-a3b-iq2xxs-think500", "model_key": "qwen3-30b-a3b-pavilion", "prompt_id": "hello", "run_idx": 3, "phase": "warm", "total_duration_ms": 11898, "prompt_tokens": 9, "completion_tokens": 64, "tokens_per_sec": 5.379055303412338, "finish_reason": "length", "reasoning_chars": 283, "content_chars": 0, "error": null}
{"type": "call", "cell_id": "pavilion:llamacpp:qwen3:30b-a3b-iq2xxs-think500", "model_key": "qwen3-30b-a3b-pavilion", "prompt_id": "P-MEDIUM", "run_idx": 0, "phase": "cold", "total_duration_ms": 132262, "prompt_tokens": 22, "completion_tokens": 461, "tokens_per_sec": 3.4855060410397543, "finish_reason": "stop", "reasoning_chars": 1700, "content_chars": 607, "error": null}
{"type": "call", "cell_id": "pavilion:llamacpp:qwen3:30b-a3b-iq2xxs-think500", "model_key": "qwen3-30b-a3b-pavilion", "prompt_id": "P-MEDIUM", "run_idx": 1, "phase": "warm", "total_duration_ms": 61800, "prompt_tokens": 22, "completion_tokens": 459, "tokens_per_sec": 7.427184466019418, "finish_reason": "stop", "reasoning_chars": 1693, "content_chars": 460, "error": null}
{"type": "call", "cell_id": "pavilion:llamacpp:qwen3:30b-a3b-iq2xxs-think500", "model_key": "qwen3-30b-a3b-pavilion", "prompt_id": "P-MEDIUM", "run_idx": 2, "phase": "warm", "total_duration_ms": 49471, "prompt_tokens": 22, "completion_tokens": 405, "tokens_per_sec": 8.186614380141902, "finish_reason": "stop", "reasoning_chars": 1552, "content_chars": 460, "error": null}
{"type": "call", "cell_id": "pavilion:llamacpp:qwen3:30b-a3b-iq2xxs-think500", "model_key": "qwen3-30b-a3b-pavilion", "prompt_id": "P-MEDIUM", "run_idx": 3, "phase": "warm", "total_duration_ms": 44543, "prompt_tokens": 22, "completion_tokens": 363, "tokens_per_sec": 8.149428641986395, "finish_reason": "stop", "reasoning_chars": 1344, "content_chars": 513, "error": null}
{"type": "call", "cell_id": "pavilion:llamacpp:qwen3:30b-a3b-iq2xxs-think500", "model_key": "qwen3-30b-a3b-pavilion", "prompt_id": "P-HARD", "run_idx": 0, "phase": "cold", "total_duration_ms": 125661, "prompt_tokens": 30, "completion_tokens": 742, "tokens_per_sec": 5.904775546907951, "finish_reason": "stop", "reasoning_chars": 1254, "content_chars": 1825, "error": null}
{"type": "call", "cell_id": "pavilion:llamacpp:qwen3:30b-a3b-iq2xxs-think500", "model_key": "qwen3-30b-a3b-pavilion", "prompt_id": "P-HARD", "run_idx": 1, "phase": "warm", "total_duration_ms": 68390, "prompt_tokens": 30, "completion_tokens": 547, "tokens_per_sec": 7.998245357508408, "finish_reason": "stop", "reasoning_chars": 1204, "content_chars": 1099, "error": null}
{"type": "call", "cell_id": "pavilion:llamacpp:qwen3:30b-a3b-iq2xxs-think500", "model_key": "qwen3-30b-a3b-pavilion", "prompt_id": "P-HARD", "run_idx": 2, "phase": "warm", "total_duration_ms": 56872, "prompt_tokens": 30, "completion_tokens": 456, "tokens_per_sec": 8.018005345336897, "finish_reason": "stop", "reasoning_chars": 898, "content_chars": 1099, "error": null}
{"type": "call", "cell_id": "pavilion:llamacpp:qwen3:30b-a3b-iq2xxs-think500", "model_key": "qwen3-30b-a3b-pavilion", "prompt_id": "P-HARD", "run_idx": 3, "phase": "warm", "total_duration_ms": 58696, "prompt_tokens": 30, "completion_tokens": 467, "tokens_per_sec": 7.956249148153196, "finish_reason": "stop", "reasoning_chars": 956, "content_chars": 983, "error": null}