layout: true class: typo, typo-selection --- count: false class: nord-dark, middle, center # π Google Benchmark vs nanobench ## Migrating C++ Benchmarking Framework in the AI Era @luk036 π¨βπ» Β· 2026 π --- ### π Agenda .pull-left[ **Part 1: Context** πΊοΈ - The ellalgo-cpp Project - Ell vs EllStable - Why Benchmark? **Part 2: Migration** π - Google Benchmark β nanobench - Source Code Transformation - Build System Changes ] .pull-right[ **Part 3: Results & Lessons** π - Correctness Verification - Performance Comparison - AI-Era Implications ] --- class: nord-light, middle, center ## πΊοΈ Part 1: Context --- ### The ellalgo-cpp Project ποΈ A C++ implementation of the **Ellipsoid Method** for convex optimization: .mermaid[ <pre> graph LR Ell["Ell\n(Standard)"] --> Algo["Cutting-Plane\nAlgorithms\nfeas / optim / q"] EllStable["EllStable\n(Numerically Stable)"] --> Algo Algo --> Oracles["Applications\nProfit, SVM,\nLowpass, LMI,\nMax-Cut"] style Ell fill:#2196f3,color:#fff style EllStable fill:#4caf50,color:#fff style Algo fill:#ff9800 style Oracles fill:#9c27b0,color:#fff </pre> ] Two ellipsoid variants with **identical interface** but different internal numerics: - **Ell** β Standard rank-one update ($O(n^2)$) - **EllStable** β LDLα΅ factorization update ($O(n^2)$, better numerical stability) The project uses both **xmake** and **CMake** build systems. π¦ --- ### Why Benchmark? π― Benchmarking answers critical questions: .mermaid[ <pre> graph TD Q1["Is EllStable slower\nthan Ell?"] --> M["Measure &\nCompare"] Q2["How does\nperformance scale\nwith dimensions?"] --> M Q3["Which framework\nis better for\nCI pipelines?"] --> M M --> R1["Informed decisions"] M --> R2["Performance budgets"] M --> R3["Regression detection"] style Q1 fill:#2196f3 style Q2 fill:#ff9800 style Q3 fill:#9c27b0 style M fill:#f44336,color:#fff style R1 fill:#4caf50,color:#fff style R2 fill:#4caf50,color:#fff style R3 fill:#4caf50,color:#fff </pre> ] The project had **4 benchmark files** using Google Benchmark. π --- ### Benchmark Suite Overview π | File | What it benchmarks | Framework | |------|-------------------|-----------| | `BM_ell.cpp` | Ell vs EllStable (profit oracle, 6 variants) | Google Benchmark | | `BM_ell_compare.cpp` | Ell vs EllStable (lowpass, scaling, verification) | Google Benchmark | | `BM_lmi.cpp` | LMI oracle optimization | Google Benchmark | | `BM_lowpass.cpp` | Lowpass filter design | Google Benchmark | The project used `add_requires("benchmark")` in xmake and `CPMAddPackage` for Google Benchmark in CMake. π οΈ --- class: nord-light, middle, center ## π Part 2: The Migration --- ### Why nanobench? π€ .mermaid[ <pre> graph TD GB["Google Benchmark\nπ§ͺ"]-->|"Migrate"| NB["nanobench\nβ‘"] GB-->|"Pros:"| G1["Industry standard\nRich feature set\nWidely known"] GB-->|"Cons:"| G2["Larger footprint\nCPM dependency\nTemplate-heavy API"] NB-->|"Pros:"| N1["Header-only\nZero dependencies\nSimpler API\nFaster compile"] NB-->|"Cons:"| N2["Smaller community\nFewer features"] style GB fill:#2196f3,color:#fff style NB fill:#4caf50,color:#fff style G1 fill:#e3f2fd style G2 fill:#ffcdd2 style N1 fill:#c8e6c9 style N2 fill:#ffcdd2 </pre> ] **Key advantage for AI-assisted development**: nanobench's simpler API means fewer tokens to process and less room for hallucination. π€ --- ### Before: Google Benchmark Pattern π΅ ```cpp #include "benchmark/benchmark.h" static void BM_MyBench(benchmark::State& state) { while (state.KeepRunning()) { auto result = do_something(); benchmark::DoNotOptimize(result); } } BENCHMARK(BM_MyBench); BENCHMARK_MAIN(); ``` - Registration via `BENCHMARK()` macro - State loop managed by framework - `DoNotOptimize` sink - `BENCHMARK_MAIN()` boilerplate --- ### After: nanobench Pattern π’ ```cpp #define ANKERL_NANOBENCH_IMPLEMENT #include <nanobench.h> int main() { ankerl::nanobench::Bench bench; bench.title("My Benchmarks").unit("op") .warmup(100).epochs(50); bench.run("my_bench", [&] { auto result = do_something(); ankerl::nanobench::doNotOptimizeAway(result); }); } ``` - No macros β pure lambdas π― - No state loop β framework handles iterations - `doNotOptimizeAway` instead of `DoNotOptimize` - Custom `main()` β full control --- ### The Transformation: BM_ell.cpp π **Before (Google Benchmark):** ```cpp static void ELL_normal(benchmark::State& state) { while (state.KeepRunning()) { Ell<Vec> ellip{100.0, Vec{0.0, 0.0}}; ProfitOracle omega{unit_price, scale, limit, elasticities, price_out}; double gamma = 0.0; auto result = cutting_plane_optim(omega, ellip, gamma); benchmark::DoNotOptimize(result); } } BENCHMARK(ELL_normal); ``` **After (nanobench):** ```cpp bench.run("ELL_normal", [&] { Ell<Vec> ellip{100.0, Vec{0.0, 0.0}}; ProfitOracle omega{unit_price, scale, limit, elasticities, price_out}; double gamma = 0.0; auto result = cutting_plane_optim(omega, ellip, gamma); ankerl::nanobench::doNotOptimizeAway(result); }); ``` --- ### The Challenge: Slow Benchmarks π’ .pull-left[ The original conversion had a **critical bug**: .mermaid[ <pre> graph TD Problem["Problem: All benchmarks used\nepochs=50, warmup=100"] --> LP["Lowpass benchmarks\nmax_iters=50000\n~250ms per run"] Problem --> SC["Scaling benchmarks\nmax_iters=4000\n~50ms per run"] LP --> Result["Result: BM_ell_compare\nwould take >5 minutes β°"] SC --> Result style Problem fill:#f44336,color:#fff style LP fill:#ff9800 style SC fill:#ff9800 style Result fill:#f44336,color:#fff </pre> ] ] .pull-right[ **Fix**: Split into two bench configs: ```cpp // Fast benchmarks β 50 epochs ankerl::nanobench::Bench fast_bench; fast_bench.warmup(100).epochs(50); // Slow benchmarks β 3 epochs (minimal repetition) ankerl::nanobench::Bench slow_bench; slow_bench.warmup(1).epochs(3).minEpochIterations(1); ``` ] --- ### Benchmark Runtime Before vs After β±οΈ .mermaid[ <pre> graph LR subgraph Before["Before (nanobench default config)"] B1["BM_ell_compare\n>5 minutes β°"] B2["BM_lowpass\n>5 minutes β°"] end subgraph After["After (epochs tuned)"] A1["BM_ell_compare\n30 seconds β "] A2["BM_lowpass\n2.5 seconds β "] end style Before fill:#ffcdd2 style After fill:#c8e6c9 style B1 fill:#f44336,color:#fff style B2 fill:#f44336,color:#fff style A1 fill:#4caf50,color:#fff style A2 fill:#4caf50,color:#fff </pre> ] **Lesson**: nanobench's automatic iteration estimation multiplies runtime for expensive benchmarks. Always tune `epochs` and `minEpochIterations` for your use case. π --- ### Build System Changes π οΈ **xmake.lua** β Before vs After: ```lua -- Before add_requires("benchmark", { alias = "benchmark" }) target("test_ell") add_packages("benchmark") -- After add_requires("nanobench", { alias = "nanobench" }) target("test_ell") add_packages("nanobench") ``` **CMake** β `bench/CMakeLists.txt`: ```cmake # Before CPMAddPackage(NAME benchmark GITHUB_REPOSITORY google/benchmark ...) target_link_libraries(${TARGET} benchmark::benchmark ...) # After CPMAddPackage(NAME nanobench GITHUB_REPOSITORY martinus/nanobench ...) target_link_libraries(${TARGET} nanobench ...) ``` Clean, simple, one-for-one replacement. β --- ### What Changed: File-by-File π .font-sm[ | File | Lines Changed | Type | |------|:------------:|------| | `bench/BM_ell.cpp` | 82 β 48 (β34) | Source migration | | `bench/BM_ell_compare.cpp` | 387 β 331 (β56) | Source migration + fix | | `bench/BM_lmi.cpp` | 218 β 86 (β132) | Source migration | | `bench/BM_lowpass.cpp` | 70 β 50 (β20) | Source migration + fix | | `bench/CMakeLists.txt` | 57 β 50 (β7) | Build system | | `xmake.lua` | 157 β 150 (β7) | Build system | | **Total** | **β256 lines** | | ] **Result**: Cleaner code, fewer dependencies, simpler build. π― --- class: nord-light, middle, center ## π Part 3: Results & Lessons --- ### Before: All Tests Passing β ```bash $ xmake run test_ellalgo [doctest] test cases: 125 | 125 passed | 0 failed | 0 skipped [doctest] assertions: 414 | 414 passed | 0 failed | [doctest] Status: SUCCESS! ``` **125 test cases, 414 assertions β all passing** before and after the migration. π§ͺ The nanobench migration was **purely a build-system and benchmarking change** β zero behavioral changes to the library code. --- ### After: Benchmark Results π ```bash $ xmake run test_ell | ns/op | op/s | Ell vs EllStable (profit oracle) |-------:|-------:|:-------------------------------- | 141,420 | 7,071 | `ELL_normal` β Ell | 129,830 | 7,702 | `ELL_stable` β EllStable $ ctest --test-dir build -C Release 1/5 EllAlgoTests .............. 0.68s β 2/5 BM_ell .................... 0.44s β 3/5 BM_ell_compare ............ 30.13s β 4/5 BM_lmi .................... 0.32s β 5/5 BM_lowpass ................ 2.56s β ``` All benchmarks pass via both **xmake** and **CMake** build systems. π --- ### Verification: Ell vs EllStable β `BM_ell_compare` includes a **correctness verification** step before benchmarking: ```text Profit normal PASS | iters: Ell=83 Stable=83 Profit robust PASS | iters: Ell=90 Stable=90 LP-32 par PASS | iters: Ell=26027 Stable=26125 LP-32 ser PASS | iters: Ell=40740 Stable=40621 LP-48 par PASS | iters: Ell=35014 Stable=35098 LP-64 par PASS | iters: Ell=27805 Stable=27926 Rand-16 PASS | iters: Ell=2067 Stable=2067 Rand-32 PASS | iters: Ell=4000 Stable=4000 Rand-64 PASS | iters: Ell=2000 Stable=2000 Result: 9/9 passed. β ``` Ell and EllStable produce **identical results** β the stable variant is truly equivalent. π― --- ### What We Learned π .mermaid[ <pre> graph TD subgraph Technical["Technical Lessons"] T1["nanobench needs epoch\ntuning for expensive\nbenchmarks"] T2["CMake + xmake dual build\nmeans updating both\nbuild systems"] end subgraph Process["Process Lessons"] P1["Simpler APIs reduce\nAI hallucination risk"] P2["Always verify correctness\nbefore comparing perf"] P3["Dead and redundant\ncode should be removed"] end style Technical fill:#e3f2fd style Process fill:#e8f5e9 style T1 fill:#2196f3 style T2 fill:#ff9800 style P1 fill:#9c27b0,color:#fff style P2 fill:#4caf50,color:#fff style P3 fill:#f44336,color:#fff </pre> ] --- ### The AI-Assisted Development Angle π€ How does this experience relate to AI-assisted coding? | Dimension | Impact | |-----------|--------| | **Tool choice matters** | Simpler APIs β fewer AI errors | | **Build system duplication** | 2Γ work for AI agent (xmake + CMake) | | **Dead code detection** | AI should flag unused/unnecessary targets | | **Epoch tuning** | AI should recognize when benchmarks are too expensive | | **Verification first** | AI should validate correctness before measuring performance | The simplest code is the code the AI doesn't have to touch. π― --- ### Git History π The migration in **2 clean commits**: ```text d6e63f9 migrate: replace Google Benchmark with nanobench in benchmark sources 79f8ecd migrate: update build system to use nanobench ``` - **4 source files** converted (BM_ell, BM_ell_compare, BM_lmi, BM_lowpass) - **2 build system files** updated (xmake.lua, bench/CMakeLists.txt) - **β256 lines** of code overall π Clean, atomic, reviewable commits. π§Ή --- ### Summary π | Metric | Before | After | Delta | |--------|:------:|:-----:|:-----:| | **Benchmark framework** | Google Benchmark | nanobench | π’ Simpler | | **Test suite** | 125/125 passing | 125/125 passing | β | | **CMake build** | β | β | β | | **Xmake build** | β | β | β | | **Dependencies** | `benchmark` + transitive | `nanobench` (header-only) | π’ Lighter | | **Lines of code** | More | β256 | π’ Less | --- ### Key Takeaway πΌοΈ .mermaid[ <pre> graph TD Q["What was achieved\nin this session?"] --> R1["π¦ Migrated benchmarking\nframework"] Q --> R2["π§ Tuned epochs for\npractical runtime"] Q --> R3["β Verified all\n125 tests pass"] Q --> R4["π Confirmed Ell == EllStable\nin correctness"] R1 --> F["β Cleaner codebase\nSimpler builds\nSame correctness"] R2 --> F R3 --> F R4 --> F style Q fill:#ff9800,stroke:#e65100,stroke-width:3px style R1 fill:#2196f3,color:#fff style R2 fill:#f44336,color:#fff style R3 fill:#4caf50,color:#fff style R4 fill:#9c27b0,color:#fff style F fill:#4caf50,color:#fff,stroke:#2e7d32,stroke-width:3px </pre> ] > **TL;DR**: Replaced Google Benchmark with nanobench across 6 files, tuned epoch configs for fast CI, verified 125 tests pass, Ell == EllStable. π― --- count: false class: nord-dark, middle, center ## π Q&A **Google Benchmark vs nanobench** *Any questions?* π€ --- count: false class: nord-dark, middle, center # π Thank You! ## Benchmark Smarter with Simpler Tools! π β‘ π