feat(benchmarks): add comprehensive performance benchmarking suite for template vs programmatic comparison - #111
Conversation
🎉 Snyk checks have passed. No issues have been found so far.✅ security/snyk check is complete. No issues have been found. (View Details) |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
- Add markdown_bench_test.go comparing original vs programmatic approaches - Include small/medium/large dataset testing with memory profiling - Implement individual method benchmarks and concurrent testing - Document 71-74% performance improvements exceeding 30-50% target - Add justfile entries for CI integration and regression testing
|
Important Review skippedBot user detected. To trigger a single review, invoke the You can disable this status message by setting the 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Join our Discord community for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
Status, Documentation and Community
|
|
This pull request introduces a comprehensive benchmarking suite and supporting documentation to evaluate the performance of the new programmatic markdown generation approach versus the original template-based method. The changes include new benchmark tests for various report generation scenarios, detailed performance results, and updates to the build scripts to make running these benchmarks easier. Benchmarking and performance evaluation enhancements:
Build and developer workflow improvements:
|
…precation
- Red mode: replace 5 fabricated metadata counters with shared
stubMarker() emitting {not_implemented: true, stub: true}. New
TestRedModeMetadata_MarksStubsExplicitly pins the invariant;
existing tests refactored to use assertStubMarker helper. (#111)
- Red-mode test annotations + stub-marker regression test. (#156)
- ValidateModeConfig delegates to ParseReportMode as single source of
truth for valid mode names. (#176)
- Delete dead sharedAuditMode / sharedSelectedPlugins / sharedPluginDir
globals in cmd/shared_flags.go. Delete buildAuditOptions function;
drop auditOpts parameter from generateOutputByFormat; delete the
unreachable `if auditOpts.AuditMode != ""` branch. Closes TODO(#457).
~58 net lines of dead code removed. (#113)
- HybridGenerator honors context.Context: threaded ctx through
Generate / GenerateToWriter / all 10 internal generator methods.
13 ctx.Err() boundary checks added (entry + per-subsystem + pre-
marshal). FormatHandler interface gains ctx as first parameter
(Go convention). (#135)
- TestHybridGenerator_*_RespectsCanceledContext: 5-format table-
driven cancellation tests against Generate and GenerateToWriter;
largeFixture(10_000) helper added; all subtests pass under -race
with 50ms timing assertion. (#155)
- Config flat fields (Verbose, Debug, Quiet, Theme, Format) marked
// Deprecated. New (*Config).DeprecationWarnings() reports YAML /
env-var usage via viper.InConfig() + os.LookupEnv() (not IsSet,
which SetDefault pollutes). 4 regression tests.
internal/config/*_test.go excluded from SA1019 in .golangci.yml
since tests legitimately exercise the deprecated surface until
v2.0 removes it. GOTCHAS §21 + CHANGELOG Deprecated entry. (#179)
- Fix 2 errcheck issues in mode_controller_test.go assertStubMarker
helper (blank-identifier type-assertion reads tightened).
Todos resolved: #111, #113, #135, #155, #156, #176, #179.
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
…precation
- Red mode: replace 5 fabricated metadata counters with shared
stubMarker() emitting {not_implemented: true, stub: true}. New
TestRedModeMetadata_MarksStubsExplicitly pins the invariant;
existing tests refactored to use assertStubMarker helper. (#111)
- Red-mode test annotations + stub-marker regression test. (#156)
- ValidateModeConfig delegates to ParseReportMode as single source of
truth for valid mode names. (#176)
- Delete dead sharedAuditMode / sharedSelectedPlugins / sharedPluginDir
globals in cmd/shared_flags.go. Delete buildAuditOptions function;
drop auditOpts parameter from generateOutputByFormat; delete the
unreachable `if auditOpts.AuditMode != ""` branch. Closes TODO(#457).
~58 net lines of dead code removed. (#113)
- HybridGenerator honors context.Context: threaded ctx through
Generate / GenerateToWriter / all 10 internal generator methods.
13 ctx.Err() boundary checks added (entry + per-subsystem + pre-
marshal). FormatHandler interface gains ctx as first parameter
(Go convention). (#135)
- TestHybridGenerator_*_RespectsCanceledContext: 5-format table-
driven cancellation tests against Generate and GenerateToWriter;
largeFixture(10_000) helper added; all subtests pass under -race
with 50ms timing assertion. (#155)
- Config flat fields (Verbose, Debug, Quiet, Theme, Format) marked
// Deprecated. New (*Config).DeprecationWarnings() reports YAML /
env-var usage via viper.InConfig() + os.LookupEnv() (not IsSet,
which SetDefault pollutes). 4 regression tests.
internal/config/*_test.go excluded from SA1019 in .golangci.yml
since tests legitimately exercise the deprecated surface until
v2.0 removes it. GOTCHAS §21 + CHANGELOG Deprecated entry. (#179)
- Fix 2 errcheck issues in mode_controller_test.go assertStubMarker
helper (blank-identifier type-assertion reads tightened).
Todos resolved: #111, #113, #135, #155, #156, #176, #179.
Signed-off-by: UncleSp1d3r <unclesp1d3r@evilbitlabs.io>
This PR implements a comprehensive benchmarking suite to validate performance improvements from the programmatic markdown generation approach compared to the original converter implementation.
What's Added
Benchmark Suite (
internal/converter/markdown_bench_test.go)AssessRiskLevel,FilterSystemTunables,CalculateSecurityScoreb.ReportAllocs()Performance Results Documentation (
PERFORMANCE_BENCHMARK_RESULTS.md)Comprehensive analysis showing:
CI Integration
New justfile commands for regression testing:
Performance Validation
The benchmarks confirm exceptional performance gains:
Memory efficiency shows similarly impressive results with 4.5x reduction in allocations (8.80MB → 1.97MB).
Example Usage
The benchmark suite validates that the programmatic approach delivers production-ready performance improvements well beyond the target requirements, making it suitable for high-throughput markdown generation workflows.
Fixes #94.
💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.