feat(converter): add streaming generation for large configurations

[unclesp1d3r]

Summary

• Implement Tier 1 streaming optimization for memory-efficient output generation
• Add code quality improvements from Go code review
• Addresses issue Enhancement: Consider streaming generation for large configurations #143

Changes

Streaming Generation (issue #143)

• Add SectionWriter interface for io.Writer-based report generation
• Add StreamingGenerator interface extending Generator with GenerateToWriter
• Implement WriteStandardReport and WriteComprehensiveReport methods
• Preserve string-based API for future HTML conversion workflows

Code Quality Improvements

• Replace init() with sync.Once pattern for global registry
• Add context cancellation support to XML parser
• Use errors.Join() for proper error accumulation
• Use slices.Sorted(maps.Keys()) for deterministic map iteration
• Remove unnecessary runtime.GC() calls
• Fix golangci-lint configuration conflict

Test plan

• All existing tests pass
• New writer tests added (writer_test.go)
• just ci-check passes
• Tested with production-size configs (1-5 MB)

Closes #143

🤖 Generated with Claude Code

[coderabbitai]

Caution

Review failed

The pull request is closed.

📝 Walkthrough

Summary by CodeRabbit

• New Features

  • Added streaming output support for Markdown reports to enable efficient memory usage during report generation.
  • Added context cancellation support for XML parsing operations, enabling timeouts and cancellation of long-running parsing tasks.

• Bug Fixes

  • Improved deterministic output by sorting map keys in reports for consistent, reproducible results.

• Chores

  • Removed unused dependencies to reduce module footprint.
  • Updated linter configuration and refactored internal plugin registry initialization.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

Walkthrough

This PR implements streaming output generation for large OPNsense configurations to address memory efficiency concerns. It introduces a new SectionWriter interface and StreamingGenerator interface with corresponding implementations, adds context cancellation support to the XML parser, refactors error handling in the convert command, removes unused dependencies, and improves deterministic output through key sorting.

Changes

Cohort / File(s)	Summary
Streaming Writer Infrastructure internal/converter/builder/writer.go, internal/converter/builder/writer_test.go	Introduces SectionWriter interface and MarkdownBuilder streaming methods to write report sections directly to io.Writer. Implements WriteSystemSection, WriteNetworkSection, WriteSecuritySection, WriteServicesSection, and WriteStandardReport/WriteComprehensiveReport with header, TOC, and footer generation. Tests verify non-empty outputs and section content accuracy.
Streaming Generator Interface internal/converter/hybrid_generator.go	Adds StreamingGenerator interface extending Generator with GenerateToWriter method. Implements writer-based generation paths for Markdown, JSON, and YAML formats with fallback to string-based generation. Routes context and writer to appropriate format handlers.
Parser Context Propagation internal/parser/xml.go	Updates Parse method signature to use named context parameter (instead of _) and adds context cancellation checks in parsing loops. Removes forced runtime.GC() calls in decodeSection and decodeSysctl. Enables cancellation/timeout support for streaming XML parsing.
Convert Command Refactoring cmd/convert.go	Refactors error collection to aggregate errors in a slice and return combined error via errors.Join instead of error chaining. Introduces new generateToWriter helper function for streaming output to io.Writer. Adds io import for writer support.
Deterministic Map Output internal/processor/report.go	Adds sorted key iteration for Service.Details and statistics maps in addStatisticsList using maps.Keys and slices.Sorted to ensure reproducible Markdown output. Imports maps and slices standard library packages.
Plugin Registry Lazy Initialization internal/audit/plugin.go	Replaces eager global registry initialization with lazy, thread-safe singleton via GetGlobalRegistry() function. Removes exported GlobalRegistry variable in favor of unexported globalRegistry and globalRegistryOnce. Updates all registry references to call GetGlobalRegistry().
Config String Building internal/config/config.go	Simplifies combineValidationErrors by consolidating two separate string builders into single local sb variable. Maintains identical behavior while reducing variable overhead.
Linter Configuration .golangci.yml	Disables gocyclo linter in favor of cyclop with descriptive comment in disabled section. No functional code changes.
Dependency Cleanup go.mod	Removes 10 unused dependencies: sprig/v3, golang-lru/v2, mergo, goutils, semver/v3, uuid, xstrings, copystructure, reflectwalk, and decimal.

Sequence Diagram(s)

sequenceDiagram
    participant Client as Client
    participant Parser as XML Parser
    participant Generator as HybridGenerator
    participant Writer as SectionWriter
    participant Output as io.Writer

    Client->>Parser: Parse(ctx, reader)
    Parser->>Parser: Check ctx.Done()
    Parser-->>Generator: OpnSenseDocument

    Client->>Generator: GenerateToWriter(ctx, writer, doc, opts)
    Generator->>Generator: Determine format (Markdown/JSON/YAML)
    
    alt Markdown Format
        Generator->>Writer: WriteStandardReport(writer, doc)
        Writer->>Output: WriteHeader()
        Writer->>Output: WriteTOC()
        Writer->>Output: WriteSystemSection()
        Writer->>Output: WriteNetworkSection()
        Writer->>Output: WriteSecuritySection()
        Writer->>Output: WriteServicesSection()
        Writer->>Output: WriteFooter()
    else JSON/YAML Format
        Generator->>Output: Marshal and write directly
    end

    Output-->>Client: Streamed output (low memory)

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• refactor: remove JSON and YAML template files and update related func… #46: Modifies JSON/YAML generation in the output pipeline, complementing this PR's new writer-based generation paths for multiple formats.
• fix(display): remove validation from display command by default #37: Updates XML parsing API and cmd/display.go to work with the new Parse method signature that now requires context parameter.
• feat(test): Create comprehensive test suite with CI-stable performance baselines for ported methods #110: Adds Markdown generation testdata and tests that align with the new streaming writer infrastructure introduced here.

Suggested labels

type:feature, go, breaking_change, testing, dependencies, performance, priority:normal

Poem

🌊 Streaming flows where megabytes once gathered,
Section by section, no heap is battered,
Context cancels, maps sort true,
OPNsense configs now parse right through! 🚀

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch 143-enhancement-consider-streaming-generation-for-large-configurations

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov]

Codecov Report

❌ Patch coverage is 41.26984% with 111 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/converter/hybrid_generator.go	0.00%	54 Missing ⚠️
internal/converter/builder/writer.go	66.33%	18 Missing and 16 partials ⚠️
internal/audit/plugin.go	0.00%	8 Missing ⚠️
internal/processor/report.go	0.00%	7 Missing ⚠️
cmd/convert.go	33.33%	6 Missing ⚠️
internal/parser/xml.go	66.66%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!