85 phase 2 implement smart migration path with hybrid generator

[unclesp1d3r]

This pull request introduces a thread-safe, configurable LRU template cache to the convert command, improving performance and memory management when using custom Go text/template files for report generation. It also adds new command-line flags for template cache size, enhances template flag completion, and refactors output generation to leverage the cache. Comprehensive unit tests are included to ensure cache correctness and concurrency safety.

Template Caching and Output Generation Enhancements:

• Added a TemplateCache type with LRU eviction, supporting thread-safe caching of parsed templates, and integrated it into the batch processing flow in cmd/convert.go. The cache size is configurable via a new --template-cache-size flag. [1] [2] [3] [4]
• Refactored output generation to use a new generateOutputByFormat function, which utilizes the cached template for markdown and template output, and gracefully falls back to programmatic generation for other formats. [1] [2]

User Experience Improvements:

• Enhanced the --custom-template flag to clarify that it accepts Go text/template files and added shell completion for .tmpl files to improve usability.
• Added new error handling for missing or invalid template paths, with explicit error messages.

Testing and Dependencies:

• Added comprehensive unit tests for the template cache, including concurrency tests, in cmd/convert_test.go.
• Introduced the github.com/hashicorp/golang-lru/v2 dependency for efficient LRU cache implementation.

Minor Cleanups:

• Removed deprecated comments from internal/converter/markdown.go for clarity. [1] [2]

[coderabbitai]

Caution

Review failed

Failed to post review comments.

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 9eca3ab and bd49d0a.

⛔ Files ignored due to path filters (1)

• go.sum is excluded by !**/*.sum

📒 Files selected for processing (8)

• cmd/convert.go (5 hunks)
• cmd/convert_test.go (2 hunks)
• cmd/shared_flags.go (4 hunks)
• go.mod (1 hunks)
• internal/converter/markdown.go (0 hunks)
• internal/markdown/hybrid_generator.go (1 hunks)
• internal/markdown/hybrid_generator_comparison_test.go (1 hunks)
• internal/markdown/hybrid_generator_test.go (1 hunks)

💤 Files with no reviewable changes (1)

• internal/converter/markdown.go

🧰 Additional context used

📓 Path-based instructions (14)

**/go.mod

📄 CodeRabbit Inference Engine (.cursor/rules/go-organization.mdc)

**/go.mod: Keep dependencies up to date
Use go.mod for dependency management

Files:

• go.mod

{go.mod,go.sum}

📄 CodeRabbit Inference Engine (.cursor/rules/go-standards.mdc)

{go.mod,go.sum}: Use Go modules for dependency management
Pin dependency versions for stability in Go modules
Regularly update dependencies for security in Go modules

Files:

• go.mod

go.mod

📄 CodeRabbit Inference Engine (.cursor/rules/go-standards.mdc)

Minimize external dependencies in Go modules

The project must include a go.mod file at the root for Go module definition and dependencies.

Files:

• go.mod

**/*.go

📄 CodeRabbit Inference Engine (.cursor/rules/core-concepts.mdc)

**/*.go: Use gofmt formatting with tabs for indentation (Go standard).
Follow Go naming conventions: camelCase for variables/functions, PascalCase for types.
Always check errors and provide meaningful context using fmt.Errorf or errors.Wrap.
Use structured logging with log/slog instead of fmt.Printf.
Implement proper error handling with context preservation.
Never hardcode API keys, passwords, or sensitive data in source code.
Use environment variables or secure vaults for configuration secrets.
Always validate and sanitize user inputs.
Default to secure configurations.
Use restrictive file permissions (0600 for config files).
Avoid exposing sensitive information in error messages.
Use encoding/xml for XML parsing.
Implement proper error handling with context preservation.

**/*.go: Include proper error handling with context
Add structured logging for important operations
Validate all inputs and handle edge cases
Document new functions and types following Go conventions
Code follows Go formatting standards (gofmt)
All linting issues resolved (golangci-lint)
Error handling includes proper context
Logging uses structured format with appropriate levels
Input validation implemented where needed
Follow Go conventions and DEVELOPMENT_STANDARDS.md
Implement proper error handling with context
Use structured logging for important operations
Address all linting issues before committing

**/*.go: Every package should have a package comment
Start package documentation with Package packagename followed by a description
Use complete sentences and proper grammar in package documentation
Place package comment before package declaration
Document all exported functions and types
Use complete sentences starting with the function name in function documentation
Describe parameters, return values, and error conditions in function documentation
Include usage examples for complex functions
Document all exported types and interfaces
Explain the purpose and usage ...

Files:

• internal/markdown/hybrid_generator_comparison_test.go
• internal/markdown/hybrid_generator_test.go
• internal/markdown/hybrid_generator.go
• cmd/shared_flags.go
• cmd/convert.go
• cmd/convert_test.go

**/*_test.go

📄 CodeRabbit Inference Engine (.cursor/rules/core-concepts.mdc)

Use table-driven tests for multiple scenarios.

**/*_test.go: Place tests in *_test.go files
Place tests in the same package as the code
Use table-driven tests for multiple scenarios
Create test helpers for common setup
Use benchmarks to measure performance

**/*_test.go: Use table-driven tests for multiple scenarios in Go tests
Use t.Run() for subtests in Go tests
Aim for greater than 80% test coverage in Go code
Test both success and failure cases in Go tests
Use descriptive test names that indicate what is being tested in Go tests
Use TestFunctionName pattern for test functions in Go tests
Use TestFunctionName_Scenario for specific test cases in Go tests
Use go test -race to detect race conditions in Go tests

**/*_test.go: Place tests in the same package as the code being tested
Use *_test.go files with descriptive names
Group related tests using t.Run() for subtests
Use table-driven tests for multiple test cases
Keep test files focused and well-organized
Test function names should be descriptive and follow the pattern: TestFunctionName_Scenario_ExpectedResult
Benchmark functions should be named with the Benchmark prefix and use the testing.B parameter
Create test helpers in *_test.go files
Use testing.TB interface for shared helpers
Create test fixtures and mock data
Use t.Helper() for helper functions
Keep helpers simple and focused
Use //go:build integration build tags for integration tests
Test component interactions in integration tests
Use real dependencies when possible in integration tests
Clean up test data after each integration test
Use separate test databases or mock external services in integration tests
Always test error conditions
Verify error messages contain useful information
Test both expected and unexpected errors
Use errors.Is() and errors.As() for error checking
Test error wrapping and context
Use constants for test data
Create realistic test scenarios
Avoid hardcoded magic numbers in tests
Use test fixtures for comp...

Files:

• internal/markdown/hybrid_generator_comparison_test.go
• internal/markdown/hybrid_generator_test.go
• cmd/convert_test.go

**/*.{go,md}

📄 CodeRabbit Inference Engine (.cursor/rules/go-documentation.mdc)

Use consistent formatting and style in documentation

Files:

• internal/markdown/hybrid_generator_comparison_test.go
• internal/markdown/hybrid_generator_test.go
• internal/markdown/hybrid_generator.go
• cmd/shared_flags.go
• cmd/convert.go
• cmd/convert_test.go

**/*.{go,_example_test.go}

📄 CodeRabbit Inference Engine (.cursor/rules/go-documentation.mdc)

Include error handling examples

Files:

• internal/markdown/hybrid_generator_comparison_test.go
• internal/markdown/hybrid_generator_test.go
• internal/markdown/hybrid_generator.go
• cmd/shared_flags.go
• cmd/convert.go
• cmd/convert_test.go

**/internal/**/*.go

📄 CodeRabbit Inference Engine (.cursor/rules/go-organization.mdc)

Use internal/ for private packages

Files:

• internal/markdown/hybrid_generator_comparison_test.go
• internal/markdown/hybrid_generator_test.go
• internal/markdown/hybrid_generator.go

internal/**/*.go

📄 CodeRabbit Inference Engine (.cursor/rules/go-standards.mdc)

Use internal/ for private application logic in Go projects

Files:

• internal/markdown/hybrid_generator_comparison_test.go
• internal/markdown/hybrid_generator_test.go
• internal/markdown/hybrid_generator.go

{cmd,internal,pkg,testdata,docs}/**

📄 CodeRabbit Inference Engine (.cursor/rules/project-structure.mdc)

Source code must be organized under the cmd/, internal/, pkg/, testdata/, and docs/ directories at the root.

Files:

• internal/markdown/hybrid_generator_comparison_test.go
• internal/markdown/hybrid_generator_test.go
• internal/markdown/hybrid_generator.go
• cmd/shared_flags.go
• cmd/convert.go
• cmd/convert_test.go

internal/**

📄 CodeRabbit Inference Engine (.cursor/rules/project-structure.mdc)

Private application logic and business rules should be placed in the internal/ directory.

Files:

• internal/markdown/hybrid_generator_comparison_test.go
• internal/markdown/hybrid_generator_test.go
• internal/markdown/hybrid_generator.go

internal/{audit,config,converter,display,export,log,markdown,model,parser,plugin,plugins,processor,templates,validator,constants}/**

📄 CodeRabbit Inference Engine (.cursor/rules/project-structure.mdc)

The internal package should be structured with subdirectories for audit, config, converter, display, export, log, markdown, model, parser, plugin, plugins, processor, templates, validator, constants, and walker.go.

Files:

• internal/markdown/hybrid_generator_comparison_test.go
• internal/markdown/hybrid_generator_test.go
• internal/markdown/hybrid_generator.go

cmd/**/*.go

📄 CodeRabbit Inference Engine (.cursor/rules/compliance-standards.mdc)

cmd/**/*.go: Use -f flag for force overwrite operations in CLI
Provide explicit validation via CLI commands

cmd/**/*.go: Use cobra v1.8.0 for command organization in CLI development.
Use charmbracelet/fang for styled help, errors, and features in CLI configuration management.
Use spf13/viper for configuration parsing.
Use charmbracelet/lipgloss for terminal output styling.
Use charmbracelet/glamour for markdown rendering.
Use cobra for command organization with consistent verb patterns.
Use charmbracelet/fang for configuration management.
Use charmbracelet/lipgloss for styled terminal output.
Use charmbracelet/glamour for markdown rendering.

Use cmd/ for CLI command definitions in Go projects

Files:

• cmd/shared_flags.go
• cmd/convert.go
• cmd/convert_test.go

cmd/*.go

📄 CodeRabbit Inference Engine (.cursor/rules/project-structure.mdc)

Command entry points should be implemented as Go files within the cmd/ directory (e.g., convert.go, display.go, validate.go, root.go).

Files:

• cmd/shared_flags.go
• cmd/convert.go
• cmd/convert_test.go

🧬 Code Graph Analysis (6)

internal/markdown/hybrid_generator_comparison_test.go (8)

internal/model/opnsense.go (1)

• OpnSenseDocument (9-46)

internal/log/logger.go (1)

• New (44-82)

internal/converter/markdown.go (1)

• NewMarkdownBuilder (61-66)

internal/markdown/hybrid_generator.go (2)

• NewHybridGenerator (28-40)
• NewHybridGeneratorWithTemplate (43-60)

internal/model/system.go (2)

• System (33-87)
• Firmware (106-114)

internal/constants/constants.go (1)

• Version (7-7)

internal/model/interfaces.go (2)

• Interfaces (57-59)
• Interface (156-196)

internal/model/security.go (5)

• Filter (93-95)
• Rule (139-157)
• InterfaceList (11-11)
• Source (160-163)
• Destination (166-170)

internal/markdown/hybrid_generator_test.go (6)

internal/converter/markdown.go (2)

• NewMarkdownBuilder (61-66)
• ErrNilOpnSenseDocument (77-77)

internal/log/logger.go (1)

• New (44-82)

internal/markdown/hybrid_generator.go (2)

• NewHybridGenerator (28-40)
• NewHybridGeneratorWithTemplate (43-60)

internal/model/opnsense.go (1)

• OpnSenseDocument (9-46)

internal/model/system.go (1)

• System (33-87)

cmd/convert.go (1)

• FormatMarkdown (129-129)

internal/markdown/hybrid_generator.go (7)

internal/converter/markdown.go (2)

• ReportBuilder (34-51)
• ErrNilOpnSenseDocument (77-77)

internal/log/logger.go (2)

• Logger (22-24)
• New (44-82)

internal/model/opnsense.go (1)

• OpnSenseDocument (9-46)

internal/markdown/options.go (1)

• Format (14-14)

cmd/convert.go (1)

• FormatMarkdown (129-129)

internal/markdown/generator.go (2)

• NewMarkdownGeneratorWithTemplates (46-67)
• Generator (24-27)

internal/model/enrichment.go (2)

• EnrichDocument (227-244)
• EnrichedOpnSenseDocument (44-64)

cmd/shared_flags.go (1)

cmd/convert.go (1)

• DefaultTemplateCacheSize (136-136)

cmd/convert.go (8)

internal/log/logger.go (2)

• New (44-82)
• Logger (22-24)

internal/model/opnsense.go (1)

• OpnSenseDocument (9-46)

internal/markdown/options.go (5)

• Options (88-143)
• Format (14-14)
• FormatMarkdown (18-18)
• FormatJSON (20-20)
• FormatYAML (22-22)

internal/display/display.go (1)

• Options (111-116)

internal/model/services.go (1)

• Options (49-55)

internal/markdown/generator.go (1)

• NewMarkdownGenerator (37-42)

internal/converter/markdown.go (1)

• NewMarkdownBuilder (61-66)

internal/markdown/hybrid_generator.go (1)

• NewHybridGenerator (28-40)

cmd/convert_test.go (1)

cmd/convert.go (2)

• NewTemplateCache (58-60)
• ErrNoTemplateSpecified (124-124)

🔇 Additional comments (1)

go.mod (1)

16-16: LRU dependency addition looks appropriate; version is pinned

Adding github.com/hashicorp/golang-lru/v2 v2.0.7 is consistent with the new template cache. Keep it pinned and run go mod tidy to ensure go.sum is up to date.

If not already done, please verify:

• go mod tidy produces no changes
• The cache usage is limited to the convert workflow to minimize transitive impact

Summary by CodeRabbit

• New Features
  • Hybrid Markdown generator combining template-based and programmatic output with graceful fallback.
  • Template caching for faster repeated conversions; cache size configurable via --template-cache-size.
  • Enhanced format handling: JSON/YAML routed through a unified generation path; unknown formats default to Markdown with a warning.
  • Shell completion for --custom-template suggests .tmpl files.
• Performance
  • Clears cache after batches to manage memory and improve long-run stability.
• Tests
  • Extensive unit and comparison tests for caching and hybrid generation.
• Chores
  • Added dependency for LRU caching.

Walkthrough

Adds a thread-safe LRU template cache and wires it into convert’s flow; introduces a HybridGenerator for markdown supporting template/programmatic modes; adds a new CLI flag for cache sizing with template file completion; updates dependency on golang-lru; removes deprecation comments; adds extensive tests for cache and hybrid generator.

Changes

Cohort / File(s)	Summary of Changes
Convert command: caching and flow cmd/convert.go	Introduces TemplateCache (LRU, thread-safe) with Get/Clear/Size, default size and format constants, and ErrNoTemplateSpecified. Integrates cache into RunE, adds loadCustomTemplate, generateOutputByFormat, and generateWithHybridGenerator; clears cache after batch; adds template and LRU imports.
Convert command tests cmd/convert_test.go	Adds unit tests for TemplateCache behavior and concurrency; validates errors, size, Clear, and instance reuse.
Shared flags and completion cmd/shared_flags.go	Adds --template-cache-size flag (default from DefaultTemplateCacheSize); updates custom-template help; implements .tmpl filename completion via os.ReadDir; no public API changes.
Markdown hybrid generator internal/markdown/hybrid_generator.go	Adds HybridGenerator supporting template-based and programmatic generation, constructors (with/without template), Generate routing, setters/getters for template and builder, helper logic for context enrichment and selection.
Hybrid generator tests (unit) internal/markdown/hybrid_generator_test.go	Tests constructors, generation paths, comprehensive mode, selection logic, error handling, and accessors.
Hybrid generator tests (comparison/parity) internal/markdown/hybrid_generator_comparison_test.go	Compares programmatic vs template outputs (standard/comprehensive), tests feature flags and fallback behavior; includes helpers and fixtures.
Markdown converter comments internal/converter/markdown.go	Removes deprecation comments; no functional changes.
Dependencies go.mod	Adds github.com/hashicorp/golang-lru/v2 v2.0.7 to require block.

Sequence Diagram(s)

sequenceDiagram
  participant User
  participant CLI as convert cmd
  participant Flags as shared flags
  participant Cache as TemplateCache (LRU)
  participant FS as Filesystem
  participant Gen as HybridGenerator

  User->>CLI: run convert --format [md/json/yaml] [--custom-template T] [--template-cache-size N]
  CLI->>Flags: parse flags
  Flags-->>CLI: format, template path, cache size
  CLI->>Cache: instantiate(size=N)
  alt custom template provided
    CLI->>Cache: Get(T)
    alt cache miss
      Cache->>FS: read T
      FS-->>Cache: template bytes
      Cache->>Cache: parse and store
    end
  end
  CLI->>Gen: create (builder, optional pre-parsed template)
  CLI->>Gen: Generate(data, options)
  Gen-->>CLI: output (md/json/yaml)
  CLI->>Cache: Clear() (post-batch)
  CLI-->>User: write output

Loading

sequenceDiagram
  participant Gen as HybridGenerator
  participant Builder as MarkdownBuilder
  participant Tmpl as Template Engine

  Gen->>Gen: shouldUseTemplate?(format, opts, template presence)
  alt Use template (markdown)
    Gen->>Tmpl: render(md with context)
    Tmpl-->>Gen: markdown string
  else Programmatic path
    Gen->>Builder: Build report (standard/comprehensive)
    Builder-->>Gen: markdown string
  end
  Gen-->>Caller: result

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~40 minutes

Suggested labels

enhancement, go

Poem

Cache spins its quiet reel, LRU at the gate,
Templates blink from disk to RAM—no need to wait.
Hybrid paths converge, where code and text agree,
Flags whisper size and .tmpl filigree.
Markdown hums, JSON nods, YAML joins along—
Offline, structured, steady strong.

✨ Finishing Touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch 85-phase-2-implement-smart-migration-path-with-hybrid-generator

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai generate unit tests to generate unit tests for this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[unclesp1d3r]

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ code/snyk check is complete. No issues have been found. (View Details)

[Copilot]

Pull Request Overview

This pull request introduces a thread-safe, configurable LRU template cache system to the convert command to improve performance and memory management when using custom Go text/template files. It implements a hybrid generator architecture that supports both template-based and programmatic markdown generation with seamless migration capabilities.

Key changes:

• Added TemplateCache with LRU eviction and configurable size via --template-cache-size flag
• Implemented HybridGenerator for dual-mode markdown generation (template vs programmatic)
• Enhanced template flag completion and error handling for better user experience

Reviewed Changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
cmd/convert.go	Added TemplateCache implementation with LRU eviction, new format constants, and refactored output generation to use hybrid approach
cmd/convert_test.go	Added comprehensive unit tests for template cache functionality including concurrency safety tests
cmd/shared_flags.go	Enhanced --custom-template flag description, added shell completion for .tmpl files, and new --template-cache-size flag
internal/markdown/hybrid_generator.go	New hybrid generator implementation supporting both template and programmatic generation modes
internal/markdown/hybrid_generator_test.go	Comprehensive unit tests for the hybrid generator covering all generation modes and edge cases
internal/markdown/hybrid_generator_comparison_test.go	Comparison tests ensuring output equivalence between generation modes and feature flag testing
internal/converter/markdown.go	Removed deprecated comments for code clarity
go.mod	Added github.com/hashicorp/golang-lru/v2 dependency for LRU cache implementation