Phase 3: Migrate Template Functions to Type-Safe Go Methods for Performance & Maintainability

[coderabbitai]

🎯 Phase 3 of Programmatic Markdown Generation Refactor

Parent Issue: #73 | Milestone: v2.0 | Type: Breaking Change

📖 Context

opnDossier currently uses Go templates with custom template functions (via Sprig) to generate markdown reports from OPNsense configuration data. While functional, this approach has several limitations:

Current Pain Points

• ⚠️ Runtime errors: Template parsing errors only caught at runtime
• 🐌 Performance overhead: Template parsing adds ~45% execution time
• 🔧 Limited IDE support: No IntelliSense or type checking for template functions
• 🧪 Testing complexity: Difficult to unit test individual template functions
• 📚 Maintenance burden: Dual code paths (template + programmatic)
• 🔍 Debugging difficulty: Template stack traces are cryptic

The Opportunity

By migrating all custom template functions to type-safe Go methods on the MarkdownBuilder, we gain:

• ✅ Compile-time safety: Catch errors during build, not at runtime
• ⚡ Performance: 30-50% faster report generation (no template parsing)
• 🛠️ Developer experience: Full IDE support with IntelliSense and debugging
• 🧪 Testability: Unit test each method independently
• 📦 Maintainability: Single, clean code path

🔍 Current State Analysis

Template Functions Inventory

The codebase currently has 20+ custom template functions in internal/markdown/generator.go:

Utility Functions:

• escapeTableContent - Escape markdown table special characters
• boolToString - Convert boolean to enabled/disabled
• formatBytes - Human-readable byte formatting
• truncate - String truncation with ellipsis
• sanitizeID - Create markdown-safe IDs

Data Transformation:

• filterTunables - Filter system tunables by security relevance
• groupServicesByStatus - Group services by running/stopped
• filterRulesByType - Filter firewall rules by type
• aggregatePackageStats - Calculate package statistics

Security Assessment:

• getRiskLevel - Assess and format security risk levels
• calculateSecurityScore - Overall security posture scoring
• assessServiceRisk - Per-service risk evaluation

Complex Formatters:

• formatInterfacesAsLinks - Generate markdown links for interfaces
• formatSystemStats - System statistics aggregation
• formatNetworkInterfaces - Network interface table formatting
• formatFirewallRules - Firewall rule table generation
• formatServices - Service status formatting
• formatUsers - User account table generation
• formatPackages - Package list formatting
• formatCertificates - Certificate information display

💡 Proposed Solution

Architecture

┌─────────────────────────────────────────────────────┐
│           OpnSense XML Configuration                │
└────────────────┬────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────┐
│              XML Parser (existing)                  │
└────────────────┬────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────┐
│          Data Model (model.OpnSenseDocument)        │
└────────────────┬────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────┐
│         MarkdownBuilder (Type-Safe Methods)         │
│  ┌────────────────────────────────────────────────┐ │
│  │ • FormatInterfaceLinks()                       │ │
│  │ • AssessRiskLevel()                            │ │
│  │ • FilterSystemTunables()                       │ │
│  │ • CalculateSecurityScore()                     │ │
│  │ • FormatNetworkInterfaces()                    │ │
│  │ • (All 20+ methods)                            │ │
│  └────────────────────────────────────────────────┘ │
└────────────────┬────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────┐
│            Markdown Report Output                   │
└─────────────────────────────────────────────────────┘

Implementation Strategy

This phase is broken into 10 detailed subtasks for systematic implementation:

📋 Planning & Mapping

• [Phase 3.1] Create template function to Go method mapping document #89 - [Phase 3.1] Create template function to Go method mapping document
  • Audit all existing template functions
  • Design Go method signatures
  • Establish implementation priority

🔧 Core Function Migration

• [Phase 3.2] Port utility functions (formatting, escaping, defaults) #90 - [Phase 3.2] Port utility functions (formatting, escaping, defaults)

  • Migrate basic formatting functions
  • Replace Sprig function dependencies

• [Phase 3.3] Port data transformation functions (filtering, aggregation) #91 - [Phase 3.3] Port data transformation functions (filtering, aggregation)

  • Migrate filtering and grouping logic
  • Optimize for large datasets

• [Phase 3.4] Port security assessment functions (risk levels, scoring) #92 - [Phase 3.4] Port security assessment functions (risk levels, scoring)

  • Implement security scoring algorithm
  • Port risk level assessment

✅ Testing & Validation

• [Phase 3.5] Create comprehensive test suite for ported methods #93 - [Phase 3.5] Create comprehensive test suite for ported methods

  • Unit tests for all methods (90%+ coverage)
  • Integration tests for parity validation
  • Edge case testing

• [Phase 3.6] Run performance benchmarks comparing template vs programmatic #94 - [Phase 3.6] Run performance benchmarks comparing template vs programmatic

  • Benchmark execution time
  • Profile memory usage
  • Validate 30-50% improvement target

🚀 Integration & Documentation

• [Phase 3.7] Update CLI to use programmatic mode by default #95 - [Phase 3.7] Update CLI to use programmatic mode by default

  • Make programmatic generation the default
  • Keep template mode with --use-template flag
  • Add backward compatibility flags

• [Phase 3.8] Update documentation for programmatic markdown generation #96 - [Phase 3.8] Update documentation for programmatic markdown generation

  • API reference for builder methods
  • Architecture diagrams
  • Usage examples

• 📘 Complete migration documentation for custom template users (v2.0 breaking change) #97 - [Phase 3.9] Add migration guide for custom template users

  • Template function → Go method mapping
  • Migration examples
  • Extension patterns

• Implement Template Mode Deprecation Notices and Migration Tooling for v2.0 #98 - [Phase 3.10] Add deprecation notices for template-only features

  • CLI warnings for template usage
  • Documentation deprecation notices
  • v3.0 removal timeline

Implementation Example

Before (Template):

{{ define "interfaces" }}
## Network Interfaces
Configured: {{ .interfaces | formatInterfacesAsLinks }}
Risk Level: {{ getRiskLevel .security.level }}
{{ end }}

After (Programmatic):

func (b *MarkdownBuilder) WriteInterfacesSection(data *model.OpnSenseDocument) {
    b.WriteHeader(2, "Network Interfaces")
    
    interfaceLinks := b.FormatInterfaceLinks(data.Interfaces)
    b.WriteParagraph("Configured: " + interfaceLinks)
    
    riskLevel := b.AssessRiskLevel(data.Security.Level)
    b.WriteParagraph("Risk Level: " + riskLevel)
}

📅 Implementation Timeline

Week 1: Core Migration

• Days 1-2: Function mapping ([Phase 3.1] Create template function to Go method mapping document #89) and utility functions ([Phase 3.2] Port utility functions (formatting, escaping, defaults) #90)
• Days 3-5: Data transformation ([Phase 3.3] Port data transformation functions (filtering, aggregation) #91) and security functions ([Phase 3.4] Port security assessment functions (risk levels, scoring) #92)

Week 2: Testing & Validation

• Days 1-3: Comprehensive test suite ([Phase 3.5] Create comprehensive test suite for ported methods #93)
• Days 4-5: Performance benchmarking ([Phase 3.6] Run performance benchmarks comparing template vs programmatic #94)

Week 3: Integration & Documentation

• Days 1-2: CLI updates ([Phase 3.7] Update CLI to use programmatic mode by default #95)
• Days 3-4: Documentation ([Phase 3.8] Update documentation for programmatic markdown generation #96, 📘 Complete migration documentation for custom template users (v2.0 breaking change) #97)
• Day 5: Deprecation notices (Implement Template Mode Deprecation Notices and Migration Tooling for v2.0 #98)

✅ Acceptance Criteria

Functional Requirements

• All 20+ template functions ported to MarkdownBuilder methods
• Each method has proper Go documentation with examples
• Zero runtime template parsing in default mode
• Template mode still available via --use-template flag
• Full backward compatibility for existing workflows

Quality Requirements

• 90%+ test coverage for all ported methods
• 100% coverage for security assessment functions
• All edge cases tested (empty data, nil values, special characters)
• Integration tests validate functional parity with templates

Performance Requirements

• 30-50% reduction in report generation time
• 20-30% reduction in memory allocations
• Benchmarks documented and tracked

Documentation Requirements

• API documentation for all builder methods
• Migration guide with examples
• Architecture documentation updated
• Deprecation timeline clearly communicated

🎯 Success Metrics

Performance Targets

• Execution Time: 30-50% faster than template mode
• Memory Usage: 20-30% reduction in allocations
• Build Time: Errors caught at compile-time

Quality Targets

• Test Coverage: >90% overall, 100% for security functions
• Bug Reduction: Eliminate runtime template errors
• Developer Velocity: Faster feature development with IDE support

Adoption Targets

• Default Usage: Programmatic mode used by default
• Migration: Clear path for custom template users
• Deprecation: Template mode removed by v3.0

🔗 Dependencies

Prerequisite Phases

• ✅ Phase 1: Core refactoring (assumed complete)
• ✅ Phase 2: Builder infrastructure (assumed complete)

Blocking Issues

• None - ready to begin implementation

Related Work

• Parent issue: Architecture: Migrate from template-based to programmatic builder-pattern markdown generation #73
• Original discussion: Architecture: Migrate from template-based to programmatic builder-pattern markdown generation #73 (comment)

🏷️ Breaking Changes (v2.0)

⚠️ This is a breaking change for users with custom templates:

1. Default Behavior Change

   • Before: Templates used by default
   • After: Programmatic generation by default
   • Migration: Add --use-template flag

2. Template Function Removal

   • Custom template functions deprecated
   • Will be removed in v3.0
   • Use MarkdownBuilder methods instead

3. API Changes

   • Template-based API deprecated
   • New programmatic API is the standard
   • Migration guide provided in 📘 Complete migration documentation for custom template users (v2.0 breaking change) #97

📚 Additional Resources

• Migration Guide: See 📘 Complete migration documentation for custom template users (v2.0 breaking change) #97
• API Documentation: See [Phase 3.8] Update documentation for programmatic markdown generation #96
• Benchmarks: See [Phase 3.6] Run performance benchmarks comparing template vs programmatic #94
• Architecture: docs/architecture.md (to be updated in [Phase 3.8] Update documentation for programmatic markdown generation #96)

---

Note: This issue tracks the overall Phase 3 effort. Implementation work is split across subtasks #89-98. Please reference the appropriate subtask when contributing.

[unclesp1d3r]

@coderabbitai

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

@unclesp1d3r, I see you've tagged me on issue #86 about porting template functions to Go methods as part of the programmatic Markdown generation refactor.

How can I assist you with this task? Would you like me to:

• Analyze the existing template functions and suggest implementation patterns for the Go methods?
• Help create unit tests for the ported methods?
• Review the current codebase to identify all template functions that need porting?
• Provide suggestions for the builder API design?

Please let me know what specific aspect of this refactoring you'd like help with!

[unclesp1d3r]

@coderabbitai Examine the issue, and its parent, and develop a proposed solution. Additionally, enhance this task with recommended labels and other metadata.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# First, let's examine the parent issue #73 to understand the full context
gh issue view 73 --repo EvilBit-Labs/opnDossier

echo "---"

# Search for template function definitions in the codebase
echo "Searching for template function definitions..."
rg -A 5 "template\.FuncMap|\.Funcs\(" --type go

echo "---"

# Look for the specific template functions mentioned
echo "Searching for specific template functions..."
rg "formatInterfacesAsLinks|getRiskLevel|filterTunables|formatSystemStats|formatNetworkInterfaces" --type go -A 3

echo "---"

# Find template files to understand usage
echo "Finding template files..."
fd -e tmpl -e tpl -e template --exec head -20 {}

echo "---"

# Look for any existing builder pattern or markdown generation code
echo "Searching for existing builder/markdown generation patterns..."
rg "type.*Builder|MarkdownBuilder|ReportBuilder" --type go -A 5

Length of output: 42046

---

@unclesp1d3r, I've examined both issue #86 and its parent issue #73. Let me provide a comprehensive solution proposal for porting the template functions to Go methods.

📋 Proposed Solution

Current State Analysis

The codebase already has a solid foundation with:

• A ReportBuilder interface and MarkdownBuilder implementation in internal/converter/markdown.go
• A hybrid generator supporting both programmatic and template modes
• Existing builder methods for core sections (System, Network, Security, Services)
• Template functions currently defined in internal/markdown/generator.go

Implementation Plan

Phase 3.1: Core Template Function Migration (Week 1)

Port the most commonly used template functions:

// internal/converter/markdown_functions.go
package converter

import (
    "fmt"
    "strings"
    "github.com/EvilBit-Labs/opnDossier/internal/model"
)

// Add these methods to MarkdownBuilder

// Port: getRiskLevel template function
func (b *MarkdownBuilder) AssessRiskLevel(severity string) string {
    switch strings.ToLower(severity) {
    case "critical":
        return "🔴 Critical Risk"
    case "high":
        return "🟠 High Risk"
    case "medium":
        return "🟡 Medium Risk"
    case "low":
        return "🟢 Low Risk"
    default:
        return "⚪ Unknown Risk"
    }
}

// Port: filterTunables template function  
func (b *MarkdownBuilder) FilterSystemTunables(tunables []model.SysctlItem, includeTunables bool) []model.SysctlItem {
    if includeTunables {
        return tunables
    }
    
    // Filter out common non-security tunables
    filtered := make([]model.SysctlItem, 0)
    securityPrefixes := []string{
        "net.inet.ip.forwarding",
        "net.inet6.ip6.forwarding", 
        "kern.securelevel",
        "security.",
        "net.inet.tcp.blackhole",
    }
    
    for _, item := range tunables {
        for _, prefix := range securityPrefixes {
            if strings.HasPrefix(item.Tunable, prefix) {
                filtered = append(filtered, item)
                break
            }
        }
    }
    return filtered
}

// Port: formatInterfacesAsLinks (already exists, enhance it)
func (b *MarkdownBuilder) FormatInterfaceLinks(interfaces model.InterfaceList) string {
    if interfaces.IsEmpty() {
        return "None"
    }
    
    links := make([]string, 0, len(interfaces.Items))
    for _, iface := range interfaces.Items {
        link := fmt.Sprintf("[%s](#%s-interface)", iface, strings.ToLower(iface))
        links = append(links, link)
    }
    return strings.Join(links, ", ")
}

Phase 3.2: Sprig Function Replacements (Week 1-2)

Replace commonly used Sprig functions with native Go implementations:

// internal/converter/markdown_helpers.go
package converter

import (
    "encoding/base64"
    "crypto/sha256"
    "fmt"
    "time"
)

// Common Sprig replacements
func (b *MarkdownBuilder) FormatDate(t time.Time, format string) string {
    if format == "" {
        format = "2006-01-02 15:04:05"
    }
    return t.Format(format)
}

func (b *MarkdownBuilder) Base64Encode(s string) string {
    return base64.StdEncoding.EncodeToString([]byte(s))
}

func (b *MarkdownBuilder) SHA256Sum(s string) string {
    h := sha256.Sum256([]byte(s))
    return fmt.Sprintf("%x", h)
}

func (b *MarkdownBuilder) DefaultValue(value, defaultVal string) string {
    if value == "" {
        return defaultVal
    }
    return value
}

func (b *MarkdownBuilder) TruncateString(s string, length int) string {
    if len(s) <= length {
        return s
    }
    return s[:length] + "..."
}

Phase 3.3: Complex Function Migration (Week 2)

Port complex formatting functions:

// internal/converter/markdown_formatters.go

func (b *MarkdownBuilder) FormatSystemStats(data *model.OpnSenseDocument) map[string]interface{} {
    stats := make(map[string]interface{})
    
    // Calculate various statistics
    stats["totalInterfaces"] = len(data.Interfaces.Items)
    stats["totalFirewallRules"] = len(data.Filter.Rules)
    stats["totalUsers"] = len(data.System.Users)
    stats["totalServices"] = b.countActiveServices(data)
    stats["securityScore"] = b.calculateSecurityScore(data)
    
    return stats
}

func (b *MarkdownBuilder) FormatNetworkInterfaces(interfaces model.Interfaces) []map[string]string {
    formatted := make([]map[string]string, 0)
    
    for name, iface := range interfaces.Items {
        entry := map[string]string{
            "name":        name,
            "device":      iface.If,
            "ip":          iface.Ipaddr,
            "subnet":      fmt.Sprintf("%s/%s", iface.Ipaddr, iface.Subnet),
            "enabled":     b.boolToString(iface.Enable),
            "description": b.escapeTableContent(iface.Descr),
        }
        formatted = append(formatted, entry)
    }
    
    return formatted
}

Phase 3.4: Testing & Benchmarking (Week 2-3)

Create comprehensive tests:

// internal/converter/markdown_functions_test.go

func TestMarkdownBuilder_AssessRiskLevel(t *testing.T) {
    builder := NewMarkdownBuilder()
    
    tests := []struct {
        name     string
        severity string
        expected string
    }{
        {"Critical", "critical", "🔴 Critical Risk"},
        {"High", "HIGH", "🟠 High Risk"},
        {"Medium", "medium", "🟡 Medium Risk"},
        {"Low", "low", "🟢 Low Risk"},
        {"Unknown", "unknown", "⚪ Unknown Risk"},
    }
    
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            result := builder.AssessRiskLevel(tt.severity)
            assert.Equal(t, tt.expected, result)
        })
    }
}

func BenchmarkTemplateVsProgrammatic(b *testing.B) {
    data := createTestData()
    builder := NewMarkdownBuilder()
    
    b.Run("Programmatic", func(b *testing.B) {
        for i := 0; i < b.N; i++ {
            _, _ = builder.BuildStandardReport(data)
        }
    })
    
    b.Run("Template", func(b *testing.B) {
        gen := createTemplateGenerator()
        for i := 0; i < b.N; i++ {
            _, _ = gen.Generate(data)
        }
    })
}

Phase 3.5: Migration & Documentation (Week 3)

1. Update CLI to use programmatic mode by default:

// cmd/convert.go
func getReportGenerator(opts markdown.Options) (markdown.Generator, error) {
    builder := converter.NewMarkdownBuilder()
    
    // Use programmatic by default, template only if specified
    if opts.TemplateDir != "" {
        return markdown.NewHybridGeneratorWithTemplate(builder, opts.TemplateDir)
    }
    
    return markdown.NewHybridGenerator(builder, nil)
}

2. Update documentation with migration guide
3. Add deprecation notices for template-only features

Success Metrics

• ✅ All 20+ template functions ported to builder methods
• ✅ Zero runtime template parsing for default operation
• ✅ Performance improvement of 30-50% (no template overhead)
• ✅ 100% test coverage for ported methods
• ✅ Full backward compatibility via --custom-template flag

🏷️ Recommended Labels

Add these labels to issue #86:

• refactoring - Major code restructuring
• enhancement - Improving existing functionality
• breaking-change - Default behavior changes (v2.0)
• priority:high - Critical for v2.0 milestone
• size:large - 2-3 weeks of work
• type:technical-debt - Addressing architectural improvements
• needs:testing - Requires comprehensive test coverage

📝 Enhanced Metadata

Milestone: v2.0
Assignee: unclesp1d3r
Priority: High
Estimated Effort: 2-3 weeks
Dependencies: 
  - Phase 1 completion (issue tracking needed)
  - Phase 2 completion (issue tracking needed)
  
Subtasks:
  - [ ] Create function mapping document (all template functions → Go methods)
  - [ ] Port utility functions (formatting, escaping, defaults)
  - [ ] Port data transformation functions (filtering, aggregation)  
  - [ ] Port security assessment functions (risk levels, scoring)
  - [ ] Create comprehensive test suite
  - [ ] Run performance benchmarks
  - [ ] Update CLI to default to programmatic mode
  - [ ] Update documentation
  - [ ] Add migration guide for custom templates
  - [ ] Deprecation notices for template-only features

🚀 Next Steps

1. Immediate: Create subtask issues for each function category
2. Week 1: Start with core functions that are most commonly used
3. Week 2: Complex formatters and data transformations
4. Week 3: Testing, benchmarking, and documentation
5. Release: Version 2.0 with breaking change notice

This refactor will significantly improve maintainability, performance, and developer experience while maintaining full backward compatibility for users who need custom templates.

[unclesp1d3r]

@coderabbitai create the proposed subtasks with all necessary implementation details