Native Interop Layer

Relevant source files

• src/Interop/Interop.Blake.cs
• src/Interop/Interop.Brotli.cs
• src/Interop/Interop.FastLzma2.cs
• src/Interop/Interop.MD.cs
• src/Interop/Interop.SHA.cs
• src/Interop/Interop.Stat.cs
• src/Interop/Interop.XXHash.cs
• src/Interop/Interop.ZStd.cs
• src/Interop/Interop.Zlib.cs

The Native Interop Layer provides the bridge between the managed .NET environment and the high-performance C implementation of the GrindCore library. This layer is responsible for P/Invoke declarations, marshaling complex structures, managing native resource lifecycles via SafeHandle, and implementing a version-dispatching mechanism to support multiple iterations of the same algorithm.

Interop Architecture

The interop system is centralized within the Nanook.GrindCore.Interop static class. It uses a consistent naming convention where the native library name is stored in a central constant, and specific algorithms are organized into nested static classes.

Core Components

• Interop.Libraries: Defines the GrindCoreLib constant string used in all DllImport attributes. This allows the library to target GrindCore.dll on Windows and libGrindCore (with appropriate extensions) on Unix-like systems src/Interop/Interop.Libraries.cs10-11
• Per-Algorithm Classes: Nested classes such as Interop.Zlib, Interop.Brotli, Interop.ZStd, Interop.SHA, and Interop.XXHash contain the specific extern method definitions and native struct layouts required for that algorithm src/Interop/Interop.Zlib.cs30-31 src/Interop/Interop.Brotli.cs15-16 src/Interop/Interop.ZStd.cs104-105 src/Interop/Interop.SHA.cs84-85 src/Interop/Interop.XXHash.cs31-32
• Versioned Dispatching: Native function names often include version prefixes (e.g., DN9_BRT_v1_1_0_... for Brotli or SZ_ZStd_v1_5_7_... for ZStd) to allow side-by-side coexistence of different algorithm versions src/Interop/Interop.Brotli.cs18-21 src/Interop/Interop.ZStd.cs107-108

Interop Entity Map

The following diagram maps the managed interop classes to their native counterparts and primary data structures.

Diagram: Interop Entity Mapping

Sources: src/Interop/Interop.Libraries.cs10-11 src/Interop/Interop.Zlib.cs17-31 src/Interop/Interop.Brotli.cs15-18 src/Interop/Interop.ZStd.cs104-108 src/Interop/Interop.SHA.cs84-88 src/Interop/Interop.XXHash.cs9-32

SafeHandle & Resource Management

GrindCore uses SafeHandle (specifically SafeBrotliEncoderHandle, SafeBrotliDecoderHandle, etc.) to ensure that native contexts are reliably freed, even if a managed exception occurs. This prevents memory leaks in the native heap.

• Initialization: Managed wrappers call native "CreateInstance" functions which return a SafeHandle. For example, DN9_BRT_v1_1_0_BrotliDecoderCreateInstance returns a SafeBrotliDecoderHandle src/Interop/Interop.Brotli.cs18-19
• Disposal: The native "DestroyInstance" functions (e.g., DN9_BRT_v1_1_0_BrotliDecoderDestroyInstance) are called when the SafeHandle is disposed or finalized src/Interop/Interop.Brotli.cs29-30
• ZLib Exception: ZLib uses a pattern where ZStream is wrapped into a SafeHandle by initialization methods like DN8_ZLib_v1_3_1_DeflateInit2_ to ensure correct disposal src/Interop/Interop.Zlib.cs11-14

Sources: src/Interop/Interop.Brotli.cs18-50 src/Interop/Interop.Zlib.cs11-14

Unsafe Pointer Patterns

To achieve high performance and avoid unnecessary data copying, GrindCore heavily utilizes unsafe blocks and fixed pointers when interacting with native libraries.

Data Flow Pattern

1. Pinning: Managed code pins buffers using the fixed keyword or GCHandle for longer-lived multi-threaded operations.
2. Pointer Pass-through: Raw pointers (byte*, byte**) are passed to native functions. For instance, BrotliDecoderDecompressStream accepts byte** for input and output tracking src/Interop/Interop.Brotli.cs21-23
3. Context Structs: Many algorithms pass a pointer to a context struct (e.g., SZ_ZStd_v1_5_7_CompressionContext* or XXH32_CTX*) which the native side uses to maintain state src/Interop/Interop.ZStd.cs108-111 src/Interop/Interop.XXHash.cs34-37

Diagram: Unsafe Buffer Interop Flow

Sources: src/Interop/Interop.Brotli.cs21-23 src/Interop/Interop.ZStd.cs141-146 src/Interop/Interop.XXHash.cs37-40

Algorithm Interop Details

ZLib & ZLib-NG

The Interop.Zlib class defines the ZStream structure, which mirrors the standard C z_stream struct src/Interop/Interop.Zlib.cs17-28 It includes enums for FlushCode, ErrorCode, CompressionLevel, and CompressionStrategy to ensure type-safe interaction with the native API src/Interop/Interop.Zlib.cs36-125

ZStandard (ZStd)

ZStd interop supports both block-based and stream-based operations. It defines specific contexts (SZ_ZStd_v1_5_7_CompressionContext) and buffer structures (SZ_ZStd_v1_5_7_InBuffer, SZ_ZStd_v1_5_7_OutBuffer) that are passed by pointer to native calls like SZ_ZStd_v1_5_7_CompressStream src/Interop/Interop.ZStd.cs63-101 src/Interop/Interop.ZStd.cs141-146

Hashing (SHA, MD, XXHash, Blake)

Hashing interop typically involves a "Context" struct that maintains the hash state between updates.

• SHA/MD: Uses structs like SHA3_CTX, SHA512_CTX, or MD5_CTX src/Interop/Interop.SHA.cs16-51 src/Interop/Interop.MD.cs28-33
• XXHash: Uses XXH32_CTX and XXH64_CTX with fixed arrays for accumulator lanes (v[4]) src/Interop/Interop.XXHash.cs9-29
• Blake: Blake2sp uses a complex CBlake2sp struct with a union for header alignment and fixed state arrays src/Interop/Interop.Blake.cs33-61 Blake3 uses a Blake3Hasher struct with a stack for chaining values (cv_stack) src/Interop/Interop.Blake.cs14-20

Fast-LZMA2

Fast-LZMA2 interop includes complex configuration structures like FL2CompressionParameters to control dictionary size, overlap fractions, and multi-threading parameters src/Interop/Interop.FastLzma2.cs12-53 It supports multi-threaded compression via FL2_compressMt src/Interop/Interop.FastLzma2.cs85-87

Sources: src/Interop/Interop.Zlib.cs1-125 src/Interop/Interop.ZStd.cs1-160 src/Interop/Interop.SHA.cs1-160 src/Interop/Interop.XXHash.cs1-50 src/Interop/Interop.Blake.cs1-107 src/Interop/Interop.FastLzma2.cs1-125