LZ4

Relevant source files

• src/DeflateZLib/DeflateBlock.cs
• src/Fast-Lzma2/FastLzma2Block.cs
• src/Interop/Interop.Lz4.cs
• src/Lz4/Lz4Block.cs
• src/Lz4/Lz4Decoder.cs
• src/Lz4/Lz4Encoder.cs

The LZ4 implementation in GrindCore.net provides high-performance compression and decompression using the LZ4 frame format. It supports both streaming via Lz4Stream and one-shot block operations via Lz4Block. The implementation leverages native bindings to LZ4 v1.10.0 and is optimized for 64-bit environments.

Implementation Overview

The LZ4 subsystem is divided into three primary layers:

1. Managed API: Lz4Stream and Lz4Block provide the public interface for consumers.
2. Internal Engine: Lz4Encoder and Lz4Decoder manage the native lifecycle, including context allocation and buffer pinning.
3. Interop Layer: Interop.Lz4 contains the P/Invoke signatures and structure definitions for the GrindCore native library.

Key Classes and Roles

Class	Role
Lz4Stream	High-level Stream implementation for LZ4 frame compression/decompression.
Lz4Block	One-shot compression/decompression for discrete data blocks src/Lz4/Lz4Block.cs16-17
Lz4Encoder	Manages SZ_Lz4F_v1_10_0_CompressionContext and frame header writing src/Lz4/Lz4Encoder.cs11-13
Lz4Decoder	Manages SZ_Lz4F_v1_10_0_DecompressionContext and frame parsing src/Lz4/Lz4Decoder.cs12-14
Interop.Lz4	Defines native structures like LZ4F_preferences_t and LZ4F_frameInfo_t src/Interop/Interop.Lz4.cs69-89

Sources: src/Lz4/Lz4Block.cs16-17 src/Lz4/Lz4Encoder.cs11-18 src/Lz4/Lz4Decoder.cs12-18 src/Interop/Interop.Lz4.cs1-91

Data Flow and Code Entity Mapping

The following diagram illustrates the relationship between the managed stream API and the underlying native interop calls during a compression operation.

Compression Data Flow

Sources: src/Lz4/Lz4Encoder.cs79-115 src/Lz4/Lz4Encoder.cs159-170 src/Interop/Interop.Lz4.cs202-225

LZ4 Stream (Lz4Stream)

Lz4Stream inherits from CompressionStream. It manages internal CompressionBuffer instances for staging data between the managed caller and the native LZ4 frame engine.

Configuration and Block Sizing

LZ4 supports specific block sizes (64KB, 256KB, 1MB, 4MB).

• Compression Levels:
  • Level 1: Linked blocks (LZ4F_blockLinked), optimized for decompression speed src/Lz4/Lz4Encoder.cs52-55
  • Level 2: Linked blocks (LZ4F_blockLinked) src/Lz4/Lz4Encoder.cs44-52
  • Level 3-12: Independent blocks (LZ4F_blockIndependent) (High Compression/HC mode) src/Lz4/Lz4Encoder.cs45-52

Sources: src/Lz4/Lz4Encoder.cs43-57

LZ4 Block (Lz4Block)

Lz4Block provides a non-streaming, one-shot compression interface src/Lz4/Lz4Block.cs16 It is used when the entire payload is available in memory.

Block Operations

• Compression: If the compression level is $\ge 3$, it uses SZ_Lz4_v1_10_0_CompressHC src/Lz4/Lz4Block.cs72-78 Otherwise, it initializes a SZ_Lz4_v1_10_0_Stream and uses SZ_Lz4_v1_10_0_CompressFastContinue with an acceleration factor src/Lz4/Lz4Block.cs81-89
• Decompression: Uses SZ_Lz4_v1_10_0_DecompressSafeContinue src/Lz4/Lz4Block.cs121-122
• Buffer Requirements: The required output size is calculated as blockSize + (blockSize / 255) + 16 src/Lz4/Lz4Block.cs52 matching the SZ_Lz4_v1_10_0_CompressBound logic src/Interop/Interop.Lz4.cs93-96

Result Mapping

Native LZ4 error codes are mapped to CompressionResultCode src/Lz4/Lz4Block.cs144-155:

LZ4 Code	Result Code	Description
0	Success	Operation completed successfully.
-1	Error	General LZ4 error.
-2	InsufficientBuffer	Output buffer too small.
-3	InvalidParameter	Parameter mismatch.
-4	InvalidData	Corrupt input data.

Sources: src/Lz4/Lz4Block.cs16-156 src/Interop/Interop.Lz4.cs10-14 src/Interop/Interop.Lz4.cs93-96

Native Interop and Contexts

The implementation uses GCHandle.Alloc with GCHandleType.Pinned to ensure that buffers passed to native code do not move during execution src/Lz4/Lz4Encoder.cs66-67 src/Lz4/Lz4Decoder.cs39-40

Context Lifecycle

The native contexts are represented by sequential structs containing an IntPtr to the internal state:

• SZ_Lz4F_v1_10_0_CompressionContext src/Interop/Interop.Lz4.cs24-27
• SZ_Lz4F_v1_10_0_DecompressionContext src/Interop/Interop.Lz4.cs29-33

Entity Relationship Diagram

Sources: src/Lz4/Lz4Encoder.cs11-18 src/Lz4/Lz4Decoder.cs12-18 src/Interop/Interop.Lz4.cs91-225

Constraints and Versioning

• Version: The implementation targets LZ4 v1.10.0 src/Interop/Interop.Lz4.cs10
• Architecture: While the managed code is platform-agnostic, the native GrindCore library handles the 64-bit logic. The Interop layer uses UIntPtr and IntPtr to maintain compatibility with native size types (size_t) src/Interop/Interop.Lz4.cs202-215
• Framing: Lz4Encoder and Lz4Decoder use the LZ4 Frame format (headers and footers) via SZ_Lz4F_v1_10_0_CompressBegin and SZ_Lz4F_v1_10_0_GetFrameInfo src/Lz4/Lz4Encoder.cs87-89 src/Lz4/Lz4Decoder.cs73-74 Lz4Block operates on raw blocks using the SZ_Lz4_v1_10_0_Stream interop src/Lz4/Lz4Block.cs81-82

Sources: src/Interop/Interop.Lz4.cs1-225 src/Lz4/Lz4Encoder.cs11-18 src/Lz4/Lz4Block.cs16-17