mop3/lz4/examples/blockStreaming_doubleBuffer.md

LZ4 Streaming API Example: Double Buffer

by Takayuki Matsuoka

blockStreaming_doubleBuffer.c is an example of the LZ4 Streaming API where we implement double buffer (de)compression.

Please note:

• Firstly, read "LZ4 Streaming API Basics".
• This is a relatively advanced application example.
• The output file is not compatible with lz4frame and is platform dependent.

What's the point of this example?

The LZ4 Streaming API can be used to handle compressing a huge file in a small amount of memory. This example shows how to use a "Double Buffer" to compress blocks (i.e. chunks) of a uniform size in a stream. The Streaming API used in this way always yields a better compression ratio than the regular Block API.

For an example with non-uniform blocks, see "LZ4 Streaming API Example: Line by Line Text Compression".

How compression works

Firstly, allocate "Double Buffer" for input and "compressed data buffer" for output. Double buffer has two pages, the "first" page (Page#1) and the "second" page (Page#2).

        Double Buffer

      Page#1    Page#2
    +---------+---------+
    | Block#1 |         |
    +----+----+---------+
         |
         v
      {Out#1}

Next, read the first block to the double buffer's first page. Compress it with LZ4_compress_continue(). On the first compression, LZ4 doesn't have any previous dependencies, so it just compresses the block without dependencies and writes the compressed block {Out#1} to the compressed data buffer. After that, write {Out#1} to the file.

      Prefix Dependency

         |         |
         v         |
    +---------+----+----+
    | Block#1 | Block#2 |
    +---------+----+----+
                   |
                   v
                {Out#2}

Next, read the second block to the double buffer's second page and compress it. This time, LZ4 can use the dependency on Block#1 to improve the compression ratio. This dependency is called "Prefix mode".

   External Dictionary Mode
         +---------+
         |         |
         |         v
    +----+----+---------+
    | Block#3 | Block#2 |
    +----+----+---------+
         |
         v
      {Out#3}

Next, read the third block to the double buffer's first page and compress it. This time LZ4 can use dependency on Block#2. This dependency is called "External Dictionary mode".

      Prefix Dependency
         +---------+
         |         |
         v         |
    +---------+----+----+
    | Block#3 | Block#4 |
    +---------+----+----+
                   |
                   v
                {Out#4}

Continue this procedure till the end of the file.

How decompression works

Decompression follows the reverse order:

• Read the first compressed block.
• Decompress it to the first page and write that page to the file.
• Read the second compressed block.
• Decompress it to the second page and write that page to the file.
• Read the third compressed block.
• Decompress it to the first page and write that page to the file.

Continue this procedure till the end of the compressed file.