kamkow1
b9e8a8bf1d
All checks were successful
Build documentation / build-and-deploy (push) Successful in 1m53s
110 lines
3.1 KiB
Markdown
110 lines
3.1 KiB
Markdown
# LZ4 Streaming API Example: Double Buffer
|
|
by *Takayuki Matsuoka*
|
|
|
|
[`blockStreaming_doubleBuffer.c`](blockStreaming_doubleBuffer.c) is an example
|
|
of the LZ4 Streaming API where we implement double buffer (de)compression.
|
|
|
|
Please note:
|
|
|
|
- Firstly, read ["LZ4 Streaming API Basics"](streaming_api_basics.md).
|
|
- This is a relatively advanced application example.
|
|
- The output file is not compatible with lz4frame and is platform dependent.
|
|
|
|
|
|
## What's the point of this example?
|
|
|
|
The LZ4 Streaming API can be used to handle compressing a huge file in a small
|
|
amount of memory. This example shows how to use a "Double Buffer" to compress
|
|
blocks (i.e. chunks) of a uniform size in a stream. The Streaming API used in
|
|
this way *always* yields a better compression ratio than the regular Block API.
|
|
|
|
For an example with non-uniform blocks, see ["LZ4 Streaming API Example: Line by
|
|
Line Text Compression"](blockStreaming_lineByLine.md).
|
|
|
|
## How compression works
|
|
|
|
Firstly, allocate "Double Buffer" for input and "compressed data buffer" for
|
|
output. Double buffer has two pages, the "first" page (`Page#1`) and the "second"
|
|
page (`Page#2`).
|
|
|
|
```
|
|
Double Buffer
|
|
|
|
Page#1 Page#2
|
|
+---------+---------+
|
|
| Block#1 | |
|
|
+----+----+---------+
|
|
|
|
|
v
|
|
{Out#1}
|
|
```
|
|
|
|
Next, read the first block to the double buffer's first page. Compress it with
|
|
`LZ4_compress_continue()`. On the first compression, LZ4 doesn't have any
|
|
previous dependencies, so it just compresses the block without dependencies and
|
|
writes the compressed block `{Out#1}` to the compressed data buffer. After that,
|
|
write `{Out#1}` to the file.
|
|
|
|
```
|
|
Prefix Dependency
|
|
|
|
| |
|
|
v |
|
|
+---------+----+----+
|
|
| Block#1 | Block#2 |
|
|
+---------+----+----+
|
|
|
|
|
v
|
|
{Out#2}
|
|
```
|
|
|
|
Next, read the second block to the double buffer's second page and compress it.
|
|
This time, LZ4 can use the dependency on `Block#1` to improve the compression
|
|
ratio. This dependency is called "Prefix mode".
|
|
|
|
```
|
|
External Dictionary Mode
|
|
+---------+
|
|
| |
|
|
| v
|
|
+----+----+---------+
|
|
| Block#3 | Block#2 |
|
|
+----+----+---------+
|
|
|
|
|
v
|
|
{Out#3}
|
|
```
|
|
|
|
Next, read the third block to the double buffer's *first* page and compress it.
|
|
This time LZ4 can use dependency on Block#2. This dependency is called "External
|
|
Dictionary mode".
|
|
|
|
```
|
|
Prefix Dependency
|
|
+---------+
|
|
| |
|
|
v |
|
|
+---------+----+----+
|
|
| Block#3 | Block#4 |
|
|
+---------+----+----+
|
|
|
|
|
v
|
|
{Out#4}
|
|
```
|
|
|
|
Continue this procedure till the end of the file.
|
|
|
|
|
|
## How decompression works
|
|
|
|
Decompression follows the reverse order:
|
|
|
|
- Read the first compressed block.
|
|
- Decompress it to the first page and write that page to the file.
|
|
- Read the second compressed block.
|
|
- Decompress it to the second page and write that page to the file.
|
|
- Read the third compressed block.
|
|
- Decompress it to the *first* page and write that page to the file.
|
|
|
|
Continue this procedure till the end of the compressed file.
|