Wild Mind - Compression: Features with heuristics, Streaming decompression, Music compression (part 2).

Compression - continued
Alone Coder

           8. Features with heuristics
   The PAQ archiver uses Neuros─
ty, and it packs not bytes, but bits. But that’s all
it is very slow and requires memory2
GB. As far as I understand, the neural network is being built
and during unpacking. But you can use
neural networks and evolutionary algorithms and for
compressors with simple unpackers,
having additional features. For example, in
Hrust1 link extension and byte insertion,in 
Laser Compact 4 copying with reversal. 

alone> archiver durilca torn! now 
leader cmix (http://mattmahoney.net/dc/
text.html ). the author refers to the book
http://mattmahoney.net/dc/dce.html
it turns out that now they do it in archivers
special methods for clamping zhpegov.
who wants to write an archiver and get
prize?http://prize.hutter1.net/
the record has been held for 5 years.
cmix uses a total of 1,723 independent
models. There are a variety of different
types of models,some specialized for cert─
ain types of data such as text, execut─
ables, or images. For each bit of input
data,each model outputs a single floating
point number, representing the probability
that the next bit of data will be a 1.cmix
uses a neural network to combine the model
predictions into a single probability.

hrumer> есть тема - при упаковке лаз. ком─ 
пактом можно выявлять непакованные байты, 
и если на них нет опоры, то заменять их 
при случае недавно встретившимися. 
Есть вероятность, что ЛК без перекодировки 
и с olzh может немного лучше жать игры с 
большим количеством графики. 
lvd>из-за каких фич? 
hrumer> Из-за перевернутости последовате─ 
льностей, возможно, более оптимально коди─ 
рованных. Эта тема с 2000 года в голове. 
alone> Не понял насчёт заменять. 
hrumer> Вот встретили АБ. Они не пакуются. 
На Бno one refers later. So we can─ 
but replace AB with AC, which was previously recently 
met, and B is not much different from C. 
alone> That is lossy compression? 
hrumer> Yeah. With losses. But with possible 
adjustment by the artist. 
alone> I remember I manually fixed it in Gunfright─ 
I put the bytes in the picture to make it better─
was my RLE)
hrumer> Well, there is one person’s experience) 
lvd>what do I think is rewriting 
games from scratch, for example, will give much more
savings than changing the pattern )
hrumer> When I made packers - one of the topics 
was that you take it, you load your head, 
what do you need to come up with? And then I have 
40 minutes of simple walking to think and 
reflect. Here I would like something worthless 
I came up with this if I had a car earlier. 

hrumer> why did you refuse one of 
versions, if I'm not mistaken, rip, from repeat 
distance? little benefit, but the depacker is swelling? 
alone> yes, I didn’t notice any benefit at all. 
hrumer> maybe the packer didn’t optimize this matter? 
alone> can only be optimized with add-ons─ 
solid passage.ZXRar 128K only.packed─
new data fits onto the packaged ones. And sliding─
That's why I didn't make a dictionary. and a 64K window.

alone> looked about image approximation 
in circles? :)http://www.youtube.com/
watch?v=dOO5XcXLxGs
hrumer> I came up with a completely new editor 
graphics for Spectrum! The picture is packed, 
Then it shows where it is poorly packed. 
The artist is given a choice where to correct and 
what does it take to pack well :) 

        9. Streaming decompression
   Make a compressor designed for
current reading of packed data. Ta─
which is convenient to use when loading MOD files─
catching in General Sound and when playing by─
current video. Now for these purposes it is used─
One page compression is usually used
or naked Huffman (ZXRar with window0K and without
header), unpacked manually.
   Streaming unpacker interface dol─
wives are allowed to unpack in one call
a piece about a kilobyte in size. Other
option - the unpacker itself will call
user program with each received
byte, it's slower.

lvd>in Penteve fpg firmware (in ngs, by the way─ 
and also) is unpacked into a ring buffer
and then loaded in pieces into the fpg. in
penteve - 100 kilos of unpacked size.
hrumer> In crunch2.1 there is also a CRC in the header) 
lvd>how many bits? 
hrumer> 16. 
lvd>ccitt I guess?http://srecord. 
sourceforge.net/crc16-ccitt.html
hrumer> You have to believe it. Maybe she does too. 
Maybe not. The calculation sources are available. 
lvd>and as it is considered, the plate is 2x256 or 
bitwise? or cleverly byte-by-byte without a table?
hrumer> There were 2 options. First the sign 
512 is considered, and then the sum. Or very 
slowly without. I looked at the crunch and 
Megals. Almost twin brothers. Megals, 
Looks like it presses a little better from behind the window at 256 
more. And at one time I, because of the roundabout 
I didn’t bother with the array. 
lvd>a ring array can be made from any 
make such a packer. in my opinion, in Megalz
links are better encoded.
in mkhmt you can set the window to 4 kilos exactly, and
compare. the crunch will still lose)))
well, yes, you can have a tablecount.
but, in my opinion, it’s easier to ink it right away)
hrumer> She doesn't pack, she's such a bitch) In those─ 
Is it possible to set a window for megaz in the current version? 
lvd>yes, you can set a window for everything. 
not just for megales. For example, for unpacking─
the window in the AVR is 2k, because there is memory
RAM is only 4k. yes, I just recently
I was bothered with the generation of the DRC in
different poses. including generation of that table
from a polynomial. CHSH one of the table elements
is the polynomial itself.
hrumer> Ring array - this is for speed─ 
This packaging was used on the ZX. 8k+512. 
lvd>a, packaging. what's the speed? 
hrumer> Links to previous occurrences of si─ 
mvola. If it's a power of two, then it's beautiful. A 
if a little more, then the stump is clear, right away 
you need 2 times more. True, the last one 
the link may just point beyond 
windows, andthis could be used for 
increasing the window by 256. 

hrumer> it would be interesting to take a look at demegas 
with a ring buffer. I have one idea, I need it 
check. 
lvd>http://svn.zxevo.ru/filedetails.php? 
repname=ngs&path=%2Fz80%2FbootFPGAOO
%2FbootFPGA.asm- times (DECHObuf)
http://svn.zxevo.ru/filedetails.php?
repname=pentevo&path=%2Favr%2Fcurrent%2Fde
packer_dirty.ctwo (also the same .h file)
http://svn.zxevo.ru/filedetails.php?
repname=pentevo&path=%2Favrboot%2Ftrunk%2F
avr%2Fboot_evo.asmthree
(around the PUTB1 label, written by DDp)
hrumer> I thought about the ring buffer, 
what if the packer won't do the coding? 
lines at the edges of the ring buffer, then you can 
without bothering to do LDIR. 
lvd>packing efficiency will decreasesame. 
hrumer> by 1-2 bytes in total (for the length 
buffer), in the worst case. 
lvd>1-2 bytes for each edge of the ring 
buffers.
hrumer> probability ~1%. you can test 
setting price = infinity when moving 
phrases in place n*window. Although, why the hell should 
if the depacker speed is sufficient) 

alone> you need to bleed the unpacker output 
another procedure, and memory to use
Have you only seen 4K cartoons about a cat? For example
https://zxaaa.net/view_demo.php?id=8593
lvd>I officially declare that for the first time 
I invented the demegasus with a buffer-window.
back in 2005 (+-1), having written a depacker
firmware for FPG in NGS.http://
svn.zxevo.ru/filedetails.php?repname=ngs&
path=%2Fz80%2FbootFPGAOO%2FbootFPGA.asm
in my opinion, the principle is already obvious. if not
obvious, see sorets on tse.alone> does it chew through the entire window? 
lvd>yes. unpacks 8 kilos then, these 8 go 
eat a kilo.
alone>, therefore, must be fixed. 
lvd>fix, the principle will not change. 
there are only 2 places where to fix: LDI and LDIR.
alone> thank you. my video is stupid 
Huffman reaped. ZXRar with window 0.
lvd>here it is, the advantage of blunt windows 
packers compared to smart dictionaries
raras and zupami.
alone> it depends on what you pack. 
lvd>if you make the buffer in F000..FFFF, then 
you can fix it easier.OR H,#F0.even set 4,h.
alone> when I was packing the cuts in 2002 
from the film, I had links back to races─
packed familiar places. but when I made a hundred─
standard acquaintances, the winnings were huge.

DDp> 
   Attributed aggression (v2)http://dlcorp.
nedopc.com/viewtopic.php?p=30304#p30304 2970 frames, 25 FPS.
 All frames in the form of a Spectrum dump
attributes(768 bytes) one after another in the form
of one stream are packed with the MegaLZ algorithm.
   Playback: in the main unpacking loop─
are transferred to a cyclic buffer8 KB (from the sky─
with a larger margin, it seems, 4 frames).In interruption─
The next LDIR frame appears on the screen.

   ASCII cartoon "Star Wars" for
"Apogee"http://zx-pk.ru/showpost.php? 
p=714968&postcount=3
Frame size67x13=871 characters (screen
Apogee buffer 78x30).Floating framerate,
individual duration for each
a long frame. Replaced in the original cartoon
characters missing from Apogee. Total
64 characters are used (0x20...Ox5f).
 Level 1:
00...3f - symbol (+0x20)
40...7b - repeat the previous one (1...60 times)
7c - scroll up (move the entire frame)
7d - scroll left
7e - scroll right
7f - clear the entire frame
80...ff - skip (1...128 acquaintances)
 Numbers of frames with scrolls are included in
packer as constants (no qualifier
movements).
   Level 2: the received stream is packed
MegaLZ algorithm with window size4 KB.
 Playback: MegaLZ unpacker used─
Uses a circular buffer4 KB. Unpacked─
these bytes are transferred to level 1, which
refreshes the screen buffer.

   Cartoon series "Simon's Cat"
 256x192 pixels, 12.5 FPS (in some
reduced to10 or lower)
 Level 1: the screen is divided into familiar areas
8x8 dots. Only changed ones are saved
familiarities.
00 - clear the entire frame
01...1f - next comes data for familiarity
 (1...31 pcs.)
20 - frame without changes
21...ff - pass acquaintances (1...223 pcs.)
 Level 2: the received stream is packed
MegaLZ algorithm with window size4 KB.
 Playback: MegaLZ unpacker used─
Uses a circular buffer4 KB. Unpacked─
these bytes are transferred to level 1, which
refreshes the screen buffer.

      10. Acceleration of the native packer
   LZ with normal hashing gives sko─
Packaging growth is at most1 KB/s at 3.5 MHz.
You can speed it up by improving the hash so that it
distributed more evenly (and heh─
shir only3 characters, 2 specially
cannot be improved - for such links you need a friend─
goy hash). Or you can speed up the main loop.

alone> How not to go through all matches by 
2, when looking for big matches? In ZXRar
triples are hashed, and twos are searched with a simple
searching no further than the 256 byte offset. But not
saves. However, I did not collect statistics,
how many addresses are moved on average?
one search. It would be possible to do separate
hash tables for other link lengths, but not
the fact is that this will speed up the search in total. Rather
on the contrary, and the more hash tables, the worse.
And hash characters 1,2,3,then 3,4,5
toowill lead to nothing. Walk two by two
chains with distance comparison
no more profitable than one long chain.
Maybe build an LZ77 on a bunch of LZ78?
Most popular algorithms are based
on LZ77, because it is derived from LZ78
The LZW algorithm was patented by the company
Unisys in 1984, after which they started
troll everyone, including even the servants
Teas for using images in the format
GIF.

alone> Interesting idea of double hashing─ 
nia:An improvement on hash-based
algorithms for searching the longest-match
string used in LZ77-type data compression
(1997) by Kunihiko Sadakane. It's interesting that
according to statistics in link texts of length 4
are more common than links of length 3.
lvd>in general, try on 3-byte hashes for 
the end of the match, and not to the beginning. if the match─
lo - go compare byte by byte. And also in mnu in
mhmt all 2-byte matches in a separate
the table contains 65536 inputs.and each input
list where this happened before.
alone> What difference does it make, towards the end or towards 
at the beginning?
lvd>such that if 2 bytes coincide at the beginning─ 
they also give 3 at the end, that is, it makes sense to compare
the entire line. I'm looking for matches on each
byte, in contrast to.
there was a thought of matching the length of en from the past
transfer bytes to the current one with length n-1.
but haven't done anything yet.
alone> Actually compare the line 
much faster than going through the list.
lvd>oh well? I have a list of all the places where 
such 2 bytes already existed.alone> And look at the inner loop. 
lvd>you take the nearest one and work. 
take the first element of the list - O(1).
alone> 2 bytes could be in 1000 places. 
lvd>find where these two bytes come before 
was - O(n). where n is the file size.
alone> Of these, the nearest 888 are incorrect. 
lvd>and so on. took it from the list, tried on the hash 
the end is a bummer. took the next one from the list
I tried on the hash - and so on to hell.
alone> And fail. Read the article. 
There is no shit even in zlib and lha.
lvd>and what do they say there, in short? 
alone> What you need to do not go by hash 
directly to the list, and to the hash of the least significant bits
following characters in this context.
And they did it, like, 40% faster.
lvd>I have a list and a hash separately. 
alone> What is your hash? 
lvd>3 bytes. 
alone> 3 bytes at your input, but what about 
coming out?
lvd>1 byte hash. 
alone> In the crunch there is the address of the chain. 
And then we go along the chain. In ZXRar too.
lvd>what chain? 
alone> Chains of all addresses where there are 3 such 
byte or similar.
lvd>I have such a chain for 2 bytes. 
alone> The trick is that you need to exclude 
such omega-long chains. For I repeat─
hrumer came up with a way to share bytes (see Info
Guide #7, not implemented).
lvd>they are excluded by max. link length 
back great.
alone> Shit are excluded if all sorts of 
The Japanese are 40% faster on texts.
lvd>faster than what? 
alone> Than zlib. 
lvd>maybe mhmt is 20% faster than the Japanese? 
alone> Hotsa on the Spectrum. 

hrumer> 
aaaaakedfhvbsdghaaaaaaadeferaaaaaaaaaaa - 
quick solution to sequence issue 
ahhhh. 
here's a piece: "I was thinking about─ for Laser Compact 
solve the problem in the following way: in a special way─ 
refer to byte sequences 
AAAAAA... and ABABABAB..., length L, longer 
certain, selected on tests (I think 
from 3..5 bytes). They very oftenmeet 
in pictures. When filling out the table - the one 
where are the addresses of previous occurrences of sim─ 
ox (or symbols, if hashing is carried out 
not one byte at a time):" 
there is actually a search from 2 bytes. 
alone> it’s a pity, that article didn’t help anyone. 
I wish I could go back to those years and write right under
256K. we didn't have a single 128K car
then! and now I’m watching the press and remembering─
niya people - it seems they were rare at all─
Stu. characteristically, the oldest archiver
on ZX - ZXZIP - supports 256K...

alone> I read an article about double 
hashing?
hrumer> eats 2 times more memory, or 
jumps on the same table, and only needs 
table of "heads" of hashes? when did I start 
read, I thought that this was exactly the alg., 
which I told you about. But it turned out that everything 
another alg. 
alone> I can’t imagine how much he eats) 
hrumer> as far as I understand, memory is 2 times 
eats more. and hashes are calculated through 0, 
+1, +2, +5, +7 +9 type bytes. 
By the way, hand compression for 95..98 years is great 
was read, I, however, at a later date 
I've been reading for years...http://compression.ru/
but I looked - it’s terribly inconvenient to read. 
95-98 was just the topic of compression. 
a lot of things were invented in those years. 
now there ishttp://encode.ru/.domain in ru, 
but everything is in English. feeling that handcomp 
press outFido moved to encode.ru. 
alone> I had an idea to speed up the main thing 
search cycle using this oak method: instead of
storing the file on one page, and lists
in the other four store in each page
a piece of a list (8K) and a piece of a file (4K).
Need 8 pages (For 32K file). It is possible
do it at 128K if you mess up the screen.
But now even more came to mind
exotic idea - don’t store it on the list
addresses, and addresses + pages, so as not to count─
poke. Then 12K list+4K file. But already
it will not be possible to move from a file to a list by
add hl,hl. But you can a) alternate them
every 4 bytes, b) alternate through the segment
256 bytes, c) page number paired with buy─
file volume, and the address will be at +#2000, for example─
measures d) e) - variations on theme c).

    11. File pre-processing
   When packing6912 screens you can use─
may spoil0, 1 bits of one byte per
7, 6 bits of the following.

hrumer> an idea that needs to be tested 
real data. and then you’ll see, it’s effective─ 
whether she is active or not. he gave me lvd here 
statistics on inserted byte in crunch1. then 
whether there are many occurrences, or not enough... from the file 
jumps to the file. but it seems unlikely to XOR in 
it can be applied. need to check. 

hrumer> Alone, seen by Lethargeek Kompakt 
0.(0)4 -image compressor while it's black─ 
white? from Lethargeek on the forum. 
alone> Not yet, but what is there? 
hrumer> yes there is something complicated in the plan 
namely graphics. Here's what Lethargeek writes: 
xored turns out like this: each cell is a sign─
place can be inverted or xored to
neighboring with a shift of 1-2 pixels (clears
solid fills and typical textures) or
to any previous (MB inverted) glue─
tku. Shifts are not taken into account now, anyway
for background objects there is usually one, so
with a shift there will be many identical cells.
Exact matches are not required, just to
the difference was smaller, in any case the choice
in favor of Xorka, which takes up less space in
prefix code (simple fixed code
left because it fits itand
the difference with the optimal Huffman is insignificant─
flax; in general, it is better if there are more zeros semi─
read). So you will still need to remember
info about the choice (here it is encoded using Hough─
fmanu, because it is unpredictable) plus, wot─
possible, distance for xorka with distant glue─
the same (except for matching the previous link in
in the case of large background objects or luck─
nia). According to the mind, it would be necessary after processing everything
compare the screen size and correction again─
tify, but I have not done this yet. Target
I had to decide on the format first. Mono─
I'll post a new and improved chrome packer on─
that's right, on Monday, then I will attribute─
do it, and then depacker on the z80.

alx_bw> packs better according to graph columns─ 
by the way. 
I always kept all the screensavers in columns. 
alone> You can stop packing graphics─ 
amount of data. And during execution it is necessary
optimize for speed.

alone> There is a picture 48x96. Every pixel 
its color. The color consists of a shade (8 pcs)
and brightness (5 pcs). The brightness probably changes
smooth, no color. But between them for sure
there is a connection. How to pack a picture? Either with─
think about how to process the data so that
crunch. The dumbest way: divide
color and brightness, pack vertically.

alone> hrumer, are there any algorithms─ 
What about fractal video compression? I used to
tried to come up with something for pure b/w
(no halftones). but it is necessary with halftones.
hrumer> There is a book by Vatolina, apparently there 
stated, but how to apply it, I even 
and I don’t know. 
alone> I think Vatolin is only talking about static ones 
writes...

alone> On Sega CD 16c the video is broken 
for 16 chunks, and the chunks are described there somewhere.
But this only compresses the familiar space by 2 times.
It would also be interesting to pack not familiar places, but
the whole screen so that everything moves like hell and
it was possible to change half a liter on the go (on
Sega CD is possible, emnip). Can be fixed
some colors and allow you to change others.
Then you can update in parts.
lvd>maybe, like in 6912, pack - picking up 
chunks, say 4x4.
alone> 4x4 will be visible. 
lvd>if the FPS is ok, it won't. besides, 
they are multi-colored. the same 4x4 can
be of different colors.
alone> IMHO we need something smarter stupid 
chunks. We need to store some kind of model,
which can then be turned into pixels.
A type of wavelet. Moreover, the areas are different
Coloring books should also live separately.
lvd>there are also key frames, and there are 
diffs. in diffs you can use the areas of previous ones
move and zoom frames. and other crap,
which I don't know about.
alone> Moving and zooming is too much 
hellish for ATM. More precisely, you can move along
whole pairs of pixels.
lvd>why not come up with jpeg chunks 
4x4 and unpack it?
alone> There is an option to make a video purely for─ 
cast, without color transitions. Plus 50% stroke─
vka. But completely floodedthe acquaintance is clearly on
Half the screen won't fit. And if they are flooded
drawings, then the packaging must be without loss.
hrumer> Alone: ​​"Multisystem2.0". presses on─ 
sequences of screens. if last buy─ 
tov did not change from picture to picture, then 
instead of these bytes, write to the compressed stream─ 
There is information about its length. With online recoding─ 
order the screen in columns inside the third screen─ 
on. with LZ77. but slow. and with RLE. 
alone> I had a split method for 6912─ 
moving to familiar places, in separate streams
standard and non-standard acquaintances and
controlling Non-standard acquaintances Kodi─
were 4 bytes (from 1 byte 2 are made).
In Time Gal (16ts) the familiar places went entirely, there
compression only due to omission of familiar places.
In color, a point still needs a very fast
unpacking, because the screen is 4 times larger.
In 6912, first instead of standard signs─
places were links back to the pool used─
familiar, but it turned out to be standard
more profitable. Even several filled with different
the level of brightness has already greatly improved the packaging─
vku.Typical grayscale video is compressed into
200-500 bytes per frame. I would like it for color
to a point on the order of kilobytes per frame.
In most cases, 4 colors are enough
on a familiar place. But listing them takes 2 bytes.
We need to play up the fact that we are familiar with
with a common palette, vertically adjacent and th─
horizontally. And so that familiar places are taken into account with─
gray in design, but it’s matan. I think they’ve fallen─
change from frame to framepointless.
hrumer> this is how you will come to jpeg. and to compression 
video mpeg. 

lvd>ptsvideos do not so much pack pictures, 
how much the previous frame is modified into a new one.
alone> this requires many, many cycles. 
lvd>if the frame is in the form of 88 or 44 chunks, 
not really. and in general, chunks of 16c can
be of any vertical size and even
horizontally.
alone> for a seed idea - 4 colors per sign─ 
comesto. this is for real cinema.
and the lines are packed completely differently.

alone> 16c video reaping using the DDp method, probably 
not the most effective way. But for flooded
The Screw method does not take into account adjacent columns.
lvd>can you press with a charset? 
alone> I think it will suck. I did it in 2002 
year packer with reference to old
familiar places, it was ugly. Figures came out
recognizable, but not from those places.
lvd>no - generate a charset for Kadyr and 
print a qadir for them. you can charset for the whole
vidu, print with precision 1pixel, different
sizes. Or dynamically update the charset.
alone> How many characters are in the charset? 
lvd>more than 256 and many. how much memory is not 
sorry.
alone> Then what's wrong? 
lvd>in summary. 
alone> You need less than a kilobyte per frame, otherwise 
there's no point in messing around.
Dimon128 has approximately 200 bytes per frame.
lvd>you want it out of your head without experiments 
can you think of less kilobytes per frame?
alone> Probably, you still need scrolling. You need 
to dance for something. Otherwise favorite digi─
animated tales and supers
people won't work on ATM, or
It will work, but it will be unreasonably fatty.
lvd>what are scrolls? 
alone> This is for the zone, indicating that the graphics 
take from the old screen from such and such a place.
lvd>you can’t stretch out the entire frame by scrolling. 
alone> A third of the frame is missing a sign─ 
seats, a third with standard acquaintances, but
the rest of the part can be scrolled,
some new graphics.12. Music compression
alone> Hrumer: haven’t you thought about 
packaging AY music?
hrumer> reflected in 1997. brought it down to 
that in the end you will get a player, like─ 
good for existing ones, and abandoned this matter :) 
and now how is it going? how many KB are there? 
to music? 
alone> Still a soundtrack player 
let's use it. There is also a WYZ tracker, but nothing there
do not convert. Spanish. It is used by Mojon
Twins. Someone on Fido said it was in 2K
the music may come from Black Raven,
but we saw no results.
  (The shortest PTZ player (less than 1 KB)
see Info Guide #11.)

jtn> can't *LZ be converted to ROM? 
lvd>well, look: the package is RND. and depacked - 
no. if you are looking for depacked in package
you will, x what you find.
jtn> first, separate the bit stream. 
lvd>so? more than 1 byte then you don't 
you will find it. if the bitstream is separated and
info about lengths and byte offsets.
jtn> not a fact, but you can try. 

hrumer> By the way, I haven’t tested this one 
thing: in LZ, do not encode links to ras─ 
forged area, and to the “packed” one. Doba─insert RLE, put unpacked bytes in one 
stream, and send the bit stream separately. 
alone> It will especially benefit if there are physical flows─ 
physically separated in memory (links are shorter).
I made a register packer for AY with LZ in UPA─
forged area (but without recursion). But not
completed (playerTFMCOM12.$H inhttp://
alonecoder.nedopc.com/zx/books/TFM.rar ).
In general, if you separate the streams, then immediately
maybe LZ78. Something has been proven for him there :)

             13. Self-control
alone> What is missing from spec packers: sra─ 
To test the block for unpackability.
lvd>if there are no errors in the code, then the block 
unpacked iron-clad, what are you talking about?
alone> that it unpacks on top of itself. 
lvd>don't unpack on top. use by─ 
current model with a buffer. True, for crunching
the buffer will fill the entire memory, but still.
you can use mhmt to reduce the maximum rear offset.
hrumer> I had a late thought. Via 
CRC check immediately after packaging+ 
on this samestage to inform how long it is necessary 
make a gap for unpacking so that there is no 
covering areas. here's what's missing - 
packer, so as not to worry about the tax─ 
the difference between the unpacked and the unpacked. 
but it can be done. 
lvd>Alone: so take the unpacker here 
and write a wrapper to set the position
depack and source data, and calculate the gap.
hrumer> it will definitely be slow. or 
don't care about speed? 
lvd>there is a whole half-megaba mod in the alonera demo─ 
Ita is unpacked via a buffer.
hrumer> no, just a piece of memory 
unpack, without buffer. if it doesn't press well 
- then at a certain moment we move out 
packages, we throw them with a special completion code 
the remaining piece. 
lvd>so in crunch it is, in fact. 
there are codes - copy a long piece
just like that.
hrumer> yeah, you just had to develop it on 
the piece is not 42 bytes, but 30 KB. Not so 
difficult, only a little more difficult in a packer 
organize. like one and a half passes. 
in a crunchthe packer sometimes fails in vain, and 
instead of 42 such bytes, it encodes unpacked 
sequence, then a couple of bytes 
packs, then encodes the unpacked ones again:( 
lvd>does it make sense to increase it to 30 kilos? 
if there are 30 kilos of random in a block, why bother?
pack?
hrumer> is, especially so that at the end 
use, but you need to understand it in time. 
Although, if you use optimal coding, 
then this is understood more simply. 
lvd>and in mhmt it always sculpts the optimal 
coding! :) however, it is not suitable for crunching
absolutely optimal. because increasing
he doesn't know how to optimize links.
hrumer> if 30 kilos of random - then you need everything 
process equally, because people want 
just squeeze the piece and release it. but if 
pure 30 kilos of random and nothing more - then 
you can't unpack it without extras. buffer :) 

         14. Finite state entropy
   Instead of Huffman and Arithmetic Cody─
roving, you can use a compromise: tse─
Lobit coding with finite state machine.
 FSE is a new kind of Entropy encoder,based  on ANS /asymmetric numeral systems/
theory, from Jarek Duda. It is designed to
compete  with  Huffman  encoder and Arith─
metic  ones. While huffman is fast but can
only represent probabilities in power of 2
(50%,  25%, etc.)  arithmetic  coding  can
represent  probabilities  with much better
accuracy, but requires multiplications and
divisions.
  FSE  solves  this  dilemna by providing
precise   probabilities,  like  arithmetic
does, but  using only additions, masks and
shifts. This makes FSE faster, on par with
Huffman  speed, and suitable for low-power
CPU environment.
https://
github.com/CyanХ973/FiniteStateEntropy
 About the asymmetric binary system is here:
https://groups.google.com/forum/#!topic/
comp.compression/_cYQFMijqXE
 Well written in Russian:
https://m.habrahabr.ru/company/playrix/
blog/324116/ But you need a large table if
can't come up with a clever data structure.
   I also came across the following publications:
http://citeseerx.ist.psu.edu/viewdoc/
summary?doi=10.1.1.168.6702
https://www.vde-verlag.de/proceedings-en/
45369047.html