C:\Compression_showdown>runme.bat C:\Compression_showdown>call Nakamichify_internal-64GB.bat Silesia_compression_corpus C:\Compression_showdown>TIMER64 Nakamichi_DD_LITE_GCC_64bit_plus-16GB-buffer.exe "Silesia_compression_corpus" "Silesia_compression_corpus.Nakamichi" 24 34000 i SMMi :MM2 0MMMMM: rMMMMMa ZMMM. 0Z :MMM7 7B rMM@ MMM 7 MMM MMM. BMMMa XZ :MM: MMM XMMX XMSANMAYCE; rMMMM rMM; MM@ WM0 2MMMMM ZMMMM MMW BMM : MMMB aMMMMW 2MM :MM8 MM7 8MMi MMMZ XMMM MMa .WMMM0 @MM rMMMMi MMMMa MM, aMM XMMB MM: MMM2 @MXMMM7 0MM MMS WMM MM MMM, MM7 7MM MM: MMX ;MMM iMM MMM8 . MM8 MMS MMMZ . iMBi :MMMX @MMa WB: XMMZ2aX ZMM MMM2 :M2 :MM ZMMM BM8 ;MMMMMaaMM ,MM XMM@MM rMMMMM2aMM XMMMM SMMSMM. MMXMMM MMMM0 aMMMa@MMr MMS ;MM;MM8 XMMMM. MMr MMM0 MM 2MMW MMM 0MM ZMS 2MM. aMMB MMM BM aMM 8Ma MM iMM BMM M7 MMZ ;MMX MMi ;MM ZMW MM0 rM 7MM ;MM MMM7 iMM ZMM2 0MMa MMX 8M: MMM ZMMS BMM2 7 MM8 ZMr :MM rMB MM@ iMM MMZ iMM WMW BM2 MMX @MM MMi ;MMM 0MS 2MM7 aMMM ;MM .8Mi MMM aMM7 ZMMM MM ;Mr XMM iM8 MMa 0MM BMM MMM MM: BMX .MM MM7 aMM BMMM MM ;MMX ZMMMZ BM@ MMM . rMM7 ZMMMZ :MM:MX 0MZ M8 XMM, MMX MM. ;W: rMM aMX ZMM 7MM MM MMMi MM MMZ 8M:MM MM; MMM, MMa 0M:MM 8MWM8 MMZM0 MMM rMM 0MM WMa,M2 MMS MM8 @M0 :MMMXMa MMM WM XMB ; :MM;M8MMM MMM @M XM0 ; MMM@ MMMW MMrZi MMS MMi MMZM0 rMM Mi MM : ;MM BMMMM: rMM MM MMSMa ZMMM@ iMMr 7MM MM MMXM2 :MMM ZMMM 8MM@@ MMZ0 7MM , :MMMM MMBMa 0MWBr ,MMMM MMMM MMS MM MMMZ MMMM MMM MMS .MM MMMZ @MM. MMM MMMM WMM@ 0MM 2M ZMMM :MMMa MMM7 iMMMMMM@ XMMM .MM XMM aMMM MMM: MMW .MM XM@ ZMMM MM0 aMMX 8MMM .MMM WMM ZMr MMMX MMM8 0MMS SMMMMi MMM, rMM,MM2 MMM: SMMM ;MM2 ;MM,MMS MMM, MMM MMM MMM. MMM .MM MMZ MMM XMMM rMMB :MMM7 MM@ BMMM aMM ;MMX aMMX 8MMM ZMM MM8 .MMX MMM2 iMMX ,MMMMMMZ .@; MMM WMM . .: : 0MMZ .2Z :7r. ;i 0MMMaSMi aMMMM7 .WW Nakamichi 'Ryuugan-ditto-1TB', written by Kaze, inspired by Haruhiko Okumura sharing, based on Nobuo Ito's LZSS source, muffinesque tips by m^2, Jim Dempsey and Kirill Kryukov enforced. This latest (2020-Nov-20) compile has B-trees only matchfinder - called 'LITE'; Downloadable at: https://twitter.com/Sanmayce Nakamichi 'Kaidanji' SUPRALITE highlights: - Back to supersimplisticism - 'Kaidanji' returns refined. Simply, ripping only orders 4..16 and then quickly trying to expand the match in the found position. Nakamichi 'Ryuugan-ditto-1TB'/'Dragoneye' LITE highlights: - The Zennish LZSS Microdeduplicator, a texttoy (file-to-file [de]compressor) written in plain C, 100% FREE - licenseless it is; - Single-threaded Non-SIMD console tool written entirely by yours truly Sanmayce; - The first Lempel-Ziv-Storer-Szymanski (LZSS) compressor using 1TB sliding window; - It targets superfast decompression of huge/tera corpora of textual data; - LITE means no memmem() (Railgun) invocations are to be made; - Compressing at Linear Rate due to Matchfinding based on B-trees only; - The Leprechaunesque (Internal/External) B-trees order 3 (2 keys MAX) are brutally-optimized; - Ability to deduplicate (as little as) 64 bytes long chunks 1TB backwards; - Ability to deduplicate 256[+] bytes long chunks 1TB backwards - when SHA3 enabled; - CHUNKABLE! Since there are no headers/metadata in .Nakamichi files, arbitrary chunks/blocks can be compressed-n-concatenated; - SCALABLE! Gets faster when more Physical or/and External RAM is available; - Dynamical (as command line parameters) selection of HASH & B-trees memory blocks; - AUTO-RESETTING PASSES (when TargetBuffer is filled then automaticaly restart building by doubling the passes); - DEFRAGMENTED B-trees are in use during compression stage, their leaves contain only BBs appearing 2[+] times; - Compileable under Windows and Linux, most suitable to run on 3TB machines; - Many thanks go to: Haruhiko Okumura, Nobuo Ito, Hamid Buzidi, Landon Noll, Aleksey Vaneev (https://github.com/avaneev/prvhash). Sanmayce, Nakamichi's author, here, some more notes: - The B-trees form the second layer, the first being HASH table handled by FNV1A-Pippip - the fastest lookup-hash; - Hashpot a.k.a. hashpools (residing in Physical RAM) could be tuned via command line parameter, thus lessening the B-trees heights; - The building of B-trees is done dynamically-n-automatically in PASSES, thus LOCALITY/LOCALIZATION leads to cache-friendliness, for example, instead of confusing/blinding the SSD controller with building 2^27 ~= 128M B-trees at a time, 'PASSES' revision lowers the "noise/mayhem" 128 times by processing e.g. (20bit) 1M B-trees at a time; - For so long, the main goal was to throw it in battle, thus to forge one battle-ready *nix/windows tool with superb hash/search/decompress capabilities. - Straight up, a wish of mine is to have the latest English Wikipedia XML dump decompressed by Nakamichi into physical RAM and then followed by Kazahana's superfast exact/wildcard/fuzzy etudes. - Through the years, such a transparent decompressor (load booster) has been losing momentum and kinda became unnecessary, but not for me, when HDDs had 40MB/s sequential read speeds it was ... boosty, now with SSDs reading at 1700+MB/s, not so smacky, yet uploading (on 128GB machines, 20GB+72GB needed) the whole Wikipedia ~20GB (instead of 72GB) is better, sizewise. - Currently, having 16 (4,6,8,10,12,14,16,18,36,64,24,28,48,56,128,256) MatchLens being hashed (plus 1 being phantomic/temporary), 29bit hashtable results in: (16+1)*8*(2^29) = (16+1)*8*(1<<29) = 69632MB hashpot (the cluster of all hashpools), each slot houses a QWORD (being an offset/address of a B-tree root). - In near future, wanna see a machine (256GB RAM) crunching the 77,265,813,300 bytes file with this command line (it requires 2x72GB + 68GB + 2930GB = 212GB physical RAM + 2.9TB SSD RAM): D:\>timer64 Nakamichi.exe enwiki-20200420-pages-articles.xml enwiki-20200420-pages-articles.xml.Nakamichi 29 3000000 e - In not so far future, wanna see a machine (4096GB RAM) crunching the 72GB file with this command line (it requires 2x72GB + 272GB + 2930GB = 3.3TB physical RAM): D:\>timer64 Nakamichi.exe enwiki-20200420-pages-articles.xml enwiki-20200420-pages-articles.xml.Nakamichi 31 3000000 i Q: - HOW TO BOOST [BUILDING/COMPRESSION] SPEED? A: - Now, the name of the game is "RAM in spades", if we have a lot then speed is going to jump a lot. How? - First, by increasing the hash-pool - 20 (number of bits) was used in 'Taishukan' resulting in 2^20 =~ 1M hash-slots a.k.a. B-trees' roots per keysize, naturally, the bigger the faster since B-trees will have less height and will be more - aiming to reach the ideal case - having only hash-slots with no actual B-trees! Back to 'Taishukan', the testfile is ~77MB, it suggests using 128M slots to lower height/branching of B-trees - 2^27 leads to using 27. Feartime, going from 20 to 27: 8*(15+1)*2^(20) = 134,217,793 bytes; with the speed-up hash needs 8*(15+1)*2^(27) = 17,179,869,184 bytes, yes 16GB. - Second, by lowering passes in building the B-trees - the trick is to increase the number in compile-time parameter -DSpeedUpBuilding=32 - (didn't make it as an execute-time parameter, lazy). If we look at the 'Taishukan' console log it tells us that (Target-Buffer 106 MB) x 64 passes = 6784MB are needed for building the B-trees in ONE PASS ONLY, so compile with -DSpeedUpBuilding=6784 instead. Enfun! Current priority class is HIGH_PRIORITY_CLASS. This compile uses B-trees only, no memmem() invocations - it compresses worse but much faster. Allocating Source-Buffer 202 MB ... Allocating Target-Buffer 16,586 MB ... Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x604d,d915 Leprechaun: Memory pool for B-trees is 34,000 MB. Leprechaun: In this revision 128MB 16-way hash is used which results in 16 x 16,777,216 internal B-Trees of order 3. Leprechaun: Allocating HASH memory 2,281,701,441 bytes = 8*(16+1)*2^(24) ... OK Leprechaun: Allocating memory for B-trees 34001 MB ... OK Leprechaun: Size of input file: 211,938,580 Leprechaun: Using (first 12 bytes of) DoubleDeuce (a.k.a. CRC64+CRC32C) for Matches 12+ long, in order to reduce memory footprint. Leprechaun: Inserting keys/BBs of order 004 into B-trees, free RAM in B-tree pool is 00,034,000 MB; Pass #000,001 of 000,001 ... DONE; 00,003,938,043 DEFRAGMENTED B-trees have been rooted so far. 4,486,929 TotalNonUnique (of length 004) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 006 into B-trees, free RAM in B-tree pool is 00,033,808 MB; Pass #000,001 of 000,001 ... DONE; 00,010,173,492 DEFRAGMENTED B-trees have been rooted so far. 7,794,309 TotalNonUnique (of length 006) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 008 into B-trees, free RAM in B-tree pool is 00,033,466 MB; Pass #000,001 of 000,001 ... DONE; 00,017,828,240 DEFRAGMENTED B-trees have been rooted so far. 10,247,335 TotalNonUnique (of length 008) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 010 into B-trees, free RAM in B-tree pool is 00,032,997 MB; Pass #000,001 of 000,001 ... DONE; 00,026,453,643 DEFRAGMENTED B-trees have been rooted so far. 12,114,174 TotalNonUnique (of length 010) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 012 into B-trees, free RAM in B-tree pool is 00,032,413 MB; Pass #000,001 of 000,001 ... DONE; 00,035,065,425 DEFRAGMENTED B-trees have been rooted so far. 12,081,422 TotalNonUnique (of length 012) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 014 into B-trees, free RAM in B-tree pool is 00,031,792 MB; Pass #000,001 of 000,001 ... DONE; 00,043,194,279 DEFRAGMENTED B-trees have been rooted so far. 11,112,573 TotalNonUnique (of length 014) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 016 into B-trees, free RAM in B-tree pool is 00,031,217 MB; Pass #000,001 of 000,001 ... DONE; 00,050,785,797 DEFRAGMENTED B-trees have been rooted so far. 10,103,874 TotalNonUnique (of length 016) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 018 into B-trees, free RAM in B-tree pool is 00,030,690 MB; Pass #000,001 of 000,001 ... DONE; 00,057,985,789 DEFRAGMENTED B-trees have been rooted so far. 9,404,275 TotalNonUnique (of length 018) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 036 into B-trees, free RAM in B-tree pool is 00,030,196 MB; Pass #000,001 of 000,001 ... DONE; 00,063,553,678 DEFRAGMENTED B-trees have been rooted so far. 6,766,031 TotalNonUnique (of length 036) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 064 into B-trees, free RAM in B-tree pool is 00,029,828 MB; Pass #000,001 of 000,001 ... DONE; 00,068,791,165 DEFRAGMENTED B-trees have been rooted so far. 6,277,787 TotalNonUnique (of length 064) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 024 into B-trees, free RAM in B-tree pool is 00,029,485 MB; Pass #000,001 of 000,001 ... DONE; 00,075,311,364 DEFRAGMENTED B-trees have been rooted so far. 8,256,241 TotalNonUnique (of length 024) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 028 into B-trees, free RAM in B-tree pool is 00,029,045 MB; Pass #000,001 of 000,001 ... DONE; 00,081,445,227 DEFRAGMENTED B-trees have been rooted so far. 7,633,940 TotalNonUnique (of length 028) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 048 into B-trees, free RAM in B-tree pool is 00,028,635 MB; Pass #000,001 of 000,001 ... DONE; 00,086,678,683 DEFRAGMENTED B-trees have been rooted so far. 6,272,817 TotalNonUnique (of length 048) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 056 into B-trees, free RAM in B-tree pool is 00,028,292 MB; Pass #000,001 of 000,001 ... DONE; 00,091,860,404 DEFRAGMENTED B-trees have been rooted so far. 6,197,016 TotalNonUnique (of length 056) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 128 into B-trees, free RAM in B-tree pool is 00,027,952 MB; Pass #000,001 of 000,001 ... DONE; 00,096,885,418 DEFRAGMENTED B-trees have been rooted so far. 5,971,432 TotalNonUnique (of length 128) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 256 into B-trees, free RAM in B-tree pool is 00,027,624 MB; Pass #000,001 of 000,001 ... DONE; 00,100,964,432 DEFRAGMENTED B-trees have been rooted so far. 4,672,829 TotalNonUnique (of length 256) keys have been inserted into DEFRAGMENTED B-trees. Histogram (rotate it 90 degrees clockwise) of Unique-Building-Blocks appearing 2[+] times (e.g. 'alfalfa-alfa' stream has 1 BB with length 4 appearing 3 times thus adding 1 to the count for {004]): 4,672,829{256] 5,971,432{128] 6,277,787{064] 6,197,016{056] 6,272,817{048] 6,766,031{036] 7,633,940{028] 8,256,241{024] 9,404,275{018] 10,103,874{016] 11,112,573{014] 12,081,422{012] 12,114,174{010] 10,247,335{008] 7,794,309{006] 4,486,929{004] Leprechaun: Total Searches-n-Inserts Per Second: 2,569,525 SNIPS = 5,401,142,273 keys in 2,102 seconds Leprechaun: RAM needed to house B-trees (relative to the file being ripped): 32N = 6,639MB Leprechaun: RAM needed to build B-trees IN ONE PASS: (Target-Buffer = 16,586 MB) x 1 passes = 16,586MB Note: In case of RAM in spades then may use compile option: -DSpeedUpBuilding=16586 Compressing 211,938,580 bytes ... /; Each rotation means 64KB are encoded; Speed: 0,148,208 B/s; Done 100%; Compression Ratio: 2.92:1; Matches(16/24/48): 1,200,952/982,538/427,845; 128[+] long matches: 144,476; ETA: 0.00 hours NumberOfFullLiterals (lower-the-better): 1359157 Tsuyo_HEURISTIC_APPLIED_thrice_back-to-back: 0 NumberOf(Tiny)Matches[Micro]Window (4)[16B]: 225626 NumberOfMatches[Bheema]Window [128GB window]: 184343 RAM-to-RAM performance: 148208 B/s. Compressed to 72,621,718 bytes. Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x604d,d915 Target-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x34ae,29ce Kernel Time = 2.468 = 0% User Time = 3524.906 = 99% Process Time = 3527.375 = 99% Virtual Memory = 53069 MB Global Time = 3530.571 = 100% Physical Memory = 25607 MB C:\Compression_showdown>Nakamichi_DD_LITE_GCC_64bit_plus-320MB-buffer.exe "Silesia_compression_corpus.Nakamichi" 1>"Silesia_compression_corpus.NKMCH" C:\Compression_showdown>fc "Silesia_compression_corpus" "Silesia_compression_corpus.NKMCH" /b Comparing files Silesia_compression_corpus and SILESIA_COMPRESSION_CORPUS.NKMCH FC: no differences encountered C:\Compression_showdown>if exist "Silesia_compression_corpus.NKMCH" del "Silesia_compression_corpus.NKMCH" C:\Compression_showdown>"Nakamichi_DD_LITE_GCC_64bit_plus-16GB-buffer.exe" Silesia_compression_corpus.Nakamichi /bench SMMi :MM2 0MMMMM: rMMMMMa ZMMM. 0Z :MMM7 7B rMM@ MMM 7 MMM MMM. BMMMa XZ :MM: MMM XMMX XMSANMAYCE; rMMMM rMM; MM@ WM0 2MMMMM ZMMMM MMW BMM : MMMB aMMMMW 2MM :MM8 MM7 8MMi MMMZ XMMM MMa .WMMM0 @MM rMMMMi MMMMa MM, aMM XMMB MM: MMM2 @MXMMM7 0MM MMS WMM MM MMM, MM7 7MM MM: MMX ;MMM iMM MMM8 . MM8 MMS MMMZ . iMBi :MMMX @MMa WB: XMMZ2aX ZMM MMM2 :M2 :MM ZMMM BM8 ;MMMMMaaMM ,MM XMM@MM rMMMMM2aMM XMMMM SMMSMM. MMXMMM MMMM0 aMMMa@MMr MMS ;MM;MM8 XMMMM. MMr MMM0 MM 2MMW MMM 0MM ZMS 2MM. aMMB MMM BM aMM 8Ma MM iMM BMM M7 MMZ ;MMX MMi ;MM ZMW MM0 rM 7MM ;MM MMM7 iMM ZMM2 0MMa MMX 8M: MMM ZMMS BMM2 7 MM8 ZMr :MM rMB MM@ iMM MMZ iMM WMW BM2 MMX @MM MMi ;MMM 0MS 2MM7 aMMM ;MM .8Mi MMM aMM7 ZMMM MM ;Mr XMM iM8 MMa 0MM BMM MMM MM: BMX .MM MM7 aMM BMMM MM ;MMX ZMMMZ BM@ MMM . rMM7 ZMMMZ :MM:MX 0MZ M8 XMM, MMX MM. ;W: rMM aMX ZMM 7MM MM MMMi MM MMZ 8M:MM MM; MMM, MMa 0M:MM 8MWM8 MMZM0 MMM rMM 0MM WMa,M2 MMS MM8 @M0 :MMMXMa MMM WM XMB ; :MM;M8MMM MMM @M XM0 ; MMM@ MMMW MMrZi MMS MMi MMZM0 rMM Mi MM : ;MM BMMMM: rMM MM MMSMa ZMMM@ iMMr 7MM MM MMXM2 :MMM ZMMM 8MM@@ MMZ0 7MM , :MMMM MMBMa 0MWBr ,MMMM MMMM MMS MM MMMZ MMMM MMM MMS .MM MMMZ @MM. MMM MMMM WMM@ 0MM 2M ZMMM :MMMa MMM7 iMMMMMM@ XMMM .MM XMM aMMM MMM: MMW .MM XM@ ZMMM MM0 aMMX 8MMM .MMM WMM ZMr MMMX MMM8 0MMS SMMMMi MMM, rMM,MM2 MMM: SMMM ;MM2 ;MM,MMS MMM, MMM MMM MMM. MMM .MM MMZ MMM XMMM rMMB :MMM7 MM@ BMMM aMM ;MMX aMMX 8MMM ZMM MM8 .MMX MMM2 iMMX ,MMMMMMZ .@; MMM WMM . .: : 0MMZ .2Z :7r. ;i 0MMMaSMi aMMMM7 .WW Nakamichi 'Ryuugan-ditto-1TB', written by Kaze, inspired by Haruhiko Okumura sharing, based on Nobuo Ito's LZSS source, muffinesque tips by m^2, Jim Dempsey and Kirill Kryukov enforced. This latest (2020-Nov-20) compile has B-trees only matchfinder - called 'LITE'; Downloadable at: https://twitter.com/Sanmayce Nakamichi 'Kaidanji' SUPRALITE highlights: - Back to supersimplisticism - 'Kaidanji' returns refined. Simply, ripping only orders 4..16 and then quickly trying to expand the match in the found position. Nakamichi 'Ryuugan-ditto-1TB'/'Dragoneye' LITE highlights: - The Zennish LZSS Microdeduplicator, a texttoy (file-to-file [de]compressor) written in plain C, 100% FREE - licenseless it is; - Single-threaded Non-SIMD console tool written entirely by yours truly Sanmayce; - The first Lempel-Ziv-Storer-Szymanski (LZSS) compressor using 1TB sliding window; - It targets superfast decompression of huge/tera corpora of textual data; - LITE means no memmem() (Railgun) invocations are to be made; - Compressing at Linear Rate due to Matchfinding based on B-trees only; - The Leprechaunesque (Internal/External) B-trees order 3 (2 keys MAX) are brutally-optimized; - Ability to deduplicate (as little as) 64 bytes long chunks 1TB backwards; - Ability to deduplicate 256[+] bytes long chunks 1TB backwards - when SHA3 enabled; - CHUNKABLE! Since there are no headers/metadata in .Nakamichi files, arbitrary chunks/blocks can be compressed-n-concatenated; - SCALABLE! Gets faster when more Physical or/and External RAM is available; - Dynamical (as command line parameters) selection of HASH & B-trees memory blocks; - AUTO-RESETTING PASSES (when TargetBuffer is filled then automaticaly restart building by doubling the passes); - DEFRAGMENTED B-trees are in use during compression stage, their leaves contain only BBs appearing 2[+] times; - Compileable under Windows and Linux, most suitable to run on 3TB machines; - Many thanks go to: Haruhiko Okumura, Nobuo Ito, Hamid Buzidi, Landon Noll, Aleksey Vaneev (https://github.com/avaneev/prvhash). Sanmayce, Nakamichi's author, here, some more notes: - The B-trees form the second layer, the first being HASH table handled by FNV1A-Pippip - the fastest lookup-hash; - Hashpot a.k.a. hashpools (residing in Physical RAM) could be tuned via command line parameter, thus lessening the B-trees heights; - The building of B-trees is done dynamically-n-automatically in PASSES, thus LOCALITY/LOCALIZATION leads to cache-friendliness, for example, instead of confusing/blinding the SSD controller with building 2^27 ~= 128M B-trees at a time, 'PASSES' revision lowers the "noise/mayhem" 128 times by processing e.g. (20bit) 1M B-trees at a time; - For so long, the main goal was to throw it in battle, thus to forge one battle-ready *nix/windows tool with superb hash/search/decompress capabilities. - Straight up, a wish of mine is to have the latest English Wikipedia XML dump decompressed by Nakamichi into physical RAM and then followed by Kazahana's superfast exact/wildcard/fuzzy etudes. - Through the years, such a transparent decompressor (load booster) has been losing momentum and kinda became unnecessary, but not for me, when HDDs had 40MB/s sequential read speeds it was ... boosty, now with SSDs reading at 1700+MB/s, not so smacky, yet uploading (on 128GB machines, 20GB+72GB needed) the whole Wikipedia ~20GB (instead of 72GB) is better, sizewise. - Currently, having 16 (4,6,8,10,12,14,16,18,36,64,24,28,48,56,128,256) MatchLens being hashed (plus 1 being phantomic/temporary), 29bit hashtable results in: (16+1)*8*(2^29) = (16+1)*8*(1<<29) = 69632MB hashpot (the cluster of all hashpools), each slot houses a QWORD (being an offset/address of a B-tree root). - In near future, wanna see a machine (256GB RAM) crunching the 77,265,813,300 bytes file with this command line (it requires 2x72GB + 68GB + 2930GB = 212GB physical RAM + 2.9TB SSD RAM): D:\>timer64 Nakamichi.exe enwiki-20200420-pages-articles.xml enwiki-20200420-pages-articles.xml.Nakamichi 29 3000000 e - In not so far future, wanna see a machine (4096GB RAM) crunching the 72GB file with this command line (it requires 2x72GB + 272GB + 2930GB = 3.3TB physical RAM): D:\>timer64 Nakamichi.exe enwiki-20200420-pages-articles.xml enwiki-20200420-pages-articles.xml.Nakamichi 31 3000000 i Q: - HOW TO BOOST [BUILDING/COMPRESSION] SPEED? A: - Now, the name of the game is "RAM in spades", if we have a lot then speed is going to jump a lot. How? - First, by increasing the hash-pool - 20 (number of bits) was used in 'Taishukan' resulting in 2^20 =~ 1M hash-slots a.k.a. B-trees' roots per keysize, naturally, the bigger the faster since B-trees will have less height and will be more - aiming to reach the ideal case - having only hash-slots with no actual B-trees! Back to 'Taishukan', the testfile is ~77MB, it suggests using 128M slots to lower height/branching of B-trees - 2^27 leads to using 27. Feartime, going from 20 to 27: 8*(15+1)*2^(20) = 134,217,793 bytes; with the speed-up hash needs 8*(15+1)*2^(27) = 17,179,869,184 bytes, yes 16GB. - Second, by lowering passes in building the B-trees - the trick is to increase the number in compile-time parameter -DSpeedUpBuilding=32 - (didn't make it as an execute-time parameter, lazy). If we look at the 'Taishukan' console log it tells us that (Target-Buffer 106 MB) x 64 passes = 6784MB are needed for building the B-trees in ONE PASS ONLY, so compile with -DSpeedUpBuilding=6784 instead. Enfun! Current priority class is HIGH_PRIORITY_CLASS. Allocating Source-Buffer 69 MB ... Allocating Target-Buffer 5,120 MB ... Decompressing 72,621,718 bytes (being the compressed stream) ... Warming up ... RAM-to-RAM performance: 1082 MB/s; 1095 MB/s; 1085 MB/s; 1087 MB/s; 1085 MB/s; 1083 MB/s; 1093 MB/s; 1148 MB/s; 1075 MB/s; 1096 MB/s; 1087 MB/s; 1088 MB/s; 1093 MB/s; 1088 MB/s; 1087 MB/s; 1085 MB/s Enforcing 17 seconds idling to avoid throttling ... 1086 MB/s; 1086 MB/s; 1095 MB/s; 1086 MB/s; 1088 MB/s; 1087 MB/s; 1089 MB/s; 1091 MB/s; 1089 MB/s; 1090 MB/s; 1085 MB/s; 1092 MB/s; 1081 MB/s; 1101 MB/s; 1090 MB/s; 1076 MB/s Enforcing 17 seconds idling to avoid throttling ... 1089 MB/s; 1090 MB/s; 1092 MB/s; 1086 MB/s; 1089 MB/s; 1085 MB/s; 1088 MB/s; 1090 MB/s; 1086 MB/s; 1081 MB/s; 1093 MB/s; 1090 MB/s; 1089 MB/s; 1074 MB/s; 1098 MB/s; 1088 MB/s Enforcing 17 seconds idling to avoid throttling ... 1089 MB/s; 1085 MB/s; 1075 MB/s; 1059 MB/s; 1119 MB/s; 1117 MB/s; 1114 MB/s; 1113 MB/s; 1115 MB/s; 1115 MB/s; 1117 MB/s; 1111 MB/s; 1097 MB/s; 1086 MB/s; 1097 MB/s; 1094 MB/s Enforcing 17 seconds idling to avoid throttling ... 1118 MB/s; 1117 MB/s; 1118 MB/s; 1119 MB/s; 1114 MB/s; 1117 MB/s; 1090 MB/s; 1105 MB/s; 1094 MB/s; 1091 MB/s; 1095 MB/s; 1095 MB/s; 1073 MB/s; 1079 MB/s; 1074 MB/s; 1075 MB/s Enforcing 17 seconds idling to avoid throttling ... 1062 MB/s; 1055 MB/s; 1073 MB/s; 1116 MB/s; 1118 MB/s; 1119 MB/s; 1116 MB/s; 1115 MB/s; 1113 MB/s; 1116 MB/s; 1116 MB/s; 1104 MB/s; 1094 MB/s; 1096 MB/s; 1099 MB/s; 1090 MB/s Enforcing 17 seconds idling to avoid throttling ... 1077 MB/s; 1047 MB/s; 1046 MB/s; 1088 MB/s; 1076 MB/s; 1093 MB/s; 1079 MB/s; 1066 MB/s; 1068 MB/s; 1068 MB/s; 1061 MB/s; 1041 MB/s; 1047 MB/s; 1129 MB/s; 1060 MB/s; 1085 MB/s Enforcing 17 seconds idling to avoid throttling ... 1089 MB/s; 1091 MB/s; 1088 MB/s; 1087 MB/s; 1075 MB/s; 1097 MB/s; 1089 MB/s; 1085 MB/s; 1160 MB/s; 1085 MB/s; 1090 MB/s; 1092 MB/s; 1090 MB/s; 1090 MB/s; 1063 MB/s; 1082 MB/s Enforcing 17 seconds idling to avoid throttling ... RAM-to-RAM (peak) performance: 1160 MB/s. Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x34ae,29ce Target-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x604d,d915 C:\Compression_showdown>call Nakamichify_internal-64GB.bat enwik8 C:\Compression_showdown>TIMER64 Nakamichi_DD_LITE_GCC_64bit_plus-16GB-buffer.exe "enwik8" "enwik8.Nakamichi" 24 34000 i SMMi :MM2 0MMMMM: rMMMMMa ZMMM. 0Z :MMM7 7B rMM@ MMM 7 MMM MMM. BMMMa XZ :MM: MMM XMMX XMSANMAYCE; rMMMM rMM; MM@ WM0 2MMMMM ZMMMM MMW BMM : MMMB aMMMMW 2MM :MM8 MM7 8MMi MMMZ XMMM MMa .WMMM0 @MM rMMMMi MMMMa MM, aMM XMMB MM: MMM2 @MXMMM7 0MM MMS WMM MM MMM, MM7 7MM MM: MMX ;MMM iMM MMM8 . MM8 MMS MMMZ . iMBi :MMMX @MMa WB: XMMZ2aX ZMM MMM2 :M2 :MM ZMMM BM8 ;MMMMMaaMM ,MM XMM@MM rMMMMM2aMM XMMMM SMMSMM. MMXMMM MMMM0 aMMMa@MMr MMS ;MM;MM8 XMMMM. MMr MMM0 MM 2MMW MMM 0MM ZMS 2MM. aMMB MMM BM aMM 8Ma MM iMM BMM M7 MMZ ;MMX MMi ;MM ZMW MM0 rM 7MM ;MM MMM7 iMM ZMM2 0MMa MMX 8M: MMM ZMMS BMM2 7 MM8 ZMr :MM rMB MM@ iMM MMZ iMM WMW BM2 MMX @MM MMi ;MMM 0MS 2MM7 aMMM ;MM .8Mi MMM aMM7 ZMMM MM ;Mr XMM iM8 MMa 0MM BMM MMM MM: BMX .MM MM7 aMM BMMM MM ;MMX ZMMMZ BM@ MMM . rMM7 ZMMMZ :MM:MX 0MZ M8 XMM, MMX MM. ;W: rMM aMX ZMM 7MM MM MMMi MM MMZ 8M:MM MM; MMM, MMa 0M:MM 8MWM8 MMZM0 MMM rMM 0MM WMa,M2 MMS MM8 @M0 :MMMXMa MMM WM XMB ; :MM;M8MMM MMM @M XM0 ; MMM@ MMMW MMrZi MMS MMi MMZM0 rMM Mi MM : ;MM BMMMM: rMM MM MMSMa ZMMM@ iMMr 7MM MM MMXM2 :MMM ZMMM 8MM@@ MMZ0 7MM , :MMMM MMBMa 0MWBr ,MMMM MMMM MMS MM MMMZ MMMM MMM MMS .MM MMMZ @MM. MMM MMMM WMM@ 0MM 2M ZMMM :MMMa MMM7 iMMMMMM@ XMMM .MM XMM aMMM MMM: MMW .MM XM@ ZMMM MM0 aMMX 8MMM .MMM WMM ZMr MMMX MMM8 0MMS SMMMMi MMM, rMM,MM2 MMM: SMMM ;MM2 ;MM,MMS MMM, MMM MMM MMM. MMM .MM MMZ MMM XMMM rMMB :MMM7 MM@ BMMM aMM ;MMX aMMX 8MMM ZMM MM8 .MMX MMM2 iMMX ,MMMMMMZ .@; MMM WMM . .: : 0MMZ .2Z :7r. ;i 0MMMaSMi aMMMM7 .WW Nakamichi 'Ryuugan-ditto-1TB', written by Kaze, inspired by Haruhiko Okumura sharing, based on Nobuo Ito's LZSS source, muffinesque tips by m^2, Jim Dempsey and Kirill Kryukov enforced. This latest (2020-Nov-20) compile has B-trees only matchfinder - called 'LITE'; Downloadable at: https://twitter.com/Sanmayce Nakamichi 'Kaidanji' SUPRALITE highlights: - Back to supersimplisticism - 'Kaidanji' returns refined. Simply, ripping only orders 4..16 and then quickly trying to expand the match in the found position. Nakamichi 'Ryuugan-ditto-1TB'/'Dragoneye' LITE highlights: - The Zennish LZSS Microdeduplicator, a texttoy (file-to-file [de]compressor) written in plain C, 100% FREE - licenseless it is; - Single-threaded Non-SIMD console tool written entirely by yours truly Sanmayce; - The first Lempel-Ziv-Storer-Szymanski (LZSS) compressor using 1TB sliding window; - It targets superfast decompression of huge/tera corpora of textual data; - LITE means no memmem() (Railgun) invocations are to be made; - Compressing at Linear Rate due to Matchfinding based on B-trees only; - The Leprechaunesque (Internal/External) B-trees order 3 (2 keys MAX) are brutally-optimized; - Ability to deduplicate (as little as) 64 bytes long chunks 1TB backwards; - Ability to deduplicate 256[+] bytes long chunks 1TB backwards - when SHA3 enabled; - CHUNKABLE! Since there are no headers/metadata in .Nakamichi files, arbitrary chunks/blocks can be compressed-n-concatenated; - SCALABLE! Gets faster when more Physical or/and External RAM is available; - Dynamical (as command line parameters) selection of HASH & B-trees memory blocks; - AUTO-RESETTING PASSES (when TargetBuffer is filled then automaticaly restart building by doubling the passes); - DEFRAGMENTED B-trees are in use during compression stage, their leaves contain only BBs appearing 2[+] times; - Compileable under Windows and Linux, most suitable to run on 3TB machines; - Many thanks go to: Haruhiko Okumura, Nobuo Ito, Hamid Buzidi, Landon Noll, Aleksey Vaneev (https://github.com/avaneev/prvhash). Sanmayce, Nakamichi's author, here, some more notes: - The B-trees form the second layer, the first being HASH table handled by FNV1A-Pippip - the fastest lookup-hash; - Hashpot a.k.a. hashpools (residing in Physical RAM) could be tuned via command line parameter, thus lessening the B-trees heights; - The building of B-trees is done dynamically-n-automatically in PASSES, thus LOCALITY/LOCALIZATION leads to cache-friendliness, for example, instead of confusing/blinding the SSD controller with building 2^27 ~= 128M B-trees at a time, 'PASSES' revision lowers the "noise/mayhem" 128 times by processing e.g. (20bit) 1M B-trees at a time; - For so long, the main goal was to throw it in battle, thus to forge one battle-ready *nix/windows tool with superb hash/search/decompress capabilities. - Straight up, a wish of mine is to have the latest English Wikipedia XML dump decompressed by Nakamichi into physical RAM and then followed by Kazahana's superfast exact/wildcard/fuzzy etudes. - Through the years, such a transparent decompressor (load booster) has been losing momentum and kinda became unnecessary, but not for me, when HDDs had 40MB/s sequential read speeds it was ... boosty, now with SSDs reading at 1700+MB/s, not so smacky, yet uploading (on 128GB machines, 20GB+72GB needed) the whole Wikipedia ~20GB (instead of 72GB) is better, sizewise. - Currently, having 16 (4,6,8,10,12,14,16,18,36,64,24,28,48,56,128,256) MatchLens being hashed (plus 1 being phantomic/temporary), 29bit hashtable results in: (16+1)*8*(2^29) = (16+1)*8*(1<<29) = 69632MB hashpot (the cluster of all hashpools), each slot houses a QWORD (being an offset/address of a B-tree root). - In near future, wanna see a machine (256GB RAM) crunching the 77,265,813,300 bytes file with this command line (it requires 2x72GB + 68GB + 2930GB = 212GB physical RAM + 2.9TB SSD RAM): D:\>timer64 Nakamichi.exe enwiki-20200420-pages-articles.xml enwiki-20200420-pages-articles.xml.Nakamichi 29 3000000 e - In not so far future, wanna see a machine (4096GB RAM) crunching the 72GB file with this command line (it requires 2x72GB + 272GB + 2930GB = 3.3TB physical RAM): D:\>timer64 Nakamichi.exe enwiki-20200420-pages-articles.xml enwiki-20200420-pages-articles.xml.Nakamichi 31 3000000 i Q: - HOW TO BOOST [BUILDING/COMPRESSION] SPEED? A: - Now, the name of the game is "RAM in spades", if we have a lot then speed is going to jump a lot. How? - First, by increasing the hash-pool - 20 (number of bits) was used in 'Taishukan' resulting in 2^20 =~ 1M hash-slots a.k.a. B-trees' roots per keysize, naturally, the bigger the faster since B-trees will have less height and will be more - aiming to reach the ideal case - having only hash-slots with no actual B-trees! Back to 'Taishukan', the testfile is ~77MB, it suggests using 128M slots to lower height/branching of B-trees - 2^27 leads to using 27. Feartime, going from 20 to 27: 8*(15+1)*2^(20) = 134,217,793 bytes; with the speed-up hash needs 8*(15+1)*2^(27) = 17,179,869,184 bytes, yes 16GB. - Second, by lowering passes in building the B-trees - the trick is to increase the number in compile-time parameter -DSpeedUpBuilding=32 - (didn't make it as an execute-time parameter, lazy). If we look at the 'Taishukan' console log it tells us that (Target-Buffer 106 MB) x 64 passes = 6784MB are needed for building the B-trees in ONE PASS ONLY, so compile with -DSpeedUpBuilding=6784 instead. Enfun! Current priority class is HIGH_PRIORITY_CLASS. This compile uses B-trees only, no memmem() invocations - it compresses worse but much faster. Allocating Source-Buffer 95 MB ... Allocating Target-Buffer 16,479 MB ... Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0xd9b6,d17a Leprechaun: Memory pool for B-trees is 34,000 MB. Leprechaun: In this revision 128MB 16-way hash is used which results in 16 x 16,777,216 internal B-Trees of order 3. Leprechaun: Allocating HASH memory 2,281,701,441 bytes = 8*(16+1)*2^(24) ... OK Leprechaun: Allocating memory for B-trees 34001 MB ... OK Leprechaun: Size of input file: 100,000,000 Leprechaun: Using (first 12 bytes of) DoubleDeuce (a.k.a. CRC64+CRC32C) for Matches 12+ long, in order to reduce memory footprint. Leprechaun: Inserting keys/BBs of order 004 into B-trees, free RAM in B-tree pool is 00,034,000 MB; Pass #000,001 of 000,001 ... DONE; 00,000,701,773 DEFRAGMENTED B-trees have been rooted so far. 716,495 TotalNonUnique (of length 004) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 006 into B-trees, free RAM in B-tree pool is 00,033,966 MB; Pass #000,001 of 000,001 ... DONE; 00,003,609,855 DEFRAGMENTED B-trees have been rooted so far. 3,194,671 TotalNonUnique (of length 006) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 008 into B-trees, free RAM in B-tree pool is 00,033,815 MB; Pass #000,001 of 000,001 ... DONE; 00,009,057,506 DEFRAGMENTED B-trees have been rooted so far. 6,593,330 TotalNonUnique (of length 008) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 010 into B-trees, free RAM in B-tree pool is 00,033,499 MB; Pass #000,001 of 000,001 ... DONE; 00,016,158,919 DEFRAGMENTED B-trees have been rooted so far. 9,237,534 TotalNonUnique (of length 010) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 012 into B-trees, free RAM in B-tree pool is 00,033,042 MB; Pass #000,001 of 000,001 ... DONE; 00,023,698,418 DEFRAGMENTED B-trees have been rooted so far. 10,012,844 TotalNonUnique (of length 012) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 014 into B-trees, free RAM in B-tree pool is 00,032,519 MB; Pass #000,001 of 000,001 ... DONE; 00,030,795,705 DEFRAGMENTED B-trees have been rooted so far. 9,228,817 TotalNonUnique (of length 014) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 016 into B-trees, free RAM in B-tree pool is 00,032,033 MB; Pass #000,001 of 000,001 ... DONE; 00,036,992,791 DEFRAGMENTED B-trees have been rooted so far. 7,735,783 TotalNonUnique (of length 016) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 018 into B-trees, free RAM in B-tree pool is 00,031,618 MB; Pass #000,001 of 000,001 ... DONE; 00,042,178,813 DEFRAGMENTED B-trees have been rooted so far. 6,203,830 TotalNonUnique (of length 018) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 036 into B-trees, free RAM in B-tree pool is 00,031,278 MB; Pass #000,001 of 000,001 ... DONE; 00,043,665,684 DEFRAGMENTED B-trees have been rooted so far. 1,556,753 TotalNonUnique (of length 036) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 064 into B-trees, free RAM in B-tree pool is 00,031,185 MB; Pass #000,001 of 000,001 ... DONE; 00,044,414,369 DEFRAGMENTED B-trees have been rooted so far. 765,900 TotalNonUnique (of length 064) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 024 into B-trees, free RAM in B-tree pool is 00,031,137 MB; Pass #000,001 of 000,001 ... DONE; 00,047,347,671 DEFRAGMENTED B-trees have been rooted so far. 3,225,048 TotalNonUnique (of length 024) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 028 into B-trees, free RAM in B-tree pool is 00,030,951 MB; Pass #000,001 of 000,001 ... DONE; 00,049,542,019 DEFRAGMENTED B-trees have been rooted so far. 2,351,658 TotalNonUnique (of length 028) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 048 into B-trees, free RAM in B-tree pool is 00,030,812 MB; Pass #000,001 of 000,001 ... DONE; 00,050,578,862 DEFRAGMENTED B-trees have been rooted so far. 1,070,339 TotalNonUnique (of length 048) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 056 into B-trees, free RAM in B-tree pool is 00,030,746 MB; Pass #000,001 of 000,001 ... DONE; 00,051,463,527 DEFRAGMENTED B-trees have been rooted so far. 908,789 TotalNonUnique (of length 056) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 128 into B-trees, free RAM in B-tree pool is 00,030,691 MB; Pass #000,001 of 000,001 ... DONE; 00,051,780,043 DEFRAGMENTED B-trees have been rooted so far. 319,546 TotalNonUnique (of length 128) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 256 into B-trees, free RAM in B-tree pool is 00,030,671 MB; Pass #000,001 of 000,001 ... DONE; 00,051,885,254 DEFRAGMENTED B-trees have been rooted so far. 105,555 TotalNonUnique (of length 256) keys have been inserted into DEFRAGMENTED B-trees. Histogram (rotate it 90 degrees clockwise) of Unique-Building-Blocks appearing 2[+] times (e.g. 'alfalfa-alfa' stream has 1 BB with length 4 appearing 3 times thus adding 1 to the count for {004]): 105,555{256] 319,546{128] 765,900{064] 908,789{056] 1,070,339{048] 1,556,753{036] 2,351,658{028] 3,225,048{024] 6,203,830{018] 7,735,783{016] 9,228,817{014] 10,012,844{012] 9,237,534{010] 6,593,330{008] 3,194,671{006] 716,495{004] Leprechaun: Total Searches-n-Inserts Per Second: 2,733,219 SNIPS = 2,694,954,662 keys in 986 seconds Leprechaun: RAM needed to house B-trees (relative to the file being ripped): 34N = 3,336MB Leprechaun: RAM needed to build B-trees IN ONE PASS: (Target-Buffer = 16,479 MB) x 1 passes = 16,479MB Note: In case of RAM in spades then may use compile option: -DSpeedUpBuilding=16479 Compressing 100,000,000 bytes ... /; Each rotation means 64KB are encoded; Speed: 0,164,473 B/s; Done 100%; Compression Ratio: 3.03:1; Matches(16/24/48): 945,737/350,360/71,620; 128[+] long matches: 7,896; ETA: 0.00 hours NumberOfFullLiterals (lower-the-better): 19324 Tsuyo_HEURISTIC_APPLIED_thrice_back-to-back: 0 NumberOf(Tiny)Matches[Micro]Window (4)[16B]: 64479 NumberOfMatches[Bheema]Window [128GB window]: 18067 RAM-to-RAM performance: 164473 B/s. Compressed to 33,044,928 bytes. Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0xd9b6,d17a Target-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x162f,01e9 Kernel Time = 1.859 = 0% User Time = 1589.171 = 99% Process Time = 1591.031 = 99% Virtual Memory = 52855 MB Global Time = 1594.229 = 100% Physical Memory = 22090 MB C:\Compression_showdown>Nakamichi_DD_LITE_GCC_64bit_plus-320MB-buffer.exe "enwik8.Nakamichi" 1>"enwik8.NKMCH" C:\Compression_showdown>fc "enwik8" "enwik8.NKMCH" /b Comparing files enwik8 and ENWIK8.NKMCH FC: no differences encountered C:\Compression_showdown>if exist "enwik8.NKMCH" del "enwik8.NKMCH" C:\Compression_showdown>"Nakamichi_DD_LITE_GCC_64bit_plus-16GB-buffer.exe" enwik8.Nakamichi /bench SMMi :MM2 0MMMMM: rMMMMMa ZMMM. 0Z :MMM7 7B rMM@ MMM 7 MMM MMM. BMMMa XZ :MM: MMM XMMX XMSANMAYCE; rMMMM rMM; MM@ WM0 2MMMMM ZMMMM MMW BMM : MMMB aMMMMW 2MM :MM8 MM7 8MMi MMMZ XMMM MMa .WMMM0 @MM rMMMMi MMMMa MM, aMM XMMB MM: MMM2 @MXMMM7 0MM MMS WMM MM MMM, MM7 7MM MM: MMX ;MMM iMM MMM8 . MM8 MMS MMMZ . iMBi :MMMX @MMa WB: XMMZ2aX ZMM MMM2 :M2 :MM ZMMM BM8 ;MMMMMaaMM ,MM XMM@MM rMMMMM2aMM XMMMM SMMSMM. MMXMMM MMMM0 aMMMa@MMr MMS ;MM;MM8 XMMMM. MMr MMM0 MM 2MMW MMM 0MM ZMS 2MM. aMMB MMM BM aMM 8Ma MM iMM BMM M7 MMZ ;MMX MMi ;MM ZMW MM0 rM 7MM ;MM MMM7 iMM ZMM2 0MMa MMX 8M: MMM ZMMS BMM2 7 MM8 ZMr :MM rMB MM@ iMM MMZ iMM WMW BM2 MMX @MM MMi ;MMM 0MS 2MM7 aMMM ;MM .8Mi MMM aMM7 ZMMM MM ;Mr XMM iM8 MMa 0MM BMM MMM MM: BMX .MM MM7 aMM BMMM MM ;MMX ZMMMZ BM@ MMM . rMM7 ZMMMZ :MM:MX 0MZ M8 XMM, MMX MM. ;W: rMM aMX ZMM 7MM MM MMMi MM MMZ 8M:MM MM; MMM, MMa 0M:MM 8MWM8 MMZM0 MMM rMM 0MM WMa,M2 MMS MM8 @M0 :MMMXMa MMM WM XMB ; :MM;M8MMM MMM @M XM0 ; MMM@ MMMW MMrZi MMS MMi MMZM0 rMM Mi MM : ;MM BMMMM: rMM MM MMSMa ZMMM@ iMMr 7MM MM MMXM2 :MMM ZMMM 8MM@@ MMZ0 7MM , :MMMM MMBMa 0MWBr ,MMMM MMMM MMS MM MMMZ MMMM MMM MMS .MM MMMZ @MM. MMM MMMM WMM@ 0MM 2M ZMMM :MMMa MMM7 iMMMMMM@ XMMM .MM XMM aMMM MMM: MMW .MM XM@ ZMMM MM0 aMMX 8MMM .MMM WMM ZMr MMMX MMM8 0MMS SMMMMi MMM, rMM,MM2 MMM: SMMM ;MM2 ;MM,MMS MMM, MMM MMM MMM. MMM .MM MMZ MMM XMMM rMMB :MMM7 MM@ BMMM aMM ;MMX aMMX 8MMM ZMM MM8 .MMX MMM2 iMMX ,MMMMMMZ .@; MMM WMM . .: : 0MMZ .2Z :7r. ;i 0MMMaSMi aMMMM7 .WW Nakamichi 'Ryuugan-ditto-1TB', written by Kaze, inspired by Haruhiko Okumura sharing, based on Nobuo Ito's LZSS source, muffinesque tips by m^2, Jim Dempsey and Kirill Kryukov enforced. This latest (2020-Nov-20) compile has B-trees only matchfinder - called 'LITE'; Downloadable at: https://twitter.com/Sanmayce Nakamichi 'Kaidanji' SUPRALITE highlights: - Back to supersimplisticism - 'Kaidanji' returns refined. Simply, ripping only orders 4..16 and then quickly trying to expand the match in the found position. Nakamichi 'Ryuugan-ditto-1TB'/'Dragoneye' LITE highlights: - The Zennish LZSS Microdeduplicator, a texttoy (file-to-file [de]compressor) written in plain C, 100% FREE - licenseless it is; - Single-threaded Non-SIMD console tool written entirely by yours truly Sanmayce; - The first Lempel-Ziv-Storer-Szymanski (LZSS) compressor using 1TB sliding window; - It targets superfast decompression of huge/tera corpora of textual data; - LITE means no memmem() (Railgun) invocations are to be made; - Compressing at Linear Rate due to Matchfinding based on B-trees only; - The Leprechaunesque (Internal/External) B-trees order 3 (2 keys MAX) are brutally-optimized; - Ability to deduplicate (as little as) 64 bytes long chunks 1TB backwards; - Ability to deduplicate 256[+] bytes long chunks 1TB backwards - when SHA3 enabled; - CHUNKABLE! Since there are no headers/metadata in .Nakamichi files, arbitrary chunks/blocks can be compressed-n-concatenated; - SCALABLE! Gets faster when more Physical or/and External RAM is available; - Dynamical (as command line parameters) selection of HASH & B-trees memory blocks; - AUTO-RESETTING PASSES (when TargetBuffer is filled then automaticaly restart building by doubling the passes); - DEFRAGMENTED B-trees are in use during compression stage, their leaves contain only BBs appearing 2[+] times; - Compileable under Windows and Linux, most suitable to run on 3TB machines; - Many thanks go to: Haruhiko Okumura, Nobuo Ito, Hamid Buzidi, Landon Noll, Aleksey Vaneev (https://github.com/avaneev/prvhash). Sanmayce, Nakamichi's author, here, some more notes: - The B-trees form the second layer, the first being HASH table handled by FNV1A-Pippip - the fastest lookup-hash; - Hashpot a.k.a. hashpools (residing in Physical RAM) could be tuned via command line parameter, thus lessening the B-trees heights; - The building of B-trees is done dynamically-n-automatically in PASSES, thus LOCALITY/LOCALIZATION leads to cache-friendliness, for example, instead of confusing/blinding the SSD controller with building 2^27 ~= 128M B-trees at a time, 'PASSES' revision lowers the "noise/mayhem" 128 times by processing e.g. (20bit) 1M B-trees at a time; - For so long, the main goal was to throw it in battle, thus to forge one battle-ready *nix/windows tool with superb hash/search/decompress capabilities. - Straight up, a wish of mine is to have the latest English Wikipedia XML dump decompressed by Nakamichi into physical RAM and then followed by Kazahana's superfast exact/wildcard/fuzzy etudes. - Through the years, such a transparent decompressor (load booster) has been losing momentum and kinda became unnecessary, but not for me, when HDDs had 40MB/s sequential read speeds it was ... boosty, now with SSDs reading at 1700+MB/s, not so smacky, yet uploading (on 128GB machines, 20GB+72GB needed) the whole Wikipedia ~20GB (instead of 72GB) is better, sizewise. - Currently, having 16 (4,6,8,10,12,14,16,18,36,64,24,28,48,56,128,256) MatchLens being hashed (plus 1 being phantomic/temporary), 29bit hashtable results in: (16+1)*8*(2^29) = (16+1)*8*(1<<29) = 69632MB hashpot (the cluster of all hashpools), each slot houses a QWORD (being an offset/address of a B-tree root). - In near future, wanna see a machine (256GB RAM) crunching the 77,265,813,300 bytes file with this command line (it requires 2x72GB + 68GB + 2930GB = 212GB physical RAM + 2.9TB SSD RAM): D:\>timer64 Nakamichi.exe enwiki-20200420-pages-articles.xml enwiki-20200420-pages-articles.xml.Nakamichi 29 3000000 e - In not so far future, wanna see a machine (4096GB RAM) crunching the 72GB file with this command line (it requires 2x72GB + 272GB + 2930GB = 3.3TB physical RAM): D:\>timer64 Nakamichi.exe enwiki-20200420-pages-articles.xml enwiki-20200420-pages-articles.xml.Nakamichi 31 3000000 i Q: - HOW TO BOOST [BUILDING/COMPRESSION] SPEED? A: - Now, the name of the game is "RAM in spades", if we have a lot then speed is going to jump a lot. How? - First, by increasing the hash-pool - 20 (number of bits) was used in 'Taishukan' resulting in 2^20 =~ 1M hash-slots a.k.a. B-trees' roots per keysize, naturally, the bigger the faster since B-trees will have less height and will be more - aiming to reach the ideal case - having only hash-slots with no actual B-trees! Back to 'Taishukan', the testfile is ~77MB, it suggests using 128M slots to lower height/branching of B-trees - 2^27 leads to using 27. Feartime, going from 20 to 27: 8*(15+1)*2^(20) = 134,217,793 bytes; with the speed-up hash needs 8*(15+1)*2^(27) = 17,179,869,184 bytes, yes 16GB. - Second, by lowering passes in building the B-trees - the trick is to increase the number in compile-time parameter -DSpeedUpBuilding=32 - (didn't make it as an execute-time parameter, lazy). If we look at the 'Taishukan' console log it tells us that (Target-Buffer 106 MB) x 64 passes = 6784MB are needed for building the B-trees in ONE PASS ONLY, so compile with -DSpeedUpBuilding=6784 instead. Enfun! Current priority class is HIGH_PRIORITY_CLASS. Allocating Source-Buffer 31 MB ... Allocating Target-Buffer 5,120 MB ... Decompressing 33,044,928 bytes (being the compressed stream) ... Warming up ... RAM-to-RAM performance: 827 MB/s; 826 MB/s; 828 MB/s; 831 MB/s; 818 MB/s; 842 MB/s; 829 MB/s; 829 MB/s; 827 MB/s; 828 MB/s; 832 MB/s; 825 MB/s; 831 MB/s; 826 MB/s; 936 MB/s; 830 MB/s Enforcing 17 seconds idling to avoid throttling ... 820 MB/s; 831 MB/s; 823 MB/s; 825 MB/s; 827 MB/s; 820 MB/s; 824 MB/s; 828 MB/s; 823 MB/s; 820 MB/s; 824 MB/s; 828 MB/s; 823 MB/s; 823 MB/s; 826 MB/s; 826 MB/s Enforcing 17 seconds idling to avoid throttling ... 825 MB/s; 827 MB/s; 817 MB/s; 825 MB/s; 834 MB/s; 828 MB/s; 830 MB/s; 827 MB/s; 819 MB/s; 815 MB/s; 811 MB/s; 807 MB/s; 813 MB/s; 807 MB/s; 811 MB/s; 809 MB/s Enforcing 17 seconds idling to avoid throttling ... 837 MB/s; 818 MB/s; 826 MB/s; 816 MB/s; 812 MB/s; 812 MB/s; 805 MB/s; 805 MB/s; 819 MB/s; 797 MB/s; 830 MB/s; 821 MB/s; 814 MB/s; 836 MB/s; 829 MB/s; 826 MB/s Enforcing 17 seconds idling to avoid throttling ... 811 MB/s; 807 MB/s; 802 MB/s; 815 MB/s; 813 MB/s; 819 MB/s; 817 MB/s; 821 MB/s; 835 MB/s; 823 MB/s; 822 MB/s; 817 MB/s; 835 MB/s; 824 MB/s; 821 MB/s; 828 MB/s Enforcing 17 seconds idling to avoid throttling ... 821 MB/s; 812 MB/s; 808 MB/s; 809 MB/s; 802 MB/s; 814 MB/s; 810 MB/s; 819 MB/s; 821 MB/s; 837 MB/s; 823 MB/s; 830 MB/s; 815 MB/s; 837 MB/s; 810 MB/s; 812 MB/s Enforcing 17 seconds idling to avoid throttling ... 811 MB/s; 817 MB/s; 797 MB/s; 788 MB/s; 799 MB/s; 788 MB/s; 798 MB/s; 794 MB/s; 798 MB/s; 829 MB/s; 826 MB/s; 821 MB/s; 835 MB/s; 823 MB/s; 828 MB/s; 826 MB/s Enforcing 17 seconds idling to avoid throttling ... 828 MB/s; 829 MB/s; 827 MB/s; 824 MB/s; 822 MB/s; 829 MB/s; 827 MB/s; 827 MB/s; 827 MB/s; 831 MB/s; 823 MB/s; 810 MB/s; 806 MB/s; 820 MB/s; 811 MB/s; 814 MB/s Enforcing 17 seconds idling to avoid throttling ... RAM-to-RAM (peak) performance: 936 MB/s. Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x162f,01e9 Target-file-Hash(FNV1A_YoshimitsuTRIAD) = 0xd9b6,d17a C:\Compression_showdown>call Nakamichify_internal-64GB.bat enwik9 C:\Compression_showdown>TIMER64 Nakamichi_DD_LITE_GCC_64bit_plus-16GB-buffer.exe "enwik9" "enwik9.Nakamichi" 24 34000 i SMMi :MM2 0MMMMM: rMMMMMa ZMMM. 0Z :MMM7 7B rMM@ MMM 7 MMM MMM. BMMMa XZ :MM: MMM XMMX XMSANMAYCE; rMMMM rMM; MM@ WM0 2MMMMM ZMMMM MMW BMM : MMMB aMMMMW 2MM :MM8 MM7 8MMi MMMZ XMMM MMa .WMMM0 @MM rMMMMi MMMMa MM, aMM XMMB MM: MMM2 @MXMMM7 0MM MMS WMM MM MMM, MM7 7MM MM: MMX ;MMM iMM MMM8 . MM8 MMS MMMZ . iMBi :MMMX @MMa WB: XMMZ2aX ZMM MMM2 :M2 :MM ZMMM BM8 ;MMMMMaaMM ,MM XMM@MM rMMMMM2aMM XMMMM SMMSMM. MMXMMM MMMM0 aMMMa@MMr MMS ;MM;MM8 XMMMM. MMr MMM0 MM 2MMW MMM 0MM ZMS 2MM. aMMB MMM BM aMM 8Ma MM iMM BMM M7 MMZ ;MMX MMi ;MM ZMW MM0 rM 7MM ;MM MMM7 iMM ZMM2 0MMa MMX 8M: MMM ZMMS BMM2 7 MM8 ZMr :MM rMB MM@ iMM MMZ iMM WMW BM2 MMX @MM MMi ;MMM 0MS 2MM7 aMMM ;MM .8Mi MMM aMM7 ZMMM MM ;Mr XMM iM8 MMa 0MM BMM MMM MM: BMX .MM MM7 aMM BMMM MM ;MMX ZMMMZ BM@ MMM . rMM7 ZMMMZ :MM:MX 0MZ M8 XMM, MMX MM. ;W: rMM aMX ZMM 7MM MM MMMi MM MMZ 8M:MM MM; MMM, MMa 0M:MM 8MWM8 MMZM0 MMM rMM 0MM WMa,M2 MMS MM8 @M0 :MMMXMa MMM WM XMB ; :MM;M8MMM MMM @M XM0 ; MMM@ MMMW MMrZi MMS MMi MMZM0 rMM Mi MM : ;MM BMMMM: rMM MM MMSMa ZMMM@ iMMr 7MM MM MMXM2 :MMM ZMMM 8MM@@ MMZ0 7MM , :MMMM MMBMa 0MWBr ,MMMM MMMM MMS MM MMMZ MMMM MMM MMS .MM MMMZ @MM. MMM MMMM WMM@ 0MM 2M ZMMM :MMMa MMM7 iMMMMMM@ XMMM .MM XMM aMMM MMM: MMW .MM XM@ ZMMM MM0 aMMX 8MMM .MMM WMM ZMr MMMX MMM8 0MMS SMMMMi MMM, rMM,MM2 MMM: SMMM ;MM2 ;MM,MMS MMM, MMM MMM MMM. MMM .MM MMZ MMM XMMM rMMB :MMM7 MM@ BMMM aMM ;MMX aMMX 8MMM ZMM MM8 .MMX MMM2 iMMX ,MMMMMMZ .@; MMM WMM . .: : 0MMZ .2Z :7r. ;i 0MMMaSMi aMMMM7 .WW Nakamichi 'Ryuugan-ditto-1TB', written by Kaze, inspired by Haruhiko Okumura sharing, based on Nobuo Ito's LZSS source, muffinesque tips by m^2, Jim Dempsey and Kirill Kryukov enforced. This latest (2020-Nov-20) compile has B-trees only matchfinder - called 'LITE'; Downloadable at: https://twitter.com/Sanmayce Nakamichi 'Kaidanji' SUPRALITE highlights: - Back to supersimplisticism - 'Kaidanji' returns refined. Simply, ripping only orders 4..16 and then quickly trying to expand the match in the found position. Nakamichi 'Ryuugan-ditto-1TB'/'Dragoneye' LITE highlights: - The Zennish LZSS Microdeduplicator, a texttoy (file-to-file [de]compressor) written in plain C, 100% FREE - licenseless it is; - Single-threaded Non-SIMD console tool written entirely by yours truly Sanmayce; - The first Lempel-Ziv-Storer-Szymanski (LZSS) compressor using 1TB sliding window; - It targets superfast decompression of huge/tera corpora of textual data; - LITE means no memmem() (Railgun) invocations are to be made; - Compressing at Linear Rate due to Matchfinding based on B-trees only; - The Leprechaunesque (Internal/External) B-trees order 3 (2 keys MAX) are brutally-optimized; - Ability to deduplicate (as little as) 64 bytes long chunks 1TB backwards; - Ability to deduplicate 256[+] bytes long chunks 1TB backwards - when SHA3 enabled; - CHUNKABLE! Since there are no headers/metadata in .Nakamichi files, arbitrary chunks/blocks can be compressed-n-concatenated; - SCALABLE! Gets faster when more Physical or/and External RAM is available; - Dynamical (as command line parameters) selection of HASH & B-trees memory blocks; - AUTO-RESETTING PASSES (when TargetBuffer is filled then automaticaly restart building by doubling the passes); - DEFRAGMENTED B-trees are in use during compression stage, their leaves contain only BBs appearing 2[+] times; - Compileable under Windows and Linux, most suitable to run on 3TB machines; - Many thanks go to: Haruhiko Okumura, Nobuo Ito, Hamid Buzidi, Landon Noll, Aleksey Vaneev (https://github.com/avaneev/prvhash). Sanmayce, Nakamichi's author, here, some more notes: - The B-trees form the second layer, the first being HASH table handled by FNV1A-Pippip - the fastest lookup-hash; - Hashpot a.k.a. hashpools (residing in Physical RAM) could be tuned via command line parameter, thus lessening the B-trees heights; - The building of B-trees is done dynamically-n-automatically in PASSES, thus LOCALITY/LOCALIZATION leads to cache-friendliness, for example, instead of confusing/blinding the SSD controller with building 2^27 ~= 128M B-trees at a time, 'PASSES' revision lowers the "noise/mayhem" 128 times by processing e.g. (20bit) 1M B-trees at a time; - For so long, the main goal was to throw it in battle, thus to forge one battle-ready *nix/windows tool with superb hash/search/decompress capabilities. - Straight up, a wish of mine is to have the latest English Wikipedia XML dump decompressed by Nakamichi into physical RAM and then followed by Kazahana's superfast exact/wildcard/fuzzy etudes. - Through the years, such a transparent decompressor (load booster) has been losing momentum and kinda became unnecessary, but not for me, when HDDs had 40MB/s sequential read speeds it was ... boosty, now with SSDs reading at 1700+MB/s, not so smacky, yet uploading (on 128GB machines, 20GB+72GB needed) the whole Wikipedia ~20GB (instead of 72GB) is better, sizewise. - Currently, having 16 (4,6,8,10,12,14,16,18,36,64,24,28,48,56,128,256) MatchLens being hashed (plus 1 being phantomic/temporary), 29bit hashtable results in: (16+1)*8*(2^29) = (16+1)*8*(1<<29) = 69632MB hashpot (the cluster of all hashpools), each slot houses a QWORD (being an offset/address of a B-tree root). - In near future, wanna see a machine (256GB RAM) crunching the 77,265,813,300 bytes file with this command line (it requires 2x72GB + 68GB + 2930GB = 212GB physical RAM + 2.9TB SSD RAM): D:\>timer64 Nakamichi.exe enwiki-20200420-pages-articles.xml enwiki-20200420-pages-articles.xml.Nakamichi 29 3000000 e - In not so far future, wanna see a machine (4096GB RAM) crunching the 72GB file with this command line (it requires 2x72GB + 272GB + 2930GB = 3.3TB physical RAM): D:\>timer64 Nakamichi.exe enwiki-20200420-pages-articles.xml enwiki-20200420-pages-articles.xml.Nakamichi 31 3000000 i Q: - HOW TO BOOST [BUILDING/COMPRESSION] SPEED? A: - Now, the name of the game is "RAM in spades", if we have a lot then speed is going to jump a lot. How? - First, by increasing the hash-pool - 20 (number of bits) was used in 'Taishukan' resulting in 2^20 =~ 1M hash-slots a.k.a. B-trees' roots per keysize, naturally, the bigger the faster since B-trees will have less height and will be more - aiming to reach the ideal case - having only hash-slots with no actual B-trees! Back to 'Taishukan', the testfile is ~77MB, it suggests using 128M slots to lower height/branching of B-trees - 2^27 leads to using 27. Feartime, going from 20 to 27: 8*(15+1)*2^(20) = 134,217,793 bytes; with the speed-up hash needs 8*(15+1)*2^(27) = 17,179,869,184 bytes, yes 16GB. - Second, by lowering passes in building the B-trees - the trick is to increase the number in compile-time parameter -DSpeedUpBuilding=32 - (didn't make it as an execute-time parameter, lazy). If we look at the 'Taishukan' console log it tells us that (Target-Buffer 106 MB) x 64 passes = 6784MB are needed for building the B-trees in ONE PASS ONLY, so compile with -DSpeedUpBuilding=6784 instead. Enfun! Current priority class is HIGH_PRIORITY_CLASS. This compile uses B-trees only, no memmem() invocations - it compresses worse but much faster. Allocating Source-Buffer 953 MB ... Allocating Target-Buffer 17,337 MB ... Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x209a,2b59 Leprechaun: Memory pool for B-trees is 34,000 MB. Leprechaun: In this revision 128MB 16-way hash is used which results in 16 x 16,777,216 internal B-Trees of order 3. Leprechaun: Allocating HASH memory 2,281,701,441 bytes = 8*(16+1)*2^(24) ... OK Leprechaun: Allocating memory for B-trees 34001 MB ... OK Leprechaun: Size of input file: 1,000,000,000 Leprechaun: Using (first 12 bytes of) DoubleDeuce (a.k.a. CRC64+CRC32C) for Matches 12+ long, in order to reduce memory footprint. Leprechaun: Inserting keys/BBs of order 004 into B-trees, free RAM in B-tree pool is 00,034,000 MB; Pass #000,001 of 000,001 ... DONE; 00,001,900,151 DEFRAGMENTED B-trees have been rooted so far. 2,015,721 TotalNonUnique (of length 004) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 006 into B-trees, free RAM in B-tree pool is 00,033,908 MB; Pass #000,001 of 000,001 ... DONE; 00,011,317,143 DEFRAGMENTED B-trees have been rooted so far. 13,823,004 TotalNonUnique (of length 006) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 008 into B-trees, free RAM in B-tree pool is 00,033,335 MB; Pass #000,001 of 000,001 ... DONE; 00,025,889,580 DEFRAGMENTED B-trees have been rooted so far. 34,113,614 TotalNonUnique (of length 008) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 010 into B-trees, free RAM in B-tree pool is 00,031,862 MB; Pass #000,001 of 000,001 ... DONE; 00,042,101,154 DEFRAGMENTED B-trees have been rooted so far. 56,945,328 TotalNonUnique (of length 010) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 012 into B-trees, free RAM in B-tree pool is 00,029,257 MB; Pass #000,001 of 000,001 ... DONE; 00,058,670,502 DEFRAGMENTED B-trees have been rooted so far. 73,709,253 TotalNonUnique (of length 012) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 014 into B-trees, free RAM in B-tree pool is 00,025,665 MB; Pass #000,001 of 000,001 ... Leprechaun: Failure! Increment 'Memory for B-trees'! Automatically increasing number of passes in order to fit B-trees into Target Buffer. Leprechaun: Inserting keys/BBs of order 014 into B-trees, free RAM in B-tree pool is 00,023,705 MB; Pass #000,002 of 000,002 ... DONE; 00,075,303,155 DEFRAGMENTED B-trees have been rooted so far. 79,860,067 TotalNonUnique (of length 014) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 016 into B-trees, free RAM in B-tree pool is 00,019,848 MB; Pass #000,002 of 000,002 ... DONE; 00,091,911,142 DEFRAGMENTED B-trees have been rooted so far. 77,086,688 TotalNonUnique (of length 016) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 018 into B-trees, free RAM in B-tree pool is 00,016,241 MB; Pass #000,002 of 000,002 ... DONE; 00,108,419,082 DEFRAGMENTED B-trees have been rooted so far. 69,362,824 TotalNonUnique (of length 018) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 036 into B-trees, free RAM in B-tree pool is 00,014,531 MB; Pass #000,001 of 000,002 ... Leprechaun: Failure! Increment 'Memory for B-trees'! Automatically increasing number of passes in order to fit B-trees into Target Buffer. Leprechaun: Inserting keys/BBs of order 036 into B-trees, free RAM in B-tree pool is 00,013,526 MB; Pass #000,004 of 000,004 ... DONE; 00,121,860,939 DEFRAGMENTED B-trees have been rooted so far. 27,100,957 TotalNonUnique (of length 036) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 064 into B-trees, free RAM in B-tree pool is 00,012,500 MB; Pass #000,004 of 000,004 ... DONE; 00,133,063,653 DEFRAGMENTED B-trees have been rooted so far. 18,478,156 TotalNonUnique (of length 064) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 024 into B-trees, free RAM in B-tree pool is 00,010,597 MB; Pass #000,004 of 000,004 ... DONE; 00,148,695,371 DEFRAGMENTED B-trees have been rooted so far. 45,029,023 TotalNonUnique (of length 024) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 028 into B-trees, free RAM in B-tree pool is 00,008,716 MB; Pass #000,004 of 000,004 ... DONE; 00,163,467,633 DEFRAGMENTED B-trees have been rooted so far. 35,644,579 TotalNonUnique (of length 028) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 048 into B-trees, free RAM in B-tree pool is 00,007,471 MB; Pass #000,004 of 000,004 ... DONE; 00,175,625,745 DEFRAGMENTED B-trees have been rooted so far. 21,638,453 TotalNonUnique (of length 048) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 056 into B-trees, free RAM in B-tree pool is 00,006,463 MB; Pass #000,004 of 000,004 ... DONE; 00,187,255,809 DEFRAGMENTED B-trees have been rooted so far. 19,819,534 TotalNonUnique (of length 056) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 128 into B-trees, free RAM in B-tree pool is 00,005,796 MB; Pass #000,004 of 000,004 ... DONE; 00,195,232,354 DEFRAGMENTED B-trees have been rooted so far. 10,822,295 TotalNonUnique (of length 128) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 256 into B-trees, free RAM in B-tree pool is 00,005,543 MB; Pass #000,004 of 000,004 ... DONE; 00,197,600,833 DEFRAGMENTED B-trees have been rooted so far. 2,553,191 TotalNonUnique (of length 256) keys have been inserted into DEFRAGMENTED B-trees. Histogram (rotate it 90 degrees clockwise) of Unique-Building-Blocks appearing 2[+] times (e.g. 'alfalfa-alfa' stream has 1 BB with length 4 appearing 3 times thus adding 1 to the count for {004]): 2,553,191{256] 10,822,295{128] 18,478,156{064] 19,819,534{056] 21,638,453{048] 27,100,957{036] 35,644,579{028] 45,029,023{024] 69,362,824{018] 77,086,688{016] 79,860,067{014] 73,709,253{012] 56,945,328{010] 34,113,614{008] 13,823,004{006] 2,015,721{004] Leprechaun: Total Searches-n-Inserts Per Second: 2,627,300 SNIPS = 53,741,424,769 keys in 20,455 seconds Leprechaun: RAM needed to house B-trees (relative to the file being ripped): 29N = 28,495MB Leprechaun: RAM needed to build B-trees IN ONE PASS: (Target-Buffer = 17,337 MB) x 4 passes = 69,348MB Note: In case of RAM in spades then may use compile option: -DSpeedUpBuilding=69348 Compressing 1,000,000,000 bytes ... -; Each rotation means 64KB are encoded; Speed: 0,114,955 B/s; Done 100%; Compression Ratio: 3.58:1; Matches(16/24/48): 6,769,704/4,809,537/1,717,079; 128[+] long matches: 389,301; ETA: 0.00 hours NumberOfFullLiterals (lower-the-better): 138025 Tsuyo_HEURISTIC_APPLIED_thrice_back-to-back: 0 NumberOf(Tiny)Matches[Micro]Window (4)[16B]: 518852 NumberOfMatches[Bheema]Window [128GB window]: 1483819 RAM-to-RAM performance: 114955 B/s. Compressed to 279,301,006 bytes. Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x209a,2b59 Target-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x5b09,d619 Kernel Time = 5.250 = 0% User Time = 29136.984 = 99% Process Time = 29142.234 = 99% Virtual Memory = 54575 MB Global Time = 29154.867 = 100% Physical Memory = 48966 MB C:\Compression_showdown>Nakamichi_DD_LITE_GCC_64bit_plus-320MB-buffer.exe "enwik9.Nakamichi" 1>"enwik9.NKMCH" C:\Compression_showdown>fc "enwik9" "enwik9.NKMCH" /b Comparing files enwik9 and ENWIK9.NKMCH FC: no differences encountered C:\Compression_showdown>if exist "enwik9.NKMCH" del "enwik9.NKMCH" C:\Compression_showdown>"Nakamichi_DD_LITE_GCC_64bit_plus-16GB-buffer.exe" enwik9.Nakamichi /bench SMMi :MM2 0MMMMM: rMMMMMa ZMMM. 0Z :MMM7 7B rMM@ MMM 7 MMM MMM. BMMMa XZ :MM: MMM XMMX XMSANMAYCE; rMMMM rMM; MM@ WM0 2MMMMM ZMMMM MMW BMM : MMMB aMMMMW 2MM :MM8 MM7 8MMi MMMZ XMMM MMa .WMMM0 @MM rMMMMi MMMMa MM, aMM XMMB MM: MMM2 @MXMMM7 0MM MMS WMM MM MMM, MM7 7MM MM: MMX ;MMM iMM MMM8 . MM8 MMS MMMZ . iMBi :MMMX @MMa WB: XMMZ2aX ZMM MMM2 :M2 :MM ZMMM BM8 ;MMMMMaaMM ,MM XMM@MM rMMMMM2aMM XMMMM SMMSMM. MMXMMM MMMM0 aMMMa@MMr MMS ;MM;MM8 XMMMM. MMr MMM0 MM 2MMW MMM 0MM ZMS 2MM. aMMB MMM BM aMM 8Ma MM iMM BMM M7 MMZ ;MMX MMi ;MM ZMW MM0 rM 7MM ;MM MMM7 iMM ZMM2 0MMa MMX 8M: MMM ZMMS BMM2 7 MM8 ZMr :MM rMB MM@ iMM MMZ iMM WMW BM2 MMX @MM MMi ;MMM 0MS 2MM7 aMMM ;MM .8Mi MMM aMM7 ZMMM MM ;Mr XMM iM8 MMa 0MM BMM MMM MM: BMX .MM MM7 aMM BMMM MM ;MMX ZMMMZ BM@ MMM . rMM7 ZMMMZ :MM:MX 0MZ M8 XMM, MMX MM. ;W: rMM aMX ZMM 7MM MM MMMi MM MMZ 8M:MM MM; MMM, MMa 0M:MM 8MWM8 MMZM0 MMM rMM 0MM WMa,M2 MMS MM8 @M0 :MMMXMa MMM WM XMB ; :MM;M8MMM MMM @M XM0 ; MMM@ MMMW MMrZi MMS MMi MMZM0 rMM Mi MM : ;MM BMMMM: rMM MM MMSMa ZMMM@ iMMr 7MM MM MMXM2 :MMM ZMMM 8MM@@ MMZ0 7MM , :MMMM MMBMa 0MWBr ,MMMM MMMM MMS MM MMMZ MMMM MMM MMS .MM MMMZ @MM. MMM MMMM WMM@ 0MM 2M ZMMM :MMMa MMM7 iMMMMMM@ XMMM .MM XMM aMMM MMM: MMW .MM XM@ ZMMM MM0 aMMX 8MMM .MMM WMM ZMr MMMX MMM8 0MMS SMMMMi MMM, rMM,MM2 MMM: SMMM ;MM2 ;MM,MMS MMM, MMM MMM MMM. MMM .MM MMZ MMM XMMM rMMB :MMM7 MM@ BMMM aMM ;MMX aMMX 8MMM ZMM MM8 .MMX MMM2 iMMX ,MMMMMMZ .@; MMM WMM . .: : 0MMZ .2Z :7r. ;i 0MMMaSMi aMMMM7 .WW Nakamichi 'Ryuugan-ditto-1TB', written by Kaze, inspired by Haruhiko Okumura sharing, based on Nobuo Ito's LZSS source, muffinesque tips by m^2, Jim Dempsey and Kirill Kryukov enforced. This latest (2020-Nov-20) compile has B-trees only matchfinder - called 'LITE'; Downloadable at: https://twitter.com/Sanmayce Nakamichi 'Kaidanji' SUPRALITE highlights: - Back to supersimplisticism - 'Kaidanji' returns refined. Simply, ripping only orders 4..16 and then quickly trying to expand the match in the found position. Nakamichi 'Ryuugan-ditto-1TB'/'Dragoneye' LITE highlights: - The Zennish LZSS Microdeduplicator, a texttoy (file-to-file [de]compressor) written in plain C, 100% FREE - licenseless it is; - Single-threaded Non-SIMD console tool written entirely by yours truly Sanmayce; - The first Lempel-Ziv-Storer-Szymanski (LZSS) compressor using 1TB sliding window; - It targets superfast decompression of huge/tera corpora of textual data; - LITE means no memmem() (Railgun) invocations are to be made; - Compressing at Linear Rate due to Matchfinding based on B-trees only; - The Leprechaunesque (Internal/External) B-trees order 3 (2 keys MAX) are brutally-optimized; - Ability to deduplicate (as little as) 64 bytes long chunks 1TB backwards; - Ability to deduplicate 256[+] bytes long chunks 1TB backwards - when SHA3 enabled; - CHUNKABLE! Since there are no headers/metadata in .Nakamichi files, arbitrary chunks/blocks can be compressed-n-concatenated; - SCALABLE! Gets faster when more Physical or/and External RAM is available; - Dynamical (as command line parameters) selection of HASH & B-trees memory blocks; - AUTO-RESETTING PASSES (when TargetBuffer is filled then automaticaly restart building by doubling the passes); - DEFRAGMENTED B-trees are in use during compression stage, their leaves contain only BBs appearing 2[+] times; - Compileable under Windows and Linux, most suitable to run on 3TB machines; - Many thanks go to: Haruhiko Okumura, Nobuo Ito, Hamid Buzidi, Landon Noll, Aleksey Vaneev (https://github.com/avaneev/prvhash). Sanmayce, Nakamichi's author, here, some more notes: - The B-trees form the second layer, the first being HASH table handled by FNV1A-Pippip - the fastest lookup-hash; - Hashpot a.k.a. hashpools (residing in Physical RAM) could be tuned via command line parameter, thus lessening the B-trees heights; - The building of B-trees is done dynamically-n-automatically in PASSES, thus LOCALITY/LOCALIZATION leads to cache-friendliness, for example, instead of confusing/blinding the SSD controller with building 2^27 ~= 128M B-trees at a time, 'PASSES' revision lowers the "noise/mayhem" 128 times by processing e.g. (20bit) 1M B-trees at a time; - For so long, the main goal was to throw it in battle, thus to forge one battle-ready *nix/windows tool with superb hash/search/decompress capabilities. - Straight up, a wish of mine is to have the latest English Wikipedia XML dump decompressed by Nakamichi into physical RAM and then followed by Kazahana's superfast exact/wildcard/fuzzy etudes. - Through the years, such a transparent decompressor (load booster) has been losing momentum and kinda became unnecessary, but not for me, when HDDs had 40MB/s sequential read speeds it was ... boosty, now with SSDs reading at 1700+MB/s, not so smacky, yet uploading (on 128GB machines, 20GB+72GB needed) the whole Wikipedia ~20GB (instead of 72GB) is better, sizewise. - Currently, having 16 (4,6,8,10,12,14,16,18,36,64,24,28,48,56,128,256) MatchLens being hashed (plus 1 being phantomic/temporary), 29bit hashtable results in: (16+1)*8*(2^29) = (16+1)*8*(1<<29) = 69632MB hashpot (the cluster of all hashpools), each slot houses a QWORD (being an offset/address of a B-tree root). - In near future, wanna see a machine (256GB RAM) crunching the 77,265,813,300 bytes file with this command line (it requires 2x72GB + 68GB + 2930GB = 212GB physical RAM + 2.9TB SSD RAM): D:\>timer64 Nakamichi.exe enwiki-20200420-pages-articles.xml enwiki-20200420-pages-articles.xml.Nakamichi 29 3000000 e - In not so far future, wanna see a machine (4096GB RAM) crunching the 72GB file with this command line (it requires 2x72GB + 272GB + 2930GB = 3.3TB physical RAM): D:\>timer64 Nakamichi.exe enwiki-20200420-pages-articles.xml enwiki-20200420-pages-articles.xml.Nakamichi 31 3000000 i Q: - HOW TO BOOST [BUILDING/COMPRESSION] SPEED? A: - Now, the name of the game is "RAM in spades", if we have a lot then speed is going to jump a lot. How? - First, by increasing the hash-pool - 20 (number of bits) was used in 'Taishukan' resulting in 2^20 =~ 1M hash-slots a.k.a. B-trees' roots per keysize, naturally, the bigger the faster since B-trees will have less height and will be more - aiming to reach the ideal case - having only hash-slots with no actual B-trees! Back to 'Taishukan', the testfile is ~77MB, it suggests using 128M slots to lower height/branching of B-trees - 2^27 leads to using 27. Feartime, going from 20 to 27: 8*(15+1)*2^(20) = 134,217,793 bytes; with the speed-up hash needs 8*(15+1)*2^(27) = 17,179,869,184 bytes, yes 16GB. - Second, by lowering passes in building the B-trees - the trick is to increase the number in compile-time parameter -DSpeedUpBuilding=32 - (didn't make it as an execute-time parameter, lazy). If we look at the 'Taishukan' console log it tells us that (Target-Buffer 106 MB) x 64 passes = 6784MB are needed for building the B-trees in ONE PASS ONLY, so compile with -DSpeedUpBuilding=6784 instead. Enfun! Current priority class is HIGH_PRIORITY_CLASS. Allocating Source-Buffer 266 MB ... Allocating Target-Buffer 5,120 MB ... Decompressing 279,301,006 bytes (being the compressed stream) ... Warming up ... RAM-to-RAM performance: 805 MB/s; 805 MB/s; 804 MB/s; 809 MB/s; 784 MB/s; 805 MB/s; 799 MB/s; 789 MB/s; 809 MB/s; 798 MB/s; 797 MB/s; 790 MB/s; 795 MB/s; 792 MB/s; 791 MB/s; 792 MB/s Enforcing 17 seconds idling to avoid throttling ... 787 MB/s; 800 MB/s; 785 MB/s; 799 MB/s; 784 MB/s; 799 MB/s; 779 MB/s; 778 MB/s; 806 MB/s; 788 MB/s; 797 MB/s; 786 MB/s; 775 MB/s; 800 MB/s; 773 MB/s; 794 MB/s Enforcing 17 seconds idling to avoid throttling ... 786 MB/s; 801 MB/s; 786 MB/s; 778 MB/s; 799 MB/s; 774 MB/s; 792 MB/s; 787 MB/s; 772 MB/s; 800 MB/s; 779 MB/s; 787 MB/s; 788 MB/s; 769 MB/s; 799 MB/s; 784 MB/s Enforcing 17 seconds idling to avoid throttling ... 790 MB/s; 789 MB/s; 770 MB/s; 800 MB/s; 781 MB/s; 785 MB/s; 794 MB/s; 771 MB/s; 766 MB/s; 799 MB/s; 789 MB/s; 797 MB/s; 771 MB/s; 765 MB/s; 811 MB/s; 779 MB/s Enforcing 17 seconds idling to avoid throttling ... 735 MB/s; 760 MB/s; 798 MB/s; 778 MB/s; 750 MB/s; 742 MB/s; 806 MB/s; 821 MB/s; 793 MB/s; 761 MB/s; 739 MB/s; 753 MB/s; 792 MB/s; 770 MB/s; 749 MB/s; 779 MB/s Enforcing 17 seconds idling to avoid throttling ... 784 MB/s; 763 MB/s; 787 MB/s; 784 MB/s; 763 MB/s; 776 MB/s; 791 MB/s; 769 MB/s; 768 MB/s; 797 MB/s; 773 MB/s; 752 MB/s; 769 MB/s; 789 MB/s; 768 MB/s; 773 MB/s Enforcing 17 seconds idling to avoid throttling ... 790 MB/s; 768 MB/s; 768 MB/s; 794 MB/s; 770 MB/s; 761 MB/s; 800 MB/s; 778 MB/s; 755 MB/s; 776 MB/s; 781 MB/s; 782 MB/s; 793 MB/s; 769 MB/s; 766 MB/s; 794 MB/s Enforcing 17 seconds idling to avoid throttling ... 769 MB/s; 796 MB/s; 777 MB/s; 759 MB/s; 791 MB/s; 783 MB/s; 765 MB/s; 779 MB/s; 790 MB/s; 769 MB/s; 800 MB/s; 780 MB/s; 757 MB/s; 792 MB/s; 791 MB/s; 762 MB/s Enforcing 17 seconds idling to avoid throttling ... RAM-to-RAM (peak) performance: 821 MB/s. Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x5b09,d619 Target-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x209a,2b59 C:\Compression_showdown>call _BENCHMARK_lzbench_turbobench.BAT Silesia_compression_corpus C:\Compression_showdown>timer64 lz4_win64_v1_9_3.exe -b1 -e12 --fast=12 -i9 --no-frame-crc "Silesia_compression_corpus" Benchmarking levels from -12 to 12 -12#mpression_corpus : 211938580 -> 126538668 (1.675), 893.2 MB/s ,3776.1 MB/s -11#mpression_corpus : 211938580 -> 124219861 (1.706), 795.8 MB/s ,3329.5 MB/s -10#mpression_corpus : 211938580 -> 123513170 (1.716), 792.7 MB/s ,3211.2 MB/s -9#ompression_corpus : 211938580 -> 121656345 (1.742), 838.3 MB/s ,3457.3 MB/s -8#ompression_corpus : 211938580 -> 120131525 (1.764), 691.2 MB/s ,3155.5 MB/s -7#ompression_corpus : 211938580 -> 117832751 (1.799), 685.8 MB/s ,3055.6 MB/s -6#ompression_corpus : 211938580 -> 116990771 (1.812), 641.0 MB/s ,2933.9 MB/s -5#ompression_corpus : 211938580 -> 114398245 (1.853), 632.9 MB/s ,3248.3 MB/s -4#ompression_corpus : 211938580 -> 112313597 (1.887), 615.5 MB/s ,3281.4 MB/s -3#ompression_corpus : 211938580 -> 109034990 (1.944), 572.4 MB/s ,3206.6 MB/s -2#ompression_corpus : 211938580 -> 107062794 (1.980), 571.5 MB/s ,3034.0 MB/s -1#ompression_corpus : 211938580 -> 104074930 (2.036), 585.0 MB/s ,3162.8 MB/s 0#ompression_corpus : 211938580 -> 100880026 (2.101), 550.9 MB/s ,3266.6 MB/s 1#ompression_corpus : 211938580 -> 100880026 (2.101), 525.2 MB/s ,2979.9 MB/s 2#ompression_corpus : 211938580 -> 100880026 (2.101), 511.5 MB/s ,2964.8 MB/s 3#ompression_corpus : 211938580 -> 81341717 (2.606), 81.2 MB/s ,2794.3 MB/s 4#ompression_corpus : 211938580 -> 79807213 (2.656), 64.6 MB/s ,2808.1 MB/s 5#ompression_corpus : 211938580 -> 78894641 (2.686), 53.8 MB/s ,2845.0 MB/s 6#ompression_corpus : 211938580 -> 78385020 (2.704), 45.6 MB/s ,2864.3 MB/s 7#ompression_corpus : 211938580 -> 78101765 (2.714), 38.3 MB/s ,2953.6 MB/s 8#ompression_corpus : 211938580 -> 77956803 (2.719), 32.4 MB/s ,2971.9 MB/s 9#ompression_corpus : 211938580 -> 77883787 (2.721), 28.5 MB/s ,3341.4 MB/s 10#ompression_corpus : 211938580 -> 77596107 (2.731), 20.6 MB/s ,2897.3 MB/s 11#ompression_corpus : 211938580 -> 77314823 (2.741), 12.1 MB/s ,3096.1 MB/s 12#ompression_corpus : 211938580 -> 77261967 (2.743), 10.1 MB/s ,3219.4 MB/s Kernel Time = 1.296 = 0% User Time = 528.734 = 97% Process Time = 530.031 = 97% Virtual Memory = 706 MB Global Time = 541.071 = 100% Physical Memory = 611 MB C:\Compression_showdown>timer64 zstd-v1.4.5-win64.exe -b1 -e22 --fast=22 --threads=1 --priority=rt --no-check -i9 "Silesia_compression_corpus" Note : switching to real-time priority -22#mpression_corpus : 211938580 -> 121818728 (1.740), 736.1 MB/s ,1845.7 MB/s Note : switching to real-time priority -21#mpression_corpus : 211938580 -> 120381965 (1.761), 688.5 MB/s ,1792.7 MB/s Note : switching to real-time priority -20#mpression_corpus : 211938580 -> 119065809 (1.780), 720.6 MB/s ,1757.3 MB/s Note : switching to real-time priority -19#mpression_corpus : 211938580 -> 117834573 (1.799), 657.0 MB/s ,1731.1 MB/s Note : switching to real-time priority -18#mpression_corpus : 211938580 -> 116994481 (1.812), 668.4 MB/s ,1722.5 MB/s Note : switching to real-time priority -17#mpression_corpus : 211938580 -> 116172251 (1.824), 704.9 MB/s ,1705.6 MB/s Note : switching to real-time priority -16#mpression_corpus : 211938580 -> 114963422 (1.844), 628.6 MB/s ,1672.5 MB/s Note : switching to real-time priority -15#mpression_corpus : 211938580 -> 113676607 (1.864), 640.0 MB/s ,1793.9 MB/s Note : switching to real-time priority -14#mpression_corpus : 211938580 -> 112731326 (1.880), 615.3 MB/s ,1700.1 MB/s Note : switching to real-time priority -13#mpression_corpus : 211938580 -> 111299312 (1.904), 575.6 MB/s ,1670.0 MB/s Note : switching to real-time priority -12#mpression_corpus : 211938580 -> 109676919 (1.932), 558.3 MB/s ,1729.8 MB/s Note : switching to real-time priority -11#mpression_corpus : 211938580 -> 108709084 (1.950), 559.3 MB/s ,1690.8 MB/s Note : switching to real-time priority -10#mpression_corpus : 211938580 -> 107004814 (1.981), 551.8 MB/s ,1757.0 MB/s Note : switching to real-time priority -9#ompression_corpus : 211938580 -> 104833303 (2.022), 530.9 MB/s ,1571.5 MB/s Note : switching to real-time priority -8#ompression_corpus : 211938580 -> 103087718 (2.056), 517.1 MB/s ,1550.8 MB/s Note : switching to real-time priority -7#ompression_corpus : 211938580 -> 101114991 (2.096), 539.0 MB/s ,1564.7 MB/s Note : switching to real-time priority -6#ompression_corpus : 211938580 -> 99607924 (2.128), 504.1 MB/s ,1541.4 MB/s Note : switching to real-time priority -5#ompression_corpus : 211938580 -> 97341468 (2.177), 472.5 MB/s ,1500.6 MB/s Note : switching to real-time priority -4#ompression_corpus : 211938580 -> 94392496 (2.245), 458.4 MB/s ,1499.8 MB/s Note : switching to real-time priority -3#ompression_corpus : 211938580 -> 91647200 (2.313), 447.1 MB/s ,1452.7 MB/s Note : switching to real-time priority -2#ompression_corpus : 211938580 -> 89428124 (2.370), 413.4 MB/s ,1408.5 MB/s Note : switching to real-time priority -1#ompression_corpus : 211938580 -> 87084097 (2.434), 406.2 MB/s ,1465.9 MB/s Note : switching to real-time priority 0#ompression_corpus : 211938580 -> 66681881 (3.178), 208.1 MB/s , 930.0 MB/s Note : switching to real-time priority 1#ompression_corpus : 211938580 -> 73521045 (2.883), 356.4 MB/s ,1097.4 MB/s Note : switching to real-time priority 2#ompression_corpus : 211938580 -> 69602633 (3.045), 272.3 MB/s , 940.4 MB/s Note : switching to real-time priority 3#ompression_corpus : 211938580 -> 66681881 (3.178), 213.5 MB/s , 922.8 MB/s Note : switching to real-time priority 4#ompression_corpus : 211938580 -> 65500408 (3.236), 178.1 MB/s , 976.4 MB/s Note : switching to real-time priority 5#ompression_corpus : 211938580 -> 63992444 (3.312), 106.8 MB/s , 930.6 MB/s Note : switching to real-time priority 6#ompression_corpus : 211938580 -> 62893475 (3.370), 86.0 MB/s , 945.9 MB/s Note : switching to real-time priority 7#ompression_corpus : 211938580 -> 61336888 (3.455), 61.6 MB/s ,1048.5 MB/s Note : switching to real-time priority 8#ompression_corpus : 211938580 -> 60756778 (3.488), 50.7 MB/s ,1076.0 MB/s Note : switching to real-time priority 9#ompression_corpus : 211938580 -> 60186629 (3.521), 36.5 MB/s ,1020.0 MB/s Note : switching to real-time priority 10#ompression_corpus : 211938580 -> 59533729 (3.560), 26.8 MB/s ,1119.9 MB/s Note : switching to real-time priority 11#ompression_corpus : 211938580 -> 59240913 (3.578), 21.0 MB/s ,1084.5 MB/s Note : switching to real-time priority 12#ompression_corpus : 211938580 -> 58775200 (3.606), 14.3 MB/s ,1028.9 MB/s Note : switching to real-time priority 13#ompression_corpus : 211938580 -> 58088767 (3.649), 9.68 MB/s ,1072.5 MB/s Note : switching to real-time priority 14#ompression_corpus : 211938580 -> 57578643 (3.681), 7.91 MB/s ,1123.8 MB/s Note : switching to real-time priority 15#ompression_corpus : 211938580 -> 57172176 (3.707), 6.08 MB/s ,1067.8 MB/s Note : switching to real-time priority 16#ompression_corpus : 211938580 -> 55684512 (3.806), 4.76 MB/s ,1048.0 MB/s Note : switching to real-time priority 17#ompression_corpus : 211938580 -> 54605847 (3.881), 3.75 MB/s , 994.3 MB/s Note : switching to real-time priority 18#ompression_corpus : 211938580 -> 53713049 (3.946), 3.03 MB/s , 964.9 MB/s Note : switching to real-time priority 19#ompression_corpus : 211938580 -> 53280719 (3.978), 2.52 MB/s , 904.7 MB/s Note : switching to real-time priority 20#ompression_corpus : 211938580 -> 52862634 (4.009), 2.37 MB/s , 815.3 MB/s Note : switching to real-time priority 21#ompression_corpus : 211938580 -> 52747874 (4.018), 1.59 MB/s , 700.3 MB/s Note : switching to real-time priority 22#ompression_corpus : 211938580 -> 52739486 (4.019), 1.52 MB/s , 695.4 MB/s Kernel Time = 5.875 = 0% User Time = 1518.765 = 99% Process Time = 1524.640 = 99% Virtual Memory = 1381 MB Global Time = 1531.268 = 100% Physical Memory = 1250 MB C:\Compression_showdown>lzbench18 -c4 -i1,15 -o3 -ebzip2,9/zstdLDM,22/brotli,1,9/crush,2/density,3/fastlz,2/gipfeli/lzo1b,999/libdeflate,1,12/lizard,19,29,39,49/lzf,1/lzfse/lzjb/lzrw,5/lzsse2,1,17/quicklz,3/snappy/xpack,1,9/yappy,99/zlib,1,9 "Silesia_compression_corpus" lzbench 1.8 (64-bit Windows) Assembled by P.Skibinski Compressor name Compress. Decompress. Orig. size Compr. size Ratio Filename memcpy 15469 MB/s 15590 MB/s 211938580 211938580 100.00 Silesia_compression_corpus bzip2 1.0.8 -9 8.63 MB/s 28 MB/s 211938580 54592577 25.76 Silesia_compression_corpus zstdLDM 1.4.3 -22 1.55 MB/s 515 MB/s 211938580 52826760 24.93 Silesia_compression_corpus brotli 2019-10-01 -1 153 MB/s 270 MB/s 211938580 73501169 34.68 Silesia_compression_corpus brotli 2019-10-01 -9 5.79 MB/s 350 MB/s 211938580 56697468 26.75 Silesia_compression_corpus crush 1.0 -2 0.55 MB/s 348 MB/s 211938580 63751130 30.08 Silesia_compression_corpus density 0.14.2 -3 306 MB/s 333 MB/s 211938580 88627492 41.82 Silesia_compression_corpus fastlz 0.1 -2 273 MB/s 507 MB/s 211938580 100905197 47.61 Silesia_compression_corpus gipfeli 2016-07-13 278 MB/s 548 MB/s 211938580 87932986 41.49 Silesia_compression_corpus lzo1b 2.10 -999 13 MB/s 695 MB/s 211938580 76593633 36.14 Silesia_compression_corpus libdeflate 1.3 -1 143 MB/s 657 MB/s 211938580 73317923 34.59 Silesia_compression_corpus libdeflate 1.3 -12 5.44 MB/s 697 MB/s 211938580 64802658 30.58 Silesia_compression_corpus lizard 1.0 -19 3.94 MB/s 3380 MB/s 211938580 77414280 36.53 Silesia_compression_corpus lizard 1.0 -29 1.72 MB/s 2159 MB/s 211938580 68694931 32.41 Silesia_compression_corpus lizard 1.0 -39 3.82 MB/s 2079 MB/s 211938580 69815764 32.94 Silesia_compression_corpus lizard 1.0 -49 1.62 MB/s 1390 MB/s 211938580 60694951 28.64 Silesia_compression_corpus lzf 3.6 -1 284 MB/s 613 MB/s 211938580 102040285 48.15 Silesia_compression_corpus lzfse 2017-03-08 59 MB/s 807 MB/s 211938580 67627385 31.91 Silesia_compression_corpus lzjb 2010 261 MB/s 421 MB/s 211938580 122670566 57.88 Silesia_compression_corpus lzrw 15-Jul-1991 -5 119 MB/s 465 MB/s 211938580 90799072 42.84 Silesia_compression_corpus lzsse2 2019-04-18 -1 17 MB/s 2964 MB/s 211938580 87974920 41.51 Silesia_compression_corpus lzsse2 2019-04-18 -17 2.94 MB/s 3261 MB/s 211938580 75684771 35.71 Silesia_compression_corpus quicklz 1.5.0 -3 47 MB/s 811 MB/s 211938580 81820424 38.61 Silesia_compression_corpus snappy 2019-09-30 381 MB/s 1235 MB/s 211938580 102138668 48.19 Silesia_compression_corpus xpack 2016-06-02 -1 132 MB/s 665 MB/s 211938580 71091029 33.54 Silesia_compression_corpus xpack 2016-06-02 -9 16 MB/s 867 MB/s 211938580 61247589 28.90 Silesia_compression_corpus yappy 2014-03-22 -99 67 MB/s 2203 MB/s 211938580 98671826 46.56 Silesia_compression_corpus zlib 1.2.11 -1 79 MB/s 281 MB/s 211938580 77259877 36.45 Silesia_compression_corpus zlib 1.2.11 -9 11 MB/s 311 MB/s 211938580 67643258 31.92 Silesia_compression_corpus done... (cIters=1 dIters=15 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB) The results sorted by column number 4: Compressor name Compress. Decompress. Orig. size Compr. size Ratio Filename zstdLDM 1.4.3 -22 1.55 MB/s 515 MB/s 211938580 52826760 24.93 Silesia_compression_corpus bzip2 1.0.8 -9 8.63 MB/s 28 MB/s 211938580 54592577 25.76 Silesia_compression_corpus brotli 2019-10-01 -9 5.79 MB/s 350 MB/s 211938580 56697468 26.75 Silesia_compression_corpus lizard 1.0 -49 1.62 MB/s 1390 MB/s 211938580 60694951 28.64 Silesia_compression_corpus xpack 2016-06-02 -9 16 MB/s 867 MB/s 211938580 61247589 28.90 Silesia_compression_corpus crush 1.0 -2 0.55 MB/s 348 MB/s 211938580 63751130 30.08 Silesia_compression_corpus libdeflate 1.3 -12 5.44 MB/s 697 MB/s 211938580 64802658 30.58 Silesia_compression_corpus lzfse 2017-03-08 59 MB/s 807 MB/s 211938580 67627385 31.91 Silesia_compression_corpus zlib 1.2.11 -9 11 MB/s 311 MB/s 211938580 67643258 31.92 Silesia_compression_corpus lizard 1.0 -29 1.72 MB/s 2159 MB/s 211938580 68694931 32.41 Silesia_compression_corpus lizard 1.0 -39 3.82 MB/s 2079 MB/s 211938580 69815764 32.94 Silesia_compression_corpus xpack 2016-06-02 -1 132 MB/s 665 MB/s 211938580 71091029 33.54 Silesia_compression_corpus libdeflate 1.3 -1 143 MB/s 657 MB/s 211938580 73317923 34.59 Silesia_compression_corpus brotli 2019-10-01 -1 153 MB/s 270 MB/s 211938580 73501169 34.68 Silesia_compression_corpus lzsse2 2019-04-18 -17 2.94 MB/s 3261 MB/s 211938580 75684771 35.71 Silesia_compression_corpus lzo1b 2.10 -999 13 MB/s 695 MB/s 211938580 76593633 36.14 Silesia_compression_corpus zlib 1.2.11 -1 79 MB/s 281 MB/s 211938580 77259877 36.45 Silesia_compression_corpus lizard 1.0 -19 3.94 MB/s 3380 MB/s 211938580 77414280 36.53 Silesia_compression_corpus quicklz 1.5.0 -3 47 MB/s 811 MB/s 211938580 81820424 38.61 Silesia_compression_corpus gipfeli 2016-07-13 278 MB/s 548 MB/s 211938580 87932986 41.49 Silesia_compression_corpus lzsse2 2019-04-18 -1 17 MB/s 2964 MB/s 211938580 87974920 41.51 Silesia_compression_corpus density 0.14.2 -3 306 MB/s 333 MB/s 211938580 88627492 41.82 Silesia_compression_corpus lzrw 15-Jul-1991 -5 119 MB/s 465 MB/s 211938580 90799072 42.84 Silesia_compression_corpus yappy 2014-03-22 -99 67 MB/s 2203 MB/s 211938580 98671826 46.56 Silesia_compression_corpus fastlz 0.1 -2 273 MB/s 507 MB/s 211938580 100905197 47.61 Silesia_compression_corpus lzf 3.6 -1 284 MB/s 613 MB/s 211938580 102040285 48.15 Silesia_compression_corpus snappy 2019-09-30 381 MB/s 1235 MB/s 211938580 102138668 48.19 Silesia_compression_corpus lzjb 2010 261 MB/s 421 MB/s 211938580 122670566 57.88 Silesia_compression_corpus C:\Compression_showdown>copy oo2core_8_win64.dll oo2core_6_win64.dll /y 1 file(s) copied. C:\Compression_showdown>date 12/04/2018 0"turbobench_v18.05_-_build_04_May_2018" "Silesia_compression_corpus" -esnappy_c/yappy/lzma,9d30:fb273:mf=bt4/oodle,89,91,95,99,111,115,119,129,139/lzturbo,19,12,10,29,22,20,39,32,30,59/brotli,1,3,5/brotli,11d30/zstd,22d30/lizard,11,19,21,29,31,39,41,49/trle -I3 -J31 -k1 -B2G C:\Compression_showdown>"turbobench_v18.05_-_build_04_May_2018" -p4 "Silesia_compression_corpus.tbb" `V? C:\Compression_showdown>call _Benchmark_ZPAQ_ZSTD_RAZOR_BROTLI_ZIP_BZIP2_7Z_BSC_BriefLZ_Nakamichi.BAT Silesia_compression_corpus C:\Compression_showdown>sha1sum "Silesia_compression_corpus" 96dddf0c357525c7739c444c6d786a19028ebb31 Silesia_compression_corpus C:\Compression_showdown>timer64 "zpaq_v7.05_x64.exe" add "Silesia_compression_corpus.method211.zpaq" "Silesia_compression_corpus" -method 211 -threads 1 zpaq v7.05 journaling archiver, compiled Apr 17 2015 Adding 211.938580 MB in 1 files -method 211 -threads 1 at 2020-12-05 12:01:14. 100.00% 0:00:00 + Silesia_compression_corpus 211938580 100.00% 0:00:00 [1..2752] 211949596 -method 211,160,0 1 +added, 0 -removed. 0.000000 + (211.938580 -> 211.938580 -> 63.151460) = 63.151460 MB 52.922 seconds (all OK) Kernel Time = 0.390 = 0% User Time = 52.515 = 99% Process Time = 52.906 = 99% Virtual Memory = 3090 MB Global Time = 52.934 = 100% Physical Memory = 1210 MB C:\Compression_showdown>timer64 "zpaq_v7.05_x64.exe" add "Silesia_compression_corpus.method511.zpaq" "Silesia_compression_corpus" -method 511 -threads 1 zpaq v7.05 journaling archiver, compiled Apr 17 2015 Adding 211.938580 MB in 1 files -method 511 -threads 1 at 2020-12-05 12:02:07. 100.00% 0:00:00 + Silesia_compression_corpus 211938580 100.00% 0:00:00 [1..2752] 211949596 -method 511,160,0 1 +added, 0 -removed. 0.000000 + (211.938580 -> 211.938580 -> 40.141591) = 40.141591 MB 533.766 seconds (all OK) Kernel Time = 0.765 = 0% User Time = 532.984 = 99% Process Time = 533.750 = 99% Virtual Memory = 4668 MB Global Time = 533.779 = 100% Physical Memory = 2704 MB C:\Compression_showdown>timer64 zstd-v1.4.5-win64.exe --single-thread --ultra -22 --zstd=wlog=31,clog=30,hlog=30,slog=26 "Silesia_compression_corpus" -o "Silesia_compression_corpus.L22W2GB.zst" Silesia_compression_corpus : 24.87% (211938580 => 52698602 bytes, Silesia_compression_corpus.L22W2GB.zst) Kernel Time = 0.750 = 0% User Time = 133.656 = 93% Process Time = 134.406 = 93% Virtual Memory = 4310 MB Global Time = 143.453 = 100% Physical Memory = 4304 MB C:\Compression_showdown>timer64 rz.exe a -d 1023M "Silesia_compression_corpus.1GB.rz" "Silesia_compression_corpus" *** RAZOR archiver 1.03.7 2018-03-22 *** *** (c) Christian Martelock christian.martelock@web.de *** *** This DEMO/TEST version is for non-commercial use only *** Scanning c:\compression_showdown\silesia_compression_corpus Found 0 dirs, 1 files, 211938580 bytes. Creating archive Silesia_compression_corpus.1GB.rz Window : 206972K (1024M..256G) Adding Silesia_compression_corpus Header : 60 Size : 43208651 Archive ok. Added 0 dirs, 1 files, 211938580 bytes. CPU time = 497.125s / wall time = 400.858s Kernel Time = 1.296 = 0% User Time = 497.125 = 124% Process Time = 498.421 = 124% Virtual Memory = 2521 MB Global Time = 400.871 = 100% Physical Memory = 2120 MB C:\Compression_showdown>timer64 brotli_gc82.exe -q 11 --large_window=30 "Silesia_compression_corpus" -o "Silesia_compression_corpus.1GB.bro" Kernel Time = 1.312 = 0% User Time = 502.453 = 99% Process Time = 503.765 = 99% Virtual Memory = 10352 MB Global Time = 503.829 = 100% Physical Memory = 1866 MB C:\Compression_showdown>timer64 "7za_x64_v1900.exe" a -tgzip -mmt1 -mx=9 "Silesia_compression_corpus.MX9.gzip" "Silesia_compression_corpus" 7-Zip (a) 19.00 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2019-02-21 Scanning the drive: 1 file, 211938580 bytes (203 MiB) Creating archive: Silesia_compression_corpus.MX9.gzip Add new data to archive: 1 file, 211938580 bytes (203 MiB) Files read from disk: 1 Archive size: 64709195 bytes (62 MiB) Everything is Ok Kernel Time = 0.062 = 0% User Time = 178.234 = 99% Process Time = 178.296 = 99% Virtual Memory = 6 MB Global Time = 178.530 = 100% Physical Memory = 9 MB C:\Compression_showdown>timer64 "7za_x64_v1900.exe" a -tbzip2 -mmt1 -mx=9 "Silesia_compression_corpus.MX9.bzip2" "Silesia_compression_corpus" 7-Zip (a) 19.00 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2019-02-21 Scanning the drive: 1 file, 211938580 bytes (203 MiB) Creating archive: Silesia_compression_corpus.MX9.bzip2 Add new data to archive: 1 file, 211938580 bytes (203 MiB) Files read from disk: 1 Archive size: 54217109 bytes (52 MiB) Everything is Ok Kernel Time = 0.062 = 0% User Time = 279.156 = 99% Process Time = 279.218 = 99% Virtual Memory = 13 MB Global Time = 279.320 = 100% Physical Memory = 16 MB C:\Compression_showdown>timer64 "7za_x64_v1900.exe" a -t7z -mmt1 -mx=9 -md=30 "Silesia_compression_corpus.MX9Dict1024.7z" "Silesia_compression_corpus" 7-Zip (a) 19.00 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2019-02-21 Scanning the drive: 1 file, 211938580 bytes (203 MiB) Creating archive: Silesia_compression_corpus.MX9Dict1024.7z Add new data to archive: 1 file, 211938580 bytes (203 MiB) Files read from disk: 1 Archive size: 48686560 bytes (47 MiB) Everything is Ok Kernel Time = 0.359 = 0% User Time = 103.781 = 99% Process Time = 104.140 = 99% Virtual Memory = 2696 MB Global Time = 104.289 = 100% Physical Memory = 2082 MB C:\Compression_showdown>timer64 "bsc_v3.1.0_x64.exe" e "Silesia_compression_corpus" "Silesia_compression_corpus.ST0Block1024.bsc" -m0 -b1024 -e2 -Tt This is bsc, Block Sorting Compressor. Version 3.1.0. 8 July 2012. Copyright (c) 2009-2012 Ilya Grebnov . Silesia_compression_corpus compressed 211938580 into 48589384 in 24.156 seconds. Kernel Time = 0.375 = 1% User Time = 23.593 = 97% Process Time = 23.968 = 99% Virtual Memory = 966 MB Global Time = 24.180 = 100% Physical Memory = 960 MB C:\Compression_showdown>timer64 BriefLZ_130_Intel_v19_64bit.exe --optimal -b2g "Silesia_compression_corpus" "Silesia_compression_corpus.2GB.blz" Kernel Time = 0.828 = 0% User Time = 228.671 = 99% Process Time = 229.500 = 99% Virtual Memory = 45403 MB Global Time = 229.574 = 100% Physical Memory = 6155 MB C:\Compression_showdown>dir "Silesia_compression_corpus.*"/os Volume in drive C is System Volume Serial Number is 1E2A-3E30 Directory of C:\Compression_showdown 12/05/2020 14:11 40,141,591 Silesia_compression_corpus.method511.zpaq 12/05/2020 14:20 43,208,651 Silesia_compression_corpus.1GB.rz 12/05/2020 14:38 48,589,384 Silesia_compression_corpus.ST0Block1024.bsc 12/05/2020 14:37 48,686,560 Silesia_compression_corpus.MX9Dict1024.7z 11/18/2020 17:59 49,538,041 Silesia_compression_corpus.1GB.bro 11/18/2020 17:59 52,698,602 Silesia_compression_corpus.L22W2GB.zst 12/05/2020 14:36 54,217,109 Silesia_compression_corpus.MX9.bzip2 12/05/2020 14:02 63,151,460 Silesia_compression_corpus.method211.zpaq 12/05/2020 14:42 63,837,817 Silesia_compression_corpus.2GB.blz 12/05/2020 14:31 64,709,195 Silesia_compression_corpus.MX9.gzip 12/05/2020 00:49 72,621,718 Silesia_compression_corpus.Nakamichi 11/18/2020 17:59 211,938,580 Silesia_compression_corpus 12 File(s) 813,338,708 bytes 0 Dir(s) 1,961,781,850,112 bytes free C:\Compression_showdown>fv "Silesia_compression_corpus" Reading Silesia_compression_corpus Drawing fv.bmp 512 by 256 from Silesia_compression_corpus (211938580 bytes) Drew part 1 of 4 in 21.91 seconds Drew part 2 of 4 in 21.99 seconds Drew part 3 of 4 in 25.01 seconds Drew part 4 of 4 in 30.01 seconds Created fv.bmp in 99.06 seconds C:\Compression_showdown>copy fv.bmp "fv_Silesia_compression_corpus.bmp" /y 1 file(s) copied. C:\Compression_showdown>del fv.bmp C:\Compression_showdown>call _BENCHMARK_lzbench_turbobench.BAT enwik8 C:\Compression_showdown>timer64 lz4_win64_v1_9_3.exe -b1 -e12 --fast=12 -i9 --no-frame-crc "enwik8" Benchmarking levels from -12 to 12 -12#nwik8 : 100000000 -> 82506211 (1.212), 916.1 MB/s ,3445.7 MB/s -11#nwik8 : 100000000 -> 81196095 (1.232), 739.1 MB/s ,2982.6 MB/s -10#nwik8 : 100000000 -> 79937137 (1.251), 709.0 MB/s ,2996.1 MB/s -9#enwik8 : 100000000 -> 78612067 (1.272), 664.0 MB/s ,2692.4 MB/s -8#enwik8 : 100000000 -> 77175896 (1.296), 648.6 MB/s ,2895.8 MB/s -7#enwik8 : 100000000 -> 75651524 (1.322), 667.1 MB/s ,2490.2 MB/s -6#enwik8 : 100000000 -> 74063491 (1.350), 644.6 MB/s ,2991.1 MB/s -5#enwik8 : 100000000 -> 71794202 (1.393), 595.7 MB/s ,2757.7 MB/s -4#enwik8 : 100000000 -> 69439249 (1.440), 495.9 MB/s ,2583.2 MB/s -3#enwik8 : 100000000 -> 66450406 (1.505), 473.1 MB/s ,2659.7 MB/s -2#enwik8 : 100000000 -> 63557747 (1.573), 423.2 MB/s ,2804.1 MB/s -1#enwik8 : 100000000 -> 60533185 (1.652), 400.2 MB/s ,2834.7 MB/s 0#enwik8 : 100000000 -> 57262281 (1.746), 406.4 MB/s ,3168.2 MB/s 1#enwik8 : 100000000 -> 57262281 (1.746), 372.7 MB/s ,2802.1 MB/s 2#enwik8 : 100000000 -> 57262281 (1.746), 399.1 MB/s ,2924.3 MB/s 3#enwik8 : 100000000 -> 43829399 (2.282), 70.1 MB/s ,2663.6 MB/s 4#enwik8 : 100000000 -> 42966510 (2.327), 55.6 MB/s ,2554.2 MB/s 5#enwik8 : 100000000 -> 42528579 (2.351), 45.8 MB/s ,2859.2 MB/s 6#enwik8 : 100000000 -> 42333109 (2.362), 38.5 MB/s ,2614.6 MB/s 7#enwik8 : 100000000 -> 42252537 (2.367), 33.8 MB/s ,2864.1 MB/s 8#enwik8 : 100000000 -> 42219435 (2.369), 31.6 MB/s ,2915.1 MB/s 9#enwik8 : 100000000 -> 42203094 (2.369), 29.6 MB/s ,2690.8 MB/s 10#enwik8 : 100000000 -> 41977797 (2.382), 19.3 MB/s ,2855.2 MB/s 11#enwik8 : 100000000 -> 41915482 (2.386), 16.5 MB/s ,2684.3 MB/s 12#enwik8 : 100000000 -> 41913140 (2.386), 13.8 MB/s ,2703.1 MB/s Kernel Time = 0.687 = 0% User Time = 496.421 = 97% Process Time = 497.109 = 97% Virtual Memory = 386 MB Global Time = 507.498 = 100% Physical Memory = 290 MB C:\Compression_showdown>timer64 zstd-v1.4.5-win64.exe -b1 -e22 --fast=22 --threads=1 --priority=rt --no-check -i9 "enwik8" Note : switching to real-time priority -22#nwik8 : 100000000 -> 83495881 (1.198), 815.7 MB/s ,2171.5 MB/s Note : switching to real-time priority -21#nwik8 : 100000000 -> 82853362 (1.207), 741.7 MB/s ,2014.3 MB/s Note : switching to real-time priority -20#nwik8 : 100000000 -> 82167574 (1.217), 690.2 MB/s ,1940.4 MB/s Note : switching to real-time priority -19#nwik8 : 100000000 -> 81415441 (1.228), 721.0 MB/s ,2036.9 MB/s Note : switching to real-time priority -18#nwik8 : 100000000 -> 80657825 (1.240), 647.4 MB/s ,1828.0 MB/s Note : switching to real-time priority -17#nwik8 : 100000000 -> 79779573 (1.253), 664.0 MB/s ,1860.0 MB/s Note : switching to real-time priority -16#nwik8 : 100000000 -> 78934674 (1.267), 614.7 MB/s ,1750.5 MB/s Note : switching to real-time priority -15#nwik8 : 100000000 -> 77948566 (1.283), 585.1 MB/s ,1766.6 MB/s Note : switching to real-time priority -14#nwik8 : 100000000 -> 76874300 (1.301), 584.5 MB/s ,1710.2 MB/s Note : switching to real-time priority -13#nwik8 : 100000000 -> 75714782 (1.321), 565.2 MB/s ,1830.3 MB/s Note : switching to real-time priority -12#nwik8 : 100000000 -> 74482449 (1.343), 509.4 MB/s ,1638.9 MB/s Note : switching to real-time priority -11#nwik8 : 100000000 -> 73121840 (1.368), 494.4 MB/s ,1569.3 MB/s Note : switching to real-time priority -10#nwik8 : 100000000 -> 71646196 (1.396), 461.7 MB/s ,1503.1 MB/s Note : switching to real-time priority -9#enwik8 : 100000000 -> 70107506 (1.426), 474.4 MB/s ,1554.1 MB/s Note : switching to real-time priority -8#enwik8 : 100000000 -> 68471148 (1.460), 422.5 MB/s ,1554.9 MB/s Note : switching to real-time priority -7#enwik8 : 100000000 -> 66755729 (1.498), 432.6 MB/s ,1423.1 MB/s Note : switching to real-time priority -6#enwik8 : 100000000 -> 64748048 (1.544), 398.1 MB/s ,1565.9 MB/s Note : switching to real-time priority -5#enwik8 : 100000000 -> 62235883 (1.607), 374.2 MB/s ,1415.6 MB/s Note : switching to real-time priority -4#enwik8 : 100000000 -> 59317244 (1.686), 351.2 MB/s ,1386.2 MB/s Note : switching to real-time priority -3#enwik8 : 100000000 -> 56413723 (1.773), 323.8 MB/s ,1292.5 MB/s Note : switching to real-time priority -2#enwik8 : 100000000 -> 53668345 (1.863), 293.5 MB/s ,1271.0 MB/s Note : switching to real-time priority -1#enwik8 : 100000000 -> 51714451 (1.934), 281.2 MB/s ,1230.0 MB/s Note : switching to real-time priority 0#enwik8 : 100000000 -> 35602698 (2.809), 150.5 MB/s , 863.2 MB/s Note : switching to real-time priority 1#enwik8 : 100000000 -> 40738526 (2.455), 248.8 MB/s , 932.0 MB/s Note : switching to real-time priority 2#enwik8 : 100000000 -> 37434671 (2.671), 187.2 MB/s , 899.0 MB/s Note : switching to real-time priority 3#enwik8 : 100000000 -> 35602698 (2.809), 149.0 MB/s , 854.1 MB/s Note : switching to real-time priority 4#enwik8 : 100000000 -> 34920953 (2.864), 131.9 MB/s , 834.0 MB/s Note : switching to real-time priority 5#enwik8 : 100000000 -> 34315441 (2.914), 96.6 MB/s , 788.5 MB/s Note : switching to real-time priority 6#enwik8 : 100000000 -> 33591155 (2.977), 75.5 MB/s , 826.9 MB/s Note : switching to real-time priority 7#enwik8 : 100000000 -> 32443027 (3.082), 47.7 MB/s , 885.6 MB/s Note : switching to real-time priority 8#enwik8 : 100000000 -> 32029317 (3.122), 42.0 MB/s , 992.2 MB/s Note : switching to real-time priority 9#enwik8 : 100000000 -> 31751130 (3.149), 26.9 MB/s , 927.2 MB/s Note : switching to real-time priority 10#enwik8 : 100000000 -> 31238385 (3.201), 18.6 MB/s , 869.7 MB/s Note : switching to real-time priority 11#enwik8 : 100000000 -> 30968326 (3.229), 13.1 MB/s , 877.9 MB/s Note : switching to real-time priority 12#enwik8 : 100000000 -> 30728213 (3.254), 7.88 MB/s , 917.1 MB/s Note : switching to real-time priority 13#enwik8 : 100000000 -> 30331609 (3.297), 6.35 MB/s , 909.3 MB/s Note : switching to real-time priority 14#enwik8 : 100000000 -> 29794982 (3.356), 4.59 MB/s , 888.1 MB/s Note : switching to real-time priority 15#enwik8 : 100000000 -> 29436073 (3.397), 3.31 MB/s , 906.5 MB/s Note : switching to real-time priority 16#enwik8 : 100000000 -> 28446554 (3.515), 3.30 MB/s , 936.2 MB/s Note : switching to real-time priority 17#enwik8 : 100000000 -> 27702787 (3.610), 2.34 MB/s , 840.1 MB/s Note : switching to real-time priority 18#enwik8 : 100000000 -> 27326331 (3.659), 2.01 MB/s , 821.8 MB/s Note : switching to real-time priority 19#enwik8 : 100000000 -> 26957405 (3.710), 1.77 MB/s , 701.8 MB/s Note : switching to real-time priority 20#enwik8 : 100000000 -> 25989368 (3.848), 1.47 MB/s , 598.2 MB/s Note : switching to real-time priority 21#enwik8 : 100000000 -> 25541097 (3.915), 1.29 MB/s , 615.2 MB/s Note : switching to real-time priority 22#enwik8 : 100000000 -> 25340452 (3.946), 1.18 MB/s , 620.1 MB/s Kernel Time = 3.343 = 0% User Time = 1269.796 = 99% Process Time = 1273.140 = 99% Virtual Memory = 1027 MB Global Time = 1280.236 = 100% Physical Memory = 932 MB C:\Compression_showdown>lzbench18 -c4 -i1,15 -o3 -ebzip2,9/zstdLDM,22/brotli,1,9/crush,2/density,3/fastlz,2/gipfeli/lzo1b,999/libdeflate,1,12/lizard,19,29,39,49/lzf,1/lzfse/lzjb/lzrw,5/lzsse2,1,17/quicklz,3/snappy/xpack,1,9/yappy,99/zlib,1,9 "enwik8" lzbench 1.8 (64-bit Windows) Assembled by P.Skibinski Compressor name Compress. Decompress. Orig. size Compr. size Ratio Filename memcpy 17589 MB/s 17582 MB/s 100000000 100000000 100.00 enwik8 bzip2 1.0.8 -9 12 MB/s 30 MB/s 100000000 29008758 29.01 enwik8 zstdLDM 1.4.3 -22 1.16 MB/s 613 MB/s 100000000 25345418 25.35 enwik8 brotli 2019-10-01 -1 137 MB/s 292 MB/s 100000000 39123256 39.12 enwik8 brotli 2019-10-01 -9 5.47 MB/s 408 MB/s 100000000 29468806 29.47 enwik8 crush 1.0 -2 0.52 MB/s 279 MB/s 100000000 31699549 31.70 enwik8 density 0.14.2 -3 298 MB/s 310 MB/s 100000000 47817692 47.82 enwik8 fastlz 0.1 -2 199 MB/s 354 MB/s 100000000 54163013 54.16 enwik8 gipfeli 2016-07-13 219 MB/s 325 MB/s 100000000 45873643 45.87 enwik8 lzo1b 2.10 -999 15 MB/s 444 MB/s 100000000 40814011 40.81 enwik8 libdeflate 1.3 -1 136 MB/s 592 MB/s 100000000 39597378 39.60 enwik8 libdeflate 1.3 -12 6.31 MB/s 591 MB/s 100000000 35100568 35.10 enwik8 lizard 1.0 -19 6.87 MB/s 3212 MB/s 100000000 41953190 41.95 enwik8 lizard 1.0 -29 1.53 MB/s 1504 MB/s 100000000 36959135 36.96 enwik8 lizard 1.0 -39 6.72 MB/s 1985 MB/s 100000000 37505492 37.51 enwik8 lizard 1.0 -49 1.47 MB/s 914 MB/s 100000000 31688480 31.69 enwik8 lzf 3.6 -1 212 MB/s 493 MB/s 100000000 53945381 53.95 enwik8 lzfse 2017-03-08 48 MB/s 713 MB/s 100000000 36157828 36.16 enwik8 lzjb 2010 216 MB/s 317 MB/s 100000000 68711273 68.71 enwik8 lzrw 15-Jul-1991 -5 98 MB/s 375 MB/s 100000000 47874203 47.87 enwik8 lzsse2 2019-04-18 -1 15 MB/s 2890 MB/s 100000000 45854485 45.85 enwik8 lzsse2 2019-04-18 -17 7.43 MB/s 3308 MB/s 100000000 38063182 38.06 enwik8 quicklz 1.5.0 -3 35 MB/s 567 MB/s 100000000 44789793 44.79 enwik8 snappy 2019-09-30 246 MB/s 905 MB/s 100000000 56555035 56.56 enwik8 xpack 2016-06-02 -1 121 MB/s 616 MB/s 100000000 39943833 39.94 enwik8 xpack 2016-06-02 -9 11 MB/s 882 MB/s 100000000 34183028 34.18 enwik8 yappy 2014-03-22 -99 74 MB/s 1776 MB/s 100000000 53654128 53.65 enwik8 zlib 1.2.11 -1 73 MB/s 254 MB/s 100000000 42298774 42.30 enwik8 zlib 1.2.11 -9 17 MB/s 258 MB/s 100000000 36475792 36.48 enwik8 done... (cIters=1 dIters=15 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB) The results sorted by column number 4: Compressor name Compress. Decompress. Orig. size Compr. size Ratio Filename zstdLDM 1.4.3 -22 1.16 MB/s 613 MB/s 100000000 25345418 25.35 enwik8 bzip2 1.0.8 -9 12 MB/s 30 MB/s 100000000 29008758 29.01 enwik8 brotli 2019-10-01 -9 5.47 MB/s 408 MB/s 100000000 29468806 29.47 enwik8 lizard 1.0 -49 1.47 MB/s 914 MB/s 100000000 31688480 31.69 enwik8 crush 1.0 -2 0.52 MB/s 279 MB/s 100000000 31699549 31.70 enwik8 xpack 2016-06-02 -9 11 MB/s 882 MB/s 100000000 34183028 34.18 enwik8 libdeflate 1.3 -12 6.31 MB/s 591 MB/s 100000000 35100568 35.10 enwik8 lzfse 2017-03-08 48 MB/s 713 MB/s 100000000 36157828 36.16 enwik8 zlib 1.2.11 -9 17 MB/s 258 MB/s 100000000 36475792 36.48 enwik8 lizard 1.0 -29 1.53 MB/s 1504 MB/s 100000000 36959135 36.96 enwik8 lizard 1.0 -39 6.72 MB/s 1985 MB/s 100000000 37505492 37.51 enwik8 lzsse2 2019-04-18 -17 7.43 MB/s 3308 MB/s 100000000 38063182 38.06 enwik8 brotli 2019-10-01 -1 137 MB/s 292 MB/s 100000000 39123256 39.12 enwik8 libdeflate 1.3 -1 136 MB/s 592 MB/s 100000000 39597378 39.60 enwik8 xpack 2016-06-02 -1 121 MB/s 616 MB/s 100000000 39943833 39.94 enwik8 lzo1b 2.10 -999 15 MB/s 444 MB/s 100000000 40814011 40.81 enwik8 lizard 1.0 -19 6.87 MB/s 3212 MB/s 100000000 41953190 41.95 enwik8 zlib 1.2.11 -1 73 MB/s 254 MB/s 100000000 42298774 42.30 enwik8 quicklz 1.5.0 -3 35 MB/s 567 MB/s 100000000 44789793 44.79 enwik8 lzsse2 2019-04-18 -1 15 MB/s 2890 MB/s 100000000 45854485 45.85 enwik8 gipfeli 2016-07-13 219 MB/s 325 MB/s 100000000 45873643 45.87 enwik8 density 0.14.2 -3 298 MB/s 310 MB/s 100000000 47817692 47.82 enwik8 lzrw 15-Jul-1991 -5 98 MB/s 375 MB/s 100000000 47874203 47.87 enwik8 yappy 2014-03-22 -99 74 MB/s 1776 MB/s 100000000 53654128 53.65 enwik8 lzf 3.6 -1 212 MB/s 493 MB/s 100000000 53945381 53.95 enwik8 fastlz 0.1 -2 199 MB/s 354 MB/s 100000000 54163013 54.16 enwik8 snappy 2019-09-30 246 MB/s 905 MB/s 100000000 56555035 56.56 enwik8 lzjb 2010 216 MB/s 317 MB/s 100000000 68711273 68.71 enwik8 C:\Compression_showdown>copy oo2core_8_win64.dll oo2core_6_win64.dll /y 1 file(s) copied. C:\Compression_showdown>date 12/04/2018 0"turbobench_v18.05_-_build_04_May_2018" "enwik8" -esnappy_c/yappy/lzma,9d30:fb273:mf=bt4/oodle,89,91,95,99,111,115,119,129,139/lzturbo,19,12,10,29,22,20,39,32,30,59/brotli,1,3,5/brotli,11d30/zstd,22d30/lizard,11,19,21,29,31,39,41,49/trle -I3 -J31 -k1 -B2G C:\Compression_showdown>"turbobench_v18.05_-_build_04_May_2018" -p4 "enwik8.tbb" `VP C:\Compression_showdown>call _Benchmark_ZPAQ_ZSTD_RAZOR_BROTLI_ZIP_BZIP2_7Z_BSC_BriefLZ_Nakamichi.BAT enwik8 C:\Compression_showdown>sha1sum "enwik8" 57b8363b814821dc9d47aa4d41f58733519076b2 enwik8 C:\Compression_showdown>timer64 "zpaq_v7.05_x64.exe" add "enwik8.method211.zpaq" "enwik8" -method 211 -threads 1 zpaq v7.05 journaling archiver, compiled Apr 17 2015 Adding 100.000000 MB in 1 files -method 211 -threads 1 at 2020-12-05 13:25:14. 100.00% 0:00:00 + enwik8 100000000 100.00% 0:00:00 [1..1457] 100005836 -method 211,105,1 1 +added, 0 -removed. 0.000000 + (100.000000 -> 100.000000 -> 30.088258) = 30.088258 MB 31.532 seconds (all OK) Kernel Time = 0.234 = 0% User Time = 31.296 = 99% Process Time = 31.531 = 99% Virtual Memory = 2551 MB Global Time = 31.553 = 100% Physical Memory = 580 MB C:\Compression_showdown>timer64 "zpaq_v7.05_x64.exe" add "enwik8.method511.zpaq" "enwik8" -method 511 -threads 1 zpaq v7.05 journaling archiver, compiled Apr 17 2015 Adding 100.000000 MB in 1 files -method 511 -threads 1 at 2020-12-05 13:25:46. 100.00% 0:00:00 + enwik8 100000000 100.00% 0:00:00 [1..1457] 100005836 -method 511,105,1 1 +added, 0 -removed. 0.000000 + (100.000000 -> 100.000000 -> 19.084598) = 19.084598 MB 279.406 seconds (all OK) Kernel Time = 0.484 = 0% User Time = 278.656 = 99% Process Time = 279.140 = 99% Virtual Memory = 3602 MB Global Time = 279.417 = 100% Physical Memory = 1583 MB C:\Compression_showdown>timer64 zstd-v1.4.5-win64.exe --single-thread --ultra -22 --zstd=wlog=31,clog=30,hlog=30,slog=26 "enwik8" -o "enwik8.L22W2GB.zst" enwik8 : 25.33% (100000000 => 25328858 bytes, enwik8.L22W2GB.zst) Kernel Time = 0.375 = 0% User Time = 95.593 = 92% Process Time = 95.968 = 92% Virtual Memory = 2151 MB Global Time = 103.289 = 100% Physical Memory = 2148 MB C:\Compression_showdown>timer64 rz.exe a -d 1023M "enwik8.1GB.rz" "enwik8" *** RAZOR archiver 1.03.7 2018-03-22 *** *** (c) Christian Martelock christian.martelock@web.de *** *** This DEMO/TEST version is for non-commercial use only *** Scanning c:\compression_showdown\enwik8 Found 0 dirs, 1 files, 100000000 bytes. Creating archive enwik8.1GB.rz Window : 97657K (512M..128G) Adding enwik8 Header : 42 Size : 21992522 Archive ok. Added 0 dirs, 1 files, 100000000 bytes. CPU time = 362.141s / wall time = 277.906s Kernel Time = 0.625 = 0% User Time = 362.140 = 130% Process Time = 362.765 = 130% Virtual Memory = 1251 MB Global Time = 277.908 = 100% Physical Memory = 1157 MB C:\Compression_showdown>timer64 brotli_gc82.exe -q 11 --large_window=30 "enwik8" -o "enwik8.1GB.bro" Kernel Time = 0.734 = 0% User Time = 263.125 = 99% Process Time = 263.859 = 100% Virtual Memory = 10333 MB Global Time = 263.852 = 100% Physical Memory = 902 MB C:\Compression_showdown>timer64 "7za_x64_v1900.exe" a -tgzip -mmt1 -mx=9 "enwik8.MX9.gzip" "enwik8" 7-Zip (a) 19.00 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2019-02-21 Scanning the drive: 1 file, 100000000 bytes (96 MiB) Creating archive: enwik8.MX9.gzip Add new data to archive: 1 file, 100000000 bytes (96 MiB) Files read from disk: 1 Archive size: 35103261 bytes (34 MiB) Everything is Ok Kernel Time = 0.078 = 0% User Time = 84.906 = 99% Process Time = 84.984 = 99% Virtual Memory = 6 MB Global Time = 85.173 = 100% Physical Memory = 9 MB C:\Compression_showdown>timer64 "7za_x64_v1900.exe" a -tbzip2 -mmt1 -mx=9 "enwik8.MX9.bzip2" "enwik8" 7-Zip (a) 19.00 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2019-02-21 Scanning the drive: 1 file, 100000000 bytes (96 MiB) Creating archive: enwik8.MX9.bzip2 Add new data to archive: 1 file, 100000000 bytes (96 MiB) Files read from disk: 1 Archive size: 29006372 bytes (28 MiB) Everything is Ok Kernel Time = 0.078 = 0% User Time = 143.218 = 99% Process Time = 143.296 = 99% Virtual Memory = 13 MB Global Time = 143.474 = 100% Physical Memory = 15 MB C:\Compression_showdown>timer64 "7za_x64_v1900.exe" a -t7z -mmt1 -mx=9 -md=30 "enwik8.MX9Dict1024.7z" "enwik8" 7-Zip (a) 19.00 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2019-02-21 Scanning the drive: 1 file, 100000000 bytes (96 MiB) Creating archive: enwik8.MX9Dict1024.7z Add new data to archive: 1 file, 100000000 bytes (96 MiB) Files read from disk: 1 Archive size: 24799182 bytes (24 MiB) Everything is Ok Kernel Time = 0.187 = 0% User Time = 89.953 = 99% Process Time = 90.140 = 99% Virtual Memory = 1045 MB Global Time = 90.644 = 100% Physical Memory = 993 MB C:\Compression_showdown>timer64 "bsc_v3.1.0_x64.exe" e "enwik8" "enwik8.ST0Block1024.bsc" -m0 -b1024 -e2 -Tt This is bsc, Block Sorting Compressor. Version 3.1.0. 8 July 2012. Copyright (c) 2009-2012 Ilya Grebnov . enwik8 compressed 100000000 into 20786218 in 13.610 seconds. Kernel Time = 0.140 = 1% User Time = 13.453 = 98% Process Time = 13.593 = 99% Virtual Memory = 487 MB Global Time = 13.606 = 100% Physical Memory = 481 MB C:\Compression_showdown>timer64 BriefLZ_130_Intel_v19_64bit.exe --optimal -b2g "enwik8" "enwik8.2GB.blz" Kernel Time = 0.609 = 0% User Time = 100.312 = 99% Process Time = 100.921 = 99% Virtual Memory = 45402 MB Global Time = 100.945 = 100% Physical Memory = 3988 MB C:\Compression_showdown>dir "enwik8.*"/os Volume in drive C is System Volume Serial Number is 1E2A-3E30 Directory of C:\Compression_showdown 12/05/2020 15:30 19,084,598 enwik8.method511.zpaq 12/05/2020 15:46 20,786,218 enwik8.ST0Block1024.bsc 12/05/2020 15:36 21,992,522 enwik8.1GB.rz 12/05/2020 15:46 24,799,182 enwik8.MX9Dict1024.7z 11/25/2020 03:37 24,808,565 enwik8.1GB.bro 11/25/2020 03:37 25,328,858 enwik8.L22W2GB.zst 12/05/2020 15:44 29,006,372 enwik8.MX9.bzip2 12/05/2020 15:25 30,088,258 enwik8.method211.zpaq 12/05/2020 15:48 30,496,733 enwik8.2GB.blz 12/05/2020 01:44 33,044,928 enwik8.Nakamichi 12/05/2020 15:42 35,103,261 enwik8.MX9.gzip 11/25/2020 03:37 100,000,000 enwik8 12 File(s) 394,539,495 bytes 0 Dir(s) 1,961,520,128,000 bytes free C:\Compression_showdown>fv "enwik8" Reading enwik8 Drawing fv.bmp 512 by 256 from enwik8 (100000000 bytes) Drew part 1 of 4 in 10.39 seconds Drew part 2 of 4 in 10.38 seconds Drew part 3 of 4 in 11.36 seconds Drew part 4 of 4 in 17.86 seconds Created fv.bmp in 50.13 seconds C:\Compression_showdown>copy fv.bmp "fv_enwik8.bmp" /y 1 file(s) copied. C:\Compression_showdown>del fv.bmp C:\Compression_showdown>call _BENCHMARK_lzbench_turbobench.BAT enwik9 C:\Compression_showdown>timer64 lz4_win64_v1_9_3.exe -b1 -e12 --fast=12 -i9 --no-frame-crc "enwik9" Benchmarking levels from -12 to 12 -12#nwik9 :1000000000 -> 731796513 (1.367), 783.1 MB/s ,2896.5 MB/s -11#nwik9 :1000000000 -> 719477242 (1.390), 790.0 MB/s ,3234.7 MB/s -10#nwik9 :1000000000 -> 708284438 (1.412), 749.0 MB/s ,2939.2 MB/s -9#enwik9 :1000000000 -> 696167133 (1.436), 692.1 MB/s ,2734.2 MB/s -8#enwik9 :1000000000 -> 683273362 (1.464), 661.9 MB/s ,2820.4 MB/s -7#enwik9 :1000000000 -> 670234161 (1.492), 652.3 MB/s ,2754.4 MB/s -6#enwik9 :1000000000 -> 656598390 (1.523), 611.2 MB/s ,2499.9 MB/s -5#enwik9 :1000000000 -> 635866364 (1.573), 560.5 MB/s ,2769.8 MB/s -4#enwik9 :1000000000 -> 614950737 (1.626), 529.8 MB/s ,2567.6 MB/s -3#enwik9 :1000000000 -> 589209799 (1.697), 514.6 MB/s ,2643.6 MB/s -2#enwik9 :1000000000 -> 564008885 (1.773), 491.5 MB/s ,2734.7 MB/s -1#enwik9 :1000000000 -> 537981995 (1.859), 436.4 MB/s ,2737.4 MB/s 0#enwik9 :1000000000 -> 509196023 (1.964), 405.5 MB/s ,3005.7 MB/s 1#enwik9 :1000000000 -> 509196023 (1.964), 407.6 MB/s ,3029.3 MB/s 2#enwik9 :1000000000 -> 509196023 (1.964), 414.1 MB/s ,2808.9 MB/s 3#enwik9 :1000000000 -> 387882359 (2.578), 76.8 MB/s ,2809.8 MB/s 4#enwik9 :1000000000 -> 380593309 (2.627), 62.9 MB/s ,2832.8 MB/s 5#enwik9 :1000000000 -> 376871181 (2.653), 51.4 MB/s ,2764.9 MB/s 6#enwik9 :1000000000 -> 375209520 (2.665), 43.9 MB/s ,2719.2 MB/s 7#enwik9 :1000000000 -> 374513912 (2.670), 39.0 MB/s ,2747.8 MB/s 8#enwik9 :1000000000 -> 374223597 (2.672), 36.7 MB/s ,3042.7 MB/s 9#enwik9 :1000000000 -> 374083516 (2.673), 33.5 MB/s ,2733.0 MB/s 10#enwik9 :1000000000 -> 372317380 (2.686), 21.7 MB/s ,2942.5 MB/s 11#enwik9 :1000000000 -> 371711094 (2.690), 18.4 MB/s ,3011.9 MB/s 12#enwik9 :1000000000 -> 371677964 (2.691), 15.4 MB/s ,3177.2 MB/s Kernel Time = 4.625 = 0% User Time = 797.578 = 97% Process Time = 802.203 = 97% Virtual Memory = 2951 MB Global Time = 821.471 = 100% Physical Memory = 2868 MB C:\Compression_showdown>timer64 zstd-v1.4.5-win64.exe -b1 -e22 --fast=22 --threads=1 --priority=rt --no-check -i9 "enwik9" Note : switching to real-time priority -22#nwik9 :1000000000 -> 731111556 (1.368), 856.6 MB/s ,2373.1 MB/s Note : switching to real-time priority -21#nwik9 :1000000000 -> 724721354 (1.380), 784.3 MB/s ,2015.5 MB/s Note : switching to real-time priority -20#nwik9 :1000000000 -> 718200263 (1.392), 758.5 MB/s ,2028.4 MB/s Note : switching to real-time priority -19#nwik9 :1000000000 -> 711374150 (1.406), 720.0 MB/s ,1912.9 MB/s Note : switching to real-time priority -18#nwik9 :1000000000 -> 704066961 (1.420), 701.7 MB/s ,1929.5 MB/s Note : switching to real-time priority -17#nwik9 :1000000000 -> 696284452 (1.436), 670.0 MB/s ,1783.9 MB/s Note : switching to real-time priority -16#nwik9 :1000000000 -> 688344299 (1.453), 650.4 MB/s ,1749.3 MB/s Note : switching to real-time priority -15#nwik9 :1000000000 -> 679373662 (1.472), 624.7 MB/s ,1812.3 MB/s Note : switching to real-time priority -14#nwik9 :1000000000 -> 669287935 (1.494), 642.1 MB/s ,1893.2 MB/s Note : switching to real-time priority -13#nwik9 :1000000000 -> 659323365 (1.517), 580.9 MB/s ,1852.2 MB/s Note : switching to real-time priority -12#nwik9 :1000000000 -> 648661080 (1.542), 569.5 MB/s ,1688.3 MB/s Note : switching to real-time priority -11#nwik9 :1000000000 -> 636843027 (1.570), 535.8 MB/s ,1715.6 MB/s Note : switching to real-time priority -10#nwik9 :1000000000 -> 623962959 (1.603), 509.5 MB/s ,1602.9 MB/s Note : switching to real-time priority -9#enwik9 :1000000000 -> 610636178 (1.638), 491.4 MB/s ,1596.7 MB/s Note : switching to real-time priority -8#enwik9 :1000000000 -> 596140541 (1.677), 481.5 MB/s ,1506.7 MB/s Note : switching to real-time priority -7#enwik9 :1000000000 -> 581427946 (1.720), 445.0 MB/s ,1571.4 MB/s Note : switching to real-time priority -6#enwik9 :1000000000 -> 564158177 (1.773), 445.9 MB/s ,1502.2 MB/s Note : switching to real-time priority -5#enwik9 :1000000000 -> 542718927 (1.843), 399.7 MB/s ,1424.2 MB/s Note : switching to real-time priority -4#enwik9 :1000000000 -> 518152244 (1.930), 386.0 MB/s ,1446.9 MB/s Note : switching to real-time priority -3#enwik9 :1000000000 -> 493689112 (2.026), 365.0 MB/s ,1376.8 MB/s Note : switching to real-time priority -2#enwik9 :1000000000 -> 470173013 (2.127), 349.6 MB/s ,1368.1 MB/s Note : switching to real-time priority -1#enwik9 :1000000000 -> 453757056 (2.204), 324.1 MB/s ,1365.5 MB/s Note : switching to real-time priority 0#enwik9 :1000000000 -> 313417226 (3.191), 163.3 MB/s , 888.7 MB/s Note : switching to real-time priority 1#enwik9 :1000000000 -> 357408189 (2.798), 294.7 MB/s ,1029.2 MB/s Note : switching to real-time priority 2#enwik9 :1000000000 -> 329020778 (3.039), 214.2 MB/s , 930.5 MB/s Note : switching to real-time priority 3#enwik9 :1000000000 -> 313417226 (3.191), 163.8 MB/s , 899.3 MB/s Note : switching to real-time priority 4#enwik9 :1000000000 -> 307416100 (3.253), 148.1 MB/s , 869.2 MB/s Note : switching to real-time priority 5#enwik9 :1000000000 -> 301594332 (3.316), 99.0 MB/s , 867.5 MB/s Note : switching to real-time priority 6#enwik9 :1000000000 -> 295044831 (3.389), 77.0 MB/s , 863.1 MB/s Note : switching to real-time priority 7#enwik9 :1000000000 -> 284665291 (3.513), 50.7 MB/s , 903.2 MB/s Note : switching to real-time priority 8#enwik9 :1000000000 -> 280669567 (3.563), 43.1 MB/s , 970.1 MB/s Note : switching to real-time priority 9#enwik9 :1000000000 -> 278213009 (3.594), 30.1 MB/s , 994.3 MB/s Note : switching to real-time priority 10#enwik9 :1000000000 -> 273413148 (3.657), 20.9 MB/s ,1026.2 MB/s Note : switching to real-time priority 11#enwik9 :1000000000 -> 270882974 (3.692), 15.1 MB/s , 961.6 MB/s Note : switching to real-time priority 12#enwik9 :1000000000 -> 268803119 (3.720), 9.29 MB/s ,1014.0 MB/s Note : switching to real-time priority 13#enwik9 :1000000000 -> 265614307 (3.765), 7.40 MB/s , 976.3 MB/s Note : switching to real-time priority 14#enwik9 :1000000000 -> 260814193 (3.834), 5.40 MB/s ,1039.6 MB/s Note : switching to real-time priority 15#enwik9 :1000000000 -> 257582679 (3.882), 3.89 MB/s , 973.3 MB/s Note : switching to real-time priority 16#enwik9 :1000000000 -> 249939606 (4.001), 3.57 MB/s , 972.5 MB/s Note : switching to real-time priority 17#enwik9 :1000000000 -> 242716889 (4.120), 2.56 MB/s , 900.2 MB/s Note : switching to real-time priority 18#enwik9 :1000000000 -> 239533930 (4.175), 2.20 MB/s , 876.3 MB/s Note : switching to real-time priority 19#enwik9 :1000000000 -> 235586643 (4.245), 1.82 MB/s , 760.2 MB/s Note : switching to real-time priority 20#enwik9 :1000000000 -> 226026294 (4.424), 1.50 MB/s , 665.5 MB/s Note : switching to real-time priority 21#enwik9 :1000000000 -> 220258348 (4.540), 1.28 MB/s , 695.3 MB/s Note : switching to real-time priority 22#enwik9 :1000000000 -> 215052701 (4.650), 1.12 MB/s , 640.9 MB/s Kernel Time = 23.078 = 0% User Time = 5749.078 = 99% Process Time = 5772.156 = 99% Virtual Memory = 3643 MB Global Time = 5777.032 = 100% Physical Memory = 3510 MB C:\Compression_showdown>lzbench18 -c4 -i1,15 -o3 -ebzip2,9/zstdLDM,22/brotli,1,9/crush,2/density,3/fastlz,2/gipfeli/lzo1b,999/libdeflate,1,12/lizard,19,29,39,49/lzf,1/lzfse/lzjb/lzrw,5/lzsse2,1,17/quicklz,3/snappy/xpack,1,9/yappy,99/zlib,1,9 "enwik9" lzbench 1.8 (64-bit Windows) Assembled by P.Skibinski Compressor name Compress. Decompress. Orig. size Compr. size Ratio Filename memcpy 16482 MB/s 16755 MB/s 1000000000 1000000000 100.00 enwik9 bzip2 1.0.8 -9 12 MB/s 31 MB/s 1000000000 253977891 25.40 enwik9 zstdLDM 1.4.3 -22 0.75 MB/s 711 MB/s 1000000000 215096273 21.51 enwik9 brotli 2019-10-01 -1 164 MB/s 173 MB/s 1000000000 346792586 34.68 enwik9 brotli 2019-10-01 -9 3.76 MB/s 232 MB/s 1000000000 257421205 25.74 enwik9 crush 1.0 -2 1.25 MB/s 310 MB/s 1000000000 279083341 27.91 enwik9 density 0.14.2 -3 297 MB/s 335 MB/s 1000000000 432910750 43.29 enwik9 fastlz 0.1 -2 239 MB/s 401 MB/s 1000000000 487260752 48.73 enwik9 gipfeli 2016-07-13 249 MB/s 369 MB/s 1000000000 411940549 41.19 enwik9 lzo1b 2.10 -999 18 MB/s 487 MB/s 1000000000 363178533 36.32 enwik9 libdeflate 1.3 -1 144 MB/s 651 MB/s 1000000000 355647167 35.56 enwik9 libdeflate 1.3 -12 5.69 MB/s 584 MB/s 1000000000 310824785 31.08 enwik9 lizard 1.0 -19 5.80 MB/s 2999 MB/s 1000000000 372092974 37.21 enwik9 lizard 1.0 -29 1.48 MB/s 1410 MB/s 1000000000 323348239 32.33 enwik9 lizard 1.0 -39 5.62 MB/s 1980 MB/s 1000000000 334314019 33.43 enwik9 lizard 1.0 -49 1.40 MB/s 920 MB/s 1000000000 277467216 27.75 enwik9 lzf 3.6 -1 214 MB/s 473 MB/s 1000000000 492987190 49.30 enwik9 lzfse 2017-03-08 51 MB/s 684 MB/s 1000000000 319756993 31.98 enwik9 lzjb 2010 207 MB/s 318 MB/s 1000000000 665072021 66.51 enwik9 lzrw 15-Jul-1991 -5 108 MB/s 366 MB/s 1000000000 436682071 43.67 enwik9 lzsse2 2019-04-18 -1 15 MB/s 2907 MB/s 1000000000 405883480 40.59 enwik9 lzsse2 2019-04-18 -17 6.60 MB/s 3350 MB/s 1000000000 340270593 34.03 enwik9 quicklz 1.5.0 -3 40 MB/s 608 MB/s 1000000000 395494056 39.55 enwik9 snappy 2019-09-30 263 MB/s 912 MB/s 1000000000 507860747 50.79 enwik9 xpack 2016-06-02 -1 137 MB/s 647 MB/s 1000000000 358741520 35.87 enwik9 xpack 2016-06-02 -9 13 MB/s 883 MB/s 1000000000 300716430 30.07 enwik9 yappy 2014-03-22 -99 76 MB/s 1718 MB/s 1000000000 492824491 49.28 enwik9 zlib 1.2.11 -1 77 MB/s 254 MB/s 1000000000 378355076 37.84 enwik9 zlib 1.2.11 -9 20 MB/s 269 MB/s 1000000000 322789230 32.28 enwik9 done... (cIters=1 dIters=15 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB) The results sorted by column number 4: Compressor name Compress. Decompress. Orig. size Compr. size Ratio Filename zstdLDM 1.4.3 -22 0.75 MB/s 711 MB/s 1000000000 215096273 21.51 enwik9 bzip2 1.0.8 -9 12 MB/s 31 MB/s 1000000000 253977891 25.40 enwik9 brotli 2019-10-01 -9 3.76 MB/s 232 MB/s 1000000000 257421205 25.74 enwik9 lizard 1.0 -49 1.40 MB/s 920 MB/s 1000000000 277467216 27.75 enwik9 crush 1.0 -2 1.25 MB/s 310 MB/s 1000000000 279083341 27.91 enwik9 xpack 2016-06-02 -9 13 MB/s 883 MB/s 1000000000 300716430 30.07 enwik9 libdeflate 1.3 -12 5.69 MB/s 584 MB/s 1000000000 310824785 31.08 enwik9 lzfse 2017-03-08 51 MB/s 684 MB/s 1000000000 319756993 31.98 enwik9 zlib 1.2.11 -9 20 MB/s 269 MB/s 1000000000 322789230 32.28 enwik9 lizard 1.0 -29 1.48 MB/s 1410 MB/s 1000000000 323348239 32.33 enwik9 lizard 1.0 -39 5.62 MB/s 1980 MB/s 1000000000 334314019 33.43 enwik9 lzsse2 2019-04-18 -17 6.60 MB/s 3350 MB/s 1000000000 340270593 34.03 enwik9 brotli 2019-10-01 -1 164 MB/s 173 MB/s 1000000000 346792586 34.68 enwik9 libdeflate 1.3 -1 144 MB/s 651 MB/s 1000000000 355647167 35.56 enwik9 xpack 2016-06-02 -1 137 MB/s 647 MB/s 1000000000 358741520 35.87 enwik9 lzo1b 2.10 -999 18 MB/s 487 MB/s 1000000000 363178533 36.32 enwik9 lizard 1.0 -19 5.80 MB/s 2999 MB/s 1000000000 372092974 37.21 enwik9 zlib 1.2.11 -1 77 MB/s 254 MB/s 1000000000 378355076 37.84 enwik9 quicklz 1.5.0 -3 40 MB/s 608 MB/s 1000000000 395494056 39.55 enwik9 lzsse2 2019-04-18 -1 15 MB/s 2907 MB/s 1000000000 405883480 40.59 enwik9 gipfeli 2016-07-13 249 MB/s 369 MB/s 1000000000 411940549 41.19 enwik9 density 0.14.2 -3 297 MB/s 335 MB/s 1000000000 432910750 43.29 enwik9 lzrw 15-Jul-1991 -5 108 MB/s 366 MB/s 1000000000 436682071 43.67 enwik9 fastlz 0.1 -2 239 MB/s 401 MB/s 1000000000 487260752 48.73 enwik9 yappy 2014-03-22 -99 76 MB/s 1718 MB/s 1000000000 492824491 49.28 enwik9 lzf 3.6 -1 214 MB/s 473 MB/s 1000000000 492987190 49.30 enwik9 snappy 2019-09-30 263 MB/s 912 MB/s 1000000000 507860747 50.79 enwik9 lzjb 2010 207 MB/s 318 MB/s 1000000000 665072021 66.51 enwik9 C:\Compression_showdown>copy oo2core_8_win64.dll oo2core_6_win64.dll /y 1 file(s) copied. C:\Compression_showdown>date 12/04/2018 0"turbobench_v18.05_-_build_04_May_2018" "enwik9" -esnappy_c/yappy/lzma,9d30:fb273:mf=bt4/oodle,89,91,95,99,111,115,119,129,139/lzturbo,19,12,10,29,22,20,39,32,30,59/brotli,1,3,5/brotli,11d30/zstd,22d30/lizard,11,19,21,29,31,39,41,49/trle -I3 -J31 -k1 -B2G C:\Compression_showdown>"turbobench_v18.05_-_build_04_May_2018" -p4 "enwik9.tbb" `V· C:\Compression_showdown>call _Benchmark_ZPAQ_ZSTD_RAZOR_BROTLI_ZIP_BZIP2_7Z_BSC_BriefLZ_Nakamichi.BAT enwik9 C:\Compression_showdown>sha1sum "enwik9" 2996e86fb978f93cca8f566cc56998923e7fe581 enwik9 C:\Compression_showdown>timer64 "zpaq_v7.05_x64.exe" add "enwik9.method211.zpaq" "enwik9" -method 211 -threads 1 zpaq v7.05 journaling archiver, compiled Apr 17 2015 Adding 1000.000000 MB in 1 files -method 211 -threads 1 at 2020-12-05 18:24:26. 100.00% 0:00:00 + enwik9 1000000000 100.00% 0:00:00 [1..14621] 1000058492 -method 211,122,1 1 +added, 0 -removed. 0.000000 + (1000.000000 -> 1000.000000 -> 244.711837) = 244.711837 MB 324.968 seconds (all OK) Kernel Time = 1.453 = 0% User Time = 323.203 = 99% Process Time = 324.656 = 99% Virtual Memory = 6776 MB Global Time = 324.960 = 100% Physical Memory = 5544 MB C:\Compression_showdown>timer64 "zpaq_v7.05_x64.exe" add "enwik9.method511.zpaq" "enwik9" -method 511 -threads 1 zpaq v7.05 journaling archiver, compiled Apr 17 2015 Adding 1000.000000 MB in 1 files -method 511 -threads 1 at 2020-12-05 18:29:51. 100.00% 0:00:00 + enwik9 1000000000 100.00% 0:00:00 [1..14621] 1000058492 -method 511,122,1 1 +added, 0 -removed. 0.000000 + (1000.000000 -> 1000.000000 -> 153.626112) = 153.626112 MB 2817.062 seconds (all OK) Kernel Time = 3.796 = 0% User Time = 2812.875 = 99% Process Time = 2816.671 = 99% Virtual Memory = 11891 MB Global Time = 2817.078 = 100% Physical Memory = 10352 MB C:\Compression_showdown>timer64 zstd-v1.4.5-win64.exe --single-thread --ultra -22 --zstd=wlog=31,clog=30,hlog=30,slog=26 "enwik9" -o "enwik9.L22W2GB.zst" enwik9 : 20.54% (1000000000 => 205362522 bytes, enwik9.L22W2GB.zst) Kernel Time = 4.078 = 0% User Time = 1270.718 = 92% Process Time = 1274.796 = 93% Virtual Memory = 9167 MB Global Time = 1366.458 = 100% Physical Memory = 9151 MB C:\Compression_showdown>timer64 rz.exe a -d 1023M "enwik9.1GB.rz" "enwik9" *** RAZOR archiver 1.03.7 2018-03-22 *** *** (c) Christian Martelock christian.martelock@web.de *** *** This DEMO/TEST version is for non-commercial use only *** Scanning c:\compression_showdown\enwik9 Found 0 dirs, 1 files, 1000000000 bytes. Creating archive enwik9.1GB.rz Window : 976563K (4096M..1024G) Adding enwik9 Header : 44 Size : 173041178 Archive ok. Added 0 dirs, 1 files, 1000000000 bytes. CPU time = 5008.172s / wall time = 3883.819s Kernel Time = 7.921 = 0% User Time = 5008.171 = 128% Process Time = 5016.093 = 129% Virtual Memory = 11263 MB Global Time = 3883.814 = 100% Physical Memory = 10101 MB C:\Compression_showdown>timer64 brotli_gc82.exe -q 11 --large_window=30 "enwik9" -o "enwik9.1GB.bro" Kernel Time = 10.015 = 0% User Time = 2873.250 = 99% Process Time = 2883.265 = 99% Virtual Memory = 10346 MB Global Time = 2883.499 = 100% Physical Memory = 8629 MB C:\Compression_showdown>timer64 "7za_x64_v1900.exe" a -tgzip -mmt1 -mx=9 "enwik9.MX9.gzip" "enwik9" 7-Zip (a) 19.00 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2019-02-21 Scanning the drive: 1 file, 1000000000 bytes (954 MiB) Creating archive: enwik9.MX9.gzip Add new data to archive: 1 file, 1000000000 bytes (954 MiB) Files read from disk: 1 Archive size: 310706136 bytes (297 MiB) Everything is Ok Kernel Time = 0.312 = 0% User Time = 865.718 = 99% Process Time = 866.031 = 99% Virtual Memory = 6 MB Global Time = 866.272 = 100% Physical Memory = 9 MB C:\Compression_showdown>timer64 "7za_x64_v1900.exe" a -tbzip2 -mmt1 -mx=9 "enwik9.MX9.bzip2" "enwik9" 7-Zip (a) 19.00 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2019-02-21 Scanning the drive: 1 file, 1000000000 bytes (954 MiB) Creating archive: enwik9.MX9.bzip2 Add new data to archive: 1 file, 1000000000 bytes (954 MiB) Files read from disk: 1 Archive size: 253971930 bytes (243 MiB) Everything is Ok Kernel Time = 0.453 = 0% User Time = 1356.000 = 99% Process Time = 1356.453 = 99% Virtual Memory = 13 MB Global Time = 1356.819 = 100% Physical Memory = 15 MB C:\Compression_showdown>timer64 "7za_x64_v1900.exe" a -t7z -mmt1 -mx=9 -md=30 "enwik9.MX9Dict1024.7z" "enwik9" 7-Zip (a) 19.00 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2019-02-21 Scanning the drive: 1 file, 1000000000 bytes (954 MiB) Creating archive: enwik9.MX9Dict1024.7z Add new data to archive: 1 file, 1000000000 bytes (954 MiB) Files read from disk: 1 Archive size: 200302539 bytes (192 MiB) Everything is Ok Kernel Time = 1.890 = 0% User Time = 1115.250 = 99% Process Time = 1117.140 = 99% Virtual Memory = 10776 MB Global Time = 1117.572 = 100% Physical Memory = 9614 MB C:\Compression_showdown>timer64 "bsc_v3.1.0_x64.exe" e "enwik9" "enwik9.ST0Block1024.bsc" -m0 -b1024 -e2 -Tt This is bsc, Block Sorting Compressor. Version 3.1.0. 8 July 2012. Copyright (c) 2009-2012 Ilya Grebnov . enwik9 compressed 1000000000 into 163879646 in 154.547 seconds. Kernel Time = 1.546 = 1% User Time = 152.593 = 98% Process Time = 154.140 = 99% Virtual Memory = 4686 MB Global Time = 154.555 = 100% Physical Memory = 4672 MB C:\Compression_showdown>timer64 BriefLZ_130_Intel_v19_64bit.exe --optimal -b2g "enwik9" "enwik9.2GB.blz" Kernel Time = 3.203 = 0% User Time = 11644.296 = 99% Process Time = 11647.500 = 99% Virtual Memory = 45403 MB Global Time = 11652.082 = 100% Physical Memory = 21367 MB C:\Compression_showdown>dir "enwik9.*"/os Volume in drive C is System Volume Serial Number is 1E2A-3E30 Directory of C:\Compression_showdown 12/05/2020 21:16 153,626,112 enwik9.method511.zpaq 12/06/2020 00:30 163,879,646 enwik9.ST0Block1024.bsc 12/05/2020 22:44 173,041,178 enwik9.1GB.rz 11/25/2020 03:37 199,103,323 enwik9.1GB.bro 12/06/2020 00:28 200,302,539 enwik9.MX9Dict1024.7z 11/25/2020 03:37 205,362,522 enwik9.L22W2GB.zst 12/05/2020 20:29 244,711,837 enwik9.method211.zpaq 12/06/2020 03:44 252,991,647 enwik9.2GB.blz 12/06/2020 00:09 253,971,930 enwik9.MX9.bzip2 12/05/2020 10:08 279,301,006 enwik9.Nakamichi 12/05/2020 23:46 310,706,136 enwik9.MX9.gzip 11/25/2020 03:37 1,000,000,000 enwik9 12 File(s) 3,436,997,876 bytes 0 Dir(s) 1,959,357,251,584 bytes free C:\Compression_showdown>fv "enwik9" Reading enwik9 Drawing fv.bmp 512 by 256 from enwik9 (1000000000 bytes) Drew part 1 of 4 in 94.06 seconds Drew part 2 of 4 in 95.35 seconds Drew part 3 of 4 in 103.26 seconds Drew part 4 of 4 in 166.82 seconds Created fv.bmp in 459.63 seconds C:\Compression_showdown>copy fv.bmp "fv_enwik9.bmp" /y 1 file(s) copied. C:\Compression_showdown>del fv.bmp C:\Compression_showdown> C:\>cd Global_Data_Compression_Competition_2020 C:\Global_Data_Compression_Competition_2020>runme C:\Global_Data_Compression_Competition_2020>call Nakamichify_internal-64GB.bat TS100.txt C:\Global_Data_Compression_Competition_2020>TIMER64 Nakamichi_DD_LITE_GCC_64bit_plus-16GB-buffer.exe "TS100.txt" "TS100.txt.Nakamichi" 24 34000 i SMMi :MM2 0MMMMM: rMMMMMa ZMMM. 0Z :MMM7 7B rMM@ MMM 7 MMM MMM. BMMMa XZ :MM: MMM XMMX XMSANMAYCE; rMMMM rMM; MM@ WM0 2MMMMM ZMMMM MMW BMM : MMMB aMMMMW 2MM :MM8 MM7 8MMi MMMZ XMMM MMa .WMMM0 @MM rMMMMi MMMMa MM, aMM XMMB MM: MMM2 @MXMMM7 0MM MMS WMM MM MMM, MM7 7MM MM: MMX ;MMM iMM MMM8 . MM8 MMS MMMZ . iMBi :MMMX @MMa WB: XMMZ2aX ZMM MMM2 :M2 :MM ZMMM BM8 ;MMMMMaaMM ,MM XMM@MM rMMMMM2aMM XMMMM SMMSMM. MMXMMM MMMM0 aMMMa@MMr MMS ;MM;MM8 XMMMM. MMr MMM0 MM 2MMW MMM 0MM ZMS 2MM. aMMB MMM BM aMM 8Ma MM iMM BMM M7 MMZ ;MMX MMi ;MM ZMW MM0 rM 7MM ;MM MMM7 iMM ZMM2 0MMa MMX 8M: MMM ZMMS BMM2 7 MM8 ZMr :MM rMB MM@ iMM MMZ iMM WMW BM2 MMX @MM MMi ;MMM 0MS 2MM7 aMMM ;MM .8Mi MMM aMM7 ZMMM MM ;Mr XMM iM8 MMa 0MM BMM MMM MM: BMX .MM MM7 aMM BMMM MM ;MMX ZMMMZ BM@ MMM . rMM7 ZMMMZ :MM:MX 0MZ M8 XMM, MMX MM. ;W: rMM aMX ZMM 7MM MM MMMi MM MMZ 8M:MM MM; MMM, MMa 0M:MM 8MWM8 MMZM0 MMM rMM 0MM WMa,M2 MMS MM8 @M0 :MMMXMa MMM WM XMB ; :MM;M8MMM MMM @M XM0 ; MMM@ MMMW MMrZi MMS MMi MMZM0 rMM Mi MM : ;MM BMMMM: rMM MM MMSMa ZMMM@ iMMr 7MM MM MMXM2 :MMM ZMMM 8MM@@ MMZ0 7MM , :MMMM MMBMa 0MWBr ,MMMM MMMM MMS MM MMMZ MMMM MMM MMS .MM MMMZ @MM. MMM MMMM WMM@ 0MM 2M ZMMM :MMMa MMM7 iMMMMMM@ XMMM .MM XMM aMMM MMM: MMW .MM XM@ ZMMM MM0 aMMX 8MMM .MMM WMM ZMr MMMX MMM8 0MMS SMMMMi MMM, rMM,MM2 MMM: SMMM ;MM2 ;MM,MMS MMM, MMM MMM MMM. MMM .MM MMZ MMM XMMM rMMB :MMM7 MM@ BMMM aMM ;MMX aMMX 8MMM ZMM MM8 .MMX MMM2 iMMX ,MMMMMMZ .@; MMM WMM . .: : 0MMZ .2Z :7r. ;i 0MMMaSMi aMMMM7 .WW Nakamichi 'Ryuugan-ditto-1TB', written by Kaze, inspired by Haruhiko Okumura sharing, based on Nobuo Ito's LZSS source, muffinesque tips by m^2, Jim Dempsey and Kirill Kryukov enforced. This latest (2020-Nov-20) compile has B-trees only matchfinder - called 'LITE'; Downloadable at: https://twitter.com/Sanmayce Nakamichi 'Kaidanji' SUPRALITE highlights: - Back to supersimplisticism - 'Kaidanji' returns refined. Simply, ripping only orders 4..16 and then quickly trying to expand the match in the found position. Nakamichi 'Ryuugan-ditto-1TB'/'Dragoneye' LITE highlights: - The Zennish LZSS Microdeduplicator, a texttoy (file-to-file [de]compressor) written in plain C, 100% FREE - licenseless it is; - Single-threaded Non-SIMD console tool written entirely by yours truly Sanmayce; - The first Lempel-Ziv-Storer-Szymanski (LZSS) compressor using 1TB sliding window; - It targets superfast decompression of huge/tera corpora of textual data; - LITE means no memmem() (Railgun) invocations are to be made; - Compressing at Linear Rate due to Matchfinding based on B-trees only; - The Leprechaunesque (Internal/External) B-trees order 3 (2 keys MAX) are brutally-optimized; - Ability to deduplicate (as little as) 64 bytes long chunks 1TB backwards; - Ability to deduplicate 256[+] bytes long chunks 1TB backwards - when SHA3 enabled; - CHUNKABLE! Since there are no headers/metadata in .Nakamichi files, arbitrary chunks/blocks can be compressed-n-concatenated; - SCALABLE! Gets faster when more Physical or/and External RAM is available; - Dynamical (as command line parameters) selection of HASH & B-trees memory blocks; - AUTO-RESETTING PASSES (when TargetBuffer is filled then automaticaly restart building by doubling the passes); - DEFRAGMENTED B-trees are in use during compression stage, their leaves contain only BBs appearing 2[+] times; - Compileable under Windows and Linux, most suitable to run on 3TB machines; - Many thanks go to: Haruhiko Okumura, Nobuo Ito, Hamid Buzidi, Landon Noll, Aleksey Vaneev (https://github.com/avaneev/prvhash). Sanmayce, Nakamichi's author, here, some more notes: - The B-trees form the second layer, the first being HASH table handled by FNV1A-Pippip - the fastest lookup-hash; - Hashpot a.k.a. hashpools (residing in Physical RAM) could be tuned via command line parameter, thus lessening the B-trees heights; - The building of B-trees is done dynamically-n-automatically in PASSES, thus LOCALITY/LOCALIZATION leads to cache-friendliness, for example, instead of confusing/blinding the SSD controller with building 2^27 ~= 128M B-trees at a time, 'PASSES' revision lowers the "noise/mayhem" 128 times by processing e.g. (20bit) 1M B-trees at a time; - For so long, the main goal was to throw it in battle, thus to forge one battle-ready *nix/windows tool with superb hash/search/decompress capabilities. - Straight up, a wish of mine is to have the latest English Wikipedia XML dump decompressed by Nakamichi into physical RAM and then followed by Kazahana's superfast exact/wildcard/fuzzy etudes. - Through the years, such a transparent decompressor (load booster) has been losing momentum and kinda became unnecessary, but not for me, when HDDs had 40MB/s sequential read speeds it was ... boosty, now with SSDs reading at 1700+MB/s, not so smacky, yet uploading (on 128GB machines, 20GB+72GB needed) the whole Wikipedia ~20GB (instead of 72GB) is better, sizewise. - Currently, having 16 (4,6,8,10,12,14,16,18,36,64,24,28,48,56,128,256) MatchLens being hashed (plus 1 being phantomic/temporary), 29bit hashtable results in: (16+1)*8*(2^29) = (16+1)*8*(1<<29) = 69632MB hashpot (the cluster of all hashpools), each slot houses a QWORD (being an offset/address of a B-tree root). - In near future, wanna see a machine (256GB RAM) crunching the 77,265,813,300 bytes file with this command line (it requires 2x72GB + 68GB + 2930GB = 212GB physical RAM + 2.9TB SSD RAM): D:\>timer64 Nakamichi.exe enwiki-20200420-pages-articles.xml enwiki-20200420-pages-articles.xml.Nakamichi 29 3000000 e - In not so far future, wanna see a machine (4096GB RAM) crunching the 72GB file with this command line (it requires 2x72GB + 272GB + 2930GB = 3.3TB physical RAM): D:\>timer64 Nakamichi.exe enwiki-20200420-pages-articles.xml enwiki-20200420-pages-articles.xml.Nakamichi 31 3000000 i Q: - HOW TO BOOST [BUILDING/COMPRESSION] SPEED? A: - Now, the name of the game is "RAM in spades", if we have a lot then speed is going to jump a lot. How? - First, by increasing the hash-pool - 20 (number of bits) was used in 'Taishukan' resulting in 2^20 =~ 1M hash-slots a.k.a. B-trees' roots per keysize, naturally, the bigger the faster since B-trees will have less height and will be more - aiming to reach the ideal case - having only hash-slots with no actual B-trees! Back to 'Taishukan', the testfile is ~77MB, it suggests using 128M slots to lower height/branching of B-trees - 2^27 leads to using 27. Feartime, going from 20 to 27: 8*(15+1)*2^(20) = 134,217,793 bytes; with the speed-up hash needs 8*(15+1)*2^(27) = 17,179,869,184 bytes, yes 16GB. - Second, by lowering passes in building the B-trees - the trick is to increase the number in compile-time parameter -DSpeedUpBuilding=32 - (didn't make it as an execute-time parameter, lazy). If we look at the 'Taishukan' console log it tells us that (Target-Buffer 106 MB) x 64 passes = 6784MB are needed for building the B-trees in ONE PASS ONLY, so compile with -DSpeedUpBuilding=6784 instead. Enfun! Current priority class is HIGH_PRIORITY_CLASS. This compile uses B-trees only, no memmem() invocations - it compresses worse but much faster. Allocating Source-Buffer 953 MB ... Allocating Target-Buffer 17,337 MB ... Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x4b2c,7bbc Leprechaun: Memory pool for B-trees is 34,000 MB. Leprechaun: In this revision 128MB 16-way hash is used which results in 16 x 16,777,216 internal B-Trees of order 3. Leprechaun: Allocating HASH memory 2,281,701,441 bytes = 8*(16+1)*2^(24) ... OK Leprechaun: Allocating memory for B-trees 34001 MB ... OK Leprechaun: Size of input file: 1,000,000,000 Leprechaun: Using (first 12 bytes of) DoubleDeuce (a.k.a. CRC64+CRC32C) for Matches 12+ long, in order to reduce memory footprint. Leprechaun: Inserting keys/BBs of order 004 into B-trees, free RAM in B-tree pool is 00,034,000 MB; Pass #000,001 of 000,001 ... DONE; 00,000,545,878 DEFRAGMENTED B-trees have been rooted so far. 554,956 TotalNonUnique (of length 004) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 006 into B-trees, free RAM in B-tree pool is 00,033,973 MB; Pass #000,001 of 000,001 ... DONE; 00,005,325,322 DEFRAGMENTED B-trees have been rooted so far. 5,624,061 TotalNonUnique (of length 006) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 008 into B-trees, free RAM in B-tree pool is 00,033,719 MB; Pass #000,001 of 000,001 ... DONE; 00,017,461,825 DEFRAGMENTED B-trees have been rooted so far. 21,601,285 TotalNonUnique (of length 008) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 010 into B-trees, free RAM in B-tree pool is 00,032,780 MB; Pass #000,001 of 000,001 ... DONE; 00,033,371,054 DEFRAGMENTED B-trees have been rooted so far. 49,713,349 TotalNonUnique (of length 010) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 012 into B-trees, free RAM in B-tree pool is 00,030,499 MB; Pass #000,001 of 000,001 ... DONE; 00,050,003,353 DEFRAGMENTED B-trees have been rooted so far. 79,779,695 TotalNonUnique (of length 012) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 014 into B-trees, free RAM in B-tree pool is 00,026,625 MB; Pass #000,001 of 000,001 ... Leprechaun: Failure! Increment 'Memory for B-trees'! Automatically increasing number of passes in order to fit B-trees into Target Buffer. Leprechaun: Inserting keys/BBs of order 014 into B-trees, free RAM in B-tree pool is 00,024,220 MB; Pass #000,002 of 000,002 ... DONE; 00,066,731,157 DEFRAGMENTED B-trees have been rooted so far. 97,811,459 TotalNonUnique (of length 014) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 016 into B-trees, free RAM in B-tree pool is 00,019,407 MB; Pass #000,002 of 000,002 ... DONE; 00,083,457,209 DEFRAGMENTED B-trees have been rooted so far. 97,277,746 TotalNonUnique (of length 016) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 018 into B-trees, free RAM in B-tree pool is 00,017,000 MB; Pass #000,001 of 000,002 ... Leprechaun: Failure! Increment 'Memory for B-trees'! Automatically increasing number of passes in order to fit B-trees into Target Buffer. Leprechaun: Inserting keys/BBs of order 018 into B-trees, free RAM in B-tree pool is 00,013,921 MB; Pass #000,004 of 000,004 ... DONE; 00,100,112,981 DEFRAGMENTED B-trees have been rooted so far. 82,668,688 TotalNonUnique (of length 018) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 036 into B-trees, free RAM in B-tree pool is 00,012,579 MB; Pass #000,004 of 000,004 ... DONE; 00,106,399,458 DEFRAGMENTED B-trees have been rooted so far. 7,878,309 TotalNonUnique (of length 036) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 064 into B-trees, free RAM in B-tree pool is 00,012,249 MB; Pass #000,004 of 000,004 ... DONE; 00,111,015,125 DEFRAGMENTED B-trees have been rooted so far. 5,399,298 TotalNonUnique (of length 064) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 024 into B-trees, free RAM in B-tree pool is 00,010,996 MB; Pass #000,004 of 000,004 ... DONE; 00,125,270,041 DEFRAGMENTED B-trees have been rooted so far. 31,776,558 TotalNonUnique (of length 024) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 028 into B-trees, free RAM in B-tree pool is 00,009,989 MB; Pass #000,004 of 000,004 ... DONE; 00,135,688,329 DEFRAGMENTED B-trees have been rooted so far. 16,280,873 TotalNonUnique (of length 028) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 048 into B-trees, free RAM in B-tree pool is 00,009,532 MB; Pass #000,004 of 000,004 ... DONE; 00,140,838,367 DEFRAGMENTED B-trees have been rooted so far. 6,150,580 TotalNonUnique (of length 048) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 056 into B-trees, free RAM in B-tree pool is 00,009,211 MB; Pass #000,004 of 000,004 ... DONE; 00,145,686,003 DEFRAGMENTED B-trees have been rooted so far. 5,719,827 TotalNonUnique (of length 056) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 128 into B-trees, free RAM in B-tree pool is 00,008,962 MB; Pass #000,004 of 000,004 ... DONE; 00,149,222,976 DEFRAGMENTED B-trees have been rooted so far. 3,972,144 TotalNonUnique (of length 128) keys have been inserted into DEFRAGMENTED B-trees. Leprechaun: Inserting keys/BBs of order 256 into B-trees, free RAM in B-tree pool is 00,008,784 MB; Pass #000,004 of 000,004 ... DONE; 00,151,773,685 DEFRAGMENTED B-trees have been rooted so far. 2,766,775 TotalNonUnique (of length 256) keys have been inserted into DEFRAGMENTED B-trees. Histogram (rotate it 90 degrees clockwise) of Unique-Building-Blocks appearing 2[+] times (e.g. 'alfalfa-alfa' stream has 1 BB with length 4 appearing 3 times thus adding 1 to the count for {004]): 2,766,775{256] 3,972,144{128] 5,399,298{064] 5,719,827{056] 6,150,580{048] 7,878,309{036] 16,280,873{028] 31,776,558{024] 82,668,688{018] 97,277,746{016] 97,811,459{014] 79,779,695{012] 49,713,349{010] 21,601,285{008] 5,624,061{006] 554,956{004] Leprechaun: Total Searches-n-Inserts Per Second: 1,811,698 SNIPS = 56,912,710,059 keys in 31,414 seconds Leprechaun: RAM needed to house B-trees (relative to the file being ripped): 26N = 25,257MB Leprechaun: RAM needed to build B-trees IN ONE PASS: (Target-Buffer = 17,337 MB) x 4 passes = 69,348MB Note: In case of RAM in spades then may use compile option: -DSpeedUpBuilding=69348 Compressing 1,000,000,000 bytes ... -; Each rotation means 64KB are encoded; Speed: 0,128,468 B/s; Done 100%; Compression Ratio: 3.15:1; Matches(16/24/48): 9,248,153/2,972,700/42,012; 128[+] long matches: 27,583; ETA: 0.00 hours NumberOfFullLiterals (lower-the-better): 23432 Tsuyo_HEURISTIC_APPLIED_thrice_back-to-back: 0 NumberOf(Tiny)Matches[Micro]Window (4)[16B]: 319001 NumberOfMatches[Bheema]Window [128GB window]: 687137 RAM-to-RAM performance: 128468 B/s. Compressed to 317,763,629 bytes. Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x4b2c,7bbc Target-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x937d,2f6b Kernel Time = 5.531 = 0% User Time = 29498.343 = 75% Process Time = 29503.875 = 75% Virtual Memory = 54575 MB Global Time = 39196.939 = 100% Physical Memory = 45727 MB C:\Global_Data_Compression_Competition_2020>Nakamichi_DD_LITE_GCC_64bit_plus-320MB-buffer.exe "TS100.txt.Nakamichi" 1>"TS100.txt.NKMCH" C:\Global_Data_Compression_Competition_2020>fc "TS100.txt" "TS100.txt.NKMCH" /b Comparing files TS100.txt and TS100.TXT.NKMCH FC: no differences encountered C:\Global_Data_Compression_Competition_2020>if exist "TS100.txt.NKMCH" del "TS100.txt.NKMCH" C:\Global_Data_Compression_Competition_2020>"Nakamichi_DD_LITE_GCC_64bit_plus-16GB-buffer.exe" TS100.txt.Nakamichi /bench SMMi :MM2 0MMMMM: rMMMMMa ZMMM. 0Z :MMM7 7B rMM@ MMM 7 MMM MMM. BMMMa XZ :MM: MMM XMMX XMSANMAYCE; rMMMM rMM; MM@ WM0 2MMMMM ZMMMM MMW BMM : MMMB aMMMMW 2MM :MM8 MM7 8MMi MMMZ XMMM MMa .WMMM0 @MM rMMMMi MMMMa MM, aMM XMMB MM: MMM2 @MXMMM7 0MM MMS WMM MM MMM, MM7 7MM MM: MMX ;MMM iMM MMM8 . MM8 MMS MMMZ . iMBi :MMMX @MMa WB: XMMZ2aX ZMM MMM2 :M2 :MM ZMMM BM8 ;MMMMMaaMM ,MM XMM@MM rMMMMM2aMM XMMMM SMMSMM. MMXMMM MMMM0 aMMMa@MMr MMS ;MM;MM8 XMMMM. MMr MMM0 MM 2MMW MMM 0MM ZMS 2MM. aMMB MMM BM aMM 8Ma MM iMM BMM M7 MMZ ;MMX MMi ;MM ZMW MM0 rM 7MM ;MM MMM7 iMM ZMM2 0MMa MMX 8M: MMM ZMMS BMM2 7 MM8 ZMr :MM rMB MM@ iMM MMZ iMM WMW BM2 MMX @MM MMi ;MMM 0MS 2MM7 aMMM ;MM .8Mi MMM aMM7 ZMMM MM ;Mr XMM iM8 MMa 0MM BMM MMM MM: BMX .MM MM7 aMM BMMM MM ;MMX ZMMMZ BM@ MMM . rMM7 ZMMMZ :MM:MX 0MZ M8 XMM, MMX MM. ;W: rMM aMX ZMM 7MM MM MMMi MM MMZ 8M:MM MM; MMM, MMa 0M:MM 8MWM8 MMZM0 MMM rMM 0MM WMa,M2 MMS MM8 @M0 :MMMXMa MMM WM XMB ; :MM;M8MMM MMM @M XM0 ; MMM@ MMMW MMrZi MMS MMi MMZM0 rMM Mi MM : ;MM BMMMM: rMM MM MMSMa ZMMM@ iMMr 7MM MM MMXM2 :MMM ZMMM 8MM@@ MMZ0 7MM , :MMMM MMBMa 0MWBr ,MMMM MMMM MMS MM MMMZ MMMM MMM MMS .MM MMMZ @MM. MMM MMMM WMM@ 0MM 2M ZMMM :MMMa MMM7 iMMMMMM@ XMMM .MM XMM aMMM MMM: MMW .MM XM@ ZMMM MM0 aMMX 8MMM .MMM WMM ZMr MMMX MMM8 0MMS SMMMMi MMM, rMM,MM2 MMM: SMMM ;MM2 ;MM,MMS MMM, MMM MMM MMM. MMM .MM MMZ MMM XMMM rMMB :MMM7 MM@ BMMM aMM ;MMX aMMX 8MMM ZMM MM8 .MMX MMM2 iMMX ,MMMMMMZ .@; MMM WMM . .: : 0MMZ .2Z :7r. ;i 0MMMaSMi aMMMM7 .WW Nakamichi 'Ryuugan-ditto-1TB', written by Kaze, inspired by Haruhiko Okumura sharing, based on Nobuo Ito's LZSS source, muffinesque tips by m^2, Jim Dempsey and Kirill Kryukov enforced. This latest (2020-Nov-20) compile has B-trees only matchfinder - called 'LITE'; Downloadable at: https://twitter.com/Sanmayce Nakamichi 'Kaidanji' SUPRALITE highlights: - Back to supersimplisticism - 'Kaidanji' returns refined. Simply, ripping only orders 4..16 and then quickly trying to expand the match in the found position. Nakamichi 'Ryuugan-ditto-1TB'/'Dragoneye' LITE highlights: - The Zennish LZSS Microdeduplicator, a texttoy (file-to-file [de]compressor) written in plain C, 100% FREE - licenseless it is; - Single-threaded Non-SIMD console tool written entirely by yours truly Sanmayce; - The first Lempel-Ziv-Storer-Szymanski (LZSS) compressor using 1TB sliding window; - It targets superfast decompression of huge/tera corpora of textual data; - LITE means no memmem() (Railgun) invocations are to be made; - Compressing at Linear Rate due to Matchfinding based on B-trees only; - The Leprechaunesque (Internal/External) B-trees order 3 (2 keys MAX) are brutally-optimized; - Ability to deduplicate (as little as) 64 bytes long chunks 1TB backwards; - Ability to deduplicate 256[+] bytes long chunks 1TB backwards - when SHA3 enabled; - CHUNKABLE! Since there are no headers/metadata in .Nakamichi files, arbitrary chunks/blocks can be compressed-n-concatenated; - SCALABLE! Gets faster when more Physical or/and External RAM is available; - Dynamical (as command line parameters) selection of HASH & B-trees memory blocks; - AUTO-RESETTING PASSES (when TargetBuffer is filled then automaticaly restart building by doubling the passes); - DEFRAGMENTED B-trees are in use during compression stage, their leaves contain only BBs appearing 2[+] times; - Compileable under Windows and Linux, most suitable to run on 3TB machines; - Many thanks go to: Haruhiko Okumura, Nobuo Ito, Hamid Buzidi, Landon Noll, Aleksey Vaneev (https://github.com/avaneev/prvhash). Sanmayce, Nakamichi's author, here, some more notes: - The B-trees form the second layer, the first being HASH table handled by FNV1A-Pippip - the fastest lookup-hash; - Hashpot a.k.a. hashpools (residing in Physical RAM) could be tuned via command line parameter, thus lessening the B-trees heights; - The building of B-trees is done dynamically-n-automatically in PASSES, thus LOCALITY/LOCALIZATION leads to cache-friendliness, for example, instead of confusing/blinding the SSD controller with building 2^27 ~= 128M B-trees at a time, 'PASSES' revision lowers the "noise/mayhem" 128 times by processing e.g. (20bit) 1M B-trees at a time; - For so long, the main goal was to throw it in battle, thus to forge one battle-ready *nix/windows tool with superb hash/search/decompress capabilities. - Straight up, a wish of mine is to have the latest English Wikipedia XML dump decompressed by Nakamichi into physical RAM and then followed by Kazahana's superfast exact/wildcard/fuzzy etudes. - Through the years, such a transparent decompressor (load booster) has been losing momentum and kinda became unnecessary, but not for me, when HDDs had 40MB/s sequential read speeds it was ... boosty, now with SSDs reading at 1700+MB/s, not so smacky, yet uploading (on 128GB machines, 20GB+72GB needed) the whole Wikipedia ~20GB (instead of 72GB) is better, sizewise. - Currently, having 16 (4,6,8,10,12,14,16,18,36,64,24,28,48,56,128,256) MatchLens being hashed (plus 1 being phantomic/temporary), 29bit hashtable results in: (16+1)*8*(2^29) = (16+1)*8*(1<<29) = 69632MB hashpot (the cluster of all hashpools), each slot houses a QWORD (being an offset/address of a B-tree root). - In near future, wanna see a machine (256GB RAM) crunching the 77,265,813,300 bytes file with this command line (it requires 2x72GB + 68GB + 2930GB = 212GB physical RAM + 2.9TB SSD RAM): D:\>timer64 Nakamichi.exe enwiki-20200420-pages-articles.xml enwiki-20200420-pages-articles.xml.Nakamichi 29 3000000 e - In not so far future, wanna see a machine (4096GB RAM) crunching the 72GB file with this command line (it requires 2x72GB + 272GB + 2930GB = 3.3TB physical RAM): D:\>timer64 Nakamichi.exe enwiki-20200420-pages-articles.xml enwiki-20200420-pages-articles.xml.Nakamichi 31 3000000 i Q: - HOW TO BOOST [BUILDING/COMPRESSION] SPEED? A: - Now, the name of the game is "RAM in spades", if we have a lot then speed is going to jump a lot. How? - First, by increasing the hash-pool - 20 (number of bits) was used in 'Taishukan' resulting in 2^20 =~ 1M hash-slots a.k.a. B-trees' roots per keysize, naturally, the bigger the faster since B-trees will have less height and will be more - aiming to reach the ideal case - having only hash-slots with no actual B-trees! Back to 'Taishukan', the testfile is ~77MB, it suggests using 128M slots to lower height/branching of B-trees - 2^27 leads to using 27. Feartime, going from 20 to 27: 8*(15+1)*2^(20) = 134,217,793 bytes; with the speed-up hash needs 8*(15+1)*2^(27) = 17,179,869,184 bytes, yes 16GB. - Second, by lowering passes in building the B-trees - the trick is to increase the number in compile-time parameter -DSpeedUpBuilding=32 - (didn't make it as an execute-time parameter, lazy). If we look at the 'Taishukan' console log it tells us that (Target-Buffer 106 MB) x 64 passes = 6784MB are needed for building the B-trees in ONE PASS ONLY, so compile with -DSpeedUpBuilding=6784 instead. Enfun! Current priority class is HIGH_PRIORITY_CLASS. Allocating Source-Buffer 303 MB ... Allocating Target-Buffer 5,120 MB ... Decompressing 317,763,629 bytes (being the compressed stream) ... Warming up ... RAM-to-RAM performance: 735 MB/s; 728 MB/s; 725 MB/s; 736 MB/s; 717 MB/s; 718 MB/s; 734 MB/s; 737 MB/s; 729 MB/s; 711 MB/s; 743 MB/s; 742 MB/s; 725 MB/s; 706 MB/s; 744 MB/s; 724 MB/s Enforcing 17 seconds idling to avoid throttling ... 717 MB/s; 739 MB/s; 719 MB/s; 716 MB/s; 738 MB/s; 715 MB/s; 717 MB/s; 738 MB/s; 717 MB/s; 718 MB/s; 738 MB/s; 718 MB/s; 719 MB/s; 736 MB/s; 715 MB/s; 720 MB/s Enforcing 17 seconds idling to avoid throttling ... 729 MB/s; 710 MB/s; 732 MB/s; 720 MB/s; 706 MB/s; 710 MB/s; 750 MB/s; 727 MB/s; 708 MB/s; 745 MB/s; 739 MB/s; 711 MB/s; 691 MB/s; 707 MB/s; 734 MB/s; 708 MB/s Enforcing 17 seconds idling to avoid throttling ... 699 MB/s; 742 MB/s; 720 MB/s; 701 MB/s; 726 MB/s; 727 MB/s; 712 MB/s; 697 MB/s; 743 MB/s; 720 MB/s; 700 MB/s; 720 MB/s; 732 MB/s; 712 MB/s; 695 MB/s; 742 MB/s Enforcing 17 seconds idling to avoid throttling ... 719 MB/s; 713 MB/s; 745 MB/s; 718 MB/s; 697 MB/s; 731 MB/s; 727 MB/s; 709 MB/s; 735 MB/s; 730 MB/s; 705 MB/s; 703 MB/s; 742 MB/s; 719 MB/s; 699 MB/s; 730 MB/s Enforcing 17 seconds idling to avoid throttling ... 724 MB/s; 704 MB/s; 709 MB/s; 737 MB/s; 717 MB/s; 720 MB/s; 736 MB/s; 718 MB/s; 721 MB/s; 735 MB/s; 716 MB/s; 694 MB/s; 739 MB/s; 728 MB/s; 705 MB/s; 740 MB/s Enforcing 17 seconds idling to avoid throttling ... 718 MB/s; 698 MB/s; 725 MB/s; 731 MB/s; 711 MB/s; 707 MB/s; 742 MB/s; 719 MB/s; 701 MB/s; 718 MB/s; 731 MB/s; 709 MB/s; 693 MB/s; 742 MB/s; 721 MB/s; 701 MB/s Enforcing 17 seconds idling to avoid throttling ... 727 MB/s; 729 MB/s; 707 MB/s; 699 MB/s; 736 MB/s; 716 MB/s; 696 MB/s; 724 MB/s; 728 MB/s; 709 MB/s; 695 MB/s; 740 MB/s; 719 MB/s; 700 MB/s; 718 MB/s; 730 MB/s Enforcing 17 seconds idling to avoid throttling ... RAM-to-RAM (peak) performance: 750 MB/s. Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x937d,2f6b Target-file-Hash(FNV1A_YoshimitsuTRIAD) = 0x4b2c,7bbc C:\Global_Data_Compression_Competition_2020>call _BENCHMARK_lzbench_turbobench.BAT TS100.txt C:\Global_Data_Compression_Competition_2020>timer64 lz4_win64_v1_9_3.exe -b1 -e12 --fast=12 -i9 --no-frame-crc "TS100.txt" Benchmarking levels from -12 to 12 -12#S100.txt :1000000000 -> 844383433 (1.184), 598.4 MB/s ,2336.1 MB/s -11#S100.txt :1000000000 -> 831333983 (1.203), 589.7 MB/s ,2617.1 MB/s -10#S100.txt :1000000000 -> 819557817 (1.220), 550.4 MB/s ,2451.6 MB/s -9#TS100.txt :1000000000 -> 806865745 (1.239), 519.9 MB/s ,2465.5 MB/s -8#TS100.txt :1000000000 -> 792740842 (1.261), 479.1 MB/s ,2299.2 MB/s -7#TS100.txt :1000000000 -> 777588302 (1.286), 461.3 MB/s ,2118.4 MB/s -6#TS100.txt :1000000000 -> 761914032 (1.312), 475.0 MB/s ,2315.1 MB/s -5#TS100.txt :1000000000 -> 741113659 (1.349), 456.9 MB/s ,2487.5 MB/s -4#TS100.txt :1000000000 -> 720750302 (1.387), 420.2 MB/s ,2639.9 MB/s -3#TS100.txt :1000000000 -> 690694244 (1.448), 409.7 MB/s ,2766.6 MB/s -2#TS100.txt :1000000000 -> 662837526 (1.509), 367.4 MB/s ,2814.6 MB/s -1#TS100.txt :1000000000 -> 633937234 (1.577), 356.8 MB/s ,2884.5 MB/s 0#TS100.txt :1000000000 -> 604293530 (1.655), 305.6 MB/s ,2630.1 MB/s 1#TS100.txt :1000000000 -> 604293530 (1.655), 306.3 MB/s ,2828.0 MB/s 2#TS100.txt :1000000000 -> 604293530 (1.655), 307.5 MB/s ,2732.5 MB/s 3#TS100.txt :1000000000 -> 450767025 (2.218), 56.5 MB/s ,2487.8 MB/s 4#TS100.txt :1000000000 -> 434425372 (2.302), 46.6 MB/s ,2588.8 MB/s 5#TS100.txt :1000000000 -> 425135506 (2.352), 34.7 MB/s ,2470.5 MB/s 6#TS100.txt :1000000000 -> 420602652 (2.378), 27.1 MB/s ,2576.2 MB/s 7#TS100.txt :1000000000 -> 418686332 (2.388), 22.0 MB/s ,2423.4 MB/s 8#TS100.txt :1000000000 -> 417925209 (2.393), 20.1 MB/s ,2719.6 MB/s 9#TS100.txt :1000000000 -> 417605152 (2.395), 18.5 MB/s ,2621.4 MB/s 10#TS100.txt :1000000000 -> 414007732 (2.415), 14.2 MB/s ,2994.1 MB/s 11#TS100.txt :1000000000 -> 412669998 (2.423), 11.9 MB/s ,3039.8 MB/s 12#TS100.txt :1000000000 -> 412587333 (2.424), 10.7 MB/s ,2988.5 MB/s Kernel Time = 4.765 = 0% User Time = 989.406 = 95% Process Time = 994.171 = 96% Virtual Memory = 2951 MB Global Time = 1033.662 = 100% Physical Memory = 2868 MB C:\Global_Data_Compression_Competition_2020>timer64 zstd-v1.4.5-win64.exe -b1 -e22 --fast=22 --threads=1 --priority=rt --no-check -i9 "TS100.txt" Note : switching to real-time priority -22#S100.txt :1000000000 -> 855097726 (1.169), 679.3 MB/s ,1944.2 MB/s Note : switching to real-time priority -21#S100.txt :1000000000 -> 849063651 (1.178), 590.4 MB/s ,1665.3 MB/s Note : switching to real-time priority -20#S100.txt :1000000000 -> 842936496 (1.186), 593.4 MB/s ,1787.0 MB/s Note : switching to real-time priority -19#S100.txt :1000000000 -> 836208739 (1.196), 588.6 MB/s ,1634.2 MB/s Note : switching to real-time priority -18#S100.txt :1000000000 -> 828742518 (1.207), 573.3 MB/s ,1558.3 MB/s Note : switching to real-time priority -17#S100.txt :1000000000 -> 820721555 (1.218), 521.2 MB/s ,1469.0 MB/s Note : switching to real-time priority -16#S100.txt :1000000000 -> 812528325 (1.231), 521.9 MB/s ,1475.9 MB/s Note : switching to real-time priority -15#S100.txt :1000000000 -> 803152889 (1.245), 481.2 MB/s ,1522.9 MB/s Note : switching to real-time priority -14#S100.txt :1000000000 -> 792008573 (1.263), 457.0 MB/s ,1468.4 MB/s Note : switching to real-time priority -13#S100.txt :1000000000 -> 780740371 (1.281), 465.5 MB/s ,1585.1 MB/s Note : switching to real-time priority -12#S100.txt :1000000000 -> 767720626 (1.303), 434.7 MB/s ,1481.0 MB/s Note : switching to real-time priority -11#S100.txt :1000000000 -> 753968913 (1.326), 412.4 MB/s ,1546.5 MB/s Note : switching to real-time priority -10#S100.txt :1000000000 -> 738725588 (1.354), 409.9 MB/s ,1408.7 MB/s Note : switching to real-time priority -9#TS100.txt :1000000000 -> 722462090 (1.384), 379.4 MB/s ,1407.0 MB/s Note : switching to real-time priority -8#TS100.txt :1000000000 -> 705825878 (1.417), 375.7 MB/s ,1355.8 MB/s Note : switching to real-time priority -7#TS100.txt :1000000000 -> 688936946 (1.452), 343.8 MB/s ,1373.6 MB/s Note : switching to real-time priority -6#TS100.txt :1000000000 -> 669914446 (1.493), 326.6 MB/s ,1412.4 MB/s Note : switching to real-time priority -5#TS100.txt :1000000000 -> 645787805 (1.548), 307.4 MB/s ,1350.3 MB/s Note : switching to real-time priority -4#TS100.txt :1000000000 -> 615968348 (1.623), 291.5 MB/s ,1302.8 MB/s Note : switching to real-time priority -3#TS100.txt :1000000000 -> 584972948 (1.709), 269.2 MB/s ,1246.7 MB/s Note : switching to real-time priority -2#TS100.txt :1000000000 -> 558091084 (1.792), 247.5 MB/s ,1174.7 MB/s Note : switching to real-time priority -1#TS100.txt :1000000000 -> 535118890 (1.869), 229.2 MB/s ,1153.7 MB/s Note : switching to real-time priority 0#TS100.txt :1000000000 -> 355912964 (2.810), 136.9 MB/s , 803.3 MB/s Note : switching to real-time priority 1#TS100.txt :1000000000 -> 399426790 (2.504), 225.3 MB/s , 935.3 MB/s Note : switching to real-time priority 2#TS100.txt :1000000000 -> 372309858 (2.686), 168.0 MB/s , 819.7 MB/s Note : switching to real-time priority 3#TS100.txt :1000000000 -> 355912964 (2.810), 136.8 MB/s , 841.4 MB/s Note : switching to real-time priority 4#TS100.txt :1000000000 -> 351027829 (2.849), 129.6 MB/s , 817.0 MB/s Note : switching to real-time priority 5#TS100.txt :1000000000 -> 350031143 (2.857), 87.4 MB/s , 781.1 MB/s Note : switching to real-time priority 6#TS100.txt :1000000000 -> 341235632 (2.931), 63.9 MB/s , 824.2 MB/s Note : switching to real-time priority 7#TS100.txt :1000000000 -> 329647022 (3.034), 43.4 MB/s , 838.9 MB/s Note : switching to real-time priority 8#TS100.txt :1000000000 -> 325489376 (3.072), 33.4 MB/s , 851.1 MB/s Note : switching to real-time priority 9#TS100.txt :1000000000 -> 321164135 (3.114), 21.6 MB/s , 906.6 MB/s Note : switching to real-time priority 10#TS100.txt :1000000000 -> 318784038 (3.137), 15.6 MB/s , 901.5 MB/s Note : switching to real-time priority 11#TS100.txt :1000000000 -> 317248237 (3.152), 11.5 MB/s , 868.2 MB/s Note : switching to real-time priority 12#TS100.txt :1000000000 -> 313442306 (3.190), 6.14 MB/s , 898.0 MB/s Note : switching to real-time priority 13#TS100.txt :1000000000 -> 306342802 (3.264), 4.64 MB/s , 942.4 MB/s Note : switching to real-time priority 14#TS100.txt :1000000000 -> 301785135 (3.314), 3.37 MB/s , 897.6 MB/s Note : switching to real-time priority 15#TS100.txt :1000000000 -> 297986173 (3.356), 2.41 MB/s , 863.6 MB/s Note : switching to real-time priority 16#TS100.txt :1000000000 -> 284978985 (3.509), 2.71 MB/s , 882.4 MB/s Note : switching to real-time priority 17#TS100.txt :1000000000 -> 280427841 (3.566), 1.98 MB/s , 908.2 MB/s Note : switching to real-time priority 18#TS100.txt :1000000000 -> 276140049 (3.621), 1.70 MB/s , 784.8 MB/s Note : switching to real-time priority 19#TS100.txt :1000000000 -> 272641321 (3.668), 1.52 MB/s , 714.7 MB/s Note : switching to real-time priority 20#TS100.txt :1000000000 -> 265663485 (3.764), 1.24 MB/s , 577.9 MB/s Note : switching to real-time priority 21#TS100.txt :1000000000 -> 261058845 (3.831), 1.04 MB/s , 575.2 MB/s Note : switching to real-time priority 22#TS100.txt :1000000000 -> 256896944 (3.893), 0.90 MB/s , 581.7 MB/s Kernel Time = 24.750 = 0% User Time = 7230.218 = 99% Process Time = 7254.968 = 99% Virtual Memory = 3643 MB Global Time = 7260.323 = 100% Physical Memory = 3511 MB C:\Global_Data_Compression_Competition_2020>lzbench18 -c4 -i1,15 -o3 -ebzip2,9/zstdLDM,22/brotli,1,9/crush,2/density,3/fastlz,2/gipfeli/lzo1b,999/libdeflate,1,12/lizard,19,29,39,49/lzf,1/lzfse/lzjb/lzrw,5/lzsse2,1,17/quicklz,3/snappy/xpack,1,9/yappy,99/zlib,1,9 "TS100.txt" lzbench 1.8 (64-bit Windows) Assembled by P.Skibinski Compressor name Compress. Decompress. Orig. size Compr. size Ratio Filename memcpy 11901 MB/s 16202 MB/s 1000000000 1000000000 100.00 TS100.txt bzip2 1.0.8 -9 11 MB/s 30 MB/s 1000000000 272660764 27.27 TS100.txt zstdLDM 1.4.3 -22 0.89 MB/s 545 MB/s 1000000000 256983011 25.70 TS100.txt brotli 2019-10-01 -1 138 MB/s 289 MB/s 1000000000 386982489 38.70 TS100.txt brotli 2019-10-01 -9 5.21 MB/s 444 MB/s 1000000000 304537916 30.45 TS100.txt crush 1.0 -2 3.96 MB/s 262 MB/s 1000000000 326051153 32.61 TS100.txt density 0.14.2 -3 375 MB/s 384 MB/s 1000000000 467345762 46.73 TS100.txt fastlz 0.1 -2 201 MB/s 342 MB/s 1000000000 571605200 57.16 TS100.txt gipfeli 2016-07-13 207 MB/s 368 MB/s 1000000000 472528661 47.25 TS100.txt lzo1b 2.10 -999 9.72 MB/s 435 MB/s 1000000000 402501072 40.25 TS100.txt libdeflate 1.3 -1 129 MB/s 660 MB/s 1000000000 403405144 40.34 TS100.txt libdeflate 1.3 -12 5.64 MB/s 630 MB/s 1000000000 345972793 34.60 TS100.txt lizard 1.0 -19 6.22 MB/s 2970 MB/s 1000000000 412856151 41.29 TS100.txt lizard 1.0 -29 1.32 MB/s 1292 MB/s 1000000000 378416320 37.84 TS100.txt lizard 1.0 -39 6.14 MB/s 2120 MB/s 1000000000 352668484 35.27 TS100.txt lizard 1.0 -49 1.28 MB/s 859 MB/s 1000000000 313277569 31.33 TS100.txt lzf 3.6 -1 197 MB/s 440 MB/s 1000000000 563981639 56.40 TS100.txt lzfse 2017-03-08 45 MB/s 598 MB/s 1000000000 368227240 36.82 TS100.txt lzjb 2010 172 MB/s 276 MB/s 1000000000 748751922 74.88 TS100.txt lzrw 15-Jul-1991 -5 91 MB/s 356 MB/s 1000000000 485039555 48.50 TS100.txt lzsse2 2019-04-18 -1 15 MB/s 2572 MB/s 1000000000 499226855 49.92 TS100.txt lzsse2 2019-04-18 -17 6.19 MB/s 3431 MB/s 1000000000 365390040 36.54 TS100.txt quicklz 1.5.0 -3 32 MB/s 558 MB/s 1000000000 481570694 48.16 TS100.txt snappy 2019-09-30 210 MB/s 799 MB/s 1000000000 600700553 60.07 TS100.txt xpack 2016-06-02 -1 128 MB/s 616 MB/s 1000000000 410950904 41.10 TS100.txt xpack 2016-06-02 -9 7.65 MB/s 909 MB/s 1000000000 340002858 34.00 TS100.txt yappy 2014-03-22 -99 65 MB/s 1426 MB/s 1000000000 555381713 55.54 TS100.txt zlib 1.2.11 -1 70 MB/s 253 MB/s 1000000000 437516967 43.75 TS100.txt zlib 1.2.11 -9 10 MB/s 262 MB/s 1000000000 364199742 36.42 TS100.txt done... (cIters=1 dIters=15 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB) The results sorted by column number 4: Compressor name Compress. Decompress. Orig. size Compr. size Ratio Filename zstdLDM 1.4.3 -22 0.89 MB/s 545 MB/s 1000000000 256983011 25.70 TS100.txt bzip2 1.0.8 -9 11 MB/s 30 MB/s 1000000000 272660764 27.27 TS100.txt brotli 2019-10-01 -9 5.21 MB/s 444 MB/s 1000000000 304537916 30.45 TS100.txt lizard 1.0 -49 1.28 MB/s 859 MB/s 1000000000 313277569 31.33 TS100.txt crush 1.0 -2 3.96 MB/s 262 MB/s 1000000000 326051153 32.61 TS100.txt xpack 2016-06-02 -9 7.65 MB/s 909 MB/s 1000000000 340002858 34.00 TS100.txt libdeflate 1.3 -12 5.64 MB/s 630 MB/s 1000000000 345972793 34.60 TS100.txt lizard 1.0 -39 6.14 MB/s 2120 MB/s 1000000000 352668484 35.27 TS100.txt zlib 1.2.11 -9 10 MB/s 262 MB/s 1000000000 364199742 36.42 TS100.txt lzsse2 2019-04-18 -17 6.19 MB/s 3431 MB/s 1000000000 365390040 36.54 TS100.txt lzfse 2017-03-08 45 MB/s 598 MB/s 1000000000 368227240 36.82 TS100.txt lizard 1.0 -29 1.32 MB/s 1292 MB/s 1000000000 378416320 37.84 TS100.txt brotli 2019-10-01 -1 138 MB/s 289 MB/s 1000000000 386982489 38.70 TS100.txt lzo1b 2.10 -999 9.72 MB/s 435 MB/s 1000000000 402501072 40.25 TS100.txt libdeflate 1.3 -1 129 MB/s 660 MB/s 1000000000 403405144 40.34 TS100.txt xpack 2016-06-02 -1 128 MB/s 616 MB/s 1000000000 410950904 41.10 TS100.txt lizard 1.0 -19 6.22 MB/s 2970 MB/s 1000000000 412856151 41.29 TS100.txt zlib 1.2.11 -1 70 MB/s 253 MB/s 1000000000 437516967 43.75 TS100.txt density 0.14.2 -3 375 MB/s 384 MB/s 1000000000 467345762 46.73 TS100.txt gipfeli 2016-07-13 207 MB/s 368 MB/s 1000000000 472528661 47.25 TS100.txt quicklz 1.5.0 -3 32 MB/s 558 MB/s 1000000000 481570694 48.16 TS100.txt lzrw 15-Jul-1991 -5 91 MB/s 356 MB/s 1000000000 485039555 48.50 TS100.txt lzsse2 2019-04-18 -1 15 MB/s 2572 MB/s 1000000000 499226855 49.92 TS100.txt yappy 2014-03-22 -99 65 MB/s 1426 MB/s 1000000000 555381713 55.54 TS100.txt lzf 3.6 -1 197 MB/s 440 MB/s 1000000000 563981639 56.40 TS100.txt fastlz 0.1 -2 201 MB/s 342 MB/s 1000000000 571605200 57.16 TS100.txt snappy 2019-09-30 210 MB/s 799 MB/s 1000000000 600700553 60.07 TS100.txt lzjb 2010 172 MB/s 276 MB/s 1000000000 748751922 74.88 TS100.txt C:\Global_Data_Compression_Competition_2020>copy oo2core_8_win64.dll oo2core_6_win64.dll /y 1 file(s) copied. C:\Global_Data_Compression_Competition_2020>date 12/04/2018 0"turbobench_v18.05_-_build_04_May_2018" "TS100.txt" -esnappy_c/yappy/lzma,9d30:fb273:mf=bt4/oodle,89,91,95,99,111,115,119,129,139/lzturbo,19,12,10,29,22,20,39,32,30,59/brotli,1,3,5/brotli,11d30/zstd,22d30/lizard,11,19,21,29,31,39,41,49/trle -I3 -J31 -k1 -B2G C:\Global_Data_Compression_Competition_2020>"turbobench_v18.05_-_build_04_May_2018" -p4 "TS100.txt.tbb" pVl C:\Global_Data_Compression_Competition_2020>call _Benchmark_ZPAQ_ZSTD_RAZOR_BROTLI_ZIP_BZIP2_7Z_BSC_BriefLZ_Nakamichi.BAT TS100.txt C:\Global_Data_Compression_Competition_2020>sha1sum "TS100.txt" d32aa36818d08568294777f8a23cae52ff240d53 TS100.txt C:\Global_Data_Compression_Competition_2020>timer64 "zpaq_v7.05_x64.exe" add "TS100.txt.method211.zpaq" "TS100.txt" -method 211 -threads 1 zpaq v7.05 journaling archiver, compiled Apr 17 2015 Adding 1000.000000 MB in 1 files -method 211 -threads 1 at 2020-12-07 03:39:56. 100.00% 0:00:00 + TS100.txt 1000000000 -> 999850263 100.00% 0:00:00 [1..14275] 999907371 -method 211,192,1 1 +added, 0 -removed. 0.000000 + (1000.000000 -> 999.850263 -> 300.132758) = 300.132758 MB 374.172 seconds (all OK) Kernel Time = 1.562 = 0% User Time = 372.390 = 99% Process Time = 373.953 = 99% Virtual Memory = 7160 MB Global Time = 374.185 = 100% Physical Memory = 5800 MB C:\Global_Data_Compression_Competition_2020>timer64 "zpaq_v7.05_x64.exe" add "TS100.txt.method511.zpaq" "TS100.txt" -method 511 -threads 1 zpaq v7.05 journaling archiver, compiled Apr 17 2015 Adding 1000.000000 MB in 1 files -method 511 -threads 1 at 2020-12-07 03:46:10. 100.00% 0:00:00 + TS100.txt 1000000000 -> 999850263 100.00% 0:00:00 [1..14275] 999907371 -method 511,192,1 1 +added, 0 -removed. 0.000000 + (1000.000000 -> 999.850263 -> 194.781915) = 194.781915 MB 2809.984 seconds (all OK) Kernel Time = 4.140 = 0% User Time = 2805.437 = 99% Process Time = 2809.578 = 99% Virtual Memory = 11891 MB Global Time = 2809.999 = 100% Physical Memory = 10283 MB C:\Global_Data_Compression_Competition_2020>timer64 zstd-v1.4.5-win64.exe --single-thread --ultra -22 --zstd=wlog=31,clog=30,hlog=30,slog=26 "TS100.txt" -o "TS100.txt.L22W2GB.zst" TS100.txt : 24.88% (1000000000 => 248823631 bytes, TS100.txt.L22W2GB.zst) Kernel Time = 5.546 = 0% User Time = 1340.718 = 91% Process Time = 1346.265 = 91% Virtual Memory = 9167 MB Global Time = 1470.131 = 100% Physical Memory = 9151 MB C:\Global_Data_Compression_Competition_2020>timer64 rz.exe a -d 1023M "TS100.txt.1GB.rz" "TS100.txt" *** RAZOR archiver 1.03.7 2018-03-22 *** *** (c) Christian Martelock christian.martelock@web.de *** *** This DEMO/TEST version is for non-commercial use only *** Scanning c:\global_data_compression_competition_2020\ts100.txt Found 0 dirs, 1 files, 1000000000 bytes. Creating archive TS100.txt.1GB.rz Window : 976563K (4096M..1024G) Adding TS100.txt Header : 46 Size : 219449490 Archive ok. Added 0 dirs, 1 files, 1000000000 bytes. CPU time = 6392.484s / wall time = 4891.059s Kernel Time = 8.812 = 0% User Time = 6392.484 = 130% Process Time = 6401.296 = 130% Virtual Memory = 11260 MB Global Time = 4891.070 = 100% Physical Memory = 10495 MB C:\Global_Data_Compression_Competition_2020>timer64 brotli_gc82.exe -q 11 --large_window=30 "TS100.txt" -o "TS100.txt.1GB.bro" Kernel Time = 11.156 = 0% User Time = 3176.718 = 99% Process Time = 3187.875 = 99% Virtual Memory = 10346 MB Global Time = 3188.460 = 100% Physical Memory = 8630 MB C:\Global_Data_Compression_Competition_2020>timer64 "7za_x64_v1900.exe" a -tgzip -mmt1 -mx=9 "TS100.txt.MX9.gzip" "TS100.txt" 7-Zip (a) 19.00 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2019-02-21 Scanning the drive: 1 file, 1000000000 bytes (954 MiB) Creating archive: TS100.txt.MX9.gzip Add new data to archive: 1 file, 1000000000 bytes (954 MiB) Files read from disk: 1 Archive size: 346488831 bytes (331 MiB) Everything is Ok Kernel Time = 0.453 = 0% User Time = 1010.281 = 99% Process Time = 1010.734 = 99% Virtual Memory = 6 MB Global Time = 1011.075 = 100% Physical Memory = 9 MB C:\Global_Data_Compression_Competition_2020>timer64 "7za_x64_v1900.exe" a -tbzip2 -mmt1 -mx=9 "TS100.txt.MX9.bzip2" "TS100.txt" 7-Zip (a) 19.00 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2019-02-21 Scanning the drive: 1 file, 1000000000 bytes (954 MiB) Creating archive: TS100.txt.MX9.bzip2 Add new data to archive: 1 file, 1000000000 bytes (954 MiB) Files read from disk: 1 Archive size: 272635725 bytes (261 MiB) Everything is Ok Kernel Time = 0.359 = 0% User Time = 1447.062 = 99% Process Time = 1447.421 = 99% Virtual Memory = 13 MB Global Time = 1447.801 = 100% Physical Memory = 15 MB C:\Global_Data_Compression_Competition_2020>timer64 "7za_x64_v1900.exe" a -t7z -mmt1 -mx=9 -md=30 "TS100.txt.MX9Dict1024.7z" "TS100.txt" 7-Zip (a) 19.00 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2019-02-21 Scanning the drive: 1 file, 1000000000 bytes (954 MiB) Creating archive: TS100.txt.MX9Dict1024.7z Add new data to archive: 1 file, 1000000000 bytes (954 MiB) Files read from disk: 1 Archive size: 244750264 bytes (234 MiB) Everything is Ok Kernel Time = 1.734 = 0% User Time = 1508.609 = 99% Process Time = 1510.343 = 99% Virtual Memory = 10776 MB Global Time = 1510.625 = 100% Physical Memory = 9614 MB C:\Global_Data_Compression_Competition_2020>timer64 "bsc_v3.1.0_x64.exe" e "TS100.txt" "TS100.txt.ST0Block1024.bsc" -m0 -b1024 -e2 -Tt This is bsc, Block Sorting Compressor. Version 3.1.0. 8 July 2012. Copyright (c) 2009-2012 Ilya Grebnov . TS100.txt compressed 1000000000 into 198105016 in 176.921 seconds. Kernel Time = 1.421 = 0% User Time = 175.343 = 99% Process Time = 176.765 = 99% Virtual Memory = 4785 MB Global Time = 176.930 = 100% Physical Memory = 4772 MB C:\Global_Data_Compression_Competition_2020>timer64 BriefLZ_130_Intel_v19_64bit.exe --optimal -b2g "TS100.txt" "TS100.txt.2GB.blz" Kernel Time = 2.937 = 0% User Time = 1530.390 = 99% Process Time = 1533.328 = 99% Virtual Memory = 45403 MB Global Time = 1534.130 = 100% Physical Memory = 21427 MB C:\Global_Data_Compression_Competition_2020>dir "TS100.txt.*"/os Volume in drive C is System Volume Serial Number is 1E2A-3E30 Directory of C:\Global_Data_Compression_Competition_2020 12/07/2020 06:33 194,781,915 TS100.txt.method511.zpaq 12/07/2020 10:21 198,105,016 TS100.txt.ST0Block1024.bsc 12/07/2020 08:19 219,449,490 TS100.txt.1GB.rz 12/07/2020 10:18 244,750,264 TS100.txt.MX9Dict1024.7z 06/09/2020 17:24 246,883,739 TS100.txt.1GB.bro 06/09/2020 17:24 248,823,631 TS100.txt.L22W2GB.zst 12/07/2020 09:53 272,635,725 TS100.txt.MX9.bzip2 12/07/2020 05:46 300,132,758 TS100.txt.method211.zpaq 12/07/2020 10:46 315,985,103 TS100.txt.2GB.blz 12/06/2020 21:34 317,763,629 TS100.txt.Nakamichi 12/07/2020 09:29 346,488,831 TS100.txt.MX9.gzip 06/09/2020 17:24 1,000,000,000 TS100.txt 12 File(s) 3,905,800,101 bytes 0 Dir(s) 1,955,273,162,752 bytes free C:\Global_Data_Compression_Competition_2020>fv "TS100.txt" Reading TS100.txt Drawing fv.bmp 512 by 256 from TS100.txt (1000000000 bytes) Drew part 1 of 4 in 111.07 seconds Drew part 2 of 4 in 111.05 seconds Drew part 3 of 4 in 115.99 seconds Drew part 4 of 4 in 211.71 seconds Created fv.bmp in 549.94 seconds C:\Global_Data_Compression_Competition_2020>copy fv.bmp "fv_TS100.txt.bmp" /y 1 file(s) copied. C:\Global_Data_Compression_Competition_2020>del fv.bmp C:\Global_Data_Compression_Competition_2020>