Use tar 1.32+ or rsync 3.2.3 for faster transfer speeds!

[eva2000]

Heads up for any folks transferring data between servers, that using newer versions of tar 1.31+ (latest is 1.33) and rsync 3.2.3 have support for faster Facebook's zstd compression in tar and zstd/lz4 compression and xxhash checksum algorithms in rsync 3.2.3. Transfer speed boost can potentially be huge.

tested network transfer speed between the servers from USA East Coast to Mid USA was ~40-50MB/s over a 1Gbps network connection due to network and geographical distance. Over an SSH encrypted and zstd compressed netcat connections, I managed to transfer:

• 144GB of file data (uncompressed size) in ~21.8 minutes with Tar + zstd and
• 65GB of MariaDB 10 MySQL data (uncompressed size) in ~8.8 minutes with MariaBackup!

Did some write ups at

• https://blog.centminmod.com/2021/01...eed-for-linux-backups-using-zstd-compression/
• https://community.centminmod.com/th...with-facebook-zstd-compression-support.16243/
• https://community.centminmod.com/th...-with-xxhash-zstd-support-for-centos-7.20768/
• https://community.centminmod.com/th...d-vs-brotli-vs-pigz-vs-bzip2-vs-xz-etc.18669/

Enjoy!

 

[TLChris]

This could be great, especially for some backup situations.

 

[haqzore]

I think I like eva2000, even though I don't understand 94% of the **** they type out.

 

[eva2000]

This could be great, especially for some backup situations.

Yeah I've converted my backup scripts so they can optionally support native tar, rsync gzip but optionally support newer tar, rsync and pigz (multi-threaded gzip), zstd compression

I think I like eva2000, even though I don't understand 94% of the **** they type out.

haha sometimes I forget who I'm conversing with and can only imagine what some not so tech savy folks think of my posts This gist of the above is there are better and faster ways of backing up and transferring data and part of the speed is due to faster compression of files which means smaller compressed files for backup storage and transfer.