Sunday, August 6, 2017

Downloading large files from a server running only SSH/SCP

Another one to add to the eternal bucket of "great simple commands to keep in mind"

rsync --partial --progress --rsh=ssh --archive user@host:/PATH/TO/SOURCE /PATH/TO/DESTINATION

It will even show a nice progress bar ;)

Additionally, assuming you have a bunch of  folders you want to transfer, but they each have thousands of little files. . .

find . -type d -maxdepth 1 -mindepth 1 -exec tar cvf {}.tar {}  \;
find ./*.tar -exec gzip -S .gz -9 {}  \;

Then you can adjust your rsync to download all .gz files