sh recursive copy (cp -r) - How to exclude subfolder

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
7
down vote

favorite
2












I need to run a remote script using ssh via Ruby (net/ssh) to recursively copy a folder and exclude a subfolder. I am looking for the fastest way to do it so rsync is not good. Also, I understand that ssh uses sh and not bash.



In bash I do:



cp -r srcdir/!(subdir) dstdir


and it works fine. However when I launch the script via ssh I receive the error



sh: 1: Syntax error: "(" unexpected


because it is using sh.



I have checked the sh man page, but there is no option to exclude files.



Is it my assumption of ssh using sh correct?
Any alternative suggestion?



EDIT 1:
In case it is useful, the output of sudo cat /etc/shells is the following:



# /etc/shells: valid login shells
/bin/sh
/bin/dash
/bin/bash
/bin/rbash
/usr/bin/tmux
/usr/bin/screen


EDIT 2:
OK. So bash it is available and that does not seems to be the problem. I have verified that the ssh is actually using bash. The issue seems to be related to the escaping of parenthesis or exclamation mark.
I have tried to run the command from the shell (macos) and this is the actual command:



ssh -i .ssh/key.pem ubuntu@X.X.X.X 'mkdir /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N; cp -r /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/!(constant) /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N; ln -s /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/constant /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N/constant'


In this way I receive a different error



cp: cannot stat '/home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/!(constant)': No such file or directory


EDIT 3:
Based on the comments I have changed my command adding extglob



If I use



ssh -i .ssh/key.pem ubuntu@X.X.X.X 'shopt -s extglob; mkdir /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N; cp -r /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/!(constant) /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N; ln -s /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/constant /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N/constant'


I receive the following error:



cp: cannot stat '/home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/!(constant)': No such file or directory


If I do not escape the parenthesis I get 



bash: -c: line 0: syntax error near unexpected token `('



shell ssh cp




share|improve this question


















  • 3




    ssh (well sshd) uses the login shell of the remote user. Could be anything.
    – Stéphane Chazelas
    Aug 19 at 17:09










  • Unix doesn't have folders, only directories. :)
    – tchrist
    Aug 19 at 18:49






  • 1




    In situations like this I often like to just develop the script on the remote host, then either 1) leave it there, ssh in (programmatically if need be) and execute it or 2) if it changes every time, scp it over, execute it via ssh, and then delete it. An extra step maybe, but you don't end up with escaping nightmares and globs expanding locally instead of remotely and all that. Otherwise I would always use heredoc format like @StéphaneChazelas uses below.
    – Josh Rumbut
    Aug 20 at 18:00














up vote
7
down vote

favorite
2












I need to run a remote script using ssh via Ruby (net/ssh) to recursively copy a folder and exclude a subfolder. I am looking for the fastest way to do it so rsync is not good. Also, I understand that ssh uses sh and not bash.



In bash I do:



cp -r srcdir/!(subdir) dstdir


and it works fine. However when I launch the script via ssh I receive the error



sh: 1: Syntax error: "(" unexpected


because it is using sh.



I have checked the sh man page, but there is no option to exclude files.



Is it my assumption of ssh using sh correct?
Any alternative suggestion?



EDIT 1:
In case it is useful, the output of sudo cat /etc/shells is the following:



# /etc/shells: valid login shells
/bin/sh
/bin/dash
/bin/bash
/bin/rbash
/usr/bin/tmux
/usr/bin/screen


EDIT 2:
OK. So bash it is available and that does not seems to be the problem. I have verified that the ssh is actually using bash. The issue seems to be related to the escaping of parenthesis or exclamation mark.
I have tried to run the command from the shell (macos) and this is the actual command:



ssh -i .ssh/key.pem ubuntu@X.X.X.X 'mkdir /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N; cp -r /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/!(constant) /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N; ln -s /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/constant /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N/constant'


In this way I receive a different error



cp: cannot stat '/home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/!(constant)': No such file or directory


EDIT 3:
Based on the comments I have changed my command adding extglob



If I use



ssh -i .ssh/key.pem ubuntu@X.X.X.X 'shopt -s extglob; mkdir /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N; cp -r /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/!(constant) /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N; ln -s /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/constant /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N/constant'


I receive the following error:



cp: cannot stat '/home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/!(constant)': No such file or directory


If I do not escape the parenthesis I get 



bash: -c: line 0: syntax error near unexpected token `('



shell ssh cp




share|improve this question


















  • 3




    ssh (well sshd) uses the login shell of the remote user. Could be anything.
    – Stéphane Chazelas
    Aug 19 at 17:09










  • Unix doesn't have folders, only directories. :)
    – tchrist
    Aug 19 at 18:49






  • 1




    In situations like this I often like to just develop the script on the remote host, then either 1) leave it there, ssh in (programmatically if need be) and execute it or 2) if it changes every time, scp it over, execute it via ssh, and then delete it. An extra step maybe, but you don't end up with escaping nightmares and globs expanding locally instead of remotely and all that. Otherwise I would always use heredoc format like @StéphaneChazelas uses below.
    – Josh Rumbut
    Aug 20 at 18:00












up vote
7
down vote

favorite
2









up vote
7
down vote

favorite
2






2





I need to run a remote script using ssh via Ruby (net/ssh) to recursively copy a folder and exclude a subfolder. I am looking for the fastest way to do it so rsync is not good. Also, I understand that ssh uses sh and not bash.



In bash I do:



cp -r srcdir/!(subdir) dstdir


and it works fine. However when I launch the script via ssh I receive the error



sh: 1: Syntax error: "(" unexpected


because it is using sh.



I have checked the sh man page, but there is no option to exclude files.



Is it my assumption of ssh using sh correct?
Any alternative suggestion?



EDIT 1:
In case it is useful, the output of sudo cat /etc/shells is the following:



# /etc/shells: valid login shells
/bin/sh
/bin/dash
/bin/bash
/bin/rbash
/usr/bin/tmux
/usr/bin/screen


EDIT 2:
OK. So bash it is available and that does not seems to be the problem. I have verified that the ssh is actually using bash. The issue seems to be related to the escaping of parenthesis or exclamation mark.
I have tried to run the command from the shell (macos) and this is the actual command:



ssh -i .ssh/key.pem ubuntu@X.X.X.X 'mkdir /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N; cp -r /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/!(constant) /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N; ln -s /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/constant /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N/constant'


In this way I receive a different error



cp: cannot stat '/home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/!(constant)': No such file or directory


EDIT 3:
Based on the comments I have changed my command adding extglob



If I use



ssh -i .ssh/key.pem ubuntu@X.X.X.X 'shopt -s extglob; mkdir /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N; cp -r /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/!(constant) /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N; ln -s /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/constant /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N/constant'


I receive the following error:



cp: cannot stat '/home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/!(constant)': No such file or directory


If I do not escape the parenthesis I get 



bash: -c: line 0: syntax error near unexpected token `('



shell ssh cp




share|improve this question














I need to run a remote script using ssh via Ruby (net/ssh) to recursively copy a folder and exclude a subfolder. I am looking for the fastest way to do it so rsync is not good. Also, I understand that ssh uses sh and not bash.



In bash I do:



cp -r srcdir/!(subdir) dstdir


and it works fine. However when I launch the script via ssh I receive the error



sh: 1: Syntax error: "(" unexpected


because it is using sh.



I have checked the sh man page, but there is no option to exclude files.



Is it my assumption of ssh using sh correct?
Any alternative suggestion?



EDIT 1:
In case it is useful, the output of sudo cat /etc/shells is the following:



# /etc/shells: valid login shells
/bin/sh
/bin/dash
/bin/bash
/bin/rbash
/usr/bin/tmux
/usr/bin/screen


EDIT 2:
OK. So bash it is available and that does not seems to be the problem. I have verified that the ssh is actually using bash. The issue seems to be related to the escaping of parenthesis or exclamation mark.
I have tried to run the command from the shell (macos) and this is the actual command:



ssh -i .ssh/key.pem ubuntu@X.X.X.X 'mkdir /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N; cp -r /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/!(constant) /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N; ln -s /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/constant /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N/constant'


In this way I receive a different error



cp: cannot stat '/home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/!(constant)': No such file or directory


EDIT 3:
Based on the comments I have changed my command adding extglob



If I use



ssh -i .ssh/key.pem ubuntu@X.X.X.X 'shopt -s extglob; mkdir /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N; cp -r /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/!(constant) /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N; ln -s /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/constant /home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/N/constant'


I receive the following error:



cp: cannot stat '/home/ubuntu/OpenFOAM/ubuntu-4.1/run/LES_New-Area_residuals2/mesh/!(constant)': No such file or directory


If I do not escape the parenthesis I get 



bash: -c: line 0: syntax error near unexpected token `('




shell ssh cp





share|improve this question













share|improve this question




share|improve this question








edited Aug 23 at 11:14

























asked Aug 19 at 11:23









Rojj

1384




1384







  • 3




    ssh (well sshd) uses the login shell of the remote user. Could be anything.
    – Stéphane Chazelas
    Aug 19 at 17:09










  • Unix doesn't have folders, only directories. :)
    – tchrist
    Aug 19 at 18:49






  • 1




    In situations like this I often like to just develop the script on the remote host, then either 1) leave it there, ssh in (programmatically if need be) and execute it or 2) if it changes every time, scp it over, execute it via ssh, and then delete it. An extra step maybe, but you don't end up with escaping nightmares and globs expanding locally instead of remotely and all that. Otherwise I would always use heredoc format like @StéphaneChazelas uses below.
    – Josh Rumbut
    Aug 20 at 18:00












  • 3




    ssh (well sshd) uses the login shell of the remote user. Could be anything.
    – Stéphane Chazelas
    Aug 19 at 17:09










  • Unix doesn't have folders, only directories. :)
    – tchrist
    Aug 19 at 18:49






  • 1




    In situations like this I often like to just develop the script on the remote host, then either 1) leave it there, ssh in (programmatically if need be) and execute it or 2) if it changes every time, scp it over, execute it via ssh, and then delete it. An extra step maybe, but you don't end up with escaping nightmares and globs expanding locally instead of remotely and all that. Otherwise I would always use heredoc format like @StéphaneChazelas uses below.
    – Josh Rumbut
    Aug 20 at 18:00







3




3




ssh (well sshd) uses the login shell of the remote user. Could be anything.
– Stéphane Chazelas
Aug 19 at 17:09




ssh (well sshd) uses the login shell of the remote user. Could be anything.
– Stéphane Chazelas
Aug 19 at 17:09












Unix doesn't have folders, only directories. :)
– tchrist
Aug 19 at 18:49




Unix doesn't have folders, only directories. :)
– tchrist
Aug 19 at 18:49




1




1




In situations like this I often like to just develop the script on the remote host, then either 1) leave it there, ssh in (programmatically if need be) and execute it or 2) if it changes every time, scp it over, execute it via ssh, and then delete it. An extra step maybe, but you don't end up with escaping nightmares and globs expanding locally instead of remotely and all that. Otherwise I would always use heredoc format like @StéphaneChazelas uses below.
– Josh Rumbut
Aug 20 at 18:00




In situations like this I often like to just develop the script on the remote host, then either 1) leave it there, ssh in (programmatically if need be) and execute it or 2) if it changes every time, scp it over, execute it via ssh, and then delete it. An extra step maybe, but you don't end up with escaping nightmares and globs expanding locally instead of remotely and all that. Otherwise I would always use heredoc format like @StéphaneChazelas uses below.
– Josh Rumbut
Aug 20 at 18:00










6 Answers
6






active

oldest

votes

















up vote
10
down vote



accepted










SSH runs your login shell on the remote system, whatever that is. But !(foo) requires shopt -s extglob, which you might not have set on the remote.



Try this to see if SSH runs Bash on the remote side: 



ssh me@somehost 'echo "$BASH_VERSION"'


If that prints anything, but your startup scripts don't set extglob, you can do it by hand on the command passed to ssh:



ssh me@somehost 'shopt -s extglob
 echo srcdir/!(subdir)' 
 # or
ssh me@somehost $'shopt -s extglobn echo srcdir/!(subdir)'


extglob affects the parsing of the command line, and only takes effect after a newline, so we have to put a literal newline there, a semicolon isn't enough.




ssh me@somehost 'shopt -s extglob; echo srcdir/!(subdir)'


Also not that if you escape the parenthesis with backslashes, they lose their special properties, like any other glob characters. This is not what you want to do in this case.



$ touch foo bar; shopt -s extglob; set +o histexpand
$ echo *
bar foo
$ echo !(foo)
bar
$ echo *
*
$ echo !(foo)
!(foo)





share|improve this answer





























    up vote
    10
    down vote













    I don't know why you think that rsync would be slow. The speed of a copy is mostly determined by the speed of the disk. Rsync has many options to specify what you want included and excluded, so it gives you much better control than shell globbing.



    As the bash manual states, the !(patter) is only recognized in bash if extglob is set. In your example you didn't set extglob. Further, a bash started as sh is still bash, but will disable some extensions for compatibility.



    The SSH server will start the user's login shell, as specified in /etc/passwd. You can either change the shell, or use that shell to start another shell that fits your needs better.






    share|improve this answer




















    • I tested with time. time cp -r mesh/!(constant) N -> real 1.04s and time rsync -a mesh/ N --exclude=constant -> real 1.8s
      – Rojj
      Aug 19 at 12:49







    • 7




      @Rojj that’s apples to oranges comparison. For one thing, you’re using -a for rsync but not for cp. That involves preservation of permissions and other attributes, so you’re not actually doing the same thing.
      – Wildcard
      Aug 19 at 19:17


















    up vote
    6
    down vote













    A few notes first:



    • the ssh server doesn't start sh to interpret the command line sent by the client, it runs the login shell of the user on the remote host, as that-shell -c <the-string-provided-by-the-client>. The login shell of the remote user could be anything. Bear in mind that some shells like tcsh, fish or rc have very different syntax from that of sh.

    • it is really a command line, or more exactly a string (that can contain newline characters, so several lines). Even if you do ssh host cmd arg1 'arg 2' where cmd, arg1 and arg 2 are three arguments passed to ssh, ssh concatenates those arguments with spaces and actually sends the cmd arg1 arg 2 string to sshd, and the remote shell would split that into cmd, arg1, arg and 2.


    • !(subdir) is a glob operator (a ksh glob operator also supported by zsh -o kshglob and bash -O extglob). Like all globs, it excludes hidden files, so beware there may be other files that it excludes.

    Here, to avoid the problem with finding out the right syntax for the remote shell, you can actually tell that other shell to start the shell you want and feed it the code via stdin (one of the options listed at How to execute an arbitrary simple command over ssh without knowing the login shell of the remote user?)



    ssh host 'bash -O extglob -O dotglob' << 'EOF'
    cp -r srcdir/!(subdir) dstdir/
    EOF


    bash -O extglob -O dotglob is a command line that is understood the same by all major shells, including Bourne-like ones, csh, rc, fish... The above would work as long as bash is installed and is in the user's $PATH (default $PATH, possibly modified by the user's login shell like with ~/.zshenv for zsh, ~/.cshrc for csh, ~/.bashrc for bash).



    POSIXly (though in practice, you may find that more systems have a bash command than a pax command), you could do:



    ssh host sh << 'EOF'
    cd srcdir && pax -rw -'s|^.//./subdir(/.*)0,1$||' .//. /path/to/destdir/
    EOF


    -s applies substitutions to the paths being transferred. When that substitution expands to nothing, the file is excluded. The problem is that substitutions also apply to target of symlinks. That's why we use .//. above to make it less likely that a symlink be affected.






    share|improve this answer





























      up vote
      4
      down vote













      I don't think ssh is limited to using sh. It rather depends on what is installed on the target system, how the user is set up, and what shells are allowed in /etc/shells.



      Did you consider the chsh command?






      share|improve this answer





























        up vote
        4
        down vote













        If you want to do it in a fast way, you can look at rsync with a different encryption algorithm. This gives you the option to easily exclude etc., at not much speed sacrifice.



        rsync -aHAXxv --numeric-ids --progress -e "ssh -T -c arcfour -o Compression=no -x" user@<source>:<source_dir> <dest_dir>



        together with adding the arcfour encryption to the line starting with Ciphers in /etc/ssh/ssh_config, if not already enabled, gives you an acceptable speed.



        WARNING: The arcfour encryption is insecure. Do NOT run this over insecure channels. If you are concerned about access to the server from insecure channels using arcfour encryption, change the etc/ssh/ssh_config with a host-specific part for your source host - Create a Host section in your ssh_config for your source host, you can use Ciphers arcfour there to mirror the above -c switch, which restricts arcfour encryption to this host only.



        For details, refer to ssh_config man pages.



        However, if your CPUs support the AES-NI instruction set, try switching to aes128-gcm@openssh.com (yes, that's the cipher name, including the @ stuff), which will use the blazingly fast (with AES-NI) AES128-GCM.



        So, with a CPU supporting AES-NI, change "ssh -T -c arcfour -o Compression=no -x" to "ssh -T -c aes128-gcm@openssh.com -o Compression=no -x" for more secure results.



        Explanation



        rsync



        • (Don't use -z, it is much slower)


        • a: archive mode - rescursive, preserves owner, preserves permissions, preserves modification times, preserves group, copies symlinks as symlinks, preserves device files.


        • H: preserves hard-links


        • A: preserves ACLs


        • X: preserves extended attributes


        • x: don't cross file-system boundaries


        • v: increase verbosity


        • --numeric-ds: don't map uid/gid values by user/group name

        • if you need to sync, add --delete: delete extraneous files from dest dirs (differential clean-up during sync)


        • --progress: show progress during transfer

        ssh




        • T: turn off pseudo-tty to decrease cpu load on destination.


        • c arcfour: use the weakest but fastest SSH encryption. Must specify "Ciphers arcfour" in sshd_config on destination.


        • o Compression=no: Turn off SSH compression.


        • x: turn off X forwarding if it is on by default.

        The beef is in the ssh options - if you just use rsync -av and the -e ssh -T -c arcfour -o Compression=no -x" part, you can get these speeds as well.



        Comparison:



        • 13.6 MB/s rsync -az

        • 16.7 MB/s scp -Cr

        • 44.8 MB/s rsync -a

        • 59.8 MB/s sftp

        • 61.2 MB/s scp -r

        • 61.4 MB/s sftp -R 128 -B 65536

        • 62.4 MB/s rsync -a -P -e "ssh -T -c arcfour -o Compression=no -x"

        • 143.5 MB/s scp -r -c arcfour

        • 144.2 MB/s sftp -oCiphers=arcfour

        Sources:



        https://gist.github.com/KartikTalwar/4393116



        http://nz2nz.blogspot.com/2018/05/rsync-scp-sftp-speed-test.html






        share|improve this answer


















        • 2




          Well, they seem to be running cp -r within the remote system, so the encryption used by the SSH connection isn't really relevant. In any case arcfour is considered rather broken and OpenSSH disables it along with others on the server by default since version 6.7 (2014-10-06). In any case, ssh -o Ciphers='aes128-ctr' gives me about 90 MB/s, which should be fast enough on a 1 Gbit/s link.
          – ilkkachu
          Aug 19 at 13:13











        • Yes, arcfour is broken, but it's not supposed to be SECURE shell for this case, but more 'comfortable shell' with no emphasis on the encryption. I wouldn't use this over insecure connections, that's correct. If 'aes128-ctr' is fast enough, it can and should be used instead.
          – emk2203
          Aug 19 at 15:23










        • See also my extended answer for usage with CPUs which support AES-NI.
          – emk2203
          Aug 19 at 16:04

















        up vote
        2
        down vote













        As per my calculations, the fastest full copy is always using 'tar' (here assuming GNU tar or compatible).



        mkdir -p photos2 &&
         tar -C photos -cf - --exclude=./.thumbcache . |
         tar -C photos2 -xpf -


        And tar has ton of options to manipulate attributes, permissions and file selection/exclusion. For example, the above command excludes the top level subfolder called .thumbcache while copying.






        share|improve this answer






















        • Note that --exclude=.thumbcache excludes all the .thumbcache files, not only the one at the top-level. With GNU tar (not bsdtar), you can use --exclude=./.thumbcache to only exclude the top-level .thumbcache file.
          – Stéphane Chazelas
          Aug 20 at 14:39










        Your Answer







        StackExchange.ready(function()
        var channelOptions =
        tags: "".split(" "),
        id: "106"
        ;
        initTagRenderer("".split(" "), "".split(" "), channelOptions);

        StackExchange.using("externalEditor", function()
        // Have to fire editor after snippets, if snippets enabled
        if (StackExchange.settings.snippets.snippetsEnabled)
        StackExchange.using("snippets", function()
        createEditor();
        );

        else
        createEditor();

        );

        function createEditor()
        StackExchange.prepareEditor(
        heartbeatType: 'answer',
        convertImagesToLinks: false,
        noModals: false,
        showLowRepImageUploadWarning: true,
        reputationToPostImages: null,
        bindNavPrevention: true,
        postfix: "",
        onDemand: true,
        discardSelector: ".discard-answer"
        ,immediatelyShowMarkdownHelp:true
        );



        );













         

        draft saved


        draft discarded


















        StackExchange.ready(
        function ()
        StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f463477%2fsh-recursive-copy-cp-r-how-to-exclude-subfolder%23new-answer', 'question_page');

        );

        Post as a guest

















        6 Answers
        6






        active

        oldest

        votes








        6 Answers
        6






        active

        oldest

        votes









        active

        oldest

        votes






        active

        oldest

        votes








        up vote
        10
        down vote



        accepted










        SSH runs your login shell on the remote system, whatever that is. But !(foo) requires shopt -s extglob, which you might not have set on the remote.



        Try this to see if SSH runs Bash on the remote side: 



        ssh me@somehost 'echo "$BASH_VERSION"'


        If that prints anything, but your startup scripts don't set extglob, you can do it by hand on the command passed to ssh:



        ssh me@somehost 'shopt -s extglob
         echo srcdir/!(subdir)' 
         # or
        ssh me@somehost $'shopt -s extglobn echo srcdir/!(subdir)'


        extglob affects the parsing of the command line, and only takes effect after a newline, so we have to put a literal newline there, a semicolon isn't enough.




        ssh me@somehost 'shopt -s extglob; echo srcdir/!(subdir)'


        Also not that if you escape the parenthesis with backslashes, they lose their special properties, like any other glob characters. This is not what you want to do in this case.



        $ touch foo bar; shopt -s extglob; set +o histexpand
        $ echo *
        bar foo
        $ echo !(foo)
        bar
        $ echo *
        *
        $ echo !(foo)
        !(foo)





        share|improve this answer


























          up vote
          10
          down vote



          accepted










          SSH runs your login shell on the remote system, whatever that is. But !(foo) requires shopt -s extglob, which you might not have set on the remote.



          Try this to see if SSH runs Bash on the remote side: 



          ssh me@somehost 'echo "$BASH_VERSION"'


          If that prints anything, but your startup scripts don't set extglob, you can do it by hand on the command passed to ssh:



          ssh me@somehost 'shopt -s extglob
           echo srcdir/!(subdir)' 
           # or
          ssh me@somehost $'shopt -s extglobn echo srcdir/!(subdir)'


          extglob affects the parsing of the command line, and only takes effect after a newline, so we have to put a literal newline there, a semicolon isn't enough.




          ssh me@somehost 'shopt -s extglob; echo srcdir/!(subdir)'


          Also not that if you escape the parenthesis with backslashes, they lose their special properties, like any other glob characters. This is not what you want to do in this case.



          $ touch foo bar; shopt -s extglob; set +o histexpand
          $ echo *
          bar foo
          $ echo !(foo)
          bar
          $ echo *
          *
          $ echo !(foo)
          !(foo)





          share|improve this answer
























            up vote
            10
            down vote



            accepted







            up vote
            10
            down vote



            accepted






            SSH runs your login shell on the remote system, whatever that is. But !(foo) requires shopt -s extglob, which you might not have set on the remote.



            Try this to see if SSH runs Bash on the remote side: 



            ssh me@somehost 'echo "$BASH_VERSION"'


            If that prints anything, but your startup scripts don't set extglob, you can do it by hand on the command passed to ssh:



            ssh me@somehost 'shopt -s extglob
             echo srcdir/!(subdir)' 
             # or
            ssh me@somehost $'shopt -s extglobn echo srcdir/!(subdir)'


            extglob affects the parsing of the command line, and only takes effect after a newline, so we have to put a literal newline there, a semicolon isn't enough.




            ssh me@somehost 'shopt -s extglob; echo srcdir/!(subdir)'


            Also not that if you escape the parenthesis with backslashes, they lose their special properties, like any other glob characters. This is not what you want to do in this case.



            $ touch foo bar; shopt -s extglob; set +o histexpand
            $ echo *
            bar foo
            $ echo !(foo)
            bar
            $ echo *
            *
            $ echo !(foo)
            !(foo)





            share|improve this answer














            SSH runs your login shell on the remote system, whatever that is. But !(foo) requires shopt -s extglob, which you might not have set on the remote.



            Try this to see if SSH runs Bash on the remote side: 



            ssh me@somehost 'echo "$BASH_VERSION"'


            If that prints anything, but your startup scripts don't set extglob, you can do it by hand on the command passed to ssh:



            ssh me@somehost 'shopt -s extglob
             echo srcdir/!(subdir)' 
             # or
            ssh me@somehost $'shopt -s extglobn echo srcdir/!(subdir)'


            extglob affects the parsing of the command line, and only takes effect after a newline, so we have to put a literal newline there, a semicolon isn't enough.




            ssh me@somehost 'shopt -s extglob; echo srcdir/!(subdir)'


            Also not that if you escape the parenthesis with backslashes, they lose their special properties, like any other glob characters. This is not what you want to do in this case.



            $ touch foo bar; shopt -s extglob; set +o histexpand
            $ echo *
            bar foo
            $ echo !(foo)
            bar
            $ echo *
            *
            $ echo !(foo)
            !(foo)






            share|improve this answer














            share|improve this answer



            share|improve this answer








            edited Aug 19 at 16:11

























            answered Aug 19 at 12:24









            ilkkachu

            50.3k677138




            50.3k677138






















                up vote
                10
                down vote













                I don't know why you think that rsync would be slow. The speed of a copy is mostly determined by the speed of the disk. Rsync has many options to specify what you want included and excluded, so it gives you much better control than shell globbing.



                As the bash manual states, the !(patter) is only recognized in bash if extglob is set. In your example you didn't set extglob. Further, a bash started as sh is still bash, but will disable some extensions for compatibility.



                The SSH server will start the user's login shell, as specified in /etc/passwd. You can either change the shell, or use that shell to start another shell that fits your needs better.






                share|improve this answer




















                • I tested with time. time cp -r mesh/!(constant) N -> real 1.04s and time rsync -a mesh/ N --exclude=constant -> real 1.8s
                  – Rojj
                  Aug 19 at 12:49







                • 7




                  @Rojj that’s apples to oranges comparison. For one thing, you’re using -a for rsync but not for cp. That involves preservation of permissions and other attributes, so you’re not actually doing the same thing.
                  – Wildcard
                  Aug 19 at 19:17















                up vote
                10
                down vote













                I don't know why you think that rsync would be slow. The speed of a copy is mostly determined by the speed of the disk. Rsync has many options to specify what you want included and excluded, so it gives you much better control than shell globbing.



                As the bash manual states, the !(patter) is only recognized in bash if extglob is set. In your example you didn't set extglob. Further, a bash started as sh is still bash, but will disable some extensions for compatibility.



                The SSH server will start the user's login shell, as specified in /etc/passwd. You can either change the shell, or use that shell to start another shell that fits your needs better.






                share|improve this answer




















                • I tested with time. time cp -r mesh/!(constant) N -> real 1.04s and time rsync -a mesh/ N --exclude=constant -> real 1.8s
                  – Rojj
                  Aug 19 at 12:49







                • 7




                  @Rojj that’s apples to oranges comparison. For one thing, you’re using -a for rsync but not for cp. That involves preservation of permissions and other attributes, so you’re not actually doing the same thing.
                  – Wildcard
                  Aug 19 at 19:17













                up vote
                10
                down vote










                up vote
                10
                down vote









                I don't know why you think that rsync would be slow. The speed of a copy is mostly determined by the speed of the disk. Rsync has many options to specify what you want included and excluded, so it gives you much better control than shell globbing.



                As the bash manual states, the !(patter) is only recognized in bash if extglob is set. In your example you didn't set extglob. Further, a bash started as sh is still bash, but will disable some extensions for compatibility.



                The SSH server will start the user's login shell, as specified in /etc/passwd. You can either change the shell, or use that shell to start another shell that fits your needs better.






                share|improve this answer












                I don't know why you think that rsync would be slow. The speed of a copy is mostly determined by the speed of the disk. Rsync has many options to specify what you want included and excluded, so it gives you much better control than shell globbing.



                As the bash manual states, the !(patter) is only recognized in bash if extglob is set. In your example you didn't set extglob. Further, a bash started as sh is still bash, but will disable some extensions for compatibility.



                The SSH server will start the user's login shell, as specified in /etc/passwd. You can either change the shell, or use that shell to start another shell that fits your needs better.







                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered Aug 19 at 12:18









                RalfFriedl

                3,0701522




                3,0701522











                • I tested with time. time cp -r mesh/!(constant) N -> real 1.04s and time rsync -a mesh/ N --exclude=constant -> real 1.8s
                  – Rojj
                  Aug 19 at 12:49







                • 7




                  @Rojj that’s apples to oranges comparison. For one thing, you’re using -a for rsync but not for cp. That involves preservation of permissions and other attributes, so you’re not actually doing the same thing.
                  – Wildcard
                  Aug 19 at 19:17

















                • I tested with time. time cp -r mesh/!(constant) N -> real 1.04s and time rsync -a mesh/ N --exclude=constant -> real 1.8s
                  – Rojj
                  Aug 19 at 12:49







                • 7




                  @Rojj that’s apples to oranges comparison. For one thing, you’re using -a for rsync but not for cp. That involves preservation of permissions and other attributes, so you’re not actually doing the same thing.
                  – Wildcard
                  Aug 19 at 19:17
















                I tested with time. time cp -r mesh/!(constant) N -> real 1.04s and time rsync -a mesh/ N --exclude=constant -> real 1.8s
                – Rojj
                Aug 19 at 12:49





                I tested with time. time cp -r mesh/!(constant) N -> real 1.04s and time rsync -a mesh/ N --exclude=constant -> real 1.8s
                – Rojj
                Aug 19 at 12:49





                7




                7




                @Rojj that’s apples to oranges comparison. For one thing, you’re using -a for rsync but not for cp. That involves preservation of permissions and other attributes, so you’re not actually doing the same thing.
                – Wildcard
                Aug 19 at 19:17





                @Rojj that’s apples to oranges comparison. For one thing, you’re using -a for rsync but not for cp. That involves preservation of permissions and other attributes, so you’re not actually doing the same thing.
                – Wildcard
                Aug 19 at 19:17











                up vote
                6
                down vote













                A few notes first:



                • the ssh server doesn't start sh to interpret the command line sent by the client, it runs the login shell of the user on the remote host, as that-shell -c <the-string-provided-by-the-client>. The login shell of the remote user could be anything. Bear in mind that some shells like tcsh, fish or rc have very different syntax from that of sh.

                • it is really a command line, or more exactly a string (that can contain newline characters, so several lines). Even if you do ssh host cmd arg1 'arg 2' where cmd, arg1 and arg 2 are three arguments passed to ssh, ssh concatenates those arguments with spaces and actually sends the cmd arg1 arg 2 string to sshd, and the remote shell would split that into cmd, arg1, arg and 2.


                • !(subdir) is a glob operator (a ksh glob operator also supported by zsh -o kshglob and bash -O extglob). Like all globs, it excludes hidden files, so beware there may be other files that it excludes.

                Here, to avoid the problem with finding out the right syntax for the remote shell, you can actually tell that other shell to start the shell you want and feed it the code via stdin (one of the options listed at How to execute an arbitrary simple command over ssh without knowing the login shell of the remote user?)



                ssh host 'bash -O extglob -O dotglob' << 'EOF'
                cp -r srcdir/!(subdir) dstdir/
                EOF


                bash -O extglob -O dotglob is a command line that is understood the same by all major shells, including Bourne-like ones, csh, rc, fish... The above would work as long as bash is installed and is in the user's $PATH (default $PATH, possibly modified by the user's login shell like with ~/.zshenv for zsh, ~/.cshrc for csh, ~/.bashrc for bash).



                POSIXly (though in practice, you may find that more systems have a bash command than a pax command), you could do:



                ssh host sh << 'EOF'
                cd srcdir && pax -rw -'s|^.//./subdir(/.*)0,1$||' .//. /path/to/destdir/
                EOF


                -s applies substitutions to the paths being transferred. When that substitution expands to nothing, the file is excluded. The problem is that substitutions also apply to target of symlinks. That's why we use .//. above to make it less likely that a symlink be affected.






                share|improve this answer


























                  up vote
                  6
                  down vote













                  A few notes first:



                  • the ssh server doesn't start sh to interpret the command line sent by the client, it runs the login shell of the user on the remote host, as that-shell -c <the-string-provided-by-the-client>. The login shell of the remote user could be anything. Bear in mind that some shells like tcsh, fish or rc have very different syntax from that of sh.

                  • it is really a command line, or more exactly a string (that can contain newline characters, so several lines). Even if you do ssh host cmd arg1 'arg 2' where cmd, arg1 and arg 2 are three arguments passed to ssh, ssh concatenates those arguments with spaces and actually sends the cmd arg1 arg 2 string to sshd, and the remote shell would split that into cmd, arg1, arg and 2.


                  • !(subdir) is a glob operator (a ksh glob operator also supported by zsh -o kshglob and bash -O extglob). Like all globs, it excludes hidden files, so beware there may be other files that it excludes.

                  Here, to avoid the problem with finding out the right syntax for the remote shell, you can actually tell that other shell to start the shell you want and feed it the code via stdin (one of the options listed at How to execute an arbitrary simple command over ssh without knowing the login shell of the remote user?)



                  ssh host 'bash -O extglob -O dotglob' << 'EOF'
                  cp -r srcdir/!(subdir) dstdir/
                  EOF


                  bash -O extglob -O dotglob is a command line that is understood the same by all major shells, including Bourne-like ones, csh, rc, fish... The above would work as long as bash is installed and is in the user's $PATH (default $PATH, possibly modified by the user's login shell like with ~/.zshenv for zsh, ~/.cshrc for csh, ~/.bashrc for bash).



                  POSIXly (though in practice, you may find that more systems have a bash command than a pax command), you could do:



                  ssh host sh << 'EOF'
                  cd srcdir && pax -rw -'s|^.//./subdir(/.*)0,1$||' .//. /path/to/destdir/
                  EOF


                  -s applies substitutions to the paths being transferred. When that substitution expands to nothing, the file is excluded. The problem is that substitutions also apply to target of symlinks. That's why we use .//. above to make it less likely that a symlink be affected.






                  share|improve this answer
























                    up vote
                    6
                    down vote










                    up vote
                    6
                    down vote









                    A few notes first:



                    • the ssh server doesn't start sh to interpret the command line sent by the client, it runs the login shell of the user on the remote host, as that-shell -c <the-string-provided-by-the-client>. The login shell of the remote user could be anything. Bear in mind that some shells like tcsh, fish or rc have very different syntax from that of sh.

                    • it is really a command line, or more exactly a string (that can contain newline characters, so several lines). Even if you do ssh host cmd arg1 'arg 2' where cmd, arg1 and arg 2 are three arguments passed to ssh, ssh concatenates those arguments with spaces and actually sends the cmd arg1 arg 2 string to sshd, and the remote shell would split that into cmd, arg1, arg and 2.


                    • !(subdir) is a glob operator (a ksh glob operator also supported by zsh -o kshglob and bash -O extglob). Like all globs, it excludes hidden files, so beware there may be other files that it excludes.

                    Here, to avoid the problem with finding out the right syntax for the remote shell, you can actually tell that other shell to start the shell you want and feed it the code via stdin (one of the options listed at How to execute an arbitrary simple command over ssh without knowing the login shell of the remote user?)



                    ssh host 'bash -O extglob -O dotglob' << 'EOF'
                    cp -r srcdir/!(subdir) dstdir/
                    EOF


                    bash -O extglob -O dotglob is a command line that is understood the same by all major shells, including Bourne-like ones, csh, rc, fish... The above would work as long as bash is installed and is in the user's $PATH (default $PATH, possibly modified by the user's login shell like with ~/.zshenv for zsh, ~/.cshrc for csh, ~/.bashrc for bash).



                    POSIXly (though in practice, you may find that more systems have a bash command than a pax command), you could do:



                    ssh host sh << 'EOF'
                    cd srcdir && pax -rw -'s|^.//./subdir(/.*)0,1$||' .//. /path/to/destdir/
                    EOF


                    -s applies substitutions to the paths being transferred. When that substitution expands to nothing, the file is excluded. The problem is that substitutions also apply to target of symlinks. That's why we use .//. above to make it less likely that a symlink be affected.






                    share|improve this answer














                    A few notes first:



                    • the ssh server doesn't start sh to interpret the command line sent by the client, it runs the login shell of the user on the remote host, as that-shell -c <the-string-provided-by-the-client>. The login shell of the remote user could be anything. Bear in mind that some shells like tcsh, fish or rc have very different syntax from that of sh.

                    • it is really a command line, or more exactly a string (that can contain newline characters, so several lines). Even if you do ssh host cmd arg1 'arg 2' where cmd, arg1 and arg 2 are three arguments passed to ssh, ssh concatenates those arguments with spaces and actually sends the cmd arg1 arg 2 string to sshd, and the remote shell would split that into cmd, arg1, arg and 2.


                    • !(subdir) is a glob operator (a ksh glob operator also supported by zsh -o kshglob and bash -O extglob). Like all globs, it excludes hidden files, so beware there may be other files that it excludes.

                    Here, to avoid the problem with finding out the right syntax for the remote shell, you can actually tell that other shell to start the shell you want and feed it the code via stdin (one of the options listed at How to execute an arbitrary simple command over ssh without knowing the login shell of the remote user?)



                    ssh host 'bash -O extglob -O dotglob' << 'EOF'
                    cp -r srcdir/!(subdir) dstdir/
                    EOF


                    bash -O extglob -O dotglob is a command line that is understood the same by all major shells, including Bourne-like ones, csh, rc, fish... The above would work as long as bash is installed and is in the user's $PATH (default $PATH, possibly modified by the user's login shell like with ~/.zshenv for zsh, ~/.cshrc for csh, ~/.bashrc for bash).



                    POSIXly (though in practice, you may find that more systems have a bash command than a pax command), you could do:



                    ssh host sh << 'EOF'
                    cd srcdir && pax -rw -'s|^.//./subdir(/.*)0,1$||' .//. /path/to/destdir/
                    EOF


                    -s applies substitutions to the paths being transferred. When that substitution expands to nothing, the file is excluded. The problem is that substitutions also apply to target of symlinks. That's why we use .//. above to make it less likely that a symlink be affected.







                    share|improve this answer














                    share|improve this answer



                    share|improve this answer








                    edited Aug 20 at 19:20

























                    answered Aug 19 at 17:43









                    Stéphane Chazelas

                    282k53521854




                    282k53521854




















                        up vote
                        4
                        down vote













                        I don't think ssh is limited to using sh. It rather depends on what is installed on the target system, how the user is set up, and what shells are allowed in /etc/shells.



                        Did you consider the chsh command?






                        share|improve this answer


























                          up vote
                          4
                          down vote













                          I don't think ssh is limited to using sh. It rather depends on what is installed on the target system, how the user is set up, and what shells are allowed in /etc/shells.



                          Did you consider the chsh command?






                          share|improve this answer
























                            up vote
                            4
                            down vote










                            up vote
                            4
                            down vote









                            I don't think ssh is limited to using sh. It rather depends on what is installed on the target system, how the user is set up, and what shells are allowed in /etc/shells.



                            Did you consider the chsh command?






                            share|improve this answer














                            I don't think ssh is limited to using sh. It rather depends on what is installed on the target system, how the user is set up, and what shells are allowed in /etc/shells.



                            Did you consider the chsh command?







                            share|improve this answer














                            share|improve this answer



                            share|improve this answer








                            edited Aug 19 at 13:08









                            ephsmith

                            5242717




                            5242717










                            answered Aug 19 at 11:28









                            RudiC

                            1,1516




                            1,1516




















                                up vote
                                4
                                down vote













                                If you want to do it in a fast way, you can look at rsync with a different encryption algorithm. This gives you the option to easily exclude etc., at not much speed sacrifice.



                                rsync -aHAXxv --numeric-ids --progress -e "ssh -T -c arcfour -o Compression=no -x" user@<source>:<source_dir> <dest_dir>



                                together with adding the arcfour encryption to the line starting with Ciphers in /etc/ssh/ssh_config, if not already enabled, gives you an acceptable speed.



                                WARNING: The arcfour encryption is insecure. Do NOT run this over insecure channels. If you are concerned about access to the server from insecure channels using arcfour encryption, change the etc/ssh/ssh_config with a host-specific part for your source host - Create a Host section in your ssh_config for your source host, you can use Ciphers arcfour there to mirror the above -c switch, which restricts arcfour encryption to this host only.



                                For details, refer to ssh_config man pages.



                                However, if your CPUs support the AES-NI instruction set, try switching to aes128-gcm@openssh.com (yes, that's the cipher name, including the @ stuff), which will use the blazingly fast (with AES-NI) AES128-GCM.



                                So, with a CPU supporting AES-NI, change "ssh -T -c arcfour -o Compression=no -x" to "ssh -T -c aes128-gcm@openssh.com -o Compression=no -x" for more secure results.



                                Explanation



                                rsync



                                • (Don't use -z, it is much slower)


                                • a: archive mode - rescursive, preserves owner, preserves permissions, preserves modification times, preserves group, copies symlinks as symlinks, preserves device files.


                                • H: preserves hard-links


                                • A: preserves ACLs


                                • X: preserves extended attributes


                                • x: don't cross file-system boundaries


                                • v: increase verbosity


                                • --numeric-ds: don't map uid/gid values by user/group name

                                • if you need to sync, add --delete: delete extraneous files from dest dirs (differential clean-up during sync)


                                • --progress: show progress during transfer

                                ssh




                                • T: turn off pseudo-tty to decrease cpu load on destination.


                                • c arcfour: use the weakest but fastest SSH encryption. Must specify "Ciphers arcfour" in sshd_config on destination.


                                • o Compression=no: Turn off SSH compression.


                                • x: turn off X forwarding if it is on by default.

                                The beef is in the ssh options - if you just use rsync -av and the -e ssh -T -c arcfour -o Compression=no -x" part, you can get these speeds as well.



                                Comparison:



                                • 13.6 MB/s rsync -az

                                • 16.7 MB/s scp -Cr

                                • 44.8 MB/s rsync -a

                                • 59.8 MB/s sftp

                                • 61.2 MB/s scp -r

                                • 61.4 MB/s sftp -R 128 -B 65536

                                • 62.4 MB/s rsync -a -P -e "ssh -T -c arcfour -o Compression=no -x"

                                • 143.5 MB/s scp -r -c arcfour

                                • 144.2 MB/s sftp -oCiphers=arcfour

                                Sources:



                                https://gist.github.com/KartikTalwar/4393116



                                http://nz2nz.blogspot.com/2018/05/rsync-scp-sftp-speed-test.html






                                share|improve this answer


















                                • 2




                                  Well, they seem to be running cp -r within the remote system, so the encryption used by the SSH connection isn't really relevant. In any case arcfour is considered rather broken and OpenSSH disables it along with others on the server by default since version 6.7 (2014-10-06). In any case, ssh -o Ciphers='aes128-ctr' gives me about 90 MB/s, which should be fast enough on a 1 Gbit/s link.
                                  – ilkkachu
                                  Aug 19 at 13:13











                                • Yes, arcfour is broken, but it's not supposed to be SECURE shell for this case, but more 'comfortable shell' with no emphasis on the encryption. I wouldn't use this over insecure connections, that's correct. If 'aes128-ctr' is fast enough, it can and should be used instead.
                                  – emk2203
                                  Aug 19 at 15:23










                                • See also my extended answer for usage with CPUs which support AES-NI.
                                  – emk2203
                                  Aug 19 at 16:04














                                up vote
                                4
                                down vote













                                If you want to do it in a fast way, you can look at rsync with a different encryption algorithm. This gives you the option to easily exclude etc., at not much speed sacrifice.



                                rsync -aHAXxv --numeric-ids --progress -e "ssh -T -c arcfour -o Compression=no -x" user@<source>:<source_dir> <dest_dir>



                                together with adding the arcfour encryption to the line starting with Ciphers in /etc/ssh/ssh_config, if not already enabled, gives you an acceptable speed.



                                WARNING: The arcfour encryption is insecure. Do NOT run this over insecure channels. If you are concerned about access to the server from insecure channels using arcfour encryption, change the etc/ssh/ssh_config with a host-specific part for your source host - Create a Host section in your ssh_config for your source host, you can use Ciphers arcfour there to mirror the above -c switch, which restricts arcfour encryption to this host only.



                                For details, refer to ssh_config man pages.



                                However, if your CPUs support the AES-NI instruction set, try switching to aes128-gcm@openssh.com (yes, that's the cipher name, including the @ stuff), which will use the blazingly fast (with AES-NI) AES128-GCM.



                                So, with a CPU supporting AES-NI, change "ssh -T -c arcfour -o Compression=no -x" to "ssh -T -c aes128-gcm@openssh.com -o Compression=no -x" for more secure results.



                                Explanation



                                rsync



                                • (Don't use -z, it is much slower)


                                • a: archive mode - rescursive, preserves owner, preserves permissions, preserves modification times, preserves group, copies symlinks as symlinks, preserves device files.


                                • H: preserves hard-links


                                • A: preserves ACLs


                                • X: preserves extended attributes


                                • x: don't cross file-system boundaries


                                • v: increase verbosity


                                • --numeric-ds: don't map uid/gid values by user/group name

                                • if you need to sync, add --delete: delete extraneous files from dest dirs (differential clean-up during sync)


                                • --progress: show progress during transfer

                                ssh




                                • T: turn off pseudo-tty to decrease cpu load on destination.


                                • c arcfour: use the weakest but fastest SSH encryption. Must specify "Ciphers arcfour" in sshd_config on destination.


                                • o Compression=no: Turn off SSH compression.


                                • x: turn off X forwarding if it is on by default.

                                The beef is in the ssh options - if you just use rsync -av and the -e ssh -T -c arcfour -o Compression=no -x" part, you can get these speeds as well.



                                Comparison:



                                • 13.6 MB/s rsync -az

                                • 16.7 MB/s scp -Cr

                                • 44.8 MB/s rsync -a

                                • 59.8 MB/s sftp

                                • 61.2 MB/s scp -r

                                • 61.4 MB/s sftp -R 128 -B 65536

                                • 62.4 MB/s rsync -a -P -e "ssh -T -c arcfour -o Compression=no -x"

                                • 143.5 MB/s scp -r -c arcfour

                                • 144.2 MB/s sftp -oCiphers=arcfour

                                Sources:



                                https://gist.github.com/KartikTalwar/4393116



                                http://nz2nz.blogspot.com/2018/05/rsync-scp-sftp-speed-test.html






                                share|improve this answer


















                                • 2




                                  Well, they seem to be running cp -r within the remote system, so the encryption used by the SSH connection isn't really relevant. In any case arcfour is considered rather broken and OpenSSH disables it along with others on the server by default since version 6.7 (2014-10-06). In any case, ssh -o Ciphers='aes128-ctr' gives me about 90 MB/s, which should be fast enough on a 1 Gbit/s link.
                                  – ilkkachu
                                  Aug 19 at 13:13











                                • Yes, arcfour is broken, but it's not supposed to be SECURE shell for this case, but more 'comfortable shell' with no emphasis on the encryption. I wouldn't use this over insecure connections, that's correct. If 'aes128-ctr' is fast enough, it can and should be used instead.
                                  – emk2203
                                  Aug 19 at 15:23










                                • See also my extended answer for usage with CPUs which support AES-NI.
                                  – emk2203
                                  Aug 19 at 16:04












                                up vote
                                4
                                down vote










                                up vote
                                4
                                down vote









                                If you want to do it in a fast way, you can look at rsync with a different encryption algorithm. This gives you the option to easily exclude etc., at not much speed sacrifice.



                                rsync -aHAXxv --numeric-ids --progress -e "ssh -T -c arcfour -o Compression=no -x" user@<source>:<source_dir> <dest_dir>



                                together with adding the arcfour encryption to the line starting with Ciphers in /etc/ssh/ssh_config, if not already enabled, gives you an acceptable speed.



                                WARNING: The arcfour encryption is insecure. Do NOT run this over insecure channels. If you are concerned about access to the server from insecure channels using arcfour encryption, change the etc/ssh/ssh_config with a host-specific part for your source host - Create a Host section in your ssh_config for your source host, you can use Ciphers arcfour there to mirror the above -c switch, which restricts arcfour encryption to this host only.



                                For details, refer to ssh_config man pages.



                                However, if your CPUs support the AES-NI instruction set, try switching to aes128-gcm@openssh.com (yes, that's the cipher name, including the @ stuff), which will use the blazingly fast (with AES-NI) AES128-GCM.



                                So, with a CPU supporting AES-NI, change "ssh -T -c arcfour -o Compression=no -x" to "ssh -T -c aes128-gcm@openssh.com -o Compression=no -x" for more secure results.



                                Explanation



                                rsync



                                • (Don't use -z, it is much slower)


                                • a: archive mode - rescursive, preserves owner, preserves permissions, preserves modification times, preserves group, copies symlinks as symlinks, preserves device files.


                                • H: preserves hard-links


                                • A: preserves ACLs


                                • X: preserves extended attributes


                                • x: don't cross file-system boundaries


                                • v: increase verbosity


                                • --numeric-ds: don't map uid/gid values by user/group name

                                • if you need to sync, add --delete: delete extraneous files from dest dirs (differential clean-up during sync)


                                • --progress: show progress during transfer

                                ssh




                                • T: turn off pseudo-tty to decrease cpu load on destination.


                                • c arcfour: use the weakest but fastest SSH encryption. Must specify "Ciphers arcfour" in sshd_config on destination.


                                • o Compression=no: Turn off SSH compression.


                                • x: turn off X forwarding if it is on by default.

                                The beef is in the ssh options - if you just use rsync -av and the -e ssh -T -c arcfour -o Compression=no -x" part, you can get these speeds as well.



                                Comparison:



                                • 13.6 MB/s rsync -az

                                • 16.7 MB/s scp -Cr

                                • 44.8 MB/s rsync -a

                                • 59.8 MB/s sftp

                                • 61.2 MB/s scp -r

                                • 61.4 MB/s sftp -R 128 -B 65536

                                • 62.4 MB/s rsync -a -P -e "ssh -T -c arcfour -o Compression=no -x"

                                • 143.5 MB/s scp -r -c arcfour

                                • 144.2 MB/s sftp -oCiphers=arcfour

                                Sources:



                                https://gist.github.com/KartikTalwar/4393116



                                http://nz2nz.blogspot.com/2018/05/rsync-scp-sftp-speed-test.html






                                share|improve this answer














                                If you want to do it in a fast way, you can look at rsync with a different encryption algorithm. This gives you the option to easily exclude etc., at not much speed sacrifice.



                                rsync -aHAXxv --numeric-ids --progress -e "ssh -T -c arcfour -o Compression=no -x" user@<source>:<source_dir> <dest_dir>



                                together with adding the arcfour encryption to the line starting with Ciphers in /etc/ssh/ssh_config, if not already enabled, gives you an acceptable speed.



                                WARNING: The arcfour encryption is insecure. Do NOT run this over insecure channels. If you are concerned about access to the server from insecure channels using arcfour encryption, change the etc/ssh/ssh_config with a host-specific part for your source host - Create a Host section in your ssh_config for your source host, you can use Ciphers arcfour there to mirror the above -c switch, which restricts arcfour encryption to this host only.



                                For details, refer to ssh_config man pages.



                                However, if your CPUs support the AES-NI instruction set, try switching to aes128-gcm@openssh.com (yes, that's the cipher name, including the @ stuff), which will use the blazingly fast (with AES-NI) AES128-GCM.



                                So, with a CPU supporting AES-NI, change "ssh -T -c arcfour -o Compression=no -x" to "ssh -T -c aes128-gcm@openssh.com -o Compression=no -x" for more secure results.



                                Explanation



                                rsync



                                • (Don't use -z, it is much slower)


                                • a: archive mode - rescursive, preserves owner, preserves permissions, preserves modification times, preserves group, copies symlinks as symlinks, preserves device files.


                                • H: preserves hard-links


                                • A: preserves ACLs


                                • X: preserves extended attributes


                                • x: don't cross file-system boundaries


                                • v: increase verbosity


                                • --numeric-ds: don't map uid/gid values by user/group name

                                • if you need to sync, add --delete: delete extraneous files from dest dirs (differential clean-up during sync)


                                • --progress: show progress during transfer

                                ssh




                                • T: turn off pseudo-tty to decrease cpu load on destination.


                                • c arcfour: use the weakest but fastest SSH encryption. Must specify "Ciphers arcfour" in sshd_config on destination.


                                • o Compression=no: Turn off SSH compression.


                                • x: turn off X forwarding if it is on by default.

                                The beef is in the ssh options - if you just use rsync -av and the -e ssh -T -c arcfour -o Compression=no -x" part, you can get these speeds as well.



                                Comparison:



                                • 13.6 MB/s rsync -az

                                • 16.7 MB/s scp -Cr

                                • 44.8 MB/s rsync -a

                                • 59.8 MB/s sftp

                                • 61.2 MB/s scp -r

                                • 61.4 MB/s sftp -R 128 -B 65536

                                • 62.4 MB/s rsync -a -P -e "ssh -T -c arcfour -o Compression=no -x"

                                • 143.5 MB/s scp -r -c arcfour

                                • 144.2 MB/s sftp -oCiphers=arcfour

                                Sources:



                                https://gist.github.com/KartikTalwar/4393116



                                http://nz2nz.blogspot.com/2018/05/rsync-scp-sftp-speed-test.html







                                share|improve this answer














                                share|improve this answer



                                share|improve this answer








                                edited Aug 19 at 16:00

























                                answered Aug 19 at 12:31









                                emk2203

                                71826




                                71826







                                • 2




                                  Well, they seem to be running cp -r within the remote system, so the encryption used by the SSH connection isn't really relevant. In any case arcfour is considered rather broken and OpenSSH disables it along with others on the server by default since version 6.7 (2014-10-06). In any case, ssh -o Ciphers='aes128-ctr' gives me about 90 MB/s, which should be fast enough on a 1 Gbit/s link.
                                  – ilkkachu
                                  Aug 19 at 13:13











                                • Yes, arcfour is broken, but it's not supposed to be SECURE shell for this case, but more 'comfortable shell' with no emphasis on the encryption. I wouldn't use this over insecure connections, that's correct. If 'aes128-ctr' is fast enough, it can and should be used instead.
                                  – emk2203
                                  Aug 19 at 15:23










                                • See also my extended answer for usage with CPUs which support AES-NI.
                                  – emk2203
                                  Aug 19 at 16:04












                                • 2




                                  Well, they seem to be running cp -r within the remote system, so the encryption used by the SSH connection isn't really relevant. In any case arcfour is considered rather broken and OpenSSH disables it along with others on the server by default since version 6.7 (2014-10-06). In any case, ssh -o Ciphers='aes128-ctr' gives me about 90 MB/s, which should be fast enough on a 1 Gbit/s link.
                                  – ilkkachu
                                  Aug 19 at 13:13











                                • Yes, arcfour is broken, but it's not supposed to be SECURE shell for this case, but more 'comfortable shell' with no emphasis on the encryption. I wouldn't use this over insecure connections, that's correct. If 'aes128-ctr' is fast enough, it can and should be used instead.
                                  – emk2203
                                  Aug 19 at 15:23










                                • See also my extended answer for usage with CPUs which support AES-NI.
                                  – emk2203
                                  Aug 19 at 16:04







                                2




                                2




                                Well, they seem to be running cp -r within the remote system, so the encryption used by the SSH connection isn't really relevant. In any case arcfour is considered rather broken and OpenSSH disables it along with others on the server by default since version 6.7 (2014-10-06). In any case, ssh -o Ciphers='aes128-ctr' gives me about 90 MB/s, which should be fast enough on a 1 Gbit/s link.
                                – ilkkachu
                                Aug 19 at 13:13





                                Well, they seem to be running cp -r within the remote system, so the encryption used by the SSH connection isn't really relevant. In any case arcfour is considered rather broken and OpenSSH disables it along with others on the server by default since version 6.7 (2014-10-06). In any case, ssh -o Ciphers='aes128-ctr' gives me about 90 MB/s, which should be fast enough on a 1 Gbit/s link.
                                – ilkkachu
                                Aug 19 at 13:13













                                Yes, arcfour is broken, but it's not supposed to be SECURE shell for this case, but more 'comfortable shell' with no emphasis on the encryption. I wouldn't use this over insecure connections, that's correct. If 'aes128-ctr' is fast enough, it can and should be used instead.
                                – emk2203
                                Aug 19 at 15:23




                                Yes, arcfour is broken, but it's not supposed to be SECURE shell for this case, but more 'comfortable shell' with no emphasis on the encryption. I wouldn't use this over insecure connections, that's correct. If 'aes128-ctr' is fast enough, it can and should be used instead.
                                – emk2203
                                Aug 19 at 15:23












                                See also my extended answer for usage with CPUs which support AES-NI.
                                – emk2203
                                Aug 19 at 16:04




                                See also my extended answer for usage with CPUs which support AES-NI.
                                – emk2203
                                Aug 19 at 16:04










                                up vote
                                2
                                down vote













                                As per my calculations, the fastest full copy is always using 'tar' (here assuming GNU tar or compatible).



                                mkdir -p photos2 &&
                                 tar -C photos -cf - --exclude=./.thumbcache . |
                                 tar -C photos2 -xpf -


                                And tar has ton of options to manipulate attributes, permissions and file selection/exclusion. For example, the above command excludes the top level subfolder called .thumbcache while copying.






                                share|improve this answer






















                                • Note that --exclude=.thumbcache excludes all the .thumbcache files, not only the one at the top-level. With GNU tar (not bsdtar), you can use --exclude=./.thumbcache to only exclude the top-level .thumbcache file.
                                  – Stéphane Chazelas
                                  Aug 20 at 14:39














                                up vote
                                2
                                down vote













                                As per my calculations, the fastest full copy is always using 'tar' (here assuming GNU tar or compatible).



                                mkdir -p photos2 &&
                                 tar -C photos -cf - --exclude=./.thumbcache . |
                                 tar -C photos2 -xpf -


                                And tar has ton of options to manipulate attributes, permissions and file selection/exclusion. For example, the above command excludes the top level subfolder called .thumbcache while copying.






                                share|improve this answer






















                                • Note that --exclude=.thumbcache excludes all the .thumbcache files, not only the one at the top-level. With GNU tar (not bsdtar), you can use --exclude=./.thumbcache to only exclude the top-level .thumbcache file.
                                  – Stéphane Chazelas
                                  Aug 20 at 14:39












                                up vote
                                2
                                down vote










                                up vote
                                2
                                down vote









                                As per my calculations, the fastest full copy is always using 'tar' (here assuming GNU tar or compatible).



                                mkdir -p photos2 &&
                                 tar -C photos -cf - --exclude=./.thumbcache . |
                                 tar -C photos2 -xpf -


                                And tar has ton of options to manipulate attributes, permissions and file selection/exclusion. For example, the above command excludes the top level subfolder called .thumbcache while copying.






                                share|improve this answer














                                As per my calculations, the fastest full copy is always using 'tar' (here assuming GNU tar or compatible).



                                mkdir -p photos2 &&
                                 tar -C photos -cf - --exclude=./.thumbcache . |
                                 tar -C photos2 -xpf -


                                And tar has ton of options to manipulate attributes, permissions and file selection/exclusion. For example, the above command excludes the top level subfolder called .thumbcache while copying.







                                share|improve this answer














                                share|improve this answer



                                share|improve this answer








                                edited Aug 25 at 2:36

























                                answered Aug 20 at 2:43









                                Lam Das

                                212




                                212











                                • Note that --exclude=.thumbcache excludes all the .thumbcache files, not only the one at the top-level. With GNU tar (not bsdtar), you can use --exclude=./.thumbcache to only exclude the top-level .thumbcache file.
                                  – Stéphane Chazelas
                                  Aug 20 at 14:39
















                                • Note that --exclude=.thumbcache excludes all the .thumbcache files, not only the one at the top-level. With GNU tar (not bsdtar), you can use --exclude=./.thumbcache to only exclude the top-level .thumbcache file.
                                  – Stéphane Chazelas
                                  Aug 20 at 14:39















                                Note that --exclude=.thumbcache excludes all the .thumbcache files, not only the one at the top-level. With GNU tar (not bsdtar), you can use --exclude=./.thumbcache to only exclude the top-level .thumbcache file.
                                – Stéphane Chazelas
                                Aug 20 at 14:39




                                Note that --exclude=.thumbcache excludes all the .thumbcache files, not only the one at the top-level. With GNU tar (not bsdtar), you can use --exclude=./.thumbcache to only exclude the top-level .thumbcache file.
                                – Stéphane Chazelas
                                Aug 20 at 14:39

















                                 

                                draft saved


                                draft discarded















































                                 


                                draft saved


                                draft discarded














                                StackExchange.ready(
                                function ()
                                StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f463477%2fsh-recursive-copy-cp-r-how-to-exclude-subfolder%23new-answer', 'question_page');

                                );

                                Post as a guest
































































                                Comments

                                Post a Comment

                                Popular posts from this blog

                                What does second last employer means? [closed]

                                Image
                                Clash Royale CLAN TAG #URR8PPP .everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty margin-bottom:0; up vote -3 down vote favorite Say, for example I have worked in the following companies, Company A => 2010 - 2011 Company B => 2011 - 2012 Company C => 2012 - Till Date Company D => Have offer Now assume Company D is asking to submit documents from my second last employer. So from the list of companies given, which company would be qualified as second last employer? software-industry employer share | improve this question asked Aug 10 '14 at 5:16 Rajaprabhu Aravindasamy 559 1 6 13 closed as off-topic by Jim G., jmort253 ♦ Aug 11 '14 at 1:58 This question appears to be off-topic. The users who voted to close gave this specific reason: "Questions seeking advice on company-specific regulations, agreements, or policies should be directed to your manager or HR department. Questions that address...
                                Read more

                                Installing NextGIS Connect into QGIS 3?

                                Image
                                Clash Royale CLAN TAG #URR8PPP .everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty margin-bottom:0; up vote 1 down vote favorite Just acquired free version of NEXTGIS and trying to install the plugin for QGIS. It tells me it was not designed for version 3 of QGIS. How do I set up the QGIS-NEXTGIS interface then? qgis qgis-plugins nextgis share | improve this question edited 3 hours ago PolyGeo ♦ 51.9k 17 77 233 asked 4 hours ago IanGS 16 4 New contributor IanGS is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct. add a comment  |  up vote 1 down vote favorite Just acquired free version of NEXTGIS and trying to install the plugin for QGIS. It tells me it was not designed for version 3 of QGIS. How do I set up the QGIS-NEXTGIS interface then? qgis qgis-plugin...
                                Read more

                                One-line joke

                                A one-liner is a joke that is delivered in a single line. A good one-liner is said to be pithy - concise and meaningful. [1] Comedians and actors use this comedic method as part of their act, e.g. Jimmy Carr, Tommy Cooper, Rodney Dangerfield, Norm Macdonald, Ken Dodd, Stewart Francis, Zach Galifianakis, Mitch Hedberg, Anthony Jeselnik, Milton Jones, Shappi Khorsandi, Jay London, Mark Linn-Baker, Demetri Martin, Groucho Marx, Dan Mintz, Emo Philips, Tim Vine, Steven Wright and Henny Youngman and Arnold Schwarzenegger. [ citation needed ] Many fictional characters are also known to deliver one-liners, including James Bond, who usually includes pithy and laconic quips after disposing of a villain. [ citation needed ] Examples "A baby seal walks into a club." "A dyslexic man walks into a bra." (Anonymous) "There are three types of people, those who can count and those who can't." "Venison's dear, isn't it?" (Jimmy Carr) "An escala...
                                Read more