tscp - Script to quickly and securely transfer files by tunneling tar through ssh

For security, pretty much every file I transfer over the network I use 'scp'.  Or more likely (to save time and traffic) I tunnel 'tar' through 'ssh'.  This compresses the data, transferes securly through the network, and then de-compress on the other end.  But it can be a pain remembering the exact syntax, and others have requested a script to help with this.

So to help I wrote the following script:


#!/bin/bash
#
#       tscp
#
#       Copyright 2011 nairb <code@nairb.us>
#
#       This program is free software; you can redistribute it and/or modify
#       it under the terms of the GNU General Public License as published by
#       the Free Software Foundation; either version 2 of the License, or
#       (at your option) any later version.
#
#       This program is distributed in the hope that it will be useful,
#       but WITHOUT ANY WARRANTY; without even the implied warranty of
#       MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#       GNU General Public License for more details.
#
#       You should have received a copy of the GNU General Public License
#       along with this program; if not, write to the Free Software
#       Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
#       MA 02110-1301, USA.

tarscp(){
  if [ $1 ] ; then
    myaddress=''
    localfile=''
    localtoremote=''
    if [ -z sshopts ]; then
      sshopts="-2 -A"
    fi
    for i in $@ ; do
      if [ $(echo $i |grep -iE '^[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}:.*') ] ; then
        myaddress=$i
        if [ $localfile ] ; then
          localtoremote="true"
        fi
      else
        if [ -f $i -o -d $i ] ; then
          localfile=$i
        fi
      fi
    done
    if [ $myaddress ] ; then
      remotepath=$(echo $myaddress |cut -d ':' -f2)
      myaddress=$(echo $myaddress |cut -d ':' -f1)
    else
      echo -e "yo, come on...\nneed to give a full address and path\nlike: 'myuser@myhost.com:~/some/folder/or/file'" && return 1
    fi
    checkremotepathcom="if [ -d $remotepath -o -f $remotepath ] ; then echo 0 ; else echo 1 ; fi"
    if [ $(ssh $sshopts $myaddress "$checkremotepathcom") == 0 ] ; then
      if [ $localfile ] ; then
        if [ $localtoremote ] ; then
          checkremotepathcom="if [ -d $remotepath ] ; then echo 0 ; else echo 1 ; fi"
          if [ $(ssh $sshopts $myaddress "$checkremotepathcom") == 0 ] ; then
            remotecom="cd $remotepath ; tar -xzf -"
            tar -czvf - $localfile |ssh $sshopts $myaddress "$remotecom"
          else
            echo "yo dude. that path aint on the server." && return 1
          fi
        else


          if [ $remotepath == '~/' ] ; then

            remotefile=$(ssh $sshopts $myaddress "pwd")

            remotefile=$(echo $remotefile |cut -d '/' -f $(echo $remotefile |sed -e 's/\//\/ /g' |wc -w))

            remotecom="cd $remotepath ; cd ../ ; tar -czvf - $remotefile"

          else

            remotefile=$(echo $remotepath |cut -d '/' -f $(echo $remotepath |sed -e 's/\//\/ /g' |wc -w))

            if [ -z $(echo $remotepath |cut -d '/' -f 1) ] ; then

              mypath='/'

            else

              mypath='~/'

            fi

            for i in $(seq 2 $(($(echo $remotepath |sed -e 's/\//\/ /g' |wc -w) - 1))) ; do

              mypath=$mypath/$(echo $remotepath | cut -d '/' -f $i)

            done

            remotepath=$mypath

            mypath=''

            remotecom="cd $remotepath ; tar -czvf - $remotefile"

          fi

          if [ -d $localfile ] ; then

            cd $localfile

          fi

          if [ $remotepath == '/' ] ; then

            remotecom="tar -czvf - /"          

          fi

          ssh $sshopts $myaddress "$remotecom" | tar -xzf -

        fi

      else

        if [ $remotepath == '~/' ] ; then

          remotefile=$(ssh $sshopts $myaddress "pwd")

          remotefile=$(echo $remotefile |cut -d '/' -f $(echo $remotefile |sed -e 's/\//\/ /g' |wc -w))

          remotecom="cd $remotepath ; cd ../ ; tar -czvf - $remotefile"

        else

          remotefile=$(echo $remotepath |cut -d '/' -f $(echo $remotepath |sed -e 's/\//\/ /g' |wc -w))

          if [ -z $(echo $remotepath |cut -d '/' -f 1) ] ; then

            mypath='/'

          else

            mypath='~/'

          fi

          for i in $(seq 2 $(($(echo $remotepath |sed -e 's/\//\/ /g' |wc -w) - 1))) ; do

            mypath=$mypath/$(echo $remotepath | cut -d '/' -f $i)

          done

          remotepath=$mypath

          mypath=''

          remotecom="cd $remotepath ; tar -czvf - $remotefile"

        fi

        if [ $remotepath == '/' ] ; then

          remotecom="tar -czvf - /"          

        fi

        ssh $sshopts $myaddress "$remotecom" | tar -xzf -

      fi

    else

      echo "dude, that path aint on the server." && return 1

    fi

  else

    echo -e "no arguments given fool\ntry again" && return 1

  fi

}




case $(echo ${0##*/} |cut -d '/' -f $(echo ${0##*/} |wc -w)) in

  tscp)

    tarscp $@

    ;;

  *)

    echo -e "come on... just call the script 'tscp'"

    ;;

esac

And, if like me, you find yourself logging into a server that doesnt have the script saved, and you dont want to make a new script file (maybe you only need to use the command once to migrate data from your old server to a new one). I converted the above into a 1-liner function that can just be pasted in and used with the same call.  Alternatively you could include this function in one of your functions rc files like mentioned in my previous post.

Regardless of the reason, here is the 1-liner function:


tscp(){ if [ $1 ] ; then myaddress=''; localfile=''; localtoremote=''; if [ -z sshopts ]; then sshopts="-2 -A"; fi; for i in $@ ; do if [ $(echo $i |grep -iE '^[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}:.*') ] ; then myaddress=$i; if [ $localfile ] ; then localtoremote="true"; fi; else if [ -f $i -o -d $i ] ; then localfile=$i; fi; fi; done; if [ $myaddress ] ; then remotepath=$(echo $myaddress |cut -d ':' -f2); myaddress=$(echo $myaddress |cut -d ':' -f1); else echo -e "yo, come on...\nneed to give a full address and path\nlike: 'myuser@myhost.com:~/some/folder/or/file'" && return 1 ; fi; checkremotepathcom="if [ -d $remotepath -o -f $remotepath ] ; then echo 0 ; else echo 1 ; fi"; if [ $(ssh $sshopts $myaddress "$checkremotepathcom") == 0 ] ; then if [ $localfile ] ; then if [ $localtoremote ] ; then checkremotepathcom="if [ -d $remotepath ] ; then echo 0 ; else echo 1 ; fi"; if [ $(ssh $sshopts $myaddress "$checkremotepathcom") == 0 ] ; then remotecom="cd $remotepath ; tar -xzf -"; tar -czvf - $localfile |ssh $sshopts $myaddress "$remotecom"; else echo "yo dude. that path aint on the server." && return 1 ; fi; else if [ $remotepath == '~/' ] ; then remotefile=$(ssh $sshopts $myaddress "pwd"); remotefile=$(echo $remotefile |cut -d '/' -f $(echo $remotefile |sed -e 's/\//\/ /g' |wc -w)); remotecom="cd $remotepath ; cd ../ ; tar -czvf - $remotefile"; else remotefile=$(echo $remotepath |cut -d '/' -f $(echo $remotepath |sed -e 's/\//\/ /g' |wc -w)); if [ -z $(echo $remotepath |cut -d '/' -f 1) ] ; then mypath='/'; else mypath='~/'; fi; for i in $(seq 2 $(($(echo $remotepath |sed -e 's/\//\/ /g' |wc -w) - 1))) ; do mypath=$mypath/$(echo $remotepath | cut -d '/' -f $i); done; remotepath=$mypath; mypath=''; remotecom="cd $remotepath ; tar -czvf - $remotefile"; fi; if [ -d $localfile ] ; then cd $localfile; fi; if [ $remotepath == '/' ] ; then remotecom="tar -czvf - /"          ; fi; ssh $sshopts $myaddress "$remotecom" | tar -xzf -; fi; else if [ $remotepath == '~/' ] ; then remotefile=$(ssh $sshopts $myaddress "pwd"); remotefile=$(echo $remotefile |cut -d '/' -f $(echo $remotefile |sed -e 's/\//\/ /g' |wc -w)); remotecom="cd $remotepath ; cd ../ ; tar -czvf - $remotefile"; else remotefile=$(echo $remotepath |cut -d '/' -f $(echo $remotepath |sed -e 's/\//\/ /g' |wc -w)); if [ -z $(echo $remotepath |cut -d '/' -f 1) ] ; then mypath='/'; else mypath='~/'; fi; for i in $(seq 2 $(($(echo $remotepath |sed -e 's/\//\/ /g' |wc -w) - 1))) ; do mypath=$mypath/$(echo $remotepath | cut -d '/' -f $i); done; remotepath=$mypath; mypath=''; remotecom="cd $remotepath ; tar -czvf - $remotefile"; fi; if [ $remotepath == '/' ] ; then remotecom="tar -czvf - /"          ; fi; ssh $sshopts $myaddress "$remotecom" | tar -xzf -; fi; else echo "dude, that path aint on the server." && return 1 ; fi; else echo -e "no arguments given fool\ntry again" && return 1 ; fi; }


Hope you find this helpful. Thanks for reading.


***** UPDATE *****
Sorry to anyone who copied this earlier today.  The functionality worked for some folders (not including '~/', or '/'), but it has now been edited to work for those. Also now including ssh options like '-2 -A' and I have fixed it to work for single file, which failed before (since you can only 'cd' into a directory, not a file)

Also the prior 'exit' lines have been changed to 'return' to prevent the terminal window being closed if you run from the function without the right arguments.

Again sorry for posting it prior to thorough testing (these things hapen with insomnia posting scripts at 5am with no sleep), but it has been fixed now, and tested on both debian, and ubuntu computers/servers

I may still update this again to make it more organized, but will likely release future revisions as file downloads on my own site instead of this blog (its a kinda long script already to copy and paste)
****

Website Security Intro - Looking for RFI attacks

If you run server-side scripting (which most major sites/applications do), it makes your site much more vulnerable to many attacks. If you are dealing with data-bases, this opens you up for other attacks. Unfortunately one may be hard pressed to find a web-site, (CMS solution) that does not have any server-side scripts, unless you have no need for dynamic content, and can program the site in just html/java-script/flash.

If you can manage to write your site up in html only (including java-script and/or flash) this is clearly the most secure solution.

But the rest out there, may want to look into putting their site up on a 3rd party managed CMS-service. Blogs (like this) can be run here on blogger (or WordPress, TypePad, ect. Also offer solutions). If blogs are not what your looking for, Google freely offers “google sites”. There are many other managed CMS hosting services available, check with your chosen hosting provider. A major benefit to these services is to leave the server-side security (and incident handling) on them instead of you. You would just need to keep up with your passwords, email, PC security just like with your facebook, or myspace pages.

For those that just cant find a managed solution for their needs (or just dont want to use them/enjoy more freedom then the provided solutions), monitoring the security of your sites can be an ongoing battle. In the following introduction, you will find helpful weapons to add to your arsenal.

I find dealing with linux servers the easiest, (and also the most common hosting solution) so I will be primarily discussing such. If your site is on a hosting account without ssh access, upgrade your account, or find a new host.

SSH, and the linux server's terminal applications (grep, awk, sed, cat, less, ect.) are your best friends. Get to know them. Google searching for their “example uses” will greatly help. If you get stuck with any of them, chances are the 'man' application will give you much more info about the options of these apps than you will ever need.

Server-side scripting exploits can be seen in the apache access logs. If you dont know where these are for you, find out. Apache log files can be viewed (searched, and their data formated more friendly) using your ssh login, and the terminal apps.

For example: RFI attacks (“remote file include” - usually the result of include statements not being sanitized correctly, allowing remote scripts to be run on your server with your users permissions) show up in the apache access logs as a line including the apache code: '200'. This is the code for successful access, although this can still be misleading if you have custom error pages, it could indicate that there was an attempt which only successfully returned the error page. Weather just an attempt, or successful, RFI attack log entries usually include a 'POST' or 'GET', and a link to a remote file such as: '=http://SomeDomain.com/MalitionsHackerScript.php'.

To run the following example (1-liner) command, to look for RFI attack attempts (including the mentioned false-positives), you would want to ssh into your webspace, and change directory to your 'logs/' (or whatever the folder name of your logs).

Example 1-liner:

zcat ./access.log.* |grep -i '=http' |grep ' 200 ' |grep -iE 'txt|php|cgi'

Now lets break down that above call:

zcat ./acess.log*

This command outputs all of the lines in your apache access logs, and then its output is “piped” to the next search command:

grep ie '=http'

Which searches for occurrences of '=http' or '=HTTP'. The output of this is then “piped” to the command:

grep ' 200 '

Which searches for occurrences of the successful apache code '200'. the output of this is then “piped” to:

grep -iE 'txt|php|cgi'

That then searches and outputs only the lines that include common extension for hacker scripts such as 'txt', 'php' or 'cgi'. There are others, or non-normal-pattern files, and scripts used also, so you could just replace this like with 'less' to just view all the posts or gets of the previous search commands. Although this will also show many potential false-positives that are just links from/to your site from/to elsewhere.

To further explain the above used grep options or “switches”:

For 'grep -i',  the 'i' option tells grep to not count the case of the characters, so picks up '=http', '=Http', '=hTtP' or '=HTTP', ect...

The 'E' option in 'grep -iE' tells it that we are looking for an expression (in this case any occurrences of: txt, php, or cgi), instead of just 1 specific text.

--
Anyhow, thats more than I meant to go into for one post, but its a starting point to one look into their websites security.

keep on keeping on, and bad guys be gone...