Mixed language (bash/python) function to deal with stdin pipes

often i find the need to be able to use one of my bash scripts to process data from a stdin pipe like:

somecommand |myscript


the standard way is to 'cat /dev/stdin' but alone doesnt do what is needed. for instance perhaps you are not piping output to the script (however this is simple to deal with as an 'if ! [ -t 0 ] ' statement). sometimes options/argument can get mixed into the pipe. the standard solution is to read (ie: 'cat /dev/stdin') using a non-breaking(/blocking, i forget) stty, but this kind of reading of stdin is tricky, and confusing, and /dev/stdin may not be available in all systems (chrooted hosting accounts for instance).

a simple solution is to create a bash function calling python (most systems that include bash also include python) to read stdin, and write directly back to stdout (in essence a simple python pager called as a bash function). this function looks like:

getstdin (){
python -c "from sys import *
if not stdin.isatty():stdout.write(stdin.read())
else:exit(1)"
}


note: ';else:exit(1)' isnt strictly required, but gives a standard error code if nothing on the stdin pipe, which is useful to further script logic (determining the error codes of commands to decide what functionality to produce)

this function takes care of the previously mentioned method's pitfalls. include this function in your own bash script, and deal with (or ignore) stdin pipe easily. a simple example bash script using this function:

#!/bin/bash
#  mystdin.sh
#  by - nairb <c0d3@nairb.us>
#  purpose: to add the text "RECIEVED: " and return stdin pipe

getstdin (){
python -c "from sys import *
if not stdin.isatty():stdout.write(stdin.read())
else:exit(1)"
}

stdin=$(getstdin)
if [ $stdin ];then
  echo "RECEIVED: $stdin"
else
  echo "didnt get anything on stdin, doing something else"
fi


you could run the above example script with a command like:

echo "this is some text" | bash mystdin.sh


the above function is also an example of the standard way to read from a stdin pipe in python scripts (not just calling python in bash).

--
hope someone finds this post useful,
- nairb

tscp - Script to quickly and securely transfer files by tunneling tar through ssh

For security, pretty much every file I transfer over the network I use 'scp'.  Or more likely (to save time and traffic) I tunnel 'tar' through 'ssh'.  This compresses the data, transferes securly through the network, and then de-compress on the other end.  But it can be a pain remembering the exact syntax, and others have requested a script to help with this.

So to help I wrote the following script:


#!/bin/bash
#
#       tscp
#
#       Copyright 2011 nairb <code@nairb.us>
#
#       This program is free software; you can redistribute it and/or modify
#       it under the terms of the GNU General Public License as published by
#       the Free Software Foundation; either version 2 of the License, or
#       (at your option) any later version.
#
#       This program is distributed in the hope that it will be useful,
#       but WITHOUT ANY WARRANTY; without even the implied warranty of
#       MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#       GNU General Public License for more details.
#
#       You should have received a copy of the GNU General Public License
#       along with this program; if not, write to the Free Software
#       Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
#       MA 02110-1301, USA.

tarscp(){
  if [ $1 ] ; then
    myaddress=''
    localfile=''
    localtoremote=''
    if [ -z sshopts ]; then
      sshopts="-2 -A"
    fi
    for i in $@ ; do
      if [ $(echo $i |grep -iE '^[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}:.*') ] ; then
        myaddress=$i
        if [ $localfile ] ; then
          localtoremote="true"
        fi
      else
        if [ -f $i -o -d $i ] ; then
          localfile=$i
        fi
      fi
    done
    if [ $myaddress ] ; then
      remotepath=$(echo $myaddress |cut -d ':' -f2)
      myaddress=$(echo $myaddress |cut -d ':' -f1)
    else
      echo -e "yo, come on...\nneed to give a full address and path\nlike: 'myuser@myhost.com:~/some/folder/or/file'" && return 1
    fi
    checkremotepathcom="if [ -d $remotepath -o -f $remotepath ] ; then echo 0 ; else echo 1 ; fi"
    if [ $(ssh $sshopts $myaddress "$checkremotepathcom") == 0 ] ; then
      if [ $localfile ] ; then
        if [ $localtoremote ] ; then
          checkremotepathcom="if [ -d $remotepath ] ; then echo 0 ; else echo 1 ; fi"
          if [ $(ssh $sshopts $myaddress "$checkremotepathcom") == 0 ] ; then
            remotecom="cd $remotepath ; tar -xzf -"
            tar -czvf - $localfile |ssh $sshopts $myaddress "$remotecom"
          else
            echo "yo dude. that path aint on the server." && return 1
          fi
        else


          if [ $remotepath == '~/' ] ; then

            remotefile=$(ssh $sshopts $myaddress "pwd")

            remotefile=$(echo $remotefile |cut -d '/' -f $(echo $remotefile |sed -e 's/\//\/ /g' |wc -w))

            remotecom="cd $remotepath ; cd ../ ; tar -czvf - $remotefile"

          else

            remotefile=$(echo $remotepath |cut -d '/' -f $(echo $remotepath |sed -e 's/\//\/ /g' |wc -w))

            if [ -z $(echo $remotepath |cut -d '/' -f 1) ] ; then

              mypath='/'

            else

              mypath='~/'

            fi

            for i in $(seq 2 $(($(echo $remotepath |sed -e 's/\//\/ /g' |wc -w) - 1))) ; do

              mypath=$mypath/$(echo $remotepath | cut -d '/' -f $i)

            done

            remotepath=$mypath

            mypath=''

            remotecom="cd $remotepath ; tar -czvf - $remotefile"

          fi

          if [ -d $localfile ] ; then

            cd $localfile

          fi

          if [ $remotepath == '/' ] ; then

            remotecom="tar -czvf - /"          

          fi

          ssh $sshopts $myaddress "$remotecom" | tar -xzf -

        fi

      else

        if [ $remotepath == '~/' ] ; then

          remotefile=$(ssh $sshopts $myaddress "pwd")

          remotefile=$(echo $remotefile |cut -d '/' -f $(echo $remotefile |sed -e 's/\//\/ /g' |wc -w))

          remotecom="cd $remotepath ; cd ../ ; tar -czvf - $remotefile"

        else

          remotefile=$(echo $remotepath |cut -d '/' -f $(echo $remotepath |sed -e 's/\//\/ /g' |wc -w))

          if [ -z $(echo $remotepath |cut -d '/' -f 1) ] ; then

            mypath='/'

          else

            mypath='~/'

          fi

          for i in $(seq 2 $(($(echo $remotepath |sed -e 's/\//\/ /g' |wc -w) - 1))) ; do

            mypath=$mypath/$(echo $remotepath | cut -d '/' -f $i)

          done

          remotepath=$mypath

          mypath=''

          remotecom="cd $remotepath ; tar -czvf - $remotefile"

        fi

        if [ $remotepath == '/' ] ; then

          remotecom="tar -czvf - /"          

        fi

        ssh $sshopts $myaddress "$remotecom" | tar -xzf -

      fi

    else

      echo "dude, that path aint on the server." && return 1

    fi

  else

    echo -e "no arguments given fool\ntry again" && return 1

  fi

}




case $(echo ${0##*/} |cut -d '/' -f $(echo ${0##*/} |wc -w)) in

  tscp)

    tarscp $@

    ;;

  *)

    echo -e "come on... just call the script 'tscp'"

    ;;

esac

And, if like me, you find yourself logging into a server that doesnt have the script saved, and you dont want to make a new script file (maybe you only need to use the command once to migrate data from your old server to a new one). I converted the above into a 1-liner function that can just be pasted in and used with the same call.  Alternatively you could include this function in one of your functions rc files like mentioned in my previous post.

Regardless of the reason, here is the 1-liner function:


tscp(){ if [ $1 ] ; then myaddress=''; localfile=''; localtoremote=''; if [ -z sshopts ]; then sshopts="-2 -A"; fi; for i in $@ ; do if [ $(echo $i |grep -iE '^[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}:.*') ] ; then myaddress=$i; if [ $localfile ] ; then localtoremote="true"; fi; else if [ -f $i -o -d $i ] ; then localfile=$i; fi; fi; done; if [ $myaddress ] ; then remotepath=$(echo $myaddress |cut -d ':' -f2); myaddress=$(echo $myaddress |cut -d ':' -f1); else echo -e "yo, come on...\nneed to give a full address and path\nlike: 'myuser@myhost.com:~/some/folder/or/file'" && return 1 ; fi; checkremotepathcom="if [ -d $remotepath -o -f $remotepath ] ; then echo 0 ; else echo 1 ; fi"; if [ $(ssh $sshopts $myaddress "$checkremotepathcom") == 0 ] ; then if [ $localfile ] ; then if [ $localtoremote ] ; then checkremotepathcom="if [ -d $remotepath ] ; then echo 0 ; else echo 1 ; fi"; if [ $(ssh $sshopts $myaddress "$checkremotepathcom") == 0 ] ; then remotecom="cd $remotepath ; tar -xzf -"; tar -czvf - $localfile |ssh $sshopts $myaddress "$remotecom"; else echo "yo dude. that path aint on the server." && return 1 ; fi; else if [ $remotepath == '~/' ] ; then remotefile=$(ssh $sshopts $myaddress "pwd"); remotefile=$(echo $remotefile |cut -d '/' -f $(echo $remotefile |sed -e 's/\//\/ /g' |wc -w)); remotecom="cd $remotepath ; cd ../ ; tar -czvf - $remotefile"; else remotefile=$(echo $remotepath |cut -d '/' -f $(echo $remotepath |sed -e 's/\//\/ /g' |wc -w)); if [ -z $(echo $remotepath |cut -d '/' -f 1) ] ; then mypath='/'; else mypath='~/'; fi; for i in $(seq 2 $(($(echo $remotepath |sed -e 's/\//\/ /g' |wc -w) - 1))) ; do mypath=$mypath/$(echo $remotepath | cut -d '/' -f $i); done; remotepath=$mypath; mypath=''; remotecom="cd $remotepath ; tar -czvf - $remotefile"; fi; if [ -d $localfile ] ; then cd $localfile; fi; if [ $remotepath == '/' ] ; then remotecom="tar -czvf - /"          ; fi; ssh $sshopts $myaddress "$remotecom" | tar -xzf -; fi; else if [ $remotepath == '~/' ] ; then remotefile=$(ssh $sshopts $myaddress "pwd"); remotefile=$(echo $remotefile |cut -d '/' -f $(echo $remotefile |sed -e 's/\//\/ /g' |wc -w)); remotecom="cd $remotepath ; cd ../ ; tar -czvf - $remotefile"; else remotefile=$(echo $remotepath |cut -d '/' -f $(echo $remotepath |sed -e 's/\//\/ /g' |wc -w)); if [ -z $(echo $remotepath |cut -d '/' -f 1) ] ; then mypath='/'; else mypath='~/'; fi; for i in $(seq 2 $(($(echo $remotepath |sed -e 's/\//\/ /g' |wc -w) - 1))) ; do mypath=$mypath/$(echo $remotepath | cut -d '/' -f $i); done; remotepath=$mypath; mypath=''; remotecom="cd $remotepath ; tar -czvf - $remotefile"; fi; if [ $remotepath == '/' ] ; then remotecom="tar -czvf - /"          ; fi; ssh $sshopts $myaddress "$remotecom" | tar -xzf -; fi; else echo "dude, that path aint on the server." && return 1 ; fi; else echo -e "no arguments given fool\ntry again" && return 1 ; fi; }


Hope you find this helpful. Thanks for reading.


***** UPDATE *****
Sorry to anyone who copied this earlier today.  The functionality worked for some folders (not including '~/', or '/'), but it has now been edited to work for those. Also now including ssh options like '-2 -A' and I have fixed it to work for single file, which failed before (since you can only 'cd' into a directory, not a file)

Also the prior 'exit' lines have been changed to 'return' to prevent the terminal window being closed if you run from the function without the right arguments.

Again sorry for posting it prior to thorough testing (these things hapen with insomnia posting scripts at 5am with no sleep), but it has been fixed now, and tested on both debian, and ubuntu computers/servers

I may still update this again to make it more organized, but will likely release future revisions as file downloads on my own site instead of this blog (its a kinda long script already to copy and paste)
****