HTML Safe Characters - Bash function using 'sed'

I find myself posting the output from commands in terminal to html pages often, so wrote a quick "1-liner" bash function to convert a string to (mostly)html safe characters. you can "pipe" a string to this function, and direct its output to another file.

The function:

tohtm(){ IFSBACK=$IFS;IFS=$(echo -en "\n\b");string2htm(){ [[ $1 ]] && echo $1|sed -e 's/\&/\&amp;/g' -e 's/\"/\&quot;/g' -e 's/</\&lt;/g' -e 's/>/\&gt;/g' -e 's/>/\&gt;/g'; };if [ ! -t 0 ];then echo -e "<html>\n<head></head>\n<body>\n<p>";for i in $(cat /dev/stdin); do string2htm $i;echo "<br/>";done;echo -e "</p>\n</body>\n</html>";fi;IFS=$IFSBACK; }

Syntax to run this function would be is similar following example commands:

ls - la |tohtm >thisfolder.html

Or

echo 'this & that is < than "something", but > than "something else"' |tohtm

Both of the above commands should work, but there may be others that don't (if the string includes single quotes for example). I still need to do some work on initially formatting such strings, but this function has been very useful for me so far.

you can also simply use cgi.escape from python. for example:

2htm(){ if ! [ -t 0 ];then myvar=$(cat /dev/stdin); else if [ "$1" ];then myvar=$@;fi;fi;pycom=$(echo -e "from cgi import escape\nprint escape(u'''$myvar''')"); python -c "$pycom"; }

That is about all I have for this post. I will post other similar functions (and perhaps this one again) in a future post.  Thanks for reading.

User owned scripts security

Since I am posting scripts (bash, pythong, ect.) up here, I figure it would be a good idea to mention security and permissions of these (or any of you own/other's) scripts.  If you make a script executable (chmod +x scriptname), Linux can use the shebang to determine the script type, and run with a call similar to any other application on the system.  However, if like me you care about security, you may not want to have executable files in your home directory.  One of the best things about linux (or any *nix including bsd, or mac) is the user permissions management.  If your account gets compromised, that doesn't mean the rest of your whole system has been "rooted" (compromised as the root user).  Most infections (linux is not immune to compromise) attack executable files that your user has access to, and so if you have executable scripts in your home directory, they can be written to.  Then the next time you call on one of these scripts, you may be running a malicious hacker's code.

Now you may ask “what alternative is there?”.  With the rest of this post, I plan to give a few.

root owned (standard for most systems):
If it is a script that others on the system would want to use (and you can become root), you could move it into one of the system folder in the users $PATH.  I don't tend to do this often (and dont really suggest it), since it puts random scripts outside of the system's package management.  Instead it may be best to use one (or a combination like I do) of the following options. You can then give the other users the file (use links, edit their rc files (~/.bashrc, ~/.profile) as root, or include them in /etc/skell before creating the users) .

rc file:
As stated in my previous post, you may create a file with a list of simple functions. If its a simple script, it may be fairly easy to convert to a function (or a few).  If you can create a few functions you would want to use together, include them into an 'rc' file.  Then when you want to use them, you only need to 'source' the file first (source MyFunctions.rc). You can then call on the functions just like any other command. If it is a list of functions you would want to use (or at least be able to without sourcing the file) every time you login, you could include sourcing the rc file through your ~/.bashrc (or ~/.profile if you might want to use these functions as shortcuts in your GUI shell such as gnome. I may go into this further in another post, but thats beyond the scope of this article).

Not every script can easily be converted to the function, and so

alias the complex scripts:
most modern linux distrobutions include a tag in your ~/.bashrc, or .profile to source a ~/.bash_aliases file.  Create, and use this file if you dont already have it. And a statement similar to the following 'if' should be included your ~/.bashrc, or ~/.profile

if [ -f ~/.bash_aliases ]; then
    . ~/.bash_aliases
fi

!!you can use a this as example of sourcing to source one of the previous mentioned functions rc files in your bashrc, or profile.

Create your ~/.bash_aliases file and fill it with statements similar to the following example to create a command to run a non-executable bash script called ~/myscripts/newscript as a command 'myscript'

alias myscript=”bash ~/myscripts/newscript”

your ~/.bashrc file is parsed every time you login to bash, your .profile is sourced every login (including most GUI shells such as gnome, kde, xfce4) so if you have non-executable scripts that use a gui toolkit (such as zenity for gnome) you may want to source another ~/.gui-aliases or similar in your ~/.profile so you can easily run said aliased script through terminal, or a shortcut in your applications menu, panel, or desktop.

Thats about all for this post. I will probably give an example rc file (with some fun included functions) that auto-aliases all scripts under a folder based on the “shebang” line in a future post.

Bash Script - Port Scanner

Please be careful! port scanning can be seen as abuse, and get you into trouble (ISP may block your traffic, ect...)

The following bash script works slowly as a port scanner, but may take a long time to scan the 1000 ports that it scanns. It is just a quick example script using 'nc'. The fastest run is with no arguments to scan your localhost. The next fastest would be IP addresses (or domain) in your local network. A slow scan would occur for a non-local IP or domain. or, the slowest scan is a list of hostnames to scan as arguments


Check it out - hope its useful to someone:

#!/bin/bash
#
#       wazopen
#    
#       @copyright - 2011 nairb <code@nairb.us>
#    
#       @license - GNU General Public License V2



findwazopen(){
portscanfile=$(tempfile -p port -s scan.txt)
nc -v -z -w 1 $1 1-1000 >>$portscanfile 2>&1
echo $(cat $portscanfile |grep succeeded|awk '{print $4}')
rm $portscanfile
}

case $(echo ${0##*/} |cut -d '/' -f $(echo ${0##*/} |wc -w)) in
wazopen)
if [ $1 ] ; then
if [ $2 ] ; then
for i in $@ ; do
echo -e "HOST: $i has the following open ports:"
findwazopen $i
done
else
findwazopen $1
fi
else
findwazopen localhost
fi
;;
*)
echo -e "wtf r u trying to do?\n\ncant call this as $0"
;;
esac

although if you have 'nmap' installed, you can do a much easier (and much faster) version like the following 1-liner function to be called on later in the same environment as zopen:

zopen(){ nmap $1 |grep \/ |grep -v nmap |awk '{print $1 " " $3}' ; }

or if you only care about the port numbers:

zopen(){ echo $(nmap $1 |grep \/ |grep -v nmap |awk '{print $1}'|cut -d '/' -f1 ) ; }

another function for a list of hosts:

zopenlist(){ for i in $@ ; do echo -e "$i:\n\t$(echo $(nmap $i |grep \/ |grep -v nmap |awk '{print $1}'|cut -d '/' -f1 ))\n" ;done ; }

or a my favorite, a much better version with better logic and usage of 'awk':

wtfsopen(){ wadopen(){ for i in $@;do echo -e "Host: $i:\nPorts\tServices\n$(nmap $i |awk '/^[0-9].*/ {print $1 " " $3}')";done ; };if [ $(which nmap) ];then if [ $1 ];then wadopen $@;else wadopen localhost;fi;else echo -e "install 'nmap' first";fi; }

and if you find you would like to use any such function often you can just include it in one of your shell's rc instead of making an executable script

Website Security Intro - Looking for RFI attacks

If you run server-side scripting (which most major sites/applications do), it makes your site much more vulnerable to many attacks. If you are dealing with data-bases, this opens you up for other attacks. Unfortunately one may be hard pressed to find a web-site, (CMS solution) that does not have any server-side scripts, unless you have no need for dynamic content, and can program the site in just html/java-script/flash.

If you can manage to write your site up in html only (including java-script and/or flash) this is clearly the most secure solution.

But the rest out there, may want to look into putting their site up on a 3rd party managed CMS-service. Blogs (like this) can be run here on blogger (or WordPress, TypePad, ect. Also offer solutions). If blogs are not what your looking for, Google freely offers “google sites”. There are many other managed CMS hosting services available, check with your chosen hosting provider. A major benefit to these services is to leave the server-side security (and incident handling) on them instead of you. You would just need to keep up with your passwords, email, PC security just like with your facebook, or myspace pages.

For those that just cant find a managed solution for their needs (or just dont want to use them/enjoy more freedom then the provided solutions), monitoring the security of your sites can be an ongoing battle. In the following introduction, you will find helpful weapons to add to your arsenal.

I find dealing with linux servers the easiest, (and also the most common hosting solution) so I will be primarily discussing such. If your site is on a hosting account without ssh access, upgrade your account, or find a new host.

SSH, and the linux server's terminal applications (grep, awk, sed, cat, less, ect.) are your best friends. Get to know them. Google searching for their “example uses” will greatly help. If you get stuck with any of them, chances are the 'man' application will give you much more info about the options of these apps than you will ever need.

Server-side scripting exploits can be seen in the apache access logs. If you dont know where these are for you, find out. Apache log files can be viewed (searched, and their data formated more friendly) using your ssh login, and the terminal apps.

For example: RFI attacks (“remote file include” - usually the result of include statements not being sanitized correctly, allowing remote scripts to be run on your server with your users permissions) show up in the apache access logs as a line including the apache code: '200'. This is the code for successful access, although this can still be misleading if you have custom error pages, it could indicate that there was an attempt which only successfully returned the error page. Weather just an attempt, or successful, RFI attack log entries usually include a 'POST' or 'GET', and a link to a remote file such as: '=http://SomeDomain.com/MalitionsHackerScript.php'.

To run the following example (1-liner) command, to look for RFI attack attempts (including the mentioned false-positives), you would want to ssh into your webspace, and change directory to your 'logs/' (or whatever the folder name of your logs).

Example 1-liner:

zcat ./access.log.* |grep -i '=http' |grep ' 200 ' |grep -iE 'txt|php|cgi'

Now lets break down that above call:

zcat ./acess.log*

This command outputs all of the lines in your apache access logs, and then its output is “piped” to the next search command:

grep ie '=http'

Which searches for occurrences of '=http' or '=HTTP'. The output of this is then “piped” to the command:

grep ' 200 '

Which searches for occurrences of the successful apache code '200'. the output of this is then “piped” to:

grep -iE 'txt|php|cgi'

That then searches and outputs only the lines that include common extension for hacker scripts such as 'txt', 'php' or 'cgi'. There are others, or non-normal-pattern files, and scripts used also, so you could just replace this like with 'less' to just view all the posts or gets of the previous search commands. Although this will also show many potential false-positives that are just links from/to your site from/to elsewhere.

To further explain the above used grep options or “switches”:

For 'grep -i',  the 'i' option tells grep to not count the case of the characters, so picks up '=http', '=Http', '=hTtP' or '=HTTP', ect...

The 'E' option in 'grep -iE' tells it that we are looking for an expression (in this case any occurrences of: txt, php, or cgi), instead of just 1 specific text.

--
Anyhow, thats more than I meant to go into for one post, but its a starting point to one look into their websites security.

keep on keeping on, and bad guys be gone...

Hacked Website or Web-space

Web-space:  If your web-space has been compromised you would want to take steps to re-secure any information which may have been collected by the hacker (such as passwords, and contact information) and re-secure the web space its self.

   A.)  Take steps to update any passwords that are used or available in your web-space or databases. You may also need to inform any other users of the potential compromise.

   B.)  As with security of your personal computer, it may be best to “re-image,” or delete all files/databases associated, and re-upload from (known secure) backup or new updated code and files.

      1.)  Make Backups frequently! This way you always have a recent secure backup to restore from.

   C.)  Update any applications (Wordpress, Joomla, or any other CMS or server-side scripts) you are running.  Contact the vendor/developer for updates if needed, and be sure to always keep up to date with security patches and advisories.

Credit Card Fraud, and Identity Theft

Have you been the victim of Online Fraud, Identity Theft, or some other form of Internet Abuse? The following outline contains steps you may take to help re-secure your information.


I.)  Your Credit Card: If your Credit Card has been compromised, contact your card provider.

   A.)  Contact the telephone number usually listed on your credit card.

   B.)  Have them cancel your card number and issue a new number. Some providers also allow you to place a "security lock", or "fraud-watch" on your account. Ask your provider if they offer any such service.

   C.)  Review your credit card statement for any fraudulent charges and dispute those charges through the card provider.

   D.)  Contact any vendors who post these charges as they may not be informed by the card provider that the purchase was a fraud, and may forward any account created with your information to a collections agency.  This may then result in a blight on your credit report.

II.)  Your Identity: If accounts are created fraudulently using your name/contact information.

   A.)  Contact your local law enforcement agency (start with your local police).

      1.)  File a claim of "Identity Theft", and provide them with all the documentation you have regarding the incident.  While the local authorities may not actively pursue smaller claims frauds, getting a police report gives legal documentation of the incident.

         i.) Changing your credit card information is easy and stops the initial credit card fraud. However, a malicious person has already pretended to be you, and likely has your contact details.  They may continue using your information fraudulently as their own.  Having police report gives you evidence that you have been the victim of Identity Theft.  You can show this documentation to future agencies attempting to collect on further accounts which may be set up by this other "you".

      2.)  The local law enforcement may also contact or direct you to contact the FBI Cyber crimes division.  At which point you would need to report a complaint through the "Internet Crime Complaint Center".  This site is available at the following URL: http://www.ic3.gov/default.aspx

PC Security - dealing with viruses, spyware, and malware

Your Personal Computer:  The security of your personal computer.

A.)  Regularly apply all security updates provided by your operating system vendor.

B.)  It is good practice to scan for viruses, spyware, and malware frequently. Im not going to go into which software I would suggest for these purposes here, since there are many to choose from, and i havnt tried them all.  There are many free ones that will manage the primary issues needed, but many payed applications have many bonus features.

    1.) On-Access protection:  Many anti-virus applications have "on-access" protection which may result in your computer running slower, but this adds an extra layer of security scanning before files get written to your hard drive.  This is good idea to keep running in most situations.

    2.) Firewall:  Monitor your firewalls (these can be hardware embedded, OS system created, or included in many anti-virus applications).  Do not enable any ports to be open that you dont need to use for a service.

    3.) No-Script:  many infections, and attacks are a result of malicious scripting on a compromised (or intentionally malicious) website.  If you use firefox (and why wouldn't you at least over IE) you can install the noscript extension which blocks all client side scripting by default.  Only after you accept the validity of the source and "white-list" it does this extension allow those scripts to be run.

C.)  If your computer is compromised, (there is NO anti-virus which is 100% effective) you may need to "re-image" the system -- this involves formatting and re-installing the operating system and any programs, before virus-scanning and restoring your files.

    1.)  Alternatively, you may wish to use a linux "live cd" -- since it is a known secure OS to boot and work from and update your passwords, and online account security.  A couple free examples that you can download are Knopix, or Ubuntu.  These can also be installed to the hard drive, and run much faster installed than from the "live cd".