Bash Line Commands & Shell Scripting

Overview

It is useful to be able to use Bash Shell commands to gather information about a test target as well as filter that information for only the necessary useful data. For example you could use “wget” to download the contents of a web page and then sort through and filter the results for links, domains, subdomains, and IP addresses that can be tested later for vulnerable open services.

#wget “http://www.some-website.com”

You can also use Bash shell commands to filter and sort through any type of resulting text output. In the example below I run an Nmap on a LAN and then filter and sort through the results.

Exercise 1 – Scan a network for IP addresses and open ports. Sort and filter your data with Bash line commands.

Try running these steps which have useful commands:

  1. In a test network environment or one that you have permission to scan with Nmap. Try the following commands in a root terminal:
  2. Run a Nmap scan on an entire network looking for responding hosts computers and open ports:
    #nmap 192.168.1.1-254  (substitute your network’s address scheme)
  3. Output/save the results to a text file:
    #nmap 192.168.1.1-254 > scan.txt
  4. Cat the results to your terminal:
    #cat scan.txt
  5. If there are active responding computers in the network you may see the following sample line: “Nmap scan report for 192.168.1.100.” Now you can use Bash to filter the results piping to a grep command:
    #cat scan.txt | grep “report”
  6. Using the cut command and specifying a delimiter of a space (-d ” “), you can segment the text string into separate pieces based on the spaces between the words and then grab the field you need (-f 5).
    #cat scan.txt | grep “report” | cut -d ” ” -f 5
    This way the result is just the responding IP addresses from the Nmap scan. Example:
         Nmap scan report for 192.168.1.100
    Nmap scan report for 192.168.1.101
    Nmap scan report for 192.168.1.102
    becomes just:
    192.168.1.100
    192.168.1.101
    192.168.1.102 
  7. If you needed to sort the data because of redundant addresses you could use the sort command to alphabetize and remove duplicate data:
           #cat scan.txt | grep “report” | cut -d ” ” -f 5
  8. The initial Nmap command that was issued against the network, would also have shown any open ports and services that could be contacted if we wanted to filter the results to show the IP addresses followed by the open port numbers we could try and format our Bash command to search for any lines with the string “report” and or any lines with the string “open” using an escape backslash and a pipe (\|) as an OR logical operator like this:
    #cat scan.txt | grep “report\|open”
  9. Now you need to pipe to the cut command to clean the results. What would your final command look like so your output would look clean? Hint, add more pipes and more cuts to the command to strip away the unnecessary words.

 

Exercise 2 – Download a webpage, sort and filter the results for subdomains and servers 

Try running these steps which have useful commands:

  1. Download the yahoo homepage using wget:
    #wget “http://yahoo.com
  2. Examine the results using cat:
    #cat index.html
  3. Now we need to try and filter our results to subdomains of yahoo.com. To do this we can search for the lines with matching text yahoo.com:
    #cat index.html | grep “yahoo.com”
  4. Your result should be be too large to be of use. If we want to continue to filter the result we can assume that most of the domains have a “http://” in front of them, and we can try cutting the lines based on a delimiter of a colon and taking the second field
    #cat index.html | grep “yahoo.com” | cut -d “:” -f 2
  5. Now most of the lines will start with // so we can cut based on a “/” and grab the 3rd field
    #cat index.html | grep “yahoo.com” | cut -d “:” -f 2 | cut -d “/” -f 3
  6. Remove duplicate lines with sort
    #cat index.html | grep “yahoo.com” | cut -d “:” -f 2 | cut -d “/” -f 3 | sort -u
  7. If you need to remove the trailing characters from a line with a domain name. Like in this example:
    info.yahoo.com” this is a sample of training text
    then you may have a problem because the trailing character starts with a quotation mark. In this case you will need to use a backslash to escape the quotation mark and have it function as a string instead of a demarcation point. See below:
    #cat index.html | grep “yahoo.com” | cut -d “:” -f 2 | cut -d “/” -f 3 | sort -u | cut -d “\”” -f 1
  8. Save your results to text
    #cat index.html | grep “yahoo.com” | cut -d “:” -f 2 | cut -d “/” -f 3 | sort -u | cut -d “\”” -f 1 > domains.txt
  9. Output your text file to the screen and continue to sort and cut and grep until you have only unique subdomains
      #cat domains.txt | sort -u | grep “yahoo.com”
  10. Output the results overwriting the text file:
      #cat domains.txt | sort -u | grep “yahoo.com” > domains.txt

You can see from the list above that there are actually many different subdomains, services and servers

Exercise 3 – Write a Bash shell script to automate host and ping commands 

Write a shell script which will loop through a list of domain names, issuing a host command on each name, in order to return the IP address that maps to each name.

  1. Using a text editor like nano or gedit or vim open a text file and name it ips.sh
  2. You will write a shell script file that will loop through the list of subdomains you created above and saved as domains.txt. Type the following lines of text into your ips.sh file:
    #!/bin/bash
    for name in $(cat domains.txt);do
    host $name;
    done
    Save the file and exit. This script will loop through the domains and issue a host command on each domain name.
  3. Give the script file executable permissions:
    #chmod 755 ips.sh
  4. Run the scripts:
    #./ips.sh
  5. Output the results to a text file:
    #./ips.sh > server-ips.txt
  6. Clean up the results. Rewrite the shell script adding pipes, grep, cut and sort commands so that it produces only ip addresses.
  7. You should now have a file with only the IP addresses that resolve to each subdomain.

Write a shell script which will loop through a list or sequence of IP addresses, issuing a single ping command on each IP address, in order to determine which IPs will respond.

  1. Using a text editor like nano, gedit or vim open a text file and name it ping.sh
  2. You will write a shell script file that will loop through a sequence or list of IP addresses like the one you just created. Type the following lines of text into your ping.sh file:
    #!/bin/bash
    for ip in $(seq 150 250);do
    ping -c 1 192.168.1.$ip
    done

    the (seq 150 250) command when issued will loop through all the numbers between 150 and 250
    or you could use the list of ips in the text file you created earlier,
       #!/bin/bash
    for ip in $(cat server-ips.txt);do
    ping -c 1 $ip
    done
  3. Save the file, give it executable permissions, and run it:
            #chmod 755 ping.sh
    #./ping.sh
  4. Edit your script file by adding grep, cut, and sort commands to clean up the results so that only replying IP addresses are listed

Additional Study

There are more powerful tools for filtering and replacing text like awk, sed, and higher order scripting languages like Perl, Python, Ruby, Php, and regular expressions . It is worth your while to learn how to use these languages.

Week 2 – Basic Tools

{loadposition adposition4}

{loadposition adposition5}Introduction

Data Mining – Can be used as a form of reconnaissance when hackers gather data from websites or scan for open services and ip addresses in networks using basic Bash Shell commands, shell scripts and higher order scripting languages. This practice can help an attacker gather necessary information about a potential victim or organization. Useful information like host IP addresses, open services and ports can be quickly ordered and filtered, using Bash Shell scripting into stripped down useful lists and saved into text files, which can later be uploaded and used in attacks. 

Week 2 Assignment

  1. a) Using BackTrack, VPN (online) or connect (SDL) to the network security lab, using NMAP scan the network for open ip addresses, services and ports. Using basic Bash shell commands, filter the output from NMAP so that only responding ip addresses and open ports are listed in the results. Save your list of open IPs and ports by sending the output to a text file. b) Download a webpage using "wget" and filter the results to find all unique subdomains. Save your list of subdomains to a text file. c) Right a Bash shell script that will automate host or ping commands to a range of ip addresses to determine which computers are responding. Go here for detailed instructions and examples.

 

{loadposition adposition8}

 

Run Services (SSH, FTPD, HTTPD)

Introduction

If we are going to penetrate victim computers (in the test lab) and establish communication to and from those clients so that we may execute commands and transfer files we will need some network services like SSH and FTP.

SSH 

SSH or secure shell allows you to open a secure terminal connection to a remote host, or simply to login to a remote computer in order to execute commands. This is done through a client server model where a SSH client connects to a SSH server. This type of connection is typically used to securely exchange encrypted data with a remote host, with the data being commands but can even be files using SFTP and SCP which utilize the SSH protocol. SSH was developed to replace non secure remote connection protocols like rlogin, telnet and rsh. SSH typically runs on port 22. SSH should be installed by default in BackTrack.

Manual pages on SSH and the SSHDaemon
   #man ssh
#man sshd

Generate the secure keys for the host service
#sshd-generate
or
#ssh-keygen

Start the SSH server using the scripts in the init.d directory
#/etc/init.d/ssh start

Check to see if SSH is listening on Port 22
#netstat -antp

Since you are now running an SSH server, you could download PuTTY to your Windows client and SSH into your BackTrack machine. Try this, was it successful? If not, where might the block be? Are you VPN’d? Are there any firewalls on the client or on an intermediary network device?

Stop the SSH server using the scripts in the init.d directory
#/etc/init.d/ssh stop

If you want instructions on how to install SSHD in Ubuntu go here.

HTTPD (Apache)

Apache webserver is the most widely used webserver on the internet. From a network penetration perspective a webserver could be used an exploitation tool serving up malicious files and scripts that will execute against a victims browser and computer. It could also be used as a way of transferring files to a victim machine once access has already been accomplished. Webservers by their very public nature are great way for a hacker getting information or reconnaissance about a company or target. Apache webserver is usually run on port 80.

Start the Apache2 server using the scripts in the init.d directory
#/etc/init.d/apache2 start

Verify if Apache is listening on Port 80
#netstat -antp

Your server’s accessible web directory is located in /var/www/ and is where you would put your webpages.

If you want instructions on how to install HTTPD Apache in Ubuntu go here.

FTPD

FTP or file transfer protocol is a protocol that allows you to transfer files to and from a remote machine. BackTrack has pure-FTPd installed by default. FTP operates on port 21 and transfers files on port 20.

The steps to setting up pure-FTPd:

Installation (pure-ftpd is installed by default in BackTrack but in case you are using another Linux distribution)
  #apt-get install pure-ftpd
#yum install pure-ftpd

Configuration to add a FTP user and setup an FTP directory
#groupadd ftpgroup
#useradd -g ftpgroup -d /dev/null -s /etc ftpuser
#pure-pw useradd <your ftp username> -u ftpuser -d /home/ftp/pub/<your ftp username>
You will be prompted to input a password twice
#pure-pw mkdb
#cd /etc/pure-ftpd/auth
#ln -s ../conf/PureDB 60pdb
#mkdir /home/ftp
#mkdir /home/ftp/pub
#mkdir /home/ftp/pub/<your ftp username>
#chown -R ftpuser:ftpgroup /home/ftp/pub/<your ftp username>
#/etc/init.d/pure-ftpd restart

Test you install of Pure-FTPd by FTPing to your loopback ip address
#ftp 127.0.0.1
key in your <ftp username>
key in your <ftp password>

Install & Configure BackTrack

Overview

You will need to download the BackTrack 4 or 5 operating system which you can find at: http://www.backtrack-linux.org/downloads/ . You can download either an .iso file which you will need to burn as a disc image or you can download a VMware image of the operating system which you can open as a virtual machine through the VMware Player in Windows. If you decide to use the VMware image you will need to download the free version of VMware Player as well. You can find it here: http://downloads.vmware.com/d/info/desktop_downloads/vmware_player/3_0

Once you have BackTrack installed and running you will want to know how to do the following things:

Login, the default is:
login: root
password: toor

Change the root password:
#passwd
<enter your new password twice>

Launch the Xwindows graphical user interface:
#startx

To bring up your network interfaces and be able to get online:
Check what interfaces you have (eth0, eth1, wlan0, etc.)
#ifconfig

For a wireless network card running DHCP
#iwconfig wlan0 mode managed key <your wep key>
#iwconfig wlan0 essid <your ssid>
#ifconfig wlan0 up
#dhclient wlan0

For a wired network card running DHCP
#ifconfig eth0 up
#dhclient eth0

For a manual configuration instead of running the DHCP client
#ifconfig eth0 <your desired ip address and /subnet mask -e.g. 192.168.1.100/24>
#route add default gw 192.168.1.1
#echo nameserver 192.168.1.1 > /etc/resolv.conf

Now test your connection by running a ping command or opening a website in your browser

Video Tutorial

 

Week 1 | Network Penetration Testing

{loadposition adposition4}

Introduction

I am really excited to be teaching this class. For a long time, I wanted to learn about network penetration and recently I got the opportunity to learn more about it. I was absolutely amazed and scared at how effective exploits can be on insecure networks, computer systems, and software.

{loadposition adposition5}In this class we will learn how to secure computer networks and clients from attack by learning the tools and tactics employed by computer attackers. The purpose of penetration testing is to learn first hand the various types of exploits and attacks used by hackers by using those same exploits in a testing environment and applying defensive measures to secure the network and defend against those attacks.

I hope you have the same eye opening experience I had learning about penetration testing and the practice of computer exploitation.

Class Materials

In this class we will use a specially designed security distribution of Linux to learn about computer exploitation and defense. You will need to download the BackTrack 4 operating system which you can download at: http://www.backtrack-linux.org/downloads/  . You can download either an .iso file which you will need to burn as a disc image or you can download a VMware image of the operating system which you can install as a virtual machine through the VMware Player in Windows. If you decide to use the VMware image you will need to download the free version of VMware Player as well, you can download it here: http://downloads.vmware.com/d/info/desktop_downloads/vmware_player/3_0

You will also have access to the Sheridan College Security Lab where you can practice network penetration attacks and defense. The lab is available for online students through VPN access 

Class Schedule and Policies

This class is offered either as Self Directed Learning-SDL, or Online. SDL students are required to be in the Lab – GMB133, a minimum of 2.5 hours per week you will log your hours in the lab. Here are some important questions you will want answered:

  • When can I come to the Lab?{loadposition adposition6}
    I am in the GMB133 class/lab over 30 hours per week, for my exact lab times click on Lab/Classroom Hours.
  • Can I work in the Lab when you are not there?
    Yes, if the door is open you are welcome to come in and work. If the door is locked try coming into the lab through GMB130 the Flex Lab. The Flex Lab opens Mon-Sat at 9am and is open late Tuesday until 9pm.
  • Can I work with BackTrack in the lab?.
    Yes, we can install BackTrack on lab computers and we have a special testing environment to practice the security labs.
  • What if I am an online student, and I can’t work in the Sheridan College Security Lab?
    If you are an online student you will be able to work in the security lab by using a VPN connection to connect into the lab from home. 
  • When should I come to class (SDL students only)?
    Remember SDL students are required to be working in the lab 2.5 hours per week. It is up to you to decide exactly what your schedule will be. For this class, I will be giving specially designed presentations on Thursday nights from 6 to 9pm. You do not want to miss these presentations if at all possible. They will be run by me and Steve B.
  • How can I assure that I will be successful?
    The students that show up regularly, week by week, both online by logging in to DansCourses and in person by coming to the lab are the ones that will be successful. The class will not be finished unless you sit down and create a schedule of times that you will work on the class and its assignments.

  • Is there a syllabus?
    The syllabus outlines the course, please download it and save it for your records: To be posted …
  • How will I turn in assignments?
    All assignments will be turned in as email attachments todan.alberghetti@gmail.com  
    • How will I be graded?
      You will be graded on a point system based on weekly assignments and projects. 
    • I am really interested in obtaining a security certification what certification and further course of study do you recommend?
      I recommend the OSCP certification through Offensive Security the makers of BackTrack. The Pentesting with BackTrack Online is an excellent course of study for those interested in an in depth study of the topic. Go here for more information: http://www.offensive-security.com/information-security-certifications/

{loadposition adposition9}

Week 1 Overview

This week you will need to get set up with tools and resources that you will need in order to be successful.

Week 1 Assignment – Install and configure BackTrack4 and OpenVPN

  1.  Install BackTrack4 – As I outlined above you will need an install of BackTrack4 to make the class easier. It is possible to install all of the necessary tools that the BackTrack4 operating system offers into any distribution of Linux. However it would be easier to install BackTrack4 as a virtual machine in the VMware player or in VirtualBox. I myself have a laptop install of BackTrack4 and also a VM that I can use on my Windows 7 computer too.
  2. Verify and Configure OpenVPN – OpenVPN is installed by default in BackTrack4, however you will need to configure it in order to connect to the lab from home. I will send you a username and password as well as the OpenVPN configuration file settings.
  3. Install and Configure Various Services – We will be using various services during the class. You will need to install, configure and know how to start and stop these services. I will post further instructions related to this during the week.
  4. Remote into Your Practice Lab Victim/Client – Everyone in the class will be given access to a computer to use as a practice target for various exploits and attacks. I will be giving you the ip address, username and password of the client assigned to you.

This assignment is finished when you have emailed me screenshots that prove the steps above have been accomplished.

{loadposition adposition8}