Redirectors and Pipes

Overview

If you are going to use bash shell scripting to create programs that will manipulate computer data, then it is useful to control what data is input to a command, control where the data is output, and also control where error messages are output.

File Descriptors

In the bash shell, command input and command output can be manipulated. For every command that can be used in bash there are three file descriptors: standard input (stdin), standard output (stdout), and standard error (stderr).

• Standard input – is the information that is passed to a command. The standard input information can be input by the user through the blinking prompt, or it can be passed to a command from a file or program. Standard input is also represented by the number 0 (see below)

Standard output – is the data output after a command has executed. Standard output is also represented by the number 1 (see below)

Standard error – are any error messages that may have been generated by a command. Standard error is also represented by the number 2 (see below)

Redirectors

In bash, standard input, standard output and standard error messages are manipulated by redirect command characters. Redirects control how data is input and output from files to commands and vice versa:

  • The output redirector (> or 1>) redirects standard output to a file instead of the screen. Example:
      $ ls -l /var > mydirectory.txt
    $ ls -l /var 1> mydirectory.txt
  • The input redirector (< or 0<) redirects standard input from a file. Example:
      $ ls -l < /var
    $ ls -l 0< /var
    $ cat < mydirectory.txt
      $ cat 0< mydirectory.txt
  • The append redirector (>> or 1>>) appends standard output to the end of a file instead of rewriting it. Example:
      $ ls -l Documents > myfiles.txt
    $ ls -l Downloads >> myfiles.txt

    $ ls -l Pictures 1>> myfiles.txt
  • The output redirector with a 2 (2>) redirects standard error to a file. Example:
      $ ls -l Pixxtures 2> errors.txt
  • The output redirector with a 1 and a 2 (1> 2>) can be use to redirect standard output and standard error to two different files. The number one in the first redirector is not necessary since a greater-than sign by itself implies the number one, standard output. Example:
      $ ls -l Documents Pixxtures > good.txt 2> errors.txt

Pipes

The pipe is a special command character (|) which can take the standard output (stdout) from one command and make it the standard input (stdin) for another command. In the example below, the command ls -l has been issued to list the contents of the /etc directory, but before the output can be sent to the screen, it is piped (|) and turned into standard input for the more command to process. The more command takes the input and executes its program, which processes the information and outputs it to the display it one screen at a time.

  $ ls -l /etc | more

In this next example, the pipe is used to take the data output from the ls -l /etc command and send it as input to the grep command to search for a specific lines of text that have the text “firefox” in them.

  $  ls -l /etc | grep “firefox”

Pipes are very useful because they allow you to send information from one command to another in sequence. Pipes and redirectors can be used together very powerfully to control how data is input and output.

Video Tutorials

In this tutorial, I demonstrate using redirector characters to control standard input, output, and error

In this video, I demonstrate how to use of the pipe special character and the
grep command, to filter output to find specific lines of content

Shell Scripting with Bash

Overview of Bash Shell Scripting

The default Linux command line interface or terminal is the Bourne Again shell or bash shell. A shell is a user interface to a computer system that relies on keyboard input, as opposed to a graphical user interface which relies on keyboard input, mouse input, and presents the user with graphical icons and windows to click on with a mouse.

Since bash is a command interpreter, it has powerful programming capabilities like a full fledged programming language. In addition to entering commands in the terminal one at a time, you can write commands into a text file, save it, and execute it like a program. An executable file with shell commands is called a shell script or shell program.

To make a shell script executable, you have to give the text file, execute permissions. You do this with the chmod command:

  $ chmod + x <filename>

If you try to run a shell script that does not have execute permissions you will get a Permission denied message. You can start by learning how to create a simple “Hello World!” shell script.

A “hello world!” shell script

1. Start by opening a Linux terminal.

2. In the terminal, create the shell script file using a text editor like Nano. If you like, you can name the file with a .sh file extension for visual reference that it is a shell script. The file extension is optional, since Linux does not rely on file extensions like Windows to determine the file type. The command structure: $ nano <filename>

  $ nano hello.sh

3.  With nano open you can type your first shell script. The first line in the script starts with a hash-bang (#!) followed by the path to the bash program. This lets the terminal know that it is bash specifically, that you want to use to execute this script. This is important since there are other terminal shells, like the korn shell, c-shell, the tcshell, etc.. Type the following text into nano, then do control+x on the keyboard, type y for yes, and press enter, and then press enter again to accept the file name.

  #!/bin/bash
echo “Hello world!”

4. Now that you have the file saved, do a cat command to verify that the text saved to the file. If it was saved, you will see the text of hello world returned. The command structure: $ cat <filename>

$ cat hello.sh
Hello World!

5. Now you need give the file execute permissions. The command structure: $ chmod +x <filename>

  $ chmod +x hello.sh 

6. Now do a ls -l command to see if execute permissions were applied. The command structure: $ ls -l <filename>

  $ ls -l hello.sh

The output should look like this. If the line leads with a “d” it means it is a directory and if it is a “-” it means it is a file. Notice that after the file character “-“, the permissions on the hello.sh line read rwxr-xr-x, indicating that the owner is read, write, execute (rwx), followed by the group permission of read and execute (r-x), and the public, or everyone, which is also read and execute (r-x). The execute bit was successfully applied to all three groups: owner, group, and everyone.

  drwxr-xr-x 2 dan dan 4096 Jan 12 16:17 Desktop
drwxr-xr-x 4 dan dan 4096 Jan 17 19:53 Documents
drwxr-xr-x 3 dan dan 4096 Jan 16 07:47 Downloads
-rwxr-xr-x 1 dan dan   43 Jan 24 19:25 hello.sh

7. Now that your shell script file has execute permissions you can run it as a program. Since the file is not saved in a directory that is included in the $PATH variable you cannot run it like a program. You can verify this by simple trying to run your shell script by typing $ hello.sh you should see the command not found returned. However, you can run your program by giving the absolute path to the file or referencing the current directory with a “./”.

  $ /home/user/hello.sh
Hello world!

or simply,
  $ ./hello.sh
Hello world!

8. You have the basics on creating and running a shell script in the bash shell. Now, expand your knowledge by  learning additional programmatic capabilities that can be used to write more advanced scripts that will automate basic to advanced system tasks.

A shell script to backup your home directory

1. Open a Linux terminal.

2. Create the file in nano

  $ nano backup.sh

3.  Start typing your shell script. This time you can use the echo command to explicitly output what it is the script is attempting to do. You will use the tar command to create a compressed file of your home folder and save it to the var directory. The command structure is: $ tar -czf <destination-directory> <file-to-backup>. After the tar command, you can use the echo command to notify the user that the backup process is complete. Control+x and save the file. You will need to change /home/dan to the path to your home directory.

  #!/bin/bash
echo “Backing up home directory”
tar -czf /var/homebackup.tgz /home/dan
echo “Done!”

4. Do a cat command to verify that the file was saved correctly. The command structure: $ cat <filename>

$ cat backup.sh

5. Give the file execute permissions. The command structure: $ chmod +x <filename>

  $ chmod +x backup.sh 

6. Now do a ls -l command to see if execute permissions were applied. The command structure: $ ls -l <filename>

  $ ls -l backup.sh

7. Now that your shell script file has execute permissions, go ahead and run it. You will notice that the script fails because you do not have super user permission to save to the /var directory. Run the script again with a sudo command in front, enter your password, and you will see that the shell script file executes successfully.

  $ ./backup.sh
  Backing up home directory
tar: Removing leading `/’ from member names
tar (child): /var/homebackup.tgz: Cannot open: Permission denied
tar (child): Error is not recoverable: exiting now
Done!

  $ sudo ./backup.sh
 
Backing up home directory
  tar: Removing leading `/’ from member names
  Done!

Video Tutorials

In this video, I write a basic shell script, give it execute permissions, and run it as a program


In part 2, I write a basic shell script to backup the user’s home directory 

Install Linux Mint 14.1

Overview

Linux Mint is a distribution of the Linux operating system that consists of both open source software and some proprietary commercially based software as well. Linux Mint has versions of its operating system that are based on both the Ubuntu Linux and the Debian Linux distributions.

Linux Mint version 1.0 was first released in 2006. In 2012, Linux Mint version 14, codename Nadia was released. Linux Mint is released with multiple versions that support different desktop environments by default. In the tutorial video below, I install Linux Mint 14.1, the Mate desktop environment that is based on the Gnome2 desktop environment.

You can find links to download the different versions of Linux Mint from their website at http://linuxmint.com.

 

Video Tutorials

In this tutorial, from within Windows 7, I install Linux Mint 14.1 in a VMware virtual machine

In this video, I install VMware Tools so I can use my Linux Mint virtual machine in full screen mode

Install & Configure the BIND DNS Server

Overview

BIND also known as NAMED is the most widely used DNS server software in the world. The Berkeley Internet Name Domain (BIND) is  domain name server software that can run on Linux, Unix, and Windows computer operating systems.


Command Steps

1. Make sure you have internet connectivity and install the BIND DNS server.
yum install bind

2. Set your DNS server setting to resolve to your loopback interface by echoing to /etc/resolv,.conf or by editing your your outside interface configuration file (example: ifcfg-eth0) adding the line: DNS1=127.0.0.1, and taking the interface down and up again (ifdown eth0, ifup eth0). The benefit to ediing the ifcfg file and then bringing the interface down and up again is that the confguration will be permanent, whereas echoing to resolv.conf is not a saved configuration. Once you set the DNS server setting to 127.0.0.1 you will lose internet connectivity until you restart BIND.
echo “nameserver 127.0.0.1” > /etc/resolv.conf 
or
vim /etc/sysconfig/network-scripts/ifcfg-eth0
add the following line to the configuration file: DNS1=127.0.0.1
ifdown eth0
ifup eth0

3. Check the resolve.conf file to verify that your DNS setting of 127.0.0.1 is available.
cat /etc/resolv.conf

4. Restart the BIND DNS server. The BIND DNS server is referenced by command as “named”  pronounced “name – d”, the name daemon.
service named restart 

{loadposition adposition6}

5. Now try to see if your DNS server can reach other DNS servers over the internet in order to resolve dns lookups.
nslookup google.com

Troubleshooting – If you get a “server can’t find google.com SERVFAIL” message you can try these troubleshooting scenarios:
1) If you are using VMware virtual machines try shutting down your Windows firewall
2) Make sure your iptables is accepting traffic on the loopback interface. Use the iptables-save command and you should see the following line: -A INPUT -i lo -j ACCEPT . If not, you can add it directly to the iptables configuration file by editing /etc/sysconfig/iptables. Then restart iptables: service iptables restart
3) Also, check that you have the right date set on your system. Use the
date command to check the date and timestamp. You can fix it with the following command: date -s ‘year-month-date time’ (e.g. 2012-4-23 12:10:00). I experienced DNS server nslookup failure due to incorrect date.
4) If you still have problems and are unable to resolve domain names using nslookup then try looking at your log files: tail /var/log/messages

6. Now put in a chkconfig command to allow BIND (NAMED) to start on system startup.
chkconfig named on

7. Before you begin configuring the DNS server you should know what your hostname. You were asked to create your hostname (computer name) during your CentOS linux installation. Your hostname follows your username and the “@” symbol in your terminal command prompt. For instance, my terminal shows the following prompt [dan@centos-server ~]$, “dan” is the username, “centos-server” is the hostname, ” ~ ” refers to my current directory – which is home, and the “$” prompt means I am in user mode as “dan” (a # would mean I am currently the root user). To verify your hostname type the command hostname in the terminal and hit enter.
hostname

You can temporarily change the hostname by issuing the hostname command. However this will not be permanent upon restart.
hostname <example-server-name>

To permanently change the system hostname even upon reboot, edit the following file in a text editor: /etc/sysconfig/network
nano /etc/sysconfig/network
and add the following line after NETWORKING=”yes”:
HOSTNAME=”example-server-hostname” 

8. Beyond knowing the server hostname, to use the full potential of the DNS server you will also need to know your fully qualified domain name (FQDN). You will probably need to configure your servers FQDN. To do this you will need to edit the configuration file of your outside facing network interface using a text editor, in my example the file is ifcfg-eth0.
nano /etc/sysconfig/network-scripts/ifcfg-eth0

and add your domain name by adding a line to the configuration file:
DOMAIN=”example.com”

{loadposition adposition6}then take the interface up and down. Check to see if the change is reflected in the /etc/resolv.conf file, and the run the hostname –fqdn command to see your FQDN. The FQDN should be your server hostname followed by a dot and your domain name (e.g. example-server-hostname.example.com). If your computer does not recognize your fully qualified domain name (fqdn), just keep going, the server should recognize the FQDN by the time you have finished configuring the DNS server conf file and zone files.
ifdown eth0
ifup eth0
cat /etc/resolv.conf
hostname –fqdn

9. Now that your hostname and fully qualified domain name are configured it is time to configure the BIND (NAMED) DNS server.The first file to configure is: /etc/named.conf
nano /etc/named.conf

10. Now you can add master lookup zones to the named.conf file like the entries in my example below. I inserted the two zones right before the line ( zone “.” IN { ). The first zone is for forward lookups and the second zone is for reverse lookups. Forward lookups resolve names to IP addresses and reverse lookups resolve ip addresses to names. Make sure to substitute your own domain name and the network portion of your IP address in reverse (outside interface).

 zone “example.com” IN {
type master;
file “example.com.zone”;
allow-update { none; };
};

zone “1.168.192.in-addr.arpa” IN {
type master;
file “example.com.rr.zone”;
allow-update { none; };
};

11. You can see in the two zones above that the third line in each zone references a file (example.com.zone and example.com.rr.zone). You will need to create two text files, one to match each of those names, and save them in the /var/named/ directory.
touch /var/named/example.com.zone
touch /var/named/example.com.rr.zone

12. Now you will need to edit and save each file with zone configurations. Make sure to substitute your own domain name and IP address (outside interface). First, the forward lookup zone file.
nano /var/named/example.com.zone

$ORIGIN example.com.
$TTL 86400
@    IN    SOA    dns1.example.com.    hostmaster.example.com. (
2001062501 ; serial
21600      ; refresh after 6 hours
3600       ; retry after 1 hour
604800     ; expire after 1 week
86400 )    ; minimum TTL of 1 day

IN    NS    dns1.example.com.

IN    MX    10    mail.example.com.

IN    A    192.168.1.113

dns1    IN    A    192.168.1.113

example-server-hostname    IN    A    192.168.1.113

ftp    IN    A    192.168.1.113

mail    IN    CNAME    example-server-hostname

www    IN    CNAME   
example-server-hostname
13. Second, the reverse lookup zone file.
nano /var/named/example.com.rr.zone

$ORIGIN 1.168.192.in-addr.arpa.
$TTL 86400
@    IN    SOA    dns1.example.com.    hostmaster.example.com. (
2001062501 ; serial
21600      ; refresh after 6 hours
3600       ; retry after 1 hour
604800     ; expire after 1 week
86400 )    ; minimum TTL of 1 day

@    IN    NS     example-server-hostname.example.com.

1    IN    PTR    example-server-hostname.example.com.

2    IN    PTR    dns1.example.com.

3    IN    PTR    ftp.example.com.

4    IN    PTR    mail.example.com.

14. Now restart your server and try resolving your domain names with nslookup. You should see that they resolve to your server!!!
service named restart
nslookup example.com
nslookup dns1.example.com
nslookup mail.example.com
nslookup ftp.example.com
nslookup www.example.com
nslookup example-server-hostname.example.com

Video Tutorials

In part 1, I install and troubleshoot the BIND (named) DNS Server in CentOS Linux

 

In part 2, I set the hostname, domain name, and edit the named.conf configuration file

In part 3, I configure the master forward and reverse lookup zones

Route, NAT, and Transparent Proxy, with CentOS Linux – Part 2

CentOS Routing and Proxy Cont.

In the previous article, I covered installing CentOS server and converting it to a router, address translator, and transparent proxy. I also covered installing a DHCP server to serve IP addresses to the interior local network. The entire process was recorded in a series of video tutorials (see previous article) and posted to YouTube.

Now that we have configured the iptables forwarding rules, manually configured our interfaces with IP addresses, as well as the default gateway and our DNS nameservers we need to save our configurations. We also need to cover installing BIND and having our CentOS server function as a DNS server as well.


Saving your iptables configuration

You can use terminal line commands to make changes to iptables which will instantly effect the linux firewall. This is similar to the difference between a Cisco router’s startup-config file and its running-config in RAM. Changes to the running-config take effect immediately but are not permanent unless saved to the startup-config. This is a similar scenario with iptables. If you have made changes to iptables using iptables terminal commands, it will directly effect the Linux firewall, but the changes will not be permanent unless they are saved to the iptables configuration file.

There are different approaches to making changes to iptables: the approach we have been using so far is to make the changes to the iptables directly, using terminal commands; another approach is to save a test iptables configuration file and then load it, to test it out. Since we have been using the first approach we need to save our configuration to a file and then replace the default iptables configuration file. Before you replace the default file it is a good idea to create a backup. This command will take the current running iptables configuration which we have manipulated and altered, and save it to a .conf text file. You can save it to the root home folder.
iptables-save > iptables.conf

Now cat your saved iptables.conf file to look it over.
cat iptables.conf

The first command listed below makes a backup of the original default iptables configuration file, and the second command replaces the iptables configuration file with the current configuration, that was just saved as iptables.conf. Now, if the computer needs to be restarted the new configuration will be maintained.
cp /etc/sysconfig/iptables /etc/sysconfig/iptables.bak
mv iptables.conf /etc/sysconfig/iptables

{loadposition adposition6}You may decide that it is smarter and safer to make changes to the Linux iptables firewall indirectly, by editing a test iptables.conf file before testing it out on the actively running iptables firewall. In this way, you make all your changes to the iptables.conf file and then load it to memory, instead of using iptables terminal commands to experiment with your running configuration and possibly compromise your security or disrupt your network users. With the following commands you save the current iptables to an iptables.conf text file, then edit it with a text editor, then restore it to a running state in active memory. The iptables-restore command is a beneficial command because it loads the edited config file to the running iptables without disrupting the service and potentially dropping the running network host connections.
iptables-save > iptables.conf
nano iptables.conf (edit the iptables.conf configuration file)
iptables-restore < iptables.conf  (load the edited iptables.conf file to the running iptables) 
mv iptables.conf /etc/sysconfig/iptables
(replace the iptables configuration file with the edited version)

Note: For a demonstration of this tutorial, see the video below: Use CentOS Linux for Routing, Proxy, NAT, DHCP – Part 7
Saving manual network interface configurations

A regular desktop installation of CentOS Server comes with the Network Manager program. Network Manager is a graphical management tool used to manage and configure network interfaces in an easy way. It was personally recommended to me to not use the Network Manager, but instead manually configure my network interfaces, meaning Ethernet ports with IP addressing, subnet masks, default gateway address, and DNS servers. Like with iptables, you can manually configure your network interfaces using terminal line commands, but if you restart the server your configurations will not be saved. In order to configure your network interfaces in a manner that will persist, you need to configure the network interface configuration files or scripts.
ls /etc/sysconfig/network-scripts/

You should see configuration files named ifcfg-eth0, ifcfg-lo, or some other variant of ifcfg-<name> where <name> refers to the network interface name. To inspect the interface configuration you can output the text of the config file using cat
cat ls /etc/sysconfig/network-scripts/ifcfg-eth0

You will see some output like the following:
DEVICE=”eth0″
NM_CONTROLLED=”yes”
ONBOOT=”no”
TYPE=Ethernet
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no

This information tells us that Network Manager is managing the interface, the interface is not activated by default when the system boots up, and that the system is using DHCP to acquire an IP address. In order to manually configure the network interface you will need to change these settings and add some additional settings as well. Before you change the settings of the interfaces it is a good idea to create backup configuration files.
cp /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth0.bak
cp /etc/sysconfig/network-scripts/ifcfg-eth1 /etc/sysconfig/network-scripts/ifcfg-eth1.bak

Now you can configure your interfaces by altering the configuration files in a text editor like Vim or Nano
nano /etc/sysconfig/network-scripts/ifcfg-eth0

{loadposition adposition7}Now alter and add the necessary settings and IP addressing for your outside interface. In my eth0 example below, I am on the 192.168.2.0/24 network using a static IP address of 192.168.2.98, a twenty four bit subnet mask, and a gateway located at 192.168.2.1. Changes are displayed in red:
DEVICE=”eth0″
NM_CONTROLLED=“no”
ONBOOT=“yes”
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPADDR=192.168.2.98  <whatever IP address your outside interface needs to be set at>
NETMASK=255.255.255.0
GATEWAY=192.168.2.1

IPV4_FAILURE_FATAL=yes
IPV6INIT=no
<addtional entries not listed>

Save the interface configuration file and exit.

On the inside network interface (in my case eth1), I used the following settings:
DEVICE=”eth1″
HWADDR=”00:0C:29:12:AE:75″
NM_CONTROLLED=“no”
ONBOOT=“yes”
IPADDR=192.168.111.1  //the IP address for your inside interface. I chose “.1” since I am the gateway for the inside network
NETMASK=255.255.255.0

Save the interface configuration file and exit. Now you can issue the following commands to verify your interface configurations and your default route
ifconfig
route

Note: For a demonstration of this tutorial, see the video below: Use CentOS Linux for Routing, Proxy, NAT, DHCP – Part 8


Manual Configuration of Network Interfaces for Trunking and VLANs

At the time of writing this article, the network manager in CentOS does not allow you to configure a network interface with subinterfaces, VLANs, and trunking. In order to configure a single network interface to function as multiple subinterfaces which allow VLANS over a trunked link you need to manually configure your network interface configuration file.

In the following scenario suppose you have one network interface eth0 and you want your router to function as a transparent proxy and a router separating an interior network from an exterior network. In the image below you can see how this would logically be arranged in the logical topology, you have the 192.168.111.0/24 network on the interior LAN side, and the 192.168.11.0/24 network on the exterior facing WAN side, of course 192.168.11.0/24 is also in a private network address range, but it is on the side of the CentOS server that is closest to the internet. However, in reality you happen to be physically limited by having only one network interface (eth0) and will therefore need to convert the one interface into multiple subinterfaces with VLANs for a trunked link (Physical Topology).

 

Looking at the physical topology in the diagram above, the CentOS server has transformed a single network interface eth0 into eth0.11 and eth0.111 for VLAN11 and VLAN111 respectively. From there the server is connected to the switch which has been configured as a trunk to allow VLAN11 and VLAN111.

Here are the commands and configuration settings needed to configure the CentOS server to fit with the scenario depicted in the diagram above. The first step is to back up your default or current network interface configuration file. THen copy the configuration file and rename it for both of the subinterface VLANs:
cp /etc/sysconfig/network-scripts/eth0 /etc/sysconfig/network-scripts/eth0.bak
cp /etc/sysconfig/network-scripts/eth0 /etc/sysconfig/network-scripts/eth0.11
cp /etc/sysconfig/network-scripts/eth0 /etc/sysconfig/network-scripts/eth0.111

Now you will need to edit all three of your configuration files ifcfg-eth0, ifcfg-eth0.11 and ifcfg-eth0.111 in turn with a text editor like vi, vim, or nano. In configuring ifcfg-eth0 you will only need to make sure that the interface is activated on startup, and that the interface is not managed by the network manager:
vim /etc/sysconfig/network-scripts/ifcfg-eth0

  DEVICE=”eth0″
NM_CONTROLLED=“no”
ONBOOT=“yes”
TYPE=Ethernet

For the inside interface, the LAN side of the router (in the diagram VLAN111 eth0.111), you will mainly need to change the interface name, the ipaddress, netmask, add a VLAN setting. The default gateway and DNS settings will be configured on the outside facing interface.
vim /etc/sysconfig/network-scripts/ifcfg-eth0.111

DEVICE=”eth0.111″
HWADDR=”00:0C:29:12:AE:75″
NM_CONTROLLED=“no”
ONBOOT=“yes”
IPADDR=192.168.111.1  //the IP address for your inside interface. I chose “.1” since I am the gateway for the inside network
NETMASK=255.255.255.0
VLAN=”yes”

For the outside facing interface, the WAN side of the router (in the diagram VLAN11 eth0.11), you will mainly need to change the interface name, the ipaddress, netmask, add a VLAN setting and add the default gateway and DNS settings.
vim /etc/sysconfig/network-scripts/ifcfg-eth0.11

 DEVICE=”eth0.11″
NM_CONTROLLED=”no”
ONBOOT=”yes”
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPADDR=192.168.11.100  <whatever IP address your outside interface needs to be set at>
NETMASK=255.255.255.0
GATEWAY=192.168.11.1
DNS=127.0.0.1  <if your CentOS server will not be running DNS services, use your normal DNS server address instead>

Note: For a demonstration of this tutorial, see the video below: Use CentOS Linux for Routing, Proxy, NAT, DHCP – Part 9



Video Tutorials

 In part 7, I demonstrate how to save your iptables configuration and make it permanent

In part 8, I manually configure and save the Ethernet network interfaces, IP addresses and gateway address

In part 9, I manually configure a single network interface to be multiple subinterfaces in order to support VLANs and trunking 

Route, NAT, and Transparent Proxy, with CentOS Linux

Overview

It is useful to protect your network, by filtering web requests and other types of traffic, a proxy server is designed to do this. You can create a Linux proxy server using Squid and SquidGuard, and configure network settings on your user’s browsers to access the proxy server which will then apply rules that will filter the requests. However, that can be bypassed by the host computer.

A stronger alternative is to create a transparent proxy server and configure your router to forward all web related requests to the proxy server, that way your network hosts are forced to go through the transparent proxy. A potential problem with that scenario is that you need to have a fairly decent router, which can forward interior traffic on a specified port, to the proxy server on the local area network. A third alternative is to make your transparent proxy server also handle routing, NAT, DHCP, and DNS. It would seem like that would be a huge undertaking, but it can actually be done fairly quickly, using a computer and a distribution of Linux.

transparentproxy-routing-nat

Lab Outline

To create a Linux proxy server that also functions as a router and more, it is recommended to use a server distribution of Linux like CentOS Linux, which is the freely distributed server equivalent to RedHat Enterprise Linux. Along with Debian ,CentOS is one the most popular server distributions of Linux available. For this lab, we download and burn CentOS 6.2, i386 (32bit) or x86_64 (64bit), ISO DVDs Parts 1 and 2. Pick a CentOS 6.2 mirror with Direct DVD downloads and download the DVD iso files, to burn as installation DVDs.

Once you have the CentOS installation DVD iso files downloaded and burned to DVD you need to install the operating system to a computer. You can choose to install to an actual computer or to a virtual computer (virtual machine). In order to do this lab from home, and if you do not have a lot of extra computers lying around, I recommend doing the whole project virtually through the use of virtual machines. For a virtualization platform, I recommend downloading either VMware Player or Virtualbox.

If you decide to follow along with the video tutorials and set up a CentOS server virtual machine and convert it into a router, remember that the IP addressing scheme will need to be modified to work with your personal network. Just because my wireless network uses a 192.168.2.0 /24 local addressing scheme does not mean that yours does also, in fact your network most likely uses a different scheme like 192.168.1.0 /24.

Network diagram of a CentOS proxy server and router using virtual machines

transparentproxy-routing-nat_in-a-virtual-environment

In the lab at the college we installed CentOS to actual physical
computers. Normally, I recommend using computers with two network
interface cards (NICs) installed, but the computers we had available to
us had only one network interface card (NIC). This posed a challenge
since the traffic would need to flow through the computers, from one
network to another, implying the need for two network interface cards.
We fixed this problem by configuring VLANs, and assigning IP addresses
to subinterfaces on the computer’s single NIC. We then connected to a
switchport configured for trunking with the same VLANs as we created on the
computer.

Network diagram of a CentOS proxy server-router implementation using a lab computer with only one network interface card (NIC) and a switch configured with VLANs and trunks

proxy-routing-nat_in-physical-env-w-single-nic

Lab Steps Overview (virtualized machines or physical machines)

a. Understand the benefits of a transparent proxy and a Linux server that can also route, NAT, etc. Plan and diagram your lab. See video tutorial part 1 below.

b. Download CentOS installation ISO files, i386 or x86_64, DVDs. See Lab Outline above and video tutorial part 2 below.

c. To do this lab on a physical computers you will need one computer for the CentOS server, preferably with two NICs, but one is okay too, and another computer for a test LAN host.

To do this lab virtually using a virtual platform like VMware or Virtualbox, first create a new virtual machine for the the Linux operating system. You will probably need to choose RedHat Enterprise 32 bit or 64 bit and walk through all the settings. At the end, configure your virtual CD/DVD to boot to the CentOS installation ISO file that you downloaded. Make sure to add or enable to virtual network interfaces (NICs). Set the first NIC to Bridged Networking mode and the second NIC to NAT mode (then once it is saved change it to again to LAN segment mode), in Virtualbox save the second NIC to Interior Network mode. See video tutorial part 2 below.

d. Configure network addressing on your server’s two network interfaces as well as your test host’s network addressing. See video tutorial part 3 below.

e. Configure routing and NATing using iptables on your server. See video tutorial part 4 below.

f. Install and configure Squid as a transparent proxy server. Configure iptables to port-forward web traffic to the server on port 3128 (Squid). See video tutorial part 5 below.

g. Secure your server by configuring iptables to accept and reject specific types of traffic. Install and configure DHCP server on your CentOS server in order to hand out IP addresses on your local network. See video tutorial part 6 below.

Lab Command Steps

Here is a list of the commands used in this lab project:

1. If you are working in my lab or have a computer with only one network card (NIC) jump to step 2. If you are using a virtual or physical computer with two NICs (e.g. eth0 and eth1) you will  need to configure each NIC with the command below, then jump to step 6. You can dynamically receive an IP address through DHCP coming from your router/gateway on eth0, click on the network manager in the upper right of the task bar and click “auto ethernet” or just click on “eth0” to activate DHCP. Then you will need to manually configure the second NIC using an ifconfig command from the terminal (e.g. ifconfig eth1 192.168.111.1) you will need to have root access to do this. After you have activated DHCP on eth0 and manually configured eth1 use the ifconfig command to check your interface ip addresses to see that you have an address for eth0 and eth1. You are now ready to jump to step 6.
su –
ifconfig eth1 192.168.111.1
ifconfig

2. If you are working in my lab or you have a computer with only one network interface card (NIC), you will need to configure trunking and two VLANs (e.g. 110, 111) on the CentOS Linux server (for students with a physical computer with only one NIC)
vconfig add eth0 110 (in my college lab we use VLAN110 for the 192.168.11.x network)
vconfig add eth0 111 

vconfig rem eth0 111 (removes a VLAN trunk)

3. To configure sub-interfaces in
order to assign an IP address to each VLAN, in our classroom lab we use
VLAN110 for the 192.168.11.0 network.
ifconfig eth0.110 192.168.11.xxx
ifconfig eth0.111 192.168.111.xxx 

4. To configure a default gateway/route
route add default gw 192.168.11.1

5. To configure DNS servers
echo “nameserver 4.2.2.2” > /etc/resolv.conf
echo “nameserver 8.8.8.8” >> /etc/resolv.conf

6. To add kernel support for IP forwarding (routing) we set a flag with the following command
sysctl -w net.ipv4.ip_forward=1

7. To set up NATing we can configure the iptables NAT table for masquerading.
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE  (assuming eth0 is the outside interface)
or
iptables -t nat -A POSTROUTING -o eth0.110 -j MASQUERADE  (assuming the eth0.110 sub-interface is the outside interface)

8. To install the Squid proxy server
yum install squid
cd /etc/squid/squid.conf

9. Edit the the squid.conf file and change the following line to enable transparent proxy mode:
http_port 3128
to
http_port 3128 intercept

10. Issuing one of the following commands will restart the Squid service or reload the configuration file
service squid restart
service squid reload

11. Add an entry to iptables NAT table to port-forward inbound traffic on the inside interface (LAN side) to the Squid server on port 3128 (assuming eth1 is the inside interface with the IP address 192.168.11.1)

iptables -t nat -A PREROUTING -i eth1 -p tcp –dport 80 -j DNAT –to (cont.) 192.168.111.1:3128

12. To get routing and forwarding to work correctly you will also need to remove some statements from the iptables tables that reject traffic. The following lines will remove reject statements from the filter INPUT iptable chain and the FORWARD iptable chain.
iptables -t filter -D INPUT -j REJECT –reject-with icmp-host-prohibited
iptables -t filter -D FORWARD -j REJECT –reject-with icmp-host-prohibited

From first glance, it seems that there is a rule in the iptables INPUT chain (third line) that accepts any protocol from any source to any destination. The following statement would delete the third line of the filter table INPUT chain, the “accept any anywhere anywhere” line: <iptables -t filter -D INPUT 3> but do not use it, because as it turns out, that particular line has information that does not appear in the output of an <iptables -L> command. Try using the following command and then compare it to the results of the <iptables -L> command:
iptables-save

You can see that the third line (-A INPUT -i lo -j ACCEPT) is actually accepting all input on the loopback (lo) interface, and since it is only the loopback interface (i.e. 127.0.0.1) it is not a problem in the iptables, indiscriminately accepting all input on the outside interface. So to clarify, in the following three line commands (see below), the first command show the current iptables rules, the second command removes line 3 (-A INPUT -i lo -j ACCEPT) which is the line that ACCEPTs packets on the loopback interface, filter table, INPUT chain. Finally, the third line below replaces it. To learn how to do this I referred the man pages for iptables <man iptables>.
iptables-save
iptables -t filter -D INPUT 3
iptables -t filter -I INPUT 3 -i lo -j ACCEPT

13. Now you can look at your iptables, default filter table, and nat table, using the following commands
iptables -L -t filter
iptables -L -t nat

14. Now you can add (append) to the iptable filter table with the following commands, to accept input on port 3128 for Squid, and reject all other types of traffic
iptables -t filter -A INPUT -p tcp –dport 3128 -j ACCEPT
iptables -t filter -A INPUT -j REJECT –reject-with icmp-host-prohibited
iptables -L -t filter

iptables 

15. The following commands and instructions are used to install and configure a DHCP server in CentOS. You will need elevate to root access, run updates, then install:
su –
yum install updates
yum install dhcp

16. You should see that the DHCP server fails on starting up, this is because we have not configured it yet. To configure the DHCP server we need to edit the dhcpd.conf file located in /etc/dhcp.
cd /etc/dhcp
ls

17. You should see a file called dhcpd.conf . You will want to edit the file in a text editor like Vim or Nano.
vim /etc/dhcp/dhcpd.conf
or
nano /etc/dhcp/dhcpd.conf

Here is a screenshot of my dhcpd.conf file after I edited it in Vim. The cat command is used to output the file to the terminal for viewing. The lines that begin with # are comments and not active configurations. You could duplicate what you see in my configuration file below, replacing every line that has 192.168.11.x ip addressing with your own network number, like 192.168.1.x, etc.:
cat /etc/dhcp/dhcpd.conf

dhcp-centos

IPtables Overview

The iptables consists of four separate tables: filter, nat, mangle, and raw. Each table has a set of rules or chains.

filter – the filter table is the default table not specified by the -t argument. The filter table has three chains: INPUT for traffic destined for a local socket, FORWARD for packets being routed through the server, and OUTPUT for packets generated locally on the machine

nat – the nat table is consulted when new connections are generated. The nat table has three chains: PREROUTING, OUTPUT, and POSTROUTING

mangle – the mangle table is used for specialized packet alteration. The mangle table has the following chains: PREROUTING, POSTROUTING, INPUT, FORWARD, and OUTPUT.

raw – the raw table is used for configuring exemptions from connection tracking.

Video Tutorials

In part 1, I outline creating a Linux proxy server and router and I diagram the network

In part 2, I create the VMware virtual machine with two NICs and install CentOS

In part 3, I configure network addressing on the CentOS server and a Fedora client

In part 4, I review network addressing, how to do the lab if the server has only one network interface.
I also set up routing and NATing on the CentOS server

In part 5, I install Squid on the CentOS server and configure it to be a transparent proxy.
I configure the iptables NAT table, to portforward all local web traffic to the CentOS server on port 3128. 

In part 6, I continue to configure iptables and I install and configure DHCP server on the CentOS server. 

Install & Configure SquidGuard in Ubuntu

Install and Configure SquidGuard Overview

It is very useful to be able to block users on your network from accessing millions of websites with nefarious content. A great way to accomplish this is with a proxy server like Squid. Squid is a free and powerful proxy server that is capable of blocking users from accessing web content. A great way of enhancing Squid’s ability to block unwanted websites, domains and IP addresses is to install SquidGuard. SquidGuard is an add-on program for the Squid proxy server (see my previous article on Squid), that’s main purpose is to block unwanted web traffic. SquidGuard works with databases of blacklists to block, filter, and redirect requested URLs and domains. You manually download and add blacklist files to SquidGuard and compile them into the SquidGuard database, then Squid can redirect web requests, checking them against SquidGuard’s database of blacklisted websites, domains and IP addresses. It seems like this process would slow down a network, but SquidGuard is an extremely fast web content filter with the ability to check web requests against millions of blacklisted sites in a matter of seconds. There is great information about SquidGuard’s capabilities on the SquidGuard website, including links to download the program, well written installation and configuration instructions, and links to websites that maintain blacklists.

Steps to manually install SquidGuard in Ubuntu

You can download and install SquidGuard using a package manager program like apt-get or yum, or even a graphical software installer tool like the software center program. Instead, I chose to outline the steps involved in manually downloading and installing SquidGuard.

1. Download the current stable version of SquidGuard at http://www.squidguard.org and save it to your downloads folder.

2. Download the Berkeley DB from Oracle at http://oracle.com. Download version 4.8.30.NC.tar.gz with no encryption and save it to your Downloads folder

3. Open a terminal and navigate to the directory where you downloaded SquidGuard and the BerkeleyDB. You should see the tar.gz files
cd ~/Downloads
ls

4. Decompress the tar.gz files (substitute the file names for the versions you downloaded)
tar -xvzf squidGuard-1.5-beta.tar.gz
tar -xvzf db-4.8.30.NC.tar.gz
ls

You should see two folders one for squidGuard and one for BerkeleyDB (e.g. squidGuard-1.5, and db-4.8.30)

5. Install the Berkeley DB first, since SquidGuard requires it for installation. By default, the Berkeley DB will install itself to a directory in /usr/local/ in a folder named BerkeleyDB.4.8 you will need this information when preparing SquidGuard for installation.
cd db-4.8.30
cd build_unix
../dist/configure
make
sudo make install

6. Install SquidGuard by navigating to the extracted SquidGuard folder and then during the configure process you will pass the configure script the location of the Berkeley DB directory and correctly change the squiduser to ‘proxy’ for Ubuntu. The squiduser and group is typically “squid” in other Linux distributions like Fedora.
cd ~/Downloads/squidGuard-1.5
./configure –with-db=/usr/local/BerkeleyDB.4.8 –with-squiduser=proxy
make
sudo make install

You should get a message that the initial SquidGuard configuration is complete. Congratulation, SquidGuard is successfully installed! Make a note of the directory locations of the SquidGurad db, log, and conf files:
/usr/local/squidGuard/db
/usr/local/squidGuard/log
/usr/local/squidGuard/squidGuard.conf

Blacklists

7. Now that SquidGuard is installed you will want to download some blacklists. The SquidGuard website provides a few options. Click on Blacklists link and download a few blacklists. I recommend going here http://cri.univ-tlse1.fr/blacklists/index_en.php and downloading the blacklists.tar.gz file from the top of the Descriptions section

Now you can move the blacklists to the SquidGuard db directory and extract them so they are ready to use.

cd ~/Downloads
sudo cp blacklists.tar.gz /usr/local/squidGuard/db/blacklists.tar.gz
cd /usr/local/squidGuard/db
sudo tar -xvf blacklists.tar.gz

Configuring SquidGuard

8. Now you are ready to configure SquidGuard you will want to open the configuration file with a text editor.
cd /usr/local/squidGuard/
ls

You should see a squidGuard.conf file. Copy the conf file to a backup and open it with a text editor
sudo cp squidGuard.conf squidGuard.conf.bak
sudo su
gedit squidGuard.conf &

If your squidGuard.conf file is janked i.e. blank, then you can copy the configuration directly from the SquidGuard website: http://www.squidguard.org/Doc/configure.html

Looking at your squidGuard.conf file in the text editor make sure that the lines beginning with dbhome and logdir point to the correct directory. For my install the dbhome and logdir lines read:
dbhome /usr/local/squidGuard/db
logdir /usr/local/squidGuard/log

So I changed the dbhome line to:
dbhome /usr/local/squidGuard/db/blacklists

Try to running squidGuard in an output to stderr mode:
squidGuard -d

I had errors showing on line 23 so I commented out lines 22 to 25 with # signs:
#rew dmz{
#          s@://admin/…
#          s@://foo.bar….
#}

Now try running squidGuard:
squidGuard -d

If squidGuard ran with no errors it is time to compile your Blacklists from text to DB with a -C all command
squidGuard -d -C all

{loadposition adposition7}I had additional errors caused by the Destination Classes area in the squidGuard.conf file. The dest adult block of code had the following lines that needed to have the “dest/” edited out, because they are not the correct directory paths following from the “/usr/local/squidGuard/db/blacklists” directory:

dest adult{
domainlist          dest/adult/domains
urllist                   dest/adult/urls
expressionlist    dest/adult/expressions
redirect               http://admin.foo.bar.de…
}

to

dest adult{
domainlist          adult/domains
urllist                   adult/urls
expressionlist    adult/expressions
redirect               http://google.com
}

I also edited the ACL block of code at the end of the config file. I commented out areas that I was not going to use, and focused on the default acl block of code, which I changed to pass only the not(!) adult sites (pass     !adult all):

acl {
#    admin {
#        pass     any
#    }
#
#    foo-clients within workhours {
#        pass     good !in-addr !adult any
#    } else {
#        pass any
#    }
#
#    bar-clients {
#        pass    local none
#    }
#
default {
pass     !adult all
#rewrite dmz
redirect http://google.com
}
}

9. After editing your config file try to compile your Blacklists from text to DB with a “-C all” command
squidGuard -d -C all

If there are no errors make sure the blacklists have correct ownership and group for Squid. You can check ownership of files and folders using the ls-l command. For Ubuntu the correct owner and group for Squid is “proxy”, in other distributions it is “squid”.
chown -R proxy:proxy /usr/local/squidGuard/db/blacklists

10. To finish the installation, add the following line to the squid.conf file in /etc/squid/squid.conf. I added the following line around line 1083 although you could add it anywhere, notice that it is directing the squidGuard program to the configuration file. If your squidGuard installation and configuration file is located in a different directory then adjust the paths in the line accordingly:

url_rewrite_program  /usr/local/bin/squidGuard  -c  /usr/local/squidGuard/squidGuard.conf

11. Now restart Squid or reload the Squid configuration file which is much faster.
service squid reload
or
pkill -9 squid
service squid start

12. In order to test if squidGuard configuration is working correctly and that Squid is passing web requests and checking them against the SquidGuard database. The SquidGuard website recommends running a dry-run test using the following command. You can substitute one of the blacklisted URLs from your blacklists instead of the “http://www.example.com” URL in the example. Also, If you do not have a “test.cfg” file, just remove the part of the line from, “-c … to … test.cfg” (see example below):

echo “http://www.example.com 10.0.0.1/ – – GET” | squidGuard -c /tmp/test.cfg -d
to
echo “http://www.blacklisted.com – – – GET” | squidGuard -d

After running the command above, if you see the following 3 messages in the output then squidGuard is functioning correctly:
– the redirected URL website address from the squidGuard.conf file
– “squidGuard ready for requests”,
– “squidguard stopped”

Now you can try using your web browser to see if it will block blacklisted domains and websites!

Note: If you are in a situation where you do not want to risk requesting blacklisted sites in your browser and having them not be filtered, then you can add one of your own entries in a blacklist, recompile the squidGuard blacklist database, and test to see if your manually entered website is blocked by squidGuard.

Setup Remote Desktop Sharing in Ubuntu with VNC

Video Tutorial – VNC

In this tutorial, I set up remote desktop sharing in Ubuntu and then
connect to Ubuntu from a Windows 7 computer using the Real VNC Client

Video Tutorial – TeamViewer

In this tutorial, I setup a remote desktop connection from Windows 7 to Ubuntu 11.10 using TeamViewer