Redirectors and Pipes

Overview

If you are going to use bash shell scripting to create programs that will manipulate computer data, then it is useful to control what data is input to a command, control where the data is output, and also control where error messages are output.

File Descriptors

In the bash shell, command input and command output can be manipulated. For every command that can be used in bash there are three file descriptors: standard input (stdin), standard output (stdout), and standard error (stderr).

• Standard input – is the information that is passed to a command. The standard input information can be input by the user through the blinking prompt, or it can be passed to a command from a file or program. Standard input is also represented by the number 0 (see below)

Standard output – is the data output after a command has executed. Standard output is also represented by the number 1 (see below)

Standard error – are any error messages that may have been generated by a command. Standard error is also represented by the number 2 (see below)

Redirectors

In bash, standard input, standard output and standard error messages are manipulated by redirect command characters. Redirects control how data is input and output from files to commands and vice versa:

  • The output redirector (> or 1>) redirects standard output to a file instead of the screen. Example:
      $ ls -l /var > mydirectory.txt
    $ ls -l /var 1> mydirectory.txt
  • The input redirector (< or 0<) redirects standard input from a file. Example:
      $ ls -l < /var
    $ ls -l 0< /var
    $ cat < mydirectory.txt
      $ cat 0< mydirectory.txt
  • The append redirector (>> or 1>>) appends standard output to the end of a file instead of rewriting it. Example:
      $ ls -l Documents > myfiles.txt
    $ ls -l Downloads >> myfiles.txt

    $ ls -l Pictures 1>> myfiles.txt
  • The output redirector with a 2 (2>) redirects standard error to a file. Example:
      $ ls -l Pixxtures 2> errors.txt
  • The output redirector with a 1 and a 2 (1> 2>) can be use to redirect standard output and standard error to two different files. The number one in the first redirector is not necessary since a greater-than sign by itself implies the number one, standard output. Example:
      $ ls -l Documents Pixxtures > good.txt 2> errors.txt

Pipes

The pipe is a special command character (|) which can take the standard output (stdout) from one command and make it the standard input (stdin) for another command. In the example below, the command ls -l has been issued to list the contents of the /etc directory, but before the output can be sent to the screen, it is piped (|) and turned into standard input for the more command to process. The more command takes the input and executes its program, which processes the information and outputs it to the display it one screen at a time.

  $ ls -l /etc | more

In this next example, the pipe is used to take the data output from the ls -l /etc command and send it as input to the grep command to search for a specific lines of text that have the text “firefox” in them.

  $  ls -l /etc | grep “firefox”

Pipes are very useful because they allow you to send information from one command to another in sequence. Pipes and redirectors can be used together very powerfully to control how data is input and output.

Video Tutorials

In this tutorial, I demonstrate using redirector characters to control standard input, output, and error

In this video, I demonstrate how to use of the pipe special character and the
grep command, to filter output to find specific lines of content

Shell Scripting with Bash

Overview of Bash Shell Scripting

The default Linux command line interface or terminal is the Bourne Again shell or bash shell. A shell is a user interface to a computer system that relies on keyboard input, as opposed to a graphical user interface which relies on keyboard input, mouse input, and presents the user with graphical icons and windows to click on with a mouse.

Since bash is a command interpreter, it has powerful programming capabilities like a full fledged programming language. In addition to entering commands in the terminal one at a time, you can write commands into a text file, save it, and execute it like a program. An executable file with shell commands is called a shell script or shell program.

To make a shell script executable, you have to give the text file, execute permissions. You do this with the chmod command:

  $ chmod + x <filename>

If you try to run a shell script that does not have execute permissions you will get a Permission denied message. You can start by learning how to create a simple “Hello World!” shell script.

A “hello world!” shell script

1. Start by opening a Linux terminal.

2. In the terminal, create the shell script file using a text editor like Nano. If you like, you can name the file with a .sh file extension for visual reference that it is a shell script. The file extension is optional, since Linux does not rely on file extensions like Windows to determine the file type. The command structure: $ nano <filename>

  $ nano hello.sh

3.  With nano open you can type your first shell script. The first line in the script starts with a hash-bang (#!) followed by the path to the bash program. This lets the terminal know that it is bash specifically, that you want to use to execute this script. This is important since there are other terminal shells, like the korn shell, c-shell, the tcshell, etc.. Type the following text into nano, then do control+x on the keyboard, type y for yes, and press enter, and then press enter again to accept the file name.

  #!/bin/bash
echo “Hello world!”

4. Now that you have the file saved, do a cat command to verify that the text saved to the file. If it was saved, you will see the text of hello world returned. The command structure: $ cat <filename>

$ cat hello.sh
Hello World!

5. Now you need give the file execute permissions. The command structure: $ chmod +x <filename>

  $ chmod +x hello.sh 

6. Now do a ls -l command to see if execute permissions were applied. The command structure: $ ls -l <filename>

  $ ls -l hello.sh

The output should look like this. If the line leads with a “d” it means it is a directory and if it is a “-” it means it is a file. Notice that after the file character “-“, the permissions on the hello.sh line read rwxr-xr-x, indicating that the owner is read, write, execute (rwx), followed by the group permission of read and execute (r-x), and the public, or everyone, which is also read and execute (r-x). The execute bit was successfully applied to all three groups: owner, group, and everyone.

  drwxr-xr-x 2 dan dan 4096 Jan 12 16:17 Desktop
drwxr-xr-x 4 dan dan 4096 Jan 17 19:53 Documents
drwxr-xr-x 3 dan dan 4096 Jan 16 07:47 Downloads
-rwxr-xr-x 1 dan dan   43 Jan 24 19:25 hello.sh

7. Now that your shell script file has execute permissions you can run it as a program. Since the file is not saved in a directory that is included in the $PATH variable you cannot run it like a program. You can verify this by simple trying to run your shell script by typing $ hello.sh you should see the command not found returned. However, you can run your program by giving the absolute path to the file or referencing the current directory with a “./”.

  $ /home/user/hello.sh
Hello world!

or simply,
  $ ./hello.sh
Hello world!

8. You have the basics on creating and running a shell script in the bash shell. Now, expand your knowledge by  learning additional programmatic capabilities that can be used to write more advanced scripts that will automate basic to advanced system tasks.

A shell script to backup your home directory

1. Open a Linux terminal.

2. Create the file in nano

  $ nano backup.sh

3.  Start typing your shell script. This time you can use the echo command to explicitly output what it is the script is attempting to do. You will use the tar command to create a compressed file of your home folder and save it to the var directory. The command structure is: $ tar -czf <destination-directory> <file-to-backup>. After the tar command, you can use the echo command to notify the user that the backup process is complete. Control+x and save the file. You will need to change /home/dan to the path to your home directory.

  #!/bin/bash
echo “Backing up home directory”
tar -czf /var/homebackup.tgz /home/dan
echo “Done!”

4. Do a cat command to verify that the file was saved correctly. The command structure: $ cat <filename>

$ cat backup.sh

5. Give the file execute permissions. The command structure: $ chmod +x <filename>

  $ chmod +x backup.sh 

6. Now do a ls -l command to see if execute permissions were applied. The command structure: $ ls -l <filename>

  $ ls -l backup.sh

7. Now that your shell script file has execute permissions, go ahead and run it. You will notice that the script fails because you do not have super user permission to save to the /var directory. Run the script again with a sudo command in front, enter your password, and you will see that the shell script file executes successfully.

  $ ./backup.sh
  Backing up home directory
tar: Removing leading `/’ from member names
tar (child): /var/homebackup.tgz: Cannot open: Permission denied
tar (child): Error is not recoverable: exiting now
Done!

  $ sudo ./backup.sh
 
Backing up home directory
  tar: Removing leading `/’ from member names
  Done!

Video Tutorials

In this video, I write a basic shell script, give it execute permissions, and run it as a program


In part 2, I write a basic shell script to backup the user’s home directory 

Intro to the Linux Shell

Shell Overview

What is the shell? Shell is actually a generic term for any interface to your computer. When talking about linux (or any other unix-like system), usually the shell refers to the command line interface (CLI). The command line interface most people are familliar with is DOS. There are actually several CLI shells that are common, however the most common default shell for the majority of the modern unix-like systems is BASH the Bourne Again Shell. The shell is sometimes referred to by several names, terminal, CLI, console, etc..

BASH Commands

Some of the most commonly used BASH commands are: cd, cat, man, ls, pwd, mkdir, touch, file, grep, help, less

Commands that give Help – One of the first things to learn how to do on any unfamiliar system is to get it to tell you how to use it. Within Linux there are a plethora of commands that provide great information on how to use the system.

help – the help command is useful in that it shows you information on what your shell can do for you. Most of this information is very specific to the shell you are using, but it will also list all of the commands built-in to your shell. It will display more information than you will be able to use right now, but it is good to know that it is there, it will help you later. Its also important to note, that if you see a command listed after help, such as cd, you can use that as an argument to help, by typing help cd.

man – the next useful command to know is man. Man is short for manual. When a program is installed, most of the time it also comes with a man page. The man pages are specific to each program. For example to access the man page for the ls command you would type man ls. You’ll notice that man pages are layed out really nice, and you can scroll up and down by using the arrow keys.

After that there is one more useful trick for getting help about commands. Almost every command in linux provides basic usage information by passing it –help. For example if you type grep –help, it will tell you how to use grep. Its usually more technical and specific than man pages so its a good resource to have in addition to man pages. In the process of learning linux there is one help source that is better than anything else: Google. If you dont know how to use something, type it into Google, it has been a personal lifesaver many times.

Commands that Navigate Files & Directories – Now that we know how to get some basic help about our system, lets see if we can start navigating.

ls – the ls command stands for list. If you type ls into your terminal, you will get a list of the files of the directory you are currently in. By default, when you open up a shell it’s current path with be your home directory. This is a place for you to store all of your files where nobody else but the root user can access them on your computer. You may notice that there are a lot of files in this directory, they are not actually files, The blue color indicates that they are directories. Directories are also sometimes referred to as folders, these terms are interchangeable.

So looking at the list of files and directories from the output of the ls command, you may want to enter a directory like the Downloads directory. In order to do this you need the change directory or CD command.

cd – the change directory command allows you to navigate into different directories. From the default home directory if you type: cd Downloads, you will enter the Downloads directory. You may notice that not much visibly changes, other than the ~, signifying the Home directory is replaced with Downloads. This means that the current working directory has changed to Downloads. This shows you the relative current directory, but how do you find out where you are in relation to anything else?

pwd – the pwd command will print the current working directory. This tells you where you are in relation to the root of the filesystem / . The single forward slash represents the root of the filesystem. The command pictured above reveals that the current directory Downloads is three directories from root. The current directory is underneath Downloads, which is underneath dan, which is underneath home, which is on the root of the filesystem /. Now that you know where you’re at, how do you get back to the home directory?

{loadposition adposition6}There are many ways to navigate a filesystem using cd. Understanding relative vs absolute paths is important. There is no right or wrong way to get to different directories. There are a few important things to note:

./  = indicates the current directory

../ = indicates one directory higher up in the filesystem

/ = indicates the root of the filesystem

~ = current user’s home directory

cd /home/dan/Downloads is an example of an absolute path – cd’ing to that will always yield the same results

cd ./Downloads or just cd Downloads, is an example of a relative path, cd’ing to that will yield different results depending on the current working directory.

Within the current example, if i wanted to get back to my home folder named “dan”, there are four basic ways of doing it.

cd ../ = going up one directory

cd /home/dan = passing the absolute path to my home directory “dan”

cd ~ = passing tilde which stands for your home directory

cd = simply issuing the cd command takes you back to your home directory

They all do the same thing, they take me back to my home directory.

Commands that find files – how do you find or search for files and directories in Linux? There are two very common commands to find files, and they work off of two different principles.

find – find is a very power program it operates by iterating through every file/folder on the entire filesystem. It has many options, and I recommend referencing the man page to master it.

find / -name bash = find a file named bash located somehwere under /

locate – locate works much more like a search engine, in that it requires indexing to occur before it returns any results. As a result, locate can find files/directories much faster, however indexing can take some time, although indexing only has to be performed after the file system changes.

updatedb – indexes the file system, updating the locate database

locate bash = find bash somewhere underneath root (/)

Video Tutorials

Linux terminal commands for Beginners –  In this series of tutorials, I cover basic, Bash shell commands for navigating, searching, getting help, creating files and directories, editing text and more. Commands that are covered are: pwd, ls, cd, mkdir, touch, rm, rmdir, man, less, more, mv, cp, cat, updatedb, sudo, locate, file.

Part 1 – Navigating and viewing directories, absolute and relative paths (pwd, cd, ls)

Part 2 – Command arguments, long format, hidden files (ls -l, ls -a)

Part 3 – Piping, viewing output, and getting help (ls, | , more, less, man, –help)

Part 4 – Making and removing files and directories (mkdir, touch, rm, rmdir)

Part 5 – Moving, copying, renaming, editing and viewing files (mv, cp, nano, cat)

Part 6 – Searching for files, elevating permissions (locate, updatedb, sudo, file)