Pipes and Redirection in Linux

1. Introduction

Most shells offer the ability to alter the way that application input and output flows. This can direct output away from the terminal and into files or other applications, or otherwise reading input from files instead of the terminal.

2. Standard Input and Output

Before we understand how redirection works in shells, we first need to understand the standard input and output process.

All applications have three unique streams that connect them to the outside world. These are referred to as Standard Input, or stdin; Standard Output, or stdout; and Standard Error, or stderr.

Standard input is the default mechanism for getting input into an interactive program. This is typically a direct link to the keyboard when running directly in a terminal, and isn’t connected to anything otherwise.

Standard output is the default mechanism for writing output from a program. This is typically a link to the output terminal but is often buffered for performance reasons. This can be especially important when the program is running over a slow network link.

The standard error is an alternative mechanism for writing output from a program. This is typically a different link to the same output terminal but is unbuffered so that we can write errors immediately while the normal output can be written one line at a time.

3. Redirecting Into Files

One common need when we run applications is to direct the output into a file instead of the terminal. Specifically, what we’re doing here is replacing the standard output stream with one that goes where we want: In this case, a file.

This is typically done with the > operator between the application to run and the file to write the output into. For example, we can send the output of the ls command into a file called files as follows:

$ ls > files

We can do this after any command, including any needed arguments:

$ ls -1 *.txt > text-files

3.1. Appending To Files

When we use >, then we are writing the output to the file, replacing all of the contents. If needed, we can also append to the file using >> instead:

$ ls -1 *.txt > files
$ ls -1 *.text >> files
$ ls -1 *.log >> files

We can use this to build up files using all manner of output if we need to, including just using echo to exact output lines:

$ echo Text Files > files
$ ls -1 *.txt >> files
$ echo Log Files >> files
$ ls -1 *.log >> files

3.2. Redirecting Standard Error

On occasion, we need to redirect standard error instead of standard output. This works in the same way, but we need to specify the exact stream.

All three of the standard streams have ID values, as defined in the POSIX specification and used in the C language. Standard input is 0, standard output is 1, and the standard error is 2.

When we use the redirect operators, by default, this applies to standard output. We can explicitly specify the stream to redirect, though, by prefixing it with the stream ID.

For example, to redirect standard error, from the cat command we would use 2>:

$ cat does-not-exist 2> log

We can be consistent and use 1> to redirect standard output if we wish, though this is identical to what we saw earlier.

We can also use &> to redirect both standard output and standard error at the same time:

$ ls -1 &> log

This sends all output from the command, regardless of which stream it is on, into the same file. This only works in certain shells, though – for example, we can use this in bash or zsh, but not in sh.

3.3. Redirecting into Streams

Sometimes we want to redirect into a stream instead of a file. We can achieve this by using the stream ID, prefixed by &, in place of the filename. For example, we can redirect into standard error by using *>&2*:

$ echo Standard Output >&1
$ echo Standard Error >&2

We can combine this with the above to combine streams, by redirecting from one stream into another. A common construct is to combine standard error into standard output, so that both can be used together. We achieve this using 2>&1 – literally redirecting stream 2 into stream 1:

# ls -1 2>&1

We can sometimes use this to create new streams, simply by using new IDs. This stream must already have been used elsewhere first though, otherwise, it is an error. Most often this is used as a stream source first.

For example, we can swap standard output and standard error by going via a third stream, using 3>&2 2>&1 1>&3:

$ ls -1 3>&2 2>&1 1>&3

This construct doesn’t work correctly in all shells. In bash, the end result is that standard output and standard error are directly swapped. In zsh the result is that both standard output and standard error have ended up on standard output instead.

3.4. Redirecting Multiple Streams

We can easily combine the above to redirect standard output and standard error at the same time. This mostly works exactly as we would expect – we simply combine the two different redirects on the same command:

$ ls -1 > stdout.log 2> stderr.log

This won’t work as desired if we try to redirect both streams into the same file. What happens here is that both streams are redirected individually, and whichever comes second wins, rather than combining both into the same file. 

If we want to redirect both into the same file, then we can use &> as we saw above, or else we can use stream combination operators. If we wish to use the stream combination operators, then we must do this after we have redirected into the file, or else only the standard output gets redirected:

$ ls -1 > log 2>&1

4. Reading From Files

Sometimes we also want to achieve the opposite – redirecting a file into an application. We do this using the < operator, and the contents of the file will replace the standard input for the application:

$ wc < /usr/share/dict/words
  235886  235886 2493109

When we do this, the only input that the application can receive comes from this source, and it will all happen immediately. It’s effectively the same as when the user types out the entire contents of the file at the very start of the application.

However, the end of the file is signaled to the application as well, which many applications can use to stop processing.

5. Piping Between Applications

The final action that we can perform is to direct the output of one application into another one. This is commonly referred to as piping, and uses the | operator instead:

$ ls | wc
      11      11     138

This directly connects the standard output of our first application into the standard input of the second one and then lets the data directly flow between them.

5.1. Handling Standard Error

The standard error isn’t connected by default, so we’ll still get anything written to that stream appearing in the console. This is by design since the standard error is designed for error reporting and not for normal application output.

If we want to redirect standard error as well then we can use the technique from above to first redirect it into standard output and then pipe into the next application:

$ ls i-dont-exist | wc
ls: i-dont-exist: No such file or directory
       0       0       0

$ ls i-dont-exist 2>&1 | wc
       1       7      44

If we want only to pipe standard error then we need to swap the standard output and standard error streams around, as we saw earlier.

$ ls 3>&2 2>&1 1>&3 | wc -l
some-file
       0

$ ls i-dont-exist 3>&2 2>&1 1>&3 | wc -l
       1

5.2. Combining Pipes

When piping between applications, we can also build arbitrary chains where we are piping between many applications to achieve a result:

$ docker images | cut -d' ' -f1 | tail -n +2 | sort -u | wc -l
      17

This command looks scary, but we can break this down to the individual parts:

  • docker images – Get a list of all Docker images

  • cut -d’ ‘ -f1 – Cut this output to only return the first column, where columns are space-separated

  • tail -n +2 – Limit this to start from line 2

  • sort -u – Sort this list, only returning unique entries

  • wc -l – Count the number of lines

So we have a command here to get the number of unique docker images, ignoring the version of the image.

Many console applications are designed for exactly this use, which is why they can often consume input from standard input and write to standard output.

Certain applications also have special modes that allow for this – git, for example, has what is termed porcelain and plumbing commands, where the plumbing commands are specially designed to be combined in this manner while the porcelain commands are designed for human consumption.

6. Summary

Here, we ‘ve seen several techniques for redirecting the input and output of applications either between files or other applications. These are compelling techniques that can give rise to complicated results from simple inputs.

Why not see what else we can do with these?