ZBlogH

Quick Links

• Streams Join Two Points

• The Linux Standard Streams: stdin, stdout, stderr

• Streams Are Handled Like Files

• Reacting to Pipes and Redirects

• Redirecting stdout and stderr

• Redirecting Both stdout and stderr

• Redirecting stdout and stderr to the Same File

• Detecting Redirection Within a Script

• Streams Of Consciousness

Key Takeaways

• Linux commands create three data streams (stdin, stdout, and stderr) that can be used to transfer data about a command
• stdin is the input stream, stdout is the output stream, and stderr is the error stream in Linux.
• Redirection allows you to redirect the output or errors to different destinations, such as files or pipes.

stdin, stdout, and stderr are three data streams created when you launch a Linux command. You can use them to tell if your scripts are being piped or redirected. We show you how.

Streams Join Two Points

As soon as you start to learn about Linux and Unix-like operating systems, you'll come across the terms stdin, stdout, and stederr . These are three standard streams that are established when a Linux command is executed. In computing, a stream is something that can transfer data. In the case of these streams, that data is text.

Data streams, like water streams, have two ends. They have a source and an outflow. Whichever Linux command you're using provides one end of each stream. The other end is determined by the shell that launched the command. That end will be connected to the terminal window, connected to a pipe, or redirected to a file or other command, according to the command line that launched the command.

The Linux Standard Streams: stdin, stdout, stderr

In Linux, stdin is the standard input stream. This accepts text as its input. Text output from the command to the shell is delivered via the stdout (standard out) stream. Error messages from the command are sent through the stderr (standard error) stream.

So you can see that there are two output streams, stdout and stderr, and one input stream, stdin. Because error messages and normal output each have their own conduit to carry them to the terminal window, they can be handled independently of one another.

Streams Are Handled Like Files

Streams in Linux---like almost everything else---are treated as though they were files. You can read text from a file, and you can write text into a file. Both of these actions involve a stream of data. So the concept of handling a stream of data as a file isn't that much of a stretch.

Each file associated with a process is allocated a unique number to identify it. This is known as the file descriptor. Whenever an action is required to be performed on a file, the file descriptor is used to identify the file.

These values are always used for stdin, stdout, and stderr:

• 0: stdin
• 1: stdout
• 2: stderr

Reacting to Pipes and Redirects

To ease someone's introduction to a subject, a common technique is to teach a simplified version of the topic. For example, with grammar, we are told that the rule is "I before E, except after C." But actually, there are more exceptions to this rule than there are cases that obey it.

In a similar vein, when talking about stdin, stdout, and stderr it is convenient to trot out the accepted axiom that a process neither knows nor cares where its three standard streams are terminated. Should a process care whether its output is going to the terminal or being redirected into a file? Can it even tell if its input is coming from the keyboard or is being piped into it from another process?

Actually, a process does know---or at least it can find out, should it choose to check---and it can change its behavior accordingly if the software author decided to add that functionality.

We can see this change in behavior very easily. Try these two commands:

ls

ls | cat

The ls command behaves differently if its output (stdout) is being piped into another command. It is ls that switches to a single column output, it isn't a conversion performed by cat. And ls does the same thing if its output is being redirected:

ls > capture.txt

cat capture.txt

Redirecting stdout and stderr

There's an advantage to having error messages delivered by a dedicated stream. It means we can redirect a command's output (stdout) to a file and still see any error messages (stderr) in the terminal window. You can react to the errors if you need to, as they occur. It also stops the error messages from contaminating the file that stdout has been redirected into.

Type the following text into an editor and save it to a file called error.sh.

#!/bin/bash
echo "About to try to access a file that doesn't exist"
cat bad-filename.txt 

Make the script executable with this command:

chmod +x error.sh

The first line of the script echoes text to the terminal window, via the stdout stream. The second line tries to access a file that doesn't exist. This will generate an error message that is delivered via stderr.

Run the script with this command:

./error.sh

We can see that both streams of output, stdout and stderr, have been displayed in the terminal windows.

Let's try to redirect the output to a file:

./error.sh > capture.txt

The error message that is delivered via stderr is still sent to the terminal window. We can check the contents of the file to see whether the stdout output went to the file.

cat capture.txt

The output from stdin was redirected to the file as expected.

The > redirection symbol works with stdout by default. You can use one of the numeric file descriptors to indicate which standard output stream you wish to redirect.

To explicitly redirect stdout, use this redirection instruction:

1>

To explicitly redirect stderr, use this redirection instruction:

2>

Let's try to our test again, and this time we'll use 2>:

./error.sh 2> capture.txt

The error message is redirected and the stdoutecho message is sent to the terminal window:

Let's see what is in the capture.txt file.

cat capture.txt

The stderr message is in capture.txt as expected.

Redirecting Both stdout and stderr

Surely, if we can redirect either stdout or stderr to a file independently of one another, we ought to be able to redirect them both at the same time, to two different files?

Yes, we can. This command will direct stdout to a file called capture.txt and stderr to a file called error.txt.

./error.sh 1> capture.txt 2> error.txt

Because both streams of output--standard output and standard error---are redirected to files, there is no visible output in the terminal window. We are returned to the command line prompt as though nothing has occurred.

Let's check the contents of each file:

cat capture.txt

cat error.txt

Redirecting stdout and stderr to the Same File

That's neat, we've got each of the standard output streams going to its own dedicated file. The only other combination we can do is to send both stdout and stderr to the same file.

We can achieve this with the following command:

./error.sh > capture.txt 2>&1

Let's break that down.

• ./error.sh: Launches the error.sh script file.
• > capture.txt: Redirects the stdout stream to the capture.txt file. > is shorthand for 1>.
• 2>&1: This uses the &> redirect instruction. This instruction allows you to tell the shell to make one stream got to the same destination as another stream. In this case, we're saying "redirect stream 2, stderr, to the same destination that stream 1, stdout, is being redirected to."

There is no visible output. That's encouraging.

Let's check the capture.txt file and see what's in it.

cat capture.txt

Both the stdout and stderr streams have been redirected to a single destination file.

To have the output of a stream redirected and silently thrown away, direct the output to /dev/null.

Detecting Redirection Within a Script

We discussed how a command can detect if any of the streams are being redirected, and can choose to alter its behavior accordingly. Can we accomplish this in our own scripts? Yes, we can. And it is a very easy technique to understand and employ.

Type the following text into an editor and save it as input.sh.

#!/bin/bash
if [ -t 0 ]; then
echo stdin coming from keyboard
else echo stdin coming from a pipe or a file
fi 

Use the following command to make it executable:

chmod +x input.sh

The clever part is the test within the square brackets. The -t (terminal) option returns true (0) if the file associated with the file descriptor terminates in the terminal window. We've used the file descriptor 0 as the argument to the test, which represents stdin.

If stdin is connected to a terminal window the test will prove true. If stdin is connected to a file or a pipe, the test will fail.

We can use any convenient text file to generate input to the script. Here we're using one called dummy.txt.

./input.sh < dummy.txt

The output shows that the script recognizes that the input isn't coming from a keyboard, it is coming from a file. If you chose to, you could vary your script's behavior accordingly.

That was with a file redirection, let's try it with a pipe.

cat dummy.txt | ./input.sh

The script recognizes that its input is being piped into it. Or more precisely, it recognizes once more that the stdin stream is not connected to a terminal window.

Let's run the script with neither pipes nor redirects.

./input.sh

The stdin stream is connected to the terminal window, and the script reports this accordingly.

To check the same thing with the output stream, we need a new script. Type the following into an editor and save it as output.sh.

#!/bin/bash
if [ -t 1 ]; then
echo stdout is going to the terminal window
else
echo stdout is being redirected or piped
fi 

Use the following command to make it executable:

chmod +x input.sh

The only significant change to this script is in the test in the square brackets. We're using the digit 1 to represent the file descriptor for stdout.

Let's try it out. We'll pipe the output through cat.

./output | cat

The script recognizes that its output is no going directly to a terminal window.

We can also test the script by redirecting the output to a file.

./output.sh > capture.txt

There is no output to the terminal window, we are silently returned to the command prompt. As we'd expect.

We can look inside the capture.txt file to see what was captured. Use the following command to do so.

cat capture.sh

Again, the simple test in our script detects that the stdout stream is not being sent directly to a terminal window.

If we run the script without any pipes or redirections, it should detect that stdout is being delivered directly to the terminal window.

./output.sh

And that's exactly what we see.

Streams Of Consciousness

Knowing how to tell if your scripts are connected to the terminal window, or a pipe, or are being redirected, allows you to adjust their behavior accordingly.

Logging and diagnostic output can be more or less detailed, depending on whether it is going to the screen or to a file. Error messages can be logged to a different file than the normal program output.

As is usually the case, more knowledge brings more options.

ncG1vNJzZmivp6x7qbvWraagnZWge6S7zGhrbG1pZYBww8eaq2aZopp6tMDDoqVmq6SZvLbAjJqlnWWjqbGmvtFmpqdlnJ67tsSO