Chapter 19. I/O Redirection

There are always three default files [1] open, stdin (the keyboard), stdout (the screen), and stderr (error messages output to the screen). These, and any other open files, can be redirected. Redirection simply means capturing output from a file, command, program, script, or even code block within a script (see Example 3-1 and Example 3-2) and sending it as input to another file, command, program, or script.

Each open file gets assigned a file descriptor. [2] The file descriptors for stdin, stdout, and stderr are 0, 1, and 2, respectively. For opening additional files, there remain descriptors 3 to 9. It is sometimes useful to assign one of these additional file descriptors to stdin, stdout, or stderr as a temporary duplicate link. [3] This simplifies restoration to normal after complex redirection and reshuffling (see Example 19-1).

   1    COMMAND_OUTPUT >
   2       # Redirect stdout to a file.
   3       # Creates the file if not present, otherwise overwrites it.
   4 
   5       ls -lR > dir-tree.list
   6       # Creates a file containing a listing of the directory tree.
   7 
   8    : > filename
   9       # The > truncates file "filename" to zero length.
  10       # If file not present, creates zero-length file (same effect as 'touch').
  11       # The : serves as a dummy placeholder, producing no output.
  12 
  13    > filename    
  14       # The > truncates file "filename" to zero length.
  15       # If file not present, creates zero-length file (same effect as 'touch').
  16       # (Same result as ": >", above, but this does not work with some shells.)
  17 
  18    COMMAND_OUTPUT >>
  19       # Redirect stdout to a file.
  20       # Creates the file if not present, otherwise appends to it.
  21 
  22 
  23       # Single-line redirection commands (affect only the line they are on):
  24       # --------------------------------------------------------------------
  25 
  26    1>filename
  27       # Redirect stdout to file "filename."
  28    1>>filename
  29       # Redirect and append stdout to file "filename."
  30    2>filename
  31       # Redirect stderr to file "filename."
  32    2>>filename
  33       # Redirect and append stderr to file "filename."
  34    &>filename
  35       # Redirect both stdout and stderr to file "filename."
  36       #
  37       #  Note that   &>>filename
  38       #+ -- attempting to redirect and *append*
  39       #+ stdout and stderr to file "filename" --
  40       #+ fails with the error message,
  41       #+ syntax error near unexpected token `>'.
  42 
  43    M>N
  44      # "M" is a file descriptor, which defaults to 1, if not explicitly set.
  45      # "N" is a filename.
  46      # File descriptor "M" is redirect to file "N."
  47    M>&N
  48      # "M" is a file descriptor, which defaults to 1, if not set.
  49      # "N" is another file descriptor.
  50 
  51       #==============================================================================
  52 
  53       # Redirecting stdout, one line at a time.
  54       LOGFILE=script.log
  55 
  56       echo "This statement is sent to the log file, \"$LOGFILE\"." 1>$LOGFILE
  57       echo "This statement is appended to \"$LOGFILE\"." 1>>$LOGFILE
  58       echo "This statement is also appended to \"$LOGFILE\"." 1>>$LOGFILE
  59       echo "This statement is echoed to stdout, and will not appear in \"$LOGFILE\"."
  60       # These redirection commands automatically "reset" after each line.
  61 
  62 
  63 
  64       # Redirecting stderr, one line at a time.
  65       ERRORFILE=script.errors
  66 
  67       bad_command1 2>$ERRORFILE       #  Error message sent to $ERRORFILE.
  68       bad_command2 2>>$ERRORFILE      #  Error message appended to $ERRORFILE.
  69       bad_command3                    #  Error message echoed to stderr,
  70                                       #+ and does not appear in $ERRORFILE.
  71       # These redirection commands also automatically "reset" after each line.
  72       #=======================================================================

   1    2>&1
   2       # Redirects stderr to stdout.
   3       # Error messages get sent to same place as standard output.
   4 
   5    i>&j
   6       # Redirects file descriptor i to j.
   7       # All output of file pointed to by i gets sent to file pointed to by j.
   8 
   9    >&j
  10       # Redirects, by default, file descriptor 1 (stdout) to j.
  11       # All stdout gets sent to file pointed to by j.

   1    0< FILENAME
   2     < FILENAME
   3       # Accept input from a file.
   4       # Companion command to ">", and often used in combination with it.
   5       #
   6       # grep search-word <filename
   7 
   8 
   9    [j]<>filename
  10       #  Open file "filename" for reading and writing,
  11       #+ and assign file descriptor "j" to it.
  12       #  If "filename" does not exist, create it.
  13       #  If file descriptor "j" is not specified, default to fd 0, stdin.
  14       #
  15       #  An application of this is writing at a specified place in a file. 
  16       echo 1234567890 > File    # Write string to "File".
  17       exec 3<> File             # Open "File" and assign fd 3 to it.
  18       read -n 4 <&3             # Read only 4 characters.
  19       echo -n . >&3             # Write a decimal point there.
  20       exec 3>&-                 # Close fd 3.
  21       cat File                  # ==> 1234.67890
  22       #  Random access, by golly.
  23 
  24 
  25 
  26    |
  27       # Pipe.
  28       # General purpose process and command chaining tool.
  29       # Similar to ">", but more general in effect.
  30       # Useful for chaining commands, scripts, files, and programs together.
  31       cat *.txt | sort | uniq > result-file
  32       # Sorts the output of all the .txt files and deletes duplicate lines,
  33       # finally saves results to "result-file".

Multiple instances of input and output redirection and/or pipes can be combined in a single command line.
   1 command < input-file > output-file
   2 
   3 command1 | command2 | command3 > output-file
See Example 15-31 and Example A-15.

Multiple output streams may be redirected to one file.
   1 ls -yz >> command.log 2>&1
   2 #  Capture result of illegal options "yz" in file "command.log."
   3 #  Because stderr is redirected to the file,
   4 #+ any error messages will also be there.
   5 
   6 #  Note, however, that the following does *not* give the same result.
   7 ls -yz 2>&1 >> command.log
   8 #  Outputs an error message and does not write to file.
   9 
  10 #  If redirecting both stdout and stderr,
  11 #+ the order of the commands makes a difference.

Closing File Descriptors

n<&-

Close input file descriptor n.

0<&-, <&-

Close stdin.

n>&-

Close output file descriptor n.

1>&-, >&-

Close stdout.

Child processes inherit open file descriptors. This is why pipes work. To prevent an fd from being inherited, close it.
   1 # Redirecting only stderr to a pipe.
   2 
   3 exec 3>&1                              # Save current "value" of stdout.
   4 ls -l 2>&1 >&3 3>&- | grep bad 3>&-    # Close fd 3 for 'grep' (but not 'ls').
   5 #              ^^^^   ^^^^
   6 exec 3>&-                              # Now close it for the remainder of the script.
   7 
   8 # Thanks, S.C.

For a more detailed introduction to I/O redirection see Appendix E.

19.1. Using exec

An exec <filename command redirects stdin to a file. From that point on, all stdin comes from that file, rather than its normal source (usually keyboard input). This provides a method of reading a file line by line and possibly parsing each line of input using sed and/or awk.


Example 19-1. Redirecting stdin using exec

   1 #!/bin/bash
   2 # Redirecting stdin using 'exec'.
   3 
   4 
   5 exec 6<&0          # Link file descriptor #6 with stdin.
   6                    # Saves stdin.
   7 
   8 exec < data-file   # stdin replaced by file "data-file"
   9 
  10 read a1            # Reads first line of file "data-file".
  11 read a2            # Reads second line of file "data-file."
  12 
  13 echo
  14 echo "Following lines read from file."
  15 echo "-------------------------------"
  16 echo $a1
  17 echo $a2
  18 
  19 echo; echo; echo
  20 
  21 exec 0<&6 6<&-
  22 #  Now restore stdin from fd #6, where it had been saved,
  23 #+ and close fd #6 ( 6<&- ) to free it for other processes to use.
  24 #
  25 # <&6 6<&-    also works.
  26 
  27 echo -n "Enter data  "
  28 read b1  # Now "read" functions as expected, reading from normal stdin.
  29 echo "Input read from stdin."
  30 echo "----------------------"
  31 echo "b1 = $b1"
  32 
  33 echo
  34 
  35 exit 0

Similarly, an exec >filename command redirects stdout to a designated file. This sends all command output that would normally go to stdout to that file.

Important

exec N > filename affects the entire script or current shell. Redirection in the PID of the script or shell from that point on has changed. However . . .

N > filename affects only the newly-forked process, not the entire script or shell.

Thank you, Ahmed Darwish, for pointing this out.


Example 19-2. Redirecting stdout using exec

   1 #!/bin/bash
   2 # reassign-stdout.sh
   3 
   4 LOGFILE=logfile.txt
   5 
   6 exec 6>&1           # Link file descriptor #6 with stdout.
   7                     # Saves stdout.
   8 
   9 exec > $LOGFILE     # stdout replaced with file "logfile.txt".
  10 
  11 # ----------------------------------------------------------- #
  12 # All output from commands in this block sent to file $LOGFILE.
  13 
  14 echo -n "Logfile: "
  15 date
  16 echo "-------------------------------------"
  17 echo
  18 
  19 echo "Output of \"ls -al\" command"
  20 echo
  21 ls -al
  22 echo; echo
  23 echo "Output of \"df\" command"
  24 echo
  25 df
  26 
  27 # ----------------------------------------------------------- #
  28 
  29 exec 1>&6 6>&-      # Restore stdout and close file descriptor #6.
  30 
  31 echo
  32 echo "== stdout now restored to default == "
  33 echo
  34 ls -al
  35 echo
  36 
  37 exit 0


Example 19-3. Redirecting both stdin and stdout in the same script with exec

   1 #!/bin/bash
   2 # upperconv.sh
   3 # Converts a specified input file to uppercase.
   4 
   5 E_FILE_ACCESS=70
   6 E_WRONG_ARGS=71
   7 
   8 if [ ! -r "$1" ]     # Is specified input file readable?
   9 then
  10   echo "Can't read from input file!"
  11   echo "Usage: $0 input-file output-file"
  12   exit $E_FILE_ACCESS
  13 fi                   #  Will exit with same error
  14                      #+ even if input file ($1) not specified (why?).
  15 
  16 if [ -z "$2" ]
  17 then
  18   echo "Need to specify output file."
  19   echo "Usage: $0 input-file output-file"
  20   exit $E_WRONG_ARGS
  21 fi
  22 
  23 
  24 exec 4<&0
  25 exec < $1            # Will read from input file.
  26 
  27 exec 7>&1
  28 exec > $2            # Will write to output file.
  29                      # Assumes output file writable (add check?).
  30 
  31 # -----------------------------------------------
  32     cat - | tr a-z A-Z   # Uppercase conversion.
  33 #   ^^^^^                # Reads from stdin.
  34 #           ^^^^^^^^^^   # Writes to stdout.
  35 # However, both stdin and stdout were redirected.
  36 # Note that the 'cat' can be omitted.
  37 # -----------------------------------------------
  38 
  39 exec 1>&7 7>&-       # Restore stout.
  40 exec 0<&4 4<&-       # Restore stdin.
  41 
  42 # After restoration, the following line prints to stdout as expected.
  43 echo "File \"$1\" written to \"$2\" as uppercase conversion."
  44 
  45 exit 0

I/O redirection is a clever way of avoiding the dreaded inaccessible variables within a subshell problem.


Example 19-4. Avoiding a subshell

   1 #!/bin/bash
   2 # avoid-subshell.sh
   3 # Suggested by Matthew Walker.
   4 
   5 Lines=0
   6 
   7 echo
   8 
   9 cat myfile.txt | while read line;
  10                  do {
  11                    echo $line
  12                    (( Lines++ ));  #  Incremented values of this variable
  13                                    #+ inaccessible outside loop.
  14                                    #  Subshell problem.
  15                  }
  16                  done
  17 
  18 echo "Number of lines read = $Lines"     # 0
  19                                          # Wrong!
  20 
  21 echo "------------------------"
  22 
  23 
  24 exec 3<> myfile.txt
  25 while read line <&3
  26 do {
  27   echo "$line"
  28   (( Lines++ ));                   #  Incremented values of this variable
  29                                    #+ accessible outside loop.
  30                                    #  No subshell, no problem.
  31 }
  32 done
  33 exec 3>&-
  34 
  35 echo "Number of lines read = $Lines"     # 8
  36 
  37 echo
  38 
  39 exit 0
  40 
  41 # Lines below not seen by script.
  42 
  43 $ cat myfile.txt
  44 
  45 Line 1.
  46 Line 2.
  47 Line 3.
  48 Line 4.
  49 Line 5.
  50 Line 6.
  51 Line 7.
  52 Line 8.

Notes

[1]

By convention in UNIX and Linux, data streams and peripherals (device files) are treated as files, in a fashion analogous to ordinary files.

[2]

A file descriptor is simply a number that the operating system assigns to an open file to keep track of it. Consider it a simplified type of file pointer. It is analogous to a file handle in C.

[3]

Using file descriptor 5 might cause problems. When Bash creates a child process, as with exec, the child inherits fd 5 (see Chet Ramey's archived e-mail, SUBJECT: RE: File descriptor 5 is held open). Best leave this particular fd alone.