Advanced Bash-Scripting Guide (PART 4)

Part 4. Commands

   Mastering the commands on your Linux machine is an indispensable
   prelude to writing effective shell scripts.

   This section covers the following commands:

     * . (See also source)
     * ac
     * adduser
     * agetty
     * agrep
     * ar
     * arch
     * at
     * autoload
     * awk (See also Using awk for math operations)
     * badblocks
     * banner
     * basename
     * batch
     * bc
     * bg
     * bind
     * bison
     * builtin
     * bzgrep
     * bzip2
     * cal
     * caller
     * cat
     * cd
     * chattr
     * chfn
     * chgrp
     * chkconfig
     * chmod
     * chown
     * chroot
     * cksum
     * clear
     * clock
     * cmp
     * col
     * colrm
     * column
     * comm
     * command
     * compgen
     * complete
     * compress
     * coproc
     * cp
     * cpio
     * cron
     * crypt
     * csplit
     * cu
     * cut
     * date
     * dc
     * dd
     * debugfs
     * declare
     * depmod
     * df
     * dialog
     * diff
     * diff3
     * diffstat
     * dig
     * dirname
     * dirs
     * disown
     * dmesg
     * doexec
     * dos2unix
     * du
     * dump
     * dumpe2fs
     * e2fsck
     * echo
     * egrep
     * enable
     * enscript
     * env
     * eqn
     * eval
     * exec
     * exit (Related topic: exit status)
     * expand
     * export
     * expr
     * factor
     * false
     * fdformat
     * fdisk
     * fg
     * fgrep
     * file
     * find
     * finger
     * flex
     * flock
     * fmt
     * fold
     * free
     * fsck
     * ftp
     * fuser
     * getfacl
     * getopt
     * getopts
     * gettext
     * getty
     * gnome-mount
     * grep
     * groff
     * groupmod
     * groups (Related topic: the $GROUPS variable)
     * gs
     * gzip
     * halt
     * hash
     * hdparm
     * head
     * help
     * hexdump
     * host
     * hostid
     * hostname (Related topic: the $HOSTNAME variable)
     * hwclock
     * iconv
     * id (Related topic: the $UID variable)
     * ifconfig
     * info
     * infocmp
     * init
     * insmod
     * install
     * ip
     * ipcalc
     * iptables
     * iwconfig
     * jobs
     * join
     * jot
     * kill
     * killall
     * last
     * lastcomm
     * lastlog
     * ldd
     * less
     * let
     * lex
     * lid
     * ln
     * locate
     * lockfile
     * logger
     * logname
     * logout
     * logrotate
     * look
     * losetup
     * lp
     * ls
     * lsdev
     * lsmod
     * lsof
     * lspci
     * lsusb
     * ltrace
     * lynx
     * lzcat
     * lzma
     * m4
     * mail
     * mailstats
     * mailto
     * make
     * MAKEDEV
     * man
     * mapfile
     * mcookie
     * md5sum
     * merge
     * mesg
     * mimencode
     * mkbootdisk
     * mkdir
     * mke2fs
     * mkfifo
     * mkisofs
     * mknod
     * mkswap
     * mktemp
     * mmencode
     * modinfo
     * modprobe
     * more
     * mount
     * msgfmt
     * mv
     * nc
     * netconfig
     * netstat
     * newgrp
     * nice
     * nl
     * nm
     * nmap
     * nohup
     * nslookup
     * objdump
     * od
     * openssl
     * passwd
     * paste
     * patch (Related topic: diff)
     * pathchk
     * pax
     * pgrep
     * pidof
     * ping
     * pkill
     * popd
     * pr
     * printenv
     * printf
     * procinfo
     * ps
     * pstree
     * ptx
     * pushd
     * pwd (Related topic: the $PWD variable)
     * quota
     * rcp
     * rdev
     * rdist
     * read
     * readelf
     * readlink
     * readonly
     * reboot
     * recode
     * renice
     * reset
     * resize
     * restore
     * rev
     * rlogin
     * rm
     * rmdir
     * rmmod
     * route
     * rpm
     * rpm2cpio
     * rsh
     * rsync
     * runlevel
     * run-parts
     * rx
     * rz
     * sar
     * scp
     * script
     * sdiff
     * sed
     * seq
     * service
     * set
     * setfacl
     * setquota
     * setserial
     * setterm
     * sha1sum
     * shar
     * shopt
     * shred
     * shutdown
     * size
     * skill
     * sleep
     * slocate
     * snice
     * sort
     * source
     * sox
     * split
     * sq
     * ssh
     * stat
     * strace
     * strings
     * strip
     * stty
     * su
     * sudo
     * sum
     * suspend
     * swapoff
     * swapon
     * sx
     * sync
     * sz
     * tac
     * tail
     * tar
     * tbl
     * tcpdump
     * tee
     * telinit
     * telnet
     * Tex
     * texexec
     * time
     * times
     * tmpwatch
     * top
     * touch
     * tput
     * tr
     * traceroute
     * true
     * tset
     * tsort
     * tty
     * tune2fs
     * type
     * typeset
     * ulimit
     * umask
     * umount
     * uname
     * unarc
     * unarj
     * uncompress
     * unexpand
     * uniq
     * units
     * unlzma
     * unrar
     * unset
     * unsq
     * unzip
     * uptime
     * usbmodules
     * useradd
     * userdel
     * usermod
     * users
     * usleep
     * uucp
     * uudecode
     * uuencode
     * uux
     * vacation
     * vdir
     * vmstat
     * vrfy
     * w
     * wait
     * wall
     * watch
     * wc
     * wget
     * whatis
     * whereis
     * which
     * who
     * whoami
     * whois
     * write
     * xargs
     * yacc
     * yes
     * zcat
     * zdiff
     * zdump
     * zegrep
     * zfgrep
     * zgrep
     * zip

   Table of Contents
   15. Internal Commands and Builtins

        15.1. Job Control Commands

   16. External Filters, Programs and Commands

        16.1. Basic Commands
        16.2. Complex Commands
        16.3. Time / Date Commands
        16.4. Text Processing Commands
        16.5. File and Archiving Commands
        16.6. Communications Commands
        16.7. Terminal Control Commands
        16.8. Math Commands
        16.9. Miscellaneous Commands

   17. System and Administrative Commands

        17.1. Analyzing a System Script

Chapter 15. Internal Commands and Builtins

   A builtin is a command contained within the Bash tool set, literally
   built in. This is either for performance reasons -- builtins execute
   faster than external commands, which usually require forking off [55]
   a separate process -- or because a particular builtin needs direct
   access to the shell internals.

   When a command or the shell itself initiates (or spawns) a new
   subprocess to carry out a task, this is called forking. This new
   process is the child, and the process that forked it off is the
   parent. While the child process is doing its work, the parent process
   is still executing.

   Note that while a parent process gets the process ID of the child
   process, and can thus pass arguments to it, the reverse is not true.
   This can create problems that are subtle and hard to track down.

   Example 15-1. A script that spawns multiple instances of itself

PIDS=$(pidof sh $0)  # Process IDs of the various instances of this script.
P_array=( $PIDS )    # Put them in an array (why?).
echo $PIDS           # Show process IDs of parent and child processes.
let "instances = ${#P_array[*]} - 1"  # Count elements, less 1.
                                      # Why subtract 1?
echo "$instances instance(s) of this script running."
echo "[Hit Ctl-C to exit.]"; echo

sleep 1              # Wait.
sh $0                # Play it again, Sam.

exit 0               # Not necessary; script will never get to here.
                     # Why not?

#  After exiting with a Ctl-C,
#+ do all the spawned instances of the script die?
#  If so, why?

# Note:
# ----
# Be careful not to run this script too long.
# It will eventually eat up too many system resources.

#  Is having a script spawn multiple instances of itself
#+ an advisable scripting technique.
#  Why or why not?

   Generally, a Bash builtin does not fork a subprocess when it executes
   within a script. An external system command or filter in a script
   usually will fork a subprocess.

   A builtin may be a synonym to a system command of the same name, but
   Bash reimplements it internally. For example, the Bash echo command
   is not the same as /bin/echo, although their behavior is almost

echo "This line uses the \"echo\" builtin."
/bin/echo "This line uses the /bin/echo system command."

   A keyword is a reserved word, token or operator. Keywords have a
   special meaning to the shell, and indeed are the building blocks of
   the shell's syntax. As examples, for, while, do, and ! are keywords.
   Similar to a builtin, a keyword is hard-coded into Bash, but unlike a
   builtin, a keyword is not in itself a command, but a subunit of a
   command construct. [56]


          prints (to stdout) an expression or variable (see Example

echo Hello
echo $a

          An echo requires the -e option to print escaped characters.
          See Example 5-2.

          Normally, each echo command prints a terminal newline, but the
          -n option suppresses this.


   An echo can be used to feed a sequence of commands down a pipe.
if echo "$VAR" | grep -q txt   # if [[ $VAR = *txt* ]]
  echo "$VAR contains the substring sequence \"txt\""


   An echo, in combination with command substitution can set a variable.
   a=`echo "HELLO" | tr A-Z a-z`
   See also Example 16-22, Example 16-3, Example 16-47, and Example

          Be aware that echo `command` deletes any linefeeds that the
          output of command generates.

          The $IFS (internal field separator) variable normally contains
          \n (linefeed) as one of its set of whitespace characters. Bash
          therefore splits the output of command at linefeeds into
          arguments to echo. Then echo outputs these arguments,
          separated by spaces.

bash$ ls -l /usr/share/apps/kjezz/sounds
-rw-r--r--    1 root     root         1407 Nov  7  2000
 -rw-r--r--    1 root     root          362 Nov  7  2000

bash$ echo `ls -l /usr/share/apps/kjezz/sounds`
total 40 -rw-r--r-- 1 root root 716 Nov 7 2000 -rw-r--r-- 1 root ro
ot ...

          So, how can we embed a linefeed within an echoed character

# Embedding a linefeed?
echo "Why doesn't this string \n split on two lines?"
# Doesn't split.

# Let's try something else.


echo $"A line of text containing
a linefeed."
# Prints as two distinct lines (embedded linefeed).
# But, is the "$" variable prefix really necessary?


echo "This string splits
on two lines."
# No, the "$" is not needed.

echo "---------------"

echo -n $"Another line of text containing
a linefeed."
# Prints as two distinct lines (embedded linefeed).
# Even the -n option fails to suppress the linefeed here.

echo "---------------"

# However, the following doesn't work as expected.
# Why not? Hint: Assignment to a variable.
string1=$"Yet another line of text containing
a linefeed (maybe)."

echo $string1
# Yet another line of text containing a linefeed (maybe).
#                                    ^
# Linefeed becomes a space.

# Thanks, Steve Parker, for pointing this out.


   This command is a shell builtin, and not the same as /bin/echo,
   although its behavior is similar.
bash$ type -a echo
echo is a shell builtin
 echo is /bin/echo

          The printf, formatted print, command is an enhanced echo. It
          is a limited variant of the C language printf() library
          function, and its syntax is somewhat different.

          printf format-string... parameter...

          This is the Bash builtin version of the /bin/printf or
          /usr/bin/printf command. See the printf manpage (of the system
          command) for in-depth coverage.


   Older versions of Bash may not support printf.

          Example 15-2. printf in action

# printf demo

declare -r PI=3.14159265358979     # Read-only variable, i.e., a constant.
declare -r DecimalConstant=31373



printf "Pi to 2 decimal places = %1.2f" $PI
printf "Pi to 9 decimal places = %1.9f" $PI  # It even rounds off correctly.

printf "\n"                                  # Prints a line feed,
                                             # Equivalent to 'echo' . . .

printf "Constant = \t%d\n" $DecimalConstant  # Inserts tab (\t).

printf "%s %s \n" $Message1 $Message2


# ==========================================#
# Simulation of C function, sprintf().
# Loading a variable with a formatted string.


Pi12=$(printf "%1.12f" $PI)
echo "Pi to 12 decimal places = $Pi12"      # Roundoff error!

Msg=`printf "%s %s \n" $Message1 $Message2`
echo $Msg; echo $Msg

#  As it happens, the 'sprintf' function can now be accessed
#+ as a loadable module to Bash,
#+ but this is not portable.

exit 0

          Formatting error messages is a useful application of printf



  printf "$@" >&2
  # Formats positional params passed, and sends them to stderr.
  exit $E_BADDIR

cd $var || error $"Can't cd to %s." "$var"

# Thanks, S.C.

          See also Example 36-15.

          "Reads" the value of a variable from stdin, that is,
          interactively fetches input from the keyboard. The -a option
          lets read get array variables (see Example 27-6).

          Example 15-3. Variable assignment, using read

# "Reading" variables.

echo -n "Enter the value of variable 'var1': "
# The -n option to echo suppresses newline.

read var1
# Note no '$' in front of var1, since it is being set.

echo "var1 = $var1"


# A single 'read' statement can set multiple variables.
echo -n "Enter the values of variables 'var2' and 'var3' "
echo =n "(separated by a space or tab): "
read var2 var3
echo "var2 = $var2      var3 = $var3"
#  If you input only one value,
#+ the other variable(s) will remain unset (null).

exit 0

          A read without an associated variable assigns its input to the
          dedicated variable $REPLY.

          Example 15-4. What happens when read has no variable



# -------------------------- #
echo -n "Enter a value: "
read var
echo "\"var\" = "$var""
# Everything as expected here.
# -------------------------- #


# ------------------------------------------------------------------- #
echo -n "Enter another value: "
read           #  No variable supplied for 'read', therefore...
               #+ Input to 'read' assigned to default variable, $REPLY.
echo "\"var\" = "$var""
# This is equivalent to the first code block.
# ------------------------------------------------------------------- #

echo "========================="

#  This example is similar to the "" script.
#  However, this one shows that $REPLY is available
#+ even after a 'read' to a variable in the conventional way.

# ================================================================= #

#  In some instances, you might wish to discard the first value read.
#  In such cases, simply ignore the $REPLY variable.

{ # Code block.
read            # Line 1, to be discarded.
read line2      # Line 2, saved in variable.
  } <$0
echo "Line 2 of this script is:"
echo "$line2"   #   #
echo            #   #!/bin/bash  line discarded.

# See also the script.

exit 0

          Normally, inputting a \ suppresses a newline during input to a
          read. The -r option causes an inputted \ to be interpreted

          Example 15-5. Multi-line input to read



echo "Enter a string terminated by a \\, then press <ENTER>."
echo "Then, enter a second string (no \\ this time), and again press <ENTER>."

read var1     # The "\" suppresses the newline, when reading $var1.
              #     first line \
              #     second line

echo "var1 = $var1"
#     var1 = first line second line

#  For each line terminated by a "\"
#+ you get a prompt on the next line to continue feeding characters into var1.

echo; echo

echo "Enter another string terminated by a \\ , then press <ENTER>."
read -r var2  # The -r option causes the "\" to be read literally.
              #     first line \

echo "var2 = $var2"
#     var2 = first line \

# Data entry terminates with the first <ENTER>.


exit 0

          The read command has some interesting options that permit
          echoing a prompt and even reading keystrokes without hitting

# Read a keypress without hitting ENTER.

read -s -n1 -p "Hit a key " keypress
echo; echo "Keypress was "\"$keypress\""."

# -s option means do not echo input.
# -n N option means accept only N characters of input.
# -p option means echo the following prompt before reading input.

# Using these options is tricky, since they need to be in the correct order.

          The -n option to read also allows detection of the arrow keys
          and certain of the other unusual keys.

          Example 15-6. Detecting the arrow keys

# Detects the arrow keys, and a few more.
# Thank you, Sandro Magi, for showing me how.

# --------------------------------------------
# Character codes generated by the keypresses.
# --------------------------------------------


echo -n "Press a key...  "
# May need to also press ENTER if a key not listed above pressed.
read -n3 key                      # Read 3 characters.

echo -n "$key" | grep "$arrowup"  #Check if character code detected.
if [ "$?" -eq $SUCCESS ]
  echo "Up-arrow key pressed."
  exit $SUCCESS

echo -n "$key" | grep "$arrowdown"
if [ "$?" -eq $SUCCESS ]
  echo "Down-arrow key pressed."
  exit $SUCCESS

echo -n "$key" | grep "$arrowrt"
if [ "$?" -eq $SUCCESS ]
  echo "Right-arrow key pressed."
  exit $SUCCESS

echo -n "$key" | grep "$arrowleft"
if [ "$?" -eq $SUCCESS ]
  echo "Left-arrow key pressed."
  exit $SUCCESS

echo -n "$key" | grep "$insert"
if [ "$?" -eq $SUCCESS ]
  echo "\"Insert\" key pressed."
  exit $SUCCESS

echo -n "$key" | grep "$delete"
if [ "$?" -eq $SUCCESS ]
  echo "\"Delete\" key pressed."
  exit $SUCCESS

echo " Some other key pressed."

exit $OTHER

# ========================================= #

#  Mark Alexander came up with a simplified
#+ version of the above script (Thank you!).
#  It eliminates the need for grep.



  read -s -n3 -p "Hit an arrow key: " x

  case "$x" in
     echo "You pressed up-arrow"
     echo "You pressed down-arrow"
     echo "You pressed left-arrow"
     echo "You pressed right-arrow"

exit $?

# ========================================= #

# Antonio Macchi has a simpler alternative.


while true
  read -sn1 a
  test "$a" == `echo -en "\e"` || continue
  read -sn1 a
  test "$a" == "[" || continue
  read -sn1 a
  case "$a" in
    A)  echo "up";;
    B)  echo "down";;
    C)  echo "right";;
    D)  echo "left";;

# ========================================= #

#  Exercise:
#  --------
#  1) Add detection of the "Home," "End," "PgUp," and "PgDn" keys.


   The -n option to read will not detect the ENTER (newline) key.

          The -t option to read permits timed input (see Example 9-4 and
          Example A-41).

          The -u option takes the file descriptor of the target file.

          The read command may also "read" its variable value from a
          file redirected to stdin. If the file contains more than one
          line, only the first line is assigned to the variable. If read
          has more than one parameter, then each of these variables gets
          assigned a successive whitespace-delineated string. Caution!

          Example 15-7. Using read with file redirection


read var1 <data-file
echo "var1 = $var1"
# var1 set to the entire first line of the input file "data-file"

read var2 var3 <data-file
echo "var2 = $var2   var3 = $var3"
# Note non-intuitive behavior of "read" here.
# 1) Rewinds back to the beginning of input file.
# 2) Each variable is now set to a corresponding string,
#    separated by whitespace, rather than to an entire line of text.
# 3) The final variable gets the remainder of the line.
# 4) If there are more variables to be set than whitespace-terminated strings
#    on the first line of the file, then the excess variables remain empty.

echo "------------------------------------------------"

# How to resolve the above problem with a loop:
while read line
  echo "$line"
done <data-file
# Thanks, Heiner Steven for pointing this out.

echo "------------------------------------------------"

# Use $IFS (Internal Field Separator variable) to split a line of input to
# "read", if you do not want the default to be whitespace.

echo "List of all users:"
OIFS=$IFS; IFS=:       # /etc/passwd uses ":" for field separator.
while read name passwd uid gid fullname ignore
  echo "$name ($fullname)"
done </etc/passwd   # I/O redirection.
IFS=$OIFS              # Restore original $IFS.
# This code snippet also by Heiner Steven.

#  Setting the $IFS variable within the loop itself
#+ eliminates the need for storing the original $IFS
#+ in a temporary variable.
#  Thanks, Dim Segebart, for pointing this out.
echo "------------------------------------------------"
echo "List of all users:"

while IFS=: read name passwd uid gid fullname ignore
  echo "$name ($fullname)"
done </etc/passwd   # I/O redirection.

echo "\$IFS still $IFS"

exit 0


   Piping output to a read, using echo to set variables will fail.
   Yet, piping the output of cat seems to work.
cat file1 file2 |
while read line
echo $line

   However, as BjЖn Eriksson shows:
   Example 15-8. Problems reading from a pipe
# This example contributed by Bjon Eriksson.

### shopt -s lastpipe

cat $0 |
while read line
    echo "{$line}"

echo "++++++++++++++++++++++"
printf "\nAll done, last: $last\n" #  The output of this line
                                   #+ changes if you uncomment line 5.
                                   #  (Bash, version -ge 4.2 required.)

exit 0  # End of code.
        # (Partial) output of script follows.
        # The 'echo' supplies extra brackets.



{cat $0 |}
{while read line}
{echo "{$line}"}
{printf "nAll done, last: $lastn"}

All done, last: (null)

The variable (last) is set within the loop/subshell
but its value does not persist outside the loop.

   The gendiff script, usually found in /usr/bin on many Linux distros,
   pipes the output of find to a while read construct.

find $1 \( -name "*$2" -o -name ".*$2" \) -print |
while read f; do
. . .


   It is possible to paste text into the input field of a read (but not
   multiple lines!). See Example A-38.


          The familiar cd change directory command finds use in scripts
          where execution of a command requires being in a specified

(cd /source/directory && tar cf - . ) | (cd /dest/directory && tar xpvf -)

          [from the previously cited example by Alan Cox]

          The -P (physical) option to cd causes it to ignore symbolic

          cd - changes to $OLDPWD, the previous working directory.


   The cd command does not function as expected when presented with two
   forward slashes.
bash$ cd //
bash$ pwd

   The output should, of course, be /. This is a problem both from the
   command-line and in a script.

          Print Working Directory. This gives the user's (or script's)
          current directory (see Example 15-9). The effect is identical
          to reading the value of the builtin variable $PWD.

   pushd, popd, dirs
          This command set is a mechanism for bookmarking working
          directories, a means of moving back and forth through
          directories in an orderly manner. A pushdown stack is used to
          keep track of directory names. Options allow various
          manipulations of the directory stack.

          pushd dir-name pushes the path dir-name onto the directory
          stack and simultaneously changes the current working directory
          to dir-name

          popd removes (pops) the top directory path name off the
          directory stack and simultaneously changes the current working
          directory to that directory popped from the stack.

          dirs lists the contents of the directory stack (compare this
          with the $DIRSTACK variable). A successful pushd or popd will
          automatically invoke dirs.

          Scripts that require various changes to the current working
          directory without hard-coding the directory name changes can
          make good use of these commands. Note that the implicit
          $DIRSTACK array variable, accessible from within a script,
          holds the contents of the directory stack.

          Example 15-9. Changing the current working directory



pushd $dir1
# Will do an automatic 'dirs' (list directory stack to stdout).
echo "Now in directory `pwd`." # Uses back-quoted 'pwd'.

# Now, do some stuff in directory 'dir1'.
pushd $dir2
echo "Now in directory `pwd`."

# Now, do some stuff in directory 'dir2'.
echo "The top entry in the DIRSTACK array is $DIRSTACK."
echo "Now back in directory `pwd`."

# Now, do some more stuff in directory 'dir1'.
echo "Now back in original working directory `pwd`."

exit 0

# What happens if you don't 'popd' -- then exit the script?
# Which directory do you end up in? Why?


          The let command carries out arithmetic operations on
          variables. [57] In many cases, it functions as a less complex
          version of expr.

          Example 15-10. Letting let do arithmetic.



let a=11            # Same as 'a=11'
let a=a+5           # Equivalent to  let "a = a + 5"
                    # (Double quotes and spaces make it more readable.)
echo "11 + 5 = $a"  # 16

let "a <<= 3"       # Equivalent to  let "a = a << 3"
echo "\"\$a\" (=16) left-shifted 3 places = $a"
                    # 128

let "a /= 4"        # Equivalent to  let "a = a / 4"
echo "128 / 4 = $a" # 32

let "a -= 5"        # Equivalent to  let "a = a - 5"
echo "32 - 5 = $a"  # 27

let "a *=  10"      # Equivalent to  let "a = a * 10"
echo "27 * 10 = $a" # 270

let "a %= 8"        # Equivalent to  let "a = a % 8"
echo "270 modulo 8 = $a  (270 / 8 = 33, remainder $a)"
                    # 6

# Does "let" permit C-style operators?
# Yes, just as the (( ... )) double-parentheses construct does.

let a++             # C-style (post) increment.
echo "6++ = $a"     # 6++ = 7
let a--             # C-style decrement.
echo "7-- = $a"     # 7-- = 6
# Of course, ++a, etc., also allowed . . .

# Trinary operator.

# Note that $a is 6, see above.
let "t = a<7?7:11"   # True
echo $t  # 7

let a++
let "t = a<7?7:11"   # False
echo $t  #     11



   The let command can, in certain contexts, return an anomalous exit
# Evgeniy Ivanov points out:

echo $?     # 0
            # As expected.

let var++
echo $?     # 1
            # The command was successful, so why isn't $?=0 ???
            # Anomaly!

let var++
echo $?     # 0
            # As expected.

# Likewise . . .

let var=0
echo $?     # 1
            # The command was successful, so why isn't $?=0 ???
            # Anomaly!

          eval arg1 [arg2] ... [argN]

          Combines the arguments in an expression or list of expressions
          and evaluates them. Any variables within the expression are
          expanded. The net result is to convert a string into a


   The eval command can be used for code generation from the
   command-line or within a script.

bash$ command_string="ps ax"
bash$ process="ps ax"
bash$ eval "$command_string" | grep "$process"
26973 pts/3    R+     0:00 grep --color ps ax
 26974 pts/3    R+     0:00 ps ax

          Each invocation of eval forces a re-evaluation of its


echo $a             # $b
                    # First level.
eval echo $a        # $c
                    # Second level.
eval eval echo $a   # d
                    # Third level.

# Thank you, E. Choroba.

          Example 15-11. Showing the effect of eval

# Exercising "eval" ...

y=`eval ls -l`  #  Similar to y=`ls -l`
echo $y         #+ but linefeeds removed because "echoed" variable is unquoted
echo "$y"       #  Linefeeds preserved when variable is quoted.

echo; echo

y=`eval df`     #  Similar to y=`df`
echo $y         #+ but linefeeds removed.

#  When LF's not preserved, it may make it easier to parse output,
#+ using utilities such as "awk".

echo "==========================================================="

eval "`seq 3 | sed -e 's/.*/echo var&=ABCDEFGHIJ/'`"

echo "==========================================================="

# Now, showing how to do something useful with "eval" . . .
# (Thank you, E. Choroba!)

version=3.4     #  Can we split the version into major and minor
                #+ part in one command?
echo "version = $version"
eval major=${version/./;minor=}     #  Replaces '.' in version by ';minor='
                                    #  The substitution yields '3; minor=4'
                                    #+ so eval does minor=4, major=3
echo Major: $major, minor: $minor   #  Major: 3, minor: 4

          Example 15-12. Using eval to select among variables


#  Passing arguments to a function to select
#+ one particular variable out of a group.

arr0=( 10 11 12 13 14 15 )
arr1=( 20 21 22 23 24 25 )
arr2=( 30 31 32 33 34 35 )
#       0  1  2  3  4  5      Element number (zero-indexed)

choose_array ()
  eval array_member=\${arr${array_number}[element_number]}
  #                 ^       ^^^^^^^^^^^^
  #  Using eval to construct the name of a variable,
  #+ in this particular case, an array name.

  echo "Element $element_number of array $array_number is $array_member"
} #  Function can be rewritten to take parameters.

array_number=0    # First array.
choose_array      # 13

array_number=2    # Third array.
choose_array      # 34

array_number=3    # Null array (arr3 not allocated).
choose_array      # (null)

# Thank you, Antonio Macchi, for pointing this out.

          Example 15-13. Echoing the command-line parameters


# Call this script with a few command-line parameters.
# For example:
#     sh first second third fourth fifth

params=$#              # Number of command-line parameters.
param=1                # Start at first command-line param.

while [ "$param" -le "$params" ]
  echo -n "Command-line parameter "
  echo -n \$$param     #  Gives only the *name* of variable.
#         ^^^          #  $1, $2, $3, etc.
                       #  Why?
                       #  \$ escapes the first "$"
                       #+ so it echoes literally,
                       #+ and $param dereferences "$param" . . .
                       #+ . . . as expected.
  echo -n " = "
  eval echo \$$param   #  Gives the *value* of variable.
# ^^^^      ^^^        #  The "eval" forces the *evaluation*
                       #+ of \$$
                       #+ as an indirect variable reference.

(( param ++ ))         # On to the next.

exit $?

# =================================================

$ sh first second third fourth fifth
Command-line parameter $1 = first
Command-line parameter $2 = second
Command-line parameter $3 = third
Command-line parameter $4 = fourth
Command-line parameter $5 = fifth

          Example 15-14. Forcing a log-off

# Killing ppp to force a log-off.
# For dialup connection, of course.

# Script should be run as root user.

#  Depending on the hardware and even the kernel version,
#+ the modem port on your machine may be different --
#+ /dev/ttyS1 or /dev/ttyS2.

killppp="eval kill -9 `ps ax | awk '/ppp/ { print $1 }'`"
#                     -------- process ID of ppp -------

$killppp                     # This variable is now a command.

# The following operations must be done as root user.

chmod 666 /dev/$SERPORT      # Restore r+w permissions, or else what?
#  Since doing a SIGKILL on ppp changed the permissions on the serial port,
#+ we restore permissions to previous state.

rm /var/lock/LCK..$SERPORT   # Remove the serial port lock file. Why?

exit $?

# Exercises:
# ---------
# 1) Have script check whether root user is invoking it.
# 2) Do a check on whether the process to be killed
#+   is actually running before attempting to kill it.
# 3) Write an alternate version of this script based on 'fuser':
#+      if [ fuser -s /dev/modem ]; then . . .

          Example 15-15. A version of rot13

# A version of "rot13" using 'eval'.
# Compare to "" example.

setvar_rot_13()              # "rot13" scrambling
  local varname=$1 varvalue=$2
  eval $varname='$(echo "$varvalue" | tr a-z n-za-m)'

setvar_rot_13 var "foobar"   # Run "foobar" through rot13.
echo $var                    # sbbone

setvar_rot_13 var "$var"     # Run "sbbone" through rot13.
                             # Back to original variable.
echo $var                    # foobar

# This example by Stephane Chazelas.
# Modified by document author.

exit 0

          The eval command occurs in the older version of indirect

eval var=\$$var


   The eval command can be risky, and normally should be avoided when
   there exists a reasonable alternative. An eval $COMMANDS executes the
   contents of COMMANDS, which may contain such unpleasant surprises as
   rm -rf *. Running an eval on unfamiliar code written by persons
   unknown is living dangerously.

          The set command changes the value of internal script
          variables/options. One use for this is to toggle option flags
          which help determine the behavior of the script. Another
          application for it is to reset the positional parameters that
          a script sees as the result of a command (set `command`). The
          script can then parse the fields of the command output.

          Example 15-16. Using set with positional parameters

# Script "set-test"

# Invoke this script with three command-line parameters,
# for example, "sh one two three".

echo "Positional parameters before  set \`uname -a\` :"
echo "Command-line argument #1 = $1"
echo "Command-line argument #2 = $2"
echo "Command-line argument #3 = $3"

set `uname -a` # Sets the positional parameters to the output
               # of the command `uname -a`

echo +++++
echo $_        # +++++
# Flags set in script.
echo $-        # hB
#                Anomalous behavior?

echo "Positional parameters after  set \`uname -a\` :"
# $1, $2, $3, etc. reinitialized to result of `uname -a`
echo "Field #1 of 'uname -a' = $1"
echo "Field #2 of 'uname -a' = $2"
echo "Field #3 of 'uname -a' = $3"
echo \#\#\#
echo $_        # ###

exit 0

          More fun with positional parameters.

          Example 15-17. Reversing the positional parameters

# Reverse positional parameters.
# Script by Dan Jacobson, with stylistic revisions by document author.

set a\ b c d\ e;
#     ^      ^     Spaces escaped
#       ^ ^        Spaces not escaped
#              ^   Saving old IFS and setting new one.


until [ $# -eq 0 ]
do          #      Step through positional parameters.
  echo "### k0 = "$k""     # Before
  k=$1:$k;  #      Append each pos param to loop variable.
#     ^
  echo "### k = "$k""      # After

set $k  #  Set new positional parameters.
echo -
echo $# #  Count of positional parameters.
echo -

for i   #  Omitting the "in list" sets the variable -- i --
        #+ to the positional parameters.
  echo $i  # Display new positional parameters.

IFS=$OIFS  # Restore IFS.

#  Question:
#  Is it necessary to set an new IFS, internal field separator,
#+ in order for this script to work properly?
#  What happens if you don't? Try it.
#  And, why use the new IFS -- a colon -- in line 17,
#+ to append to the loop variable?
#  What is the purpose of this?

exit 0

$ ./

### k0 =
### k = a b

### k0 = a b
### k = c a b

### k0 = c a b
### k = d e c a b


d e
a b

          Invoking set without any options or arguments simply lists all
          the environmental and other variables that have been

bash$ set

          Using set with the -- option explicitly assigns the contents
          of a variable to the positional parameters. If no variable
          follows the -- it unsets the positional parameters.

          Example 15-18. Reassigning the positional parameters


variable="one two three four five"

set -- $variable
# Sets positional parameters to the contents of "$variable".

shift; shift        # Shift past first two positional params.
# shift 2             also works.

echo "first parameter = $first_param"             # one
echo "second parameter = $second_param"           # two
echo "remaining parameters = $remaining_params"   # three four five

echo; echo

# Again.
set -- $variable
echo "first parameter = $first_param"             # one
echo "second parameter = $second_param"           # two

# ======================================================

set --
# Unsets positional parameters if no variable specified.

echo "first parameter = $first_param"             # (null value)
echo "second parameter = $second_param"           # (null value)

exit 0

          See also Example 11-2 and Example 16-56.

          The unset command deletes a shell variable, effectively
          setting it to null. Note that this command does not affect
          positional parameters.

bash$ unset PATH

bash$ echo $PATH


          Example 15-19. "Unsetting" a variable

# Unsetting a variable.

variable=hello                       #  Initialized.
echo "variable = $variable"

unset variable                       #  Unset.
                                     #  In this particular context,
                                     #+ same effect as:   variable=
echo "(unset) variable = $variable"  #  $variable is null.

if [ -z "$variable" ]                #  Try a string-length test.
  echo "\$variable has zero length."

exit 0


   In most contexts, an undeclared variable and one that has been unset
   are equivalent. However, the ${parameter:-default} parameter
   substitution construct can distinguish between the two.

          The export [58] command makes available variables to all child
          processes of the running script or shell. One important use of
          the export command is in startup files, to initialize and make
          accessible environmental variables to subsequent user


   Unfortunately, there is no way to export variables back to the parent
   process, to the process that called or invoked the script or shell.

          Example 15-20. Using export to pass a variable to an embedded
          awk script


#  Yet another version of the "column totaler" script (
#+ that adds up a specified column (of numbers) in the target file.
#  This uses the environment to pass a script variable to 'awk' . . .
#+ and places the awk script in a variable.


if [ $# -ne "$ARGS" ] # Check for proper number of command-line args.
   echo "Usage: `basename $0` filename column-number"
   exit $E_WRONGARGS


#===== Same as original script, up to this point =====#

export column_number
# Export column number to environment, so it's available for retrieval.

# -----------------------------------------------
awkscript='{ total += $ENVIRON["column_number"] }
END { print total }'
# Yes, a variable can hold an awk script.
# -----------------------------------------------

# Now, run the awk script.
awk "$awkscript" "$filename"

# Thanks, Stephane Chazelas.

exit 0


   It is possible to initialize and export variables in the same
   operation, as in export var1=xxx.
   However, as Greg Keraunen points out, in certain situations this may
   have a different effect than setting a variable, then exporting it.
bash$ export var=(a b); echo ${var[0]}
(a b)

bash$ var=(a b); export var; echo ${var[0]}


   A variable to be exported may require special treatment. See Example

   declare, typeset
          The declare and typeset commands specify and/or restrict
          properties of variables.

          Same as declare -r, sets a variable as read-only, or, in
          effect, as a constant. Attempts to change the variable fail
          with an error message. This is the shell analog of the C
          language const type qualifier.

          This powerful tool parses command-line arguments passed to the
          script. This is the Bash analog of the getopt external command
          and the getopt library function familiar to C programmers. It
          permits passing and concatenating multiple options [59] and
          associated arguments to a script (for example scriptname -abc
          -e /usr/local).

          The getopts construct uses two implicit variables. $OPTIND is
          the argument pointer (OPTion INDex) and $OPTARG (OPTion
          ARGument) the (optional) argument attached to an option. A
          colon following the option name in the declaration tags that
          option as having an associated argument.

          A getopts construct usually comes packaged in a while loop,
          which processes the options and arguments one at a time, then
          increments the implicit $OPTIND variable to point to the next.


         1. The arguments passed from the command-line to the script
            must be preceded by a dash (-). It is the prefixed - that
            lets getopts recognize command-line arguments as options. In
            fact, getopts will not process arguments without the
            prefixed -, and will terminate option processing at the
            first argument encountered lacking them.
         2. The getopts template differs slightly from the standard
            while loop, in that it lacks condition brackets.
         3. The getopts construct is a highly functional replacement for
            the traditional getopt external command.

while getopts ":abcde:fg" Option
# Initial declaration.
# a, b, c, d, e, f, and g are the options (flags) expected.
# The : after option 'e' shows it will have an argument passed with it.
  case $Option in
    a ) # Do something with variable 'a'.
    b ) # Do something with variable 'b'.
    e)  # Do something with 'e', and also with $OPTARG,
        # which is the associated argument passed with option 'e'.
    g ) # Do something with variable 'g'.
shift $(($OPTIND - 1))
# Move argument pointer to next.

# All this is not nearly as complicated as it looks <grin>.

          Example 15-21. Using getopts to read the options/arguments
          passed to a script

# Exercising getopts and OPTIND
#          Script modified 10/09/03 at the suggestion of Bill Gradwohl.

# Here we observe how 'getopts' processes command-line arguments to script.
# The arguments are parsed as "options" (flags) and associated arguments.

# Try invoking this script with:
#   'scriptname -mn'
#   'scriptname -oq qOption' (qOption can be some arbitrary string.)
#   'scriptname -qXXX -r'
#   'scriptname -qr'
#+      - Unexpected result, takes "r" as the argument to option "q"
#   'scriptname -q -r'
#+      - Unexpected result, same as above
#   'scriptname -mnop -mnop'  - Unexpected result
#   (OPTIND is unreliable at stating where an option came from.)
#  If an option expects an argument ("flag:"), then it will grab
#+ whatever is next on the command-line.


if [ $# -eq "$NO_ARGS" ]    # Script invoked with no command-line args?
  echo "Usage: `basename $0` options (-mnopqrs)"
  exit $E_OPTERROR          # Exit and explain usage.
                            # Usage: scriptname -options
                            # Note: dash (-) necessary

while getopts ":mnopq:rs" Option
  case $Option in
    m     ) echo "Scenario #1: option -m-   [OPTIND=${OPTIND}]";;
    n | o ) echo "Scenario #2: option -$Option-   [OPTIND=${OPTIND}]";;
    p     ) echo "Scenario #3: option -p-   [OPTIND=${OPTIND}]";;
    q     ) echo "Scenario #4: option -q-\
                  with argument \"$OPTARG\"   [OPTIND=${OPTIND}]";;
    #  Note that option 'q' must have an associated argument,
    #+ otherwise it falls through to the default.
    r | s ) echo "Scenario #5: option -$Option-";;
    *     ) echo "Unimplemented option chosen.";;   # Default.

shift $(($OPTIND - 1))
#  Decrements the argument pointer so it points to next argument.
#  $1 now references the first non-option item supplied on the command-line
#+ if one exists.

exit $?

#   As Bill Gradwohl states,
#  "The getopts mechanism allows one to specify:  scriptname -mnop -mnop
#+  but there is no reliable way to differentiate what came
#+ from where by using OPTIND."
#  There are, however, workarounds.

   Script Behavior

   source, . (dot command)
          This command, when invoked from the command-line, executes a
          script. Within a script, a source file-name loads the file
          file-name. Sourcing a file (dot-command) imports code into the
          script, appending to the script (same effect as the #include
          directive in a C program). The net result is the same as if
          the "sourced" lines of code were physically present in the
          body of the script. This is useful in situations when multiple
          scripts use a common data file or function library.

          Example 15-22. "Including" a data file


. data-file    # Load a data file.
# Same effect as "source data-file", but more portable.

#  The file "data-file" must be present in current working directory,
#+ since it is referred to by its 'basename'.

# Now, reference some data from that file.

echo "variable1 (from data-file) = $variable1"
echo "variable3 (from data-file) = $variable3"

let "sum = $variable2 + $variable4"
echo "Sum of variable2 + variable4 (from data-file) = $sum"
echo "message1 (from data-file) is \"$message1\""
# Note:                            escaped quotes

print_message This is the message-print function in the data-file.

exit 0

          File data-file for Example 15-22, above. Must be present in
          same directory.

# This is a data file loaded by a script.
# Files of this type may contain variables, functions, etc.
# It may be loaded with a 'source' or '.' command by a shell script.

# Let's initialize some variables.


message1="Hello, how are you?"
message2="Enough for now. Goodbye."

print_message ()
# Echoes any message passed to it.

  if [ -z "$1" ]
    return 1
    # Error, if argument missing.


  until [ -z "$1" ]
    # Step through arguments passed to function.
    echo -n "$1"
    # Echo args one at a time, suppressing line feeds.
    echo -n " "
    # Insert spaces between words.
    # Next one.


  return 0

          If the sourced file is itself an executable script, then it
          will run, then return control to the script that called it. A
          sourced executable script may use a return for this purpose.

          Arguments may be (optionally) passed to the sourced file as
          positional parameters.

source $filename $arg1 arg2

          It is even possible for a script to source itself, though this
          does not seem to have any practical applications.

          Example 15-23. A (useless) script that sources itself

# a script sourcing itself "recursively."
# From "Stupid Script Tricks," Volume II.

MAXPASSCNT=100    # Maximum number of execution passes.

echo -n  "$pass_count  "
#  At first execution pass, this just echoes two blank spaces,
#+ since $pass_count still uninitialized.

let "pass_count += 1"
#  Assumes the uninitialized variable $pass_count
#+ can be incremented the first time around.
#  This works with Bash and pdksh, but
#+ it relies on non-portable (and possibly dangerous) behavior.
#  Better would be to initialize $pass_count to 0 before incrementing.

while [ "$pass_count" -le $MAXPASSCNT ]
  . $0   # Script "sources" itself, rather than calling itself.
         # ./$0 (which would be true recursion) doesn't work here. Why?

#  What occurs here is not actually recursion,
#+ since the script effectively "expands" itself, i.e.,
#+ generates a new section of code
#+ with each pass through the 'while' loop',
#  with each 'source' in line 20.
#  Of course, the script interprets each newly 'sourced' "#!" line
#+ as a comment, and not as the start of a new script.


exit 0   # The net effect is counting from 1 to 100.
         # Very impressive.

# Exercise:
# --------
# Write a script that uses this trick to actually do something useful.

          Unconditionally terminates a script. [60] The exit command may
          optionally take an integer argument, which is returned to the
          shell as the exit status of the script. It is good practice to
          end all but the simplest scripts with an exit 0, indicating a
          successful run.


   If a script terminates with an exit lacking an argument, the exit
   status of the script is the exit status of the last command executed
   in the script, not counting the exit. This is equivalent to an exit


   An exit command may also be used to terminate a subshell.

          This shell builtin replaces the current process with a
          specified command. Normally, when the shell encounters a
          command, it forks off a child process to actually execute the
          command. Using the exec builtin, the shell does not fork, and
          the command exec'ed replaces the shell. When used in a script,
          therefore, it forces an exit from the script when the exec'ed
          command terminates. [61]

          Example 15-24. Effects of exec


exec echo "Exiting \"$0\"."   # Exit from script here.

# ----------------------------------
# The following lines never execute.

echo "This echo will never echo."

exit 99                       #  This script will not exit here.
                              #  Check exit value after script terminates
                              #+ with an 'echo $?'.
                              #  It will *not* be 99.

          Example 15-25. A script that exec's itself


# Note: Set permissions on this script to 555 or 755,
#       then call it with ./ or sh ./


echo "This line appears ONCE in the script, yet it keeps echoing."
echo "The PID of this instance of the script is still $$."
#     Demonstrates that a subshell is not forked off.

echo "==================== Hit Ctl-C to exit ===================="

sleep 1

exec $0   #  Spawns another instance of this same script
          #+ that replaces the previous one.

echo "This line will never echo!"  # Why not?

exit 99                            # Will not exit here!
                                   # Exit code will not be 99!

          An exec also serves to reassign file descriptors. For example,
          exec <zzz-file replaces stdin with the file zzz-file.


   The -exec option to find is not the same as the exec shell builtin.

          This command permits changing shell options on the fly (see
          Example 25-1 and Example 25-2). It often appears in the Bash
          startup files, but also has its uses in scripts. Needs version
          2 or later of Bash.

shopt -s cdspell
# Allows minor misspelling of directory names with 'cd'
# Option -s sets, -u unsets.

cd /hpme  # Oops! Mistyped '/home'.
pwd       # /home
          # The shell corrected the misspelling.

          Putting a caller command inside a function echoes to stdout
          information about the caller of that function.


function1 ()
  # Inside function1 ().
  caller 0   # Tell me about it.

function1    # Line 9 of script.

# 9 main
# ^                 Line number that the function was called from.
#   ^^^^            Invoked from "main" part of script.
#        ^^^^^^^    Name of calling script.

caller 0     # Has no effect because it's not inside a function.

          A caller command can also return caller information from a
          script sourced within another script. Analogous to a function,
          this is a "subroutine call."

          You may find this command useful in debugging.


          A command that returns a successful (zero) exit status, but
          does nothing else.

bash$ true
bash$ echo $?

# Endless loop
while true   # alias for ":"
   # Need a way to break out of loop or script will hang.

          A command that returns an unsuccessful exit status, but does
          nothing else.

bash$ false
bash$ echo $?

# Testing "false"
if false
  echo "false evaluates \"true\""
  echo "false evaluates \"false\""
# false evaluates "false"

# Looping while "false" (null loop)
while false
   # The following code will not execute.
   # Nothing happens!

   type [cmd]
          Similar to the which external command, type cmd identifies
          "cmd." Unlike which, type is a Bash builtin. The useful -a
          option to type identifies keywords and builtins, and also
          locates system commands with identical names.

bash$ type '['
[ is a shell builtin
bash$ type -a '['
[ is a shell builtin
 [ is /usr/bin/[

bash$ type type
type is a shell builtin

          The type command can be useful for testing whether a certain
          command exists.

   hash [cmds]
          Records the path name of specified commands -- in the shell
          hash table [62] -- so the shell or script will not need to
          search the $PATH on subsequent calls to those commands. When
          hash is called with no arguments, it simply lists the commands
          that have been hashed. The -r option resets the hash table.

          The bind builtin displays or modifies readline [63] key

          Gets a short usage summary of a shell builtin. This is the
          counterpart to whatis, but for builtins. The display of help
          information got a much-needed update in the version 4 release
          of Bash.

bash$ help exit
exit: exit [n]
    Exit the shell with a status of N.  If N is omitted, the exit status
    is that of the last command executed.

15.1. Job Control Commands

   Certain of the following job control commands take a job identifier
   as an argument. See the table at end of the chapter.

          Lists the jobs running in the background, giving the job
          number. Not as useful as ps.


   It is all too easy to confuse jobs and processes. Certain builtins,
   such as kill, disown, and wait accept either a job number or a
   process number as an argument. The fg, bg and jobs commands accept
   only a job number.
bash$ sleep 100 &
[1] 1384

bash $ jobs
[1]+  Running                 sleep 100 &

   "1" is the job number (jobs are maintained by the current shell).
   "1384" is the PID or process ID number (processes are maintained by
   the system). To kill this job/process, either a kill %1 or a kill
   1384 works.
   Thanks, S.C.

          Remove job(s) from the shell's table of active jobs.

   fg, bg
          The fg command switches a job running in the background into
          the foreground. The bg command restarts a suspended job, and
          runs it in the background. If no job number is specified, then
          the fg or bg command acts upon the currently running job.

          Suspend script execution until all jobs running in background
          have terminated, or until the job number or process ID
          specified as an option terminates. Returns the exit status of
          waited-for command.

          You may use the wait command to prevent a script from exiting
          before a background job finishes executing (this would create
          a dreaded orphan process).

          Example 15-26. Waiting for a process to finish before


ROOT_UID=0   # Only users with $UID 0 have root privileges.

if [ "$UID" -ne "$ROOT_UID" ]
  echo "Must be root to run this script."
  # "Run along kid, it's past your bedtime."
  exit $E_NOTROOT

if [ -z "$1" ]
  echo "Usage: `basename $0` find-string"
  exit $E_NOPARAMS

echo "Updating 'locate' database..."
echo "This may take a while."
updatedb /usr &     # Must be run as root.

# Don't run the rest of the script until 'updatedb' finished.
# You want the the database updated before looking up the file name.

locate $1

#  Without the 'wait' command, in the worse case scenario,
#+ the script would exit while 'updatedb' was still running,
#+ leaving it as an orphan process.

exit 0

          Optionally, wait can take a job identifier as an argument, for
          example, wait%1 or wait $PPID. [64] See the job id table.


   Within a script, running a command in the background with an
   ampersand (&) may cause the script to hang until ENTER is hit. This
   seems to occur with commands that write to stdout. It can be a major

ls -l &
echo "Done."

bash$ ./
 [bozo@localhost test-scripts]$ total 1
 -rwxr-xr-x    1 bozo     bozo           34 Oct 11 15:09

         As Walter Brameld IV explains it:
         As far as I can tell, such scripts don't actually hang. It jus
         seems that they do because the background command writes text
         the console after the prompt. The user gets the impression tha
         the prompt was never displayed. Here's the sequence of events:
         1. Script launches background command.
         2. Script exits.
         3. Shell displays the prompt.
         4. Background command continues running and writing text to th
         5. Background command finishes.
         6. User doesn't see a prompt at the bottom of the output, thin
     ks script
            is hanging.

   Placing a wait after the background command seems to remedy this.


ls -l &
echo "Done."

bash$ ./
 [bozo@localhost test-scripts]$ total 1
 -rwxr-xr-x    1 bozo     bozo           34 Oct 11 15:09

          Redirecting the output of the command to a file or even to
          /dev/null also takes care of this problem.

          This has a similar effect to Control-Z, but it suspends the
          shell (the shell's parent process should resume it at an
          appropriate time).

          Exit a login shell, optionally specifying an exit status.

          Gives statistics on the system time elapsed when executing
          commands, in the following form:

0m0.020s 0m0.020s

          This capability is of relatively limited value, since it is
          not common to profile and benchmark shell scripts.

          Forcibly terminate a process by sending it an appropriate
          terminate signal (see Example 17-6).

          Example 15-27. A script that kills itself


kill $$  # Script kills its own process here.
         # Recall that "$$" is the script's PID.

echo "This line will not echo."
# Instead, the shell sends a "Terminated" message to stdout.

exit 0   # Normal exit? No!

#  After this script terminates prematurely,
#+ what exit status does it return?
# sh
# echo $?
# 143
# 143 = 128 + 15
#             TERM signal


   kill -l lists all the signals (as does the file
   /usr/include/asm/signal.h). A kill -9 is a sure kill, which will
   usually terminate a process that stubbornly refuses to die with a
   plain kill. Sometimes, a kill -15 works. A zombie process, that is, a
   child process that has terminated, but that the parent process has
   not (yet) killed, cannot be killed by a logged-on user -- you can't
   kill something that is already dead -- but init will generally clean
   it up sooner or later.

          The killall command kills a running process by name, rather
          than by process ID. If there are multiple instances of a
          particular command running, then doing a killall on that
          command will terminate them all.


   This refers to the killall command in /usr/bin, not the killall
   script in /etc/rc.d/init.d.

          The command directive disables aliases and functions for the
          command immediately following it.

bash$ command ls


   This is one of three shell directives that effect script command
   processing. The others are builtin and enable.

          Invoking builtin BUILTIN_COMMAND runs the command
          BUILTIN_COMMAND as a shell builtin, temporarily disabling both
          functions and external system commands with the same name.

          This either enables or disables a shell builtin command. As an
          example, enable -n kill disables the shell builtin kill, so
          that when Bash subsequently encounters kill, it invokes the
          external command /bin/kill.

          The -a option to enable lists all the shell builtins,
          indicating whether or not they are enabled. The -f filename
          option lets enable load a builtin as a shared library (DLL)
          module from a properly compiled object file. [65].

          This is a port to Bash of the ksh autoloader. With autoload in
          place, a function with an autoload declaration will load from
          an external file at its first invocation. [66] This saves
          system resources.

          Note that autoload is not a part of the core Bash
          installation. It needs to be loaded in with enable -f (see

   Table 15-1. Job identifiers
   Notation Meaning
   %N Job number [N]
   %S Invocation (command-line) of job begins with string S
   %?S Invocation (command-line) of job contains within it string S
   %% "current" job (last job stopped in foreground or started in
   %+ "current" job (last job stopped in foreground or started in
   %- Last job
   $! Last background process

Chapter 16. External Filters, Programs and Commands

   Standard UNIX commands make shell scripts more versatile. The power
   of scripts comes from coupling system commands and shell directives
   with simple programming constructs.

16.1. Basic Commands

   The first commands a novice learns

          The basic file "list" command. It is all too easy to
          underestimate the power of this humble command. For example,
          using the -R, recursive option, ls provides a tree-like
          listing of a directory structure. Other useful options are -S,
          sort listing by file size, -t, sort by file modification time,
          -b, show escape characters, and -i, show file inodes (see
          Example 16-4).


   The ls command returns a non-zero exit status when attempting to list
   a non-existent file.
bash$ ls abc
ls: abc: No such file or directory

bash$ echo $?

          Example 16-1. Using ls to create a table of contents for
          burning a CDR disk

# (
# Script to automate burning a CDR.

SPEED=10         # May use higher speed if your hardware supports it.
# DEVICE=/dev/cdrom     For older versions of cdrecord
DEFAULTDIR=/opt  # This is the directory containing the data to be burned.
                 # Make sure it exists.
                 # Exercise: Add a test for this.

# Uses Joerg Schilling's "cdrecord" package:

#  If this script invoked as an ordinary user, may need to suid cdrecord
#+ chmod u+s /usr/bin/cdrecord, as root.
#  Of course, this creates a security hole, though a relatively minor one.

if [ -z "$1" ]
  # Default directory, if not specified on command-line.

# Create a "table of contents" file.
# The "l" option gives a "long" file listing.
# The "R" option makes the listing recursive.
# The "F" option marks the file types (directories get a trailing /).
echo "Creating table of contents."

# Create an image file preparatory to burning it onto the CDR.
echo "Creating ISO9660 file system image ($IMAGEFILE)."

# Burn the CDR.
echo "Burning the disk."
echo "Please be patient, this will take a while."
wodim -v -isosize dev=$DEVICE $IMAGEFILE
#  In newer Linux distros, the "wodim" utility assumes the
#+ functionality of "cdrecord."
echo "Exit code = $exitcode"

exit $exitcode

   cat, tac
          cat, an acronym for concatenate, lists a file to stdout. When
          combined with redirection (> or >>), it is commonly used to
          concatenate files.

# Uses of 'cat'
cat filename                          # Lists the file.

cat file.1 file.2 file.3 > file.123   # Combines three files into one.

          The -n option to cat inserts consecutive numbers before all
          lines of the target file(s). The -b option numbers only the
          non-blank lines. The -v option echoes nonprintable characters,
          using ^ notation. The -s option squeezes multiple consecutive
          blank lines into a single blank line.

          See also Example 16-28 and Example 16-24.


   In a pipe, it may be more efficient to redirect the stdin to a file,
   rather than to cat the file.
cat filename | tr a-z A-Z

tr a-z A-Z < filename   #  Same effect, but starts one less process,
                        #+ and also dispenses with the pipe.

          tac, is the inverse of cat, listing a file backwards from its

          reverses each line of a file, and outputs to stdout. This does
          not have the same effect as tac, as it preserves the order of
          the lines, but flips each one around (mirror image).

bash$ cat file1.txt
This is line 1.
 This is line 2.

bash$ tac file1.txt
This is line 2.
 This is line 1.

bash$ rev file1.txt
.1 enil si sihT
 .2 enil si sihT

          This is the file copy command. cp file1 file2 copies file1 to
          file2, overwriting file2 if it already exists (see Example


   Particularly useful are the -a archive flag (for copying an entire
   directory tree), the -u update flag (which prevents overwriting
   identically-named newer files), and the -r and -R recursive flags.
cp -u source_dir/* dest_dir
#  "Synchronize" dest_dir to source_dir
#+  by copying over all newer and not previously existing files.

          This is the file move command. It is equivalent to a
          combination of cp and rm. It may be used to move multiple
          files to a directory, or even to rename a directory. For some
          examples of using mv in a script, see Example 10-11 and
          Example A-2.


   When used in a non-interactive script, mv takes the -f (force) option
   to bypass user input.
   When a directory is moved to a preexisting directory, it becomes a
   subdirectory of the destination directory.
bash$ mv source_directory target_directory

bash$ ls -lF target_directory
total 1
 drwxrwxr-x    2 bozo  bozo      1024 May 28 19:20 source_directory/

          Delete (remove) a file or files. The -f option forces removal
          of even readonly files, and is useful for bypassing user input
          in a script.


   The rm command will, by itself, fail to remove filenames beginning
   with a dash. Why? Because rm sees a dash-prefixed filename as an
bash$ rm -badname
rm: invalid option -- b
 Try `rm --help' for more information.

   One clever workaround is to precede the filename with a " -- " (the
   end-of-options flag).

bash$ rm -- -badname

          Another method to is to preface the filename to be removed
          with a dot-slash .

bash$ rm ./-badname


   When used with the recursive flag -r, this command removes files all
   the way down the directory tree from the current directory. A
   careless rm -rf * can wipe out a big chunk of a directory structure.

          Remove directory. The directory must be empty of all files --
          including "invisible" dotfiles [67] -- for this command to

          Make directory, creates a new directory. For example, mkdir -p
          project/programs/December creates the named directory. The -p
          option automatically creates any necessary parent directories.

          Changes the attributes of an existing file or directory (see
          Example 15-14).

chmod +x filename
# Makes "filename" executable for all users.

chmod u+s filename
# Sets "suid" bit on "filename" permissions.
# An ordinary user may execute "filename" with same privileges as the file's o
# (This does not apply to shell scripts.)

chmod 644 filename
#  Makes "filename" readable/writable to owner, readable to others
#+ (octal mode).

chmod 444 filename
#  Makes "filename" read-only for all.
#  Modifying the file (for example, with a text editor)
#+ not allowed for a user who does not own the file (except for root),
#+ and even the file owner must force a file-save
#+ if she modifies the file.
#  Same restrictions apply for deleting the file.

chmod 1777 directory-name
#  Gives everyone read, write, and execute permission in directory,
#+ however also sets the "sticky bit".
#  This means that only the owner of the directory,
#+ owner of the file, and, of course, root
#+ can delete any particular file in that directory.

chmod 111 directory-name
#  Gives everyone execute-only permission in a directory.
#  This means that you can execute and READ the files in that directory
#+ (execute permission necessarily includes read permission
#+ because you can't execute a file without being able to read it).
#  But you can't list the files or search for them with the "find" command.
#  These restrictions do not apply to root.

chmod 000 directory-name
#  No permissions at all for that directory.
#  Can't read, write, or execute files in it.
#  Can't even list files in it or "cd" to it.
#  But, you can rename (mv) the directory
#+ or delete it (rmdir) if it is empty.
#  You can even symlink to files in the directory,
#+ but you can't read, write, or execute the symlinks.
#  These restrictions do not apply to root.

          Change file attributes. This is analogous to chmod above, but
          with different options and a different invocation syntax, and
          it works only on ext2/ext3 filesystems.

          One particularly interesting chattr option is i. A chattr +i
          filename marks the file as immutable. The file cannot be
          modified, linked to, or deleted, not even by root. This file
          attribute can be set or removed only by root. In a similar
          fashion, the a option marks the file as append only.

root# chattr +i file1.txt

root# rm file1.txt

rm: remove write-protected regular file `file1.txt'? y
 rm: cannot remove `file1.txt': Operation not permitted

          If a file has the s (secure) attribute set, then when it is
          deleted its block is overwritten with binary zeroes. [68]

          If a file has the u (undelete) attribute set, then when it is
          deleted, its contents can still be retrieved (undeleted).

          If a file has the c (compress) attribute set, then it will
          automatically be compressed on writes to disk, and
          uncompressed on reads.


   The file attributes set with chattr do not show in a file listing (ls

          Creates links to pre-existings files. A "link" is a reference
          to a file, an alternate name for it. The ln command permits
          referencing the linked file by more than one name and is a
          superior alternative to aliasing (see Example 4-6).

          The ln creates only a reference, a pointer to the file only a
          few bytes in size.

          The ln command is most often used with the -s, symbolic or
          "soft" link flag. Advantages of using the -s flag are that it
          permits linking across file systems or to directories.

          The syntax of the command is a bit tricky. For example: ln -s
          oldfile newfile links the previously existing oldfile to the
          newly created link, newfile.


   If a file named newfile has previously existed, an error message will

   Which type of link to use?
   As John Macdonald explains it:
   Both of these [types of links] provide a certain measure of dual
   reference -- if you edit the contents of the file using any name,
   your changes will affect both the original name and either a hard or
   soft new name. The differences between them occurs when you work at a
   higher level. The advantage of a hard link is that the new name is
   totally independent of the old name -- if you remove or rename the
   old name, that does not affect the hard link, which continues to
   point to the data while it would leave a soft link hanging pointing
   to the old name which is no longer there. The advantage of a soft
   link is that it can refer to a different file system (since it is
   just a reference to a file name, not to actual data). And, unlike a
   hard link, a symbolic link can refer to a directory.

          Links give the ability to invoke a script (or any other type
          of executable) with multiple names, and having that script
          behave according to how it was invoked.

          Example 16-2. Hello or Good-bye

# Saying "hello" or "goodbye"
#+          depending on how script is invoked.

# Make a link in current working directory ($PWD) to this script:
#    ln -s goodbye
# Now, try invoking this script both ways:
# ./
# ./goodbye


if [ $0 = "./goodbye" ]
  echo "Good-bye!"
  # Some other goodbye-type commands, as appropriate.

echo "Hello!"
# Some other hello-type commands, as appropriate.

   man, info
          These commands access the manual and information pages on
          system commands and installed utilities. When available, the
          info pages usually contain more detailed descriptions than do
          the man pages.

          There have been various attempts at "automating" the writing
          of man pages. For a script that makes a tentative first step
          in that direction, see Example A-39.

16.2. Complex Commands

   Commands for more advanced users

          -exec COMMAND \;

          Carries out COMMAND on each file that find matches. The
          command sequence terminates with ; (the ";" is escaped to make
          certain the shell passes it to find literally, without
          interpreting it as a special character).

bash$ find ~/ -name '*.txt'

          If COMMAND contains {}, then find substitutes the full path
          name of the selected file for "{}".

find ~/ -name 'core*' -exec rm {} \;
# Removes all core dump files from user's home directory.

find /home/bozo/projects -mtime -1
#                               ^   Note minus sign!
#  Lists all files in /home/bozo/projects directory tree
#+ that were modified within the last day (current_day - 1).
find /home/bozo/projects -mtime 1
#  Same as above, but modified *exactly* one day ago.
#  mtime = last modification time of the target file
#  ctime = last status change time (via 'chmod' or otherwise)
#  atime = last access time

find "$DIR" -type f -atime +5 -exec rm {} \;
#                          ^           ^^
#  Curly brackets are placeholder for the path name output by "find."
#  Deletes all files in "/home/bozo/junk_files"
#+ that have not been accessed in *at least* 5 days (plus sign ... +5).
#  "-type filetype", where
#  f = regular file
#  d = directory
#  l = symbolic link, etc.
#  (The 'find' manpage and info page have complete option listings.)

find /etc -exec grep '[0-9][0-9]*[.][0-9][0-9]*[.][0-9][0-9]*[.][0-9][0-9]*' {
} \;

# Finds all IP addresses ( in /etc directory files.
# There a few extraneous hits. Can they be filtered out?

# Possibly by:

find /etc -type f -exec cat '{}' \; | tr -c '.[:digit:]' '\n' \
| grep '^[^.][^.]*\.[^.][^.]*\.[^.][^.]*\.[^.][^.]*$'
#  [:digit:] is one of the character classes
#+ introduced with the POSIX 1003.2 standard.

# Thanks, StИphane Chazelas.


   The -exec option to find should not be confused with the exec shell

          Example 16-3. Badname, eliminate file names in current
          directory containing bad characters and whitespace.

# Delete filenames in current directory containing bad characters.

for filename in *
  badname=`echo "$filename" | sed -n /[\+\{\;\"\\\=\?~\(\)\<\>\&\*\|\$]/p`
# badname=`echo "$filename" | sed -n '/[+{;"\=?~()<>&*|$]/p'`  also works.
# Deletes files containing these nasties:     + { ; " \ = ? ~ ( ) < > & * | $
  rm $badname 2>/dev/null
#             ^^^^^^^^^^^ Error messages deep-sixed.

# Now, take care of files containing all manner of whitespace.
find . -name "* *" -exec rm -f {} \;
# The path name of the file that _find_ finds replaces the "{}".
# The '\' ensures that the ';' is interpreted literally, as end of command.

exit 0

# Commands below this line will not execute because of _exit_ command.

# An alternative to the above script:
find . -name '*[+{;"\\=?~()<>&*|$ ]*' -maxdepth 0 \
-exec rm -f '{}' \;
#  The "-maxdepth 0" option ensures that _find_ will not search
#+ subdirectories below $PWD.

# (Thanks, S.C.)

          Example 16-4. Deleting a file by its inode number

# Deleting a file by its inode number.

#  This is useful when a filename starts with an illegal character,
#+ such as ? or -.

ARGCOUNT=1                      # Filename arg must be passed to script.

if [ $# -ne "$ARGCOUNT" ]
  echo "Usage: `basename $0` filename"

if [ ! -e "$1" ]
  echo "File \""$1"\" does not exist."

inum=`ls -i | grep "$1" | awk '{print $1}'`
# inum = inode (index node) number of file
# -----------------------------------------------------------------------
# Every file has an inode, a record that holds its physical address info.
# -----------------------------------------------------------------------

echo; echo -n "Are you absolutely sure you want to delete \"$1\" (y/n)? "
# The '-v' option to 'rm' also asks this.
read answer
case "$answer" in
[nN]) echo "Changed your mind, huh?"
      exit $E_CHANGED_MIND
*)    echo "Deleting file \"$1\".";;

find . -inum $inum -exec rm {} \;
#                           ^^
#        Curly brackets are placeholder
#+       for text output by "find."
echo "File "\"$1"\" deleted!"

exit 0

          The find command also works without the -exec option.

#  Find suid root files.
#  A strange suid file might indicate a security hole,
#+ or even a system intrusion.

# Might also try /sbin, /bin, /usr/bin, /usr/local/bin, etc.
permissions="+4000"  # suid root (dangerous!)

for file in $( find "$directory" -perm "$permissions" )
  ls -ltF --author "$file"

          See Example 16-30, Example 3-4, and Example 11-9 for scripts
          using find. Its manpage provides more detail on this complex
          and powerful command.

          A filter for feeding arguments to a command, and also a tool
          for assembling the commands themselves. It breaks a data
          stream into small enough chunks for filters and commands to
          process. Consider it as a powerful replacement for backquotes.
          In situations where command substitution fails with a too many
          arguments error, substituting xargs often works. [69]
          Normally, xargs reads from stdin or from a pipe, but it can
          also be given the output of a file.

          The default command for xargs is echo. This means that input
          piped to xargs may have linefeeds and other whitespace
          characters stripped out.

bash$ ls -l
total 0
 -rw-rw-r--    1 bozo  bozo         0 Jan 29 23:58 file1
 -rw-rw-r--    1 bozo  bozo         0 Jan 29 23:58 file2

bash$ ls -l | xargs
total 0 -rw-rw-r-- 1 bozo bozo 0 Jan 29 23:58 file1 -rw-rw-r-- 1 bozo bozo 0 J

bash$ find ~/mail -type f | xargs grep "Linux"
./misc:User-Agent: slrn/ (Linux)
 ./sent-mail-jul-2005: hosted by the Linux Documentation Project.
 ./sent-mail-jul-2005: (Linux Documentation Project Site, rtf version)
 ./sent-mail-jul-2005: Subject: Criticism of Bozo's Windows/Linux article
 ./sent-mail-jul-2005: while mentioning that the Linux ext2/ext3 filesystem
 . . .

          ls | xargs -p -l gzip gzips every file in current directory,
          one at a time, prompting before each operation.


   Note that xargs processes the arguments passed to it sequentially,
   one at a time.
bash$ find /usr/bin | xargs file
/usr/bin:          directory
 /usr/bin/foomatic-ppd-options:          perl script text executable
 . . .


   An interesting xargs option is -n NN, which limits to NN the number
   of arguments passed.
   ls | xargs -n 8 echo lists the files in the current directory in 8


   Another useful option is -0, in combination with find -print0 or grep
   -lZ. This allows handling arguments containing whitespace or quotes.
   find / -type f -print0 | xargs -0 grep -liwZ GUI | xargs -0 rm -f
   grep -rliwZ GUI / | xargs -0 rm -f
   Either of the above will remove any file containing "GUI". (Thanks,
cat /proc/"$pid"/"$OPTION" | xargs -0 echo
#  Formats output:         ^^^^^^^^^^^^^^^
#  From Han Holl's fixup of ""
#+ script in "/dev and /proc" chapter.


   The -P option to xargs permits running processes in parallel. This
   speeds up execution in a machine with a multicore CPU.

ls *gif | xargs -t -n1 -P2 gif2png
# Converts all the gif images in current directory to png.

# Options:
# =======
# -t    Print command to stderr.
# -n1   At most 1 argument per command line.
# -P2   Run up to 2 processes simultaneously.

# Thank you, Roberto Polli, for the inspiration.

          Example 16-5. Logfile: Using xargs to monitor system log


# Generates a log file in current directory
# from the tail end of /var/log/messages.

# Note: /var/log/messages must be world readable
# if this script invoked by an ordinary user.
#         #root chmod 644 /var/log/messages


( date; uname -a ) >>logfile
# Time and machine name
echo ---------------------------------------------------------- >>logfile
tail -n $LINES /var/log/messages | xargs | fmt -s >>logfile
echo >>logfile
echo >>logfile

exit 0

#  Note:
#  ----
#  As Frank Wang points out,
#+ unmatched quotes (either single or double quotes) in the source file
#+ may give xargs indigestion.
#  He suggests the following substitution for line 15:
#  tail -n $LINES /var/log/messages | tr -d "\"'" | xargs | fmt -s >>logfile

#  Exercise:
#  --------
#  Modify this script to track changes in /var/log/messages at intervals
#+ of 20 minutes.
#  Hint: Use the "watch" command.

          As in find, a curly bracket pair serves as a placeholder for
          replacement text.

          Example 16-6. Copying files in current directory to another


#  Copy (verbose) all files in current directory ($PWD)
#+ to directory specified on command-line.


if [ -z "$1" ]   # Exit if no argument given.
  echo "Usage: `basename $0` directory-to-copy-to"
  exit $E_NOARGS

ls . | xargs -i -t cp ./{} $1
#            ^^ ^^      ^^
#  -t is "verbose" (output command-line to stderr) option.
#  -i is "replace strings" option.
#  {} is a placeholder for output text.
#  This is similar to the use of a curly-bracket pair in "find."
#  List the files in current directory (ls .),
#+ pass the output of "ls" as arguments to "xargs" (-i -t options),
#+ then copy (cp) these arguments ({}) to new directory ($1).
#  The net result is the exact equivalent of
#+   cp * $1
#+ unless any of the filenames has embedded "whitespace" characters.

exit 0

          Example 16-7. Killing processes by name

# Killing processes by name.
# Compare this script with

#  For instance,
#+ try "./ xterm" --
#+ and watch all the xterms on your desktop disappear.

#  Warning:
#  -------
#  This is a fairly dangerous script.
#  Running it carelessly (especially as root)
#+ can cause data loss and other undesirable effects.


if test -z "$1"  # No command-line arg supplied?
  echo "Usage: `basename $0` Process(es)_to_kill"
  exit $E_BADARGS

ps ax | grep "$PROCESS_NAME" | awk '{print $1}' | xargs -i kill {} 2&>/dev/nul
#                                                       ^^      ^^

# ---------------------------------------------------------------
# Notes:
# -i is the "replace strings" option to xargs.
# The curly brackets are the placeholder for the replacement.
# 2&>/dev/null suppresses unwanted error messages.
# Can  grep "$PROCESS_NAME" be replaced by pidof "$PROCESS_NAME"?
# ---------------------------------------------------------------

exit $?

#  The "killall" command has the same effect as this script,
#+ but using it is not quite as educational.

          Example 16-8. Word frequency analysis using xargs

# Crude word frequency analysis on a text file.

# Uses 'xargs' to decompose lines of text into single words.
# Compare this example to the "" script later on.

# Check for input file on command-line.

if [ $# -ne "$ARGS" ]
# Correct number of arguments passed to script?
  echo "Usage: `basename $0` filename"
  exit $E_BADARGS

if [ ! -f "$1" ]       # Check if file exists.
  echo "File \"$1\" does not exist."
  exit $E_NOFILE

cat "$1" | xargs -n1 | \
#  List the file, one word per line.
tr A-Z a-z | \
#  Shift characters to lowercase.
sed -e 's/\.//g'  -e 's/\,//g' -e 's/ /\
/g' | \
#  Filter out periods and commas, and
#+ change space between words to linefeed,
sort | uniq -c | sort -nr
#  Finally remove duplicates, prefix occurrence count
#+ and sort numerically.

#  This does the same job as the "" example,
#+ but a bit more ponderously, and it runs more slowly (why?).

exit $?

          All-purpose expression evaluator: Concatenates and evaluates
          the arguments according to the operation given (arguments must
          be separated by spaces). Operations may be arithmetic,
          comparison, string, or logical.

        expr 3 + 5
                returns 8

        expr 5 % 3
                returns 2

        expr 1 / 0
                returns the error message, expr: division by zero

                Illegal arithmetic operations not allowed.

        expr 5 \* 3
                returns 15

                The multiplication operator must be escaped when used in
                an arithmetic expression with expr.

        y=`expr $y + 1`
                Increment a variable, with the same effect as let y=y+1
                and y=$(($y+1)). This is an example of arithmetic

        z=`expr substr $string $position $length`
                Extract substring of $length characters, starting at

          Example 16-9. Using expr


# Demonstrating some of the uses of 'expr'
# =======================================


# Arithmetic Operators
# ---------- ---------

echo "Arithmetic Operators"
a=`expr 5 + 3`
echo "5 + 3 = $a"

a=`expr $a + 1`
echo "a + 1 = $a"
echo "(incrementing a variable)"

a=`expr 5 % 3`
# modulo
echo "5 mod 3 = $a"


# Logical Operators
# ------- ---------

#  Returns 1 if true, 0 if false,
#+ opposite of normal Bash convention.

echo "Logical Operators"

b=`expr $x = $y`         # Test equality.
echo "b = $b"            # 0  ( $x -ne $y )

b=`expr $a \> 10`
echo 'b=`expr $a \> 10`, therefore...'
echo "If a > 10, b = 0 (false)"
echo "b = $b"            # 0  ( 3 ! -gt 10 )

b=`expr $a \< 10`
echo "If a < 10, b = 1 (true)"
echo "b = $b"            # 1  ( 3 -lt 10 )
# Note escaping of operators.

b=`expr $a \<= 3`
echo "If a <= 3, b = 1 (true)"
echo "b = $b"            # 1  ( 3 -le 3 )
# There is also a "\>=" operator (greater than or equal to).


# String Operators
# ------ ---------

echo "String Operators"

echo "The string being operated upon is \"$a\"."

# length: length of string
b=`expr length $a`
echo "Length of \"$a\" is $b."

# index: position of first character in substring
#        that matches a character in string
b=`expr index $a 23`
echo "Numerical position of first \"2\" in \"$a\" is \"$b\"."

# substr: extract substring, starting position & length specified
b=`expr substr $a 2 6`
echo "Substring of \"$a\", starting at position 2,\
and 6 chars long is \"$b\"."

#  The default behavior of the 'match' operations is to
#+ search for the specified match at the BEGINNING of the string.
#       Using Regular Expressions ...
b=`expr match "$a" '[0-9]*'`               #  Numerical count.
echo Number of digits at the beginning of \"$a\" is $b.
b=`expr match "$a" '\([0-9]*\)'`           #  Note that escaped parentheses
#                   ==      ==             #+ trigger substring match.
echo "The digits at the beginning of \"$a\" are \"$b\"."


exit 0


   The : (null) operator can substitute for match. For example, b=`expr
   $a : [0-9]*` is the exact equivalent of b=`expr match $a [0-9]*` in
   the above listing.

echo "String operations using \"expr \$string : \" construct"
echo "==================================================="


echo "The string being operated upon is \"`expr "$a" : '\(.*\)'`\"."
#     Escaped parentheses grouping operator.            ==  ==

#       ***************************
#+          Escaped parentheses
#+           match a substring
#       ***************************

#  If no escaped parentheses...
#+ then 'expr' converts the string operand to an integer.

echo "Length of \"$a\" is `expr "$a" : '.*'`."   # Length of string

echo "Number of digits at the beginning of \"$a\" is `expr "$a" : '[0-9]*'`."

# ------------------------------------------------------------------------- #


echo "The digits at the beginning of \"$a\" are `expr "$a" : '\([0-9]*\)'`."
#                                                             ==      ==
echo "The first 7 characters of \"$a\" are `expr "$a" : '\(.......\)'`."
#         =====                                          ==       ==
# Again, escaped parentheses force a substring match.
echo "The last 7 characters of \"$a\" are `expr "$a" : '.*\(.......\)'`."
#         ====                  end of string operator  ^^
#  (actually means skip over one or more of any characters until specified
#+  substring)


exit 0

   The above script illustrates how expr uses the escaped parentheses --
   \( ... \) -- grouping operator in tandem with regular expression
   parsing to match a substring. Here is a another example, this time
   from "real life."
# Strip the whitespace from the beginning and end.
LRFDATE=`expr "$LRFDATE" : '[[:space:]]*\(.*\)[[:space:]]*$'`

#  From Peter Knowles' "" script
#+ for converting files to Sony Librie/PRS-50X format.
#  (

   Perl, sed, and awk have far superior string parsing facilities. A
   short sed or awk "subroutine" within a script (see Section 36.2) is
   an attractive alternative to expr.

   See Section 10.1 for more on using expr in string operations.

16.3. Time / Date Commands

   Time/date and timing

          Simply invoked, date prints the date and time to stdout. Where
          this command gets interesting is in its formatting and parsing

          Example 16-10. Using date

# Exercising the 'date' command

echo "The number of days since the year's beginning is `date +%j`."
# Needs a leading '+' to invoke formatting.
# %j gives day of year.

echo "The number of seconds elapsed since 01/01/1970 is `date +%s`."
#  %s yields number of seconds since "UNIX epoch" began,
#+ but how is this useful?

suffix=$(date +%s)  # The "+%s" option to 'date' is GNU-specific.
echo "Temporary filename = $filename"
#  It's great for creating "unique and random" temp filenames,
#+ even better than using $$.

# Read the 'date' man page for more formatting options.

exit 0

          The -u option gives the UTC (Universal Coordinated Time).

bash$ date
Fri Mar 29 21:07:39 MST 2002

bash$ date -u
Sat Mar 30 04:07:42 UTC 2002

          This option facilitates calculating the time between different

          Example 16-11. Date calculations

# Author: Nathan Coulter
# Used in ABS Guide with permission (thanks!).

MPHR=60    # Minutes per hour.
HPD=24     # Hours per day.

diff () {
        printf '%s' $(( $(date -u -d"$TARGET" +%s) -
                        $(date -u -d"$CURRENT" +%s)))
#                       %d = day of month.

CURRENT=$(date -u -d '2007-09-01 17:30:24' '+%F %T.%N %Z')
TARGET=$(date -u -d'2007-12-25 12:30:00' '+%F %T.%N %Z')
# %F = full date, %T = %H:%M:%S, %N = nanoseconds, %Z = time zone.

printf '\nIn 2007, %s ' \
       "$(date -d"$CURRENT +
        $(( $(diff) /$MPHR /$MPHR /$HPD / 2 )) days" '+%d %B')"
#       %B = name of month                ^ halfway
printf 'was halfway between %s ' "$(date -d"$CURRENT" '+%d %B')"
printf 'and %s\n' "$(date -d"$TARGET" '+%d %B')"

printf '\nOn %s at %s, there were\n' \
        $(date -u -d"$CURRENT" +%F) $(date -u -d"$CURRENT" +%T)
DAYS=$(( $(diff) / $MPHR / $MPHR / $HPD ))
CURRENT=$(date -d"$CURRENT +$DAYS days" '+%F %T.%N %Z')
HOURS=$(( $(diff) / $MPHR / $MPHR ))
CURRENT=$(date -d"$CURRENT +$HOURS hours" '+%F %T.%N %Z')
MINUTES=$(( $(diff) / $MPHR ))
CURRENT=$(date -d"$CURRENT +$MINUTES minutes" '+%F %T.%N %Z')
printf '%s days, %s hours, ' "$DAYS" "$HOURS"
printf '%s minutes, and %s seconds ' "$MINUTES" "$(diff)"
printf 'until Christmas Dinner!\n\n'

#  Exercise:
#  --------
#  Rewrite the diff () function to accept passed parameters,
#+ rather than using global variables.

          The date command has quite a number of output options. For
          example %N gives the nanosecond portion of the current time.
          One interesting use for this is to generate random integers.

date +%N | sed -e 's/000$//' -e 's/^0//'
#  Strip off leading and trailing zeroes, if present.
#  Length of generated integer depends on
#+ how many zeroes stripped off.

# 115281032
# 63408725
# 394504284

          There are many more options (try man date).

date +%j
# Echoes day of the year (days elapsed since January 1).

date +%k%M
# Echoes hour and minute in 24-hour format, as a single digit string.

# The 'TZ' parameter permits overriding the default time zone.
date                 # Mon Mar 28 21:42:16 MST 2005
TZ=EST date          # Mon Mar 28 23:42:16 EST 2005
# Thanks, Frank Kannemann and Pete Sjoberg, for the tip.

SixDaysAgo=$(date --date='6 days ago')
OneMonthAgo=$(date --date='1 month ago')  # Four weeks back (not a month!)
OneYearAgo=$(date --date='1 year ago')

          See also Example 3-4 and Example A-43.

          Time zone dump: echoes the time in a specified time zone.

bash$ zdump EST
EST  Tue Sep 18 22:09:22 2001 EST

          Outputs verbose timing statistics for executing a command.

          time ls -l / gives something like this:

real    0m0.067s
 user    0m0.004s
 sys     0m0.005s

          See also the very similar times command in the previous


   As of version 2.0 of Bash, time became a shell reserved word, with
   slightly altered behavior in a pipeline.

          Utility for updating access/modification times of a file to
          current system time or other specified time, but also useful
          for creating a new file. The command touch zzz will create a
          new file of zero length, named zzz, assuming that zzz did not
          previously exist. Time-stamping empty files in this way is
          useful for storing date information, for example in keeping
          track of modification times on a project.


   The touch command is equivalent to : >> newfile or >> newfile (for
   ordinary files).


   Before doing a cp -u (copy/update), use touch to update the time
   stamp of files you don't wish overwritten.
   As an example, if the directory /home/bozo/tax_audit contains the
   files,, and, then doing a touch spreadsheet*.data will
   protect these files from being overwritten by files with the same
   names during a cp -u /home/bozo/financial_info/spreadsheet*data

          The at job control command executes a given set of commands at
          a specified time. Superficially, it resembles cron, however,
          at is chiefly useful for one-time execution of a command set.

          at 2pm January 15 prompts for a set of commands to execute at
          that time. These commands should be shell-script compatible,
          since, for all practical purposes, the user is typing in an
          executable shell script a line at a time. Input terminates
          with a Ctl-D.

          Using either the -f option or input redirection (<), at reads
          a command list from a file. This file is an executable shell
          script, though it should, of course, be non-interactive.
          Particularly clever is including the run-parts command in the
          file to execute a different set of scripts.

bash$ at 2:30 am Friday < at-jobs.list
job 2 at 2000-10-27 02:30

          The batch job control command is similar to at, but it runs a
          command list when the system load drops below .8. Like at, it
          can read commands from a file with the -f option.

   The concept of batch processing dates back to the era of mainframe
   computers. It means running a set of commands without user

          Prints a neatly formatted monthly calendar to stdout. Will do
          current year or a large range of past and future years.

          This is the shell equivalent of a wait loop. It pauses for a
          specified number of seconds, doing nothing. It can be useful
          for timing or in processes running in the background, checking
          for a specific event every so often (polling), as in Example

sleep 3     # Pauses 3 seconds.


   The sleep command defaults to seconds, but minute, hours, or days may
   also be specified.
sleep 3 h   # Pauses 3 hours!


   The watch command may be a better choice than sleep for running
   commands at timed intervals.

          Microsleep (the u may be read as the Greek mu, or micro-
          prefix). This is the same as sleep, above, but "sleeps" in
          microsecond intervals. It can be used for fine-grained timing,
          or for polling an ongoing process at very frequent intervals.

usleep 30     # Pauses 30 microseconds.

          This command is part of the Red Hat initscripts / rc-scripts


   The usleep command does not provide particularly accurate timing, and
   is therefore unsuitable for critical timing loops.

   hwclock, clock
          The hwclock command accesses or adjusts the machine's hardware
          clock. Some options require root privileges. The
          /etc/rc.d/rc.sysinit startup file uses hwclock to set the
          system time from the hardware clock at bootup.

          The clock command is a synonym for hwclock.

16.4. Text Processing Commands

   Commands affecting text and text files

          File sort utility, often used as a filter in a pipe. This
          command sorts a text stream or file forwards or backwards, or
          according to various keys or character positions. Using the -m
          option, it merges presorted input files. The info page lists
          its many capabilities and options. See Example 11-9, Example
          11-10, and Example A-8.

          Topological sort, reading in pairs of whitespace-separated
          strings and sorting according to input patterns. The original
          purpose of tsort was to sort a list of dependencies for an
          obsolete version of the ld linker in an "ancient" version of

          The results of a tsort will usually differ markedly from those
          of the standard sort command, above.

          This filter removes duplicate lines from a sorted file. It is
          often seen in a pipe coupled with sort.

cat list-1 list-2 list-3 | sort | uniq > final.list
# Concatenates the list files,
# sorts them,
# removes duplicate lines,
# and finally writes the result to an output file.

          The useful -c option prefixes each line of the input file with
          its number of occurrences.

bash$ cat testfile
This line occurs only once.
 This line occurs twice.
 This line occurs twice.
 This line occurs three times.
 This line occurs three times.
 This line occurs three times.

bash$ uniq -c testfile
      1 This line occurs only once.
       2 This line occurs twice.
       3 This line occurs three times.

bash$ sort testfile | uniq -c | sort -nr
      3 This line occurs three times.
       2 This line occurs twice.
       1 This line occurs only once.

          The sort INPUTFILE | uniq -c | sort -nr command string
          produces a frequency of occurrence listing on the INPUTFILE
          file (the -nr options to sort cause a reverse numerical sort).
          This template finds use in analysis of log files and
          dictionary lists, and wherever the lexical structure of a
          document needs to be examined.

          Example 16-12. Word Frequency Analysis

# Crude word frequency analysis on a text file.
# This is a more efficient version of the "" script.

# Check for input file on command-line.

if [ $# -ne "$ARGS" ]  # Correct number of arguments passed to script?
  echo "Usage: `basename $0` filename"
  exit $E_BADARGS

if [ ! -f "$1" ]       # Check if file exists.
  echo "File \"$1\" does not exist."
  exit $E_NOFILE

# main ()
sed -e 's/\.//g'  -e 's/\,//g' -e 's/ /\
/g' "$1" | tr 'A-Z' 'a-z' | sort | uniq -c | sort -nr
#                           =========================
#                            Frequency of occurrence

#  Filter out periods and commas, and
#+ change space between words to linefeed,
#+ then shift characters to lowercase, and
#+ finally prefix occurrence count and sort numerically.

#  Arun Giridhar suggests modifying the above to:
#  . . . | sort | uniq -c | sort +1 [-f] | sort +0 -nr
#  This adds a secondary sort key, so instances of
#+ equal occurrence are sorted alphabetically.
#  As he explains it:
#  "This is effectively a radix sort, first on the
#+ least significant column
#+ (word or string, optionally case-insensitive)
#+ and last on the most significant column (frequency)."
#  As Frank Wang explains, the above is equivalent to
#+       . . . | sort | uniq -c | sort +0 -nr
#+ and the following also works:
#+       . . . | sort | uniq -c | sort -k1nr -k

exit 0

# Exercises:
# ---------
# 1) Add 'sed' commands to filter out other punctuation,
#+   such as semicolons.
# 2) Modify the script to also filter out multiple spaces and
#+   other whitespace.

bash$ cat testfile
This line occurs only once.
 This line occurs twice.
 This line occurs twice.
 This line occurs three times.
 This line occurs three times.
 This line occurs three times.

bash$ ./ testfile
      6 this
       6 occurs
       6 line
       3 times
       3 three
       2 twice
       1 only
       1 once

   expand, unexpand
          The expand filter converts tabs to spaces. It is often used in
          a pipe.

          The unexpand filter converts spaces to tabs. This reverses the
          effect of expand.

          A tool for extracting fields from files. It is similar to the
          print $N command set in awk, but more limited. It may be
          simpler to use cut in a script than awk. Particularly
          important are the -d (delimiter) and -f (field specifier)

          Using cut to obtain a listing of the mounted filesystems:

cut -d ' ' -f1,2 /etc/mtab

          Using cut to list the OS and kernel version:

uname -a | cut -d" " -f1,3,11,12

          Using cut to extract message headers from an e-mail folder:

bash$ grep '^Subject:' read-messages | cut -c10-80
Re: Linux suitable for mission-critical apps?
 Spam complaint
 Re: Spam complaint

          Using cut to parse a file:

# List all the users in /etc/passwd.


for user in $(cut -d: -f1 $FILENAME)
  echo $user

# Thanks, Oleg Philon for suggesting this.

          cut -d ' ' -f2,3 filename is equivalent to awk -F'[ ]' '{
          print $2, $3 }' filename


   It is even possible to specify a linefeed as a delimiter. The trick
   is to actually embed a linefeed (RETURN) in the command sequence.
bash$ cut -d'
 ' -f3,7,19 testfile
This is line 3 of testfile.
 This is line 7 of testfile.
 This is line 19 of testfile.

   Thank you, Jaka Kranjc, for pointing this out.

          See also Example 16-48.

          Tool for merging together different files into a single,
          multi-column file. In combination with cut, useful for
          creating system log files.

          Consider this a special-purpose cousin of paste. This powerful
          utility allows merging two files in a meaningful fashion,
          which essentially creates a simple version of a relational

          The join command operates on exactly two files, but pastes
          together only those lines with a common tagged field (usually
          a numerical label), and writes the result to stdout. The files
          to be joined should be sorted according to the tagged field
          for the matchups to work properly.


100 Shoes
200 Laces
300 Socks


100 $40.00
200 $1.00
300 $2.00

bash$ join

 100 Shoes $40.00
 200 Laces $1.00
 300 Socks $2.00


   The tagged field appears only once in the output.

          lists the beginning of a file to stdout. The default is 10
          lines, but a different number can be specified. The command
          has a number of interesting options.

          Example 16-13. Which files are scripts?

# Detects scripts within a directory.

TESTCHARS=2    # Test first 2 characters.
SHABANG='#!'   # Scripts begin with a "sha-bang."

for file in *  # Traverse all the files in current directory.
  if [[ `head -c$TESTCHARS "$file"` = "$SHABANG" ]]
  #      head -c2                      #!
  #  The '-c' option to "head" outputs a specified
  #+ number of characters, rather than lines (the default).
    echo "File \"$file\" is a script."
    echo "File \"$file\" is *not* a script."

exit 0

#  Exercises:
#  ---------
#  1) Modify this script to take as an optional argument
#+    the directory to scan for scripts
#+    (rather than just the current working directory).
#  2) As it stands, this script gives "false positives" for
#+    Perl, awk, and other scripting language scripts.
#     Correct this.

          Example 16-14. Generating 10-digit random numbers

# Outputs a 10-digit random number

# Script by Stephane Chazelas.

head -c4 /dev/urandom | od -N4 -tu4 | sed -ne '1s/.* //p'

# =================================================================== #

# Analysis
# --------

# head:
# -c4 option takes first 4 bytes.

# od:
# -N4 option limits output to 4 bytes.
# -tu4 option selects unsigned decimal format for output.

# sed:
# -n option, in combination with "p" flag to the "s" command,
# outputs only matched lines.

# The author of this script explains the action of 'sed', as follows.

# head -c4 /dev/urandom | od -N4 -tu4 | sed -ne '1s/.* //p'
# ----------------------------------> |

# Assume output up to "sed" --------> |
# is 0000000 1198195154\n

#  sed begins reading characters: 0000000 1198195154\n.
#  Here it finds a newline character,
#+ so it is ready to process the first line (0000000 1198195154).
#  It looks at its <range><action>s. The first and only one is

#   range     action
#   1         s/.* //p

#  The line number is in the range, so it executes the action:
#+ tries to substitute the longest string ending with a space in the line
#  ("0000000 ") with nothing (//), and if it succeeds, prints the result
#  ("p" is a flag to the "s" command here, this is different
#+ from the "p" command).

#  sed is now ready to continue reading its input. (Note that before
#+ continuing, if -n option had not been passed, sed would have printed
#+ the line once again).

#  Now, sed reads the remainder of the characters, and finds the
#+ end of the file.
#  It is now ready to process its 2nd line (which is also numbered '$' as
#+ it's the last one).
#  It sees it is not matched by any <range>, so its job is done.

#  In few word this sed commmand means:
#  "On the first line only, remove any character up to the right-most space,
#+ then print it."

# A better way to do this would have been:
#           sed -e 's/.* //;q'

# Here, two <range><action>s (could have been written
#           sed -e 's/.* //' -e q):

#   range                    action
#   nothing (matches line)   s/.* //
#   nothing (matches line)   q (quit)

#  Here, sed only reads its first line of input.
#  It performs both actions, and prints the line (substituted) before
#+ quitting (because of the "q" action) since the "-n" option is not passed.

# =================================================================== #

# An even simpler altenative to the above one-line script would be:
#           head -c4 /dev/urandom| od -An -tu4


          See also Example 16-39.

          lists the (tail) end of a file to stdout. The default is 10
          lines, but this can be changed with the -n option. Commonly
          used to keep track of changes to a system logfile, using the
          -f option, which outputs lines appended to the file.

          Example 16-15. Using tail to monitor the system log



cat /dev/null > $filename; echo "Creating / cleaning out file."
#  Creates file if it does not already exist,
#+ and truncates it to zero length if it does.
#  : > filename   and   > filename also work.

tail /var/log/messages > $filename
# /var/log/messages must have world read permission for this to work.

echo "$filename contains tail end of system log."

exit 0


   To list a specific line of a text file, pipe the output of head to
   tail -n 1. For example head -n 8 database.txt | tail -n 1 lists the
   8th line of the file database.txt.
   To set a variable to a given block of a text file:
var=$(head -n $m $filename | tail -n $n)

# filename = name of file
# m = from beginning of file, number of lines to end of block
# n = number of lines to set variable to (trim from end of block)


   Newer implementations of tail deprecate the older tail -$LINES
   filename usage. The standard tail -n $LINES filename is correct.

          See also Example 16-5, Example 16-39 and Example 32-6.

          A multi-purpose file search tool that uses Regular
          Expressions. It was originally a command/filter in the
          venerable ed line editor: g/re/p -- global - regular
          expression - print.

          grep pattern [file...]

          Search the target file(s) for occurrences of pattern, where
          pattern may be literal text or a Regular Expression.

bash$ grep '[rst]ystem.$' osinfo.txt
The GPL governs the distribution of the Linux operating system.

          If no target file(s) specified, grep works as a filter on
          stdout, as in a pipe.

bash$ ps ax | grep clock
765 tty1     S      0:00 xclock
 901 pts/1    S      0:00 grep clock

          The -i option causes a case-insensitive search.

          The -w option matches only whole words.

          The -l option lists only the files in which matches were
          found, but not the matching lines.

          The -r (recursive) option searches files in the current
          working directory and all subdirectories below it.

          The -n option lists the matching lines, together with line

bash$ grep -n Linux osinfo.txt
2:This is a file containing information about Linux.
 6:The GPL governs the distribution of the Linux operating system.

          The -v (or --invert-match) option filters out matches.

grep pattern1 *.txt | grep -v pattern2

# Matches all lines in "*.txt" files containing "pattern1",
# but ***not*** "pattern2".

          The -c (--count) option gives a numerical count of matches,
          rather than actually listing the matches.

grep -c txt *.sgml   # (number of occurrences of "txt" in "*.sgml" files)

#   grep -cz .
#            ^ dot
# means count (-c) zero-separated (-z) items matching "."
# that is, non-empty ones (containing at least 1 character).
printf 'a b\nc  d\n\n\n\n\n\000\n\000e\000\000\nf' | grep -cz .     # 3
printf 'a b\nc  d\n\n\n\n\n\000\n\000e\000\000\nf' | grep -cz '$'   # 5
printf 'a b\nc  d\n\n\n\n\n\000\n\000e\000\000\nf' | grep -cz '^'   # 5
printf 'a b\nc  d\n\n\n\n\n\000\n\000e\000\000\nf' | grep -c '$'    # 9
# By default, newline chars (\n) separate items to match.

# Note that the -z option is GNU "grep" specific.

# Thanks, S.C.

          The --color (or --colour) option marks the matching string in
          color (on the console or in an xterm window). Since grep
          prints out each entire line containing the matching pattern,
          this lets you see exactly what is being matched. See also the
          -o option, which shows only the matching portion of the

          Example 16-16. Printing out the From lines in stored e-mail


#  Emulates the useful "from" utility in Solaris, BSD, etc.
#  Echoes the "From" header line in all messages
#+ in your e-mail directory.

MAILDIR=~/mail/*               #  No quoting of variable. Why?
GREP_OPTS="-H -A 5 --color"    #  Show file, plus extra context lines
                               #+ and display "From" in color.
TARGETSTR="^From"              # "From" at beginning of line.

for file in $MAILDIR           #  No quoting of variable.
  grep $GREP_OPTS "$TARGETSTR" "$file"
  #    ^^^^^^^^^^              #  Again, do not quote this variable.

exit $?

#  Might wish to pipe the output of this script to 'more' or
#+ redirect it to a file . . .

          When invoked with more than one target file given, grep
          specifies which file contains matches.

bash$ grep Linux osinfo.txt misc.txt
osinfo.txt:This is a file containing information about Linux.
 osinfo.txt:The GPL governs the distribution of the Linux operating system.
 misc.txt:The Linux operating system is steadily gaining in popularity.


   To force grep to show the filename when searching only one target
   file, simply give /dev/null as the second file.
bash$ grep Linux osinfo.txt /dev/null
osinfo.txt:This is a file containing information about Linux.
 osinfo.txt:The GPL governs the distribution of the Linux operating system.

          If there is a successful match, grep returns an exit status of
          0, which makes it useful in a condition test in a script,
          especially in combination with the -q option to suppress

SUCCESS=0                      # if grep lookup succeeds

grep -q "$word" "$filename"    #  The "-q" option
                               #+ causes nothing to echo to stdout.
if [ $? -eq $SUCCESS ]
# if grep -q "$word" "$filename"   can replace lines 5 - 7.
  echo "$word found in $filename"
  echo "$word not found in $filename"

          Example 32-6 demonstrates how to use grep to search for a word
          pattern in a system logfile.

          Example 16-17. Emulating grep in a script

# Rudimentary reimplementation of grep.


if [ -z "$1" ]    # Check for argument to script.
  echo "Usage: `basename $0` pattern"
  exit $E_BADARGS


for file in *     # Traverse all files in $PWD.
  output=$(sed -n /"$1"/p $file)  # Command substitution.

  if [ ! -z "$output" ]           # What happens if "$output" is not quoted?
    echo -n "$file: "
    echo "$output"
  fi              #  sed -ne "/$1/s|^|${file}: |p"  is equivalent to above.



exit 0

# Exercises:
# ---------
# 1) Add newlines to output, if more than one match in any given file.
# 2) Add features.

          How can grep search for two (or more) separate patterns? What
          if you want grep to display all lines in a file or files that
          contain both "pattern1" and "pattern2"?

          One method is to pipe the result of grep pattern1 to grep

          For example, given the following file:

# Filename: tstfile

This is a sample file.
This is an ordinary text file.
This file does not contain any unusual text.
This file is not unusual.
Here is some text.

          Now, let's search this file for lines containing both "file"
          and "text" . . .

bash$ grep file tstfile
# Filename: tstfile
 This is a sample file.
 This is an ordinary text file.
 This file does not contain any unusual text.
 This file is not unusual.

bash$ grep file tstfile | grep text
This is an ordinary text file.
 This file does not contain any unusual text.

          Now, for an interesting recreational use of grep . . .

          Example 16-18. Crossword puzzle solver

# This is actually a wrapper around a one-liner (line 46).

#  Crossword puzzle and anagramming word game solver.
#  You know *some* of the letters in the word you're looking for,
#+ so you need a list of all valid words
#+ with the known letters in given positions.
#  For example: w...i....n
#               1???5????10
# w in position 1, 3 unknowns, i in the 5th, 4 unknowns, n at the end.
# (See comments at end of script.)

#                    ^^^^^^^^   Looks for word list here.
#  ASCII word list, one word per line.
#  If you happen to need an appropriate list,
#+ download the author's "yawl" word list package.
#  or

if [ -z "$1" ]   #  If no word pattern specified
then             #+ as a command-line argument . . .
  echo           #+ . . . then . . .
  echo "Usage:"  #+ Usage message.
  echo ""$0" \"pattern,\""
  echo "where \"pattern\" is in the form"
  echo "xxx..x.x..."
  echo "The x's represent known letters,"
  echo "and the periods are unknown letters (blanks)."
  echo "Letters and periods can be in any position."
  echo "For example, try:   sh w...i....n"
  exit $E_NOPATT

# ===============================================
# This is where all the work gets done.
grep ^"$1"$ "$DICT"   # Yes, only one line!
#    |    |
# ^ is start-of-word regex anchor.
# $ is end-of-word regex anchor.

#  From _Stupid Grep Tricks_, vol. 1,
#+ a book the ABS Guide author may yet get around
#+ to writing . . . one of these days . . .
# ===============================================

exit $?  # Script terminates here.
#  If there are too many words generated,
#+ redirect the output to a file.

$ sh w...i....n


          egrep -- extended grep -- is the same as grep -E. This uses a
          somewhat different, extended set of Regular Expressions, which
          can make the search a bit more flexible. It also allows the
          boolean | (or) operator.

bash $ egrep 'matches|Matches' file.txt
Line 1 matches.
 Line 3 Matches.
 Line 4 contains matches, but also Matches

          fgrep -- fast grep -- is the same as grep -F. It does a
          literal string search (no Regular Expressions), which
          generally speeds things up a bit.


   On some Linux distros, egrep and fgrep are symbolic links to, or
   aliases for grep, but invoked with the -E and -F options,

          Example 16-19. Looking up definitions in Webster's 1913


#  This script looks up definitions in the 1913 Webster's Dictionary.
#  This Public Domain dictionary is available for download
#+ from various sites, including
#+ Project Gutenberg (
#  Convert it from DOS to UNIX format (with only LF at end of line)
#+ before using it with this script.
#  Store the file in plain, uncompressed ASCII text.
#  Set DEFAULT_DICTFILE variable below to path/filename.

MAXCONTEXTLINES=50                        # Maximum number of lines to show.
                                          # Default dictionary file pathname.
                                          # Change this as necessary.
#  Note:
#  ----
#  This particular edition of the 1913 Webster's
#+ begins each entry with an uppercase letter
#+ (lowercase for the remaining characters).
#  Only the *very first line* of an entry begins this way,
#+ and that's why the search algorithm below works.

if [[ -z $(echo "$1" | sed -n '/^[A-Z]/p') ]]
#  Must at least specify word to look up, and
#+ it must start with an uppercase letter.
  echo "Usage: `basename $0` Word-to-define [dictionary-file]"
  echo "Note: Word to look up must start with capital letter,"
  echo "with the rest of the word in lowercase."
  echo "--------------------------------------------"
  echo "Examples: Abandon, Dictionary, Marking, etc."
  exit $E_BADARGS

if [ -z "$2" ]                            #  May specify different dictionary
                                          #+ as an argument to this script.

# ---------------------------------------------------------
Definition=$(fgrep -A $MAXCONTEXTLINES "$1 \\" "$dictfile")
#                  Definitions in form "Word \..."
#  And, yes, "fgrep" is fast enough
#+ to search even a very large text file.

# Now, snip out just the definition block.

echo "$Definition" |
sed -n '1,/^[A-Z]/p' |
#  Print from first line of output
#+ to the first line of the next entry.
sed '$d' | sed '$d'
#  Delete last two lines of output
#+ (blank line and first line of next entry).
# ---------------------------------------------------------

exit $?

# Exercises:
# ---------
# 1)  Modify the script to accept any type of alphabetic input
#   + (uppercase, lowercase, mixed case), and convert it
#   + to an acceptable format for processing.
# 2)  Convert the script to a GUI application,
#   + using something like 'gdialog' or 'zenity' . . .
#     The script will then no longer take its argument(s)
#   + from the command-line.
# 3)  Modify the script to parse one of the other available
#   + Public Domain Dictionaries, such as the U.S. Census Bureau Gazetteer.


   See also Example A-41 for an example of speedy fgrep lookup on a
   large text file.

          agrep (approximate grep) extends the capabilities of grep to
          approximate matching. The search string may differ by a
          specified number of characters from the resulting matches.
          This utility is not part of the core Linux distribution.


   To search compressed files, use zgrep, zegrep, or zfgrep. These also
   work on non-compressed files, though slower than plain grep, egrep,
   fgrep. They are handy for searching through a mixed set of files,
   some compressed, some not.
   To search bzipped files, use bzgrep.

          The command look works like grep, but does a lookup on a
          "dictionary," a sorted word list. By default, look searches
          for a match in /usr/dict/words, but a different dictionary
          file may be specified.

          Example 16-20. Checking words in a list for validity

# lookup: Does a dictionary lookup on each word in a data file.  # Data file from which to read words to test.


while [ "$word" != end ]  # Last word in data file.
do               # ^^^
  read word      # From data file, because of redirection at end of loop.
  look $word > /dev/null  # Don't want to display lines in dictionary file.
  lookup=$?      # Exit status of 'look' command.

  if [ "$lookup" -eq 0 ]
    echo "\"$word\" is valid."
    echo "\"$word\" is invalid."

done <"$file"    # Redirects stdin to $file, so "reads" come from there.


exit 0

# ----------------------------------------------------------------
# Code below line will not execute because of "exit" command above.

# Stephane Chazelas proposes the following, more concise alternative:

while read word && [[ $word != end ]]
do if look "$word" > /dev/null
   then echo "\"$word\" is valid."
   else echo "\"$word\" is invalid."
done <"$file"

exit 0

   sed, awk
          Scripting languages especially suited for parsing text files
          and command output. May be embedded singly or in combination
          in pipes and shell scripts.

          Non-interactive "stream editor", permits using many ex
          commands in batch mode. It finds many uses in shell scripts.

          Programmable file extractor and formatter, good for
          manipulating and/or extracting fields (columns) in structured
          text files. Its syntax is similar to C.

          wc gives a "word count" on a file or I/O stream:

bash $ wc /usr/share/doc/sed-4.1.2/README
13  70  447 README
[13 lines  70 words  447 characters]

          wc -w gives only the word count.

          wc -l gives only the line count.

          wc -c gives only the byte count.

          wc -m gives only the character count.

          wc -L gives only the length of the longest line.

          Using wc to count how many .txt files are in current working

$ ls *.txt | wc -l
#  Will work as long as none of the "*.txt" files
#+ have a linefeed embedded in their name.

#  Alternative ways of doing this are:
#      find . -maxdepth 1 -name \*.txt -print0 | grep -cz .
#      (shopt -s nullglob; set -- *.txt; echo $#)

#  Thanks, S.C.

          Using wc to total up the size of all the files whose names
          begin with letters in the range d - h

bash$ wc [d-h]* | grep total | awk '{print $3}'

          Using wc to count the instances of the word "Linux" in the
          main source file for this book.

bash$ grep Linux abs-book.sgml | wc -l

          See also Example 16-39 and Example 20-8.

          Certain commands include some of the functionality of wc as

... | grep foo | wc -l
# This frequently used construct can be more concisely rendered.

... | grep -c foo
# Just use the "-c" (or "--count") option of grep.

# Thanks, S.C.

          character translation filter.


   Must use quoting and/or brackets, as appropriate. Quotes prevent the
   shell from reinterpreting the special characters in tr command
   sequences. Brackets should be quoted to prevent expansion by the

          Either tr "A-Z" "*" <filename or tr A-Z \* <filename changes
          all the uppercase letters in filename to asterisks (writes to
          stdout). On some systems this may not work, but tr A-Z '[**]'

          The -d option deletes a range of characters.

echo "abcdef"                 # abcdef
echo "abcdef" | tr -d b-d     # aef

tr -d 0-9 <filename
# Deletes all digits from the file "filename".

          The --squeeze-repeats (or -s) option deletes all but the first
          instance of a string of consecutive characters. This option is
          useful for removing excess whitespace.

bash$ echo "XXXXX" | tr --squeeze-repeats 'X'

          The -c "complement" option inverts the character set to match.
          With this option, tr acts only upon those characters not
          matching the specified set.

bash$ echo "acfdeb123" | tr -c b-d +

          Note that tr recognizes POSIX character classes. [70]

bash$ echo "abcd2ef1" | tr '[:alpha:]' -

          Example 16-21. toupper: Transforms a file to all uppercase.

# Changes a file to all uppercase.


if [ -z "$1" ]  # Standard check for command-line arg.
  echo "Usage: `basename $0` filename"
  exit $E_BADARGS

tr a-z A-Z <"$1"

# Same effect as above, but using POSIX character set notation:
#        tr '[:lower:]' '[:upper:]' <"$1"
# Thanks, S.C.

exit 0

#  Exercise:
#  Rewrite this script to give the option of changing a file
#+ to *either* upper or lowercase.

          Example 16-22. lowercase: Changes all filenames in working
          directory to lowercase.

#  Changes every filename in working directory to all lowercase.
#  Inspired by a script of John Dubois,
#+ which was translated into Bash by Chet Ramey,
#+ and considerably simplified by the author of the ABS Guide.

for filename in *                # Traverse all files in directory.
   fname=`basename $filename`
   n=`echo $fname | tr A-Z a-z`  # Change name to lowercase.
   if [ "$fname" != "$n" ]       # Rename only files not already lowercase.
     mv $fname $n

exit $?

# Code below this line will not execute because of "exit".
# To run it, delete script above line.

# The above script will not work on filenames containing blanks or newlines.
# Stephane Chazelas therefore suggests the following alternative:

for filename in *    # Not necessary to use basename,
                     # since "*" won't return any file containing "/".
do n=`echo "$filename/" | tr '[:upper:]' '[:lower:]'`
#                             POSIX char set notation.
#                    Slash added so that trailing newlines are not
#                    removed by command substitution.
   # Variable substitution:
   n=${n%/}          # Removes trailing slash, added above, from filename.
   [[ $filename == $n ]] || mv "$filename" "$n"
                     # Checks if filename already lowercase.

exit $?

          Example 16-23. du: DOS to UNIX text file conversion.

# DOS to UNIX text file converter.


if [ -z "$1" ]
  echo "Usage: `basename $0` filename-to-convert"


CR='\015'  # Carriage return.
           # 015 is octal ASCII code for CR.
           # Lines in a DOS text file end in CR-LF.
           # Lines in a UNIX text file end in LF only.

tr -d $CR < $1 > $NEWFILENAME
# Delete CR's and write to new file.

echo "Original DOS text file is \"$1\"."
echo "Converted UNIX text file is \"$NEWFILENAME\"."

exit 0

# Exercise:
# --------
# Change the above script to convert from UNIX to DOS.

          Example 16-24. rot13: ultra-weak encryption.

# Classic rot13 algorithm,
#           encryption that might fool a 3-year old.

# Usage: ./ filename
# or     ./ <filename
# or     ./ and supply keyboard input (stdin)

cat "$@" | tr 'a-zA-Z' 'n-za-mN-ZA-M'   # "a" goes to "n", "b" to "o", etc.
#  The 'cat "$@"' construction
#+ permits getting input either from stdin or from files.

exit 0

          Example 16-25. Generating "Crypto-Quote" Puzzles

# Encrypt quotes

#  Will encrypt famous quotes in a simple monoalphabetic substitution.
#  The result is similar to the "Crypto Quote" puzzles
#+ seen in the Op Ed pages of the Sunday paper.

# The "key" is nothing more than a scrambled alphabet.
# Changing the "key" changes the encryption.

# The 'cat "$@"' construction gets input either from stdin or from files.
# If using stdin, terminate input with a Control-D.
# Otherwise, specify filename as command-line parameter.

cat "$@" | tr "a-z" "A-Z" | tr "A-Z" "$key"
#        |  to uppercase  |     encrypt
# Will work on lowercase, uppercase, or mixed-case quotes.
# Passes non-alphabetic characters through unchanged.

# Try this script with something like:
# "Nothing so needs reforming as other people's habits."
# --Mark Twain
# Output is:

# To reverse the encryption:
# cat "$@" | tr "$key" "A-Z"

#  This simple-minded cipher can be broken by an average 12-year old
#+ using only pencil and paper.

exit 0

#  Exercise:
#  --------
#  Modify the script so that it will either encrypt or decrypt,
#+ depending on command-line argument(s).

   tr variants
   The tr utility has two historic variants. The BSD version does not
   use brackets (tr a-z A-Z), but the SysV one does (tr '[a-z]'
   '[A-Z]'). The GNU version of tr resembles the BSD one.

          A filter that wraps lines of input to a specified width. This
          is especially useful with the -s option, which breaks lines at
          word spaces (see Example 16-26 and Example A-1).

          Simple-minded file formatter, used as a filter in a pipe to
          "wrap" long lines of text output.

          Example 16-26. Formatted file listing.


WIDTH=40                    # 40 columns wide.

b=`ls /usr/local/bin`       # Get a file listing...

echo $b | fmt -w $WIDTH

# Could also have been done by
#    echo $b | fold - -s -w $WIDTH

exit 0

          See also Example 16-5.


   A powerful alternative to fmt is Kamil Toman's par utility, available
   from []

          This deceptively named filter removes reverse line feeds from
          an input stream. It also attempts to replace whitespace with
          equivalent tabs. The chief use of col is in filtering the
          output from certain text processing utilities, such as groff
          and tbl.

          Column formatter. This filter transforms list-type text output
          into a "pretty-printed" table by inserting tabs at appropriate

          Example 16-27. Using column to format a directory listing

# A minor modification of the example file in the "column" man page.

; ls -l | sed 1d) | column -t
#         ^^^^^^           ^^

#  The "sed 1d" in the pipe deletes the first line of output,
#+ which would be "total        N",
#+ where "N" is the total number of files found by "ls -l".

# The -t option to "column" pretty-prints a table.

exit 0

          Column removal filter. This removes columns (characters) from
          a file and writes the file, lacking the range of specified
          columns, back to stdout. colrm 2 4 <filename removes the
          second through fourth characters from each line of the text
          file filename.


   If the file contains tabs or nonprintable characters, this may cause
   unpredictable behavior. In such cases, consider using expand and
   unexpand in a pipe preceding colrm.

          Line numbering filter: nl filename lists filename to stdout,
          but inserts consecutive numbers at the beginning of each
          non-blank line. If filename omitted, operates on stdin.

          The output of nl is very similar to cat -b, since, by default
          nl does not list blank lines.

          Example 16-28. nl: A self-numbering script.


# This script echoes itself twice to stdout with its lines numbered.

# 'nl' sees this as line 4 since it does not number blank lines.
# 'cat -n' sees the above line as number 6.

nl `basename $0`

echo; echo  # Now, let's try it with 'cat -n'

cat -n `basename $0`
# The difference is that 'cat -n' numbers the blank lines.
# Note that 'nl -ba' will also do so.

exit 0
# -----------------------------------------------------------------

          Print formatting filter. This will paginate files (or stdout)
          into sections suitable for hard copy printing or viewing on
          screen. Various options permit row and column manipulation,
          joining lines, setting margins, numbering lines, adding page
          headers, and merging files, among other things. The pr command
          combines much of the functionality of nl, paste, fold, column,
          and expand.

          pr -o 5 --width=65 fileZZZ | more gives a nice paginated
          listing to screen of fileZZZ with margins set at 5 and 65.

          A particularly useful option is -d, forcing double-spacing
          (same effect as sed -G).

          The GNU gettext package is a set of utilities for localizing
          and translating the text output of programs into foreign
          languages. While originally intended for C programs, it now
          supports quite a number of programming and scripting

          The gettext program works on shell scripts. See the info page.

          A program for generating binary message catalogs. It is used
          for localization.

          A utility for converting file(s) to a different encoding
          (character set). Its chief use is for localization.

# Convert a string from UTF-8 to UTF-16 and print to the BookList
function write_utf8_string {
    echo -n "$STRING" | iconv -f UTF8 -t UTF16 | \
    cut -b 3- | tr -d \\n >> "$BOOKLIST"

#  From Peter Knowles' "" script
#+ for converting files to Sony Librie/PRS-50X format.
#  (

          Consider this a fancier version of iconv, above. This very
          versatile utility for converting a file to a different
          encoding scheme. Note that recode is not part of the standard
          Linux installation.

   TeX, gs
          TeX and Postscript are text markup languages used for
          preparing copy for printing or formatted video display.

          TeX is Donald Knuth's elaborate typsetting system. It is often
          convenient to write a shell script encapsulating all the
          options and arguments passed to one of these markup languages.

          Ghostscript (gs) is a GPL-ed Postscript interpreter.

          Utility for processing TeX and pdf files. Found in /usr/bin on
          many Linux distros, it is actually a shell wrapper that calls
          Perl to invoke Tex.

texexec --pdfarrange --result=Concatenated.pdf *pdf

#  Concatenates all the pdf files in the current working directory
#+ into the merged file, Concatenated.pdf . . .
#  (The --pdfarrange option repaginates a pdf file. See also --pdfcombine.)
#  The above command-line could be parameterized and put into a shell script.

          Utility for converting plain text file to PostScript

          For example, enscript filename.txt -p produces the
          PostScript output file

   groff, tbl, eqn
          Yet another text markup and display formatting language is
          groff. This is the enhanced GNU version of the venerable UNIX
          roff/troff display and typesetting package. Manpages use

          The tbl table processing utility is considered part of groff,
          as its function is to convert table markup into groff

          The eqn equation processing utility is likewise part of groff,
          and its function is to convert equation markup into groff

          Example 16-29. manview: Viewing formatted manpages

# Formats the source of a man page for viewing.

#  This script is useful when writing man page source.
#  It lets you look at the intermediate results on the fly
#+ while working on it.


if [ -z "$1" ]
  echo "Usage: `basename $0` filename"

# ---------------------------
groff -Tascii -man $1 | less
# From the man page for groff.
# ---------------------------

#  If the man page includes tables and/or equations,
#+ then the above code will barf.
#  The following line can handle such cases.
#   gtbl < "$1" | geqn -Tlatin1 | groff -Tlatin1 -mtty-char -man
#   Thanks, S.C.

exit $?   # See also the "" script.

          See also Example A-39.

   lex, yacc
          The lex lexical analyzer produces programs for pattern
          matching. This has been replaced by the nonproprietary flex on
          Linux systems.

          The yacc utility creates a parser based on a set of
          specifications. This has been replaced by the nonproprietary
          bison on Linux systems.

16.5. File and Archiving Commands


          The standard UNIX archiving utility. [71] Originally a Tape
          ARchiving program, it has developed into a general purpose
          package that can handle all manner of archiving with all types
          of destination devices, ranging from tape drives to regular
          files to even stdout (see Example 3-4). GNU tar has been
          patched to accept various compression filters, for example:
          tar czvf archive_name.tar.gz *, which recursively archives and
          gzips all files in a directory tree except dotfiles in the
          current working directory ($PWD). [72]

          Some useful tar options:

         1. -c create (a new archive)
         2. -x extract (files from existing archive)
         3. --delete delete (files from existing archive)


   This option will not work on magnetic tape devices.
         4. -r append (files to existing archive)
         5. -A append (tar files to existing archive)
         6. -t list (contents of existing archive)
         7. -u update archive
         8. -d compare archive with specified filesystem
         9. --after-date only process files with a date stamp after
            specified date
        10. -z gzip the archive
            (compress or uncompress, depending on whether combined with
            the -c or -x) option
        11. -j bzip2 the archive


   It may be difficult to recover data from a corrupted gzipped tar
   archive. When archiving important files, make multiple backups.

          Shell archiving utility. The text files in a shell archive are
          concatenated without compression, and the resultant archive is
          essentially a shell script, complete with #!/bin/sh header,
          containing all the necessary unarchiving commands, as well as
          the files themselves. Shar archives still show up in Usenet
          newsgroups, but otherwise shar has been replaced by tar/gzip.
          The unshar command unpacks shar archives.

          The mailshar command is a Bash script that uses shar to
          concatenate multiple files into a single one for e-mailing.
          This script supports compression and uuencoding.

          Creation and manipulation utility for archives, mainly used
          for binary object file libraries.

          The Red Hat Package Manager, or rpm utility provides a wrapper
          for source or binary archives. It includes commands for
          installing and checking the integrity of packages, among other

          A simple rpm -i package_name.rpm usually suffices to install a
          package, though there are many more options available.


   rpm -qf identifies which package a file originates from.
bash$ rpm -qf /bin/ls


   rpm -qa gives a complete list of all installed rpm packages on a
   given system. An rpm -qa package_name lists only the package(s)
   corresponding to package_name.
bash$ rpm -qa

bash$ rpm -qa docbook-utils

bash$ rpm -qa docbook | grep docbook

          This specialized archiving copy command (copy input and
          output) is rarely seen any more, having been supplanted by
          tar/gzip. It still has its uses, such as moving a directory
          tree. With an appropriate block size (for copying) specified,
          it can be appreciably faster than tar.

          Example 16-30. Using cpio to move a directory tree


# Copying a directory tree using cpio.

# Advantages of using 'cpio':
#   Speed of copying. It's faster than 'tar' with pipes.
#   Well suited for copying special files (named pipes, etc.)
#+  that 'cp' may choke on.


if [ $# -ne "$ARGS" ]
  echo "Usage: `basename $0` source destination"
  exit $E_BADARGS


find "$source" -depth | cpio -admvp "$destination"
#               ^^^^^         ^^^^^
#  Read the 'find' and 'cpio' info pages to decipher these options.
#  The above works only relative to $PWD (current directory) . . .
#+ full pathnames are specified.

# Exercise:
# --------

#  Add code to check the exit status ($?) of the 'find | cpio' pipe
#+ and output appropriate error messages if anything went wrong.

exit $?

          This command extracts a cpio archive from an rpm one.

          Example 16-31. Unpacking an rpm archive

# Unpack an 'rpm' archive

: ${1?"Usage: `basename $0` target-file"}
# Must specify 'rpm' archive name as an argument.

TEMPFILE=$$.cpio                         #  Tempfile with "unique" name.
                                         #  $$ is process ID of script.

rpm2cpio < $1 > $TEMPFILE                #  Converts rpm archive into
                                         #+ cpio archive.
cpio --make-directories -F $TEMPFILE -i  #  Unpacks cpio archive.
rm -f $TEMPFILE                          #  Deletes cpio archive.

exit 0

#  Exercise:
#  Add check for whether 1) "target-file" exists and
#+                       2) it is an rpm archive.
#  Hint:                    Parse output of 'file' command.

          The pax portable archive exchange toolkit facilitates periodic
          file backups and is designed to be cross-compatible between
          various flavors of UNIX. It was designed to replace tar and

pax -wf daily_backup.pax ~/linux-server/files
#  Creates a tar archive of all files in the target directory.
#  Note that the options to pax must be in the correct order --
#+ pax -fw     has an entirely different effect.

pax -f daily_backup.pax
#  Lists the files in the archive.

pax -rf daily_backup.pax ~/bsd-server/files
#  Restores the backed-up files from the Linux machine
#+ onto a BSD one.

          Note that pax handles many of the standard archiving and
          compression commands.


          The standard GNU/UNIX compression utility, replacing the
          inferior and proprietary compress. The corresponding
          decompression command is gunzip, which is the equivalent of
          gzip -d.


   The -c option sends the output of gzip to stdout. This is useful when
   piping to other commands.

          The zcat filter decompresses a gzipped file to stdout, as
          possible input to a pipe or redirection. This is, in effect, a
          cat command that works on compressed files (including files
          processed with the older compress utility). The zcat command
          is equivalent to gzip -dc.


   On some commercial UNIX systems, zcat is a synonym for uncompress -c,
   and will not work on gzipped files.

          See also Example 7-7.

          An alternate compression utility, usually more efficient (but
          slower) than gzip, especially on large files. The
          corresponding decompression command is bunzip2.

          Similar to the zcat command, bzcat decompresses a bzipped2-ed
          file to stdout.


   Newer versions of tar have been patched with bzip2 support.

   compress, uncompress
          This is an older, proprietary compression utility found in
          commercial UNIX distributions. The more efficient gzip has
          largely replaced it. Linux distributions generally include a
          compress workalike for compatibility, although gunzip can
          unarchive files treated with compress.


   The znew command transforms compressed files into gzipped ones.

          Yet another compression (squeeze) utility, a filter that works
          only on sorted ASCII word lists. It uses the standard
          invocation syntax for a filter, sq < input-file > output-file.
          Fast, but not nearly as efficient as gzip. The corresponding
          uncompression filter is unsq, invoked like sq.


   The output of sq may be piped to gzip for further compression.

   zip, unzip
          Cross-platform file archiving and compression utility
          compatible with DOS pkzip.exe. "Zipped" archives seem to be a
          more common medium of file exchange on the Internet than

   unarc, unarj, unrar
          These Linux utilities permit unpacking archives compressed
          with the DOS arc.exe, arj.exe, and rar.exe programs.

   lzma, unlzma, lzcat
          Highly efficient Lempel-Ziv-Markov compression. The syntax of
          lzma is similar to that of gzip. The
          [] 7-zip Website has more

   File Information

          A utility for identifying file types. The command file
          file-name will return a file specification for file-name, such
          as ascii text or data. It references the magic numbers found
          in /usr/share/magic, /etc/magic, or /usr/lib/magic, depending
          on the Linux/UNIX distribution.

          The -f option causes file to run in batch mode, to read from a
          designated file a list of filenames to analyze. The -z option,
          when used on a compressed target file, forces an attempt to
          analyze the uncompressed file type.

bash$ file test.tar.gz
test.tar.gz: gzip compressed data, deflated,
 last modified: Sun Sep 16 13:34:51 2001, os: Unix

bash file -z test.tar.gz
test.tar.gz: GNU tar archive (gzip compressed data, deflated,
 last modified: Sun Sep 16 13:34:51 2001, os: Unix)

# Find sh and Bash scripts in a given directory:

# Bourne and Bourne-Again shell scripts

file $DIRECTORY/* | fgrep $KEYWORD

# Output:

# /usr/local/bin/burn-cd:          Bourne-Again shell script text executable
# /usr/local/bin/burnit:           Bourne-Again shell script text executable
# /usr/local/bin/      Bourne shell script text executable
# /usr/local/bin/copy-cd:          Bourne-Again shell script text executable
# . . .

          Example 16-32. Stripping comments from C program files

# Strips out the comments (/* COMMENT */) in a C program.


if [ $# -eq "$E_NOARGS" ]
  echo "Usage: `basename $0` C-program-file" >&2 # Error message to stderr.
  exit $E_ARGERROR

# Test for correct file type.
type=`file $1 | awk '{ print $2, $3, $4, $5 }'`
# "file $1" echoes file type . . .
# Then awk removes the first field, the filename . . .
# Then the result is fed into the variable "type."
correct_type="ASCII C program text"

if [ "$type" != "$correct_type" ]
  echo "This script works on C program files only."

# Rather cryptic sed script:
sed '
' $1
# Easy to understand if you take several hours to learn sed fundamentals.

#  Need to add one more line to the sed script to deal with
#+ case where line of code has a comment following it on same line.
#  This is left as a non-trivial exercise.

#  Also, the above code deletes non-comment lines with a "*/" . . .
#+ not a desirable result.

exit 0

# ----------------------------------------------------------------
# Code below this line will not execute because of 'exit 0' above.

# Stephane Chazelas suggests the following alternative:

usage() {
  echo "Usage: `basename $0` C-program-file" >&2
  exit 1

WEIRD=`echo -n -e '\377'`   # or WEIRD=$'\377'
[[ $# -eq 1 ]] || usage
case `file "$1"` in
  *"C program text"*) sed -e "s%/\*%${WEIRD}%g;s%\*/%${WEIRD}%g" "$1" \
     | tr '\377\n' '\n\377' \
     | sed -ne 'p;n' \
     | tr -d '\n' | tr '\377' '\n';;
  *) usage;;

#  This is still fooled by things like:
#  printf("/*");
#  or
#  /*  /* buggy embedded comment */
#  To handle all special cases (comments in strings, comments in string
#+ where there is a \", \\" ...),
#+ the only way is to write a C parser (using lex or yacc perhaps?).

exit 0

          which command gives the full path to "command." This is useful
          for finding out whether a particular command or utility is
          installed on the system.

          $bash which rm


          For an interesting use of this command, see Example 36-14.

          Similar to which, above, whereis command gives the full path
          to "command," but also to its manpage.

          $bash whereis rm

rm: /bin/rm /usr/share/man/man1/rm.1.bz2

          whatis command looks up "command" in the whatis database. This
          is useful for identifying system commands and important
          configuration files. Consider it a simplified man command.

          $bash whatis whatis

whatis               (1)  - search the whatis database for complete words

          Example 16-33. Exploring /usr/X11R6/bin


# What are all those mysterious binaries in /usr/X11R6/bin?

# Try also "/bin", "/usr/bin", "/usr/local/bin", etc.

for file in $DIRECTORY/*
  whatis `basename $file`   # Echoes info about the binary.

exit 0

# You may wish to redirect output of this script, like so:
# ./ >>whatis.db
# or view it a page at a time on stdout,
# ./ | less

          See also Example 11-3.

          Show a detailed directory listing. The effect is similar to ls

          This is one of the GNU fileutils.

bash$ vdir
total 10
 -rw-r--r--    1 bozo  bozo      4034 Jul 18 22:04 data1.xrolo
 -rw-r--r--    1 bozo  bozo      4602 May 25 13:58 data1.xrolo.bak
 -rw-r--r--    1 bozo  bozo       877 Dec 17  2000 employment.xrolo

bash ls -l
total 10
 -rw-r--r--    1 bozo  bozo      4034 Jul 18 22:04 data1.xrolo
 -rw-r--r--    1 bozo  bozo      4602 May 25 13:58 data1.xrolo.bak
 -rw-r--r--    1 bozo  bozo       877 Dec 17  2000 employment.xrolo

   locate, slocate
          The locate command searches for files using a database stored
          for just that purpose. The slocate command is the secure
          version of locate (which may be aliased to slocate).

          $bash locate hickson


   getfacl, setfacl
          These commands retrieve or set the file access control list --
          the owner, group, and file permissions.

bash$ getfacl *
# file: test1.txt
 # owner: bozo
 # group: bozgrp

 # file: test2.txt
 # owner: bozo
 # group: bozgrp

bash$ setfacl -m u:bozo:rw yearly_budget.csv
bash$ getfacl yearly_budget.csv
# file: yearly_budget.csv
 # owner: accountant
 # group: budgetgrp

          Disclose the file that a symbolic link points to.

bash$ readlink /usr/bin/awk

          Use the strings command to find printable strings in a binary
          or data file. It will list sequences of printable characters
          found in the target file. This might be handy for a quick 'n
          dirty examination of a core dump or for looking at an unknown
          graphic image file (strings image-file | more might show
          something like JFIF, which would identify the file as a jpeg
          graphic). In a script, you would probably parse the output of
          strings with grep or sed. See Example 11-7 and Example 11-9.

          Example 16-34. An "improved" strings command

# "word-strings" (enhanced "strings" command)
#  This script filters the output of "strings" by checking it
#+ against a standard word list file.
#  This effectively eliminates gibberish and noise,
#+ and outputs only recognized words.

# ===========================================================
#                 Standard Check for Script Argument(s)

if [ $# -ne $ARGS ]
  echo "Usage: `basename $0` filename"
  exit $E_BADARGS

if [ ! -f "$1" ]                      # Check if file exists.
    echo "File \"$1\" does not exist."
    exit $E_NOFILE
# ===========================================================

MINSTRLEN=3                           #  Minimum string length.
WORDFILE=/usr/share/dict/linux.words  #  Dictionary file.
#  May specify a different word list file
#+ of one-word-per-line format.
#  For example, the "yawl" word-list package,

wlist=`strings "$1" | tr A-Z a-z | tr '[:space:]' Z | \
tr -cs '[:alpha:]' Z | tr -s '\173-\377' Z | tr Z ' '`

# Translate output of 'strings' command with multiple passes of 'tr'.
#  "tr A-Z a-z"  converts to lowercase.
#  "tr '[:space:]'"  converts whitespace characters to Z's.
#  "tr -cs '[:alpha:]' Z"  converts non-alphabetic characters to Z's,
#+ and squeezes multiple consecutive Z's.
#  "tr -s '\173-\377' Z"  converts all characters past 'z' to Z's
#+ and squeezes multiple consecutive Z's,
#+ which gets rid of all the weird characters that the previous
#+ translation failed to deal with.
#  Finally, "tr Z ' '" converts all those Z's to whitespace,
#+ which will be seen as word separators in the loop below.

#  ****************************************************************
#  Note the technique of feeding the output of 'tr' back to itself,
#+ but with different arguments and/or options on each pass.
#  ****************************************************************

for word in $wlist                    # Important:
                                      # $wlist must not be quoted here.
                                      # "$wlist" does not work.
                                      # Why not?

  strlen=${#word}                     # String length.
  if [ "$strlen" -lt "$MINSTRLEN" ]   # Skip over short strings.

  grep -Fw $word "$WORDFILE"          #  Match whole words only.
#      ^^^                            #  "Fixed strings" and
                                      #+ "whole words" options.


exit $?


   diff, patch
          diff: flexible file comparison utility. It compares the target
          files line-by-line sequentially. In some applications, such as
          comparing word dictionaries, it may be helpful to filter the
          files through sort and uniq before piping them to diff. diff
          file-1 file-2 outputs the lines in the files that differ, with
          carets showing which file each particular line belongs to.

          The --side-by-side option to diff outputs each compared file,
          line by line, in separate columns, with non-matching lines
          marked. The -c and -u options likewise make the output of the
          command easier to interpret.

          There are available various fancy frontends for diff, such as
          sdiff, wdiff, xdiff, and mgdiff.


   The diff command returns an exit status of 0 if the compared files
   are identical, and 1 if they differ. This permits use of diff in a
   test construct within a shell script (see below).

          A common use for diff is generating difference files to be
          used with patch The -e option outputs files suitable for ed or
          ex scripts.

          patch: flexible versioning utility. Given a difference file
          generated by diff, patch can upgrade a previous version of a
          package to a newer version. It is much more convenient to
          distribute a relatively small "diff" file than the entire body
          of a newly revised package. Kernel "patches" have become the
          preferred method of distributing the frequent releases of the
          Linux kernel.

patch -p1 <patch-file
# Takes all the changes listed in 'patch-file'
# and applies them to the files referenced therein.
# This upgrades to a newer version of the package.

          Patching the kernel:

cd /usr/src
gzip -cd patchXX.gz | patch -p0
# Upgrading kernel source using 'patch'.
# From the Linux kernel docs "README",
# by anonymous author (Alan Cox?).


   The diff command can also recursively compare directories (for the
   filenames present).
bash$ diff -r ~/notes1 ~/notes2
Only in /home/bozo/notes1: file02
 Only in /home/bozo/notes1: file03
 Only in /home/bozo/notes2: file04


   Use zdiff to compare gzipped files.


   Use diffstat to create a histogram (point-distribution graph) of
   output from diff.

   diff3, merge
          An extended version of diff that compares three files at a
          time. This command returns an exit value of 0 upon successful
          execution, but unfortunately this gives no information about
          the results of the comparison.

bash$ diff3 file-1 file-2 file-3
   This is line 1 of "file-1".
   This is line 1 of "file-2".
   This is line 1 of "file-3"

          The merge (3-way file merge) command is an interesting adjunct
          to diff3. Its syntax is merge Mergefile file1 file2. The
          result is to output to Mergefile the changes that lead from
          file1 to file2. Consider this command a stripped-down version
          of patch.

          Compare and/or edit two files in order to merge them into an
          output file. Because of its interactive nature, this command
          would find little use in a script.

          The cmp command is a simpler version of diff, above. Whereas
          diff reports the differences between two files, cmp merely
          shows at what point they differ.


   Like diff, cmp returns an exit status of 0 if the compared files are
   identical, and 1 if they differ. This permits use in a test construct
   within a shell script.

          Example 16-35. Using cmp to compare two files within a script.


ARGS=2  # Two args to script expected.

if [ $# -ne "$ARGS" ]
  echo "Usage: `basename $0` file1 file2"
  exit $E_BADARGS

if [[ ! -r "$1" || ! -r "$2" ]]
  echo "Both files to be compared must exist and be readable."

cmp $1 $2 &> /dev/null  # /dev/null buries the output of the "cmp" command.
#   cmp -s $1 $2  has same result ("-s" silent flag to "cmp")
#   Thank you  Anders Gustavsson for pointing this out.
# Also works with 'diff', i.e.,   diff $1 $2 &> /dev/null

if [ $? -eq 0 ]         # Test exit status of "cmp" command.
  echo "File \"$1\" is identical to file \"$2\"."
  echo "File \"$1\" differs from file \"$2\"."

exit 0


   Use zcmp on gzipped files.

          Versatile file comparison utility. The files must be sorted
          for this to be useful.

          comm -options first-file second-file

          comm file-1 file-2 outputs three columns:

          + column 1 = lines unique to file-1
          + column 2 = lines unique to file-2
          + column 3 = lines common to both.

          The options allow suppressing output of one or more columns.

          + -1 suppresses column 1
          + -2 suppresses column 2
          + -3 suppresses column 3
          + -12 suppresses both columns 1 and 2, etc.

          This command is useful for comparing "dictionaries" or word
          lists -- sorted text files with one word per line.


          Strips the path information from a file name, printing only
          the file name. The construction basename $0 lets the script
          know its name, that is, the name it was invoked by. This can
          be used for "usage" messages if, for example a script is
          called with missing arguments:

echo "Usage: `basename $0` arg1 arg2 ... argn"

          Strips the basename from a filename, printing only the path


   basename and dirname can operate on any arbitrary string. The
   argument does not need to refer to an existing file, or even be a
   filename for that matter (see Example A-7).

          Example 16-36. basename and dirname



echo "Basename of /home/bozo/daily-journal.txt = `basename $a`"
echo "Dirname of /home/bozo/daily-journal.txt = `dirname $a`"
echo "My own home is `basename ~/`."         # `basename ~` also works.
echo "The home of my home is `dirname ~/`."  # `dirname ~`  also works.

exit 0

   split, csplit
          These are utilities for splitting a file into smaller chunks.
          Their usual use is for splitting up large files in order to
          back them up on floppies or preparatory to e-mailing or
          uploading them.

          The csplit command splits a file according to context, the
          split occuring where patterns are matched.

          Example 16-37. A script that copies itself in sections


#  A script that splits itself into chunks,
#+ then reassembles the chunks into an exact copy
#+ of the original script.

CHUNKSIZE=4    #  Size of first chunk of split files.
OUTPREFIX=xx   #  csplit prefixes, by default,
               #+ files with "xx" ...

csplit "$0" "$CHUNKSIZE"

# Some comment lines for padding . . .
# Line 15
# Line 16
# Line 17
# Line 18
# Line 19
# Line 20

cat "$OUTPREFIX"* > "$0.copy"  # Concatenate the chunks.
rm "$OUTPREFIX"*               # Get rid of the chunks.

exit $?

   Encoding and Encryption

   sum, cksum, md5sum, sha1sum
          These are utilities for generating checksums. A checksum is a
          number [73] mathematically calculated from the contents of a
          file, for the purpose of checking its integrity. A script
          might refer to a list of checksums for security purposes, such
          as ensuring that the contents of key system files have not
          been altered or corrupted. For security applications, use the
          md5sum (message digest 5 checksum) command, or better yet, the
          newer sha1sum (Secure Hash Algorithm). [74]

bash$ cksum /boot/vmlinuz
1670054224 804083 /boot/vmlinuz

bash$ echo -n "Top Secret" | cksum
3391003827 10

bash$ md5sum /boot/vmlinuz
0f43eccea8f09e0a0b2b5cf1dcf333ba  /boot/vmlinuz

bash$ echo -n "Top Secret" | md5sum
8babc97a6f62a4649716f4df8d61728f  -


   The cksum command shows the size, in bytes, of its target, whether
   file or stdout.
   The md5sum and sha1sum commands display a dash when they receive
   their input from stdout.

          Example 16-38. Checking file integrity

# Checking whether files in a given directory
#                    have been tampered with.


# Filename for storing records (database file).

set_up_database ()
  echo ""$directory"" > "$dbfile"
  # Write directory name to first line of file.
  md5sum "$directory"/* >> "$dbfile"
  # Append md5 checksums and filenames.

check_database ()
  local n=0
  local filename
  local checksum

  # ------------------------------------------- #
  #  This file check should be unnecessary,
  #+ but better safe than sorry.

  if [ ! -r "$dbfile" ]
    echo "Unable to read checksum database file!"
    exit $E_BAD_DBFILE
  # ------------------------------------------- #

  while read record[n]

    if [ "$directory_checked" != "$directory" ]
      echo "Directories do not match up!"
      # Tried to use file for a different directory.
      exit $E_DIR_NOMATCH

    if [ "$n" -gt 0 ]   # Not directory name.
      filename[n]=$( echo ${record[$n]} | awk '{ print $2 }' )
      #  md5sum writes records backwards,
      #+ checksum first, then filename.
      checksum[n]=$( md5sum "${filename[n]}" )

      if [ "${record[n]}" = "${checksum[n]}" ]
        echo "${filename[n]} unchanged."

      elif [ "`basename ${filename[n]}`" != "$dbfile" ]
             #  Skip over checksum database file,
             #+ as it will change with each invocation of script.
             #  ---
             #  This unfortunately means that when running
             #+ this script on $PWD, tampering with the
             #+ checksum database file will not be detected.
             #  Exercise: Fix this.
          echo "${filename[n]} : CHECKSUM ERROR!"
        # File has been changed since last checked.


    let "n+=1"
  done <"$dbfile"       # Read from checksum database file.


# =================================================== #
# main ()

if [ -z  "$1" ]
  directory="$PWD"      #  If not specified,
else                    #+ use current working directory.

clear                   # Clear screen.
echo " Running file integrity check on $directory"

# ------------------------------------------------------------------ #
  if [ ! -r "$dbfile" ] # Need to create database file?
    echo "Setting up database file, \""$directory"/"$dbfile"\"."; echo
# ------------------------------------------------------------------ #

check_database          # Do the actual work.


#  You may wish to redirect the stdout of this script to a file,
#+ especially if the directory checked has many files in it.

exit 0

#  For a much more thorough file integrity check,
#+ consider the "Tripwire" package,

          Also see Example A-19, Example 36-14, and Example 10-2 for
          creative uses of the md5sum command.


   There have been reports that the 128-bit md5sum can be cracked, so
   the more secure 160-bit sha1sum is a welcome new addition to the
   checksum toolkit.
bash$ md5sum testfile
e181e2c8720c60522c4c4c981108e367  testfile

bash$ sha1sum testfile
5d7425a9c08a66c3177f1e31286fa40986ffc996  testfile

          Security consultants have demonstrated that even sha1sum can
          be compromised. Fortunately, newer Linux distros include
          longer bit-length sha224sum, sha256sum, sha384sum, and
          sha512sum commands.

          This utility encodes binary files (images, sound files,
          compressed files, etc.) into ASCII characters, making them
          suitable for transmission in the body of an e-mail message or
          in a newsgroup posting. This is especially useful where MIME
          (multimedia) encoding is not available.

          This reverses the encoding, decoding uuencoded files back into
          the original binaries.

          Example 16-39. Uudecoding encoded files

# Uudecodes all uuencoded files in current working directory.

lines=35        # Allow 35 lines for the header (very generous).

for File in *   # Test all the files in $PWD.
  search1=`head -n $lines $File | grep begin | wc -w`
  search2=`tail -n $lines $File | grep end | wc -w`
  #  Uuencoded files have a "begin" near the beginning,
  #+ and an "end" near the end.
  if [ "$search1" -gt 0 ]
    if [ "$search2" -gt 0 ]
      echo "uudecoding - $File -"
      uudecode $File

#  Note that running this script upon itself fools it
#+ into thinking it is a uuencoded file,
#+ because it contains both "begin" and "end".

#  Exercise:
#  --------
#  Modify this script to check each file for a newsgroup header,
#+ and skip to next if not found.

exit 0


   The fold -s command may be useful (possibly in a pipe) to process
   long uudecoded text messages downloaded from Usenet newsgroups.

   mimencode, mmencode
          The mimencode and mmencode commands process multimedia-encoded
          e-mail attachments. Although mail user agents (such as pine or
          kmail) normally handle this automatically, these particular
          utilities permit manipulating such attachments manually from
          the command-line or in batch processing mode by means of a
          shell script.

          At one time, this was the standard UNIX file encryption
          utility. [75] Politically-motivated government regulations
          prohibiting the export of encryption software resulted in the
          disappearance of crypt from much of the UNIX world, and it is
          still missing from most Linux distributions. Fortunately,
          programmers have come up with a number of decent alternatives
          to it, among them the author's very own
          cruft (see Example A-4).

          This is an Open Source implementation of Secure Sockets Layer

# To encrypt a file:
openssl aes-128-ecb -salt -in file.txt -out file.encrypted \
-pass pass:my_password
#          ^^^^^^^^^^^   User-selected password.
#       aes-128-ecb      is the encryption method chosen.

# To decrypt an openssl-encrypted file:
openssl aes-128-ecb -d -salt -in file.encrypted -out file.txt \
-pass pass:my_password
#          ^^^^^^^^^^^   User-selected password.

          Piping openssl to/from tar makes it possible to encrypt an
          entire directory tree.

# To encrypt a directory:


tar czvf - "$sourcedir" |
openssl des3 -salt -out "$encrfile" -pass pass:"$password"
#       ^^^^   Uses des3 encryption.
# Writes encrypted file "encr-dir.tar.gz" in current working directory.

# To decrypt the resulting tarball:
openssl des3 -d -salt -in "$encrfile" -pass pass:"$password" |
tar -xzv
# Decrypts and unpacks into current working directory.

          Of course, openssl has many other uses, such as obtaining
          signed certificates for Web sites. See the info page.

          Securely erase a file by overwriting it multiple times with
          random bit patterns before deleting it. This command has the
          same effect as Example 16-60, but does it in a more thorough
          and elegant manner.

          This is one of the GNU fileutils.


   Advanced forensic technology may still be able to recover the
   contents of a file, even after application of shred.


          Create a temporary file [76] with a "unique" filename. When
          invoked from the command-line without additional arguments, it
          creates a zero-length file in the /tmp directory.

bash$ mktemp

tempfile=`mktemp $PREFIX.XXXXXX`
#                        ^^^^^^ Need at least 6 placeholders
#+                              in the filename template.
#   If no filename template supplied,
#+ "tmp.XXXXXXXXXX" is the default.

echo "tempfile name = $tempfile"
# tempfile name = filename.QA2ZpY
#                 or something similar...

#  Creates a file of that name in the current working directory
#+ with 600 file permissions.
#  A "umask 177" is therefore unnecessary,
#+ but it's good programming practice anyhow.

          Utility for building and compiling binary packages. This can
          also be used for any set of operations triggered by
          incremental changes in source files.

          The make command checks a Makefile, a list of file
          dependencies and operations to be carried out.

          The make utility is, in effect, a powerful scripting language
          similar in many ways to Bash, but with the capability of
          recognizing dependencies. For in-depth coverage of this useful
          tool set, see the GNU software documentation site.

          Special purpose file copying command, similar to cp, but
          capable of setting permissions and attributes of the copied
          files. This command seems tailormade for installing software
          packages, and as such it shows up frequently in Makefiles (in
          the make install : section). It could likewise prove useful in
          installation scripts.

          This utility, written by Benjamin Lin and collaborators,
          converts DOS-formatted text files (lines terminated by CR-LF)
          to UNIX format (lines terminated by LF only), and vice-versa.

          The ptx [targetfile] command outputs a permuted index
          (cross-reference list) of the targetfile. This may be further
          filtered and formatted in a pipe, if necessary.

   more, less
          Pagers that display a text file or stream to stdout, one
          screenful at a time. These may be used to filter the output of
          stdout . . . or of a script.

          An interesting application of more is to "test drive" a
          command sequence, to forestall potentially unpleasant

ls /home/bozo | awk '{print "rm -rf " $1}' | more
#                                            ^^^^

# Testing the effect of the following (disastrous) command-line:
#      ls /home/bozo | awk '{print "rm -rf " $1}' | sh
#      Hand off to the shell to execute . . .       ^^

          The less pager has the interesting property of doing a
          formatted display of man page source. See Example A-39.

16.6. Communications Commands

   Certain of the following commands find use in network data transfer
   and analysis, as well as in chasing spammers.

   Information and Statistics

          Searches for information about an Internet host by name or IP
          address, using DNS.

bash$ host has address

          Displays IP information for a host. With the -h option, ipcalc
          does a reverse DNS lookup, finding the name of the host
          (server) from the IP address.

bash$ ipcalc -h

          Do an Internet "name server lookup" on a host by IP address.
          This is essentially equivalent to ipcalc -h or dig -x . The
          command may be run either interactively or noninteractively,
          i.e., from within a script.

          The nslookup command has allegedly been "deprecated," but it
          is still useful.

bash$ nslookup -sil

 Non-authoritative answer:

          Domain Information Groper. Similar to nslookup, dig does an
          Internet name server lookup on a host. May be run from the
          command-line or from within a script.

          Some interesting options to dig are +time=N for setting a
          query timeout to N seconds, +nofail for continuing to query
          servers until a reply is received, and -x for doing a reverse
          address lookup.

          Compare the output of dig -x with ipcalc -h and nslookup.

bash$ dig -x
;; Got answer:
 ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 11649
 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

 ;         IN      PTR

 ;; AUTHORITY SECTION:    3600    IN      SOA
 2002031705 900 600 86400 3600

 ;; Query time: 537 msec
 ;; WHEN: Wed Jun 26 08:35:24 2002
 ;; MSG SIZE  rcvd: 91

          Example 16-40. Finding out where to report a spammer

# Look up abuse contact to report a spammer.
# Thanks, Michael Zick.

# Check for command-line arg.
if [ $# -ne "$ARGCOUNT" ]
  echo "Usage: `basename $0` domain-name"

dig +short $ -c in -t txt
# Also try:
#     dig +nssearch $1
#     Tries to find "authoritative name servers" and display SOA records.

# The following also works:
#     whois -h $1
#           ^^ ^^^^^^^^^^^^^^^  Specify host.
#     Can even lookup multiple spammers with this, i.e."
#     whois -h $spamdomain1 $spamdomain2 . . .

#  Exercise:
#  --------
#  Expand the functionality of this script
#+ so that it automatically e-mails a notification
#+ to the responsible ISP's contact address(es).
#  Hint: use the "mail" command.

exit $?

#                A known spam domain.

# ""
# ""
# ""

#  For a more elaborate version of this script,
#+ see the SpamViz home page,

          Example 16-41. Analyzing a spam domain

#! /bin/bash
# Identifying spam domains

# $Id: is-spammer, v 1.4 2004/09/01 19:37:52 mszick Exp $
# Above line is RCS ID info.
#  This is a simplified version of the "is_spammer.bash
#+ script in the Contributed Scripts appendix.

# is-spammer <>

# Uses an external program: 'dig'
# Tested with version: 9.2.4rc5

# Uses functions.
# Uses IFS to parse strings by assignment into arrays.
# And even does something useful: checks e-mail blacklists.

# Use the from the text body:
#                       ^^^^^^^^^^^
# Or the from any e-mail address:
# as the only argument to this script.
#(PS: have your Inet connection running)
# So, to invoke this script in the above two instances:

# Whitespace == :Space:Tab:Line Feed:Carriage Return:

# No Whitespace == Line Feed:Carriage Return

# Field separator for dotted decimal ip addresses

# Get the dns text resource record.
# get_txt <error_code> <list_query>
get_txt() {

    # Parse $1 by assignment at the dots.
    local -a dns
    dns=( $1 )
    if [ "${dns[0]}" == '127' ]
        # See if there is a reason.
        echo $(dig +short $2 -t txt)

# Get the dns address resource record.
# chk_adr <rev_dns> <list_server>
chk_adr() {
    local reply
    local server
    local reason

    reply=$( dig +short ${server} )

    # If reply might be an error code . . .
    if [ ${#reply} -gt 6 ]
        reason=$(get_txt ${reply} ${server} )
    echo ${reason:-' not blacklisted.'}

# Need to get the IP address from the name.
echo 'Get address of: '$1
ip_adr=$(dig +short $1)
dns_reply=${ip_adr:-' no answer '}
echo ' Found address: '${dns_reply}

# A valid reply is at least 4 digits plus 3 dots.
if [ ${#ip_adr} -gt 6 ]
    declare query

    # Parse by assignment at the dots.
    declare -a dns
    dns=( ${ip_adr} )

    # Reorder octets into dns query order.

# See: (Conservative, well maintained)
    echo -n ' says: '
    echo $(chk_adr ${rev_dns} '')

# See: (Open mail relays)
    echo -n '  says: '
    echo $(chk_adr ${rev_dns} '')

# See: (You can report spammers here)
    echo -n ' says: '
    echo $(chk_adr ${rev_dns} '')

# # # other blacklist operations # # #

# See:
    echo -n ' says: '
    echo $(chk_adr ${rev_dns} '')

# See: (Various mail relays)
    echo 'Distributed Server Listings'
    echo -n ' says: '
    echo $(chk_adr ${rev_dns} '')

    echo -n ' says: '
    echo $(chk_adr ${rev_dns} '')

    echo -n ' says: '
    echo $(chk_adr ${rev_dns} '')

    echo 'Could not use that address.'

exit 0

# Exercises:
# --------

# 1) Check arguments to script,
#    and exit with appropriate error message if necessary.

# 2) Check if on-line at invocation of script,
#    and exit with appropriate error message if necessary.

# 3) Substitute generic variables for "hard-coded" BHL domains.

# 4) Set a time-out for the script using the "+time=" option
     to the 'dig' command.

          For a much more elaborate version of the above script, see
          Example A-28.

          Trace the route taken by packets sent to a remote host. This
          command works within a LAN, WAN, or over the Internet. The
          remote host may be specified by an IP address. The output of
          this command may be filtered by grep or sed in a pipe.

bash$ traceroute
traceroute to (, 30 hops max, 38 byte packets
 1 (  191.303 ms  179.400 ms  179.767 ms
 2 (  179.536 ms  179.534 ms  169.685 ms
 3 (  189.471 ms  189.556 ms *

          Broadcast an ICMP ECHO_REQUEST packet to another machine,
          either on a local or remote network. This is a diagnostic tool
          for testing network connections, and it should be used with

bash$ ping localhost
PING localhost.localdomain ( from : 56(84) bytes of data.
 64 bytes from localhost.localdomain ( icmp_seq=0 ttl=255 time=709
 64 bytes from localhost.localdomain ( icmp_seq=1 ttl=255 time=286

 --- localhost.localdomain ping statistics ---
 2 packets transmitted, 2 packets received, 0% packet loss
 round-trip min/avg/max/mdev = 0.286/0.497/0.709/0.212 ms

          A successful ping returns an exit status of 0. This can be
          tested for in a script.
# HNAME=$HOST     # Debug: test for localhost.
  count=2  # Send only two pings.

if [[ `ping -c $count "$HNAME"` ]]
  echo ""$HNAME" still up and broadcasting spam your way."
  echo ""$HNAME" seems to be down. Pity."

          Perform a DNS (Domain Name System) lookup. The -h option
          permits specifying which particular whois server to query. See
          Example 4-6 and Example 16-40.

          Retrieve information about users on a network. Optionally,
          this command can display a user's ~/.plan, ~/.project, and
          ~/.forward files, if present.

bash$ finger
Login  Name           Tty      Idle  Login Time   Office     Office Phone
 bozo   Bozo Bozeman   tty1        8  Jun 25 16:59                (:0)
 bozo   Bozo Bozeman   ttyp0          Jun 25 16:59                (:0.0)
 bozo   Bozo Bozeman   ttyp1          Jun 25 17:07                (:0.0)

bash$ finger bozo
Login: bozo                             Name: Bozo Bozeman
 Directory: /home/bozo                   Shell: /bin/bash
 Office: 2355 Clown St., 543-1234
 On since Fri Aug 31 20:13 (MST) on tty1    1 hour 38 minutes idle
 On since Fri Aug 31 20:13 (MST) on pts/0   12 seconds idle
 On since Fri Aug 31 20:13 (MST) on pts/1
 On since Fri Aug 31 20:31 (MST) on pts/2   1 hour 16 minutes idle
 Mail last read Tue Jul  3 10:08 2007 (MST)
 No Plan.

          Out of security considerations, many networks disable finger
          and its associated daemon. [77]

          Change information disclosed by the finger command.

          Verify an Internet e-mail address.

          This command seems to be missing from newer Linux distros.

   Remote Host Access

   sx, rx
          The sx and rx command set serves to transfer files to and from
          a remote host using the xmodem protocol. These are generally
          part of a communications package, such as minicom.

   sz, rz
          The sz and rz command set serves to transfer files to and from
          a remote host using the zmodem protocol. Zmodem has certain
          advantages over xmodem, such as faster transmission rate and
          resumption of interrupted file transfers. Like sx and rx,
          these are generally part of a communications package.

          Utility and protocol for uploading / downloading files to or
          from a remote host. An ftp session can be automated in a
          script (see Example 19-6 and Example A-4).

   uucp, uux, cu
          uucp: UNIX to UNIX copy. This is a communications package for
          transferring files between UNIX servers. A shell script is an
          effective way to handle a uucp command sequence.

          Since the advent of the Internet and e-mail, uucp seems to
          have faded into obscurity, but it still exists and remains
          perfectly workable in situations where an Internet connection
          is not available or appropriate. The advantage of uucp is that
          it is fault-tolerant, so even if there is a service
          interruption the copy operation will resume where it left off
          when the connection is restored.


          uux: UNIX to UNIX execute. Execute a command on a remote
          system. This command is part of the uucp package.


          cu: Call Up a remote system and connect as a simple terminal.
          It is a sort of dumbed-down version of telnet. This command is
          part of the uucp package.

          Utility and protocol for connecting to a remote host.


   The telnet protocol contains security holes and should therefore
   probably be avoided. Its use within a shell script is not

          The wget utility noninteractively retrieves or downloads files
          from a Web or ftp site. It works well in a script.

wget -p
#  The -p or --page-requisite option causes wget to fetch all files
#+ required to display the specified page.

wget -r -O $SAVEFILE
#  The -r option recursively follows and retrieves all links
#+ on the specified site.

wget -c
#  The -c option lets wget resume an interrupted download.
#  This works with ftp servers and many HTTP sites.

          Example 16-42. Getting a stock quote

# Download a stock quote.


if [ -z "$1" ]  # Must specify a stock (symbol) to fetch.
  then echo "Usage: `basename $0` stock-symbol"
  exit $E_NOPARAMS


# Fetches an HTML file, so name it appropriately.
# Yahoo finance board, with stock query suffix.

# -----------------------------------------------------------
wget -O ${stock_symbol}${file_suffix} "${URL}${stock_symbol}"
# -----------------------------------------------------------

# To look up stuff on
# -----------------------------------------------------------
# URL="${query}"
# wget -O "$savefilename" "${URL}"
# -----------------------------------------------------------
# Saves a list of relevant URLs.

exit $?

# Exercises:
# ---------
# 1) Add a test to ensure the user running the script is on-line.
#    (Hint: parse the output of 'ps -ax' for "ppp" or "connect."
# 2) Modify this script to fetch the local weather report,
#+   taking the user's zip code as an argument.

          See also Example A-30 and Example A-31.

          The lynx Web and file browser can be used inside a script
          (with the -dump option) to retrieve a file from a Web or ftp
          site noninteractively.

lynx -dump >$SAVEFILE

          With the -traversal option, lynx starts at the HTTP URL
          specified as an argument, then "crawls" through all links
          located on that particular server. Used together with the
          -crawl option, outputs page text to a log file.

          Remote login, initates a session on a remote host. This
          command has security issues, so use ssh instead.

          Remote shell, executes command(s) on a remote host. This has
          security issues, so use ssh instead.

          Remote copy, copies files between two different networked

          Remote synchronize, updates (synchronizes) files between two
          different networked machines.

bash$ rsync -a ~/sourcedir/*txt /node1/subdirectory/

          Example 16-43. Updating FC4


# Script author: Frank Wang.
# Slight stylistic modifications by ABS Guide author.
# Used in ABS Guide with permission.

#  Download Fedora Core 4 update from mirror site using rsync.
#  Should also work for newer Fedora Cores -- 5, 6, . . .
#  Only download latest package if multiple versions exist,
#+ to save space.

# URL=rsync://
# URL=rsync://

LOG=/tmp/repo-update-$(/bin/date +%Y-%m-%d).txt

E_RETURN=85        # Something unexpected happened.

# General rsync options
# -r: recursive download
# -t: reserve time
# -v: verbose

OPTS="-rtv --delete-excluded --delete-after --partial"

# rsync include pattern
# Leading slash causes absolute path name match.
#   ^                         ^
# Quoting is necessary to prevent globbing.

# rsync exclude pattern
# Temporarily comment out unwanted pkgs using "#" . . .
#  "/4/i386/kernel-xen*"
#  "/4/i386/xen-*"

init () {
    # Let pipe command return possible rsync error, e.g., stalled network.
    set -o pipefail                  # Newly introduced in Bash, version 3.

    TMP=${TMPDIR:-/tmp}/${0##*/}.$$  # Store refined download list.
    trap "{
        rm -f $TMP 2>/dev/null
    }" EXIT                          # Clear temporary file on exit.

check_pid () {
# Check if process exists.
    if [ -s "$PID_FILE" ]; then
        echo "PID file exists. Checking ..."
        PID=$(/bin/egrep -o "^[[:digit:]]+" $PID_FILE)
        if /bin/ps --pid $PID &>/dev/null; then
            echo "Process $PID found. ${0##*/} seems to be running!"
           /usr/bin/logger -t ${0##*/} \
                 "Process $PID found. ${0##*/} seems to be running!"
            exit $E_RETURN
        echo "Process $PID not found. Start new process . . ."

#  Set overall file update range starting from root or $URL,
#+ according to above patterns.
set_range () {
    for p in "${INCLUDE[@]}"; do
        include="$include --include \"$p\""

    for p in "${EXCLUDE[@]}"; do
        exclude="$exclude --exclude \"$p\""

# Retrieve and refine rsync update list.
get_list () {
    echo $$ > $PID_FILE || {
        echo "Can't write to pid file $PID_FILE"
        exit $E_RETURN

    echo -n "Retrieving and refining update list . . ."

    # Retrieve list -- 'eval' is needed to run rsync as a single command.
    # $3 and $4 is the date and time of file creation.
    # $5 is the full package name.
    eval /bin/nice /usr/bin/rsync \
        -r $include $exclude $URL | \
        egrep '^dr.x|^-r' | \
        awk '{print $3, $4, $5}' | \
        sort -k3 | \
        { while read line; do
            # Get seconds since epoch, to filter out obsolete pkgs.
            cur_date=$(date -d "$(echo $line | awk '{print $1, $2}')" +%s)
            #  echo $cur_date

            # Get file name.
            cur_file=$(echo $line | awk '{print $3}')
            #  echo $cur_file

            # Get rpm pkg name from file name, if possible.
            if [[ $cur_file == *rpm ]]; then
                pkg_name=$(echo $cur_file | sed -r -e \
            # echo $pkg_name

            if [ -z "$pkg_name" ]; then   #  If not a rpm file,
                echo $cur_file >> $TMP    #+ then append to download list.
            elif [ "$pkg_name" != "$previous" ]; then   # A new pkg found.
                echo $pre_file >> $TMP                  # Output latest file.
                previous=$pkg_name                      # Save current.
            elif [ "$cur_date" -gt "$pre_date" ]; then
                                                #  If same pkg, but newer,
                pre_date=$cur_date              #+ then update latest pointer.
            echo $pre_file >> $TMP              #  TMP contains ALL
                                                #+ of refined list now.
            # echo "subshell=$BASH_SUBSHELL"

    }       # Bracket required here to let final "echo $pre_file >> $TMP"
            # Remained in the same subshell ( 1 ) with the entire loop.

    RET=$?  # Get return code of the pipe command.

    [ "$RET" -ne 0 ] && {
        echo "List retrieving failed with code $RET"
        exit $E_RETURN

    echo "done"; echo

# Real rsync download part.
get_file () {

    echo "Downloading..."
    /bin/nice /usr/bin/rsync \
        $OPTS \
        --filter "merge,+/ $TMP" \
        --exclude '*'  \
        $URL $DEST     \
        | /usr/bin/tee $LOG


   #  --filter merge,+/ is crucial for the intention.
   #  + modifier means include and / means absolute path.
   #  Then sorted list in $TMP will contain ascending dir name and
   #+ prevent the following --exclude '*' from "shortcutting the circuit."

    echo "Done"

    rm -f $PID_FILE 2>/dev/null

    return $RET

# -------
# Main
# -------

if [ "$RET" -eq 0 ]; then
    /usr/bin/logger -t ${0##*/} "Fedora update mirrored successfully."
    /usr/bin/logger -t ${0##*/} \
    "Fedora update mirrored with failure code: $RET"

exit $RET

          See also Example A-32.


   Using rcp, rsync, and similar utilities with security implications in
   a shell script may not be advisable. Consider, instead, using ssh,
   scp, or an expect script.

          Secure shell, logs onto a remote host and executes commands
          there. This secure replacement for telnet, rlogin, rcp, and
          rsh uses identity authentication and encryption. See its
          manpage for details.

          Example 16-44. Using ssh

# remote.bash: Using ssh.

# This example by Michael Zick.
# Used with permission.

#   Presumptions:
#   ------------
#   fd-2 isn't being captured ( '2>/dev/null' ).
#   ssh/sshd presumes stderr ('2') will display to user.
#   sshd is running on your machine.
#   For any 'standard' distribution, it probably is,
#+  and without any funky ssh-keygen having been done.

# Try ssh to your machine from the command-line:
# $ ssh $HOSTNAME
# Without extra set-up you'll be asked for your password.
#   enter password
#   when done,  $ exit
# Did that work? If so, you're ready for more fun.

# Try ssh to your machine as 'root':
#   $  ssh -l root $HOSTNAME
#   When asked for password, enter root's, not yours.
#          Last login: Tue Aug 10 20:25:49 2004 from localhost.localdomain
#   Enter 'exit' when done.

#  The above gives you an interactive shell.
#  It is possible for sshd to be set up in a 'single command' mode,
#+ but that is beyond the scope of this example.
#  The only thing to note is that the following will work in
#+ 'single command' mode.

# A basic, write stdout (local) command.

ls -l

# Now the same basic command on a remote machine.
# Pass a different 'USERNAME' 'HOSTNAME' if desired:

#  Now excute the above command-line on the remote host,
#+ with all transmissions encrypted.

ssh -l ${USER} ${HOST} " ls -l "

#  The expected result is a listing of your username's home
#+ directory on the remote machine.
#  To see any difference, run this script from somewhere
#+ other than your home directory.

#  In other words, the Bash command is passed as a quoted line
#+ to the remote shell, which executes it on the remote machine.
#  In this case, sshd does  ' bash -c "ls -l" '   on your behalf.

#  For information on topics such as not having to enter a
#+ password/passphrase for every command-line, see
#+    man ssh
#+    man ssh-keygen
#+    man sshd_config.

exit 0


   Within a loop, ssh may cause unexpected behavior. According to a
   d230] Usenet post in the comp.unix shell archives, ssh inherits the
   loop's stdin. To remedy this, pass ssh either the -n or -f option.
   Thanks, Jason Bechtel, for pointing this out.

          Secure copy, similar in function to rcp, copies files between
          two different networked machines, but does so using
          authentication, and with a security level similar to ssh.

   Local Network

          This is a utility for terminal-to-terminal communication. It
          allows sending lines from your terminal (console or xterm) to
          that of another user. The mesg command may, of course, be used
          to disable write access to a terminal

          Since write is interactive, it would not normally find use in
          a script.

          A command-line utility for configuring a network adapter
          (using DHCP). This command is native to Red Hat centric Linux


          Send or read e-mail messages.

          This stripped-down command-line mail client works fine as a
          command embedded in a script.

          Example 16-45. A script that mails itself

# Self-mailing script

adr=${1:-`whoami`}     # Default to current user, if not specified.
#  Typing ''
#+ sends this script to that addressee.
#  Just '' (no argument) sends the script
#+ to the person invoking it, for example, bozo@localhost.localdomain.
#  For more on the ${parameter:-default} construct,
#+ see the "Parameter Substitution" section
#+ of the "Variables Revisited" chapter.

# ============================================================================
  cat $0 | mail -s "Script \"`basename $0`\" has mailed itself to you." "$adr"
# ============================================================================

# --------------------------------------------
#  Greetings from the self-mailing script.
#  A mischievous person has run this script,
#+ which has caused it to mail itself to you.
#  Apparently, some people have nothing better
#+ to do with their time.
# --------------------------------------------

echo "At `date`, script \"`basename $0`\" mailed to "$adr"."

exit 0

#  Note that the "mailx" command (in "send" mode) may be substituted
#+ for "mail" ... but with somewhat different options.

          Similar to the mail command, mailto sends e-mail messages from
          the command-line or in a script. However, mailto also permits
          sending MIME (multimedia) messages.

          Show mail statistics. This command may be invoked only by

root# mailstats
Statistics from Tue Jan  1 20:32:08 2008
  M   msgsfr  bytes_from   msgsto    bytes_to  msgsrej msgsdis msgsqur  Mailer
  4     1682      24118K        0          0K        0       0       0  esmtp
  9      212        640K     1894      25131K        0       0       0  local
  T     1894      24758K     1894      25131K        0       0       0
  C      414                    0

          This utility automatically replies to e-mails that the
          intended recipient is on vacation and temporarily unavailable.
          It runs on a network, in conjunction with sendmail, and is not
          applicable to a dial-up POPmail account.

16.7. Terminal Control Commands

   Command affecting the console or terminal

          Initialize terminal and/or fetch information about it from
          terminfo data. Various options permit certain terminal
          operations: tput clear is the equivalent of clear; tput reset
          is the equivalent of reset.

bash$ tput longname
xterm terminal emulator (X Window System)

          Issuing a tput cup X Y moves the cursor to the (X,Y)
          coordinates in the current terminal. A clear to erase the
          terminal screen would normally precede this.

          Some interesting options to tput are:

          + bold, for high-intensity text
          + smul, to underline text in the terminal
          + smso, to render text in reverse
          + sgr0, to reset the terminal parameters (to normal), without
            clearing the screen

          Example scripts using tput:

    1. Example 36-13
    2. Example 36-11
    3. Example A-44
    4. Example A-42
    5. Example 27-2

   Note that stty offers a more powerful command set for controlling a


   This command prints out extensive information about the current
   terminal. It references the terminfo database.

bash$ infocmp
#       Reconstructed via infocmp from file:
 rxvt|rxvt terminal emulator (X Window System),
         am, bce, eo, km, mir, msgr, xenl, xon,
         colors#8, cols#80, it#8, lines#24, pairs#64,
         bel=^G, blink=\E[5m, bold=\E[1m,
         clear=\E[H\E[2J, cnorm=\E[?25h, cr=^M,


   Reset terminal parameters and clear text screen. As with clear, the
   cursor and prompt reappear in the upper lefthand corner of the


   The clear command simply clears the text screen at the console or in
   an xterm. The prompt and cursor reappear at the upper lefthand corner
   of the screen or xterm window. This command may be used either at the
   command line or in a script. See Example 11-25.


   Echoes commands necessary to set $TERM and $TERMCAP to duplicate the
   size (dimensions) of the current terminal.

bash$ resize
set noglob;
 setenv COLUMNS '80';
 setenv LINES '24';
 unset noglob;


   This utility records (saves to a file) all the user keystrokes at the
   command-line in a console or an xterm window. This, in effect,
   creates a record of a session.

16.8. Math Commands

   "Doing the numbers"

          Decompose an integer into prime factors.

bash$ factor 27417
27417: 3 13 19 37

          Example 16-46. Generating prime numbers


#  Generating prime numbers the quick-and-easy way,
#+ without resorting to fancy algorithms.

CEILING=10000   # 1 to 10000

is_prime ()
  local factors
  factors=( $(factor $1) )  # Load output of `factor` into array.

if [ -z "${factors[2]}" ]
#  Third element of "factors" array:
#+ ${factors[2]} is 2nd factor of argument.
#  If it is blank, then there is no 2nd factor,
#+ and the argument is therefore prime.
  return $PRIME             # 0
  return $E_NOTPRIME        # null

for n in $(seq $CEILING)
  if is_prime $n
    printf %5d $n
  fi   #    ^  Five positions per number suffices.
done   #       For a higher $CEILING, adjust upward, as necessary.



          Bash can't handle floating point calculations, and it lacks
          operators for certain important mathematical functions.
          Fortunately, bc comes to the rescue.

          Not just a versatile, arbitrary precision calculation utility,
          bc offers many of the facilities of a programming language.

          bc has a syntax vaguely resembling C.

          Since it is a fairly well-behaved UNIX utility, and may
          therefore be used in a pipe, bc comes in handy in scripts.

          Here is a simple template for using bc to calculate a script
          variable. This uses command substitution.

              variable=$(echo "OPTIONS; OPERATIONS" | bc)

          Example 16-47. Monthly Payment on a Mortgage

# Calculates monthly payment on a mortgage.

#  This is a modification of code in the
#+ "mcalc" (mortgage calculator) package,
#+ by Jeff Schmidt
#+ and
#+ Mendel Cooper (yours truly, the author of the ABS Guide).
#  [15k]

echo "Given the principal, interest rate, and term of a mortgage,"
echo "calculate the monthly payment."


echo -n "Enter principal (no commas) "
read principal
echo -n "Enter interest rate (percent) "  # If 12%, enter "12", not ".12".
read interest_r
echo -n "Enter term (months) "
read term

 interest_r=$(echo "scale=9; $interest_r/100.0" | bc) # Convert to decimal.
                 #           ^^^^^^^^^^^^^^^^^  Divide by 100.
                 # "scale" determines how many decimal places.

 interest_rate=$(echo "scale=9; $interest_r/12 + 1.0" | bc)

 top=$(echo "scale=9; $principal*$interest_rate^$term" | bc)
          #           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          #           Standard formula for figuring interest.

 echo; echo "Please be patient. This may take a while."

 let "months = $term - 1"
# ====================================================================
 for ((x=$months; x > 0; x--))
   bot=$(echo "scale=9; $interest_rate^$x" | bc)
   bottom=$(echo "scale=9; $bottom+$bot" | bc)
#  bottom = $(($bottom + $bot"))
# ====================================================================

# --------------------------------------------------------------------
#  Rick Boivie pointed out a more efficient implementation
#+ of the above loop, which decreases computation time by 2/3.

# for ((x=1; x <= $months; x++))
# do
#   bottom=$(echo "scale=9; $bottom * $interest_rate + 1" | bc)
# done

#  And then he came up with an even more efficient alternative,
#+ one that cuts down the run time by about 95%!

# bottom=`{
#     echo "scale=9; bottom=$bottom; interest_rate=$interest_rate"
#     for ((x=1; x <= $months; x++))
#     do
#          echo 'bottom = bottom * interest_rate + 1'
#     done
#     echo 'bottom'
#     } | bc`       # Embeds a 'for loop' within command substitution.
# --------------------------------------------------------------------------
#  On the other hand, Frank Wang suggests:
#  bottom=$(echo "scale=9; ($interest_rate^$term-1)/($interest_rate-1)" | bc)

#  Because . . .
#  The algorithm behind the loop
#+ is actually a sum of geometric proportion series.
#  The sum formula is e0(1-q^n)/(1-q),
#+ where e0 is the first element and q=e(n+1)/e(n)
#+ and n is the number of elements.
# --------------------------------------------------------------------------

 # let "payment = $top/$bottom"
 payment=$(echo "scale=2; $top/$bottom" | bc)
 # Use two decimal places for dollars and cents.

 echo "monthly payment = \$$payment"  # Echo a dollar sign in front of amount.

 exit 0

 # Exercises:
 #   1) Filter input to permit commas in principal amount.
 #   2) Filter input to permit interest to be entered as percent or decimal.
 #   3) If you are really ambitious,
 #+     expand this script to print complete amortization tables.

          Example 16-48. Base Conversion

# Shellscript: - print number to different bases (Bourne Shell)
# Author     :  Heiner Steven (
# Date       :  07-03-95
# Category   :  Desktop
# $Id:,v 1.2 2000/02/06 19:55:35 heiner Exp $
# ==> Above line is RCS ID info.
# Description
# Changes
# 21-03-95 stv  fixed error occuring with 0xb as input (0.2)

# ==> Used in ABS Guide with the script author's permission.
# ==> Comments added by ABS Guide author.

PN=`basename "$0"`                             # Program name
VER=`echo '$Revision: 1.2 $' | cut -d' ' -f2`  # ==> VER=1.2

Usage () {
    echo "$PN - print number to different bases, $VER (stv '95)
usage: $PN [number ...]

If no number is given, the numbers are read from standard input.
A number may be
    binary (base 2)             starting with 0b (i.e. 0b1100)
    octal (base 8)              starting with 0  (i.e. 014)
    hexadecimal (base 16)       starting with 0x (i.e. 0xc)
    decimal                     otherwise (i.e. 12)" >&2
    exit $NOARGS
}   # ==> Prints usage message.

Msg () {
    for i   # ==> in [list] missing. Why?
    do echo "$PN: $i" >&2

Fatal () { Msg "$@"; exit 66; }

PrintBases () {
    # Determine base of the number
    for i      # ==> in [list] missing...
    do         # ==> so operates on command-line arg(s).
        case "$i" in
            0b*)                ibase=2;;       # binary
            0x*|[a-f]*|[A-F]*)  ibase=16;;      # hexadecimal
            0*)                 ibase=8;;       # octal
            [1-9]*)             ibase=10;;      # decimal
                Msg "illegal number $i - ignored"

        # Remove prefix, convert hex digits to uppercase (bc needs this).
        number=`echo "$i" | sed -e 's:^0[bBxX]::' | tr '[a-f]' '[A-F]'`
        # ==> Uses ":" as sed separator, rather than "/".

        # Convert number to decimal
        dec=`echo "ibase=$ibase; $number" | bc`  # ==> 'bc' is calculator util
        case "$dec" in
            [0-9]*)     ;;                       # number ok
            *)          continue;;               # error: ignore

        # Print all conversions in one line.
        # ==> 'here document' feeds command list to 'bc'.
        echo `bc <<!
            obase=16; "hex="; $dec
            obase=10; "dec="; $dec
            obase=8;  "oct="; $dec
            obase=2;  "bin="; $dec
    ` | sed -e 's: :    :g'


while [ $# -gt 0 ]
# ==>  Is a "while loop" really necessary here,
# ==>+ since all the cases either break out of the loop
# ==>+ or terminate the script.
# ==> (Above comment by Paulo Marcel Coelho Aragao.)
    case "$1" in
        --)     shift; break;;
        -h)     Usage;;                 # ==> Help message.
        -*)     Usage;;
         *)     break;;                 # First number
    esac   # ==> Error checking for illegal input might be appropriate.

if [ $# -gt 0 ]
    PrintBases "$@"
else                                    # Read from stdin.
    while read line
        PrintBases $line


          An alternate method of invoking bc involves using a here
          document embedded within a command substitution block. This is
          especially appropriate when a script needs to pass a list of
          options and commands to bc.

variable=`bc << LIMIT_STRING


variable=$(bc << LIMIT_STRING

          Example 16-49. Invoking bc using a here document

# Invoking 'bc' using command substitution
# in combination with a 'here document'.

var1=`bc << EOF
18.33 * 19.78
echo $var1       # 362.56

#  $( ... ) notation also works.

var2=$(bc << EOF
scale = 4
a = ( $v1 + $v2 )
b = ( $v3 * $v4 )
a * b + 15.35
echo $var2       # 593487.8452

var3=$(bc -l << EOF
scale = 9
s ( 1.7 )
# Returns the sine of 1.7 radians.
# The "-l" option calls the 'bc' math library.
echo $var3       # .991664810

# Now, try it in a function...
hypotenuse ()    # Calculate hypotenuse of a right triangle.
{                # c = sqrt( a^2 + b^2 )
hyp=$(bc -l << EOF
scale = 9
sqrt ( $1 * $1 + $2 * $2 )
# Can't directly return floating point values from a Bash function.
# But, can echo-and-capture:
echo "$hyp"

hyp=$(hypotenuse 3.68 7.31)
echo "hypotenuse = $hyp"    # 8.184039344

exit 0

          Example 16-50. Calculating PI

# Approximating PI by firing cannonballs.

# Author: Mendel Cooper
# License: Public Domain
# Version 2.2, reldate 13oct08.

# This is a very simple instance of a "Monte Carlo" simulation:
#+ a mathematical model of a real-life event,
#+ using pseudorandom numbers to emulate random chance.

#  Consider a perfectly square plot of land, 10000 units on a side.
#  This land has a perfectly circular lake in its center,
#+ with a diameter of 10000 units.
#  The plot is actually mostly water, except for land in the four corners.
#  (Think of it as a square with an inscribed circle.)
#  We will fire iron cannonballs from an old-style cannon
#+ at the square.
#  All the shots impact somewhere on the square,
#+ either in the lake or on the dry corners.
#  Since the lake takes up most of the area,
#+ most of the shots will SPLASH! into the water.
#  Just a few shots will THUD! into solid ground
#+ in the four corners of the square.
#  If we take enough random, unaimed shots at the square,
#+ Then the ratio of SPLASHES to total shots will approximate
#+ the value of PI/4.
#  The reason for this is that the cannon is actually shooting
#+ only at the upper right-hand quadrant of the square,
#+ i.e., Quadrant I of the Cartesian coordinate plane.
#  (The previous explanation was a simplification.)
#  Theoretically, the more shots taken, the better the fit.
#  However, a shell script, as opposed to a compiled language
#+ with floating-point math built in, requires a few compromises.
#  This tends to lower the accuracy of the simulation.

DIMENSION=10000  # Length of each side of the plot.
                 # Also sets ceiling for random integers generated.

MAXSHOTS=1000    # Fire this many shots.
                 # 10000 or more would be better, but would take too long.
PMULTIPLIER=4.0  # Scaling factor to approximate PI.

declare -r M_PI=3.141592654
                 # Actual 9-place value of PI, for comparison purposes.

get_random ()
SEED=$(head -n 1 /dev/urandom | od -N 1 | awk '{ print $2 }')
RANDOM=$SEED                                  #  From ""
                                              #+ example script.
let "rnum = $RANDOM % $DIMENSION"             #  Range less than 10000.
echo $rnum

distance=        # Declare global variable.
hypotenuse ()    # Calculate hypotenuse of a right triangle.
{                # From "" example.
distance=$(bc -l << EOF
scale = 0
sqrt ( $1 * $1 + $2 * $2 )
#  Setting "scale" to zero rounds down result to integer value,
#+ a necessary compromise in this script.
#  This decreases the accuracy of the simulation.

# ==========================================================
# main() {
# "Main" code block, mimmicking a C-language main() function.

# Initialize variables.

while [ "$shots" -lt  "$MAXSHOTS" ]           # Main loop.

  xCoord=$(get_random)                        # Get random X and Y coords.
  hypotenuse $xCoord $yCoord                  #  Hypotenuse of
                                              #+ right-triangle = distance.

  printf "#%4d   " $shots
  printf "Xc = %4d  " $xCoord
  printf "Yc = %4d  " $yCoord
  printf "Distance = %5d  " $distance         #   Distance from
                                              #+  center of lake
                                              #+  -- the "origin" --
                                              #+  coordinate (0,0).

  if [ "$distance" -le "$DIMENSION" ]
    echo -n "SPLASH!  "
    echo -n "THUD!    "

  Pi=$(echo "scale=9; $PMULTIPLIER*$splashes/$shots" | bc)
  # Multiply ratio by 4.0.
  echo -n "PI ~ $Pi"


echo "After $shots shots, PI looks like approximately   $Pi"
#  Tends to run a bit high,
#+ probably due to round-off error and imperfect randomness of $RANDOM.
#  But still usually within plus-or-minus 5% . . .
#+ a pretty good rough approximation.
error=$(echo "scale=9; $Pi - $M_PI" | bc)
pct_error=$(echo "scale=2; 100.0 * $error / $M_PI" | bc)
echo -n "Deviation from mathematical value of PI =        $error"
echo " ($pct_error% error)"

# End of "main" code block.
# }
# ==========================================================


#  One might well wonder whether a shell script is appropriate for
#+ an application as complex and computation-intensive as a simulation.
#  There are at least two justifications.
#  1) As a proof of concept: to show it can be done.
#  2) To prototype and test the algorithms before rewriting
#+    it in a compiled high-level language.

          See also Example A-37.

          The dc (desk calculator) utility is stack-oriented and uses
          RPN ("Reverse Polish Notation"). Like bc, it has much of the
          power of a programming language.

echo "7 8 * p" | dc     # 56
#  Pushes 7, then 8 onto the stack,
#+ multiplies ("*" operator), then prints the result ("p" operator).

          Most persons avoid dc, because of its non-intuitive input and
          rather cryptic operators. Yet, it has its uses.

          Example 16-51. Converting a decimal number to hexadecimal

# Convert a decimal number to hexadecimal.

E_NOARGS=85 # Command-line arg missing.
BASE=16     # Hexadecimal.

if [ -z "$1" ]
then        # Need a command-line argument.
  echo "Usage: $0 number"
  exit $E_NOARGS
fi          # Exercise: add argument validity checking.

hexcvt ()
if [ -z "$1" ]
  echo 0
  return    # "Return" 0 if no arg passed to function.

echo ""$1" "$BASE" o p" | dc
#                  o    sets radix (numerical base) of output.
#                    p  prints the top of stack.
# For other options: 'man dc' ...

hexcvt "$1"


          Studying the info page for dc is a painful path to
          understanding its intricacies. There seems to be a small,
          select group of dc wizards who delight in showing off their
          mastery of this powerful, but arcane utility.

bash$ echo "16i[q]sa[ln0=aln100%Pln100/snlbx]sbA0D68736142snlbxq" | dc

dc <<< 10k5v1+2/p # 1.6180339887
#  ^^^            Feed operations to dc using a Here String.
#      ^^^        Pushes 10 and sets that as the precision (10k).
#         ^^      Pushes 5 and takes its square root (5v, v = square root).
#           ^^    Pushes 1 and adds it to the running total (1+).
#             ^^  Pushes 2 and divides the running total by that (2/).
#               ^ Pops and prints the result (p)
#  The result is  1.6180339887 ...
#  ... which happens to be the Pythagorean Golden Ratio, to 10 places.

          Example 16-52. Factoring

# Factor a number

MIN=2       # Will not work for number smaller than this.

if [ -z $1 ]
  echo "Usage: $0 number"
  exit $E_NOARGS

if [ "$1" -lt "$MIN" ]
  echo "Number to factor must be $MIN or greater."
  exit $E_TOOSMALL

# Exercise: Add type checking (to reject non-integer arg).

echo "Factors of $1:"
# -------------------------------------------------------
echo  "$1[p]s2[lip/dli%0=1dvsr]s12sid2%0=13sidvsr[dli%0=\
1lrli2+dsi!>.]ds.xd1<2" | dc
# -------------------------------------------------------
#  Above code written by Michel Charpentier <>
#  (as a one-liner, here broken into two lines for display purposes).
#  Used in ABS Guide with permission (thanks!).


 # $ sh 270138
 # 2
 # 3
 # 11
 # 4093

          Yet another way of doing floating point math in a script is
          using awk's built-in math functions in a shell wrapper.

          Example 16-53. Calculating the hypotenuse of a triangle

# Returns the "hypotenuse" of a right triangle.
#                (square root of sum of squares of the "legs")

ARGS=2                # Script needs sides of triangle passed.
E_BADARGS=85          # Wrong number of arguments.

if [ $# -ne "$ARGS" ] # Test number of arguments to script.
  echo "Usage: `basename $0` side_1 side_2"
  exit $E_BADARGS

AWKSCRIPT=' { printf( "%3.7f\n", sqrt($1*$1 + $2*$2) ) } '
#             command(s) / parameters passed to awk

# Now, pipe the parameters to awk.
    echo -n "Hypotenuse of $1 and $2 = "
    echo $1 $2 | awk "$AWKSCRIPT"
#   ^^^^^^^^^^^^
# An echo-and-pipe is an easy way of passing shell parameters to awk.


# Exercise: Rewrite this script using 'bc' rather than awk.
#           Which method is more intuitive?

16.9. Miscellaneous Commands

   Command that fit in no special category

   jot, seq
          These utilities emit a sequence of integers, with a
          user-selectable increment.

          The default separator character between each integer is a
          newline, but this can be changed with the -s option.

bash$ seq 5

bash$ seq -s : 5

          Both jot and seq come in handy in a for loop.

          Example 16-54. Using seq to generate loop arguments

# Using "seq"


for a in `seq 80`  # or   for a in $( seq 80 )
# Same as   for a in 1 2 3 4 5 ... 80   (saves much typing!).
# May also use 'jot' (if present on system).
  echo -n "$a "
done      # 1 2 3 4 5 ... 80
# Example of using the output of a command to generate
# the [list] in a "for" loop.

echo; echo

COUNT=80  # Yes, 'seq' also accepts a replaceable parameter.

for a in `seq $COUNT`  # or   for a in $( seq $COUNT )
  echo -n "$a "
done      # 1 2 3 4 5 ... 80

echo; echo


for a in `seq $BEGIN $END`
#  Giving "seq" two arguments starts the count at the first one,
#+ and continues until it reaches the second.
  echo -n "$a "
done      # 75 76 77 78 79 80

echo; echo


for a in `seq $BEGIN $INTERVAL $END`
#  Giving "seq" three arguments starts the count at the first one,
#+ uses the second for a step interval,
#+ and continues until it reaches the third.
  echo -n "$a "
done      # 45 50 55 60 65 70 75 80

echo; echo

exit 0

          A simpler example:

#  Create a set of 10 files,
#+ named file.1, file.2 . . . file.10.

for filename in `seq $COUNT`
  touch $PREFIX.$filename
  #  Or, can do other operations,
  #+ such as rm, grep, etc.

          Example 16-55. Letter Count"

# Counting letter occurrences in a text file.
# Written by Stefano Palmeri.
# Used in ABS Guide with permission.
# Slightly modified by document author.

MINARGS=2          # Script requires at least two arguments.

let LETTERS=$#-1   # How many letters specified (as command-line args).
                   # (Subtract 1 from number of command-line args.)

           echo Usage: `basename $0` file letters
           echo Note: `basename $0` arguments are case sensitive.
           echo Example: `basename $0` foobar.txt G n U L i N U x.

# Checks number of arguments.
if [ $# -lt $MINARGS ]; then
   echo "Not enough arguments."
   exit $E_BADARGS

# Checks if file exists.
if [ ! -f $FILE ]; then
    echo "File \"$FILE\" does not exist."
    exit $E_BADARGS

# Counts letter occurrences .
for n in `seq $LETTERS`; do
      if [[ `echo -n "$1" | wc -c` -eq 1 ]]; then             #  Checks arg.
             echo "$1" -\> `cat $FILE | tr -cd  "$1" | wc -c` #  Counting.
             echo "$1 is not a  single char."

exit $?

#  This script has exactly the same functionality as,
#+ but executes faster.
#  Why?


   Somewhat more capable than seq, jot is a classic UNIX utility that is
   not normally included in a standard Linux distro. However, the source
   rpm is available for download from the
   thena-jot-9.0-3.src.rpm] MIT repository.
   Unlike seq, jot can generate a sequence of random numbers, using the
   -r option.
bash$ jot -r 3 999

          The getopt command parses command-line options preceded by a
          dash. This external command corresponds to the getopts Bash
          builtin. Using getopt permits handling long options by means
          of the -l flag, and this also allows parameter reshuffling.

          Example 16-56. Using getopt to parse command-line options

# Using getopt

# Try the following when invoking this script:
#   sh -a
#   sh -abc
#   sh -a -b -c
#   sh -d
#   sh -dXYZ
#   sh -d XYZ
#   sh -abcd
#   sh -abcdZ
#   sh -z
#   sh a
# Explain the results of each of the above.


if [ "$#" -eq 0 ]
then   # Script needs at least one command-line argument.
  echo "Usage $0 -[options a,b,c]"
  exit $E_OPTERR

set -- `getopt "abcd:" "$@"`
# Sets positional parameters to command-line arguments.
# What happens if you use "$*" instead of "$@"?

while [ ! -z "$1" ]
  case "$1" in
    -a) echo "Option \"a\"";;
    -b) echo "Option \"b\"";;
    -c) echo "Option \"c\"";;
    -d) echo "Option \"d\" $2";;
     *) break;;


#  It is usually better to use the 'getopts' builtin in a script.
#  See ""

exit 0


   As Peggy Russell points out:
   It is often necessary to include an eval to correctly process
   whitespace and quotes.
args=$(getopt -o a:bc:d -- "$@")
eval set -- "$args"

          See Example 10-5 for a simplified emulation of getopt.

          The run-parts command [78] executes all the scripts in a
          target directory, sequentially in ASCII-sorted filename order.
          Of course, the scripts need to have execute permission.

          The cron daemon invokes run-parts to run the scripts in the
          /etc/cron.* directories.

          In its default behavior the yes command feeds a continuous
          string of the character y followed by a line feed to stdout. A
          control-C terminates the run. A different output string may be
          specified, as in yes different string, which would continually
          output different string to stdout.

          One might well ask the purpose of this. From the command-line
          or in a script, the output of yes can be redirected or piped
          into a program expecting user input. In effect, this becomes a
          sort of poor man's version of expect.

          yes | fsck /dev/hda1 runs fsck non-interactively (careful!).

          yes | rm -r dirname has same effect as rm -rf dirname


   Caution advised when piping yes to a potentially dangerous system
   command, such as fsck or fdisk. It might have unintended


   The yes command parses variables, or more accurately, it echoes
   parsed variables. For example:
bash$ yes $BASH_VERSION
 . . .

   This particular "feature" may be used to create a very large ASCII
   file on the fly:

bash$ yes $PATH > huge_file.txt

          Hit Ctl-C very quickly, or you just might get more than you
          bargained for. . . .

          The yes command may be emulated in a very simple script

yes ()
{ # Trivial emulation of "yes" ...
  local DEFAULT_TEXT="y"
  while [ true ]   # Endless loop.
    if [ -z "$1" ]
      echo "$DEFAULT_TEXT"
    else           # If argument ...
      echo "$1"    # ... expand and echo it.
  done             #  The only things missing are the
}                  #+ --help and --version options.

          Prints arguments as a large vertical banner to stdout, using
          an ASCII character (default '#'). This may be redirected to a
          printer for hardcopy.

          Note that banner has been dropped from many Linux distros.

          Show all the environmental variables set for a particular

bash$ printenv | grep HOME

          The lp and lpr commands send file(s) to the print queue, to be
          printed as hard copy. [79] These commands trace the origin of
          their names to the line printers of another era.

          bash$ lp file1.txt or bash lp <file1.txt

          It is often useful to pipe the formatted output from pr to lp.

          bash$ pr -options file1.txt | lp

          Formatting packages, such as groff and Ghostscript may send
          their output directly to lp.

          bash$ groff -Tascii | lp

          bash$ gs -options | lp

          Related commands are lpq, for viewing the print queue, and
          lprm, for removing jobs from the print queue.

          [UNIX borrows an idea from the plumbing trade.]

          This is a redirection operator, but with a difference. Like
          the plumber's tee, it permits "siphoning off" to a file the
          output of a command or commands within a pipe, but without
          affecting the result. This is useful for printing an ongoing
          process to a file or paper, perhaps to keep track of it for
          debugging purposes.

                            |----> to file
  command ---> command ---> |tee ---> command ---> ---> output of pipe

cat listfile* | sort | tee check.file | uniq > result.file
#                      ^^^^^^^^^^^^^^   ^^^^

#  The file "check.file" contains the concatenated sorted "listfiles,"
#+ before the duplicate lines are removed by 'uniq.'

          This obscure command creates a named pipe, a temporary
          first-in-first-out buffer for transferring data between
          processes. [80] Typically, one process writes to the FIFO, and
          the other reads from it. See Example A-14.

# This short script by Omair Eshkenazi.
# Used in ABS Guide with permission (thanks!).

mkfifo pipe1   # Yes, pipes can be given names.
mkfifo pipe2   # Hence the designation "named pipe."

(cut -d' ' -f1 | tr "a-z" "A-Z") >pipe2 <pipe1 &
ls -l | tr -s ' ' | cut -d' ' -f3,9- | tee pipe1 |
cut -d' ' -f2 | paste - pipe2

rm -f pipe1
rm -f pipe2

# No need to kill background processes when script terminates (why not?).

exit $?

Now, invoke the script and explain the output:

4830.tar.gz          BOZO
pipe1   BOZO
pipe2   BOZO    BOZO
Mixed.msg BOZO

          This command checks the validity of a filename. If the
          filename exceeds the maximum allowable length (255 characters)
          or one or more of the directories in its path is not
          searchable, then an error message results.

          Unfortunately, pathchk does not return a recognizable error
          code, and it is therefore pretty much useless in a script.
          Consider instead the file test operators.

          This is the somewhat obscure and much feared data duplicator
          command. Originally a utility for exchanging data on magnetic
          tapes between UNIX minicomputers and IBM mainframes, this
          command still has its uses. The dd command simply copies a
          file (or stdin/stdout), but with conversions. Possible
          conversions are ASCII/EBCDIC, [81] upper/lower case, swapping
          of byte pairs between input and output, and skipping and/or
          truncating the head or tail of the input file.

# Converting a file to all uppercase:

dd if=$filename conv=ucase > $filename.uppercase
#                    lcase   # For lower case conversion

          Some basic options to dd are:

          + if=INFILE
            INFILE is the source file.
          + of=OUTFILE
            OUTFILE is the target file, the file that will have the data
            written to it.
          + bs=BLOCKSIZE
            This is the size of each block of data being read and
            written, usually a power of 2.
          + skip=BLOCKS
            How many blocks of data to skip in INFILE before starting to
            copy. This is useful when the INFILE has "garbage" or
            garbled data in its header or when it is desirable to copy
            only a portion of the INFILE.
          + seek=BLOCKS
            How many blocks of data to skip in OUTFILE before starting
            to copy, leaving blank data at beginning of OUTFILE.
          + count=BLOCKS
            Copy only this many blocks of data, rather than the entire
          + conv=CONVERSION
            Type of conversion to be applied to INFILE data before
            copying operation.

          A dd --help lists all the options this powerful utility takes.

          Example 16-57. A script that copies itself


# This script copies itself.


dd if=$0 of=$0.$file_subscript 2>/dev/null
# Suppress messages from dd:   ^^^^^^^^^^^

exit $?

#  A program whose only output is its own source code
#+ is called a "quine" per Willard Quine.
#  Does this script qualify as a quine?

          Example 16-58. Exercising dd


# Script by Stephane Chazelas.
# Somewhat modified by ABS Guide author.

infile=$0       # This script.
outfile=log.txt # Output file left behind.

dd if=$infile of=$outfile bs=1 skip=$((n-1)) count=$((p-n+1)) 2> /dev/null
# Extracts characters n to p (3 to 5) from this script.

# --------------------------------------------------------

echo -n "hello world" | dd cbs=1 conv=unblock 2> /dev/null
# Echoes "hello world" vertically.
# Why? A newline follows each character dd emits.

exit 0

          To demonstrate just how versatile dd is, let's use it to
          capture keystrokes.

          Example 16-59. Capturing Keystrokes

# Capture keystrokes without needing to press ENTER.

keypresses=4                      # Number of keypresses to capture.

old_tty_setting=$(stty -g)        # Save old terminal settings.

echo "Press $keypresses keys."
stty -icanon -echo                # Disable canonical mode.
                                  # Disable local echo.
keys=$(dd bs=1 count=$keypresses 2> /dev/null)
# 'dd' uses stdin, if "if" (input file) not specified.

stty "$old_tty_setting"           # Restore old terminal settings.

echo "You pressed the \"$keys\" keys."

# Thanks, Stephane Chazelas, for showing the way.
exit 0

          The dd command can do random access on a data stream.

echo -n . | dd bs=1 seek=4 of=file conv=notrunc
#  The "conv=notrunc" option means that the output file
#+ will not be truncated.

# Thanks, S.C.

          The dd command can copy raw data and disk images to and from
          devices, such as floppies and tape drives (Example A-5). A
          common use is creating boot floppies.

          dd if=kernel-image of=/dev/fd0H1440

          Similarly, dd can copy the entire contents of a floppy, even
          one formatted with a "foreign" OS, to the hard drive as an
          image file.

          dd if=/dev/fd0 of=/home/bozo/projects/floppy.img

          Other applications of dd include initializing temporary swap
          files (Example 31-2) and ramdisks (Example 31-3). It can even
          do a low-level copy of an entire hard drive partition,
          although this is not necessarily recommended.

          People (with presumably nothing better to do with their time)
          are constantly thinking of interesting applications of dd.

          Example 16-60. Securely deleting a file

# Erase "all" traces of a file.

#  This script overwrites a target file alternately
#+ with random bytes, then zeros before finally deleting it.
#  After that, even examining the raw disk sectors by conventional methods
#+ will not reveal the original file data.

PASSES=7         #  Number of file-shredding passes.
                 #  Increasing this slows script execution,
                 #+ especially on large target files.
BLOCKSIZE=1      #  I/O with /dev/urandom requires unit block size,
                 #+ otherwise you get weird results.
E_BADARGS=70     #  Various error exit codes.

if [ -z "$1" ]   # No filename specified.
  echo "Usage: `basename $0` filename"
  exit $E_BADARGS


if [ ! -e "$file" ]
  echo "File \"$file\" not found."
  exit $E_NOT_FOUND

echo; echo -n "Are you absolutely sure you want to blot out \"$file\" (y/n)? "
read answer
case "$answer" in
[nN]) echo "Changed your mind, huh?"
      exit $E_CHANGED_MIND
*)    echo "Blotting out file \"$file\".";;

flength=$(ls -l "$file" | awk '{print $5}')  # Field 5 is file length.

chmod u+w "$file"   # Allow overwriting/deleting the file.


while [ "$pass_count" -le "$PASSES" ]
  echo "Pass #$pass_count"
  sync         # Flush buffers.
  dd if=/dev/urandom of=$file bs=$BLOCKSIZE count=$flength
               # Fill with random bytes.
  sync         # Flush buffers again.
  dd if=/dev/zero of=$file bs=$BLOCKSIZE count=$flength
               # Fill with zeros.
  sync         # Flush buffers yet again.
  let "pass_count += 1"

rm -f $file    # Finally, delete scrambled and shredded file.
sync           # Flush buffers a final time.

echo "File \"$file\" blotted out and deleted."; echo

exit 0

#  This is a fairly secure, if inefficient and slow method
#+ of thoroughly "shredding" a file.
#  The "shred" command, part of the GNU "fileutils" package,
#+ does the same thing, although more efficiently.

#  The file cannot not be "undeleted" or retrieved by normal methods.
#  However . . .
#+ this simple method would *not* likely withstand
#+ sophisticated forensic analysis.

#  This script may not play well with a journaled file system.
#  Exercise (difficult): Fix it so it does.

#  Tom Vier's "wipe" file-deletion package does a much more thorough job
#+ of file shredding than this simple script.

#  For an in-depth analysis on the topic of file deletion and security,
#+ see Peter Gutmann's paper,
#+     "Secure Deletion of Data From Magnetic and Solid-State Memory".

          See also the dd thread entry in the bibliography.

          The od, or octal dump filter converts input (or files) to
          octal (base-8) or other bases. This is useful for viewing or
          processing binary data files or otherwise unreadable system
          device files, such as /dev/urandom, and as a filter for binary

head -c4 /dev/urandom | od -N4 -tu4 | sed -ne '1s/.* //p'
# Sample output: 1324725719, 3918166450, 2989231420, etc.

# From example script, by StИphane Chazelas

          See also Example 9-16 and Example A-36.

          Performs a hexadecimal, octal, decimal, or ASCII dump of a
          binary file. This command is the rough equivalent of od,
          above, but not nearly as useful. May be used to view the
          contents of a binary file, in combination with dd and less.

dd if=/bin/ls | hexdump -C | less
# The -C option nicely formats the output in tabular form.

          Displays information about an object file or binary executable
          in either hexadecimal form or as a disassembled listing (with
          the -d option).

bash$ objdump -d /bin/ls
/bin/ls:     file format elf32-i386

 Disassembly of section .init:

 080490bc <.init>:
  80490bc:       55                      push   %ebp
  80490bd:       89 e5                   mov    %esp,%ebp
  . . .

          This command generates a "magic cookie," a 128-bit
          (32-character) pseudorandom hexadecimal number, normally used
          as an authorization "signature" by the X server. This also
          available for use in a script as a "quick 'n dirty" random


          Of course, a script could use md5sum for the same purpose.

# Generate md5 checksum on the script itself.
random001=`md5sum $0 | awk '{print $1}'`
# Uses 'awk' to strip off the filename.

          The mcookie command gives yet another way to generate a
          "unique" filename.

          Example 16-61. Filename generator

#  temp filename generator

BASE_STR=`mcookie`   # 32-character magic cookie.
POS=11               # Arbitrary position in magic cookie string.
LEN=5                # Get $LEN consecutive characters.

prefix=temp          #  This is, after all, a "temp" file.
                     #  For more "uniqueness," generate the
                     #+ filename prefix using the same method
                     #+ as the suffix, below.

                     #  Extract a 5-character string,
                     #+ starting at position 11.

                     # Construct the filename.

echo "Temp filename = "$temp_filename""

# sh
# Temp filename = temp.e19ea

#  Compare this method of generating "unique" filenames
#+ with the 'date' method in

exit 0

          This utility converts between different units of measure.
          While normally invoked in interactive mode, units may find use
          in a script.

          Example 16-62. Converting meters to miles


convert_units ()  # Takes as arguments the units to convert.
  cf=$(units "$1" "$2" | sed --silent -e '1p' | awk '{print $2}')
  # Strip off everything except the actual conversion factor.
  echo "$cf"

cfactor=`convert_units $Unit1 $Unit2`

result=$(echo $quantity*$cfactor | bc)

echo "There are $result $Unit2 in $quantity $Unit1."

#  What happens if you pass incompatible units,
#+ such as "acres" and "miles" to the function?

exit 0

          A hidden treasure, m4 is a powerful macro [82] processing
          filter, virtually a complete language. Although originally
          written as a pre-processor for RatFor, m4 turned out to be
          useful as a stand-alone utility. In fact, m4 combines some of
          the functionality of eval, tr, and awk, in addition to its
          extensive macro expansion facilities.

          The April, 2002 issue of Linux Journal has a very nice article
          on m4 and its uses.

          Example 16-63. Using m4

# Using the m4 macro processor

# Strings
echo "len($string)" | m4                            #   7
echo "substr($string,4)" | m4                       # A01
echo "regexp($string,[0-1][0-1],\&Z)" | m4          # 01Z

# Arithmetic
echo "incr(22)" | m4                                #  23
echo "eval(99 / 3)" | m4                            #  33


          This X-based variant of echo pops up a message/query window on
          the desktop.

xmessage Left click to continue -button okay

          The [] zenity utility is
          adept at displaying GTK+ dialog widgets and very suitable for
          scripting purposes.

          The doexec command enables passing an arbitrary list of
          arguments to a binary executable. In particular, passing
          argv[0] (which corresponds to $0 in a script) lets the
          executable be invoked by various names, and it can then carry
          out different sets of actions, according to the name by which
          it was called. What this amounts to is roundabout way of
          passing options to an executable.

          For example, the /usr/local/bin directory might contain a
          binary called "aaa". Invoking doexec /usr/local/bin/aaa list
          would list all those files in the current working directory
          beginning with an "a", while invoking (the same executable
          with) doexec /usr/local/bin/aaa delete would delete those


   The various behaviors of the executable must be defined within the
   code of the executable itself, analogous to something like the
   following in a shell script:
case `basename $0` in
"name1" ) do_something;;
"name2" ) do_something_else;;
"name3" ) do_yet_another_thing;;
*       ) bail_out;;

          The dialog family of tools provide a method of calling
          interactive "dialog" boxes from a script. The more elaborate
          variations of dialog -- gdialog, Xdialog, and kdialog --
          actually invoke X-Windows widgets.

          The sox, or "sound exchange" command plays and performs
          transformations on sound files. In fact, the /usr/bin/play
          executable (now deprecated) is nothing but a shell wrapper for

          For example, sox soundfile.wav changes a WAV
          sound file into a (Sun audio format) AU sound file.

          Shell scripts are ideally suited for batch-processing sox
          operations on sound files. For examples, see the Linux Radio
          Timeshift HOWTO and the MP3do Project.

Chapter 17. System and Administrative Commands

   The startup and shutdown scripts in /etc/rc.d illustrate the uses
   (and usefulness) of many of these comands. These are usually invoked
   by root and used for system maintenance or emergency filesystem
   repairs. Use with caution, as some of these commands may damage your
   system if misused.

   Users and Groups

          Show all logged on users. This is the approximate equivalent
          of who -q.

          Lists the current user and the groups she belongs to. This
          corresponds to the $GROUPS internal variable, but gives the
          group names, rather than the numbers.

bash$ groups
bozita cdrom cdwriter audio xgrp

bash$ echo $GROUPS

   chown, chgrp
          The chown command changes the ownership of a file or files.
          This command is a useful method that root can use to shift
          file ownership from one user to another. An ordinary user may
          not change the ownership of files, not even her own files.

root# chown bozo *.txt

          The chgrp command changes the group ownership of a file or
          files. You must be owner of the file(s) as well as a member of
          the destination group (or root) to use this operation.

chgrp --recursive dunderheads *.data
#  The "dunderheads" group will now own all the "*.data" files
#+ all the way down the $PWD directory tree (that's what "recursive" means).

   useradd, userdel
          The useradd administrative command adds a user account to the
          system and creates a home directory for that particular user,
          if so specified. The corresponding userdel command removes a
          user account from the system [84] and deletes associated


   The adduser command is a synonym for useradd and is usually a
   symbolic link to it.

          Modify a user account. Changes may be made to the password,
          group membership, expiration date, and other attributes of a
          given user's account. With this command, a user's password may
          be locked, which has the effect of disabling the account.

          Modify a given group. The group name and/or ID number may be
          changed using this command.

          The id command lists the real and effective user IDs and the
          group IDs of the user associated with the current process.
          This is the counterpart to the $UID, $EUID, and $GROUPS
          internal Bash variables.

bash$ id
uid=501(bozo) gid=501(bozo) groups=501(bozo),22(cdrom),80(cdwriter),81(audio)

bash$ echo $UID


   The id command shows the effective IDs only when they differ from the
   real ones.

          Also see Example 9-5.

          The lid (list ID) command shows the group(s) that a given user
          belongs to, or alternately, the users belonging to a given
          group. May be invoked only by root.

root# lid bozo

root# lid daemon

          Show all users logged on to the system.

bash$ who
bozo  tty1     Apr 27 17:45
 bozo  pts/0    Apr 27 17:46
 bozo  pts/1    Apr 27 17:47
 bozo  pts/2    Apr 27 17:49

          The -m gives detailed information about only the current user.
          Passing any two arguments to who is the equivalent of who -m,
          as in who am i or who The Man.

bash$ who -m
localhost.localdomain!bozo  pts/2    Apr 27 17:49

          whoami is similar to who -m, but only lists the user name.

bash$ whoami

          Show all logged on users and the processes belonging to them.
          This is an extended version of who. The output of w may be
          piped to grep to find a specific user and/or process.

bash$ w | grep startx
bozo  tty1     -                 4:22pm  6:41   4.47s  0.45s  startx

          Show current user's login name (as found in /var/run/utmp).
          This is a near-equivalent to whoami, above.

bash$ logname

bash$ whoami

          However . . .

bash$ su
Password: ......

bash# whoami
bash# logname


   While logname prints the name of the logged in user, whoami gives the
   name of the user attached to the current process. As we have just
   seen, sometimes these are not the same.

          Runs a program or script as a substitute user. su rjones
          starts a shell as user rjones. A naked su defaults to root.
          See Example A-14.

          Runs a command as root (or another user). This may be used in
          a script, thus permitting a regular user to run the script.


# Some commands.
sudo cp /root/secretfile /home/bozo/secret
# Some more commands.

          The file /etc/sudoers holds the names of users permitted to
          invoke sudo.

          Sets, changes, or manages a user's password.

          The passwd command can be used in a script, but probably
          should not be.

          Example 17-1. Setting a new password

# For demonstration purposes only.
#                      Not a good idea to actually run this script.
#  This script must be run as root.

ROOT_UID=0         # Root has $UID 0.
E_WRONG_USER=65    # Not root?


if [ "$UID" -ne "$ROOT_UID" ]
  echo; echo "Only root can run this script."; echo
  exit $E_WRONG_USER
  echo "You should know better than to run this script, root."
  echo "Even root users get the blues... "


# Check if bozo lives here.
grep -q "$username" /etc/passwd
if [ $? -ne $SUCCESS ]
  echo "User $username does not exist."
  echo "No password changed."

echo "$NEWPASSWORD" | passwd --stdin "$username"
#  The '--stdin' option to 'passwd' permits
#+ getting a new password from stdin (or a pipe).

echo; echo "User $username's password changed!"

# Using the 'passwd' command in a script is dangerous.

exit 0

          The passwd command's -l, -u, and -d options permit locking,
          unlocking, and deleting a user's password. Only root may use
          these options.

          Show users' logged in time, as read from /var/log/wtmp. This
          is one of the GNU accounting utilities.

bash$ ac
        total       68.08

          List last logged in users, as read from /var/log/wtmp. This
          command can also show remote logins.

          For example, to show the last few times the system rebooted:

bash$ last reboot
reboot   system boot  2.6.9-1.667      Fri Feb  4 18:18          (00:02)
 reboot   system boot  2.6.9-1.667      Fri Feb  4 15:20          (01:27)
 reboot   system boot  2.6.9-1.667      Fri Feb  4 12:56          (00:49)
 reboot   system boot  2.6.9-1.667      Thu Feb  3 21:08          (02:17)
 . . .

 wtmp begins Tue Feb  1 12:50:09 2005

          Change user's group ID without logging out. This permits
          access to the new group's files. Since users may be members of
          multiple groups simultaneously, this command finds only
          limited use.


   Kurt Glaesemann points out that the newgrp command could prove
   helpful in setting the default group permissions for files a user
   writes. However, the chgrp command might be more convenient for this


          Echoes the name (filename) of the current user's terminal.
          Note that each separate xterm window counts as a different

bash$ tty

          Shows and/or changes terminal settings. This complex command,
          used in a script, can control terminal behavior and the way
          output displays. See the info page, and study it carefully.

          Example 17-2. Setting an erase character

# Using "stty" to set an erase character when reading input.

echo -n "What is your name? "
read name                      #  Try to backspace
                               #+ to erase characters of input.
                               #  Problems?
echo "Your name is $name."

stty erase '#'                 #  Set "hashmark" (#) as erase character.
echo -n "What is your name? "
read name                      #  Use # to erase last character typed.
echo "Your name is $name."

exit 0

# Even after the script exits, the new key value remains set.
# Exercise: How would you reset the erase character to the default value?

          Example 17-3. secret password: Turning off terminal echoing

# secret password

echo -n "Enter password "
read passwd
echo "password is $passwd"
echo -n "If someone had been looking over your shoulder, "
echo "your password would have been compromised."

echo && echo  # Two line-feeds in an "and list."

stty -echo    # Turns off screen echo.

echo -n "Enter password again "
read passwd
echo "password is $passwd"

stty echo     # Restores screen echo.

exit 0

# Do an 'info stty' for more on this useful-but-tricky command.

          A creative use of stty is detecting a user keypress (without
          hitting ENTER).

          Example 17-4. Keypress detection

# Detect a user keypress ("hot keys").


old_tty_settings=$(stty -g)   # Save old settings (why?).
stty -icanon
Keypress=$(head -c1)          # or $(dd bs=1 count=1 2> /dev/null)
                              # on non-GNU systems

echo "Key pressed was \""$Keypress"\"."

stty "$old_tty_settings"      # Restore old settings.

# Thanks, Stephane Chazelas.

exit 0

          Also see Example 9-3 and Example A-43.

   terminals and modes
   Normally, a terminal works in the canonical mode. When a user hits a
   key, the resulting character does not immediately go to the program
   actually running in this terminal. A buffer local to the terminal
   stores keystrokes. When the user hits the ENTER key, this sends all
   the stored keystrokes to the program running. There is even a basic
   line editor inside the terminal.
bash$ stty -a
speed 9600 baud; rows 36; columns 96; line = 0;
 intr = ^C; quit = ^\; erase = ^H; kill = ^U; eof = ^D; eol = <undef>; eol2 =
 start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; flush
= ^O;
 isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt

   Using canonical mode, it is possible to redefine the special keys for
   the local terminal line editor.
bash$ cat > filexxx
wha<ctl-W>I<ctl-H>foo bar<ctl-U>hello world<ENTER>
bash$ cat filexxx
hello world
bash$ wc -c < filexxx

   The process controlling the terminal receives only 12 characters (11
   alphabetic ones, plus a newline), although the user hit 26 keys.

   In non-canonical ("raw") mode, every key hit (including special
   editing keys such as ctl-H) sends a character immediately to the
   controlling process.
   The Bash prompt disables both icanon and echo, since it replaces the
   basic terminal line editor with its own more elaborate one. For
   example, when you hit ctl-A at the Bash prompt, there's no ^A echoed
   by the terminal, but Bash gets a \1 character, interprets it, and
   moves the cursor to the begining of the line.
   StИphane Chazelas

          Set certain terminal attributes. This command writes to its
          terminal's stdout a string that changes the behavior of that

bash$ setterm -cursor off

          The setterm command can be used within a script to change the
          appearance of text written to stdout, although there are
          certainly better tools available for this purpose.

setterm -bold on
echo bold hello

setterm -bold off
echo normal hello

          Show or initialize terminal settings. This is a less capable
          version of stty.

bash$ tset -r
Terminal type is xterm-xfree86.
 Kill is control-U (^U).
 Interrupt is control-C (^C).

          Set or display serial port parameters. This command must be
          run by root and is usually found in a system setup script.

# From /etc/pcmcia/serial script:

IRQ=`setserial /dev/$DEVICE | sed -e 's/.*IRQ: //'`
setserial /dev/$DEVICE irq 0 ; setserial /dev/$DEVICE irq $IRQ

   getty, agetty
          The initialization process for a terminal uses getty or agetty
          to set it up for login by a user. These commands are not used
          within user shell scripts. Their scripting counterpart is

          Enables or disables write access to the current user's
          terminal. Disabling access would prevent another user on the
          network to write to the terminal.


   It can be quite annoying to have a message about ordering pizza
   suddenly appear in the middle of the text file you are editing. On a
   multi-user network, you might therefore wish to disable write access
   to your terminal when you need to avoid interruptions.

          This is an acronym for "write all," i.e., sending a message to
          all users at every terminal logged into the network. It is
          primarily a system administrator's tool, useful, for example,
          when warning everyone that the system will shortly go down due
          to a problem (see Example 19-1).

bash$ wall System going down for maintenance in 5 minutes!
Broadcast message from bozo (pts/1) Sun Jul  8 13:53:27 2001...

 System going down for maintenance in 5 minutes!


   If write access to a particular terminal has been disabled with mesg,
   then wall cannot send a message to that terminal.

   Information and Statistics

          Output system specifications (OS, kernel version, etc.) to
          stdout. Invoked with the -a option, gives verbose system info
          (see Example 16-5). The -s option shows only the OS type.

bash$ uname

bash$ uname -s

bash$ uname -a
Linux iron.bozo 2.6.15-1.2054_FC5 #1 Tue Mar 14 15:48:33 EST 2006
 i686 i686 i386 GNU/Linux

          Show system architecture. Equivalent to uname -m. See Example

bash$ arch

bash$ uname -m

          Gives information about previous commands, as stored in the
          /var/account/pacct file. Command name and user name can be
          specified by options. This is one of the GNU accounting

          List the last login time of all system users. This references
          the /var/log/lastlog file.

bash$ lastlog
root          tty1                      Fri Dec  7 18:43:21 -0700 2001
 bin                                     **Never logged in**
 daemon                                  **Never logged in**
 bozo          tty1                      Sat Dec  8 21:14:29 -0700 2001

bash$ lastlog | grep root
root          tty1                      Fri Dec  7 18:43:21 -0700 2001


   This command will fail if the user invoking it does not have read
   permission for the /var/log/lastlog file.

          List open files. This command outputs a detailed table of all
          currently open files and gives information about their owner,
          size, the processes associated with them, and more. Of course,
          lsof may be piped to grep and/or awk to parse and analyze its

bash$ lsof
 init         1    root  mem    REG        3,5   30748    30303 /sbin/init
 init         1    root  mem    REG        3,5   73120     8069 /lib/ld-2.1.3.
 init         1    root  mem    REG        3,5  931668     8075 /lib/libc-2.1.
 cardmgr    213    root  mem    REG        3,5   36956    30357 /sbin/cardmgr

          The lsof command is a useful, if complex administrative tool.
          If you are unable to dismount a filesystem and get an error
          message that it is still in use, then running lsof helps
          determine which files are still open on that filesystem. The
          -i option lists open network socket files, and this can help
          trace intrusion or hack attempts.

bash$ lsof -an -i tcp
 firefox 2330 bozo  32u IPv4   9956       TCP>
:http ...
 firefox 2330 bozo  38u IPv4  10535       TCP>
:http ...

          See Example 30-2 for an effective use of lsof.

          System trace: diagnostic and debugging tool for tracing system
          calls and signals. This command and ltrace, following, are
          useful for diagnosing why a given program or package fails to
          run . . . perhaps due to missing libraries or related causes.

bash$ strace df
execve("/bin/df", ["df"], [/* 45 vars */]) = 0
 uname({sys="Linux", node="bozo.localdomain", ...}) = 0
 brk(0)                                  = 0x804f5e4


          This is the Linux equivalent of the Solaris truss command.

          Library trace: diagnostic and debugging tool that traces
          library calls invoked by a given command.

bash$ ltrace df
__libc_start_main(0x804a910, 1, 0xbfb589a4, 0x804fb70, 0x804fb68 <unfinished .
 setlocale(6, "")                                 = "en_US.UTF-8"
bindtextdomain("coreutils", "/usr/share/locale") = "/usr/share/locale"
textdomain("coreutils")                          = "coreutils"
__cxa_atexit(0x804b650, 0, 0, 0x8052bf0, 0xbfb58908) = 0
getenv("DF_BLOCK_SIZE")                          = NULL


          The nc (netcat) utility is a complete toolkit for connecting
          to and listening to TCP and UDP ports. It is useful as a
          diagnostic and testing tool and as a component in simple
          script-based HTTP clients and servers.

bash$ nc localhost.localdomain 25
220 localhost.localdomain ESMTP Sendmail 8.13.1/8.13.1;
 Thu, 31 Mar 2005 15:41:35 -0700

          Example 17-5. Checking a remote server for identd

#! /bin/sh
## Duplicate DaveG's ident-scan thingie using netcat. Oooh, he'll be p*ssed.
## Args: target port [port port port ...]
## Hose stdout _and_ stderr together.
##  Advantages: runs slower than ident-scan, giving remote inetd less cause
##+ for alarm, and only hits the few known daemon ports you specify.
##  Disadvantages: requires numeric-only port args, the output sleazitude,
##+ and won't work for r-services when coming from high source ports.
# Script author: Hobbit <>
# Used in ABS Guide with permission.

# ---------------------------------------------------
E_BADARGS=65       # Need at least two args.
TWO_WINKS=2        # How long to sleep.
IDPORT=113         # Authentication "tap ident" port.
# ---------------------------------------------------

case "${2}" in
  "" ) echo "Need HOST and at least one PORT." ; exit $E_BADARGS ;;

# Ping 'em once and see if they *are* running identd.
nc -z -w $TIMEOUT0 "$1" $IDPORT || \
{ echo "Oops, $1 isn't running identd." ; exit 0 ; }
#  -z scans for listening daemons.
#     -w $TIMEOUT = How long to try to connect.

# Generate a randomish base port.
RP=`expr $$ % $RAND1 + $RAND2`


while test "$1" ; do
  nc -v -w $TIMEOUT1 -p ${RP} "$TRG" ${1} < /dev/null > /dev/null &
  sleep $THREE_WINKS
  echo "${1},${RP}" | nc -w $TIMEOUT2 -r "$TRG" $IDPORT 2>&1
  sleep $TWO_WINKS

# Does this look like a lamer script or what . . . ?
# ABS Guide author comments: "Ain't really all that bad . . .
#+                            kinda clever, actually."

  kill -HUP $PROC
  RP=`expr ${RP} + 1`

exit $?

#  Notes:
#  -----

#  Try commenting out line 30 and running this script
#+ with "localhost.localdomain 25" as arguments.

#  For more of Hobbit's 'nc' example scripts,
#+ look in the documentation:
#+ the /usr/share/doc/nc-X.XX/scripts directory.

          And, of course, there's Dr. Andrew Tridgell's notorious
          one-line script in the BitKeeper Affair:

echo clone | nc 5000 > e2fsprogs.dat

          Shows memory and cache usage in tabular form. The output of
          this command lends itself to parsing, using grep, awk or Perl.
          The procinfo command shows all the information that free does,
          and much more.

bash$ free
                total       used       free     shared    buffers     cached
   Mem:         30504      28624       1880      15820       1608       16376
   -/+ buffers/cache:      10640      19864
   Swap:        68540       3128      65412

          To show unused RAM memory:

bash$ free | grep Mem | awk '{ print $4 }'

          Extract and list information and statistics from the /proc
          pseudo-filesystem. This gives a very extensive and detailed

bash$ procinfo | grep Bootup
Bootup: Wed Mar 21 15:15:50 2001    Load average: 0.04 0.21 0.34 3/47 6829

          List devices, that is, show installed hardware.

bash$ lsdev
Device            DMA   IRQ  I/O Ports
 cascade             4     2
 dma                          0080-008f
 dma1                         0000-001f
 dma2                         00c0-00df
 fpu                          00f0-00ff
 ide0                     14  01f0-01f7 03f6-03f6

          Show (disk) file usage, recursively. Defaults to current
          working directory, unless otherwise specified.

bash$ du -ach
1.0k    ./
 1.0k    ./
 1.0k    ./random.file
 6.0k    .
 6.0k    total

          Shows filesystem usage in tabular form.

bash$ df
Filesystem           1k-blocks      Used Available Use% Mounted on
 /dev/hda5               273262     92607    166547  36% /
 /dev/hda8               222525    123951     87085  59% /home
 /dev/hda7              1408796   1075744    261488  80% /usr

          Lists all system bootup messages to stdout. Handy for
          debugging and ascertaining which device drivers were installed
          and which system interrupts in use. The output of dmesg may,
          of course, be parsed with grep, sed, or awk from within a

bash$ dmesg | grep hda
Kernel command line: ro root=/dev/hda2
 hda: IBM-DLGA-23080, ATA DISK drive
 hda: 6015744 sectors (3080 MB) w/96KiB Cache, CHS=746/128/63
 hda: hda1 hda2 hda3 < hda5 hda6 hda7 > hda4

          Gives detailed and verbose statistics on a given file (even a
          directory or device file) or set of files.

bash$ stat test.cru
  File: "test.cru"
   Size: 49970        Allocated Blocks: 100          Filetype: Regular File
   Mode: (0664/-rw-rw-r--)         Uid: (  501/ bozo)  Gid: (  501/ bozo)
 Device:  3,8   Inode: 18185     Links: 1
 Access: Sat Jun  2 16:40:24 2001
 Modify: Sat Jun  2 16:40:24 2001
 Change: Sat Jun  2 16:40:24 2001

          If the target file does not exist, stat returns an error

bash$ stat nonexistent-file
nonexistent-file: No such file or directory

          In a script, you can use stat to extract information about
          files (and filesystems) and set variables accordingly.


# Per suggestion of JoКl Bourquard and . . .

file_name=$(stat -c%n "$FILENAME")   # Same as "$FILENAME" of course.
file_owner=$(stat -c%U "$FILENAME")
file_size=$(stat -c%s "$FILENAME")
#  Certainly easier than using "ls -l $FILENAME"
#+ and then parsing with sed.
file_inode=$(stat -c%i "$FILENAME")
file_type=$(stat -c%F "$FILENAME")
file_access_rights=$(stat -c%A "$FILENAME")

echo "File name:          $file_name"
echo "File owner:         $file_owner"
echo "File size:          $file_size"
echo "File inode:         $file_inode"
echo "File type:          $file_type"
echo "File access rights: $file_access_rights"

exit 0


File name:          testfile.txt
File owner:         bozo
File size:          418
File inode:         1730378
File type:          regular file
File access rights: -rw-rw-r--

          Display virtual memory statistics.

bash$ vmstat
   procs                      memory    swap          io system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy id
 0  0  0      0  11040   2636  38952   0   0    33     7  271    88   8   3 89

          Shows how long the system has been running, along with
          associated statistics.

bash$ uptime
10:28pm  up  1:57,  3 users,  load average: 0.17, 0.34, 0.27


   A load average of 1 or less indicates that the system handles
   processes immediately. A load average greater than 1 means that
   processes are being queued. When the load average gets above 3 (on a
   single-core processor), then system performance is significantly

          Lists the system's host name. This command sets the host name
          in an /etc/rc.d setup script (/etc/rc.d/rc.sysinit or
          similar). It is equivalent to uname -n, and a counterpart to
          the $HOSTNAME internal variable.

bash$ hostname

bash$ echo $HOSTNAME

          Similar to the hostname command are the domainname,
          dnsdomainname, nisdomainname, and ypdomainname commands. Use
          these to display or set the system DNS or NIS/YP domain name.
          Various options to hostname also perform these functions.

          Echo a 32-bit hexadecimal numerical identifier for the host

bash$ hostid


   This command allegedly fetches a "unique" serial number for a
   particular system. Certain product registration procedures use this
   number to brand a particular user license. Unfortunately, hostid only
   returns the machine network address in hexadecimal, with pairs of
   bytes transposed.
   The network address of a typical non-networked Linux machine, is
   found in /etc/hosts.
bash$ cat /etc/hosts               localhost.localdomain localhost

   As it happens, transposing the bytes of, we get,
   which translates in hex to 007f0100, the exact equivalent of what
   hostid returns, above. There exist only a few million other Linux
   machines with this identical hostid.

          Invoking sar (System Activity Reporter) gives a very detailed
          rundown on system statistics. The Santa Cruz Operation ("Old"
          SCO) released sar as Open Source in June, 1999.

          This command is not part of the base Linux distribution, but
          may be obtained as part of
          the[] sysstat
          utilities package, written by Sebastien Godard.

bash$ sar
Linux 2.4.9 (        09/26/03

10:30:00          CPU     %user     %nice   %system   %iowait     %idle
10:40:00          all      2.21     10.90     65.48      0.00     21.41
10:50:00          all      3.36      0.00     72.36      0.00     24.28
11:00:00          all      1.12      0.00     80.77      0.00     18.11
Average:          all      2.23      3.63     72.87      0.00     21.27

14:32:30          LINUX RESTART

15:00:00          CPU     %user     %nice   %system   %iowait     %idle
15:10:00          all      8.59      2.40     17.47      0.00     71.54
15:20:00          all      4.07      1.00     11.95      0.00     82.98
15:30:00          all      0.79      2.94      7.56      0.00     88.71
Average:          all      6.33      1.70     14.71      0.00     77.26

          Show information and statistics about a designated elf binary.
          This is part of the binutils package.

bash$ readelf -h /bin/bash
ELF Header:
   Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
   Class:                             ELF32
   Data:                              2's complement, little endian
   Version:                           1 (current)
   OS/ABI:                            UNIX - System V
   ABI Version:                       0
   Type:                              EXEC (Executable file)
   . . .

          The size [/path/to/binary] command gives the segment sizes of
          a binary executable or archive file. This is mainly of use to

bash$ size /bin/bash
   text    data     bss     dec     hex filename
  495971   22496   17392  535859   82d33 /bin/bash

   System Logs

          Appends a user-generated message to the system log
          (/var/log/messages). You do not have to be root to invoke

logger Experiencing instability in network connection at 23:10, 05/21.
# Now, do a 'tail /var/log/messages'.

          By embedding a logger command in a script, it is possible to
          write debugging information to /var/log/messages.

logger -t $0 -i Logging at line "$LINENO".
# The "-t" option specifies the tag for the logger entry.
# The "-i" option records the process ID.

# tail /var/log/message
# ...
# Jul  7 20:48:58 localhost ./[1712]: Logging at line 3.

          This utility manages the system log files, rotating,
          compressing, deleting, and/or e-mailing them, as appropriate.
          This keeps the /var/log from getting cluttered with old log
          files. Usually cron runs logrotate on a daily basis.

          Adding an appropriate entry to /etc/logrotate.conf makes it
          possible to manage personal log files, as well as system-wide


   Stefano Falsetto has created []
   rottlog, which he considers to be an improved version of logrotate.

   Job Control

          Process Statistics: lists currently executing processes by
          owner and PID (process ID). This is usually invoked with ax or
          aux options, and may be piped to grep or sed to search for a
          specific process (see Example 15-14 and Example 29-3).

bash$  ps ax | grep sendmail
295 ?      S      0:00 sendmail: accepting connections on port 25

          To display system processes in graphical "tree" format: ps
          afjx or ps ax --forest.

   pgrep, pkill
          Combining the ps command with grep or kill.

bash$ ps a | grep mingetty
2212 tty2     Ss+    0:00 /sbin/mingetty tty2
 2213 tty3     Ss+    0:00 /sbin/mingetty tty3
 2214 tty4     Ss+    0:00 /sbin/mingetty tty4
 2215 tty5     Ss+    0:00 /sbin/mingetty tty5
 2216 tty6     Ss+    0:00 /sbin/mingetty tty6
 4849 pts/2    S+     0:00 grep mingetty

bash$ pgrep mingetty
2212 mingetty
 2213 mingetty
 2214 mingetty
 2215 mingetty
 2216 mingetty

          Compare the action of pkill with killall.

          Lists currently executing processes in "tree" format. The -p
          option shows the PIDs, as well as the process names.

          Continuously updated display of most cpu-intensive processes.
          The -b option displays in text mode, so that the output may be
          parsed or accessed from a script.

bash$ top -b
  8:30pm  up 3 min,  3 users,  load average: 0.49, 0.32, 0.13
 45 processes: 44 sleeping, 1 running, 0 zombie, 0 stopped
 CPU states: 13.6% user,  7.3% system,  0.0% nice, 78.9% idle
 Mem:    78396K av,   65468K used,   12928K free,       0K shrd,    2352K buff
 Swap:  157208K av,       0K used,  157208K free                   37244K cach

   848 bozo      17   0   996  996   800 R     5.6  1.2   0:00 top
     1 root       8   0   512  512   444 S     0.0  0.6   0:04 init
     2 root       9   0     0    0     0 SW    0.0  0.0   0:00 keventd

          Run a background job with an altered priority. Priorities run
          from 19 (lowest) to -20 (highest). Only root may set the
          negative (higher) priorities. Related commands are renice and
          snice, which change the priority of a running process or
          processes, and skill, which sends a kill signal to a process
          or processes.

          Keeps a command running even after user logs off. The command
          will run as a foreground process unless followed by &. If you
          use nohup within a script, consider coupling it with a wait to
          avoid creating an orphan or zombie process.

          Identifies process ID (PID) of a running job. Since job
          control commands, such as kill and renice act on the PID of a
          process (not its name), it is sometimes necessary to identify
          that PID. The pidof command is the approximate counterpart to
          the $PPID internal variable.

bash$ pidof xclock

          Example 17-6. pidof helps kill a process



process=xxxyyyzzz  # Use nonexistent process.
# For demo purposes only...
# ... don't want to actually kill any actual process with this script.
# If, for example, you wanted to use this script to logoff the Internet,
#     process=pppd

t=`pidof $process`       # Find pid (process id) of $process.
# The pid is needed by 'kill' (can't 'kill' by program name).

if [ -z "$t" ]           # If process not present, 'pidof' returns null.
  echo "Process $process was not running."
  echo "Nothing killed."

kill $t                  # May need 'kill -9' for stubborn process.

# Need a check here to see if process allowed itself to be killed.
# Perhaps another " t=`pidof $process` " or ...

# This entire script could be replaced by
#        kill $(pidof -x process_name)
# or
#        killall process_name
# but it would not be as instructive.

exit 0

          Identifies the processes (by PID) that are accessing a given
          file, set of files, or directory. May also be invoked with the
          -k option, which kills those processes. This has interesting
          implications for system security, especially in scripts
          preventing unauthorized users from accessing system services.

bash$ fuser -u /usr/bin/vim
/usr/bin/vim:         3207e(bozo)

bash$ fuser -u /dev/null
/dev/null:            3009(bozo)  3010(bozo)  3197(bozo)  3199(bozo)

          One important application for fuser is when physically
          inserting or removing storage media, such as CD ROM disks or
          USB flash drives. Sometimes trying a umount fails with a
          device is busy error message. This means that some user(s)
          and/or process(es) are accessing the device. An fuser -um
          /dev/device_name will clear up the mystery, so you can kill
          any relevant processes.

bash$ umount /mnt/usbdrive
umount: /mnt/usbdrive: device is busy

bash$ fuser -um /dev/usbdrive
/mnt/usbdrive:        1772c(bozo)

bash$ kill -9 1772
bash$ umount /mnt/usbdrive

          The fuser command, invoked with the -n option identifies the
          processes accessing a port. This is especially useful in
          combination with nmap.

root# nmap localhost.localdomain
 25/tcp   open  smtp

root# fuser -un tcp 25
25/tcp:               2095(root)

root# ps ax | grep 2095 | grep -v grep
2095 ?        Ss     0:00 sendmail: accepting connections

          Administrative program scheduler, performing such duties as
          cleaning up and deleting system log files and updating the
          slocate database. This is the superuser version of at
          (although each user may have their own crontab file which can
          be changed with the crontab command). It runs as a daemon and
          executes scheduled entries from /etc/crontab.


   Some flavors of Linux run crond, Matthew Dillon's version of cron.

   Process Control and Booting

          The init command is the parent of all processes. Called in the
          final step of a bootup, init determines the runlevel of the
          system from /etc/inittab. Invoked by its alias telinit, and by
          root only.

          Symlinked to init, this is a means of changing the system
          runlevel, usually done for system maintenance or emergency
          filesystem repairs. Invoked only by root. This command can be
          dangerous -- be certain you understand it well before using!

          Shows the current and last runlevel, that is, whether the
          system is halted (runlevel 0), in single-user mode (1), in
          multi-user mode (2 or 3), in X Windows (5), or rebooting (6).
          This command accesses the /var/run/utmp file.

   halt, shutdown, reboot
          Command set to shut the system down, usually just prior to a
          power down.


   On some Linux distros, the halt command has 755 permissions, so it
   can be invoked by a non-root user. A careless halt in a terminal or a
   script may shut down the system!

          Starts or stops a system service. The startup scripts in
          /etc/init.d and /etc/rc.d use this command to start services
          at bootup.

root# /sbin/service iptables stop
Flushing firewall rules:                                   [  OK  ]
 Setting chains to policy ACCEPT: filter                    [  OK  ]
 Unloading iptables modules:                                [  OK  ]


          Network mapper and port scanner. This command scans a server
          to locate open ports and the services associated with those
          ports. It can also report information about packet filters and
          firewalls. This is an important security tool for locking down
          a network against hacking attempts.


SERVER=$HOST                           # localhost.localdomain (
PORT_NUMBER=25                         # SMTP port.

nmap $SERVER | grep -w "$PORT_NUMBER"  # Is that particular port open?
#              grep -w matches whole words only,
#+             so this wouldn't match port 1025, for example.

exit 0

# 25/tcp     open        smtp

          Network interface configuration and tuning utility.

bash$ ifconfig -a
lo        Link encap:Local Loopback
           inet addr:  Mask:
           UP LOOPBACK RUNNING  MTU:16436  Metric:1
           RX packets:10 errors:0 dropped:0 overruns:0 frame:0
           TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:700 (700.0 b)  TX bytes:700 (700.0 b)

          The ifconfig command is most often used at bootup to set up
          the interfaces, or to shut them down when rebooting.

# Code snippets from /etc/rc.d/init.d/network

# ...

# Check that networking is up.
[ ${NETWORKING} = "no" ] && exit 0

[ -x /sbin/ifconfig ] || exit 0

# ...

for i in $interfaces ; do
  if ifconfig $i 2>/dev/null | grep -q "UP" >/dev/null 2>&1 ; then
    action "Shutting down interface $i: " ./ifdown $i boot
#  The GNU-specific "-q" option to "grep" means "quiet", i.e.,
#+ producing no output.
#  Redirecting output to /dev/null is therefore not strictly necessary.

# ...

echo "Currently active devices:"
echo `/sbin/ifconfig | grep ^[a-z] | awk '{print $1}'`
#                            ^^^^^  should be quoted to prevent globbing.
#  The following also work.
#    echo $(/sbin/ifconfig | awk '/^[a-z]/ { print $1 })'
#    echo $(/sbin/ifconfig | sed -e 's/ .*//')
#  Thanks, S.C., for additional comments.

          See also Example 32-6.

          Show current network statistics and information, such as
          routing tables and active connections. This utility accesses
          information in /proc/net (Chapter 29). See Example 29-4.

          netstat -r is equivalent to route.

bash$ netstat
Active Internet connections (w/o servers)
 Proto Recv-Q Send-Q Local Address           Foreign Address         State

 Active UNIX domain sockets (w/o servers)
 Proto RefCnt Flags       Type       State         I-Node Path
 unix  11     [ ]         DGRAM                    906    /dev/log
 unix  3      [ ]         STREAM     CONNECTED     4514   /tmp/.X11-unix/X0
 unix  3      [ ]         STREAM     CONNECTED     4513
 . . .


   A netstat -lptu shows sockets that are listening to ports, and the
   associated processes. This can be useful for determining whether a
   computer has been hacked or compromised.

          This is the command set for configuring a wireless network. It
          is the wireless equivalent of ifconfig, above.

          General purpose utility for setting up, changing, and
          analyzing IP (Internet Protocol) networks and attached
          devices. This command is part of the iproute2 package.

bash$ ip link show
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast qlen 1000
     link/ether 00:d0:59:ce:af:da brd ff:ff:ff:ff:ff:ff
 3: sit0: <NOARP> mtu 1480 qdisc noop
     link/sit brd

bash$ ip route list dev lo  scope link

          Or, in a script:

# Script by Juan Nicolas Ruiz
# Used with his kind permission.

# Setting up (and stopping) a GRE tunnel.

# --- ---


/sbin/ip tunnel add netb mode gre remote $REMOTE_IP \
  local $LOCAL_IP ttl 255
/sbin/ip addr add $OTHER_IFACE dev netb
/sbin/ip link set netb up
/sbin/ip route add $REMOTE_NET dev netb

exit 0  #############################################

# --- ---


/sbin/ip route del $REMOTE_NET dev netb
/sbin/ip link set netb down
/sbin/ip tunnel del netb

exit 0

          Show info about or make changes to the kernel routing table.

bash$ route
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
 pm3-67.bozosisp *      UH       40 0          0 ppp0       *            U        40 0          0 lo
 default         pm3-67.bozosisp         UG       40 0          0 ppp0

          The iptables command set is a packet filtering tool used
          mainly for such security purposes as setting up network
          firewalls. This is a complex tool, and a detailed explanation
          of its use is beyond the scope of this document. Oskar
          Andreasson's tutorial is a reasonable starting point.

          See also shutting down iptables and Example 30-2.

          Check network and system configuration. This command lists and
          manages the network and system services started at bootup in
          the /etc/rc?.d directory.

          Originally a port from IRIX to Red Hat Linux, chkconfig may
          not be part of the core installation of some Linux flavors.

bash$ chkconfig --list
atd             0:off   1:off   2:off   3:on    4:on    5:on    6:off
 rwhod           0:off   1:off   2:off   3:off   4:off   5:off   6:off

          Network packet "sniffer." This is a tool for analyzing and
          troubleshooting traffic on a network by dumping packet headers
          that match specified criteria.

          Dump ip packet traffic between hosts bozoville and caduceus:

bash$ tcpdump ip host bozoville and caduceus

          Of course, the output of tcpdump can be parsed with certain of
          the previously discussed text processing utilities.


          Mount a filesystem, usually on an external device, such as a
          floppy or CDROM. The file /etc/fstab provides a handy listing
          of available filesystems, partitions, and devices, including
          options, that may be automatically or manually mounted. The
          file /etc/mtab shows the currently mounted filesystems and
          partitions (including the virtual ones, such as /proc).

          mount -a mounts all filesystems and partitions listed in
          /etc/fstab, except those with a noauto option. At bootup, a
          startup script in /etc/rc.d (rc.sysinit or something similar)
          invokes this to get everything mounted.

mount -t iso9660 /dev/cdrom /mnt/cdrom
# Mounts CD ROM. ISO 9660 is a standard CD ROM filesystem.
mount /mnt/cdrom
# Shortcut, if /mnt/cdrom listed in /etc/fstab

          The versatile mount command can even mount an ordinary file on
          a block device, and the file will act as if it were a
          filesystem. Mount accomplishes that by associating the file
          with a loopback device. One application of this is to mount
          and examine an ISO9660 filesystem image before burning it onto
          a CDR. [85]

          Example 17-7. Checking a CD image

# As root...

mkdir /mnt/cdtest  # Prepare a mount point, if not already there.

mount -r -t iso9660 -o loop cd-image.iso /mnt/cdtest   # Mount the image.
#                  "-o loop" option equivalent to "losetup /dev/loop0"
cd /mnt/cdtest     # Now, check the image.
ls -alR            # List the files in the directory tree there.
                   # And so forth.

          Unmount a currently mounted filesystem. Before physically
          removing a previously mounted floppy or CDROM disk, the device
          must be umounted, else filesystem corruption may result.

umount /mnt/cdrom
# You may now press the eject button and safely remove the disk.


   The automount utility, if properly installed, can mount and unmount
   floppies or CDROM disks as they are accessed or removed. On
   "multispindle" laptops with swappable floppy and optical drives, this
   can cause problems, however.

          The newer Linux distros have deprecated mount and umount. The
          successor, for command-line mounting of removable storage
          devices, is gnome-mount. It can take the -d option to mount a
          device file by its listing in /dev.

          For example, to mount a USB flash drive:

bash$ gnome-mount -d /dev/sda1
gnome-mount 0.4

bash$ df
. . .
 /dev/sda1                63584     12034     51550  19% /media/disk

          Forces an immediate write of all updated data from buffers to
          hard drive (synchronize drive with buffers). While not
          strictly necessary, a sync assures the sys admin or user that
          the data just changed will survive a sudden power failure. In
          the olden days, a sync; sync (twice, just to make absolutely
          sure) was a useful precautionary measure before a system

          At times, you may wish to force an immediate buffer flush, as
          when securely deleting a file (see Example 16-60) or when the
          lights begin to flicker.

          Sets up and configures loopback devices.

          Example 17-8. Creating a filesystem in a file

SIZE=1000000  # 1 meg

head -c $SIZE < /dev/zero > file  # Set up file of designated size.
losetup /dev/loop0 file           # Set it up as loopback device.
mke2fs /dev/loop0                 # Create filesystem.
mount -o loop /dev/loop0 /mnt     # Mount it.

# Thanks, S.C.

          Creates a swap partition or file. The swap area must
          subsequently be enabled with swapon.

   swapon, swapoff
          Enable / disable swap partitition or file. These commands
          usually take effect at bootup and shutdown.

          Create a Linux ext2 filesystem. This command must be invoked
          as root.

          Example 17-9. Adding a new hard drive


# Adding a second hard drive to system.
# Software configuration. Assumes hardware already mounted.
# From an article by the author of the ABS Guide.
# In issue #38 of _Linux Gazette_,

ROOT_UID=0     # This script must be run as root.
E_NOTROOT=67   # Non-root exit error.

if [ "$UID" -ne "$ROOT_UID" ]
  echo "Must be root to run this script."
  exit $E_NOTROOT

# Use with extreme caution!
# If something goes wrong, you may wipe out your current filesystem.

NEWDISK=/dev/hdb         # Assumes /dev/hdb vacant. Check!
MOUNTPOINT=/mnt/newdisk  # Or choose another mount point.

fdisk $NEWDISK
mke2fs -cv $NEWDISK1   # Check for bad blocks (verbose output).
#  Note:           ^     /dev/hdb1, *not* /dev/hdb!
chmod 777 $MOUNTPOINT  # Makes new drive accessible to all users.

# Now, test ...
# mount -t ext2 /dev/hdb1 /mnt/newdisk
# Try creating a directory.
# If it works, umount it, and proceed.

# Final step:
# Add the following line to /etc/fstab.
# /dev/hdb1  /mnt/newdisk  ext2  defaults  1 1


          See also Example 17-8 and Example 31-3.

          Tune ext2 filesystem. May be used to change filesystem
          parameters, such as maximum mount count. This must be invoked
          as root.


   This is an extremely dangerous command. Use it at your own risk, as
   you may inadvertently destroy your filesystem.

          Dump (list to stdout) very verbose filesystem info. This must
          be invoked as root.

root# dumpe2fs /dev/hda7 | grep 'ount count'
dumpe2fs 1.19, 13-Jul-2000 for EXT2 FS 0.5b, 95/08/09
 Mount count:              6
 Maximum mount count:      20

          List or change hard disk parameters. This command must be
          invoked as root, and it may be dangerous if misused.

          Create or change a partition table on a storage device,
          usually a hard drive. This command must be invoked as root.


   Use this command with extreme caution. If something goes wrong, you
   may destroy an existing filesystem.

   fsck, e2fsck, debugfs
          Filesystem check, repair, and debug command set.

          fsck: a front end for checking a UNIX filesystem (may invoke
          other utilities). The actual filesystem type generally
          defaults to ext2.

          e2fsck: ext2 filesystem checker.

          debugfs: ext2 filesystem debugger. One of the uses of this
          versatile, but dangerous command is to (attempt to) recover
          deleted files. For advanced users only!


   All of these should be invoked as root, and they can damage or
   destroy a filesystem if misused.

          Checks for bad blocks (physical media flaws) on a storage
          device. This command finds use when formatting a newly
          installed hard drive or testing the integrity of backup media.
          [86] As an example, badblocks /dev/fd0 tests a floppy disk.

          The badblocks command may be invoked destructively (overwrite
          all data) or in non-destructive read-only mode. If root user
          owns the device to be tested, as is generally the case, then
          root must invoke this command.

   lsusb, usbmodules
          The lsusb command lists all USB (Universal Serial Bus) buses
          and the devices hooked up to them.

          The usbmodules command outputs information about the driver
          modules for connected USB devices.

bash$ lsusb
Bus 001 Device 001: ID 0000:0000
 Device Descriptor:
   bLength                18
   bDescriptorType         1
   bcdUSB               1.00
   bDeviceClass            9 Hub
   bDeviceSubClass         0
   bDeviceProtocol         0
   bMaxPacketSize0         8
   idVendor           0x0000
   idProduct          0x0000

   . . .

          Lists pci busses present.

bash$ lspci
00:00.0 Host bridge: Intel Corporation 82845 845
 (Brookdale) Chipset Host Bridge (rev 04)
 00:01.0 PCI bridge: Intel Corporation 82845 845
 (Brookdale) Chipset AGP Bridge (rev 04)
 00:1d.0 USB Controller: Intel Corporation 82801CA/CAM USB (Hub #1) (rev 02)
 00:1d.1 USB Controller: Intel Corporation 82801CA/CAM USB (Hub #2) (rev 02)
 00:1d.2 USB Controller: Intel Corporation 82801CA/CAM USB (Hub #3) (rev 02)
 00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev 42)

   . . .

          Creates a boot floppy which can be used to bring up the system
          if, for example, the MBR (master boot record) becomes
          corrupted. Of special interest is the --iso option, which uses
          mkisofs to create a bootable ISO9660 filesystem image suitable
          for burning a bootable CDR.

          The mkbootdisk command is actually a Bash script, written by
          Erik Troan, in the /sbin directory.

          Creates an ISO9660 filesystem suitable for a CDR image.

          CHange ROOT directory. Normally commands are fetched from
          $PATH, relative to /, the default root directory. This changes
          the root directory to a different one (and also changes the
          working directory to there). This is useful for security
          purposes, for instance when the system administrator wishes to
          restrict certain users, such as those telnetting in, to a
          secured portion of the filesystem (this is sometimes referred
          to as confining a guest user to a "chroot jail"). Note that
          after a chroot, the execution path for system binaries is no
          longer valid.

          A chroot /opt would cause references to /usr/bin to be
          translated to /opt/usr/bin. Likewise, chroot /aaa/bbb /bin/ls
          would redirect future instances of ls to /aaa/bbb as the base
          directory, rather than / as is normally the case. An alias XX
          'chroot /aaa/bbb ls' in a user's ~/.bashrc effectively
          restricts which portion of the filesystem she may run command
          "XX" on.

          The chroot command is also handy when running from an
          emergency boot floppy (chroot to /dev/fd0), or as an option to
          lilo when recovering from a system crash. Other uses include
          installation from a different filesystem (an rpm option) or
          running a readonly filesystem from a CD ROM. Invoke only as
          root, and use with care.


   It might be necessary to copy certain system files to a chrooted
   directory, since the normal $PATH can no longer be relied upon.

          This utility is part of the procmail package
          ([] It creates a
          lock file, a semaphore that controls access to a file, device,
          or resource.

   Definition: A semaphore is a flag or signal. (The usage originated in
   railroading, where a colored flag, lantern, or striped movable arm
   semaphore indicated whether a particular track was in use and
   therefore unavailable for another train.) A UNIX process can check
   the appropriate semaphore to determine whether a particular resource
   is available/accessible.

          The lock file serves as a flag that this particular file,
          device, or resource is in use by a process (and is therefore
          "busy"). The presence of a lock file permits only restricted
          access (or no access) to other processes.

lockfile /home/bozo/lockfiles/$0.lock
# Creates a write-protected lockfile prefixed with the name of the script.

lockfile /home/bozo/lockfiles/${0##*/}.lock
# A safer version of the above, as pointed out by E. Choroba.

          Lock files are used in such applications as protecting system
          mail folders from simultaneously being changed by multiple
          users, indicating that a modem port is being accessed, and
          showing that an instance of Firefox is using its cache.
          Scripts may check for the existence of a lock file created by
          a certain process to check if that process is running. Note
          that if a script attempts to create a lock file that already
          exists, the script will likely hang.

          Normally, applications create and check for lock files in the
          /var/lock directory. [87] A script can test for the presence
          of a lock file by something like the following.

# Application "xyzip" created lock file "/var/lock/xyzip.lock".

if [ -e "/var/lock/$appname.lock" ]
then   #+ Prevent other programs & scripts
       #  from accessing files/resources used by xyzip.

          Much less useful than the lockfile command is flock. It sets
          an "advisory" lock on a file and then executes a command while
          the lock is on. This is to prevent any other process from
          setting a lock on that file until completion of the specified

flock $0 cat $0 > lockfile__$0
#  Set a lock on the script the above line appears in,
#+ while listing the script to stdout.


   Unlike lockfile, flock does not automatically create a lock file.

          Creates block or character device files (may be necessary when
          installing new hardware on the system). The MAKEDEV utility
          has virtually all of the functionality of mknod, and is easier
          to use.

          Utility for creating device files. It must be run as root, and
          in the /dev directory. It is a sort of advanced version of

          Automatically deletes files which have not been accessed
          within a specified period of time. Usually invoked by cron to
          remove stale log files.


   dump, restore
          The dump command is an elaborate filesystem backup utility,
          generally used on larger installations and networks. [88] It
          reads raw disk partitions and writes a backup file in a binary
          format. Files to be backed up may be saved to a variety of
          storage media, including disks and tape drives. The restore
          command restores backups made with dump.

          Perform a low-level format on a floppy disk (/dev/fd0*).

   System Resources

          Sets an upper limit on use of system resources. Usually
          invoked with the -f option, which sets a limit on file size
          (ulimit -f 1000 limits files to 1 meg maximum). [89] The -t
          option limits the coredump size (ulimit -c 0 eliminates
          coredumps). Normally, the value of ulimit would be set in
          /etc/profile and/or ~/.bash_profile (see Appendix G).


   Judicious use of ulimit can protect a system against the dreaded fork
# This script is for illustrative purposes only.
# Run it at your own peril -- it WILL freeze your system.

while true  #  Endless loop.
  $0 &      #  This script invokes itself . . .
            #+ forks an infinite number of times . . .
            #+ until the system freezes up because all resources exhausted.
done        #  This is the notorious "sorcerer's appentice" scenario.

exit 0      #  Will not exit here, because this script will never terminate.

   A ulimit -Hu XX (where XX is the user process limit) in /etc/profile
   would abort this script when it exceeded the preset limit.

          Display user or group disk quotas.

          Set user or group disk quotas from the command-line.

          User file creation permissions mask. Limit the default file
          attributes for a particular user. All files created by that
          user take on the attributes specified by umask. The (octal)
          value passed to umask defines the file permissions disabled.
          For example, umask 022 ensures that new files will have at
          most 755 permissions (777 NAND 022). [90] Of course, the user
          may later change the attributes of particular files with
          chmod. The usual practice is to set the value of umask in
          /etc/profile and/or ~/.bash_profile (see Appendix G).

          Example 17-10. Using umask to hide an output file from prying

# Same as "" script, but writes output to "secure" file.

# Usage: ./ filename
# or     ./ <filename
# or     ./ and supply keyboard input (stdin)

umask 177               #  File creation mask.
                        #  Files created by this script
                        #+ will have 600 permissions.

OUTFILE=decrypted.txt   #  Results output to file "decrypted.txt"
                        #+ which can only be read/written
                        #  by invoker of script (or root).

cat "$@" | tr 'a-zA-Z' 'n-za-mN-ZA-M' > $OUTFILE
#    ^^ Input from stdin or a file.   ^^^^^^^^^^ Output redirected to file.

exit 0

          Get info about or make changes to root device, swap space, or
          video mode. The functionality of rdev has generally been taken
          over by lilo, but rdev remains useful for setting up a ram
          disk. This is a dangerous command, if misused.


          List installed kernel modules.

bash$ lsmod
Module                  Size  Used by
 autofs                  9456   2 (autoclean)
 opl3                   11376   0
 serial_cs               5456   0 (unused)
 sb                     34752   0
 uart401                 6384   0 [sb]
 sound                  58368   0 [opl3 sb uart401]
 soundlow                 464   0 [sound]
 soundcore               2800   6 [sb sound]
 ds                      6448   2 [serial_cs]
 i82365                 22928   2
 pcmcia_core            45984   0 [serial_cs ds i82365]


   Doing a cat /proc/modules gives the same information.

          Force installation of a kernel module (use modprobe instead,
          when possible). Must be invoked as root.

          Force unloading of a kernel module. Must be invoked as root.

          Module loader that is normally invoked automatically in a
          startup script. Must be invoked as root.

          Creates module dependency file. Usually invoked from a startup

          Output information about a loadable module.

bash$ modinfo hid
filename:    /lib/modules/2.4.20-6/kernel/drivers/usb/hid.o
 description: "USB HID support drivers"
 author:      "Andreas Gal, Vojtech Pavlik <>"
 license:     "GPL"


          Runs a program or script with certain environmental variables
          set or changed (without changing the overall system
          environment). The [varname=xxx] permits changing the
          environmental variable varname for the duration of the script.
          With no options specified, this command lists all the
          environmental variable settings. [91]


   The first line of a script (the "sha-bang" line) may use env when the
   path to the shell or interpreter is unknown.
#! /usr/bin/env perl

print "This Perl script will run,\n";
print "even when I don't know where to find Perl.\n";

# Good for portable cross-platform scripts,
# where the Perl binaries may not be in the expected place.
# Thanks, S.C.

   Or even ...

#!/bin/env bash
# Queries the $PATH enviromental variable for the location of bash.
# Therefore ...
# This script will run where Bash is not in its usual place, in /bin.

          Show shared lib dependencies for an executable file.

bash$ ldd /bin/ls => /lib/ (0x4000c000)
/lib/ => /lib/ (0x80000000)

          Run a command repeatedly, at specified time intervals.

          The default is two-second intervals, but this may be changed
          with the -n option.

watch -n 5 tail /var/log/messages
# Shows tail end of system log, /var/log/messages, every five seconds.


   Unfortunately, piping the output of watch command to grep does not

          Remove the debugging symbolic references from an executable
          binary. This decreases its size, but makes debugging it

          This command often occurs in a Makefile, but rarely in a shell

          List symbols in an unstripped compiled binary.

          Remote distribution client: synchronizes, clones, or backs up
          a file system on a remote server.

17.1. Analyzing a System Script

   Using our knowledge of administrative commands, let us examine a
   system script. One of the shortest and simplest to understand scripts
   is "killall," [92] used to suspend running processes at system

   Example 17-11. killall, from /etc/rc.d/init.d

# --> Comments added by the author of this document marked by "# -->".

# --> This is part of the 'rc' script package
# --> by Miquel van Smoorenburg, <>.

# --> This particular script seems to be Red Hat / FC specific
# --> (may not be present in other distributions).

#  Bring down all unneeded services that are still running
#+ (there shouldn't be any, so this is just a sanity check)

for i in /var/lock/subsys/*; do
        # --> Standard for/in loop, but since "do" is on same line,
        # --> it is necessary to add ";".
        # Check if the script is there.
        [ ! -f $i ] && continue
        # --> This is a clever use of an "and list", equivalent to:
        # --> if [ ! -f "$i" ]; then continue

        # Get the subsystem name.
        # --> Match variable name, which, in this case, is the file name.
        # --> This is the exact equivalent of subsys=`basename $i`.

        # -->  It gets it from the lock file name
        # -->+ (if there is a lock file,
        # -->+ that's proof the process has been running).
        # -->  See the "lockfile" entry, above.

        # Bring the subsystem down.
        if [ -f /etc/rc.d/init.d/$subsys.init ]; then
           /etc/rc.d/init.d/$subsys.init stop
           /etc/rc.d/init.d/$subsys stop
        # -->  Suspend running jobs and daemons.
        # -->  Note that "stop" is a positional parameter,
        # -->+ not a shell builtin.

   That wasn't so bad. Aside from a little fancy footwork with variable
   matching, there is no new material there.

   Exercise 1. In /etc/rc.d/init.d, analyze the halt script. It is a bit
   longer than killall, but similar in concept. Make a copy of this
   script somewhere in your home directory and experiment with it (do
   not run it as root). Do a simulated run with the -vn flags (sh -vn
   scriptname). Add extensive comments. Change the "action" commands to

   Exercise 2. Look at some of the more complex scripts in
   /etc/rc.d/init.d. See if you can understand parts of them. Follow the
   above procedure to analyze them. For some additional insight, you
   might also examine the file sysvinitfiles in
   /usr/share/doc/initscripts-?.??, which is part of the "initscripts"

Комментариев нет:

Отправить комментарий