Differences

This shows you the differences between two versions of the page.

--- linux:bash [2015/05/11 22:37]
mstraub
+++ linux:bash [2015/08/05 16:04] (current)
mstraub [Stream Redirection]
@@ Line 1: / Line 1: @@
+====== Bash - Bourne-Again SHell ======
+''bash'', the [[https://www.gnu.org/software/bash/bash.html|GNU Bourne-Again SHell]], is a commonly used shell a.k.a. command-line interpreter a.k.a. terminal.
+===== Configuration =====
+A shell can either be a login-shell or a non-login-shell.
+The idea behind this concept is, that the first shell instance after login (**login-shell**) is configured with various configuration files setting variables etc.
+All other shell started from the login-shell are sub-shells (**non-login-shell**). Sub-shells automatically inherit the configuration of its parent, therefore there is no need for reading the configuration files again.
+==== Configuration Files ====
+<code>
+/etc/environment         # one environment variable assignment per line; read at login only
+/etc/profile             # shell script; only executed by login-shell (in theory)
+/etc/bash.bashrc         # shell script; executed by each sub-shell (bash only!)
+</code>
+Global configuration files/scripts for all users
+<code>
+~/.pam_environment       # one environment variable assignment per line; read at login only
+~/.profile               # shell script; only executed by login-shell (in theory)
+~/.bashrc                # shell script; executed by each sub-shell (bash only!)
+</code>
+Configuration for a single user, that overrides global configuration.
+=== Limitations ===
+No login-shell is created and therefore /etc/profile and ~/.profile are not executed when:
+([[http://www.microhowto.info/howto/persistently_set_the_value_of_an_environment_variable_for_a_given_user.html|source]])
+  * executing commands via ssh non-interactively (ssh user@server 'command')
+  * sudo
+  * cron jobs
+/etc/bash.bashrc and ~/.bashrc are bash-specific and do not work with other shells like /bin/sh.
+See details in the documentation for [[https://help.ubuntu.com/community/EnvironmentVariables|Ubuntu environment variables]]
+==== Variables ====
+There are two types of variables:
+ - shell variables, which are only visible in the shell they were defined in
+ - environment variables, which are passed to child processes
+Here is how (un)setting variables works:
+<code bash>
+foo='bar bar'      # define a shell variable (no spaces between variable name and content)
+echo $foo          # print a shell or environment variable
+export foo         # make a shell variable to an environment variable
+export foo=bar     # define an environment variable
+export foo=${foo}:bar # append ':bar' to an environment variable
+export -n foo      # remove the environment variable foo, but keep the shell variable
+unset foo          # remove foo
+</code>
+Bash features a number of predefined variables. See the full list in the section "special variables" in ''man bash''. Here is a small selection:
+<code bash>
+echo $0            # name of the shell or shell script.
+echo $?            # print exit status of the most recently executed foreground pipeline.
+echo $!            # print process ID of the most recently executed background  (asynchronous) command.
+</code>
+To list currently defined variables:
+<code bash>
+export             # list current environment variables (= export -p)
+env                # does the same (but unsorted)
+</code>
+To set paths for the future (for your user) you can put them into e.g. ~/.bashrc:
+<code bash>
+export JBOSS_HOME=/opt/jboss
+export PATH=${PATH}:/opt/bin
+</code>
+The shell builtin command 'source' can then be used to update the environment of the current shell by executing the changed file:
+<code bash>
+source ~/.bashrc
+</code>
+==== Shell Options ====
+Shell options can be set with the built-in ''set'' command. They are boolean flags and can be activated with ''-'' and deactivated with ''+'' (yes, this way round!) as follows:
+<code bash>
+set -o verbose      # (or set -v) activate verbose mode
+set +o verbose      # (or set +v) deactivate verbose mode
+</code>
+The ''verbose'' mode prints the input command again before executing it. Other useful options are ''noclobber'' (avoid overwriting of regular files with stream redirection) or ''noglob'' (disable [[http://tldp.org/LDP/abs/html/globbingref.html|globbing]] - the expansion of the wildcards ? and *).
+Currently set options are stored in the variable ''$-''.
+===== Usage =====
+==== Running Programs ====
+To run/execute a program or script that lies in one of the directories of $PATH simply type its name and press enter.
+<code bash>
+ls
+</code>
+To execute other scripts or directories supply the whole path or prepend ''./'' if it resides in the current working dirctory (''pwd'').
+<code bash>
+/path/to/executable
+./executable_in_current_directory
+</code>
+The bash builtin ''exec'' will replace itself (the shell) with the command to execute. The program will have the same process id (pid) and you do not get the shell back after the command finishes. This can be useful for wrapper scripts around programs or [[http://wiki.bash-hackers.org/commands/builtin/exec|more advanced stuff]].
+<code bash>
+echo "This is a wrapper script"
+# do some vodoo here, probably change the arguments etc.
+exec "myprogram" "$@"
+</code>
+The bash builtins ''source'' and ''.'' execute all lines of a script in the current shell. The script is not required to be executable!
+<code bash>
+source /path/to/myscript.sh
+. /path/to/myscript.sh
+</code>
+With the program ''env'' the set of environment variables when running a program can be manipulated (but not changed globally).
+<code bash>
+env                                    # print environment variables
+env -i env                             # start env with a cleaned / empty environment
+env -u VAR1 -u VAR2 VAR3=X VAR4=Y env  # unset VAR1+2, set VAR3+4
+</code>
+==== Shell Expansion ====
+Have a look at the manual for [[http://www.gnu.org/software/bash/manual/bashref.html#Shell-Expansions|shell expansions]] like the tilde character (~) or brace expansion.
+==== History ====
+The command history for each user is stored in ''~/.bash_history''. It can be viewed with ''history''.
+Live-search is activated by typing CTRL-r and then a part of the command. Typing CTRL-r cycles through matching commands, ENTER executes it.
+These shortcuts can be used to execute recent commands:
+<code bash>
+!!              #run last command executed in bash (= !-1)
+!-2             #run command before last command
+!120            #run command with nr 120 as shown by ''history''
+!commandname    #run last command starting with a certain commandname
+</code>
+==== Exit Code ====
+When a program or script exits it returns an integer, which is called "exit status", "exit code", "return status",..
+By convention 0 stands for success and everything else for some kind of error.
+The exit code is stored in a special variable and can be printed like this:
+<code bash>
+echo $?
+</code>
+==== Escaping ====
+Whitespace and some additional characters are treated special by the shell. If they are not destined for the shell, they must be 'escaped' properly.
+<code>
+$&;(){}[]*?!<>"'
+</code>
+Escaping works by either preceding characters with a backslash or by putting strings in quotes:
+<code bash>
+foo=uname
+echo '$foo'     # prints "$foo"     ' completely preserves the string
+echo "$foo"     # prints "uname"    " preserves string except, dollar ($), backticks (`), backslash (\).
+echo `$foo`     # prints "Linux"    ` executes string as a command (this is called command substitution)
+echo $($foo)    # prints "Linux"    another way to use command substitution
+</code>
+==== Command Chaining ====
+The [[http://www.faqs.org/docs/artu/philosophychapter.html|Unix philosophy]] is: "Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface."
+[[http://www.linuxnix.com/2012/07/23-awesome-less-known-linuxunix-command-chaining-examples.html|Command chaining]] allows us to use these small tools to build bigger ones by executing them in a predefined order or letting execution depend on the success of previously executed commands.
+The following chaining parameters should be used like e.g. ''cmd1 | cmd2''
+<code>
+|           pipe: STDOUT of cmd1 goes into STDIN of cmd2
+;           cmd2 is executed after cmd1
+&&          cmd2 is executed after cmd1 was successful (return code = 0)
+||          cmd2 is only executed, if cmd1 was not successful
+&           cmd2 is executed after cmd1 was started in background
+()          group commands to override operator precedence
+</code>
+==== Stream Redirection ====
+Many programs support reading input from STDIN and writing output to STDOUT or STDERR. This is how the flow of streams can be controlled with [[http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-3.html|redirection]].
+In the example we use ''find'' because it writes to STDOUT (names of all readable files) as well as STDERR (unaccessible files).
+<code>
+> or >     redirect STDOUT     find /etc/apt/ /root > /dev/null
+>          redirect STDERR     find /etc/apt/ /root 2> /dev/null
+&>          redirect both       find /etc/apt/ /root &> /dev/null
+</code>
+Redirecting with a single greater than sign ''>'' causes the target file to be **overwritten**. By replacing it with a double greater than sign ''>>'' in all examples above new content is **appended** to the target.
+<code>
+>> or >>   redirect STDOUT     find /etc/apt/ /root >> /var/log/my.log
+</code>
+Output can also be redirected into another stream instead of directly to a file:
+<code>
+>&1        redirect STDERR into same stream as STDOUT      find /etc/apt/ /root > /dev/null 2>&1
+>&2        redirect STDOUT into same stream as STDERR      find /etc/apt/ /root 2> /dev/null 1>&2
+</code>
+Input redirection:
+<code>
+<           STDIN not from keyboard but from file           sort < file.txt
+<< EOT      STDIN from keyboard, but instead of CTRL-d the character sequence EOT in one line ends input (this is called: HERE document)
+</code>
+To avoid accidential overwriting of files with ''>'', the ''noclobber''-option can be set in bash. This has the same effect as the append-only file attribute (see ''chattr''), except it is valid for every file:
+<code bash>
+set +o noclobber
+</code>
+Also remember the program ''tee'', which acts as a T-junction: STDIN goes to STDOUT and into a file.
+==== Aliases ====
+Aliases are used to define shortcuts to commands invocations - including their parameters - that are used often.
+They are typically defined in a script like ~/.bashrc
+<code bash>
+alias ll='ls -alF'
+alias o='xdg-open'
+</code>
+Call ''alias'' without parameters to see which aliases are currently defined.
+===== Scripting =====
+==== Variables ====
+=== Assignment ===
+Variable assignment uses the ''='' character. There must be **no spaces** inbetween.
+<code bash>
+x=10        # assign with an equal sign without spaces
+x="a b c"   # strings with spaces must be quoted
+x=$(ls -l)  # (or x=`ls -l`)command substitution - variable stores result of command execution
+x=          # set variable to null
+unset x     # unset variable
+</code>
+Variables in bash are global except when specifically defined as ''local''. Local variables shadow global ones.
+<code bash>
+globalVariable=5
+local localVariable=6
+</code>
+=== Using Variables ===
+Prefixing a variable name with ''$'' dereferences it.
+To clarify where the variable name ends curly braces {} can be used.
+Quoting the use of variables preserves whitespace.
+<code bash>
+x="tasty   ice"
+echo $x          # prints "tasty ice"
+echo ${x}cream   # prints "tasty icecream"
+echo "${x}cream" # prints "tasty   icecream"
+</code>
+==== Evaluate Expressions ====
+=== test and [ ===
+''test'' or ''['' is a program used to e.g. evaluate comparisons typically required for if statements.
+It returns either true (return code 0) or false (return code 1).
+<code bash>
+test 10 -eq 10     # long form
+[ 10 -eq 10 ]      # short form (spaces to brackets are mandatory)
+</code>
+Integer comparisons:
+  - -eq, -ge, -le: equal, greater equal, less equal
+  - -ne, -gt, -lt: not equal, greater than, less than
+String comparisons:
+  - =, !=
+  - -z: empty string (length is zero)
+  - -n: non-empty string
+File tests (evaluated for the calling user):
+  - -r, -w, -x: read, write and executable permissions
+  - -e, -s: file exists, file exists and is greater than zero
+  - -nt, -ot: newer / older than
+  - -f, -d, -L/-h: file exists and is a regular file, directory or link
+  - -c, -b, -p, -S: file exists and is a character device, block device, pipe, socket
+Note, that most tests will return true for empty input, which may not be the expected result:
+<code bash>
+x=                                                               # x is an empty string
+[ -e $x ] && echo file $x exists || echo file $x does not exist  # output: 'file exists'
+</code>
+Expression can be grouped with parentheses () (which must be escaped!), negated with ! and combined with''-o'' (or) and ''-a'' (and)
+<code bash>
+x=5
+[ $x -gt 2 -a ! $x -eq 10 ]; echo $?
+[ $x -gt 2 -a \( ! $x -eq 10 \) ]; echo $?   # with grouping
+</code>
+== Other Comparison and Arithmetic Functions ==
+Apart from ''test'' (''['') there is an extended test command ''%%[[%%'' available in bash. Read more about it
+[[http://tldp.org/LDP/abs/html/testconstructs.html|here]] and [[http://www.ibm.com/developerworks/opensource/library/l-bash-test/index.html|here]].
+=== Integer Arithmetics ===
+== bc ==
+''bc'' is an arbitrary precision calculator language.
+<code bash>
+echo "5*4 + 1" | bc
+bc <<< "5*4 + 1"
+</code>
+== let ==
+''let'' is a bash-built-in that can do integer arithmetics. It can evaluate several expressions at once (from left to right). When using double quotes spaces can be used within the expression(s). Other variables are referenced without the dollar-sign.
+<code bash>
+let x=5*4 x=x/=2; echo $x         # several calculations resulting in 10
+a=10
+let x="15 % (a + 2)"; echo $x    # prints 3
+</code>
+== Arithmetic Expansion ==
+Another possibility to do integer arithmetic is the use of ''%%$((%%''. Within the double parentheses spaces and additional parentheses to group expressions can be used freely. Other variables are referenced without the dollar-sign. See [[http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_04.html|here]], [[http://tldp.org/LDP/abs/html/testconstructs.html|here]], and [[http://www.ibm.com/developerworks/opensource/library/l-bash-test/index.html|here]].
+<code bash>
+a=10
+x=$(( 15 % (a + 2) )); echo $x   # prints 3
+</code>
+==== Flow Control ====
+== If else ==
+Multiple line example
+<code bash>
+if [ -f $file ]
+then
+    echo $file "is a regular file"
+elif [ -d $file  ]
+then
+    echo $file "is a directory"
+else
+    echo $file "is neither a file nor a directory"
+fi
+</code>
+One-liner:
+<code bash>
+if [ -f $file ]; then echo $file "is a regular file"; elif [ -d $file  ]; then echo $file "is a directory"; else echo $file "is neither a file nor a directory"; fi
+</code>
+== While / Until ==
+A while loop runs as long as the exit status of the command in the loop head returns 0.
+An until loop runs as long as the exit status of the command in the loop head returns 1.
+Multiple line example
+<code bash>
+i=1
+while [[ $i < 6 ]]; do
+    echo $((i++))
+done
+i=1
+until [[ $i > 5 ]]; do
+    echo $((i++))
+done
+</code>
+One-liner
+<code bash>
+i=1
+while [[ $i < 6 ]]; do echo $((i++)); done
+i=1
+until [[ $i > 5 ]]; do echo $((i++)); done
+</code>
+== For ==
+Multiple line example
+<code bash>
+for i in $(seq 1 5); do
+    echo $i
+done
+</code>
+<code bash>
+for i in var1 var2 var3; do
+    echo $i
+done
+</code>
+One-liner
+<code bash>
+for i in $(seq 1 5); do echo $i; done
+</code>
+== Case ==
+After the first match, case terminates with the exit status of the last command that was executed. So there is no need for ''break''s after each switch.
+<code bash>
+i=5
+case $i in
+)
+     echo five
+     ;;
+     banana)
+     echo monkey
+     ;;
+     *)
+     echo "something else - $i to be precise"
+     ;;
+esac
+</code>
+== Break & Continue ==
+''break'' and ''continue'' can be used to break out of a loop / case or to continue at the next iteration of a loop.
+== Exit ==
+To stop the script immediately and decleare a return code (exit value) use ''exit''.
+<code bash>
+exit 0   # exit with success
+exit 1   # exit with failure (any code except 0)
+</code>
+==== Functions ====
+Bash supports basic [[http://tldp.org/LDP/abs/html/functions.html|functions]] with arguments and an exit status (integer only). The function definition must precede the first call to it (but using a not yet defined function within another function is OK - only when calling the function both must be defined).
+''unset -f'' will get rid of a defined function.
+<code bash>
+function myfunc() {
+    echo "hello"
+}
+function myfuncArgs() {
+    echo $1 $2
+    return 0              # exit status of the function
+}
+myfunc
+myfuncArgs a b c # will print a b, because $3 is not used.
+unset -f myfunc
+</code>
+==== Command sequences ====
+Parentheses ( () ) execute commands in a subshell. The current shell environment is therefore not influenced by the executed commands, e.g. a directory change
+<code bash>
+(cd /tmp; touch file)
+</code>
+Braces ( {} ) execute commands in the current shell context. The braces must be separated with a space and a semicolon before the closing brace is required.
+<code bash>
+{ cd /tmp; touch file; }
+</code>
+Check out [[http://stackoverflow.com/a/8552128|this nice overview of what brackets, braces and parentheses can mean in bash.]]
+==== Scripts ====
+Scripts should be executable files and start with a sha-bang (#!) in the first line. The sha-bang defines which program should execute the script, e.g.
+<code>
+#!/bin/bash                  # using absolute location of program
+#!/usr/bin/env bash          # using platform-independent way to find program
+</code>
+This simple script prints '1234' and uses local/global variables and a function:
+<code bash>
+#!/bin/bash
+#global variable
+x=1
+echo -n $x
+#simple function
+function myFunction() {
+     #local variable
+     local y=2
+     echo -n $y
+     #changing the global variable
+     x=3
+     #creating another global variable
+     z=4
+}
+#invocing function
+myFunction
+echo -n $x
+echo -n $z
+</code>
+More resources:
+[[http://tldp.org/LDP/abs/html/|Advanced Bash-Scripting Guide]]
+[[http://mywiki.wooledge.org/BashGuide|Bash Guide]]
+=== Debugging ===
+With ''sleep'' a script can be paused for a certain amount of time:
+<code bash>
+sleep 10           # sleep for 10 seconds
+sleep 1h 10m 59s   # sleep for 1 hour, 10 minutes and 59 seconds
+</code>
+''time'' can measure the execution time of a program and is available as bash built-in or program (that can display more information) in ''/usr/bin/time''.
+<code bash>
+time program              # print summary of execution time (bash buit-in)
+/usr/bin/time -v program  # print more information with program time
+</code>
+=== Trace Mode ===
+To trace scripts use the bash option 'x', which prints commands and their arguments as they are executed.
+<code bash>
+set -x # enable trace-mode
+var=1
+echo $var
+set +x # disable trace-mode
+</code>

Small heaps of code

User Tools

Site Tools

Differences

Page Tools