Bash tricks
src:
Basic stuffs
ctrl+r: search your history- command with a
spacedon’t go in your history, good if there’s a 🔒password 🔒
# loops
for i in *.png; do
convert $i $i.jpg
done
# magical braces
convert file.{jpg,png}
# expands to
convert file.jpg file.png
# {1..5} expands to {1 2 3 4 5}
for i in {1..100}; do echo $i; done
# $( ) gives the output of a command
# e.g. create a file named file-2022-03-08
touch file-$(date -I)
# !! expands the last command run
sudo !!
# changes to the directory you were last in
cd -
# <( ) process substitution
# treat process output like a file (no more temp file)
diff <(ls) <(ls -a)
# fix command: open the last command you ran in an editor
fc
# tells you if something is a builtin, program or alias
type pingtput to colorize messages in bash scripts
The tput Unix program offers commands to control the current terminal screen. It provides commands to change the terminal cursor position, retrieve terminal information, and change text styles. We can invoke the tput command with the echo command to print various text styles on all Unix-like and Unix-based operating systems.
Look at the following example that prints different text styles:
#!/usr/bin/env bash
bold=$(tput bold)
underline=$(tput smul)
italic=$(tput sitm)
info=$(tput setaf 2)
error=$(tput setaf 160)
warn=$(tput setaf 214)
reset=$(tput sgr0)
echo "${info}INFO${reset}: This is an ${bold}info${reset} message"
echo "${error}ERROR${reset}: This is an ${underline}error${reset} message"
echo "${warn}WARN${reset}: This is a ${italic}warning${reset} message"1) `` vs $()
These two operators do the same thing. Compare these two lines:
$ echo `ls`
$ echo $(ls)Why these two forms existed confused me for a long time.
If you don’t know, both forms substitute the output of the command contained within it into the command.
The principal difference is that nesting is simpler.
Which of these is easier to read (and write)?
$ echo `echo \`echo \\\`echo inside\\\`\`` or:
$ echo $(echo $(echo $(echo inside)))If you’re interested in going deeper, see here or here.
2) globbing vs regexps
Another one that can confuse if never thought about or researched.
While globs and regexps can look similar, they are not the same.
Consider this command:
$ rename -n 's/(.*)/new$1/' *The two asterisks are interpreted in different ways.
The first is ignored by the shell (because it is in quotes), and is interpreted as ‘0 or more characters’ by the rename application. So it’s interpreted as a regular expression.
The second is interpreted by the shell (because it is not in quotes), and gets replaced by a list of all the files in the current working folder. It is interpreted as a glob.
So by looking at man bash can you figure out why these two commands produce different output?
$ ls *
$ ls .*The second looks even more like a regular expression. But it isn’t!
3) Exit Codes
Not everyone knows that every time you run a shell command in bash, an ‘exit code’ is returned to bash.
Generally, if a command ‘succeeds’ you get an error code of 0. If it doesn’t succeed, you get a non-zero code. 1 is a ‘general error’, and others can give you more information (eg which signal killed it, for example).
But these rules don’t always hold:
$ grep not_there /dev/null
$ echo $?$? is a special bash variable that’s set to the exit code of each command after it runs.
Grep uses exit codes to indicate whether it matched or not. I have to look up every time which way round it goes: does finding a match or not return 0?
Grok this and a lot will click into place in what follows.
4) if statements, [ and [[
Here’s another ‘spot the difference’ similar to the backticks one above.
What will this output?
if grep not_there /dev/null
then
echo hi
else
echo lo
figrep’s return code makes code like this work more intuitively as a side effect of its use of exit codes.
Now what will this output?
a) hihi
b) lolo
c) something else
if [ $(grep not_there /dev/null) = '' ]
then
echo -n hi
else
echo -n lo
fi
if [[ $(grep not_there /dev/null) = '' ]]
then
echo -n hi
else
echo -n lo
fiThe difference between [ and [[ was another thing I never really understood. [ is the original form for tests, and then [[ was introduced, which is more flexible and intuitive. In the first if block above, the if statement barfs because the $(grep not_there /dev/null) is evaluated to nothing, resulting in this comparison:
[ = '' ]
which makes no sense. The double bracket form handles this for you.
This is why you occasionally see comparisons like this in bash scripts:
if [ x$(grep not_there /dev/null) = 'x' ]
so that if the command returns nothing it still runs. There’s no need for it, but that’s why it exists.
5) sets
Bash has configurable options which can be set on the fly. I use two of these all the time:
set -e
exits from a script if any command returned a non-zero exit code (see above).
This outputs the commands that get run as they run:
set -x
So a script might start like this:
#!/bin/bash
set -e
set -x
grep not_there /dev/null
echo $?What would that script output?
6) <()
This is my favourite. It’s so under-used, perhaps because it can be initially baffling, but I use it all the time.
It’s similar to $() in that the output of the command inside is re-used.
In this case, though, the output is treated as a file. This file can be used as an argument to commands that take files as an argument.
Confused? Here’s an example.
Have you ever done something like this?
$ grep somestring file1 > /tmp/a
$ grep somestring file2 > /tmp/b
$ diff /tmp/a /tmp/bThat works, but instead you can write:
diff <(grep somestring file1) <(grep somestring file2)Isn’t that neater?
7) Quoting
Quoting’s a knotty subject in bash, as it is in many software contexts.
Firstly, variables in quotes:
A='123' echo "$A" echo '$A'Pretty simple – double quotes dereference variables, while single quotes go literal.
So what will this output?
mkdir -p tmp
cd tmp
touch a
echo "*"
echo '*'Surprised? I was.
8) Top three shortcuts
There are plenty of shortcuts listed in man bash, and it’s not hard to find comprehensive lists. This list consists of the ones I use most often, in order of how often I use them.
Rather than trying to memorize them all, I recommend picking one, and trying to remember to use it until it becomes unconscious. Then take the next one. I’ll skip over the most obvious ones (eg !! – repeat last command, and ~ – your home directory).
**!$**
I use this dozens of times a day. It repeats the last argument of the last command. If you’re working on a file, and can’t be bothered to re-type it command after command it can save a lot of work:
grep somestring /long/path/to/some/file/or/other.txt
vi !$
**`!:1-$`**This bit of magic takes this further. It takes all the arguments to the previous command and drops them in. So:
grep isthere /long/path/to/some/file/or/other.txt
egrep !:1-$
fgrep !:1-$The ! means ‘look at the previous command’, the : is a separator, and the 1 means ‘take the first word’, the - means ‘until’ and the $ means ‘the last word’.
Note: you can achieve the same thing with !*. Knowing the above gives you the control to limit to a specific contiguous subset of arguments, eg with !:2-3.
**:h**
I use this one a lot too. If you put it after a filename, it will change that filename to remove everything up to the folder. Like this:
grep isthere /long/path/to/some/file/or/other.txt
cd !$:hwhich can save a lot of work in the course of the day.
9) startup order
The order in which bash runs startup scripts can cause a lot of head-scratching. I keep this diagram handy (from this great page):
It shows which scripts bash decides to run from the top, based on decisions made about the context bash is running in (which decides the colour to follow).
So if you are in a local (non-remote), non-login, interactive shell (eg when you run bash itself from the command line), you are on the ‘green’ line, and these are the order of files read:
/etc/bash.bashrc
~/.bashrc[bash runs, then terminates]
~/.bash_logoutThis can save you a hell of a lot of time debugging.
10) getopts (cheapci)
If you go deep with bash, you might end up writing chunky utilities in it. If you do, then getting to grips with getopts can pay large dividends.
For fun, I once wrote a script called cheapci which I used to work like a Jenkins job.
The code here implements the reading of the two required, and 14 non-required arguments. Better to learn this than to build up a bunch of bespoke code that can get very messy pretty quickly as your utility grows.
1) ^x^y^
A gem I use all the time.
Ever typed anything like this?
$ **grp somestring somefile**
-bash: grp: command not foundSigh. Hit ‘up’, ‘left’ until at the ‘p’ and type ‘e’ and return.
Or do this:
$ **^rp^rep^**
grep 'somestring' somefile
$One subtlety you may want to note though is:
$ grp rp somefile
$ ^rp^rep^
$ grep rp somefileIf you wanted rep to be searched for, then you’ll need to dig into the man page and use a more powerful history command:
$ grp rp somefile
$ !!:gs/rp/rep
grep rep somefile
$I won’t try and explain this one here…
2) pushd / popd
This one comes in very handy for scripts, especially when operating within a loop.
Let’s say you’re in a for loop moving in and out of folders like this:
for d1 in $(ls -d */)
do
# Store original working directory.
original_wd="$(pwd)" cd "$d1"
for d2 in $(ls -d */)
do
pushd "$d2"
# Do something
popd
done
# Return to original working directory
cd "${original_wd}"
doneYou can rewrite the above using the pushd stack like this:
for d1 in $(ls -d *)
do
pushd "$d1"
for d2 in $(ls -d */)
do
pushd "$d2"
# Do something
popd
done
popd
doneWhich tracks the folders you’ve pushed and popped as you go.
Note that if there’s an error in a pushd you may lose track of the stack and popd too many time. You probably want to set -e in your script as well (see previous post)
There’s also cd -, but that doesn’t ‘stack’ – it just returns you to the previous folder:
cd ~
cd /tmp
cd blah
cd - # Back to /tmp
cd - # Back to 'blah'
cd - # Back to /tmp
cd - # Back to 'blah' ...3) shopt vs set
This one bothered me for a while.
What’s the difference between set and shopt?
sets we saw before, but shopts look very similar. Just inputting shopt shows a bunch of options:
$ shopt
cdable_vars off
cdspell on
checkhash off
checkwinsize on
cmdhist on
compat31 off
dotglob offI found a set of answers here.
Essentially, it looks like it’s a consequence of bash (and other shells) being built on sh, and adding shopt as another way to set extra shell options.
But I’m still unsure… if you know the answer, let me know.
4) Here Docs and Here Strings
‘Here docs’ are files created inline in the shell.
The ‘trick’ is simple. Define a closing word, and the lines between that word and when it appears alone on a line become a file.
Type this:
$ cat > afile << SOMEENDSTRING
> here is a doc
> it has three lines
> SOMEENDSTRING alone on a line will save the doc
> SOMEENDSTRING
$ cat afile
here is a doc
it has three lines
SOMEENDSTRING alone on a line will save the doc
$Notice that:
- the string could be included in the file if it was not ‘alone’ on the line
- the string
SOMEENDSTRINGis more normallyEND, but that is just convention
Lesser known is the ‘here string’:
$ cat > asd <<< 'This file has one line'5) String Variable Manipulation
You may have written code like this before, where you use tools like sed to manipulate strings:
$ VAR='HEADERMy voice is my passwordFOOTER'
$ PASS="$(echo $VAR | sed 's/^HEADER(.*)FOOTER/1/')"
$ echo $PASSBut you may not be aware that this is possible natively in bash.
This means that you can dispense with lots of sed and awk shenanigans.
One way to rewrite the above is:
$ VAR='HEADERMy voice is my passwordFOOTER'
$ PASS="${VAR#HEADER}"
$ PASS="${PASS%FOOTER}"
$ echo $PASS - The
#means ‘match and remove the following pattern from the start of the string’ - The
%means ‘match and remove the following pattern from the end of the string
The second method is twice as fast as the first on my machine. And (to my surprise), it was roughly the same speed as a similar python script.
If you want to use glob patterns that are greedy (see globbing here) then you double up:
$ VAR='HEADERMy voice is my passwordFOOTER'
$ echo ${VAR##HEADER*}
$ echo ${VAR%%*FOOTER}
$6) Variable Defaults
These are very handy for knocking up scripts.
If you have a variable that’s not set, you can ‘default’ them by using this. Create a file called default.sh with these contents
#!/bin/bash
FIRST_ARG="${1:-no_first_arg}"
SECOND_ARG="${2:-no_second_arg}"
THIRD_ARG="${3:-no_third_arg}"
echo ${FIRST_ARG}
echo ${SECOND_ARG}
echo ${THIRD_ARG}Now run chmod +x default.sh and run the script with ./default.sh first second.
Observer how the third argument’s default has been assigned, but not the first two.
You can also assign directly with ${VAR:**=**defaultval} (equals sign, not dash) but note that this won’t work with positional variables in scripts or functions. Try changing the above script to see how it fails.
7) Traps
The trap builtin can be used to ‘catch’ when a signal is sent to your script.
Here’s an example I use in my own [cheapci](https://github.com/ianmiell/cheapci) script:
function cleanup() { rm -rf “{LOCK_FILE}” # get rid of /tmp detritus, leaving anything accessed 2 days ago+ find ”${BUILD_DIR_BASE}”/* -type d -atime +1 | rm -rf echo “cleanup done” } trap cleanup TERM INT QUIT
Any attempt to CTRL-C, CTRL- or terminate the program using the TERM signal will result in cleanup being called first.
Be aware:
- Trap logic can get very tricky (eg handling signal race conditions)
- The KILL signal can’t be trapped in this way
But mostly I’ve used this for ‘cleanups’ like the above, which serve their purpose.
8) Shell Variables
It’s well worth getting to know the standard shell variables available to you. Here are some of my favourites:
RANDOM
Don’t rely on this for your cryptography stack, but you can generate random numbers eg to create temporary files in scripts:
$ echo ${RANDOM}
16313
$ # Not enough digits?
$ echo ${RANDOM}${RANDOM}
113610703
$ NEWFILE=/tmp/newfile_${RANDOM}
$ touch $NEWFILEREPLY
No need to give a variable name for read…
$ read
my input
$ echo ${REPLY}LINENO and SECONDS
Handy for debugging
$ echo ${LINENO} 115 $ echo ${SECONDS}; sleep 1; echo ${SECONDS}; echo $LINENO 174380 174381 116Note that there are two ‘lines’ above, even though you used ; to separate the commands.
TMOUT
You can timeout reads, which can be really handy in some scripts
#!/bin/bash TMOUT=5 echo You have 5 seconds to respond... read echo ${REPLY:-noreply}9) Extglobs
If you’re really knee-deep in bash, then you might want to power up your globbing. You can do this by setting the extglob shell option. Here’s the setup:
shopt -s extglob A="12345678901234567890" B=" ${A} " Now see if you can figure out what each of these does:
echo "B |${B}|" echo "B#+( ) |${B#+( )}|" echo "B#?( ) |${B#?( )}|"
echo "B#*( ) |${B#*( )}|" echo "B##+( )|${B##+( )}|" echo "B##*( )|${B##*( )}|"
echo "B##?( )|${B##?( )}|"Now, potentially useful as it is, it’s hard to think of a situation where you’d absolutely want to do it this way. Normally you’d use a tool better suited to the task (like sed) or just drop bash and go to a ‘proper’ programming language like python.
10) Associative Arrays
Talking of moving to other languages, a rule of thumb I use is that if I need arrays then I drop bash to go to python (I even created a Docker container for a tool to help with this here).
What I didn’t know until I read up on it was that you can have associative arrays in bash.
Type this out for a demo:
$ declare -A MYAA=([one]=1 [two]=2 [three]=3)
$ MYAA[one]="1"
$ MYAA[two]="2"
$ echo $MYAA
$ echo ${MYAA[one]}
$ MYAA[one]="1"
$ WANT=two
$ echo ${MYAA[$WANT]}Note that this is only available in bashes 4.x+.