BASHing the Linux console

Posted in Projectblog ∙ 30 mins read

The Unix shell tricks for the Command Line Interface

Important Commands / Apps
Globbing
Compression
Cron jobs
1. Examples
Redirecting input/output/error IO
Command chains
Grep - searching within files
1. Regex for grep and more
2. Examples
Writing / executing bash scripts
1. Examples
If-conditions
1. Operators
2. Examples
Exit codes
Case-conditions
For-loops
1. Examples
Functions
1. Returning Data and Arguments
csvkit data processing

The following writeup contains general bash knowledge I have written down over the years and finally decided to translate into markdown.

More flags and examples are found in the internets.

Important Commands / Apps

Name	Name Description	Description	Flags	Example
ls, ll	list short, list long	Display the contents of the current directory -> dir sizes fake		ls [dir]
		long description	-l	ls -l [dir]
		human-readable file-sizes	-h	ls -h [dir]
		sort by modification time	-t	ls -t [dir]
		all hidden paths/dirs	-a	ls -a [dir]
		all files and dirs, no subdir contents	-d	ls -d [dir]
		all files and dirs, with sub-directories (recursive) similar to “find” or “tree”	-R	ls -R [dir]
		sort by extension	-X	ls -lX [dir]
		sort by size	-S	ls -S [dir]
		reverse sorting	-r	ls -r [dir]
tree		prints graphic representation of files and sub-directories (non-std app)		tree [directory]
		root needs sudo privileges		sudo tree -F /
		tree depth, level of recursion	-L	sudo tree -L 2 /
du	disk usage	print dir content like “ls -lh [dir]”, but with real dir sizes
pwd	print working directory	shows currently set working directory
cd	change directory	changes to a different directory		cd [dir]
../..		root dir
~		logged-in user’s home directory
.		current directory
..		parent directory
cat	concatenate	view contents of single file		cat [File]
		view contents of multi files		cat [File1] [File2]
		viewing a file’s content that doesn’t exit yet creates it.		cat > [File]
cut		removes sections from a text file
cp	copy	copies one or multi files, warning: if dest-file exists then it is overwritten		cp [Src-file] [Dest-file]
		directory copy (recursively)	-r	cp -r [Src-dir] [Dest-dir]
		copy files to dir		cp [Src-file1] [Src-file2] [Dest-dir]
mv	move	move or rename		mv [filename] [dest-dir]
		inform overwrite	-i	mv -i [file] [new_file]
mkdir	make directory	create a new directory		mkdir [newdir]
		new dir within new dir:		mkdir -p [newdir1]/[newdir2]
touch		creates empty file or updates modification date		touch [file1] [file2] [file3]
		avoid create file if not exists	-c	touch -c [file]
echo		print to terminal		echo [string]
		write to file (overwrites previous content)		echo hello > file.txt
wc	word count	count number of words	-w	wc -w [file]
		count number of lines	-l	wc -l [file]
passwd	password	changes a user’s password
rm	remove	delete a file OR directory
		remove dir with sub-dirs	-r	rm -r [myfolder]
		remove with inform y/n prompt	-i	rm -i [myfile]
rmdir	remove directory	deletes a directory, by default only empty dirs		rmdir [directory]
more		view text files one screen at a time (superseded by ‘less’)
less		views text files, allows scrolling up and down a line or page at a time
whereis		display the file path to a specified program and related manual files
head		show the first n lines of a text file		head -n 5 [file]
tail		show the last few lines of a file		tail -n 5 [file]
sort		order line by line (default alphabetical)		sort [file]
		numerical order	-n	sort -n [file]
		reverse order	-r	sort -r [file]
tr	translate	replace or removes characters		tr “[:lower:]” “[:upper:]”
chmod		changes a file permissions (read, write, execute)
grep	g/re/p	search within a file (global regular expression print)
		case sensitive search	-i
		recursive in subfolder files	-r
		counts matches	-c
		invert, show what NOT matches	-v	$ grep -v “[sawgtfixk]” [file]
		extended regex	+ ?	-E	grep -E “e+$” [file]
find		search for file in in home dir		find ~ -name [file]
		search in current dir		find . -name [file]
		find all files ending with number		find ~ -name “*[0-9]”
locate		search for file in index database (update db with sudo updatedb)
man		open manual page of a command		man [command]
info		open info page of a command		info [command]
sed		works on lines (modifies words or other parts of lines, inserts, deletes)
awk		work on records with fields
which		show path of an executable specified in PATH variable		which netcat
expr	expression	integer math (not supported in native bash)		expr 4 + 1
				echo $((4+1))
bc	basic calculator	decimal-place math, default rounded to 0 decimal places		echo “4.1 + 2.3” \| bc
		define decimal places		echo “scale=3; 10/3” \| bc

Globbing

Globbing is using wildcards to select multiple files and is similar to regex (see below).

* any number of any character, including no characters. Example: ls question20*3
? any ONE character. Example: ls question?3
[] class/range of characters. Example: ls file[1-3]
^ excluding. Example: ls file[^a]
[:digit:] digits. Example: ls file[[:digit:]a]
[:alpha:] alphabetical
[:alnum:] alphabetrical + numerical (digits)
[:blank:] space, tabs
[:space:] space, tabs
[:cntrl:] backspace, bell, NAK, escape
[:lower:] lowercase
[:upper:] uppercase
[:punct:] punctuation, period, comma, semicolon, exclamation mark, ampersand
[:print:] all printable symbols = alphabetical + numerical + space
[:xdigit:]hexadecimal

Compression

bzip2, gzip, xy

bzip2 [file]` --> `bunzip2 [file2.bz2]

gzip [file]` –> `gunzip [file3.gz]

xz [file]` --> `unxz [file.xz]

Gzipping deletes the original, gunzipping deletes the zipped file

Compression levels

gzip -1 [file]` or `gzip -9 [file]

1 = weak, 9 = strong

Access files while they are still compressed

Bzip2: bzcat, bzgrep,bzdiff, bzless, bzmore

Gzip: zcat, zgrep, zdiff, zless, zmore

Xy: xycat, xygrep,xydiff, xyless, xymore

tar archives

Crating a tarball archive

tar -cf [new-tar-dir.tar] [file/dir-worked-on 1] [file/dir-worked-on 2]

c = create, f = filename

View contents of tarball

tar -tf [file.tar]

t = view

Add/update file to exiting tarball archive (only works uncompressed)

tar -uf [file.tar] [file/dir-worked-on 1] [file/dir-worked-on 2]

u = update

Extract tarball archive

tar -xf [file.tar]

x = extract

Extract single file of tarball archive

tar -xf [file.tar] [compression/hosts.gz]

x = extract with f = filename inside the tarball

Compressed tar archives (double ending .tar.gz or .tgz)

tar -czf gzip.tar.gz [file/dir-worked-on 1] [file/dir-worked-on 2]

tar -cjf bzip2.tar.bz2 [file/dir-worked-on 1] [file/dir-worked-on 2]

tar -cJf xz.tar.xz [file/dir-worked-on 1] [file/dir-worked-on 2]

c = create, z = gzip, j = bzip2, J = xy, f = filename

Unzipping .tgz

gunzip [archive.tgz]

tar xf [archive.tar]

Managing zip files

zip

zip -9 [zipfile.zip][file-worked-on] # 0-9 compression level

zip -r [zipfile.zip] [file/dir-worked-on 1] [file/dir-worked-on 2]

r = recursively all subfolders

rm -rf [file/dir-worked-on 1]

Unlike the other file formats, zipping does NOT automatically delete the uncompressed file.

unzip

unzip [zipfile.zip]

Cron jobs

crontab -l List all cron jobs

crontab -e Edit list of cron jobs, select editor

Alternative: Open crontab file with nano crontab or vim crontab.

crontab -r Delete the current cron jobs

nano /etc/crontab

* * * * * /path/to/command arg1 arg2

* * * * * /root/backup.txt

	Increment	Range
first *	minutes	[0-59]
second *	hours	[0-23]
third *	day	[0-31]
tourth *	month	[0-12]
fifth *	day of week	[0-7]

Examples

Example 1:

Task: Run backup script on first day of each month at 21h.

0 21 1 * * /path/to/script/backup-script.sh

Example 2:

Task: Run php script every day at 10 AM.

0 10 * * * /path/to/myphpscript.php

Example 3:

Task: Run every hour at minute 0.

0 * * * * /path/to/myphpscript.php

Redirecting input/output/error IO

Keyboard input (stdout, channel 0)

Screen print (stdout, channel 1)

Crate/overwrite a text file with (string) output contents using >

echo "Hello World!" > text.txt

Create/append a text file with (string) output contents using >>

echo "Hello to you too!" >> text.txt

Error (stderr, channel 2)

Create/overwrite a text containing error message using 2>

find /usr games 2> text-error.txt

If no error occurs, nothing will be saved to text file.

Create/append a text containing error message using 2>>

sort /etc 2>> text-error.txt

Redirecting to ‘/dev/null’ bit bucket (suppress output)

sort /etc 2> /dev/null

Substituting keyboard input with file contents

Input data to a command, from file instead of a keyboard. Used with commands that don’t accept file arguments.

cat < text.txt

Translate delete letter l in text

tr -d "l" < text.txt

Combo redirects (channel 1 screen & channel 2 error)

find /usr admin &> newfile.txt

Command chains

Sequential runs in a command line with ;

; links commands together and runs sequentially. Inputs and outputs are separate.

Piping | redirect output to next command as input

Redirect output to next input.

Show contents of passwd file -> make page scrollable

cat /etc/passwd | less

Show all files -> only top 10 -> word count only words

ls -l | head | wc -w

Redirect output only if successful

&& links commands together, but only runs the 2nd command if the 1st succeeds.

csvlook SpotifyData_All.csv && csvstat SpotifyData_All.csv

Grep - searching within files

grep “[regex-term]” [file]

grep “exact-string” /etc/passwd

-i case sensitive search

-r recursive in subfolder files

-c counts matches

-v invert, show what NOT matches

-E extended regex | + ?

Regex for grep and more

Expression	Description
.	any char except newline
[abcABC]	match any char in brackets
[^abcABC]	match any char except the ones in brackets
[a-z]	match any char in range
[^a-z]	match any char except in range
sun\|moon	or
^	start of line
$	end of line
*	Zero or more of the preceding pattern (requires -E)
+	One or more of the preceding pattern (requires -E)
?	Zero or one of the preceding pattern (requires -E)

Examples

1 All lines containing the word cat anywhere on the line.

grep "cat" /usr/share/hunspell/en_US.dic

2 All lines that do not contain (inverse) any of the following characters: sawgtfixk.

grep -v "[sawgtfixk]" /usr/share/hunspell/en_US.dic

3 All lines that start with any 3 letters and the word dig.

grep "^...dig" /usr/share/hunspell/en_US.dic

4 All lines that end with at least one e.

grep -E "e+$" /usr/share/hunspell/en_US.dic

5 All lines that contain one of the following words: org , kay or tuna.

grep -E "org|kay|tuna" /usr/share/hunspell/en_US.dic

6 Number count of lines that start with one or no c followed by the string ati.

grep -cE "^c?ati" /usr/share/hunspell/en_US.dic

Writing / executing bash scripts

All scripts should begin with a shebang, which defines the path to the interpreter (#!/bin/bash).

All scripts should include comments to describe their use (#…).

Edit scripts with cat -n [script.sh] (n=line numbers)

No spaces before/after = assigning variables.
Call variable with $ before variable ($username).
$# env. variable with count of arguments passed.
$? env. variable with error codes (0=no error).
$@ or #* env. variable containing ALL passed arguments.
$1 - $9 = explicit call of positional parameters 1-9
Single quotes (‘ ‘). Literal interpretation (variable substitution won’t work).
Double quotes (“ “). Variable substitution works. Example: “$var” -> varcontent.
Back ticks (` `). Content gets fetched from a sub-shell.
Parenthesis method $(). Content gets fetched from a sub-shell. Example: var=“Today’s date is $(date).”
Variables are always considered type=str -> math doesn’t work out of the box.

echo 'echo "Hello World!"' > new_script.sh

Specify interpreter in first line of script -> comment with shebang! (bash folder)

echo '#!/bin/bash \n echo "Hello World!"' > new_script.sh

Add access permissions to script file + execute script

chmod +x [new_script.sh]

Examples

Single quotes vs double quotes

#!/bin/bash

username=”Carol Smith”
echo "Hello $username!"
echo ‘Hello $username!’

Hello Carol Smith! Hello $username!

Script Arguments

Input: new_script.sh Carol Dave

#!/bin/bash

username1=$1
username2=$2
echo "Hello $username1 and $username2!"

Hello Carol and Dave!

Number of Script Arguments ($#)

Input: new_script.sh Carol Dave

#!/bin/bash

username=$1
echo "Hello $username!"
echo "Number of arguments: $#."

Hello Carol! Number of arguments: 2.

Shell-within-a-shell calculation

#!/bin/bash

model1=87.56
model2=89.20
echo “Total score is $(echo $model1 + $model2 | bc)”
echo “Average score is $(echo ‘scale=2; ($model1 + $model2) / 2’ | bc)”

Converting temperature units (pass °F as parameter)

Input: tempconverter.sh 84

#!/bin/bash

temp_f=$1
temp_f2=$(echo "scale=2; $temp_f - 32" | bc)
temp_c=$(echo "scale=2; $temp_f2 * 5 / 9" | bc)
echo $temp_c

28

Script creating variables from files

Input: pull_temp_files.sh

#!/bin/bash

temp_a=$(cat temps/region_A)
temp_b=$(cat temps/region_B)
temp_c=$(cat temps/region_C)
echo "The three temperatures were $temp_a, $temp_b, and $temp_c"

The three temperatures were 45, 22, and 32

If-conditions

Operators

Scalar comparison operators

-eq equal (=)

-ne not equal (!=)

-gt greater than (>)

-ge greater or equal (>=)

-lt less than (<)

-le less or equal(<=)

String Comparison Operators

abc == abc true

abc == ABC false

1 == 1 true

1+1 == 2 false

Examples

Example 1

If one argument is passed [ $# -eq 1 ], then echo greeting, else raise error, finish if, print $#

Input: new_script.sh Carol

#!/bin/bash

if [ $# -eq 1 ]
then
	username=$1
	echo "Hello $username!"
else
	echo "Please enter only one argument."
fi
echo "Number of arguments: $#."

Hello Carol! Number of arguments: 1.

Example 2

Input: guided1.sh 3 0

#!/bin/bash

fruit1=Apples
fruit2=Oranges

if [ $1 -lt $# ]
then
	echo "This is like comparing $fruit1 and $fruit2!"
elif [ $1 -gt $2 ]
then
	echo "$fruit1 win!"
else
	echo "$fruit2 win!"
fi

Apples win!

Exit codes

Make your own exit codes and end script there

Input: script.sh Carol

#!/bin/bash

# A simple script to greet a single user.
if [ $# -eq 1 ]
then
	username=$1
	echo "Hello $username!"
	exit 0
else
	echo "Please enter only one argument."
	exit 1
fi
echo "Number of arguments: $#."  #never executed, since exited before

Output:

Hello Carol!

echo $? check exit code

0

Case-conditions

Indsead of long chains of if/else-statements, case-statements are simpler to write.

Feed in filename as script argument ($1)

#!/bin/bash

case $1 in
	# Match on all months with OR pipes
	January|March|May|July|September|November)
	echo "It is an odd month!";;
  
	# Match on all even months
	February|April|June|August|October|December)
	echo "It is an even month!";;
  
	# Default case if no match (i.e., else)
	*)
	echo "Not a month!";;
esac

Feed in filename as script argument ($1) cat through the file contents and move or remove the files.

#!/bin/bash

case $(cat $1) in
	*Frankfurt*)
	mv $1 frankfurt/ ;;

	*Berlin*)
	rm $1 ;;

	*Hamburg*)
	mv $1 "SPECIAL_$1" ;;
	
	# Defaultcase if no match
	*)
	echo "No city found" ;;
esac

For-loops

Pattern: `for [indexing_var] in [iterable] do [something] done

Examples

Example 1

#!/bin/bash

FILES="/usr/sbin/accept /usr/sbin/pwck/ usr/sbin/chroot"

for file in $FILES
do
	ls -lh $file
done

Example 2

Loop through parameters and print each parameter

Input: friendly2.sh Carol Dave Henry

#!/bin/bash

# if there are no parameters passed
if [ $# -eq 0 ]		
then
	echo "Please enter at least one user to greet."
	exit 1
else
	for username in $@
	do
		echo "Hello $username!"
	done
	exit 0
fi

Output:

Hello Carol! Hello Dave! Hello Henry!

Example 3

Loop through parameters and check for alphabetical symbols

Input: friendly2.sh Carol Dave Henry

#!/bin/bash

if [ $# -eq 0 ]
then
	echo "Please enter at least one user to greet."
	exit 1
else
	for username in $@
	do
		echo $username | grep "^[A-Za-z]*$" > /dev/null
		# if  the line above produced an error code
		if [ $? -eq 1 ]  
		then
			echo "ERROR: Names must only contain letters."
			exit 2
		else
			echo "Hello $username!"
		fi
	done
	exit 0
fi

Output:

Hello Carol! Hello Dave! Hello Henry!

friendly2.sh 42 Carol Dave Henry

ERROR: Names must only contain letters. echo $? check number of passed arguments 2

Example 4

A script that will take any number of arguments from the user, and print only those arguments which are numbers greater than 10.

#!/bin/bash

for arg in $@
do
	# strip all numerical symbols
	echo $arg | grep "^[0-9]*$" > /dev/null
	
	# if the above produced NO error code
	if [ $? -eq 0 ]								
	then
		if [ arg -lt 10 ]
		then
			echo $arg
		fi
	fi
done

Example 5

A script combining for-loop and case-statements to sort files into folders based on contents or remove them.

#!/bin/bash

# loop through files in folder
for file in model_output/*
do
    # cat into file's content
    case $(cat $file) in

		# Matched tree models are moved
		*"Random Forest"*|*GBM*|*XGBoost*)
		mv $file tree_model/ ;;
		
		# Other Models are deleted/removed
		*KNN*|*Logistic*)
		rm $file ;;

		# Default case if no match (i.e., else)
		*) 
		echo "Unknown model in $file" ;;
    esac
done

Functions

A function with no parameters gets set up with the keyword function and curly braces.

Variables defined within a function are always in global scope, unlike other programming languages. Instead, they have to be declared as local.

The return option is not for returning data, but for error codes (0 = no error, 1-255 = failure). To get data out of a function, the variable can be echoed and/or declared a global variable.

Example 1

function cloud_load () {
	# loop through all files
	for file in output_dir/*results*
	do
		# do something
		echo "Uploading $file to cloud"
	done
}

# call
cloud_load

Example 2

However, it also works without the keyword.

what_day_is_it () {

	# Parse date
	local 	current_day=$(date | cut -d " " -f1)

	echo $current_day
}

# call
what_day_is_it

Returning Data and Arguments

Arguments for functions are handled with $1, $2…$@, $# as with shell scripts. They do not get declared when writing the function, but all get passed as positional parameters.

function return_percentage () {

  # Do calculation on passed parameters
  percent=$(echo "scale=2; 100 * $1 / $2" | bc)

  # print the data
  echo $percent
}

# Call function
return_test=$(return_percentage 410 55)
echo "The percentage is $return_test%"

The percentage is 745.45%

csvkit data processing

csv kit can do csv processing similarl to Pandas data frames.

Install using Python pip

pip install csvkit

in2csv converting files to csvkit

Print excel spreadsheet

in2csv My_data.xlsx

Print excel sheet names with flag -n

in2csv -n My_Data.xlsx

Excel to csv

in2csv My_Data.xlsx > My_Data.csv

Excel sheet to csv

in2csv My_Data.xlsx --sheet "Worksheet1" > My_Data_worksheet1.csv

csvlook csv preview in shell

Print a csv like a markdown table.

csvlook My_data.csv

id	size
118GQ70Sp6pMqn6w1oKuki	7
6S7cr72a7a8RVAXzDCRj6m	7
7h2qWpMJzIVtiP30E8VDW4	7
3KVQFxJ5CWOcbxdpPYdi4o	7
0JjNrI1xmsTfhaiU1R6OVc	7
3HjTcZt29JUHg5m60QhlMw	7

csvstat descriptive statistics

Similar to the pandas command pandas.DataFrame.describe

csvstat My_data.csv

“id” Type of data: Text Contains null values: False Unique values: 24 Longest value: 22 characters Most common values: 118GQ70Sp6pMqn6w1oKuki (1x) 6S7cr72a7a8RVAXzDCRj6m (1x)

csvcut column cutting

Print column names with flag -n

csvcut -n My_Data.csv

Print first, third, fifth columns by positional argument

csvcut -c 1,3,5 My_Data.csv

Print a column by name

csvcut -c "id","size" My_Data.csv

csvgrep search in csv

-m exact row value to filter

-c select column

csvgrep -c "id" -m 5RCPsfzmEpTXMCTNk7wEfQ My_Data.csv

-r regex pattern

-f path to a file/filename

csvstack vertical stacking

csvstack My_data1.csv My_data2.csv > My_data_complete.csv

csvlook My_data_complete.csv

id	size
7JYCpIzpoidDOnnmxmHwtj	6
0mmFibEg5NuULMwTVN2tRU	6
118GQ70Sp6pMqn6w1oKuki	7
6S7cr72a7a8RVAXzDCRj6m	7

Add a group column that traces the original file with flag -g and a group name with flag -n

csvstack -g "data1","data2" -n "source" My_data1.csv My_data2.csv > My_data_complete.csv

csvlook My_data_complete.csv

source	id	size
data1	7JYCpIzpoidDOnnmxmHwtj	6
data1	0mmFibEg5NuULMwTVN2tRU	6
data2	118GQ70Sp6pMqn6w1oKuki	7
data2	6S7cr72a7a8RVAXzDCRj6m	7

sql2csv load from sql database

Executes an SQL query on a large variety of SQL databases (e.g., MS SQL, MySQL, Oracle, PostgreSQL, Sqlite) and outputs result to csv.

Establishing database connection

--db is followed by the database connection string

SQLite: starts with sqlite:/// and ends with .db

Postgres: starts with postgres:/// and no .db

MySQL: starts with mysql:/// and with no .db

Querying against the database

--query is followed by the SQL query string

Use SQL syntax compatible with the database

Write query in one line with no line breaks

Saving the output

> re-directs output to new local csv

sql2csv --db "sqlite:///TwitterDatabase.db" \
        --query "SELECT * FROM Twitter_Size" \
        > Twitter_size.csv

csvsql

Write SQL queries against csv

Applies SQL statements to one or more csv. Creates an in-memory SQL database that temporarily hosts the Suitable for small to medium files only

--query contains the sql statement

csvsql --query "SELECT * FROM Twitter_data LIMIT 1" \
       data/Twitter_data.csv | csvlook

Query joined tables/csv files

The sql query must be written in one line without breaks.

The order of files in the query and the csv files after that must match.

csvsql --query "SELECT * FROM file_a INNER JOIN file_b" file_a.csv file_b.csv

Upload csv to database

Create tables and insert into tables.

--insert csv file to insert into table --db address --no-inference Disable type inference when parsing the input --no-constraints Generate a schema without length limits or null checks

csvsql --no-inference --no-constraints \
       --db "sqlite:///TwitterDatabase.db" \
       --insert Twitter_data.csv

Important Commands / Apps

Globbing

Compression

bzip2, gzip, xy

tar archives

Compressed tar archives (double ending .tar.gz or .tgz)

Managing zip files

Cron jobs

Examples

Redirecting input/output/error IO

Keyboard input (stdout, channel 0)

Screen print (stdout, channel 1)

Error (stderr, channel 2)

Substituting keyboard input with file contents

Combo redirects (channel 1 screen & channel 2 error)

Command chains

Sequential runs in a command line with ;

Piping | redirect output to next command as input

Redirect output only if successful

Grep - searching within files

Regex for grep and more

Examples

Writing / executing bash scripts

Examples

If-conditions

Operators

Examples

Exit codes

Case-conditions

For-loops

Examples

Functions

Returning Data and Arguments

csvkit data processing

in2csv converting files to csvkit

csvlook csv preview in shell

csvstat descriptive statistics

csvcut column cutting

csvgrep search in csv

csvstack vertical stacking

sql2csv load from sql database

Establishing database connection

Querying against the database

Saving the output

csvsql

Write SQL queries against csv

Query joined tables/csv files

Upload csv to database

Templates (for web app):

Error