Linux Shell Scripting Cookbook
Solve real-world shell scripting problems with over 110 simple but incredibly effective recipes
- Master the art of crafting one-liner command sequence to perform tasks such as text processing, digging data from files, and lot more
- Practical problem solving techniques adherent to the latest Linux platform
- Packed with easy-to-follow examples to exercise all the features of the Linux shell scripting language
- Part of Packt’s Cookbook series: Each recipe is a carefully organized sequence of instructions to complete the task as efficiently as possible.
Disk usage hacks
Disk space is a limited resource. We frequently perform disk usage calculation on hard disks or any storage media to find out the free space available on the disk. When free space becomes scarce, we will need to find out large-sized files that are to be deleted or moved in order to create free space. Disk usage manipulations are commonly used in shell scripting contexts. This recipe will illustrate various commands used for disk manipulations and problems where disk usages can be calculated with a variety of options.
Getting ready
df and du are the two significant commands that are used for calculating disk usage in Linux. The command df stands for disk free and du stands for disk usage. Let’s see how we can use them to perform various tasks that involve disk usage calculation.
How to do it…
To find the disk space used by a file (or files), use:
$ du FILENAME1 FILENAME2 . .
For example:
$ du file.txt
4
The result is, by default, shown as size in bytes.
In order to obtain the disk usage for all files inside a directory along with the individual disk usage for each file showed in each line, use:
$ du -a DIRECTORY
-a outputs results for all files in the specified directory or directories recursively.
Running du DIRECTORY will output a similar result, but it will show only the size consumed by subdirectories. However, they do not show the disk usage for each of the files. For printing the disk usage by files, -a is mandatory.
For example:
$ du -a test
4 test/output.txt
4 test/process_log.sh
4 test/pcpu.sh
16 test
An example of using du DIRECTORY is as follows:
$ du test
16 test
There’s more…
Let’s go through additional usage practices for the du command.
Displaying disk usage in KB, MB, or Blocks
By default, the disk usage command displays the total bytes used by a file. A more human-readable format is when disk usage is expressed in standard units KB, MB, or GB. In order to print the disk usage in a display-friendly format, use –h as follows:
du -h FILENAME
For example:
$ du -sh test/pcpu.sh
4.0K test/pcpu.sh
# Multiple file arguments are accepted
Or:
# du -h DIRECTORY
$ du -h hack/
16K hack/
Finding the 10 largest size files from a given directory
Finding large-size files is a regular task we come across. We regularly require to delete those huge size files or move them. We can easily find out large-size files using du and sort commands. The following one-line script can achieve this task:
$ du -ak SOURCE_DIR | sort -nrk 1 | head
Here -a specifies all directories and files. Hence du traverses the SOURCE_DIR and calculates the size of all files. The first column of the output contains the size in Kilobytes since -k is specified and the second column contains the file or folder name.
sort is used to perform numerical sort with column 1 and reverse it. head is used to parse the first 10 lines from the output.
For example:
$ du -ak /home/slynux | sort -nrk 1 | head -n 4
50220 /home/slynux
43296 /home/slynux/.mozilla
43284 /home/slynux/.mozilla/firefox
43276 /home/slynux/.mozilla/firefox/8c22khxc.default
One of the drawbacks of the above one-liner is that it includes directories in the result. However, when we need to find only the largest files and not directories we can improve the one-liner to output only the large-size files as follows:
$ find . -type f -exec du -k {} ; | sort -nrk 1 | head
We used find to filter only files to du rather than allow du to traverse recursively by itself.
Calculating execution time for a command
While testing an application or comparing different algorithms for a given problem, execution time taken by a program is very critical. A good algorithm should execute in minimum amount of time. There are several situations in which we need to monitor the time taken for execution by a program. For example, while learning about sorting algorithms, how do you practically state which algorithm is faster? The answer to this is to calculate the execution time for the same data set. Let’s see how to do it.
How to do it…
time is a command that is available with any UNIX-like operating systems. You can prefix time with the command you want to calculate execution time, for example:
$ time COMMAND
The command will execute and its output will be shown. Along with output, the time command appends the time taken in stderr. An example is as follows:
$ time ls
test.txt
next.txt
real 0m0.008s
user 0m0.001s
sys 0m0.003s
It will show real, user, and system times for execution. The three different times can be defined as follows:
- Real is wall clock time—the time from start to finish of the call. This is all elapsed time including time slices used by other processes and the time that the process spends when blocked (for example, if it is waiting for I/O to complete).
- User is the amount of CPU time spent in user-mode code (outside the kernel) within the process. This is only the actual CPU time used in executing the process. Other processes and the time that the process spends when blocked do not count towards this figure.
- Sys is the amount of CPU time spent in the kernel within the process. This means executing the CPU time spent in system calls within the kernel, as opposed to library code, which is still running in the user space. Like ‘user time’, this is only the CPU time used by the process.
An executable binary of the time command is available at /usr/bin/time as well as a shell built-in named time exists. When we run time, it calls the shell built-in by default. The shell built-in time has limited options. Hence, we should use an absolute path for the executable (/usr/bin/time) for performing additional functionalities.
We can write this time statistics to a file using the -o filename option as follows:
$ /usr/bin/time -o output.txt COMMAND
The filename should always appear after the –o flag.
In order to append the time statistics to a file without overwriting, use the -a flag along with the -o option as follows:
$ /usr/bin/time -a -o output.txt COMMAND
We can also format the time outputs using format strings with the -f option. A format string consists of parameters corresponding to specific options prefixed with %. The format strings for real time, user time, and sys time are as follows:
- Real time – %e
- f User – %U
- f sys – %S
By combining parameter strings, we can create formatted output as follows:
$ /usr/bin/time -f “FORMAT STRING” COMMAND
For example:
$ /usr/bin/time -f “Time: %U” -a -o timing.log uname
Linux
Here %U is the parameter for user time.
When formatted output is produced, the formatted output of the command is written to the standard output and the output of the COMMAND, which is timed, is written to standard error. We can redirect the formatted output using a redirection operator (>) and redirect the time information output using the (2>) error redirection operator. For example:
$ /usr/bin/time -f “Time: %U” uname> command_output.txt 2>time.log
$ cat time.log
Time: 0.00
$ cat command_output.txt
Linux
Many details regarding a process can be collected using the time command. The important details include, exit status, number of signals received, number of context switches made, and so on. Each parameter can be displayed by using a suitable format string.
The following table shows some of the interesting parameters that can be used:
For example, the page size can be displayed using the %Z parameters as follows:
$ /usr/bin/time -f “Page size: %Z bytes” ls> /dev/null
Page size: 4096 bytes
Here the output of the timed command is not required and hence the standard output is directed to the /dev/null device in order to prevent it from writing to the terminal.
