Learning Linux Shell Scripting

Table of Contents

Learning Linux Shell Scripting

Credits

About the Author

Acknowledgments

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the color images of this book

Errata

Piracy

Questions

1. Getting Started and Working with Shell Scripting

Comparison of shells

Tasks done by shell

Working in shell

Learning basic Linux commands

Our first script – Hello World

Compiler and interpreter – difference in process

When not to use scripts

Various directories

Working more effectively with shell – basic commands

Working with permissions

Changing file permissions

Command chmod

Technique one – the symbolic method

Technique two – the numeric method

Setting umask

Setuid

Setgid

Sticky bit

Summary

2. Drilling Deep into Process Management, Job Control, and Automation

Introducing process basics

Monitoring processes using ps

Process management

Process monitoring tools – top, iostat, and vmstat

Understanding at

Understanding crontab

Summary

3. Using Text Processing and Filters in Your Scripts

Text filtering tools

Head and tail

The diff command

The cut command

The paste command

The join command

The uniq command

The comm command

The tr command

IO redirection

File descriptors

Redirection

Brace expansion

Pattern matching with the vi editor

Pattern searching using grep

Summary

4. Working with Commands

Learning shell interpretation of commands

Checking and disabling shell internal commands

The exit status

Command substitution

Command separators

Command1; command2

Command grouping

Logical operators

Command1 & command2

Command1 && command2

Command1 || command2

Pipes

Summary

5. Exploring Expressions and Variables

Understanding variables

Working with environment variables

The local variable and its scope

Exporting variables

Working with read-only variables

Working with command line arguments (special variables, set and shift, getopt)

Understanding set

Understanding shift

Resetting positional parameters

Understanding getopts

Understanding default parameters

Working with arrays

Creating an array and initializing it

Accessing array values

Summary

6. Neat Tricks with Shell Scripting

Interactive Shell scripts – reading user input

Summarizing the read command with options

The here document and the << operator

The here operator with the sort command

The here operator with the wc command

The utility ed and here operator

A script for sending messages to all logged-in users

Using the << here operator for FTP usage and data transfer

Turning off variable substitution

The here string and the <<< operator

File handling

Introducing file handling

Using exec to assign file descriptor (fd) to file

Understanding the opening, writing, and closing of a file

Understanding reading from a file

Understanding reading and writing to a file

Using command read on file descriptor (fd)

Reading from one file and writing to another file

Displaying the file descriptor information from the /proc folder

File handling – reading line by line

Executing the command and storing the results in a file

Summarizing usage of the exec command

Debugging

Debugging mode – disabling the shell (option -n)

Debugging mode – displaying commands (option -v)

Debugging mode – the tracing execution (option -x)

Summarizing the debugging options for the Bash shell

Using the set command

Summary of debugging options for set command

The vi editor setting for debugging

Good practices for Shell scripts

Summary

7. Performing Arithmetic Operations in Shell Scripts

Using a command declare for arithmetic

Listing integers

Using the let command for arithmetic

Using the expr command for arithmetic

Using an arithmetic expansion

Binary, octal, and hex arithmetic operations

A floating-point arithmetic

Summary

8. Automating Decision Making in Scripts

Checking the exit status of commands

Understanding the test command

Using the test command with single brackets

Using the test command with double brackets

String comparison options for the test command

Numerical comparison operators for the test command

File test options for the test command

File testing binary operators

Logical test operators

Conditional constructs – if else

Numerical handling if constructs

Using the command exit and the ? variable

String handling with the if construct

Checking for null values

File handling with the if command

Multiple test commands and if constructs

The if/elif/else command

The null command

Switching case

Implementing simple menus with select

Looping with the for command

Exiting from the current loop iteration with the continue command

Exiting from a loop with a break

Working with the do while loop

Using until

Piping the output of a loop to a Linux command

Running loops in the background

The IFS and loops

Summary

9. Working with Functions

Understanding functions

Displaying functions

Removing functions

Passing arguments or parameters to functions

Sharing the data by many functions

Declaring local variables in functions

Returning information from functions

Returning a word or string from a function

Running functions in the background

Command source and period (.)

Creating a library of functions

Summary

10. Using Advanced Functionality in Scripts

Understanding signals and traps

Using the trap command

Ignoring signals

Resetting signals

Listing traps

Using traps in function

Running scripts or processes even if the user logs out

Creating dialog boxes with the dialog utility

Creating a message box (msgbox)

Creating a message box (msgbox) with a title

The yes/no box (yesno)

The input box (inputbox)

The textbox (textbox)

A password box

The menu box (menu)

The checklist box (checklist)

The radiolist box (radiolist)

The progress meter box (gauge)

Summary

11. System Startup and Customizing a Linux System

System startup, inittab, and run levels

The kernel startup and init process

Understanding run levels

System initialization boot scripts

User initialization scripts

Systemwide settings scripts

User level settings – default files

Summary

12. Pattern Matching and Regular Expressions with sed and awk

The basics of regular expressions

sed – noninteractive stream editor

Understanding sed

Understanding regular expression usage in sed

Addressing in sed

How to modify a file with sed

Printing – the p command

Deleting – the d command

Substitution – the s command

Range of selected lines: the comma

Multiple edits – the e command

Reading from files – the r command

Writing to files – the w command

Appending – the a command

Inserting – the i command

Changing – the c command

Transform – the y command

Quit – the q command

Holding and getting – the h and g commands

Holding and exchanging – the h and x commands

sed scripting

Using awk

The meaning of awk

Using awk

Input from files

Input from commands

How awk works

awk commands from within a file

Records and fields

Records

The record separator

The $0 variable

The NR variable

Fields

Field separators

The input field separator

Patterns and actions

Patterns

Actions

Regular expressions

Writing the awk script file

Using variables in awk

Decision making using an if statement

Using the for loop

Using the while loop

Using the do while loop

Summary

Index

Learning Linux Shell Scripting

Learning Linux Shell Scripting

Copyright © 2015 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: December 2015

Production reference: 1211215

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78528-621-6

www.packtpub.com

Credits

Author

Ganesh Sanjiv Naik

Reviewers

Advait Deo

Eax Melanhovich

Shawn Solomon

Commissioning Editor

Nadeem Bagban

Acquisition Editor

Tushar Gupta

Content Development Editor

Nikhil Potdukhe

Technical Editor

Rupali R. Shrawane

Copy Editor

Charlotte Carneiro

Project Coordinator

Izzat Contractor

Proofreader

Safis Editing

Indexer

Rekha Nair

Production Coordinator

Manu Joseph

Cover Work

Manu Joseph

About the Author

Ganesh Sanjiv Naik is an author, consultant, and corporate trainer for embedded Android, embedded Linux, and Internet of Things related product development.

He completed his computer engineering in 1988. Since then, he has worked in this industry. He has worked on projects including micro-controller based projects to advanced Embedded Android projects. He has more than 20 years of professional experience and project accomplishment in information technology.

Ganesh has a passion and deep desire for teaching. He has trained 1,000 engineers in Linux and Android product development. He has developed a lot of training material as well as curriculum for various universities and training institutes.

He has an interest in spiritual study and practices such as meditation. He is a certified yoga teacher. His hobbies include yoga and martial arts.

He has worked as a corporate trainer for Indian Space Research Organization, Intel, GE, Samsung, Motorola, Penang Skill Development Center (Malaysia), various companies in Singapore as well as various other corporates in India and other countries. He has started a company called Levana Technologies, which works with the Penang Skill Development Center (Malaysia) for consulting and training activities. If you would like to send feedback, suggestions, or corrections in the book, he can be contacted at https://in.linkedin.com/in/naikganesh.

This book is his real-life experience….

He has worked as a consultant and corporate trainer in the following skills:

Internet of Things

Embedded Android, Android internals, and device driver development

USB and PCI device driver development in Linux

Embedded Linux and device driver development

Unix Shell scripting with sed and awk

Embedded C++ and C programming

Operating systems, software engineering, and networking

Problem solving—analysis, reasoning, and solution techniques for software engineers

Acknowledgments

I would like to thank my wife, Vishalakshi, for putting up with my late night writing sessions. Even though Vishalakshi is from microbiology background, on my request, she proofed all the chapters to spot language shortcomings. She patiently read my complete book, and even after not being a programmer, she gave me many valuable suggestions. Without her support and continuous motivation, this book would not have been possible. My professional friend, Kuldeep Vaity, has also read all chapters from a developer's perspective. His feedback was very valuable from technical angles, such as unclear code, insufficient explanation, and similar.

I feel proud to say that the book's cover picture was designed by my daughter, Roopali. She conceptualized the theme and took a lot of effort to properly convey the meaning of the book to readers.

My special thanks to the technical reviewers of the book. Their valuable suggestions have helped me add value to this book.

I would like to thank the entire Packt Publishing team. I would especially like to thank, Aditya Nair and Nikhil Potdukhe of Packt Publishing for the positive, motivating support given to me during the initial period to the end of the completion of this book. Nikhil was very helpful and patient in following up with me for chapters. Due to my busy schedule of consulting and training activities, I was not able to complete my chapters as per schedule; but Nikhil used to follow up very nicely, understandingly, and patiently for the chapter's completion. Without his patient follow up, either I would have completed this book very late, or I would have left the writing work incomplete.

About the Reviewers

Advait Deo has more than 10 years of experience in database domain, has spent time on many aspects of databases until now, starting from Oracle version 8 until 12c. He mainly focuses on database performance tuning, integrating database with front end application, scripting, and automation. He is currently working as a senior database administrator for (world leader in retail business).

Prior to this book, Advait has reviewed Oracle Database 11g R2 Performance Tuning Cookbook, Packt Publishing (ISBN 1849682607). He also publishes some of his work and learnings on his website at http://avdeo.com.

Eax Melanhovich is 27 and lives in Moscow, Russia. Most people don't know his real name or where he works since Eax is concerned about his privacy. Eax is an author of probably the most popular Russian standalone technical blog eax.me. He is also one of the co-hosts of the popular Russian IT podcast devzen.ru. Eax is an Ubuntu user and a functional programming enthusiast.

Shawn Solomon is a technology veteran with a broad background of experience from more than 20 years of pioneering in various technologies. While working in the ISP, VoIP, educational, open source development, and disaster recovery fields, his skillset has adapted and broadened over the years.

www.PacktPub.com

Support files, eBooks, discount offers, and more

For support files and downloads related to your book, please visit www.PacktPub.com.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

https://www2.packtpub.com/books/subscription/packtlib

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can search, access, and read Packt's entire library of books.

Why subscribe?

Fully searchable across every book published by Packt

Copy and paste, print, and bookmark content

On demand and accessible via a web browser

Free access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view 9 entirely free books. Simply use your login credentials for immediate access.

I wish to dedicate this book to my Gurudev His Holiness Dr. Jayant Balaji Athavale. I wish to express gratitude for his guidance, which I have received for, how to become good human being, good professional and a seeker on the path of spiritual progress.

- Ganesh Sanjiv Naik

Preface

Shell scripts are an essential part of any modern operating system, such as UNIX, Linux, Windows, and similar. The scripting language or its syntax may vary from OS to OS; but the fundamental principles remain the same. I first encountered Linux Shell scripts during the development of embedded Linux product development. Shell scripts were initializing the complete product from the basic booting procedure until users logged in and a complete operating system was initialized. Another situation was in the automation of regular activities, such as the build and release management of source codes of very complex products, where more than 10,000 files were a part of a single project. Similarly, another very common requirement comes while using the make utility, which is used to compile and build complex product source codes.

Initially, I had learned scripts to solve practical problems and customize already existing products. This book is the summary of what I have learned over the years in Linux Shell scripting during project development work, consultancy, and corporate trainings and their Q&A sessions.

In this book, you will learn about the very basics of Shell scripting to complex, customized automation. By the end of this book, you will be able to confidently use your own Shell scripts for the real-world problems out there. The idea is to be as practical as possible and give you the look and feel of what real-world scripting looks like.

This book covers bash, the GNU Bourne-Again Shell scripting. You can use the knowledge gained by reading this book for any shell of any of the UNIX flavors or distributions. You will need to take care of a few syntax changes if you are working in other shells, such as Korn, and similar. You should be able to read this book cover to cover, or you can just pick it up and read anything that you might find interesting. But perhaps most importantly, if you have a question about how to solve a particular problem or you need a hint, you will find it easy to find the right solution or something close enough to save your time and energy.

What this book covers

Chapter 1, Getting Started and Working with Shell Scripting, you will learn different ways to write and run Shell scripts. You will also learn ways to handle files and directories, and you will learn about working with permissions.

Chapter 2, Drilling Deep into Process Management, Job Control, and Automation, you will learn about basic process management. You will learn about command ps and job management using commands such as jobs, fg, bg, kill, and pkill. Later on, you will learn about process monitoring tools: top, iostat, vmstat and sar.

Chapter 3, Using Text Processing and Filters in Your Scripts, you will learn about using more, less, head, and tail commands. You will also learn text processing tools such as, cut, paste, comm, and uniq. You will learn about standard input, output, and error. Later on, you will learn about metacharacters and pattern matching using vi and grep.

Chapter 4, Working with Commands, you will learn about how shell interprets any command entered on the command line. You will also learn command substitution, separators, and pipes in detail.

Chapter 5, Exploring Expressions and Variables, you will learn about variables—environment variables. This will include how to export environment variables, set, shift,