How to see what's going on with your Linux system right now
If you're a system administrator responsible for Linux servers, it's essential that you learn the Bash shell well, especially the commands that show what software is running. That's necessary for troubleshooting, obviously, but it is always a good idea to know what's happening on a server for other reasons—not the least of which is security awareness.
Previously, I summarized 16 Linux server monitoring commands you really need to know, what to do when a Linux server keels over, and Linux commands you should never use. Here are some more commands to further your system administration skills—and help you identify the current status of your Linux server. I consider these commands to be fundamentals, whether your servers run on bare iron, a virtual machine, or a container with a lifespan of only a few hours.
To put these in some kind of context, let's follow sysadmin Sammie as she goes through a typical day.
First, Sammie wants to check out exactly what a command does, so she turns to
Yes, I know I’m starting with the most obvious command. There's a reason that one of the knee-jerk put-downs in tech circles is Read the F(ine) Manual (RTFM). The secrets of the universe, or Linux/Unix anyway, are hidden inside the
man searches to search for the manual page for a given command. So, for example,
# man top
shows you in great detail how to run the top command. (While I covered
top in an earlier article, it’s worth a short review: top shows all active processes so you can see the most CPU-intensive tasks running on the server, and it provides a fast and easy way to see if any process is starting to lurch out of control.)
If Sammie needed a quick summary of what a command does, she could run
whatis. But, for the down and dirty of what a command does—including its options and flags, and examples of how to use it—
man is the command to use. And then, too, you won’t have to put up with someone telling you to read the freaking manual again. Because you have.
head and tail
Sammie finds herself staring at a log file that simply keeps scrolling and scrolling. Her answer? Turn to
head and tail, two simple but powerful Linux commands. Head displays the first part of a file, while tail displays its last part. By default, they show the first 10 or last 10 lines.
So what? That's a lot actually!
Say you want to troubleshoot a program that's going wrong. Do you want to read its entire log? Probably not. Often, what matters is the information about the software’s last few actions. So Sammie starts troubleshooting with the command:
# tail /var/log/messages
If something was going awry, Sammie would keep an eye on its log file by using
tail with the
# tail -f /var/log/messages
With this command, every new message that appears in that log file displays at a refresh rate of once per second.
Head isn't as useful, but it can be helpful for peeking at data. For, instance, it may be handy to get a quick look at the first few lines listed in a password file, such as:
# head /etc/passwd
With either command, you're not limited to the first or last 10 lines. The
-n flag lets you display as many lines as you'd like. Thus, to see the last 20 lines in that log file:
#head -n 20 /var/log/syslog
Sammie wants to do more with another log file. She knows another powerful, simple command for looking at files is
cat, which concatenates and prints files. In other words, it displays all the contents of a given file. So, what's the use of that, you say, as you enter…
…and your screen is instantly filled with hundreds or thousands of lines? Lots. Usually you use cat in conjunction with other commands. For example, you can pipe data from cat through the
less commands to view a file's contents more easily:
# cat /etc/passwd | less
Of the pair,
less is more functional. For instance, you can go backward from screen to screen.
Sammie can also use
cat to redirect a file's standard output into a new file. So, to create a time-dated copy of a password file, she would type:
# cat /etc/passwd > passwd07042018
However, the most practical use for
cat is to use it to pipe a file's content through to
grep or another pattern-searching program such as
sed. For instance, to show any error messages about your PCI Express bus or devices, type:
# cat /var/log/kern.log | grep PCIe
So, as you can see, it's very handy to have a cat in your system.
The grep program is so powerful that it is worth a book about itself—and it has one: the grep Pocket Reference. Sammie has a copy, and so should you.
Sammie wants to track down an active process that's eating more CPU cycles than it should. So she uses grep to find a pattern within a line of text. That line can reside in a file or be the output result of another command. Thus, grep is often used to filter data from another program. For example, one favorite way to use grep is in this one liner:
# ps -ef | grep <trouble-making-program-name>
In this case, grep is used with
ps, which displays data about active processes. The
-ef flags tell ps to show all the active processes with standard Linux syntax. (If you were doing this on a BSD system, you'd use
-aux rather than
-ef.) When you pipe these
ps results to
, you include the name of the text (in this case, the misbehaving program you want to know about). That lets you examine the lines of text that are associated with that application, so you can figure out the source of your pain.
Sammie is also fond of
env. This is a useful little command that can run another program in a custom environment without modifying the current shell.
For example, Sammie is logged in as Joe to see what's what with his
bash_profile file. Joe's default editor is
emacs, but Sammie prefers to use
vim. So, she simply enters:
And she can use
vim to edit his file.
More commonly, you use
env without any arguments to see the current environment variables and their values. This includes such variables as HOME, SHELL, and PATH. For example, Sammie knows that if Joe's shell script isn't working as expected, there's a decent chance it's because he's calling a command that isn't in his PATH. Sure enough, that proved to be the case.
df and du
Next, Sammie has some containers that are running out of space. She knows there are many ways to learn how much storage your files are consuming. She likes to use
du (disk usage) and
df (display free disk space) because they're the simplest file system utilities.
To troubleshoot disk space problems, Sammie starts with
df. Even though storage is cheap nowadays, this is still relevant. With the rise of containers, it's becoming more common to run into error messages about running out of free space on the container host. With
df, it's easy to work if you're indeed out of room.
The simplest way to use
df is with the
-h option. This shows you the information in human-readable format. For example, to learn the display file system type, run:
# df -T
In this case, our intrepid sysadmin uses the following command to see how much room her DBA's MySQL databases are taking up:
# df /var/lib/mysql
Sammie follows this up by using the related
du utility, which shows how much space each file takes up. As with
df, you can use it with the
-h flag for results that are more useful to the naked eye. When she wants to know if a file or directory is growing out of control (wait, how did that go from 34 MB to 304 GB?!),
du is the easiest tool.
There's no rest for a sysadmin. Now someone's workstation is hiccuping on the network. Sammie knows she could turn to any number of useful Linux network tools, such as
tcpdump, but she regularly uses one great do-it-all program: ip. This collection of TCP/IP utilities is for networking and traffic control in Linux. Using
ip with the command
ip address displays the server's interfaces and IP addresses.
For example, if you have a container attached to two networks, the
-a flag (which is short for address) shows which interface connects to which network.
# ip a
What's this? Is sjvn connected to a wireless network? That will never do! It's time to knock out that connection with the command:
ip link set dev wlp2s0 down
That will show him! (Ed. Probably not.)
(Image courtesy of the author.)
Sammie also knows
ip can do far more than just show addresses. It includes other networking utilities, such as adding and deleting addresses; managing the routing tables; monitoring the state of devices, addresses, and routes; and creating IPv4 tunnels. Really,
iproute is a powerful utility worth studying in detail.
If your system doesn't have the
ip utility, install it as part of the
Who on the network is running that application? Sammie can find the answer to that question and many more with the
id command. By itself,
id displays the currently logged-in user, the user ID, group ID and name, and other groups the current user belongs to. You can also feed it another user's name to get their data.
So how is this information useful? Say you're in a container and you want to install a program within it. Are you logged in as root for the container or as the admin for an application? It’s frustrating to try to install a program and fail because you don’t have sufficient user rights. Running id before you get started can save you from that annoyance.
Simple commands, big results
Most of these commands are basic. If you want to become a professional sysadmin like Sammie, you must have a good grasp of them.
True, DevOps tools such as Ansible, Chef, and Puppet make it easy to install, implement, monitor, and manage servers. DevOps makes it easy to define server configurations and maintain them with version control. But at their root, you find shell commands.
Sure, there are sexier sysadmin programs, such as the DevOps SaltStack, but these commands are fundamental—and powerful. Once you have a grip on them, you're well on your way to being a kick-butt sysadmin.
This article/content was written by the individual writer identified and does not necessarily reflect the view of Hewlett Packard Enterprise Company.