Sysadmin survival guide

It's hard to predict how different sysadmins jobs will be in the coming years. But here are some guidelines gathered during a sysadmin career spanning more than three decades.

System administration is a tough job with ever-changing requirements. In the future, if you’re just starting your career as a sysadmin, you will undoubtedly use tools that you can’t even imagine today. Applications make increasingly complex decisions, servers turn into virtual servers, services move into the cloud, new languages and protocols appear, security threats keep getting worse, and the staff you support could be down the hall or thousands of miles away.

You will almost always have a lot of responsibility without a lot of control over resources. The demands will be great and the surprises endless. The key to thriving in this challenging field of work is keeping your skills relevant, fostering good work habits that will help you get a lot done, and gaining a sense of accomplishment from your work.

When I started in systems administration, sysadmins were managing user accounts, installing applications, and running backups. We were writing scripts and setting up tasks to run overnight while we slept. Some of us began to manage our networks as well. The applications we managed became more complex. Soon we were dealing with encrypted connections, replacing telnet with SSH, managing keys, learning new languages, and tightening security.

[Also read: 13 ways to tank your IT career | How to make your company more attractive to women | Most valuable certifications for infrastructure pros]

We began setting up and managing firewalls. We doubled down on looking for threats and tightening file permissions. We rolled up our sleeves and got ready to manage new types of file systems—shared file systems, virtual file systems, and clustered files systems. We began playing a role in cybersecurity, looking for flaws in our systems that might permit them to be attacked and looking for signs of intruders.

We started dealing with access governance (managing access on the corporate scale), auditing our servers, and promoting certifications such as ISO 27001 that moved us from doing the right thing to proving that we were doing the right thing. At the same time, services were moving to the cloud, servers were being virtualized, and devices that fit in our pockets began connecting to our networks.

Today’s sysadmins face all of this and more. The constantly evolving IT world requires sysadmins who are also constantly evolving. So what do you do?

Managing your career and keeping your skill set relevant requires some planning. It’s hard to predict how different our jobs will be in another five, 10, or 20 years. But here are some guidelines that have helped me over the past three and a half decades.

Emerging technologies are changing our future. A new world is coming.

Learning is key

Keeping your skills up to date is the single most important thing you can do to stay relevant. Whether you’re aiming to be better at the kind of work you’re already doing or looking to add new skills to your repertoire, the need to continuously improve your skill set is more important than ever. New tools, techniques, and challenges are appearing at an increasing pace. It’s almost a cliché, but still true: If you’re not acquiring new skills, you’re falling behind.

As an example, I’ve added depth to my understanding of how Unix systems collect random data, a key element of cryptography. In the process, I also discovered options that can provide enough truly random numbers to supply an entire data center and some exciting facts about the physics behind how this is done. Good cryptography depends on randomness to keep it from being predictable and, therefore, at risk.

I’ve also taught myself to take advantage of the Unix nohup (no hangup) command to get more work done on my home network.

There are so many ways to stay ahead. You can take formal classes—a week away from your job or every Thursday night at the local community college. You can find an online class or work your way through a good hands-on book.

You can take on some extra tasks at work, maybe shadowing a co-worker or devoting some time to experimenting with new languages and tools on your home network.

Whether you can get your company to pay for classes or have to foot the expenses yourself, you need to keep learning and finding ways to put your new skills to work. If you’re learning on your own, committing to a certain number of hours every week is a very good idea.

Avoid getting stuck

Avoid getting stuck in a position that doesn’t offer growth potential—e.g., managing an application that's practically on life support. Try not to get dropped into a role that you can’t easily hand off when better opportunities come along. Instead, focus on tasks that will help you develop new skills.

My old rule was that the ideal position would have me spending 90 percent of my time doing things that I could nearly do in my sleep (because I was so skilled) and 10 percent doing things that forced me to work hard and pick up new skills. These days, that 10 percent seems far too modest.

One way to stay on top of which skills are most in demand is to periodically check job openings. Find out which skills are sought after, especially for companies and positions that appeal to you.

Document the tasks you manage well enough that you can turn them over to someone else when the opportunity to move on to a more interesting and challenging assignment shows up.

Know when to jump ship—too much of the same thing and no opportunities for growth does not bode well for your career. One of my former colleagues never stayed in a position for more than a year or two unless he was picking up new skills. After I hadn’t seen him for about four years, I was amazed at how many new tools he had mastered.

Welcome problems

While I never look for problems, I have to admit that some of my most significant learning opportunities have come about when I ran into a problem that at first seemed like a brick wall. The denser a problem first appears to be, the more you can expect to gain from solving it. To the extent you can, welcome problems that force you to take a deeper look at how the technology you deal with works. When you fully understand how things work, you’ll also understand how and why they break, how to best test them, and what it takes to fix them.

I once had a script that wouldn’t run because of a ^M in the "#!/bin/bash" line. It took a lot of head scratching to figure it out because everything looked right. The extra character kept the script from working, yet was very easy to overlook. I made regular use of dos2unix after that to be sure this wouldn’t happen again.

Early in my career, I struggled with subnet mask settings and remember the “Aha!” moment when I understood that they didn’t have to break on a byte boundary. This allows subnets of almost any size to be configured.

I also worked with JPG files many years ago and was looking for a way to convert color images to grayscale because many of my customers used large grayscale monitors. Eventually, I broke down and used the Unix od (octal dump) command to figure out how JPG files’ color maps work. I was then able to write a simple program that converted the images from color to grayscale (all during a flight from Maryland to California), taking advantage of the od printout I’d brought along with me, the lack of distractions, and the pad and pencil I kept in my bag.

Develop good troubleshooting skills

Beef up your troubleshooting skills. Ask yourself questions like these: What kind of tools and techniques can help me determine what went wrong? How can I break down a problem? What can I rule out?

Develop a process:

  • What errors or unusual behaviors am I seeing?
  • What evidence am I seeing in the log files?
  • What just changed that might have brought about the problem?
  • How is the system different from similar systems that don’t show the problem?
  • How can the problem be divided into parts? For example, might disk space, memory contention, configuration changes, or network problems be playing a role?
  • What troubleshooting tools do I have on hand to help? Do I know how to examine processes, disk usage, log files, open files, network connections, etc.?
  • Might the problem I’m investigating be a side effect of a larger problem?

Also, be careful to preserve evidence when you can (e.g., if a configuration file has changed, make a copy before fixing it so you can more easily track down how the change happened). It might be very handy if you or someone else runs into the same kind of problem again.

Be proactive

Don’t wait to be told what to do. When you can, take the lead. Notice what’s working and what’s not working. Suggest improvements. Become familiar with how systems and applications work before there are problems and you’ll be in much better shape to deal with them when they’re breaking down in some way. The best way to spot poor system performance is to know what good system performance looks like.

Think security, security, security

Make system security part of everything you do. It’s always important and becoming only more so. Review accounts privileges, close or lock accounts that are no longer needed, and review trust relationships between systems.

Explore your company’s reach

Make a point of understanding your company’s reach—its products and customers. See the big picture and make it part of your landscape. Know what matters most not just to your boss, but his or her boss.

Know your customers

Know who your customers are whether they are internal or external. To the extent possible, tune in to their problems. Track the problems they run into and always give feedback so they don’t feel ignored or overlooked. Keep track of requests and problems your customers report. Use some kind of trouble ticketing system—even if it’s just a spreadsheet. Sysadmins almost always end up juggling a lot of different tasks. Find a way to ensure that nothing important falls through one of those cracks in your busy floor.

Be adaptable

Remember what Darwin said: It’s not necessarily the smartest or the strongest that survive, but the most adaptable. Your repertoire of skills should allow you to fit into many different career slots. Be very good at your sysadmin role, but also strive to be adaptable.

I have worked at companies with as few as four staff members and organizations with many thousands of employees. I have been a do-everything-IT employee and a fill-this-narrow-slot employee. There are advantages to both.

To the extent possible, practice your wizardry in multiple areas. Here are some of the most in-demand skills for sysadmins: 

  • Account and server management
  • File system management
  • Customer service
  • Workload management (tickets, priority tasks)
  • Scripting
  • Troubleshooting
  • Security, compliance, and governance
  • Disaster planning
  • Project management

Try to be good at as many of these basic skills as you can. And be someone who can work under pressure.

Avoid, if you can, getting drenched in organizational politics. Stay focused on what really matters. And learn from anyone with something to teach.

Reach beyond your job

Don’t let your boss, your current position, or your company be your sole locus of professional activity. Develop relationships with other professionals in your field—through conferences, online forums, community groups, etc. Connect with people in similar positions. You can gain a lot from their perspectives, and the contacts can come in handy when you need a new job. 

Probably the single best step I ever took in my career was getting involved with the Sun User Group back in the 1980s. From there, I started writing for SunExpert and then IDG. I met a lot of very interesting technology leaders in the process of helping to organize conferences and writing about the technology that I love. I broadened my horizons and got a better sense of where I could take my career. Essentially, I took responsibility for my professional reputation and my employment outlook.

Pursue your interests

In conclusion: Work isn’t just work. It’s what you do with your life. Strive to do work that’s both interesting and challenging. Find ways to enjoy what you do or move on. You can’t spend your career hating your job and then expect to thrive in it.

This article/content was written by the individual writer identified and does not necessarily reflect the view of Hewlett Packard Enterprise Company.