Sometimes you just need to get stuff done quickly and there’s nary a replacement better than a quick shell one-liner.  Recently I’ve needed to feed some large, multi-variable commands into an external program for processing.  Here’s some simple shell one-liners and loops that have helped me along the way.

Starting Small – Simple For Loops
At the base of all iterative loops is the for loop.  I’m not going to go into too much detail because it’s documented extensively everywhere much better than I will be able to explain it here.  It still warrants basic explanation for folks starting out however.

At its most simple construct we have:

for name [in word ...] do list done

You’re saying for every name or variable in some action do something.  Here’s a practical example using sequential numbers and echo.

for x in $(seq 0 2); do echo $x; done

0
1
2

Above, we designated the variable x to signify an iterative numeric range between 0 and 2 using the seq command and perform the action of echo on each one of the numbers in that range.

Let’s look at another using an external file:  /tmp/people.  This will contain a single list of names, one on each line.

cat /tmp/people 

jane
fred
antoine
marek
bubba

We will now use a for loop to echo this into an English sentence.

for people in $(cat /tmp/people); do echo "$people is my friend."; done

jane is a friend.
fred is a friend.
antoine is a friend.
marek is a friend.
bubba is a friend.

For Loops with Multiple Variables
Let’s look at some more advanced usage that fits more into real-world examples.  I need to create a for loop iterating through a file containing two different sets of variables: MAC addresses and their corresponding hostnames.

In this example I am trying to run a series of Foreman commands to create host entries (IP/DNS entries, DHCP records) for a set of new machines.  I certainly don’t want to do this by hand or in a web UI so I’ll use the Foreman Hammer CLI command.

Here’s the file I want to operate on containing two columns: MAC address and hostname and I want to pass line1 column A along with line1 column B for each command and so on.  There are a lot here so I’ll just show you a few to understand the structure.

cat /tmp/hosts

52:54:00:cb:12:44 cap3.rdu.openstack.example.com
52:54:00:cb:12:55 cap4.rdu.openstack.example.com
52:54:00:cb:12:66 cap5.rdu.openstack.example.com
---- snip ----

Now I’m going to iterate through the file assigning variable: mac and variable: hostname for each line and pass this to the Foreman hammer CLI to create my host IP/DNS/DHCP entries.

cat /tmp/hosts | while read mac hostname ; do hammer host create --architecture x86_64 --build 0 --domain rdu.openstack.example.com --environment production --hostgroup provisioning --name $hostname --mac=$mac ; done

Host created.
Host created.
Host created.
Host created.
-- snip --

Let’s break this down by the numbers: we are sending the stream of our file /tmp/hosts line by line to a pipe and reading it.  We assign mac for column1 line1 and hostname for column2 line1 above.  We can then refer to these variables to complete the rest of our commands.

cat /tmp/hosts | while read var1 var2; do command --var1 $var1 --var2 $var2 ; done

Here is another more similiar example (deserves it’s own blog post) for creating lots of iSCSI LUNs on Netapp very quickly using for loops.

Logical OR Comparisons
Using the shell built-in || operator you can have a command execute only if the first command has a non-zero exit status or fails.  This is infinitely useful in one-liners.  Here’s an example where I want to print SUCCESS/FAIL of pinging a handful of servers:

for host in 01 02 03 04 05; do ping -c2 server$host &>/dev/null && echo server$host SUCCESS || echo server$host FAIL; done

server01 SUCCESS
server02 SUCCESS
server03 SUCCESS
server04 SUCCESS
server05 FAIL

Above I am pinging five hosts, if they are reachable it will echo SUCCESS and if they are down/unavailable it will echo FAIL.  Uh-oh, server05 is down again!

We could also pass a much larger list of hosts from a file:

for host in $(cat /tmp/checkhosts); do ping -c2 $host &>/dev/null && echo $host SUCCESS || echo $host FAIL; done

Loop Counts
You can create an ordered loop that starts and ends based on predefined range or a number of files or targets present (or really anything measurable).  Here’s an example:

We will create a 9 files called document1 through document9

for x in $(seq 1 9); do touch document$x; done

Now let’s print an arbitrary label of “File” next to each file using a loop count (the total amount of items is the number of files in this directory.  We will the * glob character to include any item in the current directory.

i=1 ; for document in * ; do echo "File $((i++)) : $document"; done

File 1 : document1
File 2 : document2
File 3 : document3
File 4 : document4
File 5 : document5
File 6 : document6
File 7 : document7
File 8 : document8
File 9 : document9

Infinite Loops
You can take this a step further and create an infinite loop, stopping it with control + c.  This might be useful if you want to run a repeating command to ensure a machine is alive or set some sort of a quick and dirty timer, it simply iterates from 1 and echoes the number every 10 seconds.

i=1 ; for (( ; ; )); do sleep 10; echo "Counting: $((i++))"; done

Counting: 1
Counting: 2
Counting: 3
^C

Detecting Pass/Fail Effectively
Sometimes you need to query a set of hosts for certain conditions and then have them report something is either expected (ok) or not expected (something is wrong).  Using the operators && and || you can achieve this quickly.

Just like our ping / logical comparison example above let’s do this for checking whether a set of hosts has or does not have configuration files in a certain location.

Objective:  Check each host in a list if they have files located in /etc/sysconfig/network-scripts/ with enp* in the name.  I’ve added \ to break up the line to make it more readable.

for host in $(cat /tmp/myhosts); \
do echo "=== $host ===" ; ssh root@$host \
"ls -lah /etc/sysconfig/network-scripts/ifcfg-enp*" \
1>/dev/null && echo "I'm good" || \
echo "I have no ifcfg-* files for NIC2,3,4"; done 2>/dev/null

Let’s de-construct what is happening.

• For each host in /tmp/myhosts ssh and check whether or not /etc/sysconfig/network-scripts/ifcfg-enp* exists, with the * being a wildcard to match any variation of enp in a filename.
• 1>/devnull after our pass/fail check will filter stdout to /dev/null so we don’t see it (we don’t need to see the actual return of the ls command, only record if it found a match).
• && means run this command if previous is successful in this case it will echo “I’m good”.
• || is the logical or separator, if the previous command was not successful then and only then do something.  In this case it was echo “I have no ifcfg-* files for NIC2,3,4”
• 2>/dev/null filters out stderr, we don’t want to see any output other than our one or two purposeful lines for success or failure.

A logical representation would be:

1. For each of the hosts in my list;
   1. repeat the name of the host
2. SSH to each host and ls for a particular file
   1. If there’s a match then say “I’m good” and quit
   2. If there’s not a match proceed to #3
3. If I got this far I failed and the ls command didn’t find a match
   1. say “I couldn’t find a match” and quit.

End result would look something like below, we first echo the hostname so we know the success or failure status for each machine and then we see the result.

=== b06-h01-1029u.example.com ===
I have no ifcfg-* files for NIC2,3,4
=== b06-h02-1029u.example.com ===
I'm good
=== b06-h03-1029u.example.com ===
I'm good

Remembering History and One-Liners
I add the following snippet to my ~/.bashrc so that it creates a file called ~/.bash_eternal_history so I record and keep every command and one-liner ever typed.

export HISTTIMEFORMAT="%s "
PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND ; }"'echo $$ $USER "$(history 1)" >> ~/.bash_eternal_history'

When It’s Too Much?
Usually if I have to stop to debug several layers of pipes, awk, sed and double and single quotes it’s time to just use Ansible or Python.  While it’s cool to show off a particularly sick masterpiece of a regex one-liner the question you should be asking is am I saving time?  I believe you can do almost anything in shell but should you really?

An example might be copying large sets of SSH public keys in an ad-hoc fashion to many servers.  For me the best way to do this is with the Ansible SSH module but sometimes I’ll do this with a Python tool if it’s a one-off situation.  You can do this in a one-liner but it may become too much to manage, and really you should use config management for this sort of stuff anyway.

Extending Further
This is just a very small set of examples.  I keep my miscellaneous scripts on Github but there’s always a time to use one-liners to accomplish tasks.  You might also take a look here or at any of the many many shell scripting examples and guides on the internet for more ideas.

Do you have some favorite shell one-liners?  Let me know in the comments and I’ll be happy to add them here.