Ten Cool Coreutils Commands
6 January 2007
While a few commands, such as 'cd' are built directly into bash; many of the most important commands come from coreutils, a GNU package containing over a hundred commands.
Some are the well known commands such as 'ls', 'mv' and 'cat'. While there are some more obscure ones that are not always discovered because there are also more modern programs that can do similar jobs.
These less well known commands often reflect the history of computing where once upon a time you would do everything possible in the Bourne or Bash shell, rather than waiting a long time for program such as emacs or vi to start, which at the time were considered to be memory intensive but now seem lightning fast (now you can even run them from a mobile phone).
Here we will look at ten useful commands that might come in handy one day.
1. tac
One of the key commands is 'cat', short for concatenate, which you can use to add the contents of one file to another, or just to print a file or set of files out to the screen.
cat 1.txt 2.txt > 3.txt # Adds 1 and 2 together as a new file 3.
cat 3.txt # Prints file 3 out to the screen.
However, tac is cat backwards, it lets you concatenate and print files in reverse. It could come in very handy for some uses, for example if you want to reverse a list or a log file.
tac 1.txt > 2.txt # 2 is a reversed copy of 1
2. tee
Often you will pipe the output of one program as the input of another:
ps -e | grep apache
This command will feed to grep the output of ps, thus showing what apache processes are currently running.
Or you may redirect the output of a command to a file:
ps -e | grep apache > apache-processes.txt
However, what if you want to see the results and write the file or files? This is where the tee command is useful:
ps -e | grep bash | tee bash-processes.txt # Print the results to the screen but also to a file.
You can enter more than one filename if you want multiple copies.
3. pr
Most printers these days will attempt to print out anything that you throw at them. However, you still may want to format a file in a certain way before sending it to the printer. For example, you may regularly discuss a log or some records at a meeting and want them to look the same every week.
The pr command gets a text file ready for the printer. For example you may want a set page or column format.
pr +10 -h"Apache Errors" -l25 error_log | lpr -# 5
Reading the pr command from left to right, the options are to start from page 10, then add a header to each page, make each page 25 lines long and lastly the filename. The result is then piped to lpr which will submit the file to the printer with a request for 5 copies.
4. stat
Using 'ls -l' will give a lot of information about a file, enough for me at least. If you are a glutten for punishment, you can use the 'stat' command to get more information:
$ stat stat
File: `stat'
Size: 34684 Blocks: 72 IO Block: 4096 regular file
Device: 302h/770d Inode: 7586763 Links: 1
Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2007-01-06 00:35:15.000000000 +0000
Modify: 2006-09-18 09:50:24.000000000 +0100
Change: 2007-01-04 14:51:47.000000000 +0000
5. yes
'yes' is one of those strange legacy commands, the most comic featured here. 'yes' prints a string and a newline and then repeats until interrupted.
As dumb as it sounds, it does have a couple of uses. Firstly, if you want to turn an interactive command into a non-interactive one.
There are some command line programs which you run and then it asks something like "Are you sure?". For example:
yes 5 | command
This will run command then send the string 5 and a newline (i.e. as if you had pressed return).
It almost goes without saying that you can also use yes to turn your processor up to 100% usage, for example if you are testing some fans or cooling system or if you otherwise want to punish your machine.
6. expand
So you wrote lots of nice Python files, indenting them with tabs. But shock horror, you found out that the current fashion is to use four spaces not tabs!
No problem, the expand command converts tabs to spaces, here we have chosen to use 4 spaces:
expand -4 uncoolscript.py > coolerscript.py
To go from spaces to tabs, you can use 'unexpand' command.
7. split
'split' takes a file and splits it into chunks for you. For example:
split -l 20 access_log part
This command will split the file access_log into chunks of 20 lines each, the name of each file will begin with part.
8. uniq
Sometimes you will have a file that is a long list of items and you want to remove all the repeated lines, or possibly you want to group the items into sets with a number of occurrences.
So to remove all repeated lines:
uniq file.txt
You can also use the output of another command:
cat file.txt | uniq
To include the occurrence count:
uniq -c file.txt
9. wc
wc allows you to count words, lines and bytes. The default is to show all three:
wc ulyss12.txt
32758 267235 1561677 ulyss12.txt
[lines] [words] [bytes]
Using -l just gives you lines:
wc -l ulyss12.txt
32758 ulyss12.txt
Using -w just gives you words:
wc -w ulyss12.txt
267235 ulyss12.txt
10. shred
'rm', the remove command, unlinks a file so the space can be reused. However, files deleted can sometimes be recovered with a bit of persistence and luck.
Our final command is shred. 'shred' overwrites a file repeatedly, making it much harder to recover. This can be useful for personal financial information such as your credit card details.
shred -u mastercard.txt
You can even use shred on a device such as a partition. If you want to completely clear your home partition which happens to be stored on /dev/sda6. Then you would use:
shred /dev/sda6
Alert readers will note that I used -u in the first example but not the second. This because you want to unlink (i.e. rm) a file but not a device - you will want empty the /dev/sda6 to exist.
Now there are some complications to this. Firstly, a complex RAID setup might interpret this as hardware failure and replace the data. Likewise some corporate setups will sync the files with a server. Shred will not remove these copies.
Lastly, some fancy modern journaled filesystems such as reiserfs will have a backup in the journal, you will need to mount the partition in a non- journaled mode for shred to work completely.
There we go, ten commands that you may not have heard of, I hope you can find at least one that is useful in your computing activities.
1 Eddy Mulyono says...
Love 'em.
Thanx for sharing.
Posted at 8:40 a.m. on January 8, 2007
2 tony says...
Thanks for the tip about shread. I'm learning unix and enjoying the shell, but am still trying to wrap my head around uses for all this wonderful exe gems. You're blog is great.
Posted at 6:59 p.m. on January 8, 2007
3 Jason says...
Nice list. Definitely going into my bookmarks.
I've run into more circumstances where I've needed to to a numerical or alphabetical unique sort, which I don't think uniq can do. In this case a sort -u or sort -un would be better:
cat file.txt | sort -u cat file.txt | sort -un
Posted at 5:31 p.m. on January 29, 2007
4 Volatile says...
Nice! Didn't know half of them... expand/unexpand is going to make my life soo much easier!
Posted at 12:43 a.m. on March 7, 2007
5 Mad Hatter says...
tac is definatly a new one to me, pretty cool. I've used knoppix and shred for all laptops I have returned to companies I have quit. Makes sure that some nosy IT guy can't pull up anything.
Posted at 1:51 p.m. on March 21, 2007
6 Jeroen says...
Nice list, thanx!
Being who I am... you have a little typo in your shred command (shread)
And to Mad Hatter, the nosy IT guy would have done that while you where working on it and where connected to the company network, or just take a good look at your server copy of your profile and homeshare. So unless you never connected long to the company network and your sysadmin was already a moron it's not gonna help much towards the goal you mention.
Posted at 8:18 a.m. on March 29, 2007
7 Jeroen says...
Forgot to mention why I like the list, and the mentioning of the source package (CoreUtils):
http://gnuwin32.sourceforge.net/packages/coreutils.htm
They work on Windhose too :)
Posted at 8:20 a.m. on March 29, 2007
8 AJS says...
I always use -n1 with the shred command. This option limits the number of overwrite passes, and one is enough.
Yes, one overwrite pass is enough.
(Please don't quote the Gutmann paper! Many of the assumptions on which it relies have been invalidated since it was written. Voice coil actuators have replaced stepper motors, thus improving the precision of tracking; and data densities have increased by several orders of magnitude, thanks to the more precise tracking.)
If it were really possible reliably to recover overwritten data from magnetic storage devices, somebody by now would have built a storage device which used the phenomenon to increase its capacity. If you can recover data after one overwrite, you can effectively fit two bits into the space of one. Given the history of computers, it is next to certain that (1) such a device would have been economically feasible at some stage in the past, and (2) some manufacturer would have committed heavily to it, probably just after it ceased to be economically feasible.
Posted at 12:57 p.m. on January 15, 2008
9 Greg Gann says...
I have found the commands head and tail useful.
By default, head outputs the first 10 lines of a file, and tail, the last 10. However, you can combine them to produce any section of the file you need to see.
Does your compiler report an error at line 253?
The command:
will display lines 246 through 265 of foo.c
Useful, that.
Posted at 9:01 p.m. on April 14, 2008
10 Greg Gann says...
I have found the commands head and tail useful.
By default, head outputs the first 10 lines of a file, and tail, the last 10. However, you can combine them to produce any section of the file you need to see.
Does your compiler report an error at line 253?
The command:
will display lines 246 through 265 of foo.c
Useful, that.
Posted at 9:10 p.m. on April 14, 2008
11 harsha says...
I'm interested in working with GNU coreutils where i can add new commands in it .
Can anybody help me with some ideas for creating new commands so that i can develop on that .
Posted at 5:49 p.m. on June 6, 2008
12 atul says...
paste is also a cool one... The following is a single line...
$ for ((i=1; i<10; ++i)) ; do echo "$i"; done | paste -s -d '*' 1*2*3*4*5*6*7*8*9 $
It is very useful to put something in BETWEEN two numbers...
passing it to bc obviously illustrates the power of linux ;-)
$ for ((i=1; i<100; ++i)) ; do echo "$i"; done | paste -s -d '*' | bc 933262154439441526816992388562667004907159682643816214685929638952175999932299156089414639761565182862536979208272237582511852109168640000000000000000000000
just some fun ;-)
-- thx atul
Posted at 10:28 a.m. on June 17, 2008
13 Sovok says...
Very nice, thanks. The equivalent to shred on Mac OS X is srm. Overwrites the file once, with 7 DoD compliant passes or 35 times (Gutman algorithm).
Posted at 1:05 p.m. on February 7, 2009
14 Doug says...
AJS: no, one overwrite pass is not necessarily enough.
Your logic doesn't take into account that the recovery of deleted data may be allowed to take vastly longer than normal reads/writes.
The ability to recover multiply-overwritten data at extremely slow speeds does not imply the feasibility (economically or not) to recover that same data at the required very high speeds of normal operation.
Posted at 5:36 p.m. on February 7, 2009
15 someone says...
Please don't encourage the use of "shred". It doesn't work on file systems used on modern Linux systems. It says so right in the man page.
Posted at 7:03 p.m. on February 27, 2009
16 Mike says...
When using shred, it has been recommended to mount the filesystem as ext2, which has no journaling feature.
Posted at 8:28 p.m. on February 27, 2009
17 Caleb Cushing ( xenoterracide ) says...
generally I'm not the spelling police. but dude SPELL CHECK. s/shread/shred/ I can live with typo's out of code... but in code it tends to bother me.
Posted at 5:27 a.m. on March 4, 2009
18 Zeth says...
Oh man, that typo has been there for two years and no one has noticed it.
Thanks Caleb, now fixed.
Posted at 9:42 a.m. on March 4, 2009
19 PhilSchmil says...
Develop a command-line email program that fully supports: 1) ReadReceipts (assuming, of course, that the recipient elects to acknowledge receipt); 2) DeliveryReceipts, which would be receiving server dependent; 3) Multiple attachments, as in more than one.
I can't begin to tell you how non-existent a community-based (aka free to use, NOT free to accept credit for) email program with the above features really is out there. I have searched all ends for something like this.
Before anyone begins flaming or otherwise attempts to degrade my need, I have a very legitimate use for this, and NO, it is not and never will be used for spamming!
I have to email a dozen unique dept performance reports to each of 60+ depts throughout our organization every month. The boss wants to know who received the reports (reason for the delivery receipts) and who does not bother to read the reports (read receipts).
Obviously, after the first month you start looking for something, anything to automate this mundane and error-prone task. To this day I have found nothing that doesn't have some kind of limit (no receipts, only 1 attachment, works for 1st 10 mailings, etc.) built-in.
So, how about it, Harsha (or anyone else): is this do-able?
Posted at 4:57 p.m. on June 2, 2009
20 apithla says...
Generally, a file needs to be sorted if uniq is to remove all the duplicate lines.
Exceptions to sorting are left as an exercise.
Posted at 10:16 a.m. on August 30, 2009
21 Marnix van Ammers says...
I find "uniq -c" very useful in a pipeline between two sorts:
This would be useful for instance if you were extracting IP addresses from a log of login attempts, or extracting a list of URLs from a web server access log.
Posted at 6:32 p.m. on September 16, 2009
22 kunju says...
what is working of "chcon" command in redhat linux? how it works an its implimentations,plz suggest me where i find all details about it?
Posted at 2:18 p.m. on October 8, 2009