Ten Cool Coreutils Commands

6 January 2007

While a few commands, such as 'cd' are built directly into bash; many of the most important commands come from coreutils, a GNU package containing over a hundred commands.

Some are the well known commands such as 'ls', 'mv' and 'cat'. While there are some more obscure ones that are not always discovered because there are also more modern programs that can do similar jobs.

These less well known commands often reflect the history of computing where once upon a time you would do everything possible in the Bourne or Bash shell, rather than waiting a long time for program such as emacs or vi to start, which at the time were considered to be memory intensive but now seem lightning fast (now you can even run them from a mobile phone).

Here we will look at ten useful commands that might come in handy one day.

1. tac

One of the key commands is 'cat', short for concatenate, which you can use to add the contents of one file to another, or just to print a file or set of files out to the screen.

cat 1.txt 2.txt > 3.txt # Adds 1 and 2 together as a new file 3.

cat 3.txt # Prints file 3 out to the screen.

However, tac is cat backwards, it lets you concatenate and print files in reverse. It could come in very handy for some uses, for example if you want to reverse a list or a log file.

tac 1.txt > 2.txt # 2 is a reversed copy of 1

2. tee

Often you will pipe the output of one program as the input of another:

ps -e | grep apache

This command will feed to grep the output of ps, thus showing what apache processes are currently running.

Or you may redirect the output of a command to a file:

ps -e | grep apache > apache-processes.txt

However, what if you want to see the results and write the file or files? This is where the tee command is useful:

ps -e | grep bash | tee bash-processes.txt # Print the results to the screen but also to a file.

You can enter more than one filename if you want multiple copies.

3. pr

Most printers these days will attempt to print out anything that you throw at them. However, you still may want to format a file in a certain way before sending it to the printer. For example, you may regularly discuss a log or some records at a meeting and want them to look the same every week.

The pr command gets a text file ready for the printer. For example you may want a set page or column format.

pr +10 -h"Apache Errors" -l25 error_log | lpr -# 5

Reading the pr command from left to right, the options are to start from page 10, then add a header to each page, make each page 25 lines long and lastly the filename. The result is then piped to lpr which will submit the file to the printer with a request for 5 copies.

4. stat

Using 'ls -l' will give a lot of information about a file, enough for me at least. If you are a glutten for punishment, you can use the 'stat' command to get more information:

$ stat stat

File: `stat'

System Message: WARNING/2 (<string>, line 79); backlink

Inline interpreted text or phrase reference start-string without end-string.

Size: 34684 Blocks: 72 IO Block: 4096 regular file

Device: 302h/770d Inode: 7586763 Links: 1

Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)

Access: 2007-01-06 00:35:15.000000000 +0000

Modify: 2006-09-18 09:50:24.000000000 +0100

Change: 2007-01-04 14:51:47.000000000 +0000

5. yes

'yes' is one of those strange legacy commands, the most comic featured here. 'yes' prints a string and a newline and then repeats until interrupted.

As dumb as it sounds, it does have a couple of uses. Firstly, if you want to turn an interactive command into a non-interactive one.

There are some command line programs which you run and then it asks something like "Are you sure?". For example:

yes 5 | command

This will run command then send the string 5 and a newline (i.e. as if you had pressed return).

It almost goes without saying that you can also use yes to turn your processor up to 100% usage, for example if you are testing some fans or cooling system or if you otherwise want to punish your machine.

6. expand

So you wrote lots of nice Python files, indenting them with tabs. But shock horror, you found out that the current fashion is to use four spaces not tabs!

No problem, the expand command converts tabs to spaces, here we have chosen to use 4 spaces:

expand -4 uncoolscript.py > coolerscript.py

To go from spaces to tabs, you can use 'unexpand' command.

7. split

'split' takes a file and splits it into chunks for you. For example:

split -l 20 access_log part

This command will split the file access_log into chunks of 20 lines each, the name of each file will begin with part.

**8. uniq **

System Message: WARNING/2 (<string>, line 135); backlink

Inline strong start-string without end-string.

Sometimes you will have a file that is a long list of items and you want to remove all the repeated lines, or possibly you want to group the items into sets with a number of occurrences.

So to remove all repeated lines:

uniq file.txt

You can also use the output of another command:

cat file.txt | uniq

To include the occurrence count:

uniq -c file.txt

9. wc

wc allows you to count words, lines and bytes. The default is to show all three:

wc ulyss12.txt

32758 267235 1561677 ulyss12.txt

[lines] [words] [bytes]

Using -l just gives you lines:

wc -l ulyss12.txt

32758 ulyss12.txt

Using -w just gives you words:

wc -w ulyss12.txt

267235 ulyss12.txt

10. shred

'rm', the remove command, unlinks a file so the space can be reused. However, files deleted can sometimes be recovered with a bit of persistence and luck.

Our final command is shred. 'shred' overwrites a file repeatedly, making it much harder to recover. This can be useful for personal financial information such as your credit card details.

shread -u mastercard.txt

You can even use shred on a device such as a partition. If you want to completely clear your home partition which happens to be stored on /dev/sda6. Then you would use:

shread /dev/sda6

Alert readers will note that I used -u in the first example but not the second. This because you want to unlink (i.e. rm) a file but not a device - you will want empty the /dev/sda6 to exist.

Now there are some complications to this. Firstly, a complex RAID setup might interpret this as hardware failure and replace the data. Likewise some corporate setups will sync the files with a server. Shred will not remove these copies.

Lastly, some fancy modern journaled filesystems such as reiserfs will have a backup in the journal, you will need to mount the partition in a non- journaled mode for shread to work completely.

There we go, ten commands that you may not have heard of, I hope you can find at least one that is useful in your computing activities.

1 Eddy Mulyono says...

Love 'em.

Thanx for sharing.

Posted at 8:40 a.m. on January 8, 2007


2 tony says...

Thanks for the tip about shread. I'm learning unix and enjoying the shell, but am still trying to wrap my head around uses for all this wonderful exe gems. You're blog is great.

Posted at 6:59 p.m. on January 8, 2007


3 Jason says...

Nice list. Definitely going into my bookmarks.

I've run into more circumstances where I've needed to to a numerical or alphabetical unique sort, which I don't think uniq can do. In this case a sort -u or sort -un would be better:

cat file.txt | sort -u cat file.txt | sort -un

Posted at 5:31 p.m. on January 29, 2007


4 Volatile says...

Nice! Didn't know half of them... expand/unexpand is going to make my life soo much easier!

Posted at 12:43 a.m. on March 7, 2007


5 Mad Hatter says...

tac is definatly a new one to me, pretty cool. I've used knoppix and shred for all laptops I have returned to companies I have quit. Makes sure that some nosy IT guy can't pull up anything.

Posted at 1:51 p.m. on March 21, 2007


6 Jeroen says...

Nice list, thanx!

Being who I am... you have a little typo in your shred command (shread)

And to Mad Hatter, the nosy IT guy would have done that while you where working on it and where connected to the company network, or just take a good look at your server copy of your profile and homeshare. So unless you never connected long to the company network and your sysadmin was already a moron it's not gonna help much towards the goal you mention.

Posted at 8:18 a.m. on March 29, 2007


7 Jeroen says...

Forgot to mention why I like the list, and the mentioning of the source package (CoreUtils):

http://gnuwin32.sourceforge.net/packages/coreutils.htm

They work on Windhose too :)

Posted at 8:20 a.m. on March 29, 2007


8 AJS says...

I always use -n1 with the shred command. This option limits the number of overwrite passes, and one is enough.

Yes, one overwrite pass is enough.

(Please don't quote the Gutmann paper! Many of the assumptions on which it relies have been invalidated since it was written. Voice coil actuators have replaced stepper motors, thus improving the precision of tracking; and data densities have increased by several orders of magnitude, thanks to the more precise tracking.)

If it were really possible reliably to recover overwritten data from magnetic storage devices, somebody by now would have built a storage device which used the phenomenon to increase its capacity. If you can recover data after one overwrite, you can effectively fit two bits into the space of one. Given the history of computers, it is next to certain that (1) such a device would have been economically feasible at some stage in the past, and (2) some manufacturer would have committed heavily to it, probably just after it ceased to be economically feasible.

Posted at 12:57 p.m. on January 15, 2008


9 Greg Gann says...

I have found the commands head and tail useful.

By default, head outputs the first 10 lines of a file, and tail, the last 10. However, you can combine them to produce any section of the file you need to see.

Does your compiler report an error at line 253?

The command:

head -n 265 foo.c | tail -n 20

will display lines 246 through 265 of foo.c

Useful, that.

Posted at 9:01 p.m. on April 14, 2008


10 Greg Gann says...

I have found the commands head and tail useful.

By default, head outputs the first 10 lines of a file, and tail, the last 10. However, you can combine them to produce any section of the file you need to see.

Does your compiler report an error at line 253?

The command:

head -n 265 foo.c | tail -n 20

will display lines 246 through 265 of foo.c

Useful, that.

Posted at 9:10 p.m. on April 14, 2008


11 harsha says...

I'm interested in working with GNU coreutils where i can add new commands in it .

Can anybody help me with some ideas for creating new commands so that i can develop on that .

Posted at 5:49 p.m. on June 6, 2008


12 atul says...

paste is also a cool one... The following is a single line...

$ for ((i=1; i<10; ++i)) ; do echo "$i"; done | paste -s -d '*' 1*2*3*4*5*6*7*8*9 $

It is very useful to put something in BETWEEN two numbers...

passing it to bc obviously illustrates the power of linux ;-)

$ for ((i=1; i<100; ++i)) ; do echo "$i"; done | paste -s -d '*' | bc 933262154439441526816992388562667004907159682643816214685929638952175999932299156089414639761565182862536979208272237582511852109168640000000000000000000000

just some fun ;-)

-- thx atul

Posted at 10:28 a.m. on June 17, 2008


What do you have to say?

Show Editing Help

Europython

About

Hello, my name is Zeth, I'll be your host here.

Command Line Warriors is about taking control of your own technology, it looks at our experiences of computing; especially using GNU/Linux, the Python programming language, the command-line and issues such as techno-ethics, best practices and whatever is cool now. If you take control of your technology then you are a Warrior too!

This site is your site too which means that you can contribute and get involved. You can leave comments using the facility provided. For me, the comments and discussions are by far the best part of the site. So please do have your say!

Latest Discussions

picsus

January 5, 2009
Monique, a Leaf fan, originate this plumb persistent to believe. Now, let me regarding out that this was in no way an try to articulate one cooperate is more wisely ...
This week in the world of the Command Line; The Friday Round up!

QuickSilver

January 5, 2009
Nice! Is there anyway to implement a ServerAliveInterval for long processes? This is because my our firewall keeps closing the connection based on inactive connections. Thanks,
SFTP in Python: Really Simple SSH

Tun

January 5, 2009
Hi, Do You know, haw can i get start date for tasks evolution? If exists the similar way to your example: i.get_due() ? I would like to have sth like ...
Three Useful Python Bindings - ClamAV, Apt and Evolution

MurreiM

January 5, 2009
This is great! http://www.youtube.com/MurreiM Buy Alli Orlistat online cheap
Filing cabinets 101 - An introduction to disk partitions

sarah

January 5, 2009
I recently came across your blog and have been reading along. I thought I would leave my first comment. I don't know what to say except that I have enjoyed ...
This Week: Freedom not Time-Bombs

jnfrlast

January 4, 2009
Hi! http://www.youtube.com/jnfrlast buy cheap viagra online
Filing cabinets 101 - An introduction to disk partitions

Samuel Huckins

January 4, 2009
Great tips! I have had occasion to do a lot of MySQL instance migrations lately, so here is an improvement for Trick 1: mysqldump <DATABASE_NAME> [mysqldump_options] | gzip -c | ...
Five useful command one liners

George Glass

December 31, 2008
I don't really see the point in trying to make linux user-friendly or take over the desktop. We rule the servers the most important element of the entire game. Let ...
Give Linux a chance

bug

December 31, 2008
@Zeth: The hidden field does block some. Not perfect, but it does release some weight from the filtering system, as those are 100% false comments. Acctually, if you would have ...
On Comment Spam

Zeth

December 31, 2008
Hi Eion, Yes that is an interesting approach also. It is the only approach given by default in the stock Django comments module, though it does not stop all comment ...
On Comment Spam

Bug

December 30, 2008
Well... Sadly, and I guess you hate me for it, I use captcha. But at least it's not an image, so even if you visit using w3m [yey!] you can ...
On Comment Spam

Eion

December 30, 2008
Other than server-side processing of comments, I like to add additional <input>'s and hide them in external css. Most of the time the fields are populated by spam-bots, and if ...
On Comment Spam

Nostoc

December 27, 2008
... Mate possible because of the dull Kg8
Ruy Lopez, Berlin defence, open variation

Nostoc

December 27, 2008
My bad, I meant the picture beneath 15, after close inspection my suggestion would be on 18. Instead of 18 : Qe2, I would have taken that knight with my ...
Ruy Lopez, Berlin defence, open variation

Zeth

December 27, 2008
Nostoc, white takes the rook on 15, the rook is a better kill than a knight.
Ruy Lopez, Berlin defence, open variation

Nostoc

December 26, 2008
I'm not that good at chess, but I have a question. At 15, why doesn't white simply take black's knight in C6 with the bishop? It's an easy kill, since ...
Ruy Lopez, Berlin defence, open variation

Zeth

December 26, 2008
CorkyAgain, good question, I don't have a FreeBSD box available at the moment so I can't comment. On Linux at least watch does as I have described.
Five useful command one liners

CorkyAgain

December 25, 2008
Is the watch command you're describing a Linuxism? On my FreeBSD box, "man watch" seems to be describing something completely different.
Five useful command one liners

Binny V A

December 25, 2008
I have actually setup a site to store just short commands... http://txt.binnyva.com/
Five useful command one liners

Bassam essa

December 25, 2008
i try this line command elinks -source "http://www.e51g.com/" > resulthtml.txt its work done :) thx
Command the Web - an ELinks tutorial