Sunday, March 20, 2011

Here Strings, echo, sed and ed

It's not uncommon to see people (mis)using sed in simple scripts to modify files. A pretty typical pattern is something like:

for i in $({some operation that returns file names}) ; do 
    sed -i -e 's/old/new/' ${1}
done

For (at least) the BSD and GNU versions of sed: -i means edit this file in place and -e simply tells it to execute the following command(s) on the file(s) given.

UPDATE (03/22/11): To clarify, my point here is that using sed to edit files in place is bad practice, generally speaking. Creating a stream with which to edit the contents of one file as it's placed into a new file is not, here, my gripe.

I've never been a fan of using sed in this manner. The reasons are many but let's go with "sed is a stream editor and that's not a stream there". Basically the usage above is turning a specialized tool (stream editor) into a generic tool (file editor). Instead of overloading sed with this type of work, I generally use ed which was designed to edit files (not streams). I'd use something like:

for i in $({some operation that returns file names}) ; do
    ed ${i} <<-EOF
%s/old/new/
w
q
EOF
done


This usage is a bit more verbose but avoids ambiguity and possible incompatibility with -i. While both BSD and GNU version of sed have -i they implement it differently. The BSD version expects an extension for the backup file that's created. For GNUbies this probably isn't an issue. All of their systems are GNU/Linux and they only ever have to work with GNU sed. For those of us who work on unix systems of various flavors it's a bit more pertinent.

If you're working on the command line and don't want to bother with a multi-line here document just to do a simple file edit you do have options.

printf (both the shell built-in and standalone utility) can be used to reliably send control characters (including the newline character) to STDOUT which can then be piped into the ed command:


for i in $({some operation that returns file names}) ; do
    printf '\%s/old/new/\nw\nq\n' | ed ${i}
done


Unfortunately you have to escape the % character that you want passed through because otherwise it looks like you're saying '%s' which has special meaning for printf.

Another option if you're using a modern ksh93 work-a-like is to use here strings in combination with string formatting:


for i in $({some operation that returns file names}) ; do
    ed ${i} <<< $'%s/old/new/\nw\nq\n'
done