Essential Sed one liners

By
JP White
– Wednesday, August 26, 2009 Posted in: , , ,

sed (Stream EDitor) refers to a Unix utility for parsing text files and the programming language it uses to apply textual transformations to a sequential stream of data. It reads input files line by line, applying the operation which has been specified via the command line (or a sed script), and then outputs the line. Getting started with sed can be a real pain if you are unfamiliar with perl for regular expressions. Here are a several sed 'one liners' that I use all the time for test file manipulation. More can be found here.






FILE SPACING:



  • Double space a file


    • sed '/^$/d;G'


  • Undo double-spacing (assumes even-numbered lines are always blank)


    • sed 'n;d'






    NUMBERING:



  • Number each line of a file (number on left, right-aligned)


    • sed = filename | sed 'N; s/^/ /; s/ *(.{6,})n/1 /'


  • Number each line of file, but only print numbers if line is not blank


    • sed '/./=' filename | sed '/./N; s/n/ /'


  • Count lines (emulates "wc -l")


    • sed -n '$='






    TEXT CONVERSION AND SUBSTITUTION:



  • Convert DOS newlines (CR/LF) to Unix format.


    • sed 's/.$//'


  • Convert Unix newlines (LF) to DOS format.


    • sed 's/$'"/`echo r`/"


  • Delete leading whitespace (spaces, tabs) from front of each line aligns all text flush left


    • sed 's/^[ t]*//'


  • Delete trailing whitespace (spaces, tabs) from end of each line


    • sed 's/[ t]*$//'


  • Delete BOTH leading and trailing whitespace from each line


    • sed 's/^[ t]*//;s/[ t]*$//'


  • Substitute (find and replace) "foo" with "bar" on each line


    • sed 's/foo/bar/'     # replaces only 1st instance


    sed 's/foo/bar/4'     # replaces only 4th instance


    sed 's/foo/bar/g'     # replaces ALL instances


  • Change "scarlet" or "ruby" or "puce" to "red"


    • sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g'






    SELECTIVE PRINTING OF CERTAIN LINES:



  • Print first 10 lines of file (emulates behavior of "head")


    • sed 10q


  • Print first line of file (emulates "head -1")


    • sed q


  • Print the last 10 lines of a file (emulates "tail")


    • sed -e :a -e '$q;N;11,$D;ba'


  • Print the last 2 lines of a file (emulates "tail -2")


    • sed '$!N;$!D'


  • Print the last line of a file (emulates "tail -1")


    • sed -n '$p'


  • Print section of file based on line numbers (lines 8-12, inclusive)


    • sed -n '8,12p'


  • Print only lines which match regular expression (emulates "grep")


    • sed '/regexp/!d'


  • Grep for AAA and BBB and CCC (in any order)


    • sed '/AAA/!d; /BBB/!d; /CCC/!d'


  • Grep for AAA and BBB and CCC (in that order)


    • sed '/AAA.*BBB.*CCC/!d'






    SELECTIVE DELETION OF CERTAIN LINES:



  • Delete duplicate, consecutive lines from a file (emulates "uniq").


    • sed '$!N; /^(.*)n1$/!P; D'


  • Delete duplicate, nonconsecutive lines from a file.


    • sed -n 'G; s/n/&&/; /^([ -~]*n).*n1/d; s/n//; h; P'


  • Delete all lines except duplicate lines (emulates "uniq -d").


    • sed '$!N; s/^(.*)n1$/1/; t; D'


  • Delete the first line of a file


    • sed '1d'


  • Delete the first 10 lines of a file


    • sed '1,10d'


  • Delete the last line of a file


    • sed '$d'


  • Delete the last 2 lines of a file


    • sed 'N;$!P;$!D;$d'


  • Delete the last 10 lines of a file


    • sed -n -e :a -e '1,10!{P;N;D;};N;ba'



  • Delete lines matching pattern


    • sed '/pattern/d'


  • Delete ALL blank lines from a file (same as "grep '.' ")


    • sed '/^$/d'


  • Remove most HTML tags (accommodates multiple-line tags)


    • sed -e :a -e 's/<[^>]*>//g;/