echo Modifies Your Strings

When you use the command echo, it doesn’t print out exactly what you specify. Here, I talk about the new line it adds and how to create exactly the bytes you want.

The echo command is a commonly used command for spitting out arbitrary text in your shell.

> echo "Hello world!"
Hello world!
>

Sometimes you might even find advice to place some contents in a file using the echo command and redirections:

> echo "Hello world!" > hello_world.txt
>

But, you have to be careful with this, because what echo actually spits out is the specified string followed by a newline. We can see this by passing the output of the command to hd which displays the hexadecimal representation of a file, which here, by default, is /dev/stdin:

> echo "Hello world!" | hd
00000000  48 65 6c 6c 6f 20 77 6f  72 6c 64 21 0a           |Hello world!.|
0000000d

The . on the right-hand side indicates a character that can’t be displayed, but we see the 0a on the left which is the ASCII and UTF-8 code point for a new line.

Oct   Dec   Hex   Char                        Oct   Dec   Hex   Char
────────────────────────────────────────────────────────────────────────
007   7     07    BEL '\a' (bell)             107   71    47    G
010   8     08    BS  '\b' (backspace)        110   72    48    H
011   9     09    HT  '\t' (horizontal tab)   111   73    49    I
012   10    0A    LF  '\n' (new line)         112   74    4A    J
013   11    0B    VT  '\v' (vertical tab)     113   75    4B    K
014   12    0C    FF  '\f' (form feed)        114   76    4C    L
015   13    0D    CR  '\r' (carriage ret)     115   77    4D    M

In this excerpt from the ASCII man page (man 7 ascii), you can see new line specified on the line starting with 012, which is octal for 10, or 0a in hexadecimal.

But, why is this such a big deal? Well, computers are very precise and if you were to, say, drop in a password or an API token using this strategy into a file, you’ll find that using the contents will likely give you access denied errors.

There are a few ways to avoid this. The easy one is the -n flag for echo that explicitly tells echo not to print out the new line.

> echo -n "Hello world!" | hd
00000000  48 65 6c 6c 6f 20 77 6f  72 6c 64 21              |Hello world!|
0000000c

You might also note that the second line has changed. It's just changed to reflect that the input is one byte smaller.

You can also consider turning to printf. printf is designed to explicitly print out what you specify, but that f is really important as it stands for format. While, replacing echo with printf works for this use case:

> printf "Hello world!" | hd
00000000  48 65 6c 6c 6f 20 77 6f  72 6c 64 21              |Hello world!|
0000000c

things get weird if your string include % symbols:

> printf "Hello 50% of the world!"
Hello 500f the world!>

That trailing `>` is actually our prompt. Because we didn't output a new line, it showed up right after the text we wrote.

> printf "Hello 50% of the world!" | hd
00000000  48 65 6c 6c 6f 20 35 30  30 66 20 74 68 65 20 77  |Hello 500f the w|
00000010  6f 72 6c 64 21                                    |orld!|
00000015

This is because the % indicates that we are going to add things to the format string. This particular format string in particular is % o, which says place the octal representation of the passed in parameter here. We didn’t pass a parameter, and so it ended up with the default value 0.

That space also holds value: it tells printf to prepend the octal string with spaces to fill the specified width, though we never passed a width, so there wasn’t any reason to prepend anything. But, if we did, we’d have gotten a bunch of spaces:

> printf "Hello 50% 8of the world!" | hd
00000000  48 65 6c 6c 6f 20 35 30  20 20 20 20 20 20 20 30  |Hello 50       0|
00000010  66 20 74 68 65 20 77 6f  72 6c 64 21              |f the world!|
0000001c

And you can do more, like right-aligning or prepending with zeroes:

> printf "%-8o %08o\n" "21" "22"
25       00000026

Here we have multiple formatting fields and pass multiple arguments to fill in each field. If we had passed in more than there were fields, `printf` would reuse the format string until it had consumed all arguments.

Wikipedia actually has a pretty detailed section on format strings. In fact, this syntax is not just specific to this tool, but also by the identically named C function, Python’s string formatting, and all sorts of other places.

Now, we’ve wandered a bit far from where we started of avoiding adding surprise new lines to strings. But, hopefully this guide has given you some extra insight into displaying strings that can help prime your exploration next time you need to get fancy.

> printf "% 10s %-10s\n" "abc" "efghi" "jklmno" "pqrstuv" "wxy" "z"
       abc efghi
    jklmno pqrstuv
       wxy z

And, as some final tips, if you want to get even better at managing your secrets, consider reading directly from your clipboard instead of typing the secret on the command line:

# Linux running X11
xclip -selection clipboard -o > secret.txt
# MacOS
pbpaste > secret.txt

And to go further, use a tool like sops to keep your secrets from being exposed while at rest.

xclip -selection clipboard -o | sops -e /dev/stdin > secret.txt