Efficient Linux at the CLI  ## Metadata - Author: [[The Changelog: Software Development, Open Source]] - Full Title: Efficient Linux at the CLI - Category: #podcasts - URL: https://share.snipd.com/episode/4b68092b-6c71-4996-ab44-6ef360eacb44 ## Document Notes ## Highlights - **Inefficient Workflow** - Daniel J. Barrett was inspired to write his book after observing a colleague's inefficient Python workflow. - The colleague repeatedly quit and relaunched the editor instead of using job control. Transcript: Adam Stacoviak So in your blog post, you're writing about why you wrote this new book, Efficient Linux as a Command Line. And you mentioned how you were working with one of your colleagues there at Google on Python and you were watching them kind of do Python code. And just the way that they were going about it, what I think was like quitting the editor, going back to the command line, probably up arrowing to some test command or something, executing That, and then relaunching the editor and then finding their spot again. And it was kind of driving you nuts because like there's just better ways to do it. Right. But there are, that resonated with me because I cut my teeth on like SSH into a machine and coding on that machine in Vim, not because I wanted to, because my teacher in college made us, Which I'm appreciative for now. But I remember just being like, I don't even know what, like, it's just a prompt. And I'm like, I'll learn a few commands. I learned how to CD. I learned how to nano back then. Eventually he made us use Vim and I learned Vim, but there's so much I didn't know. And then I realized you could up arrow to old commands. And I was like, okay, now we're talking, you know, and I learned the history command. So then I realized you could take history, which is your history of commands you've typed before. And then you could like pipe it to grep and search for something. Like you start to kind of learn these things slowly, but that was back in like the early two thousands. And I'm telling you just a few months ago, I learned about control R, which, which like all nerds already know about, but I just didn't like somehow I just never knew about it. And you can just like start typing out and fuzzy match and hit enter. And you, anyways, my efficiency's up, you know, even after 20 years of doing this stuff, like in the last year, I just like doubled. And so I was reading that and I was like, okay, Daniel's onto something here. There's so much efficiency gains you can have if you just have someone tell you, here's how to do it. You ([Time 0:08:31](https://share.snipd.com/snip/3515064d-0853-4a6c-81f0-75ef50a96370)) - **Double Your Speed** - Learn command-line skills to double your speed and efficiency. - Straightforward techniques can significantly improve your workflow. Transcript: Daniel J. Barrett Have totally hit the nail on the head there. I got into efficient command line use largely because of the experiences like the one that you just cited about the engineer who was quitting the editor and restarting it and wasn't Aware of job control where you can suspend and resume commands. And I should mention that was not at Google, by the way. My Google colleagues are wonderfully adept at Linux. But it's amazing how many folks who use Linux have learned it through trial and error. And it's a fun operating system. And for, you know, you're a hobbyist, you want to try something out, that's great. But at a certain point, it will help to be a little more formal so you can learn these capabilities like Control-R, the bash shell feature you just mentioned that searches dynamically Through your command history. It's so fast, right? It's so much better. Adam Stacoviak Yeah. Yeah. Why didn't anybody tell me about this for years? Daniel J. Barrett Yeah. So there are probably, I can think of maybe three or four completely transformative things that I learned at the command line that each one of them just made me so much more efficient And I never looked back. And that's kind of why I wrote the book Efficient Linux at the command line, because you can definitely double your speed at the command line. And there are fairly straightforward ways to do it if you're willing to put in five minutes or an hour or whatever, just to learn some new ([Time 0:10:16](https://share.snipd.com/snip/1be1bf5c-50e9-4400-8772-ee0f2eb2511e)) - **CLI as Puzzles** - Command-line interfaces are puzzles where you invent commands to accomplish tasks. - Combining commands lets you create new tools, making Linux a joyful experience. Transcript: Daniel J. Barrett Okay. All right. Before I jump into that, let me just sort of set the context. Now, command line interfaces, a lot of times people think about them as kind of basic, you know, compared to these wonderful windowing icon and based OSs where you can click on something Or tap and make things happen. But actually, command line interfaces are the most interesting because every time you type a command, you are solving a puzzle, maybe a little puzzle. You have a task that you want to do, like I want to list my files or I want to edit this document or whatever it might be. And you have to invent on the spot a command to do it. And sometimes it seems kind of trivial, like listing your files, you just type ls and hit enter. But then there are other times where there is no Linux command for what you want to do. Like, let's say you're a Python programmer, and you want to know how many Python files are in your current directory tree. There is no single command that will count Python files. So you have to take one command like ls or find and pipe it to a command like word count that counts lines in standard output. And when you mash them together with a pipeline, you've invented a command that didn't exist before. And so this is one reason I just find Linux in some way a really joyful user experience because you're constantly solving puzzles. And who doesn't like puzzles in our community, right? It's fun. But the thing is, it takes a little while to become a good puzzle solver. So when somebody asks you something a little more complicated, like what's the most common initial in the last names of the users in this system? There's absolutely no command to find the most popular last initial. But with a pipeline of five or six commands, you can do it. And if you can instantly produce that pipeline, because you've been learning the concepts and so forth, you can move really quickly and solve fairly complicated sounding challenges Like puzzles right at the command line. And those are the skills I try to teach in efficient Linux at the command line. But I do want to get to the question you asked too. ([Time 0:11:53](https://share.snipd.com/snip/49641793-fca4-44a2-9c99-c6c6827e73a7)) - **Command Substitution** - Use command substitution to inject one command's output into another command. - Edit files containing specific words using the editor with `grep -wl`. Transcript: Daniel J. Barrett Cool. So some of the techniques I've learned that have really made a difference, the first one's called command substitution. And it is a way of taking the output of one command and injecting it into the text of the command that you are typing. So it's not like a pipe where you're sending standard output of command one into standard input of command two. But I'll give you an example. Suppose you want to edit all the files in the current directory that contain a particular word. And that's not an uncommon problem. You want to find a word and change it to something else in all these files. And maybe you want to do it interactively to make sure you're not having any false positives on your matching. So there is a command that will tell you all the names of the files that contain a particular word. That will be grep-w for word matching and dash l for just printing the file name. So grep-wl. And there's a command for editing files. It could be nano or vim or emacs or what have you. And you can combine these two pieces of information. Sometimes people use backticks on the command line. So you would say name of your editor like emacs and then backtick grep-wl and the word you're looking for, close backtick. And that will produce the list of names as if you had typed them on the command line. Everything between the backticks, which is taken as a command, is replaced by the output of that command. Adam Stacoviak Yeah, that is really powerful. I have one of those that I do all the time. So I have various scripts in various folders that are all in my path that I've either written or whatever. And sometimes I want to just edit and tweak a script and then execute it. But I don't really know where it lives and I'm not anywhere near that. And so I'll type which, W-H right? The which command and the script name. And that will print out the entire path to where that is on disk. And then instead of just, I'll take that and I'll wrap it in back ticks and I'll say Vim back tick, which command, and it will then take the path and give it to Vim. And so that's an example of just quickly editing that file without having to navigate to it or anything else, or even know where it is. I just know that the output of which can be substituted in to the command line for Vim. And that's cool. ([Time 0:14:24](https://share.snipd.com/snip/1c0b923c-d306-4ff5-8056-5a1c38583c16)) - **Process Substitution** - Process substitution creates temporary files for commands that need disk files. - Compare command outputs without creating and deleting actual disk files. Transcript: Daniel J. Barrett Yeah, that's exactly a perfect use case for a command substitution. And you can build some really complicated or complex computations using it. And in fact, there are ways of nesting it so that you produce output that then produces other output that then goes into your command line. And we have lots of examples of that in my book. And then there's another one that has a similar name to command substitution called process substitution. And the first time I saw this one, I didn't know what the heck was going on. I just didn't understand it. So I'll explain it. And the understanding will come probably in a minute as I'm speaking. So some commands don't work well in pipelines. An example would be the move command. Like you don't pipe things into move, you're moving files, right? Or the diff command, you're diffing the contents of two files. So these are programs that really work just on disk files. But sometimes you want to use these kinds of commands with the output of other programs or commands. I'll give you a specific example. Let's say you're a Python programmer and you're trying to debug the flags that you're passing into your program. And so you run your program twice, once with the flag and once without, and you want to compare the output. So the slow way to do this is you run the command the first time, redirect the output to file one, you run it a second time with your flag, redirect to file two, and then you diff the two files And then you delete the two files. That's how I would do it. Adam Stacoviak Exactly. Daniel J. Barrett So process substitution is this brilliant technique that I only learned, I only learned 10 years ago, that doesn't, I guess with 40 years of Unix, I didn't know it for the first 30. Process substitution allows you to create a sort of pretend file, a pretend disk file that fits right into the command line with you. So you wind up typing the word diff, the first command that would have produced your first output file, the second command that would have produced your second output file, all in the Same command line. But you're surrounding those two commands with a particular syntax. It happens to be a less than sign, a left parenthesis, and at the other end, a right parenthesis. So it winds up looking like diff less than paren first command, close paren, and then a space, and then left less than left paren the second command, close paren and what that says is each Of those two commands when they produce their output that output will behave as if it were in a disk file that doesn't have a name okay and diff will diff those two pieces of output right There in a single command. Adam Stacoviak So inside the less than parans section, like the substitution section, is your actual Python execution process, right? In your example, in your example. Daniel J. Barrett Yes, you'd say less than paren Python, whatever you want to run it with. Foo.py, close paren, and then less than paren, sorry, Python, foo.py, dash, x, y, z, paren. Adam Stacoviak Okay, so it's effectively like taking the output from a process and like giving it some sort of virtual file thing that only exists until the command ends kind of a thing? Daniel J. Barrett Exactly, and that's why it's called process substitution. Okay, ([Time 0:16:40](https://share.snipd.com/snip/456d05d7-45c7-458b-8311-47a99d34c567)) - **CDPATH Hack** - Use CDPATH to simplify navigation by defining a list of frequently visited directories. - Quickly access nested directories without typing full paths, like the command search path. Transcript: Daniel J. Barrett Hit us. So since you mentioned the search path, I want to talk about one that a lot of people don't know about, but is a real game changer for navigating your file system. And that is called the CD path. I don't know if you're familiar with it, but what do you think about what your search path is. It's a list of directories where executable programs are kept. So when you type just the bare name of a command at your command line, behind the scenes, the shell searches that entire list of directories until it finds the first match and it runs that Program. LS is sitting in your user bin or your bin directory. That's in your path. You type LS and the shell finds the proper LS executable. There is a second variable, not path, but CD path that does the same thing for just the command CD. Okay. It searches a list of directories for a destination that you type. So let's say that you've got a directory that you often visit of your own somewhere deep in your home directory. Let's say it's your home directory and then you have a subdirectory called finances and under that you have a subdirectory called bank. And you often go to the bank subdirectory because you want to look stuff up about your finances, let's say. So it's home, finances, bank. If you are off somewhere in the file system doing your work and you want to get to that deeply nested directory, you have to type the full path. So cd tilde slash finances slash bank. And you can imagine 10 levels deep instead of two levels deep. The CD path is a shortcut that lets you say, I've got a collection of directories that I often visit. So I want you to search for the thing that I'm, search for the subdirector I'm looking for in all of those directories till you find the first match. Just like you search for a command and find the first match. So I could set up a CD path that includes directory finances. And then no matter where I am in the file system, no matter where I am, I can type CD bank and it'll bring me to tilde slash finances slash bank because the finances directory is in my CD path. And if you are aware of this and you set up a CD path, let's say for all the first level directories inside home, you can get to every directory two levels deep that you own with a single CD With no path typing. ([Time 0:25:03](https://share.snipd.com/snip/d822901b-d6b4-480e-a1e2-aa94fc0bd7a4)) - **Piping to Bash** - Pipe command sequences to Bash for execution. - Generate and execute multiple commands dynamically. Transcript: Daniel J. Barrett Oh, yeah. I could keep going all afternoon. But another one that was really transformative to me was it's a little bit inspired by the Lisp community where, you know, in Lisp, code and data are fairly equivalent because you can Emit strings and execute them as code and so forth. You can do similar things on the command line because your shell, I'm going to assume your shell is bash just for ease of conversation. Bash reads from standard input. When you launch it, it's just a regular old command. Linux launches it for you when you log in, so it's kind of hidden in that way, but you can run B-A and hit enter and it will do something. It'll start a shell and then you hit control D and exit and the shell is done. It's just a plain old command. And if you know this, you can use that command to your advantage. For example, we all know the echo command. It just prints its arguments. So you can echo hello world and hello world prints on the screen. You can also echo LS. Think about that for a minute. You echo ls. All that does is print the word ls on the screen. Not very useful, but you can also say echo ls pipe bash. What do you think that does? Adam Stacoviak Probably tells bash to execute that command. Daniel J. Barrett It executes the ls command that it received on standard in. So what this means, that's a trivial example. What this means is you can use other Linux commands to create sequences of commands that you would like to execute and ultimately pipe them to bash for execution. Now, you may have seen things like this. There are a few software packages out there that ask you to run a curl command to download them and pipe them into a shell to install. And that always feels a little risky when folks do that. But that's an example of sending the output of a Linux command into bash. And I'll give you a practical example of something you can do on your own, because when you're running a curl command, you're getting stuff from some third party. You don't know what's in it. But if it's your own stuff that you're generating, you're generating your own lines of text. You can check them, you know, 100 times before you send them to bash. So here's an example that I ran into actually as I was writing Efficient Linux at the command line. At a certain point, I had 10 or 12 chapter files, and I needed to create a new file and insert it into the middle of the book, a new chapter. So my chapters 5 through 10 had to become chapters 6 through 11. And I could sit there and type six or seven move commands by hand to make that happen and move those files into names with integers that are one higher and then create my new file. But I wanted to do this all in one shot just for the challenge of it. And so I wrote a small script that generated the move commands I wanted. So it output the lines MV space, chapter five, chapter six, MV chapter four, chapter five, and move chapter three to chapter four and so forth. And that script was actually one command on the command line. It wasn't like a loop or anything. And with that output, I just put at the end of it, pipe into bash and they ran. Adam Stacoviak Instead of like saving that as a file and then executing the file, you just. Daniel J. Barrett Exactly. Or instead of writing a loop, like for I going from five to 10, you know, do this move command. And this, it's probably easier when you, to understand when you see it in the book, but the idea that you can generate any text you want and feed it to bash for execution means that you can Create, you can use all kinds of Linux tricks to produce sequences of commands that you can then run in one shot. Actually, now that I've talked about process substitution and command substitution and piping to bash, there are probably 15 different ways you can run commands to produce various Effects. ([Time 0:35:18](https://share.snipd.com/snip/34b13ad6-9aa9-44f1-81cc-3f5d47d49054)) - **Flexibility is Key** - Knowing multiple ways to perform tasks increases flexibility. - Different approaches can solve problems when one method fails (e.g. listing many files). Transcript: Daniel J. Barrett You've got those three. You've got pipes. You've got a whole bunch of others that I haven't gone into yet. And the key here is flexibility. If you know how to do something in multiple ways, your toolbox is set up for you to work more efficiently. Like here's a trivial example. Suppose you just wanted to list all the Python files in your current directory. Ls star dot py. That's what 99.9% of people would write. But if you've got 100 million files in your directory, that ls command is going to choke. Actually, the shell is going to choke because it's got a limited amount of buffer space to hold those file names after it expands that wildcard before it can pass them to ls what do you Do now well you could also just list the files just ls straight no wildcards no anything and pipe that to grep to find file names that end with dot py that has no length limitations because Now we're talking about lines of text, not one line of text. And so the fact that you know two ways to list the files in your directory means that you can do things when you run into trouble and one of them doesn't work. It's flexibility. And that's a skill that I try to communicate through a lot of conceptual examples in the book. I actually show, I think, 15 ways to list files, Python files in your current directory. And some of them are absolutely wacko, okay? Adam Stacoviak Right, like you probably would never do this, but here's another one. Daniel J. Barrett Yeah, but if you know these techniques, you will at some point find a ([Time 0:39:21](https://share.snipd.com/snip/53117e73-51e5-4418-bc3f-61f4571fc3c2)) - **Sudo with Bash -c** - Use `sudo bash -c` to run entire command lines with root privileges. - Avoid permission errors when redirecting output or using shell constructs. Transcript: Daniel J. Barrett Not at all. So since you were kind enough to create a curl options puzzle, I'd like to pose a puzzle as well. Sweet. And this will be related to what we were just talking about. So suppose you've created a wonderful program and you would like to have it send its output into a log file. So you might write, type the name of your program, greater than slash var slash log slash like my file dot log. And of course, that's going to fail because you don't have write permission in the log directory. So you throw a sudo in front of it. And you type sudo, my program, greater than path to the log file. That should do it, right? Well. But it doesn't. It fails. It fails. And why does it fail? That's a good question. Adam Stacoviak You're escalating your privileges, but you're actually, yeah, I'll say you're not actually switching users. I don't know why it would fail. My intuition is that it would fail, but I don't know why the intuition is that way. Jerod Santo Adam, do you have any guesses? Maybe because the output. It's maybe sudo on the first thing, but not on the second thing. Daniel J. Barrett You are close. You're on the right track. The sudo command, let's make it a little more concrete. I'll give my command a name. Let's just say it's the who command, which prints a list of users on the system. So sudo who, it sounds like a Dr. Seuss book, sudo who greater than var log my file dot log. Sudo does escalate the privileges of the who command, but the greater than symbol is a construct of the shell, not of the who command. So you have to escalate. You have to give root privileges for the whole command line. And the best way to do that is our friend bash dash C. Okay. Just like in Homebrew. So you say sudo bash dash C quote who greater than var log,.log close quote. And that will work because now the shell that is being invoked by bash to run the string inside the quotes now has root privileges. Gotcha. ([Time 0:48:40](https://share.snipd.com/snip/989cb58a-fb57-4876-9796-38030fcaf5cb)) - **ChatGPT Dangers** - Articles using ChatGPT for Linux commands often contain dangerous oversights. - One example involved a chmod command that recursively changed file permissions. Transcript: Daniel J. Barrett Glad you brought up ChatGPT because I've seen a number of articles recently about exactly what you were describing, asking ChatGPT for a Linux command. So you give an English description, a description in whatever your home language is, and it tells you what command to run. And in every single one of the articles that I've seen, there's been at least one fatal, dangerous command produced by ChatGPT that is not noticed by the writer of the article. I'll give you one example. I saw one article where a writer asked for, I think, a CHMOD command to make all the files in their current directory read only. And I don't remember exactly the way they phrased it to chat GPT, but what it returned to the person was a recursive command that changes the permissions on all files in the entire tree. And the way the question was phrased, it could have meant either one, just the current directory or current directory and all subdirectories. And if you are just taking instruction from chat TPT and don't know what the options mean, you can destroy the permissions on far more files than you meant to change with no way to restore Them. ([Time 0:52:24](https://share.snipd.com/snip/7cc7fe0e-8e23-4ad9-b9f2-9f60d9ec3b3a)) - **Job Control** - Use job control to suspend and resume processes in a single terminal. - Quickly switch between editing and running your code with fg. Transcript: Daniel J. Barrett Yeah, so the use case we're talking about is you are working in a single terminal, so maybe you're over an SSH connection to a server, because if you're able to create multiple windows, This whole problem goes away, because you can compile in one window and edit in another. So what this engineer was doing was in a single window, they would jump into their editor. They would fix whatever the next thing they wanted to do in their code. They'd quit the editor. They would run the program or compile or whatever they needed to do and see what happened. And then they would restart their editor, find where they had been before and continue editing. So there's a lot of stopping and starting and reestablishing the context of where they had previously been. But the Linux shells all have a feature called job control, which allows you to temporarily suspend commands and bring them back into the foreground, as they're called. So when you're in your editor, for example, you can type a keystroke that will cause the shell to suspend the editor, which will just give you your prompt back. The editor is still running in memory. I should say it's still in memory, but it's been stopped. It has a different process state. It's stopped. And then you can run your Python program or compile or whatever you want. And when you want to go back to restore the state of where you'd been, you simply type FG, that's short for foreground, and hit enter. And it brings back into the foreground the process that had been suspended, which is your editor. And so that is a much quicker way to jump back where you were when you're editing than quitting and restarting and trying to find where you were. And that particular individual shaved hours off of their coding time from this. ([Time 0:56:49](https://share.snipd.com/snip/aa49c1b8-1238-4e81-ab1d-0df2d796a6ec)) - **Directory Stack** - Use the directory stack (pushd, popd, dirs) for efficient navigation. - Push frequently used directories onto the stack and pop them off to quickly switch. Transcript: Daniel J. Barrett I've got one other win, and this was also one of the transformative concepts. And that was the use of what's called the directory stack for moving around the file system. So these are the commands called pushd, popd, and dirs. Have you run across those? Adam Stacoviak I'm aware of them. I think mostly in reference with some of these other third-party CD replacements because they kind of have that whole stack thing. And they're like, you don't have to use PushD and PopD, but please, expand. Daniel J. Barrett Well, this changed my whole use of Linux. I think I type PushD and PopD more often than I type CD these days. But before I geek out too much, let's give a little bit of context. So normally when you are working in Linux, you can CD by an absolute path, you know, starting with slash all the way from the root up. So you can say CD slash A slash B slash D slash D slash F and move long distances in the file system. The problem is when you do that and you need to get back where you were, you've got to keep typing these really long paths. And honestly, one of the biggest obstacles, I think, to new Linux users is typing all these long paths. And so the directory stack is a way to reduce your typing significantly when you are working in a collection of directories that you use frequently. So an example I give in the book Efficient Linux at the command line goes like this. Let's say you're a web developer. And web developers frequently work in at least four different common directories on the Linux system. You've got the directory where you're developing. That could be anywhere in your home directory, let's say. Then you've got the Apache directory where you've got to configure your web server. That's slash etc slash Apache 2 or whatever it might be. Then you've got a directory where SSL certificates are kept. That's like etc SSL certs. And then you've got the directory where your web files get deployed which is like slash var www blah blah blah so you've got to keep moving between all these different directories while You're working and if you have four windows open you know one to each directory that's great but if you're over an ssh connection or you just want to work in one window you got a lot of CD-ing And a lot of slashes. The directory stack gives you a quick way to say, bring me back to that place I was working a minute ago. It doesn't matter where it was or what it was called, just get me back. The simplest use of this is to type CD slash and then a hyphen, just a dash. Adam Stacoviak And that, a lot of people know that one. That's a good trick for... I didn't know that was part of this system, but yeah, I use that all the time. Daniel J. Barrett I'm not sure that it is, but it's all part of the shell. So I suppose in some sense it's part of the system, but that's the easy one that says, take me back where I was. So if you were in your home work directory and you need to go to the Apache directory, you can type in CD dash, CD dash, CD dash, and bounce back and forth between those two. The problem is if you cd anywhere else, now you've lost context of where you are, and you can't do that anymore. And now you're going to have to type some long path to get you back. The directory stack is not limited to two directories. It can be as many as you want, and they're arranged like a stack, like a traditional computer science stack. In computer science, it's last in, first out, and you can keep pushing directories onto this stack, piling them up, and then popping them off to go back where you were. And so the command pushd, P-U is a substitute for cd that says cd where I want to go, but also push a directory onto a stack. And you keep doing pushds. It keeps the stack grows, and then you can pop them off. And this is a great way to move around the file system because not only can you push and pop, you can also swap. So you can have the same effect as cd- but not lose your context. So it's so great. Once you start using these commands, you will be absolutely hooked, and you'll probably want to alias them to shorter names than pushd and popd. Adam Stacoviak I was about to say that. It seemed like too long to be using that often. Daniel J. Barrett Yeah, yeah. So I do gd. My pushd is gd for like get directory and pd for popd. And so, yeah, all the time. And then you can examine your stack. You can manipulate the stack. You can take things out of the middle of the stack. So it's really not a stack. It's more like a linked list. But you can still call the stack. And this, this completely changed my navigation that, and the CD path I mentioned earlier, it's so easy now to get anywhere I need to go in very little typing. ([Time 1:05:19](https://share.snipd.com/snip/3336c5ff-132b-459e-b36b-91db2659f69d))