Process Section 1: Command line representation

Simultaneously enrolled students are responsible for attending section and self-reporting their attendance at www.tinyurl.com/cs61sections.

Ahead-of-time work

The topic on command line grammars relies on knowledge of BNF grammars, a mechanism commonly used in systems to define little languages. Please scan the BNF grammar material before section.

Command line scientists

How do shells behave? It’s possible to learn a lot by constructing specific “experiments”, namely command lines, and observing their behavior!

Unix offers a toolbox of utilities useful for command line science.

Program	Description
true	Silently succeed (exit with status 0).
false	Silently fail (exit with status 1).
cat	Write file(s) to standard output.
wc	Count lines, words, and characters in standard input.
head -n N	Print first N lines of standard input.
tail -n N	Print last N lines of standard input.
yes	Print y (or any arguments) forever.
sleep N	Pause for N seconds.
exit [STATUS]	Exit the shell with status STATUS (default 0).
sh -c "COMMANDLINE"	Run another shell executing "COMMANDLINE".

For example, the behaviors of the && and || conditional operators can be explored with different combinations of programs like false, true, and echo.

# Hypothesis: `A && B` runs `B` iff `A` exits with status 0.
$ true && echo X
X
$ false && echo X
$ sh -c "exit 2" && echo X
# Hypothesis not falsified!


# Hypothesis: `A || B` runs `B` iff `A` does not exit with status 0.
$ true || echo X
$ false || echo X
X
$ sh -c "exit 2" || echo X
X
# Hypothesis not falsified!


# Hypothesis: The `sleep` command exits with status 1.
$ sleep 10 && echo Done
[10 seconds go by, then:] Done
# Hypothesis falsified 😢: it looks like `sleep` exits with status 0.

In these exercises, you’ll attempt to explore more complex shell properties.

EXERCISE. Investigate the (true) hypothesis that in a conditional chain, commands are grouped left to right. For instance, given the chain A || B && C:

1. A || B runs first.
   1. So A runs first.
   2. Then, depending on A’s status, B runs or is skipped.
2. Then, depending on A || B’s status, C runs or is skipped.

Alternatives. This grouping might seem obvious, but others are possible too. For example, commands could be grouped right to left:

1. A would run first.
2. Then, depending on A’s status, B && C would run or be skipped.

Or && could take precedence over ||, etc.

To investigate this question and distinguish different groupings, we need long chains—more than 2 processes—and mixtures of && and ||. (If all the operators are the same, grouping won’t make any difference.) For instance, consider false && true || echo t1. If grouped left to right, as in ( false && true ) || echo t1, this would print t1: the exit status of false && true is 1—the same as false alone—and false || echo t1 prints t1. But grouped right to left, this would not print t1: false && ( true || echo t1 ) doesn’t execute the right-hand grouping.

$ false && true || echo t1
t1
$ true && true || echo t2
$ true && false || echo t3
t3
$ true || true && echo t4
t4
$ true || false && echo t5
t5
$ false || true && echo t6
t6

The above results can be explained by reasoning from left to right. If conditional chains were executed in some other way we would expect different results.

EXERCISE. Investigate the (true) hypothesis that in a two-process pipeline, such as A | B, processes A and B execute in parallel.

# If the right-hand command started after the left-hand command finished,
# then `foo` would never appear. It appears immediately.
$ yes | echo foo
foo

# If the left-hand command started after the right-hand command finished,
# then the error message would appear after 10 seconds (but it appears immediately).
$ cat /tmp/nonexistentfile | sleep 10
cat: /tmp/nonexistentfile: No such file or directory

OFFLINE EXERCISE. Investigate a similar hypothesis for longer pipelines.

EXERCISE. Investigate the (true) hypothesis that the exit status of a conditional equals the exit status of the last-executed command in that conditional.

Hint. The $? shell variable equals the exit status of the most-recently-executed foreground conditional. For instance:

$ true
$ echo $?
0
$ false
$ echo $?
1
$ sh -c "exit 2"
$ echo $?
2

$ true && false
$ echo $?
1
$ true && sh -c "exit 2"
$ echo $?
2
$ sh -c "exit 3" && true
$ echo $?
3

EXERCISE. Investigate the (true) hypothesis that the exit status of a pipeline equals the exit status of its rightmost command. (Try to avoid using $?.)

Conditionals can broadly test a command’s exit status (in the sense of “zero or not”).

$ false | false | false | false | true && echo foo
foo
$ false | false | true || echo foo
$ true | true | false && echo foo
$ true | true | false || echo foo
foo

Command line grammars

The shell grammar defines the underlying hierarchical structure of command lines. The overall structure of a line, as described by the grammar, is:

• A list is composed of conditionals joined by ; or & (and the last command in the list might or might not end with &).
• A conditional is composed of pipelines joined by && or ||.
• A pipeline is composed of commands joined by |.
• A command is composed of words and redirections.

So lists are comprised of conditionals, which are comprised of pipelines, which are comprised of commands.

There are two common ways to represent our parsed command line in a data structure appropriate for our shell, a tree representation and a list representation. Students often are biased toward the tree representation, which precisely represents the structure of the grammar, but the list representation is simpler in some ways. It’s useful to consider them both.

Neither the tree representation nor the list representation will exactly match the parent/child relationships implied by parent process IDs for the shell process and its children! In-memory command line representations should be designed for ease of traversal; parent/child process relationships are determined by other constraints, such as which processes need to wait for others.

Before going further:

EXERCISE SET. Work through the BNF grammar exercises.

Tree representation

The following tree-formatted data structure precisely represents this grammar structure.

• A command contains its executable, arguments, and any redirections.
• A pipeline is a linked list of commands, all of which are joined by |.
• A conditional is a linked list of pipelines. But since adjacent conditionals can be connected by either && or ||, we need to store whether the link is && or ||.
• A list is a linked list of conditionals, each flagged as foreground (i.e., joined by ;) or background (i.e., joined by &). Note that while the conditional link type doesn’t matter for the last pipeline in a conditional, the background flag does matter for the last conditional in a list, since the last conditional in a list might be run in the foreground or background.

For instance, consider this tree structure for a command line (the words and redirections that are there, but not being shown for the sake of simplifying the image somewhat):

EXERCISE. Assuming that a, b, c, d, e, and f are distinct commands, what could be a command line that would produce this tree structure?

a & b | c && d ; e | f &

EXERCISE. Sketch C++ structure definitions corresponding to this design, starting from this:

struct command {
    std::vector<std::string> args;
    command* next_in_pipeline = nullptr;
    command* prev_in_pipeline = nullptr;
};

Here’s one:

struct pipeline {
    command* command_child = nullptr;
    pipeline* next_in_conditional = nullptr;
    bool next_is_or = false;
};
struct conditional {
    pipeline* pipeline_child = nullptr;
    conditional* next_in_list = nullptr;
    bool is_background = false;
};

There are many other solutions. Logically, we need a struct for conditional chains that indicates whether the chain should be run in the foreground or background; a struct for pipelines; and a struct for commands. Each struct needs to hold the necessary pointers to form a linked list of siblings. Furthermore, conditionals and pipelines need a pointer to the first item of their sublists.

EXERCISE. Given a C++ structure corresponding to a conditional chain, explain how to tell whether that chain should be executed in the foreground or the background.

For our solution, that’s as simple as c->is_background.

EXERCISE. Given a C++ structure corresponding to a command, how can one determine whether the command is on the left-hand side of a pipe?

For our solution, a command c is on the left-hand side of a pipe iff c->next_in_pipeline != nullptr.

EXERCISE. Given a C++ structure corresponding to a command, how can one determine whether the command is on the right-hand side of a pipe?

If c->prev_in_pipeline != nullptr.

EXERCISE. How should we destroy a conditional, in other words, free all its memory and the memory of related structures? How should the conditional::~conditional() destructor function be implemented?

Linked list representation

Alternatively, we can create a single linked list of all of the commands. In this case, we also store the connecting operator (one of & ; | && or ||). For our example command line above, the flat linked list structure might look something like:

Traversing this structure is slightly more time-consuming than in the tree representation, but the structure is much easier to build, and the additional CPU time spent traversing it is in the noise given how short command lines typically are.

EXERCISE. Write C++ structure definitions corresponding to this design.

Here’s an example:

struct command {
    std::vector<std::string> args;
    command* next = nullptr;
    command* prev = nullptr;
    int link = TYPE_SEQUENCE;  // Operator following this command.
                    // Always TYPE_SEQUENCE or TYPE_BACKGROUND for last in list.
};

Much simpler!

Another good one, with fewer pointers and thus fewer opportunities for memory mistakes, but harder to traverse (you need C++ iterators):

struct command {
    std::vector<std::string> args;
    int op;         // same as link above
};
using command_list = std::list<command>;

EXERCISE. Given a C++ structure corresponding to a conditional chain, explain how to tell whether that chain should be executed in the foreground or the background.

bool chain_in_background(command* c) {
    while (c->op != TYPE_SEQUENCE && c->op != TYPE_BACKGROUND) {
        c = c->next;
    }
    return c->op == TYPE_BACKGROUND;
}

Things are slightly more complicated with a flat linked list. We are still executing commands sequentially, but some subsequences of commands may need to be executed in the foreground or the background. During parsing, we can skip ahead in the command line to find the end of our current list (i.e., our current chain of conditionals).

EXERCISE. Given a C++ structure corresponding to a command, how can one determine whether the command is on the left-hand side of a pipe?

If c->op == TYPE_PIPE.

EXERCISE. Given a C++ structure corresponding to a command, how can one determine whether the command is on the right-hand side of a pipe?

If c->prev != nullptr && c->prev->op == TYPE_PIPE.

EXERCISE. Sketch out code for parsing a command line into these C++ structures.

This solution builds up the command linked list in chead.

command* parse_line(const char* s) {
    shell_parser parser(s);
    command* chead = nullptr;    // first command in list
    command* clast = nullptr;    // last command in list
    command* ccur = nullptr;     // current command being built
    for (auto it = parser.begin(); it != parser.end(); ++it) {
        switch (it.type()) {
        case TYPE_NORMAL:
            // Add a new argument to the current command.
            // Might require creating a new command.
            if (!ccur) {
                ccur = new command;
                if (clast) {
                    clast->next = ccur;
                    ccur->prev = clast;
                } else {
                    chead = ccur;
                }
            }
            ccur->args.push_back(it.str());
            break;
        case TYPE_SEQUENCE:
        case TYPE_BACKGROUND:
        case TYPE_PIPE:
        case TYPE_AND:
        case TYPE_OR:
            // These operators terminate the current command.
            assert(ccur);
            clast = ccur;
            clast->op = it.type();
            ccur = nullptr;
            break;
        }
    }
    return chead;
}

(Offline) Testing your shell

In our experience, students experience two different kinds of error with this problem set: first, creating incorrect parsed representations for a command line; and second, executing incorrect system calls for a given representation. It is worth thinking explicitly with your peers about how you will test these aspects of your problem set! There’s no one right answer.

EXERCISE. Discuss what general tools you might use for testing your code. Are there kinds of error the compiler might help you find?

Sanitizers can catch many kinds of undefined behavior error, including missing nullptr assignments, use-after-free bugs, and so forth. Try make SAN=1 check!

EXERCISE. Discuss how you might test your parsing function, parse_line. What does it mean for the function to be correct? How could you test it for different kinds of error?

parse_line is correct if it parses command lines into the corresponding representations, without executing undefined behavior or leak memory.

(We promise that we never provide unparsable strings to parse_line, but a maximally robust parse_line would also check for input errors. If the user provided an invalid command line, such as “&& echo foo”, parse_line would report the error.)

One example plan to test that parse_line produces the right representations would be to add functions to your representation that unparse the representation into string format. For instance, these functions unparse the tree representation:

std::string command::unparse() {
    std::string s = "", separator = "";
    for (auto arg : this->args) {
        s += separator + arg;
        separator = " ";
    }
    if (this->next_in_pipeline) {
        s += " | " + this->next_in_pipeline->unparse();
    }
    return s;
}

std::string pipeline::unparse() {
    std::string s = this->command->unparse();
    if (this->next_in_conditional) {
        s += (this->next_is_or ? " || " : " && ");
        s += this->next_in_conditional->unparse();
    }
    return s;
}

std::string conditional::unparse() {
    std::string s = this->pipeline->unparse();
    if (this->next_in_list) {
        s += (this->is_background ? " & " : " ; ");
        s += this->next_in_list->unparse();
    } else if (this->is_background) {
        s += " &";
    }
    return s;
}

Then you could write a test function that, for each of a set of sample command line strings, (1) parses the string into a representation, (2) unparses the representation into a string, and (3) validates that the string and the representation string are equal.

EXERCISE. Discuss how you might test your execution functions, command::run and run_list. What does it mean for the functions to be correct? How could you test it for different kinds of error?

Since you’re aiming to implement a shell compatible with the normal Unix shells, these functions are correct if they call roughly the same system calls as the normal shell. And our friend strace is a good way to investigate these system calls!

For example, say you’re curious about how the shell executes the command chain cat /dev/null | wc. Then try this:

$ strace -f -b execve -o s.out sh -c "cat /dev/null | wc"

Here’s what the arguments mean:

• -f: Follow forks. This causes the strace to follow forked children, so you can see exactly what the forked child processes do.
• -b execve: Detach when execve is encountered. This causes strace to stop following a child once it loads a new program image.

When -f is provided, every line in the strace starts with the process ID of the relevant process. It’s important to remember this or you may get confused about which actions are being taken in child processes as opposed to the parent shell! Consider sorting the strace by process ID, so that all of a process’s actions are brought together:

$ sort -sk1,1 s.out | less

Here’s what the arguments mean:

• -k1,1: Sort by the first blank-separated key, which is the process ID.
• -s: Stable sort. This means that sort will not reorder lines that have the same process ID.

You will also need to know that some system calls have different names in strace than in the standard library. For instance:

• The fork system call appears as clone.
• The _exit system call appears as exit_group.
• The waitpid system call appears as wait4.

Putting all this together, here is a part of the sorted strace -f -b execve output for the command sh -c "/bin/echo foo | /usr/bin/wc". (Providing full pathnames for the commands avoids uninteresting calls to stat.) Compare these system calls to the pipe dance!

249   pipe([3, 4])                      = 0
249   clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f6dd7a48850) = 250
249   close(4 <unfinished ...>
249   <... close resumed>)              = 0
249   clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD <unfinished ...>
249   <... clone resumed>, child_tidptr=0x7f6dd7a48850) = 251
249   close(3 <unfinished ...>
249   <... close resumed>)              = 0
249   close(-1 <unfinished ...>
249   <... close resumed>)              = -1 EBADF (Bad file descriptor)
249   wait4(-1,  <unfinished ...>
249   <... wait4 resumed>[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 250
249   --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=250, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
249   rt_sigreturn({mask=[]})           = 250
249   --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=251, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
249   rt_sigreturn({mask=[]})           = 250
249   wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 251
249   exit_group(0)                     = ?
249   +++ exited with 0 +++
250   close(3 <unfinished ...>
250   <... close resumed>)              = 0
250   dup2(4, 1 <unfinished ...>
250   <... dup2 resumed>)               = 1
250   close(4 <unfinished ...>
250   <... close resumed>)              = 0
250   execve("/bin/echo", ["/bin/echo", "foo"], 0x564ffc49fc38 /* 13 vars */ <unfinished ...>
251   dup2(3, 0 <unfinished ...>
251   <... dup2 resumed>)               = 0
251   close(3)                          = 0
251   execve("/usr/bin/wc", ["/usr/bin/wc"], 0x564ffc49fc18 /* 13 vars */ <unfinished ...>

(Offline) Subprocess

Finally, this exercise (which is not about command line representation) will test your understanding of file descriptor system calls as well as processes.

EXERCISE. Write a function that implements a version of Python’s subprocess.Popen functionality. Your function should have the following signature:

// subprocess(file, argv, pfd)
//    Run the command `file` with arguments `argv` in a child process.
//    Three pipes are opened between this process and the child process:
//    one for the child’s standard input, one for its standard output,
//    and one for its standard error. The `pfd` argument is populated
//    with this process’s pipe ends.
//
//    * Data written by this process to `pfd[0]` is read from the child’s
//      standard input.
//    * Data written to the child’s standard output is read by this
//      process from `pfd[1]`.
//    * Data written to the child’s standard error is read by this
//      process from `pfd[2]`.
//
//    Returns the process ID of the child or -1 on error.
pid_t subprocess(const char* file, char* const argv[], int pfd[3]);

You will use the pipe and dup2 system calls, among others.

Here’s one solution. It uses a common pattern in C programs called “goto error”: all the error handling code is in one place, and goto is used to jump to that place from multiple locations where errors could occur.

pid_t subprocess(const char* file, char* const argv[], int pfd[3]) {
    // create pipes
    int inpfd[2] = {-1, -1}, outpfd[2] = {-1, -1}, errpfd[2] = {-1, -1};
    pid_t p = -1;
    if (pipe(inpfd) < 0
        || pipe(outpfd) < 0
        || pipe(errpfd) < 0) {
        goto error;
    }

    // create child
    p = fork();
    if (p == 0) {
        dup2(inpfd[0], STDIN_FILENO);
        close(inpfd[0]);
        close(inpfd[1]);
        dup2(outpfd[1], STDOUT_FILENO);
        close(outpfd[0]);
        close(outpfd[1]);
        dup2(errpfd[1], STDERR_FILENO);
        close(errpfd[0]);
        close(errpfd[1]);
        execvp(file, argv);
        _exit(1);
    } else if (p < 0) {
        goto error;
    }

    // clean up file descriptors
    close(inpfd[0]);
    pfd[0] = inpfd[1];
    close(outpfd[1]);
    pfd[1] = outpfd[0];
    close(errpfd[1]);
    pfd[2] = errpfd[0];

    // return pid
    return p;

  error:
    if (inpfd[0] >= 0) {
        close(inpfd[0]);
        close(inpfd[1]);
    }
    if (outpfd[0] >= 0) {
        close(outpfd[0]);
        close(outpfd[1]);
    }
    if (errpfd[0] >= 0) {
        close(errpfd[0]);
        close(errpfd[1]);
    }
    return -1;
}