Process 2: Inter-process communication

Overview

In this lecture, we discuss the primary system calls shells use to arrange inter-process communication.

Full lecture notes — Textbook readings

Primary process control system calls

fork creates a new process

execvp runs a new program

_exit terminates this process

waitpid monitors a child process for completion

Exit notification as a communication channel

API design

Primary IPC system calls for shells

The pipe system call

pipe illustration

Before pipe
After pipe

selfpipe

Pipe privacy

Pipe throughput

Pipe end-of-file

childpipe and dangling ends

Pipes and the shell

Full pipeline

Problem: How to move file descriptors around?

Initial shell
After pipe system call
Full pipeline

dup2

Pipe dance

Full pipeline

Pipe dance explained

  1. Here’s the initial state of the shell.

    Initial state

  2. Child processes inherit file descriptors from their parents only at fork time. Since pipes must be accessible to children in the pipeline, the pipe must be created first, before the child processes.

    Step 1: pipe() in parent

  3. Now the pipe exists, we can create the left-hand child. It is still running shell code.

    Step 2: fork() in parent

  4. We switch our attention to the left-hand child. Inside the code for the child (if (result_of_fork == 0) { ... }), we must reconfigure the process environment to match the requirements of the pipeline. We start by setting up the standard output file descriptor with dup2(pfd[1], 1) (recall that standard output’s file descriptor number is 1). It’s important not to do this in the parent shell—the parent needs its stdout!

    Step 3: dup2(4, 1) in child

  5. That done, we no longer need the pfd[1] file descriptor…

    Step 4: close(4) in child

  6. …or the pfd[0] file descriptor. (Not all these steps are order-sensitive—we could have closed pfd[0] first.)

    Step 5: close(3) in child

  7. Now the child’s environment matches all requirements, it is safe to replace its program image via execvp. We must set up the environment before calling execvp because the new program, /bin/echo, can run in many environments and simply accepts its initial environment as a given. An advantage of this separation of concerns is that the shell does not rely on /bin/echo (a program it does not control) to perform the dup2/close system calls correctly.

    Step 6: execvp() in child

  8. The left-hand child is on its merry way and is no longer running shell code. (It may already execute, writing its output into the pipe buffer!) We switch our attention back to the parent shell, which has been continuing in the meantime.

    The pipe’s write end is no longer needed once the child on the left-hand side of that pipe is forked. Once a file descriptor is no longer needed by the parent shell or any of its future children, it’s safe for the parent to close it with close(pfd[1]). (You could close that file descriptor later, too, but that might affect future child processes: remember that all redundant file descriptors to pipe ends require closing.)

    Step 7: close(4) in parent

  9. The parent can now create the right-hand child. In your code, this will typically happen with another call to command::run. You’ll need to figure out how to pass the read end of the pipe to that call.

    Step 8: fork() in parent

  10. The right-hand child now proceeds to reconfigure its environment, first with dup2, …

    Step 9: dup2(3, 0) in child

  11. …and then close.

    Step 10: close(3) in child

  12. The child’s environment matches our goal, so the child calls execvp to replace its program image with the new program.

    Step 11: execvp() in child

  13. Meanwhile, the parent shell closes its now-redundant read end of the pipe with close.

    Step 12: close(3) in parent

  14. We have arrived at our goal: a two-process pipeline. (Can you work out how to set up a three-process pipeline on your own?)

    Final state

Pipe philosophy

Literate programming

The literate programming challenge (1986)

“Given a text file and an integer k, print the k most common words in the file (and the number of their occurrences) in decreasing frequency.”

Jon Bentley; source

Donald Knuth’s solution

A literate program

Source

Doug McIlroy’s critique

tr -cs A-Za-z '\n' |
tr A-Z a-z |
sort |
uniq -c |
sort -rn |
head -n $1

“The … shell script was written on the spot and worked on the first try. … Knuth has shown us here how to program intelligibly, but not wisely. I buy the discipline. I do not buy the result. He has fashioned a sort of industrial-strength Faberge egg—intricate, wonderfully worked, refined beyond all ordinary desires, a museum piece from the start.”

Source