Unix Programming

For a DevOps engineer this may be the most directly practical subject of the whole degree. Linux is a Unix-like system, and almost everything you do on servers, in containers and in CI pipelines is the shell, files, processes, permissions and pipes described here. This article revisits the five units of the JNTUK course, and each unit ends with how it shows up at work.

Unit 1: The Unix system and the shell

History and philosophy

Unix was created at Bell Labs in 1969 (Ken Thompson and Dennis Ritchie) and rewritten in C. Linux (1991) is a Unix-like kernel, combined with GNU tools. The Unix philosophy:

• Write programs that do one thing well.
• Make programs work together, through plain text and pipes.
• Treat everything as a file: ordinary files, directories, devices, pipes and sockets.

Architecture

Layer	Role
Hardware	CPU, memory, devices
Kernel	Process, memory, file and device management; system calls
Shell	The command interpreter (bash, zsh, sh)
Utilities and applications	ls, grep, editors, compilers

The file system layout

A single tree rooted at /.

Directory	Contents
/bin, /usr/bin	Commands
/etc	Configuration files
/home	User directories
/var	Variable data: logs, spool, caches
/tmp	Temporary files
/dev	Device files
/proc, /sys	Kernel and process information (virtual)
/lib, /usr/lib	Libraries

Basic commands

ls, cd, pwd, cp, mv, rm, mkdir, cat, less, head, tail, wc, man, who, date. Commands take options and arguments; man shows the manual. Wildcards (*, ?, [a-z]) are expanded by the shell, and quoting controls that.

Users, groups and permissions

Every file has an owner, a group and permissions for user, group and others: read (r = 4), write (w = 2), execute (x = 1).

Example	Meaning
rwxr-xr-x = 755	Owner can read, write, execute; others read and execute
rw-r--r-- = 644	Typical file
chmod 600 key.pem	Only the owner can read and write (needed for SSH keys)

chmod changes permissions, chown changes ownership, umask sets defaults, and the special bits setuid, setgid and the sticky bit (as on /tmp) exist for special cases. sudo runs a command as root.

Today: SSH refusing a private key because permissions are “too open” (needs 600), Permission denied in a container and chmod +x script.sh are everyday examples. The “everything is a file” idea is why /proc and /sys are used for monitoring, and /dev/null for discarding output.

Unit 2: Shell scripting and text processing

Redirection and pipes

Every process has three standard streams: stdin (0), stdout (1) and stderr (2).

Syntax	Effect
cmd > file	Write stdout to a file (overwrite)
cmd >> file	Append
cmd < file	Read stdin from a file
cmd 2> err.log	Redirect errors
cmd > out 2>&1	Both to one file
cmd1 | cmd2	Pipe: stdout of one becomes stdin of the next

Example: ps aux | grep nginx | wc -l counts nginx processes.

Text tools (filters)

Tool	Purpose
grep	Search lines that match a pattern (regular expressions; -i, -v, -r, -n)
sed	Stream editor: substitute (sed 's/old/new/g'), delete lines
awk	Pattern-action processing of columns (awk '{print $1}', sums, averages)
sort, uniq, cut, tr, tee, xargs, find	Order, deduplicate, extract fields, translate, split output, build commands, search files

Shell scripting (bash)

A script starts with #!/bin/bash and is made executable with chmod +x.

Feature	Example
Variables	name="web", use as $name; command substitution $(date)
Arguments	$1, $2, $@, $#; exit status $?
Conditions	if [ -f file ]; then ... fi; case
Loops	for f in *.log; do ...; done; while read line
Functions	backup() { ...; }
Tests	-f file exists, -d directory, -z empty string, -eq number equal
Safety options	set -euo pipefail stops on errors and unset variables

A tiny example: a loop that compresses old logs: for f in /var/log/app/*.log; do gzip "$f"; done. Always quote variables ("$f") so spaces do not break the command. Cron runs scripts on a schedule (0 2 * * * /opt/backup.sh), and regular expressions appear in grep, sed and awk.

Today: CI jobs, Dockerfile RUN lines, Jenkins stages, entrypoint scripts and cron-based backups are shell scripts. set -euo pipefail, kubectl get pods | grep -v Running and awk one-liners over logs are part of daily troubleshooting. For anything complex, Python or Go is a better choice than a long shell script.

Unit 3: Files and directories: the file-system API

Files, inodes and links

A Unix file system stores each file as an inode (metadata and block pointers) plus data blocks. The directory maps names to inode numbers. A hard link is another name for the same inode (cannot cross file systems); a symbolic link is a small file pointing to a path. A file is deleted when its link count reaches zero and no process has it open.

System calls for files

Files are accessed through file descriptors: small integers (0, 1, 2 are the standard streams).

Call	Purpose
open(path, flags, mode)	Get a file descriptor
read(fd, buf, n), write(fd, buf, n)	Transfer data
lseek(fd, offset, whence)	Move the file position
close(fd)	Release the descriptor
stat(path, &st)	Read metadata (size, owner, mode, times)
dup, dup2	Duplicate descriptors; used to implement redirection
unlink, rename, link, symlink, mkdir, rmdir, chmod, chown	Manage the namespace and attributes
opendir, readdir, closedir	Read a directory
fcntl, flock	File control and locking

The C standard I/O library (fopen, fread, fprintf, fclose) adds buffering on top of these calls. Unbuffered (system call) and buffered (stdio) I/O differ in speed and in when data reaches disk (fsync forces it).

Mounting and devices

A file system is mounted at a directory with mount; df shows usage and du shows directory size. Special files for devices are block or character devices. /etc/fstab lists mounts at boot. Tools such as find, lsof (lists open files) and tar and rsync (archive and copy) are part of the daily kit.

Today: “No space left on device” can mean inodes, not bytes (df -i); a disk that stays full after deleting a big log means a process still holds the file open (lsof | grep deleted). “Too many open files” is a file-descriptor limit (ulimit -n). Docker volumes and Kubernetes PersistentVolumes are mounts, and fsync behaviour matters to databases.

Unit 4: Processes and signals

Process creation

Call	Purpose
fork()	Create a child process that is a copy of the parent; returns 0 in the child and the child’s PID in the parent
exec family (execl, execvp, …)	Replace the current program with a new one
wait(), waitpid()	Parent collects the child’s exit status
exit()	End the process, with a status code (0 = success)

The shell runs a command by fork then exec. An orphaned child is adopted by init (PID 1); a child that has exited but not yet been waited for is a zombie. A daemon is a background process detached from any terminal. Each process has a PID, a parent (PPID), real and effective user IDs, an environment and a working directory. ps, top, htop, kill, nice and jobs, bg, fg manage them.

Signals

A signal is an asynchronous notification sent to a process.

Signal	Number	Meaning	Can it be caught?
SIGINT	2	Interrupt (Ctrl+C)	Yes
SIGTERM	15	Polite request to terminate (the default of kill)	Yes
SIGKILL	9	Forced kill	No
SIGSTOP / SIGCONT	19 / 18	Pause and resume	SIGSTOP: no
SIGHUP	1	Terminal hung up; often “reload configuration”	Yes
SIGSEGV	11	Invalid memory access	Yes (usually fatal)
SIGCHLD	17	A child process ended	Yes
SIGPIPE	13	Writing to a closed pipe	Yes

A program installs a handler with signal() or the more reliable sigaction(). Handlers must do very little (they interrupt normal code), and signals can be blocked or masked. kill -9 should be the last resort because the process cannot clean up.

Sessions and job control

A process group and session tie a shell’s jobs together, with a controlling terminal. nohup and & run commands in the background; tmux and screen keep sessions alive.

Today: when Kubernetes stops a pod it sends SIGTERM, waits for the grace period and then SIGKILL, so applications must handle SIGTERM to shut down gracefully. In a container, the main process is PID 1 and needs to forward signals and reap zombies (use tini or --init). systemd manages services on modern Linux, replacing the traditional init. kill -HUP makes nginx reload its configuration.

Unit 5: Inter-process communication and sockets

Processes are isolated, so they need explicit ways to exchange data.

Mechanism	Description	Use
Pipe (pipe())	One-way byte stream between related processes	Shell pipelines
FIFO (named pipe)	A pipe with a file-system name; works between unrelated processes	Simple local channels
Message queues	Typed messages in a kernel queue	Structured local messaging
Shared memory	A memory region mapped into several processes	The fastest IPC; needs synchronisation
Semaphores	Counters for synchronisation (SysV and POSIX)	Protect shared memory
Signals	Simple notifications	Control, not data
Sockets	Two-way communication, local or across the network	Everything networked

Sockets

A socket is an endpoint for network communication. The classic TCP server sequence is:

1. socket() creates the endpoint.
2. bind() attaches it to an address and port.
3. listen() starts accepting connections.
4. accept() returns a new socket for each client.
5. read() / write() (or recv/send) exchange data; close() ends it.

The client calls socket() then connect(). UDP uses sendto and recvfrom with no connection. Unix domain sockets (a path such as /var/run/docker.sock) provide local IPC. Addresses use network byte order (htons, htonl) and name lookup (getaddrinfo).

Handling many clients

Approach	Idea
Fork per connection	Simple, heavy
Threads	Lighter, need synchronisation
I/O multiplexing	select, poll, and on Linux epoll: one thread watches many sockets
Event-driven, non-blocking I/O	The model behind nginx and Node.js

Threads (POSIX)

pthread_create, pthread_join, mutexes (pthread_mutex_lock), condition variables and semaphores give shared-memory parallelism inside one process.

Today: the Docker daemon is controlled through a Unix socket, and access to it is effectively root access. ss -tlnp lists listening sockets with their processes, “address already in use” is a bind error and a high number of connections in TIME_WAIT or CLOSE_WAIT is a socket-handling problem. nginx, Redis and Node.js use epoll to serve thousands of connections per thread. Pipes, sockets and shared memory are also how containers and sidecars talk on the same host.

One-minute recap

Unit	Key idea	You meet it in
1. Unix and the shell	Everything is a file; permissions rwx	SSH key permissions, /proc, /dev/null
2. Shell scripting	Pipes, redirection, grep, sed, awk, bash scripts	CI jobs, entrypoint scripts, cron, log analysis
3. Files and directories	Inodes, links, file descriptors, open/read/write	df -i, deleted-but-open files, ulimit -n
4. Processes and signals	fork and exec, zombies, SIGTERM vs SIGKILL	Graceful pod shutdown, PID 1 in containers
5. IPC and sockets	Pipes, shared memory, socket calls, epoll	Docker socket, ss, nginx and Node.js servers

Next in Engineering Revisited: Distributed Databases.

---

Written by Claude, Anthropic’s AI assistant, as part of Engineering Revisited, a quick refresher of the B.Tech subjects one at a time.