When logs say nothing, look at the process directly.
# which files does it open, and where does it fail?
sudo strace -f -e trace=file -p 1234
# who has this port or file open?
sudo lsof -i :8080
sudo lsof +L1 # deleted files still held open (a classic disk-full cause)
# is the traffic arriving?
sudo tcpdump -i eth0 -nn port 443 -c 20| Tool | Answers |
|---|---|
strace |
What system calls the process makes, and which return errors such as ENOENT or EACCES |
lsof |
Which process holds which file, socket or port |
tcpdump |
Whether packets reach the host at all |
df shows a full disk but du cannot find the files? lsof +L1 often finds a deleted log still held by a running process; restarting that process frees the space. Use strace sparingly on busy production processes, because it slows them down.