Tracking Down Deleted-But-Still-Open Files

The situation: df (which shows disk usage) disagrees with du (which shows disk space taken up by files). So you have disk space which is "used", but it is not taken up by any files!? What gives!?

For example:

root@xps:~# df -h /tmp
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/xps--vg-tmp  4.9G  4.6G     0 100% /tmp
root@xps:~# du -sh /tmp
708K    /tmp

Here we have a discrepancy of at least 21 Mb - more than can be accounted for by filesystem overhead. So there has to be another explanation.

If df and du ran at different times on an active system, it is possible that things changed in between those times. So it is usually worth double-checking that you are looking at up-to-date information before jumping to conclusions. But since you are still reading this, let us assume this that the information is up-to-date...

du and df can disagree due to open files which have been deleted: Linux allows files to be deleted while they are still open (the disk space allocated to the file will not be returned to the "free" pool until the file is closed) This makes the file "invisible" to du (because it no longer appears in any directory), but df shows the disk space as occupied.

So how do we find such files? They will not appear in any directory because they have been deleted, so looking for them in the file system is futile.

Here lsof (in the Debian package of the same name) comes to the rescue.

lsof /tmp | grep deleted | sort -n -k7 -r

This might give output looking like this:

bash      11164       karl    4w   REG 253,12 4909318144   57 /tmp/somefile (deleted)
firefox-e 31391       karl  144r   REG 253,12     939693   46 /tmp/tmp-blm.xpi (deleted)
gnome-ter 24045       karl   15u   REG 253,12     262144   50 /tmp/#50 (deleted)
gnome-ter 24045       karl   23u   REG 253,12     196608   60 /tmp/#60 (deleted)
gnome-ter 24045       karl   20u   REG 253,12     196608   21 /tmp/#21 (deleted)
gnome-ter 24045       karl   17u   REG 253,12      65536   61 /tmp/#61 (deleted)
gnome-ter 24045       karl   16u   REG 253,12      65536   58 /tmp/#58 (deleted)
apache2    7264   www-data    9u   REG 253,12          0   42 /tmp/.ZendSem.Agz5mK (deleted)
apache2    7263   www-data    9u   REG 253,12          0   42 /tmp/.ZendSem.Agz5mK (deleted)
apache2    7262   www-data    9u   REG 253,12          0   42 /tmp/.ZendSem.Agz5mK (deleted)
apache2    7261   www-data    9u   REG 253,12          0   42 /tmp/.ZendSem.Agz5mK (deleted)
apache2    7260   www-data    9u   REG 253,12          0   42 /tmp/.ZendSem.Agz5mK (deleted)
apache2    7259   www-data    9u   REG 253,12          0   42 /tmp/.ZendSem.Agz5mK (deleted)
apache2    1338       root    9u   REG 253,12          0   42 /tmp/.ZendSem.Agz5mK (deleted)

This lists the process which has the deleted files open - in this case, the disk space was being used by /tmp/somefile which was open by process id 11164. So stopping/killing that process would free up the disk space.