Weird disk space usage results.

[AK_Dude]

I am one of the administrators for a particular FreeBSD server that is having some bizarre file system usage errors. I received an alarm from a cron job that monitors disk usage, which stated that /tmp was at 100%, so I logged on to the box to find out why. All I got was confused: :D

AnonymousServer# df -k
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/aacd0s1a 257998 76290 161070 32% /
/dev/aacd0s1e 257998 236292 1068 100% /tmp
/dev/aacd0s1f 4129310 2242094 1556872 59% /usr
/dev/aacd0s1g 25700108 10090302 13553798 43% /var
procfs 4 4 0 100% /proc
AnonymousServer#

Okay, /tmp is still full. Let's see why:

AnonymousServer# du -k /tmp | sort -n
2 /tmp
AnonymousServer# du -k /tmp
2 /tmp
AnonymousServer# cd /tmp
AnonymousServer# du -k
2 .
AnonymousServer#

WT*? If /tmp is full, why isn't du -k showing anything? Okay, let's just look in the directory:

AnonymousServer# ls -a
. ..
AnonymousServer#

Okay, that's odd...Am I just on crack?

AnonymousServer# pwd
/tmp
AnonymousServer# ls -a
. ..
AnonymousServer#

Okay, I don't appear to be. Is /tmp still full?

AnonymousServer# df -k
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/aacd0s1a 257998 76290 161070 32% /
/dev/aacd0s1e 257998 237332 28 100% /tmp
/dev/aacd0s1f 4129310 2242094 1556872 59% /usr
/dev/aacd0s1g 25700108 10112416 13531684 43% /var
procfs 4 4 0 100% /proc
AnonymousServer#


Yep--more so than earlier. What's going on here?

[sharth]

a file could still be in use by some other program. run lsof (usually not installed, but it's a handy program) like this...

lsof | grep "/tmp"

[AK_Dude]

Originally posted by sharth lsof | grep "/tmp"

Doh! I should have thought of that.

We "solved" the problem with the Windows approach--reboot the machine. I still wish I knew what caused it, though...

[ph34r]

It is possible to have completely hidden files - I saw an example in C recently, just forget where.

Basically, open a file handle, then delete the file created. File handle is still open, so you can keep reading/writing to the file. When the app exists or you close the handle, it is truly gone... but in the mean time, it will eat whatever space you define.

Good luck hunting.

[sharth]

That's what lsof would solve. It shows all open files by programs, and then you pipe that through grep to see what has stuff open within /tmp.