September 10, 2008

Finding deleted files in solaris

Today I was doing some routine cleaning-up of /tmp directories, when I ran into a problem:

I spotted a big file that could be deleted: /tmp/logfiles/00000001/00000001 . It’s 1.6GB, and all it’s doing is eating up swap space.

So, I run rm /tmp/logfiles/00000001/00000001, and the file goes, but df -h /tmp shows that the space is in use.

Bugger. Some process somewhere still has an open file handle on the file. The space won’t get reclaimed until I can find and kill that process.

lsof could do this, but for various tedious reasons I don’t have lsof on this box.

However, 1.6GB is unusually big for a file. There can’t be many processes with a handle on a file that’s more than 1GB in size can there?

Procfs to the rescue! /proc in solaris has a directory for every process running on the system, and within that an FD directory with a magical link to every open file handle. Even if the file has been rm-ed, the df entry will still be able to retreive it’s bytes. fd entries don’t name the files, but they do list their sizes.

So, all I need to do is look through all the FD directories, for a file of ~ 1.6 GB in size, and that will give me the process ID that’s got it open…

take 1:

for file in `ls -l  | grep root | awk '{print $9}' `; do echo "PID $file"; ls -l $file/fd 2>/dev/null; done | awk '$5 > 1000000 || /^PID/' | grep -v ','

clunky, but it works! It gives me back a PID, and running ‘pargs’ on the PID, I get

-bash-3.00# pargs 1859
1859:   less logfiles/00000001/00000001
argv[0]: less
argv[1]: logfiles/00000001/00000001

- someone {ahem} tried to less the file earlier, and gave up when it didn’t work :-).

A quick kill, and my swap space is back again.

Now, I’m fairly sure that I could do the same thing with a single ‘find’ incantation. That can wait for v2, though…

- No comments Not publicly viewable

Add a comment

You are not allowed to comment on this entry as it has restricted commenting permissions.

Most recent entries


Search this blog

on twitter...


    Not signed in
    Sign in

    Powered by BlogBuilder
    © MMXXI