Unmounting a volume with open file handles to deleted files without killing the process

By samson7point1
Recently we've been turning on compression in our Oracle databases because it saves us a ton of disk space and actually improves the performance of the databases.  Among other things the process involves writing the newly compressed tablespaces to new data files (the systems in question are not using ASM).  Because we're moving the DB's from where they are, that gives us the opportunity to do some housecleaning before moving them back, such as un-mounting, checking and possibly resizing the existing filesystems.

Occasionally it will occur that after the data has been migrated away and the old data files deleted, that Oracle still has open handles to the deleted files.

So the DBA tells me the filesystem is clear and I can have it, but then this happens:

[root@kwt-r3oql00 E1Q]# umount /dev/mapper/vg_kwt_r3oql20_s00-oracle_E1Q_sapdata7
umount: /oracle/E1Q/sapdata7: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

And when we go to see who the culprit is:

[root@kwt-r3oql00 E1Q]# lsof /oracle/E1Q/sapdata7
COMMAND     PID   USER   FD   TYPE DEVICE    SIZE/OFF     NODE NAME
oracle_13 13214 oracle  520u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_13 13254 oracle  519u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_13 13258 oracle  520u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_13 13286 oracle  517u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_13 13294 oracle  520u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_13 13526 oracle  506u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_13 13996 oracle  512u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_14 14424 oracle  493u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_14 14488 oracle  515u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_14 14574 oracle  403u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_14 14712 oracle  506u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_23 23296 oracle  273u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_24 24260 oracle  277u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)


This is super annoying because the DB seems to have forgotten about those open handles and internally has no mechanism for gracefully releasing them.  If you can tolerate the outage restarting the DB will release those handles, but if you can't there is another solution.

The GNU debugger (gdb) will allow you to attach to the process and close the file handles.  It's not without risk, but I've been using it successfully for years.  The main drawback is how tedious it is to sift through /proc looking for the open file handles and closing them one by one with the debugger interactively.

Laziness being the father of invention, I wrote the following script to do this for you.  Simply provide it the path to check for open deleted file handles, and it will find the processes, attach the debugger and remove the handles.


#!/bin/bash

FS=$1

if [[ -z $FS ]]; then
   echo "Please provide a filesystem path to check for open file descriptors to deleted files"
   exit 1
fi

echo "WARNING: This is super-dangerous.  Please don't use it in Prod without a "
echo "         really good reason and a change request/blackout"
read -p "Type 'C' and Enter to continue, anything else to abort: " CHECK

if [[ "$CHECK" != "C" ]]; then
   echo "Aborted by user.  No changes made."
   exit 0
fi

# Get a list of processes with open file handles to the given directory
PIDLIST=`lsof $FS | egrep -v 'PID' | awk '{print $2}' | sort -u`


for PID in $PIDLIST; do

  unset DESCLIST

  # Get a list of file descriptors for that PID that refer to deleted files
  DESCLIST=`ls -l /proc/${PID}/fd | grep deleted | awk '{print $9}'`

  # Display the list
  echo "The Process $PID has open file descriptors for deleted files:"
  echo "${DESCLIST}" | sed 's/^/  /g'

  # Create a name for the script for gdb
  DEBUGSCRIPT=/tmp/.$PID

  # Remove any previous version of the script
  if [[ -f $DEBUGSCRIPT ]]; then /bin/rm $DEBUGSCRIPT; fi

  # Write a close command for each deleted file descriptor
  for DFD in $DESCLIST; do
     echo "p close(${DFD})" >> $DEBUGSCRIPT
  done

  # Detach and close the debugger
  echo "detach" >> $DEBUGSCRIPT
  echo "quit" >> $DEBUGSCRIPT

  echo "Forcibly closing handles for deleted files on process $PID"

  # Run the debugger in batch mode to execute the script and close the handles
  /usr/bin/gdb --pid $PID --batch -x $DEBUGSCRIPT

  # Wait before deleting the script
  sleep 1

  # Clean up the script
  /bin/rm $DEBUGSCRIPT

done



Comments

Post a Comment

Popular posts from this blog

What I've Learned Fixing Optical Drives

By samson7point1
So You Want to Fix an Optical Drive If you found this page looking for advice on how to fix an optical drive, you might be in luck.  There's a lot of information and prognostication out there these days that might tell you fixing a laser mechanism is a losing proposition, but the fact is that in many cases these drives are completely fixable. Over the past decade I've begun fixing a optical drive failures in a wide variety of hardware. In the process I've learned a thing or two about a thing or two and I've compiled my findings here in the hopes they may help someone else. Full disclosure: I'm a technical professional, just not in the field of electrical engineering.  I do this as a hobby.  My skills and information comes from a lifetime of curiosity and decades of reading manuals, reading books, learning from experts and practical application. My success rate in repairing these drives and restoring them to original function seems to be in direct opposition to popul
Read more

FM Towns PC ODE - Grey Tower Wizard Installation Notes And Review

By samson7point1
Image
The Gray Tower Wizard is an Optical Drive Emulator (ODE) for FM Towns PCs, not to be confused with the FM Towns Marty (which is a stripped-down and consolized version of the PCs) or the Doc Brown ODE for Marty (which is made by the same person).  The FM Towns PCs have dual floppy drives and the ability to add RAM which makes them capable of running more complex or resource-heavy games than the Marty. How Do You Get One? The kit including shipping ran me 150 Euro.  Ordering one is basically the same process as the GDEmu, Rhea, Phoebe and Doc Brown.  I had never ordered anything from Deunan before but I'm more than familiar with the popular complaints about it being difficult.  It's not like ordering something from Amazon, but I didn't find it to be very difficult at all.   I signed up for blog updates, waited until he announced he was going to sell them, then refreshed the Wizard Ordering page  until he activated it.  When he did, the ordering page changed to a form to submi
Read more

ColecoVision RGB Mod [Updated]

By samson7point1
Image
[Updated June 2020]  As with all hobbies these things can be learning processes. After initially completing this mod, I discovered some new things and made some changes.  Primarily, I've re-incorporated the RF box for actual composite video, figured out the black & white picture problem, added a buffering capacitor to the audio output and tapped the audio from a different location to maintain compatibility with the ColecoVision Expansion Module #1.  The Expansion module doesn't work with RGB out, but now it's at least usable with composite Genesis 2 cables.  Background ColecoVision wasn't something I had or even played growing up, but several years ago someone gave me one.  Since its only AV output is RF and that makes it hard to have multiple systems connected at once, it sat in a box for about 10 years until I stumbled across Ben Heck's tutorial on a very straightforward composite video mod. The Picture Quality Produced by the Ben Heck composite vide
Read more