Home Page
Posts > cPanel Hard Linked VirtFS Causing Quota Problems
cPanel Hard Linked VirtFS Causing Quota Problems
Quota problems are annoying and expensive to debug :-\
Time: 12/18/09 1:35:25 am
Tags: Quota, VirtFS, cPanel [6], Servers [31]
Most relevent 4 of 32 posts with shared tags:

For a really long time I’ve seen many cPanel servers have improper space reporting and quota problems due to the VirtFS system, which is used for the “jailshell” login for users (if a user logs into SSH [or presumably Telnet] whom has their shell set to “/usr/local/cpanel/bin/jailshell” in “/etc/passwd”). Whenever a user logs into their jailshell, it creates a virtual directory structure to chroot them into, and “hard links” their home directory inside this virtfs directory, which doubles their apparent hard disk usage from their home directory (does not include their sql files, for example). Sometimes, the virtfs directory is not unmounted (presumably if the user does not log out correctly, which I have not confirmed), so the doubled hard disk usage is permanent, causing them to go over their quota if a full quota update is ran.

I had given this up as a lost cause a while back, because all the information I could find on the Internet said to just leave it alone and not worry about it, and that it couldn’t be fixed. I picked the problem back up today when one of our servers decided to do a quota check update and a bunch of accounts went over. It seems a script was added recently at “/scripts/clear_orphaned_virtfs_mounts” that fixes the problem :-) (not sure when it was added, but the creation, modification, and access times for the files on all 3 of my cPanel servers shows as today...). Surprisingly, I could not find this file documented or mentioned anywhere on the Internet, and still no mentions anywhere on how to fix this problem.

So the simplest way to fix the problem is to run
/scripts/clear_orphaned_virtfs_mounts --clearall
I did some digging and found that that script calls the function “clean_user_virtfs” in “/scripts/cPScript/Filesys/Virtfs”, so you can just clear one user directory “properly” by running
cd /scripts
perl -e 'use cPScript::Filesys::Virtfs();cPScript::Filesys::Virtfs::clean_user_virtfs("ACCOUNTNAME");'
All this script really does is unmount the home directory inside the jailed failed system (so the hard link is done through a “mount” that doesn’t show up in “df”... interesting). So the problem can also be fixed by
umount -f /home/virtfs/USERNAME/home/USERNAME

After this is done, a simple
can be used to set the users’ quotas back to what they should be. Do note that this operation goes through every file on the file system to count users’ disk usages.

To add comments, please go to the forum page for this post (guest comments are allowed for the Projects, Posts, and Updates Forums).
Comments are owned by the user who posted them. We accept no responsibility for the contents of these comments.

Re: cPanel Hard Linked VirtFS Causing Quota Problems on 12/18/09 6:58am by Dakusan
I just noticed the VirtFS mounts are shown in /proc/mounts (cat /proc/mounts to view it). I'm not sure, but I think cPanel updated its jail system to use mounts now instead of some crazy form of directory hard links (which i think they used to) to help fix these problems, especially with a reboot.

Do note the last solution above (doing a manual umount) would only fix one of the many mounted directories. There include: lib, opt, usr/lib, usr/sbin, usr/share, usr/bin, usr/man, usr/X11R6, usr/kerberos, usr/libexec, usr/local/bin, usr/local/share, usr/local/Zend, usr/local/IonCube, usr/include, usr/local/lib, var/spool, var/lib, var/cpanel, usr/local/cpanel/Cpanel, var/run, var/log, bin, tmp, dev, proc

Also, it seems even after the user exits the jailshell, the mounted directories do not go away at all immediately. I'm not sure how long it takes for them to be cleaned up, if ever. Presumably when daily cron scripts run, perhaps?