Always make sure your envelope sender is correct

Otherwise things may not work as you expect

When you send an email there may be multiple fields in the email header that specify the email address that it came from and how to reply back to that address. Some of these are:

• From: This is the field that the user sees in their email client as the "From" address. This field is the most easily (and most often) spoofable as you can put anything you want in this field and it doesn't change how the email is received or responded to. Most systems, in my experience, don't try to protect this field either.
• Envelope sender: This is used internally by email software to see who the email was really from. Different systems (i.e. spam blockers) can use this field for different purposes.
• Return path: This field specifies the email address to reply to when you click the "reply" button on your email client.

There can be multiple problems if the latter 2 field are not properly set. Some of these are:

• Spam blockers may be more likely to identify the email as spam
• The email might be sent from the wrong IP address. Exim (which cPanel uses by default) might be configured to check /etc/mailips to determine what IP address to send from depending on the domain of the envelope sender.
• The recipient might reply to the wrong email address when replying to the email.

When sending an email from PHP via the mail function through Exim you can only manually set the "From" header field (of the three) through the "additional_headers" (4th) parameter. This might be possible to remedy on some systems however.

If your server is configured to allow it (it may require privileged user permission), you can pass to the "additional_parameters" (5th) parameter of the mail function the -f Exim option, which sets the envelope sender and return path. For example:

mail('example@gmail.com', 'This is an example', 'Example!', 'From: example@yourdomain.com', '-f example@yourdomain.com');

On a related security note, if you think an email may not be legitimate, don't forget to check the email headers by viewing the original email source. Our servers include many useful headers in emails to help combat fraud including (depending on circumstances) the account the email was sent from, the IP address it was sent from, if it was sent from PHP, and if so, the script it was sent from.

cPanel Hard Linked VirtFS Causing Quota Problems

Quota problems are annoying and expensive to debug :-\

For a really long time I’ve seen many cPanel servers have improper space reporting and quota problems due to the VirtFS system, which is used for the “jailshell” login for users (if a user logs into SSH [or presumably Telnet] whom has their shell set to “/usr/local/cpanel/bin/jailshell” in “/etc/passwd”). Whenever a user logs into their jailshell, it creates a virtual directory structure to chroot them into, and “hard links” their home directory inside this virtfs directory, which doubles their apparent hard disk usage from their home directory (does not include their sql files, for example). Sometimes, the virtfs directory is not unmounted (presumably if the user does not log out correctly, which I have not confirmed), so the doubled hard disk usage is permanent, causing them to go over their quota if a full quota update is ran.

I had given this up as a lost cause a while back, because all the information I could find on the Internet said to just leave it alone and not worry about it, and that it couldn’t be fixed. I picked the problem back up today when one of our servers decided to do a quota check update and a bunch of accounts went over. It seems a script was added recently at “/scripts/clear_orphaned_virtfs_mounts” that fixes the problem :-) (not sure when it was added, but the creation, modification, and access times for the files on all 3 of my cPanel servers shows as today...). Surprisingly, I could not find this file documented or mentioned anywhere on the Internet, and still no mentions anywhere on how to fix this problem.

So the simplest way to fix the problem is to run

/scripts/clear_orphaned_virtfs_mounts --clearall

I did some digging and found that that script calls the function “clean_user_virtfs” in “/scripts/cPScript/Filesys/Virtfs”, so you can just clear one user directory “properly” by running

cd /scripts
perl -e 'use cPScript::Filesys::Virtfs();cPScript::Filesys::Virtfs::clean_user_virtfs("ACCOUNTNAME");'

All this script really does is unmount the home directory inside the jailed failed system (so the hard link is done through a “mount” that doesn’t show up in “df”... interesting). So the problem can also be fixed by

umount -f /home/virtfs/USERNAME/home/USERNAME

After this is done, a simple

/scripts/fixquotas

can be used to set the users’ quotas back to what they should be. Do note that this operation goes through every file on the file system to count users’ disk usages.

Core Dump Files

Not all OSs crash in the same way :-)

If you ever find a file named “core.#” when running Linux, where # is replaced by a number, it means something crashed at some point. Most of the time, you will probably just want to delete the file, but sometimes you may wonder what crashed. To do this, you use gdb (The GNU debugger), a very power tool, to analyze the core dump file.

Near the very bottom of the blob of outputted text after running this command, you should see a line that says “Core was generated by `...'.”. This tells you the command line of what crashed. To exit gdb, enter “quit”. You can also use gdb to find out what actually happened and troubleshoot/debug the problem, but that’s a very long and complex topic.

Recently, I started seeing hundreds of core dump files taking up gigabytes of space showing up in “/usr/local/cpanel/whostmgr/docroot/” on multiple of our web servers. According to several online sources, it seems cPanel (web hosting made easy!) likes to dump many, if not all, of its programs' core files into this directory. In our case, it has been “dnsadmin” doing the crashing. We’ve been having some pretty major DNS problems lately, this kind on the name server level, so I may have to rebuild our DNS cluster in the next few days. Joy.