Prior to Exchange 2003 SP1, if your SMTP protocol virtual server didn't have anywhere to go with a timed-out email, it would drop it in the BadMail directory on your server. This directory could potentially grow and grow without bounds until it filled up the disk, etc. There are two big changes to Badmail in the SP1 timeframe: one in SP1 directly, and one in the WebRelease “WR1”.
In the Web Release, a new script “Badmail Deletion and Archiving” was released that will aid in maintining the size and growth of this Badmail folder on your server. This will allow you to “trim” the directory down on a scheduled basis to keep the size under control.
However, even bigger is that by default in Exchange 2003 SP1 retaining undeliverable messages in the Badmail folder is now disabled and its behavior is controlled by a set of registry keys.
Set under: HKLM\System\CurrentControlSet\Services\SMTPSVC\Queuing (note the spelling: “Queuing“)
Value: MaxBadMailFolderSize This is a DWORD which defaults to 0 if not set. It is the max size (in KB) that the Badmail folder can grow to before Badmail retension is stopped; thus setting it to 0 turns off the Badmail folder and prevents possible DoS attacks. Messages that would have been sent to Badmail will simply be discarded if the MaxBadMailFolderSize is set to 0. If you set this to -1 (0xFFFFFFFF in hex) it will revert to the pre-SP1 behavior of growing without bound.
Value: BadMailSyncPeriodThis is a DWORD which specifies how often we should check the size of the badmail folder (in minutes). If the BadMailSyncPeriod has not been reached, we won't rescan the directory and discover if Badmail items have been removed manually by the Admin.