DSpam: WebGUI modifications - Javascript goodies.

Ive been using dspam for two years now. It has been set up as a broad filter which checks all the emails passing through our systems. It has worked very well over the past year although going through the thousands of emails has been a chore.

The WebGUI for dspam is rudimentary, and is really built for single users to review their own spam/hams. It really isnt designed for large volumes. So I had to make some modifications.

Modification #1: Looking for missed SPAM.

The problem with the WebUI when you view the History tab is that it displays ALL the emails which pass through the system. This would be a normal requirement, but if you really are just interested in reviewing the False Negatives, i.e. SPAM that got away, it really takes ages to scroll through the pages one after another. If your domain has attracted alot of spam, over 70% of the entries are spam anyway, and you arent really interested in that information.
So the solution is to ignore the SPAM entries from the /var/dspam/.../dspam.log file. To do this, you will need to modify the /var/www/html/dspam.cgi file.

This is the patch (dspam.skipSPAM.patch):

With this patch, you can toggle the ability to skip through Spams by adding in another url argument &skipSPAM=true on the URL address. Otherwise, if you want this on by default, just make $skipSPAM = "true" in the perl script.

The result should look something like this:
Notice how the Spams are ignored, giving you a clear view of what to retrain or allow. I also skip through Whitelisted emails so that means less lines to review. I use Firefox's tab browsing, and just middle click the entries which I want to flag off as Spam. The process is very fast, and I probably need to click through about 3 pages of history to mark off any significant growth in spam.

Modification #2: Marking Dead meat the brute force way
One of the hassles of clearing False Positives is the process of going through each and every spam item and checking it off. Early on, I modified the nav_quarantine.html template file with this small Javascript which checked off the first 200 items. Here is the patch for the "Select 200" modification for the templates/nav_quarantine.html file (nav_quarantine.select200.patch)

This worked well to a certain extent, but when you have over 10K entries, refreshing the page 500 times is certainly not an option. There MUST be a better way. And there is ...

Modification #3: Marking Dead meat the elegant way.
Blindly selecting the first 200 entries isnt really an efficient way of culling the confirmed spam. I needed an almost automated way to handle this. So I embedded more information from dspam into the WebUI and written some Javascript to make this process alot more bearable.

The first requirement is to remove all the marked spam of a given percentage of certainty. Throughout the entire production usage of DSpam, I have yet to see a False Positive with a certainty score of more than 70%. What would be great is to check off all entries given a score automatically. This is now possible by entering a confidence number, and simply clicking on the "Mark Rating" button.
What the script does is that it uses XPath to query out all rows which have a rating of more than what is entered. The Javascript code looks something like this:

var pRate = parseFloat( document.getElementById("rating_val").value ) / 100;
var xpath = document.evaluate( "//tr[\@rating > "+pRate+" ]", document, null, +XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null );

I have also added an extra feature in that it will also mark of similar items which may have less of a confidence than provided. This will be elaborated below, in the "hash" I generate with all the entries later. Because of this 'recursive' behaviour, the script will take a while to complete, so you may need to increase the timeout for Firefox (otherwise it may complain with a "A script on this page may be busy, or it may have stopped responding. You can stop the script right now, or you can continue to see if it completes.") To do so, type about:config in the URL bar, and adjust the dom.max_script_runtime from the default of 10 to something larger like 500.

Modification #4: Ajax waits for no refresh.
Another tedious part about using the WebUI is that whenever you need to purge the quarantine of the caught spam, it takes ages because it causes an entire page refresh. Its OK if the list is less than a thousand, but when it reaches 20K or more, its just too much.

Then we come to another problem when deleting the entries from the /var/dspam/.../dspam.mbox file. As you remove entries from the file, if at any point during that time, a new email arrives, the deletion will cancel and the file will roll back to its original state plus new email. So realistically on a busy system, you can't delete more than 50 spams at a go. This means we will have to endure ALOT of page refreshes.

What I implemented then was a AJAX type handler for dspam.cgi to execute. I added the Javascript features in the WebUI, and it looks like this:
As you click the button, the Javascript will scroll through the checked list, and when it compiles 25 entries, it forms a query back to dspam.cgi to execute in the background. It will alert the user by stating it is currently "deleting 25". When the call is successful, it will state "deleted 25". It will then repeat the process if there are still items to be checked.

The figure 25 is something which I found to be small enough to cater for non roll backs, and because the process is automated, it doesnt need to be large. So to clear off 15K entries, it takes about 30 minutes to an hour.

Modification #5: Hashing up spam
For the remaining spam which isnt obvious, I have included two little clickies on the end of the table. "del" deletes the entry immediately, while "hash" checks the entry's checkbox, and all entries with similar subjects. This means you can click off multiple spams with just one click as demonstrated below:

This makes marking off spam almost ... fun!

Patching the WebGUI
I include with this post three patches. In the dspam.cgi directory, run this:
# patch < dspam.skipSPAM.patch
# patch < dspam.ajax.patch

and in the template directory
# patch < nav_quarantine.select200.patch

Otherwise dspam.cgi and template/nav_quarantine.html are also available.

How I use these modifications
Whenever I have the time to review the spam collection

1. I load up the quarantine page until its fully loaded.
   
2. I then click on "Mark Spam" with the default rate of 85%.
   
3. This takes a few seconds depending on your PC.
   
4. I then click on the "Ajax delete" button to start the deletion process in the background.
   
5. In the meantime, I reduce the rate down to 70% and sometimes 60% to clear off further spam.
   
6. I also start from the top, i.e. 47% confidence spam items, and slowly review the items up to about 53%, clicking on the "hash" to remove the spam items.
   
7. After I clean off the False Positives (if any) I click on the "Select 200" and eyeball the remaining items until there are no entries left.
   
8. It still takes some time, but at least its a whole less time than before!

I hope this helps!

yk

How to make nice looking diffs

I was wondering how to make nice looking diff patch files yesterday, as using "diff " gave really cryptic outputs and not very user friendly. I IM'ed Aizat who happened to be online in Chile. He just said use "svn diff". I told him that I was working on files local on my machine, so svn was not appropriate.

Googling didn't help much. So I just submitted the standard diff output as my patch.

Then this morning, Ow had a blog post about his patch, and he included his command line. The answer is "diff -Nau"!

So here is the patch for the archivemail-dspam script:

yky@x1407:~/dspam$ diff -Nau archivemail archivemail-dspam
--- archivemail 2007-12-18 19:13:34.000000000 +0800
+++ archivemail-dspam   2007-12-18 19:02:47.000000000 +0800
@@ -187,6 +187,8 @@
    min_size             = None
    verbose              = 0
    warn_duplicates      = 0
+    """ 071218   yky  DSPAM-Confidence setting """
+    spam_confidence = 0.00

    def parse_args(self, args, usage):
        """Set our runtime options from the command-line arguments.
@@ -206,7 +208,7 @@
                             "filter-append=", "pwfile=", "dont-mangle",
                            "archive-name=",
                             "preserve-unread", "quiet", "size=", "suffix=",
-                             "verbose", "version", "warn-duplicate"])
+                             "verbose", "version", "warn-duplicate", "spam=" ])
        except getopt.error, msg:
            user_error(msg)

@@ -256,6 +258,8 @@
                self.verbose = 1
            if o == '--archive-name':
                self.archive_name = a;
+            if o == '--spam':
+                self.spam_confidence = float(a)
            if o in ('-V', '--version'):
                print __version__ + "\n\n" + __copyright__
                sys.exit(0)
@@ -265,7 +269,7 @@
        """Complain bitterly about our options now rather than later"""
        if self.output_dir:
            check_sane_destdir(self.output_dir)
-        if self.days_old_max <>= 10000:
            user_error("--days argument must be less than 10000")
@@ -661,6 +665,7 @@
      --include-flagged messages flagged important can also be archived
      --no-compress     do not compress archives with gzip
      --warn-duplicate  warn about duplicate Message-IDs in the same mailbox
+      --spam=FLOAT      SPAM Confidence levels ( e.g. 0.80  )
  -v, --verbose         report lots of extra debugging information
  -q, --quiet           quiet mode - print no statistics (suitable for crontab)
  -V, --version         display version information
@@ -737,6 +742,22 @@
    mbox_from = "From %s %s\n" % (address, date_string)
    return mbox_from

+
+def get_spam_confidence(message):
+       """Returns the DSPAM_Confidence from the message headers. Zero by default"""
+       """ 071218  yky  Created """
+
+       assert(message != None)
+
+       for header in ('X-DSPAM-Confidence', 'SPAM-Confidence'):
+               confidence = message.get(header)
+               if confidence:
+                               confidence_val = float( confidence )
+                               if confidence_val:
+                                       vprint("Spam Confidence: %f " % confidence_val)
+                                       return confidence_val
+
+       return 0.0

def guess_return_path(message):
    """Return a guess at the Return Path address of an rfc822 message"""
@@ -987,6 +1008,11 @@
        return 0
    if options.preserve_unread and is_unread(message):
        return 0
+
+        # 071218  yky  Filtering by SPAM Confidence       
+    if (options.spam_confidence > 0) and (options.spam_confidence > get_spam_confidence(message)):
+               return 0
+   
    return 1
       
   
@@ -1019,7 +1045,7 @@
    max_days -- maximum number of days before message is considered old
      
    """
-    assert(max_days >= 1)
+    assert(max_days >= 0)

    time_now = time.time()
    if time_message > time_now:

Thanks Ow!

yk.

Making Archivemail work with DSpam

Ive got an dspam "appliance" where the enterprise emails filter through. I've set it up so that only one dspam user is used to filter all the emails. This has worked well over the past few years, but managing it has been quite a chore. Every morning, I'd have to wade through the emails in the quarantine (about 15K), and free up any False Positives which were caught.

Beyond the 58% spam confidence as reported by DSpam is pretty much spam. Below that, between the 47% - 57% there may exist one or two False Positives.

After freeing them up, deleting the remaining emails is a huge chore, because the DSpam UI will not allow deleting the quarantine file when new spam pops in.

So I needed a little program which would scan the quarantine mbox file and delete off any messages which are 58% or higher spam confidence.

I tried the most obvious program called 'archivemail', which was readily available in all distros, but was disappointed that it only allowed filtering on the messages age. There was a mysterious "Filter" switch but it only applied to IMAP mailboxes.

The great thing about this is that archivemail, like the entire emailling stack on my servers, is its completely Free Software. I just had to invest some time to look at the code. archivemail lived in /usr/bin/. I had a look at the file, and its a very small 1500 line python script!

I haven't programmed in python before, but looking at the code, it didn't look too scary. It had classes, but no colons. Indentation seemed to be important here. I scanned the code, and I found the little function called "should_archive(message)". And sure enough, the crux of the logic which defines whether a message is to be archived away or not, was there.

So I added this line:

if (options.spam_confidence > 0)
  and (options.spam_confidence > get_spam_confidence(message)):
     return 0

And modified the options class to include the spam_confidence field. Did some modifications on the code to read in the command line options, and then had to create the section which extracts the spam confidence from the message headers. Doing this was relatively easy, because the rest of the code basically does the same things: reading things off the headers and using the information. So my new function looked like this:

def get_spam_confidence(message):
"""Returns the DSPAM_Confidence from the message headers. Zero by default"""
""" 071218  yky  Created """

assert(message != None)

for header in ('X-DSPAM-Confidence', 'SPAM-Confidence'):
 confidence = message.get(header)
 if confidence:
   confidence_val = float( confidence )
   if confidence_val:
    vprint("Spam Confidence: %f " % confidence_val)
    return confidence_val    
   
return 0.0

Thats it!

I also set some cronjobs to run against the quarantine file; to kill 88% and above spams every hour, kill 58% spams after 3 days, and kill the rest if they are more than 14 days old.

I then followed up with my corporate responsibility duties, and submitted the patch back to the archivemail project in sourceforge. This didn't take me long, and it is worth while whether they accept it or not. At least the source is available online.

I hope this helps other dspam admins out there too!

yk.

Little Tricks with Xen images from jailtime.org

Ive been busy setting up a CentOS server. After the successful installation of dspam on our FC5 Xen virtual machine, we have decided to move it from our development server (a frankenstein of a machine) to a proper host. Yes, Ive been testing it for over a year now!

Postfix with MySQL support in CentOS 5

So I downloaded a prebuilt CentOS image from jailtime and installed the necessary tools. A little trick I learnt was the inclusion of the extra repositories. dspam requires a postfix installation with MySQL support. To do this, you can either compile from source, but being the lazy git I am, I'd rather download the binaries.

To do this, you just need to enable the CentOSPlus repositories, which can be done via a command line switch:

# yum --enablerepo=centosplus upgrade postfix

Yum will then work out the requisites and download the stuff for you. Unfortunately it also brings down the postgresql binaries. So unless you are in dire need of hard disk space, don't do it this way.

We aren't in Kansas anymore, Toto

Another thing about the jailtime images is that the hwclock scripts are modified to return 0. This is because there is no hardware clock. It uses the host's (Dom0) time. So to get the correct time, make sure you change the zone information in /etc/localtime. For us users in Kuala Lumpur, I have to do this:

# ln -s /usr/share/zoneinfo/Asia/Kuala_Lumpur /etc/localtime

Keeping it 32bits

My Xen host is a AMD 64 server. My Xen clients are i386 virtual machines. This is for "portability" reasons. My previous post about using yum to install i386 binaries can be improved. Now use setarch. To install:

# yum install setarch.i386

subsequently make sure you run it before yumming some more.

# setarch i386

If you forget and accidentally call yum, mixing i386 and x86_64 info, the downloaded headers will be "confused". You'd have to do a clean:

# yum clean all


yk.

SPAM: No Listing

Ive read about the concept on No Listing awhile back, but never ventured in setting it up for the domains I look after. Reading xWingz post inspired me to try it out for real.

We currently have a few lines and therefore a few separate IP addresses within the MX groupings. So it was relatively easy for us to try this out. We just disabled Port 25 (smtp) on the main mail server. We initially just left it at that, without a response to the request to the sender. This meant that the delivery was delayed, as the sender had to wait one to two minutes before timing out to the "dummy" mailserver.

You can test this by telnetting to the port: 'telnet mail.host.com 25' and that would take forever before it returns any results.

Therefore, to be a better 'netizen, and as recommended by the No listing page, we have requested the server to return a "Reject" on any connections to port 25. This means that the connections are immediately terminated, and non-spam senders will immediately try MX record #2.

Additionally, instead of doing this blind, we have current statistics of spam rates collected over the year on some selected email accounts. So here is the graph of spam (red) vs ham (everything else) for the past two weeks:


As you can see spam for these accounts are at a "healthy" 80-90%. We shall see in a few weeks if the changes made to the MX records make a difference to our spam rates!

yk.