How to make nice looking diffs

I was wondering how to make nice looking diff patch files yesterday, as using "diff " gave really cryptic outputs and not very user friendly. I IM'ed Aizat who happened to be online in Chile. He just said use "svn diff". I told him that I was working on files local on my machine, so svn was not appropriate.

Googling didn't help much. So I just submitted the standard diff output as my patch.

Then this morning, Ow had a blog post about his patch, and he included his command line. The answer is "diff -Nau"!

So here is the patch for the archivemail-dspam script:

yky@x1407:~/dspam$ diff -Nau archivemail archivemail-dspam
--- archivemail 2007-12-18 19:13:34.000000000 +0800
+++ archivemail-dspam   2007-12-18 19:02:47.000000000 +0800
@@ -187,6 +187,8 @@
    min_size             = None
    verbose              = 0
    warn_duplicates      = 0
+    """ 071218   yky  DSPAM-Confidence setting """
+    spam_confidence = 0.00

    def parse_args(self, args, usage):
        """Set our runtime options from the command-line arguments.
@@ -206,7 +208,7 @@
                             "filter-append=", "pwfile=", "dont-mangle",
                            "archive-name=",
                             "preserve-unread", "quiet", "size=", "suffix=",
-                             "verbose", "version", "warn-duplicate"])
+                             "verbose", "version", "warn-duplicate", "spam=" ])
        except getopt.error, msg:
            user_error(msg)

@@ -256,6 +258,8 @@
                self.verbose = 1
            if o == '--archive-name':
                self.archive_name = a;
+            if o == '--spam':
+                self.spam_confidence = float(a)
            if o in ('-V', '--version'):
                print __version__ + "\n\n" + __copyright__
                sys.exit(0)
@@ -265,7 +269,7 @@
        """Complain bitterly about our options now rather than later"""
        if self.output_dir:
            check_sane_destdir(self.output_dir)
-        if self.days_old_max <>= 10000:
            user_error("--days argument must be less than 10000")
@@ -661,6 +665,7 @@
      --include-flagged messages flagged important can also be archived
      --no-compress     do not compress archives with gzip
      --warn-duplicate  warn about duplicate Message-IDs in the same mailbox
+      --spam=FLOAT      SPAM Confidence levels ( e.g. 0.80  )
  -v, --verbose         report lots of extra debugging information
  -q, --quiet           quiet mode - print no statistics (suitable for crontab)
  -V, --version         display version information
@@ -737,6 +742,22 @@
    mbox_from = "From %s %s\n" % (address, date_string)
    return mbox_from

+
+def get_spam_confidence(message):
+       """Returns the DSPAM_Confidence from the message headers. Zero by default"""
+       """ 071218  yky  Created """
+
+       assert(message != None)
+
+       for header in ('X-DSPAM-Confidence', 'SPAM-Confidence'):
+               confidence = message.get(header)
+               if confidence:
+                               confidence_val = float( confidence )
+                               if confidence_val:
+                                       vprint("Spam Confidence: %f " % confidence_val)
+                                       return confidence_val
+
+       return 0.0

def guess_return_path(message):
    """Return a guess at the Return Path address of an rfc822 message"""
@@ -987,6 +1008,11 @@
        return 0
    if options.preserve_unread and is_unread(message):
        return 0
+
+        # 071218  yky  Filtering by SPAM Confidence       
+    if (options.spam_confidence > 0) and (options.spam_confidence > get_spam_confidence(message)):
+               return 0
+   
    return 1
       
   
@@ -1019,7 +1045,7 @@
    max_days -- maximum number of days before message is considered old
      
    """
-    assert(max_days >= 1)
+    assert(max_days >= 0)

    time_now = time.time()
    if time_message > time_now:

Thanks Ow!

yk.

Making Archivemail work with DSpam

Ive got an dspam "appliance" where the enterprise emails filter through. I've set it up so that only one dspam user is used to filter all the emails. This has worked well over the past few years, but managing it has been quite a chore. Every morning, I'd have to wade through the emails in the quarantine (about 15K), and free up any False Positives which were caught.

Beyond the 58% spam confidence as reported by DSpam is pretty much spam. Below that, between the 47% - 57% there may exist one or two False Positives.

After freeing them up, deleting the remaining emails is a huge chore, because the DSpam UI will not allow deleting the quarantine file when new spam pops in.

So I needed a little program which would scan the quarantine mbox file and delete off any messages which are 58% or higher spam confidence.

I tried the most obvious program called 'archivemail', which was readily available in all distros, but was disappointed that it only allowed filtering on the messages age. There was a mysterious "Filter" switch but it only applied to IMAP mailboxes.

The great thing about this is that archivemail, like the entire emailling stack on my servers, is its completely Free Software. I just had to invest some time to look at the code. archivemail lived in /usr/bin/. I had a look at the file, and its a very small 1500 line python script!

I haven't programmed in python before, but looking at the code, it didn't look too scary. It had classes, but no colons. Indentation seemed to be important here. I scanned the code, and I found the little function called "should_archive(message)". And sure enough, the crux of the logic which defines whether a message is to be archived away or not, was there.

So I added this line:

if (options.spam_confidence > 0)
  and (options.spam_confidence > get_spam_confidence(message)):
     return 0

And modified the options class to include the spam_confidence field. Did some modifications on the code to read in the command line options, and then had to create the section which extracts the spam confidence from the message headers. Doing this was relatively easy, because the rest of the code basically does the same things: reading things off the headers and using the information. So my new function looked like this:

def get_spam_confidence(message):
"""Returns the DSPAM_Confidence from the message headers. Zero by default"""
""" 071218  yky  Created """

assert(message != None)

for header in ('X-DSPAM-Confidence', 'SPAM-Confidence'):
 confidence = message.get(header)
 if confidence:
   confidence_val = float( confidence )
   if confidence_val:
    vprint("Spam Confidence: %f " % confidence_val)
    return confidence_val    
   
return 0.0

Thats it!

I also set some cronjobs to run against the quarantine file; to kill 88% and above spams every hour, kill 58% spams after 3 days, and kill the rest if they are more than 14 days old.

I then followed up with my corporate responsibility duties, and submitted the patch back to the archivemail project in sourceforge. This didn't take me long, and it is worth while whether they accept it or not. At least the source is available online.

I hope this helps other dspam admins out there too!

yk.