Compiz works on Heron!

After the initial dissapointment with my clean installation of Ubuntu Hardy Heron, I finally got compiz to work. It was related to the strange core dumps which happens after I run the glx applications. Even running glxinfo causes a segmentation fault. Fortunately it provided a nugget of information:

direct rendering: No (If you want to find out why, try setting LIBGL_DEBUG=verbose)

So set an environment variable as such:

# export LIBGL_DEBUG=verbose

and running glxinfo will result in a more verbose error:

libGL: XF86DRIGetClientDriverName: 5.3.0 r200 (screen 0)
libGL: OpenDriver: trying /usr/lib/dri/r200_dri.so
libGL error: dlopen /usr/lib/dri/r200_dri.so failed (/usr/lib/dri/r200_dri.so: undefined symbol: _glapi_Dispatch)
libGL error: unable to load driver: r200_dri.so

I googled for this, and found this very interesting 'bug report'

Basically the issue is that I erroneously installed the fglrx drivers which I dont need to because my ATI Radeon Mobility 9200 is fully supported by the "ati" Open Source drivers provided by xorg.

So the solution for all this is to remove fglrx with this:

# apt-get remove xorg-driver-fglrx

However there is one more hurdle. Compiz by itself will not run properly "out of the box" on my system. If you run it on the command line

# /usr/bin/compiz --replace &

You will get this error:

Checking for Xgl: not present.
Found laptop using ati driver.
aborting and using fallback: /usr/bin/metacity

And therefore fallback to metacity. No compiz goodness.

To get this to run, according to this post, you will need to modify /usr/bin/compiz by adding these two lines at the top of the file:

LIBGL_ALWAYS_INDIRECT="true"
SKIP_CHECKS="yes"

This will bypass the checks compiz runs which may abort if it thinks your system isnt good enough. Its something like the "Vista Capable" programme, but only in reverse.

Running compiz again on the command line confirms that its really working, and using the System/Preferences/Appearance/Visual Effects also correctly boots up compiz. The performance is really fast, my wobbly windows are sufficiently wobbly, and all the compiz effects are really smooth.

This inspired me to make my own compiz capable logo.

yk.

[Update: Removed the beryl logo]

Hardy Heron - works!

Last Friday, I started my migration from Gutsy Gibbon to Hardy Heron. I took this opportunity to wipe out my entire system, starting from scratch with a new hard disk. Yeah, a 160GB notebook IDE hard disk is now only RM200, so I thought it was a worthwhile upgrade (and effort). I used to have two huge partitions (20-30GB each) formatted as FAT32 for me to share data if I so happened to boot into Windows XP. However for the past year, I never found the need to do that, so Im finally getting rid of FAT32 and by extension, issues with user and datetime information when rsynching.

Popped out the old HDD, replaced with the new, popped in the Install CD and was in the Installation screen booted up relatively quickly. The partitioning part of the installation process was quite tedious, because changing the partitions sizes was quite manual. You had to delete the partitions to make changes to the sizes. No slidey thingies as we are used to with partition management UIs.

After the CD was copied to the hard disk, Hardy booted up really fast. Im not sure its because I was so used to the relatively slow bootup of Gutsy, because it was an upgrade of Feisty Fawn, but I was well impressed with the speed. Things just worked out of the box. Wifi connected without a hitch, sound worked well, special buttons worked, and battery indicators seemed accurate. Suspend gave a problem, but fortunately Hibernate worked! Although it did give some worrying messages in between.

The new Firefox 3 beta 5 looks and feels real good. There are lots of little details which really make it look polished. My greasemonkey scripts appear to run faster than before. Im very happy that greasemonkey and firebug are easily installed via the Add/Remove software app. However some other browser plugins still arent ready, e.g. Google Sync and Permatabs.

I loved the new look to it, fonts seemed a lot more crisp (make sure this is set: System/Preferences/Fonts from BestShapes to SubPixels for your laptop/LCD screens), and windows moved alot faster. I checked out the screensavers, and the 3D ones seemed to be smoother too. However, strangely, compiz was not working. My ATI Mobility 9200 doesn't seem to work well with it, even though I have seen it work with Gutsy. Running glxgears gives a pretty good 1500 FPS. However when you close the app, it core dumps and a new dialog appears. This appears for other 3D screensavers too.

Installing the fglrx drivers doesn't seem to work either. Placing "fglrx" in the xorg.conf causes X to not start. Thats really annoying.

Also horizontal scrolling on the Mighty Mouse is not working. Whenever I change the Driver in the xorg.conf file from "mouse" to "evdev", the mouse will not work at all. Ive tried all the new tricks as described in this relatively new document, but getting horizontal scroll to work seems to be a puzzle.

Option "SendCoreEvents" "true"

It seems strange that things which used to work almost out of the box isnt working now. Whats interesting about Firefox 3 is that it is Mighty Mouse friendly: the Backward/Forward (mousewheel.horizscroll.withnokey.action from "2" to "0") and Horizontal Scroll reversal ( mousewheel.horizscroll.withnokey.numlines from "-1" to "1") settings are already corrected, so that one less fiddle in the about:config step to go through.

Another final touch is getting multimedia to work. So this command downloads the necessary unfree codecs:

# sudo apt-get install ubuntu-restricted-extras

It will then download 37MB of files (and more) for your viewing pleasure.

So overall, Im very happy with the new breath of fresh performance on my slightly old laptop (3 years old), Im dismayed that I can't scroll horizontally, and my compiz eyecandy isnt working. But I'm confident that I'll eventually find a way to rectify the current features as it used to work before.

It's definitely a worthy update for experts now, but for n00bs, please wait a month for workarounds and upgrades to plugins are found.

yk.

winmail.dat

Just got this email from a frustrated colleague:

any idea why this keeps happening? mike's attachments keep appearing as winmail.dat files when i receive them.

The winmail.dat file is a 84KB file of unknown origins. So I detached the file, and looked at it via the command line:

yky@x1407:~/Desktop$ more winmail.dat
x�>"



���� IPM.Microsoft Mail.Note

�^L^L7

&

blegh.

Googling "winmail.dat attachments" brought me to this informative page.

Problem

The Microsoft Outlook e-mail program sometimes sends e-mails in the Transport Neutral Encapsulation Format (TNEF). Most other e-mail programs, including Thunderbird, do not understand TNEF.

If your e-mail program doesn't understand TNEF, instead of seeing the e-mail and/or attachment, you may only see an attachment named "winmail.dat" or "Part 1.2" that you cannot open. Also, sometimes you may receive a TNEF attachment with a generic name such as ATT00008.dat or ATT00005.eml instead.

The solution is to have a TNEF decoder. I typed "tnef" on my ubuntu shell, and it was already installed. 'man'ed it and found that the syntax is:

tnef -vf winmail.dat

'v' is for 'verbose' so that it lists what file names it successfully decodes, and the 'f' flag is to indicate which input file to use (otherwise it uses stdin).

Thats all there is to it. The encoded file will then appear, forward it to your colleagues and their frustrations will subside ... at least till yet another Outlook user decides to harass em.

yk.

A new day.

Its now 3am, and time to sleep. Been following the results of the Malaysian Elections via the super-fast malaysiakini site, probably because the majority of normal browsers would have been denied access to the website, because the DNS entries for malaysiakini.com pointed back to localhost at 127.0.0.1.

A usable address for malaysiakini is http://122.0.17.30/index.php

In the meantime, the results have been surprising. Lets just hope that this 'mandate' as small as it may be, would not be squandered like it was 4 years ago.

This is the first elections I have felt so interested in, and I haven't felt so Malaysian before. Its all good!


Congrats, elected members, but remember that the public WILL chose from now on.

yk.

Malaysian Elections

Today is Election day for Malaysia, and the news is certainly interesting. National TV is broadcasting like nothing has changed since 2004, while alternative news sites like malaysiakini is showing sweeping changes. The biggest of course is about Penang, where the current Opposition has taken control of the state govt. We dont see any updates of this big news in the mainstream news.

Anyway, the disparity is quite interesting, and I just hope its not a repeat of this guy:



yk.

[Update: 11pm. Looks like malaysiakini is down. 'wget http://malaysiakini.com/index.php' gives nothing.]

[Update:11:15pm. Thats strange. I do a ping www.malaysiakini.com, and I get localhost! 127.0.0.1 Anybody else getting the same thing?]

[Update: 11:40pm, Thanks to the power of google/whois, malaysiakini is available here via static IP: http://122.0.17.30/index.php ]

[Update: 11:45pm, Mel has some interesting info on the downtime. "Breaking News - Malaysiakini DNS Suspended!"]

eeePC dialing into maxis via bluetooth.

For real mobility, I needed to configure the Asus eeePC to connect to the internet via my e61. I basically followed the instructions I compiled back in May 2007, "Ubuntu dialup via bluetooth" which was configured for my laptop, with some corrections.

I have updated the post with the most current information. Items being:

1. apt-get doesnt require blues-pin anymore.
2. a bluetooth service needs to be restarted to register the rfcomm0 device

I got myself a AVF bluetooth adaptor, and its really slim - about 3mm thick. It cost RM39.00 which I thought was a great price. It also presents ample opportunity for it to be embedded within the eee PC casing if I were so bold as to solder it directly into the motherboard. heh. One of these days, I guess.


yk.

Asus eeePC, ubuntu, firefox and the synaptic touchpad

Im playing around with the Asus eeePC and have since tried out different linux flavours on it. Ive settled with Ubuntu mainly because of familiarity. And after a reinstall, its fast enough.

A surprising discovery is that the synaptics touchpad features horizontal scrolling too. To enable it,

1) Go to System / Preference / Mouse
2) Click the "Touchpad" tab
3) Enable "Horizontal Scrolling"

Firefox will then become impossible to use, because the horizontal action is defaulted to "back" and "forward". So just like the Mighty Mouse settings, adjust these about:config settings:

4) mousewheel.horizscroll.withnokey.action from "2" to "0"
5) mousewheel.horizscroll.withnokey.numlines from "-1" to "1"

And there you have it, vertical and horizontal scrolling on the little eeePC.

My thoughts on this device in a later post.

yk

Canon ImageRunner and Ubuntu

We have gone away from individual departments printers with a centralised printer/photocopier/scanner device. We chose Canon's ImageRunner 3300 as our solution. Installing the printer drivers for Windows was easy. Installing it on Ubuntu was even easier - no drivers necessary.

1) Select System / Administration / Printing.
2) Click on "New Printer". This would take about 10 seconds before this screen appears:

As it scans the network for available printers. The Printer I wanted was the Canon iR3300.
3) Clicking "Forward" would display the available drivers.

The Manufacturer was automatically selected, but the model was not. There was is an entry for imageRunner 330s which I selected.
4) Clicking "Forward" again was a screen for descriptive information for the printer.

Its not entirely necessary to fill it in...
5) and finally, when the "Apply" button is clicked, the new printer is displayed:

And the printer is immediately available! The test print worked, and a spreadsheet from OpenOffice.org Calc worked. Duplex (double sided) printing also works.

So that was relatively straightforward!

However, now Id have to figure out how to get the network scanning feature to work ...

yk.

How to make nice looking diffs

I was wondering how to make nice looking diff patch files yesterday, as using "diff " gave really cryptic outputs and not very user friendly. I IM'ed Aizat who happened to be online in Chile. He just said use "svn diff". I told him that I was working on files local on my machine, so svn was not appropriate.

Googling didn't help much. So I just submitted the standard diff output as my patch.

Then this morning, Ow had a blog post about his patch, and he included his command line. The answer is "diff -Nau"!

So here is the patch for the archivemail-dspam script:

yky@x1407:~/dspam$ diff -Nau archivemail archivemail-dspam
--- archivemail 2007-12-18 19:13:34.000000000 +0800
+++ archivemail-dspam   2007-12-18 19:02:47.000000000 +0800
@@ -187,6 +187,8 @@
    min_size             = None
    verbose              = 0
    warn_duplicates      = 0
+    """ 071218   yky  DSPAM-Confidence setting """
+    spam_confidence = 0.00

    def parse_args(self, args, usage):
        """Set our runtime options from the command-line arguments.
@@ -206,7 +208,7 @@
                             "filter-append=", "pwfile=", "dont-mangle",
                            "archive-name=",
                             "preserve-unread", "quiet", "size=", "suffix=",
-                             "verbose", "version", "warn-duplicate"])
+                             "verbose", "version", "warn-duplicate", "spam=" ])
        except getopt.error, msg:
            user_error(msg)

@@ -256,6 +258,8 @@
                self.verbose = 1
            if o == '--archive-name':
                self.archive_name = a;
+            if o == '--spam':
+                self.spam_confidence = float(a)
            if o in ('-V', '--version'):
                print __version__ + "\n\n" + __copyright__
                sys.exit(0)
@@ -265,7 +269,7 @@
        """Complain bitterly about our options now rather than later"""
        if self.output_dir:
            check_sane_destdir(self.output_dir)
-        if self.days_old_max <>= 10000:
            user_error("--days argument must be less than 10000")
@@ -661,6 +665,7 @@
      --include-flagged messages flagged important can also be archived
      --no-compress     do not compress archives with gzip
      --warn-duplicate  warn about duplicate Message-IDs in the same mailbox
+      --spam=FLOAT      SPAM Confidence levels ( e.g. 0.80  )
  -v, --verbose         report lots of extra debugging information
  -q, --quiet           quiet mode - print no statistics (suitable for crontab)
  -V, --version         display version information
@@ -737,6 +742,22 @@
    mbox_from = "From %s %s\n" % (address, date_string)
    return mbox_from

+
+def get_spam_confidence(message):
+       """Returns the DSPAM_Confidence from the message headers. Zero by default"""
+       """ 071218  yky  Created """
+
+       assert(message != None)
+
+       for header in ('X-DSPAM-Confidence', 'SPAM-Confidence'):
+               confidence = message.get(header)
+               if confidence:
+                               confidence_val = float( confidence )
+                               if confidence_val:
+                                       vprint("Spam Confidence: %f " % confidence_val)
+                                       return confidence_val
+
+       return 0.0

def guess_return_path(message):
    """Return a guess at the Return Path address of an rfc822 message"""
@@ -987,6 +1008,11 @@
        return 0
    if options.preserve_unread and is_unread(message):
        return 0
+
+        # 071218  yky  Filtering by SPAM Confidence       
+    if (options.spam_confidence > 0) and (options.spam_confidence > get_spam_confidence(message)):
+               return 0
+   
    return 1
       
   
@@ -1019,7 +1045,7 @@
    max_days -- maximum number of days before message is considered old
      
    """
-    assert(max_days >= 1)
+    assert(max_days >= 0)

    time_now = time.time()
    if time_message > time_now:

Thanks Ow!

yk.

Making Archivemail work with DSpam

Ive got an dspam "appliance" where the enterprise emails filter through. I've set it up so that only one dspam user is used to filter all the emails. This has worked well over the past few years, but managing it has been quite a chore. Every morning, I'd have to wade through the emails in the quarantine (about 15K), and free up any False Positives which were caught.

Beyond the 58% spam confidence as reported by DSpam is pretty much spam. Below that, between the 47% - 57% there may exist one or two False Positives.

After freeing them up, deleting the remaining emails is a huge chore, because the DSpam UI will not allow deleting the quarantine file when new spam pops in.

So I needed a little program which would scan the quarantine mbox file and delete off any messages which are 58% or higher spam confidence.

I tried the most obvious program called 'archivemail', which was readily available in all distros, but was disappointed that it only allowed filtering on the messages age. There was a mysterious "Filter" switch but it only applied to IMAP mailboxes.

The great thing about this is that archivemail, like the entire emailling stack on my servers, is its completely Free Software. I just had to invest some time to look at the code. archivemail lived in /usr/bin/. I had a look at the file, and its a very small 1500 line python script!

I haven't programmed in python before, but looking at the code, it didn't look too scary. It had classes, but no colons. Indentation seemed to be important here. I scanned the code, and I found the little function called "should_archive(message)". And sure enough, the crux of the logic which defines whether a message is to be archived away or not, was there.

So I added this line:

if (options.spam_confidence > 0)
  and (options.spam_confidence > get_spam_confidence(message)):
     return 0

And modified the options class to include the spam_confidence field. Did some modifications on the code to read in the command line options, and then had to create the section which extracts the spam confidence from the message headers. Doing this was relatively easy, because the rest of the code basically does the same things: reading things off the headers and using the information. So my new function looked like this:

def get_spam_confidence(message):
"""Returns the DSPAM_Confidence from the message headers. Zero by default"""
""" 071218  yky  Created """

assert(message != None)

for header in ('X-DSPAM-Confidence', 'SPAM-Confidence'):
 confidence = message.get(header)
 if confidence:
   confidence_val = float( confidence )
   if confidence_val:
    vprint("Spam Confidence: %f " % confidence_val)
    return confidence_val    
   
return 0.0

Thats it!

I also set some cronjobs to run against the quarantine file; to kill 88% and above spams every hour, kill 58% spams after 3 days, and kill the rest if they are more than 14 days old.

I then followed up with my corporate responsibility duties, and submitted the patch back to the archivemail project in sourceforge. This didn't take me long, and it is worth while whether they accept it or not. At least the source is available online.

I hope this helps other dspam admins out there too!

yk.

Gutsy Gibbonized

Started the command line upgrade process on Friday, with all the packages downloaded from the pretty fast Taiwan mirror site (averaging at about 50kBs). I left it to download for about 5 hours. When I was done at work, it started unpacking the packages. Because my home is not far from the office, I just picked up the laptop and let it continue its installation all the way home.
About an hour later, the laptop was ready to be rebooted.
Restarted the machine, and waited in anticipation. The boot screen flicked to the text console (as normal, because fsck always has something to say about my two FAT32 partitions), and after that, blackness...

Uh oh... problems with X. The keyboard was non responsive (Caps Lock didnt toggle the LEDs), so the only alternative is to reboot into the recovery state.

Good chance to test out the bulletproof X, I thought, so I renamed /etc/X11/xorg.conf to something else, and rebooted. This time it worked, except that after logging in, the brownness of ubuntu changed to the harshness of raw X (cursor and all). Then the computer hung there for quite a while. I broke into another console (Ctrl-Alt-F2) and tried to dig around, but after about a minute, the familiar sounds of the ubuntu drums indicated that things were ok again. (Ctrl-Alt-F7) brought back the familiar gnome desktop.

But things were very different. Windows were very slow. Scrolling through webpages was actually painful! It was like running Windows XP without the appropriate video card drivers. I downloaded the new displayconfig-gtk tool to set the "Screen and Graphics Preferences", which created a new xorg.conf file. That didnt help much either. I tried using some settings from my fiesty xorg.conf, but certain Options would kill X (blank black screen).

Did a google, and found this: "ATI Radeon 9200 extremely slow in Gutsy" which described my situation to the dot. One recommendation is to disable Xgl, which is not a default install. To do so, do this:

touch ~/.config/xserver-xgl/disable

Make sure the "xserver-xgl" directory is created as well, because usually it isnt.

Rebooting made the performance of 2D X similar to the fiesty days.

Things are still not entirely back to normal: my Mighty Mouse is not scrolling sideways, and compiz isnt working. Need to allocate some time to dig into xorg.conf yet again. Also the timelag between logins and desktop appearing is very annoying.

yk.

[Update: I just found out that my xorg.conf was removed because in an hour of desperation, I tried to install the ATI proprietary drivers, fglrx. One of the steps involved was running aticonfig --initial, which clobbers the xorg.files. That core dumped and I gave up on that. So I have been running without an xorg.conf file all this while. So bulletproof X really works, and part of the delay in showing the gnome desktop was because of this.

My xorg.conf from fiesty still gives a dead black screen on bootup, so I have to work on a cut down version. I will have to slowly work on this to get my dual screen back up. Wonder how easy that would be ... ?]

Driving License extended

Ow has an interesting factoid. When we met at aizat's birthday a few months back, he said that someone in a room of 10 is bound to have a driving license expired, or close to expiring. And true enough mine was going to expire (then) on the 2nd of October. Of course being the serial procrastinator that I am, I subsequently forgot about renewing it.

On the 4th of October, two days after the expiry, I checked the JPJ website and noticed that there is a 'CDL Renewal Online' feature hosted by a third party, MyEG.com.my. Throwing caution to the wind, I decided to test it out. I filled in my details, paying by Credit Card, and submitted the information.

I opted for a 5 year extension (RM150) and for the new license to be delivered to my office (RM5). MyEG seems to be charging RM2 for the service, which sounds fair. But considering the income generating nature of JPJ, citizens should expect service as this to be completely free. Additionally any form of automation would reduce overheads for JPJ.

At the end of it all, you get a official looking receipt with all the JPJ logos and all:

I double checked this by going to the main site, and clicking on 'License Expiry Date Inquiry' tab. The interface changed from English to Malay (because the URL seems to be hardcoded with the language settings). I punched in my IC number, and this is what I got:


Confirmation of a 5 year extension! I was still skeptical of the renewal because it seemed all too effortless. Additionally, my recently expired license is already 10 years old, and the picture of me was taken another 10 years prior.

So I waited for a few days... 7 days in all. I received a Registered Post letter from JPJ, which enclosed a laminated card stating that my old license indeed has been extended for another five years. This extra card (which they call a 'Slip') is 'To Be Produced With Driving License'. It makes my wallet marginally thicker, but I guess its a fair trade for an afternoon of bureaucracy.

Overall, I was impressed with this. Seems like a small thing to most IT enabled countries where doing things online is of the norm. But when Ive been disappointed with the online services in Malaysia so many times todate, I guess my standards are far lower.

Here are some points for JPJ to improve on:

1. Do not hardcode the IP address of the servers on the website. Its currently now pointing to '202.190.64.96'. It makes linking directly to your services difficult especially in the future when you change your servers.
2. Someone messed up with the sidebar links. Remove the '/lang,ms/' option in the URLs. This forces the language to flip from English to Malay on subsequent clicks.
   
3. Its unnerving to be moved to another site, and another window when clicking on the CDL extension. Make it clear that MyEG is a trusted partner or embed the payment page within the main JPJ site. It would be alot better if this feature was seamless to the user.
4. Don't making using the online service any less attractive to drivers. There should not be any additional costs (RM2). In fact online applications should be encouraged, and there should even be a small rebate.

Because only 5 people will read this blog post, I cannot guarantee that one of you will have an expiring or expired license. But if you do, you'd be glad to know you can now renew it online.

yk.