Thursday, May 29, 2008

Airport Security

Saw this sign in Heathrow Airport:

What is this suppose to mean?
  1. The rest of the airport is NOT secure
  2. Because of me, they now have to secure it
  3. Im a terrorist, and I now need to be aware that I shouldnt be
  4. Its a reminder to airport staff to be more vigilant
Argh. Whatever that sign is suppose to mean, it does NOT make me, an innocent passenger feel any better.



Copying Pictures from the iPhone

Im thinking of upgrading the firmware of the iPhone to 1.1.4. Before I do that however, I needed to move out the pictures Ive taken. For some strange reason, my Windows machine, which has iTunes on, refused to recognise the iPhone as a camera. Or rather it did, but when I accessed it as a 'drive', it didn't copy over the photos. It would display the listing of the files, but when I do a drag-n-drop, the copied files would be 0 bytes. Very strange.

Also iTunes doesnt make it obvious on how to sync the pictures which I've taken. Doubly strange.

Giving up on that, its back to Linux to help me out on this rather simple task
  1. Make sure your iPhone is connected to wifi.
  2. Find out the address by going to Settings / Wi-Fi, and tapping on the wifi details. It should display the IP address there, e.g.
  3. Also in Settings, go to the General / Auto-Lock menu and set the time-out to "Never". Otherwise the wireless will cut off. Don't know why the options are restricted to only 5 minutes.
  4. On the linux shell, use this command:
    • scp -rp root@ .
  5. This should copy all the files in the directory where the photos are stored.
    • -r is for recursive and
    • -p is to preserve the time and date
  6. The default password for ssh in an iPhone is 'alpine' for the 1.1.3 firmware.
Thats all to it. You can also do some interesting stuff with rsync, but I just wanted my pictures out before the machine gets wiped.


Wednesday, May 21, 2008

Ballmer strikes back.

So Ballmer went to Hungary ...
... an this student started shouting ...
... and threw eggs at the most powerful CEO in America ...
... and made Ballmer look like he was cowering in fear ...
... and while the student was taken away ...
.... Ballmer strikes back!!!
.... "heh, you don't mess with the best baby!
Ive got the best throwing arm in the world!

See it on YouTube.


DSpam: WebGUI modifications - Javascript goodies.

Ive been using dspam for two years now. It has been set up as a broad filter which checks all the emails passing through our systems. It has worked very well over the past year although going through the thousands of emails has been a chore.

The WebGUI for dspam is rudimentary, and is really built for single users to review their own spam/hams. It really isnt designed for large volumes. So I had to make some modifications.

Modification #1: Looking for missed SPAM.

The problem with the WebUI when you view the History tab is that it displays ALL the emails which pass through the system. This would be a normal requirement, but if you really are just interested in reviewing the False Negatives, i.e. SPAM that got away, it really takes ages to scroll through the pages one after another. If your domain has attracted alot of spam, over 70% of the entries are spam anyway, and you arent really interested in that information.
So the solution is to ignore the SPAM entries from the /var/dspam/.../dspam.log file. To do this, you will need to modify the /var/www/html/dspam.cgi file.

This is the patch (dspam.skipSPAM.patch):

With this patch, you can toggle the ability to skip through Spams by adding in another url argument &skipSPAM=true on the URL address. Otherwise, if you want this on by default, just make $skipSPAM = "true" in the perl script.

The result should look something like this:
Notice how the Spams are ignored, giving you a clear view of what to retrain or allow. I also skip through Whitelisted emails so that means less lines to review. I use Firefox's tab browsing, and just middle click the entries which I want to flag off as Spam. The process is very fast, and I probably need to click through about 3 pages of history to mark off any significant growth in spam.

Modification #2: Marking Dead meat the brute force way
One of the hassles of clearing False Positives is the process of going through each and every spam item and checking it off. Early on, I modified the nav_quarantine.html template file with this small Javascript which checked off the first 200 items. Here is the patch for the "Select 200" modification for the templates/nav_quarantine.html file (nav_quarantine.select200.patch)

This worked well to a certain extent, but when you have over 10K entries, refreshing the page 500 times is certainly not an option. There MUST be a better way. And there is ...

Modification #3: Marking Dead meat the elegant way.
Blindly selecting the first 200 entries isnt really an efficient way of culling the confirmed spam. I needed an almost automated way to handle this. So I embedded more information from dspam into the WebUI and written some Javascript to make this process alot more bearable.

The first requirement is to remove all the marked spam of a given percentage of certainty. Throughout the entire production usage of DSpam, I have yet to see a False Positive with a certainty score of more than 70%. What would be great is to check off all entries given a score automatically. This is now possible by entering a confidence number, and simply clicking on the "Mark Rating" button.
What the script does is that it uses XPath to query out all rows which have a rating of more than what is entered. The Javascript code looks something like this:

var pRate = parseFloat( document.getElementById("rating_val").value ) / 100;
var xpath = document.evaluate( "//tr[\@rating > "+pRate+" ]", document, null, +XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null );
I have also added an extra feature in that it will also mark of similar items which may have less of a confidence than provided. This will be elaborated below, in the "hash" I generate with all the entries later. Because of this 'recursive' behaviour, the script will take a while to complete, so you may need to increase the timeout for Firefox (otherwise it may complain with a "A script on this page may be busy, or it may have stopped responding. You can stop the script right now, or you can continue to see if it completes.") To do so, type about:config in the URL bar, and adjust the dom.max_script_runtime from the default of 10 to something larger like 500.

Modification #4: Ajax waits for no refresh.
Another tedious part about using the WebUI is that whenever you need to purge the quarantine of the caught spam, it takes ages because it causes an entire page refresh. Its OK if the list is less than a thousand, but when it reaches 20K or more, its just too much.

Then we come to another problem when deleting the entries from the /var/dspam/.../dspam.mbox file. As you remove entries from the file, if at any point during that time, a new email arrives, the deletion will cancel and the file will roll back to its original state plus new email. So realistically on a busy system, you can't delete more than 50 spams at a go. This means we will have to endure ALOT of page refreshes.

What I implemented then was a AJAX type handler for dspam.cgi to execute. I added the Javascript features in the WebUI, and it looks like this:
As you click the button, the Javascript will scroll through the checked list, and when it compiles 25 entries, it forms a query back to dspam.cgi to execute in the background. It will alert the user by stating it is currently "deleting 25". When the call is successful, it will state "deleted 25". It will then repeat the process if there are still items to be checked.

The figure 25 is something which I found to be small enough to cater for non roll backs, and because the process is automated, it doesnt need to be large. So to clear off 15K entries, it takes about 30 minutes to an hour.

Modification #5: Hashing up spam
For the remaining spam which isnt obvious, I have included two little clickies on the end of the table. "del" deletes the entry immediately, while "hash" checks the entry's checkbox, and all entries with similar subjects. This means you can click off multiple spams with just one click as demonstrated below:

This makes marking off spam almost ... fun!

Patching the WebGUI
I include with this post three patches. In the dspam.cgi directory, run this:
# patch < dspam.skipSPAM.patch
# patch < dspam.ajax.patch

and in the template directory
# patch < nav_quarantine.select200.patch

Otherwise dspam.cgi and template/nav_quarantine.html are also available.

How I use these modifications
Whenever I have the time to review the spam collection
  1. I load up the quarantine page until its fully loaded.
  2. I then click on "Mark Spam" with the default rate of 85%.
  3. This takes a few seconds depending on your PC.
  4. I then click on the "Ajax delete" button to start the deletion process in the background.
  5. In the meantime, I reduce the rate down to 70% and sometimes 60% to clear off further spam.
  6. I also start from the top, i.e. 47% confidence spam items, and slowly review the items up to about 53%, clicking on the "hash" to remove the spam items.
  7. After I clean off the False Positives (if any) I click on the "Select 200" and eyeball the remaining items until there are no entries left.
  8. It still takes some time, but at least its a whole less time than before!

I hope this helps!


Thursday, May 15, 2008

Converting video: Realmedia to DivX

I have a video which is encoded in rmvb. Realmedia (who they now?). I want to convert it to a divx file so that I can burn it on a DVD and watch it on my DivX-DVD player at home. This is what I tried to view it with ffmpeg:
# ffmpeg -i videofile.rmvb -vcodec mpeg4 out.mpg

FFmpeg version SVN-rUNKNOWN, Copyright (c) 2000-2007 Fabrice Bellard, et al.
configuration: --enable-gpl --enable-pp --enable-swscaler --enable-pthreads --enable-libvorbis --enable-libtheora --enable-libogg --enable-libgsm --enable-dc1394 --disable-debug --enable-shared --prefix=/usr
libavutil version: 1d.49.3.0
libavcodec version: 1d.51.38.0
libavformat version: 1d.51.10.0
built on Mar 12 2008 14:31:53, gcc: 4.2.3 (Ubuntu 4.2.3-2ubuntu4)
[NULL @ 0xb7e086e8]Unsupported video codec
[NULL @ 0xb7e086e8]Unsupported video codec
Unsupported codec (id=72) for input stream #0.1
So that didn't work. FFmpeg doesnt seem to understand Realmedia. A bit of googling brought me to this command:
# mencoder videofile.rmvb -o outfile.avi -ovc xvid -xvidencopts fixed_quant=5 -oac mp3lame -lameopts abr:br=128

Opening video decoder: [realvid] RealVideo decoder
Error: /usr/lib/win32/ cannot open shared object file: No such file or directory
Win32 LoadLibrary failed to load:, /usr/lib/win32/, /usr/local/lib/win32/
Error loading dll
So that doesn't work either; looks like Im missing some DLLs. Abit more googling, and I am recommended to visit this page "MPlayer Codecs", which is a compilation of all the codec dll's for quite a few video formats. So I downloaded essential-20071007.tar.bz2 and which was extracted into the /usr/bin/win32 directory as root.

After which, the mencoder command worked without a flaw in converting between Realmedia to DivX.

What joy, this multitude of proprietary file formats.


Wednesday, May 14, 2008

Windows Terminal Server - Maximum connections blegh.

Don't you just love proprietary software? It restricts you in the most irritating manner.

I unfortunately had to do something on a Windows 2003 server, and I normally go about by using the wonderful Remote desktop protocol. Linux has an excellent client implementation of it, and I use it more often than VNC for Windows servers.

This time however, I was alerted with this error:
"The terminal server has exceeded the maximum number of allowed connections."
Translation: "Nye-nye! Buy CALs!"

Not wanting to be a neanderthal by rebooting the machine, I decided to see if I could circumvent this "problem" with a bit of Googleducation. Well, it seems its possible with Windows 2003. You can actually make use of the "console" session if your login has the role of an administrator. To access the console, or "session zero", in Linux its like this:
# rdesktop -0
... where you replace your server IP address with the example provided above. The "-0" flag indicates that you want the console session. In MS-DOS's, supposedly, because I havent tried it, its:
C:\> mstsc -v: /f -console
Once you are in, and get some scary errors about kicking out the currently logged in person, you can view the RDP session which have been hanging around, taking up the precious license seats.

The commandline command is:
C:\> query session
Subsequently, you can delete the Disconnected / Orphaned / Stray / Dismembered sessions using this command, which is very similar to vncserver -kill :[screen #]
C:\> reset session [session ID]
You can disconnect ALL sessions by selecting the highest ID (usually >65536) e.g.
C:\> reset session 65538
Obviously I tried it once and got kicked out of by remote desktop. Heh. Cheap thrills.

So there you have it: how to retain access, and a simple way of 'resetting' your Windows 2003 server's remote desktops without having to reboot the darn thing, because of the artificial restrictions placed on software based on user access licensing.


Monday, May 05, 2008

VirtualBox: WinXP on Ubuntu. Fast. Finally.

Now that my Ubuntu laptop is beefed up, I had the opportunity to recreate my old old proprietary development desktop. Also, Ive been using VMWare ESX Server alot at work lately, so just to be different, I decided to use something new. Something free preferably.

VirtualBox, from Innotek Gmbh, yet another German company snapped up by Sun, has a great solution. In Ubuntu, I naively did an "Application/Add Software..." and selected "Virtual Box OSE" to be installed. It downloaded, did its thing, and in the "Application/System Tools/Virtual Box OSE" menu item, launched.

I created a 7GB Virtual Harddisk, and clicked on "Start" to boot up the Virtual Machine.

Immediately I got this error:

Yup, "vboxdrv kernel module was either not loaded or /dev/vboxdrv was not created for some reason." It then tells me to install "virtualbox-ose-modules". Bokay:
# apt-get install virtualbox-ose-modules
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package virtualbox-ose-modules is a virtual package provided by:
  • virtualbox-ose-modules-2.6.24-16-virtual 24
  • virtualbox-ose-modules-2.6.24-16-server 24
  • virtualbox-ose-modules-2.6.24-16-rt 24
  • virtualbox-ose-modules-2.6.24-16-openvz 24
  • virtualbox-ose-modules-2.6.24-16-generic 24
  • virtualbox-ose-modules-2.6.24-16-386 24
You should explicitly select one to install.
E: Package virtualbox-ose-modules has no installation candidate

GGgg! So many choices! I guess its right that "Its not about choosing, but about having a choice"! So I just guessed:
# apt-get install virtualbox-ose-modules-2.6.24-16-generic
And it proceeds to download 326KB worth of extra precompiled modules. Now you'd have thought that they would include it anyway, so it would make the steps alot easier, but I guess sysadmins need to make their living.

I also had to add myself in the vboxusers group:
# gpasswd -a yky vboxusers
Finally, the VM booted up, and I successfully installed the glorious Windows XP SP2 into it. Brings back so many memories. Its been quite some time since Ive relied on a proprietary desktop to do my daily work.

The performance of VirtualBox is quite good. I tried Qemu a long time ago, and that was sluggish. This was fast and responsive. Installation from CD was not a problem. Everything worked well. There is also a feature to "Install Guest Additions", and the VirtualBox manager will then download a 5MB ISO which can be mounted as a CD.

Run "Setup.exe" from the drive, and you should have a more "seamless" experience, in that your mouse will not be caught (and have to be released with the Right Cntrl key), and also the desktop screen resolution can be changed on-the-fly, just by resizing the VirtualBox window! Neat feature.

I resized the desktop to a little. WinXP now looks like a Windows CE emulator.

I installed my Windows dev tools, and response is better than I could ask for from a real Free (as in Freedom) Virtual Machine. Whats great is that it supports my non-VT Centrino processor. Load on the CPU is low, and a 512MB VM chugs along quite well on my 1.7GHz 1.2GB host.

So I wholly recommend this application for el proprietary desktop application needs. Its quite good! Installation could be easier, but its a hell of a lot easier years ago. Well done Sun! Must have been Colin's fault yet again.