Playing around with Ordered Dictionary (collections.OrderedDict)

The new and cool feature added to Python 3.1.x family is the Ordered Dictionary. Some implementations have been lying around for sometime, it's good to see one made it into  the Python standard library.

Dictionaries are general purpose container data structure that are accessed through keys. It uses a hashing mechanism to achieve fast retrieval of information by key on typical cases. The annoying thing about standard dictionary dict object is that, whenever we retrieve the keys, it is not guaranteed to be sorted.
In Java platform, usually this dictionary is called map. java.util.Map provides interface to implementations of map such as java.util.HashMap, java.util.TreeMap.

Every value is associated with value, written in : pair.

Try to paste this code to your Python 3.1 interpreter:

[sourcecode lang="python"]<br />ud = dict()<br />ud['c']='1'<br />ud['b']='2'<br />ud['a']='3'<br />[/sourcecode]

This interactive session below highlight the thing:

[sourcecode lang="shell"]<br />>>> ud = dict()<br />>>> ud['c']='1'<br />>>> ud['b']='2'<br />>>> ud['a']='3'<br />>>> ud<br />{'a': '3', 'c': '1', 'b': '2'}<br />[/sourcecode]

Nice general purpose container for all of our applications. The dictionary keys doesn't display in sorted order.

Instead when we paste this code into Python interpreter:

[sourcecode lang="python"]<br />import collections as coll<br />od = coll.OrderedDict()<br />od['c']='1'<br />od['b']='2'<br />od['a']='3'<br />[/sourcecode]

We will get this result:

[sourcecode lang="shell"]<br />>>> import collections as coll<br />>>> od = coll.OrderedDict()<br />>>> od['c']='1'<br />>>> od['b']='2'<br />>>> od['a']='3'<br />>>> od<br />OrderedDict([('c', '1'), ('b', '2'), ('a', '3')])<br />[/sourcecode]

As we can see in the result above, the collections.OrderedDict objects displayed the keys according to the sequence that they were inserted. When an insertion done on an already existing key, the new insertion will overwrite the value but the sequence is left untouched. A deletion followed by addition will move the key to the end of the list, as show in the session below (continuation of the previous one).

[sourcecode lang="shell"]<br />>>> od['d']='4'<br />>>> od<br />OrderedDict([('c', '1'), ('b', '2'), ('a', '3'), ('d', '4')])<br />>>> od['b']='5'<br />>>> od<br />OrderedDict([('c', '1'), ('b', '5'), ('a', '3'), ('d', '4')])<br />>>> od.pop('a')<br />'3'<br />>>> od<br />OrderedDict([('c', '1'), ('b', '5'), ('d', '4')])<br />>>> od['a']='6'<br />>>> od<br />OrderedDict([('c', '1'), ('b', '5'), ('d', '4'), ('a', '6')])<br />>>><br />[/sourcecode]

I guess "battery included" still holds, or probably, "more organised battery included...".
Hooray for Python.

Python 3.1.2 Released (bug fix released)

Python 3.1.2 (bugfix release) is out now! We can download from this site: http://www.python.org/download/

Spring Python 1.1.0.M1 Released

SpringSource has has just announced last  week of a milestone release of SpringPython 1.1.0.M1. This announcement made aware about this project. There is a project that has nearly reached its 1.1 version that tries to develop Spring-like framework in Python. I would like to look into this project.

It is different from invoking Spring Java from Jython in  that, this project brings the concept of Spring framework into the CPython platform in general. Some of the configuration are really Pythonic!

I think I will explore more on this project.

Setting Up Your Own Jython 2.5.1 Environment

In this post I'll show you how to setup your own Jython environment.

Prerequisite to this, you must have a Java Runtime Environment (JRE) installed in your system. Chances are when you already have browser running some Java applet or Java Web Start application, you already have JRE installed on your system.
Get the latest JRE from http://java.com/en/download/manual.jsp.

First you need to download a Jython binaries installer, currently the latest one is version 2.5.1, which you could download from this link.

After download completion you should have a file named jython_installer-2.5.1.jar, click on the file, or run from command line:

[sourcecode lang="shell"]<br />$ java -jar jython_installer-2.5.1.jar<br />[/sourcecode]

Or in some other platform:

[sourcecode lang="shell"]<br />c:\&gt; java -jar jython_installer-2.5.1.jar<br />[/sourcecode]

Now you need to add you Jython path to your OS Path variable, so that the shell or command line interpreter is able to find Jython. In my sample Windows installation case:

[sourcecode lang="shell"]<br />C:\&gt; set PATH=%PATH%;c:\opt\env\jython2.5.1<br />[/sourcecode]

Alternatively, if you are running bash shell (assuming you installed Jython to /opt/env/jython2.5.1 folder):

[sourcecode lang="shell"]<br />$ export PATH=$PATH:/opt/env/jython2.5.1<br />[/sourcecode]

Now you may proceed with the first invocation of Jython.

[sourcecode lang="shell"]<br />C:\&gt; jython<br />*sys-package-mgr*: processing new jar, 'C:\opt\env\jython2.5.1\jython.jar'<br />*sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre6\lib\resources.jar'<br />*sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre6\lib\rt.jar'<br />*sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre6\lib\jsse.jar'<br />*sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre6\lib\jce.jar'<br />*sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre6\lib\charsets.jar'<br />*sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre6\lib\ext\dnsns.jar'<br />*sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre6\lib\ext\localedata.jar'<br />*sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre6\lib\ext\QTJava.zip'<br />*sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre6\lib\ext\sunjce_provider.jar'<br />*sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre6\lib\ext\sunmscapi.jar'<br />*sys-package-mgr*: processing new jar, 'C:\Program Files\Java\jre6\lib\ext\sunpkcs11.jar'<br />Jython 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54)<br />[Java HotSpot(TM) Client VM (Sun Microsystems Inc.)] on java1.6.0_16<br />Type "help", "copyright", "credits" or "license" for more information.<br />&gt;&gt;&gt;<br />[/sourcecode]

The jython command line actually invoked a Windows batch file jython.bat, which
When first time invoked, Jython interpreter will activate sys-package-mgr to add standard Java library classes as Python package.

 

After the installation, just play around with simple thing such as:

[sourcecode lang="shell"]<br />Jython 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54)<br />[Java HotSpot(TM) Client VM (Sun Microsystems Inc.)] on java1.6.0_16<br />Type "help", "copyright", "credits" or "license" for more information.<br />&gt;&gt;&gt; from java.lang import System<br />&gt;&gt;&gt; System.out.println(&amp;quot;Hello, world!&amp;quot;)<br />Hello, world!<br />&gt;&gt;&gt; System.out.println('Hello, world!')<br />Hello, world!<br />&gt;&gt;&gt; System.currentTimeMillis()<br />1257145977410L<br />&gt;&gt;&gt; from java.lang import String<br />&gt;&gt;&gt; s = String(&amp;quot;abc&amp;quot;)<br />&gt;&gt;&gt; s.toString()<br />u'abc'<br />[/sourcecode]

Look at how System.out.println is able to receive both single quote and double quote Python string, and also the java.lang.String type by a Java class method is a Unicode string (u'abc').

PyDev 1.5 on Eclipse Galileo 3.5 SR1

I begin to have more time to explore the Eclipse Galileo Service Release 1 (Eclipse 3.5.1). This time I want to set it up for my Python development, using the PyDev plugin.

PyDev is an Eclipse plugin for Python language application development. It supports CPython, Jython and IronPython runtimes. It supports the interactive as well as non-interactive Python development on Eclipse IDE.

Eclipse IDE (and platform) has become the de facto industry standard IDE. More and more free software, open source software, and proprietary software makers build their systems on Eclipse platform. Galileo is the codename for 3.5 release, and now the latest one is the SR1 (Service Release 1).

There has been interesting development on PyDev since its last 1.4.7 version. PyDev has moved to be sheltered under the umbrella of Aptana. PyDev and PyDev Extension now has been merged and both now open source (previously the PyDev Extension, the advanced version of PyDev was not an open source).

The update site has been moved from domain www.fabioz.com (http://www.fabioz.com/pydev) to domain pydev.org (http://pydev.org/updates) . Aptana  This development looks good for me, means that PyDev now being maintained by its own organization, and hopefully in more serious efforts.

In Eclipse Galileo, you need to go to the menu and choose: Help > Install New Software...

It will show the Install dialog.

Click the [Add] button on the upper of menu dialog. You will get the "Add Site" dialog.

Type name and location.
Name: "PyDev Update  Site" (you can feel with any arbitrary name though)
Location: http://pydev.org/updates

Back to the "Install" dialog, the previous "Add Site" will fill up the "Work with" field for you.

Now tick the Checkbox "PyDev for Eclipse" under "PyDev". You may also optionally tick the "Pydev Mylyn Integration". In this example we don't tick.

Click the [Next >] button. It will show you the Install Details:

Proceed by clicking [Next >] button. Review Licenses dialog will be displayed.

Click the radio button "I accept the terms of the license  agreement" when you are agree with the licenses. Proceed by clicking [Finish] button.

You will see the progress bar. Somewhere around the progress, you will be asked whether to install software that contains unsigned content.

Click [OK] to proceed (we have no choice as long as the provider of this plugin hasn't signed the content).

Click the [Yes] button to restart your Eclipse IDE.

Chardet Python Library: Determining Character Encoding of Text File or Text Stream

Recently I have came into a disagreement with fellow programmer. He tried to create view simple codes in different programming languageshe knows: C, C++, Perl, Python, PHP, and Java (I think it's for self-actualization, obviously not for money or wealth). It turns out to be he created XML parser codes on those (programming) languages.

The code doesn't produce as expected, except of course in Python. He ascertained that it is because most those languages make use of libxml2 library (from GNU). He came into a conclusion that the Java library might also make use of the libxml2 library from GNU, which I believe most unlikely. Java wouldn't normally pick GNU stuffs for its library. It really hit into my Java developer pride, and I need to prove that it is not the library that is incorrect, instead the input file is not correctly encoded.

Although in Western world most text files usually play along well most of the time, characters other than ordinary characters will be displayed incorrectly. Most of the case it will be displayed incorrectly in countries that uses special characters!

Since the first time I work in Singapore, I absorb it quickly that the ability to display these special characters are one important thing a web application should have, especially when they cater for the Asian market! You will see that they will require you to do i18n, display this in Traditional Chinese characters, in Simplified Chinese characters, Tamil characters, Thai characters, to name a few...

It brought me to this question:
If I have an arbitrary text file, how could I know which encoding the text uses?

I googled around and found  this: Universal Encoding Detector (chardet) library. The chardet site shows how to do it for web sites.

[sourcecode lang="python"]<br />&gt;&gt;&gt; import urllib<br />&gt;&gt;&gt; urlread = lambda url: urllib.urlopen(url).read()<br />&gt;&gt;&gt; import chardet<br />&gt;&gt;&gt; chardet.detect(urlread("http://google.cn/"))<br />{'encoding': 'GB2312', 'confidence': 0.99}<br /><br />&gt;&gt;&gt; chardet.detect(urlread("http://yahoo.co.jp/"))<br />{'encoding': 'EUC-JP', 'confidence': 0.99}<br />[/sourcecode]

Now it here comes the next question for me: if it is not a web site, but a file instead, how to detect the encoding?

So here is the solution:

[sourcecode lang="python"]<br />&gt;&gt;&gt; import chardet<br />&gt;&gt;&gt; fileread = lambda filename: open(filename, "r").read()<br />&gt;&gt;&gt; chardet.detect(fileread("italian-english.xml"))<br />{'confidence': 0.9899999999999999, 'encoding': 'utf-8'}<br /><br />&gt;&gt;&gt; chardet.detect(fileread("utf8_demo.txt"))<br />{'confidence': 0.9899999999999999, 'encoding': 'utf-8'}<br /><br />[/sourcecode]

So this way, I checked that the text file is most likely an UTF-8 encoded text.

By the way, the encoding detection engine library was derived from (or ported from) Mozilla auto-detection code, which has been widely used. So more or less I believe it is mature and powerful enough for most usages.

Another thing to note is that those codes above are using lambda notation, a feature of Python language borrowed from pure functional programming syntax.

Python 3.1 File I/O open() Is No Longer Binary Mode By Default

After long holiday on my wedding preparation, D day and honeymoon, I returned back in shape.

Some things happened during the off days, such as the acquisition of SpringSource by VMWare, Inc.

I try to run my piece of XML parsing code like this in Python 3.1:

[sourcecode lang="python"]
import xml.parsers.expat
import sys
import os.path

filename = "companies.xml"
parser = xml.parsers.expat.ParserCreate()
f = open(filename, "r")
parser.ParseFile(f)
f.close()
[/sourcecode]

The code above throws an error like this:

[sourcecode lang="shell"]
TypeError: read() did not return a bytes object (type=str)
[/sourcecode]

This source code works in Python 2.6, but failed when running on Python 3.1.

After comparing the manual for the built-in open() function between those 2 versions (2.6 and 3.x versions), I found out that there is a new feature in Python 3.x which is not backward compatibility (unlike transitions between 2.x versions, the transition from 2.x to 3.x may break the backward compatibility).

Python 3.x added the "t" for text, "b" for binary, "+" for updating (read/write) and "U" modifier to the open file mode.

It turns out to be that Python 3.1 no more handles file open as binary by default.Now the text ("t") mode become default for Python 3.1 meanwhile Python 2.6 assume all file access are binary accesses. Python 3.x has implemented modes of more similarity to its C stdio library counterpart than its previous 2.6 version.

Since Python 3.x added additional parameter to the open() function, we must now specify "b" to make it binary access, so that the read() method will return bytes, otherwise it will return str.

[sourcecode lang="python"]
import xml.parsers.expat
import sys
import os.path

xmlFilename = "companies.xml"
p = xml.parsers.expat.ParserCreate()
f = open(xmlFilename, "rb")
p.ParseFile(f)
f.close()
[/sourcecode]

Now the ParseFile() method get what it asks for, a file handler with read() method that return bytes instead of str.