Jeroens blog

Maps and Semantic Maps 0.6.5 are now available for download. This release contains mainly internal changes to improve code modularity and fix some security concerns. Several bugs have been fixed as well, and a new hook has been added to Semantic Maps. This hook will get you the map format as default one for queries where you only ask for coordinates when using SMW 1.5.2 or above. For a full list of changes since 0.6.4 see changes to Maps and changes to SM. Everyone running 0.6.2 or older is advised to upgrade as soon as possible. 

This release is notable for it being the first one in which I’m happy with the code-base as a whole. It took me a year to get here, but now I think the way the mapping extensions work is good and solid. This means you can now extend Maps and not be afraid the code will be incompatible in a few weeks due to changes. This also means that I’ll be focusing more on actual functionality rather then refactoring in future releases. I’ll be progressively building a little guide that explains how the extensions work from a developers perspective and how to extend them.

I might release another minor update in the 0.6.x series if any significant issues are found in 0.6.5. Further plans are finishing up a bunch of changes I’ve started to make in Validator, which I’ll probably release as 0.4 then, and to start working on Maps and Semantic Maps 0.7, which would aim at adding new features and improving existing ones. A likely new feature I’m particularly looking forward to implementing is several tag extensions that do the equivalent of the current parser functions added by Maps. The timetable for all this depends a lot on which other things I get cough up in (I’ll probably continue putting effort into the deployment stuff for my GSoC project) and what kind of funding will be available.

Downloads:

• Maps 0.6.5 [zip - 7z]
• Maps and Semantic Maps 0.6.5 [zip - 7z]

You can also view the release announcement at the documentation wiki.

I got a new diagram!!!1!11!! It’s based on my previous one, but slightly more elaborate, and a lot less messy, as I now used Dia to create it

Legend:

• Striped lines: Existing components where code will be copied from, or based upon.
• Full lines: Components of the complete deployment model.
• Thick full lines: Core components (of the deployment model) that I definitely want to have completed during GSoC.

Since my previous post about my Google Summer of Code project I have been poking at the new MediaWiki installer to see what’s there already, how it is there, and how I can integrate it with the above deployment model. I’ve made a bunch of style and documentation improvements while going over the code, and renamed some things to make more sense. And I had Tim Starling clean up a bad svn commit I made

So what I’m doing now is splitting the current ‘Installer’ class, which is part of the new installer, into 2: Installer and CoreInstaller. Installer will hold general installer functionality and be part of the whole deployment model, while CoreInstaller will hold installer functionality specific to core, and will be part of the new installer. After that I can create an initial version of CoreInstallers counterpart: ExtensionInstaller.

Today my MediaWiki.org user page is one year old – I created the first version on July 20, 2009. With my SVN account also approaching it’s first birthday, I can now say I’m doing MediaWiki development for a year. A lot has happened in this year.

I created the Maps and Semantic Maps extensions, and have continues releasing big and small updates the whole year long. At the end of 2009 I created the Validator extension to facilitate parameter handling in Maps and Semantic Maps. In early 2010 I was contracted by the Wikimedia Foundation to create the Storyboard extension, and by the Karlsruhe Institute of Technology to do work on Semantic MediaWiki. In May 2010 I started working on my Google Summer of Code 2010 project to create an extension management platform for MediaWiki. In between all these things I made various contributions to other extensions, including Semantic Forms, Semantic Internal Objects, Page Object Model, Semantic Compound Queries, Semantic Result Formats and Approved Revisions.

Next to all the code I created and released, I also attended several events and gave a number of presentations. These events include SMWCamp 2009 in Karlsruhe, the Berlin developers workshop in April and Wikimania 2010 in Gdansk.

I’m currently all time MediaWiki comitter #18, with 1080 commits. Looking forward to all the awesome stuff I can do in the coming year

I figured having some unit tests for Maps, the MediaWiki extension to work with geographical data and display it by embedding dynamic maps into your articles, would be beneficial to it’s quality. It’s pretty hard to try cover all possible use cases with manual tests, and consumes a lot of time in any case. I therefore decided to try create some tests for the coordinate parser and formatter class, as it’s arguably the core feature of Maps.

I started off by trying to install plain PHPUnit, which is the most commonly used unit testing framework for PHP. This took me a while, as you are supposed to install it using PEAR (PHP Extension and Application Repository), a repository tool for PHP applications, and has never used this before. After two hours or so of messing around, I got both installed Then I went on investigating how I could best integrate this into my work-flow, and discovered that PHPUnit comes bundled with Zend Studio, seamlessly integrated, working completely out of the box o_O.

I then wrote a test case for the coordinate parsing and formatting class of Maps. I had a hard time getting it to work, as I needed to include MW itself, as the class uses MW functions. After some non-constructive discussion with several fellow MW devs I found a way to get it to work by including the maintenance script entry point, and tricking MW into thinking the call was made from a CLI. I now have a test case for the coordinate class, with tests for most of it’s functionality. Some more test data, and maybe some extra tests would be nice. A tricky thing in the case of this class is founding errors, which are hard to take into account, especially if you only want to allow them to a certain degree.

This particular test case is already paying off, as it made me find 3 subtle errors in coordinate parsing or formatting, that did not show up in my manual tests, as I was not covering the test data causing the issues.

I’m now planning to maybe write test cases for the distance parser to, which should be rather easy to do. I probably won’t create any others for Maps, as it’s rather time consuming, and I have a lot of other things to do right now. When I create new classes that are suited for unit tests in the future, I’ll definitely write tests for them as I build them up though, as it’ll not cost a lot more time then doing manual tests, and will ensure the classes are really solid.

About a year back I decided I had to change my development environment from working on a remote server with a simple text editor to something more solid, if I wanted to do serious PHP development.

I went for Eclipse as IDE, which is without a doubt the most awesome IDE out there, as it’s completely open source, robust, and has a lot of extensions that make it usable for an awful lot of languages. For PHP there is the open source PHP Development Tools Project, and the commercial Eclipse based IDE build on top of this, Zend Studio. At that point having my development environment as mobile as possible was rather important, as I often occupied machines other then my laptop or my own desktop. Therefore I put both Eclipse and my server on an usb.

Now, a year later, a lot has changed, and I finally updated my tools accordingly. First of all, I’m now also using Linux, and plan to leave Windows into the dust bin soon, so prefer tools that work on both operating systems. Secondly, I now only develop on my laptop and own desktop machine, so portability is not as important any more. Also, I’m now doing a lot more serious PHP development as I was doing a year ago, and would benefit a lot from more decent debugging, testing and profiling tools.

What I did was throwing out my mobile web server and install Zend Studio, together with Zend Server Community Edition (which is free). They integrate in such a way that you can do code tracing, work with breakpoints, profile code, ect, all out of the box. If you are developing PHP applications like me and in search for a good tool, I can definitely recommend this. Zend Studio isn’t free, but it’s worth the price. Not going into a complete list of awesome stuff PDT and Zend Studio include, but this blog post, although a little dates, does a good job at it.

It’ll take me a while to integrate these features into my work-flow, as I’m not used to having them available, but I expect this to start paying off rather soon then late Also waiting on a new release of Zend Studio build on Eclipse 3.6 (Helios).

(O yeah, can’t write a post about Eclipse without saying: NetBeans fails )

A lot has happened in my Google Summer of Code project since my last blog post about it, so here is another update. I did give a short presentation about it at Wikimania 2010, but that did not go very well unfortunately. You can get the slides though.

I changed the goals of my project again, from building an extension management platform build onto the Deployment Framework and Configure to creating a more general Deployment base on which the new MediaWiki installer and extension management can run. The reason for this change is that it makes a lot more sense from a technical perspective. The work done by the MediaWiki installer, and the still-to-be-created realized extension management is very similar after all. Because of this change I renamed the project from Extension Management Platform to Deployment.

Goals

So let’s have a look at what my idea of the end product is, from an users perspective. There would be 4 new special pages with the extension management functionality, largely based on what you can do in WordPress:

• Special:Install: On this page administrators can browse and search through extensions that are in the connected repository. This can be very basic to start with, but should eventually include filtering on categories and keywords, popularity, rating, ect.
• Special:Update: Checks for updates for both core and extensions, and shows update options for individual components, or the whole deal.
• Special:Extensions: A page listing all installed extensions, with options to uninstall, disable and upgrade them, as well as links to documentation, ect. Once MediaWiki has a configuration database, links to configure the extensions can also be included here.
• Special:Dashboard: A dashboard for administrators containing update information and fancy stuff like statistics.

The new MediaWiki installer would also have support for installation extension on initial run, and be able to do a complete core upgrade, instead of just a database update as is now the case.

Technical

This is a rough draft of how I see the structure of the end product, and where the code is coming from:

I’m in search of a quick and easy diagram tool to make a cleaner and more elaborate version

Stuff we already have

The underneath items have been completed, or require only a little work.

• Web interface for the core installer.
• Database abstraction for the installer.
• CLI interface for the core installer? (in progress)
• Core installer class with database install and upgrade capabilities.
• Filesystem abstraction (I ported this from WP (see code here), mostly done, no testing done yet though).

Stuff that’s still needed

• Everything related to detecting updates, fetching packages and instructions, ect. The DF has some nice stuff that can be used here, so does WP. It’s be nice to also have an abstraction layer here, so multiple mechanisms can be used here. An extension repository also needs to be set up, preferably on mediawiki.org.
• Filesystem support for the installer, so it can be used to upgrade MW by clicking a button and then just fetching the new release and doing all the work. This can be achieved by creating the generic installer class and making the core installer inherit from it.
• Extension installer class and the special pages that provide an interface to it.
• Extension support for the core installer: installation and upgrade. This can be done by re-using the code of the special pages.
• CLI support for extension management

Work done

This is a list of the work I have so far done during this project, oldest first.

• I researched all involved components and created the above draft.
• I ported the WordPress filesystem abstraction classes (base class, FTP and direct one).
• I got core commit access (yay!), so I now can make changes to the new installer.

Roadmap

A rather loose planning of what I’m planning to do next:

• Finish porting SSH2 filesystem abstraction class.
• Figure out how to make the whole system secure.
• Create installer class, adapt to core installer to work with this, and also create the extension installer class.
• Take care of the fetching stuff.
• Create the interfaces.

I’ll update the deployment wiki page as I make progress, but probably won’t make a lot of blog posts about it, as I want to focus on the work itself. Suggestions are welcome on the discussion page.

I’ve been “switching” to linux for about 2 months now, and am making this post because I feel that at this point I can just drop Windows while continuing to develop and do other stuff at a normal rate.

I’ve finally made the dicision to switch to Linux after being semi-lynched for using Windows at the local hackerspaces for a while and then finally my windows install failing fatally, requiring me to re-install. I got so tired of the ridiculous Windows bootloader and all other Windows crap I decided to give Ubuntu a go, as Lucid Lynx, the 10.04 LTS release, was just out. Soon after having messed with most basics, I decided to try out KDE, as Gnome was lacking some Windows 7 window manager like features. I then sicked with KDE as I find the window manager a lot better, as you can drag windows to a side of a screen to pack them there, or to the top to maximise, and can lock windows and applications to a certain virtual desktop, or make them appear there initially, which saves you moving around all your startup applications every time you login.

It took me quite a while to set up my whole dev environment, as I was not familiar with quite some of the needed apps, and more significantly, had next to no command line experience. I used to think command lines where evil, as clicking buttons works out of the box, while for the command line you have to learn the commands. Although this is true to some extend, I now love command line, as it allows you to chain scripts together, and even write your own, which you obviously can’t do with a GUI app. The self-written script I’m enjoying the most at the moment is one that svn updates my MediaWiki trunk and several MediaWiki tags, and then loops through my MW extension directory to update all extensions there. Similarly, I have a script that svn commits all that stuff. A lot faster (and more awesome) then doing it via TortoiseSVN on Windows!

The most notable application switch I needed to make was getting rid of U3 and RoboForm2Go, which I’m rather happy I finally did. U3 is a portable applications platform that runs only on Windows and was nice when it came out, but then all development of apps for it failed. RoboForm2Go is a portable, U3 version, of RoboForm, a rather nice cross-browser plugin (but not for Chrome ), that inserts a toolbar into your browser that stores bookmars and ‘passcards’. I have now switched to using Mozilla Sync (part of Mozilla Weave), or rather Firefox Sync, as it was recently renamed, which is a Firefox plugin that synchronizes user data such as bookmarks, history and passwords between different Firefox installs. It’s an awesome plugin, as the data is encrypted in such a way you are the only one that can decode it (Mozilla can’t, so no privacy issues) and is developed by one of the awesomest orgs out there, which is the Mozilla Foundation. As this does not run on other browsers (I hope it soon will, at least for Chrome), and does not work offline on a machine where you don’t have your data synchronized yet, I’m also using KeePass, a cool password manager, which has a port that works on Linux, called KeePassX, to store all my passwords and user names. To have security on my usb stick, instead of the U3 lock, I’m now using TrueCrypt, a cross-platform on-the-fly encryption application. I’m very happy with it, as it provides theoretical security by really encrypting the data, where U3 only locks access to your drive partition, and have also encrypted most of the partitions I have on my external hard drives (~1.5TiB encrypted now). It requires a virtual partition to mount each encrypted one on, which eats up drive letters on Windows, proving drive letters totally suck.

I’m having issues with getting (k)ubuntu to work nicely on my dual screen desktop machine though, which is making me stick to windows there for the moment. I’m going to put some more effort in fully switching to Kubuntu after my GSoC project has ended.

Now I can call people Windows-fags – yay!

Last 3 days I’ve been at the 2010 edittion of Wikimania, one of the biggest yearly wiki events. This year it was in Gdansk, Poland. I went by Plane from Eindhoven, together with Maarten, a moderator on the Dutch Wikipedia with >5 year history, who drove me to the airport.

I’ll start with the things that where not nice, basically the heat and the dormitories. It was 30+ °C practically the whole time, and the dormitory I stayed in, which was reserved by the event organizers for the attendees, was more like a sauna then a dormitory. It was located right next to a viaduct, so when opening the window at night, you’d get a lot of noise. On the last night the people from the event could not stay at that dormitory, and had to go somewhere else. Me and another attendee from Belgium got assigned some dormitory a few km away from the event. Saying this was a crappy one is an understatement. Furthermore, the payment was not arranged, so we had to pay for staying, while it should have been done by the event organizers as we paid for it?!! In any case, this amounted to me having less then 10 hours of sleep over 4 nights And they don’t have Club Mate in Gdansk o_O!!!

I think the event itself was great, with lots of interesting talks, awesome people and good food.

On the first day I missed the keynote since my flight arrived to late to see it, and only followed some non-technical talks, of which the most notable one was about Liquid Threads, the totally awesome extension by Andrew Garrett and now also some other people. I left early, skipping the deinner and following events, and went straight to the dormitory, to get some sleep (which failed >_>)

I gave 2 talks, one about my Google Summer of Code project, now titled Deployment, and one about Maps and Semantic Maps, both on the second day. The Deployment talk went pretty bad, as I was extremely tired. Hopefully I got the core message across of the importance of having a solid and user friendly way of deployment. Apologies to all the people I forgot to give credits to! Daniel Kinzler had some interesting security concerns which we discussed later on the day, together with Markus.

On the second day of the event there was a series of Semantic MediaWiki talks and workshops, kicked off by a presentation about the concept and what SMW has become over the past 5 years by Markus. This was followed by a workshop by Hans-Jörg Happel and Frank Dengler about Semantic Result Formats. Daniel Herzig gave a talk about AskTheWiki, and promising extension he is developing. There where two more SMW related talks focusing on the advantages of SMW and how it can be deployed. Later on the second day I gave my talk about Maps and Semantic Maps, preceded by one of Tim Alder, who presented the capabilities of the geo-related work on the toolserver. Luckily I was awake to some extend during this talk, and it went reasonably well IMO, although I had to rush through it, cause of to little time.

The second day was closed by the word premier of the movie “Truth in Numbers“, which aims at giving people an idea about the goals of the Wikimedia Foundation, and how it works. It shows arguments from both people involved or enthusiastic about the foundation, and those who think the world is going to explode if you go to Wikipedia for whatever reason. It will be interesting to see how this movie gets distributed and what effect it will have. I’m a little sad about how the creators choose to distribute it though, as it will require you to pay for it. It would be a lot more awesome if it was free with a big donate button. All the footage, which is a lot more then what’s seen in the movie itself includes, but not edited, will be released for free under a creative commons (or similar?) licence though.

The third day I found the most enjoyable as I did get some amount of sleep the night before. (In other words, I was awake enough again to program during the talks : ) The most interesting talk for me I followed here was definitely the one by Roan Kattouw about writing MediaWiki extensions, which mentioned some things I did not know about yet (such as build in MediaWiki support for Memcached o_O), as well as provided a nice overview of the things you need to keep in mind. I wish I had seen it a year earlier though, as I would have learned a lot more from it then, and not made a lot of the noted beginner mistakes. In the next session I attended several strategy sessions, which gave me a nice idea of what all the strategy fuss is about. The last session I attended was about Wikimedia credibility, including a talk by Maarten about how information (esp the incorrect) spreads to other media. The other talks in this session where similar, and although obviously none of the issues addresses are technical in nature, I found several of them rather amusing.

It was great to meet all the people involved with SEMANTIC MediaWiki in person, as well as a bunch of people I only knew from IRC and other online communication tools. Sadly enough Yaron Koren and Brion Vibber and several other people I’d like to have seen there could not be at the event.

On the way back to Belgium I tackled a lot of small design issues that have been present in Maps and Semantic Maps for months, resulting in one big refactoring commit for each extension, which I made as soon as I was back home. After that I went to sleep, and woke up 19 hours later O_o

Next years Wikimania will be in Haifa, Israel. I’m probably going, but seriously hope it won’t be so insanely warm there

Maps and Semantic Maps 0.6.4 are now available for download. This release contains several new features, amongst which basic KML support for Google Maps, a new OSM service implementation and re-added service links support to the Geographical Coordinates data-type in Semantic Maps. Everyone running 0.6.2 or older is advised to upgrade as soon as possible.

Lets have a closer look at some of the changes:

• The Google Maps (v2) service now has a KML parameter, which allows you to specify one or more KML files to use. Files are separated with comma’s. The base for this feature was submitted by Andy Chow, who created it for Transit Unlimited wiki. A neat example of a map there using KML is this one. Thanks for the patch Andy!
• There again is an OpenStreetMap service, although this one is different from the one that got removed in 0.6. The implementation is done by embedding (in an iframe) a page from the WMF toolserver, which renders a map with OSM tiles and Wikipedia overlay. Example. It’s basically the same what’s done on the German Wikipedia. Credits here go to Tim Alder and all other people who are making the awesome toolserver stuff work. The reason the creation of the map is not done on the server running the MediaWiki installation itself is that it’s rather difficult to fetch the Wikipedia layer from the toolsever. Hopefully a solution is found for this, as that would be a lot cooler for non WMF usage.
• I re-added service links support for the Geographical Coordinates data-type in Semantic Maps after someone requested this. It got removed in 0.6, as the old implementation was not working anymore, and I was unsure at that point how to create a new one. You can find the updated documentation on this here, although the examples are not working yet (semantic-mediawiki.org is using an alpha of 0.6). I will migrate the examples to the mapping documentation wiki at some later date.
• OpenLayers i18n (translation of the controls) should now be working. A style fix has also been made which caused OpenLayers to render badly in Chrome.
• Multiple small bugfixes have been made in both extensions, and a few in Validator. I guess this is now be the most stable release to date yet, so be sure to upgrade!

For a full list of changes since 0.6.3 see changes to Maps and changes to SM.

Downloads

• Maps 0.6.4: zip, 7z
• Maps and Semantic Maps 0.6.4: zip, 7z

See also

• The release notes on the documentation wiki

Andy Chow