Jeroens blog

With only 2 days till the suggested Google Summer of Code ‘pencils down’ date, and a week longer until the firm one, I’m using my remaining time to get some basic functionality working for my GSoC project. I’ve started creating a new extension called Distribution that will provide an API module which can be used to query extension meta-data. Later on it should also be able to do this for core, and provide archived packages that can be downloaded to your MediaWiki server, and directly installed. Initially I’ll probably simply be pointing to the ExtensionDistributor extension on MediaWiki.org, which can create archives for extensions on request. The Deployment extension which I started on earlier on will use data obtained via the API Distribution provides to display available extensions on your local installation. Right now I’m attempting to get the ExtensionDistributor working locally, so I can develop the Distribution extension alongside it, and confirm everything works. This will take some time as it uses Linux/UNIX specific commands, forcing me to use my Kubuntu install, on which my dev environment is only partially set up at the moment.

Maps and Semantic Maps 0.6.5 are now available for download. This release contains mainly internal changes to improve code modularity and fix some security concerns. Several bugs have been fixed as well, and a new hook has been added to Semantic Maps. This hook will get you the map format as default one for queries where you only ask for coordinates when using SMW 1.5.2 or above. For a full list of changes since 0.6.4 see changes to Maps and changes to SM. Everyone running 0.6.2 or older is advised to upgrade as soon as possible. 

This release is notable for it being the first one in which I’m happy with the code-base as a whole. It took me a year to get here, but now I think the way the mapping extensions work is good and solid. This means you can now extend Maps and not be afraid the code will be incompatible in a few weeks due to changes. This also means that I’ll be focusing more on actual functionality rather then refactoring in future releases. I’ll be progressively building a little guide that explains how the extensions work from a developers perspective and how to extend them.

I might release another minor update in the 0.6.x series if any significant issues are found in 0.6.5. Further plans are finishing up a bunch of changes I’ve started to make in Validator, which I’ll probably release as 0.4 then, and to start working on Maps and Semantic Maps 0.7, which would aim at adding new features and improving existing ones. A likely new feature I’m particularly looking forward to implementing is several tag extensions that do the equivalent of the current parser functions added by Maps. The timetable for all this depends a lot on which other things I get cough up in (I’ll probably continue putting effort into the deployment stuff for my GSoC project) and what kind of funding will be available.

Downloads:

• Maps 0.6.5 [zip - 7z]
• Maps and Semantic Maps 0.6.5 [zip - 7z]

You can also view the release announcement at the documentation wiki.

I got a new diagram!!!1!11!! It’s based on my previous one, but slightly more elaborate, and a lot less messy, as I now used Dia to create it

Legend:

• Striped lines: Existing components where code will be copied from, or based upon.
• Full lines: Components of the complete deployment model.
• Thick full lines: Core components (of the deployment model) that I definitely want to have completed during GSoC.

Since my previous post about my Google Summer of Code project I have been poking at the new MediaWiki installer to see what’s there already, how it is there, and how I can integrate it with the above deployment model. I’ve made a bunch of style and documentation improvements while going over the code, and renamed some things to make more sense. And I had Tim Starling clean up a bad svn commit I made

So what I’m doing now is splitting the current ‘Installer’ class, which is part of the new installer, into 2: Installer and CoreInstaller. Installer will hold general installer functionality and be part of the whole deployment model, while CoreInstaller will hold installer functionality specific to core, and will be part of the new installer. After that I can create an initial version of CoreInstallers counterpart: ExtensionInstaller.

I figured having some unit tests for Maps, the MediaWiki extension to work with geographical data and display it by embedding dynamic maps into your articles, would be beneficial to it’s quality. It’s pretty hard to try cover all possible use cases with manual tests, and consumes a lot of time in any case. I therefore decided to try create some tests for the coordinate parser and formatter class, as it’s arguably the core feature of Maps.

I started off by trying to install plain PHPUnit, which is the most commonly used unit testing framework for PHP. This took me a while, as you are supposed to install it using PEAR (PHP Extension and Application Repository), a repository tool for PHP applications, and has never used this before. After two hours or so of messing around, I got both installed Then I went on investigating how I could best integrate this into my work-flow, and discovered that PHPUnit comes bundled with Zend Studio, seamlessly integrated, working completely out of the box o_O.

I then wrote a test case for the coordinate parsing and formatting class of Maps. I had a hard time getting it to work, as I needed to include MW itself, as the class uses MW functions. After some non-constructive discussion with several fellow MW devs I found a way to get it to work by including the maintenance script entry point, and tricking MW into thinking the call was made from a CLI. I now have a test case for the coordinate class, with tests for most of it’s functionality. Some more test data, and maybe some extra tests would be nice. A tricky thing in the case of this class is founding errors, which are hard to take into account, especially if you only want to allow them to a certain degree.

This particular test case is already paying off, as it made me find 3 subtle errors in coordinate parsing or formatting, that did not show up in my manual tests, as I was not covering the test data causing the issues.

I’m now planning to maybe write test cases for the distance parser to, which should be rather easy to do. I probably won’t create any others for Maps, as it’s rather time consuming, and I have a lot of other things to do right now. When I create new classes that are suited for unit tests in the future, I’ll definitely write tests for them as I build them up though, as it’ll not cost a lot more time then doing manual tests, and will ensure the classes are really solid.

A lot has happened in my Google Summer of Code project since my last blog post about it, so here is another update. I did give a short presentation about it at Wikimania 2010, but that did not go very well unfortunately. You can get the slides though.

I changed the goals of my project again, from building an extension management platform build onto the Deployment Framework and Configure to creating a more general Deployment base on which the new MediaWiki installer and extension management can run. The reason for this change is that it makes a lot more sense from a technical perspective. The work done by the MediaWiki installer, and the still-to-be-created realized extension management is very similar after all. Because of this change I renamed the project from Extension Management Platform to Deployment.

Goals

So let’s have a look at what my idea of the end product is, from an users perspective. There would be 4 new special pages with the extension management functionality, largely based on what you can do in WordPress:

• Special:Install: On this page administrators can browse and search through extensions that are in the connected repository. This can be very basic to start with, but should eventually include filtering on categories and keywords, popularity, rating, ect.
• Special:Update: Checks for updates for both core and extensions, and shows update options for individual components, or the whole deal.
• Special:Extensions: A page listing all installed extensions, with options to uninstall, disable and upgrade them, as well as links to documentation, ect. Once MediaWiki has a configuration database, links to configure the extensions can also be included here.
• Special:Dashboard: A dashboard for administrators containing update information and fancy stuff like statistics.

The new MediaWiki installer would also have support for installation extension on initial run, and be able to do a complete core upgrade, instead of just a database update as is now the case.

Technical

This is a rough draft of how I see the structure of the end product, and where the code is coming from:

I’m in search of a quick and easy diagram tool to make a cleaner and more elaborate version

Stuff we already have

The underneath items have been completed, or require only a little work.

• Web interface for the core installer.
• Database abstraction for the installer.
• CLI interface for the core installer? (in progress)
• Core installer class with database install and upgrade capabilities.
• Filesystem abstraction (I ported this from WP (see code here), mostly done, no testing done yet though).

Stuff that’s still needed

• Everything related to detecting updates, fetching packages and instructions, ect. The DF has some nice stuff that can be used here, so does WP. It’s be nice to also have an abstraction layer here, so multiple mechanisms can be used here. An extension repository also needs to be set up, preferably on mediawiki.org.
• Filesystem support for the installer, so it can be used to upgrade MW by clicking a button and then just fetching the new release and doing all the work. This can be achieved by creating the generic installer class and making the core installer inherit from it.
• Extension installer class and the special pages that provide an interface to it.
• Extension support for the core installer: installation and upgrade. This can be done by re-using the code of the special pages.
• CLI support for extension management

Work done

This is a list of the work I have so far done during this project, oldest first.

• I researched all involved components and created the above draft.
• I ported the WordPress filesystem abstraction classes (base class, FTP and direct one).
• I got core commit access (yay!), so I now can make changes to the new installer.

Roadmap

A rather loose planning of what I’m planning to do next:

• Finish porting SSH2 filesystem abstraction class.
• Figure out how to make the whole system secure.
• Create installer class, adapt to core installer to work with this, and also create the extension installer class.
• Take care of the fetching stuff.
• Create the interfaces.

I’ll update the deployment wiki page as I make progress, but probably won’t make a lot of blog posts about it, as I want to focus on the work itself. Suggestions are welcome on the discussion page.

Last 3 days I’ve been at the 2010 edittion of Wikimania, one of the biggest yearly wiki events. This year it was in Gdansk, Poland. I went by Plane from Eindhoven, together with Maarten, a moderator on the Dutch Wikipedia with >5 year history, who drove me to the airport.

I’ll start with the things that where not nice, basically the heat and the dormitories. It was 30+ °C practically the whole time, and the dormitory I stayed in, which was reserved by the event organizers for the attendees, was more like a sauna then a dormitory. It was located right next to a viaduct, so when opening the window at night, you’d get a lot of noise. On the last night the people from the event could not stay at that dormitory, and had to go somewhere else. Me and another attendee from Belgium got assigned some dormitory a few km away from the event. Saying this was a crappy one is an understatement. Furthermore, the payment was not arranged, so we had to pay for staying, while it should have been done by the event organizers as we paid for it?!! In any case, this amounted to me having less then 10 hours of sleep over 4 nights And they don’t have Club Mate in Gdansk o_O!!!

I think the event itself was great, with lots of interesting talks, awesome people and good food.

On the first day I missed the keynote since my flight arrived to late to see it, and only followed some non-technical talks, of which the most notable one was about Liquid Threads, the totally awesome extension by Andrew Garrett and now also some other people. I left early, skipping the deinner and following events, and went straight to the dormitory, to get some sleep (which failed >_>)

I gave 2 talks, one about my Google Summer of Code project, now titled Deployment, and one about Maps and Semantic Maps, both on the second day. The Deployment talk went pretty bad, as I was extremely tired. Hopefully I got the core message across of the importance of having a solid and user friendly way of deployment. Apologies to all the people I forgot to give credits to! Daniel Kinzler had some interesting security concerns which we discussed later on the day, together with Markus.

On the second day of the event there was a series of Semantic MediaWiki talks and workshops, kicked off by a presentation about the concept and what SMW has become over the past 5 years by Markus. This was followed by a workshop by Hans-Jörg Happel and Frank Dengler about Semantic Result Formats. Daniel Herzig gave a talk about AskTheWiki, and promising extension he is developing. There where two more SMW related talks focusing on the advantages of SMW and how it can be deployed. Later on the second day I gave my talk about Maps and Semantic Maps, preceded by one of Tim Alder, who presented the capabilities of the geo-related work on the toolserver. Luckily I was awake to some extend during this talk, and it went reasonably well IMO, although I had to rush through it, cause of to little time.

The second day was closed by the word premier of the movie “Truth in Numbers“, which aims at giving people an idea about the goals of the Wikimedia Foundation, and how it works. It shows arguments from both people involved or enthusiastic about the foundation, and those who think the world is going to explode if you go to Wikipedia for whatever reason. It will be interesting to see how this movie gets distributed and what effect it will have. I’m a little sad about how the creators choose to distribute it though, as it will require you to pay for it. It would be a lot more awesome if it was free with a big donate button. All the footage, which is a lot more then what’s seen in the movie itself includes, but not edited, will be released for free under a creative commons (or similar?) licence though.

The third day I found the most enjoyable as I did get some amount of sleep the night before. (In other words, I was awake enough again to program during the talks : ) The most interesting talk for me I followed here was definitely the one by Roan Kattouw about writing MediaWiki extensions, which mentioned some things I did not know about yet (such as build in MediaWiki support for Memcached o_O), as well as provided a nice overview of the things you need to keep in mind. I wish I had seen it a year earlier though, as I would have learned a lot more from it then, and not made a lot of the noted beginner mistakes. In the next session I attended several strategy sessions, which gave me a nice idea of what all the strategy fuss is about. The last session I attended was about Wikimedia credibility, including a talk by Maarten about how information (esp the incorrect) spreads to other media. The other talks in this session where similar, and although obviously none of the issues addresses are technical in nature, I found several of them rather amusing.

It was great to meet all the people involved with SEMANTIC MediaWiki in person, as well as a bunch of people I only knew from IRC and other online communication tools. Sadly enough Yaron Koren and Brion Vibber and several other people I’d like to have seen there could not be at the event.

On the way back to Belgium I tackled a lot of small design issues that have been present in Maps and Semantic Maps for months, resulting in one big refactoring commit for each extension, which I made as soon as I was back home. After that I went to sleep, and woke up 19 hours later O_o

Next years Wikimania will be in Haifa, Israel. I’m probably going, but seriously hope it won’t be so insanely warm there

Maps and Semantic Maps 0.6.4 are now available for download. This release contains several new features, amongst which basic KML support for Google Maps, a new OSM service implementation and re-added service links support to the Geographical Coordinates data-type in Semantic Maps. Everyone running 0.6.2 or older is advised to upgrade as soon as possible.

Lets have a closer look at some of the changes:

• The Google Maps (v2) service now has a KML parameter, which allows you to specify one or more KML files to use. Files are separated with comma’s. The base for this feature was submitted by Andy Chow, who created it for Transit Unlimited wiki. A neat example of a map there using KML is this one. Thanks for the patch Andy!
• There again is an OpenStreetMap service, although this one is different from the one that got removed in 0.6. The implementation is done by embedding (in an iframe) a page from the WMF toolserver, which renders a map with OSM tiles and Wikipedia overlay. Example. It’s basically the same what’s done on the German Wikipedia. Credits here go to Tim Alder and all other people who are making the awesome toolserver stuff work. The reason the creation of the map is not done on the server running the MediaWiki installation itself is that it’s rather difficult to fetch the Wikipedia layer from the toolsever. Hopefully a solution is found for this, as that would be a lot cooler for non WMF usage.
• I re-added service links support for the Geographical Coordinates data-type in Semantic Maps after someone requested this. It got removed in 0.6, as the old implementation was not working anymore, and I was unsure at that point how to create a new one. You can find the updated documentation on this here, although the examples are not working yet (semantic-mediawiki.org is using an alpha of 0.6). I will migrate the examples to the mapping documentation wiki at some later date.
• OpenLayers i18n (translation of the controls) should now be working. A style fix has also been made which caused OpenLayers to render badly in Chrome.
• Multiple small bugfixes have been made in both extensions, and a few in Validator. I guess this is now be the most stable release to date yet, so be sure to upgrade!

For a full list of changes since 0.6.3 see changes to Maps and changes to SM.

Downloads

• Maps 0.6.4: zip, 7z
• Maps and Semantic Maps 0.6.4: zip, 7z

See also

• The release notes on the documentation wiki

Andy Chow

Maps and Semantic Maps 0.6.3 are now available for download. This release is one aimed to improve stability and addresses a number of bugs that where present in 0.6.x. Some notable fixed issues are a failure of parsing any coordinates containing a degree symbol that surfaced in 0.6.2, and the failing of non-Google Maps form inputs. Some internal rewriting has also been done, which was needed to address some of these bugs, and will enable future awesomeness increase. The included OpenLayers library has also been upgraded to 2.9.1. (changes to Maps, changes to SM). People running any 0.6.x are advised to upgrade. Especially people using Semantic Maps in conjunction with Semantic Forms, as both 0.6 and 0.6.1 contain undesired behaviour in the forms.

A new release of Semantic Bundle, which will include these versions of the mapping extensions, is likely to soon be released as well

Downloads

• Maps 0.6.3 (7z)
• Maps and Semantic Maps 0.6.3 (7z)

See also

• The release notes on the documentation wiki

After doing a pile of research on how to best create an extension management platform for MediaWiki as my Google Summer of Code project, I realized that a lot of the work that I wanted to do was already done in some form or another, and decided to somewhat augment my goals. Since I can build upon Configure, and the Deployment Framework of Ontoprise, it should now be possible to also take care of the MediaWiki and extension configuration that I put as optional in my original proposal. To have some transparency here, and not to cause misconceptions, I created an awesome new roadmap.

I did an attempt to get some feedback by posting my roadmap on wikitech-l, but apparently everyone is either happy with it, or more likely, Parkinson’s Law of Triviality is in play here. So feedback and suggestions are definitely welcome, you can post them on the discussion page.

As Configure is doing an awesome job already without making significant changes to any existing code, I decided to start off with the Deployment Framework. I’m currently in the process of figuring out how it works exactly, so I’m able to extend it’s features, and build a GUI in the form of several MediaWiki special pages on top of it. This will have some immediate pay-off as Ontoprise will be able to use this improved version directly.

Both my presentations for Wikimania 2010 have been accepted, of which one is about my project, which will give me a change to explain to people what I’m doing and why it’s so awesome

Maps and Semantic Maps 0.6.2 are now available for download.

This release contains improvements to parameter handling, case insensitivity has been re-added, and using default parameters together with named ones will work more intuitive. In Maps the handling of distances has also been considerably improved: there is a new #distance parser function, additional distance related parameters in #geodistance, and new settings that allow you to customize the default distance behaviour. In Semantic Maps a rather important bugfix has been made which solves a problem with editing articles with forms that contain maps that arose in 0.6.1. (changes to Maps, changes to SM). People running 0.6 and 0.6.1 are advised to upgrade. Especially people using Semantic Maps in conjunction with Semantic Forms, as both 0.6 and 0.6.1 contain undesired behaviour in the forms.

Read this announcement on the documentation wiki