Jeroens blog

As Google Summer of Code (GSoC) 2010 has ended, I’m writing this blog post to outline what I’ve done during the coding period and what the results are. Thanks go to the Wikimedia Foundation and Google for providing the opportunity to do this project, Brion Vibber, who mentored me, and to all other people who helped me out, especially Yaron Koren who I bugged the most

What I did during GSoC

My initial proposal was to create an awesome extension management platform for MediaWiki that would allow for functionality similar to what you have in the WordPress admin panel. After doing some research I realized this would require significant effort in two areas: configuration and deployment. After looking at some already existing tools such as the Configure extension and the Deployment Framework of Ontoprise, I decided to completely drop the configuration part and concentrate on the deployment work.

I started with porting the filesystem abstraction classes from WordPress, which are needed for doing any upgrade or installation operations that include changes to the codebase. (The current MediaWiki installer can do upgrades, but only to the database.) I created a new extension called Deployment, where I put in this code, and which was intended as a place to experiment with all the MediaWiki-installation side deployment stuff. As You obviously want this functionality to be part of MediaWiki itself, I wrote it with the idea of moving over the code to MediaWiki core once it was finished. It turned out that doing filesystem upgrades securely is not an easy task though, so after finishing the port, I quitted work on this as I decided to do other functionality first.

I then poked somewhat at the new MediaWiki installer, which is a complete rewrite of the current installer with a lot of new cool stuff and a totally more awesome interface. I made some minor imrpovements there, and split the Installer class, which held core installer functionality, into a more generic Installer class and a CoreInstaller. This allows for creating an ExtensionInstaller that uses the same base code, such as database, filesystem and LocalsSttings manipulation.

After this I started thinking about how to best structure a package repository for MediaWiki and extensions to get updates and new extensions from. I had a look at PEAR and CPAN, as well as WordPress, although I don’t learn a lot about the later. Apparently their repository code is not freely available After discussion with Brion I decided to just create the repository from scratch, and started working on another extension, titled Distribution, for this purpose. I merged it together with a rewritten version of the MWReleases extension written by Chad, which already had core update detection functionality.

After the Distribution API’s where working decently I started work on the Special pages in Distribution that would serve as the equivalent of the WordPress admin panel. As I put of the configuration work, and also the file-system manipulation for the initial version, this came down to simply listing currently installed software, update detection and browsing through extensions available in the repository.

On top of my GSoC project itself, I did quite some other MediaWiki work in “my free time”. I released 5 new versions of Maps and Semantic Maps, starting with 0.6 and ending at 0.6.5. As I finally got core commit access, I also poked at some other things, such as Special:Version, which now will automatically put all extensions of unknown type in the “other” category, and will display this category as the last one. Plus misc minor improvements to a verity of extensions. This all amounts into a little over 550 commits to the MediaWiki SVN repository during the GSoC coding period.

State of the code

The Distribution extension has the infrastructure for storing and providing extension and core data via the MediaWiki API basically ready for use. It adds 4 database tables to MediaWiki:

• distribution_units: This table stores non-version specific info of ‘release units’ . Currently these unit’s are extensions only – the reason I went for a more general name is to allow for adding other things such as skins and content packages later on. The info here consist of a name, a URL, a description and a pointer to the “current version”.
• distribution_unit_versions: Entries in this table contain info about a specific version of a ‘release unit’. The info here consists of a version number, release status (beta, rc, stable, deprecated, …), release data, authors, description and some installation data.
• distribution_mwreleases: This table contains MediaWiki releases. It has been merged in from MWReleases, so all credit for it goes to Chad.
• distribution_packages: This table is not in use yet, and needs some more work. The goal is to be able to install a “package” onto your wiki which can contain multiple ‘distribution units’. This would basically be the same as Semantic Bundle is doing now, but a lot easier to set up and maintain.

The API modules added are:

• ApiQueryExtensions: Returns a list of extensions matching certain search criteria, which can include keywords, tags and authors. Only extensions with a version that has a release state acceptable for your installation are returned.
• ApiMWReleases: Gets the current MediaWiki releases. Like this distribution_mwreleases database table, this has been merged in from MWReleases and all credits go to Chad.
• ApiUpdates: This API module returns update information for the extensions you give it, and does the same for MediaWiki itself if a core version number is provided. The only info that’s returned is a version number for each unit or core, if there is an update. Otherwise nothing will be returned for that unit or core.

To populate the database with existing extension info I wrote a maintenance script “getSvnMetadata”, which goes through a local checkout of the MediaWiki extensions directory and get’s the names from the extensions. I haven’t found a good way yet to also get other extension data though.

The Deployment extension contains an abstraction layer for repository interaction and several interfaces that use this. The abstraction layer allows for supporting different kinds of repositories. The only implementation it currently has is for interaction with repositories provided by the Distribution extension.  It’s also a convenient point to implement caching, as you probably don’t want to send the requests for available updates every time you view a page on the admin panel, and allows for changes to the format the repository uses without any effects in other parts of Deployment. The interfaces that are finished to some extend are:

• Special:Extensions: This page lists all installed extensions and allows you to filter on extension type. It’s based on the WordPress “plugins” page and is currently only an improved version of the extension list in Special:Version. It’s the only special page added by Deployment that can be viewed by non administrators. When logged in however, every extension has a list of links allowing you do various actions. The extension info is handled by a new class ExtensionInfo, which parses the info of individual extensions in $wgExtensionCredits, and provides a more convenient way to work with it. This allows for adding support for a new, better, extension info format later on. A planned feature for this special page is showing update notifications in each extension row.

• Special:Install: This page allows you to search through available extensions in the repository. The interface is based on the “plugin-install” page of WordPress and allows for searching extensions based on term, tag or author. After performing a search you get a list of matching extensions showing their name, version, authors, description, link to the documentation, and a link to download them. Later on this download link will be replaced by an “Install” one.

• Special:Update: This page will inform you of any updates to both core and extensions. It’s behaves basically identical the WordPress “update” page.

The extension allows you to configure several aspects of the repository interaction:

• $wgRepositoryApiLocation: This might be an obvious one, but also a very important one, as it allows you to use a repository other the the Wikimedia Foundation one on mediawiki.org, which will be the default.
• $wgRepositoryLocation: This is similar to the previous setting, but links to a web interface providing browsing capabilities through the repository, or at least some additional info.
• $wgRepositoryPackageStates: This is a list of allow release states. By default these will only be “stable” and “beta”. Early adopters can also add “dev” and “alpha”, and there also is “rc” and “deprecated”.

What’s next?

Although some very basic functionality is working, quite some work still needs to be done to get this to the WordPress-awesomeness level. Let’s first have a look at Distribution and then Deployment:

The most basic issue with Distribution currently is that there is no script yet that allows collecting all current extension data, which is needed for it to be of any use. I’m not sure how gathering current data can be properly automated, which is the main reason the script doesn’t exist yet. Any suggestions here are very welcome! After the initial version it should be possible for extension authors to edit their extensions data, and create new releases. For this we’ll need some new special pages. The data itself can then be used to populate the extension pages on mediawiki.org, and some new magic words such as “current MediaWiki version”, can automate a bunch of stuff. After these things new features can be added, such as the management of packages, and more detailed extension information, including things such as dependencies and compatibility info.

Deployment mainly needs interface work, and will need additions to support any new information provided by the Distirbution repository. A cool feature that could be added is supplying the repository with installation information (obviously optionally), which would allow the developers to get an idea of which versions of MediaWiki core and extensions people are using. After the whole MediaWiki deployment model has been revised and is up and running, it’s configuration can similarly be reinvented. The interfaces added by Deployment can then be adapted to allow contain extension configuration.

GSoC 2011?

This was my last GSoC as a student, as I no longer qualify, since I quitted my official studies. If I’m still doing MediaWiki development next year, which I guess is pretty likely, there is a lot of change I’ll be signing up as a mentor though If you are interested in being a student in 2011, you can already put your name on the 2011 GSoC page

Some useful links

• My blog posts about this project
• The Wikimedia 2010 GSoC page with all GSoC projects
• Distribution and Deployment extensions
• My MediaWiki.org user page

Just a few quick screenshots of Special:Extensions, on which I’ve been working today. The first screenshot shows Special:Extensions page displaying a list of all the extensions I have installed on my local wiki:

As you can see, you can now filter on extension type with the control right below the “Installed extensions” title. Here I have filtered on the SEMANTIC extensions:

An interesting change I made is that you can now access this page without having the siteadmin permission. Doing this will get the above, but without the add new button and administration controls (currently only “Deactivate” which is there only for show so far). This way this page will be a nice addition to Special:Version.

Tomorrow is the last coding day in Google Summer of Code 2010, during which I’m planning to focus on the update detection functionality, or rather creating the interface for it, as the plumbing for it is all but done. I also want to move several classes from Deployment over to MediaWiki core, as they make more sense to have there, and would allow for some nice improvements.

Since my last blog post about my GSoC project, which aims to bring more awesome deployment capabilities to MediaWiki, I’ve been putting my time into both the Distribution and Deployment extensions. I was pleased to find a bunch of stuff was easier to do then I had imagined, and now I finally have some functionality you can actually see working – yay It’s not a lot, as this is just very rudimentary and even uses demo data at places. Still it’s very nice to be able to post some screenshots after months of doing research and poking at code. I also got a new crappy diagram (although I think I’m succeeding in getting them less crappy each iteration) that shows the architecture of the initial versions I’m working towards.

The following screenshots show an import script running. This script is part of Distribution and is meant to get data from a checked out copy of the extensions directory and store it into the database tables provided by Distribution. I haven’t found a good way to actually get the extension data other then their path names (for example ‘SemanticMediaWiki’), so have a function that’s just returning some demo data.

Via Deployment you can search for extensions by keyword, author or tag. This is done by making a request to the API provided by Distribution which serves the data collected by the script. This screenshot shows the interface I have created so far:

And after clicking the button, you get:

As I’m doing with all special pages in Deployment, Special:Install’s layout and functionality is based on what WordPress has in it’s admin panel.

I also put some work in Special:Extensions already, although it’s basically just Special:Version with just the extensions for now.

I’m not wring a real comprehensive overview of what features there are already, which are planned, and how I plan to create them for now, as there is little time left in GSoC (only 3 days!), and I want to get some more work finished before then

With only 2 days till the suggested Google Summer of Code ‘pencils down’ date, and a week longer until the firm one, I’m using my remaining time to get some basic functionality working for my GSoC project. I’ve started creating a new extension called Distribution that will provide an API module which can be used to query extension meta-data. Later on it should also be able to do this for core, and provide archived packages that can be downloaded to your MediaWiki server, and directly installed. Initially I’ll probably simply be pointing to the ExtensionDistributor extension on MediaWiki.org, which can create archives for extensions on request. The Deployment extension which I started on earlier on will use data obtained via the API Distribution provides to display available extensions on your local installation.  Right now I’m attempting to get the ExtensionDistributor working locally, so I can develop the Distribution extension alongside it, and confirm everything works. This will take some time as it uses Linux/UNIX specific commands, forcing me to use my Kubuntu install, on which my dev environment is only partially set up at the moment.

I got a new diagram!!!1!11!! It’s based on my previous one, but slightly more elaborate, and a lot less messy, as I now used Dia to create it

Legend:

• Striped lines: Existing components where code will be copied from, or based upon.
• Full lines: Components of the complete deployment model.
• Thick full lines: Core components (of the deployment model) that I definitely want to have completed during GSoC.

Since my previous post about my Google Summer of Code project I have been poking at the new MediaWiki installer to see what’s there already, how it is there, and how I can integrate it with the above deployment model. I’ve made a bunch of style and documentation improvements while going over the code, and renamed some things to make more sense. And I had Tim Starling clean up a bad svn commit I made

So what I’m doing now is splitting the current ‘Installer’ class, which is part of the new installer, into 2: Installer and CoreInstaller. Installer will hold general installer functionality and be part of the whole deployment model, while CoreInstaller will hold installer functionality specific to core, and will be part of the new installer. After that I can create an initial version of CoreInstallers counterpart: ExtensionInstaller.

Today my MediaWiki.org user page is one year old – I created the first version on July 20, 2009. With my SVN account also approaching it’s first birthday, I can now say I’m doing MediaWiki development for a year. A lot has happened in this year.

I created the Maps and Semantic Maps extensions, and have continues releasing big and small updates the whole year long. At the end of 2009 I created the Validator extension to facilitate parameter handling in Maps and Semantic Maps. In early 2010 I was contracted by the Wikimedia Foundation to create the Storyboard extension, and by the Karlsruhe Institute of Technology to do work on Semantic MediaWiki. In May 2010 I started working on my Google Summer of Code 2010 project to create an extension management platform for MediaWiki. In between all these things I made various contributions to other extensions, including Semantic Forms, Semantic Internal Objects, Page Object Model, Semantic Compound Queries, Semantic Result Formats and Approved Revisions.

Next to all the code I created and released, I also attended several events and gave a number of presentations. These events include SMWCamp 2009 in Karlsruhe, the Berlin developers workshop in April and Wikimania 2010 in Gdansk.

I’m currently all time MediaWiki comitter #18, with 1080 commits. Looking forward to all the awesome stuff I can do in the coming year

A lot has happened in my Google Summer of Code project since my last blog post about it, so here is another update. I did give a short presentation about it at Wikimania 2010, but that did not go very well unfortunately. You can get the slides though.

I changed the goals of my project again, from building an extension management platform build onto the Deployment Framework and Configure to creating a more general Deployment base on which the new MediaWiki installer and extension management can run. The reason for this change is that it makes a lot more sense from a technical perspective. The work done by the MediaWiki installer, and the still-to-be-created realized extension management is very similar after all. Because of this change I renamed the project from Extension Management Platform to Deployment.

Goals

So let’s have a look at what my idea of the end product is, from an users perspective. There would be 4 new special pages with the extension management functionality, largely based on what you can do in WordPress:

• Special:Install: On this page administrators can browse and search through extensions that are in the connected repository. This can be very basic to start with, but should eventually include filtering on categories and keywords, popularity, rating, ect.
• Special:Update: Checks for updates for both core and extensions, and shows update options for individual components, or the whole deal.
• Special:Extensions: A page listing all installed extensions, with options to uninstall, disable and upgrade them, as well as links to documentation, ect. Once MediaWiki has a configuration database, links to configure the extensions can also be included here.
• Special:Dashboard: A dashboard for administrators containing update information and fancy stuff like statistics.

The new MediaWiki installer would also have support for installation extension on initial run, and be able to do a complete core upgrade, instead of just a database update as is now the case.

Technical

This is a rough draft of how I see the structure of the end product, and where the code is coming from:

I’m in search of a quick and easy diagram tool to make a cleaner and more elaborate version

Stuff we already have

The underneath items have been completed, or require only a little work.

• Web interface for the core installer.
• Database abstraction for the installer.
• CLI interface for the core installer? (in progress)
• Core installer class with database install and upgrade capabilities.
• Filesystem abstraction (I ported this from WP (see code here), mostly done, no testing done yet though).

Stuff that’s still needed

• Everything related to detecting updates, fetching packages and instructions, ect. The DF has some nice stuff that can be used here, so does WP. It’s be nice to also have an abstraction layer here, so multiple mechanisms can be used here. An extension repository also needs to be set up, preferably on mediawiki.org.
• Filesystem support for the installer, so it can be used to upgrade MW by clicking a button and then just fetching the new release and doing all the work. This can be achieved by creating the generic installer class and making the core installer inherit from it.
• Extension installer class and the special pages that provide an interface to it.
• Extension support for the core installer: installation and upgrade. This can be done by re-using the code of the special pages.
• CLI support for extension management

Work done

This is a list of the work I have so far done during this project, oldest first.

• I researched all involved components and created the above draft.
• I ported the WordPress filesystem abstraction classes (base class, FTP and direct one).
• I got core commit access (yay!), so I now can make changes to the new installer.

Roadmap

A rather loose planning of what I’m planning to do next:

• Finish porting SSH2 filesystem abstraction class.
• Figure out how to make the whole system secure.
• Create installer class, adapt to core installer to work with this, and also create the extension installer class.
• Take care of the fetching stuff.
• Create the interfaces.

I’ll update the deployment wiki page as I make progress, but probably won’t make a lot of blog posts about it, as I want to focus on the work itself. Suggestions are welcome on the discussion page.

Last 3 days I’ve been at the 2010 edittion of Wikimania, one of the biggest yearly wiki events. This year it was in Gdansk, Poland. I went by Plane from Eindhoven, together with Maarten, a moderator on the Dutch Wikipedia with >5 year history, who drove me to the airport.

I’ll start with the things that where not nice, basically the heat and the dormitories. It was 30+ °C practically the whole time, and the dormitory I stayed in, which was reserved by the event organizers for the attendees, was more like a sauna then a dormitory. It was located right next to a viaduct, so when opening the window at night, you’d get a lot of noise. On the last night the people from the event could not stay at that dormitory, and had to go somewhere else. Me and another attendee from Belgium got assigned some dormitory a few km away from the event. Saying this was a crappy one is an understatement. Furthermore, the payment was not arranged, so we had to pay for staying, while it should have been done by the event organizers as we paid for it?!!  In any case, this amounted to me having less then 10 hours of sleep over 4 nights And they don’t have Club Mate in Gdansk o_O!!!

I think the event itself was great, with lots of interesting talks,  awesome people and good food.

On the first day I missed the keynote since my flight arrived to late to see it, and only followed some non-technical talks, of which the most notable one was about Liquid Threads, the totally awesome extension by Andrew Garrett and now also some other people. I left early, skipping the deinner and following events, and went straight to the dormitory, to get some sleep (which failed >_>)

I gave 2 talks, one about my Google Summer of Code project, now titled Deployment, and one about Maps and Semantic Maps, both on the second day. The Deployment talk went pretty bad, as I was extremely tired. Hopefully I got the core message across of the importance of having a solid and user friendly way of deployment. Apologies to all the people I forgot to give credits to! Daniel Kinzler had some interesting security concerns which we discussed later on the day, together with Markus.

On the second day of the event there was a series of Semantic MediaWiki talks and workshops, kicked off by a presentation about the concept and what SMW has become over the past 5 years by Markus. This was followed by a workshop by Hans-Jörg Happel and Frank Dengler about Semantic Result Formats. Daniel Herzig gave a talk about AskTheWiki, and promising extension he is developing. There where two more SMW related talks focusing on the advantages of SMW and how it can be deployed. Later on the second day I gave my talk about Maps and Semantic Maps, preceded by one of Tim Alder, who presented the capabilities of the geo-related work on the toolserver. Luckily I was awake to some extend during this talk, and it went reasonably well IMO, although I had to rush through it, cause of to little time.

The second day was closed by the word premier of the movie “Truth in Numbers“, which aims at giving people an idea about the goals of the Wikimedia Foundation, and how it works. It shows arguments from both people involved or enthusiastic about the foundation, and those who think the world is going to explode if you go to Wikipedia for whatever reason. It will be interesting to see how this movie gets distributed and what effect it will have. I’m a little sad about how the creators choose to distribute it though, as it will require you to pay for it. It would be a lot more awesome if it was free with a big donate button. All the footage, which is a lot more then what’s seen in the movie itself includes, but not edited, will be released for free under a creative commons (or similar?) licence though.

The third day I found the most enjoyable as I did get some amount of sleep the night before. (In other words, I was awake enough again to program during the talks : ) The most interesting talk for me I followed here was definitely the one by Roan Kattouw about writing MediaWiki extensions, which mentioned some things I did not know about yet (such as build in MediaWiki support for Memcached o_O), as well as provided a nice overview of the things you need to keep in mind. I wish I had seen it a year earlier though, as I would have learned a lot more from it then, and not made a lot of the noted beginner mistakes. In the next session I attended several strategy sessions, which gave me a nice idea of what all the strategy fuss is about. The last session I attended was about Wikimedia credibility, including a talk by Maarten about how information (esp the incorrect) spreads to other media. The other talks in this session where similar, and although obviously none of the issues addresses are technical in nature, I found several of them rather amusing.

It was great to meet all the people involved with SEMANTIC MediaWiki in person, as well as a bunch of people I only knew from IRC and other online communication tools. Sadly enough Yaron Koren and Brion Vibber and several other people I’d like to have seen there could not be at the event.

On the way back to Belgium I tackled a lot of small design issues that have been present in Maps and Semantic Maps for months, resulting in one big refactoring commit for each extension, which I made as soon as I was back home. After that I went to sleep, and woke up 19 hours later O_o

Next years Wikimania will be in Haifa, Israel. I’m probably going, but seriously hope it won’t be so insanely warm there

After doing a pile of research on how to best create an extension management platform for MediaWiki as my Google Summer of Code project, I realized that a lot of the work that I wanted to do was already done in some form or another, and decided to somewhat augment my goals. Since I can build upon Configure, and the Deployment Framework of Ontoprise, it should now be possible to also take care of the MediaWiki and extension configuration that I put as optional in my original proposal. To have some transparency here, and not to cause misconceptions, I created an awesome new roadmap.

I did an attempt to get some feedback by posting my roadmap on wikitech-l, but apparently everyone is either happy with it, or more likely, Parkinson’s Law of Triviality is in play here. So feedback and suggestions are definitely welcome, you can post them on the discussion page.

As Configure is doing an awesome job already without making significant changes to any existing code, I decided to start off with the Deployment Framework. I’m currently in the process of figuring out how it works exactly, so I’m able to extend it’s features, and build a GUI in the form of several MediaWiki special pages on top of it. This will have some immediate pay-off as Ontoprise will be able to use this improved version directly.

Both my presentations for Wikimania 2010 have been accepted, of which one is about my project, which will give me a change to explain to people what I’m doing and why it’s so awesome

I’ve been working on Maps and Semantic Maps 0.6, the next big, awesome, release of both MediaWiki extensions, for over a month now. I also released an early alpha on the 3th of April. All planned new features have been implemented, and known bugs from 0.5.x have been fixed. So you might wonder why 0.6 is still not out.

In response to the possibility of someone doing work on Maps and Semantic Maps during the upcoming Google Summer of Code, I took a critical look at the current structure, holding into account all the things I learned over the last two month while working on Storyboard. I came to the conclusion that a bunch of things ought to be handled in a cleaner fashion, and started to refactor this code. The most difficult part here is changing how the display_map and display_point(s) parser functions handle their mapping service parameter and validate the provided location(s). I’ve been wanting to change this since 0.4, but didn’t since it’s rather tricky to do. I decided to finally get this done now, and have done most of the work. To complete these changes, I’ll have to make some rather complex modifications to Validator, which can take a while to complete. That’s the last thing that needs to be done before the 0.6 release though

I estimate this should be done in approximately 2 weeks, maybe sooner. After that I’m planning to release at least one RC, to ensure stability and complete awesomeness, cause really a lot has been changed. I figure about three quarters of the codebase has been changed in one way or another!

After this release a lot of cool new features can be added, which I might do before the GSoC coding period starts. There are also a few people who wanted to do a mapping project for GSoC at the WMF, but did not get accepted, which are planning to do some effort here after all, which is totally awesome