Jeroens blog

As Google Summer of Code (GSoC) 2010 has ended, I’m writing this blog post to outline what I’ve done during the coding period and what the results are. Thanks go to the Wikimedia Foundation and Google for providing the opportunity to do this project, Brion Vibber, who mentored me, and to all other people who helped me out, especially Yaron Koren who I bugged the most

What I did during GSoC

My initial proposal was to create an awesome extension management platform for MediaWiki that would allow for functionality similar to what you have in the WordPress admin panel. After doing some research I realized this would require significant effort in two areas: configuration and deployment. After looking at some already existing tools such as the Configure extension and the Deployment Framework of Ontoprise, I decided to completely drop the configuration part and concentrate on the deployment work.

I started with porting the filesystem abstraction classes from WordPress, which are needed for doing any upgrade or installation operations that include changes to the codebase. (The current MediaWiki installer can do upgrades, but only to the database.) I created a new extension called Deployment, where I put in this code, and which was intended as a place to experiment with all the MediaWiki-installation side deployment stuff. As You obviously want this functionality to be part of MediaWiki itself, I wrote it with the idea of moving over the code to MediaWiki core once it was finished. It turned out that doing filesystem upgrades securely is not an easy task though, so after finishing the port, I quitted work on this as I decided to do other functionality first.

I then poked somewhat at the new MediaWiki installer, which is a complete rewrite of the current installer with a lot of new cool stuff and a totally more awesome interface. I made some minor imrpovements there, and split the Installer class, which held core installer functionality, into a more generic Installer class and a CoreInstaller. This allows for creating an ExtensionInstaller that uses the same base code, such as database, filesystem and LocalsSttings manipulation.

After this I started thinking about how to best structure a package repository for MediaWiki and extensions to get updates and new extensions from. I had a look at PEAR and CPAN, as well as WordPress, although I don’t learn a lot about the later. Apparently their repository code is not freely available After discussion with Brion I decided to just create the repository from scratch, and started working on another extension, titled Distribution, for this purpose. I merged it together with a rewritten version of the MWReleases extension written by Chad, which already had core update detection functionality.

After the Distribution API’s where working decently I started work on the Special pages in Distribution that would serve as the equivalent of the WordPress admin panel. As I put of the configuration work, and also the file-system manipulation for the initial version, this came down to simply listing currently installed software, update detection and browsing through extensions available in the repository.

On top of my GSoC project itself, I did quite some other MediaWiki work in “my free time”. I released 5 new versions of Maps and Semantic Maps, starting with 0.6 and ending at 0.6.5. As I finally got core commit access, I also poked at some other things, such as Special:Version, which now will automatically put all extensions of unknown type in the “other” category, and will display this category as the last one. Plus misc minor improvements to a verity of extensions. This all amounts into a little over 550 commits to the MediaWiki SVN repository during the GSoC coding period.

State of the code

The Distribution extension has the infrastructure for storing and providing extension and core data via the MediaWiki API basically ready for use. It adds 4 database tables to MediaWiki:

• distribution_units: This table stores non-version specific info of ‘release units’ . Currently these unit’s are extensions only – the reason I went for a more general name is to allow for adding other things such as skins and content packages later on. The info here consist of a name, a URL, a description and a pointer to the “current version”.
• distribution_unit_versions: Entries in this table contain info about a specific version of a ‘release unit’. The info here consists of a version number, release status (beta, rc, stable, deprecated, …), release data, authors, description and some installation data.
• distribution_mwreleases: This table contains MediaWiki releases. It has been merged in from MWReleases, so all credit for it goes to Chad.
• distribution_packages: This table is not in use yet, and needs some more work. The goal is to be able to install a “package” onto your wiki which can contain multiple ‘distribution units’. This would basically be the same as Semantic Bundle is doing now, but a lot easier to set up and maintain.

The API modules added are:

• ApiQueryExtensions: Returns a list of extensions matching certain search criteria, which can include keywords, tags and authors. Only extensions with a version that has a release state acceptable for your installation are returned.
• ApiMWReleases: Gets the current MediaWiki releases. Like this distribution_mwreleases database table, this has been merged in from MWReleases and all credits go to Chad.
• ApiUpdates: This API module returns update information for the extensions you give it, and does the same for MediaWiki itself if a core version number is provided. The only info that’s returned is a version number for each unit or core, if there is an update. Otherwise nothing will be returned for that unit or core.

To populate the database with existing extension info I wrote a maintenance script “getSvnMetadata”, which goes through a local checkout of the MediaWiki extensions directory and get’s the names from the extensions. I haven’t found a good way yet to also get other extension data though.

The Deployment extension contains an abstraction layer for repository interaction and several interfaces that use this. The abstraction layer allows for supporting different kinds of repositories. The only implementation it currently has is for interaction with repositories provided by the Distribution extension. It’s also a convenient point to implement caching, as you probably don’t want to send the requests for available updates every time you view a page on the admin panel, and allows for changes to the format the repository uses without any effects in other parts of Deployment. The interfaces that are finished to some extend are:

• Special:Extensions: This page lists all installed extensions and allows you to filter on extension type. It’s based on the WordPress “plugins” page and is currently only an improved version of the extension list in Special:Version. It’s the only special page added by Deployment that can be viewed by non administrators. When logged in however, every extension has a list of links allowing you do various actions. The extension info is handled by a new class ExtensionInfo, which parses the info of individual extensions in $wgExtensionCredits, and provides a more convenient way to work with it. This allows for adding support for a new, better, extension info format later on. A planned feature for this special page is showing update notifications in each extension row.

• Special:Install: This page allows you to search through available extensions in the repository. The interface is based on the “plugin-install” page of WordPress and allows for searching extensions based on term, tag or author. After performing a search you get a list of matching extensions showing their name, version, authors, description, link to the documentation, and a link to download them. Later on this download link will be replaced by an “Install” one.

• Special:Update: This page will inform you of any updates to both core and extensions. It’s behaves basically identical the WordPress “update” page.

The extension allows you to configure several aspects of the repository interaction:

• $wgRepositoryApiLocation: This might be an obvious one, but also a very important one, as it allows you to use a repository other the the Wikimedia Foundation one on mediawiki.org, which will be the default.
• $wgRepositoryLocation: This is similar to the previous setting, but links to a web interface providing browsing capabilities through the repository, or at least some additional info.
• $wgRepositoryPackageStates: This is a list of allow release states. By default these will only be “stable” and “beta”. Early adopters can also add “dev” and “alpha”, and there also is “rc” and “deprecated”.

What’s next?

Although some very basic functionality is working, quite some work still needs to be done to get this to the WordPress-awesomeness level. Let’s first have a look at Distribution and then Deployment:

The most basic issue with Distribution currently is that there is no script yet that allows collecting all current extension data, which is needed for it to be of any use. I’m not sure how gathering current data can be properly automated, which is the main reason the script doesn’t exist yet. Any suggestions here are very welcome! After the initial version it should be possible for extension authors to edit their extensions data, and create new releases. For this we’ll need some new special pages. The data itself can then be used to populate the extension pages on mediawiki.org, and some new magic words such as “current MediaWiki version”, can automate a bunch of stuff. After these things new features can be added, such as the management of packages, and more detailed extension information, including things such as dependencies and compatibility info.

Deployment mainly needs interface work, and will need additions to support any new information provided by the Distirbution repository. A cool feature that could be added is supplying the repository with installation information (obviously optionally), which would allow the developers to get an idea of which versions of MediaWiki core and extensions people are using. After the whole MediaWiki deployment model has been revised and is up and running, it’s configuration can similarly be reinvented. The interfaces added by Deployment can then be adapted to allow contain extension configuration.

GSoC 2011?

This was my last GSoC as a student, as I no longer qualify, since I quitted my official studies. If I’m still doing MediaWiki development next year, which I guess is pretty likely, there is a lot of change I’ll be signing up as a mentor though If you are interested in being a student in 2011, you can already put your name on the 2011 GSoC page

Some useful links

• My blog posts about this project
• The Wikimedia 2010 GSoC page with all GSoC projects
• Distribution and Deployment extensions
• My MediaWiki.org user page

After doing a pile of research on how to best create an extension management platform for MediaWiki as my Google Summer of Code project, I realized that a lot of the work that I wanted to do was already done in some form or another, and decided to somewhat augment my goals. Since I can build upon Configure, and the Deployment Framework of Ontoprise, it should now be possible to also take care of the MediaWiki and extension configuration that I put as optional in my original proposal. To have some transparency here, and not to cause misconceptions, I created an awesome new roadmap.

I did an attempt to get some feedback by posting my roadmap on wikitech-l, but apparently everyone is either happy with it, or more likely, Parkinson’s Law of Triviality is in play here. So feedback and suggestions are definitely welcome, you can post them on the discussion page.

As Configure is doing an awesome job already without making significant changes to any existing code, I decided to start off with the Deployment Framework. I’m currently in the process of figuring out how it works exactly, so I’m able to extend it’s features, and build a GUI in the form of several MediaWiki special pages on top of it. This will have some immediate pay-off as Ontoprise will be able to use this improved version directly.

Both my presentations for Wikimania 2010 have been accepted, of which one is about my project, which will give me a change to explain to people what I’m doing and why it’s so awesome

Like last year, I’ve been accepted for GSoC 2010 – yay!

The Google Summer of Code (GSoC) is an annual program, first held from May to August 2005, in which Google awards stipends to hundreds of students who successfully complete a requested free software / open-source coding project during the summer.

My project is creating an awesome extension management platform for MediaWiki, facilitating the installation, updating, removal and configuration of extensions. I’ll get mentored by Brion Vibber, which is probably the best known MediaWiki developer there is. The underneath paragraphs are out-takes of my actual proposal.

Project summary

The goal of this project is to create an administration panel from where wiki administrators can update, install and remove extensions. A second goal would be to allow management of the installed extensions.

A panel where wiki administrators can install, update and remove extensions would have huge benefits. First of all, people would not have to manually download an extension and put an includes in LocalSettings, neither would they need to worry about compatibility and dependencies. Hitting an update button also takes considerably less time then doing the whole download routine again, and will cause people to run more up to date extensions. Another important advantage is that people will get extensions recommended, and can easily browse them. This way people will find extensions that do something they wanted but did not know about, and in general have extensions that better suit their needs. A third advantage is that extension developers won’t need to do extreme efforts to let people know there is a new version (and probably still only reach part of the relevant public). This is inspired on the way Word-press does things.

The second goal of this project is to add setting management for individual extensions. Currently extension settings are managed via LocalSettings. The aim here is to completely remove the need of editing any file directly by storing the configuration the the MediaWiki database, and creating a GUI to modify these settings. This would involve creating API modules so extensions can add and update their own settings. Work on this will only be started after the first goal is completed, and is seen as an “if time permits to-do”.

A third, also optional, goal would be to create a management interface for the wiki’s configuration itself. This is very similar to the second goal, and should be kept in mind while creating the management for extension settings. I do not expect to complete this to-do during GSoC, but want to provide the foundations for it, so this can be completed after the project itself is finished.

Deliverables

Note: the beneath list is a guideline only.

Things the administration panel should be capable of:

Required deliverables

• Automatic checking for extension updates (and updates to MW itself)

• Download and install option for extension updates

• Download and install option for extensions that have not yet been installed

• Removal of installed extensions

• Dependency and compatibility checking (both for installation, removal and updating)

• Extension browsing
  • Recommendation of similar extensions
  • Showing a list of most used and most recommended extensions
  • Finding extensions by category or keywords

• Enabling and disabling of installed extensions (so without removing them)

If time permits

• Management of settings specific to each extension.

• Management of MediaWiki configuration.

Project schedule

I’m for a loose schedule, since I believe this is the most efficient. I have no doubt that the to-do list will change a lot during the project, items will be changed, moved, multiple new ones will be added, and some might be removed. A fixes schedule would take away flexibility and stand in the way of efficiency. One of the main reasons to have a schedule is to ensure the student does not take the project to lightly, end ends up making insufficient progress. I like to believe I have clearly demonstrated that I will put considerable effort in such projects, even without any schedule, during last yeas GSoC, and with all the commits I’ve made since then.

This list contains some loose planning without any dates:

• Discuss the best way to structure the platform with mentor, and other relevant people.

• Investigate how similar functionality works with other software, and which aspects of these implementations can be used.

• Get the requited knowledge of MW to be able to create the platform in an efficient and modular fashion.

• Create the panel with the most basic features, then release and document it.

• Add the other features, possibly spread over several releases.

• (Start on the extension setting management functionality.)

• (Start on the MediaWiki configuration management functionality.)

• (Add other awesome things to the platform.)

• Take over the worlds and make everyone use this platform.

My current planning

I have to finish up the work I’m currently doing for the Wikimedia Foundation before I can fully start on this, and also like to release Maps and Semantic Maps 0.6 before then. This will probably be in two to tree weeks from now.

Also see the Wikimedia tech blog post which links to the other accepted projects.

Yay! I finally got my Google Summer of Code T-shirt!

With my T-shirt, I also got my GSoC 2009 certificate, which marks the end of any involvement with GSoC for me (see original post). That is, until next year. When I reflect back to what I learned and achieved during GSoC, I’m more then happy with it. I’m planning to do another (awesome) Wikimedia Foundation project next year, if I manage to get accepted again. This (awesome) project will be more focused on changes to MW’s core, and so provide me with an (awesome) opportunity to learn more about the inner working of MW, which I’m currently still unfamiliar with. I’m not going to make my exact plans public yet, cause it’s possible the things I want to improve will already be taken care off, and I don’t want everyone to go rip-off my idea I’ve already put my name on the (awesome) participants list for MW, and was FIRST (awesome!) since I created the (awesome) page.

I’ve been accepted for GSoC 2009 – yay!

This will enable me to work over 2 months on open source while getting payed by Google. For more info about GSoC, check out the official GSoC site.

I’ll be working on in my opinion one of the most exciting open source projects out there: the MediaWiki engine. More precice, I’ll work on the Semantic Maps extension, which provides the ability to view and edit coordinate data stored through the Semantic MediaWiki extension, using Google Maps, Open Layers, and variouse others. Yaron Koren, main author of the Semantic Google Maps extension, and one of the most important contributors to the SMW community, will mentor me.

The coding period starts on the 23th of may, but I’m already doing some experimenting with the relevant code. I’m also doing some effort to get some of my other projects finished by then, to not have to put them on hold for over 2 months.