Jeroens blog

It’s been a while since I last wrote a blog post, and this is in part due to the last extension I wrote sadly enough not being available under an open license due to client demands :/ This blog post is about what I’m currently working on, new infrastructure for a project that I hope you’ll agree upon is totally awesome.

Open knowledge and open learning tools go hand in hand, thus Wikipedia can be an amazing resource for education further then simply looking up things. The Wikipedia Education Program strives to involve university students with the Wikipedia community, by creating and editing articles as part of their coursework. This is a real win-win for everyone involved. The students get to do something which is actually useful, as their work won’t disappear into some dusty archive of their university, and they learn working collaboratively with other people in the Wikipedia community. The gains for Wikipedia, and it’s community, are probably pretty obvious; more contributors, more content, more people to help increase the quality of the encyclopaedia as a whole.

If you are interested in bringing this program to your university, definitely check out the Wikipedia Education Program (WEP). Also, you can find more comprehensive posts about the program itself, as well as many success stories, on the Wikimedia blog.

The WEP started several years ago with a handful of universities participating. Such participation and enrolment was kept track of manually on wiki pages. Obviously, this does not scale well, and now that the program has two or three orders of magnitude more students, new tools are urgently needed. The main goal is having a way for students to enrol themselves, and then be tracked by their mentors and people of the community wanting to provide assistance where needed. Further a lot of nice things can be done, such as keeping track of contributions, drop-out rates, ect, so that problems can be spotted, and measures can be taken to avoid these in the future. You can find an initial draft of the requirements here.

I’m implementing this as a new MediaWiki extension, which will when completed, be placed onto Wikipedia. The extension is, probably unsurprisingly, called Education Program. It’s currently in early alpha stage, so not much to see there yet. However, a draft of it’s functionality can be found here. Management wise it’ll be somewhat similar to Contest, but more extensive (since there is more stuff to manage), and hopefully improved along multiple dimensions. I still need to put more thoughts into the exact flow for students though, and discuss this with the other people involved.

This is a screenshot of one of the many special pages making up the management interface. It lists all courses, allows you to browse through them (paged) and filter on criteria. Program administrators and mentors also get to see a control to add new courses, which then takes them to a new page with a form. Very similar pages exist for other types of objects, such as Institutions, Terms, Mentors and Students (although the later two are somewhat different, since they cannot be modified by people other then the students or mentors themselves).

I will likely be working on this for another 2 months, after which I will start work on an even more awesome project (don’t get me wrong, the WEP is definitely awesome), on which I’ll post more later on.

As it’s been 2 months since my last blog post, I figured it was time for another one. Quite a few things I could write about (SMWCon, my new awesome laptop, Stanfords AI and ML classes, me moving to Berlin, …), but I decided to give some introduction to my most recent MediaWiki extension: Contest.

Contest extension that allows users to participate in admin defined contest challenges. Via a judging interface, judges can discuss and vote on submissions. I created it for the Wikimedia October coding challenge, so it got a nice amount of review, uncovering some minor misconceptions I had about some core MW code, and it got deployed on MediaWiki.org. The coding challenge is quite awesome, but I won’t discuss it any further here, so check out the linked blog post if you’re curious/interested.

Feature overview:

• Admin interface for managing contests and their challenges.
• Landing and signup pages for each contest.
• Personal contest list and submission interface for each user.
• Summary pages per contest listing contestants, which can be filtered and sorted.
• Judging interface that allows for rating and commenting on each participant.
• All contests, challenges, contestants, comments and votes can be queried and exported via the API.
• Signup and reminder emails.

Requirements

• MediaWiki 1.18 or above – yeah! my first awesome MediaWiki 1.18 or later extension
• PHP 5.2 or above
• MySQL

Some screenshots

Contest welcome interface

Contest submission and management interface for a single user

Contest administration and judging interface listing all contests

Contest administration and judging interface for a single contest

Contest editing interface

Judging interface for a single submission

Some background

When starting with this extension, it was clear pretty quickly that it could really use the awesome DBObject class I’ve been incrementally creating over my last few extensions and mostly finished in Survey. This class is a wrapper for objects of a certain type, which is equivalent to a row in some db table. Even though it’s in essence very simple – it just has a field that is an associative array with field => value – it’s also very powerful and flexible. When I started with this, I had no idea it would turn out to be so neat. The bad news was that I could not use PHP 5.3 or later for Contest, while the DBObject class uses late static binding, which was introduces in PHP 5.3. I came up with a simple hack: all static methods in the base class have been made-non static, but are marked as “should be static”. Then, every deriving class has a public static function s(), which returns a (cached) instance of the class. So then for every “static” method you need to call, instead of ClassName::methodName(), you do ClassName::s()->methodName(). If you know this, and do not misuse the non-static-but-should-be-static methods in the base class, it retains all it’s niceness, at the cost of something that’s pretty much a tiny bit of syntactic sugar. And it’s quite obvious how easy it will be to replace this with actual LSB usage once this becomes possible

• Survey DBObject class – deriving class
• Contest DBObject class – deriving class

Download

• Release tarballs
• SVN tags
• SVN trunk

What’s next?

There are various small additions that could be made, but one things really stands out: contest configuration with version history. Right now, you can create contests and challenges, modify them and delete them. But once you make a change, the previous version is lost. You cannot revert. You cannot compare. You can’t even see who made the change or when. Implementing such a thing is not trivial, especially if you want to have a generic system that can be used by any extension that wants to store data and have version history for it. And if you think about it, quite a few extensions could use this. Let’s have a look at my extensions, latest first, and see if they can use it:

• Contest – well yeah, obviously
• Survey – yes, for the surveys, their configuration, and their questions
• Semantic Watchlist – yeah, for the watchlist groups
• IncludeWP – no
• SubPageList – no
• Live Translate – yeah, for the translation memory admin
• Push – no, although it could, if it had an admin interface for the push targets
• Validator – no
• Maps and SM – yeah, for the map layer definitions currently done as key=value pairs, one per line, in the Layer namespace

That’s 6 out of 10 just for the extensions I wrote. What it comes down to is that pretty much any extension that has some sort of settings interface that is not user-specific, could use this. And maybe even the user specific ones, which would obviously include the user preferences in MediaWiki core. So why not store this data in wiki pages, such as done by Maps for layers? You could even store it as JSON or serialized PHP objects if you need more complexity… The things with this is that it only works for simple use cases (such as the layers in Maps), and even then is limited. You cannot query over the data as you do not have it in relational form. And you cannot have fine grained access and write rights control over the data, which in a lot of cases is quite important. So a generic solution here would be an awesome addition to MediaWiki if you ask me.

More info on Contest can be found on it’s documentation page.

Somewhere in the last two weeks I quickly wrote up a small new MediaWiki extension to include content from Wikipedia or some other MediaWiki install into pages on your wiki. It’s titled Include WP.

The extension does not import anything (so nothing is stored), but rather fetches content from the remote wiki every time the page is loaded. This means it takes a second or two for it to actually load, but ensures you got the latest content. Some clean up of the remote content happens and there are several options that allow you to customize how the included content is displayed. It makes use of the Validator extension.

Feature overview:

• Include articles from Wikipedia or any other MediaWiki wiki into your pages.
• Show only a limited amount of paragraphs on page load, with an option to show the full article.
• Partial conversion to plain-text.

1. Removal of templates (such as infoboxes), ref tags, comments, categories and images.
2. Both internal and external links are rendered as plain-text.
3. Tables, lists, table of content, section headers and more are retained.

• Usage of the MediaWiki Resource Loader when available with backward support for MW 1.16.x.

Some screenshots

Download

• Stable releases (archives)
• SVN tags
• SVN trunk

The last 2 days I’ve been fixing a whole bunch of issues in Maps and Semantic Maps that surfaced since 0.4.x. I’ve also taken some time to implement a new, and IMO pretty awesome, feature for Google Maps.

You can now specify overlays for any Google Maps map. These overlays will be held in a new ‘overlays’ control you can add to the map, by which the user can select and deselect the available overlays. ATM there are 4 supported overlays: Wikipedia, Webcams, Photos and Videos. A new setting in Maps allows you to set the default overlays that should be selectable on a map, and also which of them should be shown on page load. A new ‘overlays=’ parameter allows you to override this default behaviour. This example demonstrates the use of this parameter:

|overlays=wikipedia-1,webcams-1,photos,videos

The additional -1 for the first 2 overlays indicates they should be shown at pageload. The underneath image is a screenshot of a map generated with this parameter:

Using this feature together with the Google Earth plugin, which is also supported by Maps, results in awesome interaction with the map. This screenshot gives you a glimpse, but you should try it out for yourself to get all awesomeness.

As you can see, the overlays control will not be shown when viewing the Earth view. You can change the overlays by switching to another map type, changing them, and then changing the map type back to Earth.

Like the title of this blog post indicates, this feature is new in Maps 0.4.2. This version of Maps has not been released yet, but an RC is available for download (click here for Maps & Semantic Maps 0.4.2 rc1). This release will include some other changes, and a load of big bug fixes. I’ll post more about this after the release, which will probably be tomorrow, or Saturday.

Me and Yaron, my former Google Summer of Code mentor, agreed that Maps would be a good choice as mapping extension on the wikimedia wiki’s, like Wikipedia and Mediawiki.org. There are various reasons why we think Maps is the best choice, but I’m not going to list them here.

I’ve recently discussed this with Ævar, one of the front people in the effort of getting mapping onto wikimedia’s wiki’s. The current plan is to display maps provided by OpenStreetMaps via the OpenLayers API. This includes creating a map tile rendering infrastructure, since the OSM service would simply not be able to cope with all requests if maps are all over Wikipedia. Basically, the complete OSM database will be copied to Wikimedia servers, from where it’ll be processed. Later on data might be directly added to that, loose from OSM. This would allow to add things like bird migration patterns to maps, which is not, and mpost likely will not, be present on OSM. Another big difference with the current usage of maps is that the OL maps won’t be simply ‘shown’ to a user. A requirement to put the mapping onto Wikipedia and the other wiki’s is that it should be usable by people that do not have JavaScript enabled, or use a browser that does not support it (that can happen when you live in the time zone that’s still in the stone age apparently). Concretely this means a rendered static image will be displayed of the map, without any of the map controls. Users will not be able to zoom or pan around since it’s a plain static image (.jpg, .gif, .png, stuff like that). When you click on that image however, it’ll be replaced by the map, after which you will have all functionality that comes with OL available.

The biggest part of the mapping effort is going to the stuff that happens ‘behind the scenes’, meaning the rendering and messing around the data. ATM SlippyMap, a small OL-spesific mapping extension, is used to display the actual results. Ævar agreed that Maps probably would suite the needs as well. This list contains some of the most important changes that will need to be made to Maps to make it suitable for large-scale deployment:

• Maps should be able to handle images from the rendering infrastructure of Wikimedia Foundation.
• Maps should support static display functionality, meaning display a regular image on page load, and the dynamic map after it’s clicked.
• The performance of Maps should be optimized – or at least the part relevant to the display of OSM maps via OL.
• Maps should be able to display maps without any markers on them.

It would also be nice to have the ability to view high resolution images via OL, the so called image-as-layer feature. This was already on Map’s (and Semantic Map’s) to-do list, but has now gotten a higher priority, since it has pretty much been requested by Brion Vibber, the Wikimedia Foundation CTO (he’s totally awesome at giving awesome presentations about awesome stuff btw, which is of course awesomely awesome).

The underneath video is from Wikimania 2009, in Buenos Aires, showing Ævar talking about the rendering infrastructure and SlippyMap.

Play video

If you are interested in Wikimania, be sure to check out Yaron’s email report about it.

This is a great opportunity for me, but will require a lot of work before it’s possible. In any case, I’m looking foreward to adding the new functionality to maps, and see how the performance of both the PHP and JS can be increased.