Jeroens blog

I’ve been neglecting my blog in favour of microblogging status updates and release notes, and have not written anything here for over a month and a half. Therefore I’m going to provide an overview of all the awesome (now also on StatusNet!) new things that have come out since then, along with other interesting news from the Semantic MediaWiki world.

Semantic MediaWiki 1.5.4

Yes, 1.5.4. Since my previous blog post, I have made 2 minor updates to the SMW 1.5.x branch: SMW 1.5.3 on November 4th and SMW 1.5.4 on December 1st.  The focus of these releases is mainly on fixing bugs, increasing security and compatibility with MediaWiki 1.17.  The only significant new feature are explicit strict and non-strict greater-then and less-then comparators, which Saruman paid me to add.

The Semantic MediaWiki documentation wiki also got a facelift, including a new logo and redesigned Style. Most of the work here was done by Markus Krötzsch and Yaron Koren. See Yarons blog post for more info on the new look.

Maps and Semantic Maps 0.7.3

After releasing Maps and Semantic Maps 0.7.1, I continued refining the new images-as-layers functionality that this release introduced. As I had accidentally broken compatibility with MediaWiki 1.15.x without noticing, I ended up doing a pretty quick 0.7.2 release, about a week later. In late November I got contracted to add KML export functionality to Semantic Maps, which I implemented in the form of a new ‘kml’ result format, and then released in version 0.7.3. Some additional options for the KML format are available when using the code which is on svn trunk, but that’s not enough of a change to justify a new release just yet.

Semantic Bundle

Several important releases of other SMW related extensions that are in the Semantic Bundle have also been made, most notable to Semantic Forms, which has progressed no less then 5 minor releases since my last post. Each of these was followed by a Semantic Bundle release, which also include updates to Semantic Internal Objects and ApprovedRevs. Several compatibility improvements have been made to Semantic Result Formats, but this extension hasn’t brought a new version yet.

Push

The last few days I’ve been working on an awesome new extension that’s meant to facilitate pushing page content from one wiki to one or more other wikis, called ‘Push’. It makes heavy use of the MediaWiki API, and takes advantage of the MW 1.17 Resource Loader when available (but is compatible with MW 1.16). You can use it on-page by clicking the ‘push’ tab, which will get you a list of targets to push to (specified in LocalSettings).

It also supports bulk operations via Special:Push, which allows you to select pages in a similar fashion to Special:Export, and then, after submitting, pushes the pages one by one in a nice ajax-y way.

I created this extension as WikiWorks consultant for Texas Instruments.

What’s next?

MediaWiki 1.17 has just branched, so hopefully it won’t take to long for it to get released.

Maps 0.7.1 and Semantic Maps 0.7.1 are now available for download. The main new feature in this release is the long awaited images as layers. It allows you to display images with OpenLayers, so users can pan and zoom around, and markers can be placed to draw attention to certain area’s and provide additional information. Check out the examples. Some issues have also been fixed, in both extensions, mainly OpenLayers related.

What’s next?

Improving the pages in the layer namespace and providing a way to group markers in different overlays in OpenLayers. For a complete overview, see the roadmap. Feel free to propose new features, or help out creating them.

Also check out the wiki version of this post.

Version 0.7 of both the Maps and Semantic Maps extensions is now available for download. This release is made after 3 beta’s and a release candidate, so should be stable.

The most notable new features in this release are tag extension support for all the Maps parser functions, more consistent error handling via Validator 0.4 and compatibility with the upcoming MediaWiki 1.17. Most changes are internal improvements, but there are also several bug fixes and minor new features.

Whats next?

There only big new feature on the roadmap for 0.7.x is the infamous images-as-layers one. This feature was supported by the now long obsolete Semantic Layers extension (example), and has been requested by dozens of people since the first releases of Maps. Next to fixing the bugs that show up, the focus will mainly be on adding new functionality. Feel free to propose new features, or help out creating them.

An early beta of the next big release of the Maps and Semantic Maps extensions is now available for download. The most notable new features in this release are tag extension support for all the Maps parser functions, more consistent error handling via Validator 0.4 and compatibility with the upcoming MediaWiki 1.17. Most changes are internal improvements, but there are also several bug fixes and minor new features.

Both extensions need some more polishing work before the 0.7 release, some testing needs to be done, and some small new features might still be added. This is a list of changes so far:

Maps

New features

Tag support for these parser hooks (which previously only had parser function support):

• Coordinates
• Distance
• Finddestination
• Geocode
• Geodistance

Bug fixes

• Fixed compatibility with the MW 1.17 resource loader.
• Fixed i18n issue with the overlays control for Google Maps v2 maps.
• Fixed default zoom level for Yahoo! Maps maps.
• Increased the maximum decimals for DMS coordinates from 2 to 20.

Removed features

• #geocodelong and #geocodelat parser functions – you can obtain their functionality using #geocode.

Internal improvements

• Rewrote the geocoding functionality. It’s now an integral part of the extension that can not be just pulled out, while the reverse is true for individual geocoders. Geocoder interaction now uses the same model as mapping service interaction.
• Use of Validator 0.4, allowing for more robust and consistent error reporting.
• Rewrote the parser hooks to use the ParserHook class provided by Validator.
• Restructured the directory structure of the extension to better match it’s architecture.

Semantic Maps

New features

• Added ‘locations’ parameter to the map query printers that allows for displaying static locations in addition to query results in queries. It behaves the same as the locations parameter in display_points.

Semantic Maps displaying the result of a query on an OpenLayers map together with a static point.

Bug fixes

• Fixed compatibility with the MW 1.17 resource loader.

Internal improvements

• Use of Validator 0.4, allowing for more robust and consistent error reporting.
• Restructured the directory structure of the extension to better match it’s architecture.

Notice

Although the big features should work, this release will probably contain multiple issues. Please report any you might find.

For the most current version of this announcement, see the original on the mapping wiki.

Today I finished work on an initial version of a script I created to be able to update the Belgian Hackerspace wiki’s from my development environment. It took me quite a while to create this, as it’s my first bash script, and I had to figure out all the basic syntax stuff. Fixing up bad configuration and differences in set up also took up quite some time. Now I update the codebase of all wiki’s on the server weekly by running a single command, which is totally awesome Screenshot of the script running on my laptop:

As you can see, it allows updating a single wiki, or all of them, and the update can consist of everything, MediaWiki core, all extensions or just a single extension.

A week ago I finally created the project form, template and category on the 0×20 wiki, leaving only recurring event support on the immediate wishlist. When that has been taken care off, the semantic datastructures can be copied to a new wiki on hackerspaces.be, which will serve as a general Belgian wiki. Here the datastructures can be refined further and copied to somewhere before actual contents is put in (which allows other wiki’s to be created with it, avoiding a lot of work). This is probably a good chance to get the WikiSpaces project rolling again. After the datastructures have been copied, the wiki can be used to put information that is not specific to a single Belgian hackerspace, such as Pamela, the Mate supply, hacker events, ect. It’d also be the logical place to pull events from the other wiki’s into.

Just under a month after the 0.6.5 release of both mapping extensions, the next minor update, 0.6.6, is available for download. No spectacular new features, but several important bugfixes. Several issues with coordinate parsing have been fixed, you can now using geocoding when behind a proxy, and wikitext should finally(!) behave correctly in marker pop-ups. Some internal changes have also been made, mainly rounding off the many changes I made in the 0.6.x branch. I expect this release to be the most stable one to date, and have therefore changed the extensions status from ‘beta’ to ‘stable’ on the documentation pages.

A lot of improvements have been made to the documentation as well. Both the Maps examples and Semantic Maps examples are now comprehensive and complete. There now are finally examples of using query templates, of distance queries and of some nice compound queries. Some more work is needed though, a lot of which is explaining basic functionality and fixing minor issues all over the place. I’ll be taking care of the most important things, but I’d be great if people using the extensions could help me out improving the documentation

This release is probably the last one before 0.7, in which I expect to be focusing on new functionality. I’m looking for people that want to fund the development of new features, so please contact me if you are such a person

Downloads:

• Maps 0.6.6 zip archive
• Maps 0.6.6 7z archive
• Maps and Semantic Maps 0.6.6 zip archive
• Maps and Semantic Maps 0.6.6 7z archive

As Google Summer of Code (GSoC) 2010 has ended, I’m writing this blog post to outline what I’ve done during the coding period and what the results are. Thanks go to the Wikimedia Foundation and Google for providing the opportunity to do this project, Brion Vibber, who mentored me, and to all other people who helped me out, especially Yaron Koren who I bugged the most

What I did during GSoC

My initial proposal was to create an awesome extension management platform for MediaWiki that would allow for functionality similar to what you have in the WordPress admin panel. After doing some research I realized this would require significant effort in two areas: configuration and deployment. After looking at some already existing tools such as the Configure extension and the Deployment Framework of Ontoprise, I decided to completely drop the configuration part and concentrate on the deployment work.

I started with porting the filesystem abstraction classes from WordPress, which are needed for doing any upgrade or installation operations that include changes to the codebase. (The current MediaWiki installer can do upgrades, but only to the database.) I created a new extension called Deployment, where I put in this code, and which was intended as a place to experiment with all the MediaWiki-installation side deployment stuff. As You obviously want this functionality to be part of MediaWiki itself, I wrote it with the idea of moving over the code to MediaWiki core once it was finished. It turned out that doing filesystem upgrades securely is not an easy task though, so after finishing the port, I quitted work on this as I decided to do other functionality first.

I then poked somewhat at the new MediaWiki installer, which is a complete rewrite of the current installer with a lot of new cool stuff and a totally more awesome interface. I made some minor imrpovements there, and split the Installer class, which held core installer functionality, into a more generic Installer class and a CoreInstaller. This allows for creating an ExtensionInstaller that uses the same base code, such as database, filesystem and LocalsSttings manipulation.

After this I started thinking about how to best structure a package repository for MediaWiki and extensions to get updates and new extensions from. I had a look at PEAR and CPAN, as well as WordPress, although I don’t learn a lot about the later. Apparently their repository code is not freely available After discussion with Brion I decided to just create the repository from scratch, and started working on another extension, titled Distribution, for this purpose. I merged it together with a rewritten version of the MWReleases extension written by Chad, which already had core update detection functionality.

After the Distribution API’s where working decently I started work on the Special pages in Distribution that would serve as the equivalent of the WordPress admin panel. As I put of the configuration work, and also the file-system manipulation for the initial version, this came down to simply listing currently installed software, update detection and browsing through extensions available in the repository.

On top of my GSoC project itself, I did quite some other MediaWiki work in “my free time”. I released 5 new versions of Maps and Semantic Maps, starting with 0.6 and ending at 0.6.5. As I finally got core commit access, I also poked at some other things, such as Special:Version, which now will automatically put all extensions of unknown type in the “other” category, and will display this category as the last one. Plus misc minor improvements to a verity of extensions. This all amounts into a little over 550 commits to the MediaWiki SVN repository during the GSoC coding period.

State of the code

The Distribution extension has the infrastructure for storing and providing extension and core data via the MediaWiki API basically ready for use. It adds 4 database tables to MediaWiki:

• distribution_units: This table stores non-version specific info of ‘release units’ . Currently these unit’s are extensions only – the reason I went for a more general name is to allow for adding other things such as skins and content packages later on. The info here consist of a name, a URL, a description and a pointer to the “current version”.
• distribution_unit_versions: Entries in this table contain info about a specific version of a ‘release unit’. The info here consists of a version number, release status (beta, rc, stable, deprecated, …), release data, authors, description and some installation data.
• distribution_mwreleases: This table contains MediaWiki releases. It has been merged in from MWReleases, so all credit for it goes to Chad.
• distribution_packages: This table is not in use yet, and needs some more work. The goal is to be able to install a “package” onto your wiki which can contain multiple ‘distribution units’. This would basically be the same as Semantic Bundle is doing now, but a lot easier to set up and maintain.

The API modules added are:

• ApiQueryExtensions: Returns a list of extensions matching certain search criteria, which can include keywords, tags and authors. Only extensions with a version that has a release state acceptable for your installation are returned.
• ApiMWReleases: Gets the current MediaWiki releases. Like this distribution_mwreleases database table, this has been merged in from MWReleases and all credits go to Chad.
• ApiUpdates: This API module returns update information for the extensions you give it, and does the same for MediaWiki itself if a core version number is provided. The only info that’s returned is a version number for each unit or core, if there is an update. Otherwise nothing will be returned for that unit or core.

To populate the database with existing extension info I wrote a maintenance script “getSvnMetadata”, which goes through a local checkout of the MediaWiki extensions directory and get’s the names from the extensions. I haven’t found a good way yet to also get other extension data though.

The Deployment extension contains an abstraction layer for repository interaction and several interfaces that use this. The abstraction layer allows for supporting different kinds of repositories. The only implementation it currently has is for interaction with repositories provided by the Distribution extension.  It’s also a convenient point to implement caching, as you probably don’t want to send the requests for available updates every time you view a page on the admin panel, and allows for changes to the format the repository uses without any effects in other parts of Deployment. The interfaces that are finished to some extend are:

• Special:Extensions: This page lists all installed extensions and allows you to filter on extension type. It’s based on the WordPress “plugins” page and is currently only an improved version of the extension list in Special:Version. It’s the only special page added by Deployment that can be viewed by non administrators. When logged in however, every extension has a list of links allowing you do various actions. The extension info is handled by a new class ExtensionInfo, which parses the info of individual extensions in $wgExtensionCredits, and provides a more convenient way to work with it. This allows for adding support for a new, better, extension info format later on. A planned feature for this special page is showing update notifications in each extension row.

• Special:Install: This page allows you to search through available extensions in the repository. The interface is based on the “plugin-install” page of WordPress and allows for searching extensions based on term, tag or author. After performing a search you get a list of matching extensions showing their name, version, authors, description, link to the documentation, and a link to download them. Later on this download link will be replaced by an “Install” one.

• Special:Update: This page will inform you of any updates to both core and extensions. It’s behaves basically identical the WordPress “update” page.

The extension allows you to configure several aspects of the repository interaction:

• $wgRepositoryApiLocation: This might be an obvious one, but also a very important one, as it allows you to use a repository other the the Wikimedia Foundation one on mediawiki.org, which will be the default.
• $wgRepositoryLocation: This is similar to the previous setting, but links to a web interface providing browsing capabilities through the repository, or at least some additional info.
• $wgRepositoryPackageStates: This is a list of allow release states. By default these will only be “stable” and “beta”. Early adopters can also add “dev” and “alpha”, and there also is “rc” and “deprecated”.

What’s next?

Although some very basic functionality is working, quite some work still needs to be done to get this to the WordPress-awesomeness level. Let’s first have a look at Distribution and then Deployment:

The most basic issue with Distribution currently is that there is no script yet that allows collecting all current extension data, which is needed for it to be of any use. I’m not sure how gathering current data can be properly automated, which is the main reason the script doesn’t exist yet. Any suggestions here are very welcome! After the initial version it should be possible for extension authors to edit their extensions data, and create new releases. For this we’ll need some new special pages. The data itself can then be used to populate the extension pages on mediawiki.org, and some new magic words such as “current MediaWiki version”, can automate a bunch of stuff. After these things new features can be added, such as the management of packages, and more detailed extension information, including things such as dependencies and compatibility info.

Deployment mainly needs interface work, and will need additions to support any new information provided by the Distirbution repository. A cool feature that could be added is supplying the repository with installation information (obviously optionally), which would allow the developers to get an idea of which versions of MediaWiki core and extensions people are using. After the whole MediaWiki deployment model has been revised and is up and running, it’s configuration can similarly be reinvented. The interfaces added by Deployment can then be adapted to allow contain extension configuration.

GSoC 2011?

This was my last GSoC as a student, as I no longer qualify, since I quitted my official studies. If I’m still doing MediaWiki development next year, which I guess is pretty likely, there is a lot of change I’ll be signing up as a mentor though If you are interested in being a student in 2011, you can already put your name on the 2011 GSoC page

Some useful links

• My blog posts about this project
• The Wikimedia 2010 GSoC page with all GSoC projects
• Distribution and Deployment extensions
• My MediaWiki.org user page

Just a few quick screenshots of Special:Extensions, on which I’ve been working today. The first screenshot shows Special:Extensions page displaying a list of all the extensions I have installed on my local wiki:

As you can see, you can now filter on extension type with the control right below the “Installed extensions” title. Here I have filtered on the SEMANTIC extensions:

An interesting change I made is that you can now access this page without having the siteadmin permission. Doing this will get the above, but without the add new button and administration controls (currently only “Deactivate” which is there only for show so far). This way this page will be a nice addition to Special:Version.

Tomorrow is the last coding day in Google Summer of Code 2010, during which I’m planning to focus on the update detection functionality, or rather creating the interface for it, as the plumbing for it is all but done. I also want to move several classes from Deployment over to MediaWiki core, as they make more sense to have there, and would allow for some nice improvements.

As I wrote a post when I reached 10k credits for the BOINC projects I’m following, and another one for the 20k threshold, here is a third one, as I now passed 30k

SETI@Home is far behind with only 16k.

Since my last blog post about my GSoC project, which aims to bring more awesome deployment capabilities to MediaWiki, I’ve been putting my time into both the Distribution and Deployment extensions. I was pleased to find a bunch of stuff was easier to do then I had imagined, and now I finally have some functionality you can actually see working – yay It’s not a lot, as this is just very rudimentary and even uses demo data at places. Still it’s very nice to be able to post some screenshots after months of doing research and poking at code. I also got a new crappy diagram (although I think I’m succeeding in getting them less crappy each iteration) that shows the architecture of the initial versions I’m working towards.

The following screenshots show an import script running. This script is part of Distribution and is meant to get data from a checked out copy of the extensions directory and store it into the database tables provided by Distribution. I haven’t found a good way to actually get the extension data other then their path names (for example ‘SemanticMediaWiki’), so have a function that’s just returning some demo data.

Via Deployment you can search for extensions by keyword, author or tag. This is done by making a request to the API provided by Distribution which serves the data collected by the script. This screenshot shows the interface I have created so far:

And after clicking the button, you get:

As I’m doing with all special pages in Deployment, Special:Install’s layout and functionality is based on what WordPress has in it’s admin panel.

I also put some work in Special:Extensions already, although it’s basically just Special:Version with just the extensions for now.

I’m not wring a real comprehensive overview of what features there are already, which are planned, and how I plan to create them for now, as there is little time left in GSoC (only 3 days!), and I want to get some more work finished before then