Jeroens blog

As Google Summer of Code (GSoC) 2010 has ended, I’m writing this blog post to outline what I’ve done during the coding period and what the results are. Thanks go to the Wikimedia Foundation and Google for providing the opportunity to do this project, Brion Vibber, who mentored me, and to all other people who helped me out, especially Yaron Koren who I bugged the most

What I did during GSoC

My initial proposal was to create an awesome extension management platform for MediaWiki that would allow for functionality similar to what you have in the WordPress admin panel. After doing some research I realized this would require significant effort in two areas: configuration and deployment. After looking at some already existing tools such as the Configure extension and the Deployment Framework of Ontoprise, I decided to completely drop the configuration part and concentrate on the deployment work.

I started with porting the filesystem abstraction classes from WordPress, which are needed for doing any upgrade or installation operations that include changes to the codebase. (The current MediaWiki installer can do upgrades, but only to the database.) I created a new extension called Deployment, where I put in this code, and which was intended as a place to experiment with all the MediaWiki-installation side deployment stuff. As You obviously want this functionality to be part of MediaWiki itself, I wrote it with the idea of moving over the code to MediaWiki core once it was finished. It turned out that doing filesystem upgrades securely is not an easy task though, so after finishing the port, I quitted work on this as I decided to do other functionality first.

I then poked somewhat at the new MediaWiki installer, which is a complete rewrite of the current installer with a lot of new cool stuff and a totally more awesome interface. I made some minor imrpovements there, and split the Installer class, which held core installer functionality, into a more generic Installer class and a CoreInstaller. This allows for creating an ExtensionInstaller that uses the same base code, such as database, filesystem and LocalsSttings manipulation.

After this I started thinking about how to best structure a package repository for MediaWiki and extensions to get updates and new extensions from. I had a look at PEAR and CPAN, as well as WordPress, although I don’t learn a lot about the later. Apparently their repository code is not freely available After discussion with Brion I decided to just create the repository from scratch, and started working on another extension, titled Distribution, for this purpose. I merged it together with a rewritten version of the MWReleases extension written by Chad, which already had core update detection functionality.

After the Distribution API’s where working decently I started work on the Special pages in Distribution that would serve as the equivalent of the WordPress admin panel. As I put of the configuration work, and also the file-system manipulation for the initial version, this came down to simply listing currently installed software, update detection and browsing through extensions available in the repository.

On top of my GSoC project itself, I did quite some other MediaWiki work in “my free time”. I released 5 new versions of Maps and Semantic Maps, starting with 0.6 and ending at 0.6.5. As I finally got core commit access, I also poked at some other things, such as Special:Version, which now will automatically put all extensions of unknown type in the “other” category, and will display this category as the last one. Plus misc minor improvements to a verity of extensions. This all amounts into a little over 550 commits to the MediaWiki SVN repository during the GSoC coding period.

State of the code

The Distribution extension has the infrastructure for storing and providing extension and core data via the MediaWiki API basically ready for use. It adds 4 database tables to MediaWiki:

• distribution_units: This table stores non-version specific info of ‘release units’ . Currently these unit’s are extensions only – the reason I went for a more general name is to allow for adding other things such as skins and content packages later on. The info here consist of a name, a URL, a description and a pointer to the “current version”.
• distribution_unit_versions: Entries in this table contain info about a specific version of a ‘release unit’. The info here consists of a version number, release status (beta, rc, stable, deprecated, …), release data, authors, description and some installation data.
• distribution_mwreleases: This table contains MediaWiki releases. It has been merged in from MWReleases, so all credit for it goes to Chad.
• distribution_packages: This table is not in use yet, and needs some more work. The goal is to be able to install a “package” onto your wiki which can contain multiple ‘distribution units’. This would basically be the same as Semantic Bundle is doing now, but a lot easier to set up and maintain.

The API modules added are:

• ApiQueryExtensions: Returns a list of extensions matching certain search criteria, which can include keywords, tags and authors. Only extensions with a version that has a release state acceptable for your installation are returned.
• ApiMWReleases: Gets the current MediaWiki releases. Like this distribution_mwreleases database table, this has been merged in from MWReleases and all credits go to Chad.
• ApiUpdates: This API module returns update information for the extensions you give it, and does the same for MediaWiki itself if a core version number is provided. The only info that’s returned is a version number for each unit or core, if there is an update. Otherwise nothing will be returned for that unit or core.

To populate the database with existing extension info I wrote a maintenance script “getSvnMetadata”, which goes through a local checkout of the MediaWiki extensions directory and get’s the names from the extensions. I haven’t found a good way yet to also get other extension data though.

The Deployment extension contains an abstraction layer for repository interaction and several interfaces that use this. The abstraction layer allows for supporting different kinds of repositories. The only implementation it currently has is for interaction with repositories provided by the Distribution extension.  It’s also a convenient point to implement caching, as you probably don’t want to send the requests for available updates every time you view a page on the admin panel, and allows for changes to the format the repository uses without any effects in other parts of Deployment. The interfaces that are finished to some extend are:

• Special:Extensions: This page lists all installed extensions and allows you to filter on extension type. It’s based on the WordPress “plugins” page and is currently only an improved version of the extension list in Special:Version. It’s the only special page added by Deployment that can be viewed by non administrators. When logged in however, every extension has a list of links allowing you do various actions. The extension info is handled by a new class ExtensionInfo, which parses the info of individual extensions in $wgExtensionCredits, and provides a more convenient way to work with it. This allows for adding support for a new, better, extension info format later on. A planned feature for this special page is showing update notifications in each extension row.

• Special:Install: This page allows you to search through available extensions in the repository. The interface is based on the “plugin-install” page of WordPress and allows for searching extensions based on term, tag or author. After performing a search you get a list of matching extensions showing their name, version, authors, description, link to the documentation, and a link to download them. Later on this download link will be replaced by an “Install” one.

• Special:Update: This page will inform you of any updates to both core and extensions. It’s behaves basically identical the WordPress “update” page.

The extension allows you to configure several aspects of the repository interaction:

• $wgRepositoryApiLocation: This might be an obvious one, but also a very important one, as it allows you to use a repository other the the Wikimedia Foundation one on mediawiki.org, which will be the default.
• $wgRepositoryLocation: This is similar to the previous setting, but links to a web interface providing browsing capabilities through the repository, or at least some additional info.
• $wgRepositoryPackageStates: This is a list of allow release states. By default these will only be “stable” and “beta”. Early adopters can also add “dev” and “alpha”, and there also is “rc” and “deprecated”.

What’s next?

Although some very basic functionality is working, quite some work still needs to be done to get this to the WordPress-awesomeness level. Let’s first have a look at Distribution and then Deployment:

The most basic issue with Distribution currently is that there is no script yet that allows collecting all current extension data, which is needed for it to be of any use. I’m not sure how gathering current data can be properly automated, which is the main reason the script doesn’t exist yet. Any suggestions here are very welcome! After the initial version it should be possible for extension authors to edit their extensions data, and create new releases. For this we’ll need some new special pages. The data itself can then be used to populate the extension pages on mediawiki.org, and some new magic words such as “current MediaWiki version”, can automate a bunch of stuff. After these things new features can be added, such as the management of packages, and more detailed extension information, including things such as dependencies and compatibility info.

Deployment mainly needs interface work, and will need additions to support any new information provided by the Distirbution repository. A cool feature that could be added is supplying the repository with installation information (obviously optionally), which would allow the developers to get an idea of which versions of MediaWiki core and extensions people are using. After the whole MediaWiki deployment model has been revised and is up and running, it’s configuration can similarly be reinvented. The interfaces added by Deployment can then be adapted to allow contain extension configuration.

GSoC 2011?

This was my last GSoC as a student, as I no longer qualify, since I quitted my official studies. If I’m still doing MediaWiki development next year, which I guess is pretty likely, there is a lot of change I’ll be signing up as a mentor though If you are interested in being a student in 2011, you can already put your name on the 2011 GSoC page

Some useful links

• My blog posts about this project
• The Wikimedia 2010 GSoC page with all GSoC projects
• Distribution and Deployment extensions
• My MediaWiki.org user page

Maps and Semantic Maps 0.6.5 are now available for download. This release contains mainly internal changes to improve code modularity and fix some security concerns. Several bugs have been fixed as well, and a new hook has been added to Semantic Maps. This hook will get you the map format as default one for queries where you only ask for coordinates when using SMW 1.5.2 or above. For a full list of changes since 0.6.4 see changes to Maps and changes to SM. Everyone running 0.6.2 or older is advised to upgrade as soon as possible. 

This release is notable for it being the first one in which I’m happy with the code-base as a whole. It took me a year to get here, but now I think the way the mapping extensions work is good and solid. This means you can now extend Maps and not be afraid the code will be incompatible in a few weeks due to changes. This also means that I’ll be focusing more on actual functionality rather then refactoring in future releases. I’ll be progressively building a little guide that explains how the extensions work from a developers perspective and how to extend them.

I might release another minor update in the 0.6.x series if any significant issues are found in 0.6.5. Further plans are finishing up a bunch of changes I’ve started to make in Validator, which I’ll probably release as 0.4 then, and to start working on Maps and Semantic Maps 0.7, which would aim at adding new features and improving existing ones. A likely new feature I’m particularly looking forward to implementing is several tag extensions that do the equivalent of the current parser functions added by Maps. The timetable for all this depends a lot on which other things I get cough up in (I’ll probably continue putting effort into the deployment stuff for my GSoC project) and what kind of funding will be available.

Downloads:

• Maps 0.6.5 [zip - 7z]
• Maps and Semantic Maps 0.6.5 [zip - 7z]

You can also view the release announcement at the documentation wiki.

I got a new diagram!!!1!11!! It’s based on my previous one, but slightly more elaborate, and a lot less messy, as I now used Dia to create it

Legend:

• Striped lines: Existing components where code will be copied from, or based upon.
• Full lines: Components of the complete deployment model.
• Thick full lines: Core components (of the deployment model) that I definitely want to have completed during GSoC.

Since my previous post about my Google Summer of Code project I have been poking at the new MediaWiki installer to see what’s there already, how it is there, and how I can integrate it with the above deployment model. I’ve made a bunch of style and documentation improvements while going over the code, and renamed some things to make more sense. And I had Tim Starling clean up a bad svn commit I made

So what I’m doing now is splitting the current ‘Installer’ class, which is part of the new installer, into 2: Installer and CoreInstaller. Installer will hold general installer functionality and be part of the whole deployment model, while CoreInstaller will hold installer functionality specific to core, and will be part of the new installer. After that I can create an initial version of CoreInstallers counterpart: ExtensionInstaller.

Maps and Semantic Maps 0.6.3 are now available for download. This release is one aimed to improve stability and addresses a number of bugs that where present in 0.6.x. Some notable fixed issues are a failure of parsing any coordinates containing a degree symbol that surfaced in 0.6.2, and the failing of non-Google Maps form inputs. Some internal rewriting has also been done, which was needed to address some of these bugs, and will enable future awesomeness increase. The included OpenLayers library has also been upgraded to 2.9.1. (changes to Maps, changes to SM). People running any 0.6.x are advised to upgrade. Especially people using Semantic Maps in conjunction with Semantic Forms, as both 0.6 and 0.6.1 contain undesired behaviour in the forms.

A new release of Semantic Bundle, which will include these versions of the mapping extensions, is likely to soon be released as well

Downloads

• Maps 0.6.3 (7z)
• Maps and Semantic Maps 0.6.3 (7z)

See also

• The release notes on the documentation wiki

Maps and Semantic Maps 0.6.2 are now available for download.

This release contains improvements to parameter handling, case insensitivity has been re-added, and using default parameters together with named ones will work more intuitive. In Maps the handling of distances has also been considerably improved: there is a new #distance parser function, additional distance related parameters in #geodistance, and new settings that allow you to customize the default distance behaviour. In Semantic Maps a rather important bugfix has been made which solves a problem with editing articles with forms that contain maps that arose in 0.6.1. (changes to Maps, changes to SM). People running 0.6 and 0.6.1 are advised to upgrade. Especially people using Semantic Maps in conjunction with Semantic Forms, as both 0.6 and 0.6.1 contain undesired behaviour in the forms.

Read this announcement on the documentation wiki

Maps and Semantic Maps 0.6.1 are now available for download. This release does not add any new features to 0.6, but contains multiple important bugfixes (Maps, SM). People running 0.6 are advised to upgrade. Especially people using Semantic Maps 0.6 in conjunction with Semantic Forms, as the mapping form inputs in 0.6 will output invalid coordinates when editing existing ones or inserting new ones.

Maps and Semantic Maps 0.6 are now available for download. Maps 0.6 requires Validator 0.3, which is included in the release distribution, and can also be found on SVN. Semantic Maps 0.6 requires Maps 0.6, Validator 0.3 and Semantic MediaWiki 1.5.1 or above. See the download page for full dependency and compatibility tables.

This is a big update, including a lot of new features, bug fixes, security patches, and most of all, internal improvements, making both extensions more modular and extendible (these changes are not covered here, see the relevant change logs for more info). It is also the first release of Semantic Maps that requires you to run the SMW update script, as it requires a new table layout to store coordinates (more info on this).

Let’s have a look at the various new features.

• Maps now supports real coordinate parsing and formatting, which allows you to input coordinates in any of the supported notations at any point in the extension, and can also request any output to be in the notation of your choice. This means you can now choose what format Semantic MediaWiki shows coordinates in, such as in ask queries. The supported notations are DMS, decimal degrees, decimal minutes and floats. All those can be either directional or non directional. A new parser function has been added that allows you to convert between any of these formats: #coordinates.

• New geographical functions: #geodistance and #finddestination. You can use the #geodistance parser function to calculate the geographical distance between two points, from and to any of the supported formats. The #finddestination parser function can be used to find a destination given a starting point, an initial bearing and a distance.

• Related to the new geographical functions is a rewritten distance query in Semantic Maps. It now takes into account performance and is scalable, which the old query was not, by using the new storage structure for coordinates. The notation for distance queries has also changed. Instead of using the like operator and a global distance parameter ( like #ask:[[Property: :~coordinates]]|distance=42 ) you now only have to specify your distance locally in the coordinate criteria itself like #ask:[[Property: :coordinates (42 km)]]. Like you can see you can now also specify a unit, which can be any of the supported ones.

• Support for various width and height notations. Previously Maps only accepted width and height values is px, forcing you to use maps of fixed sizes. Since most people want to have their complete page width visible even on small screens, this resulted in a lot of people using rather small maps, and so wasting screen space. 0.6 allows you to specify the size in px, ex, em, and most importantly, in %. The syntax is what you’d expect: width=”420px”, width=”420em”, width=”42%”. width=”420″ will default to using px, so is backward compatible. When using the % values, maps will even adapt their size when the screen width or the height of the container they are in is changed after the page has loaded

• Not a new feature, but rather one that’s removed: OSM support. The OSM service has been completely removed from Maps and Semantic Maps as it was rather broken and not easy to upgrade to the internal structure of Maps 0.6. I’m planning to add it back later on, rewritten from ground up, in 0.6.1 or 0.6.2 or so. Note that you can still view OSM maps on your wiki using the OpenLayers service, which has build in OSM layers, and also allows you to define your own layers since 0.5.5.

• And many more

The most notable bugfixes are:

• Fixed conflict with prototype library that caused compatibility problems with the Halo extension.
• Added automatic icon image sizing for Google Maps and Yahoo! Maps markers.
• Various security fixes, mostly preventing XSS attacks.

If no serious bugs are found in this release, a minor update can be expected in a month or so.

View announcement on the mapping wiki.

Downloads

• Maps 0.6 [ zip, 7z ]
• Maps and Semantic Maps 0.6 [ zip, 7z ]

One of the big changes in the upcoming 0.6 release of Semantic Maps will be the from the ground up rewritten semantic datatype for Geographical coordinates. Although the changes themselves do not directly add any value for the user, they enable some pretty neat improvements to existing features, and the creation of many new ones. In this blog post I’ll first go over the changes that are made, in a technical manner, and then sum up the effects they have from an users point of view.

The SMWDataValue extending class for coordinates, which was previously part of Semantic MediaWiki, and only recently moved over to Semantic Maps, has had a big overhaul. This class, now called SMGeoCoordsValue, used to parser coordinates all by itself. Considering Maps also parses coordinates, this is pretty dumb, since the code is redundant, and even worse, does not behave the same way, resulting into incompatible output. I rewrote all the parsing and formatting code in an as modular fashion as possible, and put it in a new class in Maps, which is MapsCoordinateParser. SMGeoCoordsValue now does all formatting and parsing of values via this class

Markus recently added a hook to Semantic MediaWiki that allows for extensions to define their own database tables for storage of semantic values with a specific signature, which will be part of the upcoming SMW 1.5.1. Semantic Maps is using this hook for storing coordinates in a table with lat and lon fields, both floats. This is a huge improvement to simply storing the coordinates in non-directional float notation with the elements separated by a comma, in a string field. The new layout allows for much faster lookups since the values can be index, esp. for things like the soon to be re-implemented distance query, and opens up a lot of new possibilities.

As a user, you will not notice any of these changes, except maybe that things that where broken will be magically fixed when using this new code. Semantic Maps now has settings that allow you to specify the output format of coordinates in queries though, so you can now view them in a different format then non-directional floats, like you where obliged to in the previous releases. You can pick any format supported by Maps, for example directional DMS, as show on the screenshot. Another change is that you will see new functionality based on the new table layout, like, but not limited to, the distance query.

Edit: There actually is one thing you’ll definitely notice as a user when upgrading – you need to update your semantic data via Special:SMWAdmin in order for your wikis coordinates to be stored in the new table. If you don’t do this, queries for coordinates won’t return anything. It’s sort of annoying you need to do this, but there is no way around it.

I’ve been working on Maps and Semantic Maps 0.6, the next big, awesome, release of both MediaWiki extensions, for over a month now. I also released an early alpha on the 3th of April. All planned new features have been implemented, and known bugs from 0.5.x have been fixed. So you might wonder why 0.6 is still not out.

In response to the possibility of someone doing work on Maps and Semantic Maps during the upcoming Google Summer of Code, I took a critical look at the current structure, holding into account all the things I learned over the last two month while working on Storyboard. I came to the conclusion that a bunch of things ought to be handled in a cleaner fashion, and started to refactor this code. The most difficult part here is changing how the display_map and display_point(s) parser functions handle their mapping service parameter and validate the provided location(s). I’ve been wanting to change this since 0.4, but didn’t since it’s rather tricky to do. I decided to finally get this done now, and have done most of the work. To complete these changes, I’ll have to make some rather complex modifications to Validator, which can take a while to complete. That’s the last thing that needs to be done before the 0.6 release though

I estimate this should be done in approximately 2 weeks, maybe sooner. After that I’m planning to release at least one RC, to ensure stability and complete awesomeness, cause really a lot has been changed. I figure about three quarters of the codebase has been changed in one way or another!

After this release a lot of cool new features can be added, which I might do before the GSoC coding period starts. There are also a few people who wanted to do a mapping project for GSoC at the WMF, but did not get accepted, which are planning to do some effort here after all, which is totally awesome

During the past two weeks I’ve been putting some effort in making some big changes to the source of Maps and Semantic Maps, and adding some awesome new features. Although not every to-do for the 0.6 release has been completed, I’ve made an early alpha version available which allows you to try out some of the exiting new functionality. So let’s have a look at the most notable changes since 0.5.5.

New features

• Support for various width and height notations. Previously Maps only accepted width and height values is px, forcing you to use maps of fixed sizes. Since most people want to have their complete page width visible even on small screens, this resulted in a lot of people using rather small maps, and so wasting screen space. 0.6 allows you to specify the size in px, ex, em, and most importantly, in %. The syntax is what you’d expect: width=”420px”, width=”420em”, width=”42%”. width=”420″ will default to using px, so is backward compatible. When using the % values, maps will even adapt their size when the screen width or the height of the container they are in is changed after the page has loaded See my wiki for some cool examples. Although this is really basic functionality in a way, it’s awesome to finally have it available.
• Added full support for DMS, DD, DM and float coordinate notations, both their directional and non-directional variants. This means you can now enter coordinates in these notations in the display_point(s) and display_map parser functions, as well as make Semantic Maps recognize them as geographical coordinates. Smart geocoding support has also been added to #geocode, meaning you can now pass along coordinates as well as addresses, so you don’t have to double check if your passing along an address to prevent losing the value.
• Coordinate formatting support. Along with the smart geocoding support, #geocode now also accepts 2 parameters to specify the output notation. Since you can pass along coordinates, this means you can change the notation of a set of coordinates using #geocode. A new parser function, #coordinates, has been added specifically for this purpose, so for pure formatting, you should use this one rather then #geocode. Both parser functions work with a hybrid parameter system that supports several ‘default’ parameter (ones that consist of only a value, and no name) and named parameters. You can work with names or without for the default ones, and the default ones in #geocode are the same as the old parameter order, so backward compatibility is retained.
• Rewrote map html and js output. This is now done in a cleaner, safer and more consistent way, for all mapping services in both Maps and Semantic Maps. This is not a new feature, but such a big refactoring it’s worth noting.

More to come…

• Native support for geographical proximity queries in Semantic Maps. This will require quite some work to do in a clean and efficient way, so will probably one of the last features finished for 0.6.
• OpenLayers 2.9 support. Since Maps 0.1, it has been supporting OpenLayers 2.8, and 2.9 is just about to be released, so It’ll be nice to see the progress made there.

On top of these changes to the extensions, I’m drastically redoing the documentation and examples. I’ll post more about that later on though