Jeroens blog

It’s been a little over half a year since the last mayor release of the Maps and Semantic Maps extensions, but now 1.0, featuring many new features and internal improvements, is here! This is the most significant release since version 0.1, which quite coincidentally, was released exactly 2 years ago today

Let’s have a look at all the new awesomeness:

Google Maps v3 support

Maps has had some very basic support for Google Maps v3 since version 0.5.3, back when the v3 API was still in beta. Right now the v3 API is out of beta, and the v2 one has been deprecated a few months back, so it was definitely time to further implement support for v2. This new version of Google Maps brings many improvements, focusing mostly on performance (loading speed of the maps, esp on mobile devices), but also several other cool things such as streetview support and easy adding of layers such as traffic. Since the v2 API has been deprecated it makes only sense that the default mapping service in Maps has changed from Google Maps v2 to Google Maps v3. It’s not really a new feature, but awesome nonetheless: you don’t need an API key for Google Maps v3! So when setting up Maps on a new wiki (using Google Maps), you won’t have to bother with any API key configuration any more, it’ll just work Support for the former is still there, so if you want to retain the exact same functionality as you have with v0.7 of Maps (or earlier), you can still get those maps using service=googlemaps2 (or format=googlemaps2 in SMW queries). One feature has been removed from the v2 implementation, which is the overlays control. This has to do with internal changes and performance optimizations discussed later on in this blog post. Be sure to check out the Google Maps v3 Maps documentation to discover all the cool features it supports.

Improved form inputs

Since it’s first release Semantic Maps has had form inputs using Google Maps v2, Yahoo! Maps and OpenLayers. These inputs allow for entering geographical coordinates in Semantic Forms forms via a nice GUI with a map and an option to geocode an address. Nothing much has changed to these inputs since that initial release, until now. All input, including a new Google Maps v3 one, now have a “set location” button next to the coordinates box, which sets the map to that location, as was already the case with the geocoding button next to the address field. Some minor layout improvements have also been made, and hitting enter in the coordinates or geocoding fields will result in what you’d expect, rather then submitting the form. The OpenLayers form input now also supports geocoding, making use of the new GeoNames API; more on that later.

Use of the MediaWiki resource loader

MediaWiki 1.17 introduces a resource loader for JavaScript and CSS, which both Maps and Semantic Maps now make use of for all their JS and CSS resources. The resource loader does several neat things, the most important thing being delaying loading (and execution) of resources until after page load as well as combining and minifying them, which is very important for performance (page load time). This means that when you have over 9000 maps on your page, it’ll actually load before protons decay away, initially showing only gray boxes where the maps should be, and then one-by-one loading the maps into them. The resource loader does several other cool things, such as automatic right-to-left conversion of CSS and neat conversion of i18n messages from PHP to JS. Making use of the RL when available but at the same time retaining compatibility with pre-RL MediaWiki turned out to be a bit difficult and it would greatly complicate the code, so I decided to simply not do this. This obviously means you will need MediaWiki 1.17 in order to use (Semantic) Maps 0.8.x and later. Can’t use 1.17 yet for some reason? Don’t panic! I’ll continue to support the 0.7.x for a while longer, fixing bugs as they are found. Don’t expect any new features there though.

Support for the new GeoNames API

Maps has a new geocoding service: the new GeoNames API. It already had support for GeoNames, but it seems this service is now only offered as legacy support. (I’m not completely sure about this, if someone better familiar with the service knows, please poke me.) A big change with the new service is that you need a GeoNames API account to use it, which you can create here. You then need to set your accounts user name in your LocalSettings.php file. Maps still has the ‘geonames’ service as default for #geocode and OpenLayers maps. If you set the account name, the new service will be used, if not, Maps will fall back to the old one, so you can upgrade to 0.8 without geocoding suddenly failing because you don’t have a GeoNames account. For more info see the GeoNames documentation for Maps.

JavaScript overhaul

Similarly to the form inputs in Semantic Maps, all the JavaScript in both Maps and Semantic Maps has seen quite little attention since the initial releases of the extensions. Many additions have been made to add new functionality, but the structure has remained the same ever since. All of it has now been rewritten to jQuery plugins, making it a lot more orderly and easy to extend.

More internal improvements

Not only the JavaScript has seen significant improvements, but some legacy code has also been thrown out of the PHP, making it a lot less complex, more easy to track the code flow and definitely makes it easier to add new functionality. This rewrite is very much a follow up to internal improvements made in versions 0.6, 0.7 and 0.7.3, and completes getting rid of some bad old architecture in the core Maps code. Future releases will therefore most likely focus a lot more on simply adding new features

And more…

For a full list of changes, see the release announcement. These do not list the huge amount of internationalization updates and small improvements made by a lot of contributors by reporting issues and providing patches. Thanks to all!

How stable is it?

A lot of internal changes have been made, but at the same time, most of these have been made about 3 months back. Several wikis have been using alpha versions of 1.0 ever since, and a release candidate was made a few days back. Since there are no known issues at this point, I decided to release 1.0. So yes, it should be pretty stable, although you might run into minor issues with less frequently used components. If you do, please report it, so they can be addressed quickly in a 1.0.1 release.

Legacy support for 0.7.x

As an extension developer and MediaWiki consultant, I’m quite aware that a lot of people are not in positions to update their MediaWiki to 1.17 just yet, preventing them from upgrading Maps and Semantic Maps. For this reason I decided way back when starting the development of version 1.0 to continue limited support of version 0.7.x for a while. Versions 0.7.4 to 0.7.7 have been released especially for this purpose, and I’ll continue to backport important fixes. Don’t expect any new features to show up for 0.7.x though.

What’s next?

Although the current set of functionality is pretty solid, there are many other geographical features one can imagine. Features such as marker clustering, static maps, route plotting (without the use of KML), ect, have been on the wishlist practically since the inception of the Maps extension. There is nothing really standing out for me enough to go ahead and implement it in my free time. If any such feature is important to you and you can fund it’s development, definitely contact me. Of course I’ll continue to support the extension and make fixes where needed.

The Semantic MediaWiki 1.6 release will be followed by a new Semantic Bundle, which will include this new version of Maps and Semantic Maps.

Downloads

• Maps 1.0 (archive)
• Maps 1.0 (svn tag)
• Maps and Semantic Maps 1.0 (archive)
• Semantic Maps 1.0 (svn tag)

• Maps 0.7.7 (archive)
• Maps 0.7.7 (svn tag)
• Maps and Semantic Maps 0.7.7 (archive)
• Semantic Maps 0.7.7 (svn tag)

Referata, which runs the documentation wiki for the mapping extensions has upgraded to version 1.0, so you can have a look at and try out the new features yourself on the example/demo pages.

I just released a new version of the Live Translate extension for MediaWiki.

Live Translate is a simple extension that allows live translation of wiki pages using Google Translate or (as of version 1.1) Microsoft Translator. It also enables you to define a “dictionary” of certain words or phrases and their translations; any word or phrase in the original text found in the “dictionary” will be translated using that dictionary, instead of using the Google/Microsoft translation service.

As you can deduce from the above description, one of the new features in version 1.1 is the ability to choose Microsoft Translator as translation service instead of Google Translate, which remains the default for now. The reason for the addition of this service is that Google has announced that it will be retiring it’s free machine translation services in a few months, which Live Translate is using. Another new feature is the ability to place comments into translation memories using the Live Translate Format. Full release notes can be found here.

Downloads

• LiveTranslate1.1.zip
• SVN tag
• SVN trunk

It’s been a while since I posted an update on Semantic MediaWiki and extensions in general, and my work specifically. This is due to a pile of work that has been done on different components, each of which I’ll address at the point it’s released.  In this blog post I’ll provide you with a short overview of what”s (been) going on in the SMW world.

Semantic MediaWiki 1.6

Almost two years after the latest big release, SMW 1.5, 1.6 comes with many important internal changes focused on performance, stability and extensibility and several new features. Here you have an extract of the release notes as they currently are on SVN trunk:

* Full support for synchronizing RDF stores with SMW, and for answering #ask queries based on this data. The communication happens via SPARQL (1.1), and all SPARQL-capable stores should be supported.
* The Type namespace has been abolished. Builtin types now are displayed by the special page Special:Types, and there are no “custom types” any longer. By default, the Type namespace is gone and existing pages in this namespace can no longer be accessed. This can be changed by setting $smwgHistoricTypeNamespace = true in LocalSettings.php before including SMW.
* Changed the way in which units of measurement work. Type:Number now does not accept any units, and a new type “Quantity” is used for numbers with units. Units must be declared on the property page (not on the Type page as before), and only units that have a declared conversion factor are accepted.
* The declaration of Type:Record properties has changed. Instead of a list of datatypes, the declaration now requires a list of properties that are to be used for the fields of the record. The declaration is still done with the property “has fields” as before.
* Changed the way parameters in query printers are specified and handled using the Validator extension. This includes improvements to the parameter options in the Special:Ask GUI and better error reporting for ask queries.
* Added UNIX-style DSV (Delimiter-separated values) result format.
* Reworked internal data model, cleaning up and re-implementing SMWDataValue and all of its subclasses, and introducing new data item classes to handle data. The class SMWCompatibilityHelpers provides temporal help for extensions that still depend on the old format and APIs.
* Fixed PostGre SQL issues with the installation and upgrade code.
* Added API module (smwinfo) via which statistics about the semantic data can be obtained.

That’s a lot of awesomeness no?

As you can deduce from the above notes, this release is not fully backwards compatibility with SMW 1.5.x, so it’s possible you’ll need to do some migration work. The Validator extension is also introduced as an extra dependency, but it will come bundled with SMW, so you’ll only need to care about this when getting the code from SVN.

SMW 1.6 has been in testing phase for 2 weeks or so now, and most bugs have been taken care of. With some luck, the new version will be released in a week or two Do feel free to try out the new version on non-critical wikis and report any issues you might find.

I already stated that SMW 1.6 is not fully feature compatible with SMW 1.5.x, but it’s also most definitely not compatible with earlier versions code-wise for extensions. This means that quite some SMW extensions released before the development on SMW 1.6 started won’t be compatible anymore, and will require you to also update them to their latest release when upgrading SMW to 1.6 or later. The ones that are in the Semantic Bundle are all compatible already on SVN trunk, so you should be able to upgrade everything as soon as SMW 1.6 is released.

Maps and Semantic Maps 1.0

Another very significant release is the one of Maps and Semantic Maps. I’ve been working on this version for quite a while; it was branched from version 0.7.3, and it’s the most significant (and awesome) release since the creation of both extensions, hence the bump from 0.7.x to 1.0. These are the changes:

* Added full Google Maps v3 support and set it as the default mapping service.
* Added new geocoder making use of the new GeoNames API.
* Added support for the auto-documentation features for parser hooks introduced in Validator 0.4.3.
* Added resizeable parameter to all mapping services except OSM.
* Removed compatibility with pre MediaWiki 1.17.
* Removed overlays parameter for Google Maps v2.
* Usage of the Resource Loader for all scripts and stylesheets.
* Rewrote all the map JavaScript to jQuery plugins.
* Rewrote the way parameters are translated to JavaScript. Now one big PHP object is json_encoded.
* Improved KML formatter.
* Use of Google Maps geocoding service v3 instead of v2.
* Fixed geocoding service overriding based on mapping service (merged in from Maps 0.7.5).
* Fixed fatal error occurring when running maintenance/refreshLinks.php.
* Improved default width of maps (merged in from Maps 0.7.5).
* Improved map query parameter support in the Special:Ask GUI
* Rewrote the map printers to use the SMQueryHandler class.
* Added geocoding capability to the OpenLayers form input when having a GeoNames API account.
* Added ‘update map’ button to all form inputs.

This release of the mapping extensions requires MediaWiki 1.17 or later and the new SMW 1.6, or later. For people not running an MW older then 1.17 (which currently is still not released, pretty much blocking this release of Maps and SM), the 0.7.x branch still remains supported for a while. It’s currently at 0.7.6, and I plan to release 0.7.7 soonish. Do note that Semantic Maps 0.7.x is NOT compatible with SMW 1.6 or later, it needs SMW 1.5.1 – 1.5.7 alpha.

Semantic Forms 2.2

Yaron Koren has been working on the next big release of the most popular SMW extension, Semantic Forms. It brings compatibility with SMW 1.6, and adds several new features, including:

• #autoedit parser function that allows creating a link that, when clicked, automatically sets one or more fields in another page to certain values.
• “Save and continue” button
• Handling of boolean properties can now also be done using radiobuttons and dropdowns, instead of only checkboxes.

Semantic Watchlist

Semantic Watchlist is a new SMW extension I’ve developed for the IEEE as WikiWorks consultant. Semantic Watchlist enables users to watch semantic properties by adding a new watchlist page that lists changes to these properties. Users can choose to follow one or more watchlist groups, which are administrator defined, and cover a set of properties and a set of pages (category, namespace, or SMW concept). Notification of changes to watched properties is also possible via email. I think it’s totally awesome.

It’s main features are:

• A watchlist page listing changes to properties watched by the user.
• Per-user optional email notification per edit that changes properties.
• Integration with user preferences to allow users to specify which watchlist groups they want to follow, and if they want to receive emails on changes.
• Special:WatchListConditions as administration interface for watchlist groups.
• API module to query property changes grouped by edit for a single user.
• API modules to add, modify and delete the watchlist groups.

It requires MediaWiki 1.17 or later and SMW 1.6 or later, and still has to see an initial release. It’s pretty much ready for it, and can be seen as beta right now.

SMWCon

The Spring 2011 SMWCon was held on April 28-30, 2011 at the Raytheon BBN Technologies office in Arlington, Virginia, in the Washington, DC area, and it was a great success. You can read more about it in Yarons writeup.

The next SMWCon, SMWCon Fall 2011, will be held on September 21–23, 2011 in Berlin, Germany. Berlin – yay! It’s going to be awesome, and I’ll be attending, probably giving some talk about Maps and Semantic Maps, and possibly other extensions as well (Semantic Watchlist being a good candidate).

Like I already noted, I’ll be posting more comprehensive (and official) release announcements for each extension when they are released I’d also like to point out that this is definitely not everything that’s been going on in the SMW world. For example there are 2 Google Summer of Code students doing SMW related work, about which I might write later on, and many people are doing SMW projects that I’m simply not aware of or am not closely following.

Yesterday I quickly wrote up a simple (but awesome) MediaWiki extension that allows you to make use of the Spark library in your wiki.

Spark as described on the Spark website:

The web is not only growing in sheer size, but it also grows in how much it is interconnected. Where once the Web was a set of more or less separated sites, today sites are more and more being connected. More and more data is being offered on the Web in a way that can be further processed, and more and more sites and applications are using external data. More and more mashups are created, where data from different sources is integrated and displayed with novel visualisations.

Spark is a library that enables HTML authors to create mashups more easily than ever before. Using standard Web technologies like SPARQL, RDF, HTML5, and JavaScript, Spark can query external knowledge sources (so called triple stores or SPARQL endpoints), and then visualise the results.

With Spark, website developers can create visually appealing mashups without having to write a single line of JavaScript, but merely using some markup elements describing the source of the data that is to be shown, a query to select the appropriate data, and selecting one from an expandable set of visualisations and their parameters.

Spark is developed by Denny Vrandečić and Andreas Harth.

This MediaWiki extension, unsurprisingly titled Spark, adds a <spark> tag to MediaWiki which is equivalent to <div class=”spark”> as described in the spark library documentation. All parameters (except the class=”spark” one) can just be copied over between spark divs in web pages, and the <spark> tag in MediaWiki. It is currently at version 0.1, which is a beta release. It includes a still experimental version of the Spark library, so you should probably not use this extension on production websites just yet. The Spark people are looking for developers to help out, so if you want to play around with SPARQL a bit, like I basically did with this extension, be sure to poke them

The extension required MediaWiki 1.17 or above (as it makes use of the new Resource Loader) and PHP 5.2 or later.

Download

• The current Spark (MediaWiki extension) release (zip)
• svn co http://svn.wikimedia.org/svnroot/mediawiki/tags/extensions/Spark/REL_0_1 (tag for 0.1, including Spark lib trunk)
• svn co http://svn.wikimedia.org/svnroot/mediawiki/trunk/extensions/Spark (trunk, including Spark lib trunk)
• svn co http://rdf-spark.googlecode.com/svn/trunk/src/ (Spark lib trunk)

Further possibilities

Right now you can embed mashups with SPARQL queries that get their data from some SPARQL endpoint. This opens up a whole bunch of possibilities, but is a bit silly when you are running your own Semantic MediaWiki instance and want to visualize structured data stored by it using Spark. A possible addition to the Spark MediaWiki extension therefore is having support for Spark as a so called SMW result format. For this translation from the SMW ask query language to SPARQL is needed, which is some work. I might implement this at some future point, but have several other things I want to poke at, so it won’t be soonish

A few weeks back I started work on a new MediaWiki extension to provide decent rating functionality. The reason: I got sick of all the crappy rating extensions out there and decided to write one that both works and has sane code. The new extension is called “Ratings“.

The Ratings extension provides a tag extension that when embedded in a page allows users to rate different “properties” of the page. Quite simple. It also adds a votesummary hook which allows embedding a summary as seen below into pages for any property of any page.

The extension is written in such a way it’s easy to add additional types of rating (interface) elements. It exposes 2 API modules, one to obtain all rating data needed on a page, and one to submit votes.

It’s not quite finished yet, as I can’t get the rating interface quite right. I tried 2 jQuery rating star plugins, both messing up their position in the page after onload. Therefore I have not made any release yet, and definitely recommend against using this extension on any production wiki. I’m not planning to put any more time into this issue (I already wasted some hours on it >_>), so feel absolutely free to fix the layout issues (you can contact me for details, or try out the extension yourself and spot the obvious), or to put in your own favourite rating element. (source code)

Yesterday I for some reason decided to have some fun with Python by writing a simple script to Find Dead Translation keys in MediaWiki extensions. The resulting script, titled FDT, can be found on GitHub, and is licenced under the GNU GPL v3+ (yes, the later probably comes as a shock to you!).

What is does, and how

The script works by first obtaining a list of defined language keys from the specified i18n file (which is done via an evil PHP subprocess, to avoid messy parsing of array keys), and then looping over all php files in the directory (recursively ofc) to check if they contain any of the keys, after which a list of not-found keys is returned. If there are not-found keys, the script offers to delete those from the i18n file, which also saves quite some work if they are assigned to for many languages. One important limitation is that the script finds key usage by doing a simple “in string” search on the whole contents of each file, which obviously will miss dynamically constructed strings. An example of the work this script did (for Semantic Maps) can be found in this commit.

Some more points of interest

This was a nice little exercise in Python for me, making me more familiar with the language basics, which after several similar small projects, I now think I pretty much have mastered. Also fun was that this is the first project I did using Apatana Studio 3, which was released very recently. It’s an Eclipse based IDE aimed at Python and Ruby web development, including excellent support for JavaScript and various JS libraries such as jQuery and Ext. It also has some support for PHP, but that’s a bit meagre compared to Zend Studio or PhpStorm, an IntelliJ based IDE I’ve been trying out lately.

More posts to come

Over the past 2 years I’ve changed my blog posting style from posting casual updates about small stuff like this post, to only posting about new MediaWiki extensions or important updates to existing ones. I’ve decided to reverse this trend and post more lightweight for fun things, as opposed to only big release announcements

A few days back I created my second ever Android application, basically a re-do of the first one that was titled “Pamela for Android“, now with the name “PAMELA Widget“. It has the same function: display a list of people at 0×20, the hackerspace in Ghent, or HSB, the space in Brussels,obtained via the PAMELA webservice. More info on PAMELA can be found here.

I couldn’t really get the interface in the previous app quite working as I wanted, and then got inspired by the iRail Liveboard app, created by Christophe Versieux, to write a new version, based on the code of the Liveboard app.

You can get the app from the Android market by searching “pamela widget”. You can also get the code, which is GPL3+ licenced, from GitHub.

Somewhere in the last two weeks I quickly wrote up a small new MediaWiki extension to include content from Wikipedia or some other MediaWiki install into pages on your wiki. It’s titled Include WP.

The extension does not import anything (so nothing is stored), but rather fetches content from the remote wiki every time the page is loaded. This means it takes a second or two for it to actually load, but ensures you got the latest content. Some clean up of the remote content happens and there are several options that allow you to customize how the included content is displayed. It makes use of the Validator extension.

Feature overview:

• Include articles from Wikipedia or any other MediaWiki wiki into your pages.
• Show only a limited amount of paragraphs on page load, with an option to show the full article.
• Partial conversion to plain-text.

1. Removal of templates (such as infoboxes), ref tags, comments, categories and images.
2. Both internal and external links are rendered as plain-text.
3. Tables, lists, table of content, section headers and more are retained.

• Usage of the MediaWiki Resource Loader when available with backward support for MW 1.16.x.

Some screenshots

Download

• Stable releases (archives)
• SVN tags
• SVN trunk

Earlier today I released a new version of the Semantic Result Formats extension. SRF bundles a number of so called “result formats” that can be used in conjunction with Semantic MediaWiki, and allow visualization of queried data.

New tagcloud format

This release introduces a new “tagcloud” format which I created for WikiWorks. It allows displaying queried values in a tag cloud based on how many times they occur in the result set. Several options allow you to specify the minimum and maximum tags sizes, how to increase (logarithmic or linear), how to order the tags and more. The format is enabled by default, so when updating SRF to the latest version, you can immediately use it.

Improved gallery format

Some significant improvements have been made to the gallery format. Previously it was only able to display images that are returned as subjects in the query result. This is useful in some cases, but forces you to have semantic properties on your image pages, which is not a good approach if you want to have this info on the page the images belong to. Now it’s possible to point to images using a page property and then do a query in which you specify this property points to the images you want to display. An option to disable automatic captions has also been added.

And more…

Several other improvements have also been made, including fixes to the jqplot format by Yaron Koren and some clean-up of the timeline and eventline formats.

Download

Since SRF was lacking a place for release downloads, I created a new Google Code project where they are put now.

• Stable releases
• Releases as part of Semantic Bundle
• SVN tags
• SVN trunk

It’s been a while since the previous release of Validator, but this minor update includes some cool new functionality.

As I was creating the SubPageList extension and it’s documentation (at 27c3!), it occurred to me that to document the usage of the <subpages /> parser hook, all the info I needed was already available via Validator. Since there was no way to get this info out in a meaningful way, I decided to create a new parser hook just for this purpose. It’s titled “describe” and describes one or more parser hooks implemented using the ParserHook class provided by Validator. This description includes a short text about the parser hooks functionality, a table with it’s parameters and their meta-data, syntax examples, a list of aliases (if any) and more. It’s possible to get the description in wikitext, as opposed directly in the page, so you can copy it, and use it (or parts of it) somewhere else. By default it’ll list all parser hooks it knows about directly in the page. An example can be seen here.

Many more not yet implemented features that make use of the parameter handling framework that is Validator can be imagined. One I might implement in the near future is linking to a special page that generates a description of how to use a certain parser hook based on something the user is doing wrong, and link to that from error messages. This can be done easily by building on top of the already existing error message system and the describe parser hook

I think Validator adds a lot of Value when creating parser hooks, and other parameter handling objects. The main problem currently is that when using it’s features in an extension, this extension obviously becomes dependent on Validator. From a technical perspective this does not really matter, but it complicates installation for users (especially since MediaWiki lacks any sort of extension management facilities). I therefore hope to get Validators functionality into MediaWiki core, or at least in the default MediaWiki distribution. I’m rather sceptical to if I can make this happen as a non core-developer, but if I succeed, it would be a great advantage for extension (and core) development, so it’s definitely worth a try.

SubPageList 0.1 comes bundled with an alpha version of Validator 0.4.3, containing most of the new functionality. Maps and Semantic Maps will come bundled with it (or a newer version) on their next release.

Downloads

• Zip and 7z archives
• SVN tag
• SVN trunk
• There will be a new Semantic Bundle soonish which will include the latest Validator.