Jeroens blog

That’s right, the next big release of Semantic MediaWiki is finally here! A little over a year and six minor releases after SMW 1.5. And a big release it is, packing my improvements and several new features. It has been in the making over the past four months and contains contributions by many people. So let’s have a look at all the new awesomeness

Support for RDF triplestores

Full support added for storing SMW data directly within an RDF triplestore, and for answering #ask queries based on this data. The communication happens via SPARQL (1.1), and all SPARQL-capable stores should be supported. The following settings are needed in LocalSettings.php:

$smwgDefaultStore = ‘SMWSparqlStore’;
$smwgSparqlDatabase = ‘SMWSparqlDatabase’;
// The following should be set to the URLs to reach the store:
$smwgSparqlQueryEndpoint = ‘http://localhost:8080/sparql/’;
$smwgSparqlUpdateEndpoint = ‘http://localhost:8080/update/’;
$smwgSparqlDataEndpoint = ‘http://localhost:8080/data/’; // can be empty

The specific support that SMW used to have for the RAP RDF store has been discontinued.

The Type namespace has been abolished

Builtin types now are displayed by the special page Special:Types, and there are no “custom types” any longer. By default, the Type namespace is gone and existing pages in this namespace can no longer be accessed. This can be changed by setting $smwgHistoricTypeNamespace = true in LocalSettings.php before including SMW.

Changes to units of measurement

Changed the way in which units of measurement work. Type:Number now does not accept any units, and a new type “Quantity” is used for numbers with units. Units must be declared on the property page (not on the Type page as before), and only units that have a declared conversion factor are accepted.

Type:Record changes

The declaration of Type:Record properties has changed. Instead of a list of datatypes, the declaration now requires a list of properties that are to be used for the fields of the record. The declaration is still done with the property “has fields” as before. Properties must not be used more than once in has_fields, or the order of values will be random.

Reworked internal data model

Reworked internal data model, cleaning up and re-implementing SMWDataValue and all of its subclasses, and introducing new data item classes to handle data. The class SMWCompatibilityHelpers provides temporary help for extensions that still depend on the old format and APIs.

This change is very important compatibility wise, as it removes support for some older behavior and deprecates a bunch more. Older versions of a lot of SMW extensions will not work with SMW 1.6. However, since 1.6 has been in the making for quite some time, most, if not all, of these extensions have released one or more versions that work just fine with 1.6. So if you stick with the latest releases, you should be good.

There are already 2 SMW extensions that now require SMW version 1.6 or later: Semantic Maps 1.0 and the new kid in town: Semantic Watchlist (see Semantic Watchlist release post).

Result formats make use of Validator

Semantic MediaWiki now makes use of the Validator extension to handle parameters passed to result printers. This opens the door to numerous awesome new functionality. For one, it allows for better parameter error feedback to the user. In SMW < 1.6 when you enter a text value for a numeric parameter, it’ll simple be ignored (without giving any warning). With this new approach the user will get a warning along the lines of “The value ‘foobar’ is not valid for parameter limit, it must be an integer.”. Another advantage of using Validator here is that the query argument GUI on Special:Ask can show more fitting controls and display default values. A third win, is that writing new query printers has now become easier, as you just have to specify the parameters they should accept, and then don’t have to bother with cleaning, validating and formatting them. Further things that can be done with this in the future are automatic generation of documentation per result format, which would be a huge improvement over the current documentation and smarter behavior (such as autocompletion and live validation) in interfaces where the values for these parameters can be specified.

New DSV format

I added a new result format to SMW that outputs data as UNIX-style Delimiter Separated Values. For those not familiar with the differences between DSV and CSV (which was already supported), it comes down to sane escaping of the delimiter characters. Instead of the weird and hard to parse approach taken in CSV, the familiar backlashing technique is used in DSV. You can use this format by adding “format=dsv” to any ask query.

New API module for version info

This is a very small feature I added geared towards developers and the SMW registry tool in particular; it has not relevance to regular end users. Up until now the SMW registry tool obtained info about SMW installs by scraping Special:Version, which is a HTML page intended for humans to look at, and by scraping Special:SemanticStatistics, which is also a HTML page. Now the info provided by these pages and used by the SMW registry, ie the version of SMW and the number of properties, users and pages, can be obtained via the MediaWiki API using action=smwinfo.

New smwdoc parser hook

Similar to the new API module, this new parser hook is not geared to most end users, but rather to documenters and site admins. This hook utilizes Validators auto documentation capabilities to display tables with parameter info, such as name, type, default value and description, for a specified result format. It’s in essence a result format oriented  version of Validators more general purpose describe parser hook. As soon as the SMW documentation wiki updates to 1.6, it’ll be possible to use this hook to fill part of the documentation gap there unfortunately is.

Some noteworthy fixes

• PostgreSQL support
• Output issues in the JSON result format

I’m probably forgetting others here

Downloads

• Semantic MediaWiki 1.6 release, zip archive
• Semantic MediaWiki 1.6 release, 7z archive
• Semantic MediaWiki 1.6 release, svn tag
• Semantic MediaWiki, svn trunk

Do note that as of this release, SMW requires Validator, which is included in the release packages, but obviously needs to be obtained separately when checking out with SVN.

What’s next?

Devayon Das, a Google Summer of Code student, mentored by Markus Krötzsch, is working on improving the search and browse interfaces of SMW (read all about it here). This work will probably end up in one of the upcoming 1.6.x releases. I’d also like to make some improvements to the automatic documentation generation via the smwdoc parser hook by the next release, mainly focusing on internationalization. I’m not aware of any other concrete plans to make changes to SMW itself for now. There are many things that can be done, and several projects that are being looked at, but only time will tell which of these get any traction.

In any case, if you are interested in the future of SMW, I definitely recommend attending SMWCon, the SMW event that’s held once a year in both the US and the EU. The next one will take place from September 21st to September the 23rd, in Berlin, Germany.

More frequent updates

In an effort to provide the community with more frequent updates on the status of SMW development and other news, a Twitter and an Identi.ca account have been set up. You can follow these to stay up to date on the latest SMW developments

Further info

• Semantic MediaWiki 1.6 page on the SMW wiki
• Installation and upgrade instructions
• Raw release notes

It’s been a while since I posted an update on Semantic MediaWiki and extensions in general, and my work specifically. This is due to a pile of work that has been done on different components, each of which I’ll address at the point it’s released.  In this blog post I’ll provide you with a short overview of what”s (been) going on in the SMW world.

Semantic MediaWiki 1.6

Almost two years after the latest big release, SMW 1.5, 1.6 comes with many important internal changes focused on performance, stability and extensibility and several new features. Here you have an extract of the release notes as they currently are on SVN trunk:

* Full support for synchronizing RDF stores with SMW, and for answering #ask queries based on this data. The communication happens via SPARQL (1.1), and all SPARQL-capable stores should be supported.
* The Type namespace has been abolished. Builtin types now are displayed by the special page Special:Types, and there are no “custom types” any longer. By default, the Type namespace is gone and existing pages in this namespace can no longer be accessed. This can be changed by setting $smwgHistoricTypeNamespace = true in LocalSettings.php before including SMW.
* Changed the way in which units of measurement work. Type:Number now does not accept any units, and a new type “Quantity” is used for numbers with units. Units must be declared on the property page (not on the Type page as before), and only units that have a declared conversion factor are accepted.
* The declaration of Type:Record properties has changed. Instead of a list of datatypes, the declaration now requires a list of properties that are to be used for the fields of the record. The declaration is still done with the property “has fields” as before.
* Changed the way parameters in query printers are specified and handled using the Validator extension. This includes improvements to the parameter options in the Special:Ask GUI and better error reporting for ask queries.
* Added UNIX-style DSV (Delimiter-separated values) result format.
* Reworked internal data model, cleaning up and re-implementing SMWDataValue and all of its subclasses, and introducing new data item classes to handle data. The class SMWCompatibilityHelpers provides temporal help for extensions that still depend on the old format and APIs.
* Fixed PostGre SQL issues with the installation and upgrade code.
* Added API module (smwinfo) via which statistics about the semantic data can be obtained.

That’s a lot of awesomeness no?

As you can deduce from the above notes, this release is not fully backwards compatibility with SMW 1.5.x, so it’s possible you’ll need to do some migration work. The Validator extension is also introduced as an extra dependency, but it will come bundled with SMW, so you’ll only need to care about this when getting the code from SVN.

SMW 1.6 has been in testing phase for 2 weeks or so now, and most bugs have been taken care of. With some luck, the new version will be released in a week or two Do feel free to try out the new version on non-critical wikis and report any issues you might find.

I already stated that SMW 1.6 is not fully feature compatible with SMW 1.5.x, but it’s also most definitely not compatible with earlier versions code-wise for extensions. This means that quite some SMW extensions released before the development on SMW 1.6 started won’t be compatible anymore, and will require you to also update them to their latest release when upgrading SMW to 1.6 or later. The ones that are in the Semantic Bundle are all compatible already on SVN trunk, so you should be able to upgrade everything as soon as SMW 1.6 is released.

Maps and Semantic Maps 1.0

Another very significant release is the one of Maps and Semantic Maps. I’ve been working on this version for quite a while; it was branched from version 0.7.3, and it’s the most significant (and awesome) release since the creation of both extensions, hence the bump from 0.7.x to 1.0. These are the changes:

* Added full Google Maps v3 support and set it as the default mapping service.
* Added new geocoder making use of the new GeoNames API.
* Added support for the auto-documentation features for parser hooks introduced in Validator 0.4.3.
* Added resizeable parameter to all mapping services except OSM.
* Removed compatibility with pre MediaWiki 1.17.
* Removed overlays parameter for Google Maps v2.
* Usage of the Resource Loader for all scripts and stylesheets.
* Rewrote all the map JavaScript to jQuery plugins.
* Rewrote the way parameters are translated to JavaScript. Now one big PHP object is json_encoded.
* Improved KML formatter.
* Use of Google Maps geocoding service v3 instead of v2.
* Fixed geocoding service overriding based on mapping service (merged in from Maps 0.7.5).
* Fixed fatal error occurring when running maintenance/refreshLinks.php.
* Improved default width of maps (merged in from Maps 0.7.5).
* Improved map query parameter support in the Special:Ask GUI
* Rewrote the map printers to use the SMQueryHandler class.
* Added geocoding capability to the OpenLayers form input when having a GeoNames API account.
* Added ‘update map’ button to all form inputs.

This release of the mapping extensions requires MediaWiki 1.17 or later and the new SMW 1.6, or later. For people not running an MW older then 1.17 (which currently is still not released, pretty much blocking this release of Maps and SM), the 0.7.x branch still remains supported for a while. It’s currently at 0.7.6, and I plan to release 0.7.7 soonish. Do note that Semantic Maps 0.7.x is NOT compatible with SMW 1.6 or later, it needs SMW 1.5.1 – 1.5.7 alpha.

Semantic Forms 2.2

Yaron Koren has been working on the next big release of the most popular SMW extension, Semantic Forms. It brings compatibility with SMW 1.6, and adds several new features, including:

• #autoedit parser function that allows creating a link that, when clicked, automatically sets one or more fields in another page to certain values.
• “Save and continue” button
• Handling of boolean properties can now also be done using radiobuttons and dropdowns, instead of only checkboxes.

Semantic Watchlist

Semantic Watchlist is a new SMW extension I’ve developed for the IEEE as WikiWorks consultant. Semantic Watchlist enables users to watch semantic properties by adding a new watchlist page that lists changes to these properties. Users can choose to follow one or more watchlist groups, which are administrator defined, and cover a set of properties and a set of pages (category, namespace, or SMW concept). Notification of changes to watched properties is also possible via email. I think it’s totally awesome.

It’s main features are:

• A watchlist page listing changes to properties watched by the user.
• Per-user optional email notification per edit that changes properties.
• Integration with user preferences to allow users to specify which watchlist groups they want to follow, and if they want to receive emails on changes.
• Special:WatchListConditions as administration interface for watchlist groups.
• API module to query property changes grouped by edit for a single user.
• API modules to add, modify and delete the watchlist groups.

It requires MediaWiki 1.17 or later and SMW 1.6 or later, and still has to see an initial release. It’s pretty much ready for it, and can be seen as beta right now.

SMWCon

The Spring 2011 SMWCon was held on April 28-30, 2011 at the Raytheon BBN Technologies office in Arlington, Virginia, in the Washington, DC area, and it was a great success. You can read more about it in Yarons writeup.

The next SMWCon, SMWCon Fall 2011, will be held on September 21–23, 2011 in Berlin, Germany. Berlin – yay! It’s going to be awesome, and I’ll be attending, probably giving some talk about Maps and Semantic Maps, and possibly other extensions as well (Semantic Watchlist being a good candidate).

Like I already noted, I’ll be posting more comprehensive (and official) release announcements for each extension when they are released I’d also like to point out that this is definitely not everything that’s been going on in the SMW world. For example there are 2 Google Summer of Code students doing SMW related work, about which I might write later on, and many people are doing SMW projects that I’m simply not aware of or am not closely following.

It’s been a while since the previous release of Validator, but this minor update includes some cool new functionality.

As I was creating the SubPageList extension and it’s documentation (at 27c3!), it occurred to me that to document the usage of the <subpages /> parser hook, all the info I needed was already available via Validator. Since there was no way to get this info out in a meaningful way, I decided to create a new parser hook just for this purpose. It’s titled “describe” and describes one or more parser hooks implemented using the ParserHook class provided by Validator. This description includes a short text about the parser hooks functionality, a table with it’s parameters and their meta-data, syntax examples, a list of aliases (if any) and more. It’s possible to get the description in wikitext, as opposed directly in the page, so you can copy it, and use it (or parts of it) somewhere else. By default it’ll list all parser hooks it knows about directly in the page. An example can be seen here.

Many more not yet implemented features that make use of the parameter handling framework that is Validator can be imagined. One I might implement in the near future is linking to a special page that generates a description of how to use a certain parser hook based on something the user is doing wrong, and link to that from error messages. This can be done easily by building on top of the already existing error message system and the describe parser hook

I think Validator adds a lot of Value when creating parser hooks, and other parameter handling objects. The main problem currently is that when using it’s features in an extension, this extension obviously becomes dependent on Validator. From a technical perspective this does not really matter, but it complicates installation for users (especially since MediaWiki lacks any sort of extension management facilities). I therefore hope to get Validators functionality into MediaWiki core, or at least in the default MediaWiki distribution. I’m rather sceptical to if I can make this happen as a non core-developer, but if I succeed, it would be a great advantage for extension (and core) development, so it’s definitely worth a try.

SubPageList 0.1 comes bundled with an alpha version of Validator 0.4.3, containing most of the new functionality. Maps and Semantic Maps will come bundled with it (or a newer version) on their next release.

Downloads

• Zip and 7z archives
• SVN tag
• SVN trunk
• There will be a new Semantic Bundle soonish which will include the latest Validator.

During 27c3 someone asked me to install an extension to list subpages onto the hackerspace.be wiki. I picked SubPageList3, as it seemed to most decent one. I did a simple test to see if it was working, and immediately found a namespace bug. So I decided to quickly rewrite it and also let it use Validator for additional awesomeness.

Strangely enough, although there where SubPageList2 and SubPageList3 extensions, there was no SubPageList extension to be found anywhere in the WMF SVN repo or the mediawiki.org wiki, so I decided to simply take that name. The extension should be backwards compatible with SubPageList3.

I spend 5 hours or so doing the rewrite, and have released an initial version yesterday. Seems to work fine and is currently in use on hackerspace.be. It requires Validator 0.4.2 or above, and is the first extension making full use of new the auto-documentation functionality Validator 0.4.3 introduces (more on that in a later blog post).

Download

• Version 0.1 archives
• Version 0.1 svn tag
• svn trunk

Maps 0.7.1 and Semantic Maps 0.7.1 are now available for download. The main new feature in this release is the long awaited images as layers. It allows you to display images with OpenLayers, so users can pan and zoom around, and markers can be placed to draw attention to certain area’s and provide additional information. Check out the examples. Some issues have also been fixed, in both extensions, mainly OpenLayers related.

What’s next?

Improving the pages in the layer namespace and providing a way to group markers in different overlays in OpenLayers. For a complete overview, see the roadmap. Feel free to propose new features, or help out creating them.

An early beta of the next big release of the Maps and Semantic Maps extensions is now available for download. The most notable new features in this release are tag extension support for all the Maps parser functions, more consistent error handling via Validator 0.4 and compatibility with the upcoming MediaWiki 1.17. Most changes are internal improvements, but there are also several bug fixes and minor new features.

Both extensions need some more polishing work before the 0.7 release, some testing needs to be done, and some small new features might still be added. This is a list of changes so far:

Maps

New features

Tag support for these parser hooks (which previously only had parser function support):

• Coordinates
• Distance
• Finddestination
• Geocode
• Geodistance

Bug fixes

• Fixed compatibility with the MW 1.17 resource loader.
• Fixed i18n issue with the overlays control for Google Maps v2 maps.
• Fixed default zoom level for Yahoo! Maps maps.
• Increased the maximum decimals for DMS coordinates from 2 to 20.

Removed features

• #geocodelong and #geocodelat parser functions – you can obtain their functionality using #geocode.

Internal improvements

• Rewrote the geocoding functionality. It’s now an integral part of the extension that can not be just pulled out, while the reverse is true for individual geocoders. Geocoder interaction now uses the same model as mapping service interaction.
• Use of Validator 0.4, allowing for more robust and consistent error reporting.
• Rewrote the parser hooks to use the ParserHook class provided by Validator.
• Restructured the directory structure of the extension to better match it’s architecture.

Semantic Maps

New features

• Added ‘locations’ parameter to the map query printers that allows for displaying static locations in addition to query results in queries. It behaves the same as the locations parameter in display_points.

Semantic Maps displaying the result of a query on an OpenLayers map together with a static point.

Bug fixes

• Fixed compatibility with the MW 1.17 resource loader.

Internal improvements

• Use of Validator 0.4, allowing for more robust and consistent error reporting.
• Restructured the directory structure of the extension to better match it’s architecture.

Notice

Although the big features should work, this release will probably contain multiple issues. Please report any you might find.

For the most current version of this announcement, see the original on the mapping wiki.

Maps and Semantic Maps 0.6.4 are now available for download. This release contains several new features, amongst which basic KML support for Google Maps, a new OSM service implementation and re-added service links support to the Geographical Coordinates data-type in Semantic Maps. Everyone running 0.6.2 or older is advised to upgrade as soon as possible.

Lets have a closer look at some of the changes:

• The Google Maps (v2) service now has a KML parameter, which allows you to specify one or more KML files to use. Files are separated with comma’s. The base for this feature was submitted by Andy Chow, who created it for Transit Unlimited wiki. A neat example of a map there using KML is this one. Thanks for the patch Andy!
• There again is an OpenStreetMap service, although this one is different from the one that got removed in 0.6. The implementation is done by embedding (in an iframe) a page from the WMF toolserver, which renders a map with OSM tiles and Wikipedia overlay. Example. It’s basically the same what’s done on the German Wikipedia. Credits here go to Tim Alder and all other people who are making the awesome toolserver stuff work. The reason the creation of the map is not done on the server running the MediaWiki installation itself is that it’s rather difficult to fetch the Wikipedia layer from the toolsever. Hopefully a solution is found for this, as that would be a lot cooler for non WMF usage.
• I re-added service links support for the Geographical Coordinates data-type in Semantic Maps after someone requested this. It got removed in 0.6, as the old implementation was not working anymore, and I was unsure at that point how to create a new one. You can find the updated documentation on this here, although the examples are not working yet (semantic-mediawiki.org is using an alpha of 0.6). I will migrate the examples to the mapping documentation wiki at some later date.
• OpenLayers i18n (translation of the controls) should now be working. A style fix has also been made which caused OpenLayers to render badly in Chrome.
• Multiple small bugfixes have been made in both extensions, and a few in Validator. I guess this is now be the most stable release to date yet, so be sure to upgrade!

For a full list of changes since 0.6.3 see changes to Maps and changes to SM.

Downloads

• Maps 0.6.4: zip, 7z
• Maps and Semantic Maps 0.6.4: zip, 7z

See also

• The release notes on the documentation wiki

Andy Chow

I’ve been working on Maps and Semantic Maps 0.6, the next big, awesome, release of both MediaWiki extensions, for over a month now. I also released an early alpha on the 3th of April. All planned new features have been implemented, and known bugs from 0.5.x have been fixed. So you might wonder why 0.6 is still not out.

In response to the possibility of someone doing work on Maps and Semantic Maps during the upcoming Google Summer of Code, I took a critical look at the current structure, holding into account all the things I learned over the last two month while working on Storyboard. I came to the conclusion that a bunch of things ought to be handled in a cleaner fashion, and started to refactor this code. The most difficult part here is changing how the display_map and display_point(s) parser functions handle their mapping service parameter and validate the provided location(s). I’ve been wanting to change this since 0.4, but didn’t since it’s rather tricky to do. I decided to finally get this done now, and have done most of the work. To complete these changes, I’ll have to make some rather complex modifications to Validator, which can take a while to complete. That’s the last thing that needs to be done before the 0.6 release though

I estimate this should be done in approximately 2 weeks, maybe sooner. After that I’m planning to release at least one RC, to ensure stability and complete awesomeness, cause really a lot has been changed. I figure about three quarters of the codebase has been changed in one way or another!

After this release a lot of cool new features can be added, which I might do before the GSoC coding period starts. There are also a few people who wanted to do a mapping project for GSoC at the WMF, but did not get accepted, which are planning to do some effort here after all, which is totally awesome

Yesterday I released Maps and Semantic Maps 0.5.4. This is a minor update which mainly brings additional stability and security to the 0.5.x branch. It contains several bug fixes, a few code improvements and some security fixes. Check out the Maps change log and Semantic Maps change log.

The only new functionality is that you can now define OpenLayers layers in your LocalSettings file, as well as layer groups. This is done by modifying the $egMapsOLAvailableLayers, $egMapsOLLayerGroups and $egMapsOLLayerDependencies arrays, which are defined in Maps_Settings.php.

Another noteworthy point is that Semantic Maps now contains all the geographical coordinate format handling that was part of Semantic MediaWiki earlier on. The code has been removed from SMW itself, and the SMW 1.5 release will be the first one without it. A nice side effect of this is that people still using Semantic Google Maps (one of the extensions Maps and Semantic Maps are based on, which is now obsolete (and really should not be used any more!)) will be forced to switch over to Maps and SM if they want to get the latest SMW.

Both extensions now use Validator 0.2.2, which is a minor update to 0.2.1, which was used in 0.5.3.

Downloads

• Maps 0.5.4 [zip, 7z, svn tag]
• Maps and Semantic Maps 0.5.4 [zip, 7z]
• Semantic Maps [svn tag]
• Validator 0.2.2 [zip, 7z, svn tag]

Today I released Validator version 0.2, on which I’ve been working the last 2 days. It features massive rewriting to make it more flexible, and has some added functionality. Let’s have a look at what changed.

The most important change is, without any doubt, the new list support. Validator 0.1 had a list type, which allowed you to have enumerations of values and do crude validation on them. This version now supports lists of a type, instead of seeing list as a type. So you can now create lists of strings, lists of integers, and even lists of custom types you add. This new approach also allows per-item-validation and per-item-defaulting. This means you can set an in_array criteria, which will then be enforced for every value in your list. Closely related to this new form of list support are the new list criteria, which allow you to validate lists as a whole. At the moment the only 2 build in list criteria are item_count and unique_items, but like for regular criteria, you can hook into Validator and add your own.

Another important change are the output formats. Output formats allow you to specify additional formatting that needs to be done with the parameter value, before it is retrieved from Validator. There are currently 6 build in types, which are array, list, boolean, boolstr, string and unique_items, but again you can hook into this list via Validator. The awesome thing about output formats is that it greatly reduces the mess you otherwise have with converting your parameters from user input to true data structures. You can even specify multiple output formats, which will then do their formatting one by one.

Other things brought by Validator 0.2 include some new criteria (is_boolean, has_length and regex), a new error level, Validator_ERRORS_MINIMAL, new parameter types (boolean, number and char) and support for Validator_ERRORS_WARN in ValidatorManager.

Equally important as the changes made is that the documentation has been completely updated, to give in-depth cover of how Validator works, and how you should use it.

Both Maps and Semantic Maps 0.5.1 use Validator 0.2, allowing them to throw away a lot of repetitive, dumb manipulation, code that has been their since the initial versions of those extensions

Downloads:

• Validator 0.2 (zip)
• Validator 0.2 (7z)
• Validator 0.2 SVN tag