It’s been a while since I posted an update on Semantic MediaWiki and extensions in general, and my work specifically. This is due to a pile of work that has been done on different components, each of which I’ll address at the point it’s released. In this blog post I’ll provide you with a short overview of what”s (been) going on in the SMW world.
Almost two years after the latest big release, SMW 1.5, 1.6 comes with many important internal changes focused on performance, stability and extensibility and several new features. Here you have an extract of the release notes as they currently are on SVN trunk:
* Full support for synchronizing RDF stores with SMW, and for answering #ask queries based on this data. The communication happens via SPARQL (1.1), and all SPARQL-capable stores should be supported.
* The Type namespace has been abolished. Builtin types now are displayed by the special page Special:Types, and there are no “custom types” any longer. By default, the Type namespace is gone and existing pages in this namespace can no longer be accessed. This can be changed by setting $smwgHistoricTypeNamespace = true in LocalSettings.php before including SMW.
* Changed the way in which units of measurement work. Type:Number now does not accept any units, and a new type “Quantity” is used for numbers with units. Units must be declared on the property page (not on the Type page as before), and only units that have a declared conversion factor are accepted.
* The declaration of Type:Record properties has changed. Instead of a list of datatypes, the declaration now requires a list of properties that are to be used for the fields of the record. The declaration is still done with the property “has fields” as before.
* Changed the way parameters in query printers are specified and handled using the Validator extension. This includes improvements to the parameter options in the Special:Ask GUI and better error reporting for ask queries.
* Added UNIX-style DSV (Delimiter-separated values) result format.
* Reworked internal data model, cleaning up and re-implementing SMWDataValue and all of its subclasses, and introducing new data item classes to handle data. The class SMWCompatibilityHelpers provides temporal help for extensions that still depend on the old format and APIs.
* Fixed PostGre SQL issues with the installation and upgrade code.
* Added API module (smwinfo) via which statistics about the semantic data can be obtained.
That’s a lot of awesomeness no? 
As you can deduce from the above notes, this release is not fully backwards compatibility with SMW 1.5.x, so it’s possible you’ll need to do some migration work. The Validator extension is also introduced as an extra dependency, but it will come bundled with SMW, so you’ll only need to care about this when getting the code from SVN.
SMW 1.6 has been in testing phase for 2 weeks or so now, and most bugs have been taken care of. With some luck, the new version will be released in a week or two Do feel free to try out the new version on non-critical wikis and report any issues you might find.
I already stated that SMW 1.6 is not fully feature compatible with SMW 1.5.x, but it’s also most definitely not compatible with earlier versions code-wise for extensions. This means that quite some SMW extensions released before the development on SMW 1.6 started won’t be compatible anymore, and will require you to also update them to their latest release when upgrading SMW to 1.6 or later. The ones that are in the Semantic Bundle are all compatible already on SVN trunk, so you should be able to upgrade everything as soon as SMW 1.6 is released.
Maps and Semantic Maps 1.0
Another very significant release is the one of Maps and Semantic Maps. I’ve been working on this version for quite a while; it was branched from version 0.7.3, and it’s the most significant (and awesome) release since the creation of both extensions, hence the bump from 0.7.x to 1.0. These are the changes:
* Added full Google Maps v3 support and set it as the default mapping service.
* Added new geocoder making use of the new GeoNames API.
* Added support for the auto-documentation features for parser hooks introduced in Validator 0.4.3.
* Added resizeable parameter to all mapping services except OSM.
* Removed compatibility with pre MediaWiki 1.17.
* Removed overlays parameter for Google Maps v2.
* Usage of the Resource Loader for all scripts and stylesheets.
* Rewrote all the map JavaScript to jQuery plugins.
* Rewrote the way parameters are translated to JavaScript. Now one big PHP object is json_encoded.
* Improved KML formatter.
* Use of Google Maps geocoding service v3 instead of v2.
* Fixed geocoding service overriding based on mapping service (merged in from Maps 0.7.5).
* Fixed fatal error occurring when running maintenance/refreshLinks.php.
* Improved default width of maps (merged in from Maps 0.7.5).
* Improved map query parameter support in the Special:Ask GUI
* Rewrote the map printers to use the SMQueryHandler class.
* Added geocoding capability to the OpenLayers form input when having a GeoNames API account.
* Added ‘update map’ button to all form inputs.
This release of the mapping extensions requires MediaWiki 1.17 or later and the new SMW 1.6, or later. For people not running an MW older then 1.17 (which currently is still not released, pretty much blocking this release of Maps and SM), the 0.7.x branch still remains supported for a while. It’s currently at 0.7.6, and I plan to release 0.7.7 soonish. Do note that Semantic Maps 0.7.x is NOT compatible with SMW 1.6 or later, it needs SMW 1.5.1 – 1.5.7 alpha.
Semantic Forms 2.2
Yaron Koren has been working on the next big release of the most popular SMW extension, Semantic Forms. It brings compatibility with SMW 1.6, and adds several new features, including:
- #autoedit parser function that allows creating a link that, when clicked, automatically sets one or more fields in another page to certain values.
- “Save and continue” button
- Handling of boolean properties can now also be done using radiobuttons and dropdowns, instead of only checkboxes.
Semantic Watchlist
Semantic Watchlist is a new SMW extension I’ve developed for the IEEE as WikiWorks consultant. Semantic Watchlist enables users to watch semantic properties by adding a new watchlist page that lists changes to these properties. Users can choose to follow one or more watchlist groups, which are administrator defined, and cover a set of properties and a set of pages (category, namespace, or SMW concept). Notification of changes to watched properties is also possible via email. I think it’s totally awesome.
It’s main features are:
- A watchlist page listing changes to properties watched by the user.
- Per-user optional email notification per edit that changes properties.
- Integration with user preferences to allow users to specify which watchlist groups they want to follow, and if they want to receive emails on changes.
- Special:WatchListConditions as administration interface for watchlist groups.
- API module to query property changes grouped by edit for a single user.
- API modules to add, modify and delete the watchlist groups.
It requires MediaWiki 1.17 or later and SMW 1.6 or later, and still has to see an initial release. It’s pretty much ready for it, and can be seen as beta right now.
The Spring 2011 SMWCon was held on April 28-30, 2011 at the Raytheon BBN Technologies office in Arlington, Virginia, in the Washington, DC area, and it was a great success. You can read more about it in Yarons writeup.
The next SMWCon, SMWCon Fall 2011, will be held on September 21–23, 2011 in Berlin, Germany. Berlin – yay! It’s going to be awesome, and I’ll be attending, probably giving some talk about Maps and Semantic Maps, and possibly other extensions as well (Semantic Watchlist being a good candidate).
Like I already noted, I’ll be posting more comprehensive (and official) release announcements for each extension when they are released I’d also like to point out that this is definitely not everything that’s been going on in the SMW world. For example there are 2 Google Summer of Code students doing SMW related work, about which I might write later on, and many people are doing SMW projects that I’m simply not aware of or am not closely following.
Categories: Programming, Software
Yesterday I quickly wrote up a simple (but awesome) MediaWiki extension that allows you to make use of the Spark library in your wiki.
Spark as described on the Spark website:
The web is not only growing in sheer size, but it also grows in how much it is interconnected. Where once the Web was a set of more or less separated sites, today sites are more and more being connected. More and more data is being offered on the Web in a way that can be further processed, and more and more sites and applications are using external data. More and more mashups are created, where data from different sources is integrated and displayed with novel visualisations.
Spark is a library that enables HTML authors to create mashups more easily than ever before. Using standard Web technologies like SPARQL, RDF, HTML5, and JavaScript, Spark can query external knowledge sources (so called triple stores or SPARQL endpoints), and then visualise the results.
With Spark, website developers can create visually appealing mashups without having to write a single line of JavaScript, but merely using some markup elements describing the source of the data that is to be shown, a query to select the appropriate data, and selecting one from an expandable set of visualisations and their parameters.
Spark is developed by Denny VrandeÄić and Andreas Harth.
This MediaWiki extension, unsurprisingly titled Spark, adds a <spark> tag to MediaWiki which is equivalent to <div class=”spark”> as described in the spark library documentation. All parameters (except the class=”spark” one) can just be copied over between spark divs in web pages, and the <spark> tag in MediaWiki. It is currently at version 0.1, which is a beta release. It includes a still experimental version of the Spark library, so you should probably not use this extension on production websites just yet. The Spark people are looking for developers to help out, so if you want to play around with SPARQL a bit, like I basically did with this extension, be sure to poke them
The extension required MediaWiki 1.17 or above (as it makes use of the new Resource Loader) and PHP 5.2 or later.
Download
- The current Spark (MediaWiki extension) release (zip)
- svn co http://svn.wikimedia.org/svnroot/mediawiki/tags/extensions/Spark/REL_0_1 (tag for 0.1, including Spark lib trunk)
- svn co http://svn.wikimedia.org/svnroot/mediawiki/trunk/extensions/Spark (trunk, including Spark lib trunk)
- svn co http://rdf-spark.googlecode.com/svn/trunk/src/ (Spark lib trunk)
Further possibilities
Right now you can embed mashups with SPARQL queries that get their data from some SPARQL endpoint. This opens up a whole bunch of possibilities, but is a bit silly when you are running your own Semantic MediaWiki instance and want to visualize structured data stored by it using Spark. A possible addition to the Spark MediaWiki extension therefore is having support for Spark as a so called SMW result format. For this translation from the SMW ask query language to SPARQL is needed, which is some work. I might implement this at some future point, but have several other things I want to poke at, so it won’t be soonish
Categories: Programming, Software
Somewhere in the last two weeks I quickly wrote up a small new MediaWiki extension to include content from Wikipedia or some other MediaWiki install into pages on your wiki. It’s titled Include WP.
The extension does not import anything (so nothing is stored), but rather fetches content from the remote wiki every time the page is loaded. This means it takes a second or two for it to actually load, but ensures you got the latest content. Some clean up of the remote content happens and there are several options that allow you to customize how the included content is displayed. It makes use of the Validator extension.
Feature overview:
- Include articles from Wikipedia or any other MediaWiki wiki into your pages.
- Show only a limited amount of paragraphs on page load, with an option to show the full article.
- Partial conversion to plain-text.
- Removal of templates (such as infoboxes), ref tags, comments, categories and images.
- Both internal and external links are rendered as plain-text.
- Tables, lists, table of content, section headers and more are retained.
- Usage of the MediaWiki Resource Loader when available with backward support for MW 1.16.x.
Some screenshots
Download
- Stable releases (archives)
- SVN tags
- SVN trunk
Categories: Programming, Software
Earlier today I released a new version of the Semantic Result Formats extension. SRF bundles a number of so called “result formats” that can be used in conjunction with Semantic MediaWiki, and allow visualization of queried data.
New tagcloud format
This release introduces a new “tagcloud” format which I created for WikiWorks. It allows displaying queried values in a tag cloud based on how many times they occur in the result set. Several options allow you to specify the minimum and maximum tags sizes, how to increase (logarithmic or linear), how to order the tags and more. The format is enabled by default, so when updating SRF to the latest version, you can immediately use it.
Improved gallery format
Some significant improvements have been made to the gallery format. Previously it was only able to display images that are returned as subjects in the query result. This is useful in some cases, but forces you to have semantic properties on your image pages, which is not a good approach if you want to have this info on the page the images belong to. Now it’s possible to point to images using a page property and then do a query in which you specify this property points to the images you want to display. An option to disable automatic captions has also been added.
And more…
Several other improvements have also been made, including fixes to the jqplot format by Yaron Koren and some clean-up of the timeline and eventline formats.
Download
Since SRF was lacking a place for release downloads, I created a new Google Code project where they are put now.
Categories: Programming, Software
It’s been a while since the previous release of Validator, but this minor update includes some cool new functionality.
As I was creating the SubPageList extension and it’s documentation (at 27c3!), it occurred to me that to document the usage of the <subpages /> parser hook, all the info I needed was already available via Validator. Since there was no way to get this info out in a meaningful way, I decided to create a new parser hook just for this purpose. It’s titled “describe” and describes one or more parser hooks implemented using the ParserHook class provided by Validator. This description includes a short text about the parser hooks functionality, a table with it’s parameters and their meta-data, syntax examples, a list of aliases (if any) and more. It’s possible to get the description in wikitext, as opposed directly in the page, so you can copy it, and use it (or parts of it) somewhere else. By default it’ll list all parser hooks it knows about directly in the page. An example can be seen here.
Many more not yet implemented features that make use of the parameter handling framework that is Validator can be imagined. One I might implement in the near future is linking to a special page that generates a description of how to use a certain parser hook based on something the user is doing wrong, and link to that from error messages. This can be done easily by building on top of the already existing error message system and the describe parser hook
I think Validator adds a lot of Value when creating parser hooks, and other parameter handling objects. The main problem currently is that when using it’s features in an extension, this extension obviously becomes dependent on Validator. From a technical perspective this does not really matter, but it complicates installation for users (especially since MediaWiki lacks any sort of extension management facilities). I therefore hope to get Validators functionality into MediaWiki core, or at least in the default MediaWiki distribution. I’m rather sceptical to if I can make this happen as a non core-developer, but if I succeed, it would be a great advantage for extension (and core) development, so it’s definitely worth a try.
SubPageList 0.1 comes bundled with an alpha version of Validator 0.4.3, containing most of the new functionality. Maps and Semantic Maps will come bundled with it (or a newer version) on their next release.
Downloads
- Zip and 7z archives
- SVN tag
- SVN trunk
- There will be a new Semantic Bundle soonish which will include the latest Validator.
Categories: Programming, Software
During 27c3 someone asked me to install an extension to list subpages onto the hackerspace.be wiki. I picked SubPageList3, as it seemed to most decent one. I did a simple test to see if it was working, and immediately found a namespace bug. So I decided to quickly rewrite it and also let it use Validator for additional awesomeness.
Strangely enough, although there where SubPageList2 and SubPageList3 extensions, there was no SubPageList extension to be found anywhere in the WMF SVN repo or the mediawiki.org wiki, so I decided to simply take that name. The extension should be backwards compatible with SubPageList3.
I spend 5 hours or so doing the rewrite, and have released an initial version yesterday. Seems to work fine and is currently in use on hackerspace.be. It requires Validator 0.4.2 or above, and is the first extension making full use of new the auto-documentation functionality Validator 0.4.3 introduces (more on that in a later blog post).
Download
Categories: Programming, Software
I’m happy to announce the release of a new MediaWiki extension I’ve been working on over the past two weeks. It’s titled Live Translate and allows live page translation via the Google Translate API. It also enables you to define a “dictionary” of certain words or phrases and their translations; any word or phrase in the original text found in the “dictionary” will be translated using that dictionary, instead of using Google Translate.
The main features are:
- Live translation of page contents using Google Translate.
- Ability to define translations of special words in-wiki that will then be left alone by Google Translate.
Some screenshots
A wiki page with the translation control of the Live Translate extension at the right top corner:
The same page after translating it to Dutch:
The dictionary page briefly summarizing how many words and languages it contains:
Editing the special words dictionary works just like editing any other page:
Funding
I created this extension as WikiWorks consultant for Texas Instruments. Thanks to TI for funding this and allowing licensing under the GNU GPL.
Points of interest
For me the most interesting part of creating this extension was figuring out how to walk through the page DOM and send only the actual text to the Google Translate API using JavaScript (and jQuery). This step was needed because the GT API limits translation requests to 500 characters, so it’s not definitely possible to send the whole page. It took me a while to figure this out, and I still think it’s a rather lame limit, as it leads to loss of context, and thus worse translations.
The rest of the extension is rather simple and does not contain anything I haven’t done before. The special words dictionary is stored in a simple db table (fields: word id, word text, word language and word primary) and can be accessed via 2 API modules. One is to query a list of the special words that are defined for a language, the other is to get translations of a set of special words from one language to the other. When you request your first translation of a page, the first API module is hit and the result is used to insert notranslate spans, which make Google Translate ignore stuff, around the special words. After that the other module is hit, the special words are replaced by their translations, and finally the script sends a ton of requests to the GT API.
Ah, and not to forgot, I used a XOR. I kid you not, source code or it didn’t happen (see the last function).
State and future
Live Translate is currently at version 0.2 and contains all the features initially requested by TI. It appears to be stable, and ready for production usage. Of course, if bugs pop up, they will be addressed and a new minor version will be released. A cool new feature I’ve been considering, and might add at some point, is having __LIVETRANSLATE__ and __NOLIVETRANSLATE__ magic words, that allow per-page showing or hiding of the translation control. Yaron suggested also having a per-namespace setting. If you have some cool new features in mind, feel free to suggest them on the Live Translate discussion page (where you also can ask for support and point out bugs). If you want to fund any new functionality, please contact WikiWorks.
You can obtain the latest version of Live Translate here.
Categories: Programming, Software
I’m happy to announce the release of a new MediaWiki extension I’ve been working on over the past two weeks. As you’ve might already have guessed from it’s name, Push, it enables you to push content of wiki pages to one or more other MediaWiki installs.
The main features are:
- Pushing page content to other wikis via a tab on the page.
- Bulk push via Special:Push.
- Remote authentication support.
- Automatic transfer of included files.
- Support for ApprovedRevs. If there is an approved revision, it will be pushed, if not, the latest one is pushed.
- AdminLinks integration.
- Usage of the new MediaWiki Resource Loader when available with backward support for MW 1.16.x.
The tab interface
When logged in, a new tab or action (on vector based skins) will be added which leads to the push interface for the page you are on. This interface consists of a table listing the available target wikis and also informs you of the status of the target pages. A big push button in each row allows you to do the actual push. When there is more then one target wiki, a convenient ‘Push all’ button will also be shown.Underneath this table the available options will be displayed. With the current version you can choose to include the templates used on the page in the push, as well as transferring the embedded images.
In this screenshot of the tab interface, you can see an actual ‘Push’ tab, rather then an action in the collapsed actions dropdown. You can choose this behaviour, which is useful when you do very frequent pushing, using one of the settings made available by the Push extension.
After initiating the push to one or more targets, the work will happen in the background, and the interface will update to show you progress, completion, and possible errors (such as not having sufficient rights to edit on the target wiki).
Special:Push
Push provides bulk push capabilities via a special page aptly titled ‘Special:Push’. The layout and workings of this page should be familiar to anyone having used MediaWikis native Special:Export, as it’s largely based on this special page. A big textbox allows you to specify the pages you want to push (one per line), and you can choose to add all pages from a category or namespace to it. Under the texbox are the same options as in the tab interface: automatic inclusion of templates and files. Finally you are able to select one or more target wikis.
Note that the file inclusion option was not added yet in the version the below screenshot was taken at. I obtained the above list of pages by entering ‘Locations’ in the category box and submitting it.
When submitting the push request, you’ll be shown the pushing progress in the form of a list to which items get added as pushes complete.
Funding
I created this extension as WikiWorks consultant for Texas Instruments. Thanks to TI for funding this and allowing licensing under the GNU GPL.
Status and future
Push is currently at version 0.5 and contains all the features requested by TI. It appears to be stable, and ready for production usage. Of course, if bugs pop up, they will be addressed and a new minor version will be released. Many useful additions to push can be imagined for various use-cases. Feel free to suggest them on the Push discussion page (where you also can ask for support and point out bugs). If you want to fund any new functionality, please contact WikiWorks.
You can obtain the latest version of Push here.
Categories: Programming, Software
An early beta of the next big release of the Maps and Semantic Maps extensions is now available for download. The most notable new features in this release are tag extension support for all the Maps parser functions, more consistent error handling via Validator 0.4 and compatibility with the upcoming MediaWiki 1.17. Most changes are internal improvements, but there are also several bug fixes and minor new features.
Both extensions need some more polishing work before the 0.7 release, some testing needs to be done, and some small new features might still be added. This is a list of changes so far:
Maps
New features
Tag support for these parser hooks (which previously only had parser function support):
- Coordinates
- Distance
- Finddestination
- Geocode
- Geodistance
Bug fixes
- Fixed compatibility with the MW 1.17 resource loader.
- Fixed i18n issue with the overlays control for Google Maps v2 maps.
- Fixed default zoom level for Yahoo! Maps maps.
- Increased the maximum decimals for DMS coordinates from 2 to 20.
Removed features
- #geocodelong and #geocodelat parser functions – you can obtain their functionality using #geocode.
Internal improvements
- Rewrote the geocoding functionality. It’s now an integral part of the extension that can not be just pulled out, while the reverse is true for individual geocoders. Geocoder interaction now uses the same model as mapping service interaction.
- Use of Validator 0.4, allowing for more robust and consistent error reporting.
- Rewrote the parser hooks to use the ParserHook class provided by Validator.
- Restructured the directory structure of the extension to better match it’s architecture.
Semantic Maps
New features
- Added ‘locations’ parameter to the map query printers that allows for displaying static locations in addition to query results in queries. It behaves the same as the locations parameter in display_points.
Semantic Maps displaying the result of a query on an OpenLayers map together with a static point.
Bug fixes
- Fixed compatibility with the MW 1.17 resource loader.
Internal improvements
- Use of Validator 0.4, allowing for more robust and consistent error reporting.
- Restructured the directory structure of the extension to better match it’s architecture.
Notice
Although the big features should work, this release will probably contain multiple issues. Please report any you might find.
For the most current version of this announcement, see the original on the mapping wiki.
Categories: Programming
Just under a month after the 0.6.5 release of both mapping extensions, the next minor update, 0.6.6, is available for download. No spectacular new features, but several important bugfixes. Several issues with coordinate parsing have been fixed, you can now using geocoding when behind a proxy, and wikitext should finally(!) behave correctly in marker pop-ups. Some internal changes have also been made, mainly rounding off the many changes I made in the 0.6.x branch. I expect this release to be the most stable one to date, and have therefore changed the extensions status from ‘beta’ to ‘stable’ on the documentation pages.
A lot of improvements have been made to the documentation as well. Both the Maps examples and Semantic Maps examples are now comprehensive and complete. There now are finally examples of using query templates, of distance queries and of some nice compound queries. Some more work is needed though, a lot of which is explaining basic functionality and fixing minor issues all over the place. I’ll be taking care of the most important things, but I’d be great if people using the extensions could help me out improving the documentation 
This release is probably the last one before 0.7, in which I expect to be focusing on new functionality. I’m looking for people that want to fund the development of new features, so please contact me if you are such a person
Downloads:
Categories: Programming
Previous Entries
Categories
Tag Cloud
Blog RSS
Comments RSS
Last 50 Posts
Back
Void 
Life 
Earth 
Wind 
Water 
Fire