Jeroens blog

Maps and Semantic Maps 0.6.5 are now available for download. This release contains mainly internal changes to improve code modularity and fix some security concerns. Several bugs have been fixed as well, and a new hook has been added to Semantic Maps. This hook will get you the map format as default one for queries where you only ask for coordinates when using SMW 1.5.2 or above. For a full list of changes since 0.6.4 see changes to Maps and changes to SM. Everyone running 0.6.2 or older is advised to upgrade as soon as possible. 

This release is notable for it being the first one in which I’m happy with the code-base as a whole. It took me a year to get here, but now I think the way the mapping extensions work is good and solid. This means you can now extend Maps and not be afraid the code will be incompatible in a few weeks due to changes. This also means that I’ll be focusing more on actual functionality rather then refactoring in future releases. I’ll be progressively building a little guide that explains how the extensions work from a developers perspective and how to extend them.

I might release another minor update in the 0.6.x series if any significant issues are found in 0.6.5. Further plans are finishing up a bunch of changes I’ve started to make in Validator, which I’ll probably release as 0.4 then, and to start working on Maps and Semantic Maps 0.7, which would aim at adding new features and improving existing ones. A likely new feature I’m particularly looking forward to implementing is several tag extensions that do the equivalent of the current parser functions added by Maps. The timetable for all this depends a lot on which other things I get cough up in (I’ll probably continue putting effort into the deployment stuff for my GSoC project) and what kind of funding will be available.

Downloads:

• Maps 0.6.5 [zip - 7z]
• Maps and Semantic Maps 0.6.5 [zip - 7z]

You can also view the release announcement at the documentation wiki.

Last 3 days I’ve been at the 2010 edittion of Wikimania, one of the biggest yearly wiki events. This year it was in Gdansk, Poland. I went by Plane from Eindhoven, together with Maarten, a moderator on the Dutch Wikipedia with >5 year history, who drove me to the airport.

I’ll start with the things that where not nice, basically the heat and the dormitories. It was 30+ °C practically the whole time, and the dormitory I stayed in, which was reserved by the event organizers for the attendees, was more like a sauna then a dormitory. It was located right next to a viaduct, so when opening the window at night, you’d get a lot of noise. On the last night the people from the event could not stay at that dormitory, and had to go somewhere else. Me and another attendee from Belgium got assigned some dormitory a few km away from the event. Saying this was a crappy one is an understatement. Furthermore, the payment was not arranged, so we had to pay for staying, while it should have been done by the event organizers as we paid for it?!!  In any case, this amounted to me having less then 10 hours of sleep over 4 nights And they don’t have Club Mate in Gdansk o_O!!!

I think the event itself was great, with lots of interesting talks,  awesome people and good food.

On the first day I missed the keynote since my flight arrived to late to see it, and only followed some non-technical talks, of which the most notable one was about Liquid Threads, the totally awesome extension by Andrew Garrett and now also some other people. I left early, skipping the deinner and following events, and went straight to the dormitory, to get some sleep (which failed >_>)

I gave 2 talks, one about my Google Summer of Code project, now titled Deployment, and one about Maps and Semantic Maps, both on the second day. The Deployment talk went pretty bad, as I was extremely tired. Hopefully I got the core message across of the importance of having a solid and user friendly way of deployment. Apologies to all the people I forgot to give credits to! Daniel Kinzler had some interesting security concerns which we discussed later on the day, together with Markus.

On the second day of the event there was a series of Semantic MediaWiki talks and workshops, kicked off by a presentation about the concept and what SMW has become over the past 5 years by Markus. This was followed by a workshop by Hans-Jörg Happel and Frank Dengler about Semantic Result Formats. Daniel Herzig gave a talk about AskTheWiki, and promising extension he is developing. There where two more SMW related talks focusing on the advantages of SMW and how it can be deployed. Later on the second day I gave my talk about Maps and Semantic Maps, preceded by one of Tim Alder, who presented the capabilities of the geo-related work on the toolserver. Luckily I was awake to some extend during this talk, and it went reasonably well IMO, although I had to rush through it, cause of to little time.

The second day was closed by the word premier of the movie “Truth in Numbers“, which aims at giving people an idea about the goals of the Wikimedia Foundation, and how it works. It shows arguments from both people involved or enthusiastic about the foundation, and those who think the world is going to explode if you go to Wikipedia for whatever reason. It will be interesting to see how this movie gets distributed and what effect it will have. I’m a little sad about how the creators choose to distribute it though, as it will require you to pay for it. It would be a lot more awesome if it was free with a big donate button. All the footage, which is a lot more then what’s seen in the movie itself includes, but not edited, will be released for free under a creative commons (or similar?) licence though.

The third day I found the most enjoyable as I did get some amount of sleep the night before. (In other words, I was awake enough again to program during the talks : ) The most interesting talk for me I followed here was definitely the one by Roan Kattouw about writing MediaWiki extensions, which mentioned some things I did not know about yet (such as build in MediaWiki support for Memcached o_O), as well as provided a nice overview of the things you need to keep in mind. I wish I had seen it a year earlier though, as I would have learned a lot more from it then, and not made a lot of the noted beginner mistakes. In the next session I attended several strategy sessions, which gave me a nice idea of what all the strategy fuss is about. The last session I attended was about Wikimedia credibility, including a talk by Maarten about how information (esp the incorrect) spreads to other media. The other talks in this session where similar, and although obviously none of the issues addresses are technical in nature, I found several of them rather amusing.

It was great to meet all the people involved with SEMANTIC MediaWiki in person, as well as a bunch of people I only knew from IRC and other online communication tools. Sadly enough Yaron Koren and Brion Vibber and several other people I’d like to have seen there could not be at the event.

On the way back to Belgium I tackled a lot of small design issues that have been present in Maps and Semantic Maps for months, resulting in one big refactoring commit for each extension, which I made as soon as I was back home. After that I went to sleep, and woke up 19 hours later O_o

Next years Wikimania will be in Haifa, Israel. I’m probably going, but seriously hope it won’t be so insanely warm there

Maps and Semantic Maps 0.6.4 are now available for download. This release contains several new features, amongst which basic KML support for Google Maps, a new OSM service implementation and re-added service links support to the Geographical Coordinates data-type in Semantic Maps. Everyone running 0.6.2 or older is advised to upgrade as soon as possible.

Lets have a closer look at some of the changes:

• The Google Maps (v2) service now has a KML parameter, which allows you to specify one or more KML files to use. Files are separated with comma’s. The base for this feature was submitted by Andy Chow, who created it for Transit Unlimited wiki. A neat example of a map there using KML is this one. Thanks for the patch Andy!
• There again is an OpenStreetMap service, although this one is different from the one that got removed in 0.6. The implementation is done by embedding (in an iframe) a page from the WMF toolserver, which renders a map with OSM tiles and Wikipedia overlay. Example. It’s basically the same what’s done on the German Wikipedia. Credits here go to Tim Alder and all other people who are making the awesome toolserver stuff work. The reason the creation of the map is not done on the server running the MediaWiki installation itself is that it’s rather difficult to fetch the Wikipedia layer from the toolsever. Hopefully a solution is found for this, as that would be a lot cooler for non WMF usage.
• I re-added service links support for the Geographical Coordinates data-type in Semantic Maps after someone requested this. It got removed in 0.6, as the old implementation was not working anymore, and I was unsure at that point how to create a new one. You can find the updated documentation on this here, although the examples are not working yet (semantic-mediawiki.org is using an alpha of 0.6). I will migrate the examples to the mapping documentation wiki at some later date.
• OpenLayers i18n (translation of the controls) should now be working. A style fix has also been made which caused OpenLayers to render badly in Chrome.
• Multiple small bugfixes have been made in both extensions, and a few in Validator. I guess this is now be the most stable release to date yet, so be sure to upgrade!

For a full list of changes since 0.6.3 see changes to Maps and changes to SM.

Downloads

• Maps 0.6.4: zip, 7z
• Maps and Semantic Maps 0.6.4: zip, 7z

See also

• The release notes on the documentation wiki

Andy Chow

Maps and Semantic Maps 0.6.3 are now available for download. This release is one aimed to improve stability and addresses a number of bugs that where present in 0.6.x. Some notable fixed issues are a failure of parsing any coordinates containing a degree symbol that surfaced in 0.6.2, and the failing of non-Google Maps form inputs. Some internal rewriting has also been done, which was needed to address some of these bugs, and will enable future awesomeness increase. The included OpenLayers library has also been upgraded to 2.9.1. (changes to Maps, changes to SM). People running any 0.6.x are advised to upgrade. Especially people using Semantic Maps in conjunction with Semantic Forms, as both 0.6 and 0.6.1 contain undesired behaviour in the forms.

A new release of Semantic Bundle, which will include these versions of the mapping extensions, is likely to soon be released as well

Downloads

• Maps 0.6.3 (7z)
• Maps and Semantic Maps 0.6.3 (7z)

See also

• The release notes on the documentation wiki

Maps and Semantic Maps 0.6.2 are now available for download.

This release contains improvements to parameter handling, case insensitivity has been re-added, and using default parameters together with named ones will work more intuitive. In Maps the handling of distances has also been considerably improved: there is a new #distance parser function, additional distance related parameters in #geodistance, and new settings that allow you to customize the default distance behaviour. In Semantic Maps a rather important bugfix has been made which solves a problem with editing articles with forms that contain maps that arose in 0.6.1. (changes to Maps, changes to SM). People running 0.6 and 0.6.1 are advised to upgrade. Especially people using Semantic Maps in conjunction with Semantic Forms, as both 0.6 and 0.6.1 contain undesired behaviour in the forms.

Read this announcement on the documentation wiki

Maps and Semantic Maps 0.6.1 are now available for download. This release does not add any new features to 0.6, but contains multiple important bugfixes (Maps, SM). People running 0.6 are advised to upgrade. Especially people using Semantic Maps 0.6 in conjunction with Semantic Forms, as the mapping form inputs in 0.6 will output invalid coordinates when editing existing ones or inserting new ones.

Maps and Semantic Maps 0.6 are now available for download. Maps 0.6 requires Validator 0.3, which is included in the release distribution, and can also be found on SVN. Semantic Maps 0.6 requires Maps 0.6, Validator 0.3 and Semantic MediaWiki 1.5.1 or above. See the download page for full dependency and compatibility tables.

This is a big update, including a lot of new features, bug fixes, security patches, and most of all, internal improvements, making both extensions more modular and extendible (these changes are not covered here, see the relevant change logs for more info). It is also the first release of Semantic Maps that requires you to run the SMW update script, as it requires a new table layout to store coordinates (more info on this).

Let’s have a look at the various new features.

• Maps now supports real coordinate parsing and formatting, which allows you to input coordinates in any of the supported notations at any point in the extension, and can also request any output to be in the notation of your choice. This means you can now choose what format Semantic MediaWiki shows coordinates in, such as in ask queries. The supported notations are DMS, decimal degrees, decimal minutes and floats. All those can be either directional or non directional. A new parser function has been added that allows you to convert between any of these formats: #coordinates.

• New geographical functions: #geodistance and #finddestination. You can use the #geodistance parser function to calculate the geographical distance between two points, from and to any of the supported formats. The #finddestination parser function can be used to find a destination given a starting point, an initial bearing and a distance.

• Related to the new geographical functions is a rewritten distance query in Semantic Maps. It now takes into account performance and is scalable, which the old query was not, by using the new storage structure for coordinates. The notation for distance queries has also changed. Instead of using the like operator and a global distance parameter ( like #ask:[[Property: :~coordinates]]|distance=42 ) you now only have to specify your distance locally in the coordinate criteria itself like #ask:[[Property: :coordinates (42 km)]]. Like you can see you can now also specify a unit, which can be any of the supported ones.

• Support for various width and height notations. Previously Maps only accepted width and height values is px, forcing you to use maps of fixed sizes. Since most people want to have their complete page width visible even on small screens, this resulted in a lot of people using rather small maps, and so wasting screen space. 0.6 allows you to specify the size in px, ex, em, and most importantly, in %. The syntax is what you’d expect: width=”420px”, width=”420em”, width=”42%”. width=”420″ will default to using px, so is backward compatible. When using the % values, maps will even adapt their size when the screen width or the height of the container they are in is changed after the page has loaded

• Not a new feature, but rather one that’s removed: OSM support. The OSM service has been completely removed from Maps and Semantic Maps as it was rather broken and not easy to upgrade to the internal structure of Maps 0.6. I’m planning to add it back later on, rewritten from ground up, in 0.6.1 or 0.6.2 or so. Note that you can still view OSM maps on your wiki using the OpenLayers service, which has build in OSM layers, and also allows you to define your own layers since 0.5.5.

• And many more

The most notable bugfixes are:

• Fixed conflict with prototype library that caused compatibility problems with the Halo extension.
• Added automatic icon image sizing for Google Maps and Yahoo! Maps markers.
• Various security fixes, mostly preventing XSS attacks.

If no serious bugs are found in this release, a minor update can be expected in a month or so.

View announcement on the mapping wiki.

Downloads

• Maps 0.6 [ zip, 7z ]
• Maps and Semantic Maps 0.6 [ zip, 7z ]

One of the big changes in the upcoming 0.6 release of Semantic Maps will be the from the ground up rewritten semantic datatype for Geographical coordinates. Although the changes themselves do not directly add any value for the user, they enable some pretty neat improvements to existing features, and the creation of many new ones. In this blog post I’ll first go over the changes that are made, in a technical manner, and then sum up the effects they have from an users point of view.

The SMWDataValue extending class for coordinates, which was previously part of Semantic MediaWiki, and only recently moved over to Semantic Maps, has had a big overhaul. This class, now called SMGeoCoordsValue, used to parser coordinates all by itself. Considering Maps also parses coordinates, this is pretty dumb, since the code is redundant, and even worse, does not behave the same way, resulting into incompatible output. I rewrote all the parsing and formatting code in an as modular fashion as possible, and put it in a new class in Maps, which is MapsCoordinateParser. SMGeoCoordsValue now does all formatting and parsing of values via this class

Markus recently added a hook to Semantic MediaWiki that allows for extensions to define their own database tables for storage of semantic values with a specific signature, which will be part of the upcoming SMW 1.5.1. Semantic Maps is using this hook for storing coordinates in a table with lat and lon fields, both floats. This is a huge improvement to simply storing the coordinates in non-directional float notation with the elements separated by a comma, in a string field. The new layout allows for much faster lookups since the values can be index, esp. for things like the soon to be re-implemented distance query, and opens up a lot of new possibilities.

As a user, you will not notice any of these changes, except maybe that things that where broken will be magically fixed when using this new code. Semantic Maps now has settings that allow you to specify the output format of coordinates in queries though, so you can now view them in a different format then non-directional floats, like you where obliged to in the previous releases. You can pick any format supported by Maps, for example directional DMS, as show on the screenshot. Another change is that you will see new functionality based on the new table layout, like, but not limited to, the distance query.

Edit: There actually is one thing you’ll definitely notice as a user when upgrading – you need to update your semantic data via Special:SMWAdmin in order for your wikis coordinates to be stored in the new table. If you don’t do this, queries for coordinates won’t return anything. It’s sort of annoying you need to do this, but there is no way around it.

I’ve been working on Maps and Semantic Maps 0.6, the next big, awesome, release of both MediaWiki extensions, for over a month now. I also released an early alpha on the 3th of April. All planned new features have been implemented, and known bugs from 0.5.x have been fixed. So you might wonder why 0.6 is still not out.

In response to the possibility of someone doing work on Maps and Semantic Maps during the upcoming Google Summer of Code, I took a critical look at the current structure, holding into account all the things I learned over the last two month while working on Storyboard. I came to the conclusion that a bunch of things ought to be handled in a cleaner fashion, and started to refactor this code. The most difficult part here is changing how the display_map and display_point(s) parser functions handle their mapping service parameter and validate the provided location(s). I’ve been wanting to change this since 0.4, but didn’t since it’s rather tricky to do. I decided to finally get this done now, and have done most of the work. To complete these changes, I’ll have to make some rather complex modifications to Validator, which can take a while to complete. That’s the last thing that needs to be done before the 0.6 release though

I estimate this should be done in approximately 2 weeks, maybe sooner. After that I’m planning to release at least one RC, to ensure stability and complete awesomeness, cause really a lot has been changed. I figure about three quarters of the codebase has been changed in one way or another!

After this release a lot of cool new features can be added, which I might do before the GSoC coding period starts. There are also a few people who wanted to do a mapping project for GSoC at the WMF, but did not get accepted, which are planning to do some effort here after all, which is totally awesome

During the past two weeks I’ve been putting some effort in making some big changes to the source of Maps and Semantic Maps, and adding some awesome new features. Although not every to-do for the 0.6 release has been completed, I’ve made an early alpha version available which allows you to try out some of the exiting new functionality. So let’s have a look at the most notable changes since 0.5.5.

New features

• Support for various width and height notations. Previously Maps only accepted width and height values is px, forcing you to use maps of fixed sizes. Since most people want to have their complete page width visible even on small screens, this resulted in a lot of people using rather small maps, and so wasting screen space. 0.6 allows you to specify the size in px, ex, em, and most importantly, in %. The syntax is what you’d expect: width=”420px”, width=”420em”, width=”42%”. width=”420″ will default to using px, so is backward compatible. When using the % values, maps will even adapt their size when the screen width or the height of the container they are in is changed after the page has loaded See my wiki for some cool examples. Although this is really basic functionality in a way, it’s awesome to finally have it available.
• Added full support for DMS, DD, DM and float coordinate notations, both their directional and non-directional variants. This means you can now enter coordinates in these notations in the display_point(s) and display_map parser functions, as well as make Semantic Maps recognize them as geographical coordinates. Smart geocoding support has also been added to #geocode, meaning you can now pass along coordinates as well as addresses, so you don’t have to double check if your passing along an address to prevent losing the value.
• Coordinate formatting support. Along with the smart geocoding support, #geocode now also accepts 2 parameters to specify the output notation. Since you can pass along coordinates, this means you can change the notation of a set of coordinates using #geocode. A new parser function, #coordinates, has been added specifically for this purpose, so for pure formatting, you should use this one rather then #geocode. Both parser functions work with a hybrid parameter system that supports several ‘default’ parameter (ones that consist of only a value, and no name) and named parameters. You can work with names or without for the default ones, and the default ones in #geocode are the same as the old parameter order, so backward compatibility is retained.
• Rewrote map html and js output. This is now done in a cleaner, safer and more consistent way, for all mapping services in both Maps and Semantic Maps. This is not a new feature, but such a big refactoring it’s worth noting.

More to come…

• Native support for geographical proximity queries in Semantic Maps. This will require quite some work to do in a clean and efficient way, so will probably one of the last features finished for 0.6.
• OpenLayers 2.9 support. Since Maps 0.1, it has been supporting OpenLayers 2.8, and 2.9 is just about to be released, so It’ll be nice to see the progress made there.

On top of these changes to the extensions, I’m drastically redoing the documentation and examples. I’ll post more about that later on though