Jeroens blog

Maps and Semantic Maps 0.6.1 are now available for download. This release does not add any new features to 0.6, but contains multiple important bugfixes (Maps, SM). People running 0.6 are advised to upgrade. Especially people using Semantic Maps 0.6 in conjunction with Semantic Forms, as the mapping form inputs in 0.6 will output invalid coordinates when editing existing ones or inserting new ones.

Maps and Semantic Maps 0.6 are now available for download. Maps 0.6 requires Validator 0.3, which is included in the release distribution, and can also be found on SVN. Semantic Maps 0.6 requires Maps 0.6, Validator 0.3 and Semantic MediaWiki 1.5.1 or above. See the download page for full dependency and compatibility tables.

This is a big update, including a lot of new features, bug fixes, security patches, and most of all, internal improvements, making both extensions more modular and extendible (these changes are not covered here, see the relevant change logs for more info). It is also the first release of Semantic Maps that requires you to run the SMW update script, as it requires a new table layout to store coordinates (more info on this).

Let’s have a look at the various new features.

• Maps now supports real coordinate parsing and formatting, which allows you to input coordinates in any of the supported notations at any point in the extension, and can also request any output to be in the notation of your choice. This means you can now choose what format Semantic MediaWiki shows coordinates in, such as in ask queries. The supported notations are DMS, decimal degrees, decimal minutes and floats. All those can be either directional or non directional. A new parser function has been added that allows you to convert between any of these formats: #coordinates.

• New geographical functions: #geodistance and #finddestination. You can use the #geodistance parser function to calculate the geographical distance between two points, from and to any of the supported formats. The #finddestination parser function can be used to find a destination given a starting point, an initial bearing and a distance.

• Related to the new geographical functions is a rewritten distance query in Semantic Maps. It now takes into account performance and is scalable, which the old query was not, by using the new storage structure for coordinates. The notation for distance queries has also changed. Instead of using the like operator and a global distance parameter ( like #ask:[[Property: :~coordinates]]|distance=42 ) you now only have to specify your distance locally in the coordinate criteria itself like #ask:[[Property: :coordinates (42 km)]]. Like you can see you can now also specify a unit, which can be any of the supported ones.

• Support for various width and height notations. Previously Maps only accepted width and height values is px, forcing you to use maps of fixed sizes. Since most people want to have their complete page width visible even on small screens, this resulted in a lot of people using rather small maps, and so wasting screen space. 0.6 allows you to specify the size in px, ex, em, and most importantly, in %. The syntax is what you’d expect: width=”420px”, width=”420em”, width=”42%”. width=”420″ will default to using px, so is backward compatible. When using the % values, maps will even adapt their size when the screen width or the height of the container they are in is changed after the page has loaded

• Not a new feature, but rather one that’s removed: OSM support. The OSM service has been completely removed from Maps and Semantic Maps as it was rather broken and not easy to upgrade to the internal structure of Maps 0.6. I’m planning to add it back later on, rewritten from ground up, in 0.6.1 or 0.6.2 or so. Note that you can still view OSM maps on your wiki using the OpenLayers service, which has build in OSM layers, and also allows you to define your own layers since 0.5.5.

• And many more

The most notable bugfixes are:

• Fixed conflict with prototype library that caused compatibility problems with the Halo extension.
• Added automatic icon image sizing for Google Maps and Yahoo! Maps markers.
• Various security fixes, mostly preventing XSS attacks.

If no serious bugs are found in this release, a minor update can be expected in a month or so.

View announcement on the mapping wiki.

Downloads

• Maps 0.6 [ zip, 7z ]
• Maps and Semantic Maps 0.6 [ zip, 7z ]

One of the big changes in the upcoming 0.6 release of Semantic Maps will be the from the ground up rewritten semantic datatype for Geographical coordinates. Although the changes themselves do not directly add any value for the user, they enable some pretty neat improvements to existing features, and the creation of many new ones. In this blog post I’ll first go over the changes that are made, in a technical manner, and then sum up the effects they have from an users point of view.

The SMWDataValue extending class for coordinates, which was previously part of Semantic MediaWiki, and only recently moved over to Semantic Maps, has had a big overhaul. This class, now called SMGeoCoordsValue, used to parser coordinates all by itself. Considering Maps also parses coordinates, this is pretty dumb, since the code is redundant, and even worse, does not behave the same way, resulting into incompatible output. I rewrote all the parsing and formatting code in an as modular fashion as possible, and put it in a new class in Maps, which is MapsCoordinateParser. SMGeoCoordsValue now does all formatting and parsing of values via this class

Markus recently added a hook to Semantic MediaWiki that allows for extensions to define their own database tables for storage of semantic values with a specific signature, which will be part of the upcoming SMW 1.5.1. Semantic Maps is using this hook for storing coordinates in a table with lat and lon fields, both floats. This is a huge improvement to simply storing the coordinates in non-directional float notation with the elements separated by a comma, in a string field. The new layout allows for much faster lookups since the values can be index, esp. for things like the soon to be re-implemented distance query, and opens up a lot of new possibilities.

As a user, you will not notice any of these changes, except maybe that things that where broken will be magically fixed when using this new code. Semantic Maps now has settings that allow you to specify the output format of coordinates in queries though, so you can now view them in a different format then non-directional floats, like you where obliged to in the previous releases. You can pick any format supported by Maps, for example directional DMS, as show on the screenshot. Another change is that you will see new functionality based on the new table layout, like, but not limited to, the distance query.

Edit: There actually is one thing you’ll definitely notice as a user when upgrading – you need to update your semantic data via Special:SMWAdmin in order for your wikis coordinates to be stored in the new table. If you don’t do this, queries for coordinates won’t return anything. It’s sort of annoying you need to do this, but there is no way around it.

I’ve been working on Maps and Semantic Maps 0.6, the next big, awesome, release of both MediaWiki extensions, for over a month now. I also released an early alpha on the 3th of April. All planned new features have been implemented, and known bugs from 0.5.x have been fixed. So you might wonder why 0.6 is still not out.

In response to the possibility of someone doing work on Maps and Semantic Maps during the upcoming Google Summer of Code, I took a critical look at the current structure, holding into account all the things I learned over the last two month while working on Storyboard. I came to the conclusion that a bunch of things ought to be handled in a cleaner fashion, and started to refactor this code. The most difficult part here is changing how the display_map and display_point(s) parser functions handle their mapping service parameter and validate the provided location(s). I’ve been wanting to change this since 0.4, but didn’t since it’s rather tricky to do. I decided to finally get this done now, and have done most of the work. To complete these changes, I’ll have to make some rather complex modifications to Validator, which can take a while to complete. That’s the last thing that needs to be done before the 0.6 release though

I estimate this should be done in approximately 2 weeks, maybe sooner. After that I’m planning to release at least one RC, to ensure stability and complete awesomeness, cause really a lot has been changed. I figure about three quarters of the codebase has been changed in one way or another!

After this release a lot of cool new features can be added, which I might do before the GSoC coding period starts. There are also a few people who wanted to do a mapping project for GSoC at the WMF, but did not get accepted, which are planning to do some effort here after all, which is totally awesome

During the past two weeks I’ve been putting some effort in making some big changes to the source of Maps and Semantic Maps, and adding some awesome new features. Although not every to-do for the 0.6 release has been completed, I’ve made an early alpha version available which allows you to try out some of the exiting new functionality. So let’s have a look at the most notable changes since 0.5.5.

New features

• Support for various width and height notations. Previously Maps only accepted width and height values is px, forcing you to use maps of fixed sizes. Since most people want to have their complete page width visible even on small screens, this resulted in a lot of people using rather small maps, and so wasting screen space. 0.6 allows you to specify the size in px, ex, em, and most importantly, in %. The syntax is what you’d expect: width=”420px”, width=”420em”, width=”42%”. width=”420″ will default to using px, so is backward compatible. When using the % values, maps will even adapt their size when the screen width or the height of the container they are in is changed after the page has loaded See my wiki for some cool examples. Although this is really basic functionality in a way, it’s awesome to finally have it available.
• Added full support for DMS, DD, DM and float coordinate notations, both their directional and non-directional variants. This means you can now enter coordinates in these notations in the display_point(s) and display_map parser functions, as well as make Semantic Maps recognize them as geographical coordinates. Smart geocoding support has also been added to #geocode, meaning you can now pass along coordinates as well as addresses, so you don’t have to double check if your passing along an address to prevent losing the value.
• Coordinate formatting support. Along with the smart geocoding support, #geocode now also accepts 2 parameters to specify the output notation. Since you can pass along coordinates, this means you can change the notation of a set of coordinates using #geocode. A new parser function, #coordinates, has been added specifically for this purpose, so for pure formatting, you should use this one rather then #geocode. Both parser functions work with a hybrid parameter system that supports several ‘default’ parameter (ones that consist of only a value, and no name) and named parameters. You can work with names or without for the default ones, and the default ones in #geocode are the same as the old parameter order, so backward compatibility is retained.
• Rewrote map html and js output. This is now done in a cleaner, safer and more consistent way, for all mapping services in both Maps and Semantic Maps. This is not a new feature, but such a big refactoring it’s worth noting.

More to come…

• Native support for geographical proximity queries in Semantic Maps. This will require quite some work to do in a clean and efficient way, so will probably one of the last features finished for 0.6.
• OpenLayers 2.9 support. Since Maps 0.1, it has been supporting OpenLayers 2.8, and 2.9 is just about to be released, so It’ll be nice to see the progress made there.

On top of these changes to the extensions, I’m drastically redoing the documentation and examples. I’ll post more about that later on though

Today I released Validator version 0.2, on which I’ve been working the last 2 days. It features massive rewriting to make it more flexible, and has some added functionality. Let’s have a look at what changed.

The most important change is, without any doubt, the new list support. Validator 0.1 had a list type, which allowed you to have enumerations of values and do crude validation on them. This version now supports lists of a type, instead of seeing list as a type. So you can now create lists of strings, lists of integers, and even lists of custom types you add. This new approach also allows per-item-validation and per-item-defaulting. This means you can set an in_array criteria, which will then be enforced for every value in your list. Closely related to this new form of list support are the new list criteria, which allow you to validate lists as a whole. At the moment the only 2 build in list criteria are item_count and unique_items, but like for regular criteria, you can hook into Validator and add your own.

Another important change are the output formats. Output formats allow you to specify additional formatting that needs to be done with the parameter value, before it is retrieved from Validator. There are currently 6 build in types, which are array, list, boolean, boolstr, string and unique_items, but again you can hook into this list via Validator. The awesome thing about output formats is that it greatly reduces the mess you otherwise have with converting your parameters from user input to true data structures. You can even specify multiple output formats, which will then do their formatting one by one.

Other things brought by Validator 0.2 include some new criteria (is_boolean, has_length and regex), a new error level, Validator_ERRORS_MINIMAL, new parameter types (boolean, number and char) and support for Validator_ERRORS_WARN in ValidatorManager.

Equally important as the changes made is that the documentation has been completely updated, to give in-depth cover of how Validator works, and how you should use it.

Both Maps and Semantic Maps 0.5.1 use Validator 0.2, allowing them to throw away a lot of repetitive, dumb manipulation, code that has been their since the initial versions of those extensions

Downloads:

• Validator 0.2 (zip)
• Validator 0.2 (7z)
• Validator 0.2 SVN tag

Earlier today, versions 0.5 of Maps and Semantic Maps where released. Some mayor new features where added, and a whole bunch of things have been refactored. I also did some effort to improve the documentation by adding some screencasts and revising the developer docs. Version 0.4.2 proved to be pretty stable, since only 2 bugs have been found and fixed.

Let’s have a look at the new, awesome, functionality:

• Strict parameter validation which allows you to get specific errors or warnings when entering invalid values or parameters. This functionality is obtained by relying on the Validator extension, which saw it’s first release today. This makes Maps dependent on Validator. Every distribution of Maps does include Validator, and Maps will automatically load Validator when it’s not loaded yet, so no extra installation work is required (unless you get the code from SVN).

• Static map support makes you able to display maps as plain images instead of via a JavaScript (or other non-html language, for example flash) mapping API. This is very important since it makes viewing the maps possible for people who are browsing the web with a browser that does not support JavaScript (or this other language), or simply have it disabled for some reason.

• Added a query printer that handles the osm result format.

• Added support for GUI parameter selection of Yaron’s new version of SMW’s Special:Ask page to the query printers.

• Added smart ‘autopanzoom’ like control for Google Maps and Yahoo! Maps.

• Added internationalization to the OSM service, and an extra parameter to define per-map languages.

The list of all things that have been refactored is rather long, so I’ll only cover the most interesting things here:

• Complete rewrite of the parameter handling. To make Maps work with Validator, this was required. The result is that Maps does not have a whole mess of specific validation and defaulting functions any more, since that’s now handled by Validator.

• Moved the geographical coordinate data type handling from Semantic MediaWiki to Semantic Maps. This will make the SMW codebase smaller by moving the geographical coordinate data type out, which is logical, since if you use coordinates, you very likely also want maps.

• Added code to unload any services from the service hook that are not present in the list of allowed services. This ensures they don’t get initialized, and makes any check to see if the service is one of the allowed ones further on unneeded.

As for documentation, I created 2 screencasts, both covering a different aspect of Maps. This way people can learn how to work with Maps in a more interactive way then just reading the documentation.

• Screencast: display_map

• Screencast: display_point

The developer documentation on how you can extend Maps using it’s hooks has been completely rewritten. This was needed since the previous version was created for Maps 0.3.3, since which a lot has been changed to the hook systems of Maps.

For a complete list of changes, see the Maps change log, and Semantic Maps change log.

Downloads:

• Maps 0.5 (includes Validator 0.1) [zip, 7z]
• Semantic Maps 0.5 [zip, 7z]
• Maps and Semantic Maps 0.5 (includes Validator 0.1) [zip, 7z]
• Maps 0.5 SVN tag
• Semantic Maps 0.5 SVN tag

One of the big new features in Maps 0.5 will be strict parameter validation. This means Maps will allow you to get specific errors or warnings when entering invalid values or parameters.

The setting determining the strictness of the validation, which can be changes in your LocalSettings file, currently accepts 4 values:

• Maps_ERRORS_NONE : Maps will not show any errors, and make the best of the imput it got.
• Maps_ERRORS_WARN : Maps will make the best of the imput it got, but will show warnings for omitted coordinates.
• Maps_ERRORS_SHOW : Maps will make the best of the imput it got, but will show a list of all errors.
• Maps_ERRORS_STRICT: Maps will only show a map when there are no errors, if there are, a list of them will be shown.

The underneath example demonstrates an error list that can be generated when the validation level is on Maps_ERRORS_SHOW or Maps_ERRORS_STRICT. In case of the former, it’ll be shown below the map,while in case of the later, it’ll be shown instead of any map. The error messages are of course fully internationalized.

The validation is done via a new class dedicated to parameter validation. To be able to validate anything, you need to feed it two things: an associative array containing the raw parameter names and their values, and a somewhat more complex, nested, array containing the allowed parameter and their meta data, such as aliases and default values. The class also provides a hook for validation types, allowing you to do specific or complex validation that is not build in. The handling of the different strictness levels and generation of the actual error messages is done by another class that uses the first to validate and get the errors. Both classes do not contain any Maps specific code, so can be used to validate the parameter of any parser function. I’m planning to split this code, after it has reached a beta level, into a separate extension, that will probably be named “Validator”. This extension will be bundled with Maps, and will not any additional steps to the installation process.

Together with implementing this new feature, I did a big overhaul of the parameter handling in Maps and Semantic Maps. Instead of the two level system, containing general parameters, and service specific parameters, that was used in Maps, there now is a four level system. The first level are the general parameters, shared by everything. These include things like width, height and zoom. Feature specific parameters make up the second level, while the third one holds service specific parameters. The last level are the parameters specific to a combination of service and feature. Maps goes through these levels, starting with the upper one, and overriding it with the following. This allows more specific behaviour and is required to be able to validate the parameters in some instances.

The changes I made to Maps and Semantic Maps during these rewrites are responsible for what are probably my biggest commits to both extensions yet.

• Changes to Maps (rev 59531)
• Changes to Semantic Maps (rev 59532)

Due to the extend of changes I made, and the lack of thorough tests yet, I expect multiple issues with this code, including several severe ones, so I advice against using the latest SVN code for the moment, except for testing purposes of course. I hope to have the code refined and bug hunted in the coming week, so I can put it in a new extension and release it. During this period I’ll also start working on the other new features planned for 0.5, so you can expect more news on this soon.

It’s been almost 2 months since Maps version 0.3.4 was released. Although I did waste a lot of my time at school, I managed to get quite some important work done in that period. Version 0.4 mainly addressed new functionality, and some thorough refactoring for Maps, and only introduced 2 bug fixes for Semantic Maps. So lets have a look at everything that changed

New features

• I added a display_map parser function, that’s obviously meant to display maps. The code handling this parser function is optimized to display maps, and does not hold into account any possible markers, ect. The main reason for adding this function is that it’ll be used for the Wikimedia Foundation implementation of Maps.
• Smart geocoding support. Maps is now able to determine if a value is a set of coordinates, or an address. This enables automatic geocoding of addresses, making the specification of the value type unneeded, and allows you to mix coordinates and addresses in location lists. This functionality is implemented for display_map’s default parameter, the one of display_point(s), and display_point(s) centre property.
• OpenStreetMap service. This service uses OpenLayers to display OpenStreetMap tiles. It’s optimized for OSM, and does not allow you to display other tile sources with it. This is another of the requirements for WF usage that’s now been completed. This service is only implemented in Maps ATM, and will be added to Semantic Maps in version 0.5.
• Support for DM (Decimal Minutes) and DD (Decimal Degrees) coordinate notations. Someone requested these to be added. Several other variants, such as GPS coordinates, might be put into Maps in the future.
• Added a setting specifying the minimum and maximum width and height of maps. When a value falls out of this range, it’ll be changed to the nearest allowed value. This is to prevent post-stamp or wallpaper maps, and is inspired by the similar functionality in the SlippyMap extension.
• Parsing of marker-specific title and label values. I actually marked the lack of this feature a bug a while ago, which I changed at the point where I remembered I simply did not add such a thing yet, and only map-wide title and label value’s got parsed.

Refactoring (“under the hood” changes)

• A new hook system for the parser functions, allowing the adding or removing of additional parser function support. Each mapping service will now have one class for handling on specific parser function. Again, the main reason behind this change is that it’s required for the WF usage of Maps, where at the start, only display_map will be used. Another great advantage is that you can now very easily add new parser functions without having to mess up the core extension code. An example of such a function is display_route, which might be added at some point to maps.
• Change the geocoding functionality into a true feature hook element, so it can be easily removed. This is not easy to do, since the geocoding functionality is currently used at various points by the parser functions. So before I can start making it a feature hook element, the parser function code will have to be restructured.
• Create service hook for the geocoding feature – separate from the mapping services hook. This will allow adding and removing support for a specific geocoding service.
• Since Maps is now able to distinguish coordinates from addresses, the display_address and display_addresses parser functions have been removed. For backward compatibility, calls to them will be routed to display_point(s). This will of course eventually be removed.
• Removed a redundant path variable. One of the messy remains of the initial Maps release, which even caused problems on some installations.

Bug fixes

• Fixed a mayor bug in the initialization method causing hook code to get executed at a probably wrong moment. This bug can be the cause of some weird problems that surfaced since 0.3.3.
• Fixed a rather annoying issue with size of popups in Google Maps. They did not stretch far enough vertically for large contents. I’m happy I got this fixed now, since I’ve been aware of it for quite a while now, but simply could not find a solution. Yaron pointed me to a helpful blog where a certain post set me on the right track. The solution was ridiculously simple, and one line (in the JS code) long.
• Fixed some problems with the display of the mapping result formats on the Special:Ask page of Semantic MediaWiki. I’ve done this by adding an aliasing system for result formats to SMW itself. So to be able to benefit from this, you need to have that version of SMW, or later. Maps has a build in check to see if the aliasing functionality of SMW is present, and if not, uses the old, but a lot less efficient, solution.

For the change log listing, see Maps version history and Semantic Maps version history.

The documentation has been completely updated for both extensions. I only have to change the demo’s on my demo wiki to reflect the new syntaxis of the parser functions. After that I can get to work on the to-do’s for version 0.5, which will include the long awaited static map support

Did I mention Semantic Maps now supports over 50 languages? :p

Downloads

• Maps 0.4 (zip, 7z)
• Semantic Maps 0.4 (zip, 7z)
• Maps and Semantic Maps 0.4 (zip, 7z)
• Maps 0.4 SVN tag
• Semantic Maps 0.4 SVN tag

Since the 0.3.4 release of both Maps and Semantic Maps, I’ve been putting the little free time I have to use by working on the to-do’s for the next release. Originally this would have been 0.3.5, but I’ve added several things to it, that made me decide to make it 0.4 recently, since it involved some relatively big changes for the users. Here are a few of the most noteworthy changes that are planned for 0.4. Some of them are already completed, while still have to start on the others. For a list of changes I’ve made since 0.3.4, view revisions 57704, 57585 and 56614 (newest first).

New Features

• A new display_map parser function. This will enable you to simply display a map without any coordinates indicated on it. Since none of the marker related code will be used, this will be a more efficient and clean way of displaying maps. The main reason I’m implementing this is cause it’s a requirement for getting Maps suitable for Wikimedia Foundation usage.
• An OSM mapping service, which uses OL, but only allows OSM layers and is optimized for OSM. This is an other to-do that got prioritized by the WF usage requirements.
• Parsing of marker-specific title and label values. I actually marked the lack of this feature a bug a while ago, which I changed at the point where I remembered I simply did not add such a thing yet, and only map-wide title and label value’s got parsed.

Refactoring

• A new hook system for the parser functions, allowing the adding or removing of additional parser function support. Each mapping service will now have one class for handling on specific parser function.
• Change the geocoding functionality into a true feature hook element, so it can be easily removed. This is not easy to do, since the geocoding functionality is currently used at various points by the parser functions. So before I can start making it a feature hook element, the parser function code will have to be restructured.
• Create service hook for the geocoding feature – separate from the mapping services hook. This will allow adding and removing support for a specific geocoding service.
• Restructure parser functions: change display_point(s) and display_address(es) to display_point(s), with auto detect to see if the provided value are coordinates or addresses. Retain display_address(es) for backward compatibility, but remove from docs.

Bug Fixes

• Fix a major bug in the initialization method causing hook code to get executed at a probably wrong moment. This bug can be the cause of some weird problems that surfaced since 0.3.3.
• Fix a rather annoying issue with size of popups in Google Maps. They did not stretch far enough vertically for large contents. I’m happy I got this fixed now, since I’ve been aware of it for quite a while now, but simply could not find a solution. Yaron pointed me to a helpful blog where a certain post set me on the right track. The solution was ridiculously simple, and one line (in the JS code) long.

After the actual release, I’ll post a complete list of changes. In any case, these changes will force me to make a heap of changes to the docs, and also to the powerpoint’s of the presentations related to Maps and SM I’m giving.