Jeroens blog

Somewhere in the last two weeks I quickly wrote up a small new MediaWiki extension to include content from Wikipedia or some other MediaWiki install into pages on your wiki. It’s titled Include WP.

The extension does not import anything (so nothing is stored), but rather fetches content from the remote wiki every time the page is loaded. This means it takes a second or two for it to actually load, but ensures you got the latest content. Some clean up of the remote content happens and there are several options that allow you to customize how the included content is displayed. It makes use of the Validator extension.

Feature overview:

• Include articles from Wikipedia or any other MediaWiki wiki into your pages.
• Show only a limited amount of paragraphs on page load, with an option to show the full article.
• Partial conversion to plain-text.

1. Removal of templates (such as infoboxes), ref tags, comments, categories and images.
2. Both internal and external links are rendered as plain-text.
3. Tables, lists, table of content, section headers and more are retained.

• Usage of the MediaWiki Resource Loader when available with backward support for MW 1.16.x.

Some screenshots

Download

• Stable releases (archives)
• SVN tags
• SVN trunk

I’m happy to announce the release of a new MediaWiki extension I’ve been working on over the past two weeks. It’s titled Live Translate and allows live page translation via the Google Translate API. It also enables you to define a “dictionary” of certain words or phrases and their translations; any word or phrase in the original text found in the “dictionary” will be translated using that dictionary, instead of using Google Translate.

The main features are:

• Live translation of page contents using Google Translate.
• Ability to define translations of special words in-wiki that will then be left alone by Google Translate.

Some screenshots

A wiki page with the translation control of the Live Translate extension at the right top corner:

The same page after translating it to Dutch:

The dictionary page briefly summarizing how many words and languages it contains:

Editing the special words dictionary works just like editing any other page:

Funding

I created this extension as WikiWorks consultant for Texas Instruments. Thanks to TI for funding this and allowing licensing under the GNU GPL.

Points of interest

For me the most interesting part of creating this extension was figuring out how to walk through the page DOM and send only the actual text to the Google Translate API using JavaScript (and jQuery). This step was needed because the GT API limits translation requests to 500 characters, so it’s not definitely possible to send the whole page. It took me a while to figure this out, and I still think it’s a rather lame limit, as it leads to loss of context, and thus worse translations.

The rest of the extension is rather simple and does not contain anything I haven’t done before. The special words dictionary is stored in a simple db table (fields: word id, word text, word language and word primary) and can be accessed via 2 API modules. One is to query a list of the special words that are defined for a language, the other is to get translations of a set of special words from one language to the other. When you request your first translation of a page, the first API module is hit and the result is used to insert notranslate spans, which make Google Translate ignore stuff, around the special words. After that the other module is hit, the special words are replaced by their translations, and finally the script sends a ton of requests to the GT API.

Ah, and not to forgot, I used a XOR. I kid you not, source code or it didn’t happen (see the last function).

State and future

Live Translate is currently at version 0.2 and contains all the features initially requested by TI. It appears to be stable, and ready for production usage. Of course, if bugs pop up, they will be addressed and a new minor version will be released. A cool new feature I’ve been considering, and might add at some point, is having __LIVETRANSLATE__ and __NOLIVETRANSLATE__ magic words, that allow per-page showing or hiding of the translation control. Yaron suggested also having a per-namespace setting. If you have some cool new features in mind, feel free to suggest them on the Live Translate discussion page (where you also can ask for support and point out bugs). If you want to fund any new functionality, please contact WikiWorks.

You can obtain the latest version of Live Translate here.

I’m happy to announce the release of a new MediaWiki extension I’ve been working on over the past two weeks. As you’ve might already have guessed from it’s name, Push, it enables you to push content of wiki pages to one or more other MediaWiki installs.

The main features are:

• Pushing page content to other wikis via a tab on the page.
• Bulk push via Special:Push.
• Remote authentication support.
• Automatic transfer of included files.
• Support for ApprovedRevs. If there is an approved revision, it will be pushed, if not, the latest one is pushed.
• AdminLinks integration.
• Usage of the new MediaWiki Resource Loader when available with backward support for MW 1.16.x.

The tab interface

When logged in, a new tab or action (on vector based skins) will be added which leads to the push interface for the page you are on. This interface consists of a table listing the available target wikis and also informs you of the status of the target pages. A big push button in each row allows you to do the actual push. When there is more then one target wiki, a convenient ‘Push all’ button will also be shown.Underneath this table the available options will be displayed. With the current version you can choose to include the templates used on the page in the push, as well as transferring the embedded images.

In this screenshot of the tab interface, you can see an actual ‘Push’ tab, rather then an action in the collapsed actions dropdown. You can choose this behaviour, which is useful when you do very frequent pushing, using one of the settings made available by the Push extension.

After initiating the push to one or more targets, the work will happen in the background, and the interface will update to show you progress, completion, and possible errors (such as not having sufficient rights to edit on the target wiki).

Special:Push

Push provides bulk push capabilities via a special page aptly titled ‘Special:Push’. The layout and workings of this page should be familiar to anyone having used MediaWikis native Special:Export, as it’s largely based on this special page. A big textbox allows you to specify the pages you want to push (one per line), and you can choose to add all pages from a category or namespace to it. Under the texbox are the same options as in the tab interface: automatic inclusion of templates and files. Finally you are able to select one or more target wikis.

Note that the file inclusion option was not added yet in the version the below screenshot was taken at. I obtained the above list of pages by entering ‘Locations’ in the category box and submitting it.

When submitting the push request, you’ll be shown the pushing progress in the form of a list to which items get added as pushes complete.

Funding

I created this extension as WikiWorks consultant for Texas Instruments. Thanks to TI for funding this and allowing licensing under the GNU GPL.

Status and future

Push is currently at version 0.5 and contains all the features requested by TI. It appears to be stable, and ready for production usage. Of course, if bugs pop up, they will be addressed and a new minor version will be released. Many useful additions to push can be imagined for various use-cases. Feel free to suggest them on the Push discussion page (where you also can ask for support and point out bugs). If you want to fund any new functionality, please contact WikiWorks.

You can obtain the latest version of Push here.

I’ve been working on the Wikimedia Storyboard extension for a week now, and like to provide the people who are interested in it with some more details about what the project goal is exactly, and what my current progress is.

The idea of the Wikimedia Storyboard is to develop a fundraising landing page on http://wikimediafoundation.org/ with moderated but up-to-date stories from users and donors explaining the role Wikimedia has played in their lives. The motivation of building such a landing page is that donor and supporter stories can help to convince and persuade people of the impact an organization is having. It’s essentially an extended, moderated version of the real-time contribution history.

Basically the extension can be divided into 3 separate parts:

The storyboard interface

The storyboard interface will contain several stories stacked vertically in a box that follows the “eternal load” paradigm, much like Google Reader. Each story will consist of a picture with some text next to it, and some “share” features for stuff like Facebook and Twitter. For people with a special storyreview permission there would also be a button allowing them to hide the stories from being displayed in the storyboard. This part of the UI will be implemented using the tag extension <storyboard/>. You can see a rough layout on the underneath mock-up (on the left).

Story submission interface

As the above image reveals, there will also be an interface where you can submit stories, that will be placed on the right of the storyboard itself, in a tab gadget, together with a donate page. Like the storyboard itself, this will be implemented using a tag extension: <storysubmission/>.

Story review interface

The story review interface will allow users that have the ‘storyreview’ permission to review, edit, publish, and hide stories. Hidden stories could still be published later; they would simply be available through a separate queue. This interface will be implemented as a special page (Special:StoryReview), but will be transcludable, meaning it can be put into other pages.

My progress

Over the past few week, most of my dev time has gone to reviewing documentation about how to correctly create the desired features in the Storyboard extension. Things new for me include, tag extensions, special pages, database interaction, and permission management. I now also better understand some other stuff like how variables should be escaped properly, which I’ll be applying to Maps, Semantic Maps and Validator for their next release. I’ve set up the basic structure of the extension, as you can see on svn trunk, and am now working on creating the dynamic aspects of the interface using jQuery and jQuery UI.

Some stuff I’ll still have to review further at some point is how to make the stories searchable, how to best internationalize the stories and how I’ll implement the “share” features. I’m looking forward to getting more familiar with MW core though, and getting the hang of these things should not be to difficult with the awesome support I’m getting from the devs at the MW IRC.