How to cross-compile snapcraft plugins

I implemented support for cross-compiling when using the Go plugin and the Rust plugin (bonus points for using a LXD container) in snapcraft. On the part of the user of snapcraft it’s completely transparent but each plugin needs to explicitly support this, so here’s how you enable your favorite snapcraft plugin to make this work.

snapcraft --target-arch=armhf --debug

This will result in the following error if a part uses a plugin that doesn’t support cross-compilation yet:

Setting target machine to 'armhf'
Building for a different target architecture requires a plugin specific implementation in the 'foo' plugin

Let’s enable the fictional foo plugin.

snapcraft/plugins/foo.py

First of all, we tell snapcraft that we know how to cross-compile by implementing one additional method. We don’t need to do anything here. Although I’ve found one good use of the method to be checking that the requested target is supported by the toolchain. For a language like Rust or Go the target is different to the Debian architecture. Rust even supports using a different libc (glibc or musl). That means the plugin needs to map the target and raise an error if it can’t. In the following example I’ll also be seeting self._target so that it can be used in other steps of the lifecycle, for example to set environment variables or when installing target-specific dependencies of the toolchain.

def enable_cross_compilation(self):
    # Cf. rustc --print target-list
    rust_targets = {
        'armhf': 'armv7-{}-{}eabihf',
        'arm64': 'aarch64-{}-{}',
        'i386': 'i686-{}-{}',
        'amd64': 'x86_64-{}-{}',
        'ppc64el': 'powerpc64le-{}-{}',
    }
    self._target = rust_targets.get(self.project.deb_arch).format(
        'unknown-linux', 'gnu')
    if not self._target:
        raise NotImplementedError(
            '{!r} is not supported as a target architecture when '
            'cross-compiling with the rust plugin'.format(
                self.project.deb_arch))

Second of all, we’ll want to adjust the environment variables. This depends on the build system. Typical candidates are compiler names and pkgconfig – we don’t need to worry about the latter since snapcraft already takes care of it. Go for example would also require GOARCH to be set.

In the following code I’m assuming there’s a self._goarch variable that was set in enable_cross_compilation.

def env(self, root):
    env = super().env(root)
    if self.project.is_cross_compiling:
        env.extend([
            'CC={}-gcc'.format(self.project.arch_triplet),
            'CXX={}-g++'.format(self.project.arch_triplet),
            'GOARCH={}'.format(self._goarch),
          ])
    return env

The example uses the triplet of the target architecture. Commonly useful values here are, using armhf values for demonstration purposes:

cross_compiler_prefix: arm-linux-gnueabihf-
arch_triplet: arm-linux-gnueabihf
deb_arch: armhf
kernel_arch: arm

Note that snapcraft will automatically install the compiler and libc headers needed. Should you need extra packages it’s only one command away:

self.build_packages.append('foo-armhf-cross')

It goes without saying that we’ll want to have an integration test checking that this new code actually continues to work in the future.

integration_tests/plugins/test_foo_plugin.py

You’ll need to have a snap in integration_tests/snaps/foo which you may already have. Otherwise you’ll need to add it. It’s a good idea to try and make use of more advanced use cases that may need special care in the context of cross-compilation, such as using C code in the case of Go.

import snapcraft
from snapcraft.tests.matchers import HasArchitecture
[...]
def test_cross_compiling(self):
    if snapcraft.ProjectOptions().deb_arch != 'amd64':
        self.skipTest('The test only handles amd64 to armhf')

    arch = 'arm64'
    self.run_snapcraft('stage', 'foo', '--target={}'.format(arch))
    binary = os.path.join(self.parts_dir, 'foo', 'install', 'bin',
                          os.path.basename(self.path))
    self.assertThat(binary, HasArchitecture('aarch64'))

You’ll also want to unit test the new use cases supported by the plugin. There will be a snapcraft/tests/plugins/test_foo.py if it’s an existing plugin where you can verify the build environment and commands used for building insofar as they differ (Go would require you to use go build for example instead of go install).

I’m not going to go into all the details here since this is very plugin-specific but one thing you’ll generally want to add is scenarios and mock is_cross_compiling:

scenarios = [
    ('armv7l', dict(deb_arch='armhf', go_arch='arm')),
    ('aarch64', dict(deb_arch='arm64', go_arch='arm64')),
    ('i386', dict(deb_arch='i386', go_arch='386')),
    ('x86_64', dict(deb_arch='amd64', go_arch='amd64')),
    ('ppc64le', dict(deb_arch='ppc64el', go_arch='ppc64le')),
]

def setUp(self):
    super().setUp()

    self.project_options = snapcraft.ProjectOptions(
        target_deb_arch=self.deb_arch)
    patcher = mock.patch('snapcraft.ProjectOptions.is_cross_compiling')
patcher.start()
self.addCleanup(patcher.stop)

If all went well I hope you’re going to propose a PR to snapcraft adding support for the foo plugin – a real one of course, not the fictional one that doesn’t exist

Preparing for Chemnitz 19 March 2016

Freshly printed Midori stickers just arrived, in November the last ones of the old batch were being given out to visitors of the OpenRheinRuhr. So I ordered new ones at FlyerAlarm, a print company based in Würzburg, Austria. This time around I went for circular ones and slightler smaller than they used to be. This size also fits better on phones and tablets for those who like to decorate theirs with stickers. You might also notice that the colors are better and the graphic comes out better – fixes to the SVG will be finding their way into the repository soon.

photo221454754825480153

With the new stickers I’m well-prepared for Chemnitz Linux Days this march, in a little more than a fortnight. There will be a Midori booth there. Be sure to add the weekend of March 19 and 20 to your calendars!

Heads or tails

Midori featuring headerbar and Adblock in the toolbarAs things go we diverged a bit from the original plan of making a big compatibility break with this release. Instead much of the original branch has been broken off into pieces that made their way into trunk. We did however bump WebKit2, libSoup and Zeitgeist dependencies. The most visible change is support for CSD, client side decorations, also called header bars after the API in GTK+3.10 (enabled via GTK_CSD=1). With no further adue here goes Midori 0.5.11 with a whole lot of nice things and a ton of bug fixes! Peek at the change log if you want more details.

 

So download Midori v0.5.11 now (1.2 MB)

The source tarball is up. Windows is going to follow suit shortly, and others are still being refreshed, so be patient if the option of your choice isn’t there just yet.

You are welcome to join #midori on irc.freenode.net and party for the release!

A leap of leopards (or: 0.5.10)

Whilst work on the compatibility breaking branch is coming along nicely it seemed sensible to do another bug fix release. Thus Midori 0.5.10.

So download Midori v0.5.10 now (1.2 MB)

The source tarball is up. Windows is going to follow suit shortly, and others are still being refreshed, so be patient if the option of your choice isn’t there just yet.

You are welcome to join #midori on irc.freenode.net or Slack and party for the release!

WebKitGTK+ 2.8.0

We are excited and proud of announcing WebKitGTK+ 2.8.0, your favorite web rendering engine, now faster, even more stable and with a bunch of new features and improvements.

Gestures

Touch support is one the most important features missing since WebKitGTK+ 2.0.0. Thanks to the GTK+ gestures API, it’s now more pleasant to use a WebKitWebView in a touch screen. For now only the basic gestures are implemented: pan (for scrolling by dragging from any point of the WebView), tap (handling clicks with the finger) and zoom (for zooming in/out with two fingers). We plan to add more touch enhancements like kinetic scrolling, overshot feedback animation, text selections, long press, etc. in future versions.

HTML5 Notifications

notifications

Notifications are transparently supported by WebKitGTK+ now, using libnotify by default. The default implementation can be overridden by applications to use their own notifications system, or simply to disable notifications.

WebView background color

There’s new API now to set the base background color of a WebKitWebView. The given color is used to fill the web view before the actual contents are rendered. This will not have any visible effect if the web page contents set a background color, of course. If the web view parent window has a RGBA visual, we can even have transparent colors.

webkitgtk-2.8-bgcolor

A new WebKitSnapshotOptions flag has also been added to be able to take web view snapshots over a transparent surface, instead of filling the surface with the default background color (opaque white).

User script messages

The communication between the UI process and the Web Extensions is something that we have always left to the users, so that everybody can use their own IPC mechanism. Epiphany and most of the apps use D-Bus for this, and it works perfectly. However, D-Bus is often too much for simple cases where there are only a few  messages sent from the Web Extension to the UI process. User script messages make these cases a lot easier to implement and can be used from JavaScript code or using the GObject DOM bindings.

Let’s see how it works with a very simple example:

In the UI process, we register a script message handler using the WebKitUserContentManager and connect to the “script-message-received-signal” for the given handler:

webkit_user_content_manager_register_script_message_handler (user_content, 
                                                             "foo");
g_signal_connect (user_content, "script-message-received::foo",
                  G_CALLBACK (foo_message_received_cb), NULL);

Script messages are received in the UI process as a WebKitJavascriptResult:

static void
foo_message_received_cb (WebKitUserContentManager *manager,
                         WebKitJavascriptResult *message,
                         gpointer user_data)
{
        char *message_str;

        message_str = get_js_result_as_string (message);
        g_print ("Script message received for handler foo: %s\n", message_str);
        g_free (message_str);
}

Sending a message from the web process to the UI process using JavaScript is very easy:

window.webkit.messageHandlers.foo.postMessage("bar");

That will send the message “bar” to the registered foo script message handler. It’s not limited to strings, we can pass any JavaScript value to postMessage() that can be serialized. There’s also a convenient API to send script messages in the GObject DOM bindings API:

webkit_dom_dom_window_webkit_message_handlers_post_message (dom_window, 
                                                            "foo", "bar");

 

Who is playing audio?

WebKitWebView has now a boolean read-only property is-playing-adio that is set to TRUE when the web view is playing audio (even if it’s a video) and to FALSE when the audio is stopped. Browsers can use this to provide visual feedback about which tab is playing audio, Epiphany already does that :-)

ephy-is-playing-audio

HTML5 color input

Color input element is now supported by default, so instead of rendering a text field to manually input the color  as hexadecimal color code, WebKit now renders a color button that when clicked shows a GTK color chooser dialog. As usual, the public API allows to override the default implementation, to use your own color chooser. MiniBrowser uses a popover, for example.

mb-color-input-popover

APNG

APNG (Animated PNG) is a PNG extension that allows to create animated PNGs, similar to GIF but much better, supporting 24 bit images and transparencies. Since 2.8 WebKitGTK+ can render APNG files. You can check how it works with the mozilla demos.

webkitgtk-2.8-apng

SSL

The POODLE vulnerability fix introduced compatibility problems with some websites when establishing the SSL connection. Those problems were actually server side issues, that were incorrectly banning SSL 3.0 record packet versions, but that could be worked around in WebKitGTK+.

WebKitGTK+ already provided a WebKitWebView signal to notify about TLS errors when loading, but only for the connection of the main resource in the main frame. However, it’s still possible that subresources fail due to TLS errors, when using a connection different to the main resource one. WebKitGTK+ 2.8 gained WebKitWebResource::failed-with-tls-errors signal to be notified when a subresource load failed because of invalid certificate.

Ciphersuites based on RC4 are now disallowed when performing TLS negotiation, because it is no longer considered secure.

Performance: bmalloc and concurrent JIT

bmalloc is a new memory allocator added to WebKit to replace TCMalloc. Apple had already used it in the Mac and iOS ports for some time with very good results, but it needed some tweaks to work on Linux. WebKitGTK+ 2.8 now also uses bmalloc which drastically improved the overall performance.

Concurrent JIT was not enabled in GTK (and EFL) port for no apparent reason. Enabling it had also an amazing impact in the performance.

Both performance improvements were very noticeable in the performance bot:

webkitgtk-2.8-perf

 

The first jump on 11th Feb corresponds to the bmalloc switch, while the other jump on 25th Feb is when concurrent JIT was enabled.

Plans for 2.10

WebKitGTK+ 2.8 is an awesome release, but the plans for 2.10 are quite promising.

  • More security: mixed content for most of the resources types will be blocked by default. New API will be provided for managing mixed content.
  • Sandboxing: seccomp filters will be used in the different secondary processes.
  • More performance: FTL will be enabled in JavaScriptCore by default.
  • Even more performance: this time in the graphics side, by using the threaded compositor.
  • Blocking plugins API: new API to provide full control over the plugins load process, allowing to block/unblock plugins individually.
  • Implementation of the Database process: to bring back IndexedDB support.
  • Editing API: full editing API to allow using a WebView in editable mode with all editing capabilities.

Ballet (or: 0.5.9)

Seven months of sweat and tears… oh well, not quite so dramatic. In any event Midori 0.5.9 is out!

We’re already scheming… I mean planning for the next cycle. We want to go WebKit2 and GTK+3 only now and do away with the fourfold compatibility setup. Anyone who finds this thrilling is more than welcome to join in; the fun is going to start soon.

So download Midori v0.5.9 now (1.2 MB)

The source tarball and Windows download are up, while others are still being refreshed, so be patient if the option of your choice isn’t there just yet.

You are welcome to join #midori on irc.freenode.net or Slack and party for the release!

Midori booth at OpenRheinRuhr 2014

At OpenRheinRuhr 2014 we had a Midori booth again. Almost didn’t happen due to the strike of Deutsche Bahn which caused many people to miss the event. Fortunately my humble self made it.

In addition to stickers, this time around we even had edible gummi paws!

Midori booth

Amongst the most posed questions was process isolation, something WebKit2 provides, and which will be available to everyone soon once our plan to move fully towards GTK+3, WebKit2 and Vala is put in motion. The other one was why Midori crashed so often – which is due to the at least 4 primary build configurations currently available. I’m very much looking forward to seeing this go away at some point in the future.

Pretty happy with the results personally, was able to get a decent number of people curious to try out Midori and in one instance even install it on site.

Xubuntu 14.10 “Utopic Unicorn”

After six months of development, the latest version of Xubuntu has been released! Xubuntu 14.10 “Utopic Unicorn” features the latest in Xfce development and is the first step towards the next Long Term Support release in 2016.

Here’s 10 new things to look for in the latest release.  For a more comprehensive list of changes and a list of download links, please see the official release announcement and release notes.

Appearance Updates

  1. The login screen received a minor visual refresh and greater customization options.
    LightDM GTK+ Greeter with the latest Greybird theme.
    LightDM GTK+ Greeter with the latest Greybird theme.
  2. The new wallpaper for this release adds a splash of pink to the Xfce mouse.
    The new default wallpaper for Xubuntu 14.10 "Utopic Unicorn"
    The new default wallpaper for Xubuntu 14.10 “Utopic Unicorn”
  3. To celebrate the 14.10 codename “Utopic Unicorn”, pink highlights have been added.  These highlights can be reverted or easily changed to another color with the installed “Theme Configuration” utility.
    Using “Theme Configuration”, you can easily change Xubuntu’s colors to match your mood.

Xfce Updates

  1. Xfce Power Manager 1.4 sports several improvements over previous releases. Brightness controls have been extended to better support backlit keyboards and new laptop displays. The updated panel plugin shows device charge status, adds display brightness controls, and fixes “Presentation Mode” — letting you disable automatic screensavers.
    Xfce Power Manager replaces the Power Indicator for Xubuntu 14.10
    Xfce Power Manager replaces the Power Indicator for Xubuntu 14.10
  2. With the latest Xfce Display Settings, managing multiple monitors is no longer a hassle. Just drag and rearrange the displays to your liking.
    The new drag-and-drop display settings greatly improves multihead support.
    The new drag-and-drop display settings greatly improves multihead support.
  3. With the updated Xfce window manager, the Alt-Tab switcher has been updated with a refreshed appearance and the ability to select windows with your mouse or by touch.
    Even minor features like the Alt-Tab switcher have been improved.
    Even minor features like the Alt-Tab switcher have been improved.
  4. With the latest Whisker Menu and changes to the default configuration, applications in the Settings Manager are now searchable.
    Quickly find and launch any application with the Whisker Menu.
    Quickly find and launch any application with the Whisker Menu.

Application Updates

  1. With Catfish 1.2, previewing files has been greatly simplified. Easily switch between details and preview mode. When the search index becomes outdated, Catfish will also notify you to update.
    Catfish makes searching for images easier with the latest release.
    Catfish makes searching for images easier with the latest release.
  2. Parole 0.7 introduces a new Clutter-based backend and finally supports video playback in Virtualbox. The media controls are now contained in a slide-over overlay (with a configurable timeout).
    Parole's interface is further refined with each new release.
    Parole’s interface is further refined with each new release.
  3. Light Locker Settings has been improved, further integrating with Xfce Power Manager to handle screensaver settings.
    Light Locker Settings can now integrate with other screensaver managers.
    Light Locker Settings can now integrate with other screensaver managers.

 
That’s it for this release, now to get ready for 15.04 “Vivid Vervet”!

Kitten looking to be adopted

The following unfortunately comes up regularly these days:

  • This Midori looks interesting, maybe I should try it out.
  • Aha, there’s a Debian package for it.
  • sudo apt-get install midori
  • midori (0.4.3+dfsg-0.1)
  • Meh, pretty crashy, I better ask some devs for help.
  • Ah, so 0.5.8 is the latest.
  • So why isn’t this packaged!?

The answer is simple, Midori needs a Debian package maintainer! Thousands of Debian and Ubuntu users are facing the above situation, including lots of Raspberry Pi users. A single person stepping up to it can literally change the world at this point. Now if fame isn’t motivation enough, there may be a t-shirt to get.

GTK+ 3 Plugins in WebKitGTK+ and Evince Browser Plugin

GTK+ 3 plugins in WebKitGTK+

The WebKit2 GTK+ API has always been GTK+ 3 only, but WebKitGTK+ still had a hard dependency on GTK+ 2 because of the plugin process. Some popular browser plugins like flash or Java use GTK+ 2 unconditionally (and it seems they are not going to be ported to GTK+ 3, at least not in the short term). These plugins stopped working in Epiphany when it switched to GTK+ 3 and started to work again when Epiphany moved to WebKit2.

To support GTK+ 2 plugins we had to build the plugin process with GTK+ 2, but also some parts of WebCore and WebKit2 (the ones depending on GTK+ and used by the plugin process) were built twice. As a result we had a WebKitPluginProcess binary of ~40MB, that was always used for all the plugins. This kind of made sense, since there were no plugins using GTK+ 3, and the GTK+ 2 dependency was harmless for plugins not using GTK+ at all. However, we realized we were making a rule for the exception, since most of the plugins don’t even use GTK+, and there weren’t plugins using GTK+ 3 because they were not supported by any browser (kind of chicken-egg problem).

Since WebKitGTK+ 2.5.1 we have two binaries for the plugin process: WebKitPluginProcess2 which is exactly the same 40MB binary using GTK+ 2 that we have always had, but that now is only used to load plugins using GTK+ 2; and WebKitPluginProcess, a 7,4K binary that is now used by default for everything except loading plugins that use GTK+ 2. And since it links to GTK+ 3, it might load plugins using GTK+ 3 as well. Another side effect is that now we can make GTK+ 2 optional, WebKitPluginProcess2 wouldn’t be built and only plugins using GTK+ 2 wouldn’t be supported.

Evince Browser Plugin

For a long time, we have maintained that PDF documents shouldn’t be opened inside the browser, but downloaded and then opened by the default document viewer. But then the GNOME design team came up with new mockups for Epiphany were everything was integrated in the browser, including PDF documents. It’s something all the major browsers do nowadays, using different approaches though (Custom PDF plugin inside the web engine, JavaScript libraries, etc.).

At the WebKitGTK+ hackfest in 2012 we started to think about how to implement the integrated document reading in Epiphany based on the design mockups. We quickly discarded the idea of implementing it as a NPAPI plugin, because that would mean we had to use a very old evince version using GTK+ 2. We can’t implement it inside WebKit using libevince because it’s a GPL library, so the first approach was to implement it inside Epiphany using libevince. I wrote a first patch, it was mostly a proof of concept hack, that added a new view widget based on EvView to be used instead of a WebView when a document supported by evince was requested. This approach has a lot of limitations, since it only works when the main resource is a document, but not for documents embedded in a HTML page or an iframe, and a lot of integration problems that makes it quite difficult to maintain inside Epiphany. All of these issues would be solved by implementing it as a NPAPI plugin and it wouldn’t require any change in Epiphany. Now that WebKitGTK+ supports GTK+ 3 plugins, there’s no reason not to do so.

Epiphany Evince Plugin

Thanks to a project in Igalia I’ve been able to work on it, and today I’ve landed an initial implementation of the browser plugin to Evince git master. It’s only a first implementation (written in C++ 11) with the basic features (page navigation, view modes, zoom and printing), and a very simple UI that needs to be updated to match the mockups. It can be disabled at compile time like all other frontends inside Evince (thumbnailer, previewer, nautilus properties page).

Epiphany embedded PDF document Epiphany standalone PDF document

Another advantage of being a NPAPI plugin is that it’s scriptable so that you can control the viewer using JavaScript.

Epiphany scriptable PDF

And you can pass initial parameters (like current page, zoom level, view mode, etc.) from the HTML tag.

<object data="test.pdf" type="application/pdf" width="600" height="300" 
                currentPage="2" zoomMode="fit-page" continuous="false">
  The pdf could not be rendered.
</object>

You can even hide the default toolbar and build your own one using HTML and JavaScript.