01 Oct 2018, 20:37

A Quick Look at WorkerDOM

In the browser, many operations occur on a single main thread. Due to the number of things we need to handle as web developers such as styling, DOM updates, fetches, data transforms, timers and so forth we can end up with a lot going on that thread. Unfortunately, this thread is also responsible for handling user inputs and rendering to the screen, which are user critical requirements and have large impacts on their experience. Here if we have tasks that run for long periods this can block responding to user input and rendering new frames.

In the browser we have access to threading via Web Workers. Web Workers allow us to run JavaScript in a separate thread whilst also being able to message back to the main thread. The limitations of Web Workers are that they have transfer times for data between the main and worker thread, limited transfer types (i.e. Functions and Errors aren’t allowed) and also perhaps most importantly no DOM access (i.e. document.getElementByID for example). I have experimented and written about transfer times for workers, showing that for small loads the fear is probably over played. However a core aspect of frontend development is, maybe somewhat obviously, trying to update what is happening on the screen (render new content). This makes the DOM limitation pretty painful for those using Web Workers, as it mostly limits it to computational logic which we then proxy back to the main thread.

On mobile devices, where performance perhaps matters the most, we’re now seeing low-end devices with multiple cores (for example a Nokia 1 and the Micromax Bharat Go both have quadcore processors). Arguably this is where multiple threaded code would probably have the largest benefit. However, we still have a particularly single-threaded approach to writing browser code. This is why it’s interesting to see the emergence of the recently announced WorkerDOM library from the AMP team, which aims to let developers leverage workers more easily, by removing the aforementioned limitation of lack of DOM access (or at least perceived lack of DOM access).

What is WorkerDOM?

WorkerDOM is a library which you can include in your applications and pages. It provides much of the regular DOM API and in turn proxied access to the real DOM on the main thread, passing mutations to the DOM over to the main-thread to be handled. This encourages a lot of logic that was previously happening on the main thread to be handled by the Web Worker, leaving the main thread to handle user input and DOM changes. The library itself is written in TypeScript and is compiled down to both global variable and module formats.

Setting up WorkerDOM

You can include WorkerDOM by installing it via NPM like so:

npm install @ampproject/worker-dom

or if you use Yarn then:

yarn add @ampproject/worker-dom

Alternatively you could use a script tag directly, using a CDN, in the following fashion:

<script src="https://unpkg.com/@ampproject/worker-dom@0.1.2/dist/index.js"></script>

We can actually take this a step further as WorkerDOM is distributed as both a global (aliased as MainThread) and an ES Module, allowing for the import syntax. This means you can do the following;

<script src="https://unpkg.com/@ampproject/worker-dom@0.1.2/dist/index.mjs" type="module"></script>
<script src="https://unpkg.com/@ampproject/worker-dom@0.1.2/dist/index.js" nomodule defer></script>

Here only browsers that understand modules (all modern browsers, except Samsung Internet now support modules) will receive the module script.

First Steps

WorkerDOM allows us to expose a specific part of the DOM to be upgraded, we can do this by doing something we would probably not normally do; we set the src attribute on the containing element we want to upgraded to work with WorkerDOM. The src attribute is updated to be the name of the script we’re interested in running as our worker script. For the purposes of this blog, we are going to generate some prime numbers and render them into the DOM. For our index.html file, we might start with something like this in our case:

<body>
    <div src="primes.js" id="primes">
    </div>
</body>

So now we have an element that we can upgrade in our body. We need to declare explicitly that we want to upgrade the element using the upgradeElement method. We can do that like so:


<script type="module">
    import {upgradeElement} from '/dist/index.mjs';
    upgradeElement(document.getElementById('primes'), '/dist/primes.mjs');
</script>

<script nomodule async=false defer>
    document.addEventListener('DOMContentLoaded', function() {
        MainThread.upgradeElement(document.getElementById('primes'), '/dist/primes.js');
    }, false);
</script>

You can see we take both the module and none module code paths allowing us to handle both scenarios. This means we can now begin looking at our actual worker logic!

The Worker File

The coolest part about WorkerDOM is that it allows us to behave as if we have DOM access in the worker thread. That means we have access to properties like document.createElement for example. Continuing on with the concept that we want to generate primes and add them to the DOM, we could do something like this:


const startNumber = 1;

function generatePrimes() {

    while (document.body.firstChild) {
        document.body.removeChild(document.body.firstChild);
    }

    const numDivs = 1000;
    const limit = startNumber + numDivs;
    const primes = sieveOfEratosthenes(limit); // An algorithm for generating primes up to 'limit'

    const div = document.createElement('div');
    div.className = 'parent';

    for (let i = startNumber; i < limit; i++) {
        const numberDiv = document.createElement('div');
        numberDiv.className = "number";
        const numberText = document.createTextNode(i);
        if (primes.has(i)) {
        numberDiv.style.fontWeight = 'bold';
        numberDiv.style.color = '#240098';
        }
        numberDiv.appendChild(numberText);
        div.appendChild(numberDiv);
    }

    document.body.appendChild(div);
    startNumber += numDivs;

}

setTimeout(generatePrimes, 0); // Not sure why we need this in a timeout?
document.body.addEventListener('click', generatePrimes);

This code will create divs with numbers in them, with prime numbers being highlighted and put in bold. The numbers will update on click on the document. Notice how we can behave as if this worker code is on the main thread, with access to DOM APIs. It’s worth pointing out the main difference here is that the "primes" div from the index.html is considered our document.body here.

As it stands, WorkerDOM is currently in alpha, and as such is still being worked on. Hopefully, this post has given you a reasonable overview of the library, and if you are interested in learning more about WorkerDOM I would recommend these resources:

31 Jul 2018, 20:37

Creating Your Own OS Zoomstack Tiles

In a past life I worked for the Ordnance Survey, specifically at the Geovation Hub in London working with startups doing interesting things with geospatial tools and data. One thing we experimented with there was Ordnance Survey based Vector Tiles. Since I left the OS has now invested a full open data Vector Tile product called Zoomstack. I thought it was a good chance to give it a whirl!

You can get access to Zoomstack by signing up for the trial. Once you have the trial you will receive an email where you can download the tiles in a variety of formats including a Vector Tile API, GeoPackage, a MBtiles file or a PostGIS dump. The purpose of this post is to produce another format, which is the raw Vector Tiles themselves (protobuf files/ .pbfs). To do this we are going to download the MBTiles file and then run it through a tool called tile-join which comes with Mapbox’s tippecanoe. There are probably many other ways we could potentially perform this extraction, and probably even serve them straight from a database directly using a server tool like t-rex or tegola, but in this case will just be creating the raw protobuf files.

Installing tippecanoe and tile-join

On Mac this is fairly straight forward using Brew:

brew install tippecanoe

On Ubuntu (and flavours) you can get the necessary dependencies, compile and install doing:

sudo apt-get install build-essential libsqlite3-dev zlib1g-dev
make
make install

On Windows this is more complicated but there are some thoughts on doing it on the GitHub issues page for tippecanoe.

To confirm that tile-join is successfully installed, and type:

tile-join

into your command line. You should see a corresponding output with some usage hints. Something like this:

Usage: tile-join [-f] [-i] [-pk] [-pC] [-c joins.csv] [-x exclude ...] -o new.mbtiles source.mbtiles ...

Generating the Vector Tiles

Now that we know it works, we can use tile-join to generate the necessary files like so:

tile-join --output-to-directory="./tiles" OS-Open-Zoomstack.mbtiles -z14 --no-tile-size-limit --no-tile-compression

You could also remove --no-tile-compression if your web server can support serving gzipped files. This is definitely recommended for production deployments as gzip tiles will be much lighter over the wire, but for local development uncompressed is fine. One last thing; this process might take a little while!

Locally hosting tiles

Now we are going to host them locally. I chose to use http-server, which you can install using npm (node package manager) like so:

npm install -g http-server

Now http-server should be available as a global command, and we can run it in our tile root directory like so:

http-server --cors

--cors provides us with Cross Origin Resource Sharing support which can be handy if we need to speak from other domains (for example Maputnik). If you did want local gzip compression you could rename all the compressed .pbf files to .pbf.gz with a script and add --gzip as a command flag and http-server should pick them up. You could also add http caching using -c60 (where 60 is the number of seconds) but locally this might cause you some pain if you’re trying to change things rapidly.

Now, assuming you were serving tiles from a tiles folder and you called the above command from the root folder contain tiles, you could get at a specific tile like so:

http://127.0.0.1:8080/tiles/14/8128/5494.pbf

Try opening that in your browser (Chrome, Firefox etc) to prove that it works!

Plugging it into a library

For simplicity in styling here I am going to use Mapbox GL, but we could easily be using the Esri JavaScript API, OpenLayers or potentially even Leaflet (with the right plugin). Basically any library which supports Vector Tiles.

Here is how we’d setup a map in JavaScript:


    mapboxgl.accessToken = 'pk.eyJ1IjoiamFtZXNtaWxuZXIiLCJhIjoiY2prNzA2aGprMWt0MTNydGhuMWs3NG13NSJ9.hRj9NRx4ROa0QDkT4t9XdQ';
    var map = new mapboxgl.Map({
        container: 'map',
        style: 'zoomstack.json'
    });

For styling purposes I took the a premade style from the Ordnance Survey styling repository. You can just download to the root directory where your index.html is and save it with a sensible name as .json file. You will need to make some adjustments to the original OS styling file to make it work with our new setup, such as change the tile url to be the new tiles API format like so:

	"composite": {
		"tiles": [
			"http://127.0.0.1:8080/tiles/pbftiles/{z}/{x}/{y}.pbf"
		],
		"type": "vector"
	}

Using sprites locally

You can download the sprites from the Ordnance Survey folder Mapbox GL styles folder on GitHub, place them locally for your web server to pick up. You can reference them in your style json like so:

	"sprite": "http://127.0.0.1:8080/sprites/sprites",

Custom Fonts

Custom glpyhs can be provided to your styling file to add flair to your map. Let’s say you have a font tha you like from a provider like Google Fonts and then use those with your tiles. You can do this by downloading a fonts ttf file and then running it through fontnik. You can install fontnik locally using npm (npm install fontnik) and then run the build-glyphs in the node_modules folder for fontnik. For example let’s say we downloaded the Roboto TTF file, we can build those glyphs using:

mkdir fonts/Roboto\ Regular
node_modules/fontnik/bin/build-glyphs path/to/source/Roboto-Regular.ttf path/to/fonts/Roboto\ Regular

Note escaping the spaces in the path names using the backslash. After this you can edit your style file to provide the web server glyph URL like so:

	"glyphs": "http://127.0.0.1:8080/path/to/fonts/{fontstack}/{range}.pbf",

Then in a layer layout property you could defined font-text like this:

	"text-font": [
		"Roboto Regular"
	]

Custom styling using Maputnik

Maputnik provides a GUI styling tool you can use to create your own custom styles. You can load your local style (zoomstack-json for example) right into Maputnik and making edits. Check it out at https://maputnik.github.io/editor. Here’s an example of a little styling I did using Material Design based colours and fonts:

Conclusion

Using the .mtiles file you can generate you own tiles to play around with. You could use these locally or host them somewhere on the web like an Amazon S3 bucket, or a web server of your choosing. You’ll want to figure out how to serve the compressed files for best performance. You can also use fantastic base stylings from Ordnance Survey as starting points for your own custom styling for the maps, and edit them manually using a text editor or using a GUI like Maputnik.

04 Jul 2018, 20:37

Service Worker State Management

With this post I want to build upon a previous technique I wrote about; message passing with Service Workers. Here I will be looking at how you might integrate this concept to manage application state across tabs for a site. For those of you unfamiliar, Service Workers are a separate thread that sit at in the background at browser level (rather than the page level like Web Workers) allowing pages and workers on the same domain scope (i.e. example.com) to interact with it. This makes them a great host for doing things such as intercepting and caching requests, handling offline requests, triaging push notifications and managing background syncing. This is even more true now that they are supported in all modern browsers!

Off the Beaten Track with Service Workers

As well as these more run-of-the-mill Service Worker patterns, there are also some more experimental ideas, like on the fly WebP support, or caching from a ZIP file. It’s definitely cool that Service Workers enable this interesting applications. Previously I wrote about passing messages between tabs using a Service Worker, inspired by some tweets and in turn a blog post by Craig Russell. I also recently realised Arnelle Balane wrote some similar ideas, albeit a slight different approach, which are worth a read.

In this post I want to take this further by exploring the idea of state management in a Service Worker. Although many of you might be versed in writing complex applications, I wanted to run through state management at high level before we kick off. For the TL;DR skip to the Mixing State Management with Service Workers section.

State Management

We can think of state management as the remits of how we organise and update the state of our application. For example if our web application was a simple counter, state management would entail concepts like:

  • What is it’s default value?
  • Where do we store it’s value?
  • How do we update the counter?
  • In which ways can the counter increment/decrement?

You can see how for even a arguably straightforward application such as a counter the cognitive load we have to endure as developers can quickly stack up. This is where state management libraries come in.

State management libraries are tools that allow you to centralise you state management, allowing updates to your data to become more predictable as they are funnelled through very specific and narrow channels. This in theory reduce bugs, decrease cognitive load, and in turn make it quicker and easier to scale a web application. With this being said, although state management can simplify scaling complex applications, they may actually make smaller applications more complicated than necessary (see this great post from Dan Abrimov for a deeper insight on that). Many state management libraries are based (or loosely based on the concepts of), Facebook’s Flux pattern. Of these Redux is perhaps the most popular. Another popular state management library is MobX which takes a more reactive/observer based approach to state management.

Mixing State Management with Service Workers

Now for the interesting part; putting state management in a Service Worker. Because Service Workers exist outside of a page and/or worker context for a given domain scope, we can use them to pass data to each other. So what if we took this a step further and stored the apps state in Service Worker? So I know what you’re potentially thinking, which is ‘is this a good idea?’ and in honesty I’m not even sure, but it’s definitely fun and foreseeably useful in specific cases.

Initially I tried to use Redux as the demonstration state manager, however I hit a hurdle. My proof-of-concept appeared to work great in Chrome, but in Firefox it would fail when changing between tabs. What was going on? As it currently stands (June 2018) it looks like Firefox kills off idle Service Workers after 30 seconds, although from my experimenting it seems actually less than that. This means when the tab is idle for a certain period, the script is re-executed when a new message is sent to the worker. State is wiped during this process, making it a none viable approach. There is some potential Chrome might be doing this in the future.

So, what to do? One suggestion in the above issue suggests is sending some sort of message on a timer to keep the Service Worker alive. I’m not a massive fan of this approach though as it feels a bit flakey and in general think timers should be avoided where possible. So what else can we do? Jeff Posnick recommends using IndexDB for persisting state, which got me looking into IndexedDB backed Redux libraries. I came across another Redux library called redux-persist. However this didn’t work out, as the state didn’t seem to persist the data in a way that was conducive to syncing state in the way I wanted. So instead, I rolled my own state library based on idb by Jake Archibald.

The Web Page

Let’s start with the web page first, let’s assume we are building our counter application and we have a standard HTML page. Next we’re going to want to register our Service Worker (let’s assume it’s wrapped in a ('serviceWorker' in navigator):


	navigator.serviceWorker.register('serviceworker.js')
		.then((reg) => {

			// Here we add the event listener for receiving messages
			navigator.serviceWorker.addEventListener('message', function(event){
				// Some function that renders the state to the page
				render(event.data.state.count);
			});

			messageServiceWorker({ GET_STATE: true});

		}).catch(function(error) {
			console.error('Service Worker registration error : ', error);
		});

	// When a new SW registration becomes available
	navigator.serviceWorker.oncontrollerchange = function() {
		messageServiceWorker({ GET_STATE: true});
	}

Here we are going to do something interesting; we’re going to tell the page that when it closes, we want to fire an event to the Service Worker letting it know that tab died. Because Service Workers can exist even when the page isn’t open, we need to a way to reset the state when no tabs are open. Again lets assume we use feature detection for the Service Worker:


	// Event on tab/window closed, so we can then check for no tabs/window.
	// If we wanted we could make this false to permanently persist state
	if (RESET_ON_NO_CLIENTS) {
		window.onunload = function() {
			// postMessage should be synchronous in this context?
			navigator.serviceWorker.controller.postMessage({
				TAB_KILLED: true
			});
		};
	}

We’ll also need a way to post our actions to our Service Worker so the Redux store and dispatch them, so lets add that:


	// Send generic messages to the Service Worker
	function messageServiceWorker(data){
		if (navigator.serviceWorker && navigator.serviceWorker.controller) {
			navigator.serviceWorker.controller.postMessage(data);
		}
	}

	// Pass actions specifically to the Service Worker
	function actionToServiceWorker(action) {
		messageServiceWorker({ ACTION: action })
	}

Let’s also say for the sake of simplicity that we only want to increment the counter, we could do it like this:


    document.getElementById('increment')
		.addEventListener('click', function () {
			actionToServiceWorker('INCREMENT');
		});

The Service Worker

A Service Worker exists as a single file, although may import others with the importScripts function. Let’s setup a Service Worker that can handle our state changes are persist them. Because Service Workers are only supported in modern browsers, I’ve written these in ES6 syntax. Firstly lets handle the incoming messages to the worker:


	initialiseOnMessage() {
		if (!self) {
			console.error("Self undefined, are you sure this is a worker context?");
			return;
		}
		self.onmessage = (message) => {
			if (message.data.GET_STATE) {
				this.store.getState().then((state) => {
					this.syncTabState(state);
				});
			} else if (message.data.TAB_KILLED) {
				this.checkIfAllTabsKilled(actions.RESET)
			} else if (message.data.ACTION) {
				this.dispatchToStore(message.data.ACTION)
			}
		}
	}

Next lets handle syncing state to the tabs. Here we need to be able to be able to dispatch events to our store, sync that store with new state, and also reset that store when all the tabs have been closed. Let’s see how we can do that:


		// Get all the tabs for the current domain scope
		getTabs() {
			return self.clients.claim().then(() => {
				return clients.matchAll(
					{
						includeUncontrolled: true,
						type: "window"
					}
				);
			})
		}


		// Dispatch a store event and sync that back to the tabs
		dispatchToStore(action, clientId) {
			this.store.dispatch(action).then(() => {
				this.store.getState().then((state) => {
					this.syncTabState(state);
				})
			})
		}

		// Check if all the tabs have died and if so reset the state
		checkIfAllTabsKilled(RESET) {

			this.getTabs().then((clients) => {

				// Sometimes the new client exists before we can check if there
				// are no tabs open at all. Essentially we need to handle the refresh case
				const isRefresh = clients.length === 1 && this.lastKnownNumClients < 2;
				const shouldReset = clients.length === 0 || isRefresh;

				if (shouldReset) {
					// Reset state back to normal
					this.store.dispatch(RESET);
				}

				this.lastKnownNumClients = clients.length;

			});

		}

		// Sync the state back to all the available tabs and windows
		syncTabState(newState) {

			this.getTabs().then((clients) => {
				// Loop over all available clients
				clients.forEach((client) => {
					const data = { state: newState }
					client.postMessage(data);
				});

				this.lastKnownNumClients = clients.length;

			});

		}

This code misses out the logic for actually updating our IndexedDB store, but under the hood it’s a mix of a Redux-esque pattern and the idb library I mentioned for persisting that store. The state will only update if the persistence part is successful. You can find the full code for the store logic in the GitHub link below.

Pulling it All Together

Now I’ve explained the page and Service Worker parts, let’s see how it looks in practice! You can find a link to a live demo here, and a link to the full code here.

Conclusion

Is it possible to put your state management in a Service Worker? Totally, if you’re willing to persist state. Is it sensible? I’m not entirely sure. Some obvious shortcomings are that you’ll have to write a fallback for none SW support browsers, you can’t use it in incognito in Firefox and it’s going to increase the complexity of the app with the message passing / asynchronosity aspect. Also in theory there’s more points of failure as you’re introducing a Service Worker and IndexedDB into the mix. This being said, if having tabs in sync is a critical part of your application, this may be a reasonable approach to solving that specific problem. Another way might be to just broadcast the actions to all other pages which, in theory should keep them in sync whilst keeping the Service Worker stateless.