24 Dec 2017, 14:53

Getting going with ES6 Maps

As we move into a world where modern browsers are supporting the nearly the entire ES6 specification, we have a variety of data structures to use for storing and accessing our data as native built-in primitives. In ES6 we specifically gain access to ‘keyed collections’. Keyed collections encompasses four built-ins, these are namely Set, Map, WeakSet and WeakMap. Here we’ll be talking about the Map in particular. Let’s get stuck in!

What is a Map?

A Map is a collection of key value pairs, where both key and values can be of any type. This differs from how one might have traditionally used a JavaScript Object as a map, as historically keys have been strings only (this has changed with the introduction of Symbols but that’s for a different blog post to come!).

const obj = { 1 : "one" }
typeof Object.keys(obj)[0] // Evaluates to string

So with Maps, we have the additional power of using anything we like as our key, even a HTMLElement, which is powerful because it makes it easy to associate data with a DOM node. To illustrate the use of none String keys, we can see this example using booleans:

const map = new Map([
    [true, "Things that are true"],
    [false, "Things that are false"]

Here true and false are the respective keys in our collection. So how does JavaScript know how to evaluate something as matching a Map key? Well the approach is very similar to the === (identity) operator, with the minor caveat that NaN (Not a Number) equates to NaN even though NaN !== NaN.

How do we use it?

Maps provide a set of methods for interfacing with them such as get, set and delete. These methods provide a clean inteface for interacting with our data. Let’s see how that works in practice:

const one = "one";
const map = new Map([ ["one", 1] ])
map.get("one"); // Returns 1
map.get(one); // Returns 1

map.set("two", 2);
// Returns the Map with "two" set to 2

// Returns true if successful or false if unsuccessful

// Returns undefined

As you can see there’s a little gotcha here, get, set and delete all have different return behaviours. Get will return the associated value, set will return the Map itself, and delete will return a boolean, and clear returns undefined. Something to keep in mind!

We also have some really nice convenience methods to help us interface. For example, has which will tell us if a key exists in a Map:

const map = new Map([ ["one", 1], ["two", 2] ]);
map.has("two") // true

This is very useful for quickly checking the existence of key in a Map. What about if we want to iterate over a Maps contents? Firstly there are a few convenient methods in this regard: values, entries and forEach. Let’s look at values and keys. Both these methods return an iterable. Iterables are a little outside the scope of this post, but think of them as JavaScript variables that have defined behaviours for what happens when you iterate over them (for example using for…of):

// Entries
const map = new Map([ ["one", 1], ["two", 2] ]);

for (const num of map.entries()) {
    // Logs ["one", 1]
    // Then ["two", 2]

// Values
for (const key of map.values()) {
    // Logs 1
    // Then 2

Alongside this we have the forEach method that some of you may be familiar with from the Array method of the same name. It behaves like so:

const map = new Map([ ["one", 1], ["two", 2] ]);
map.forEach((key, value) => console.log(key, value));
// Logs ["one", 1]
// Then ["two", 2]

Lastly we can quickly and conveniently get the size of a map with the size method. It is fairly straight forward and operates like so:

const pets = Map([ ["dog", "woof!"], ["cat", "meow"], ["goldfish", "bop bop bop"] ]);
animals.size(); // Evaluates to 3

Which would have been less elegant using an Object:

var pets = {"dog", "woof!", "cat": "meow", "goldfish": "bop bop bop"};

Why use a Map?

Simply put, the Map is more idiomatic and effective primitive than an Object for use as a mutable key-value pair data structure. We have the added flexibility of using anything we please as our keys, alongside a clear set of a powerful methods that we have shown above. These methods allow us to be more semantic and straightforward about interacting and manipulating our data. It also avoids some of the potential pitfalls of the Object, for example, because of the interface of a Map, we can avoid the possibility of colliding with default Object properties such as toString or hasOwnProperty. This is obviously an edge case but provides us with extra reassurance:

// With a traditional Object
const obj = {};
obj["toString"] = "woops";
// Evaluates to woops

// With a Map
const map = Map();
map.get("toString") // undefined
map.set("toString", "This is fine");

Overall it expresses a lot of cool properties that suite it better to behaviours we wanted from using an Object as a Map previously. Happy Mapping!

14 Jan 2016, 19:16

On Well Known Text and the Web

Over the past week I have made a start on looking back at the problem of transferring spatial data out of databases into front end applications. This issue stemed from a project I worked on UCL called Lacuna. Lacuna was a 3D, WebGL based Geographic Information System that I worked on for my masters dissertation. I have since deprecated Lacuna because, in all honesty I dived straight in the deep end and at the time I had only a superficial knowledge of JavaScript and PHP. As such it accrued a lot of technical debt from trying to finish my dissertation in a reasonable time frame.

With the knowledge I have accrued since then I have decided to give tackling CRUD operations on spatial databases via web services. One decision I made was it would require something more performant for the backend. For the language I knew PHP5 wasn’t going to cut it this time around. I weighed up the pros and cons of various languages such as Node and PHP7 but decided that Go was a solid choice. However for the backend I would keep using PostGIS due to it and PostgreSQL relative maturity.

Go is an interesting language, and I’m equal parts enjoying and bashing my head on the desk playing with it for this latest project. The backend of Lacuna was responsible predominately for getting the 3D geometries stored in PostGIS and bringing them to the front end to be rendered by Three.js.

This shouldn’t be an overly complex task, but this time around, I realised that to do it properly is actually quite a significant pain. The major reason for this is Well Known Text. For those of you that aren’t familiar Well Known Text is a markup language for geometries. Specifically it is the text based, human readable version of Well Known Binary (WKB), which is the standard for storing geometries in spatial databases. Let me show you an example of what Well Known Text looks like:

  ((0 0 0, 0 1 0, 1 1 0, 1 0 0, 0 0 0)),
  ((0 0 0, 0 1 0, 0 1 1, 0 0 1, 0 0 0)),
  ((0 0 0, 1 0 0, 1 0 1, 0 0 1, 0 0 0)),
  ((1 1 1, 1 0 1, 0 0 1, 0 1 1, 1 1 1)),
  ((1 1 1, 1 0 1, 1 0 0, 1 1 0, 1 1 1)),
  ((1 1 1, 1 1 0, 0 1 0, 0 1 1, 1 1 1))

You can find a full specification here. As it stands there 18 different Geometry types and 4 variations on each (2D, Z, M, ZM). My job as the developer is to convert these Well Known Text fragments so that I can use clientside in a 3D geometry viewer. This on the surface seems like a plausible task; I mean, firstly someone must have hit this problem before, there’s got to be a library out there that does this right? It turns out there are a few and they’re great, but inevitably they have their limitations. One recurring issue is the 3D support is varying (either they lack Z or ZM geometries, PolyhedralSurfaces, TIN Z or a combination). Generally this is okay if you are doing things with Leaflet, OpenLayers, Google or Esri APIs and working in a 2D environment. However, as the GI industry progresses we are incrementally seeing more 3D geographic information, with demand increasing for both data and systems that can handle 3D.

In reality most web systems will mostly likely require that the data at some point make it’s way into JSON for consumption by the ubiquity that is JavaScript. As an example Terraformer (an open source library from Esri) tackles this by converting to GeoJSON. Again this is great stuff, but unfortunately GeoJSON is a 2D specification (as it stands).

OK you say, well stop being lazy and write your own parser and come up with your own JSON schema. So I am, and I’m about 40% of the way there, but I felt compelled to point out some issues I see with WKT. One of the major ones is the way the text itself is delimited. WKT doesn’t appear (to my knowledge) to leverage any obvious standard for structuring data (i.e. JSON, XML or YAML). As such you more-or-less have to write the parser from scratch. Lets take the example above of the POLYHEDRALSURFACE Z: the X Y Z coordinates are delimited by spaces which, amongst other things makes converting them into something more usable like JSON quite a complex task. Delimitation between coordinates in the same part of the geometry, different parts of the same geometry and separate geometries in their entirety are all delimited by commas. This in turn makes it tough to parse effectively. What characters should you split on? Is that even a good methodology? How can I ensure the integrity of the regex if I go down that path? These are of course rhetorical questions but I hope you get my point.

The approach I took in the end was using a combination of string manipulation and regex, but it feels fragile (read: hacky), and by all means this may not the most optimal. Perhaps a better approach would be to break down the parentheses but it’s hard to tell which ones are superficial and which are integral to the structure of the text. Unfortunately the specification is quite dense for developers to digest in comparison to something like GeoJSON (MapBox have poked a little fun at this on the wellknown GitHub README).

Again though this poses another problem, now I’ve parsed the WKT into JSON, but how should the JSON schema look? Maybe like GeoJSON? But really there’s no standard so it feels somewhat arbitrary. The real point of this post is to open up a dialogue on how to handle WKT going forward. It seems like a standard that has been adopted by spatial database and as such is not going anywhere fast. Storing 3D data in databases makes sense, but in honesty WKT is a tough thing to transition to something meaningful for clientside apps (unlike JSON, XML, YAML). Should we be working towards better support for 3D in projects like wellknown from MapBox, Terraformer from Esri and GeoScript? And should we be looking at ways to produce a standard for 3D geo data in the web i.e. by extending GeoJSON? I am really keen to hear what approaches people have taken to handling WKT in development.

As an after thought, how about something like this for modelling WKT as JSON?:

    dimensions : "Z"
    geometries : [
        [[ {x: 0, y: 0, z: 0}, {x: 0, y: 1, z: 0}, {x: 1, y: 1, z: 0}, {x: 1, y: 0, z: 0}, {x: 0, y: 0, z: 0}]],
        [[ {x: 0, y: 0, z: 0}, {x: 0, y: 1, z: 0}, {x: 0, y: 1, z: 1}, {x: 0, y: 0, z: 1}, {x: 0, y: 0, z: 0}]],
        [[ {x: 0, y: 0, z: 0}, {x: 1, y: 0, z: 0}, {x: 1, y: 0, z: 1}, {x: 0, y: 0, z: 1}, {x: 0, y: 0, z: 0}]],
        [[ {x: 1, y: 1, z: 1}, {x: 1, y: 0, z: 1}, {x: 0, y: 0, z: 1}, {x: 0, y: 1, z: 1}, {x: 1, y: 1, z: 1}]],
        [[ {x: 1, y: 1, z: 1}, {x: 1, y: 0, z: 1}, {x: 1, y: 0, z: 0}, {x: 1, y: 1, z: 0}, {x: 1, y: 1, z: 1}]],
        [[ {x: 1, y: 1, z: 1}, {x: 1, y: 1, z: 0}, {x: 0, y: 1, z: 0}, {x: 0, y: 1, z: 1}, {x: 1, y: 0, z: 1}]]

I know what you’re thinking; this is longer than the original! That is true, however after gzipping and transfer over the wire the difference will be very negligible because it’s a repeating pattern (x, y, z).