Building simple interpreter in ClojureScript



In rerenderer I need an intermediate language that can be interpreted in browser and on Android. For browser part I chose ClojureScript. Quick note about the language, it’s an EDSL on Clojure macros, for example:

(let [event (r/new Event "click")
      canvas (r/.. document (createElement "canvas"))
      width (r/.. window -innerWidth)]
  (r/set! (r/.. canvas -width) width)
  (r/.. canvas (dispatchEvent event)))

It contains only .., set! and new macros. It’s very similar to Clojure interop with java and ClojureScript interop with JavaScript. This code is an equivalent of JavaScript code:

var event = new Event("click");
var canvas = document.createElement("canvas");
var width = window.innerWidth;
canvas.width = width;
canvas.dispatchEvent(event):

And macros are just an user friendly facade, inside it produces something like bytecode:

[[:new [:ref "a"] [:static "Event"] [[:val "click"]]]
 [:call [:ref "b"] [:static "document"] "createElement" [[:val "canvas"]]]
 [:get [:ref "c"] [:static "window"] "innerWidth"]
 [:set [:ref "b"] "width" [:ref "c"]]
 [:call [:ref "d"] [:ref "b"] "dispatchEvent" [[:ref "a"]]]]

We’ll write interpreter for it. You can notice that bytecode has four instructions:

  • [:new <result-var> <class-var> [<arguments-vars>]] – creates new object;
  • [:get <result-var> <object-var> <attribute>] – gets attribute of object;
  • [:set <object-var> <attribute> <value-var>] – sets attribute of object;
  • [:call <result-var> <object-var> <method> [<arguments-vars>]] – calls object method.

And also bytecode has one utility instruction:

  • [:free <object-var>] – removes reference from registry.

Variable can be in three forms:

  • [:val <value>] – value of primitive type, like double, string or bool, aka pass by value;
  • [:static <value>] – value from global scope, like window or document, similar to pass by reference;
  • [:ref <reference>] – reference to value in registry, aka pass by reference.

So, let’s start with variables and implement function that returns variable value, there we’ll use match macro from core.match:

(defn extract-var
  [registry var]
  (match var
    [:static x] (aget js/window x)  ; variable from global scope, from `window`
    [:ref x] (registry x)  ; variable from registry
    [:val x] x))  ; value

It’s simple, now it’s time to functions that’ll handle instructions. All that functions should have signature like (registry, *instruction-params) -> registry. We start with function that’ll handle :new:

(defn create-instance
  [registry [_ ref-id] cls args]
  (let [cls (extract-var registry cls)
        js-args (clj->js (into [nil] (map #(extract-var registry %) args)))
        constructor (.. js/Function -prototype -bind (apply cls js-args))
        inst (new constructor)]
    (assoc registry ref-id inst)))

So it’s just creating new instance of cls with args and puts result to registry. You can notice obscure:

js-args (clj->js (into [nil] (map #(extract-var registry %) args)))
constructor (.. js/Function -prototype -bind (apply cls js-args))
inst (new constructor)

We need this because we can’t use apply with new, like (apply new cls args) and also not all constructors have apply method, so we can’t just call (new (.apply cls js-args)). Interop isn’t nice here and this code is an equivalent of JavaScript:

new Function.prototype.bind.apply(cls, [null].concat(args))

Next is a function, that’ll handle :set, it’s very simple, we just use aset:

(defn set-attr
  [registry obj attr value]
  (aset (extract-var registry obj) attr (extract-var registry value))
  registry)

And similar for :get with aget:

(defn get-attr
  [registry [_ result-ref] ref attr]
  (assoc registry result-ref (aget (extract-var registry ref) attr)))

Function for :call is a bit more complicated:

(defn call-method
  [registry [_ result-ref] var method args]
  (let [obj (extract-var registry var)
        js-args (clj->js (mapv #(extract-var registry %) args))
        call-result (.apply (aget obj method) obj js-args)]
    (assoc registry result-ref call-result)))

We need the obscure part with .apply here, because not all JavaScript functions can be called with ClojureScript’s apply.

Functions for :free is the easiest:

(defn free
  [registry [_ ref]]
  (dissoc registry ref))

So now it’s time to write function, that can handle all instructions, it should have signature like (registry, instruction) -> registry. And it’s very easy to write it with match:

(defn interpret-instruction
  [registry instruction]
  (try
    (match instruction
      [:new result-var cls args] (create-instance registry result-var cls args)
      [:set var attr value] (set-attr registry var attr value)
      [:get result-var var attr] (get-attr registry result-var var attr)
      [:call result-var var method args] (call-method registry result-var var
                                                      method args)
      [:free var] (free registry var))
    (catch js/Error e
      (.warn js/console "Can't execute instruction" instruction ":" e)
      (throw e))))

So now we can interperete instructions one by one, and interpret script with reduce:

(reduce interpret-instruction {} script)

But we forgot about one significant part. How we handle references registry? It’ll be just atom with hash-map:

(def registry-cache (atom {}))

And function, that interpret script and update registry will be just:

(defn interpret!
  [script]
  (swap! registry-cache #(reduce interpret-instruction % script)))

You can try bytecode interpreter in action:

(new window)

Full code of interpreter.

Make youtube video look like a gif



Sometimes I need to insert youtube video looped, without sound and without controls, basically it should work like a gif. I’ve tried imgur video to gif, but it limited to 15 seconds video and a bit buggy (need to use http link instead of https).

And I thought that it would be easier to just insert youtube player without controls and info, and use IFrame API for looping (loop param works only with flash player) and muting.

So we need to insert IFrame API script and our script:

<script src="//www.youtube.com/iframe_api"></script>
<script>
    function onYouTubeIframeAPIReady() {  // called automatically when IFrame API loaded
        [].forEach.call(document.querySelectorAll('.gifify'), function(el) {
            var player = new YT.Player(el, {
                events: {
                    'onReady': function() {
                        player.mute();
                        player.playVideo();
                    },
                    'onStateChange': function(state) {
                        if (state.data === 0) {  // video ended
                            player.seekTo({seconds: 0});
                            player.playVideo();
                        }
                    }
                }
            });
        });
    }
</script>

And then insert video with ?enablejsapi=1&showinfo=0&controls=0 params and gifify class, like:

<iframe class="gifify" src="https://www.youtube.com/embed/4YKx9z6j1aA?enablejsapi=1&showinfo=0&controls=0" frameborder="0"></iframe>

In action:

Fuzzy assertions with cljs.test and core.match



Sometimes I need to test content of a big structure, where not all content meaningful, so I need something like fuzzy assertions. And I use core.match for that, like:

(ns example.test
  (:require [cljs.test :refer-macros [deftest is]]
            [cljs.core.match :refer-macros [match]]
            [example.core :refer [get-script]]))

(deftest test-render
  (let [script (get-script)]
    (is (match script
          [[:new [:ref _] "Canvas" []]
           [:call [:ref _] [:ref _] "getContext" [[:val "2d"]]]
           [:set [:ref _] "width" [:val 30]]
           [:set [:ref _] "height" [:val 40]]
           [:set [:ref _] "fillStyle" [:val "red"]]] true
          _ false))))

But it’s redundant, and failure messages are a bit ugly and it doesn’t even show content of not matched structure:

That can be easily fixed, because cljs.test is very extendable. So we just need to create macros for simpler matching and implement assert-expr method for it:

(ns example.test-utils
  (:require [cljs.core.match :refer [match]]
            [cljs.test :refer [assert-expr]]))

(defmacro match?
  [x pattern]
  `(match ~x
     ~pattern true
     _# false))

(defmethod assert-expr 'match? [_ msg form]
  (let [[_ x pattern] form]
    `(if ~form
       (cljs.test/do-report {:type :pass
                             :message ~msg
                             :expected '~form
                             :actual nil})
       (cljs.test/do-report {:type :fail
                             :message ~msg
                             :expected '~pattern
                             :actual ~x}))))

Then update test:

(ns example.test
  (:require [cljs.test :refer-macros [deftest is]]
            [example.test :refer-macros [match?]]
            [example.core :refer [get-script]]))

(deftest test-render
  (let [script (get-script)]
    (is (match? script
          [[:new [:ref _] "Canvas" []]
           [:call [:ref _] [:ref _] "getContext" [[:val "2d"]]]
           [:set [:ref _] "width" [:val 30]]
           [:set [:ref _] "height" [:val 40]]
           [:set [:ref _] "fillStyle" [:val "red"]]]))))

After that failure message will be much nicer:

Bonnie Eisenman: Learning React Native



book cover white Lately I’m working a lot with React, and mobile development looks interesting to me. So I decided to read something about React Native and chosen Learning React Native by Bonnie Eisenman. It’s a short book, mostly in tutorial format. Explains differences with React and platform specific things.

But the book have a few downsides, there’s too much screenshots of XCode and app stores, book mostly written for OS X users and contains not so much info for Android development.

Sums up everything, it seems to be a nice introductory book.

Better ClojureScript tests output with Chrome and cljs-devtools



For testing ClojureScript code I use cljs.test with figwheel. It’s very cool, tests runs when every .cljs file changed, but frequently output in DevTools is a mess. And it would be nice to have highlighted code in expected/actual part.

Then I found a nice lib – cljs-devtools. For using it you need to add it to your project and configure Chrome DevTools.

It’s nice, but it doesn’t work with print/println, only with console.log. And only when clojure data structures passed to console.log. cljs.test uses print and formatters for printing expected/actual. So we need to hack cljs.test a bit. Assume your test runner looks like:

(ns ^:figwheel-always rerenderer.test
  (:require [cljs.test :refer-macros [run-all-tests]]))

(enable-console-print!)
(run-all-tests #"rerenderer\..*-test")

And we need to redefine cljs/test.print-comparison:

(ns ^:figwheel-always rerenderer.test
  (:require [cljs.test :refer-macros [run-all-tests] :as test]
            [devtools.core :as devtools]))

(enable-console-print!)
(devtools/install!)

(defn print-comparison
  [{:keys [expected actual]}]
  (.log js/console "expected:" expected)
  (.log js/console "  actual:" actual))

(with-redefs [test/print-comparison print-comparison]
  (run-all-tests #"rerenderer\..*-test"))

And that’s all, example output:

Andreas Oehlke: Learning LibGDX Game Development



book cover I’m a bit interested in game development and LibGDX seems to be a good framework, so I read Learning LibGDX Game Development by Andreas Oehlke. I’ve planned to read about best practices, edge cases and how it works inside. And it’s not that kind of book. It’s just an entry level tutorial with a lot of printed code.

But the author described development of a little game touching almost all LibGDX parts. So maybe it’s a nice tutorial for newcomers.

Michael L. Scott: Programming Language Pragmatics



book cover Yesterday I’ve finished reading Programming Language Pragmatics by Michael L. Scott and it’s a big book with a lot of exercises (I’ve tried to do most of them) and with a ton of additional material on PLP CD. This book covers most of PL related concepts from grammar to concurrency and memory management. Also it explains why something implemented the way it implemented in exists programming languages. And has information about ton of languages, from Algol to Haskell.

But some chapters is outdated, for example chapter about in-browser scripting. And some chapters covers not all aspects, it’s like 50 pages about mutexes and locks and only 2 pages about message passing in chapter about concurrency.

Despite this, I guess it’s must read book for software developers.

Talking with Hubot through Google Glass



For home automation I use Hubot, it’s something like a framework for creating chat bots, it’s very easy to use and rules for it can be written in CoffeeScript, like:

robot.hear /pause move/i, (res) ->
  exec 'player pause'
  res.send 'Sure!'

So I planned to create an app for phone, which would allow me to say “Ok Google, Hubot, next song”, or something similar. But it’s not possible, Android API not allow to create custom voice actions like “Hubot”, only commands from predefined list allowed.

But I remembered that I have useless Google Glass, which GDK API allows to create custom “Ok Glass” commands, like “Ok Glass, Hubot, next song”, at least in development mode with:

<uses-permission android:name="com.google.android.glass.permission.DEVELOPMENT" />

First of all I created Hubot adapter which support something like polling with simple API:

  • POST /polling/subscribe/ with {} respond s{user: user} – subscribe to polling;
  • POST /polling/message/ with {user: user, text: message-text} responds {ok: ok} – send message to bot;
  • GET /polling/response/:user/ responds {responses: [response]} – get unread bot responses.

It’s not so interesting, written in CoffeeScript (Hubot requires it), code is very simple.

For Glass part I’ve used Kotlin with a few nice libraries:

And it’s very great combination, with them making http request is much nicer than with java and just DefaultHttpClient, like:

Fuel.post("$url/polling/message/")
    .body(jsonObject("user" to user, "text" to text).toString())
    .header("Content-Type" to "application/json")
    .jsonPromise()
    .success { info("Message $text sent") }
    .fail { warn("Can't send message $text because $it") }

Just promises in comparison with Clojure core.async and Scala Akka it’s a bit way back, it’s like writing in pre ES7 JavaScript. So why Kotlin? It’s simpler to use on Android, struggling with dex errors with Scala isn’t fun. Performance is similar to apps written in Java, sometimes startup time of Clojure apps is annoying. And there’s far less magic then in Scala scaloid and Clojure Neko.

In action:

Sources on github.

Import python modules straight from github



In Go we have ability to import modules from github, like:

import "github.com/parnurzeal/gorequest"

It’s a bit controversial feature, but sometimes it’s useful. And I was interested, is it possible to implement something like that in python. TLDR it’s possible with import_from_github_com package:

from github_com.kennethreitz import requests

assert requests.get('https://github.com').status_code == 200

So, how it works, according to PEP-0302 we have special sys.meta_path with importer objects and every importer should implement finder protocol with find_module(module_name: str, package_path: [str]) -> Loader|None. Now we need to implement finder that handles modules, which path starts with github_com, like:

class GithubComFinder:
    def find_module(self, module_name, package_path):
        if module_name.startswith('github_com'):
            return GithubComLoader()
            
sys.meta_path.append(GithubComFinder())

And now we need GithubComLoader that implements loader protocol with load_module(fullname: str) -> None, I’ll skip private methods of the loader here, they’re straightforward and not interesting in context of the article:

class GithubComLoader:
    def load_module(self, fullname):
        if self._is_repository_path(fullname):
            self._install_module(fullname)

        if self._is_intermediate_path(fullname):
            module = IntermediateModule(fullname)
        else:
            module = self._import_module(fullname)

        sys.modules[fullname] = module

So what’s IntermediateModule, it’s a dummy module/package for paths like github_com.nvbn, it’s used only in intermediate steps and shouldn’t be used by end user. Installation happens in _install_module method, it just calls pip with git url, like:

import pip

pip.main(['install', 'git+https://github.com/kennethreitz/requests'])

All looks very simple, let’s try it in action:

>>> from github_com.kennethreitz import requests
Collecting git+https://github.com/kennethreitz/requests
  Cloning https://github.com/kennethreitz/requests to /tmp/pip-8yyvh7kr-build
Installing collected packages: requests
  Running setup.py install for requests
Successfully installed requests-2.9.1
>>> requests.get('https://github.com').status_code
200

Sources on github.

Three months in Bali, Indonesia



Rice field nearby my house

From September to December I was in Indonesia, most of the time I was living in Ubud (Bali), but also was in Jakarta, Kuta (Bali) and Gili Trawangan.

I was entering Indonesia by VOA, it allowed me to be in the country for 30 days. So every month I was going to a third country, visited Kuala Lumpur (Malaysia), Bangkok (Thailand) and Singapore.

So in Ubud I was renting part of a house with kitchen, A/C, hot water, pool (shared between 6 houses) and 8 MBit/s internet. With all bills it costs 3000000 IDR ($216). The house was beside nice rice fields, but a bit far from shops, beaches and attractions. So I was renting motorbike (Honda Scoopy) for 600000 IDR ($43).

Back to internet, it worked well in dry season, but very bed in wet season. It wasn’t working for an hour after every thunderstorm, and in some days it was like five thunderstorms. So as a backup I’ve used mobile internet by Telkomsel simPATI and paid 160000 IDR ($11) for 6GB. I’ve used 3G, but heard that now LTE available.

In Bali it’s cheaper to buy prepared food on night markets. It costs 150000-200000 IDR ($10-15) for week of eating three times a days local food like nasi/mi goreng, satay and etc. But I guess it’s not healthy, so mostly I cooked by myself from ingredients from supermarket. So with every day meat/fish/seafood it was 300000-500000 IDR ($20-35) for week.

Alcohol is a bit expensive, like 20000-30000 IDR ($1.5-2) for a can of local beer and 70000-120000 IDR ($5-8) for a bottle of local rice vodka (arac).

Tons of articles already written about Bali’s attractions, so I just mention that some beaches, waterfalls and temples are quite nice here.