Livecoding for Android with ClojureScript and Bocko



Not so long ago I found a cool library that allows us to draw simple graphic — Bocko. It works in browsers, on desktop and on iOS, but not on Android. So I decided to port it. Not by putting it in a WebView, I wanted a native app. Not by using Clojure on Android, it was a bit laggy when I tried it, it was an year or more ago, so I guess I should try it again. But by running ClojureScript inside a WebView and using a proxy object for drawing on a native canvas.

And one of the greatest advantages of this method – I can use figwheel just by changing url that opens in the WebView. First of all, it gives use REPL. And code, evaluated in the REPL, executes on Android device and changes painting on the screen:

scale

Also, figwheel automatically pushes changes in the code to the device, so when the code changes, painting on the screen changes too:

scale

And it’s simple to configure. First of all you need to changed method getUrl from BockoAndroid/app/src/main/java/com/nvbn/bockoandroid/BockoView.java to something like this, but with your ip address:

String getUrl() {
    return "http://192.168.0.107:3449/";
}

And write your ip address in :websocket-host in :figwheel section of :cljsbuild build configuration, so your project.clj will be like:

(defproject bocko-example "0.1.0-SNAPSHOT"
            :license {:name "Eclipse Public License"
                      :url "http://www.eclipse.org/legal/epl-v10.html"}
            :dependencies [[org.clojure/clojure "1.7.0-beta3"]
                           [org.clojure/clojurescript "0.0-3269"]
                           [bocko "0.3.0"]
                           [bocko-android "0.1.3-1"]]
            :plugins [[lein-cljsbuild "1.0.6"]
                      [lein-figwheel "0.3.3"]]
            :cljsbuild {:builds {:main {:source-paths ["src"]
                                        :figwheel {:websocket-host "192.168.0.107"}
                                        :compiler {:output-to "resources/public/compiled/main.js"
                                                   :output-dir "resources/public/compiled"
                                                   :asset-path "/compiled"
                                                   :main bocko-example.core
                                                   :source-map true
                                                   :optimizations :none
                                                   :pretty-print false}}}})

Bocko on Android github.

Reactive animation with SVG, ClojureScript and Om



Animation in a browser is a very complicated subject, it requires tons of timers, hard-to-track imperative stuff like drawRect. But can’t it be simplified to just drawing specific items in some places at certain times? Sounds complicated too, but it’s not. In SVG it’s just a changing of attributes like x, y, width and etc, or inserting DOM nodes with desired attributes. Sounds like React will be useful here, I’ll use not just React, but Om with om-tools (for better syntax).

For example, what if we want to draw a rotating red rectangle? Looks like we just need to change transform attribute of the rectangle every n milliseconds:

(defcomponent rotated-rect
  [{:keys [x y width height]} owner]
  (init-state [_] {:angle 0})
  (will-mount [_]
    (go-loop []
      ; Increases angle every 10 milliseconds:
      (om/update-state! owner :angle #(-> % inc (mod 360)))
      (<! (timeout 10))
      (recur)))
  (render-state [_ {:keys [angle]}]
    (let [center-x (+ x (/ width 2))
          center-y (+ y (/ height 2))]
      (dom/rect {:width width
                 :height height
                 :fill "red"
                 ; Rotates rectangle for `angle`:
                 :transform (str "rotate(" angle ", " center-x ", " center-y ")")
                 :x x
                 :y y}))))

(defcomponent scene-1
  [_ _]
  (render [_]
    (dom/svg {:width "100%" :height "100%"}
             (om/build rotated-rect {:x 50
                                     :y 50
                                     :width 100
                                     :height 100}))))

Isn’t it simple? And it works:

What if we want to move the rectangle across the scene? We can do it without changing rotated-rect:

(defcomponent scene-2
  [_ owner]
  (init-state [_] {:x 0})
  (will-mount [_]
    (go-loop []
      ; Increases x every 10 milliseconds:
      (om/update-state! owner :x #(-> % inc (mod 600)))
      (<! (timeout 10))
      (recur)))
  (render-state [_ {:keys [x]}]
    (dom/svg {:width "100%" :height "100%"}
             (om/build rotated-rect {:x x
                                     :y 50
                                     :width 100
                                     :height 100}))))

And the rectangle continued to rotate:

Sprites are very useful for animation, and in SVG it can be used with combination of pattern and image tags, but both of them doesn’t supported by Om (because not supported by React). But we can use ugly dangerouslySetInnerHTML attribute:

(defcomponent sprite
  [{:keys [x y img-width img-height scale sprite-x sprite-y sprite-w sprite-h href]} _]
  (render-state [_ _]
    (let [id (gensym)]
      (dom/g (dom/defs {:dangerouslySetInnerHTML {:__html (str "
            <pattern id='" id "'
                     patternUnits='userSpaceOnUse'
                     x='" x "'
                     y='" y "'
                     width='" sprite-w "'
                     height='" sprite-h "'>
              <image x='" (- 0 sprite-x) "'
                     y='" (- 0 sprite-y) "'
                     xlink:href='" href "'
                     width='" img-width "'
                     height='" img-height "'
                     transform='scale(" scale ")' />
            </pattern>")}})
             (dom/rect {:x x :y y :width sprite-w :height sprite-h
                        :fill (str "url(#" id ")")})))))

(defcomponent scene-3
  [_ _]
  (render [_]
    (dom/svg {:width "100%" :height "100%"}
             (om/build sprite {:img-width 406
                               :img-height 1507
                               :scale 2
                               :sprite-w 40
                               :sprite-h 80
                               :href "mario.png"
                               :sprite-x 214
                               :sprite-y 240}))))

And render Mario from the sprite:

So we have the sprite with a few poses of Mario, why not create a component for him? Assume that he can stand, run and jump to left and right, and running is an animation of changing three images. So Mario component should render himself depending on state and direction:

(defn mario-sprite
  [& opts]
  (om/build sprite (assoc (apply hash-map opts)
                     :img-width 406
                     :img-height 1507
                     :scale 2
                     :sprite-w 40
                     :sprite-h 80
                     :href "mario.png")))

(def sprites
  {:right {:run [328 320]
           :run-1 [354 320]
           :run-2 [378 320]
           :jump [335 240]
           :stand [214 240]}
   :left {:run [60 320]
          :run-1 [34 320]
          :run-2 [8 320]
          :jump [54 240]
          :stand [174 240]}})

(defcomponent mario
  [{:keys [x y]} owner]
  (init-state [_] {:state :stand
                   :direction :left})
  (will-mount [_]
    (go-loop []
      (let [state (om/get-props owner :state)
            direction (om/get-props owner :direction)
            drawing-state (om/get-state owner :state)
            drawing-direction (om/get-state owner :direction)
            next-state (if (and (= direction drawing-direction) (= state :run))
                         ; Changes drawing state (and sprite) when Mario running:
                         (condp = drawing-state
                           :run :run-1
                           :run-1 :run-2
                           :run)
                         state)]
        (om/set-state! owner :state next-state)
        (om/set-state! owner :direction direction))
      (<! (timeout 100))
      (recur)))
  (render-state [_ {:keys [state direction]}]
    (let [[sx sy] (get-in sprites [direction state])]
      (mario-sprite :x x
                    :y y
                    :sprite-x sx
                    :sprite-y sy))))

(defcomponent scene-4
  [_ _]
  (render [_]
    (dom/svg {:width "100%" :height "100%"}
             (om/build mario {:x 10 :y 10 :state :stand :direction :right})
             (om/build mario {:x 60 :y 10 :state :run :direction :right})
             (om/build mario {:x 110 :y 10 :state :jump :direction :right})
             (om/build mario {:x 10 :y 100 :state :stand :direction :left})
             (om/build mario {:x 60 :y 100 :state :run :direction :left})
             (om/build mario {:x 110 :y 100 :state :jump :direction :left}))))

It works and now we can see Mario rendered with all available states and directions:

So let’s write a simple animation with Mario: he jumps, goes to the right end, jumps, goes to the left end and repeats. With core.async it’s very simple to write “scenario” for this:

(defcomponent scene-5
  [_ owner]
  (init-state [_] {:mario-state :stand
                   :mario-x 20
                   :mario-y 10
                   :mario-direction :right})
  (will-mount [_]
    (go-loop []
      ; Stand a half
      (om/set-state! owner :mario-state :stand)
      (<! (timeout 500))
      ; Jump
      (om/set-state! owner :mario-state :jump)
      (dotimes [_ 20]
        (<! (timeout 5))
        (om/update-state! owner :mario-y dec))
      (dotimes [_ 20]
        (<! (timeout 5))
        (om/update-state! owner :mario-y inc))
      ; Stand a half
      (om/set-state! owner :mario-state :stand)
      (<! (timeout 500))
      ; Go right
      (om/set-state! owner :mario-state :run)
      (dotimes [_ 300]
        (<! (timeout 5))
        (om/update-state! owner :mario-x inc))
      ; Stand a second
      (om/set-state! owner :mario-state :stand)
      (<! (timeout 500))
      (om/set-state! owner :mario-direction :left)
      (<! (timeout 500))
      ; Jump
      (om/set-state! owner :mario-state :jump)
      (dotimes [_ 20]
        (<! (timeout 5))
        (om/update-state! owner :mario-y dec))
      (dotimes [_ 20]
        (<! (timeout 5))
        (om/update-state! owner :mario-y inc))
      ;Stand a half
      (om/set-state! owner :mario-state :stand)
      (<! (timeout 500))
      ; Go back
      (om/set-state! owner :mario-state :run)
      (dotimes [_ 300]
        (<! (timeout 5))
        (om/update-state! owner :mario-x dec))
      ; Stand a half
      (om/set-state! owner :mario-direction :right)
      (<! (timeout 500))
      (recur)))
  (render-state [_ {:keys [mario-state mario-direction mario-x mario-y]}]
    (dom/svg {:width "100%" :height "100%"}
             (om/build mario {:x mario-x
                              :y mario-y
                              :state mario-state
                              :direction mario-direction})
             (dom/rect {:fill "green"
                        :x 0
                        :y 70
                        :width 400
                        :height 20}))))

It’s simple and it works:

Looks cool, but it isn’t. Animation is laggy and isn’t smooth, and it glitches. So I guess it isn’t an appropriate solution for an SVG animation. And only working solution is, I guess – <animate> and <animateTransform>, further I’ll try to make them work with Om, dangerouslySetInnerHTML isn’t enough for them.

Gist with the source code.

Sam Newman: Building Microservices



book cover Few days ago I finished reading Building Microservices by Sam Newman and I can say is the one of the greatest book I read in the last year. It describes rationale of microservices, technical problems and possible solutions, view to the theme from management side and a lot more. The book describes development of modern distributed system from all aspects, but not goes deep in details.

Definitely worth reading!

Three months in Budva, Montenegro



Work from beach sucks!

The latest three month I was living In Budva, Montenegro. It’s a small town between the sea and the mountains with great climate (in comparison with weather in Saint-Petersburg, Russia).

I was working remotely and lived by a tourist visa (maybe not a visa, just a stamp in a passport), because of that I was required to leave the country once a month (probably origin country depend). It’s a bit troublesome, but I visited Istanbul (Turkey) and Belgrade (Serbia), which is nice.

Also in Montenegro It’s obligatory to receive the document called “beli karton”. For that you need to pay around €25 (for a month) in a bank, and receive this document in tourists office.

As a landline internet only ADSL available with 7Mbps/768kbps (works more like 3-4Mbps), which is enough for most of tasks, but pulling docker images and pushing code to GAE was a pain. Costs around €25.

For a mobile internet I used packs for 1GB for €3 for 15 days, I don’t remember the speed, but according to the indicator on the phone it was HSDPA.

About prices:

  • one-bedroom apartments for a month if you rent for three – €200;
  • utilities (including internet and mobile phone) – €50-70;
  • food for one person for a week, meat/fish/seafood everyday – €30-50;
  • can of beer – €1-1.25;
  • bottle of local wine – €4;
  • burger/kebab – €0.8-1.2.

Also I tried to work from a beach, and with my laptop screen it wasn’t possible.

Use SVG <image> tag with Om



Few days ago I had to use <image> for rendering image inside an SVG “scene” inside an Om component, but I ran into a problem, Om doesn’t provide om.dom/image. Then I found that neither do React, and there’s already outdated pull request for that.

But React has an ugly solution for using raw html – dangerouslySetInnerHTML. So it’s easy to create Om component for an image:

(defn image
  [{:keys [href x y width height]} _]
  (reify
    om/IRender
    (render [_]
      (let [html (str "<image x='" x "' y='" y "'"
                      "width='" width "' height='" height "'"
                      "xlink:href='" href "'/>")]
        (dom/g #js {:dangerouslySetInnerHTML #js {:__html html}})))))

It just put a raw html inside <g> SVG tag. Usage:

(om/build image {:href "image.png"
                 :x 10
                 :y 20
                 :width 200
                 :height 300})

Solution is a bit ugly and unsafe, but it works:

Gist with the example sources from the iframe.

Brainfuck compiler in Clojure



Brainfuck is one of the simplest languages to implement, so why not creates special compiler which translates Brainfuck code to composition of pure (actually not, . isn’t pure) functions?

At first let’s implement simple version without loops ([]), and write functions for +-<>.. I think it’s a good place for using a multimethod:

(defmulti run-symbol
  (fn [symbol _] symbol))

(defmethod run-symbol \+
  [_ {:keys [pos] :as state}]
  (update-in state [:stack pos] inc))

(defmethod run-symbol \-
  [_ {:keys [pos] :as state}]
  (update-in state [:stack pos] dec))

(defmethod run-symbol \>
  [_ {:keys [stack pos] :as state}]
  (let [new-pos (inc pos)]
    (assoc state :pos new-pos
                 :stack (if (>= new-pos (count stack))
                          (conj stack 0)
                          stack))))

(defmethod run-symbol \<
  [_ {:keys [pos] :as state}]
  (let [new-pos (dec pos)]
    (if (neg? new-pos)
      (update-in state [:stack] into [0])
      (assoc state :pos new-pos))))

(defmethod run-symbol \.
  [_ {:keys [pos] :as state}]
  (-> (get-in state [:stack pos])
      char
      print)
  state)

(defmethod run-symbol \,
  [_ {:keys [pos] :as state}]
  (->> (read-line)
       first
       (assoc-in state [:stack pos])))

(defmethod run-symbol :default [_ state] state)

Each method gets state and returns new one, state is a map with keys :pos and :stack. And now is simple to write simple translator using this methods:

(defn compile-simple
  "Creates composition of functions from Brainfuck code." 
  [code]
  (->> (map #(partial run-symbol %) code)
       reverse
       (apply comp)))

(defn run-code
  "Compiles Brainfuck code and runs it with default state."
  [code]
  ((compile-simple code) {:stack [0]
                          :pos 0}))

Let’s test it with Hello World!:

user=> (run-code "+++++++++++++++++++++++++++++++++++++++++++++
  #_=>  +++++++++++++++++++++++++++.+++++++++++++++++
  #_=>  ++++++++++++.+++++++..+++.-------------------
  #_=>  ---------------------------------------------
  #_=>  ---------------.+++++++++++++++++++++++++++++
  #_=>  ++++++++++++++++++++++++++.++++++++++++++++++
  #_=>  ++++++.+++.------.--------.------------------
  #_=>  ---------------------------------------------
  #_=>  ----.-----------------------.")
Hello World!
{:pos 0, :stack [10]}

It works, so now it’s time to add support of loops, and I guess simplest way to do this – extract code inside [] and compile it’s separately, so now symbol can be a function and when it’s a function – it’s always loop (a bit hackish), so we need to rewrite :default:

(defmethod run-symbol :default
  [symbol state]
  (if (fn? symbol)
    (loop [{:keys [pos stack] :as state} state]
      (if (zero? (stack pos))
        state
        (recur (symbol state))))
    state))

And code of extractor and updated code of the compiler:

(defn update-last
  [coll & args]
  (apply update-in coll [(dec (count coll))] args))

(defn extract-loops
  [code]
  (loop [[current & rest] code
         loops []
         result []]
    (cond
      ; Returns result when all code processed
      (nil? current) result
      ; Start of a new loop
      (= current \[) (recur rest (conj loops []) result)
      ; End of a loop when it inside another loop
      (and (= current \]) (> (count loops) 1)) (recur rest
                                                      (butlast loops)
                                                      (update-last result conj
                                                                   (compile-simple (last loops))))
      ; End of a top level loop
      (= current \]) (recur rest
                            (butlast loops)
                            (conj result (compile-simple (last loops))))
      ; Code inside a loop
      (seq loops) (recur rest
                         (update-last loops conj current)
                         result)
      ; Code outside a loop
      :else (recur rest loops (conj result current)))))

(defn compile-code
  [code]
  (-> (extract-loops code)
      compile-simple))

(defn run-code
  [code]
  ((compile-code code) {:stack [0]
                        :pos 0}))

So now we can test it with Hello World! with loops:

user=> (run-code "++++++++++[>+++++++>++++++++++>+++>+<<<<-]>++
  #_=>  .>+.+++++++..+++.>++.<<+++++++++++++++.>.+++.
  #_=>  ------.--------.>+.>.")
Hello World!
{:pos 4, :stack [0 87 100 33 10]}

Yep, it works!

Gist with source code.

George T. Heineman, Gary Pollice, Stanley Selkow: Algorithms in a Nutshell



book cover white Sometimes I have a strange feeling that I forgot almost all algorithms, in the previous time when that occurred I watched a course about it and now I read Algorithms in a Nutshell by George T. Heineman, Gary Pollice, Stanley Selkow. So it’s a bit boring book (like all other books about algorithms), but worth reading. It contains explanation of well known and even less famous algorithms with code examples (they said examples will be in Ruby, but almost all of them in C++ and Java), places when a reader can use them and benchmarks.

The downside of this book – it contains less than nothing about data structures.

REPL driven development on pyboard



All heard the stories about Smalltalk and LISP apps which worked for decades and was developed/updated via REPL (or similar), so why don’t do something similar on microcontroller board?

On pyboard we have a REPL, but we can’t run it and code on pyboard concurrently, so first of all we need to develop simple REPL with which we can do it. And for making it more interesting – make this REPL to work through bluetooth:

from pyb import UART

uart = UART(1)
uart.init(115200)


def exec_command(command):
    """Runs command and returns output for REPL."""
    try:
        return eval(command)
    except SyntaxError:
        try:
            # For stuff like `a = 10`:
            return exec(command)
        except Exception as e:
            return e
    except Exception as e:
        return e


def handle_repl():
    """Tries to read command from uart and exec it."""
    command = uart.read().decode()
    if command:
        result = exec_command(command)
        if result is not None:
            uart.write('{}\n'.format(result))


def iteration():
    """Function for overriding."""


while True:
    handle_repl()
    iteration()

So now we can test it:

echo "1 + 1" > /dev/rfcomm1
➜ head -n 1 /dev/rfcomm1
2

It works, so let’s try to override iteration for sending Hello World! on each iteration to us through bluetooth:

echo "globals()['iteration'] = lambda: uart.write('Hello World\n')" > /dev/rfcomm1
➜ cat /dev/rfcomm1
Hello World
Hello World
^C%  

Or we can do something more practical – send measurements from accel sensors:

echo "from pyb import Accel

def _send_accel():
    accel = Accel()
    xyz = accel.filtered_xyz()
    uart.write('{}\n'.format(xyz))
    
globals()['iteration'] = _send_accel" > /dev/rfcomm1
➜ cat /dev/rfcomm1    
(9, -6, 90)
(7, -4, 91)
(6, -5, 91)
(5, -4, 92)
^C%

That’s not all, we can also modify the way that REPL works, for example – display all REPL commands/results on little screen (ssd1306 module):

echo "from ssd1306 import Display

display = Display(pinout={'sda': 'Y10',
                          'scl': 'Y9'},
                  height=64,
                  external_vcc=False)

orig = exec_command

def wrapper(command):
    display.write('>>> {}'.format(command))
    result = orig(command)
    if result is not None:
        display.write('{}\n'.format(result))
    return result

globals()['exec_command'] = wrapper" > /dev/rfcomm1
➜ echo "a = 1" > /dev/rfcomm1
➜ echo "b = 2" > /dev/rfcomm1
➜ echo "a + b" > /dev/rfcomm1
➜ echo "[x ** 2 for x in range(a + b)]" > /dev/rfcomm1 

And it works:

photo

So it’s cool and maybe can be useful for developing/updating some hard-to-reach devices.

Simple DSL for creating html in Python



In Clojure world we have hiccup for creating html:

[:div.top
  [:h1 "Hello world]
  [:p hello-text]]

In JS world we have JSX (it’s not internal DSL, but it’s relevant):

var html = (
    <div className="top">
        <h1>Hello world</h1>
        <p>{helloText}</p>
    </div>
);

But in Python we don’t have similar DSL (upd: actually we have: lxml.E, pyxl, Dominate and The DOM), and isn’t it be cool (actually it isn’t, I don’t recommend to do something like this, it’s just an experiment) to write something like this:

h.div(klass='top')[
    h.h1["Hello word"],
    h.p[hello_text]]

Let’s start with simplest part, implement ability to call h.p and h.div, for this I’ll use magic of metaclasses and __getattr__:

class hBase(type):
    def __getattr__(cls, name):
        return cls(name)
        
        
class h(metaclass=hBase):
    def __init__(self, name):
        self._name = name
        
    def __str__(self):
        return '<{name}></{name}>'.format(name=self._name)
        
    def __repr__(self):
        return str(self)
        
        
In [3]: h.div
Out [3]: <div></div>

It’s very simple, now is the time to add ability to define childs for html element with h.div[h.h2, h.p], magic of __getitem__ will help me:

class hBase(type):
    def __getattr__(cls, name):
        return cls(name)


class h(metaclass=hBase):
    def __init__(self, name, childs=None):
        self._name = name
        self._childs = childs
        
    def __getitem__(self, childs):
        if not hasattr(childs, '__iter__'):
            childs = [childs]
        return type(self)(self._name, childs)
        
    def _format_childs(self):
        if self._childs is None:
            return ''
        if isinstance(self._childs, str):
            return self._childs
        else:
            return '\n'.join(map(str, self._childs))
        
    def __str__(self):
        return '<{name}>{childs}</{name}>'.format(
            name=self._name,
            childs=self._format_childs())
            
    def __repr__(self):
        return str(self)


In [7]: h.div[h.h2['Hello world'], h.p['Just text.']]
Out [7]:
<div><h2>Hello world</h2>
<p>Just text.</p></div>

Cool, it works! So now let’s add ability to define attributes with h.div(id="my-id"), but before I need to notice that in python we not allowed to use class as a name of argument, so I’ll use klass instead. So here I’ll use magic of __call__:

class hBase(type):
    def __getattr__(cls, name):
        return cls(name)


class h(metaclass=hBase):
    def __init__(self, name, childs=None, attrs=None):
        self._name = name
        self._childs = childs
        self._attrs = attrs
        
    def __getitem__(self, childs):
        if not hasattr(childs, '__iter__'):
            childs = [childs]
        return type(self)(self._name, childs, self._attrs)
        
    def __call__(self, **attrs):
        return type(self)(self._name, self._childs, attrs)
        
    def _format_attr(self, name, val):
         if name == 'klass':
             name = 'class'
         return '{}="{}"'.format(name, str(val).replace('"', '\"'))
        
    def _format_attrs(self):
        if self._attrs:
            return ' ' + ' '.join([self._format_attr(name, val)
                                   for name, val in self._attrs.items()])
        else:
            return ''
        
    def _format_childs(self):
        if self._childs is None:
            return ''
        if isinstance(self._childs, str):
            return self._childs
        else:
            return '\n'.join(map(str, self._childs))
        
    def __str__(self):
        return '<{name}{attrs}>{childs}</{name}>'.format(
            name=self._name,
            attrs=self._format_attrs(),
            childs=self._format_childs())
            
    def __repr__(self):
        return str(self)
            
            
In [19]: hello_text = 'Hi!'
In [20]: h.div(klass='top')[
          h.h1["Hello word"],
          h.p[hello_text]]
Out [20]:
<div class="top"><h1>Hello word</h1>
<p>Hi!</p></div>

Yep, it’s working, and it’s a simple DSL/template language just in 44 lines of code, thanks to Python magic methods. It can be used in more complex situations, for example – blog page:

from collections import namedtuple


BlogPost = namedtuple('BlogPost', ('title', 'text'))
posts = [BlogPost('Title {}'.format(n),
                  'Text {}'.format(n))
         for n in range(5)]

In [30]: h.body[
    h.div(klass='header')[
        h.h1['Web page'],
        h.img(klass='logo', src='logo.png')],
    h.div(klass='posts')[(
        h.article[
            h.h2(klass='title')[post.title],
            post.text]
        for post in posts)]]
Out [30]:
<body><div class="header"><h1>Web page</h1>
<img class="logo" src="logo.png"></img></div>
<div class="posts"><article><h2 class="title">Title 0</h2>
Text 0</article>
<article><h2 class="title">Title 1</h2>
Text 1</article>
<article><h2 class="title">Title 2</h2>
Text 2</article>
<article><h2 class="title">Title 3</h2>
Text 3</article>
<article><h2 class="title">Title 4</h2>
Text 4</article></div></body>

And after that little experiment I have to say that everything is a LISP if you’re brave enough =)

Redis RPOP-LPUSH as a core.async channel



Redis has RPOP and LPUSH commands, which often used for creating simpler messaging queue, for example, open two redis-cli:

# first cli
127.0.0.1:6379> LPUSH queue "test"
(integer) 1

# second cli
127.0.0.1:6379> RPOP queue
"test"

And semantic of this commands are a bit like >! (LPUSH) and <! (RPOP) from core.async. So why not implement special channel which will use Redis lists?

As a library for working with Redis I’ll use carmine because It’s most popular and alive.

Let’s start with >!, for doing it we should implement method put! of WritePort protocol, and call LPUSH command inside of the method:

(require '[clojure.core.async.impl.protocols :refer [WritePort]]
         '[taoensso.carmine :refer [wcar lpush]])
         
(defn redis-chan
  [conn id]
  (reify
    WritePort
    (put! [_ val _]
      (atom (wcar conn
              (lpush id val))))))

And try it:

user=> (require '[clojure.core.async :refer [>!!]])
nil
user=> (def ch (redis-chan {} :queue))
#'user/ch
user=> (>!! ch "test-data")
1

Check result in redis-cli:

127.0.0.1:6379> RPOP "queue"
"test-data"

Yep, it’s working and it’s very simple.

So now the time for <!, we should implement method take! of ReadPort protocol. We have two variants for popping value from Redis list: use RPOP and poll Redis for new values in list, or just use blocking BRPOP. I chose simplest solution – BRPOP, but for non-blocking semantic of go and <! we should call that command in separate thread, I don’t recommend doing stuff like this in your production code, but this is just an experiment. So redis-chan with ability to take! values will be:

(require '[clojure.core.async.impl.protocols :refer [ReadPort WritePort take!]]
         '[clojure.core.async :refer [thread]]
         '[taoensso.carmine :refer [wcar brpop lpush]])

(defn redis-chan
  [conn id]
  (reify
    ReadPort
    (take! [_ handler]
      (take! (thread (last (wcar conn
                             (brpop id 0))))
             handler))
    WritePort
    (put! [_ val _]
      (atom (wcar conn
              (lpush id val))))))

Try it:

user=> (require '[clojure.core.async :refer [>!! <!!]])
nil
user=> (def ch (redis-chan {} :queue))
#'user/ch
user=> (>!! ch "new-data")
1
user=> (<!! ch)
"new-data"
user=> (>!! ch "other-data")
1

And ensure that all works correctly from redis-cli:

127.0.0.1:6379> RPOP "queue"
"other-data

It’s working!