Mark Jason Dominus: Higher-Order Perl



book cover As I sometimes use Perl, I decided to read something about it. And Higher-Order Perl by Mark Jason Dominus was looking interesting. The book is really nice, it shows that Perl can be kind of a functional programming language and that it’s possible to implement almost every feature from other languages in Perl. Also, after reading the book, I think there’s copious amount of ways to shot yourself in the leg in Perl.

The problems in the book are interesting and a bit challenging, so I think it can be worth reading even for people who don’t work with Perl.

Soundlights with ESP8266 and NeoPixel Strip



About a year ago I made soundlights with Raspberry Pi. But RPI is a bit of an overkill for this simple task and it’s quite big, doesn’t have WiFi out of the box and practically can’t be used without a power adapter.

So I decided to port soundlights to ESP8266. The main idea was to reuse as much as possible from the previous implementation, so the parts with patched audio visualizer and colors generation are the same. In a few words, I’ve patched cava to print numbers instead of showing pretty bars in a terminal. And I’ve generated colors with a code found on Quora.

And in current implementation I decided to make it very simple to use, the only requirement is to have a machine with cava and ESP8266 on the same WiFi network. So I chose UDP broadcasting as a way to send data to ESP8266. And because there’s just 60 LEDs and color of a LED is three values from 0 to 255, colors for all strip is just 180 bytes. So it fits in one UDP packet.

Let’s start with the part with cava:

import sys
import socket
import array


COLORS_COUNT = 256
COLORS_OFFSET = 50


def get_spaced_colors(n):
    max_value = 16581375
    interval = int(max_value / n)
    colors = [hex(i)[2:].zfill(6) for i in range(0, max_value, interval)]

    return [(int(color[:2], 16),
             int(color[2:4], 16),
             int(color[4:6], 16)) for color in colors]


def send(colors):
    line = array.array('B', colors).tostring()
    sock.sendto(line, ('255.255.255.255', 42424))


sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)

colors = get_spaced_colors(COLORS_COUNT)

while True:
    try:
        nums = map(int, sys.stdin.readline()[:-1].split())
        led_colors = [c for num in nums for c in colors[num]]
        send(led_colors)
    except Exception as e:
        print(e)

It can be used like:

unbuffer ./cava -p soundlights/cava_config | python cava/soundlights/esp/client.py

And it just reads numbers from cava output, generates colors, transforms them to bytes and broadcasts them at 42424 port.

The ESP8266 part is even simpler:

import socket
import machine
import neopixel


np = neopixel.NeoPixel(machine.Pin(5), 60)
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(('', 42424))


while True:
    line, _ = sock.recvfrom(180)

    if len(line) < 180:
        continue

    for i in range(60):
        np[i] = (line[i * 3], line[i * 3 + 1], line[i * 3 + 2])

    np.write()

It just receives broadcasts from 42424 port and changes colors of LEDs.

At the end, this version has less code and just works. It’s even some sort of IoT and with some effort can become a wearable device.

Github.

Language agnostic REPL driven development with Visual Studio Code



A few years ago I was using Light Table, the integration with Clojure REPL was so nice. But the other parts of the editor and other languages support weren’t that good. And at the moment the editor looks almost dead.

After that, for Clojure, I switched to Cursive, and I still use it. It has a nice feature – send to REPL, which allows users to execute selected code in REPL. But it’s Clojure/ClojureScript only and requires some hassle to configure.

Nowadays for some stuff, I use Visual Studio Code. It doesn’t have a nice integration with REPL, but it has integrated terminal. So I thought, wouldn’t it be nice to just open any REPL in a terminal and somehow send selected code to the REPL. Without any configuration, even with REPL on a remote server.

So I wrote a little extension – SendToREPL. In action with Python REPL:

How does it work? Let’s look at the initial version of the extension:

const vscode = require('vscode');
let terminal;

function activate(context) {
    terminal = vscode.window.createTerminal('SendToREPL terminal');
    terminal.show();

    const command = vscode.commands.registerTextEditorCommand('extension.sendToREPL', (textEditor) => {
        const code = textEditor.document.getText(textEditor.selection);
        terminal.sendText(code);
    });

    context.subscriptions.push(command);
}

exports.activate = activate;

function deactivate() {
    if (terminal) {
        terminal.dispose();
    }
}

exports.deactivate = deactivate;

It just creates a terminal when extension loaded and registers extension.sendToREPL command. When the command is triggered (by Ctrl+’/Cmd+’ hotkey or from Quick Open) it gets selected code and sends it to the terminal.

The current version is a bit more advanced, it sends the line with cursor if nothing selected and squash code in one line for some languages e.g. Perl.

Marketplace, github.

Michael Feathers: Working Effectively with Legacy Code



book cover Recently I wanted to read something about refactoring and about working with not so good code, so I decided to read Working Effectively with Legacy Code by Michael Feathers. And it seems to be a good book, it contains a lot of recipes and techniques for making the code more testable, for removing dependencies and for making the code better generally.

Although the book is a bit too OOPish and a bit old, I think it’s still useful.

Change iTerm2 tab and window title colors depending on ssh host



At my work, I use macOS with iTerm2 as a terminal. And iTerm2 has fancy escape codes for changing tab and window titles colors:

\033]6;1;bg;red;brightness;255\a
\033]6;1;bg;green;brightness;255\a
\033]6;1;bg;blue;brightness;255\a

So I thought that it will be nice to distinguish different ssh hosts by color. I found on Stack Overflow how to generate color from a string and wrote a python script that extracts host from command line arguments and prints fancy sequences:

#!/usr/bin/python

import sys


def get_host():
    for arg in sys.argv[1:]:
        if not arg.startswith('-'):
            return arg


def str_to_color(s):
    hash = 0
    for c in s:
        hash = ord(c) + ((hash << 5) - hash)

    for i in range(3):
        yield (hash >> (i * 8)) & 0xff


def generate_seqs(color):
    seq = '\033]6;1;bg;{};brightness;{}\a'
    names = ['red', 'green', 'blue']
    for name, v in zip(names, color):
        yield seq.format(name, v)


if __name__ == '__main__':
    host = get_host()
    if host:
        color = str_to_color(host)
        for seq in generate_seqs(color):
            sys.stdout.write(seq)

In action:

➜ ./ssh_color.py mrw.wtf
]6;1;bg;red;brightness;173]6;1;bg;green;brightness;84]6;1;bg;blue;brightness;51

Now we need to create a bash/zsh function that will call our script, run ssh and reset color on exit:

ssh_color () {
    ssh_color.py $*  # I put script in /usr/local/bin/
    trap 'echo -e "\033]6;1;bg;*;default\a"' INT EXIT
    ssh $*
}

alias ssh=ssh_color

And it just works:

screenshot

Nigel Poulton: The Kubernetes Book



book cover white Recently I decided to read something more about Kubernetes and found The Kubernetes Book by Nigel Poulton. And I’ve made a wrong choice because it’s an introductory book explaining basic concepts with very simple examples.

I don’t know, maybe it’s a good book to start working with Kubernetes.

Jim Webber, Ian Robinson, Emil Eifrem: Graph Databases



book cover white Recently I wanted to read something about graph databases and in the Humble Book Bundle, I found Graph Databases by Jim Webber, Ian Robinson and Emil Eifrem. The book is mostly focused on neo4j but has a bit of information about other databases. It has examples of graph data models and real world use cases, also the book contains an information about theoretical parts and neo4j internals. And there’s a lot about Cypher language.

Although in some chapters the book can be described by anything’s a graph if you’re brave enough.

Building a graph of flights from airport codes in tweets



A lot of people (at least me) tweet airports codes like PRG ✈ AMS before flights. So I thought it will be interesting to draw a directed graph of flights and airports. Where airports are nodes and flights are edges.

First of all, I created a twitter application, authorized my account within it and got all necessary credentials:

TWITTER_CONSUMER_KEY = ''
TWITTER_CONSUMER_SECRET = ''
TWITTER_ACCESS_TOKEN = ''
TWITTER_ACCESS_TOKEN_SECRET = ''
USER_ID = ''

As a special marker I chose airplane emoji:

MARKER = '✈'

Then I tried to receive all my tweets with that marker but stuck with a huge problem, twitter REST API doesn’t work with emojis in a search query. So I decided to receive a whole timeline and filter it manually. So only the last 3200 tweets will be parsed. Working with twitter API is very easy with tweepy:

import tweepy


def get_tweets():
    auth = tweepy.OAuthHandler(TWITTER_CONSUMER_KEY, TWITTER_CONSUMER_SECRET)
    auth.set_access_token(TWITTER_ACCESS_TOKEN, TWITTER_ACCESS_TOKEN_SECRET)
    api = tweepy.API(auth)
    cursor = tweepy.Cursor(api.user_timeline,
                           user_id=USER_ID,
                           exclude_replies='true',
                           include_rts='false',
                           count=200)
    return cursor.items()
>>> for tweet in get_tweets():
...     print(tweet)
... 
Status(_api=<tweepy.api.API object at 0x7f876a303ac8>, ...)

Then I filtered tweets with in its text:

flight_texts = (tweet.text for tweet in get_tweets()
                if MARKER in tweet.text)
>>> for text in flight_texts:
...     print(text)
...
ICN ✈️ IKT
IKT ✈️ ICN
DME ✈️ IKT

As some tweets may contain more than one flight, like LED ✈ DME ✈ AUH, it’s convenient to extract all three letter parts and build flights like LED ✈ DME and DME ✈ AUH:

def get_flights(text):
    parts = [part for part in text.split(' ') if len(part) == 3]
    if len(parts) < 2:
        return []

    return zip(parts[:-1], parts[1:])


flights = [flight for text in flight_texts
           for flight in get_flights(text)]
uniq_flights = list(set(flights))
>>> uniq_flights
[('ICN', 'IKT'), ('IKT', 'ICN'), ('DME', 'IKT')]

From edges in uniq_flights it’s very easy to get all nodes:

airports = [airport for flight in flights
            for airport in flight]
uniq_airports = list(set(airports))
>>> uniq_airports
['ICN', 'IKT', 'DME']

So now it’s possible to create a graph with networkx and draw it with matplotlib:

import networkx
from matplotlib import pyplot


graph = networkx.DiGraph()
graph.add_nodes_from(uniq_airports)
graph.add_edges_from(uniq_flights)
networkx.draw(graph, with_labels=True, node_size=1000)
pyplot.draw()
pyplot.show()

The graph is very ugly:

But it’s simple to improve it by using different colors depending on nodes and edges weight, and by using graphviz.

from collections import Counter
from matplotlib import cm


def get_colors(all_records, uniq_records):
    counter = Counter(all_records)
    max_val = max(counter.values())
    return [counter[record] / max_val
            for record in uniq_records]


networkx.draw(graph, 
              with_labels=True,
              node_size=1000,
              width=1.5,
              pos=networkx.nx_pydot.graphviz_layout(graph, prog='neato'),
              cmap=cm.get_cmap('Pastel1'),
              edge_cmap=cm.get_cmap('Pastel2'),
              edge_color=get_colors(flights, uniq_flights),
              node_color=get_colors(airports, uniq_airports))
pyplot.draw()
pyplot.show()

So now it’s much nicer:

Gist with sources.

Baron Schwartz, Peter Zaitsev, Vadim Tkachenko: High Performance MySQL



book cover white Apart from using Cloud SQL, I haven’t touched MySQL for a while, so I decided to freshen up things and read High Performance MySQL by Baron Schwartz, Peter Zaitsev, and Vadim Tkachenko. The book feels solid, it explains how MySQL works (and worked before) inside, what problems storage engines/parser/optimizer/etc have and how to leverage them. It’s kind of nice that a big part of the book is about MySQL scaling. And it’s also good that the book has a lot of information about troubleshooting, debugging, profiling and some MySQL related tools.

Although the book is probably a bit outdated, it covers MySQL versions up to 5.5, but nowadays the latest version is 5.7.