Tom White: Hadoop: The Definitive Guide



book cover Around a month ago I’ve started working with Hadoop, so I decided that I need to read something about it, and chose Hadoop: The Definitive Guide by Tom White that I already had from the Humble Book Bundle. The book contains a lot of information about Hadoop internals and related stack with use cases and examples. And I guess there are too much huge code samples with all boilerplate. And too many pages are like some option – description, there’s already the documentation for things like that.

But overall the book is good. After reading it I feel way more confident with Hadoop.

Christmas lights on MacBook TouchBar



MacBook TouchBar is almost useless, so I decided to put Christmas lights on it. And apparently it was very easy. Electron has very simple API for TouchBar.

As Christmas lights should have distinctive colors, I hardcoded just seven of them:

const colors = [
    '#ff0000',
    '#00ff00',
    '#0000ff',
    '#ffff00',
    '#ff00ff',
    '#00ffff',
    '#ffffff',
];

And as only one colorable element in electron API is TouchBarButton, and there’s space for only eight buttons on the TouchBar, I generated them with defined colors:

const count = 8;

const lights = [];
for (let i = 0; i < count; i++) {
    lights.push(
        new TouchBarButton({
            backgroundColor: colors[i * 3 % colors.length],
        })
    );
}

Centered them with an ugly hack with TouchBarLabel:

const touchBar = new TouchBar([
    new TouchBarLabel({label: "      "}),
    ...lights,
]);

Assigned TouchBar to a dummy electron app:

app.once('ready', () => {
  window = new BrowserWindow({
    frame: false,
    titleBarStyle: 'hiddenInset',
    width: 300,
    height: 100,
  });
  window.loadURL('javascript:document.write("<br><h1>Christmas lights!!!</h1>")');
  window.setTouchBar(touchBar);
});

The last part is the logic of the lights. I made it very simple, every five milliseconds I’m increasing tick and updating colors of buttons with number have the same modulo of three as the tick:

const interval = 500;

let tick = 0;
setInterval(() => {
    for (let i = 0; i < count; i++) {
        if (i % 3 === tick % 3) {
            let index = colors.indexOf(lights[i].backgroundColor);
            index += 1;
            if (index >= colors.length) {
                index = 0;
            }

            lights[i].backgroundColor = colors[index];
        }
    }

    tick++;
}, interval);

By the end I just run the script with electron and got semi-nice Christmas lights and kind of Christmas spirit:

electron macbook_touchbar_christmas_lights.js

Gist with sources.

Make KissCartoon usable with Chromecast



KissCartoon is a nice place to watch cartoons, but it’s not usable with Chromecast at all. It doesn’t play next episode automatically. But it’s very easy to fix with a small Chrome extension. TLDR: Castable KissCartoon.

So how it works?

When you open a page with a player, it’s starting to listen for an ended event:

const enableAutoplay = (player) => player.addEventListener('ended', () => {
  window.localStorage.setItem('autoPlayingBefore', location.href);
  document.getElementById('btnNext').click();
}, false);

When the event emits, it puts current URL in localStorage and clicks next button. After the next page is loaded, it ensures that the previous page is the page with the previous episode:

const isContinuingPlaying = () => {
  const previous = window.localStorage.getItem('autoPlayingBefore');
  window.localStorage.removeItem('autoPlayingBefore');

  const [previousBtn] = document.getElementsByClassName('preev');

  return previousBtn.href === previous;
};

If the extension is sure that we on the page with the next episode, it toggles fullscreen. But we can’t actually toggle fullscreen, it’s not possible to call requestFullscreen because it can be called only from a callback initiated by a user event. So the extension uses a little hack. It dims other elements on the page and expands the player by setting position: fixed. And it works well with Chromecast, the player fills the whole TV.

const enableFullscreen = (player) => {
  const offlight = document.getElementById('offlight');
  player.style.setProperty('position', 'fixed');
  offlight.click();

  let isExited = false;
  document.addEventListener('keyup', ({keyCode}) => {
    if (keyCode === 27 && !isExited) {
      player.style.setProperty('position', '');
      offlight.click();
      isExited = true;
    }
  });
};

And that’s all! Source code, Chrome extension.

Loading/progress indicator for shell with aging emojis



Recently, while waiting for a long-running script to finish, I thought that it would be nice to have some sort of loader with aging emojis. TLDR: we-are-waiting.

The “life” of an emoji is simple:

👶🏿 → 👧🏿 → 👩🏿 → 👱🏿‍♀️ → 👩🏿‍⚕️ → 👵🏿

It contains aging from a baby to grown-up person, one profession, and oldness.

And as we have four colors, two genders, five ages, and 22 professions. We can have a great variety of lives. So as the first thing to do I decided to generate all those variants. Initially, I was planning to implement everything in Go, but it’s not possible to use emojis in Go code, only codepoints. Because of that, I decided to write a little Python script, that will generate Go code with all variants with codepoints instead of emojis.

For that I just copied lines with emojis from getemoji.com and put them in lists:

ages = [
    "👶 👦 👧 👨 👩 👱‍♀️ 👱 👴 👵",
    "👶🏻 👦🏻 👧🏻 👨🏻 👩🏻 👱🏻‍♀️ 👱🏻 👴🏻 👵🏻",
    "👶🏼 👦🏼 👧🏼 👨🏼 👩🏼 👱🏼‍♀️ 👱🏼 👴🏼 👵🏼",
    "👶🏽 👦🏽 👧🏽 👨🏽 👩🏽 👱🏽‍♀️ 👱🏽 👴🏽 👵🏽",
    "👶🏾 👦🏾 👧🏾 👨🏾 👩🏾 👱🏾‍♀️ 👱🏾 👴🏾 👵🏾",
    "👶🏿 👦🏿 👧🏿 👨🏿 👩🏿 👱🏿‍♀️ 👱🏿 👴🏿 👵🏿",
]
ages = [x.split(' ') for x in ages]

roles = [
    "👮‍♀️ 👮 👷‍♀️ 👷 💂‍♀️ 💂 🕵️‍♀️ 🕵️ 👩‍⚕️ 👨‍⚕️ 👩‍🌾 👨‍🌾 👩‍🍳 👨‍🍳 👩‍🎓 👨‍🎓 👩‍🎤 👨‍🎤 👩‍🏫 👨‍🏫 👩‍🏭 👨‍🏭 👩‍💻 👨‍💻 👩‍💼 👨‍💼 👩‍🔧 👨‍🔧 👩‍🔬 👨‍🔬 👩‍🎨 👨‍🎨 👩‍🚒 👨‍🚒 👩‍✈️ 👨‍✈️ 👩‍🚀 👨‍🚀 👩‍⚖️ 👨‍⚖️ 🤶 🎅 👸 🤴",
    "👮🏻‍♀️ 👮🏻 👷🏻‍♀️ 👷🏻 💂🏻‍♀️ 💂🏻 🕵🏻‍♀️ 🕵🏻 👩🏻‍⚕️ 👨🏻‍⚕️ 👩🏻‍🌾 👨🏻‍🌾 👩🏻‍🍳 👨🏻‍🍳 👩🏻‍🎓 👨🏻‍🎓 👩🏻‍🎤 👨🏻‍🎤 👩🏻‍🏫 👨🏻‍🏫 👩🏻‍🏭 👨🏻‍🏭 👩🏻‍💻 👨🏻‍💻 👩🏻‍💼 👨🏻‍💼 👩🏻‍🔧 👨🏻‍🔧 👩🏻‍🔬 👨🏻‍🔬 👩🏻‍🎨 👨🏻‍🎨 👩🏻‍🚒 👨🏻‍🚒 👩🏻‍✈️ 👨🏻‍✈️ 👩🏻‍🚀 👨🏻‍🚀 👩🏻‍⚖️ 👨🏻‍⚖️ 🤶🏻 🎅🏻 👸🏻 🤴🏻",
    "👮🏼‍♀️ 👮🏼 👷🏼‍♀️ 👷🏼 💂🏼‍♀️ 💂🏼 🕵🏼‍♀️ 🕵🏼 👩🏼‍⚕️ 👨🏼‍⚕️ 👩🏼‍🌾 👨🏼‍🌾 👩🏼‍🍳 👨🏼‍🍳 👩🏼‍🎓 👨🏼‍🎓 👩🏼‍🎤 👨🏼‍🎤 👩🏼‍🏫 👨🏼‍🏫 👩🏼‍🏭 👨🏼‍🏭 👩🏼‍💻 👨🏼‍💻 👩🏼‍💼 👨🏼‍💼 👩🏼‍🔧 👨🏼‍🔧 👩🏼‍🔬 👨🏼‍🔬 👩🏼‍🎨 👨🏼‍🎨 👩🏼‍🚒 👨🏼‍🚒 👩🏼‍✈️ 👨🏼‍✈️ 👩🏼‍🚀 👨🏼‍🚀 👩🏼‍⚖️ 👨🏼‍⚖️ 🤶🏼 🎅🏼 👸🏼 🤴🏼",
    "👮🏽‍♀️ 👮🏽 👷🏽‍♀️ 👷🏽 💂🏽‍♀️ 💂🏽 🕵🏽‍♀️ 🕵🏽 👩🏽‍⚕️ 👨🏽‍⚕️ 👩🏽‍🌾 👨🏽‍🌾 👩🏽‍🍳 👨🏽‍🍳 👩🏽‍🎓 👨🏽‍🎓 👩🏽‍🎤 👨🏽‍🎤 👩🏽‍🏫 👨🏽‍🏫 👩🏽‍🏭 👨🏽‍🏭 👩🏽‍💻 👨🏽‍💻 👩🏽‍💼 👨🏽‍💼 👩🏽‍🔧 👨🏽‍🔧 👩🏽‍🔬 👨🏽‍🔬 👩🏽‍🎨 👨🏽‍🎨 👩🏽‍🚒 👨🏽‍🚒 👩🏽‍✈️ 👨🏽‍✈️ 👩🏽‍🚀 👨🏽‍🚀 👩🏽‍⚖️ 👨🏽‍⚖️ 🤶🏽 🎅🏽 👸🏽 🤴🏽",
    "👮🏾‍♀️ 👮🏾 👷🏾‍♀️ 👷🏾 💂🏾‍♀️ 💂🏾 🕵🏾‍♀️ 🕵🏾 👩🏾‍⚕️ 👨🏾‍⚕️ 👩🏾‍🌾 👨🏾‍🌾 👩🏾‍🍳 👨🏾‍🍳 👩🏾‍🎓 👨🏾‍🎓 👩🏾‍🎤 👨🏾‍🎤 👩🏾‍🏫 👨🏾‍🏫 👩🏾‍🏭 👨🏾‍🏭 👩🏾‍💻 👨🏾‍💻 👩🏾‍💼 👨🏾‍💼 👩🏾‍🔧 👨🏾‍🔧 👩🏾‍🔬 👨🏾‍🔬 👩🏾‍🎨 👨🏾‍🎨 👩🏾‍🚒 👨🏾‍🚒 👩🏾‍✈️ 👨🏾‍✈️ 👩🏾‍🚀 👨🏾‍🚀 👩🏾‍⚖️ 👨🏾‍⚖️ 🤶🏾 🎅🏾 👸🏾 🤴🏾",
    "👮🏿‍♀️ 👮🏿 👷🏿‍♀️ 👷🏿 💂🏿‍♀️ 💂🏿 🕵🏿‍♀️ 🕵🏿 👩🏿‍⚕️ 👨🏿‍⚕️ 👩🏿‍🌾 👨🏿‍🌾 👩🏿‍🍳 👨🏿‍🍳 👩🏿‍🎓 👨🏿‍🎓 👩🏿‍🎤 👨🏿‍🎤 👩🏿‍🏫 👨🏿‍🏫 👩🏿‍🏭 👨🏿‍🏭 👩🏿‍💻 👨🏿‍💻 👩🏿‍💼 👨🏿‍💼 👩🏿‍🔧 👨🏿‍🔧 👩🏿‍🔬 👨🏿‍🔬 👩🏿‍🎨 👨🏿‍🎨 👩🏿‍🚒 👨🏿‍🚒 👩🏿‍✈️ 👨🏿‍✈️ 👩🏿‍🚀 👨🏿‍🚀 👩🏿‍⚖️ 👨🏿‍⚖️ 🤶🏿 🎅🏿 👸🏿 🤴🏿",
]
roles = [x.split(' ') for x in roles]

As emojis have a strange order, generation of all variants is a bit tricky, but it’s easier than rearranging them in code because my editor doesn’t work quite well with emojis:

def get_life(color, gender, role):
    yield ages[color][0]
    yield ages[color][1 + gender]
    yield ages[color][3 + gender]
    yield ages[color][6 - gender]
    yield roles[color][role * 2 + 1 - gender]
    yield ages[color][7 + gender]
>>> list(get_life(0, 0, 0))
['👶', '👦', '👨', '👱', '👮', '👴']
def get_variants():
    for color in range(len(ages)):
        for gender in (0, 1):
            for role in range(len(roles[0]) // 2):
                yield color, gender, role
>>> list(get_life(*list(get_variants())[23]))
['👶', '👧', '👩', '👱\u200d♀️', '👷\u200d♀️', '👵']

And after that it’s very easy to generate Go package with all possible variants:

code = b'package variants\n\nvar All = [][]string{\n'

for variant in get_variants():
    code += b'\t{\n'
    for emoji in get_life(*variant):
        code += b'\t\t"' + emoji.encode('unicode-escape') + b'",\n'
    code += b'\t},\n'

code += b'}\n'

with open('variants/variants.go', 'wb') as f:
    f.write(code)

So we’ll have something like this in variants/variants.go:

package variants

var All = [][]string{
	{
		"\U0001f476",
		"\U0001f466",
		"\U0001f468",
		"\U0001f471",
		"\U0001f46e",
		"\U0001f474",
	},
	...
}

The logic of the loader isn’t that interesting, although I want to highlight some moments. At the high level we just read lines from a pipe, if there’s no new line arrived before tick seconds, we update our emojis:

func main() {
        ...
        go watchApp(lines)
        
        for {
            select {
            case line, isOpen := <-lines:
                ...
                os.Stdout.WriteString(line)
                ...
                printPeople(people)
            case <-time.After(time.Duration(*tick) * time.Second):
                people = updatePeople(people, *count)
                printPeople(people)
            }
        }
}

While updating, we can add new emoji, make one emoji older or “kill” the oldest:

func updatePeople(people []*human, count int) []*human {
	addNew := rand.Intn(5) == 0
	toMakeOlder := canMakeOlder(people)

	if addNew || len(toMakeOlder) == 0 {
		people = append(people, getRandomHuman())
	} else {
		index := toMakeOlder[rand.Intn(len(toMakeOlder))]
		people[index].position += 1
	}

	if len(people) > count {
		oldest := getOldest(people)
		return append(people[:oldest], people[oldest+1:]...)
	} else {
		return people
	}
}

And that’s all. You can find the source code on GitHub.

Mark Jason Dominus: Higher-Order Perl



book cover As I sometimes use Perl, I decided to read something about it. And Higher-Order Perl by Mark Jason Dominus was looking interesting. The book is really nice, it shows that Perl can be kind of a functional programming language and that it’s possible to implement almost every feature from other languages in Perl. Also, after reading the book, I think there’s copious amount of ways to shot yourself in the leg in Perl.

The problems in the book are interesting and a bit challenging, so I think it can be worth reading even for people who don’t work with Perl.

Soundlights with ESP8266 and NeoPixel Strip



About a year ago I made soundlights with Raspberry Pi. But RPI is a bit of an overkill for this simple task and it’s quite big, doesn’t have WiFi out of the box and practically can’t be used without a power adapter.

So I decided to port soundlights to ESP8266. The main idea was to reuse as much as possible from the previous implementation, so the parts with patched audio visualizer and colors generation are the same. In a few words, I’ve patched cava to print numbers instead of showing pretty bars in a terminal. And I’ve generated colors with a code found on Quora.

And in current implementation I decided to make it very simple to use, the only requirement is to have a machine with cava and ESP8266 on the same WiFi network. So I chose UDP broadcasting as a way to send data to ESP8266. And because there’s just 60 LEDs and color of a LED is three values from 0 to 255, colors for all strip is just 180 bytes. So it fits in one UDP packet.

Let’s start with the part with cava:

import sys
import socket
import array


COLORS_COUNT = 256
COLORS_OFFSET = 50


def get_spaced_colors(n):
    max_value = 16581375
    interval = int(max_value / n)
    colors = [hex(i)[2:].zfill(6) for i in range(0, max_value, interval)]

    return [(int(color[:2], 16),
             int(color[2:4], 16),
             int(color[4:6], 16)) for color in colors]


def send(colors):
    line = array.array('B', colors).tostring()
    sock.sendto(line, ('255.255.255.255', 42424))


sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)

colors = get_spaced_colors(COLORS_COUNT)

while True:
    try:
        nums = map(int, sys.stdin.readline()[:-1].split())
        led_colors = [c for num in nums for c in colors[num]]
        send(led_colors)
    except Exception as e:
        print(e)

It can be used like:

unbuffer ./cava -p soundlights/cava_config | python cava/soundlights/esp/client.py

And it just reads numbers from cava output, generates colors, transforms them to bytes and broadcasts them at 42424 port.

The ESP8266 part is even simpler:

import socket
import machine
import neopixel


np = neopixel.NeoPixel(machine.Pin(5), 60)
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(('', 42424))


while True:
    line, _ = sock.recvfrom(180)

    if len(line) < 180:
        continue

    for i in range(60):
        np[i] = (line[i * 3], line[i * 3 + 1], line[i * 3 + 2])

    np.write()

It just receives broadcasts from 42424 port and changes colors of LEDs.

At the end, this version has less code and just works. It’s even some sort of IoT and with some effort can become a wearable device.

Github.

Language agnostic REPL driven development with Visual Studio Code



A few years ago I was using Light Table, the integration with Clojure REPL was so nice. But the other parts of the editor and other languages support weren’t that good. And at the moment the editor looks almost dead.

After that, for Clojure, I switched to Cursive, and I still use it. It has a nice feature – send to REPL, which allows users to execute selected code in REPL. But it’s Clojure/ClojureScript only and requires some hassle to configure.

Nowadays for some stuff, I use Visual Studio Code. It doesn’t have a nice integration with REPL, but it has integrated terminal. So I thought, wouldn’t it be nice to just open any REPL in a terminal and somehow send selected code to the REPL. Without any configuration, even with REPL on a remote server.

So I wrote a little extension – SendToREPL. In action with Python REPL:

How does it work? Let’s look at the initial version of the extension:

const vscode = require('vscode');
let terminal;

function activate(context) {
    terminal = vscode.window.createTerminal('SendToREPL terminal');
    terminal.show();

    const command = vscode.commands.registerTextEditorCommand('extension.sendToREPL', (textEditor) => {
        const code = textEditor.document.getText(textEditor.selection);
        terminal.sendText(code);
    });

    context.subscriptions.push(command);
}

exports.activate = activate;

function deactivate() {
    if (terminal) {
        terminal.dispose();
    }
}

exports.deactivate = deactivate;

It just creates a terminal when extension loaded and registers extension.sendToREPL command. When the command is triggered (by Ctrl+’/Cmd+’ hotkey or from Quick Open) it gets selected code and sends it to the terminal.

The current version is a bit more advanced, it sends the line with cursor if nothing selected and squash code in one line for some languages e.g. Perl.

Marketplace, github.

Michael Feathers: Working Effectively with Legacy Code



book cover Recently I wanted to read something about refactoring and about working with not so good code, so I decided to read Working Effectively with Legacy Code by Michael Feathers. And it seems to be a good book, it contains a lot of recipes and techniques for making the code more testable, for removing dependencies and for making the code better generally.

Although the book is a bit too OOPish and a bit old, I think it’s still useful.

Change iTerm2 tab and window title colors depending on ssh host



At my work, I use macOS with iTerm2 as a terminal. And iTerm2 has fancy escape codes for changing tab and window titles colors:

\033]6;1;bg;red;brightness;255\a
\033]6;1;bg;green;brightness;255\a
\033]6;1;bg;blue;brightness;255\a

So I thought that it will be nice to distinguish different ssh hosts by color. I found on Stack Overflow how to generate color from a string and wrote a python script that extracts host from command line arguments and prints fancy sequences:

#!/usr/bin/python

import sys


def get_host():
    for arg in sys.argv[1:]:
        if not arg.startswith('-'):
            return arg


def str_to_color(s):
    hash = 0
    for c in s:
        hash = ord(c) + ((hash << 5) - hash)

    for i in range(3):
        yield (hash >> (i * 8)) & 0xff


def generate_seqs(color):
    seq = '\033]6;1;bg;{};brightness;{}\a'
    names = ['red', 'green', 'blue']
    for name, v in zip(names, color):
        yield seq.format(name, v)


if __name__ == '__main__':
    host = get_host()
    if host:
        color = str_to_color(host)
        for seq in generate_seqs(color):
            sys.stdout.write(seq)

In action:

➜ ./ssh_color.py mrw.wtf
]6;1;bg;red;brightness;173]6;1;bg;green;brightness;84]6;1;bg;blue;brightness;51

Now we need to create a bash/zsh function that will call our script, run ssh and reset color on exit:

ssh_color () {
    ssh_color.py $*  # I put script in /usr/local/bin/
    trap 'echo -e "\033]6;1;bg;*;default\a"' INT EXIT
    ssh $*
}

alias ssh=ssh_color

And it just works:

screenshot

Nigel Poulton: The Kubernetes Book



book cover white Recently I decided to read something more about Kubernetes and found The Kubernetes Book by Nigel Poulton. And I’ve made a wrong choice because it’s an introductory book explaining basic concepts with very simple examples.

I don’t know, maybe it’s a good book to start working with Kubernetes.