The blog of dlaa.me

Make the Pi Higher [Configuring a Raspberry Pi for Node.js and web development]

Since the beginning of the year, I've been doing all my OSS project development on a Raspberry Pi 4 running Raspberry Pi OS (née Raspbian). It's not as powerful a machine as my Intel-based desktop PC running Windows, but it's much smaller, it uses much less power, it's completely silent, and it cost about 5% of the price. To be clear, I still do photo editing on Windows and the extra speed of the PC is nice for running unit tests - but most of the time I'm perfectly happy using the Pi.

There are some great resources for Setting up your Raspberry Pi. If you want to configure it for development work like I did, here are steps that worked for me. (They assume some familiarity with Linux, but not much.)

  1. Install Raspberry Pi OS (32-bit) with desktop
  2. sudo apt update
  3. sudo apt upgrade
  4. If you want to use the Xfce desktop environment:
    1. sudo apt install xfce4 xfce4-terminal
    2. sudo update-alternatives --config x-session-manager
    3. Choose startxfce4
  5. Install the Git UI tools and configure Git
    1. sudo apt install git-gui gitk
    2. Set your username in Git
    3. Set your commit email address in Git
    4. Connect to GitHub with SSH
  6. Install a current version of Node.js
  7. Install a community build of Visual Studio Code
    1. Trust the corresponding GPG key: wget -qO - https://packagecloud.io/headmelted/codebuilds/gpgkey | sudo apt-key add -
  8. Install the ARM build of Visual Studio Code
  9. Install Visual Studio Code via apt install code

Note: The community build of VS Code is the only part of this process that uses an unofficial release. I'd love for VS Code to support the Pi officially, but this issue from 2016 has not been addressed. Everything is official now that VS Code supports Raspberry Pi!

That's pretty much it! I really enjoy this minimalist configuration and find there are very few compromises with the Pi relative to a "real" computer. Plus, it's easy to swap drives (SD cards) and keep a set of different OS'es / configurations handy for experimenting and trying new things.

If you thought the Pi was just a toy, maybe it's time for another look!

Updated 2020-11-22 to link to official builds of Visual Studio Code for ARM.

Without geometry, life is pointless [Another interesting programming puzzle worked through]

I happened across another coding puzzle recently that caught my interest. (If you're curious, here is a link to the puzzle I discussed a few weeks ago about implementing a weird kind of addition.) The problem this time is as follows:

You are given an array of coordinates, coordinates[i] = [x,y], where [x,y] represents the coordinate of a point. Check if these points make a straight line in the XY plane.

I saw an approach to solving this that didn't thrill me, so I wanted to solve it for myself.

If you're going to do the same, stop reading now because there are spoilers ahead!

As an aside, one thing I don't love about this problem is that it requires a certain amount of geometry knowledge to solve and furthermore the solution I use benefits from experience with trigonometric functions. However, one can be a perfectly good programmer without knowing either of these topics and they're not likely to come up in most roles. I try to avoid questions like this when I'm interviewing because they're likely to favor some backgrounds over others. That said, I didn't come up with this scenario and it's still interesting to think about, so let's go.

The first thing is to figure out how to approach the problem. If you remember much about linear equations, you might think to use the equation for a line to solve this: y = mx + b. If the slope, m, and the y-intercept, b, are the same for all pairs of points, then they all lie on the same line. This is helpful, but the equation starts to fall apart for points that lie on the same vertical line (for example, along the y-axis) because the slope there is infinite. Now, maybe you are comfortable dealing with infinity or risking division by zero, but I am not. That makes the vertical line scenario a special case and "special case" is often another phrase for "bug farm".

We don't want to use the slope equation, but we're in luck because there's something similar that's easier to work with: angle. Specifically:

The Math.atan2() function returns the angle in the plane (in radians) between the positive x-axis and the ray from (0,0) to the point (x,y), for Math.atan2(y,x).

That's not a solution as-is, but if we choose two points from the input and subtract the corresponding x and y coordinates, we get coordinates that define a ray in the manner described above. Now, two sets of points on line segments sharing the same angle (as calculated above) will be parallel, but not necessarily collinear. However, if we use the same point to calculate the angle to every other point of the input, that should be sufficient to determine collinearity of the set. In words:

If the angle of the line segment made up by pairing the first point of the input array with every other point of the array is the same, then we know all those rays point in the same direction and all originate at the same point (the first point) and therefore all the points are on the same line!

We're nearly there, but there's one more observation: two rays that go in exactly opposite directions from the same point are also on the same line. For example, the line segment defined by two points on a ray with an angle of 10° to the x-axis are parallel to the segment corresponding to two points on a ray with an angle of 190° to the x-axis. So instead of worrying about all 360°, we can focus on just the first 180°. The modulo operator provides an easy way to map angles between 180° and 360° into the range of 0° to 180°. (Except that the atan2 function returns values in radians instead of degrees because "Why not?" so we're going to be dealing with π.)

With all that behind us, the following algorithm should make sense. There are a few subtleties so it correctly handles empty lists as well as lists with just one point and also lists where the same point appears multiple times. In addition, it bails out as soon as it finds a non-collinear point for efficiency. Here's the code - as is the custom with JavaScript, there's no parameter validation, but this is intended to work for all valid inputs:

function arePointsCollinear(points) {
  let angle = null;
  const first = points[0];
  for (let i = 1; i < points.length; i++) {
    const current = points[i];
    const deltaX = current[0] - first[0];
    const deltaY = current[1] - first[1];
    if (deltaX || deltaY) {
      const theta = (Math.atan2(deltaY, deltaX) + Math.PI) % Math.PI;
      if (angle === null) {
        angle = theta;
      } else if (angle !== theta) {
        return false;
      }
    }
  }
  return true;
}

Interviewing for Fun and Profit [An unreasonably detailed consideration of a practical programming puzzle]

A few days ago, Brent Simmons published a thoughtful blog post about one aspect of programming interviews. As someone who conducts interviews regularly for work, the topic is of interest to me. Rather than summarizing the post, I encourage you to go read Practicing the Coding Challenges now. (You might also enjoy some of the replies to Brent's tweet linking to it.)

After reading the post, I tweeted a response:

I agree with the spirit of this post - that clarity beats premature optimization - but feel the conclusion skips an important point. The "efficient" solution is correct for arbitrarily large numbers while the "clear" one fails after MAX_INT (or similar).

Brent replied:

I thought about discussing that, but felt like I'd be muddying up the post too much.

And I responded:

Understood, thanks! I don't know how it was posed on LeetCode, so that difference may not even be relevant. But if I were discussing this with a candidate during an interview, it's something I'd want them to raise for discussion as an indication they're thinking about edge cases.

And I kept thinking about this problem in the context of a programming interview. When I saw Olof Hellman's well-reasoned follow-up post Practicing the Coding Challenges, I realized I wasn't the only one.

I was curious what the "clear" and "efficient" solutions might look like in practice, so I decided to write them and see. I used JavaScript because it's my "go to" language. Things will look a little different in other languages, but I had to pick one and I chose something popular. I'm not including comments in the code below because they'll distract from it and I'm not validating parameters (for null-ness or range) per typical JavaScript practice.

As a reminder, here's the problem definition:

You need to add two numbers and get a sum. But, importantly, the digits of those numbers are stored in arrays, and they're backwards.

The return value also needs to be in a backwards array.

If inputs are [4,2,6] and [3,8,0,8], the function should return [7,0,7,8] - because 624 + 8083 == 8707.

Let's start with the "clear" approach. The point here is to be obvious and straightforward - like you normally want your code to be. Recall Brent's pseudocode proposal:

let num1 = number(from: array1)
let num2 = number(from: array2)
let sum = num1 + num2
return array(from: sum)

Now, in order to run that, we need to implement those two conversion functions. As John Siracusa pointed out, we can do anything we want inside there and because we want to handle arbitrarily large inputs, we'll use JavaScript's BigInt built-in.

As an aside, BigInt does not have widespread support across browsers. In fact, none of the three JavaScript tools I use on iOS support it and I had to switch to a "real" computer to finish this post.

Okay, here's what I came up with:

function clear(array1, array2) {
  const num1 = toBigInt(array1);
  const num2 = toBigInt(array2);
  const sum = num1 + num2;
  return toArray(sum);
}
function toBigInt(arr) {
  const copy = [...arr];
  copy.reverse();
  const numberAsString = copy.join("");
  return BigInt(numberAsString);
}
function toArray(num) {
  const numString = num.toString();
  const charArray = numString.split("");
  const digitArray = charArray.map(Number);
  digitArray.reverse();
  return digitArray;
}

Because this is the "clear" implementation, each line does a single thing and I've mostly avoided idiomatic JavaScript. You could argue map(Number) is a bit subtle, but it should be familiar to most JavaScript programmers. The BigInt constructor takes a Number or a String, so we need to bounce the input through a String on the way into/out of BigInt. We copy the input array to avoid mutating data we don't own.

Okay, so what does the "efficient" code look like? Here's what I came up with:

function efficient(array1, array2) {
  const length = Math.max(array1.length, array2.length);
  const res = new Array(length);
  let carry = 0;
  for (let i = 0; (i < length) || carry; i++) {
    const sum = (array1[i] || 0) + (array2[i] || 0) + carry;
    carry = (sum >= 10) ? 1 : 0;
    res[i] = sum % 10;
  }
  return res;
}

Because this isn't the "clear" solution, I took some liberties with this implementation in the interest of compactness and efficiency. The output array is pre-allocated (though it may need to grow by one) and there is a single loop over the input with no redundancy and fairly little overhead I see. I'll acknowledge some folks might be uncomfortable with the (deliberate) ignorance of array bounds and that the use of carry in the loop condition may be too clever by half. (It can be removed at the cost of a single additional line of code, but I like how this approach reuses the core logic for the final overflow.) Otherwise, I tried not to be unnecessarily obscure.

Conclusions?

The "clear" code is definitely clearer, but it's not as simple as it originally seemed to be. The need to implement helper functions basically tripled the amount of code that needed to be written. The "efficient" code is smaller and ought to be quite a bit cheaper to run considering the algorithmic efficiency and the fact that it doesn't require JIT-ing BigInt or the need for String parsing. In fact, by eschewing those dependencies, it's able to run in many environments (like iOS) where BigInt doesn't exist - so it's also the more "flexible" implementation!

Of course, there's no "right" answer here - just inspiration and food for thought. My thanks go out to everyone who contributed to the discussion and especially Brent, whose NetNewsWire app is my favorite RSS reader!

Existence is random [JavaScript code to efficiently generate a random (version 4) UUID]

A few months ago, I dashed out a JavaScript function to efficiently generate a random (version 4) UUID per RFC 4122. This was inspired by code I saw for the same purpose that was not especially efficient - or random. (For more on randomness in JavaScript, see my post about polyfilling Math.random().) My UUID generator uses crypto.getRandomValues() for the best randomness and tries to do as little extra work as possible for efficiency.

The generateRandomUUID page on GitHub has the code and a link to the generateRandomUUID sample page that runs in your browser and generates a new UUID every time it's loaded.

Because it's so small, I've included the code here for reference:

// generateRandomUUID.js
// https://github.com/DavidAnson/generateRandomUUID
// 2019-10-27

"use strict";

// JavaScript code to efficiently generate a random UUID per RFC 4122
function generateRandomUUID() {
  // UUIDs have 16 byte values
  const bytes = new Uint8Array(16);
  // Seed bytes with cryptographically random values
  crypto.getRandomValues(bytes);
  // Set required fields for an RFC 4122 random UUID
  bytes[6] = (bytes[6] & 0x0f) | 0x40;
  bytes[8] = (bytes[8] & 0x3f) | 0x80;
  // Convert bytes to hex and format appropriately
  const uuid = Array.prototype.map.call(bytes, (b, i) => {
    // Left-pad single-character values with 0,
    // Convert to hexadecimal,
    // Add dashes
    return ((b < 16) ? "0" : "") +
      b.toString(16) +
      (((i % 2) && (i < 10) && (i > 2)) ? "-" : "");
  }).join("");
  // Return the string
  return uuid;
}

Let's go to the video tape! [tape-player is a simple, terse, in-process reporter for the tape test harness for Node.js]

I've been a happy user of the nodeunit test harness for a long time, but it was deprecated a few years ago. Recently, I went looking for a similar Node.js test harness to replace it. I prefer small, simple packages and settled on the tape test harness. I enjoy nearly everything about it, but didn't like having to pipe output to a formatter (more on this below). So I wrote a quick bit of code to create an in-process reporter. Then I realized what I'd done could have broader applicability (in my own projects, if nowhere else!) and published a reusable package after adding scenario tests to ensure formatted output for all of the the tape primitives is reasonable.

If this seems interesting, the README goes into more detail:

The Test Anything Protocol (TAP) used by many test harnesses is versatile, but it's not much to look at - or rather, it's too much to look at. There are many custom formatters that work with the tape test harness, but most work by piping process output. This is a useful technique, but interferes with the exit status of the test harness which is a problem in scripts that are meant to fail when tests fail (like npm test). (Though there are workarounds for this, they are shell- and platform-specific.)

Fortunately, tape offers an alternative logging mechanism via its createStream API. This technique is easy to use and runs in-process so it doesn't interfere with the exit status of the test harness. tape-player takes advantage of this to produce a concise test log that's easy to enable.

You can find directions to install and enable tape-player on the GitHub project for tape-player.

Good, cheap, fast: you only get one and you don't get to pick [CoLR, the Camera of Last Resort]

One way I learn about new technologies is by using them. Sometimes I have a compelling scenario that yearns for a creative solution. Other times I don't. This was one of those times... So I created what's probably the worst camera app ever as a way of experimenting with four new (to me) technologies.

Excerpting from the README:

  • CSS Grid Layout - A modern layout system based on grid cells with the great working reference A Complete Guide to Grid
  • MediaDevices.getUserMedia() API - The part of the WebRTC Media Streams API that allows websites to access the device camera for video and image capture
    • Verdict: A somewhat cumbersome API that's nonetheless quite powerful
  • Preact - Exactly what it says on the box, a "Fast 3kB alternative to React with the same modern API"
    • Verdict: Works like a charm, easy for someone with React.js experience
  • Dexie.js - More truth in advertising, a "Minimalistic Wrapper for IndexedDB", where IndexedDB is "a JavaScript-based object-oriented database" that runs in the browser
    • Verdict: Easy to use, even for someone with no prior IndexedDB experience

If you want to learn more, please visit the CoLR project on GitHub. There, you'll find complete source code, instructions, links to resources, and some FAQ's.

If you just want to see how bad a camera app can be (and your device has a camera and your browser supports ECMAScript 2015), please open the CoLR web app in your browser.

Oops, I did it again [A rewrite of my website/blog platform - now public, open-source, and on GitHub as simple-website-with-blog]

Almost 5 years ago, I moved my blog (and website) from a hosted environment to a Node.js implementation of my own creation. At the time, I was new to Node and this was a great way to learn. That code has served me well, but over time I've found a few things I wanted to change - in part due to the rapid growth of the Node platform. Some of the changes were foundational, so I chose to do a rewrite instead of multiple revisions.

The rewrite ended up taking about twice as long as I anticipated, but I'm glad I did it. The new architecture is similar to what it was before - and the user interface almost identical - but there are some nice improvements and modernizations under the hood. Of note, the rendering was separated so it's easy to customize (there are samples for a text blog and a photo blog; the unit tests are implemented as a blog, too), search is much more powerful (with inclusion, exclusion, partial matches, etc.), and the code is simplified by the removal of some undue complexity (mostly features I thought were neat but that added little). Finally, because the new project was written to be reused for other purposes and by other people, I'm able to share it.

For context, the project goals are:

  • An easy way to create a simple, secure website with a blog
  • Support for text-based and photo-based blog formats
  • Easy authoring in HTML, Markdown (with code formatting), or JSON
  • Ordering of posts by publish date or content date
  • Easy customization of site layout and formatting
  • High resolution (2x) support for photo blog images
  • Support for Windows and Linux hosting with Node.js
  • Simple post format that separates content and metadata
  • Ability to author hidden posts and schedule a publish date
  • Ability to create posts that never show up in the timeline
  • Support for archive links and tagging of posts by category
  • Quick search of post content, including simple search queries
  • Automatic Twitter and Open Graph metadata for social media
  • Automatic cross-linking of related posts
  • No JavaScript requirement for client browsers

To learn more about the code, its dependencies, and how to use it, please visit: simple-website-with-blog on GitHub

To see it in action, just browse this blog and site. :)

A picture is worth a thousand words [A small script to update contact photos on iOS]

I'm always looking for new ways to develop code. My latest adventure was writing a script to update contact photos on iOS for a prettier experience in the Messages app conversation list. The script is called update-ios-contact-images.js and it runs in Scriptable, a great app for interacting with the iOS platform from JavaScript. The idea is:

There are various conditions where Apple iOS can't (or won't) synchronize Contact photos between iPhone/iPad devices. If you're in this situation and want it to "just work", you can configure each device manually. Or you can run this script to do that for you.

update-ios-contact-images.js takes a list of email addresses and optional image links and sets the photo for matching contacts in your address book. If an image link is provided, it's used as-is; if not, the Gravatar for that email address is used instead.

You can find more in the update-ios-contact-images.js repository on Github, but the code is short enough that I've included it below.

Notes

  • As I said on Twitter, this was the first meaningful programming project I did completely on iPad (and iPhone). Research, prototyping, coding, debugging, documentation, and posting to GitHub were all done on an iOS device (using a Bluetooth keyboard at times). The overall workflow has some rough edges, but many of the pieces are there today to do real-world development tasks.
  • I'm accustomed to using JavaScript Promises directly, but took this opportunity to try out async and await. The latter are definitely easier to use - and probably easier to understand (though the syntax error JavaScriptCore gives for using await outside an async function is not obvious to me). However, the lack of support for "parallelism" by async/await means you still need to know about Promises and be comfortable using helpers like Promise.all (so I wonder how much of a leaky abstraction this ends up being).
    • Yes, I know JavaScript is technically single-threaded in this context; that's why I put the word "parallelism" in quotes above. :)

Code

// update-ios-contact-images
// A Scriptable (https://scriptable.app/) script to update contact photos on iOS
// Takes a list of email accounts and image URLs and assigns each image to the corresponding contact
// https://github.com/DavidAnson/update-ios-contact-images

// List of email accounts and images to update
const accounts = [
  {
    email: "test1@example.com"
    // No "image" property; uses Gravatar
  },
  {
    email: "test2@example.com",
    image: "https://example.com/images/test2.jpg"
  }
];

// MD5 hash for Gravatar (see https://github.com/blueimp/JavaScript-MD5 and https://cdnjs.com/libraries/blueimp-md5)
eval(await (new Request("https://cdnjs.cloudflare.com/ajax/libs/blueimp-md5/2.10.0/js/md5.min.js")).loadString());

// Load all address book contacts
const contacts = await Contact.all(await ContactsContainer.all());

// For all accounts...
await Promise.all(accounts.map(account => {
  // Normalize email address
  const emailLower = account.email.trim().toLowerCase();
  // For all contacts with that email...
  return Promise.all(contacts.
    filter(contact => contact.emailAddresses.some(address => address.value.toLowerCase() === emailLower)).
    map(async contact => {
      // Use specified image or fallback to Gravatar (see https://en.gravatar.com/site/implement/images/)
      const url = account.image || `https://www.gravatar.com/avatar/${md5(emailLower)}`;
      // Load image from web
      contact.image = await (new Request(url).loadImage());
      // Update contact
      Contact.update(contact);
      console.log(`Updated contact image for "${emailLower}" to "${url}"`);
    }));
}));

// Save changes
Contact.persistChanges();
console.log("Saved changes");

Looking for greener pastures [A Practical Comparison of Mastodon and Micro.blog]

I've been looking into Twitter alternatives Mastodon and Micro.blog recently. I couldn't find a good comparison of the two services, so I created one and put it on GitHub to quickly iterate on any feedback. That's happened, so I'm posting the comparison here where it's easier to find. Enjoy!

A Practical Comparison of Mastodon and Micro.blog

Many of us are looking at Twitter alternatives and there are two services that stand out: Micro.blog and Mastodon.

These services take different approaches, so choosing one is challenging. This page highlights some of the differences and is meant for non-nerds who don't want to get bogged down by implementation details. Every attempt has been made to be accurate, but some technical details are deliberately glossed over.

For more about similar services, see the Comparison of microblogging services on Wikipedia

Mastodon Micro.blog
Web site https://joinmastodon.org/ https://micro.blog/
Sales pitch "Social networking, back in your hands. Follow friends and discover new ones. Publish anything you want: links, pictures, text, video. All on a platform that is community-owned and ad-free." "A network of independent microblogs. Short posts like tweets but on your own web site that you control. Micro.blog is a safe community for microblogs. A timeline to follow friends and discover new posts. Blog hosting built on open standards."
Best price Free Free, but requires a separate blog for posting
Actual price Free $5 per month, no blog needed
Harassment and abuse https://blog.joinmastodon.org/2018/07/cage-the-mastodon/ https://help.micro.blog/2018/twitter-differences/
Code of conduct Depends on the server (Example) https://help.micro.blog/2017/community-guidelines/
Privacy policy Depends on the server https://help.micro.blog/2018/privacy-policy/
Hashtags in posts Yes No
Replies handled differently No Yes
Able to export content Yes Yes
Cross-posting to Twitter No Yes, with a $2/month or $5/month subscription
Import Twitter friends Yes No
Official iOS or Android app No iOS only
Official Mac or Windows app No Mac only
Third-party clients Yes Yes
Security User name + password (2FA is optional) Email address only

Deliberately omitted from above: User counts, open source status, federation details

I'm not part of either project, so there may be mistakes in the table above. That's why this is on GitHub - please open an issue or send a pull request to correct any problems you find. If you are adding content, please do so for both platforms and link to your sources.

For other questions or to start a discussion, contact me on:

Convert *all* the things [PowerShell script to convert RAW and HEIC photos to JPEG]

Like most people, I take lots of photos. Like many people, I save them in the highest-quality format (often RAW). Like some people, I edit those pictures on a desktop computer.

Support for RAW images has gotten better over the years, but there are still many tools and programs that do not support these bespoke formats. So it's handy to have a quick and easy way to convert such photos into a widely-supported format like JPEG. There are many tools to do so, but it's hard to beat a command line script for simplicity and ease of use.

I didn't know of one that met my criteria, so I wrote a PowerShell script:

ConvertTo-Jpeg - A PowerShell script that converts RAW (and other) image files to the widely-supported JPEG format

Notes:

  • This script uses the Windows.Graphics.Imaging API to decode and encode. That API supports a variety of file formats and when new formats are added to the list, they are automatically recognized by the script. Because the underlying implementation is maintained by the Windows team, it is fast and secure.
    • As it happens, support for a new format showed up in the days since I wrote this script: Windows 10's April 2018 Update added support for HEIC/HEIF images such as those created by iPhones running iOS 11.
  • The Windows.Graphics.Imaging API is intended for use by Universal Windows Platform (UWP) applications, but I am using it from PowerShell. This is unfortunately harder than it should be, but allowed me to release a single script file which anybody can read and audit.
    • Transparency is not a goal for everyone, but it's important to me - especially in today's environment where malware is so prevalent. I don't trust random code on the Internet, so I prefer to use - and create - open implementations when possible.
  • The choice of PowerShell had some drawbacks. For one, it is not a language I work with often, so I spent more time looking things up than I normally do. For another, it's clear that interoperating with UWP APIs is not a core scenario for PowerShell. In particular, calling asynchronous methods is tricky, and I did a lot of searching before I found a solution I liked: Using WinRT's IAsyncOperation in PowerShell.
  • There are some obvious improvements that could be made, but I deliberately started simple and will add features if/when the need arises.