Joy-Con Clicker

October 26, 2019

Turning Switch's Joy-Con 🕹 into a clicker is non-trivial effort because you need to poll keypress events by yourself with the Gamepad API.

Note: the context of this note is building the slides features on my site.

This note is not yet completed. But I am working on it and hope to finish this soon(-ish). Most recent update (Nov 3, 2019) is roughly what you can see in this tweet.

Before you may want to read the remaining of this article, I'd like to point out that

the MDN web docs already has a thorough guide Using the Gamepad API, you should be able to learn all what I have to share by reading that amazing guide
the direction I go is more personal towards the stuff I want to build, it may not be systematic enough, and there can be mistakes (which I'll fix if you will kindly let me know)
having said the above, the main point is I invite you to try this by yourselves

Why we absolutely have to do this

Ever since I started building the slides features, I also from time to time toy with the ideas of controlling my slides in some funny ways.

One of the silly ideas is to brew in a few magic words such as, "next page", or, "ok, next" - the words that you will likely say during your talk when you're about to slide to the next page.

Here another one that I discussed with my friend Huijing is to turn her favorite toy, a mechanical key tester that she keeps clicking like a maniac, into a bluetooth connected key(board). My feature request is to have two keys, j and k, respectively, because that's how my slides page forward and backward.

Then we were both at JSConfBP earlier this year. And during the closing keynote by Surma and Jake on the first day she texted me:

Jake and Surma are each holding a switch controller 😁

My thoughts then went a round trip from that point back to first hearing of the idea from Jinjiang who talked about this as he prepared for his talk on keyboard accessibility (slides in Chinese / 挖掘鍵盤的潛能) for Modern Web 19, and brilliantly recognizes accessibility issues as extended explorations of how we interact with devices and hence web pages. Considering the fact that any time I give a talk I consider myself half disabled (brainless and one hand holding on to a mic, likely), I find this idea a perfect fit.

Plus, the idea of holding on to a Joy-Con to control your slides is simply too cool and fun to miss.

The Gamepad API

Before I played with this, my naïve imagination was that I write something similar to keyboard event handlers and be done with it.

But no, WRONG. The only event handlers that the Gamepad API exposes are gamepadconnected and gamepaddisconnected. What essentially we have to do is to start polling keypresses (normally done by starting a gameloop) on gamepadconnected and stop this process on gamepaddisconnected.

Connecting the controller(s)

Browsers will receive the gamepadconnected event when the controllers connect. I think a normal idea is to use this chance to

get notified that a gamepad is connected and display some information
start the game loop (to poll key presses)

window.addEventListener('gamepadconnected', evt => {
  document.getElementById(
    'gamepad-display'
  ).innerHTML = `Gamepad connected at index ${evt.gamepad.index}: ${evt.gamepad.id}. ${evt.gamepad.buttons.length} buttons, ${evt.gamepad.axes.length} axes.`;

  // gameloop() // start game loop
});

And when the controllers disconnect, normally we

get notified
stop gameloop

window.addEventListener('gamepaddisconnected', () => {
  console.log('Gamepad disconnected.');
  // cancelAnimationFrame(start); // related with game loop
});

I find it more readable to give those functions names and later on put them together. You may want to use an object-oriented style and put them in an object or a class later on. Or maybe they can be a separate hook.

// gamepad.js
export const gamepadConnect = evt => {
  document.getElementById(
    'gamepad-display'
  ).innerHTML = `Gamepad connected at index ${evt.gamepad.index}: ${evt.gamepad.id}. ${evt.gamepad.buttons.length} buttons, ${evt.gamepad.axes.length} axes.`;
};

export const gamepadDisconnect = () => {
  console.log('Gamepad disconnected.');
};

Describing the current status

If you also have very simple brain like I do, you'll probably also like the gamepad API. There's no fancy wrapping / controller / whatever, just that evt object containing real time data regarding the status of your Joy-Cons.

I'm also brain washed by the two keywords in the React community, "immutable" and "re-render" and so for a brief moment I could not comprehend the fact that that evt object just quietly gets updated to the Joy Con's current state. There's no "re-rendering" and everything mutates.

Let's take a closer look at what's inside that evt object:

{
  "axes": [1.2857142857142856, 0],
  "buttons": [
    {
      "pressed": false,
      "touched": false,
      "value": 0
    }
    // ... more (altogether 17) buttons
  ],
  "connected": true,
  "displayId": 0,
  "hand": "",
  "hapticActuators": [],
  "id": "57e-2006-Joy-Con (L)",
  "index": 0,
  "mapping": "standard",
  "pose": {
    "angularAcceleration": null,
    "angularVelocity": null,
    "hasOrientation": false,
    "hasPosition": false,
    "linearAcceleration": null,
    "linearVelocity": null,
    "orientation": null,
    "position": null
  },
  "timestamp": 590802
}

Those fields together describe the concurrent status of the joy con. And in particular:

axes: array describing the joystick navigation
buttons: array of "buttons" object, each describing whether it's pressed or touched, value being from 0 to 1 describing the physical force
timestamp: can be used to decide which update comes latest
pose: handhold data, looking very powerful ah

What is different from our normal mindset of interactions with browsers is that, the browsers have already done a lot of work for us, such as polling key presses and exposes a declarative API where we can say onKeyPress then do this.

Essentially, what we want to do is to keep polling this object to acquire the most current information. And in this context we do so by implementing a "game loop".

Game loop

The main idea is game loop is that it's a snake that eats it's own tail.

const gameLoop = () => {
  // does things

  // run again
  gameLoop(); // but what we actually do is requestAnimationFrame(gameLoop)
};

And if you remember the gamepadConnect we wrote earlier, besides announcing that our Joy Con is connected, we start the game loop there:

const gamepadConnect = evt => {
  console.log('connected');
  // start the loop
  requestAnimationFrame(gameLoop);
};

And then, we define the actual interactions inside our game loop.

I happen to have drawn a Switch using CSS roughly a year ago 😅 So convenient, let's use that. If you're interested, here is the original CodePen for the Nintendo Switch art. And once again, the original design credits to its original creator Louie Mantia.

Original design Nintendo Switch art by Louie Mantia

And inside each run of the game loop, we take the information we need. For example, if we want to acquire whether certain button is pressed, we can use the example code from MDN directly:

const buttonPressed = key => {
  if (typeof key == 'object') {
    return key.pressed;
  }
  return false;
};

And since each button's statuses is stored in the evt data object in its own corresponding buttons array, in a particular order¹, we then can poll each particular button:

// 8 is 'minus'
if (buttonPressed(gp.buttons[8])) {
  console.log('pressed minus!');
}

And since a button is either pressed or not pressed, we can then use this to toggle a "shake" UI on those buttons:

// 8 is 'minus'
if (buttonPressed(gp.buttons[8])) {
  document.getElementById('minus').classList.add('active');
} else {
  document.getElementById('minus').classList.remove('active');
}

More complex interactions can be acquired by waiting a little between actions, which I won't go into at the moment. But you can 😉

You can check out this CodeSandbox for a demo of the key pollers. Of course this is not the tersest implementation, but there's no mind burning function passes. It's good for brain health.

This section is WiP.

highlight pointer - moving sth using the joy stick

Debouncing

This section is WiP.

needed by slides (it slided all the way to the end, then all the way back up, before debouncing it)

Revisit game loop

This section is WiP.

https://gameprogrammingpatterns.com/game-loop.html

We're not actually writing game loop. We're interopting with the browsers' requestAnimationFrame.

Maybe we can borrow some of the idea and optimize the interaction a little bit.

Usages with React

This section is WiP.

making it have peace with react lifecycle and hook
the joy con highlight pointer mover janks when controlling the position using React
having the gameloop directly write to DOM keeps the framerate acceptably high

import * as React from 'react';
import { gamepadConnect, gamepadDisconnect } from './gamepad';

const JoyConController = () => {
  React.useEffect(() => {
    window.addEventListener('gamepadconnected', gamepadConnect);
    window.addEventListener('gamepaddisconnected', gamepadDisconnect);
  }, []);
  return <div className="gamepad-display" />;
};

There are multiple ways to design this API.

we can pass a ref to the game loop, with which our lower level implementation can directly write to that element
we can pass a callback function, i.e., setState, that our lower level implementation can call to update data in our React component, which will cause re-render

Caveats

¹Each browser has different mappings. In some cases, even whether you connect two Joy Cons at the same time result in different mappings.

the Joy-Con never turns off ah?
FireFox crashes >90% of the time when you connect two JoyCon to two tabs

Takeaways

learn browser APIs
do not copy other people's abstractions, start from scratch and come up with your own

Joy-Con Clicker

Why we absolutely have to do this

The Gamepad API

Connecting the controller(s)

Describing the current status

Game loop

Debouncing

Revisit game loop

Usages with React

Caveats

Takeaways

Links