Playdate integration

[midouest]

What do folks think about adding Playdate integration to Norns? I have a feature complete implementation running on my Norns, but I'd like to get some feedback on the big picture before I ask anyone to look at the code.

Motivation

From the perspective of Norns, Playdate is basically a retro game controller with two notable exceptions: it has a crank and it runs Lua. I'm primarily interested in using the crank to interact with Norns and Crow. There are lots of possibilities in the other direction too, such as controlling Playdate music apps with a midi keyboard, a grid or control voltage signals.

Implementation

The Playdate supports serial connections over USB. Official documentation for the serial connection can be found in the Playdate Lua serial API docs and Playdate C serial API docs. There is also unofficial documentation of the USB serial capabilities by the Playdate Reverse Engineering project.

The serial connection can be used to enable a controller mode that streams the button, crank and accelerometer state over serial. It does not (to my knowledge) support the HID protocol, so it could not be used with the Norns gamecontroller API without modifications.

Proposed API

Methods

playdate.connected(): true if Playdate is connected

• The Playdate serial connection is only available if the Playdate is connected to USB and unlocked.

playdate.send( line ): send a string message to Playdate using the msg serial command

playdate.run( path ): launch the PDX at the given path on Playdate

• This could be convenient if there is a custom PDX associated with the Norns script. The user would be able to start the Norns script and automatically have the Playdate in the expected state

playdate.controller_start(): start controller mode
playdate.controller_stop(): stop controller mode

• These methods allow Norns scripts to react to Playdate hardware state changes by overriding certain callback functions.
• While controller mode is enabled, the Playdate screen stops updating and no longer responds to button presses except for the lock button. The lock button will stop controller mode on the Playdate side. There is no indication over the serial connection that controller mode has stopped other than no longer receiving updates.
• Controller mode might ultimately be a shortcut to getting date from Playdate into Norns without having to write a custom PDX. Users could also replicate or extend the behavior from controller mode in their custom PDX using print() in Lua or playdate->system->logToConsole() in C with the expected message format.

Device Management Callbacks

function playdate.add( id, name, dev )
function playdate.remove( id )

• These could be used to reactivate controller mode if the device auto-locks and is later unlocked.

Controller Event Callbacks

function playdate.accel( x, y, z )

• received once per frame with the current accelerometer vector
• x, y, and z are floats between -1.0 and 1.0

function playdate.button( b, s )

• received on the frame that any button except the lock button is depressed or released
• b is one of A, B, U, D, L, R or M, and s is 1 if depressed and 0 if released

function playdate.crank( d )

• received each frame that the crank position is updated
• d is the current angle of the crank in degrees (0.0 to 360.0)

function playdate.crankdock( s )

• received on the frame that the crank is docked or undocked
• s is 1 if docked and 0 if undocked

Serial Event Callbacks

function playdate.event( line )

• called for all non-controller-mode serial messages
• line is a string sent from the PDX using print or logToConsole
• used to implement custom serial connection between a Playdate app and a Norns script

Other interesting serial APIs:

buttons

• This enables the button testing mode of the playdate. It could be used to implement Norns event callbacks with slightly different behavior from the controller mode implementation:
  • It sends the current state of all buttons and the crank every frame
  • It does not send accelerometer data
  • It sends the state of the lock button and prevents the device from being locked
• This might some things easier like measuring how the crank position changes over time, but I don't think it's a net positive over using controller mode or implementing the behavior you need in a custom PDX.

bitmap

• Send a 400x240 1-bit bitmap to the Playdate's screen
• I could see this being nice when controller mode is enabled since the Playdate screen is non-interactive at this point, but probably not worth it.

screen

• Take a screenshot of the Playdate
• Might be nice for documentation, debugging or issue reporting

eval

• Send a compiled Lua function to playdate and execute it
• Possibly a shortcut to implementing serial message handling on the Playdate side, but most apps would probably be better off responding to msg serial events.
• Could also be a way to interact with apps that don't explicitly support serial communication, but that may require reverse engineering the PDX.

stream

• This continually streams the playdates screen, audio and buttons over the serial connection. It is used to implement the Mirror app which mirrors the Playdate on a connected computer.
• The audio stream might be the most interesting thing here, but you could also just connect the Playdate headphone output to the Norns audio input.

btn / changecrank / dockcrank / accel

• Simulate button presses on Playdate
• Could be an interesting way to let Norns interact with Playdate, but in most cases users would probably want to implement the serial message handlers.

autolock

• Change the autolock setting on Playdate
• It might be desirable to change Playdate to only autolock on battery when connected to Norns. I could also see users preferring to set this themselves and not wanting Norns script mess with this setting on their device.

Open Questions / Other Considerations

1. The implementation in my Norns fork is mostly a copy of the crow implementation with some additional C APIs exposed to Lua. While I was writing it I was wondering if it might be worth having a more generic serial device interface that can read/write messages and defer to Lua for more device-specific logic. That could eliminate the need to implement this sort of integration in the Norns codebase and instead move it to a user script or library. This might not be possible without changes to device monitoring and initialization though.
2. The playdate.event( line ) callback would be called for all messages logged from the Playdate. I debated if it should only be called for messages that use some sort of prefix like ~msg: , similar to how the controller protocol works (~ctl: ). Ultimately I'm leaning towards leaving this up to the script/PDX author.
3. There are official APIs supporting the serial connection, so that feels fairly stable to me. As for the serial commands other than msg, these are not officially documented outside of using the help command. I'm not sure if commands like controller start and controller stop can be considered stable.

Thanks for considering my proposal!

[Dewb]

As a playdate user: very cool!

From a systems perspective, I completely agree with your first point in Open Questions. I’ve interfaced some other generic serial devices with norns (Arduinos, rs422 ptz controllers) and it’s a little tricky to do reliably with just lua and shell calls. A generic serial core API that manages the Linux bookkeeping and allows lua to interface with arbitrary devices would be a great core addition imo, and then playdate support could be provided as a mod or script lib.

[tehn]

generic serial device is an interesting prospect, i agree. (and also, playdate seems like a fun interface!)

here's where we differentiate TTY devices:

https://github.com/monome/norns/blob/main/matron/src/device/device_monitor.c#L270

[midouest]

Nice! I'm gonna take a stab at the generic serial device, although it might be a little bit beyond my skills. @Dewb, do you have any of these scripts on GitHub, or examples you can share?

[Dewb]

Here’s a script that uses a couple of serial devices (“servo” is an Arduino Uno, and “camera” is a USB to RS422 adapter.)

https://gist.github.com/Dewb/12c09eb5cf80c8b432c7b4a0f9287d13

(NB: I am also not an expert on Linux serial, this was a “just enough to work for the performance” last-minute script.)