Fix Nix builds: force use of pyproject, fix pynrfjprog dependency spec

[lschuermann]

With recent Nixpkgs revisions, Tockloader fails to build. This is because Nix attempts to use setup.py to build it, instead of using pyproject.

Additionally, the version of the nrf-command-line-tools pinned in default.nix is more recent than the version required in the pyproject.toml. This PR makes this dependency specification less strict, so that these versions are deemed compatible.

Once this is merged, it'd be great if we could do a patch release of Tockloader, which we can then pin in the other Tock repositories.

[lschuermann]

I have also make pynrfjprog an optional dependency, as otherwise Tockloader cannot build without proprietary dependencies. Last time I checked, it worked fine without this dependency for most features.

@bradjc Is this still true?

It turns out that there is no way to declare a dependency as optional in pyproject.toml and still include it in the default set of dependencies (as you can do with Rust crates). So, unfortunately, this does mean that developers would need to use something like tockloader[nrfjprog] to include this dependency when installing it via pip, etc.

[bradjc]

pynrfjprog is used to 1) detect which serial port is the correct serial port when two are created for an nrf52840dk and 2) for accessing the flash chip attached to the nrf52840dk. Yes, both should be optional.

[bradjc]

As in literally pipx install tockloader[nrfjprog] that exact string? How does this work? It's so hard to even communicate that because [ implies "substitute your thing here".

[lschuermann]

As in literally pipx install tockloader[nrfjprog] that exact string? How does this work? It's so hard to even communicate that because [ implies "substitute your thing here".

I'm no Python expert, but yes, I believe that is how that works. See: https://stackoverflow.com/a/13681679

For what its worth, I agree with your frustration and really dislike Python's approach. On the other hand, with pyproject I don't think there's any other way that we can support optional dependencies.

I can see whether there's a way around this. https://github.com/NordicSemiconductor/pynrfjprog itself is licensed under the "Nordic 5-Clause License" and ships with the required pre-built libraries (even though they are from the nrf-command-line-tools, which is an unfree package)...

[bradjc]

I have to say I'm pretty displeased with Python at this point. I wish there was a fork that kept the same language but got rid of all of these packaging headaches and just implemented reasonable features. For example, this proposal is frankly obvious and does not require a lengthy discussion.

That all being said, unfortunately pynrfjprog makes tockloader listen just work for the nrf52840dk, which is a pretty big quality of life feature, especially for new users. I would be more inclined to create some sort of fork of tockloader for users who care about this, rather than burden the average user.

[lschuermann]

@bradjc I managed to get the package switched over to this repository (away from the binary nrf-command-line-tools), but it's no use: https://github.com/NordicSemiconductor/pynrfjprog/

This is licensed under the Nordic 5-Clause license, which, amongst other restrictions does not allow reverse engineering, or using the software for non-Nordic chips: https://devzone.nordicsemi.com/nordic/nordic-blog/b/blog/posts/introducing-nordics-new-software-licensing-schemes

As such, it's still very much unfree software. Therefore, Tockloader, with this as a required dependency, cannot be used without installing unfree software. This is a shame generally, but also means that users of package managers like Nix will be unable to install Tockloader without explicitly enabling unfree packages. It also disqualifies Tockloader from inclusion in many distribution's repositories.

So I'd really like for us to figure out how to make this an optional dependency.

[lschuermann]

That all being said, unfortunately pynrfjprog makes tockloader listen just work for the nrf52840dk, which is a pretty big quality of life feature, especially for new users.

I believe that the best and easiest path forward is to try and replicate whatever features currently rely on pynrfjprog without this dependency. I don't know exactly what diff there is. Here's what (and how) things work without pynrfjprog and just openocd installed on a Raspberry Pi with an nRF52840DK:

• tockloader listen works just fine:

  (tockenv) tml@raspberrypi:~/tockloader $ python3 -m tockloader.main listen
  [INFO   ] No device name specified. Using default name "tock".
  [INFO   ] No serial port with device name "tock" found.
  [INFO   ] Found 2 serial ports.
  Multiple serial port options found. Which would you like to use?
  [0]     /dev/ttyACM0 - J-Link - CDC
  [1]     /dev/ttyAMA10 - ttyAMA10
  
  Which option? [0] 0
  [INFO   ] Using "/dev/ttyACM0 - J-Link - CDC".
  [INFO   ] Listening for serial
  
  tock$ list
   PID    ShortID    Name                Quanta  Syscalls  Restarts  Grants  State
   0      Unique     malloc_test02            0        13         0   0/15   Terminated
  

• tockloader read (and, presumably, most other commands interacting
  with the flash) work fine, but do require specifying the --openocd
  flag:

  (tockenv) tml@raspberrypi:~/tockloader $ python3 -m tockloader.main read --openocd 0 1
  [INFO   ] Using settings from KNOWN_BOARDS["nrf52dk"]
  [STATUS ] Reading flash from the board...
  [STATUS ]   Address: 0x0
  [STATUS ]   Length:  1 bytes
  00000000  00                                                |.|
  
  [INFO   ] Finished in 0.111 seconds
  

... am I missing something? Do we need pynrfjprog on platforms other than Linux?

(I do think that needing to pass the --openocd flag is unfortunate; it'd be great if we could flip the search order and, if OpenOCD is installed, try to use those boards first instead of the Tock bootloader protocol.)

[alevy]

I don't think replacing pynrfjprog is a blocker for this PR.

[lschuermann]

I don't think replacing pynrfjprog is a blocker for this PR.

@alevy Unfortunately, the pyproject build infrastructure won't build if it's marked as optional, but @bradjc (somewhat understandably) objects to making it an optional dependency because then you have to install Tockloader with pip install tockloader[nrfjprog] if you want to have that dependency (which most users probably do want to have).

So this is not something we want to merge at this point.

[lschuermann]

@alevy I don't think we need this change. We're just exposing the unfree packages for convenience here, and passthru is ignored for regular package builds. Because of Nix's lazy eval, you wouldn't need to pull in any unfree dependencies for Tockloader alone, even if we unconditionally expose these packages here.

In other words, withUnfreePkgs should really only influence Tockloader's build inputs alone.

[alevy]

I mean... it was failing for me without that I think? But sure, can revert my commit

[bradjc]

pynrfjprog is required for two things:

• Reading/writing the external qspi flash on the nrf52840dk
• Determining which serial port is the UART serial port when two are created on the host.

[lschuermann]

Determining which serial port is the UART serial port when two are created on the host.

If we get that reliably supported some other way, would you be fine making pynrfjprog an optional dependency? I think read/writing the external QSPI chip will be harder to implement ourselves, but also doesn't seem like a commonly used feature. We could print a message that this requires an installation of tockloader[nrfjprog] instead.

[bradjc]

I don't understand why we don't have something like pipx install tockloader[limited] or something. Is it that somehow doesn't work? It seems like since you are asking for this feature that it would be reasonable for you to have the extra complexity, not average users.

[alevy]

I don't understand why we don't have something like pipx install tockloader[limited] or something. Is it that somehow doesn't work? It seems like since you are asking for this feature that it would be reasonable for you to have the extra complexity, not average users.

I think there are two things going on here.

The first, and I think the more "blocking" one is that pynrfjprog is non-free (it has an open source license but you're only allowed to use it on Nordic hardware), and Segger JLink is also non-free (it's also rife with CVEs, but that's a slightly different story).

As it stands, it is not possible to "install" tockloader without also installing these unfree packages. It's possible not to use them, but because they are in the project dependencies, they must be installed, even if you never use them and even if you are the kind of person who either prefers not to or is not allowed to install non-free software on your machine.

The second is that Nix happens to make this distinction more explicit. So, if you want to install software marked non-free, you need to set an environment variable (or have a .config entry, as I do). It would be totally possible to have tockloader use non-free packages in the Nix expression by default or exclusively. It would require people using Nix to set this variable, and maybe that's ultimately the right answer.

But it is tockloader (unrelated to Nix) not marking this dependencies as optional that prevents installing like a tockloader-limited or something.

[alevy]

Note, btw, that if tockloader used the nrfjprog command, instead of the library, that could be relied on at runtime, rather than installation time, and would be as optional as JLinkExe is at the moment.

Also, is the functionality that pynrfjprog used for really just equivalent to choosing the tty with a smaller number? Or is the UART sometimes the higher numbered one?

[lschuermann]

Or is the UART sometimes the higher numbered one?

As far as I understand, the UART numbers are not reliable, at least on some platforms. I don't know whether this is across all platforms, or e.g., only on Windows or Linux.

I've captured the USB traffic going to the J-Link and don't see any messages being exchanged, so it seems like nrfjprog uses just the descriptors to distinguish the serial ports. However, I don't know which fields it relies on (yet).

Note, btw, that if tockloader used the nrfjprog command, instead of the library, that could be relied on at runtime, rather than installation time, and would be as optional as JLinkExe is at the moment.

I think that would be a good, minimally invasive workaround.

[lschuermann]

Edit: nevermind, the below error was because J-Link wasn't installed. J-Link is shipped with the nrf-command-line-tools Debian package which includes pynrfjprog and all Nordic shared libraries, and can be installed without a SEGGER license prompt. However, apparently the pynrfjprog package includes the Nordic proprietary shared libraries, but not the SEGGER ones.

All of this is very confusing, and means that just installing pynrfjprog doesn't really buy the user anything -- they need to have J-Link anyways.

However, on the bright side, the new nrfutil command actually provides VCOM detection and works without J-Link installed, and has a JSON interface for interfacing with it from other programs. So the best step forward seems to just hook into that and write a BoardInterface implementation for that.

---

I'm giving this a shot right now, but here's something interesting along the way. This "automatically pick the right serial port" has never worked for me (hence wondering what pynrfjprog does), and at least on a Raspberry Pi 5 default Raspberry Pi OS install with Tockloader, it seems like pynrfjprog doesn't work at all:

(tockloader-env) tml@raspberrypi:~/tockloader $ python3
Python 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pynrfjprog
>>> from pynrfjprog import LowLevel
>>> api = pynrfjprog.LowLevel.API()
>>> api.open()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tml/tockloader-env/lib/python3.11/site-packages/pynrfjprog/LowLevel.py", line 239, in open
    raise APIError(result, error_data=self.get_errors())
pynrfjprog.APIError.APIError: An error was reported by NRFJPROG DLL: -101 JLINKARM_DLL_COULD_NOT_BE_OPENED.

Now, Tockloader puts all of this in a large try / except block, so this error has always been masked. But now I'm curious whether there are more platforms where this doesn't work.

Anyways, this means testing whether I'm not breaking the existing pynrfjprog integration will be tricky on Treadmill, which only has nRFs on ARM64 hosts. I'll continue once I'm back on an AMD64 host with an nRF with two serial ports... yuck