PyControl and evdev via API

[joshs08]

Hello,

I am trying to use PyControl in conjunction with evdev via the PyControl API, and am encountering an issue that typically results in the PyControl_GUI no longer responding, force quit required.

I want to use PyControl to trigger acquisition on a camera and a 2p (using pulse), and I want to concurrently use evdev to read events from an optical computer mouse. As soon as I try to read events from the optical mouse, the GUI typically crashes with erratic behaviour. Most frequently it just stops responding, requiring a force quit. Occasionally the optical mouse reading events seem to occur, but the camera triggering fails, and occasionally it crashes and I get the error message: "Error: uncaught exception of type: <class 'BlockingIOError'>". I see this error is "Raised when an operation would block on an object (e.g. socket) set for non-blocking operation." If anyone has any advice, I would greatly appreciate it.

I have attached both relevant scripts (please save as .py, unsupported file type upload hence the .txt), which I have in the folders as specified in the API example.
I am using Python 3.12.3 on Ubuntu 22.04.4
Micropython version: 1.2
Framework version: 2.0.1

motor_input_task_clean.txt
treadmill_api_class.txt

Kind regards,
Josh

[alustig3]

Hi Josh,

*(I haven't had time to try and replicate, but at first glance of the code...) I think what is happening is that get_treadmill() is getting called repeatedly before it is finished. You have this while loop that seems to be blocking the script for 10s, and when process_data_user() gets called things get messy I think.

  while currenttime < (starttime + duration):
      self.print_message("while occurring")
      #separate events by mouse 
      if device.path == self.patha:
          #separate x and y events
          if event.code == evdev.ecodes.REL_X:
              # synchronisation events (which the mice output when x and y movements coincide) 
              # have the same event code as rel_x (i.e. 0). event.value is delta
              if event.value != 0:
                  self.print_message("appending")
                  ax.append([event.value, event.timestamp])

Every time there is an event in the task, the api's process_data_user() function will get called. In this case, I suspect there is high frequency of "cam_frame_on", "cam_frame_off","twop_frame_on", "twop_frame_off" events coming in, and each time process_data_user()->get_treadmill() is getting called.

One suggestions is to use FrameTrigger and Framelogger :https://pycontrol.readthedocs.io/en/latest/user-guide/synchronisation/#frame_trigger to reduce the number of events being processed by the state machine. Additionally you may need to use Python's asyncio or multithreading to avoid blocking code, or ensure that you have 10s in between events that would call process_data_user().

Another option may be to use a optical mouse module and arduino library to read the data with a separate microcontroller, and then connect/output the data to a pyControl port adaptor board which would then be plugged directly into your pyControl breakout board and be treated like any other device.

Or maybe you can adapt your treadmill to use a pyControl rotary encoder for positional tracking instead of using the optical mouse.

Hope that gives you some ideas on how to progress. Let us know how it goes.

Best,
Andy

[joshs08]

Hi Andy,

Thanks so much for your swift and helpful reply! I have taken your advice regarding the high frequency of events, and simplified my code, removing all events besides the button press, and in the API class only trying to loop through events from a single mouse, and simply increment a counter. Unfortunately it still stops responding upon execution of the following line:

for event in g502a.read_loop():
                 i = i +1   

I tried adding a time.sleep line in that for loop to slow down event reading, but it made no difference. Do you think that multithreading/asyncio may provide the solution, or that it is another issue?

If I am not able to get evdev to work I will certainly look into the optical mouse module idea. Thank you for this good advice. Ideally my treadmill would move in all axes (air-supported ball), so the rotary encoder would not work unfortunately.

Thanks again and all the best,
Josh

treadmill_api_class.txt
motor_input_task_miceintegrated.txt

[alustig3]

I kinda sorta got something working, but it is not great and the more I think about it, really not a approach I would recommend. I'll share it though if if you want to play around with it.

I couldn't get evdev so instead am using pynput to get mouse data. One downside to this that I think it can only read in cursor position, not raw sensor data, so the position data is restricted the display size since the cursor can't go outside the bounds of the display. Also the library won't distinguish between multiple mice connected to the computer, so your computer mouse movement will mix with your treadmill sensor data. The more important part may be the threading portion of the code, so hopefully it will be useful to you / you'll be able to adapt it to use with evdev if you want.

# mouse_api_task.py

from pyControl.utility import *
from devices import *

# States and events.
states = ["start_state"]
events = ["poll_sensor"]

initial_state = "start_state"

# Variables
v.position = [0, 0]  # mouse position [x, y]
v.duration = 500 * ms
v.api_class = "Mouse_sensor"  


# State behaviour functions
def start_state(event):
    if event == "entry":
        set_timer("poll_sensor", v.duration)
    elif event == "poll_sensor":
        set_timer("poll_sensor", v.duration)

# Mouse_sensor.py

from source.gui.api import Api
from pynput import mouse
from threading import Thread


class MouseMovement:
    def __init__(self):
        self.x_pos = 0
        self.y_pos = 0

    def on_move(self, x, y):
        self.x_pos, self.y_pos = x, y

    def start_listener(self):
        with mouse.Listener(on_move=self.on_move) as self.listener:
            self.listener.join()


class Mouse_sensor(Api):
    def __init__(self):
        self.off_count = 0

    # this runs at the start of session
    def run_start(self):
        # you can print directly to the log from this user class using print_to_log
        self.print_to_log("\nMessage from config/user_classes/Example_user_class.py at the start of the run")
        self.mouse_movement = MouseMovement()
        self.listener_thread = Thread(target=self.mouse_movement.start_listener)
        self.listener_thread.start()

    # use this function
    def process_data_user(self, data):
        get_treadmill_event_happened = [state.name == "poll_sensor" for state in data["events"]]
        if get_treadmill_event_happened:
            self.set_variable("position", [self.mouse_movement.x_pos, self.mouse_movement.y_pos])

    def run_stop(self):
        self.mouse_movement.listener.stop()
        self.print_to_log("\nMessage from config/user_classes/Example_user_class.py at the end of the run")

I think it would be much better to read in the optical mouse sensor data directly into the pyboard so it can be properly processed as an analog channel where the timestamped data will automatically be streamed, plotted, and saved as a separate file and you can easily setup threshold events. I believe @AtMostafa has achieved this in his setup (see here: #33). Hopefully he can chime in with a good solution 😄 .

[AtMostafa]

We have been reading the data from the PMW3360DM mouse sensor directly with the microcontroller. It works reliably and consistently. The trick I used was to modify the analog channel in the pycontrol framework (see here) and then stream the x and y coordinates as 2 analog channels to the GUI that gets saved to file as well. this way we could raise an interrupt every few cm of movement and avoid overwhelming the microcontroller (code here).
While this is working well, the problem is that it is based on an earlier version of pycontrol (without any API), I unfortunately haven't had the time to adapt it to the newer release. And also, as the setup grew, now we feel like this is taking a lot of resources and we're wondering if/how we can offload this whole process to an Arduino.
If you have any questions about the data streaming of the mouse sensor, I'm happy to help out

[joshs08]

Thank you both for the detailed advice and for the time taken to assist! I will have a look at these options today and see what I can get to work.

[ThomasAkam]

I agree that connecting the optical mouse sensor directly to the pyboard is likely to give a beter solution that trying to route the data throught the API.

It should be pretty straightforward to update @AtMostafa's implementation to work with the latest version of pyControl. One disadvantage of the PMW3360DM is that it has an SPI serial interface, and pyControl does not break out the pins needed for SPI to any of the standard pyControl breakout board ports. It might be possible to use the software SPI functionality in machine.softSPI but this would use more microcontroller resources.

Annother option that looks interesting is the Sparkfun optical tracking sensor, which has an I2C serial interface (available on the standard pyControl breakout board ports 3 and 4), and a python library which should hopefully make developing a pyControl device class for it fairly straighforward.