Recordings are sped up significantly

[nkhi]

Describe the bug
When using the example code, the output .mp4 has a much faster speed than the recording length. For example, I use a 20s sleep to invoke the stop_recording() method, but the output file is 4s with everything sped up. Using 60fps, and default fps. Using standard file name. Not sure what's causing this behavior.

To Reproduce
Run the following chunk

import time
import pyscreenrec
from tkinter import filedialog
from tkinter import *
from datetime import datetime

# get file save location from user
root = Tk()
root.withdraw()
folder_selected = filedialog.askdirectory()

# get timestamp and config save loc
now = datetime.now().strftime("%d-%m-%y-%H%M%S")
fullpath = f"{folder_selected}/recording-{now}.mp4"

# Start recording
print(f"File will be saved in {fullpath}. Recording starting...\n")

rec = pyscreenrec.ScreenRecorder()
rec.start_recording(fullpath)

# Stop recording
time.sleep(20)
rec.stop_recording()

print(f"File should be saved in {fullpath}. Recording done.\n")

Expected behavior
A 20s mp4 in my fullpath directory. Actual behavior: 1s.

Desktop (please complete the following information):

• macOS Big Sur 11.3
• Python 3.8
• Pyscreenrec 0.3

Additional context
Would be happy to contribute a fix, just not sure why it is doing this.

[nkhi]

Just replicated on Windows 10, with Python 3.8.3. Output seems to be discarding frames and/or dropping frames significantly?

[damies13]

I think i figured out part of the problem,

it seems to be related to the time it takes to call screenshot(os.path.join(self.screenshot_folder, f"s{i}.jpg"))

I added import time to the top of the file and made this temporary change:

            if self.__start_mode == "start":
                self.__running = True
                i = 1

                # starting screenshotting
                while self.__running:
                    sss = round(time.time() * 1000)
                    screenshot(os.path.join(self.screenshot_folder, f"s{i}.jpg"))
                    sse = round(time.time() * 1000)
                    sst = sse - sss
                    if sst < (1/self.fps):
                        sleep((1/self.fps)-sst)
                    i += 1

Then my video which should be 10 sec went from 1 sec to 6 sec

Also to help diagnose the issue I added this print statement in def _save_video:

        # writing all the images to a video
        for image in images:
            print("image:", image)
            video.write(cv2.imread(os.path.join(self.screenshot_folder, image)))

My 10 sec video at 10 fps should result in 100 frames, before my change above I got 37 frames, after the change i got 68 frames, still not 100 but closer

Python 3.10.12 on Ubuntu 22.04

[shravanasati]

I am sorry that I am very late in responding to this issue. I debugged this a while back, and my conclusion aligns with what @damies13 found out. The problem is that there are not enough screenshots generated during the recording phase as OpenCV expects. For example, with 10fps recording for 10 seconds, there should be 100 frames but we always get less than that. OpenCV writes the video at 10fps resulting in less duration.

The approach pyscreenrec uses to record the screen, i.e., screenshotting, sleeping, and then combining the frames into a video is naive. This work is done in a separate thread to ensure that the main thread isn't blocked. Since Python threads don't run in parallel, this thread will not get all the time it needs. Using processes instead of threads won't solve this either, since processes would be subject to OS context switches too (a hypothesis, there would still be a difference tho).

Ideally, it should use the OS-specific screen capture APIs, which is a lot of work.

[damies13]

My approach of subtracting the execution time from the sleep, while not perfect at least improved it, so until you find an OS-specific screen capture API library, this at least would be an improvement, though my code was pretty rough so would need to be cleaned up before publishing, though I'm sure you got the idea and can do that.

[shravanasati]

I'd try experimenting with that. Thanks for your active participation in resolving this issue.

[shravanasati]

A new release has been published incorporating the fix suggested by @damies13. I'll keep this issue open though, to encourage further discussion on this issue.

[StepaniaH]

I attempted to locate the specific source of the delay and received the following output:

We will record for 10 seconds
Capture: 78.4ms, Convert: 4.0ms, Write: 130.7ms
Capture: 74.3ms, Convert: 3.7ms, Write: 112.5ms
Capture: 62.1ms, Convert: 3.5ms, Write: 111.1ms
Time: 1s / 10s, Frames: 3, Actural FPS: 7.4, Frame processing time: 190.5ms
Capture: 59.7ms, Convert: 3.3ms, Write: 110.4ms
Capture: 60.2ms, Convert: 3.3ms, Write: 113.2ms
Capture: 58.9ms, Convert: 3.3ms, Write: 114.1ms
Capture: 61.3ms, Convert: 3.3ms, Write: 114.0ms
Capture: 59.5ms, Convert: 3.4ms, Write: 115.8ms
Capture: 55.8ms, Convert: 3.3ms, Write: 114.3ms
...[other output]

As a result, I tried adding frame scaling during capture to attempt to reduce write pressure:

def _start_recording(self) -> None:
        with mss.mss() as sct:
            if self.__running.value != 0:
                warn("Screen recording is already running.", ScreenRecordingInProgress)
            else:
                self.__running.value = 1
                while self.__running.value != 0:
                    st_start = time.perf_counter()
                    img = sct.grab(self.mon)
                    frame = np.array(img)

                    # Scale image dimensions
                    scale_percent = 50 # Scaling percentage, adjust as needed
                    width = int(frame.shape[1] * scale_percent / 100)
                    height = int(frame.shape[0] * scale_percent / 100)
                    dim = (width, height)
                    resized_frame = cv2.resize(frame, dim, interpolation=cv2.INTER_AREA)

                    # Put the scaled frame into the queue
                    self.queue.put(resized_frame)

                    st_total = time.perf_counter() - st_start
                    time.sleep(max(0, 1 / self.fps - st_total))

This approach significantly reduced the write pressure, but since this solution affects the target video quality, I haven't submitted a PR for the code modification yet. I'm currently working on quality restoration processing that minimizes the impact as much as possible.

[shravanasati]

Sounds great, I'll be looking forward to another PR :).

[damies13]

@StepaniaH's comment is really interesting, most of the time is taken in write, I wonder if there's a library that can be used for buffered writing in a separate thread? If so it might allow a frame rate of around 12fps (enough for smooth motion, 12 fps is what Disney's original mick mouse cartoons were)

[StepaniaH]

@StepaniaH's comment is really interesting, most of the time is taken in write, I wonder if there's a library that can be used for buffered writing in a separate thread? If so it might allow a frame rate of around 12fps (enough for smooth motion, 12 fps is what Disney's original mick mouse cartoons were)

Maybe python can not achieve enough speed to record a video like 30 fps or even 60 fps, cause its threads don't run in parallel? I do NOT sure, I just a rookie of the python. 🤔

[shravanasati]

Python threads do not run in parallel. The Python interpreter switches between the threads, as per the docs:

CPython implementation detail: In CPython, due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation). If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing or concurrent.futures.ProcessPoolExecutor. However, threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously.

I tried replacing threads with processes for v0.6, but processes are much more complex—for example, they maintain their memory. Communication between processes is also possible only with pickle-able objects. Maybe I should try exploring this option again now that the library code has been simplified.

I like the idea of a buffered writer tho - it might reduce the write time.