feat: support WebM media fragments in Anki exports for local video#924
Open
khajiitvaper2017 wants to merge 36 commits intokillergerbah:mainfrom
Open
feat: support WebM media fragments in Anki exports for local video#924khajiitvaper2017 wants to merge 36 commits intokillergerbah:mainfrom
khajiitvaper2017 wants to merge 36 commits intokillergerbah:mainfrom
Conversation
…ideo export bitrate
…bplayer into feat-webm-v2
- Increase video readiness fallback timeout to 7000ms - Reduce WebM seek, frame-rate sampling, and render-stall watchdog timeouts to 3000ms - Simplify WebM render watchdog to a fixed per-frame stall timeout - Clarify watchdog comment to document stall-based behavior
- Update WebmFileMediaFragmentData.atTimestamp to stop reusing the current instance video/canvas/context - Create timestamp-derived WebM fragments with fresh rendering resources - Keep existing in-flight cancellation behavior while removing shared mutable DOM state between instances - Prevent races where concurrent/stale renders could draw to the same canvas or mutate the same video element
- Replace fixed sampling timeout with playback-rate-aware calculation - Keep sampling timeout capped at 3000ms maximum - Introduce base and max sampling timeout constants for clearer tuning - Ignore implausibly small media-time deltas based on max capture frame rate - Prevent duplicate/near-duplicate frame callbacks from inflating sample count
- Add SavedVideoState and VideoStateGuard helpers to centralize save/restore behavior - Capture playbackRate, muted, volume, onerror, onended, and optional preservesPitch in one place - Replace duplicated manual property save/restore in _renderWebm with guard apply/restore - Replace duplicated manual property save/restore in _sampleCaptureFrameRate with guard apply/restore - Reduce risk of future state-restore regressions when additional video properties are introduced
- Add _rendering instance flag to detect same-instance concurrent _renderWebm calls - Throw a descriptive error when _renderWebm is invoked concurrently - Split _renderWebm into guarded wrapper and locked implementation method - Remove _seekVideo resolveWithCleanup alias and wire onseeked directly to finish - Use finish() directly for epsilon seek short-circuit path
Perform an early findNotes("added:1") check in Anki.export for updateLast mode before starting audio/image base64 generation.
If no recent note is found, throw the existing "Could not find note to update" error immediately, avoiding expensive WebM/audio capture work that was previously done before failing.
- hard-lock WebM capture to 24fps and remove frame-rate sampling machinery - simplify WebmFileMediaFragmentData flow by splitting mime resolution, capture settings, and capture execution - remove VideoStateGuard and _blobPromiseReject state in favor of abort-driven cancellation and cleaner resource ownership - fix atTimestamp lifecycle: dispose when render is active, transfer owned resources only when idle - make recorder startup fail fast by rejecting recorderStarted on early MediaRecorder errors - update MediaFragment.fromWebmFile to use the simplified WebM constructor
…retries recoverable - replace _getCanvas Promise(async ...) pattern with _renderCanvas + cached promise wrapper - reset cached _canvasPromise and reject callback when rendering rejects so future calls can retry - clear cached promise/rejector when cancelling via atTimestamp to avoid permanently rejected state - preserve existing seek/render behavior for follow-up event lifecycle hardening
…tener lifecycle - replace video.onseeked/video.onerror assignments with addEventListener-based handlers - register seek/error listeners before mutating currentTime to avoid missed events - remove loadedmetadata/seeked/error listeners on settle to prevent stale callbacks on reused video elements - extract frame drawing into _drawCanvas for clearer render flow
- remove redundant Promise wrappers around _getCanvas in base64() and blob() - keep toBlob callback bridging as the only explicit Promise in blob() - preserve output format and quality handling while reducing nesting and error-path indirection
…lation, and listener lifecycle - add jpeg-file-media-fragment-data.test.ts with isolated mocked video/canvas behavior - verify render retries succeed after createVideoElement failure - verify atTimestamp cancels in-flight render and old instance can render again - verify seek/error listeners are present before currentTime seek and fully cleaned after settle - run full @project/common test suite to validate integration
# Conflicts: # common/anki/anki.ts
- prefer saved media fragments before regenerating clips while preserving legacy image upgrade behavior - consolidate Anki media encoding and upload handling to reduce duplicated export branching - simplify image preview loading and timestamp slider updates in the dialog flow - pause and resume WebM frame capture when the tab becomes hidden and visible again - relax the WebM frame watchdog to reduce false timeouts during tab swaps and transient stalls - add regression tests for saved media fragment selection and hidden-tab WebM resume behavior
Owner
|
Hey sorry I'm dragging my feet on this. Will try to review carefully on the weekend. |
- expose MediaFragment.endTimestamp and remove duplicate WebM range handling in AnkiDialog - localize the new capture-format and clip-trim labels, and simplify the UI copy - remove the createImageBitmap fallback path - simplify WebM capture internals by dropping unnecessary defensive canvas checks - preserve unknown recorder/render errors in fallback messages - remove the extra setTimeout scheduler fallback - remove the leftover preservesPitch mutation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
At first I tried adding GIF capture (#574 #187). It worked, but GIFs were too heavy and took too long to generate, so I switched to WebM instead. WebM ended up working surprisingly well. I also parallelized video and audio capture, and since they take about the same amount of time, it doesn't really feel any slower in practice (I used a laptop disconnected from power and with power saving on for a week with this). I tested this mainly on the latest Chrome.
The main caveat is that the Anki note template needs to support the
<video>tag.MediaFragmentpipeline that supports both JPEG screenshots and generated WebM clipsupdateLastwhen there is no recent note to updateDetails
common/src/media-fragment.tsand split file-backed rendering intojpeg-file-media-fragment-data.tsandwebm-file-media-fragment-data.tsMediaRecorderhandling, and fallback scheduling for browsers withoutrequestVideoFrameCallbackmediaFragmentFormat,mediaFragmentTrimStart, andmediaFragmentTrimEndAnkiDialog, preview components, and hooks to handle both still images and video previews; clipboard copy is still image-only<video autoplay loop muted playsinline ...>, while preserving compatibility with existing serializedimagepayloads