Syncing single audio

[RKelln]

Ran into a use case where I want do actually want the audio from a single video to be played. What's a decent workaround?

Some sort of external player that can send the video to shmdata?

Perhaps the window grab input? (Currently not working for me but could be fixed).

[brunoherbelin]

Thanks for bringing this idea back to life.

In practice, nothing prevents vimix from playing audio : gstreamer pipelines are very capable of doing so.

In effect however, the user needs once audio is supported in vimix would grow exponentially with the (normal) expectations when comparing with other software. So it is more a design choice.

One thing I had in mind:

• define in settings the name of the gstreamer pipeline sink for audio (e.g. allow to specify shm, or autoaudiosink, or anything that connects to another device, etc.)
• allow ONE video to be sending its audio when playing: no modulation of volume, no effects.

But opening this pandora box was a bit scary :)
You would have other suggestions?

[RKelln]

Both options sound good. One source of truth for audio with no bells or whistles seems to cover most of the bases.

My weird use case is doing visuals for a dance performance, a few of the pieces have some real-time video elements that I'm compositing using vimix. The main video track needs to be perfectly in sync with the music, so ideally played in vimix.

For the external audio, you get into the issue of how to get it sync'd but that option should be more for doing VJing to a DJ set you've previously recorded and support gapless audio playback while transitioning sessions. While audio in video is good for my sort of adding/compositing videos on to a base video.

[brunoherbelin]

Additional comment regarding metronome synchronization:

Although the features are not yet fully empowered in vimix, there is in the background a connection to Ableton Link that allows a performed of video to synchronize with a performer of music when playing together (and connected to the same local network). E.g. a video can wait for the beat to loop, ensuring a tempo regular loop in sync with music.

Now, playing an audio would naturally lead to the request : can the metronome adapt to the audio to set the tempo?
That would be something I would like, but its not so straightforward.. I am looking at options like https://aubio.org/ but am welcoming input for how to merge audio playing and metronome

[RKelln]

As someone who tries to only use open source software it would be nice to have an alternate to Ableton, can other things use Link? I think it's fine to force people to use their own beat detection and sending to vimix, just need a way to input in an open way.

[brunoherbelin]

Agreed and am the first to advocate for open source !
Link technology is actually not bad and the API is open source : https://github.com/Ableton/link
it is widespread with other music compositing tools (not only Ableton) so I didn't want to reinvent the wheel...

[RKelln]

Yeah if the protocol is open that works for me.

[RKelln]

Here is my ugly workaround for this a shell script to be started once vimix session is loaded and ready to go. uses mpv to play the audio and OSC to start playing everything that needs to start (that has been put into batch#0).

SOCKET=./mpv/socket
AUDIO_FILE="audio.mp3"

# reset
oscsend localhost 7000 '/vimix/batch#0/play' f 0.0
oscsend localhost 7000 '/vimix/batch#0/seek' f 0.0
killall -q mpv
sleep 0.5
rm -f $SOCKET

# if any arg is passed, then exit
if [ $# -gt 0 ]; then
    exit
fi

# start audio paused in background
mpv --no-video --no-loop --pause --no-terminal --profile=low-latency --input-ipc-server="$SOCKET" "$AUDIO_FILE" > /dev/null 2>&1 &

# wait for audio to load
sleep 1

# start video in vimix using OSC
oscsend localhost 7000 '/vimix/batch#0/play' f 1.0

# start mpv playing through socket
echo cycle pause | socat - "$SOCKET"

[RKelln]

Here is a fancier version easier to do rehearsal with and can use the same video:

#!/bin/bash

# script.sh         // start at beginning
# script.sh "1:23"  // start at 1 minute 23 seconds
# script.sh stop    // stop and reset vimix to start of video

SOCKET=./mpv/socket
OSC_PORT=7070
FILE="./video.mp4"
MPV_OPTIONS="--no-video --no-loop --pause --no-terminal --profile=low-latency"
START=0
START_SEC=0.0

# reset
oscsend localhost 7000 '/vimix/batch#0/play' f 0.0
oscsend localhost 7000 '/vimix/batch#0/seek' f 0.0
killall -q mpv
sleep 0.5
rm -f $SOCKET

# if any arg is passed assume it is a time in seconds to seek to before starting
# exit if < 0
if [ $# -gt 0 ]; then
    # if arg contains stop or quit then exit
    if [[ $1 == *"stop"* ]] || [[ $1 == *"quit"* ]]; then
        exit
    fi
    START=$1

    # if arg contains : then assume it is a time in hh:mm:ss format
    # convert it to seconds:
    #   1. split on :
    #   2. reverse the array
    #   3. multiply each element by 60^(index)
    #   4. sum the array
    if [[ $1 == *":"* ]]; then
        START_SEC=$(echo $1 | awk -F: '{for(i=NF;i>0;i--)s+=$i*60^(NF-i);print s}')
    fi
    oscsend localhost 7000 '/vimix/batch#0/seek' f $START_SEC
fi

# start audio paused in background
mpv $MPV_OPTIONS --input-ipc-server="$SOCKET" "$FILE" > /dev/null 2>&1 &

# wait for audio to load
sleep 1

# seek to audio start time
echo seek $START exact | socat - "$SOCKET"

# start video in vimix using OSC
oscsend localhost $OSC_PORT '/vimix/batch#0/play' f 1.0

# start mpv playing through socket
echo cycle pause | socat - "$SOCKET"

[brunoherbelin]

Excellent, thanks for sharing your approach !

Indeed, that is a nice idea too to sync two programs : that could be done with a GUI program for playing audio that is sending events to vimix through OSC... Or to make an intermediate with scripting, it could be a 'button box' that is configurable : each button could be associated to a gstreamer pipeline (which can be for playing audio) and to OSC events sent to vimix...

Let me think about this... I will try to propose something soon...

[brunoherbelin]

Okay, so here we go !

I opened the Pandora box of audio support in vimix !
The idea is to individually select if you enable the audio for a media source, and set a volume:

This can be disabled in the application settings.

It is in Beta for now : thank you for testing this by compiling the Beta branch, e.g. by making a flatpak package following these instructions:
https://github.com/brunoherbelin/vimix/blob/master/flatpak/README.md

[RKelln]

Nice, I can confirm it works in a local (non-flatpak) build. There may be a regression with opening a file by double clicking the name of it in Recent files, as that never finished loading, staying on "Video file loading..." forever. Opening with a dialog worked however and audio played. For your first audio bug there was a small click/pop on the start of the audio (using headphones) and when seeking or stopping. Also after playing with the seek to listen for pops (it doesn't seem to be consistent) I got into a state where the play button didn't work any longer in the Player.
However, my local build has some oddness with hardware decoding and video file format that has never been resolved (so testing was done after failed to load video and then switching to software decoding), so my system is pretty untrustworthy for testing.

[PhilippKuhn]

Hi, I really like your programm! To add the posiibility for audio would be very important for me because we work with film and live music. I tried to get the beta version with the audio working but cannot get more far than:
flatpak install --user vimix Suchen nach Übereinstimmungen … Fehler: No remote refs found for ‘vimix’
Any ideas what could be wrong?

[brunoherbelin]

Thanks !
I suggest installing from flathub https://flathub.org/apps/io.github.brunoherbelin.Vimix
The setup is quite easy and available for almost all linux distribs https://flathub.org/setup
More info on https://github.com/brunoherbelin/vimix/wiki/Quick-Installation-Guide

[PhilippKuhn]

Hi Bruno,
thanks for the reply!
I installed it from Flathub and the Audio Option is available but no sound is coming out...

any ideas why it's not working?

[PhilippKuhn]

Hi Bruno,

I did a bit of research and if possible I would really like to have some kind of audio support. It is longtime that I did programming but maybe I will try to add a small audio-mixer. It would be great to have it! What do you think? Or do you know any other possibilities how to achieve using audio with it?

[brunoherbelin]

Hi ! Thanks for keeping motivated to use vimix!

I think I have the answer to your problem : if you installed the flathub version, you have to authorize it to use audio;

   sudo flatpak override --socket=pulseaudio io.github.brunoherbelin.Vimix

Another way to do that easily is to install Flatseal and to enable the 'PuseAudio sound server' in the vimix app :

Thanks for pointing this problem; For the next release this will be set by default.

[brunoherbelin]

Hi again,

I did a bit of research and if possible I would really like to have some kind of audio support. It is longtime that I did programming but maybe I will try to add a small audio-mixer. It would be great to have it! What do you think? Or do you know any other possibilities how to achieve using audio with it?

It would be indeed very welcome if you wish to contribute to improving vimix audio. In the background, everything is handled by the gstreamer library: so in theory it is not difficult to do.

What is important is the UX integration : I would like to avoid messing up with the logic of image processing in vimix and I do not want to add all the knobs and audio mixing buttons as done in other audio dedicated programs. I was however imagining that adding one global audio player (e.g. support for mp3) could make things easier to record a mix based on music. To be discussed

[PhilippKuhn]

Hi Bruno, I could make the audio work with the flatpak command. Our shows are mainly Live-Music mixed with video, so sound is kind of Important! What I am thinking about ist just a small Mixer with whom one can control and manipulate the sound easily with a Midi Controller or something similar. Send it to another application would be a solution but there my probem ist synchronicity. The video must absolutely play synchronous image and sound what means they have to be started and played at the very same time. How I mentioned, it´ s quite a long time that I learned C++ but I am in the mood to refresh it. So if it´s ok. for you, I will read the code of Vimix until I understand it, than I will think again how to solve maybe the practcial demands :) Anyway, thanks for writing the program, I like it! Philipp

…

--- Philipp Kuhn Gailhöfe 6 88699 Frickingen-Altheim Am 24.03.2024 21:53 schrieb BHBN:

Hi again, > I did a bit of research and if possible I would really like to have > some kind of audio support. It is longtime that I did programming > but maybe I will try to add a small audio-mixer. It would be great > to have it! What do you think? Or do you know any other > possibilities how to achieve using audio with it? It would be indeed very welcome if you wish to contribute to improving vimix audio. In the background, everything is handled by the gstreamer library: so in theory it is not difficult to do. What is important is the UX integration : I would like to avoid messing up with the logic of image processing in vimix and I do not want to add all the knobs and audio mixing buttons as done in other audio dedicated programs. I was however imagining that adding one global audio player (e.g. support for mp3) could make things easier to record a mix based on music. To be discussed -- Reply to this email directly, view it on GitHub [1], or unsubscribe [2]. You are receiving this because you commented.Message ID: ***@***.***> Links: ------ [1] #104 (comment) [2] https://github.com/notifications/unsubscribe-auth/APTXVV5HE6JBBFDZGRX5J4LYZ44N7AVCNFSM6AAAAAA6QFSHPGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DQOJVGA4TK

[brunoherbelin]

Great ! Thanks for your positive feedback! Feel free to explore the code and improve it! It is not so much documented but should have enough comments to say what code does what when its not self-explanatory.

You can start with the MediaPlayer that controls the gstreamer playbin, and the MediaSource that encapsulates the MediaPlayer as a vimix source.

In the ImGuiVisitor you will find the code to render with imgui the left panel options for MediaSource.