@gavin1818 Thanks for the report! Are you able to share the input file that reproduces this?

@gavin1818 Could you also provide a little info on the model, device, an os you're running on? There could be a number of different potential issues depending on that.

The realtime demo seems to be running Whisper over the full recorded file, and using seeking to transcribe the most recent bit.

Would that be slowing things down? Would it make sense to cut it off - accepting that we would lose some of the context?

For realtime it is looking for a specific amount of confirmations to say that it effectively transcribed some set of audio without any cutoff in the middle of words. That is set to 2 segments currently which does end up being slower because we need to fix the timestamp filter to get smaller segments. You can configure it to only use 1 segment confirmation which might help. We also have a task to drop audio after transcribe is completed, but that wouldn't impact performance.

For realtime it is looking for a specific amount of confirmations to say that it effectively transcribed some set of audio without any cutoff in the middle of words. That is set to 2 segments currently which does end up being slower because we need to fix the timestamp filter to get smaller segments. You can configure it to only use 1 segment confirmation which might help. We also have a task to drop audio after transcribe is completed, but that wouldn't impact performance.

@ZachNagengast Do you mean that the current word-level timestamp might be inaccurate? Because if it were accurate, then cutting the buffer based on the end time of a word shouldn’t result in splitting a word in the middle, right?

Word-level timestamps have to be accurate for realtime (eager) mode to work, but cutting the buffer changes the start time of the audio, and since we are using seeking (aka clipTimestamps) to decode the next segment of real time audio as it comes in from the microphone, the seek value would also have to be reset if we're dropping audio from the buffer, in order for the clipTimestamps to correctly seek to the point in the buffer that has yet to be transcribed. None of that is super complicated to build into the system, so we're keeping all these kinds of things in mind for when we take eager mode out of the experimental stage.

Word-level timestamps have to be accurate for realtime (eager) mode to work, but cutting the buffer changes the start time of the audio, and since we are using seeking (aka clipTimestamps) to decode the next segment of real time audio as it comes in from the microphone, the seek value would also have to be reset if we're dropping audio from the buffer, in order for the clipTimestamps to correctly seek to the point in the buffer that has yet to be transcribed. None of that is super complicated to build into the system, so we're keeping all these kinds of things in mind for when we take eager mode out of the experimental stage.

Thank you for your reply