Standalone Faster-Whisper - AI auto-transcription-translation

[VoodooFX]

Whisper is a state of the art auto-transcription-translation model - Robust Speech Recognition via Large-Scale Weak Supervision



There are my compiled binaries for newbies: https://github.com/Purfview/whisper-standalone-win

[StainlessS]

WOW! VX, you be da man.

[VoodooFX]

@StainlessS How is internet at the pub, did you downloaded a release with GPU?

[StainlessS]

Not yet, I thought that it seemed a little bit wierd, what with all of the various downloads necessary.
I did not have a clue what to do with stuff on the github site, did not seem to be anything downloadable.
However, I did find some stuff here:- https://github.com/openai/whisper/discussions/63
which would seem to be the model thingies.
Maybe I down them in pub, but in no great hurry at the moment.
I'll also down the GPU thingy.

EDIT: Yes I know it can auto download the models, but I want offline download and want to know where they come from.

[VoodooFX]

Originally Posted by StainlessS

I did not have a clue what to do with stuff on the github site, did not seem to be anything downloadable.

It seemed that you knew where to download:

Originally Posted by StainlessS

EDIT: What kind of speed can one expect (I downed the 170MB-ish CPU version, 1.6-ish GB for GPU ver$ is a bit rich for me, is that much faster ?) ?.

Anyway, in GitGub at the right side you should see "Releases" button.

Model[s] are downloaded separately, automatically or manually, link for the models is in the front page of the repo.

You don't want "OpenAI" stuff, it's very slow.

[StainlessS]

OK, I got it working [with auto download of the model, I downloaded the pytorch models [.pt extension] by mistake, in pub].

I tested with both CPU and GPU versions, GPU pleasantly faster,
01:33:xx movie under GPU/medium.en model took 269 seconds for 933 subtiles.

Thanks for prodding me in this direction.

[Emulgator]

CPU: I get

2023-05-04 23:28:28.0660903 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:1671 onnxruntime::python::CreateInferencePybindStateModule] Init provider bridge failed.

but after that recognition starts. Aborted manually to see CUDA blow...

CUDA: the same report, then it starts munching...

Still it repeats some lines over and over, but
WOW IS THAT QUICK ! Like 10-fold going from a .wav file.
635s for a 1:43:40 musical movie in English, 1769 subs, songs included.
103 mins playing time transcribed in 10,5min with ~95% accuracy.
Found slang stuff I was helpless to translate before.

CPU 11900K 17%, GPU RTX3080 1% How, well...

Awesome work, many thanks, VoodooFX !

[VoodooFX]

That message from onnxruntime is "normal", expectable nonsense from Microsoft's lib.

Originally Posted by Emulgator

Still it repeats some lines over and over

Can you cut and share shorter sample of that audio where it happens? [Test if cut sample still loops]

EDIT:

Originally Posted by Emulgator

going from a .wav file

Use original audio, don't convert it to anything, results from ffmpeg conversion are noticeably worse for some reason.

EDIT2:

Btw, some people reported that on CPU it's a bit more accurate.

[Emulgator]

No time to continue right now but I guess it will be the same behaviour as with the other Whisper versions within SubtitleEdit:
If I just restrict and give only a smaller range it works most of the time.

[Emulgator]

Now let's have this engine on Android (I once had to help a hearing-disabled person using their mobile phone for live transcription),
and finally Big Goo's online-listening would be out of the water...

[StainlessS]

Btw, some people reported that on CPU it's a bit more accurate.

My CPU and GPU outputs were definitely different, GPU quite often split longer single CPU sub
into two shorter GPU subs. [I think]

EDIT:
I wonder how it would do on movie SNATCH, Brad Pitt's irish gypsy accent [fantastic job by Pitt to make it totally unintelligible].

Snatch/Pitt: https://www.youtube.com/watch?v=Gfzxz7asbZs

[VoodooFX]

On English audio I get better results with multilingual "medium" model than with "medium.en".


@Emulgator
@StainlessS
Could you test if new Faster-Whisper r117 runs OK on CUDA? [Executable is small]
And I would be interested in transcription accuracy and speed benchmark vs previous version.

[StainlessS]

[Executable is small]

Yeah, maybe, but requires
cuBLAS 11.x @ 3.4GB, and
cuDNN 8.x @ unknown size (I have an nVidia devs account somewhere, I'll havta find it).
I'm currently on 50GB data per month [EE @ £20/month], I think maybe I'll up it to 130GB/month next month [think next up is 130GB for £30/month].
Anyway, I'll down them in pub next time I'm there.

EDIT: The models were updated 22hours ago.
EDIT: Actually EE 125GB for £30/month for my 4G+ Router. https://shop.ee.co.uk/sim-only/pay-as-you-go-phones#

[VoodooFX]

Originally Posted by StainlessS

Yeah, maybe, but requires
cuBLAS 11.x @ 3.4GB, and
cuDNN 8.x @ unknown size (I have an nVidia devs account somewhere, I'll havta find it).

EDIT: The models were updated 22hours ago.

Maybe it would work out of the box. Shouldn't that be already installed with CUDA drivers/stuff?

EDIT:
Actually all that stuff should be present in previous version, copying dlls to the same folder from there should work.

[VoodooFX]

Originally Posted by StainlessS

I'm currently on 50GB data per month [EE @ £20/month], I think maybe I'll up it to 130GB/month next month [think next up is 130GB for £30/month].

Get GiffGaff PAYG, unlimited* for £25. [GiffGaff is basically O2]

*Full 4G speed till 80GB, after that it limits to 386kb at daytime, but at night it's full speed again.

[StainlessS]

Thanx VFX, but I'll stick with faster EE, says in my given link that speed is max 25mbps.
EtherNet, USB etc, tend to have a management overhead of 20%, I assume same for 4G,
So to convert mbps to MB/s, just divide by 10.
Despite what EE says (max 25 mbps) I regularly get 3 or 4 MB/s (as for 30 or 40 mbps), and have
once noticed it at 8MB/s during the night.
Thats pretty good speed considering that I'm quite a way from city urban area (I'm near green parkland),
and only get 2 out of 5 bar signal.

EDIT: I presume that we get charged for the management overhead, and that it is included in max 25 mbps.

*Full 4G speed till 80GB, after that it limits to 386kb at daytime, but at night it's full speed again.

Did not know that, thanx.

[VoodooFX]

@StainlessS
I could make a release [or separate download] including Nvidia libs, but I don't know which libs are actually needed, I don't wanna include whole 4GB stuff.
Could someone check which libs are actually needed by copying dlls [to same folder with whisper.exe from "r117"] one by one on error from "b103" release, there libs are located at "Whisper-Faster\torch\lib"?
That would need Windows with only Nividia drivers installed without CUDA Toolkit & cuDNN.

[Emulgator]

If swapped into the same python-laden folder as r103, r117 throws error:
"Could not load cudnn_ops_infer64_8.dll. Error code 126.
Make sure that cudnn_ops_infer64_8.dll is in your path."
That file was indeed in torch\lib, together with 36 more .dlls.
I copied that side by side to 117, then
"Could not load cudnn_cnn_infer64_8.dll. Error code 126.
Make sure that cudnn_cnn_infer64_8.dll is in your path."
I copied it side by side, then still the same fault.

Win10P64, i9-11900K, RTX3080, cudart64_110.dll 6.14.11.11080 in system32.
P.S. This is my replacement system, so no CUDA 11.8 installed yet.

[VoodooFX]

@Emulgator Thanks for testing it.

r117 is standalone single executable, no need to copy it to r103 folder. [It's not some incremental patch]

"cudnn_cnn_infer64_8.dll" has dependency on "zlibwapi.dll", so copy it too.

[Emulgator]

Ah, ok. Continuing.
Removed cudart64_110.dll 6.14.11.11080 from system32,
(still no CUDA 11.8 installed).

Now running from a separate folder, containing only the .exe, the .bat and the _models folder,
and adding dependencies as we speak.

We had
cudnn_ops_infer64_8.dll
cudnn_cnn_infer64_8.dll

You hinted
zlibwapi.dll
I added that.

Now it asked for
cublasLt64_11.dll
I added that.

Then it asked for
cublas64_11.dll
I added that.

Start: Success ! Now it runs with just these 5 dependencies.
14..17..18% on CPU, 3 of 16 cores (1 is 90%, 1 is 66%, 1 is 33%), 1% GPU.
Load distribution looks the same as with r103.

Speed: will tell when it is finished. Feels the same range as with r103.
Then I will compare again r103 vs. r117 apples-to-apples.

Finished: 949s for the same movie. 50%slower.
Was stuck quite a bit at 01:37:40.700 (song beginning)
generating of subtitles ended there at #1757,
so it did not reach the movie's end at 01:43:40

Repeated r103: 633s for the same movie, ran until the end.
At the moment r103 within its full dependency bag looks better.

nVidia Driver on this (older, clone father, now replacement) system SSD: 462.75