RGB to YUV using GPU not CPU

[efschu]

Hi,

I'm using the following script:

core.num_threads = 32

clip = video_in

stream0 = core.std.SelectEvery(core.trt.Model(core.resize.Bicubic(clip, width=1280, height=720, format=vs.RGBS, matrix_in_s='709'), engine_path="/root/Downloads/realesr-general-wdn-x4v3_opset16_V100_574_720.engine", num_streams=3, device_id=0), cycle=2, offsets=0)
stream1 = core.std.SelectEvery(core.trt.Model(core.resize.Bicubic(clip, width=1280, height=720, format=vs.RGBS, matrix_in_s='709'), engine_path="/root/Downloads/realesr-general-wdn-x4v3_opset16_V100_574_720.engine", num_streams=3, device_id=1), cycle=2, offsets=1)

clip = core.std.Interleave([stream0, stream1])

clip.set_output()

which gives me 27 fps using vspipe.

now i want to use this in realtime with MPV, so I must convert the clip to YUV at the end. but this takes to much CPU power, and the fps are dropping to 23fps, which makes it not usable in realtime.

clip = core.resize.Bicubic(clip, format=vs.YUV420P8, matrix_s='709')

is it possible to convert format on the GPU rather the CPU - or is it possible to use RGB output with MPV?

thnx in advance

[Z2697]

mpv can take RGBS output just fine

edit: huh, so the problem is that the vapoursynth filtering in mpv doesn't support RGB output? I think you can use vs-placebo to do the conversion on GPU, but I don't think it can make such a difference... the data needs to be transfered between VRAM and RAM yet again, and in the meantime your GPU is busy doing upscale.

[efschu]

I have a third GPU idling meanwhile, while CPU is nearly 100% load, so I would like to outsource the scaling and colorspace conversion. I have looked @vs-placebo, but havent found out til now how i can use it for yuv to rgbs, rgbs to yuv conversion. Have to have a deeper look.

Thnx for the suggestion.

[Z2697]

Originally Posted by efschu

I have a third GPU idling meanwhile, while CPU is nearly 100% load, so I would like to outsource the scaling and colorspace conversion. I have looked @vs-placebo, but havent found out til now how i can use it for yuv to rgbs, rgbs to yuv conversion. Have to have a deeper look.

Thnx for the suggestion.

I'd assume CPU is idling for the given script, what is eating your CPU resource?

Apparently vs-placebo didn't provide a easy way of doing matrix conversion, sorry.

[efschu]

top command shows me vspipe is consuming 16 cores while running, and if I do not do a colorspace conversion from RGB to YUV at the end the speed of script is fine, but if i do the conversion (which is done on the CPU) the speed drops below realtime (which makes it useless for my usecase)

[Z2697]

That's sounds weird. I can't imagine a 5K color matrix conversion @23FPS hitting a 16-core CPU that hard. What is your CPU?
I mean, it does consumes quite a lot CPU resource, but 100% is just too unreal (it was 100% even without this final converion?)

[efschu]

The load is nearly at 100% befor the rgb2yuv conversion. So there is no power left. Forsure if the CPU is idling, it's no problem to do the matrix conversion. Its an AMD Ryzen 5950X with 3800MHz RAM.

[Z2697]

I just have a hard time to understand why the CPU load was 100% already...
You can try to use FP16 (vs.RGBH), your CPU has F16C instruction, that can reduce some cost on memory bandwidth at least, but you need to re-build the TRT engine.

[efschu]

Wow. Creating the engine with

--inputIOFormats=fp16:chw --outputIOFormats=fp16:chw

and using RGBH, increased performance by 38%.

Thank you so much.