StaxRip 2.0 Support Thread - Page 39

Atak_Snajpera · 28th August 2019, 15:02

I just use ffmpeg as piping tool. Works without any issues so far.
Piping in general is better than direct support in encoder.
Reasons:
1) decoding/encoding overlapping ( https://forum.doom9.org/showthread.php?t=175258 )
2) separation of processes on diffrent NUMA nodes ( decoding can run on NUMA 0 while encoding on NUMA 1 )

jlw_4049 · 28th August 2019, 15:19

My log files are pending approval apparently. Maybe I didn't add them correctly?

I went ahead and put them on a pastebin. Here are the pastebin links!

https://pastebin.com/EgM0pat8
https://pastebin.com/tL61JyXM

stax76 · 28th August 2019, 15:25

Quote:

Originally Posted by jlw_4049

My log files are pending approval apparently. Maybe I didn't add them correctly?

I went ahead and put them on a pastebin. Here are the pastebin links!

https://pastebin.com/EgM0pat8
https://pastebin.com/tL61JyXM

Thanks for helping to debug this, I hope it can be fixed.

jlw_4049 · 28th August 2019, 15:28

Quote:

Originally Posted by stax76

Thanks for helping to debug this, I hope it can be fixed.

No problem man! I love the program for 4K encoding.

It seems to hang and slowly build the AVS file 1 at a time between like 3 minute segments. I know that sounds crazy since they are >1kbs.

Let me know if I can help more/test more builds. I glanced over the debugging log, granted I have no knowledge of code, but I didn't see anything that stood out.

EDIT: Going to be out for a bit. If you would like to download this source file from me to test yourself, let me know and I'll set you up a login to my ftpServer with access to that file.

lifta19 · 28th August 2019, 16:48

Quote:

Originally Posted by stax76

Maybe if somebody is used to do things in a certain way or want to try out a different way, maybe just for fun.

I like Wolfberry's builds because they support avs input, for that it's good that the piping tool can be disabled. Some people might want to use another x265 build that don't has avs input support or maybe Wolfberry stops making builds.

Maybe one of the piping tools has a rare bug, people can then try another piping tool then.

I think it's always good when an application is feature rich, of course when an application has many features some care has to be taken to structure everything and add some search features so users can find the features easily.

If I recall correctly then the default in staxrip is avs2pipemod for x264 and x265 using AviSynth and for VapourSynth it's vspipe.

What's new is that x264 avs input use avs2pipemod, before it was using direct avs input, I changed it in order to support 10 bit.

I remember the reason why I first added avs2pipemod was because something with ffmpeg was not working, I don't remember what it was, maybe I just missed a setting like strict -1, today maybe ffmpeg would be sufficient but since everything works very well since a long time I just leave the old defaults.

I think megui and ribbot use other piping tools, can't remember the name, there are many solutions it seems. Not sure what is most popular or fastest or most stable. What's your favorite way these days?

I've used several encoders.
But I liked the StaxRip the best. It's because the GUI is very intuitive and easy to use It's a great encoder with a user-centered signature. So I only use the staxrip.
Thank you so much for providing me with a good encoder.
I'm sorry for my poor English.

ps:
And I tested it.
ffmpeg pipe was faster than x265 none

stax76 · 28th August 2019, 18:10

@lifta19, jlw_4049

Thanks for the positive feedback, it's very helpful for motivation.

@jlw_4049

I can see now that the issue is probably related to AviSynth, the main thread (thread of the main dialog) is opening the avs file to retrieve information from it, the opening happens at 20:55:44 and the closing happens minutes after at 20:59:09, here with an 18 GB file it was taking 0.5 seconds. Why it takes so long is unclear, I can try to analyze the code, maybe make some changes and add some more trace printing. It would be helpful to know if the issue persists when VapourSynth is used instead of AviSynth.

jlw_4049 · 28th August 2019, 18:13

Quote:

Originally Posted by stax76

@lifta19, jlw_4049

Thanks for the positive feedback, it's very helpful for motivation.

@jlw_4049

I can see now that the issue is probably related to AviSynth, the main thread (thread of the main dialog) is opening the avs file to retrieve information from it, the opening happens at 20:55:44 and the closing happens minutes after at 20:59:09, here with an 18 GB file it was taking 0.5 seconds. Why it takes so long is unclear, I can try to analyze the code, maybe make some changes and add some more trace printing. It would be helpful to know if the issue persists when VapourSynth is used instead of AviSynth.

The issue doesn't happen with vapoursynth. Its instant.

Sent from my SM-G965U1 using Tapatalk

stax76 · 28th August 2019, 19:00

Is the source file on a fixed local drive or on a network drive? Maybe it takes so long because it's again indexing.

edit:

To find out if it was indexing again you can look at the last write date of the index file (.ffindex).

jlw_4049 · 28th August 2019, 19:21

Quote:

Originally Posted by stax76

Is the source file on a fixed local drive or on a network drive? Maybe it takes so long because it's again indexing.

edit:

To find out if it was indexing again you can look at the last write date of the index file (.ffindex).

I've tried it with my 2nd PC locally and on the network drive. I have tried it on 3 local drives a nvme ssd and another ssd.

It's not just 1 file, it's like almost every 4k hdr remux.

Going to record what it does, so you can see. Give me a few minutes.

Sent from my SM-G965U1 using Tapatalk

stax76 · 28th August 2019, 19:32

After the main dialog comes back please check the last write time of the ffindex file, it must match with the end time of the index process found in the log file. If there is a difference of minutes then we know that the problem is that it indexes a second time.

----------------------- Indexing using ffmsindex -----------------------

"C:\Users\jlwServer\Desktop\Stax 2.0.3.1 test\Apps\Plugins\both\ffms2\ffmsindex.exe" "E:\temp\iron man 3 UHD\Iron man 3 UHD.mkv" "E:\temp\iron man 3 UHD\Iron man 3 UHD_temp\Iron man 3 UHD.ffindex"

Writing index... done.

Start: 4:50:31 PM
End: 4:55:44 PM

jlw_4049 · 28th August 2019, 19:52

Quote:

Originally Posted by stax76

After the main dialog comes back please check the last write time of the ffindex file, it must match with the end time of the index process found in the log file. If there is a difference of minutes then we know that the problem is that it indexes a second time.

----------------------- Indexing using ffmsindex -----------------------

"C:\Users\jlwServer\Desktop\Stax 2.0.3.1 test\Apps\Plugins\both\ffms2\ffmsindex.exe" "E:\temp\iron man 3 UHD\Iron man 3 UHD.mkv" "E:\temp\iron man 3 UHD\Iron man 3 UHD_temp\Iron man 3 UHD.ffindex"

Writing index... done.

Start: 4:50:31 PM
End: 4:55:44 PM

Here is the video. It's processing, but it should be done soon. Pretty compressed video file.

https://youtu.be/PEzNkNW0Zi8

Will add new log files in just a second

Here you go:
https://pastebin.com/5pb5a20M
https://pastebin.com/dXgKEzXD

Also, index times look accurate. There is no HDD activity/or updated index file while it's waiting the additional 8 minutes.

stax76 · 28th August 2019, 20:20

If there is no hdd activity then a second indexing is probably not happening. Maybe check the timestamps anyway after the waiting, the index file isn't written while indexing, it's only written after the indexing.

jlw_4049 · 28th August 2019, 20:23

Quote:

Originally Posted by stax76

If there is no hdd activity then a second indexing is probably not happening. Maybe check the timestamps anyway after the waiting, the index file isn't written while indexing, it's only written after the indexing.

It's not indexing that is the issue. :/

Watch the video and take a look!

stax76 · 28th August 2019, 20:28

I'll watch it. To be sure that it's not indexing only the HDD activity or the timestamps can tell. The filter can index using the staxrip process not using ffmsindex.exe

stax76 · 28th August 2019, 20:38

I've watched the video, there is about 7% CPU time, that would be about that same what indexing uses. :-) I'll watch it again.

jlw_4049 · 28th August 2019, 21:15

Quote:

Originally Posted by stax76

I've watched the video, there is about 7% CPU time, that would be about that same what indexing uses. :-) I'll watch it again.

Let me know what you think it is.

EDIT: Also, wouldn't the HDD be under load though if it was indexing a 2nd time? Like I said, I've done a lot of tests and IDK what it is.

I am pretty sure it does it with this file on 3 out of the 4 computers I have tested it on. I even reinstalled Windows 10 on my gaming machine to try with fresh avisynth etc, still does the same thing. Takes about 7-8 minutes each time after the index. Usually the Index takes about 5 minutes.

(Just double checked. No HDD usage during the 8 minute period where StaxRip isn't there)

jlw_4049 · 28th August 2019, 22:05

Quote:

Originally Posted by stax76

I've watched the video, there is about 7% CPU time, that would be about that same what indexing uses. :-) I'll watch it again.

Just noticed, on the build I'm using, I went to go ahead and try another file (which did take the 8 minutes exta), however I was going to do a 4K encode, I clicked crop and it's doing the exact same thing that the Indexer is doing. I'm going to wait and see if it's waiting for the 8 minutes before the crop window comes up for me to crop.

Just took the program about 7 minutes to open the cropping window.

stax76 · 28th August 2019, 22:55

If the preview or cropping dialog takes minutes to load then it's an indication that's a general avs issue, maybe try to load the avs scripts generated by staxrip in another app like VirtualDub2, if that takes also minutes to load then it's a general avs issue out of the control of staxrip.

jlw_4049 · 28th August 2019, 23:32

Quote:

Originally Posted by stax76

If the preview or cropping dialog takes minutes to load then it's an indication that's a general avs issue, maybe try to load the avs scripts generated by staxrip in another app like VirtualDub2, if that takes also minutes to load then it's a general avs issue out of the control of staxrip.

I'll uninstall/reinstall AviSynth. Have you made any head way on that issue after indexing?

Atlantis · 28th August 2019, 23:38

I just installed 2.0.3 and want to thank you. This is not just an improvement, this is a huge first step. For the very first time Staxrip (or at least me) is encoding H265 10bit sources correctly.

For many years, don't remember maybe since 1.0.7 or 1.0.6 I have been encoding H265 10bit HDR sources incorrectly. I was always dissatisfied with the results and I knew something was wrong but did not know what.

Now I have discovered that before 2.0.3 I got

avs2pipemod[info]: writing 1146 frames of 24000/1001 fps, 3840x2160,
sar 0:0, YUV-420-planar-8bit progressive video.
y4m [info]: 3840x2160 fps 24000/1001 i420p8 unknown frame count

now I get this
avs2pipemod[info]: writing 1146 frames of 24000/1001 fps, 3840x2160,
sar 0:0, YUV-420-planar-10bit progressive video.
y4m [info]: 3840x2160 fps 24000/1001 i420p10 unknown frame count

So I'm suspecting that for the past few years, staxrip was converting H265 10bit HDR sources to 8bit and again to 10bit H265 for encoding.

Now my encodes with H265 10bit sources, finally look acceptable to my eyes.

The side-effect is that my encodes have gone extremely slower. A 16hr encode now takes 25 hours. I suspect that in the past when it was sending 8bit information, there was less information to encode and it was faster. Now for the first time it is sending the correct full 10bit information to encode and it is extremely slow.

28th August 2019, 15:02	#761 \| Link
Atak_Snajpera RipBot264 author Join Date: May 2006 Location: Poland Posts: 7,816	I just use ffmpeg as piping tool. Works without any issues so far. Piping in general is better than direct support in encoder. Reasons: 1) decoding/encoding overlapping ( https://forum.doom9.org/showthread.php?t=175258 ) 2) separation of processes on diffrent NUMA nodes ( decoding can run on NUMA 0 while encoding on NUMA 1 ) __________________ Windows 7 Image Updater - SkyLake\KabyLake\CoffeLake\Ryzen Threadripper Last edited by Atak_Snajpera; 28th August 2019 at 15:07.

28th August 2019, 18:10	#766 \| Link
stax76 Registered User Join Date: Jun 2002 Location: On thin ice Posts: 6,837	@lifta19, jlw_4049 Thanks for the positive feedback, it's very helpful for motivation. @jlw_4049 I can see now that the issue is probably related to AviSynth, the main thread (thread of the main dialog) is opening the avs file to retrieve information from it, the opening happens at 20:55:44 and the closing happens minutes after at 20:59:09, here with an 18 GB file it was taking 0.5 seconds. Why it takes so long is unclear, I can try to analyze the code, maybe make some changes and add some more trace printing. It would be helpful to know if the issue persists when VapourSynth is used instead of AviSynth. __________________ https://github.com/stax76/software-list https://www.youtube.com/@stax76/playlists

28th August 2019, 19:00	#768 \| Link
stax76 Registered User Join Date: Jun 2002 Location: On thin ice Posts: 6,837	Is the source file on a fixed local drive or on a network drive? Maybe it takes so long because it's again indexing. edit: To find out if it was indexing again you can look at the last write date of the index file (.ffindex). __________________ https://github.com/stax76/software-list https://www.youtube.com/@stax76/playlists Last edited by stax76; 28th August 2019 at 19:05.

28th August 2019, 19:32	#770 \| Link
stax76 Registered User Join Date: Jun 2002 Location: On thin ice Posts: 6,837	After the main dialog comes back please check the last write time of the ffindex file, it must match with the end time of the index process found in the log file. If there is a difference of minutes then we know that the problem is that it indexes a second time. ----------------------- Indexing using ffmsindex ----------------------- "C:\Users\jlwServer\Desktop\Stax 2.0.3.1 test\Apps\Plugins\both\ffms2\ffmsindex.exe" "E:\temp\iron man 3 UHD\Iron man 3 UHD.mkv" "E:\temp\iron man 3 UHD\Iron man 3 UHD_temp\Iron man 3 UHD.ffindex" Writing index... done. Start: 4:50:31 PM End: 4:55:44 PM __________________ https://github.com/stax76/software-list https://www.youtube.com/@stax76/playlists

28th August 2019, 20:20	#772 \| Link
stax76 Registered User Join Date: Jun 2002 Location: On thin ice Posts: 6,837	If there is no hdd activity then a second indexing is probably not happening. Maybe check the timestamps anyway after the waiting, the index file isn't written while indexing, it's only written after the indexing. __________________ https://github.com/stax76/software-list https://www.youtube.com/@stax76/playlists

28th August 2019, 15:19	#762 \| Link
jlw_4049 Registered User Join Date: Sep 2018 Posts: 391	My log files are pending approval apparently. Maybe I didn't add them correctly? I went ahead and put them on a pastebin. Here are the pastebin links! https://pastebin.com/EgM0pat8 https://pastebin.com/tL61JyXM

28th August 2019, 20:28	#774 \| Link
stax76 Registered User Join Date: Jun 2002 Location: On thin ice Posts: 6,837	I'll watch it. To be sure that it's not indexing only the HDD activity or the timestamps can tell. The filter can index using the staxrip process not using ffmsindex.exe __________________ https://github.com/stax76/software-list https://www.youtube.com/@stax76/playlists

28th August 2019, 20:38	#775 \| Link
stax76 Registered User Join Date: Jun 2002 Location: On thin ice Posts: 6,837	I've watched the video, there is about 7% CPU time, that would be about that same what indexing uses. :-) I'll watch it again. __________________ https://github.com/stax76/software-list https://www.youtube.com/@stax76/playlists

28th August 2019, 22:55	#778 \| Link
stax76 Registered User Join Date: Jun 2002 Location: On thin ice Posts: 6,837	If the preview or cropping dialog takes minutes to load then it's an indication that's a general avs issue, maybe try to load the avs scripts generated by staxrip in another app like VirtualDub2, if that takes also minutes to load then it's a general avs issue out of the control of staxrip. __________________ https://github.com/stax76/software-list https://www.youtube.com/@stax76/playlists

28th August 2019, 23:38	#780 \| Link
Atlantis Registered User Join Date: Feb 2002 Posts: 758	I just installed 2.0.3 and want to thank you. This is not just an improvement, this is a huge first step. For the very first time Staxrip (or at least me) is encoding H265 10bit sources correctly. For many years, don't remember maybe since 1.0.7 or 1.0.6 I have been encoding H265 10bit HDR sources incorrectly. I was always dissatisfied with the results and I knew something was wrong but did not know what. Now I have discovered that before 2.0.3 I got avs2pipemod[info]: writing 1146 frames of 24000/1001 fps, 3840x2160, sar 0:0, YUV-420-planar-8bit progressive video. y4m [info]: 3840x2160 fps 24000/1001 i420p8 unknown frame count now I get this avs2pipemod[info]: writing 1146 frames of 24000/1001 fps, 3840x2160, sar 0:0, YUV-420-planar-10bit progressive video. y4m [info]: 3840x2160 fps 24000/1001 i420p10 unknown frame count So I'm suspecting that for the past few years, staxrip was converting H265 10bit HDR sources to 8bit and again to 10bit H265 for encoding. Now my encodes with H265 10bit sources, finally look acceptable to my eyes. The side-effect is that my encodes have gone extremely slower. A 16hr encode now takes 25 hours. I suspect that in the past when it was sending 8bit information, there was less information to encode and it was faster. Now for the first time it is sending the correct full 10bit information to encode and it is extremely slow.