Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > New and alternative a/v containers

Reply
 
Thread Tools Search this Thread Display Modes
Old 24th March 2019, 11:52   #321  |  Link
mkver
Registered User
 
Join Date: May 2016
Posts: 197
Quote:
Originally Posted by Perenista View Post
It's odd. I tried to add a dubbing to a Matroska video and told MKVToolnix to add + 7000 ms as delay, to sync said dubbing audio (AC3) with the video. This time didn't work. However when I open MPC-HC (and both files at the same time) and add 7000 ms manually the audio matches. Now it seems 7000 ms is too much because the audio is still 0.5 second or 1 second out of sync.

(I know the best way to do this is adding these +7 seconds in the dubbing audio and then adding to the MKV without any delay option enabled in MKVToolnix. The problem is that I might have to do this with subtitle streams, too, so it would be a hassle if this didn't work in all cases.

Maybe this time didn't work because 7 seconds is too much?
Is your audio an AC3 elementary stream (that doesn't have explicit timestamps, so that its first audio frame gets a timestamp of zero by default) or a Matroska file (or another container)? If so, what is the lowest timestamp of said audio file? And what about the timestamps of the other file (only the first few timestamps are of interest here)? See my answers here for a possible reason why this is happening.
mkver is offline   Reply With Quote
Old 28th March 2019, 16:08   #322  |  Link
robena
Registered User
 
Join Date: May 2007
Posts: 29
Quote:
Originally Posted by Mosu View Post
One last idea: when starting mkvmerge manually, do you perchance start it with a lower process priority (e.g. because the process you start it from, such as cmd.exe, is already running with a lower process priority)? Or do you start the GUI (and therefore mkvmerge-started-from-the-GUI) with a higher one? See Windows 10's task manager's "detail" page, right-click on the column header & enable the "base priority" column.
I use 3 kind of drives: a QNAP basic NAS (600/260 speed), an SSD (500/500) and a SAS RAID Areca DAS (500/500).

Using "sync -r" before each test, remuxing a 60GB UHD movie in command line mode takes 1872s on the Areca, 2053s on the SSD (its cache is full and slows it down) and 523s on the QNAP! With the GUI, the Areca takes 350s. The GUI on the QNAP takes 420s.

These are completely reproducible results. The command line is almost as fast as the GUI over a 10Gbs networked NAS, but almost 6 times slower on a Sata or SAS hard drive (the Areca RAID is seen by Windows like a single SAS drive).

Thoughts?
robena is offline   Reply With Quote
Old 28th March 2019, 16:28   #323  |  Link
mkver
Registered User
 
Join Date: May 2016
Posts: 197
What about the actual CPU time used by the underlying mkvmerge process for each task? Does it make a difference when started from the GUI?
mkver is offline   Reply With Quote
Old 29th March 2019, 03:47   #324  |  Link
Perenista
Registered User
 
Join Date: Oct 2013
Posts: 205
Quote:
Originally Posted by mkver View Post
Is your audio an AC3 elementary stream (that doesn't have explicit timestamps, so that its first audio frame gets a timestamp of zero by default) or a Matroska file (or another container)? If so, what is the lowest timestamp of said audio file? And what about the timestamps of the other file (only the first few timestamps are of interest here)? See my answers here for a possible reason why this is happening.
I did more tests and 19 others (and different ones) were perfectly fine. There were times I even added 7000 or more than 8000 ms in terms of delay.

The file number 20 was the one with this issue. However I managed to figure out the correct delay to insert: 6350 ms. I checked at the beginning and after 58 minutes and now it's 100% OK. Not sure what went wrong before.

The audio stream I was trying to correct the sync (to be added in my Matroska video) is AC3:

Format : AC-3
Format/Info : Audio Coding 3
Commercial name : Dolby Digital
File size : 352 MiB
Duration : 1 h 49 min
Overall bit rate mode : Constant
Overall bit rate : 448 kb/s

Audio
Format : AC-3
Format/Info : Audio Coding 3
Commercial name : Dolby Digital
Duration : 1 h 49 min
Bit rate mode : Constant
Bit rate : 448 kb/s
Channel(s) : 6 channels
Channel layout : L R C LFE Ls Rs
Sampling rate : 44.1 kHz
Frame rate : 28.711 FPS (1536 SPF)
Bit depth : 16 bits
Compression mode : Lossy
Stream size : 352 MiB (100%)
Service kind : Complete Main

Info about the MKV:
https://pastebin.com/U4beAwL1
Perenista is offline   Reply With Quote
Old 29th March 2019, 05:32   #325  |  Link
mkver
Registered User
 
Join Date: May 2016
Posts: 197
1. This is actually not all the data I wanted, because it doesn't tell me anything about the first few timestamps of Matroska file. Use mkvinfo -s -r <output log> <Matroska file> for this; only the first few (say, the first second) of timestamps is of interest here.
2. And you used the elementary stream during determining the offset? Not the stream remuxed into Matroska or so? (Seeking in elementary streams that don't provide timestamps is a PIA, so it is often not instantaneous.)
mkver is offline   Reply With Quote
Old 29th March 2019, 10:25   #326  |  Link
dREV
Registered User
 
dREV's Avatar
 
Join Date: Jan 2019
Location: Antarctica
Posts: 74
Hello, not sure if this has been brought up or not. I wanted to bring an issue that I wanted to clarify or just ignore. I made a post on mkvtoolnix page a few months ago asking about this but was told to come here instead to ask my question. Not sure if I should also post this on HEVC's thread as well?

I use an older version of mkvmerge (prefer simple gui then the new one) and also tried the latest portable of version 32.0.0 "Astral Progressions" 64 bit and both still got the same results.

My issue is concerning HEVC's implementation of the code when viewed in Media Info. It's "total-frames=". As I cut frames with Avisynth then append them together to fix problematic scenes, however, when muxing I would receive these messages only on HEVC.

Quote:
#version 8.3 "Over the Horizon" 64 bit version

mkvmerge finished with a return code of 1. There were warnings, or the process was terminated

Warning: The track number 0 from the file 'C:\Users\Public\# 000\4003, 4146) #banding\# 000 4003, 4146) #banding.mkv' can probably not be appended correctly to the track number 0 from the file 'C:\Users\Public\#000\0, 4002) #Combined\# 000 0, 4002) #Combined.mkv': The codec's private data does not match (lengths: 2277 and 2278). Please make sure that the resulting file plays correctly the whole time. The author of this program will probably not give support for playback issues with the resulting file.
Quote:
# version 32.0.0

--- Warnings emitted by job 'Multiplexing to file "# 000 - done.mkv" in directory "C:\Users\Public"' started on 2019-03-29 02:14:27 ---The track number 0 from the file 'C:\Users\Public\# 000\4003, 4146) #banding\# 000 4003, 4146) #banding.mkv' can probably not be appended correctly to the track number 0 from the file 'C:\Users\Public\# 000\0, 4002) #Combined\# 000 0, 4002) #Combined.mkv': The codec's private data does not match (lengths: 2277 and 2278). Please make sure that the resulting file plays correctly the whole time. The author of this program will probably not give support for playback issues with the resulting file.
I understand there's no support for the older mkvtoolnix version but as I wrote it's carrying on in the latest. If I try to mess with the total frame line via command line's in MeGUI for HEVC it'll stop working.

When appending then muxing them together regardless of how many frames there are in total now it'll only go by the first mkv cut (ex: frames 0, 4003) and will display that under media info ignoring the appended frames added.

Quote:
# HEVC x265 3.0_Au+14-c7e5878bdd31

Encoding settings : cpuid=1111039 / frame-threads=3 / numa-pools=12 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=3 / input-res=1440x1080 / interlace=0 / total-frames=4003
So my question is, should I just ignore this as thus far there's been no issues during playback. Is this even an issue or not? It's kinda odd that it's able to recalculate the "Bits/(Pixel*Frame)" for the complete frame count of the video (at least I believe it's so) likewise with H264 but not do the same with HEVC's "total-frames=". There's no issues when doing these cuts/appends on H264 since it doesn't exist there.

Hopefully devs know what I am talking about concerning this and clue me in.

Last edited by dREV; 29th March 2019 at 10:36. Reason: so it's not widewide screen
dREV is offline   Reply With Quote
Old 29th March 2019, 11:12   #327  |  Link
sneaker_ger
Registered User
 
Join Date: Dec 2002
Posts: 5,565
You can ignore this in this case as long as the rest of the x265 settings are identical. It's not an issue. The "encoding settings" line you see in MediaInfo is x265's custom "SEI". It's only for information, not part of any standard. Mkvmerge will never alter it in any way. The bitrate on the other hand is taken from Matroska statistic tags (when present) which are in fact updated when appending.

When appending mkvmerge compares the header data of both tracks. When they are not identical it displays a "warning" that there might be a problem. It doesn't actually check whether or not the files are compatible in any meaningful way.

Last edited by sneaker_ger; 29th March 2019 at 11:15.
sneaker_ger is offline   Reply With Quote
Old 29th March 2019, 15:04   #328  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 3,078
This is my experience,too, using H.264 streams.

I often add 5 seconds of blank video with silent audio to the beginning and the end of my converted MKV files, and as long as I create the "Silence" file with exactly the same settings as the movie then MKVMerge has no complaints.

But recently I noticed that after replacing x264.exe with a newer build I do get these warnings about the codec's private data not matching. The weird thing is that the lengths are identical so I really do not see any mismatch.

The combined file plays perfectly, but just to be on the safe side I reencoded the "Silence" file using the newer x264 build, and the warnings disappeared.
manolito is offline   Reply With Quote
Old 29th March 2019, 17:33   #329  |  Link
sneaker_ger
Registered User
 
Join Date: Dec 2002
Posts: 5,565
"sky", "oil". 2 words that are not identical but both have length of 3 letters.
sneaker_ger is offline   Reply With Quote
Old 29th March 2019, 18:45   #330  |  Link
mkver
Registered User
 
Join Date: May 2016
Posts: 197
You can use e.g. ffmpeg's trace_headers bitstream filter to see what is in the relevant parameter sets: ffmpeg -i <Inputfile> -c copy -bsf:v trace_headers -t <appropriately short time like 0.1> -report -f null -
mkver is offline   Reply With Quote
Old 29th March 2019, 21:02   #331  |  Link
robena
Registered User
 
Join Date: May 2007
Posts: 29
Quote:
Originally Posted by mkver View Post
What about the actual CPU time used by the underlying mkvmerge process for each task? Does it make a difference when started from the GUI?
No, it's pretty low in both cases, below 20%. None of the 6 cores goes over 50%.
robena is offline   Reply With Quote
Old 29th March 2019, 21:41   #332  |  Link
mkver
Registered User
 
Join Date: May 2016
Posts: 197
I actually meant the accumulated CPU time, i.e. for the whole remuxing, not at a single instant in time. If the average CPU utilization is the same in both cases, but the CLI version uses it longer, then that would be really strange.
mkver is offline   Reply With Quote
Old 30th March 2019, 05:17   #333  |  Link
dREV
Registered User
 
dREV's Avatar
 
Join Date: Jan 2019
Location: Antarctica
Posts: 74
Quote:
Originally Posted by sneaker_ger View Post
You can ignore this in this case as long as the rest of the x265 settings are identical. It's not an issue. The "encoding settings" line you see in MediaInfo is x265's custom "SEI". It's only for information, not part of any standard. Mkvmerge will never alter it in any way. The bitrate on the other hand is taken from Matroska statistic tags (when present) which are in fact updated when appending.

When appending mkvmerge compares the header data of both tracks. When they are not identical it displays a "warning" that there might be a problem. It doesn't actually check whether or not the files are compatible in any meaningful way.
OK. Should I post this in HEVC thread to let them know about this? Seems like an oversight for those who cut frames.


Quote:
Originally Posted by manolito View Post
This is my experience,too, using H.264 streams.

I often add 5 seconds of blank video with silent audio to the beginning and the end of my converted MKV files, and as long as I create the "Silence" file with exactly the same settings as the movie then MKVMerge has no complaints.

But recently I noticed that after replacing x264.exe with a newer build I do get these warnings about the codec's private data not matching. The weird thing is that the lengths are identical so I really do not see any mismatch.

The combined file plays perfectly, but just to be on the safe side I reencoded the "Silence" file using the newer x264 build, and the warnings disappeared.
I too use .h264 streams and don't receive that. At least with version 8.3 64 bit. I don't get those when appending files together either in .h264 streams or mkv. Even when I've accidentally muxed in the wrong frames it gave me no warning about that. So that's odd you are receiving that. Have you by accident changed the x264 settings from one set of frames to the next? I've accidentally done that and would receive that message as well and as sneaker_ger wrote seems to be so. The x264 I'm using is x264 core 152 r2851kMod ba24899. http://komisar.gin.by/.

I think I tried updating to the newer ones but I think MeGUI couldn't use it since I'm using an old version (ver 2525) in order to still use .h264 streams via DGI as lsmash takes a bit longer to index.

Last edited by dREV; 30th March 2019 at 05:48. Reason: extra quote
dREV is offline   Reply With Quote
Old 30th March 2019, 07:40   #334  |  Link
sneaker_ger
Registered User
 
Join Date: Dec 2002
Posts: 5,565
Quote:
Originally Posted by dREV View Post
OK. Should I post this in HEVC thread to let them know about this? Seems like an oversight for those who cut frames.
Let who know? It's not a bug.
sneaker_ger is offline   Reply With Quote
Old 30th March 2019, 09:24   #335  |  Link
robena
Registered User
 
Join Date: May 2007
Posts: 29
Quote:
Originally Posted by mkver View Post
I actually meant the accumulated CPU time, i.e. for the whole remuxing, not at a single instant in time. If the average CPU utilization is the same in both cases, but the CLI version uses it longer, then that would be really strange.
Well, I not sure how to measure it precisely, but since the CLI lasts 6 times more with about the same average, the accumulated should be higher.
If you know a tool to measure the accumulated time of a process, I'll make the measurement.
Anyway, I have decided to replace my DAS by another NAS, more powerful with a 20Gbs aggregated Ethernet link, so the problem will be soon moot I hope.
robena is offline   Reply With Quote
Old 30th March 2019, 11:17   #336  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,823
Quote:
Originally Posted by manolito View Post
This is my experience,too, using H.264 streams.

I often add 5 seconds of blank video with silent audio to the beginning and the end of my converted MKV files, and as long as I create the "Silence" file with exactly the same settings as the movie then MKVMerge has no complaints.

But recently I noticed that after replacing x264.exe with a newer build I do get these warnings about the codec's private data not matching. The weird thing is that the lengths are identical so I really do not see any mismatch.

The combined file plays perfectly, but just to be on the safe side I reencoded the "Silence" file using the newer x264 build, and the warnings disappeared.
I always include --stitchable in the x264 command line. I never know when I'm going to need to split off a bit off an encode and re-do it to fix a problem and/or do some appending.

For reasons I don't understand, when x264 is writing directly to MKV, or you're appending MKV's, the warning is more likely to occur.
I've never understood why sometimes I can extract the raw h264 streams from each MKV to be appended (after MKVToolNix produced the codec private data warning), append the raw streams, and if the encoder settings were identical sometimes that'll make the codec private data warning go away, even if --stitchable wasn't used.

I'm happy to learn why... should anyone would care to explain.

Cheers.

PS. I'm using x264 core 155 r2901 7d0ff22 (with 32 bit MeGUI), as last time I tried to upgrade it, the newer x264 wouldn't run on XP. Is that still the case with the latest version?

Last edited by hello_hello; 30th March 2019 at 11:23.
hello_hello is offline   Reply With Quote
Old 30th March 2019, 12:36   #337  |  Link
sneaker_ger
Registered User
 
Join Date: Dec 2002
Posts: 5,565
Mkvmerge displays a warning if "CodecPrivate" element of 2 (or more) mkv files you wish to append aren't identical. "CodecPrivate" is an element of a Matroska container. So if you append 2 raw H.264 streams there is no "CodecPrivate". So there is no warning. The underlying problem remains the same, though.

Note that x264cli behaves a bit different depending on how you output:
to .mkv: H.264 headers (SPS/PPS) only at beginning of file
to .264: SPS/PPS at every keyframe
The latter can mitigate problems when appending files with different headers because the decoder can always access the correct PPS/SPS.


Note that not using --stitchable doesn't necessarily make files incompatible. But I think it is indeed a good idea to activate in case you want to make editing later.

Last edited by sneaker_ger; 30th March 2019 at 12:41.
sneaker_ger is offline   Reply With Quote
Old 30th March 2019, 12:48   #338  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 3,078
Quote:
Originally Posted by hello_hello View Post
PS. I'm using x264 core 155 r2901 7d0ff22 (with 32 bit MeGUI), as last time I tried to upgrade it, the newer x264 wouldn't run on XP. Is that still the case with the latest version?
The behavior I was talking about happened with LigH's latest build r2969 under Win7-64. My XP computer does not really handle HD sources. The latest x264 version which runs under XP and does not need SSE2 is 0.148.2579.0 from 2015.

I never used the --stitchable parameter in x264 because at the time when I created this "Silence" clip I used the same x264 build which was used for the movie conversion. The warning started to appear after I updated x264 to a newer build. For the movie conversion the parameters were absolutely identical. So I believe that every new build probably creates different "private data" even when identical parameters are used.

Maybe mkvmerge should be a little less strict when it issues this warning...


Cheers
manolito

Last edited by manolito; 30th March 2019 at 12:56.
manolito is offline   Reply With Quote
Old 30th March 2019, 14:01   #339  |  Link
dREV
Registered User
 
dREV's Avatar
 
Join Date: Jan 2019
Location: Antarctica
Posts: 74
Quote:
Originally Posted by sneaker_ger View Post
Let who know? It's not a bug.
To the HEVC developers.

I also did a test by doing two different frames. Cutting one with (0, 1) and the other with (2, 900) and the info showed total frames= 899 so I wanted to know what the documentation means by
Quote:
https://x265.readthedocs.io/en/latest/cli.html

If --total-frames is 1, then a stillpicture variant will be signaled, but this parameter is not always set by applications, particularly not when the CLI uses stdin streaming or when libx265 is used by third-party applications.
So wanted to let them know for those who cut frames and append and see if it wasn't an oversight.

Quote:
Originally Posted by manolito View Post
The behavior I was talking about happened with LigH's latest build r2969 under Win7-64. My XP computer does not really handle HD sources. The latest x264 version which runs under XP and does not need SSE2 is 0.148.2579.0 from 2015.
Didn't know you use XP. I'm on Windows 7 so I see why I am not expiriencing the same issue as you're going through.

Last edited by dREV; 30th March 2019 at 14:08. Reason: forgot quote
dREV is offline   Reply With Quote
Old 30th March 2019, 14:03   #340  |  Link
mkver
Registered User
 
Join Date: May 2016
Posts: 197
Quote:
Originally Posted by robena View Post
Well, I not sure how to measure it precisely, but since the CLI lasts 6 times more with about the same average, the accumulated should be higher.
If you know a tool to measure the accumulated time of a process, I'll make the measurement.
Anyway, I have decided to replace my DAS by another NAS, more powerful with a 20Gbs aggregated Ethernet link, so the problem will be soon moot I hope.
The easiest way to measure this is by using the Taskmanager: Make it show the CPU Time of your processes and take a look at the value of this counter right before the process ends.
(This is not a perfect benchmark: If a process takes (say) 10 % of a core's CPU time, then after 10s the CPU time will be 1s higher -- regardless of whether your CPU is currently running at its maximum/default frequency or not.)
mkver is offline   Reply With Quote
Reply

Tags
matroska

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 06:17.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.