MKVToolNix v24.0.0 released

[hubblec4]

Hi Mosu
and many thanks for this new version.

In the ChapterEditor tool, is a button "Open Matroska, WebM- or Chapter files".
But WebM files doesn't contain chapters (says the specs).

[sneaker_ger]

# Version 14.0.0 "Flow" 2017-07-23

## New features and enhancements

* mkvmerge, mkvpropedit, MKVToolNix GUI (chapter editor): added support for
  chapters in WebM files that is spec-compliant by removing all tag elements
  not supported by the WebM spec. Implements #2002.

[Mosu]

Originally Posted by hubblec4

But WebM files doesn't contain chapters (says the specs).

Chapters in WebM are spec-compliant. The valid elements are a subset of all the chapter elements Matroska supports.

[Mosu]

Originally Posted by sneaker_ger

When flagging H.264 frames from mp4 as discardable is mkvmerge using mp4 container info (is there such info?) or is it parsing the H.264 bitstream?

For normal h.264 bitstreams: container information. In the case of MP4 this information is incomplete as the container level doesn't really provide it

In the case of Annex B bitstreams inside MP4: the bitstream information.

[hubblec4]

Originally Posted by Mosu

Chapters in WebM are spec-compliant. The valid elements are a subset of all the chapter elements Matroska supports.

Ok, thanks for this link.

[mkver]

Originally Posted by Mosu

For normal h.264 bitstreams: container information. In the case of MP4 this information is incomplete as the container level doesn't really provide it

In the case of Annex B bitstreams inside MP4: the bitstream information.

This is how I proposed it in #2047. But now that I think about it again I think my proposal was based on a misconception: That parsing framed input would be slow. In fact, for an eternity you have already looked at the first byte of every NAL unit in order to discard filler bytes and now AUDs as well. What do you think?

[Mosu]

Not interested. Due to how Matroska works, mkvmerge has to calculate all reference timestamps properly, which basically means treating any h.264 stream the way an Annex B stream is treated (safe for finding the unit boundaries, of course): calculating the frame order etc. The flags in the "simple block" structure such as "key" and "discardable" are only derived from those reference timestamps. And for situations such as simple blocks being disabled the reference timestamps will actually have to be written to the output file (in the block groups).

So no, it's not just looking at the slice type.

[manolito]

Just stumbled upon a weird interaction between TSDoctor and MKVMerge.

The source is a downloaded TS file from the ARTE Mediathek. I used the SD version, but the HD version has the same problem.

The downloaded TS file has a frame rate of 25 fps, it plays without issues. After running it through TSDoctor it still shows 25 fps and plays nicely. But after repacking this TSDoctor fixed TS using MKVMerge GUI the properties change to VFR, and playback shows heavy artifacts.

If I feed MKVMerge with the original downloaded TS without using TSDoctor then the resulting MKV has 25 fps CFR and plays without problems.

I uploaded my results here:
http://www83.zippyshare.com/v/cndrA2sS/file.html


Any idea what is happening here? I tend to believe that MKVMerge is to blame, because when I use FFmpeg to repack the TSDoctor fixed TS file then the resulting MKV does not show any issues.


Cheers
manolito

[Selur]

Looks to me like TSDoctor is not properly adjusting the headers, mkvmerge keeps the messed up headers and delivers vfr, ffmpeg ignores the headers partially and returns cfr. So I would blame TSDoctor and not mkvtoolnix.

[sneaker_ger]

Mkvmerge drops some frames. E.g. frame #16 display/#13 coded, P frame.

[mkver]

Beginning with mkvmerge 20.0 mkvmerge nows uses the information from the continuity counter that the transport stream provides to decide whether a PES packet in the input is corrupted; corrupted PES packets are dropped. ProjectX says that there are some discontinuities in the continuity counter, so this is what's happening here. It also complains a lot about bit errors in packets.
Why do you use tsdoctor at all? What can it do that mkvmerge can't (apart from outputting a transport stream what mkvmerge obviously won't)?
[Edit]: It's actually slightly different to what I said above: There are some packets in your file that look like this:

47 01 00 39 B7 00 FF FF ... FF FF

(... = 178 more 0xFF)
According to the TS header this packet has an adaption field followed by a payload. The B7 (=183) says that the whole rest of the ts packet belongs to said adaption field, i.e. contrary to what has been signalled the packet doesn't even contain a payload. Now the thing is that the continuity counter is only supposed to increase for packets with payload. If one ignores this restriction, then there is no discontinuity in the continuity counter; but if one expects the continuity counter not to increase with such packets, then one has a discontinuity in the continuity counter which leads to the rejection of the entire PES packet.
Now I have to admit that I don't know whether the continuity counter should increment if the header signals the presence of a payload despite there being none (or does a 0 B payload count?) or if it should only increment when there is an actual payload. Anyway, this faulty header information leads to ProjectY's reports of bit errors in the packet; therefore it drops said packets and reports a discontinuity in the continuity counter afterwards. mkvmerge is likely to do something similar.
PS: I'd really liked to know what TSDoctor is supposed to achieve.

[manolito]

I have been using TSDoctor for a long time for video broadcasts in TS containers mainly to repair broadcast glitches and remove padding data. You are probably correct to question its usefulness for non-broadcast streams like downloaded files or ripped files from a BD.

You cannot really compare TSDoctor to ProjectX. ProjectX converts MPEG2 transport streams to program streams, it repairs broadcast glitches and takes care of keeping A/V sync. TSDoctor keeps the transport stream format when repairing the source, and when there was a glitch in the transmission TSDoctor will issue a warning, but it will not repair the stream. The resulting "fixed" stream will still exhibit A/V sync problems (which ProjectX would have repaired).

Still my question remains valid. Why does the TSDoctor fixed TS file work perfectly, but after repacking it to MKV with mkvmerge the resulting MKV is broken? And doing the repacking of the same file to MKV with FFmpeg does not introduce any problems?


Cheers
manolito

[mkver]

1. mkvmerge does also remove padding data by default. And has done this for a very long time. So I don't see a point in using TSDoctor for broadcast streams, too. You yourself said that mkvmerge when fed with the original ts file produces a flawless file.
2. I did not compare ProjectX to TSDoctor; I just said that I used ProjectX to analyze your sample.
3. I don't see that TSDoctor fixed your TS file; the file it created is corrupted as I have explained. And since when does a mediathek use filler data at all? That would be news to me. So why do they exist in the version produced by TSDoctor? Really strange.
4. ffmpeg ignores the continuity counter, therefore it doesn't care about discontinuities in the continuity counter.

[Mosu]

Originally Posted by mkver

Now I have to admit that I don't know whether the continuity counter should increment if the header signals the presence of a payload despite there being none (or does a 0 B payload count?) or if it should only increment when there is an actual payload. Anyway, this faulty header information leads to ProjectY's reports of bit errors in the packet; therefore it drops said packets and reports a discontinuity in the continuity counter afterwards. mkvmerge is likely to do something similar.
PS: I'd really liked to know what TSDoctor is supposed to achieve.

ISO/IEC 13818-1 (the standard document describing MPEG program and transport streams) has this to say about the continuity_counter:

continuity_counter – The continuity_counter is a 4-bit field incrementing with each Transport Stream packet with the
same PID. The continuity_counter wraps around to 0 after its maximum value. The continuity_counter shall not be
incremented when the adaptation_field_control of the packet equals '00' or '10'.

The lower bit of the two is what signals the presence of data payload, the higher one the presence of the adaptation field. The standard doesn't say anything about the actual number of payload bytes having anything to do with the decision whether or not the continuity_counter should be incremented; and indeed, mkvmerge doesn't base its decision on it either.

That 0xb7 = 0b10110111 indicates: 0b10 = transport_scrambling_control; 11 = adaptation_field_control ( = adaption field present, payload present), 01111 = current continuity_counter. The "payload present" flag is set, therefore the condition mentioned above is not met, meaning the continuity_counter must be increased.

If TSDoctor is writing such header fields but isn't increasing the continuity_counter, then it produces broken files and should not be used until that issue's been fixed.

Originally Posted by manolito

Still my question remains valid. Why does the TSDoctor fixed TS file work perfectly, but after repacking it to MKV with mkvmerge the resulting MKV is broken?

We've just explained this to you.

Originally Posted by monilito

And doing the repacking of the same file to MKV with FFmpeg does not introduce any problems?

Just guessing: maybe ffmpeg can recognize such situations and fixes them silently.

[nevcairiel]

FFmpeg reads the continuity counter, and it warns if it mis-matches, but it doesn't automatically drop data.

PS:
Your analyzes is one byte off Mosu, the important header starts at 39, but the conclusion remains the same.

[mkver]

Originally Posted by Mosu

That 0xb7 = 0b10110111 indicates: 0b10 = transport_scrambling_control; 11 = adaptation_field_control ( = adaption field present, payload present), 01111 = current continuity_counter. The "payload present" flag is set, therefore the condition mentioned above is not met, meaning the continuity_counter must be increased.

You are off by 1 B: 0x39 is the header byte containing the information concerning TSC, adaption field, payload and continuity counter. It says that both an adaption field and a payload is there (and no scrambling is used); the 0xb7 is the size of the rest of the adaption field (following 0xb7) and it says that 183 B are this rest. Together with the length field and the 4 bytes of header this is the complete packet.

Originally Posted by Mosu

If TSDoctor is writing such header fields but isn't increasing the continuity_counter, then it produces broken files and should not be used until that issue's been fixed.

TSDoctor is writing packets that contain payload according to the header, but in reality don't contain any payload. For these packets it is incrementing the continuity counter which is in line with the spec and the code you cited. I thought that mkvmerge checks whether a ts packet has any real payload, but if it just checks the header then I actually don't understand why this frame is dropped; because then there is no discontinuity in the continuity counter. Version 19.0 doesn't have any problems with it, so it must be the recent changes to the ts input module.
@Nevcairiel: I have seen ffmpeg outputting error messages that PES packet are incomplete, but I never got an error message (or a warning) informing me of discontinuities in the continuity counter.

[Mosu]

Yes, I was off one byte (oops!), but as Nev said, it doesn't actually change the analysis: the adaptation_field is still 0b11, and therefore the continuity_counter must be incremented.

Version 19.0 doesn't have any problems with it, so it must be the recent changes to the ts input module.

Well, that may very well be, but I'm not really interested in spending time on this. There are three methods described in this thread that don't pose a problem (not using TSDoctor in the first place, using ffmpeg on the file produced by TSDoctor, using an older mkvmerge). If anyone wants to debug mkvmege and propose a fix, I'll gladly apply it, of course.

[Mosu]

Well, I did look into my source code. And yes, this is a bug in mkvmerge. From what I can tell, mkvmerge doesn't update its expectation about what the next continuity_counter is supposed to be if a TS packet doesn't contain payload bytes. The line I've linked to above handles that case correctly, it simply isn't called in that case (and that's likely the issue here). I'll see if I can get it fixed.

[Mosu]

Should be fixed in the latest pre-builds.

[mkver]

It is. Except for the tags the output is the same as the one from 19.0.