Xin1generator: Create Ordered Chapter mkv's from Seamless Branching BluRay's

[Fullmetal Encoder]

robpdotcom, do you know if the author is currently accepting requests for improvements on this or is he pretty much done with it?

[robpdotcom]

Originally Posted by Fullmetal Encoder

robpdotcom, do you know if the author is currently accepting requests for improvements on this or is he pretty much done with it?

I haven't contacted him in a while, but he was very open to suggestions when I did. I'm thinking of asking him to try to incorporate mmg - it can now open m2ts files, so it should be possible to create an mkv from the disc in one step.

What were you thinking of requesting?

[Fullmetal Encoder]

Originally Posted by robpdotcom

I haven't contacted him in a while, but he was very open to suggestions when I did. I'm thinking of asking him to try to incorporate mmg...

Well, regarding using mkvmerge I was going to bring that up. When I discovered that it could so easily handle .m2ts file demuxing I was impressed. Not just the video but it perfectly handles the HD audio and subtitles and packages them all into nice, neat little MKV's for us. But this highlights a problem I have found with the way Xin1Generator produces h264 files. It is using eac3to for this. What I have found is that eac3to apparently does not properly write the h264 file. I think it does something to the stream because I compared a DGIndex dga file produced from eac3to's h264 with a dga file created directly from the .m2ts and they differ. But if eac3to isn't modifying the video stream then I would think they should be same. Incidentally, testing a dga file generated from an h264 stream created by mkvmerge against the m2ts dga yielded a perfect match. This leads me to believe that mkvmerge would be a much better choice for demuxing the video stream.

Originally Posted by robpdotcom

...it can now open m2ts files, so it should be possible to create an mkv from the disc in one step.

Yes, it could give MakeMKV a run for its money At least for those of us who have their disks already on their computer. In fact being able to demux everything out into a single MKV would be extremely helpful to me for my particular workflow as it would pretty much eliminate file clutter. This could be done with mkvmerge now but it would be very nice indeed to have the option to specify multiple titles to be packaged like that all in one go through Xin1Generator's CLI.

Originally Posted by robpdotcom

What were you thinking of requesting?

Well, my goal is to be able to do Avisynth processing on my blu-rays excluding certain segments within an .m2ts title while maintaining chapters in the final product with start points that land precisely on I-frames (totally smooth chapter navigation). The problem is that television shows will almost always have an OP and an ED. If I demux and process the entire title the OP and ED get processed and encoded for every single episode over and over again. Naturally this wastes a tremendous amount of processing time and drive space and requires a lot of drive thrashing for no reason. Thus, I'd like to be able to use Xin1Generator to specify which chapters get extracted into respective audio, video and subtitle streams (with each chapter as a separate file).

Of course, if certain chapters have been excluded from the title then the original chapter start points are no longer useful but that's why the goal is to get each chapter demuxed from the m2ts into it's own separate file. I can then attach an "empty" chapter file (with 0 for a start time) to each chapter file and combine them into whole episodes using mkvmerge. When doing this, mkvmerge automagically assigns the correct chapter start points to land perfectly at the beginning of each chapter segment.

As luck would have it, mkvmerge has the ability to split the video, audio and subtitle streams like this. All you have to do is provide it with the appropriate chapter time points which Xin1Generator already has perfect access to

Of course, for this to be truly powerful it would need to be accessible in CLI. It shouldn't be too difficult to do this as Xin1Generator already can tell how many chapters are in a title. If they were identified by Xin1Generator as, say, chapters 1-6, then using a command line switch and comma separated list of the chapters I want to keep should be all that's needed and the program could just impose that on mkvmerge. In fact, it could go a step further as sometimes the middle of an episode will have multiple chapter points that we may want to be extracted as one long chapter. For example, for a title with a chapter structure that looks like this:

Chapter 1 - Prologue
Chapter 2 - OP
Chapter 3 - Part A
Chapter 4 - Part B
Chapter 5 - ED
Chapter 6 - Next Episode Preview

the command line might look like this:

-c 1,3-4,6

There is one problem with that though and that is that mkvmerge's method for demuxing and splitting a stream by time values doesn't explicitly allow you to forgo demuxing a segment within a title that you do not want to demux. So for example, if I tell mkvmerge to split an .m2ts episode after the prologue (before the OP) and after the OP (before part A) it will still write the OP in a separate, demuxed file. I don't know any way to stop it from writing the OP without forcibly aborting the splitting process. But maybe the developer of Xin1Generator has a better way.

Oh well, this is probably too much to be implemented but since I have some access to the developer I figured why not go for the gold.

I appreciate your acting as the go-between for us but as a matter of curiosity why doesn't he appear here himself? Or maybe they are too busy at MIT to bother with us :P

[robpdotcom]

I think the author is not here because he mostly created the program for his own personal use, but is kind enough to share it with everyone else. Of course, that's just a guess, but it's completely his choice if that's the case.

As far as the way eac3to handles h264 files, I seriously doubt it does anything wrong. Do you extract them, or mux them to mkv? From what I have been told, it's not a good idea to extract raw h264 files, so I always mux them. I've used it on over 100 h264 encoded discs, and never had any problems.

[Fullmetal Encoder]

Originally Posted by robpdotcom

I think the author is not here because he mostly created the program for his own personal use, but is kind enough to share it with everyone else. Of course, that's just a guess, but it's completely his choice if that's the case.

As far as the way eac3to handles h264 files, I seriously doubt it does anything wrong. Do you extract them, or mux them to mkv? From what I have been told, it's not a good idea to extract raw h264 files, so I always mux them. I've used it on over 100 h264 encoded discs, and never had any problems.

My process to test the eac3to demux of the h264 stream was to first use DGAVCIndex to demux a raw h264 stream from an m2ts file as well as create the dga index file. The next step was to create a dga from the h264 stream that eac3to produced from the same segment in the same m2ts file. I did that and then compared both of the dga files in a text editor. Since DGIndex should be operating the same way when working with each h264 stream I would expect the dga's to be the same. Interestingly enough the h264 streams themselves are different with the one produced by DGAVCIndex being about 2.5 MB larger. So for me this begs the question, where did the 2.5 MB go?

At first I wondered if this might be due (somehow) to the fact that one dga was created from the m2ts file directly while the other dga was created from the elementary stream produced by eac3to. So I tried creating a dga from the elementary stream produced by DGAVCIndex (as opposed to the m2ts) and obtained the same results as before. Actually, looking at the files just now I noticed that the DGAVCIndex stream has the extension ".264" while the eac3to stream has the extension ".H264" Perhaps there is some padding in the .264 file or something. I am not aware of the intricacies of the two extensions.

TL;DR

In any case the testing was valuable because it strongly indicates to me that there would be no trouble trusting mkvmerge's demux of the video stream should any option to use mkvmerge be implemented in Xin1Generator.

If you get the chance, thank the developer for me for providing this tool to us. It really is a golden nugget because it's the only tool I know of that can incorporate extra versions from a blu-ray into a single mkv.

[DMosc]

Hi. It's me, the author. Let me first address why I haven't been active in this thread. Xin1Generator started out as a tool for a certain private community, and I still provide support there. About a year ago I made the program publicly available via Google Code. This allowed me to provide support in two additional ways:

• Google Code's Issues functionality
• Email; my email address can be found on the Google Code page

I initially did not want to provide support in another way, but apparently many people can't find my contact information on Google Code or simply prefer to post on this forum. I'll monitor this thread more closely from now on.

Originally Posted by mbcd

A nice suggestion would be to have a option in X1, or a seperate Tool for demuxing.

It would be best to have this as a separate tool. If anyone is interested in picking this up, feel free to take a look at a tool I wrote a few years ago that deals with files created by Xin1Generator, among other Matroska files with unordered chapters: ChapterUnorderer. In its current state, it not very suited for use on Xin1 files, simply because it take a really long time and consumes a lot of space. It might work, though!

Originally Posted by hubblec4

a "Stop working" button will be nice.

I agree. Unfortunately, that's a functionality I hadn't considered while writing the program, and it's not so simple to add at this point due to the way Xin1Generator handles threads and processes. If any developer knows of a relatively simple way to do this in a way more sophisticated than "kill every instance of eac3to and xport", I'm all ears.

Originally Posted by robpdotcom

I'm thinking of asking him to try to incorporate mmg - it can now open m2ts files, so it should be possible to create an mkv from the disc in one step.

I like this! One reason everyone loves eac3to is the way it flawlessly deals with delays and overlaps. Can you confirm that mkvmerge handles appending M2TS files equally well?

Originally Posted by Fullmetal Encoder

Well, my goal is to be able to do Avisynth processing on my blu-rays excluding certain segments within an .m2ts title while maintaining chapters in the final product with start points that land precisely on I-frames (totally smooth chapter navigation). The problem is that television shows will almost always have an OP and an ED. If I demux and process the entire title the OP and ED get processed and encoded for every single episode over and over again. Naturally this wastes a tremendous amount of processing time and drive space and requires a lot of drive thrashing for no reason. Thus, I'd like to be able to use Xin1Generator to specify which chapters get extracted into respective audio, video and subtitle streams (with each chapter as a separate file).

Of course, if certain chapters have been excluded from the title then the original chapter start points are no longer useful but that's why the goal is to get each chapter demuxed from the m2ts into it's own separate file. I can then attach an "empty" chapter file (with 0 for a start time) to each chapter file and combine them into whole episodes using mkvmerge. When doing this, mkvmerge automagically assigns the correct chapter start points to land perfectly at the beginning of each chapter segment.

As luck would have it, mkvmerge has the ability to split the video, audio and subtitle streams like this. All you have to do is provide it with the appropriate chapter time points which Xin1Generator already has perfect access to

Of course, for this to be truly powerful it would need to be accessible in CLI. It shouldn't be too difficult to do this as Xin1Generator already can tell how many chapters are in a title. If they were identified by Xin1Generator as, say, chapters 1-6, then using a command line switch and comma separated list of the chapters I want to keep should be all that's needed and the program could just impose that on mkvmerge. In fact, it could go a step further as sometimes the middle of an episode will have multiple chapter points that we may want to be extracted as one long chapter. For example, for a title with a chapter structure that looks like this:

Chapter 1 - Prologue
Chapter 2 - OP
Chapter 3 - Part A
Chapter 4 - Part B
Chapter 5 - ED
Chapter 6 - Next Episode Preview

the command line might look like this:

-c 1,3-4,6

There is one problem with that though and that is that mkvmerge's method for demuxing and splitting a stream by time values doesn't explicitly allow you to forgo demuxing a segment within a title that you do not want to demux. So for example, if I tell mkvmerge to split an .m2ts episode after the prologue (before the OP) and after the OP (before part A) it will still write the OP in a separate, demuxed file. I don't know any way to stop it from writing the OP without forcibly aborting the splitting process. But maybe the developer of Xin1Generator has a better way.

Oh well, this is probably too much to be implemented but since I have some access to the developer I figured why not go for the gold.

While an interesting use case, that's honestly too much work for me to consider implementing right now. Sorry. Patches welcome!

[robpdotcom]

Hey! Glad to see you stopping by here. I think you're right that some people have trouble finding your contact info, and I imagine that some are also hesitant to use email, out of fear they would be bugging you.

Originally Posted by DMosc

I like this! One reason everyone loves eac3to is the way it flawlessly deals with delays and overlaps. Can you confirm that mkvmerge handles appending M2TS files equally well?

I can't confirm it, but I will do some testing. I guess a good way would be to use Xin1 to create the chapter file and then use mmg to manually append the tracks?

I looked into this a bit a while back, and did see one thing that may be helpful. MKVmerge has two ways of appending files, as described below:

--append-mode mode

Determines how timecodes are calculated when appending files. The parameter mode can have two values: 'file' which is also the default and 'track'.

When mkvmerge appends a track (called 'track2_1' from now on) from a second file (called 'file2') to a track (called 'track1_1') from the first file (called 'file1') then it has to offset all timecodes for 'track2_1' by an amount. For 'file' mode this amount is the highest timecode encountered in 'file1' even if that timecode was from a different track than 'track1_1'. In track mode the offset is the highest timecode of 'track1_1'.

Unfortunately mkvmerge cannot detect which mode to use reliably. Therefore it defaults to 'file' mode. 'file' mode usually works better for files that have been created independently of each other; e.g. when appending AVI or MP4 files. 'track' mode may work better for sources that are essentially just parts of one big file, e.g. for VOB and EVO files.

Subtitle tracks are always treated as if 'file' mode were active even if 'track' mode actually is.

Anyway, I'll test out using mmg to append the files, probably tonight, and report back the results.

Oh, and btw, thanks again for sharing your program, and now for providing feedback here. Of course, you've probably opened yourself up to even more requests, and probably even some criticism, just as other developers here subject themselves to.

[Mosu]

I like this! One reason everyone loves eac3to is the way it flawlessly deals with delays and overlaps. Can you confirm that mkvmerge handles appending M2TS files equally well?

mkvmerge doesn't handle appending M2TS files well at the moment, unlike multiple VOBs from a single DVD for example. What I'm talking about is when a logical M2TS file is split into several files and which should really be read as one file. It's on my TODO list, though.

[robpdotcom]

Just an fyi, I tried using mmg to append the m2ts files of a seamless branching disc, and the results were not too good.

Guess we'll have to wait for Mosu to release a version that handles them better.

[hubblec4]

it will help if the output-file(s) gets the name of the language.

is there a way to do this?

[DMosc]

Originally Posted by hubblec4

it will help if the output-file(s) gets the name of the language.

is there a way to do this?

Not right now, but it's an easy thing to implement once I have a bit more time on my hands. Harass me again in the upcoming weeks or file an issue on Google Code to help me remember.

[hubblec4]

i have found a way to get streams with the correct titles like eac3to do.

after generating the files (chapter, tag and extract.cmd) i open the extract.cmd file in an editor an delete all to extracted streams (1:track1 2:track2...) and type only the switch
-demux

now all streams will be demuxed with the right names of language and other informations.



PS:

or you unselect all streams. then set xin1-generater in the extract.cmd automaticly the switch "-demux".

[Farfel]

Ok, so I think I understand how this seamless branching/ordered chapters thing works, but I still have some questions.

In your example (and every example I've read about so far) you're using a single video track. That track has a "real" timeline and the ordered chapters are then used to jump around it, creating separate "Cuts." But what if I have a Blu-Ray disc that uses seamless branching with different video tracks? e.g. pretend there's a "Super Duper Cut" that includes all deleted scenes, even ones that not fully post-processed (lacking CGI, lighting, music, etc). These deleted scenes will most likely NOT be 1080p, but rather 480i or else. Also, what if I wanted to include Behind-the-scenes clips in the MKV file, how would I go about doing that if it contained different video characteristics as the main film?

I should point out some of this is hypothetical as I haven't encountered a Blu-Ray disc yet that has a Cut as crazy as I described above (with the different video sizes/framerates/codecs, etc) but I'm assuming it's possible? If it is, how would I stuff all that craziness into an ordered-chapter MKV file? Thanks.

[hubblec4]

with differnet videosize,- codec and other sounds, i think its not possible to create a single ordered chapter mkv.

maybe yo can create multiple linked files.

[Farfel]

Does the Blu-ray spec even allow this in seamless branching? Has anyone encountered a disc yet with different video size/codec in the same cut?

[DMosc]

The "craziest" seamless branching I've seen was on Certifiable: Live in Buenos Aires by The Police. It has TrueHD/AC3 audio tracks in most of the files, but plain AC3 on two very small files with a total length of 6 seconds. This messed up Xin1Generator because eac3to would ignore those files in demuxing, while Xin1Generator did use those files to determine the number of frames. Temporarily renaming the two files so eac3to wouldn't pick them up in the first place provided a workaround.

I can't guarantee you that other similar problems don't exist, but it's rare enough that we can handle them on a case-by-case basis.

[Thunderbolt8]

^^even that didnt work out properly in my case, because for some reason, my chapterfile was different from yours, although our eac3to and xin1generator versions were the same -.-

[szabi]

Hi

Can I use this program for DVD as well?
If it is not possible, could anyone upload these workable files created by the program?

chapters.xml
tags.xml
qpfile.txt
extract.cmd

Perhaps I can understand how to use chapters to reach my goal.
Thnx.

bye
szabi

[DMosc]

Xin1Generator itself could be fairly easily modified to support DVDs, but unfortunately eac3to (which Xin1Generator relies heavily on) simply doesn't support the DVD structure.

Here's an example of Xin1Generator output: http://www.mediafire.com/?74urzmlg09en0lu

[szabi]

Thnx for upolading.
I tried to do it manually.
There are 19 differences between theatrical and directory version, in the movie.
Some scene is not only longer but exchanged as well.
When i realised that i postponed the project.
If it had been a simple extended version i could have done it.
But different scenes, no idea.

bye
szabi