Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
|
|
Thread Tools | Search this Thread | Display Modes |
7th May 2010, 02:17 | #1 | Link |
Registered User
Join Date: Apr 2010
Posts: 14
|
tscombine: a simple tool to combine two transport streams for error reduction
hi there
i wrote this tool a few months ago and feel that it's become pretty usable now, so i thought i would post it here. i wrote it because my clear-qam cable captures had annoying glitches in them here and there, and it was very difficult for me to get an error-free stream capture. likewise, over-the-air atsc had the same problem. however since both streams were from the same source, i was able to write a tool to detect and correct what errors it could at the transport stream level, resulting in a (potentially) bit-exact replication of the source broadcast. to use the tool: take two recordings of the same showing (the tool will cut off any leading and trailing parts which don't occur in both recordings), and run "tscombine.exe file1.ts file2.ts out.ts". it will provide information regarding progress as it processes the files. i'm also going to include the source code, in the hopes that if someone else finds this useful, they can develop it further. i've made a few comments in the source as far as a few ideas of my own for improvements, but as it servers my current needs fine already i'm unmotivated to actually code them . my only request is that if you do change the code to something better, that you post the code changes in this thread so others can benefit as well. one final note: it should be possible to recompile this into linux without too much difficulty (i originally developed it on an ubuntu box). |
7th May 2010, 15:41 | #3 | Link |
Registered User
Join Date: Apr 2010
Posts: 14
|
it uses a couple methods to detect errors. firstly, the transport stream multiplexer is _supposed_ to set a bit in a packet indicating if there was a transmission error. however it seems this doesn't always happen, so additionally it detects if there's a mismatch between the two streams, in which case it will choose the stream with the least amount of missing packets until it can re-sync them.
[edit] also, since this works on the transport stream level, it supports any codec which you can mux into a .ts. so mpeg2, avc, ac3, etc. |
7th May 2010, 18:49 | #4 | Link |
Registered User
Join Date: Aug 2008
Location: The Land Of Dracula (Romania - EU)
Posts: 934
|
thanks a lot...
_
__________________
if you ask a question and somebody give you the correct answer don't forget to leave a "thank you" note... Visit The Land Of Dracula (Romania - EU)! |
8th May 2010, 14:19 | #5 | Link |
Registered User
Join Date: Mar 2009
Location: Germany
Posts: 5,769
|
Does tscombine make use of a sort of internal synch in aligning the two recordings? The real life situation says that the two recordings won't start exactly at the same microsecond, and if the studios reencode the final output before airing it, the streams won't have the same structure....
|
8th May 2010, 15:21 | #6 | Link |
Registered User
Join Date: Apr 2010
Posts: 14
|
right, in this case a much more involved restructuring would be needed, and probably a re-encoding if you wanted the final output to be a .ts file. (not to mention would probably significantly increase the combining time...) this is for multiple recordings of the same source, such as atsc over-the-air + clear-qam rebroadcasts, or multiple over-the-air recordings of the same station. it is able to tolerate PID differences and will automatically map the differing PIDs to the correct values. one of the possible improvements i think i mentioned in the code would be for if the packets were aligned differently, or if they were multiplexed in a different order. fortunately in my tests it seems cable companies don't do this.
|
25th September 2010, 04:40 | #7 | Link |
Registered User
Join Date: Apr 2010
Posts: 14
|
i've been working on this bit by bit... one thing i changed was that it seems sometimes the error flag isn't marked on a packet that had an error, so in the version i posted here i hardcoded something like every time it sees a packet with errors, it drops 2 packets before and 3 after, to be safe... however with some streams this isn't enough, so i added a cmdline parameter to specify how big a buffer size you want.
one thing i keep getting though is that sometimes it syncs to the incorrect position in the files... all i can guess is that it's finding that some generic dummy packets match up before the actual bad spot is finished, and this causes problems... i was thinking of making it match at least 2 in a row before determining that the error region is finished. unless anyone has a better idea? |
26th September 2010, 15:58 | #9 | Link |
Registered User
Join Date: Mar 2009
Location: Germany
Posts: 5,769
|
A TS is supposed to be tolerant to transmission errors. First thing to do is to reorder the packets in their native order, only then perform the "puzzling" (the compare function should work, at least better than before).
|
28th September 2010, 07:24 | #10 | Link |
Registered User
Join Date: Apr 2010
Posts: 14
|
this would essentially mean demux the ts first, correct? the problem i see with doing that is you might lose audio sync, since i think the order of packets in the ts determines sync points... on the other hand, testing the data for decodability would be a more accurate indicator of the presence of errors. (and a lot more work, as then it would need to know how to decode codecs generally muxed into ts)
|
20th October 2010, 21:41 | #11 | Link |
Registered User
Join Date: Apr 2010
Posts: 14
|
i added additional checks to make sure it only copies streams with pid entries in the program map table. (sometimes muxers screw up and throw in random streams apparently). also i put in the part about matching up to k packets (k being some user inputted number) to determine a resync instead of just one. i would upload the new code but there's still something going horribly wrong sometimes, where it resyncs and copies over the data from one stream, but then somehow ends up copying the second streams data after that (it should simply skip over it), and i haven't been able to find where the problem is. (if someone here would be willing to take a look at it, i could upload the buggy code). if not then i'll upload it when i get around to fixing the problem
|
|
|