Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 20th November 2015, 15:29   #1  |  Link
raj2569
Registered User
 
Join Date: Aug 2005
Posts: 18
Decoding H.264 in MPEG-2 transport streams

Hi,

I am writing a program to identify IDR frames in an H.264 in MPEG-2 transport stream. The source video is streamed by vlc (v2.2.0) with "RTP/MPEG Transport Stream" destination and H.264+MP3 (MP4) profile.

So far I am not able to find a resource which documents the packet format. I can see documentation about packet structure of H.264: NAL Units, but documentation about how they are contained in MPEG-TS container is not to be found.

I can see the TS headers starting with 0x47, but what comes next is a mystery. Any pointers is much appreciated.


Following is the raw dump of some of the packets I am seeing for reference:

Start of dump of packet 386
47004518 246D6246 D9639AB8 76C15270
1F7001F4 EE89F77A D529056A 1D6F6B47
2A9DCDBB 52C06C9A 362D435A FA28E400
6D64E5C5 020F8A25 3EB4D0BB 2C8E8269
B2B67D81 B4E1EBDF 2D0AB171 1EE6B73E
C17C6AE8 07DDD566 A0F5748F DA9AA472
8E46ABC9 73C22BE8 0E9DB7BB 1E326861
B3727009 B65700E1 2CB401F9 33A7277E
2774C958 EE19A561 913FBE92 BDA1E1C1
9B7183FC 129CC115 8FD6ACB5 AFDFF8F9
D9C1F74A D1E0BB65 1A67C9D3 B277B486
B3852A9C B3BC3367 BAC141C6 00000000
Start of dump of packet 387
47404415 000001C0 01A98080 052190E7
97E3FFFD 80044433 33322232 22222211
11249249 00000900 00000000 ABAAABB9
AAAAA6AE 1C739138 E4145565 9761959A
5D765761 86DC6DB7 1C71D71B 6DD7227D
F420E461 3564DC8D 5CCE2B71 2BE94EEB
B7E38DE8 91020292 CDB4DAD5 36CD33A2
77E2DD4A A2E17252 592072C0 A1FA3438
76870ECE CE5836AC 646B74EA 63B5DDA5
6379DAF6 2F2C7152 384DB9AD B9F533CD
D5374042 391AD528 ED7354A6 BAA178DA
10E04347 16DFA8ED 7AED2045 00000000
Start of dump of packet 388
47004519 3A98BBB1 56DE0C76 DF359974
3AD6C36E B5C92FE6 91FB9282 1C773F28
EFE4EB90 FD25A18F C28A81C7 01437068
AEAF6D9E 04F7BAC8 D228DB46 839A6A85
F16403F7 FF9B7E7A 0788B29C B72FE010
A79996E6 BE7C3893 517602B0 8D7A5378
4067D8E4 7C0BAC0E 2CD85D5F 05BE42F9
83381659 505E63F7 5739C3C4 BBD138E9
22DB602C 7DC82BA7 463D8171 A0311912
80B6EEC5 7F08DE97 ED158BA4 14D7C036
E9784122 2C297F09 5C5690E5 18694202
F62D5E89 7B99AE71 B2869145 00000000
Start of dump of packet 389
4700451A 2071EA49 3F285273 36CF3308
877BA784 45E5D173 4B2F52F7 697A0C6F
A72DFF3E C0A5986A C26B33FD 6C37656F
22F40C96 E369F566 34E284B7 C63376E8
26AA7840 47582682 C26F1A18 6D41C1F5
384AE189 D04C85FB 3FD1F2DC 38DECC2F
A025086A 6396A8B3 D63BED58 AB81A922
1C9102CE DD484BC2 86267476 562F7B34
E0A559D7 26A517B5 2F4D1700 4FDA8C55
28992B03 8CD43F93 F06C5E40 498C1931
31410C40 90497216 6FD7D172 93D32599
B01F0D61 CB98B500 997EFE12 00000000
Start of dump of packet 390
4700451B F60EF8B1 20FC5EC4 0C3B2071
C60362E1 DA4CFEFA 87ACB0D5 87A685C3
41FBE318 759B5256 0782FA1C 04EB47FB
C21D1BDB 1CBBDD6B 96FFE3EC 73CF7CD4
4E8D42FD 476AEB66 A8BA60EA E01AAFD2
2FC4686F D45E5477 6195ED33 46C0D2E5
2B469519 5AF4AED1 D377F5F6 3C1279BE
5CAA6FB4 CE33AC74 FE50253C 6FF1DCE2
8039B20E E7ED7D5F 138B4F3E 7A506E4C
8E075283 F9BBF801 96EC2F5F 56F560D5
8EE93BDF 492C351B 52F23525 BF753B6A
8DBEFA08 EDCBBC6B CD8E05F4 00000000
Start of dump of packet 391
4700453C 8600FFFF FFFFFFFF FFFFFFFF
FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
FFFFFFFF FFFFFFFF FFFFFF2E 2D287D58
DC1829A7 A6196089 C59CC23A 29CDD44D
EBB4558F 3B070489 E9E65C7A 5C02F6AC
3E2BD107 ED6BAE24 3F924B20 00000000
Start of dump of packet 392
4740451D 000001E0 146A80C0 0A3190E7
D72F1190 E79C9700 00000109 E0000000
01419A33 49E10F26 53027FB0 F6E417BB
6BCB3929 2E359B04 5CB03DB6 4157981D
EBAEE340 A72CF089 81C653AD 4A191FC3
CE534FFD 7E3545F4 0512613F 1BE17DE0
DBA9648D B8D18DB6 22E5881F D904B78F
D3244E21 DC5D46A0 D2925288 4AF04550
F54AEFA3 E49F9CCA B4C34D1F 460CFFEA
D98BB2F0 19EF5D17 2B081543 AB16867E
1C968C7C 3C55156D F2135C80 1CD2B37B
4D5C40D1 4611341F 09C08504 00000000
raj2569 is offline   Reply With Quote
Old 20th November 2015, 19:42   #2  |  Link
pandy
Registered User
 
Join Date: Mar 2006
Posts: 1,049
There is no such thing as MPEG-2 TS - there is MPEG TS and it is covered by http://www.itu.int/rec/T-REC-H.222.0 - you should be able to find all information's there.
Side to this you may analyze code for existing open source TS demuxers:ffmpeg , https://code.google.com/p/tsdemuxer/ , http://www.w6rz.net/ xport etc.
pandy is offline   Reply With Quote
Old 21st November 2015, 11:29   #3  |  Link
SeeMoreDigital
Life's clearer in 4K UHD
 
SeeMoreDigital's Avatar
 
Join Date: Jun 2003
Location: Notts, UK
Posts: 12,219
Quote:
Originally Posted by pandy View Post
There is no such thing as MPEG-2 TS - there is MPEG TS and it is covered by http://www.itu.int/rec/T-REC-H.222.0 - you should be able to find all information's there.
Indeed...

That being said, the 'transport stream' is specified within Part-1 of the MPEG-2 ISO/IEC 13818 standard
__________________
| I've been testing hardware media playback devices and software A/V encoders and decoders since 2001 | My Network Layout & A/V Gear |
SeeMoreDigital is offline   Reply With Quote
Old 21st November 2015, 22:59   #4  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,558
Short answer: Search for 00000165, 00000145, or 00000125. You're guaranteed to find all IDR, but you might accidentally find something else if you happen to be in the middle of a non-AVC stream. If you write a TS, PES, and ES parser all of that information will naturally be available to you. You can also piggyback on ffmpeg's demuxer.

Long answer: You posted a limited number of packets that makes it hard to parse, but 386, 388, 389, 390, and 392 are all video stream, 387 is audio, and 391 is something else, based on the PIDs. It's obvious which are which because 387 and 392 have 000001 PES packet start codes (Cx-Dx is audio, Ex-Fx is video); the 474x in the TS header requires them there anyway. Since 392 is the only actual start of a video frame, we'll look at that.

The full PES header is 000001E0 146A80C0 0A3190E7 D72F1190 E79C97. That's not too important for now, except it tells us the frame size is 5216, or 185 TS packets. Following that is the ES, with 00 00000109, the (optional) AUD, the start of a H.264 frame. AUDs (and IDRs) almost always have 3 zero bytes instead of 2, just just as a gentleman's agreement, to make it easy to find new frames: Just search for every 00000001. E0 means any slice type.

Now you get to 000000 0141. 41 parses out to "referenced slice of non-IDR pic". If it was an IDR, it would be 45 (or 25 or 65, damn you AVC). x264 always uses 65 for IDR and 4x for referenced non-IDR, but some other encoders just use 4 for everything. Anyway, now you have your answer to seeking IDR startcodes.

Last edited by foxyshadis; 21st November 2015 at 23:01.
foxyshadis is offline   Reply With Quote
Old 23rd November 2015, 10:07   #5  |  Link
raj2569
Registered User
 
Join Date: Aug 2005
Posts: 18
Quote:
Originally Posted by foxyshadis View Post
now you have your answer to seeking IDR startcodes.
Thank you for your detailed answer. I am trying to understand what you have just said. Will get back with questions if any.

Once again, thanks!
raj2569 is offline   Reply With Quote
Old 23rd November 2015, 15:29   #6  |  Link
raj2569
Registered User
 
Join Date: Aug 2005
Posts: 18
foxyshadis,

I could understand the parts upto PES headers, could not understand what comes after that, before H.264 frame.

Quote:
Originally Posted by foxyshadis View Post
Following that is the ES, with 00 00000109, the (optional) AUD, the start of a H.264 frame. AUDs (and IDRs) almost always have 3 zero bytes instead of 2, just just as a gentleman's agreement, to make it easy to find new frames: Just search for every 00000001. E0 means any slice type.
What is the meaning of bytes 00 00000109 E0? Which std gives information about this bytes?

What is AUD?

For confirmation, does H.264 frame starts after 00 00000109 E0?

Quote:
Originally Posted by foxyshadis View Post
Now you get to 000000 0141. 41 parses out to "referenced slice of non-IDR pic". If it was an IDR, it would be 45
The 41 would be the forbidden_zero_bit, nal_ref_idc and nal_unit_type defined in 14496-10?

Thanks again for your help.
raj2569 is offline   Reply With Quote
Old 24th November 2015, 09:06   #7  |  Link
raj2569
Registered User
 
Join Date: Aug 2005
Posts: 18
I was looking for 00000165 to check for IDR frames, and found that it occurs deep inside the packet (marked in red). The dump is:

4740471B 000001E0 5A2980C0 0A3D61FB
0EE91D61 F9D45100 00000109 E0000000
0167
6400 1FACD940 400E7E7C 04400000
03004000 000C23C6 0C658000 00000168
EBE3CB22 C0000000 01658884 0023FFFC
39A9F2E1 6E7DA87E 0F17B2F6 65276E25
314443AC 423F1DF6 04E9BEC1 E15D492D
63E87C54 8AA37A53 3F8CD9B6 4586A380
62992B50 8C9F1FD0 E84CB72B 7D4A3811
B3AA15AB 296C88F8 1250C5E6 8FE42B97
DC0597A3 E9B6D01C 5C476777 F5C0E773
19C26B92 E3BEE60C DCC0828C 00000000


While I was expecting it at location marked in green. Just wondering if this is where it should be expected?
raj2569 is offline   Reply With Quote
Old 24th November 2015, 10:22   #8  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,558
Quote:
Originally Posted by raj2569 View Post
What is the meaning of bytes 00 00000109 E0? Which std gives information about this bytes?

What is AUD?
The H.264 standard, it just means "a new frame starts now." It's completely optional, does nothing, and exists only to enable fast seeking to frames. Otherwise, you're stuck looking at every 000001 startcode, testing what the NALU is (SPS, PPS, Slice, etc) and then having to do more decoding to piece together whether it's the first slice of the frame or not.

Quote:
Originally Posted by raj2569 View Post
For confirmation, does H.264 frame starts after 00 00000109 E0?

The 41 would be the forbidden_zero_bit, nal_ref_idc and nal_unit_type defined in 14496-10?

Thanks again for your help.
Immediately after E0 is the beginning of the first slice of the frame, yes, the byte being the three fields you stated.
foxyshadis is offline   Reply With Quote
Old 24th November 2015, 10:26   #9  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,558
Quote:
Originally Posted by raj2569 View Post
I was looking for 00000165 to check for IDR frames, and found that it occurs deep inside the packet (marked in red). The dump is:

4740471B 000001E0 5A2980C0 0A3D61FB
0EE91D61 F9D45100 00000109 E0000000
0167
6400 1FACD940 400E7E7C 04400000
03004000 000C23C6 0C658000 00000168
EBE3CB22 C0000000 01658884 0023FFFC
39A9F2E1 6E7DA87E 0F17B2F6 65276E25
314443AC 423F1DF6 04E9BEC1 E15D492D
63E87C54 8AA37A53 3F8CD9B6 4586A380
62992B50 8C9F1FD0 E84CB72B 7D4A3811
B3AA15AB 296C88F8 1250C5E6 8FE42B97
DC0597A3 E9B6D01C 5C476777 F5C0E773
19C26B92 E3BEE60C DCC0828C 00000000


While I was expecting it at location marked in green. Just wondering if this is where it should be expected?
67 is an SPS; the bitstream can send an SPS or PPS at anytime, and it's very common in streams to send both immediately before every IDR (some standards actually require this). This allows greater error resilience and the ability to begin decoding as soon as the IDR is reached. Immediately following it is the corresponding PPS, then the IDR.
foxyshadis is offline   Reply With Quote
Old 24th November 2015, 11:20   #10  |  Link
raj2569
Registered User
 
Join Date: Aug 2005
Posts: 18
Thank you for the detailed answer!

Just trying to map the bytes I am seeing with whats on standards.

Quote:
Originally Posted by foxyshadis View Post
The H.264 standard, it just means "a new frame starts now."
Is that "Access unit delimiter" nal_unit_type ?

Quote:
Originally Posted by foxyshadis View Post
E0 means any slice type.
I am reading through ISO/IEC 14496-10 standard doc, but could not figure out the values and its meaning of primary_pic_type.

Last edited by raj2569; 24th November 2015 at 14:40.
raj2569 is offline   Reply With Quote
Old 24th November 2015, 18:31   #11  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,558
Quote:
Originally Posted by raj2569 View Post
Is that "Access unit delimiter" nal_unit_type ?
Yup.

Quote:
Originally Posted by raj2569 View Post
I am reading through ISO/IEC 14496-10 standard doc, but could not figure out the values and its meaning of primary_pic_type.
Its only meaning is that if you see a value of 0, it's definitely an I-frame, if it's 1, it's an I or P frame, and if it's anything else it's probably not useful. Decoders may use it to seek to frame types they're interested in while seeking, if the encoder set the values to the minimum necessary. In your case it seems to just tag every frame E, even the IDR, so it's pointless to consider.

I haven't worked with multi-slice video in a long time, so it's always more useful to me to just read a few more bytes and grab the actual frame type instead of the AUD set. The AUD set is only potentially useful in the case where a frame has multiple slices and each slice has a different type and it's vitally important to know that.

A similar marker that's useless to decoders is that if the first slice is 5-9, every slice in the frame must be the same type. I rarely see this, but it's more informative than the AUD set.
foxyshadis is offline   Reply With Quote
Old 27th November 2015, 09:00   #12  |  Link
raj2569
Registered User
 
Join Date: Aug 2005
Posts: 18
Thanks a lot foxyshadis, I am now able to identify IDR frames correctly from the stream.
raj2569 is offline   Reply With Quote
Reply

Tags
h.264, mpeg-ts

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 14:18.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.