Current Patches, Where to get them, How they affect speed/output

[Dark Shikari]

Originally Posted by Rodger

I can´t sugguest this release as it totally sucks doing an encode 1280*720 at 4Mbit! Total blockyness! everything washed out.
Just my 2 cents.

Does it give particularly *different* results from a non-experimental build? Most of the extra patches applied are off by default...

[MythCreator]

Originally Posted by Rodger

I can´t sugguest this release as it totally sucks doing an encode 1280*720 at 4Mbit! Total blockyness! everything washed out.
Just my 2 cents.

Maybe you use FGO?Or something else.I doing an encode 1280*720 at 1.5Mbps,everything was fine....

Here's my settings

--no-psnr --no-ssim --cqmfile "C:\AutoMKV\profiles\x264-profiles\matrix\M4G_High_Detail_V3.1.cfg" --no-fast-pskip --no-dct-decimate --mixed-refs --scenecut 40 -I 90 -i 25 --me tesa --merange 16 --sar 1:1 --threads 3 -r 6 --nr 0 -f -3:0 -b 16 --direct "auto" --bime --b-bias 0 --b-pyramid -w --deadzone-inter 6 --deadzone-intra 6 --aq-mode 0 --chroma-qp-offset 0 --direct-8x8 1 -B 1500 --qpmin 10 --qpmax 51 --qpstep 4 --qcomp 0.600000 --ipratio 1.400000 --pbratio 1.300000 --8x8dct -A p8x8,p4x4,b8x8,i8x8,i4x4 -m 7 --b-rdo --ratetol 1.000000 --vbv-init 0.9 --vbv-bufsize 0 --vbv-maxrate 0

[Rodger]

MeGui´s current release, Jarods 826, was okay doing the same job.
Nobody else experiencing the same? I´ll have a second look on that.


--bitrate 4000 --level 4.1 --keyint 25 --min-keyint 2 --ref 2 --mixed-refs --bframes 2 --bime --weightb --nf --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --ipratio 1.1 --pbratio 1.1 --vbv-bufsize 24000 --vbv-maxrate 32000 --qcomp 0.5 --merange 12 --threads auto --thread-input --progress --no-psnr --no-ssim --output "output" "input" --mvrange 511 --aud --nal-hrd

[Inventive Software]

Originally Posted by Rodger

I can´t sugguest this release as it totally sucks doing an encode 1280*720 at 4Mbit! Total blockyness! everything washed out.
Just my 2 cents.

Post a command line before posting such a damning verdict.

[chriszxl]

Originally Posted by Inventive Software

Post a command line before posting such a damning verdict. ;-)

DDDamnin' right! posting the command line 1st. then we can find out the problem

[lexor]

Originally Posted by chriszxl

DDDamnin' right! posting the command line 1st. then we can find out the problem

you realize that you both asked him for the line at least half an hour after he posted it? where you writing your post for all that time?

[Inventive Software]

Pedanticism aside lexor (we have other lives too, we're not on here 24/7, so quit bugging when we ask something a bit "late" ), those are quite conservative settings. 2 Refs and 2 B-frames for 4 Mbits? If it's very fast motion, 4 Mbits won't be enough, so either increase the bitrate, or increase the Refs and B-frames. 8 for each would be a could number.

[lexor]

Originally Posted by Inventive Software

Pedanticism aside lexor (we have other lives too, we're not on here 24/7, so quit bugging when we ask something a bit "late" )

Dunno about pedantic, but I'm pretty sure that your argument that you don't have time to read posts after you asked him to post more is a logical fallacy and now you have 2 posts instead of 1 you would otherwise make, doesn't fit the time saving mantra either.

Also I don't see the problem with b-frames and refs that you mentioned. At most he can raise each by 1 to maintain CE compatibility (which by the looks of the other settings is probably the reason it's so low to begin with). That won't give him enough, not to mention that 3,3 for those 2 works wonders on every DVD5 HD backup I've done (and Italian Job for one has some fast motion with blur and wacky colours to boot). I doubt +1 to both will save the picture from being a horrid mess.

Rodger, do other movies playback fine? i.e. do you know the decoder isn't the issue, like deblocking being skipped maybe?

[Rodger]

So here is my final "judgement".

TO ME this release tends more to blockyness than the "current" release for Megui.

This monster blockyness I got out of that new release that one time must have been a bad encode or something...I don´t have that source anymore, so I can´t rebuild the error.

But to repeat myself...I´d say the new version just tends more to blockyness than the old one.

Try yourself...If it´s just me....than everything has to be fine.
I just wanted to share my thoughts with you.

Infos: Same settings used as above. The "effect" does come to get noticed more on green grass like gardens.

[MythCreator]

Just for more test

x264.839.modified.experimental.1.exe

contains the above patches:
x264.gaussian.cplxblur.01.diff
x264_me-prepass_DeathTheSheep.01.diff
x264_2pass_vbv.7.diff
x264_hrd_pulldown.04_interlace.diff
x264_fix_win_stdin.diff



And then:
x264.839.modified.experimental.2.exe

Contains these:
x264.gaussian.cplxblur.01.diff
x264_me-prepass_DeathTheSheep.01.diff
x264_2pass_vbv.7.diff
x264_hrd_pulldown.04_interlace.diff
x264_fix_win_stdin.diff
x264_fgo.01.826.diff
32x32samples_crash.diff
cosmetic.diff
debug-defines.diff
fix_stats_file_work_for_cli.diff
frames_memoryleak.diff
multithreading_Nth_pass_ratecontrol.diff




both are make fprofiled by GCC 4.4.0 20080331 experimental

[chriszxl]

Originally Posted by MythCreator

Just for more test

x264.839.modified.experimental.1.exe

contains the above patches:
x264.gaussian.cplxblur.01.diff
x264_me-prepass_DeathTheSheep.01.diff
x264_2pass_vbv.7.diff
x264_hrd_pulldown.04_interlace.diff
x264_fix_win_stdin.diff



And then:
x264.839.modified.experimental.2.exe

Contains these:
x264.gaussian.cplxblur.01.diff
x264_me-prepass_DeathTheSheep.01.diff
x264_2pass_vbv.7.diff
x264_hrd_pulldown.04_interlace.diff
x264_fix_win_stdin.diff
x264_fgo.01.826.diff
32x32samples_crash.diff
cosmetic.diff
debug-defines.diff
fix_stats_file_work_for_cli.diff
frames_memoryleak.diff
multithreading_Nth_pass_ratecontrol.diff




both are make fprofiled by GCC 4.4.0 20080331 experimental

WOO...nice...can not wait to try.....
But someone real shd be pay attention to the red characters before da final judgement

[Sharktooth]

bobor, any chance for a fresh modified build?

[lexor]

Originally Posted by Sharktooth

bobor, any chance for a fresh modified build?

fresh? x264.nl changelog matches git, and I don't see any updated patches, so what is there to build?

[Sharktooth]

oh, sorry. i missed it.

[bob0r]

Originally Posted by Sharktooth

bobor, any chance for a fresh modified build?

Never announced but:
x264.839.modified.exe
=)

I was waiting for more patches, guess they will be with the next pengvado patching spree!

[jefrey]

hi boboir, have you an build of x264 64 bit edition?
i've found an old x264build on my hdd and it is for 64 bit systems but not much patched .

Would be very nice if you can create a 64bit build of x264

[bob0r]

I am oldschool, nothing is 64bit here

[jefrey]

damned, Would you tell me(pm) how can i add some patches to the 64 build?
i Wanna see if my 32bit build is much faster that the 64bit build on my dual core2quad cpus

[IgorC]

Can anyone please make an experimental build with following patches:
1. VAQ 2.0
2. FGO
3. me prepass
...and others

Thanks.

[Sharktooth]

x264_2pass_vbv.8.diff (taken form the x264 ML)

diff --git a/encoder/encoder.c b/encoder/encoder.c
old mode 100644
new mode 100755
index a328fdf..05dcc55
--- a/encoder/encoder.c
+++ b/encoder/encoder.c
@@ -659,6 +659,7 @@ x264_t *x264_encoder_open   ( x264_param_t *param )
           || h->param.rc.i_rc_method == X264_RC_CRF
           || h->param.b_bframe_adaptive
           || h->param.b_pre_scenecut );
+    h->frames.b_have_lowres |= (h->param.rc.b_stat_read && h->param.rc.i_vbv_buffer_size > 0);
 
     h->frames.i_last_idr = - h->param.i_keyint_max;
     h->frames.i_input    = 0;
diff --git a/encoder/ratecontrol.c b/encoder/ratecontrol.c
old mode 100644
new mode 100755
index a0e0859..7e252fd
--- a/encoder/ratecontrol.c
+++ b/encoder/ratecontrol.c
@@ -43,6 +43,7 @@ typedef struct
     int p_tex_bits;
     int misc_bits;
     uint64_t expected_bits;
+    double expected_vbv;
     float new_qscale;
     int new_qp;
     int i_count;
@@ -331,7 +332,7 @@ int x264_ratecontrol_new( x264_t *h )
         rc->rate_tolerance = 0.01;
     }
 
-    h->mb.b_variable_qp = (rc->b_vbv && !rc->b_2pass) || h->param.rc.i_aq_mode;
+    h->mb.b_variable_qp = rc->b_vbv || h->param.rc.i_aq_mode;
 
     if( rc->b_abr )
     {
@@ -851,17 +852,25 @@ double predict_row_size( x264_t *h, int y, int qp )
     return (pred_s + pred_t) / 2;
 }
 
-double predict_row_size_sum( x264_t *h, int y, int qp )
+double row_bits_so_far( x264_t *h, int y )
 {
     int i;
     double bits = 0;
     for( i = 0; i <= y; i++ )
         bits += h->fdec->i_row_bits[i];
+    return bits;
+}
+
+double predict_row_size_sum( x264_t *h, int y, int qp )
+{
+    int i;
+    double bits = row_bits_so_far(h, y);
     for( i = y+1; i < h->sps->i_mb_height; i++ )
         bits += predict_row_size( h, i, qp );
     return bits;
 }
 
+
 void x264_ratecontrol_mb( x264_t *h, int bits )
 {
     x264_ratecontrol_t *rc = h->rc;
@@ -873,7 +882,7 @@ void x264_ratecontrol_mb( x264_t *h, int bits )
     rc->qpa_rc += rc->qpm;
     rc->qpa_aq += h->mb.i_qp;
 
-    if( h->mb.i_mb_x != h->sps->i_mb_width - 1 || !rc->b_vbv || rc->b_2pass )
+    if( h->mb.i_mb_x != h->sps->i_mb_width - 1 || !rc->b_vbv)
         return;
 
     h->fdec->i_row_qp[y] = rc->qpm;
@@ -883,9 +892,8 @@ void x264_ratecontrol_mb( x264_t *h, int bits )
         /* B-frames shouldn't use lower QP than their reference frames */
         if( y < h->sps->i_mb_height-1 )
         {
-            rc->qpm = X264_MAX( rc->qp,
-                      X264_MIN( h->fref0[0]->i_row_qp[y+1],
-                                h->fref1[0]->i_row_qp[y+1] ));
+            int avg_qp = (h->fref0[0]->i_row_qp[y+1]+h->fref1[0]->i_row_qp[y+1])*0.5+rc->pb_offset * ((h->fenc->i_type == X264_TYPE_BREF) ? 0.5 : 1);
+            rc->qpm = X264_MIN(X264_MAX( rc->qp, avg_qp), 51); //avg_qp could go higher than 51 due to pb_offset
         }
     }
     else
@@ -901,20 +909,42 @@ void x264_ratecontrol_mb( x264_t *h, int bits )
             int i_qp_max = X264_MIN( prev_row_qp + h->param.rc.i_qp_step, h->param.rc.i_qp_max );
             int i_qp_min = X264_MAX( prev_row_qp - h->param.rc.i_qp_step, h->param.rc.i_qp_min );
             float buffer_left_planned = rc->buffer_fill - rc->frame_size_planned;
+            float rc_tol = 1;
+            float headroom = 0;
+
+           /* Don't modify the row QPs until a sufficent amount of the bits of the frame have been processed, in case a flat */
+           /* area at the top of the frame was measured inaccurately. */
+            if(row_bits_so_far(h,y) < 0.05 * rc->frame_size_planned)
+            {
+                return;
+            }
+ 
+            headroom = buffer_left_planned/rc->buffer_size;
+            if(h->sh.i_type != SLICE_TYPE_I)
+                headroom /= 2;
+            rc_tol += headroom;
 
             if( !rc->b_vbv_min_rate )
                 i_qp_min = X264_MAX( i_qp_min, h->sh.i_qp );
 
             while( rc->qpm < i_qp_max
-                   && (b1 > rc->frame_size_planned * 1.15
+                   && (b1 > rc->frame_size_planned * rc_tol
                     || (rc->buffer_fill - b1 < buffer_left_planned * 0.5)))
             {
                 rc->qpm ++;
                 b1 = predict_row_size_sum( h, y, rc->qpm );
             }
 
+            /* avoid VBV underflow */
+            while( (rc->qpm < h->param.rc.i_qp_max)
+                   && (rc->buffer_fill - b1 < rc->buffer_size * 0.005))
+            {
+                rc->qpm ++;
+                b1 = predict_row_size_sum( h, y, rc->qpm );
+            }
+
             while( rc->qpm > i_qp_min
-                   && buffer_left_planned > rc->buffer_size * 0.4
+                   && ((buffer_left_planned > rc->buffer_size * 0.4) || rc->qpm > h->fdec->i_row_qp[0])
                    && ((b1 < rc->frame_size_planned * 0.8 && rc->qpm <= prev_row_qp)
                      || b1 < (rc->buffer_fill - rc->buffer_size + rc->buffer_rate) * 1.1) )
             {
@@ -1249,7 +1279,7 @@ static void update_vbv( x264_t *h, int bits )
         return;
 
     rct->buffer_fill_final += rct->buffer_rate - bits;
-    if( rct->buffer_fill_final < 0 && !rct->b_2pass )
+    if( rct->buffer_fill_final < 0 )
         x264_log( h, X264_LOG_WARNING, "VBV underflow (%.0f bits)\n", rct->buffer_fill_final );
     rct->buffer_fill_final = x264_clip3f( rct->buffer_fill_final, 0, rct->buffer_size );
 }
@@ -1425,6 +1455,23 @@ static float rate_estimate_qscale( x264_t *h )
                 double w = x264_clip3f( time*100, 0.0, 1.0 );
                 q *= pow( (double)total_bits / rcc->expected_bits_sum, w );
             }
+            if( rcc->b_vbv )
+            {
+                double expected_size = qscale2bits(&rce, q);
+                double expected_vbv = rcc->buffer_fill + rcc->buffer_rate - expected_size;
+                if( (expected_vbv < rcc->buffer_size*.4) && (expected_vbv < rce.expected_vbv) )
+                {
+                    double qmax = (expected_vbv < rcc->buffer_size*.15) ? lmax : q*1.5;
+                    double size_constraint = 1 + rce.expected_vbv/rcc->buffer_size;
+                    while( (expected_vbv < rce.expected_vbv/size_constraint) && (q < qmax) )
+                    {
+                        q *= 1.05;
+                        expected_size = qscale2bits(&rce, q);
+                        expected_vbv = rcc->buffer_fill + rcc->buffer_rate - expected_size;
+                    }
+                }
+                rcc->last_satd = x264_rc_analyse_slice( h );
+            }
             q = x264_clip3f( q, lmin, lmax );
         }
         else /* 1pass ABR */
@@ -1509,10 +1556,13 @@ static float rate_estimate_qscale( x264_t *h )
         rcc->last_qscale_for[pict_type] =
         rcc->last_qscale = q;
 
-        if( !rcc->b_2pass && h->fenc->i_frame == 0 )
+        if( !(rcc->b_2pass && !rcc->b_vbv) && h->fenc->i_frame == 0 )
             rcc->last_qscale_for[SLICE_TYPE_P] = q;
 
-        rcc->frame_size_planned = predict_size( &rcc->pred[h->sh.i_type], q, rcc->last_satd );
+        if( rcc->b_2pass && rcc->b_vbv)
+            rcc->frame_size_planned = qscale2bits(&rce, q);
+        else
+            rcc->frame_size_planned = predict_size( &rcc->pred[h->sh.i_type], q, rcc->last_satd );
         return q;
     }
 }
@@ -1555,6 +1605,133 @@ void x264_thread_sync_ratecontrol( x264_t *cur, x264_t *prev, x264_t *next )
     /* the rest of the variables are either constant or thread-local */
 }
 
+static int find_underflow( x264_t *h, double *fills, int *t0, int *t1, int over )
+{
+    /* find an interval ending on an overflow or underflow (depending on whether
+     * we're adding or removing bits), and starting on the earliest frame that
+     * can influence the buffer fill of that end frame. */
+    x264_ratecontrol_t *rcc = h->rc;
+    const double buffer_min = (over ? .1 : .1) * rcc->buffer_size;
+    const double buffer_max = .9 * rcc->buffer_size;
+    double fill = fills[*t0-1];
+    double parity = over ? 1. : -1.;
+    int i, start=-1, end=-1;
+    for(i=*t0; i<rcc->num_entries; i++)
+    {
+        fill += (rcc->buffer_rate - qscale2bits(&rcc->entry[i], rcc->entry[i].new_qscale)) * parity;
+        fill = x264_clip3f(fill, 0, rcc->buffer_size);
+        fills[i] = fill;
+        if(fill <= buffer_min || i == 0)
+        {
+            if(end >= 0)
+                break;
+            start = i;
+        }
+        else if(fill >= buffer_max && start >= 0)
+            end = i;
+    }
+    *t0 = start;
+    *t1 = end;
+    return start>=0 && end>=0;
+}
+
+static int fix_underflow( x264_t *h, int t0, int t1, double adjustment, double qscale_min, double qscale_max)
+{
+    x264_ratecontrol_t *rcc = h->rc;
+    double qscale_orig, qscale_new;
+    int i;
+    int adjusted = 0;
+    if(t0 > 0)
+        t0++;
+    for(i=t0; i<=t1; i++) {
+        qscale_orig = rcc->entry[i].new_qscale;
+        qscale_new =  qscale_orig * adjustment;
+        qscale_new = x264_clip3f(qscale_new, qscale_min, qscale_max);
+        rcc->entry[i].new_qscale  = qscale_new;
+        adjusted = adjusted || (qscale_new != qscale_orig);
+    }
+    return adjusted;
+}
+
+static double count_expected_bits( x264_t *h )
+{
+    x264_ratecontrol_t *rcc = h->rc;
+    double expected_bits = 0;
+    int i;
+    for(i=0; i<rcc->num_entries; i++)
+    {
+        ratecontrol_entry_t *rce = &rcc->entry[i];
+        rce->expected_bits = expected_bits;
+        expected_bits += qscale2bits(rce, rce->new_qscale);
+    }
+    return expected_bits;
+}
+
+static void vbv_pass2( x264_t *h )
+{
+    /* foreach interval of buffer_full .. underflow
+     *   uniformly increase the qp of all frames in the interval until either
+     *     buffer is full at some intermediate frame
+     *     or the last frame in the interval no longer underflows
+     * recompute intervals and repeat
+     * then do the converse to put bits back into overflow areas until target size is met */
+
+    x264_ratecontrol_t *rcc = h->rc;
+    double *fills = x264_malloc((rcc->num_entries+1)*sizeof(double));
+    double all_available_bits = h->param.rc.i_bitrate * 1000. * rcc->num_entries / rcc->fps;
+    double expected_bits = 0;
+    double adjustment;
+    double prev_bits = 0;
+    int i, t0, t1;
+    double qscale_min = qp2qscale(h->param.rc.i_qp_min);
+    double qscale_max = qp2qscale(h->param.rc.i_qp_max);
+    int iterations = 0;
+    int adj_min, adj_max;
+
+    fills++;
+
+    //adjust overall stream size
+    do {
+        iterations++;
+        prev_bits = expected_bits;
+
+        if (expected_bits != 0) { //not first iteration
+            adjustment = X264_MAX(X264_MIN(expected_bits / all_available_bits, 0.999), 0.9);
+            fills[-1] = rcc->buffer_size * h->param.rc.f_vbv_buffer_init;
+            t0 = 0;
+            //fix overflows
+            adj_min = 1;
+            while(adj_min && find_underflow(h, fills, &t0, &t1, 1))
+            {
+                adj_min = fix_underflow(h, t0, t1, adjustment, qscale_min, qscale_max);
+                t0 = t1;
+            }
+        }
+
+        fills[-1] = rcc->buffer_size * (1. - h->param.rc.f_vbv_buffer_init);
+        t0 = 0;
+        //fix underflows - should be done after overflow, as we'd better undersize target than underflowing VBV
+        adj_max = 1;
+        while(adj_max && find_underflow(h, fills, &t0, &t1, 0))
+        {
+            adj_max = fix_underflow(h, t0, t1, 1.001, qscale_min, qscale_max);
+        }
+
+        expected_bits = count_expected_bits(h);
+    } while(expected_bits < .995*all_available_bits && expected_bits > prev_bits);
+
+    if (!adj_max)
+        x264_log( h, X264_LOG_WARNING, "vbv-maxrate issue, qpmax or vbv-maxrate too low\n");
+
+    //store expected vbv filling values for tracking when encoding
+    for(i=0; i<rcc->num_entries; i++)
+        rcc->entry[i].expected_vbv = rcc->buffer_size - fills[i];
+
+//    x264_log( h, X264_LOG_INFO, "VBV RC initial iterations: %d \n", iterations);
+
+    x264_free(fills-1);
+}
+
 static int init_pass2( x264_t *h )
 {
     x264_ratecontrol_t *rcc = h->rc;
@@ -1643,7 +1820,6 @@ static int init_pass2( x264_t *h )
         rcc->last_non_b_pict_type = -1;
         rcc->last_accum_p_norm = 1;
         rcc->accum_p_norm = 0;
-        rcc->buffer_fill = rcc->buffer_size * h->param.rc.f_vbv_buffer_init;
 
         /* find qscale */
         for(i=0; i<rcc->num_entries; i++){
@@ -1680,18 +1856,11 @@ static int init_pass2( x264_t *h )
         /* find expected bits */
         for(i=0; i<rcc->num_entries; i++){
             ratecontrol_entry_t *rce = &rcc->entry[i];
-            double bits;
             rce->new_qscale = clip_qscale(h, rce->pict_type, blurred_qscale[i]);
             assert(rce->new_qscale >= 0);
-            bits = qscale2bits(rce, rce->new_qscale);
-
-            rce->expected_bits = expected_bits;
-            expected_bits += bits;
-            update_vbv(h, bits);
-            rcc->buffer_fill = rcc->buffer_fill_final;
+            expected_bits += qscale2bits(rce, rce->new_qscale);
         }
 
-//printf("expected:%llu available:%llu factor:%lf avgQ:%lf\n", (uint64_t)expected_bits, all_available_bits, rate_factor);
         if(expected_bits > all_available_bits) rate_factor -= step;
     }
 
@@ -1699,6 +1868,10 @@ static int init_pass2( x264_t *h )
     if(filter_size > 1)
         x264_free(blurred_qscale);
 
+    if(rcc->b_vbv)
+        vbv_pass2(h);
+    expected_bits = count_expected_bits(h);
+
     if(fabs(expected_bits/all_available_bits - 1.0) > 0.01)
     {
         double avgq = 0;
@@ -1706,7 +1879,8 @@ static int init_pass2( x264_t *h )
             avgq += rcc->entry[i].new_qscale;
         avgq = qscale2qp(avgq / rcc->num_entries);
 
-        x264_log(h, X264_LOG_WARNING, "Error: 2pass curve failed to converge\n");
+        if ((expected_bits > all_available_bits) || (!rcc->b_vbv))
+            x264_log(h, X264_LOG_WARNING, "Error: 2pass curve failed to converge\n");
         x264_log(h, X264_LOG_WARNING, "target: %.2f kbit/s, expected: %.2f kbit/s, avg QP: %.4f\n",
                  (float)h->param.rc.i_bitrate,
                  expected_bits * rcc->fps / (rcc->num_entries * 1000.),
@@ -1725,7 +1899,7 @@ static int init_pass2( x264_t *h )
             else
                 x264_log(h, X264_LOG_WARNING, "try increasing target bitrate\n");
         }
-        else
+        else if(!(rcc->b_2pass && rcc->b_vbv))
             x264_log(h, X264_LOG_WARNING, "internal error\n");
     }