Jekyll2023-07-02T18:51:29+00:00https://andsobegins.com/feed.xmlAnd so begins…A supplementary blogZram Setup2020-11-20T12:58:00+00:002020-11-20T12:58:00+00:00https://andsobegins.com/article/2020/11/20/zram-setup<p>People have a habit of over-specifying parameters, zram is particularly bad for it, but really, you don’t need to tell it a lot and it will take care of it for you.</p>
<p>The other misunderstanding with zram is that when you set the size you are setting how big the swap partition itself is, rather than how much memory it is allowed to use. You can’t really set how much memory to devote to zram as that depends on the compression ratio.</p>
<p>For example, setting up zram like tmpfs with half the size of your memory, will in tmpfs’ case mean that it will use up to half your memory. With zram it means you’re limiting the swap used to half your memory, if you get a compression ratio of 3:1 then you’ll be using 1/6 of your memory for swapping. Which likely isn’t what you want. Instead, you should set your zram up so it can compress much more of your memory than that.</p>
<p>I would recommend setting up zram at two to three times your memory size instead as there isn’t a lot of point in going bigger than your compression ratio of your memory.</p>
<p>In my case, performance was more important than compressibility, so I’m using the lz4 algorithm - and because that’s also not going to net you higher compression ratios, I stick to two times my memory size.</p>
<p>An extreme example of zram usage involves compiling chromium.</p>
<p>An average compression ration of just over 3 was attained with a maximum used swap of 15GiB for an 8-thread Chromium (86.0.4240.193) compile so it works in practice and under extreme conditions. The computer did slow down during the process, but not nearly as badly as with disk-based swap of course.</p>
<p>Personally, I only use a single zram for swap as that will improve the system as a whole. A tmpfs /tmp directory will swap out when pressured and I keep /var/tmp as disk-based because I’d have to write exceptions for big packages and there’s only a negligible advantage to putting it into memory over letting the kernel cache it.</p>
<p>You need the module loading at start-up and a service file (you can of course adapt this for other init systems).</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#/etc/modules-load.d/zram.conf</span>
zram
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># /etc/systemd/system/zram-setup.service</span>
<span class="o">[</span>Unit]
<span class="nv">Description</span><span class="o">=</span>Setup zram
<span class="o">[</span>Service]
<span class="nv">Type</span><span class="o">=</span>oneshot
<span class="nv">RemainAfterExit</span><span class="o">=</span><span class="nb">true
</span><span class="nv">ExecStart</span><span class="o">=</span>/sbin/zramctl <span class="nt">-a</span> lz4 <span class="nt">-s</span> 16G /dev/zram0
<span class="nv">ExecStart</span><span class="o">=</span>/sbin/mkswap <span class="nt">-L</span> Swap /dev/zram0
<span class="nv">ExecStart</span><span class="o">=</span>/sbin/swapon <span class="nt">-L</span> Swap <span class="nt">-p</span> 100
<span class="nv">ExecStop</span><span class="o">=</span>/sbin/swapoff /dev/zram0
<span class="o">[</span>Install]
<span class="nv">WantedBy</span><span class="o">=</span>multi-user.target
</code></pre></div></div>PeterPeople have a habit of over-specifying parameters, zram is particularly bad for it, but really, you don’t need to tell it a lot and it will take care of it for you.VMAF comparisons for 2160 upscaled Content on YouTube2020-09-22T11:50:00+00:002020-09-22T11:50:00+00:00https://andsobegins.com/article/2020/09/22/VMAF-comparisons-for-2160-upscaled-Content-on-YouTube<ul id="markdown-toc">
<li><a href="#introduction" id="markdown-toc-introduction">Introduction</a></li>
<li><a href="#capture" id="markdown-toc-capture">Capture</a></li>
<li><a href="#quality" id="markdown-toc-quality">Quality</a></li>
<li><a href="#2160-transcoding" id="markdown-toc-2160-transcoding">2160 Transcoding</a></li>
<li><a href="#how-did-1080-go" id="markdown-toc-how-did-1080-go">How did 1080 go?</a></li>
<li><a href="#2160-vmaf-and-bitrates" id="markdown-toc-2160-vmaf-and-bitrates">2160 vmaf and bitrates</a></li>
<li><a href="#youtube-downscaling" id="markdown-toc-youtube-downscaling">YouTube downscaling</a></li>
<li><a href="#2160-1440-1080-vmaf" id="markdown-toc-2160-1440-1080-vmaf">2160, 1440, 1080 vmaf</a></li>
<li><a href="#conclusion" id="markdown-toc-conclusion">Conclusion</a></li>
<li><a href="#more-to-do" id="markdown-toc-more-to-do">More to do</a></li>
<li><a href="#appendix-a-tools" id="markdown-toc-appendix-a-tools">Appendix A Tools</a></li>
<li><a href="#appendix-z-notes" id="markdown-toc-appendix-z-notes">Appendix Z Notes</a></li>
</ul>
<h1 id="introduction">Introduction</h1>
<p>The purpose of these tests was to disprove the notion that uploading higher than 1080 content would give better quality results for the 1080 version of an upload than just uploading at 1080 by showing the vp9 version where supported instead of the h264 version. Turned out it does actually work. The turning point for me was the observation that some channel’s videos show you the vp9 version and some the h264, yet Youtube always encodes to both. Since I did a bunch of vmaf comparisons of video for the previous article, <a href="/article/2020/03/08/encoding-for-youtube.html">Encoding for Youtube</a> - I thought I would do something similar for 2160 upscales of the same source material.</p>
<h1 id="capture">Capture</h1>
<p>Nine games were chosen, from slow-paced 30fps games with flat block-colour menus to a 60fps colourful racing game as well as some games I thought would be a challenge for both the encoders and the quality assessment.</p>
<ul>
<li>Death Stranding</li>
<li>Detroit: Become Human</li>
<li>Diablo 3</li>
<li>Flower</li>
<li>Forza Horizon 4</li>
<li>Untitled Goose Game</li>
<li>Persona 4</li>
<li>Resident Evil 7</li>
<li>Rogue Legacy</li>
</ul>
<p>These representative 3-minute clips will be referred to by shorthand names: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue.</p>
<p>Capturing is further detailed in the <a href="/article/2020/03/08/encoding-for-youtube.html">previous article</a>. Whilst transcoding is lossless, the initial captures are not true lossless for practicality’s sake as they are limited range, yuv420p and were in 60000/1001fps.</p>
<h1 id="quality">Quality</h1>
<p>VMAF Blurb: VMAF (Video Multi-Method Assessment Fusion) is a perceptual video quality assessment algorithm developed by Netflix. VMAF Development Kit (VDK) is a software package that contains the VMAF algorithm implementation, as well as a set of tools that allows a user to train and test a custom VMAF model. Read <a href="https://medium.com/netflix-techblog/toward-a-practical-perceptual-video-quality-metric-653f208b9652">this</a> techblog post for an overview, or <a href="https://medium.com/netflix-techblog/vmaf-the-journey-continues-44b51ee9ed12">this</a> post for the latest updates and tips for best practices.</p>
<p>Basically, vmaf is a rating for comparing a distorted video to its original and telling you how close it is. It is an objective measure of how close the video matches, one that takes many methods into account and is specifically for video. The scale itself is more subjective and based upon how real humans would rate the content.</p>
<h1 id="2160-transcoding">2160 Transcoding</h1>
<p>I thought it would be interesting to upscale with (ffmpeg default is bicubic) and without filters (nearest neighbour). The x265 encoder was used at the slow preset at crf 18 (constant rate factor, going from 0, lossless, to 51, awful). The last set of tests showed that using lossless doesn’t really affect the final vmaf rating from Youtube, so this time only the crf version was used, and is more feasible to upload outside of testing. The slow preset is just so it encodes in a reasonable time across the network.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>parallel <span class="nt">--nice</span> 20 <span class="nt">--eta</span> <span class="nt">-S</span> node1,node2,node3 <span class="nt">-j1</span> ffmpeg <span class="nt">-i</span> <span class="o">{}</span>_ll.mkv <span class="nt">-crf</span> 18 <span class="nt">-c</span> libx265 <span class="nt">-preset</span> slow <span class="nt">-vf</span> <span class="nv">scale</span><span class="o">=</span>3840x2160:flags<span class="o">=</span>bicubic <span class="o">{}</span>_265_s_18_4k.mkv ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue
parallel <span class="nt">--nice</span> 20 <span class="nt">--eta</span> <span class="nt">-S</span> node1,node2,node3 <span class="nt">-j1</span> ffmpeg <span class="nt">-i</span> <span class="o">{}</span>_ll.mkv <span class="nt">-crf</span> 18 <span class="nt">-c</span> libx265 <span class="nt">-preset</span> slow <span class="nt">-vf</span> <span class="nv">scale</span><span class="o">=</span>3840x2160:flags<span class="o">=</span>neighbor <span class="o">{}</span>_265_s_18_4kn.mkv ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue
</code></pre></div></div>
<p>The naming convention is just the game name, video type (265), crf value (18) and 4k for default upscaling (bicubic) and 4kn for upscaling with none or (nearest) neighbor.</p>
<p>Despite working with upscaled videos we can still perform vmaf comparisons by simply running the reference lossless videos through the same upscale filter when comparing. We even have a new comparison to make since vmaf provides a 4k (2160) model (vmaf_4k_v0.6.1) as well as the usual 1080 (vmaf_v0.6.1) one.</p>
<blockquote>
<p>Most recently, we added a new 4K VMAF model which predicts the subjective quality of video displayed on a 4K TV and viewed from a distance of 1.5H. A viewing distance of 1.5H is the maximum distance for the average viewer to appreciate the sharpness of 4K content. The 4K model is similar to the default model in the sense that both models capture quality at the critical angular frequency of 1/60 degree/pixel. However, the 4K model assumes a wider viewing angle, which affects the foveal vs peripheral vision that the subject uses.</p>
</blockquote>
<p><em><a href="https://netflixtechblog.com/vmaf-the-journey-continues-44b51ee9ed12">VMAF: The Journey Continues</a></em></p>
<p>To give a little context, H is height of device, and for the default model, it’s a 1080 display viewed at 3H.</p>
<h1 id="how-did-1080-go">How did 1080 go?</h1>
<table>
<thead>
<tr>
<th>Name</th>
<th>Transcode VMAF</th>
<th>Transcode Rate (kb/s)</th>
<th>YT.264 VMAF</th>
<th>YT.264 Rate (kb/s)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Death</td>
<td>99.023438</td>
<td>13962</td>
<td>57.284986</td>
<td>4242</td>
</tr>
<tr>
<td>Detroit</td>
<td>97.94216</td>
<td>6447</td>
<td>67.874461</td>
<td>3490</td>
</tr>
<tr>
<td>Diablo</td>
<td>96.761145</td>
<td>10799</td>
<td>64.274843</td>
<td>5477</td>
</tr>
<tr>
<td>Flower</td>
<td>96.238418</td>
<td>11387</td>
<td>61.241824</td>
<td>5118</td>
</tr>
<tr>
<td>Forza</td>
<td>97.810374</td>
<td>26326</td>
<td>52.031377</td>
<td>5401</td>
</tr>
<tr>
<td>Goose</td>
<td>96.301997</td>
<td>1695</td>
<td>75.384589</td>
<td>2565</td>
</tr>
<tr>
<td>Persona</td>
<td>97.644543</td>
<td>3768</td>
<td>84.632011</td>
<td>2517</td>
</tr>
<tr>
<td>RE7</td>
<td>96.152931</td>
<td>5012</td>
<td>64.376164</td>
<td>4227</td>
</tr>
<tr>
<td>Rogue</td>
<td>97.675072</td>
<td>3067</td>
<td>88.606968</td>
<td>5037</td>
</tr>
</tbody>
</table>
<p><img src="https://andsobegins.com/assets/images/1080Transcode_Upload.svg" alt="1080 Transcode & Upload" title="1080 Transcode & Upload" /><br />
<em>1080 Transcode & Upload</em></p>
<p>These are the realistic results from last time, I do use this encode to produce video, although I don’t start with lossless, but quicksync’s highest quality. This is what happens when you transcode from lossless with x265, crf18, using the slow preset at 1080p and then upload to Youtube.</p>
<p>The videos you are presented with when you watch Youtube are in h264. The vp9 versions are generated, but not shown. As expected they are both lower bitrate and higher quality which is why vp9 is so desirable (and begs the question of why google doesn’t always show the vp9 version of every video where supported).</p>
<h1 id="2160-vmaf-and-bitrates">2160 vmaf and bitrates</h1>
<p>Let’s see how the 2160 versions turned out.</p>
<p>Here’s how they we got the vmaf numbers from the transcodes (4k & 4kn) and from the downloaded youtube files (using youtube-dl) respectively.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>parallel <span class="nt">--nice</span> 20 <span class="nt">--eta</span> <span class="nt">-j1</span> <span class="s1">'ffmpeg -i /mnt/LPWorking/vmaf/{}_265_s_18_4k.mkv -i /mnt/LPWorking/vmaf/{}_ll.mkv -filter_complex "[1]scale=3840x2160:flags=bicubic[ref];[0][ref]libvmaf=model_path=/usr/share/model/vmaf_4k_v0.6.1.pkl" -f null -t 180 /dev/null'</span> ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue &> /mnt/LPWorking/vmaf/vmafscores_2160-2160.txt
parallel <span class="nt">--nice</span> 20 <span class="nt">--eta</span> <span class="nt">-j1</span> <span class="s1">'ffmpeg -i /mnt/LPWorking/vmaf/{}_265_s_18_4kn.mkv -i /mnt/LPWorking/vmaf/{}_ll.mkv -filter_complex "[1]scale=3840x2160:flags=neighbor[ref];[0][ref]libvmaf=model_path=/usr/share/model/vmaf_4k_v0.6.1.pkl" -f null -t 180 /dev/null'</span> ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue &> /mnt/LPWorking/vmaf/vmafscores_2160-2160n.txt
parallel <span class="nt">--nice</span> 20 <span class="nt">--eta</span> <span class="nt">-j1</span> <span class="s1">'ffmpeg -i /mnt/LPWorking/vmaf/yt/{}_265_s_18_4k-2160.webm -i /mnt/LPWorking/vmaf/{}_ll.mkv -filter_complex "[1]scale=3840x2160:flags=bicubic[ref];[0][ref]libvmaf=model_path=/usr/share/model/vmaf_4k_v0.6.1.pkl" -f null -t 180 /dev/null'</span> ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue &> /mnt/LPWorking/vmaf/yt/vmafscores_4k_vp9-2160-2160.txt
parallel <span class="nt">--nice</span> 20 <span class="nt">--eta</span> <span class="nt">-j1</span> <span class="s1">'ffmpeg -i /mnt/LPWorking/vmaf/yt/{}_265_s_18_4kn-2160.webm -i /mnt/LPWorking/vmaf/{}_ll.mkv -filter_complex "[1]scale=3840x2160:flags=neighbor[ref];[0][ref]libvmaf=model_path=/usr/share/model/vmaf_4k_v0.6.1.pkl" -f null -t 180 /dev/null'</span> ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue &> /mnt/LPWorking/vmaf/yt/vmafscores_4k_vp9-2160-2160n.txt
</code></pre></div></div>
<table>
<thead>
<tr>
<th>Name</th>
<th>2160-2160 VMAF</th>
<th>Trans Rate (kb/s)</th>
<th>2160-2160 YT.vp9 VMAF</th>
<th>2160-2160 YT.vp9 Rate (kb/s)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Death_265_s_18_4k</td>
<td>99.631391</td>
<td>34757</td>
<td>88.30421</td>
<td>13366</td>
</tr>
<tr>
<td>Death_265_s_18_4kn</td>
<td>99.092667</td>
<td>40346</td>
<td>87.80756</td>
<td>13542</td>
</tr>
<tr>
<td>Detroit_265_s_18_4k</td>
<td>99.06393</td>
<td>28424</td>
<td>92.226379</td>
<td>9982</td>
</tr>
<tr>
<td>Detroit_265_s_18_4kn</td>
<td>98.51933</td>
<td>26449</td>
<td>91.717363</td>
<td>10317</td>
</tr>
<tr>
<td>Diablo_265_s_18_4k</td>
<td>99.396409</td>
<td>26308</td>
<td>93.658602</td>
<td>17674</td>
</tr>
<tr>
<td>Diablo_265_s_18_4kn</td>
<td>99.037009</td>
<td>30518</td>
<td>93.195276</td>
<td>18916</td>
</tr>
<tr>
<td>Flower_265_s_18_4k</td>
<td>97.879916</td>
<td>30788</td>
<td>92.673437</td>
<td>17500</td>
</tr>
<tr>
<td>Flower_265_s_18_4kn</td>
<td>96.749556</td>
<td>33642</td>
<td>91.738545</td>
<td>17982</td>
</tr>
<tr>
<td>Forza_265_s_18_4k</td>
<td>99.154993</td>
<td>61127</td>
<td>86.989702</td>
<td>22880</td>
</tr>
<tr>
<td>Forza_265_s_18_4kn</td>
<td>98.742877</td>
<td>69084</td>
<td>86.440331</td>
<td>23332</td>
</tr>
<tr>
<td>Goose_265_s_18_4k</td>
<td>98.723693</td>
<td>4099</td>
<td>93.148316</td>
<td>3544</td>
</tr>
<tr>
<td>Goose_265_s_18_4kn</td>
<td>98.46808</td>
<td>6880</td>
<td>93.093757</td>
<td>5599</td>
</tr>
<tr>
<td>Persona_265_s_18_4k</td>
<td>99.781146</td>
<td>9372</td>
<td>96.129109</td>
<td>5695</td>
</tr>
<tr>
<td>Persona_265_s_18_4kn</td>
<td>99.755002</td>
<td>11047</td>
<td>96.165361</td>
<td>6529</td>
</tr>
<tr>
<td>RE7_265_s_18_4k</td>
<td>98.350062</td>
<td>14046</td>
<td>90.456835</td>
<td>9482</td>
</tr>
<tr>
<td>RE7_265_s_18_4kn</td>
<td>97.564546</td>
<td>16557</td>
<td>89.861327</td>
<td>10323</td>
</tr>
<tr>
<td>Rogue_265_s_18_4k</td>
<td>99.816733</td>
<td>6971</td>
<td>99.145447</td>
<td>6291</td>
</tr>
<tr>
<td>Rogue_265_s_18_4kn</td>
<td>99.837137</td>
<td>11038</td>
<td>99.074572</td>
<td>8689</td>
</tr>
</tbody>
</table>
<p><img src="https://andsobegins.com/assets/images/4kTranscode_Upload.svg" alt="4k Transcode & Upload" title="4k Transcode & Upload" /><br />
<em>4k Transcode & Upload</em></p>
<p>The transcodes this time have higher vmaf scores, with big increases in bitrate. The increase in bitrate I was expecting - we are storing quadruple resolution video after all, although I was surprised at the unfiltered upscaling, expecting it to make uniformly smaller files rather than usually bigger.</p>
<p>When you ignore the transcoded size and just look at the default 1080 versus the default 2160 and also add an average the difference in 1080 and upscaled 2160 on Youtube becomes more apparent.</p>
<table>
<thead>
<tr>
<th>Game</th>
<th>4k2160 to vp9 vmaf</th>
<th>4k2160 to vp9 rate</th>
<th>1080 to 264 vmaf</th>
<th>1080 to 264 rate</th>
</tr>
</thead>
<tbody>
<tr>
<td>Death</td>
<td>88.922144</td>
<td>13366</td>
<td>57.284986</td>
<td>4242</td>
</tr>
<tr>
<td>Detroit</td>
<td>91.718805</td>
<td>9982</td>
<td>67.874461</td>
<td>3490</td>
</tr>
<tr>
<td>Diablo</td>
<td>91.723392</td>
<td>17674</td>
<td>64.274843</td>
<td>5477</td>
</tr>
<tr>
<td>Flower</td>
<td>96.413496</td>
<td>17500</td>
<td>61.241824</td>
<td>5118</td>
</tr>
<tr>
<td>Forza</td>
<td>88.858607</td>
<td>22880</td>
<td>52.031377</td>
<td>5401</td>
</tr>
<tr>
<td>Goose</td>
<td>91.507133</td>
<td>3544</td>
<td>75.384589</td>
<td>2565</td>
</tr>
<tr>
<td>Persona</td>
<td>94.266461</td>
<td>5695</td>
<td>84.632011</td>
<td>2517</td>
</tr>
<tr>
<td>RE7</td>
<td>89.171265</td>
<td>9482</td>
<td>64.376164</td>
<td>4227</td>
</tr>
<tr>
<td>Rogue</td>
<td>97.913176</td>
<td>6291</td>
<td>88.606968</td>
<td>5037</td>
</tr>
<tr>
<td>Average</td>
<td>92.27716433</td>
<td>11824</td>
<td>68.41191367</td>
<td>4230</td>
</tr>
</tbody>
</table>
<p><img src="https://andsobegins.com/assets/images/4kTranscode_1080Upload_Comparison.svg" alt="4k Transcode & 1080 Upload Comparison" title="4k Transcode & 1080 Upload Comparison" /><br />
<em>4k Transcode & 1080 Upload Comparison</em></p>
<p>2160 on YouTube doesn’t just provide enough extra bitrate to provide a similar quality to 1080 at an increased resolution, it provides enough to bump up the average vmaf score from 68 to 92, to the point that you have to look closely and know where to look to distinguish the difference between Youtube and upload in some cases.</p>
<h1 id="youtube-downscaling">YouTube downscaling</h1>
<p>When you upload a video, it isn’t available in just the resolution you upload, but also downscaled into different resolutions for viewing. To find out what kind of downscaling Youtube uses you can compare the 1080p versions of your 2160p uploads with an ffmpeg downscaled version of that same 2160p Youtube upload. No need for vmaf comparisons here, the obvious result was lanczos when placed one on top of the other. I could have used lanczos instead of the default, bicubic, for upscaling too, (in fact this has become my default approach).</p>
<p>This means when comparing 2160p content to 1080p, the comparison of vmaf numbers is a bit fairer by using the same 1080 model and emulating the same sort of downscale that Youtube itself would perform.</p>
<p>Luckily, the vmaf scores are comparable - a file upscaled from 1080 to 2160 will have a <em>very</em> similar vmaf score (under 1.7% different) in the 4k model to that same file scaled back down to 1080 in the default 1080 model.</p>
<p>More usefully, downscaling to 1080 can also be used to test youtube’s different resolution transcodes. So not only can we compare 2160 and 1080, but also 1440.</p>
<h1 id="2160-1440-1080-vmaf">2160, 1440, 1080 vmaf</h1>
<p>So here’s how the youtube video’s 1440 resolution videos were compared once downloaded via youtube-dl (other versions skipped for brevity).</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>parallel <span class="nt">--nice</span> 20 <span class="nt">--eta</span> <span class="nt">-S</span> Soma,Stardew,Portal <span class="nt">-j1</span> <span class="s1">'ffmpeg -i /mnt/LPWorking/vmaf/yt/{}_265_s_18_4k-1440.webm -i /mnt/LPWorking/vmaf/{}_ll.mkv -filter_complex "[0]scale=1920x1080:flags=lanczos[main];[1]scale=3840x2160:flags=bicubic[up];[up]scale=1920x1080:flags=lanczos[ref];[main][ref]libvmaf=model_path=/usr/share/model/vmaf_v0.6.1.pkl" -f null -t 180 /dev/null'</span> ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue &> /mnt/LPWorking/vmaf/yt/vmafscores_4k_vp9-1440-1080_scaled.txt
parallel <span class="nt">--nice</span> 20 <span class="nt">--eta</span> <span class="nt">-S</span> Soma,Stardew,Portal <span class="nt">-j1</span> <span class="s1">'ffmpeg -i /mnt/LPWorking/vmaf/yt/{}_265_s_18_4kn-1440.webm -i /mnt/LPWorking/vmaf/{}_ll.mkv -filter_complex "[0]scale=1920x1080:flags=lanczos[main];[1]scale=3840x2160:flags=neighbor[up];[up]scale=1920x1080:flags=lanczos[ref];[main][ref]libvmaf=model_path=/usr/share/model/vmaf_v0.6.1.pkl" -f null -t 180 /dev/null'</span> ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue &> /mnt/LPWorking/vmaf/yt/vmafscores_4k_vp9-1440-1080n_scaled.txt
</code></pre></div></div>
<table>
<thead>
<tr>
<th>Name</th>
<th>“1080 YT.vp9 VMAF”</th>
<th>“1080 YT.vp9 Rate (kb/s)”</th>
<th>“1440-1080 YT.vp9 VMAF”</th>
<th>“1440 YT.vp9 Rate (kb/s)”</th>
<th>“2160-1080 YT.vp9 VMAF”</th>
<th>“2160 YT.vp9 Rate (kb/s)”</th>
</tr>
</thead>
<tbody>
<tr>
<td>Death_265_s_18_4k</td>
<td>72.264176</td>
<td>2382</td>
<td>85.825568</td>
<td>7339</td>
<td>88.922144</td>
<td>13366</td>
</tr>
<tr>
<td>Death_265_s_18_4kn</td>
<td>71.148515</td>
<td>2363</td>
<td>85.365538</td>
<td>7274</td>
<td>88.329157</td>
<td>13542</td>
</tr>
<tr>
<td>Detroit_265_s_18_4k</td>
<td>86.224406</td>
<td>1549</td>
<td>91.167497</td>
<td>5199</td>
<td>91.718805</td>
<td>9982</td>
</tr>
<tr>
<td>Detroit_265_s_18_4kn</td>
<td>84.761498</td>
<td>1508</td>
<td>90.855967</td>
<td>5162</td>
<td>91.383948</td>
<td>10317</td>
</tr>
<tr>
<td>Diablo_265_s_18_4k</td>
<td>79.263833</td>
<td>3437</td>
<td>90.008917</td>
<td>10202</td>
<td>91.723392</td>
<td>17674</td>
</tr>
<tr>
<td>Diablo_265_s_18_4kn</td>
<td>77.8263</td>
<td>3411</td>
<td>89.448375</td>
<td>10274</td>
<td>91.204057</td>
<td>18916</td>
</tr>
<tr>
<td>Flower_265_s_18_4k</td>
<td>82.283623</td>
<td>3244</td>
<td>93.480704</td>
<td>9669</td>
<td>96.413496</td>
<td>17500</td>
</tr>
<tr>
<td>Flower_265_s_18_4kn</td>
<td>81.132132</td>
<td>3161</td>
<td>93.064839</td>
<td>9575</td>
<td>95.783452</td>
<td>17982</td>
</tr>
<tr>
<td>Forza_265_s_18_4k</td>
<td>70.939047</td>
<td>4176</td>
<td>83.72851</td>
<td>11200</td>
<td>88.858607</td>
<td>22880</td>
</tr>
<tr>
<td>Forza_265_s_18_4kn</td>
<td>69.784144</td>
<td>4134</td>
<td>83.251745</td>
<td>11210</td>
<td>88.021732</td>
<td>23332</td>
</tr>
<tr>
<td>Goose_265_s_18_4k</td>
<td>85.233304</td>
<td>650</td>
<td>90.33181</td>
<td>1626</td>
<td>91.507133</td>
<td>3544</td>
</tr>
<tr>
<td>Goose_265_s_18_4kn</td>
<td>84.380214</td>
<td>660</td>
<td>90.401845</td>
<td>1769</td>
<td>91.553551</td>
<td>5599</td>
</tr>
<tr>
<td>Persona_265_s_18_4k</td>
<td>90.565604</td>
<td>1228</td>
<td>93.979172</td>
<td>3427</td>
<td>94.266461</td>
<td>5695</td>
</tr>
<tr>
<td>Persona_265_s_18_4kn</td>
<td>89.361618</td>
<td>1246</td>
<td>93.895276</td>
<td>3600</td>
<td>94.260319</td>
<td>6529</td>
</tr>
<tr>
<td>RE7_265_s_18_4k</td>
<td>79.695647</td>
<td>1543</td>
<td>87.706611</td>
<td>5469</td>
<td>89.171265</td>
<td>9482</td>
</tr>
<tr>
<td>RE7_265_s_18_4kn</td>
<td>78.559084</td>
<td>1508</td>
<td>87.21628</td>
<td>5485</td>
<td>88.701008</td>
<td>10323</td>
</tr>
<tr>
<td>Rogue_265_s_18_4k</td>
<td>96.160856</td>
<td>1936</td>
<td>97.730732</td>
<td>3723</td>
<td>97.913176</td>
<td>6291</td>
</tr>
<tr>
<td>Rogue_265_s_18_4kn</td>
<td>95.042382</td>
<td>1957</td>
<td>97.703936</td>
<td>4152</td>
<td>97.911519</td>
<td>8689</td>
</tr>
</tbody>
</table>
<p><img src="https://andsobegins.com/assets/images/4kUpload_Allres_Comparison.svg" alt="4k Upload Allres Comparison" title="4k Upload Allres Comparison" /><br />
<em>4k Upload Allres Comparison</em></p>
<p>As you can see, 1440p is a reasonable compromise, needing smaller files, but still producing significantly higher vmaf scores at 1080 and 1440 than 1080 on its own.</p>
<h1 id="conclusion">Conclusion</h1>
<p>For new Youtube uploaders, it’s up to them to decide if the significant increase in upload size is worth the increase in fidelity at 1080p that you get from getting the vp9 to show by default, and even then, that vp9 version <em>may</em> not look as good as an established channel’s. Both new channels and established channels can benefit from the higher achievable fidelity at above 1080p resolutions. It isn’t just people with 4k hardware that benefit though, it’s trivial on PCs at least to select 2160p and have Youtube downscale the video to their display.</p>
<p>This will let you view higher quality 1080 on Youtube. It isn’t just quantisation errors that disappear though - viewing a 2160p Youtube video downscaled to your 1080p display also lets you see a video that effectively has no chroma subsampling since the yuv420p that Youtube (and everywhere else) uses quarter-resolution colour, and 2160p to 1080p is a quarter of the resolution, you get full colour resolution in 1080p.</p>
<p>Saying that, image quality isn’t everything, and it’s strange seeing Youtube embrace the very lastest in video technology, crazy resolutions like 8k, mind-numbingly expensive to encode av1, yet not implementing decades old sound technology as whilst you can upload a 5.1 surround track, you can’t listen to a video in it. The only way you can get 5.1 surround out of Youtube is using Dolby Pro Logic II for which there is no full-featured free encoder.</p>
<p>At least sound quality is a solved issue as opus@160kbps is pretty much transparent.</p>
<h1 id="more-to-do">More to do</h1>
<p>Repeat but with lanczos filter.</p>
<p>It would also be nice to re-upload the test files to a larger channel to find out what differences there are other than default codec to show a viewer. I can’t attribute the slightly higher vmaf and bitrate to the upscaling process unless I can compare to an upload that directly shows the vp9 version by default when a 1080 video is uploaded.</p>
<p>av1? 8k? sub 1080p?</p>
<h1 id="appendix-a-tools">Appendix A Tools</h1>
<p>ffmpeg-4.1.3 for transcoding.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ffmpeg version 4.1.3 Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 9.2.0 (Gentoo 9.2.0-r2 p3)
configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --docdir=/usr/share/doc/ffmpeg-4.1.3/html --mandir=/usr/share/man --enable-shared --cc=x86_64-pc-linux-gnu-gcc --cxx=x86_64-pc-linux-gnu-g++ --ar=x86_64-pc-linux-gnu-ar --optflags='-march=native -O2 -pipe' --disable-static --enable-avfilter --enable-avresample --enable-libvmaf --enable-version3 --disable-stripping --disable-optimizations --disable-libcelt --disable-indev=v4l2 --disable-outdev=v4l2 --disable-indev=oss --disable-indev=jack --disable-outdev=oss --enable-bzlib --disable-runtime-cpudetect --disable-debug --disable-gcrypt --disable-gnutls --disable-gmp --enable-gpl --enable-hardcoded-tables --enable-iconv --disable-libtls --disable-libxml2 --disable-lzma --enable-network --disable-opencl --disable-openssl --enable-postproc --disable-libsmbclient --enable-ffplay --enable-sdl2 --enable-vaapi --enable-vdpau --enable-xlib --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-zlib --disable-libcdio --disable-libiec61883 --disable-libdc1394 --disable-libcaca --disable-openal --enable-opengl --disable-libv4l2 --enable-libpulse --disable-libdrm --disable-libjack --disable-libopencore-amrwb --disable-libopencore-amrnb --disable-libcodec2 --disable-libfdk-aac --disable-libopenjpeg --disable-libbluray --disable-libgme --disable-libgsm --disable-mmal --disable-libmodplug --enable-libopus --disable-libilbc --disable-librtmp --disable-libssh --disable-libspeex --disable-libsrt --enable-librsvg --disable-ffnvcodec --enable-libvorbis --disable-libvpx --disable-libzvbi --disable-appkit --disable-libbs2b --disable-chromaprint --disable-libflite --disable-frei0r --disable-libfribidi --disable-fontconfig --disable-ladspa --disable-libass --disable-lv2 --enable-libfreetype --disable-librubberband --disable-libzmq --disable-libzimg --disable-libsoxr --enable-pthreads --disable-libvo-amrwbenc --enable-libmp3lame --disable-libkvazaar --disable-libaom --disable-libopenh264 --disable-libsnappy --disable-libtheora --disable-libtwolame --enable-libwavpack --disable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --disable-armv5te --disable-armv6 --disable-armv6t2 --disable-neon --disable-vfp --disable-vfpv3 --disable-armv8 --disable-mipsdsp --disable-mipsdspr2 --disable-mipsfpu --disable-altivec --disable-amd3dnow --disable-amd3dnowext --disable-avx2 --cpu=host --disable-doc --disable-htmlpages --enable-manpages
libavutil 56. 22.100 / 56. 22.100
libavcodec 58. 35.100 / 58. 35.100
libavformat 58. 20.100 / 58. 20.100
libavdevice 58. 5.100 / 58. 5.100
libavfilter 7. 40.101 / 7. 40.101
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 3.100 / 5. 3.100
libswresample 3. 3.100 / 3. 3.100
libpostproc 55. 3.100 / 55. 3.100
</code></pre></div></div>
<p>vmaf-1.3.15 for the vmaf library.</p>
<p>gnu parallel-20191022 for pushing jobs around the network.</p>
<p>obs-24.0.3 for capturing the video from the Black Magic card.</p>
<p>youtube-dl-2020.01.24 for downloading the videos from YouTube for comparison.</p>
<h1 id="appendix-z-notes">Appendix Z Notes</h1>
<p>There is a channel dedicated to this article where you can find all the uploads featured within. <a href="https://www.youtube.com/channel/UCwXGFTOhuMJ1jQzOXRx3Fbg">https://www.youtube.com/channel/UCwXGFTOhuMJ1jQzOXRx3Fbg</a></p>
<p>Original videos are too large to share.</p>
<p>Our actual gaming channel is <a href="https://www.youtube.com/c/AndSoBegins">https://www.youtube.com/c/AndSoBegins</a></p>PeterEncoding for YouTube2020-03-08T12:16:00+00:002020-03-08T12:16:00+00:00https://andsobegins.com/article/2020/03/08/encoding-for-youtube<ul id="markdown-toc">
<li><a href="#introduction" id="markdown-toc-introduction">Introduction</a></li>
<li><a href="#capture" id="markdown-toc-capture">Capture</a></li>
<li><a href="#lossless-versions" id="markdown-toc-lossless-versions">Lossless Versions</a></li>
<li><a href="#quality" id="markdown-toc-quality">Quality</a></li>
<li><a href="#youtube-transcodes" id="markdown-toc-youtube-transcodes">YouTube Transcodes</a></li>
<li><a href="#producing-the-videos" id="markdown-toc-producing-the-videos">Producing the Videos</a></li>
<li><a href="#comparing-quality-targets-versus-bitrate-targets" id="markdown-toc-comparing-quality-targets-versus-bitrate-targets">Comparing quality targets versus bitrate targets</a></li>
<li><a href="#comparing-fixed-bitrate-methods" id="markdown-toc-comparing-fixed-bitrate-methods">Comparing Fixed-bitrate methods</a></li>
<li><a href="#comparing-crf-methods" id="markdown-toc-comparing-crf-methods">Comparing CRF methods</a></li>
<li><a href="#double-baking-crf" id="markdown-toc-double-baking-crf">Double-baking crf</a></li>
<li><a href="#what-about-4k" id="markdown-toc-what-about-4k">What about 4k?</a></li>
<li><a href="#appendix-a-tools" id="markdown-toc-appendix-a-tools">Appendix A Tools</a></li>
<li><a href="#appendix-b-creating-transcodes" id="markdown-toc-appendix-b-creating-transcodes">Appendix B Creating Transcodes</a></li>
<li><a href="#appendix-c-transcoded-vmaf-comparisons" id="markdown-toc-appendix-c-transcoded-vmaf-comparisons">Appendix C Transcoded VMAF comparisons</a></li>
<li><a href="#appendix-d-vmaf-comparisons-from-youtube" id="markdown-toc-appendix-d-vmaf-comparisons-from-youtube">Appendix D VMAF Comparisons from YouTube</a></li>
<li><a href="#appendix-x-results-tables" id="markdown-toc-appendix-x-results-tables">Appendix X Results Tables</a></li>
<li><a href="#appendix-z-notes" id="markdown-toc-appendix-z-notes">Appendix Z Notes</a></li>
</ul>
<h1 id="introduction">Introduction</h1>
<p>The purpose of these tests and this document is somewhat to justify my own settings and find out if I need to change them. However, the results are of general interest, so I thought I’d turn them into an article.</p>
<h1 id="capture">Capture</h1>
<p>Nine games were chosen, from slow-paced 30fps games with flat block-colour menus to a 60fps colourful racing game as well as some games I thought would be a challenge for both the encoders and the quality assessment. Games were chosen because they represent things YouTube gamers would actually be uploading rather than using open source movies.</p>
<p>We all love <a href="https://peach.blender.org/">Big Buck Bunny</a> and his wacky adventures, but it’s almost, but not quite, entirely unlike game footage.</p>
<ul>
<li>Death Stranding</li>
<li>Detroit: Become Human</li>
<li>Diablo 3</li>
<li>Flower</li>
<li>Forza Horizon 4</li>
<li>Untitled Goose Game</li>
<li>Persona 4</li>
<li>Resident Evil 7</li>
<li>Rogue Legacy</li>
</ul>
<p>These will be referred to by shorthand names: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue</p>
<p>The games were captured via OBS from a Black Magic Intensity Pro 4k device, all at 60000/1001 fps as that’s what the PS4 outputs in and the PC captures were set to match. NVEnc was used in lossless mode, however, the capture was set to limited / movie colour range and used nv12 (yuv420) as the pixel format, so it wasn’t true lossless, but matches the capabilities of the final destination of YouTube.</p>
<h1 id="lossless-versions">Lossless Versions</h1>
<p>After the initial capture, a three minute section that displayed the best cross-section of gameplay was chosen. These sections were stream-copied for a bitrate comparison to the original capture, and then transcoded losslessly to some working files at non-fractional framerates to make sure indexing works right and to not drop any more frames, these were then transcoded again losslessly (x264 veryslow qp 0) into three minute clips.</p>
<table>
<thead>
<tr>
<th>Game</th>
<th>Capture rate* (kb/s)</th>
<th>Transcode rate (kb/s)</th>
<th>Final Fps</th>
</tr>
</thead>
<tbody>
<tr>
<td>Death</td>
<td>307012</td>
<td>176507</td>
<td>30</td>
</tr>
<tr>
<td>Detroit</td>
<td>434886</td>
<td>217563</td>
<td>30</td>
</tr>
<tr>
<td>Diablo</td>
<td>281281</td>
<td>211467</td>
<td>60</td>
</tr>
<tr>
<td>Flower</td>
<td>317912</td>
<td>238585</td>
<td>60</td>
</tr>
<tr>
<td>Forza</td>
<td>500664</td>
<td>389333</td>
<td>60</td>
</tr>
<tr>
<td>Goose</td>
<td>241264</td>
<td>189136</td>
<td>60</td>
</tr>
<tr>
<td>Persona</td>
<td>81923</td>
<td>43857</td>
<td>30</td>
</tr>
<tr>
<td>RE7</td>
<td>298612</td>
<td>221580</td>
<td>60</td>
</tr>
<tr>
<td>Rogue</td>
<td>78416</td>
<td>33123</td>
<td>60</td>
</tr>
</tbody>
</table>
<p><img src="https://andsobegins.com/assets/images/Capture_rate_and_Transcode_rate.svg" alt="Capture rate* (kb/s) and Transcode rate (kb/s)" title="Capture rate* kb/s and Transcode rate kb/s" /><br />
<em>Capture rate* (kb/s) and Transcode rate (kb/s)</em></p>
<p>*Stream-copying works on i-frame boundaries so these figures are close, but will have up to 3 seconds of extra footage, the rate takes these extra seconds into consideration.</p>
<p>Rates here are calculated from mkv file containing only the video steam, as embedded stream rates aren’t always accurate. The downside is there’ll be a touch of overhead.</p>
<p>Two interesting facts to note.</p>
<p>The capture and transcoded files have the same video frames in them, but very different bitrates. NVEnc was running in real-time and so needed to use more bits to store the information. Transcodes were not done in real-time, but at the ‘veryslow’ preset (the slowest reasonable preset, the only slower one is called placebo for good reason). All this work was done on a variety of commodity equipment in parallel, so no transcode times are available.</p>
<p>The variation between games is <em>enormous</em>. If you compare the smallest and largest transcoded bitrates, Forza is <strong>more than 10 times</strong> bigger than Rogue. Rogue Legacy is a four-way scrolling sprite-based game and was specifically chosen as easy to encode (along with Persona).</p>
<h1 id="quality">Quality</h1>
<p>I also want to get across early on that <em>bitrate does not equal quality</em>. Here we can see wildly different bitrates all representing lossless information and thus the same quality. Bitrate comparisons are often very misleading for this reason - this is why we’re using VMAF and its 0-100 scale.</p>
<p>And you don’t care about bitrate unless you’re streaming or targeting a specific size of file.</p>
<p>You care about <em>Quality</em>, how good it looks in comparison to what you see on a screen. This is also a good demonstration though that one thing we do care about is overall filesize. If we didn’t, we’d all happily stick to a completely lossless workflow and upload enormous files, but that would take forever, and we wouldn’t be able to archive the videos. The best file is one that looks sufficiently good and is sufficiently small (and renders in a sufficiently prompt time). That will of course vary between people. In the case of this document, several different quality settings and a few recommend settings will be trialled.</p>
<p>VMAF Blurb: VMAF (Video Multi-Method Assessment Fusion) is a perceptual video quality assessment algorithm developed by Netflix. VMAF Development Kit (VDK) is a software package that contains the VMAF algorithm implementation, as well as a set of tools that allows a user to train and test a custom VMAF model. Read <a href="https://medium.com/netflix-techblog/toward-a-practical-perceptual-video-quality-metric-653f208b9652">this</a> techblog post for an overview, or <a href="https://medium.com/netflix-techblog/vmaf-the-journey-continues-44b51ee9ed12">this</a> post for the latest updates and tips for best practices.</p>
<p>Basically, vmaf is a rating for comparing a distorted video to its original and telling you how close it is. It is an objective measure of how close the video matches, one that takes many methods into account and is specifically for video. The scale itself is more subjective and based upon how real humans would rate the content.</p>
<h1 id="youtube-transcodes">YouTube Transcodes</h1>
<p>YouTube transcodes your videos to a lower quality stream-friendly version. It does this to two codecs, h264 & vp9.</p>
<p>By uploading our lossless videos to YouTube, downloading the transcoded versions and using vmaf to compare the videos, we can get an idea of what sort of quality we can get out of YouTube and put a number to it.</p>
<table>
<thead>
<tr>
<th>Name</th>
<th>YT.264 VMAF</th>
<th>YT.264 Rate (kb/s)</th>
<th>YT.vp9 VMAF</th>
<th>YT.vp9 Rate (kb/s)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Death_ll</td>
<td>57.606028</td>
<td>4257</td>
<td>65.894301</td>
<td>1961</td>
</tr>
<tr>
<td>Detroit_ll</td>
<td>68.334414</td>
<td>3536</td>
<td>79.690493</td>
<td>1716</td>
</tr>
<tr>
<td>Diablo_ll</td>
<td>64.417506</td>
<td>5496</td>
<td>70.17594</td>
<td>3207</td>
</tr>
<tr>
<td>Flower_ll</td>
<td>61.560809</td>
<td>5142</td>
<td>67.273151</td>
<td>2939</td>
</tr>
<tr>
<td>Forza_ll</td>
<td>54.373016</td>
<td>5654</td>
<td>56.147891</td>
<td>3285</td>
</tr>
<tr>
<td>Goose_ll</td>
<td>76.11744</td>
<td>2625</td>
<td>85.667461</td>
<td>1325</td>
</tr>
<tr>
<td>Persona_ll</td>
<td>85.245592</td>
<td>2607</td>
<td>89.414688</td>
<td>1473</td>
</tr>
<tr>
<td>RE7_ll</td>
<td>69.8519</td>
<td>4034</td>
<td>77.311906</td>
<td>2287</td>
</tr>
<tr>
<td>Rogue_ll</td>
<td>96.449077</td>
<td>3751</td>
<td>93.799547</td>
<td>2202</td>
</tr>
</tbody>
</table>
<p><img src="https://andsobegins.com/assets/images/Lossless_uploads_to_youtube.svg" alt="Lossless uploaded to YouTube" title="Lossless uploaded to YouTube" /><br />
<em>Lossless uploaded to YouTube</em></p>
<p>vp9 gives better results than h264 <em>despite</em> the bitrate being lower.</p>
<p>The bitrate is not <em>consistent</em>. An assumption might be that YouTube has a certain bitrate limit and that, should you feed it lossless content, it will use say, up to 5mb/s, but that’s not the case, YouTube like most modern encoders will encode targeting a quality - otherwise the bitrates would be flat.</p>
<p>You can see how YouTube will reflect hard-to-encode things and easy-to-encode things in much the same way as lossless. Coincidentally, difficult to encode things appear on the left, whereas the right-side games are easier to encode, requiring fewer bits for better quality. After a certain point, the video’s quality really starts dropping, so it isn’t a strict quality it targets, likely some form of bitrate constrained crf (constant rate factor, default quality targeting in most software encoders) is happening. It is also the case that YouTube segments its videos and encodes each segment separately these are the parts you need to download and assemble to create the finished YouTube video.</p>
<p>You may expect that lossless to YouTube would get you the best possible results from YouTube in terms of quality. In actuality, we can (and do) get higher vmaf ratings from lossy transcodes, but usually not by much and you can see a definite, but weak correlation between the vmaf rating of the Transcode and the vmaf rating of the YouTube version. A stronger correlation would likely be shown with lower quality trancodes. Local transcodes produce more predictable results.</p>
<h1 id="producing-the-videos">Producing the Videos</h1>
<p>Here are how the rest of the videos were produced for testing.</p>
<p>Two codecs, h264 and h265 encoded with libx264 and libx265.</p>
<p>Two speeds, s: slow, vs: veryslow.</p>
<p>x264 and x265 both allow you to select a speed preset with one of the following values:</p>
<ul>
<li>placebo</li>
<li>veryslow</li>
<li>slower</li>
<li>slow</li>
<li>medium (default)</li>
<li>fast</li>
<li>faster</li>
<li>veryfast</li>
<li>superfast</li>
<li>ultrafast</li>
</ul>
<p>We’re using veryslow for x264 as generally, this will produce the smallest files. Generally. For x265 we’re using slow because that’s what I actually use, but it is arbitrary, it represents what is fast enough. x264 is also fast enough at veryslow, but there is no slower speed (worth using). I can still use x265 after upgrading with a slower preset. Ultrafast is used just to make the point that even when following guidelines on how to encode, there is still room to tune for speed / efficiency within those parameters, but the results aren’t that interesting - so you can see them in the Google Sheet, but I haven’t used them in this write-up.</p>
<p>Just to quickly explain the quality settings, here are some terms.</p>
<p><a href="https://trac.ffmpeg.org/wiki/Encode/H.264">ffmpeg’s h264 encoding guide</a> says of crf:</p>
<p>Constant Rate Factor (CRF)</p>
<p>“Use this rate control mode if you want to keep the best quality and care less about the file size. This is the recommended rate control mode for most uses.” Note that it ranges from 0 (lossless) to 51 (hideous). Lower values mean higher quality. It will just use whatever bitrate gives you, perceptually, the required arbitrary quality. It is vbr (variable bitrate).</p>
<p>With cbr (constant bitrate), you assign a bitrate, and that’s how much it uses. x264 doesn’t actually have a cbr mode, but you can emulate it with x264 by specifying nal-hrd=cbr and the bitrate limitations. This is what OBS does too.</p>
<p>Several Quality settings, crf 15,18,23,28, strict YouTube recommended with bitrate recommendations (8Mbps for 30fps, and 12Mbps for 60fps), and YouTube recommended, but swapping in cbr instead of vbr. Since I’d also seen recommendations like record in cbr at 8 or 12Mbps, then encode in cbr at the same bitrate for YouTube, and thought that was terrible advice, that I’d do that too. The idea I’m guessing being that the encoder will preserve the same bits each pass.</p>
<p>Finally since part of what I want to be able to do is make a mistake and then re-render an episode once the source files are gone. So I wanted to see how “double-baked video” would do. So x265 was encoded at crf 18 and then that, in turn, was encoded at crf 18.</p>
<h1 id="comparing-quality-targets-versus-bitrate-targets">Comparing quality targets versus bitrate targets</h1>
<p>Turns out that YouTube’s encoding process takes away so much quality that only the crf 28 encodes caused a particularly sharp decline in the final vmaf of the YouTube encodes, even double cbr, whilst consistently worse than crf, was using fairly high bitrates that mostly survived the process. So it seems like you can’t go <em>too</em> wrong at least.</p>
<p>A good demonstration of the difference between picking a quality setting and picking a bitrate can be shown by taking the same codec and using both modes and showing the vmaf versus the bitrate. So let’s take x264, use the same speed preset of veryslow and encode crf 18 versus YouTube strict using 8 or 12Mbps. That gets us this.</p>
<table>
<thead>
<tr>
<th>Game</th>
<th>CRF18 VMAF</th>
<th>CRF18 Rate (kb/s)</th>
<th>YTRec VMAF</th>
<th>YTRec Rate (kb/s)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Death</td>
<td>99.179093</td>
<td>16859</td>
<td>69.120015</td>
<td>8386</td>
</tr>
<tr>
<td>Detroit</td>
<td>98.028856</td>
<td>8889</td>
<td>77.208784</td>
<td>8228</td>
</tr>
<tr>
<td>Diablo</td>
<td>96.894216</td>
<td>13035</td>
<td>80.557112</td>
<td>12282</td>
</tr>
<tr>
<td>Flower</td>
<td>96.282884</td>
<td>14280</td>
<td>94.192944</td>
<td>12412</td>
</tr>
<tr>
<td>Forza</td>
<td>98.319477</td>
<td>32144</td>
<td>68.470782</td>
<td>12083</td>
</tr>
<tr>
<td>Goose</td>
<td>96.063484</td>
<td>2618</td>
<td>82.46231</td>
<td>12218</td>
</tr>
<tr>
<td>Persona</td>
<td>97.764895</td>
<td>3826</td>
<td>89.441311</td>
<td>7988</td>
</tr>
<tr>
<td>RE7</td>
<td>96.013602</td>
<td>6867</td>
<td>76.679025</td>
<td>11783</td>
</tr>
<tr>
<td>Rogue</td>
<td>97.993974</td>
<td>3205</td>
<td>98.423263</td>
<td>12430</td>
</tr>
<tr>
<td>Average</td>
<td>97.39338678</td>
<td>11303</td>
<td>81.83950511</td>
<td>10868</td>
</tr>
</tbody>
</table>
<p><img src="https://andsobegins.com/assets/images/Quality_versus_bitrate.svg" alt="Quality versus Bitrate" title="Quality versus Bitrate" /><br />
<em>Comparing setting quality versus selecting target bitrate for x264</em></p>
<p>Instead of being fair, let’s be realistic and use x265 instead and compare that to the YouTube recommendations. Saying that though, we’ll still use the slow preset in x265 so it doesn’t take forever.</p>
<table>
<thead>
<tr>
<th>Game</th>
<th>265_18 VMAF</th>
<th>265_18 Rate (kb/s)</th>
<th>YTRec VMAF</th>
<th>YTRec Rate (kb/s)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Death</td>
<td>99.023438</td>
<td>13962</td>
<td>69.120015</td>
<td>8386</td>
</tr>
<tr>
<td>Detroit</td>
<td>97.94216</td>
<td>6447</td>
<td>77.208784</td>
<td>8228</td>
</tr>
<tr>
<td>Diablo</td>
<td>96.761145</td>
<td>10799</td>
<td>80.557112</td>
<td>12282</td>
</tr>
<tr>
<td>Flower</td>
<td>96.238418</td>
<td>11387</td>
<td>94.192944</td>
<td>12412</td>
</tr>
<tr>
<td>Forza</td>
<td>97.810374</td>
<td>26326</td>
<td>68.470782</td>
<td>12083</td>
</tr>
<tr>
<td>Goose</td>
<td>96.301997</td>
<td>1695</td>
<td>82.46231</td>
<td>12218</td>
</tr>
<tr>
<td>Persona</td>
<td>97.644543</td>
<td>3768</td>
<td>89.441311</td>
<td>7988</td>
</tr>
<tr>
<td>RE7</td>
<td>96.152931</td>
<td>5012</td>
<td>76.679025</td>
<td>11783</td>
</tr>
<tr>
<td>Rogue</td>
<td>97.675072</td>
<td>3067</td>
<td>98.423263</td>
<td>12430</td>
</tr>
<tr>
<td>Average</td>
<td>97.283342</td>
<td>9163</td>
<td>81.83950511</td>
<td>10868</td>
</tr>
</tbody>
</table>
<p><img src="https://andsobegins.com/assets/images/Quality_265_versus_264_bitrate.svg" alt="Quality versus Bitrate" title="Quality versus Bitrate" /><br />
<em>Comparing setting quality in x265 versus selecting target bitrate for x264</em></p>
<p><em>This</em> is why you don’t select a bitrate when encoding for YouTube, it won’t give consistent results, in fact it’s almost the opposite as games will vary in quality massively and that is then passed on to YouTube to source its own video from, whereas selecting a quality will simply use more bits for more difficult content, and fewer bits when it isn’t being challenged so much. It’s also a vindication of both crf and vmaf as crf is supposed to target a perceptual quality and vmaf is supposed to measure it.</p>
<p>12Mbps just isn’t enough to take on Forza, and 8Mbps isn’t enough to take on Death Stranding, but 12Mbps is completely overkill for Rogue Legacy. The average is especially telling. x265 uses fewer bits overall, but maintains a much higher and consistent quality. Only Rogue Legacy will look better on transcode, and only by a sliver, and not remotely proportional to the four times the bits being used to produce it. There are also plenty of examples of games using a much lower bitrate in x264 <em>and</em> x265 and producing far better results than the much higher bitrates of the YouTube recommended bitrates in x264. Untitled Goose Game, Persona 4, and Resident Evil 7 show just how inefficient selecting a bitrate target can be. Please stop.</p>
<h1 id="comparing-fixed-bitrate-methods">Comparing Fixed-bitrate methods</h1>
<p>Anyway, there were a bunch of different fixed bitrate transcodes, let’s have a look at them all.</p>
<p><img src="https://andsobegins.com/assets/images/Bitrate_targets.svg" alt="Targetting Bitrate" title="Targetting Bitrate" /><br />
<em>Comparing selecting target bitrate for x264 with Strict YouTube versus cbr and cbr into cbr</em></p>
<p>Unsurprisingly, CBR is always worse than VBR, and CBR into CBR is always worse than just CBR on its own.</p>
<p>Surprisingly, CBR wasn’t <em>much</em> worse than VBR, but then I suppose it isn’t really CBR, but an emulation of it that makes it pad the bitrate over a certain framesize, whereas most slow spots or black screen are going to be shorter than that framesize.</p>
<p>The conclusion however is still avoid CBR when transcoding for YouTube, and actually, avoid it all together. Even for streaming you’re better off using crf with a constrained bitrate limit.</p>
<h1 id="comparing-crf-methods">Comparing CRF methods</h1>
<p><img src="https://andsobegins.com/assets/images/crf_comparison.svg" alt="Comparing x265 crf values" title="Comparing x265 crf values" /><br />
<em>Comparing targetting quality with x265</em></p>
<p>My process has been to use crf 15 for 60fps and crf 18 for 30fps. Turns out, that’s overkill and the quality gain from 18 to 15 isn’t worth the rate increase, and 18 is fine for 30 or 60fps. So from now I will be using only crf 18.</p>
<h1 id="double-baking-crf">Double-baking crf</h1>
<p><img src="https://andsobegins.com/assets/images/Double-baked_18_2.svg" alt="Double-baking crf 18" title="Double-baking crf 18" /><br />
<em>Double-baking crf 18</em></p>
<p>I’m also quite happy that crf 18 can survive a further transcode of itself with an expected generational loss, but at least the bitrate also goes down. There’s barely any effect on the YouTube encodes, here we’re showing the better vp9 codec.</p>
<h1 id="what-about-4k">What about 4k?</h1>
<p><a href="/article/2020/09/22/VMAF-comparisons-for-2160-upscaled-Content-on-YouTube.html">Next article</a>, VMAF comparisons for 2160 upscaled Content on YouTube.</p>
<p><img src="https://andsobegins.com/assets/images/4k_preview.svg" alt="Uploading 2160p" title="Uploading 2160p" /><br />
<em>Uploading 2160p</em></p>
<p>Turns out the encoding done by YouTube when you upload 4k is so much better that even uploading upscaled 1080p content will result in significantly better YouTube vmaf scores for the same resolution. Here we show the realistic scenario. h264 version of the YouTube video shown by default for me versus the 2160p vp9 versions which are the default when uploading the upscaled to 2160p video and viewing in various resolution settings. Not shown, but comparing 1080 vp9 to 1080 vp9 versions on YouTube still shows a significant yet humble quality increase and 2160p viewed in its native resolution gives comparable results to 2160p viewed as 1080p.</p>
<p>So in actuality, all future videos will be quick rendered in 2160p with a lanzos upscale and then in 1080p crf 18 slow for archival. Because YouTube’s encoder is better when handling 2160p content.</p>
<h1 id="appendix-a-tools">Appendix A Tools</h1>
<p>ffmpeg-4.1.3 for transcoding.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ffmpeg version 4.1.3 Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 9.2.0 (Gentoo 9.2.0-r2 p3)
configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --docdir=/usr/share/doc/ffmpeg-4.1.3/html --mandir=/usr/share/man --enable-shared --cc=x86_64-pc-linux-gnu-gcc --cxx=x86_64-pc-linux-gnu-g++ --ar=x86_64-pc-linux-gnu-ar --optflags='-march=native -O2 -pipe' --disable-static --enable-avfilter --enable-avresample --enable-libvmaf --enable-version3 --disable-stripping --disable-optimizations --disable-libcelt --disable-indev=v4l2 --disable-outdev=v4l2 --disable-indev=oss --disable-indev=jack --disable-outdev=oss --enable-bzlib --disable-runtime-cpudetect --disable-debug --disable-gcrypt --disable-gnutls --disable-gmp --enable-gpl --enable-hardcoded-tables --enable-iconv --disable-libtls --disable-libxml2 --disable-lzma --enable-network --disable-opencl --disable-openssl --enable-postproc --disable-libsmbclient --enable-ffplay --enable-sdl2 --enable-vaapi --enable-vdpau --enable-xlib --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-zlib --disable-libcdio --disable-libiec61883 --disable-libdc1394 --disable-libcaca --disable-openal --enable-opengl --disable-libv4l2 --enable-libpulse --disable-libdrm --disable-libjack --disable-libopencore-amrwb --disable-libopencore-amrnb --disable-libcodec2 --disable-libfdk-aac --disable-libopenjpeg --disable-libbluray --disable-libgme --disable-libgsm --disable-mmal --disable-libmodplug --enable-libopus --disable-libilbc --disable-librtmp --disable-libssh --disable-libspeex --disable-libsrt --enable-librsvg --disable-ffnvcodec --enable-libvorbis --disable-libvpx --disable-libzvbi --disable-appkit --disable-libbs2b --disable-chromaprint --disable-libflite --disable-frei0r --disable-libfribidi --disable-fontconfig --disable-ladspa --disable-libass --disable-lv2 --enable-libfreetype --disable-librubberband --disable-libzmq --disable-libzimg --disable-libsoxr --enable-pthreads --disable-libvo-amrwbenc --enable-libmp3lame --disable-libkvazaar --disable-libaom --disable-libopenh264 --disable-libsnappy --disable-libtheora --disable-libtwolame --enable-libwavpack --disable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --disable-armv5te --disable-armv6 --disable-armv6t2 --disable-neon --disable-vfp --disable-vfpv3 --disable-armv8 --disable-mipsdsp --disable-mipsdspr2 --disable-mipsfpu --disable-altivec --disable-amd3dnow --disable-amd3dnowext --disable-avx2 --cpu=host --disable-doc --disable-htmlpages --enable-manpages
libavutil 56. 22.100 / 56. 22.100
libavcodec 58. 35.100 / 58. 35.100
libavformat 58. 20.100 / 58. 20.100
libavdevice 58. 5.100 / 58. 5.100
libavfilter 7. 40.101 / 7. 40.101
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 3.100 / 5. 3.100
libswresample 3. 3.100 / 3. 3.100
libpostproc 55. 3.100 / 55. 3.100
</code></pre></div></div>
<p>vmaf-1.3.15 for the vmaf library.</p>
<p>gnu parallel-20191022 for pushing jobs around the network.</p>
<p>obs-24.0.3 for capturing the video from the Black Magic card.</p>
<p>youtube-dl-2020.01.24 for downloading the videos from YouTube for comparison.</p>
<h1 id="appendix-b-creating-transcodes">Appendix B Creating Transcodes</h1>
<p>All capturing was done via hmdi into a Black Magic Intensity Pro 4k in 1080p full rgb @60000/1001. OBS was used to record the stream in yuv420p limited colour.</p>
<p>Transcode to lossless just to smooth out any possible issues with indexing or the like and to strip the audio.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>parallel <span class="nt">--nice</span> 20 <span class="nt">--eta</span> <span class="nt">-S</span> node1,node2,node3 <span class="nt">-j1</span> ffmpeg <span class="nt">-i</span> <span class="o">{}</span> <span class="nt">-qp</span> 0 <span class="nt">-an</span> <span class="o">{</span>.<span class="o">}</span>_ll_medium.mkv ::: /mnt/LPWorking/vmaf/ORIG/<span class="k">*</span>.mkv
</code></pre></div></div>
<p>Framecopy out the ‘br’ versions, which are specifically to get an idea of the bitrate of the original captures for the same three minutes. These will be slightly longer than 180 seconds, but when we work out the bitrate, we take these extra seconds into consideration. This is why it’s not 100% accurate.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>parallel <span class="nt">--eta</span> <span class="nt">-j1</span> ffmpeg <span class="nt">-ss</span> <span class="o">{</span>1<span class="o">}</span> <span class="nt">-nostdin</span> <span class="nt">-i</span> <span class="o">{</span>2<span class="o">}</span>.mkv <span class="nt">-c</span> copy <span class="nt">-an</span> <span class="nt">-t</span> 180 <span class="nt">-y</span> <span class="o">{</span>2<span class="o">}</span>_br.mkv ::: 237 141 90 16 5 7 210 7 14 :::+ Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue
</code></pre></div></div>
<p>Get three minute clips from the transcodes of the original captures - also set the framerate to what the content should be in, rather than the obs one. Non-fractional framerates for simplicity’s sake. These were used as reference videos for vmaf.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>parallel <span class="nt">--eta</span> <span class="nt">-S</span> node1,node2,node3 <span class="nt">-j1</span> ffmpeg <span class="nt">-ss</span> <span class="o">{</span>1<span class="o">}</span> <span class="nt">-nostdin</span> <span class="nt">-i</span> /mnt/LPWorking/vmaf/ORIG/<span class="o">{</span>2<span class="o">}</span>_ll_medium.mkv <span class="nt">-qp</span> 0 <span class="nt">-preset</span> veryslow <span class="nt">-r</span> <span class="o">{</span>3<span class="o">}</span> <span class="nt">-t</span> 180 <span class="nt">-y</span> /mnt/LPWorking/vmaf/<span class="o">{</span>2<span class="o">}</span>_ll.mkv ::: 237 141 90 16 5 7 210 7 14 :::+ Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue :::+ 30 30 60 60 60 60 30 60 60
</code></pre></div></div>
<p>CRF versions were made like this.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>parallel <span class="nt">--eta</span> <span class="nt">-j1</span> <span class="nt">-S</span> node1,node2,node3 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_ll.mkv <span class="nt">-preset</span> veryslow <span class="nt">-crf</span> <span class="o">{</span>2<span class="o">}</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_264_vs_<span class="o">{</span>2<span class="o">}</span>.mkv ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue ::: 15 18 23 28
parallel <span class="nt">--eta</span> <span class="nt">-j1</span> <span class="nt">-S</span> node1,node2,node3 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_ll.mkv <span class="nt">-preset</span> slow <span class="nt">-c</span> libx265 <span class="nt">-crf</span> <span class="o">{</span>2<span class="o">}</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_265_s_<span class="o">{</span>2<span class="o">}</span>.mkv ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue ::: 15 18 23 28
</code></pre></div></div>
<p>Strict YouTube versions.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>parallel <span class="nt">-j1</span> <span class="nt">--eta</span> <span class="nt">-S</span> node1,node2,node3 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_ll.mkv <span class="nt">-g</span> <span class="o">{</span>3<span class="o">}</span> <span class="nt">-bf</span> 2 <span class="nt">-profile</span>:v high <span class="nt">-movflags</span> faststart <span class="nt">-coder</span> 1 <span class="nt">-preset</span> veryslow <span class="nt">-b</span>:v <span class="o">{</span>2<span class="o">}</span>M <span class="nt">-y</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_264_vs_ytrec.mp4 ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue :::+ 8 8 12 12 12 12 8 12 12 :::+ 15 15 30 30 30 30 15 30 30
parallel <span class="nt">-j1</span> <span class="nt">--eta</span> <span class="nt">-S</span> node1,node2,node3 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_ll.mkv <span class="nt">-g</span> <span class="o">{</span>3<span class="o">}</span> <span class="nt">-bf</span> 2 <span class="nt">-profile</span>:v high <span class="nt">-movflags</span> faststart <span class="nt">-coder</span> 1 <span class="nt">-preset</span> ultrafast <span class="nt">-b</span>:v <span class="o">{</span>2<span class="o">}</span>M <span class="nt">-y</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_264_uf_ytrec.mp4 ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue :::+ 8 8 12 12 12 12 8 12 12 :::+ 15 15 30 30 30 30 15 30 30
</code></pre></div></div>
<p>CBR versions.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>parallel <span class="nt">-j1</span> <span class="nt">--eta</span> <span class="nt">-S</span> node1,node2,node3 <span class="nt">--nice</span> 20 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_ll.mkv <span class="nt">-g</span> <span class="o">{</span>3<span class="o">}</span> <span class="nt">-bf</span> 2 <span class="nt">-profile</span>:v high <span class="nt">-movflags</span> faststart <span class="nt">-coder</span> 1 <span class="nt">-preset</span> ultrafast <span class="nt">-x264-params</span> <span class="s2">"nal-hrd=cbr"</span> <span class="nt">-b</span>:v <span class="o">{</span>2<span class="o">}</span>M <span class="nt">-minrate</span> <span class="o">{</span>2<span class="o">}</span>M <span class="nt">-maxrate</span> <span class="o">{</span>2<span class="o">}</span>M <span class="nt">-bufsize</span> 2M <span class="nt">-y</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_264_uf_cbr.mp4 ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue :::+ 8 8 12 12 12 12 8 12 12 :::+ 15 15 30 30 30 30 15 30 30
parallel <span class="nt">-j1</span> <span class="nt">--eta</span> <span class="nt">-S</span> node1,node2,node3 <span class="nt">--nice</span> 20 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_ll.mkv <span class="nt">-g</span> <span class="o">{</span>3<span class="o">}</span> <span class="nt">-bf</span> 2 <span class="nt">-profile</span>:v high <span class="nt">-movflags</span> faststart <span class="nt">-coder</span> 1 <span class="nt">-preset</span> veryslow <span class="nt">-x264-params</span> <span class="s2">"nal-hrd=cbr"</span> <span class="nt">-b</span>:v <span class="o">{</span>2<span class="o">}</span>M <span class="nt">-minrate</span> <span class="o">{</span>2<span class="o">}</span>M <span class="nt">-maxrate</span> <span class="o">{</span>2<span class="o">}</span>M <span class="nt">-bufsize</span> 2M <span class="nt">-y</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_264_vs_cbr.mp4 ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue :::+ 8 8 12 12 12 12 8 12 12 :::+ 15 15 30 30 30 30 15 30 30
</code></pre></div></div>
<p>CBR to CBR versions.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>parallel <span class="nt">-j1</span> <span class="nt">--eta</span> <span class="nt">-S</span> node1,node2,node3 <span class="nt">--nice</span> 20 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_264_vs_cbr.mp4 <span class="nt">-g</span> <span class="o">{</span>3<span class="o">}</span> <span class="nt">-bf</span> 2 <span class="nt">-profile</span>:v high <span class="nt">-movflags</span> faststart <span class="nt">-coder</span> 1 <span class="nt">-preset</span> veryslow <span class="nt">-x264-params</span> <span class="s2">"nal-hrd=cbr"</span> <span class="nt">-b</span>:v <span class="o">{</span>2<span class="o">}</span>M <span class="nt">-minrate</span> <span class="o">{</span>2<span class="o">}</span>M <span class="nt">-maxrate</span> <span class="o">{</span>2<span class="o">}</span>M <span class="nt">-bufsize</span> 2M <span class="nt">-y</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_264_vs_cbrcbr.mp4 ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue :::+ 8 8 12 12 12 12 8 12 12 :::+ 15 15 30 30 30 30 15 30 30
parallel <span class="nt">-j1</span> <span class="nt">--eta</span> <span class="nt">-S</span> node1,node2,node3 <span class="nt">--nice</span> 20 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_264_uf_cbr.mp4 <span class="nt">-g</span> <span class="o">{</span>3<span class="o">}</span> <span class="nt">-bf</span> 2 <span class="nt">-profile</span>:v high <span class="nt">-movflags</span> faststart <span class="nt">-coder</span> 1 <span class="nt">-preset</span> ultrafast <span class="nt">-x264-params</span> <span class="s2">"nal-hrd=cbr"</span> <span class="nt">-b</span>:v <span class="o">{</span>2<span class="o">}</span>M <span class="nt">-minrate</span> <span class="o">{</span>2<span class="o">}</span>M <span class="nt">-maxrate</span> <span class="o">{</span>2<span class="o">}</span>M <span class="nt">-bufsize</span> 2M <span class="nt">-y</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_264_uf_cbrcbr.mp4 ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue :::+ 8 8 12 12 12 12 8 12 12 :::+ 15 15 30 30 30 30 15 30 30
</code></pre></div></div>
<p>… and finally, Double-baked CRF.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>parallel <span class="nt">--eta</span> <span class="nt">-j1</span> <span class="nt">--nice</span> 20 <span class="nt">-S</span> node1,node2,node3 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{}</span>_265_s_18.mkv <span class="nt">-preset</span> slow <span class="nt">-crf</span> 18 /mnt/LPWorking/vmaf/<span class="o">{}</span>_265_s_1818.mkv ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue
</code></pre></div></div>
<h1 id="appendix-c-transcoded-vmaf-comparisons">Appendix C Transcoded VMAF comparisons</h1>
<p>Self comparison for lossless</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>parallel <span class="nt">--nice</span> 20 <span class="nt">--eta</span> <span class="nt">-j1</span> <span class="nt">-S</span> node1,node2,node3 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_ll.mkv <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_ll.mkv <span class="nt">-lavfi</span> <span class="nv">libvmaf</span><span class="o">=</span><span class="s2">"model_path=/usr/share/model/vmaf_v0.6.1.pkl"</span> <span class="nt">-f</span> null /dev/null ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue &> /mnt/LPWorking/vmaf/vmafscores_self.txt
</code></pre></div></div>
<p>Comparisons for all the CRF files.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>parallel <span class="nt">--nice</span> 20 <span class="nt">--eta</span> <span class="nt">-j1</span> <span class="nt">-S</span> node1,node2,node3 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_<span class="o">{</span>3<span class="o">}</span>_<span class="o">{</span>4<span class="o">}</span>_<span class="o">{</span>2<span class="o">}</span>.mkv <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_ll.mkv <span class="nt">-lavfi</span> <span class="nv">libvmaf</span><span class="o">=</span><span class="s2">"model_path=/usr/share/model/vmaf_v0.6.1.pkl"</span> <span class="nt">-f</span> null /dev/null ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue ::: 15 18 23 28 ::: 264 265 :::+ vs s &> /mnt/LPWorking/vmaf/vmafscores_crf.txt
</code></pre></div></div>
<p>Comparisons for all the bitrate targetted files.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>parallel <span class="nt">--nice</span> 20 <span class="nt">--eta</span> <span class="nt">-j1</span> <span class="nt">-S</span> node1,node2,node3 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_264_<span class="o">{</span>2<span class="o">}</span>_<span class="o">{</span>3<span class="o">}</span>.mp4 <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_ll.mkv <span class="nt">-lavfi</span> <span class="nv">libvmaf</span><span class="o">=</span><span class="s2">"model_path=/usr/share/model/vmaf_v0.6.1.pkl"</span> <span class="nt">-f</span> null <span class="nt">-t</span> 180 /dev/null ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue ::: uf vs ::: ytrec cbr cbrcbr &> /mnt/LPWorking/vmaf/vmafscores_mp4.txt
</code></pre></div></div>
<p>Comparisons for Twice-baked CRF.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>parallel <span class="nt">--eta</span> <span class="nt">-j1</span> <span class="nt">-S</span> node1,node2,node3 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_265_s_1818.mkv <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_ll.mkv <span class="nt">-lavfi</span> <span class="nv">libvmaf</span><span class="o">=</span><span class="s2">"model_path=/usr/share/model/vmaf_v0.6.1.pkl"</span> <span class="nt">-f</span> null <span class="nt">-t</span> 180 /dev/null ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue &> /mnt/LPWorking/vmaf/vmafscores_1818.txt
</code></pre></div></div>
<h1 id="appendix-d-vmaf-comparisons-from-youtube">Appendix D VMAF Comparisons from YouTube</h1>
<p>Lossless upload.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>parallel <span class="nt">-j1</span> <span class="nt">--eta</span> <span class="nt">-S</span> node1,node2,node3 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/yt/<span class="o">{}</span>_ll.mp4 <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{}</span>_ll.mkv <span class="nt">-lavfi</span> <span class="nv">libvmaf</span><span class="o">=</span><span class="s2">"model_path=/usr/share/model/vmaf_v0.6.1.pkl"</span> <span class="nt">-f</span> null <span class="nt">-t</span> 180 /dev/null ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue &> vmafscores_ll_264.txt
parallel <span class="nt">-j1</span> <span class="nt">--eta</span> <span class="nt">-S</span> node1,node2,node3 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/yt/<span class="o">{}</span>_ll.webm <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{}</span>_ll.mkv <span class="nt">-lavfi</span> <span class="nv">libvmaf</span><span class="o">=</span><span class="s2">"model_path=/usr/share/model/vmaf_v0.6.1.pkl"</span> <span class="nt">-f</span> null <span class="nt">-t</span> 180 /dev/null ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue &> vmafscores_ll_vp9.txt
</code></pre></div></div>
<p>CRF comparisons.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>parallel <span class="nt">--eta</span> <span class="nt">-j1</span> <span class="nt">-S</span> node1,node2,node3 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/yt/<span class="o">{</span>1<span class="o">}</span>_<span class="o">{</span>3<span class="o">}</span>_<span class="o">{</span>4<span class="o">}</span>_<span class="o">{</span>2<span class="o">}</span>.mp4 <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_ll.mkv <span class="nt">-lavfi</span> <span class="nv">libvmaf</span><span class="o">=</span><span class="s2">"model_path=/usr/share/model/vmaf_v0.6.1.pkl"</span> <span class="nt">-f</span> null <span class="nt">-t</span> 180 /dev/null ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue ::: 15 18 23 28 ::: 264 265 :::+ vs s &> vmafscores_264.txt
parallel <span class="nt">--eta</span> <span class="nt">-j1</span> <span class="nt">-S</span> node1,node2,node3 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/yt/<span class="o">{</span>1<span class="o">}</span>_<span class="o">{</span>3<span class="o">}</span>_<span class="o">{</span>4<span class="o">}</span>_<span class="o">{</span>2<span class="o">}</span>.webm <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_ll.mkv <span class="nt">-lavfi</span> <span class="nv">libvmaf</span><span class="o">=</span><span class="s2">"model_path=/usr/share/model/vmaf_v0.6.1.pkl"</span> <span class="nt">-f</span> null <span class="nt">-t</span> 180 /dev/null ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue ::: 15 18 23 28 ::: 264 265 :::+ vs s &> vmafscores_vp9.txt
</code></pre></div></div>
<p>Comparisons for all the bitrate targetted files.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>parallel <span class="nt">--nice</span> 20 <span class="nt">--eta</span> <span class="nt">-j1</span> <span class="nt">-S</span> node1,node2,node3 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/yt/<span class="o">{</span>1<span class="o">}</span>_264_<span class="o">{</span>2<span class="o">}</span>_<span class="o">{</span>3<span class="o">}</span>.mp4 <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_ll.mkv <span class="nt">-lavfi</span> <span class="nv">libvmaf</span><span class="o">=</span><span class="s2">"model_path=/usr/share/model/vmaf_v0.6.1.pkl"</span> <span class="nt">-f</span> null <span class="nt">-t</span> 180 /dev/null ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue ::: uf vs ::: ytrec cbr cbrcbr &> /mnt/LPWorking/vmaf/yt/vmafscores_rec_264.txt
parallel <span class="nt">--nice</span> 20 <span class="nt">--eta</span> <span class="nt">-j1</span> <span class="nt">-S</span> node1,node2,node3 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/yt/<span class="o">{</span>1<span class="o">}</span>_264_<span class="o">{</span>2<span class="o">}</span>_<span class="o">{</span>3<span class="o">}</span>.webm <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_ll.mkv <span class="nt">-lavfi</span> <span class="nv">libvmaf</span><span class="o">=</span><span class="s2">"model_path=/usr/share/model/vmaf_v0.6.1.pkl"</span> <span class="nt">-f</span> null <span class="nt">-t</span> 180 /dev/null ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue ::: uf vs ::: ytrec cbr cbrcbr &> /mnt/LPWorking/vmaf/yt/vmafscores_rec_vp9.txt
</code></pre></div></div>
<p>Comparisons for Twice-baked CRF.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>parallel <span class="nt">--nice</span> 20 <span class="nt">--eta</span> <span class="nt">-j1</span> <span class="nt">-S</span> node1,node2,node3 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/yt/<span class="o">{</span>1<span class="o">}</span>_265_s_1818.mp4 <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_ll.mkv <span class="nt">-lavfi</span> <span class="nv">libvmaf</span><span class="o">=</span><span class="s2">"model_path=/usr/share/model/vmaf_v0.6.1.pkl"</span> <span class="nt">-f</span> null <span class="nt">-t</span> 180 /dev/null ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue &> /mnt/LPWorking/vmaf/yt/vmafscores_1818_264.txt
parallel <span class="nt">--nice</span> 20 <span class="nt">--eta</span> <span class="nt">-j1</span> <span class="nt">-S</span> node1,node2,node3 ffmpeg <span class="nt">-i</span> /mnt/LPWorking/vmaf/yt/<span class="o">{</span>1<span class="o">}</span>_265_s_1818.webm <span class="nt">-i</span> /mnt/LPWorking/vmaf/<span class="o">{</span>1<span class="o">}</span>_ll.mkv <span class="nt">-lavfi</span> <span class="nv">libvmaf</span><span class="o">=</span><span class="s2">"model_path=/usr/share/model/vmaf_v0.6.1.pkl"</span> <span class="nt">-f</span> null <span class="nt">-t</span> 180 /dev/null ::: Death Detroit Diablo Flower Forza Goose Persona RE7 Rogue &> /mnt/LPWorking/vmaf/yt/vmafscores_1818_vp9.txt
</code></pre></div></div>
<h1 id="appendix-x-results-tables">Appendix X Results Tables</h1>
<p>Here are the tables of data for completeness.</p>
<p>If you just want to see the data for charting etc. Please use the Google Doc https://docs.google.com/spreadsheets/d/1D3000VzF7XZ1HKsLKYdK2CULOyII0hFwo257vxUzLnY/edit?usp=sharing</p>
<table>
<thead>
<tr>
<th>Game</th>
<th>Capture rate* (kb/s)</th>
<th>Transcode rate (kb/s)</th>
<th>Final Fps</th>
<th>Capture Size</th>
<th>Transcode Size</th>
</tr>
</thead>
<tbody>
<tr>
<td>Death</td>
<td>307012</td>
<td>176507</td>
<td>30</td>
<td>7040946293</td>
<td>3971409494</td>
</tr>
<tr>
<td>Detroit</td>
<td>434886</td>
<td>217563</td>
<td>30</td>
<td>9970854162</td>
<td>4895175886</td>
</tr>
<tr>
<td>Diablo</td>
<td>281281</td>
<td>211467</td>
<td>60</td>
<td>6415678692</td>
<td>4758011796</td>
</tr>
<tr>
<td>Flower</td>
<td>317912</td>
<td>238585</td>
<td>60</td>
<td>7293305749</td>
<td>5368162899</td>
</tr>
<tr>
<td>Forza</td>
<td>500664</td>
<td>389333</td>
<td>60</td>
<td>11318766196</td>
<td>8759995647</td>
</tr>
<tr>
<td>Goose</td>
<td>241264</td>
<td>189136</td>
<td>60</td>
<td>5514996855</td>
<td>4255550684</td>
</tr>
<tr>
<td>Persona</td>
<td>81923</td>
<td>43857</td>
<td>30</td>
<td>1858737306</td>
<td>986790988</td>
</tr>
<tr>
<td>RE7</td>
<td>298612</td>
<td>221580</td>
<td>60</td>
<td>6825892227</td>
<td>4985544618</td>
</tr>
<tr>
<td>Rogue</td>
<td>78416</td>
<td>33123</td>
<td>60</td>
<td>1779155973</td>
<td>745274044</td>
</tr>
<tr>
<td>Average</td>
<td>282441</td>
<td>191239</td>
<td>50</td>
<td>6446481495</td>
<td>4302879562</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>Name</th>
<th>Transcode VMAF</th>
<th>Transcode Rate (kb/s)</th>
<th>YT.264 VMAF</th>
<th>YT.264 Rate (kb/s)</th>
<th>YT.vp9 VMAF</th>
<th>YT.vp9 Rate (kb/s)</th>
<th>Trancoded Size</th>
<th>YT.264 Size</th>
<th>YT.vp9 Size</th>
</tr>
</thead>
<tbody>
<tr>
<td>Death_264_uf_cbr</td>
<td>67.170819</td>
<td>8002</td>
<td>56.039215</td>
<td>4438</td>
<td>59.186626</td>
<td>1927</td>
<td>180041664</td>
<td>99844109</td>
<td>43355383</td>
</tr>
<tr>
<td>Death_264_uf_cbrcbr</td>
<td>64.361487</td>
<td>8002</td>
<td>54.460046</td>
<td>4416</td>
<td>57.944661</td>
<td>1928</td>
<td>180039429</td>
<td>99363584</td>
<td>43384830</td>
</tr>
<tr>
<td>Death_264_uf_ytrec</td>
<td>67.914076</td>
<td>8310</td>
<td>56.244398</td>
<td>4423</td>
<td>59.311758</td>
<td>1932</td>
<td>186965479</td>
<td>99514490</td>
<td>43480780</td>
</tr>
<tr>
<td>Death_264_vs_15</td>
<td>99.694551</td>
<td>25197</td>
<td>57.55067</td>
<td>4256</td>
<td>65.716546</td>
<td>1959</td>
<td>566931155</td>
<td>95754503</td>
<td>44073333</td>
</tr>
<tr>
<td>Death_264_vs_18</td>
<td>99.179093</td>
<td>16859</td>
<td>57.404738</td>
<td>4242</td>
<td>65.547333</td>
<td>1957</td>
<td>379330750</td>
<td>95453399</td>
<td>44038592</td>
</tr>
<tr>
<td>Death_264_vs_23</td>
<td>95.273895</td>
<td>8251</td>
<td>57.720714</td>
<td>4427</td>
<td>60.503374</td>
<td>1930</td>
<td>185637158</td>
<td>99609686</td>
<td>43422512</td>
</tr>
<tr>
<td>Death_264_vs_28</td>
<td>84.922153</td>
<td>4016</td>
<td>50.188563</td>
<td>4186</td>
<td>58.311813</td>
<td>1964</td>
<td>90369193</td>
<td>94186195</td>
<td>44185395</td>
</tr>
<tr>
<td>Death_264_vs_cbr</td>
<td>68.247038</td>
<td>8004</td>
<td>57.331133</td>
<td>4445</td>
<td>60.304952</td>
<td>1934</td>
<td>180097959</td>
<td>100007254</td>
<td>43507926</td>
</tr>
<tr>
<td>Death_264_vs_cbrcbr</td>
<td>65.563274</td>
<td>8003</td>
<td>56.280751</td>
<td>4466</td>
<td>59.675803</td>
<td>1938</td>
<td>180058613</td>
<td>100481041</td>
<td>43607225</td>
</tr>
<tr>
<td>Death_264_vs_ytrec</td>
<td>69.120015</td>
<td>8386</td>
<td>57.491718</td>
<td>4433</td>
<td>60.3571</td>
<td>1936</td>
<td>188685066</td>
<td>99733146</td>
<td>43569036</td>
</tr>
<tr>
<td>Death_265_s_15</td>
<td>99.622277</td>
<td>21093</td>
<td>58.426482</td>
<td>4434</td>
<td>60.789081</td>
<td>1935</td>
<td>474599657</td>
<td>99758224</td>
<td>43548322</td>
</tr>
<tr>
<td>Death_265_s_18</td>
<td>99.023438</td>
<td>13962</td>
<td>57.284986</td>
<td>4242</td>
<td>65.486753</td>
<td>1955</td>
<td>314153911</td>
<td>95434306</td>
<td>43982566</td>
</tr>
<tr>
<td>Death_265_s_1818</td>
<td>97.502328</td>
<td>12807</td>
<td>57.852953</td>
<td>4427</td>
<td>60.483807</td>
<td>1932</td>
<td>288152248</td>
<td>99612477</td>
<td>43473458</td>
</tr>
<tr>
<td>Death_265_s_23</td>
<td>95.377011</td>
<td>6852</td>
<td>57.575012</td>
<td>4435</td>
<td>60.462077</td>
<td>1932</td>
<td>154161249</td>
<td>99784141</td>
<td>43468481</td>
</tr>
<tr>
<td>Death_265_s_28</td>
<td>86.784151</td>
<td>3263</td>
<td>50.371678</td>
<td>4194</td>
<td>58.649406</td>
<td>1960</td>
<td>73407268</td>
<td>94363557</td>
<td>44089549</td>
</tr>
<tr>
<td>Death_ll</td>
<td>99.937872</td>
<td>176507</td>
<td>57.606028</td>
<td>4257</td>
<td>65.894301</td>
<td>1961</td>
<td>3971409494</td>
<td>95774239</td>
<td>44121968</td>
</tr>
<tr>
<td>Detroit_264_uf_cbr</td>
<td>76.297497</td>
<td>8002</td>
<td>68.886341</td>
<td>3504</td>
<td>79.071504</td>
<td>1711</td>
<td>180039724</td>
<td>78842052</td>
<td>38504205</td>
</tr>
<tr>
<td>Detroit_264_uf_cbrcbr</td>
<td>75.42974</td>
<td>8002</td>
<td>68.311523</td>
<td>3510</td>
<td>78.361377</td>
<td>1715</td>
<td>180039500</td>
<td>78966842</td>
<td>38590540</td>
</tr>
<tr>
<td>Detroit_264_uf_ytrec</td>
<td>77.022013</td>
<td>8238</td>
<td>67.31789</td>
<td>3484</td>
<td>78.597533</td>
<td>1712</td>
<td>185351940</td>
<td>78398301</td>
<td>38516595</td>
</tr>
<tr>
<td>Detroit_264_vs_15</td>
<td>98.552945</td>
<td>19016</td>
<td>68.155937</td>
<td>3499</td>
<td>79.59872</td>
<td>1714</td>
<td>427869845</td>
<td>78730566</td>
<td>38560415</td>
</tr>
<tr>
<td>Detroit_264_vs_18</td>
<td>98.028856</td>
<td>8889</td>
<td>68.128538</td>
<td>3511</td>
<td>79.433685</td>
<td>1711</td>
<td>200009536</td>
<td>78999786</td>
<td>38497687</td>
</tr>
<tr>
<td>Detroit_264_vs_23</td>
<td>96.208758</td>
<td>3345</td>
<td>67.317815</td>
<td>3467</td>
<td>78.501758</td>
<td>1701</td>
<td>75273720</td>
<td>77997467</td>
<td>38269453</td>
</tr>
<tr>
<td>Detroit_264_vs_28</td>
<td>90.955378</td>
<td>1788</td>
<td>55.609121</td>
<td>3291</td>
<td>64.554201</td>
<td>1665</td>
<td>40221670</td>
<td>74043579</td>
<td>37458385</td>
</tr>
<tr>
<td>Detroit_264_vs_cbr</td>
<td>76.568403</td>
<td>8004</td>
<td>69.547736</td>
<td>3500</td>
<td>79.966464</td>
<td>1718</td>
<td>180094594</td>
<td>78753355</td>
<td>38656200</td>
</tr>
<tr>
<td>Detroit_264_vs_cbrcbr</td>
<td>75.886727</td>
<td>8005</td>
<td>69.196849</td>
<td>3494</td>
<td>79.506965</td>
<td>1717</td>
<td>180105807</td>
<td>78615884</td>
<td>38622669</td>
</tr>
<tr>
<td>Detroit_264_vs_ytrec</td>
<td>77.208784</td>
<td>8228</td>
<td>67.962808</td>
<td>3508</td>
<td>79.230978</td>
<td>1715</td>
<td>185140499</td>
<td>78919264</td>
<td>38587640</td>
</tr>
<tr>
<td>Detroit_265_s_15</td>
<td>98.438399</td>
<td>13375</td>
<td>67.999537</td>
<td>3504</td>
<td>79.347754</td>
<td>1705</td>
<td>300943133</td>
<td>78849315</td>
<td>38372501</td>
</tr>
<tr>
<td>Detroit_265_s_18</td>
<td>97.94216</td>
<td>6447</td>
<td>67.874461</td>
<td>3490</td>
<td>79.145717</td>
<td>1701</td>
<td>145062821</td>
<td>78529570</td>
<td>38278091</td>
</tr>
<tr>
<td>Detroit_265_s_1818</td>
<td>97.094961</td>
<td>5485</td>
<td>67.328119</td>
<td>3453</td>
<td>78.57657</td>
<td>1689</td>
<td>123405557</td>
<td>77691690</td>
<td>38002460</td>
</tr>
<tr>
<td>Detroit_265_s_23</td>
<td>96.48004</td>
<td>2552</td>
<td>67.211333</td>
<td>3462</td>
<td>78.484413</td>
<td>1693</td>
<td>57418441</td>
<td>77906170</td>
<td>38095093</td>
</tr>
<tr>
<td>Detroit_265_s_28</td>
<td>92.530513</td>
<td>1278</td>
<td>56.030748</td>
<td>3320</td>
<td>65.086951</td>
<td>1666</td>
<td>28756460</td>
<td>74709380</td>
<td>37483895</td>
</tr>
<tr>
<td>Detroit_ll</td>
<td>99.504689</td>
<td>217563</td>
<td>68.334414</td>
<td>3536</td>
<td>79.690493</td>
<td>1716</td>
<td>4895175886</td>
<td>79567396</td>
<td>38608798</td>
</tr>
<tr>
<td>Diablo_264_uf_cbr</td>
<td>77.996417</td>
<td>12004</td>
<td>63.051692</td>
<td>5496</td>
<td>68.32209</td>
<td>3195</td>
<td>270100246</td>
<td>123649023</td>
<td>71886724</td>
</tr>
<tr>
<td>Diablo_264_uf_cbrcbr</td>
<td>76.400558</td>
<td>12004</td>
<td>62.572641</td>
<td>5528</td>
<td>67.713572</td>
<td>3193</td>
<td>270099321</td>
<td>124371913</td>
<td>71842697</td>
</tr>
<tr>
<td>Diablo_264_uf_ytrec</td>
<td>79.272198</td>
<td>12361</td>
<td>69.590782</td>
<td>5550</td>
<td>76.076226</td>
<td>3163</td>
<td>278123593</td>
<td>124875197</td>
<td>71174661</td>
</tr>
<tr>
<td>Diablo_264_vs_15</td>
<td>98.106589</td>
<td>19346</td>
<td>64.318407</td>
<td>5482</td>
<td>70.04671</td>
<td>3203</td>
<td>435275210</td>
<td>123337158</td>
<td>72069890</td>
</tr>
<tr>
<td>Diablo_264_vs_18</td>
<td>96.894216</td>
<td>13035</td>
<td>64.269661</td>
<td>5498</td>
<td>69.925102</td>
<td>3196</td>
<td>293280333</td>
<td>123711748</td>
<td>71905850</td>
</tr>
<tr>
<td>Diablo_264_vs_23</td>
<td>92.804394</td>
<td>6737</td>
<td>63.775731</td>
<td>5450</td>
<td>69.352453</td>
<td>3185</td>
<td>151587428</td>
<td>122620534</td>
<td>71664092</td>
</tr>
<tr>
<td>Diablo_264_vs_28</td>
<td>84.837268</td>
<td>3476</td>
<td>62.118481</td>
<td>5433</td>
<td>67.233671</td>
<td>3151</td>
<td>78213679</td>
<td>122232518</td>
<td>70906575</td>
</tr>
<tr>
<td>Diablo_264_vs_cbr</td>
<td>79.399942</td>
<td>12004</td>
<td>64.161044</td>
<td>5519</td>
<td>69.760989</td>
<td>3201</td>
<td>270099790</td>
<td>124188605</td>
<td>72029145</td>
</tr>
<tr>
<td>Diablo_264_vs_cbrcbr</td>
<td>78.043105</td>
<td>12004</td>
<td>63.753272</td>
<td>5488</td>
<td>69.280496</td>
<td>3201</td>
<td>270099749</td>
<td>123469217</td>
<td>72016828</td>
</tr>
<tr>
<td>Diablo_264_vs_ytrec</td>
<td>80.557112</td>
<td>12282</td>
<td>64.132311</td>
<td>5465</td>
<td>69.837804</td>
<td>3204</td>
<td>276350877</td>
<td>122972209</td>
<td>72093012</td>
</tr>
<tr>
<td>Diablo_265_s_15</td>
<td>97.950499</td>
<td>16153</td>
<td>72.790804</td>
<td>5330</td>
<td>74.668467</td>
<td>3121</td>
<td>363439250</td>
<td>119935055</td>
<td>70224019</td>
</tr>
<tr>
<td>Diablo_265_s_18</td>
<td>96.761145</td>
<td>10799</td>
<td>64.274843</td>
<td>5477</td>
<td>70.00497</td>
<td>3189</td>
<td>242971955</td>
<td>123228196</td>
<td>71743989</td>
</tr>
<tr>
<td>Diablo_265_s_1818</td>
<td>95.238331</td>
<td>10344</td>
<td>64.005861</td>
<td>5456</td>
<td>69.729306</td>
<td>3174</td>
<td>232734805</td>
<td>122763424</td>
<td>71425082</td>
</tr>
<tr>
<td>Diablo_265_s_23</td>
<td>93.006921</td>
<td>5392</td>
<td>63.979798</td>
<td>5468</td>
<td>69.619533</td>
<td>3160</td>
<td>121330646</td>
<td>123020259</td>
<td>71104048</td>
</tr>
<tr>
<td>Diablo_265_s_28</td>
<td>85.919885</td>
<td>2622</td>
<td>62.563314</td>
<td>5427</td>
<td>68.010864</td>
<td>3110</td>
<td>58984142</td>
<td>122100782</td>
<td>69969486</td>
</tr>
<tr>
<td>Diablo_ll</td>
<td>99.495108</td>
<td>211467</td>
<td>64.417506</td>
<td>5496</td>
<td>70.17594</td>
<td>3207</td>
<td>4758011796</td>
<td>123665492</td>
<td>72164165</td>
</tr>
<tr>
<td>Flower_264_uf_cbr</td>
<td>90.820921</td>
<td>12003</td>
<td>60.289722</td>
<td>5112</td>
<td>65.51911</td>
<td>2905</td>
<td>270074770</td>
<td>115020107</td>
<td>65365803</td>
</tr>
<tr>
<td>Flower_264_uf_cbrcbr</td>
<td>87.23414</td>
<td>12003</td>
<td>59.52696</td>
<td>5136</td>
<td>64.528292</td>
<td>2897</td>
<td>270076071</td>
<td>115567271</td>
<td>65189777</td>
</tr>
<tr>
<td>Flower_264_uf_ytrec</td>
<td>92.20385</td>
<td>12482</td>
<td>60.31722</td>
<td>5103</td>
<td>65.603094</td>
<td>2912</td>
<td>280851844</td>
<td>114812752</td>
<td>65522093</td>
</tr>
<tr>
<td>Flower_264_vs_15</td>
<td>98.033976</td>
<td>21657</td>
<td>61.476664</td>
<td>5142</td>
<td>67.105189</td>
<td>2930</td>
<td>487284073</td>
<td>115699067</td>
<td>65929188</td>
</tr>
<tr>
<td>Flower_264_vs_18</td>
<td>96.282884</td>
<td>14280</td>
<td>78.859306</td>
<td>5196</td>
<td>70.496349</td>
<td>2925</td>
<td>321303667</td>
<td>116912183</td>
<td>65811665</td>
</tr>
<tr>
<td>Flower_264_vs_23</td>
<td>90.309735</td>
<td>7280</td>
<td>60.627877</td>
<td>5145</td>
<td>66.016645</td>
<td>2907</td>
<td>163791603</td>
<td>115753544</td>
<td>65397501</td>
</tr>
<tr>
<td>Flower_264_vs_28</td>
<td>79.213158</td>
<td>3784</td>
<td>57.865338</td>
<td>5072</td>
<td>62.783757</td>
<td>2877</td>
<td>85132045</td>
<td>114109923</td>
<td>64737343</td>
</tr>
<tr>
<td>Flower_264_vs_cbr</td>
<td>93.089044</td>
<td>12003</td>
<td>61.070225</td>
<td>5127</td>
<td>66.610977</td>
<td>2920</td>
<td>270076943</td>
<td>115348193</td>
<td>65701206</td>
</tr>
<tr>
<td>Flower_264_vs_cbrcbr</td>
<td>90.262987</td>
<td>12003</td>
<td>60.685497</td>
<td>5119</td>
<td>66.097981</td>
<td>2913</td>
<td>270077386</td>
<td>115169337</td>
<td>65544217</td>
</tr>
<tr>
<td>Flower_264_vs_ytrec</td>
<td>94.192944</td>
<td>12412</td>
<td>61.119528</td>
<td>5132</td>
<td>66.688747</td>
<td>2927</td>
<td>279263084</td>
<td>115470547</td>
<td>65864494</td>
</tr>
<tr>
<td>Flower_265_s_15</td>
<td>97.890013</td>
<td>17416</td>
<td>61.383645</td>
<td>5135</td>
<td>67.058359</td>
<td>2921</td>
<td>391866568</td>
<td>115542709</td>
<td>65729248</td>
</tr>
<tr>
<td>Flower_265_s_18</td>
<td>96.238418</td>
<td>11387</td>
<td>61.241824</td>
<td>5118</td>
<td>66.91126</td>
<td>2916</td>
<td>256212430</td>
<td>115157382</td>
<td>65599187</td>
</tr>
<tr>
<td>Flower_265_s_1818</td>
<td>93.792025</td>
<td>10549</td>
<td>77.822758</td>
<td>5187</td>
<td>69.782112</td>
<td>2898</td>
<td>237347369</td>
<td>116700825</td>
<td>65214516</td>
</tr>
<tr>
<td>Flower_265_s_23</td>
<td>91.079349</td>
<td>5639</td>
<td>60.673012</td>
<td>5115</td>
<td>66.230594</td>
<td>2899</td>
<td>126871689</td>
<td>115078549</td>
<td>65231405</td>
</tr>
<tr>
<td>Flower_265_s_28</td>
<td>82.067773</td>
<td>2775</td>
<td>58.792273</td>
<td>5092</td>
<td>64.040091</td>
<td>2872</td>
<td>62440546</td>
<td>114562620</td>
<td>64612975</td>
</tr>
<tr>
<td>Flower_ll</td>
<td>99.523266</td>
<td>238585</td>
<td>61.560809</td>
<td>5142</td>
<td>67.273151</td>
<td>2939</td>
<td>5368162899</td>
<td>115706202</td>
<td>66135515</td>
</tr>
<tr>
<td>Forza_264_uf_cbr</td>
<td>64.729647</td>
<td>12004</td>
<td>50.81258</td>
<td>5440</td>
<td>60.342175</td>
<td>3279</td>
<td>270100444</td>
<td>122407182</td>
<td>73780840</td>
</tr>
<tr>
<td>Forza_264_uf_cbrcbr</td>
<td>61.770599</td>
<td>12004</td>
<td>50.810904</td>
<td>5725</td>
<td>53.410165</td>
<td>3228</td>
<td>270099542</td>
<td>128812514</td>
<td>72636182</td>
</tr>
<tr>
<td>Forza_264_uf_ytrec</td>
<td>66.114957</td>
<td>12195</td>
<td>51.122722</td>
<td>5423</td>
<td>60.825468</td>
<td>3280</td>
<td>274381014</td>
<td>122022612</td>
<td>73803256</td>
</tr>
<tr>
<td>Forza_264_vs_15</td>
<td>99.29691</td>
<td>48754</td>
<td>54.167064</td>
<td>5639</td>
<td>56.009519</td>
<td>3280</td>
<td>1096954002</td>
<td>126867156</td>
<td>73803418</td>
</tr>
<tr>
<td>Forza_264_vs_18</td>
<td>98.319477</td>
<td>32144</td>
<td>52.112966</td>
<td>5423</td>
<td>63.959207</td>
<td>3404</td>
<td>723248160</td>
<td>122018809</td>
<td>76589940</td>
</tr>
<tr>
<td>Forza_264_vs_23</td>
<td>92.537892</td>
<td>15131</td>
<td>51.751005</td>
<td>5385</td>
<td>63.348598</td>
<td>3408</td>
<td>340451509</td>
<td>121152707</td>
<td>76668786</td>
</tr>
<tr>
<td>Forza_264_vs_28</td>
<td>80.466659</td>
<td>6753</td>
<td>52.019913</td>
<td>5628</td>
<td>54.765883</td>
<td>3277</td>
<td>151935408</td>
<td>126640535</td>
<td>73723302</td>
</tr>
<tr>
<td>Forza_264_vs_cbr</td>
<td>66.84449</td>
<td>12005</td>
<td>51.670847</td>
<td>5398</td>
<td>62.687511</td>
<td>3441</td>
<td>270103243</td>
<td>121462450</td>
<td>77421072</td>
</tr>
<tr>
<td>Forza_264_vs_cbrcbr</td>
<td>64.543753</td>
<td>12007</td>
<td>51.390652</td>
<td>5408</td>
<td>61.763106</td>
<td>3434</td>
<td>270165096</td>
<td>121678591</td>
<td>77265584</td>
</tr>
<tr>
<td>Forza_264_vs_ytrec</td>
<td>68.470782</td>
<td>12083</td>
<td>51.676026</td>
<td>5423</td>
<td>62.87408</td>
<td>3452</td>
<td>271868686</td>
<td>122015814</td>
<td>77674164</td>
</tr>
<tr>
<td>Forza_265_s_15</td>
<td>99.097435</td>
<td>40023</td>
<td>52.005312</td>
<td>5391</td>
<td>64.009681</td>
<td>3400</td>
<td>900506342</td>
<td>121299280</td>
<td>76505252</td>
</tr>
<tr>
<td>Forza_265_s_18</td>
<td>97.810374</td>
<td>26326</td>
<td>52.031377</td>
<td>5401</td>
<td>63.894881</td>
<td>3399</td>
<td>592338478</td>
<td>121519709</td>
<td>76481172</td>
</tr>
<tr>
<td>Forza_265_s_1818</td>
<td>95.234687</td>
<td>24286</td>
<td>53.618469</td>
<td>5620</td>
<td>55.63266</td>
<td>3265</td>
<td>546438965</td>
<td>126458972</td>
<td>73458497</td>
</tr>
<tr>
<td>Forza_265_s_23</td>
<td>92.052077</td>
<td>12271</td>
<td>51.672386</td>
<td>5402</td>
<td>63.338261</td>
<td>3392</td>
<td>276093174</td>
<td>121541573</td>
<td>76321839</td>
</tr>
<tr>
<td>Forza_265_s_28</td>
<td>81.483807</td>
<td>5268</td>
<td>51.731833</td>
<td>5656</td>
<td>54.616867</td>
<td>3253</td>
<td>118523102</td>
<td>127252331</td>
<td>73187594</td>
</tr>
<tr>
<td>Forza_ll</td>
<td>99.731102</td>
<td>389333</td>
<td>54.373016</td>
<td>5654</td>
<td>56.147891</td>
<td>3285</td>
<td>8759995647</td>
<td>127216993</td>
<td>73909804</td>
</tr>
<tr>
<td>Goose_264_uf_cbr</td>
<td>82.44302</td>
<td>12004</td>
<td>75.741353</td>
<td>2585</td>
<td>85.238116</td>
<td>1318</td>
<td>270100006</td>
<td>58166935</td>
<td>29648678</td>
</tr>
<tr>
<td>Goose_264_uf_cbrcbr</td>
<td>82.060309</td>
<td>12004</td>
<td>75.574072</td>
<td>2584</td>
<td>85.035081</td>
<td>1326</td>
<td>270100653</td>
<td>58142795</td>
<td>29835382</td>
</tr>
<tr>
<td>Goose_264_uf_ytrec</td>
<td>82.679568</td>
<td>12506</td>
<td>75.854697</td>
<td>2593</td>
<td>85.376269</td>
<td>1314</td>
<td>281391131</td>
<td>58345790</td>
<td>29565474</td>
</tr>
<tr>
<td>Goose_264_vs_15</td>
<td>96.859109</td>
<td>4094</td>
<td>75.811661</td>
<td>2564</td>
<td>85.318066</td>
<td>1300</td>
<td>92104733</td>
<td>57696445</td>
<td>29245506</td>
</tr>
<tr>
<td>Goose_264_vs_18</td>
<td>96.063484</td>
<td>2618</td>
<td>75.558384</td>
<td>2537</td>
<td>85.026407</td>
<td>1292</td>
<td>58913753</td>
<td>57091305</td>
<td>29064982</td>
</tr>
<tr>
<td>Goose_264_vs_23</td>
<td>93.689916</td>
<td>1530</td>
<td>63.995765</td>
<td>2668</td>
<td>70.673336</td>
<td>1433</td>
<td>34426215</td>
<td>60019415</td>
<td>32239266</td>
</tr>
<tr>
<td>Goose_264_vs_28</td>
<td>89.087909</td>
<td>1033</td>
<td>62.342069</td>
<td>2634</td>
<td>68.754366</td>
<td>1417</td>
<td>23248165</td>
<td>59270440</td>
<td>31878260</td>
</tr>
<tr>
<td>Goose_264_vs_cbr</td>
<td>82.210953</td>
<td>12005</td>
<td>78.012663</td>
<td>2328</td>
<td>84.943394</td>
<td>1329</td>
<td>270103419</td>
<td>52390287</td>
<td>29894900</td>
</tr>
<tr>
<td>Goose_264_vs_cbrcbr</td>
<td>81.879387</td>
<td>12008</td>
<td>75.672767</td>
<td>2558</td>
<td>85.171845</td>
<td>1305</td>
<td>270170378</td>
<td>57552529</td>
<td>29366799</td>
</tr>
<tr>
<td>Goose_264_vs_ytrec</td>
<td>82.46231</td>
<td>12218</td>
<td>75.882753</td>
<td>2583</td>
<td>85.412708</td>
<td>1310</td>
<td>274902604</td>
<td>58108981</td>
<td>29481612</td>
</tr>
<tr>
<td>Goose_265_s_15</td>
<td>96.970247</td>
<td>2604</td>
<td>75.593086</td>
<td>2575</td>
<td>85.068361</td>
<td>1292</td>
<td>58583974</td>
<td>57936822</td>
<td>29071221</td>
</tr>
<tr>
<td>Goose_265_s_18</td>
<td>96.301997</td>
<td>1695</td>
<td>75.384589</td>
<td>2565</td>
<td>84.802974</td>
<td>1284</td>
<td>38133039</td>
<td>57709265</td>
<td>28893136</td>
</tr>
<tr>
<td>Goose_265_s_1818</td>
<td>95.157821</td>
<td>1915</td>
<td>76.78748</td>
<td>2228</td>
<td>83.586637</td>
<td>1285</td>
<td>43095519</td>
<td>50132383</td>
<td>28902174</td>
</tr>
<tr>
<td>Goose_265_s_23</td>
<td>94.421579</td>
<td>931</td>
<td>63.916998</td>
<td>2694</td>
<td>70.601952</td>
<td>1441</td>
<td>20939142</td>
<td>60614632</td>
<td>32417698</td>
</tr>
<tr>
<td>Goose_265_s_28</td>
<td>90.766738</td>
<td>572</td>
<td>74.276496</td>
<td>2254</td>
<td>80.978117</td>
<td>1276</td>
<td>12862051</td>
<td>50716476</td>
<td>28703598</td>
</tr>
<tr>
<td>Goose_ll</td>
<td>98.769961</td>
<td>189136</td>
<td>76.11744</td>
<td>2625</td>
<td>85.667461</td>
<td>1325</td>
<td>4255550684</td>
<td>59058303</td>
<td>29818338</td>
</tr>
<tr>
<td>Persona_264_uf_cbr</td>
<td>87.87474</td>
<td>8002</td>
<td>83.464212</td>
<td>2780</td>
<td>87.824627</td>
<td>1527</td>
<td>180040874</td>
<td>62560046</td>
<td>34347722</td>
</tr>
<tr>
<td>Persona_264_uf_cbrcbr</td>
<td>87.050873</td>
<td>8002</td>
<td>82.802116</td>
<td>2823</td>
<td>87.204416</td>
<td>1555</td>
<td>180040618</td>
<td>63523033</td>
<td>34995319</td>
</tr>
<tr>
<td>Persona_264_uf_ytrec</td>
<td>89.250364</td>
<td>8150</td>
<td>84.758816</td>
<td>2583</td>
<td>88.954543</td>
<td>1476</td>
<td>183376795</td>
<td>58106608</td>
<td>33213109</td>
</tr>
<tr>
<td>Persona_264_vs_15</td>
<td>98.071494</td>
<td>5589</td>
<td>85.021831</td>
<td>2562</td>
<td>89.209192</td>
<td>1467</td>
<td>125755818</td>
<td>57634444</td>
<td>33004785</td>
</tr>
<tr>
<td>Persona_264_vs_18</td>
<td>97.764895</td>
<td>3826</td>
<td>84.822397</td>
<td>2545</td>
<td>89.023112</td>
<td>1462</td>
<td>86079699</td>
<td>57259079</td>
<td>32895970</td>
</tr>
<tr>
<td>Persona_264_vs_23</td>
<td>96.791966</td>
<td>2084</td>
<td>76.347832</td>
<td>2452</td>
<td>80.029167</td>
<td>1400</td>
<td>46897687</td>
<td>55178369</td>
<td>31500447</td>
</tr>
<tr>
<td>Persona_264_vs_28</td>
<td>94.271684</td>
<td>1189</td>
<td>75.307659</td>
<td>2415</td>
<td>78.92975</td>
<td>1387</td>
<td>26749320</td>
<td>54338683</td>
<td>31205736</td>
</tr>
<tr>
<td>Persona_264_vs_cbr</td>
<td>89.112521</td>
<td>8002</td>
<td>84.798437</td>
<td>2575</td>
<td>89.017621</td>
<td>1478</td>
<td>180038928</td>
<td>57941045</td>
<td>33249331</td>
</tr>
<tr>
<td>Persona_264_vs_cbrcbr</td>
<td>88.70434</td>
<td>8002</td>
<td>84.544804</td>
<td>2585</td>
<td>88.769201</td>
<td>1492</td>
<td>180039239</td>
<td>58165181</td>
<td>33559401</td>
</tr>
<tr>
<td>Persona_264_vs_ytrec</td>
<td>89.441311</td>
<td>7988</td>
<td>84.956704</td>
<td>2576</td>
<td>89.161317</td>
<td>1472</td>
<td>179727730</td>
<td>57956970</td>
<td>33122168</td>
</tr>
<tr>
<td>Persona_265_s_15</td>
<td>97.951096</td>
<td>5515</td>
<td>84.842391</td>
<td>2534</td>
<td>89.056717</td>
<td>1458</td>
<td>124084162</td>
<td>57023780</td>
<td>32806520</td>
</tr>
<tr>
<td>Persona_265_s_18</td>
<td>97.644543</td>
<td>3768</td>
<td>84.632011</td>
<td>2517</td>
<td>88.865603</td>
<td>1446</td>
<td>84771267</td>
<td>56623656</td>
<td>32523764</td>
</tr>
<tr>
<td>Persona_265_s_1818</td>
<td>97.157507</td>
<td>3560</td>
<td>85.743572</td>
<td>2466</td>
<td>87.748083</td>
<td>1409</td>
<td>80099914</td>
<td>55491841</td>
<td>31696960</td>
</tr>
<tr>
<td>Persona_265_s_23</td>
<td>96.762617</td>
<td>2030</td>
<td>76.251858</td>
<td>2452</td>
<td>79.936432</td>
<td>1393</td>
<td>45679867</td>
<td>55168405</td>
<td>31341544</td>
</tr>
<tr>
<td>Persona_265_s_28</td>
<td>94.731936</td>
<td>1118</td>
<td>75.446565</td>
<td>2399</td>
<td>79.116052</td>
<td>1377</td>
<td>25146715</td>
<td>53969481</td>
<td>30991059</td>
</tr>
<tr>
<td>Persona_ll</td>
<td>98.573126</td>
<td>43857</td>
<td>85.245592</td>
<td>2607</td>
<td>89.414688</td>
<td>1473</td>
<td>986790988</td>
<td>58665095</td>
<td>33149132</td>
</tr>
<tr>
<td>RE7_264_uf_cbr</td>
<td>75.888887</td>
<td>12004</td>
<td>64.227227</td>
<td>4214</td>
<td>74.419716</td>
<td>2213</td>
<td>270100090</td>
<td>94807880</td>
<td>49790816</td>
</tr>
<tr>
<td>RE7_264_uf_cbrcbr</td>
<td>74.956536</td>
<td>12005</td>
<td>63.790475</td>
<td>4212</td>
<td>73.843212</td>
<td>2208</td>
<td>270106343</td>
<td>94760349</td>
<td>49672323</td>
</tr>
<tr>
<td>RE7_264_uf_ytrec</td>
<td>76.594269</td>
<td>11797</td>
<td>64.487254</td>
<td>4238</td>
<td>74.829886</td>
<td>2220</td>
<td>265428700</td>
<td>95364168</td>
<td>49950325</td>
</tr>
<tr>
<td>RE7_264_vs_15</td>
<td>97.105435</td>
<td>10552</td>
<td>64.780621</td>
<td>4257</td>
<td>75.20065</td>
<td>2214</td>
<td>237430576</td>
<td>95790898</td>
<td>49823799</td>
</tr>
<tr>
<td>RE7_264_vs_18</td>
<td>96.013602</td>
<td>6867</td>
<td>64.542449</td>
<td>4232</td>
<td>74.924611</td>
<td>2204</td>
<td>154499455</td>
<td>95213185</td>
<td>49586747</td>
</tr>
<tr>
<td>RE7_264_vs_23</td>
<td>92.261366</td>
<td>3589</td>
<td>63.415114</td>
<td>4097</td>
<td>73.529831</td>
<td>2160</td>
<td>80760447</td>
<td>92189174</td>
<td>48598981</td>
</tr>
<tr>
<td>RE7_264_vs_28</td>
<td>84.265748</td>
<td>2050</td>
<td>50.471717</td>
<td>3472</td>
<td>57.670474</td>
<td>1940</td>
<td>46131371</td>
<td>78122796</td>
<td>43646861</td>
</tr>
<tr>
<td>RE7_264_vs_cbr</td>
<td>76.056284</td>
<td>12006</td>
<td>64.662748</td>
<td>4251</td>
<td>75.006354</td>
<td>2209</td>
<td>270139875</td>
<td>95648587</td>
<td>49702661</td>
</tr>
<tr>
<td>RE7_264_vs_cbrcbr</td>
<td>75.264467</td>
<td>12008</td>
<td>64.335408</td>
<td>4221</td>
<td>74.613173</td>
<td>2201</td>
<td>270188428</td>
<td>94973634</td>
<td>49530895</td>
</tr>
<tr>
<td>RE7_264_vs_ytrec</td>
<td>76.679025</td>
<td>11783</td>
<td>64.79921</td>
<td>4277</td>
<td>75.185211</td>
<td>2220</td>
<td>265114062</td>
<td>96242045</td>
<td>49945085</td>
</tr>
<tr>
<td>RE7_265_s_15</td>
<td>97.120023</td>
<td>7661</td>
<td>64.637117</td>
<td>4264</td>
<td>75.060625</td>
<td>2211</td>
<td>172362662</td>
<td>95930665</td>
<td>49751101</td>
</tr>
<tr>
<td>RE7_265_s_18</td>
<td>96.152931</td>
<td>5012</td>
<td>64.376164</td>
<td>4227</td>
<td>74.762484</td>
<td>2204</td>
<td>112763295</td>
<td>95112842</td>
<td>49584275</td>
</tr>
<tr>
<td>RE7_265_s_1818</td>
<td>94.367464</td>
<td>5074</td>
<td>67.990275</td>
<td>3946</td>
<td>75.536917</td>
<td>2226</td>
<td>114161155</td>
<td>88784903</td>
<td>50088347</td>
</tr>
<tr>
<td>RE7_265_s_23</td>
<td>93.104052</td>
<td>2546</td>
<td>63.48773</td>
<td>4186</td>
<td>73.661428</td>
<td>2169</td>
<td>57289274</td>
<td>94189994</td>
<td>48794086</td>
</tr>
<tr>
<td>RE7_265_s_28</td>
<td>86.943132</td>
<td>1335</td>
<td>51.266253</td>
<td>3575</td>
<td>58.684662</td>
<td>1964</td>
<td>30033787</td>
<td>80427882</td>
<td>44194339</td>
</tr>
<tr>
<td>RE7_ll</td>
<td>99.259192</td>
<td>221580</td>
<td>69.8519</td>
<td>4034</td>
<td>77.311906</td>
<td>2287</td>
<td>4985544618</td>
<td>90764454</td>
<td>51446797</td>
</tr>
<tr>
<td>Rogue_264_uf_cbr</td>
<td>96.925727</td>
<td>12003</td>
<td>86.99517</td>
<td>5302</td>
<td>90.175933</td>
<td>2689</td>
<td>270077750</td>
<td>119284444</td>
<td>60507087</td>
</tr>
<tr>
<td>Rogue_264_uf_cbrcbr</td>
<td>96.030996</td>
<td>12003</td>
<td>85.911095</td>
<td>5435</td>
<td>89.050071</td>
<td>2741</td>
<td>270078488</td>
<td>122278877</td>
<td>61683684</td>
</tr>
<tr>
<td>Rogue_264_uf_ytrec</td>
<td>98.027889</td>
<td>12711</td>
<td>88.544829</td>
<td>5153</td>
<td>91.696659</td>
<td>2575</td>
<td>285996470</td>
<td>115933980</td>
<td>57933800</td>
</tr>
<tr>
<td>Rogue_264_vs_15</td>
<td>98.269626</td>
<td>4315</td>
<td>89.015472</td>
<td>5132</td>
<td>92.124669</td>
<td>2541</td>
<td>97084851</td>
<td>115478779</td>
<td>57177302</td>
</tr>
<tr>
<td>Rogue_264_vs_18</td>
<td>97.993974</td>
<td>3205</td>
<td>88.866763</td>
<td>5122</td>
<td>91.956588</td>
<td>2538</td>
<td>72106078</td>
<td>115234619</td>
<td>57115175</td>
</tr>
<tr>
<td>Rogue_264_vs_23</td>
<td>97.065343</td>
<td>1984</td>
<td>77.474442</td>
<td>4933</td>
<td>80.302464</td>
<td>2626</td>
<td>44633382</td>
<td>110985633</td>
<td>59081494</td>
</tr>
<tr>
<td>Rogue_264_vs_28</td>
<td>94.769673</td>
<td>1293</td>
<td>92.951124</td>
<td>3607</td>
<td>90.688775</td>
<td>2209</td>
<td>29097238</td>
<td>81155353</td>
<td>49696367</td>
</tr>
<tr>
<td>Rogue_264_vs_cbr</td>
<td>98.042362</td>
<td>12003</td>
<td>88.87322</td>
<td>5173</td>
<td>91.995923</td>
<td>2572</td>
<td>270076402</td>
<td>116399110</td>
<td>57871595</td>
</tr>
<tr>
<td>Rogue_264_vs_cbrcbr</td>
<td>97.708131</td>
<td>12003</td>
<td>88.541975</td>
<td>5217</td>
<td>91.678204</td>
<td>2588</td>
<td>270077320</td>
<td>117373447</td>
<td>58240322</td>
</tr>
<tr>
<td>Rogue_264_vs_ytrec</td>
<td>98.423263</td>
<td>12430</td>
<td>89.049659</td>
<td>5187</td>
<td>92.17192</td>
<td>2557</td>
<td>279668682</td>
<td>116710275</td>
<td>57540070</td>
</tr>
<tr>
<td>Rogue_265_s_15</td>
<td>98.020741</td>
<td>4181</td>
<td>88.762654</td>
<td>5099</td>
<td>91.878941</td>
<td>2530</td>
<td>94080135</td>
<td>114718029</td>
<td>56930200</td>
</tr>
<tr>
<td>Rogue_265_s_18</td>
<td>97.675072</td>
<td>3067</td>
<td>88.606968</td>
<td>5037</td>
<td>91.652086</td>
<td>2514</td>
<td>69006708</td>
<td>113343466</td>
<td>56554417</td>
</tr>
<tr>
<td>Rogue_265_s_1818</td>
<td>97.155745</td>
<td>2910</td>
<td>95.185235</td>
<td>3670</td>
<td>92.564231</td>
<td>2160</td>
<td>65471586</td>
<td>82578171</td>
<td>48596652</td>
</tr>
<tr>
<td>Rogue_265_s_23</td>
<td>96.665464</td>
<td>1851</td>
<td>77.045894</td>
<td>4903</td>
<td>79.874038</td>
<td>2612</td>
<td>41654962</td>
<td>110318033</td>
<td>58771227</td>
</tr>
<tr>
<td>Rogue_265_s_28</td>
<td>94.167452</td>
<td>1127</td>
<td>92.415327</td>
<td>3659</td>
<td>90.186826</td>
<td>2134</td>
<td>25364215</td>
<td>82326855</td>
<td>48019095</td>
</tr>
<tr>
<td>Rogue_ll</td>
<td>98.665875</td>
<td>33123</td>
<td>96.449077</td>
<td>3751</td>
<td>93.799547</td>
<td>2202</td>
<td>745274044</td>
<td>84406256</td>
<td>49539400</td>
</tr>
</tbody>
</table>
<h1 id="appendix-z-notes">Appendix Z Notes</h1>
<p>I suspect if I tried big bunny, a lot more of these results would have been expected, because that’s the sort of content these things are tested with, whereas, Detroit, Flower, and Untitled Goose Game are not like most animations or films that netflix or x264 testing will have come across. These more unusual visual experiences are part of the draw of modern games.</p>
<p>There is a channel dedicated to this article where you can find all the uploads featured within. <a href="https://www.youtube.com/channel/UCwXGFTOhuMJ1jQzOXRx3Fbg">https://www.youtube.com/channel/UCwXGFTOhuMJ1jQzOXRx3Fbg</a></p>
<p>Original videos are too large to share.</p>
<p>Our actual gaming channel is <a href="https://www.youtube.com/c/AndSoBegins">https://www.youtube.com/c/AndSoBegins</a></p>PeterGoodbye Windows 72020-01-18T18:55:00+00:002020-01-18T18:55:00+00:00https://andsobegins.com/blog/2020/01/18/goodbye-windows-7<p><img src="https://andsobegins.com/assets/images/Win7.webp" alt="Windows 7 Partition" /></p>
<p>It is with somewhat of a heavy heart that Windows 7 is finally being cleared out and the recovered space handed over to Windows 10. Windows 7 is just obsolete now and won’t get proper updates any more. More importantly for us is that Windows 10 is now running Windows 7-era games more competently than when it was a fresher OS. Not that we actually like Windows 10 more than 7 though. To be honest, Windows 10 has always felt somewhat incomplete, and not very friendly to Let’s Playing with its habit of wanting to update in the middle of a recording session. Since most games not on consoles are really made to run on Windows and a lot of the acceleration stuff works better in Windows 10 than Linux, we do stream, record and play games using Windows. For everything else, Linux.</p>
<p>It’s hard not to feel a little emotional for an environment that you’ve been using and helping you get so many unique experiences as a gamer.</p>
<p>For a Windows OS, Windows 7 was stable, light on bundled crap, and more so than any Windows OS after it, consistent.</p>
<p>Windows 7 joins Windows 98 and XP as one of the good ones. Here’s to you.</p>PeterPenumbra: Overture review2009-12-27T15:54:17+00:002009-12-27T15:54:17+00:00https://andsobegins.com/article/2009/12/27/penumbra-overture<p><img src="https://andsobegins.com/assets/images/penumbra_header.jpg" alt="Penumbra: Overture" /></p>
<p>One of my favourite games is Call of Cthulhu: Dark Corners of the Earth, I have a soft spot for games that are actually scary, that make you cringe in fear, make you dread what happens next, yet drive you forward by using attraction combined with revulsion. It isn’t easy to do, few games, or films for that matter have managed it. Arguably, games being interactive, they have the advantage, Silent Hills 1, 2, and 4 being good examples that aren’t Dark Corners of the Earth. Oh and System Shock 2.</p>
<p>Wait, that’s all the games that have managed it.</p>
<p><a href="https://www.youtube.com/watch?v=SEk8RjgaDQk">Penumbra: Overture Review - youtube.com</a></p>
<p>Until now. Or until 2007 at least, but I’ve played it now, and that’s what counts.</p>
<p>Penumbra: Overture is horror, not survival horror where zombies go boo, but actual just horror. Speaking of interactive, it really goes much further than most games now, System Shock 2 had an interesting idea where you can go into a cursor mode to give yourself a finer control over the manipulation of the game’s environment. It was a great idea, but seemed like a bit of a chore, and it has taken a relative newcomer to the industry to be able to bring it back, better than before. It is also entirely optional, so you don’t like it, well, you don’t have to use it. However, when you grasp a drawer in the game and drag it open with your mouse after grasping the front, it’s a joy and not a chore (it becomes a chore later since there are a <em>lot</em> of drawers), and you don’t need to switch to cursor mode to do it anyway. Doors work in much the same way, grasp with your cursor and then draw your mouse like opening a door, this may sound over the top, but in a horror game being able to control how quietly and slowly you open a door is a real treat. Occasionally it won’t work right and you’ll have to approach an object at a different angle, but most of the time it’s a welcome addition.</p>
<p><img src="https://andsobegins.com/assets/images/penumbra_Doorpull.jpg" alt="Doorpull" title="A door being opened into a room" />
<em>Doors are opened by click-dragging mouse gestures</em></p>
<p>With that said, it’ll take you a little while to get into it, unfortunately the game seems a little amateur, and could have really used with some better voice acting as the opening demonstrates. I mean, it isn’t terrible, but does detract from the presentation a little. You could argue that it’s refreshing to hear someone who is close to the age of the protagonist, someone who just sounds like an everyday person. Really though, it isn’t possible to make an everyperson, even when the main guy is silent, he still isn’t you. I would have been happier with the older guy with the cool voice from down the road, I don’t ask for much, but a horror game is about atmosphere, it is forgiveable and something they can work on in the future, the upcoming game for 2010 for example: Amnesia: The Dark Descent. I look forward to it.</p>
<p>So you take the role of Philip, really cynical English guy. Your dead father writes to you with some instructions to burn some papers he’s left you. Of course you don’t, and unlike most plots where characters do weird stuff for no reason, his opening actions are completely believable. He gets curious.</p>
<p>This is one of the many points I love about this game, for they have a good story, and this is what I think more developers should consider doing. Don’t make clones, don’t just make plot-less pablum for new platforms, you have an advantage over established game developers, you can take risks and do something with feeling and story, that’s where you can succeed where others fail, and Penumbra does this, not to say it’s hugely original, what is these days? We want to have fun on games, yes we could be original and write something new, but it probably won’t be fun, engaging, or entertaining, story-lines are game weaknesses though, a Fallout 3, or a Silent Hill 2 quality story is very rare indeed, so rare you don’t have to be even near that good to produce something worthwhile.</p>
<p>Philip’s curiosity (and cynicism, don’t forget that) lead him out to where his father tells him not to go, he was to destroy all evidence, but couldn’t. And now he’s in a part of the game with graphics. They aren’t bad, they have that flat look to them but really, that’s a small complaint, and the grainy filter helps with both graphics and atmosphere. The interactivity shines through immediately as you can grasp, pull, open lids all with gesturing with the mouse and moving. It might not sound like much at first, but you’ll get completely taken in all the more. As all horror stories, it starts off innocently enough, but soon gets creepy. Isolation is one of the games strengths as you open the game in desolate, uninhabited Greenland. Might as well get used to it, it’s an ever-present theme. The howling of the wind, the sometimes poignant, always excellent <a href="https://mikkotarmia.bandcamp.com/album/penumbra-soundtrack">music</a>, the lack of anything approaching human warmth drive home your (almost) complete desolation. This leaves you free to worry about the ‘other’.</p>
<p><img src="https://andsobegins.com/assets/images/penumbra_Isolation.jpg" alt="Isolated Greenland" title="A snowy view" />
<em>A snowy view</em></p>
<p>You soon enough, after an introductory section, get stuck with no way back in an abandoned structure with only a few meagre possessions, including the obligatory flashlight. This flashlight is a little crappy however, as Philip comments, and eats batteries like there’s no tomorrow (which there might not be anyhow), so you also have a glowstick, and the difference is big enough that you want the batteries, whilst the flashlight’s bright cone lights up the darkest corners of the earth, the glowstick’s ghoulish glow doesn’t reach very far leading you blindly searching out the edges of the room. Of course, they’re both wickedly atmospheric. There are also other light sources which are optional and it’s quite nice to not have everything you pick up be of vital importance, they’re in there purely for atmosphere, like the flare.</p>
<p>The scares come fast and hard and you get the nagging suspicion that the Cthulhu Mythos solution is the most sensible, just kill yourself now, get it over and done with whilst your sanity remains intact. That would take all the fun out of it though. Whilst the game isn’t actually set in the Mythos, it does borrow heavily from it in mood. Strange bangs in the distance and sometimes in front of your face come just infrequently enough for exhaustion to be avoided, tight areas give you a very uncomfortable sense of claustrophobia. It isn’t all good news, the combat in the game, whilst extremely rare is far too awkward and some chase sequences are tremendously difficult, with the annoyance that some enemies must be avoided too many times. The sheer thrill of discovering more about your surroundings compensates for some thoroughly frustrating times, and the sometimes slog is worth it. The sneaking system helps, when you crouch in the darkness, you become much harder for the enemies to see, and crouching still is even better, you’re compensated the lack of light sources (the enemies would see you) by your eyes natural adjustment to the darkness represented by a blue tinged field of view change. Looking too much at the whatever-the-hell-it-is stalking you will make you panic and stand, alerting it to your presence. Mind you, the moment when an enemy sniffs you out and detects you should be experienced at least once. Being chased into a new area you don’t know spotting dead ends is intensely exhilarating.</p>
<p><img src="https://andsobegins.com/assets/images/penumbra_BlueTinged.jpg" alt="BlueTinged" title="Very dark blue-tinged corridor shot" />
<em>Very dark blue-tinged corridor shot</em></p>
<p>What horror game worth its salt doesn’t have great iconic moments though? Silent Hill 2’s Pyramid Head sequences, the moment when Shodan reveals herself, or the section in Dark Corners of the Earth when you’re being pursued through the hotel. This game unfortunately does not reach the heights of those games, but it does do very, very well, there’s a certain scene with an incinerator which is just unforgettable.</p>
<p>If you’ve been pining for a successor to Dark Corners of the Earth or lamenting for a game to tide you over until Shattered Memories, get this.</p>PeterS.T.A.L.K.E.R.: Shadow of Chernobyl review2009-08-26T23:02:54+00:002009-08-26T23:02:54+00:00https://andsobegins.com/article/2009/08/26/s-t-a-l-k-e-r-shadow-of-chernobyl<p><img src="https://andsobegins.com/assets/images/stalker_header.jpg" alt="S.T.A.L.K.E.R.: Shadow of Chernobyl" /></p>
<p><a href="https://www.youtube.com/watch?v=ibCLrEP3OZ4">Stalker: Shadow of Chernobyl Review - youtube.com</a></p>
<p>This game is quite highly rated, it is also somewhat suspiciously rated. The ratings given vary widely, there is good reason for this.</p>
<p>It is irredeemably awful.</p>
<p>A little history, I have no idea of this game’s history, I don’t really care if it comes from a previous successful and good game, I don’t particularly care if it comes from a protracted period of development hell, the game must stand on its own without history, and the game falls flat on its stupid face. It features an absolutely dreadful tutorial which tells you almost nothing useful that wasn’t already intuitive. I don’t really need an explanation of how a map works, or how the obvious buttons do the obvious things, what I really wanted to know is what I press to get the PDA to pop up on screen, what the hot keys turn out to be to get it to go to each section with a key press. Some other useful information should have also been provided, for example that the game has an inventory system. That’s right, it forgets to mention that the game has a system for managing what you carry. They may have glossed over it because it’s rather poor, it looks and handles like poo, and is an unpleasant throwback to amateuristic 1990s RPGs. I should probably mention now that this game isn’t really an RPG, nor is it an FPS, it combines both in new-found and awkward ways.</p>
<p><img src="https://andsobegins.com/assets/images/stalker_Inventory.jpg" alt="Mostly empty inventory screen" title="The secret inventory screen" />
<em>The secret inventory screen</em></p>
<p>For example, I suspect that at some point in its development, it was decided that while this game would have a single-player mode, it would be mostly designed as a multi-player experience, then it was decided after a lot of groundwork was completed that the multi-player sucked and single-player was the way to go. Never before in a game have I been talking to someone only for them to notice a dog three hundred miles away and stop mid-conversation to go and stand in the corner of a building facing in the general direction of it. This is not good. I’m playing single player, can we just pause it, and don’t give me any crap about realism. The behaviour it actually does isn’t exactly realistic is it? Oh and the mid-conversational bit you were trying to read (yes read, most conversations aren’t voiced), that’s lost forever of course.</p>
<p>Anyway, the game.</p>
<p>It’s set in a world where there was another Chernobyl disaster, you are the “Marked One”, so called because you have a tattoo, which must therefore be an incredibly rare thing for Stalkers to have. Actually you have the acronym S.T.A.L.K.E.R. tattooed on your arm, this is pretty stupid as this is his occupation. You don’t often find people with their occupations tattooed to their forearm, anyway, after clicking new game we get treated to a pretty cool opening movie where almost nothing is explained, but that’s okay because you’re supposed to be an unknown element, presumably all the story I missed by quitting and exiting in disgust is the Marked One’s journey of self-discovery where they discover they grew up in an agrarian paradise with no radiation or something. So you get rescued from a crashed truck by a bloke who knows a guy who can get money for dead people. After the guy vigorously eats chicken leg he asks you to work for him since you “owe him”. After a brief tutorial mentioned above and a conversation where he moves his lips for the first half of the voiced conversation, you emerge into the world after wondering whether or not you are supposed to notice that the mission in your PDA is Kill Strelok when the opening movie’s had Kill The Strelok and if that’ll be a huge ‘surprise’ plot point later in the game.</p>
<p><img src="https://andsobegins.com/assets/images/stalker_ChickenMan.jpg" alt="Russian man sitting behind a desk" title="After the guy vigorously eats chicken leg" />
<em>After the guy vigorously eats chicken leg</em></p>
<p>You’re now in the land of 1990’s graphics where the poly-count of the other actors in the game are low enough that you wonder why the dog-mutant thing has trapezoidal legs. Almost immediately you notice the lag in movement and controls so you go back to the graphics menu to wonder if it’s the graphics settings being too high despite the age of the game. Holy crap it is! The game is intensely inefficient graphically and should work better than this at full settings with much crappier hardware. Especially since the graphics are so bad. This is not a pretty game at all, and looks dreadful even comparing it to much older games. Not to mention that artistically it’s very bland too with the boring browns of modern shooters. You’re given a bunch of tasks where you must travel at broken neck speeds around the allegedly open-plan world (that’s no fast travel, you go everywhere on foot, and with the added bonus that lots of the areas require loading and going through a single narrow point you can’t go around).</p>
<p><img src="https://andsobegins.com/assets/images/stalker_OpenWorld.jpg" alt="House behind wooded field" title="Stalker's open(ish) world" />
<em>Stalker’s open(ish) world</em></p>
<p>Then you die. A lot. The weapons are pathetic and it feels like you’re shooting people with a squirt gun for all the death-dealing they do. Oh except when people shoot at you. Then they’re in possession of … well, guns. Yes, another game has tried replacing game-play with difficulty. It’s really hard, yes, even on easy. Well done. Except that no-one will be able to play it without modification. And it still isn’t good.</p>PeterPrince of Persia review2008-12-29T10:24:07+00:002008-12-29T10:24:07+00:00https://andsobegins.com/article/2008/12/29/prince-of-persia<p><img src="https://andsobegins.com/assets/images/princeofpersia_header.jpg" alt="Prince of Persia" /></p>
<p>Dull.</p>Peter