# Magic formula to create h264/ac3/mp4?



## orangeboy (Apr 19, 2004)

Over the years, I've collected a variety of different videos: xvid, divx, mpeg2, etc. Is there a way to determine the optimal bitrate to re-encode into TiVo friendly h264/ac3/mp4, given the source codec(s) and bitrate(s)?

For example:
if vcodec = xvid, h264 bitrate = xvid bitrate x .66
if vcodec = divx, h264 bitrate = divx bitrate x .72
if acodec = mp3, ac3 bitrate = mp3 bitrate x .5

Is that what 2-pass encoding does? I've done searches and found tools to re-encode whatever to mp4, but there always seems to be a field for "target" bitrate. I don't want to set a target bitrate of say, 5000kbps when the source is xvid @3000kbps. I don't want to set anything really, I just want "whatever" to figure it out on it's own. Target file size isn't a big factor at the moment, but I would like to benefit from smaller file sizes with the more efficient codecs.

On another topic, what's the best way to change h264 "[email protected]" to h264 "[email protected]"? Doing the hex edit method (not unexpectedly) does NOT make the video play on my Premiere. Just re-encode using ffmpeg?

I have no clue about trellis/cabac/me/subme/etc. settings, but I think I can get all that from mediainfo, and reuse those settings - good idea or not?

Thanks!


----------



## moyekj (Jan 24, 2006)

I don't think there is a magic formula. Constant quality encoding (-crf option with ffmpeg, -q option with handbrake) gives predictable output quality for 1-pass encodes but unpredictable file sizes. Theoretically for constant quality the bit rate is adjusted as needed to maintain constant quality. In practice it may not always work out that way visually but is easier than going with potentially overkill fixed bit rates. If you don't really care about file sizes it may be a good starting point to explore.


----------



## moyekj (Jan 24, 2006)

P.S. I still find mpeg2 is the best format for playback on TiVos. The trick play functions work best with mpeg2 (1xFF doesn't seem to work very well with H.264 for example, and I don't know if aspect ratio weirdness is still an issue with H.264 but at least with series 3 they can be depending on resolution). Hence just letting pyTivo transcode on the fly if necessary is still the easiest option regardless of source format (including Blu Ray m2ts). i.e. If you don't care about file sizes mpeg2 program stream is still the best format to generate for TiVo playback, especially now that Premiere units are fast enough to receive mpeg2 HD formats at better than real time speeds. Previously if you wanted to push HD video to Series 3 TiVos at reasonable speeds you really had not much choice but to use H.264 with AC3, but that's no longer necessary for Premiere units.


----------



## txporter (Sep 18, 2006)

orangeboy said:


> Over the years, I've collected a variety of different videos: xvid, divx, mpeg2, etc. Is there a way to determine the optimal bitrate to re-encode into TiVo friendly h264/ac3/mp4, given the source codec(s) and bitrate(s)?
> 
> For example:
> if vcodec = xvid, h264 bitrate = xvid bitrate x .66
> ...


I agree with what Kevin posted above. If you have a bunch of video with multiple formats, just use CRF based encoding. You will not get a known bitrate, but quality will be good.

Also, I agree that MPEG2 does still play back best on premiere. I find that h.264 playback is satisfactory on the premiere though and hopefully they can continue to improve it.

What do you mean by using a hex editor to update the header info? Have you tried using this to remux your video and update the header info? I doubt that you have 1080p video at 100+ fps that actually NEEDS that level.

As for other settings, I would download a copy of a recent x264 build from x264.nl and direct a copy of the help file into a text document (x264 --fullhelp). Look through the presets. I would recommend using those rather than trying to come up with your own set of encoding parameters. I find the default preset quality with CRF 19 for SD and CRF 21 for HD to be sufficient and pretty fast for transcoding.


----------



## larrs (May 2, 2005)

In Handbrake (at least the newest ver. 9.5), you can just set the CRF to 1 and get a perfect copy. It may change the bitrate slightly, but I would think that is only a function of going from one format to the other.


----------



## fyodor (Sep 19, 2006)

I concur heartily.

Transcoding to h.264 just isn't worth the time and effort if you can transfer with realtime transcoding fast enough to watch. I've wasted too many hours playing with codecs for no real benefit. If takes up more space on the Tivo, but I'd rather just store it off the Tivo and transfer as needed.



moyekj said:


> P.S. I still find mpeg2 is the best format for playback on TiVos. The trick play functions work best with mpeg2 (1xFF doesn't seem to work very well with H.264 for example, and I don't know if aspect ratio weirdness is still an issue with H.264 but at least with series 3 they can be depending on resolution). Hence just letting pyTivo transcode on the fly if necessary is still the easiest option regardless of source format (including Blu Ray m2ts). i.e. If you don't care about file sizes mpeg2 program stream is still the best format to generate for TiVo playback, especially now that Premiere units are fast enough to receive mpeg2 HD formats at better than real time speeds. Previously if you wanted to push HD video to Series 3 TiVos at reasonable speeds you really had not much choice but to use H.264 with AC3, but that's no longer necessary for Premiere units.


----------



## orangeboy (Apr 19, 2004)

Thanks guys - lots of good info here.

Regarding the Hex Edit stuff, here's a link to a tool that changes the reported level: http://forum.videohelp.com/threads/...-profile-4-1?p=2045621&viewfull=1#post2045621

Basically at the beginning of the file, these bytes get changed:
67 4D 40 33
changes to:
67 4D 40 29


----------



## jcthorne (Jan 28, 2002)

fyodor said:


> I concur heartily.
> 
> Transcoding to h.264 just isn't worth the time and effort if you can transfer with realtime transcoding fast enough to watch. I've wasted too many hours playing with codecs for no real benefit. If takes up more space on the Tivo, but I'd rather just store it off the Tivo and transfer as needed.


But since most HD material is in h264 already, there usually is no transcoding, just muxing into the correctly formatted container. Unless you are suggesting transcoding HD material FROM h264 TO mpeg2.

For DVD sourced SD material, I agree that transcoding to h264 is mostly a waste of time these days.


----------



## orangeboy (Apr 19, 2004)

Hmm... This is a neat page: http://mirror05.x264.nl/Dark/website/compare.html
h264 comparisons against other various codecs at the same bitrate. Click the codec names below the image to see how the encoding looks!


----------



## Dan203 (Apr 17, 2000)

I just finished coding a quality factor mode into VideoReDo. It uses a combination of the source bitrate, frame rate and resolution to calculate a quality factor. It then increases/reduces that factor based on a user set conversion percentage. Then recalculates the output bitrate using the adjusted QF and the output resolution and frame rate.

I literally just finished coding it about an hour ago, but so far my tests have produced very good results. Provided I don't run into any major issues it should be in the next beta of TVSuite v4.

Dan


----------



## moyekj (Jan 24, 2006)

Dan203 said:


> I just finished coding a quality factor mode into VideoReDo. It uses a combination of the source bitrate, frame rate and resolution to calculate a quality factor. It then increases/reduces that factor based on a user set conversion percentage. Then recalculates the output bitrate using the adjusted QF and the output resolution and frame rate.
> 
> I literally just finished coding it about an hour ago, but so far my tests have produced very good results. Provided I don't run into any major issues it should be in the next beta of TVSuite v4.
> 
> Dan


 Sounds good Dan, looking forwards to checking it out.


----------



## dlfl (Jul 6, 2006)

jcthorne said:


> But since most HD material is in h264 already, ......


Hmmmm.... All Cable and OTA broadcast in the USA (and captured by TiVo or tuner cards) is mpeg2. What source(s) furnish "most HD material" in your viewpoint? Are you thinking of the 800 lb gorilla that many people use but don't often mention, initials BT?


----------



## jcthorne (Jan 28, 2002)

No, I think he was referring to BR (BluRay) but it would apply to net sourced video as well.


----------



## Dan203 (Apr 17, 2000)

There are actually a lot of BluRays that use VC-1 instead of H.264. So if your plan is to convert everything to H.264 you'll need to watch out for that.

Dan


----------



## wmcbrine (Aug 2, 2003)

VC-1 is also pushable, although the TiVo is even pickier about it than about MP4.


----------



## larrs (May 2, 2005)

fyodor said:


> I concur heartily.
> 
> Transcoding to h.264 just isn't worth the time and effort if you can transfer with realtime transcoding fast enough to watch. I've wasted too many hours playing with codecs for no real benefit. If takes up more space on the Tivo, but I'd rather just store it off the Tivo and transfer as needed.


Gosh I kow this to be true. I have at least 40 hours of tinkering and finally have something, but it is still not perfect. Tivos via tivo desktop do not like mp4s with 5.1 audio, so you have to use streambaby or pyTivo (both excellent programs as attested all over the boards) as well as figuring out the best encoding method.


----------



## lrhorer (Aug 31, 2003)

moyekj said:


> especially now that Premiere units are fast enough to receive mpeg2 HD formats at better than real time speeds.


 I'm not sure why you say that. S3 and even THD TiVos are perfectly capable of real-time transfer of HD programming.



moyekj said:


> Previously if you wanted to push HD video to Series 3 TiVos at reasonable speeds you really had not much choice but to use H.264 with AC3


That's just not true. I have only ever had trouble transferring at better than real-timewith the very highest resolution 1080i material (~20 Mbps) or with 720p material on my S3 TiVos. Even my THD doesn't have any problem transferring at real time for most HD content, especially if the tuners are not active. It's true the Premier is fast enough to transfer all HD material regardless of the type or the activity of the unit, while the S3 and especially the THD are much closer to the ragged edge, but I almost never get a pause when watching a movie straight off the server.


----------



## moyekj (Jan 24, 2006)

You always reported faster transfers than most, for both MRV and TTG (perhaps PROM hacked units or some other optimization). For HDXL unit when I had one TiVo<->PC transfers were ~ 10 Mbps, for my S3s ~ 13 Mbps under normal circumstances (both tuners buffering HD channels). It's not just me either - the speeds reported in the TivoHD FAQ are right about the same as I was seeing on average, so I consider those typical speeds to expect.


FAQ said:


> 12. How fast can I download TiVo recordings to my computer?
> 
> As with MRV, the more you are doing on the DVR, the slower recordings will download to your PC / Mac. For example, if you are recording two different HD programs while watching a third, previously recorded HD program, download speeds will be up to 50% slower than if you were simply watching live TV with both tuners set to SD channels.
> 
> ...


Yes you could play some games and tune to music channels or channels you don't subscribe, but then I was still lucky to get > 15 Mbps which is marginal for high quality mpeg2. With the Premiere it's easy to get 30+ Mbps for mpeg2, 45+ Mbps for H.264 under normal conditions so it's not a concern any more. I'm not too fond of the Premiere I have compared to my S3 OLED unit which I love, but the Premiere is much faster at transfers and does a much better job playing H.264/AC3. Other than that I prefer my S3 OLED still for almost everything else. The HDXL I only had when participating in a hardware beta and was never fond of that slow unit.


----------



## fyodor (Sep 19, 2006)

lrhorer said:


> I'm not sure why you say that. S3 and even THD TiVos are perfectly capable of real-time transfer of HD programming.
> 
> That's just not true. I have only ever had trouble transferring at better than real-timewith the very highest resolution 1080i material (~20 Mbps) or with 720p material on my S3 TiVos. Even my THD doesn't have any problem transferring at real time for most HD content, especially if the tuners are not active. It's true the Premier is fast enough to transfer all HD material regardless of the type or the activity of the unit, while the S3 and especially the THD are much closer to the ragged edge, but I almost never get a pause when watching a movie straight off the server.


Are your systems hacked? On my S3s I was never able to get realtime transfer of 1080i mpeg-2 material even from locally networked machines. It always capped out around 10-12 megabits/s.


----------



## jcthorne (Jan 28, 2002)

fyodor said:


> Are your systems hacked? On my S3s I was never able to get realtime transfer of 1080i mpeg-2 material even from locally networked machines. It always capped out around 10-12 megabits/s.


Same for me at about 11 for my TivoHD. The Premiere has no trouble with streaming mpeg2 1080i video though. Still don't think storing all hd on my server as mpeg2 is the way to go. Just too wasteful of space and the conversion is more onerous than remuxing to mp4 for most of my hd content. I have actually only come across a few VC-1 BR disks and those I do recode to h264. Most already are.


----------



## lrhorer (Aug 31, 2003)

moyekj said:


> You always reported faster transfers than most, for both MRV and TTG (perhaps PROM hacked units or some other optimization).


Nope, as I reported to you previously, the results I posted were from a 100% unhacked S3 TiVo. The only modification is a hard drive upgrade (to a 1.5TB WD "green" drive):










Although lower, the numbers to / from my THD were also reasonable. Note I do have two hacked TiVos, but neither is employing back-ported network drivers or any such utilities. I am not, however, using TiVo Desktop. The transfers are via pyTivo or Galleon. Note also when I actually do transfer programs from the hacked TiVos to the server, I ordinarily do not use TTG (which is what is measured above), but rather tserver which does indeed result in a significant gain in throughput: usually 45 - 52 Mbps. Jamie has achieved 112 Mbps using a USB / Gig-E adapter and back-ported USB drivers. For watching the video, though, it's pure pyTivo using the stock network drivers.



moyekj said:


> For HDXL unit when I had one TiVo<->PC transfers were ~ 10 Mbps, for my S3s ~ 13 Mbps under normal circumstances (both tuners buffering HD channels).


The tuners being active does indeed impact systems resources and transfer speed, but I have never seen numbers quite that low. That said, unless the TiVo is recording something about which I care, then those resources would be merely wasted, so there is no major down side to disabling the tuners. Of course, as I mentioned, it is actually necessary for 720p content and for some - but by no means all - 1080i content, so I do it habitually. It only takes a moment.



moyekj said:


> It's not just me either - the speeds reported in the TivoHD FAQ are right about the same as I was seeing on average, so I consider those typical speeds to expect.


I can't really speak to that. None of my three TiVos has performance that poor.



moyekj said:


> Yes you could play some games and tune to music channels or channels you don't subscribe, but then I was still lucky to get > 15 Mbps which is marginal for high quality mpeg2.


All I can do is refer you to the above and repeat that I very rarely ever encounter any pauses except with 720p material.



moyekj said:


> With the Premiere it's easy to get 30+ Mbps for mpeg2, 45+ Mbps for H.264 under normal conditions so it's not a concern any more. I'm not too fond of the Premiere I have compared to my S3 OLED unit which I love, but the Premiere is much faster at transfers and does a much better job playing H.264/AC3.


That, OTOH, is true. It is the only reason I might like to have a Premier, but its other drawbacks override the desire, even if I were in the market for a TiVo at this time, which I am not.



moyekj said:


> Other than that I prefer my S3 OLED still for almost everything else. The HDXL I only had when participating in a hardware beta and was never fond of that slow unit.


It's implementation dependent. The THD is fine for its purpose where it sits in my house. It would not be acceptable in the Theater. I could live with one in the living room, but that unit happens to be an S3, as well.


----------



## lrhorer (Aug 31, 2003)

fyodor said:


> Are your systems hacked?


See above. In the context of this discussion, the answer is, "No".



fyodor said:


> On my S3s I was never able to get realtime transfer of 1080i mpeg-2 material even from locally networked machines. It always capped out around 10-12 megabits/s.


Well, first of all, a lot of 1080i material averages about 12 mbps. For example, picking a few titles at random, I have the following:

Creator 9,725,818,880 bytes 6453.18 seconds => 12,057,086 bps

Demolition Man 9,583,634,432 bytes 6895.27 seconds => 11,119,082 bps

Good Will Hunting 11,424,301,056 bytes 7592.11 seconds => 12,038,077 bps

Judgment at Nuremberg 24,342,040,576 bytes 10,709.29 seconds => 18,183,869 bps

Lawrence of Arabia 27,191,769,088 bytes 12,985.05 seconds => 16,752,661 bps

October Sky 5,310,377,984 bytes 6177.15 seconds => 6,877,477 bps

Practical Magic 7,306,844,160 bytes 6252.08 seconds => 9,349,649 bps.

While You Were Sleeping 9,309,399,040 bytes 6193.18 seconds => 12,025,355 bps

With no tuners active, I experience no pauses watching any of these programs in the theater on an S3 Tivo, excepting possibly Judgment at Nuremberg. Allowing 2 or 3 minutes of buffer prior to starting the movie will alleviate even that. If the tuners were not disabled, I might experience pauses with Lawrence of Arabia, although even then only a 2 or 3 minute pause at the beginning will allow the remainder of the movie to pass through without further pauses.

OTOH, material such as Planet Earth: The Great Plains which averages 17.5 Mbps will very likely produce pauses unless I allow a healthy buffer to accumulate before starting the program. Of course, since the program itself is only 42 minutes, a 1 or 2 minute buffer is sufficient.

Of course, to engender maximal performance, one must not have any significant bottlenecks in the network. All of the switches are capable of 1000 Mbps, and the server employs a 3.0 GHz dual core AMD Athlon processor running Linux, with a RAID array capable of delivering more than 2500 Mbps. The Tivo is most certainly the bottleneck.


----------



## lrhorer (Aug 31, 2003)

jcthorne said:


> Same for me at about 11 for my TivoHD. The Premiere has no trouble with streaming mpeg2 1080i video though.


I just ran two tests on my TiVo HD. The first was a short (5 minute) MPEG2 video transfer from the server to the THD. It squirted across at 14.0 Mbps. I then began transferring and immediately watching The Milagro Beanfield War, which averages 12.02 Mbps. 'No pauses, and indeed the transfer is out-pacing the display by considerably more than 10%. After 21 minutes, more than 25 minutes of video have transferred. Of course the tuners are disabled. It's nothing like the speed of which the server or the network is capable, but it's a bit better than real time for this (rather typical) 1080i HD video.


----------



## jcthorne (Jan 28, 2002)

I do not disable the tuners, nor would the wife do so if I was brain dead enough to ask. The tivo takes care of it self. If the vid will not transfer and play straight from the menu without a bunch of mucking around, then it does not work. For us though, its a moot point. Very little hd is stored in mpeg2 format for us nor am I about to expand a film to that format for the tivo to stream it.

For now I'll stick with pushing the vids to the tivo and storing them there.


----------



## lrhorer (Aug 31, 2003)

jcthorne said:


> I do not disable the tuners, nor would the wife do so if I was brain dead enough to ask.


Brain dead? We're not talking about overhauling the Space Shuttle, here, just checking the tuners to make sure they are not unnecessarily recording. In fact, after selecting the program to transfer, the 10 - 20 seconds it takes to make sure the tuners are not recording anything help to insure the user won't get any pauses during playback for marginal videos.



jcthorne said:


> The tivo takes care of it self.


Unfortunately, in this instance, not. Actually, it would not be a bad idea to allow the user to set up live buffer disabling so that whenever the user is watching a program, the live buffers are automatically disabled. I'm not holding my breath for this feature, though.



jcthorne said:


> If the vid will not transfer and play straight from the menu without a bunch of mucking around, then it does not work.


That's ridiculous. It's a minor limitation, and one very easily circumvented. It certainly comes nowhere near being "a bunch of mucking around." I'm about as lazy as a person can get, but even I am nowhere nearly that lazy.



jcthorne said:


> For us though, its a moot point. Very little hd is stored in mpeg2 format for us nor am I about to expand a film to that format for the tivo to stream it.


Stream? As in "streambaby"? You've got to be joking. Unless they have worked around the pauses inherent in streaming using streambaby, said pauses are far more disruptive and annoying than taking a few seconds to make sure the tuners are disabled prior to watching a movie.

OTOH, if you are actually talking about transferring, especially via pyTiVo Pulls, then there is a good chance you actually are "expanding" the files. PyTivo transcodes on the fly.



jcthorne said:


> For now I'll stick with pushing the vids to the tivo and storing them there.


That's not the best option if one has more than one TiVo. It's not an option at all when one has upwards of 10 Terabytes of programs served among three TiVos.


----------



## jcthorne (Jan 28, 2002)

In fairness, I do a bit of both. Current movies and tv shows are pushed to the tivo. This is 95&#37; + of what is watched on the tivo. Works perfectly every time from the user perspective. I'll do whatever it takes behind the scenes to make the tivo interface 'just work' Asking my wife, mother or house guest to remember to disable the live tuners before watching a movie (which would also require the tivo to have nothing scheduled to record) is not an option. Just does not work.

Archive video, music and photos are all stored on the server and remote accessed by the tivo. Pulling a vid does result in tivo needing to buffer for a few minutes but its seldom used. if pulling of h264 tivo files is eventually able to be implemented this all goes away. The speed for pulling a vid that requires transcoding is limited by the processor in the file server as it cannot keep up with the premiere in feeding data while transcoding on the fly.

No, we do not use streambaby for the very reasons you stated.


----------

