# mp4 transcoding appears to be eliminated



## larry99

Starting this week I have seen another jump in speed for mp4 transfers since Desktop version 2.7 came out. Transfer time has been reduced from about 50 minutes to about 5 minutes! File size of the transferred file is the same as the original mp4. And [bit rate] *transfer rate to DVR* has about doubled to 2 mbs. Is this part of the Desktop 2.7 change that is just now being implemented? Has TIVO finally elminated mp4 transcoding?


----------



## moyekj

That would be great news if so. If you transfer from Tivo back to PC again what do you get (and what size is it)?


----------



## larry99

I can't transfer back because of the TIVO copyright restriction. Lets me transfer from PC to DVR but not back from DVR to PC.


----------



## NA9D

larry99 said:


> I can't transfer back because of the TIVO copyright restriction. Lets me transfer from PC to DVR but not back from DVR to PC.


That's odd. Why would something you transferred in be flagged as copyrighted?


----------



## gonzotek

larry99 said:


> Starting this week I have seen another jump in speed for mp4 transfers since Desktop version 2.7 came out. Transfer time has been reduced from about 50 minutes to about 5 minutes! File size of the transferred file is the same as the original mp4. And bit rate has about doubled to 2 mbs. Is this part of the Desktop 2.7 change that is just now being implemented? Has TIVO finally elminated mp4 transcoding?


What's the software version on the TiVo?
Messages & Settings > Account & System Information > System Information

Latest Software versions by model:
http://www.tivo.com/setupandsupport...tyhelp/TiVo_software_version_information.html

There's been at least one report in the forums of a 11.0b "Spring 2009" release. I wonder if the new software is a pre-req for mp4-native transfers?


----------



## larry99

Version on my S3 DVR; 11.0-01-2-652. TIVO website shows 11.0......On copyright, everything has some copyright even if its free to download. Why TIVO lets copyrighted material transfer one way and not the other, I have no idea. The particular program I used this time was Democracy Now for Feb 18 th. But I have encountered this problem before.


----------



## moyekj

larry99, do you have a link to an mp4 file which exhibits this behavior by any chance?
If not, can you post detailed specs of an mp4 file which is doing this? Ideally if you can use something like mediainfo to collect all specs on the file that would be great. Thanks!


----------



## berkshires

larry99 said:


> File size of the transferred file is the same as the original mp4. And bit rate has about doubled to 2 mbs.


That isn't physically possible

Look deeper, you may have a problem you don't realize yet.


----------



## larry99

Try mp4 download of this link via bittorrent or miro;
http://www.democracynow.org/2009/2/18/stream

Berkshires: Which isn't possible, the 2 mbs upload speed or the file being the same size? The mp4 file size is 441 mb, the DVR shows 0.44 Gb after transfer. I watched the download via Sygate on my PC and it averaged about 2 mbs. That's about twice what I have experienced before over my wireless g network with TIVO. I didn't time the upload exactly, but it was definetly under 15 minutes for a 1 hour video. I did it for two mp4s including the link above. I have watched about half of that one. Will watch the rest tommorow and post if there are any problems.

Mediainfo: MPEG-4 441 MiB 59mn 5s, video stream AVC, audio stream AAC. English, 900 Kbps, 640*480(4/3) at 29.970 fps, AVC


----------



## moyekj

I'm not home so can't easily try transfer to Tivo yet, but here's detailed specs on the file:
(In summary it's h.264 video with 2 channel AAC audio in mp4 container)


Code:


General

Complete name                    : C:\home\Demnow-DemocracyNowWednesdayFebruary182009386.mp4

Format                           : MPEG-4

Format profile                   : Base Media / Version 2

Codec ID                         : mp42

File size                        : 95.3 MiB

Duration                         : 59mn 5s

Overall bit rate                 : 225 Kbps

Encoded date                     : UTC 2009-02-17 13:06:59

Tagged date                      : UTC 2009-02-17 13:06:59



Video

ID                               : 201

Format                           : AVC

Format/Info                      : Advanced Video Codec

Format profile                   : [email protected]

Format settings, CABAC           : No

Format settings, ReFrames        : 6 frames

Codec ID                         : avc1

Codec ID/Info                    : Advanced Video Coding

Duration                         : 59mn 4s

Bit rate mode                    : Variable

Bit rate                         : 120 Kbps

Maximum bit rate                 : 1 045 Kbps

Width                            : 320 pixels

Height                           : 240 pixels

Display aspect ratio             : 4/3

Frame rate mode                  : Constant

Frame rate                       : 14.985 fps

Original frame rate              : 25.000 fps

Resolution                       : 24 bits

Colorimetry                      : 4:2:0

Scan type                        : Progressive

Bits/(Pixel*Frame)               : 0.104

Stream size                      : 50.7 MiB (53&#37;)

Writing library                  : x264 core 59

Encoding settings                : cabac=0 / ref=6 / deblock=1:0:0 / analyse=0x1:0x111 / me=umh / subme=6 / brdo=0 / mixed_ref=0 / me_range=16 / chroma_me=1 / trellis=0 / 8x8dct=0 / cqm=0 / deadzone=21,11 / chroma_qp_offset=0 / threads=12 / nr=0 / decimate=1 / mbaff=0 / bframes=0 / keyint=90 / keyint_min=15 / scenecut=40(pre) / rc=crf / crf=26.0 / rceq='blurCplx^(1-qComp)' / qcomp=1.00 / qpmin=10 / qpmax=51 / qpstep=4 / ip_ratio=1.40 / aq=2:1.00

Encoded date                     : UTC 2009-02-17 13:06:59

Tagged date                      : UTC 2009-02-17 13:07:01



Audio

ID                               : 101

Format                           : AAC

Format/Info                      : Advanced Audio Codec

Format version                   : Version 4

Format profile                   : LC

Format settings, SBR             : No

Codec ID                         : 40

Duration                         : 59mn 5s

Bit rate mode                    : Variable

Bit rate                         : 104 Kbps

Maximum bit rate                 : 158 Kbps

Channel(s)                       : 2 channels

Channel positions                : L R

Sampling rate                    : 32.0 KHz

Resolution                       : 16 bits

Stream size                      : 43.8 MiB (46%)

Encoded date                     : UTC 2009-02-17 13:07:00

Tagged date                      : UTC 2009-02-17 13:07:01


----------



## moyekj

It just occurred to me that one may need to have Tivo Desktop Plus to transfer mp4 files from PC->Tivo using Tivo Desktop, so I may not be able to try this out (certainly not willing to spend the money on Plus). So if someone else out there with Plus can try this out and get the same results as larry99 that would be greatly appreciated.
(If mp4 is indeed being transferred and stored natively on Tivo then that capability should be possible to enable in pyTivo).


----------



## moyekj

larry99, I should be able to generate a short test clip with similar encoding profile that is DRM free that should be possible to transfer back from Tivo to PC. Will post a URL for it after I get a chance to generate one and if you could try it out that would be great.
Come to think of it, the issue may not be DRM at all but rather Tivo doesn't have a TTG mechanism for transferring mp4 files from Tivo->PC


----------



## larry99

Moyekj: Your parameters are different from mine. There are two mp4 versions on the link. I am using the Bittorrent. Its 640*480 and 900 Kbs. I will do more testing tommorow. Is there a way to attach a jpg? I could then show you exactly what I have.


----------



## moyekj

Just for grins can you try the following encoding I just generated? It's a short 1 minute or so clip. (I don't think the exact bit rate etc. is as important as the video and audio types. As long as that matches the torrent version then should be sufficient for our testing purposes).
http://tivostream.googlecode.com/files/test.mp4


----------



## Allanon

Moyekj, I tested your file using Tivo Desktop Plus 2.7 and here are the results:

File size on computer is 13.7MB

*Transfer From Computer to Tivo HD:*
File size on Tivo reads 0.050GB
File Size in Tivo Desktop Plus reads 60MB

*Transfer From Tivo To Computer:*
File Size on Computer is 52MB

I will test the video larry99 linked to and report back.


----------



## moyekj

Thanks Allanon. Going by the size looks like the test clip is being transcoded to mpeg2. If you decrypt the .TiVo file transferred back and look at the specs you can determine for sure if that's the case.


----------



## fyodor

I'd love to give this a try. How do you check file sizes on the Tivo?

F



Allanon said:


> Moyekj, I tested your file using Tivo Desktop Plus 2.7 and here are the results:
> 
> File size on computer is 13.7MB
> 
> *Transfer From Computer to Tivo HD:*
> File size on Tivo reads 0.050GB
> File Size in Tivo Desktop Plus reads 60MB
> 
> *Transfer From Tivo To Computer:*
> File Size on Computer is 52MB
> 
> I will test the video larry99 linked to and report back.


----------



## berkshires

larry99 said:


> Transfer time has been reduced from about 50 minutes to about 5 minutes! File size of the transferred file is the same as the original mp4. And bit rate has about doubled to 2 mbs.


I see - contextually your 2mbs refered to the bit rate of the video, not the transfer speed, but you meant the transfer speed.

In that case, '50min to 5min' and 'double transfer speed' don't make sense. Either way, all three statements can't be simultaneously correct.


----------



## fyodor

Why? As I understand him, he is transferring the MP4 file directly instead of transcoding it to a larger MPEG-2 file and transferring that to his Tivo. So, if he's getting twice the transfer rate with substantially smaller file, he could be seeing 1/10 the transfer time.



berkshires said:


> I see - contextually your 2mbs refered to the bit rate of the video, not the transfer speed, but you meant the transfer speed.
> 
> In that case, '50min to 5min' and 'double transfer speed' don't make sense. Either way, all three statements can't be simultaneously correct.


----------



## moyekj

fyodor said:


> I'd love to give this a try. How do you check file sizes on the Tivo?
> 
> F


 From Now Playing List "right" into a show then click "info" to see extended info about the show. The file size is listed towards the bottom of the screen (for some shows you have to page down to see it).


----------



## Allanon

I downloaded the video using bit torrent and transfered it to the Tivo using Tivo Desktop Plus 2.7:

File size on computer is 440MB
File size on Tivo is 2.66GB

My conclusion is it's being re-encoded.


----------



## moyekj

Allanon said:


> I downloaded the video using bit torrent and transfered it to the Tivo using Tivo Desktop Plus 2.7:
> 
> File size on computer is 440MB
> File size on Tivo is 2.66GB
> 
> My conclusion is it's being re-encoded.


 Thanks for posting and sorry to hear it. Not sure what's going on with larry99's setup then... BTW does it show up as copy protected for transferring back to PC? (I would guess not).
From a previous post looks like larry99 is using a THD (model 652). Did you try with THD, S3 or HDXL unit? (Not that it probably matters but just for completeness would be good to know).


----------



## fyodor

I got the same results as Allanon when I tried transferring the test movie (requesting from the Tivo). The downloaded video was 3.26 gb from a 440MB movie.

Annoyed that larry99 had wasted my time with his foolish foolery, I set some other mp4s (that I was planning to watch) to auto-transfer, expecting to watch them after the slow transcoding process.

I came downstairs to find the first transfer nearly halfway done. Checking the size, it was the same size as the source file. I tried setting some other mp4s to auto-transfer-same result. Super fast with a Tivo file that is about the same size as the source file (not exactly because of the way the Tivo measures gigabytes).

I tested both ways with a few other files. When files are requested from the Tivo, it transcodes. When it's set to auto-transfer from Tivo Desktop, it transfers in mp4 format.

There are some very slight differences in appearance too between the native mp4 files and the transcoded mpeg-2 files.



moyekj said:


> Thanks for posting and sorry to hear it. Not sure what's going on with larry99's setup then... BTW does it show up as copy protected for transferring back to PC? (I would guess not).
> From a previous post looks like larry99 is using a THD (model 652). Did you try with THD, S3 or HDXL unit? (Not that it probably matters but just for completeness would be good to know).


----------



## Allanon

I did an auto transfer using Tivo Desktop Plus 2.7 with the test file moyekj posted and got these results:

File size on computer is 13.7MB

*Transfer From Computer to Tivo HD using Auto Transfer:*
File size on Tivo reads 0.01GB
File Size in Tivo Desktop Plus reads 20MB

The file is marked as copy protected so I couldn't transfer the file back to the computer. Also it took less than 30 seconds to transfer the file to the Tivo. I have a feeling auto transfer uses the same protocol as the Tivo subscriptions because those are also copy protected when transfered to the Tivo.


----------



## moyekj

Very interesting indeed! Thanks for the discovery guys and especially thanks to larry99 for 1st posting about it!
NOTE: When streaming mp4 files to Tivo the transfer speeds are quite a bit higher than mpeg2 so that jives well with what you guys are seeing as well.

The real exciting part about this is that HD downloads to Tivo in H.264 format now seems to be a reality. HD mpeg2 internet downloads to Tivo were never really viable because of large file sizes but now this opens up the door for HD internet downloads. I'll have to try generating some different encodings such as H.264 video with multi-channel AC3 audio to see if that works as well... May have to go ahead and fork up the money for TD Plus to investigate more easily. It's too bad there doesn't seem to be a way to get these non-mpeg2 videos back off the Tivo again.

One more thing to check please is to look at the Tivo Now Playing List XML for 1 of these shows to see if there is anything interesting there. i.e. (replace IP below with your Tivo IP)


Code:


https://192.168.1.101/TiVoConnect?Command=QueryContainer&Container=&#37;2FNowPlaying&Recurse=Yes
login=tivo
password=MAK

Do ContentType & SourceFormat will still show video/x-raw-tivo-tts for these?

Now the hard part of how to emulate this with something like pyTivo...


----------



## Allanon

Here is the information, the title is the name of the folder I used for auto transfers.



Code:


- <Item>
   - <Details>
      <ContentType>video/x-tivo-raw-tts</ContentType> 
      <SourceFormat>video/x-tivo-raw-tts</SourceFormat> 
      <Title>Testing</Title> 
      <CopyProtected>Yes</CopyProtected> 
      <SourceSize>20971520</SourceSize> 
      <Duration>71000</Duration> 
      <CaptureDate>0x499D051E</CaptureDate> 
      <EpisodeTitle>test</EpisodeTitle> 
      <Description>Transferred by TiVo Desktop Copyright Tribune Media Services, Inc.</Description> 
      <HighDefinition>Yes</HighDefinition> 
      <ProgramId>BS15000101</ProgramId> 
      <SeriesId>BS1315814499</SeriesId> 
      <ByteOffset>0</ByteOffset> 
      <RecordingQuality>255</RecordingQuality> 
   </Details>
   - <Links>
      - <Content>
         <Url>http://192.168.0.10:80/download/Testing.TiVo?Container=&#37;2FNowPlaying&id=1480724</Url> 
         <ContentType>video/x-tivo-raw-tts</ContentType> 
         <Available>No</Available> 
      </Content>
      - <TiVoVideoDetails>
         <Url>https://192.168.0.10:443/TiVoVideoDetails?id=1480724</Url> 
         <ContentType>text/xml</ContentType> 
         <AcceptsParams>No</AcceptsParams> 
      </TiVoVideoDetails>
   </Links>
</Item>




> May have to go ahead and fork up the money for TD Plus to investigate more easily.


Doesn't pyTivo have the push capabilities that is used by Tivo Desktop Plus auto transfers? Might try your test files to see what happens.

http://pytivo.krkeegan.com/new-development-auto-transfer-t677.html


----------



## wmcbrine

Allanon said:


> Doesn't pyTivo have the push capabilities that is used by Tivo Desktop Plus auto transfers?


Yes, but it doesn't yet know about pushing MP4 without transcoding it.


----------



## larry99

I am glad some people have been able to duplicate. I may not have time until Friday to do more testing myself. Notes:

- Up until this week the mp4s were being transcoded. The file size on the DVR was over 3 GB, about 5 times the mpg2 size. If you take twice the transfer speed and reduce the file size by a factor of 5 you do get a 10 times increase. Hence 50 minutes to 5 minutes makes sense.

- There are both auto and manual modes available for the transfer from PC to TIVO. Is this the auto mode that is being talked about in prior posts? (As opposed to the auto transfer of files from DVR to PC.) I will test in both modes as soon as I get a chance. (I think I was using auto-transfer from PC to DVR last week as well as this week. So this would not explain why the improvement this week.)

- The Bps I referred to is as seen in the Sygate firewall main menu. 

- My TIVO is the HD one selling for $299.99 on the TIVO website.

- The one change I made this week was the installation of VideoReDO. Is it possible that had some effect?

- Do we know if TIVO made any software changes subsequent to release of Desktop 2.7?


----------



## berkshires

larry99 said:


> - Up until this week the mp4s were being transcoded. The file size on the DVR was over 3 GB, about 5 times the mpg2 size. If you take twice the transfer speed and reduce the file size by a factor of 5 you do get a 10 times increase. Hence 50 minutes to 5 minutes makes sense.
> 
> - Do we know if TIVO made any software changes subsequent to release of Desktop 2.7?


Now your numbers make sense, although a 5X increase from mpeg4 to mpeg2 seems like quite a waste - not that that had anything to do with you.

Its perfectly likely its been there a while, at least from the release of 2.7 (and 11.x) and you just noticed it first. You don't happen to have the 11.x beta update I received last week on my S3, do you?


----------



## larry99

See prior post in this thread (*DVR Version *, 3:39 PM Wednesday) for my software version. How do I know if its a beta?


----------



## fyodor

I think that this scales somewhat depending on the source MP4 quality. I was seeing 3.5x for higher resolution mp4s, while the test file that larry99 linked to was about 6x. I'm wondering if Tivo Desktop doesn't do some upscaling for lower resolution videos.

F



berkshires said:


> Now your numbers make sense, although a 5X increase from mpeg4 to mpeg2 seems like quite a waste - not that that had anything to do with you.
> 
> Its perfectly likely its been there a while, at least from the release of 2.7 (and 11.x) and you just noticed it first. You don't happen to have the 11.x beta update I received last week on my S3, do you?


----------



## fyodor

moyekj said:


> Very interesting indeed! Thanks for the discovery guys and especially thanks to larry99 for 1st posting about it!
> NOTE: When streaming mp4 files to Tivo the transfer speeds are quite a bit higher than mpeg2 so that jives well with what you guys are seeing as well.


I was unfortunately too excited by Larry's discovery and forgot to check the transfer speed in the Tivo network diagnostics, but my qualitative impression was that it didn't seem to be quite as fast as the buffer-fills I was seeing with Streambaby. The speed improvements were more consistent with similar transfer speeds but a smaller file. I'm using a pair of MOCA adapters.

I'm also not sure why Larry would be seeing higher transfer speeds on a wireless G network. My understanding is that even with the added slowdown that comes with transferring MPEG-2 files back to the Tivo, that it's more than fast enough to max out a wireless G network.

Anyway, I wasn't monitoring that carefully so I may be mistaken. I'll try to get some transfer stats tonight.


----------



## fyodor

I'll add that I'd love to see a KMTTG feature that returns a generated MP4 file to the Tivo.
F



moyekj said:


> The real exciting part about this is that HD downloads to Tivo in H.264 format now seems to be a reality. HD mpeg2 internet downloads to Tivo were never really viable because of large file sizes but now this opens up the door for HD internet downloads. I'll have to try generating some different encodings such as H.264 video with multi-channel AC3 audio to see if that works as well... May have to go ahead and fork up the money for TD Plus to investigate more easily. It's too bad there doesn't seem to be a way to get these non-mpeg2 videos back off the Tivo again.
> 
> Now the hard part of how to emulate this with something like pyTivo...


----------



## berkshires

fyodor said:


> I think that this scales somewhat depending on the source MP4 quality. I was seeing 3.5x for higher resolution mp4s, while the test file that larry99 linked to was about 6x. I'm wondering if Tivo Desktop doesn't do some upscaling for lower resolution videos.
> 
> F


Not having actually used this stuff myself, I'd say that with the DT you can expect TiVo will make the mpeg2 transcodes either 352,480 or 720 x 480i60. So you can do the math from your mp4 source as to the increase in pixels per second.

As far as the increase in transfer bit rate: are we dealing with a slow computer that transcoded too slowly?


----------



## sinanju

My old S3 seems to be in on the party. Pulling the test clip down taxed the transcoder service for a while and resulted in a file reported by the desktop to be 55 MB on the TiVo. Pushing it down was in a wink resulted in a 15MB file.


----------



## berkshires

Never mind the last reference to a DT. Larry has an S3 and fy hasn't said I think.

Anyway, try that Select-Play-Select-BackSkip-Select thing that posts the video format on the bottom of the screen.


----------



## fyodor

I'm using an S3.

The key sequence you gave gives the resolution and framerate but doesn't say anything about the file format.

Regarding download speeds, the Tivo network diagnostics are showing download speeds of about 18.5 Mbit/s for the MP4 transfers as compared to 11.7 Mbit/s for my previous TTCB transfers. I am using a MoCA (coax) connection so both are still slower than the network link would allow.



berkshires said:


> Never mind the last reference to a DT. Larry has an S3 and fy hasn't said I think.
> 
> Anyway, try that Select-Play-Select-BackSkip-Select thing that posts the video format on the bottom of the screen.


----------



## moyekj

Well I broke down and got a TD+ key and setup for auto transfers for experimenting. Did some network sniffing on auto transfer of my short H.264 test clip. There's a lot of encrypted handshaking going on so not much to be deciphered, but this is my Tivo retrieving the mp4 clip from my PC. Note the *?Format=video/mp4* modifier. (I intentionally x'd out last 8 digits of tsn)


Code:


192.168.1.107 (Tivo) -> 192.168.1.110 (PC)	HTTP
---------------------------------------------------
GET /&#37;7BA118900A-B075-4E62-98C1-3822C57439A2%7D/%7BACCBA1D2-83C0-4FF6-96F4-6B9DE4D0B9E2%7D?Format=video%2Fmp4 HTTP/1.1
Host: 192.168.1.110:8080
Range: bytes=0-
User-Agent: TvHttpClient
tsn: 6480001xxxxxxxx
Connection: close

I'm going to try some other formats such as mpeg2 & wmv w/ vc-1 video just to note the format modifiers for those (though I think they will be obvious) and perhaps h.264 with multi-channel AC3 audio just to confirm that is possible natively as well.

How exactly the Tivo is determining the type I still don't know for sure...

EDIT:
* mpeg2 transfers happen without any format modifier

* h.264 with AC3 audio is not accepted by Tivo Desktop and won't transfer (which is pretty silly since for streaming purposes that works just fine). I guess whatever codecs Tivo Desktop is using aren't setup to handle mp4 with AC3 audio which is a real shame.

* wmv files do auto transfer but are transcoded to mpeg2

* I can't seem to get .TiVo files to auto transfer for some reason...

So this is not looking as hopeful as I originally thought.


----------



## moyekj

mp4 container with HD H.264 video & 6 channel AAC audio auto transfers natively to Tivo as well and plays fine. So looks like any mp4 file with H.264 video & AAC audio works fine. Haven't tried with mpeg audio but I would guess that works fine as well.

NOTE: For all these mp4 files the moov atom needs to be at front of file, so one has to use an encoder that does this automatically or run qt-faststart on the encoding before attempting to auto transfer to Tivo. If you don't do this Tivo will try to auto transfer and fail with message "This program was not downloaded onto this DVR because the downloaded file size does not match the expected file size"


----------



## berkshires

moyekj you are awesome. Thorough in your approach.

fy - with your transfers converted to mpeg2, how does the resolution/framerate and file size compare to the original mp4? with the ones that remain mp4, does the resolution/framerate or file size change any?

What happens if you transfer an mp4 with a resolution/framerate not typical of the ATSC standards?


----------



## PaulS

berkshires said:


> moyekj you are awesome. Thorough in your approach.


+1

Is it possible to see any differences in the requests being sent by the TiVo between HMO "pushes" versus "pulls" ? I'm thinking of how this seemingly new TiVo feature applies to pyTivo...

Thanks again for all the hard work.


----------



## ozdoc

I've just reinstalled TiVo Desktop Plus 2.7 and noticed this in the release notes:

This version has the following improvements in TiVo Desktop Plus:

	Videos from Flip camcorders can be viewed on your DVR.
	You can select a portable device conversion when defining an automatic transfer. 
*	TiVo Desktop supports transferring some files in the MKV format to your DVR.*
	TiVo Desktop may recognize additional video types based on other software installed on your system.
	TiVo recordings that were broadcast as movies will be added to the Movies category when added to iTunes


----------



## sinanju

It appears there are some geometry glitches. There are some HD H.264 movie trailers here: http://www.h264info.com/clips.html

If you transfer the Serenity (1280x720p) clip, it looks great. If you download any of the clips with a wider aspect ratio, like the Silver Surfer (1280x544,720p) clip, it appears (on my S3, at least... can't speak for HDs) that the TiVo guesses wrong about the ratio.. after some display glitches, playback starts but the TiVo claims the clip is 4:3. The result is that only the left half of the picture is shown. So, Panavision-type aspects aren't being handled correctly.


----------



## moyekj

PaulS said:


> Is it possible to see any differences in the requests being sent by the TiVo between HMO "pushes" versus "pulls" ? I'm thinking of how this seemingly new TiVo feature applies to pyTivo...


 The same mp4 test clip that transfers natively with auto push looks like this during a pull. Note the *?Format=video/x-tivo-mpeg* tag which is what is causing it to be converted to mpeg2 by Tivo Desktop before transfer. My guess is if the tag was video/mp4 instead as in the case of auto pushing that no conversion would happen.


Code:


GET /TiVoConnect/TivoNowPlaying/&#37;3EIauto_done.lnk%3CItest_h264_aac2.mp4?Format=video%2Fx-tivo-mpeg HTTP/1.1
Host: 192.168.1.110:8080
Range: bytes=0-
User-Agent: TvHttpClient
tsn: 6480001xxxxxxxx
Connection: close

So I think it's pretty clear that pulls are using the classic HMO protocol while auto pushes are using a different protocol presumably similar to how tivocasts are handled.

pyTivo acting as the file host of course is perfectly capable of ignoring the video/x-tivo-mpeg tag and transferring mp4 files natively. Whether that actually works or not we will have to find out. To be safe that experiment should be done with a known good mp4 that transfers natively to Tivo via auto push. My test clip given in this thread is such a case.
I'm hoping one of the pyTivo developers can give this a shot?


----------



## kearygriffin

moyekj said:


> ..
> How exactly the Tivo is determining the type I still don't know for sure...
> ..


I haven't done much pyTivo hacking, and I am not really even sure how to use the webvideo portion of pyTivo (the portion that I believe is able to do a "push" of a video using some kind of XMPP communications), but there is a line in mind.py (which is what communicates using your tivo.com username and password):

'encodingType': 'mpeg2ProgramStream',

I imagine if we can figure out what encodingType should be for an MP4, it will cause the TiVo to make the request for a mp4 as outlined above.

I believe from some discussions I saw on various forums about deciphering TiVo's XMPP protocol, the TiVo service does not (did not) require a specific SSL certificate, so it should be possible to do something like the following to intercept the traffic. (I'm saying this like it is easy, but I'm not 100% sure how to set something like this up). Hijack the requests from TivoDesktop (by spoofing the DNS name more than likely) and redirecting requests to our server:

TiVoDesktop(SSL) --> InterceptingProxy (SSL) --> tivo.com communications

So the TivoDesktop is actually talking to our proxy so we can see the data, and the proxy is passing along the data to tivo.com (and vice versa) I believe a tool like this will do it:
http://grinder.sourceforge.net/g3/tcpproxy.html#ssl

I may give this a shot this weekend, but I'm not sure I will have the time. (I have some MP4's that streambaby doesn't like and that could end up eating most of the weekend)


----------



## kearygriffin

moyekj said:


> pyTivo acting as the file host of course is perfectly capable of ignoring the video/x-tivo-mpeg tag and transferring mp4 files natively. Whether that actually works or not we will have to find out. To be safe that experiment should be done with a known good mp4 that transfers natively to Tivo via auto push.


Just gave this a quick shot using the demnow video clip from the very original post (which I had transferred succesfully using auto-push last night).

Didn't seem to work. I made changes to pyTivo to not transcode, and changed Content-Type to video/mp4. The TiVo attempts to download it without complaint, but you end up with an "empty' video.

Note: It probably makes sense for someone who has a little more familiarity with the pyTivo code to give this a go also, as I was guessing what I needed to do.


----------



## moyekj

kearygriffin said:


> Just gave this a quick shot using the demnow video clip from the very original post (which I had transferred succesfully using auto-push last night).
> 
> Didn't seem to work. I made changes to pyTivo to not transcode, and changed Content-Type to video/mp4. The TiVo attempts to download it without complaint, but you end up with an "empty' video.
> 
> Note: It probably makes sense for someone who has a little more familiarity with the pyTivo code to give this a go also, as I was guessing what I needed to do.


 Interesting, seems to be similar to issue posted here for auto-transfers of mp4 podcasts which broke in TD 2.7:
http://www.tivocommunity.com/tivo-vb/showthread.php?t=418732
Does the file size indicated by Tivo Info button approximately match the source mp4 file size?


----------



## fyodor

moyekj said:


> So I think it's pretty clear that pulls are using the classic HMO protocol while auto pushes are using a different protocol presumably similar to how tivocasts are handled.


Yeah, if you check the download in the transfer daignostics, it actually shows up as an internet download, not a PC-to-Tivo transfer.


----------



## Allanon

ozdoc said:


> *	TiVo Desktop supports transferring some files in the MKV format to your DVR.*


I tried pushing a MKV file and it does transfer but it also gets transcoded so you lose the speed and size benefit of a MP4 file.


----------



## moyekj

kearygriffin said:


> I haven't done much pyTivo hacking, and I am not really even sure how to use the webvideo portion of pyTivo (the portion that I believe is able to do a "push" of a video using some kind of XMPP communications), but there is a line in mind.py (which is what communicates using your tivo.com username and password):
> 
> 'encodingType': 'mpeg2ProgramStream',
> 
> I imagine if we can figure out what encodingType should be for an MP4, it will cause the TiVo to make the request for a mp4 as outlined above.
> 
> I believe from some discussions I saw on various forums about deciphering TiVo's XMPP protocol, the TiVo service does not (did not) require a specific SSL certificate, so it should be possible to do something like the following to intercept the traffic. (I'm saying this like it is easy, but I'm not 100% sure how to set something like this up). Hijack the requests from TivoDesktop (by spoofing the DNS name more than likely) and redirecting requests to our server:
> 
> TiVoDesktop(SSL) --> InterceptingProxy (SSL) --> tivo.com communications
> 
> So the TivoDesktop is actually talking to our proxy so we can see the data, and the proxy is passing along the data to tivo.com (and vice versa) I believe a tool like this will do it:
> http://grinder.sourceforge.net/g3/tcpproxy.html#ssl
> 
> I may give this a shot this weekend, but I'm not sure I will have the time. (I have some MP4's that streambaby doesn't like and that could end up eating most of the weekend)


 Searching through my packet dump for interactions with mind.tivo.com (204.176.49.65) unfortunately nothing in ASCII is revealed (it's all encrypted) so it looks like that may be the only viable approach to be able to snoop the data and have a chance at finding the right push request.


----------



## moyekj

Actually we should be able to use mind.py to get information about a pending mp4 push. Something like:
- Add 'encodingType' to NEEDED_VALUES in getDownloadRequests procedure.
- Add following to bottom of __init__ procedure:
s = self.getDownloadRequests()
print s
- prepare a check.py script which simply does:
import mind
m = mind.getMind()
- In Tivo Desktop auto push folder drop in your mp4
- Run the check.py script repeatedly until something shows up and we can get the 'encodingType' information

NOTE: I know very little about python so the details may not be quite right but hopefully something along these lines...


----------



## moyekj

Well the mind.py idea didn't yield anything so I'm not sure that Tivo Desktop is using the same method as pyTivo for scheduling pushes via mind.tivo.com. Hopefully the pyTivo experts and comment and/or experiment further.


----------



## wmcbrine

mind.py was based on reverse-engineering TiVo Desktop. I don't think there's any other method.

armooo said he used "a simple MITM attack" to intercept the SSL communications. But he didn't give details, and rest of us have not found it quite as simple to reproduce.


----------



## justintime

Doing this isn't too hard, once you get all the right tools.

I'm a little short on time this weekend, otherwise I'd volunteer to setup some captures. If someone can do it this weekend, great, if not, let me know and I'll setup a box in front of my tivo and start doing some dumps.

Basically, the easiest place to start is with the "BackTrack" linux live CD. Setup your Linux box with two nics with a loopback cable from eth0 to the tivo, then plug eth1 into your switch/router/whatever. Boot the backtrack cd, and follow the general outlines described here.

Using the Grinder mentioned above may work, but I know that ettercap can do it.


----------



## moyekj

Just for grins and to test my check.py script I tried a push of an mpeg2 via pyTivo and found that even though the push worked the getDownloadRequests doesn't return anything that way either, so I must be misunderstanding what that's supposed to do.


----------



## kearygriffin

Ok, so I booted up grinder, and I think the encoding type is: avcL41MP4

Here is the relevant packet (with some stuff removed...):


Code:


<?xml version="1.0" encoding="utf-8"?><bodyOffer><bodyId>tsn:xxxxxxbodyId><bodyOfferId>tivo:bo.15000xxx</bodyOfferId><createDate>2009-02-21 18:15:54</createDate><description>Transferred by TiVo Desktop</description><duration>3545</duration><encodingType>avcL41MP4</encodingType><levelOfDetail>high</levelOfDetail><offerId>tivo:of.bs.1500xxxx</offerId><partnerId>tivo:pt.3187</partnerId><pcBodyId>tivo:pc.10005xxxxx</pcBodyId><publishDate>2009-02-21 18:15:50</publishDate><size>99920801</size><source>file:/C&#37;3A%2Fkeary%20stuff%2Fx</source><state>complete</state><subtitle>demo</subtitle><title>x</title><updateDate>2009-02-21 18:15:54</updateDate><url>http://192.168.1.33:8080/%XXXXXX-43CA-4E60-BD89-613CXXXXXXXX%7D/%XXXXXXXX-481E-45C3-BF83-514FXXXXXXX%7D?Format=video%2Fmp4</url></bodyOffer>


----------



## kearygriffin

And I got my first MP4 push working...

I edited mind.py, changing the encodingType line in pushVideo to:
'encodingType': 'avcL41MP4',

and ran this little python script:


Code:


#!/usr/bin/env python

import mind
m = mind.getMind()
m.pushVideo('652000xxxxxxxxxx', 'http://192.168.1.37/keary/videos/demnow.mp4', 'KGTransfer', 3545, 99920801, 'demonow-title', 'demnow-subtitle');

Replace 6520xxxxx with your tsn, the URL with the correct URL for your transfer, 3545 with the duration and 99920801 with the file length.
Works great...


----------



## moyekj

Bingo! Thanks Keary! It should be easy to try out mp4 with ac3 audio now as well. I will give that a shot shortly.


----------



## fyodor

Neat-do these show up as copy protected when transferred?

Thanks to you and MoyekJ for all of your work on this.



kearygriffin said:


> And I got my first MP4 push working...
> 
> I edited mind.py, changing the encodingType line in pushVideo to:
> 'encodingType': 'avcL41MP4',
> 
> and ran this little python script:
> 
> 
> Code:
> 
> 
> #!/usr/bin/env python
> 
> import mind
> m = mind.getMind()
> m.pushVideo('652000xxxxxxxxxx', 'http://192.168.1.37/keary/videos/demnow.mp4', 'KGTransfer', 3545, 99920801, 'demonow-title', 'demnow-subtitle');
> 
> Replace 6520xxxxx with your tsn, the URL with the correct URL for your transfer, 3545 with the duration and 99920801 with the file length.
> Works great...


----------



## Dan203

kearygriffin said:


> Ok, so I booted up grinder, and I think the encoding type is: avcL41MP4


That gives us some incite into what exactly it's looking for too..

avc = H.264

L41 = Level 4.1

MP4 = MP4 container

If you look at the chart on this page...

http://en.wikipedia.org/wiki/H.264

It shows the specifications for Level 4.1 video, so you know what the max bitrate, resolution, etc.. is.

Since it is expecting an MP4 container it is unlikely that it will support AC3 audio. (although not impossible) An official specification for storing AC3 in an MP4 container was only just released a few months ago and most muxers/demuxers don't support it yet. If anyone wants to run a test the program Handbrake is the only one I know of that stores AC3 in an MP4 container according the the specification.

Dan


----------



## moyekj

I can confirm that mp4 container with h.264 video + ac3 audio works fine as well as long as you use pyTivo as the video server. Tivo Desktop doesn't allow it probably because the codecs it's using don't support it.

Dan, one can use ffmpeg to generate such a video so handbrake is not the only option. The key is the moov atom must be at front of file for it to work, so if that's not the case then you can run qt-faststart on the file.

And yes unfortunately the pushes still make the mp4 files on the Tivo copy protected so you can't transfer off the Tivos.


----------



## moyekj

So for others who may want to try this out but aren't clear on the details here's a condensed summary of changes to get it going for now (until pyTivo developers have a chance to properly integrate changes to support native mp4 pushes)
NOTE: Keep backup copies of all the files since these changes will break pyTivo for mpeg2 transfers. I would recommend installing a new pyTivo installation in a separate location and then make these changes.

pyTivo file changes:
1/ *mind.py*
- Change 'mpeg2ProgramStream' to 'avcL41MP4' in both places in the file

2/ *plugins/video/video.py* (send_file procedure):
change Content-Type from 'video/x-tivo-mpeg' to 'video/mp4'

3/ *plugins/video/transcode.py* (tivo_compatible procedure):
- add following after 1st vInfo line of the procedure:
message = True, 'mp4'
logger.debug('%s, %s' % (message, inFile))
return message

4/ Make sure your *pyTivo.conf* file has proper tivo_username & tivo_password settings (your tivo login information) in the [Server] section. Also make sure you have proper ffmpeg path set in your pyTivo.conf as that is needed to collect video information.

5/ Put your test mp4 clips in pyTivo video share directory

6/ Finally start pyTivo & initiate a push from browser:
http://localhost:9032
click on Video shares link
select the mp4 video and Tivo to push to and click on 'Send to Tivo'
NOTE: You should leave pyTivo running while the push process happens and it can take a few minutes to start which you'll know when the blue light appears on your Tivo.

CHECKING TRANSFER RATE
You can check the transfer rate of your mp4 pushes from Network Diagnostics:
TC->Messages&Settings->Settings->Phone&Network->View Network Diagnostics->Transfer history->Video Download
You will note that you can get quite significantly higher transfer rates for mp4 pushes compared to mpeg2 pulls & pushes. I'm getting 20+ Mbps for my mp4 pushes to my S3 compared to about 13 Mbps for mpeg2.


----------



## PaulS

moyekj said:


> So for others who may want to try this out but aren't clear on the details here's a condensed summary of changes to get it going for now (until pyTivo developers have a chance to properly integrate changes to support native mp4 pushes)


Nice! Great work guys!


----------



## westside_guy

moyekj said:


> And yes unfortunately the pushes still make the mp4 files on the Tivo copy protected so you can't transfer off the Tivos.


Personally I don't see this as a significant issue, given the circumstances.

Thanks to all of you for the work you've done so far on this! I prefer to rip my DVDs to x.264, since the space savings is substantial and the quality drop isn't significant enough to matter to me (I could always watch the original DVD if it really mattered anyway). This way there's no additional tradeoff to using the same repository/server for streambaby and pyTivo (unless I've missed something).


----------



## westside_guy

Oops, never mind. Pay no attention to that man behind the curtain...


----------



## ryanrk

moyekj said:


> 6/ Finally start pyTivo & initiate a push from browser:
> localhost:9032
> click on Video shares link
> select the mp4 video and Tivo to push to


When I click on the Video Share link all i get is an XML feed back. Nothing to click on or select or anything to push it back.


----------



## moyekj

ryanrk said:


> When I click on the Video Share link all i get is an XML feed back. Nothing to click on or select or anything to push it back.


 That's probably stupid Internet Explorer at work. If you try a proper browser like Firefox then it works fine.


----------



## wmcbrine

ryanrk said:


> When I click on the Video Share link all i get is an XML feed back. Nothing to click on or select or anything to push it back.


Internet Explorer, right? The Push page doesn't work there -- try Firefox or some other browser (I can only vouch for Firefox). Sorry.

K & K, exciting stuff! 

Keary, while you have your spy gear hooked up, there are a couple more things you might want to look at:

1. There are, or used to be, some TiVoCasts that aren't copy prohibited. If we can find one of those, we might be able to get the info we need to turn it off on pyTivo pushes. Unfortunately the only one that I know had this was DL.TV.

2. The HME side of Netflix and YouTube, which are also SSL'd.


----------



## moyekj

wmcbrine said:


> 2. The HME side of Netflix and YouTube, which are also SSL'd.


 Perhaps Tivo/Netflix as well? Would be wonderful to figure out how to get rid of that pesky 1.1GB streaming limit


----------



## ryanrk

wmcbrine said:


> Internet Explorer, right? The Push page doesn't work there -- try Firefox or some other browser (I can only vouch for Firefox). Sorry.
> 
> K & K, exciting stuff!
> 
> Keary, while you have your spy gear hooked up, there are a couple more things you might want to look at:
> 
> 1. There are, or used to be, some TiVoCasts that aren't copy prohibited. If we can find one of those, we might be able to get the info we need to turn it off on pyTivo pushes. Unfortunately the only one that I know had this was DL.TV.
> 
> 2. The HME side of Netflix and YouTube, which are also SSL'd.


Yet it works in Firefox.

However the push didn't. I got this error for some reason. I can pull this file



Code:


----------------------------------------
Exception happened during processing of request from ('192.168.0.185', 51127)
Traceback (most recent call last):
  File "/usr/lib/python2.5/SocketServer.py", line 464, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.5/SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.5/SocketServer.py", line 522, in __init__
    self.handle()
  File "/usr/lib/python2.5/BaseHTTPServer.py", line 316, in handle
    self.handle_one_request()
  File "/usr/lib/python2.5/BaseHTTPServer.py", line 310, in handle_one_request
    method()
  File "/home/ryan/pyTivo/httpserver.py", line 103, in do_GET
    method(self, query)
  File "/home/ryan/pyTivo/plugins/video/video.py", line 288, in Push
    file_info['valid'] = transcode.supported_format(file_path)
  File "/home/ryan/pyTivo/plugins/video/transcode.py", line 707, in supported_format
    if video_info(inFile)['Supported']:
  File "/home/ryan/pyTivo/plugins/video/transcode.py", line 507, in video_info
    stdin=subprocess.PIPE)
  File "/usr/lib/python2.5/subprocess.py", line 594, in __init__
    errread, errwrite)
  File "/usr/lib/python2.5/subprocess.py", line 1091, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
----------------------------------------


----------



## westside_guy

wmcbrine said:


> Internet Explorer, right? The Push page doesn't work there -- try Firefox or some other browser (I can only vouch for Firefox).


Doesn't work in Safari, either - surprisingly. Fortunately Firefox is cross platform.


----------



## moyekj

ryanrk said:


> However the push didn't. I got this error for some reason. I can pull this file


 I got that same error when trying to send a wmv file (I know that's not mp4 but I was trying it anyway). I bet you if you try with a known good file such as the ones posted earlier in this thread you will find those work OK.
(Pull works because it transcodes to mpeg2 for an unmodified pyTivo installation)


----------



## westside_guy

One thing that messed me up at first - despite it having been mentioned more than once in this thread - is not making sure qt-faststart had been run on the video in question. 

This is kinda cool!


----------



## moyekj

westside_guy said:


> One thing that messed me up at first - despite it having been mentioned more than once in this thread - is not making sure qt-faststart had been run on the video in question.


 That's one advantage of Keary's streambaby - it takes care of the moov atom so you don't have to worry about qt-faststart. Actually, aside from the 1.1GB limitation streaming mp4 is better than transfers in many other respects:
* random access anywhere in the file instantly, even beyond buffer
* flexible jump n minutes forwards/backwards
* closed captioning capability
* can access your files in a flexible folder structure of your choosing
* fancy metadata with graphics
* easy access from any series 3 tivo
...

Still, it's nice to have the option to transfer mp4 files natively as well. Would be nice if it would work with the pull method instead of just push and hopefully something Tivo should be looking into for next TD update.
Nice to have capability in pyTivo now and ironic that it can already do better than TD (supports mp4 with ac3 audio that doesn't work in TD) - I already uninstalled the Tivo Desktop monster.


----------



## westside_guy

moyekj said:


> That's one advantage of Keary's streambaby - it takes care of the moov atom so you don't have to worry about qt-faststart. Actually, aside from the 1.1GB limitation streaming mp4 is better than transfers in many other respects:
> * random access anywhere in the file instantly, even beyond buffer
> * flexible jump n minutes forwards/backwards
> * closed captioning capability
> * can access your files in a flexible folder structure of your choosing
> * fancy metadata with graphics
> * easy access from any series 3 tivo


I totally agree, and it's nice to have options. Thanks to this new ability that you guys have developed, I'm thinking I'll keep using streambaby for all my TV/anime/etc. rips (which all fall well under the 1.1GB limit) but start pushing movies when I want to watch them.

The most ideal solution IMNSHO is if, eventually, someone figures out how to take advantage of whatever way Netflix et. al. stream without being limited by the 1.1GB limit (as you mentioned earlier). But all the currently available options are already great; and speaking as someone who's basically mooching off of all the hard work you guys are doing... I'm certainly not going to complain!


----------



## wmcbrine

westside_guy said:


> Doesn't work in Safari, either - surprisingly.


OK, I've fixed this. It was just some missing MIME types. (Imagine, Internet Explorer caring more about MIME types than Firefox -- what is the world coming to?) Tested now in IE7, Opera and Safari, as well as Firefox.


----------



## westside_guy

wmcbrine said:


> OK, I've fixed this. It was just some missing MIME types. (Imagine, Internet Explorer caring more about MIME types than Firefox -- what is the world coming to?) Tested now in IE7, Opera and Safari, as well as Firefox.


I'm beginning to think you guys who develop pyTivo, streambaby, and the like never sleep!  Thank you very much for fixing this.

Regarding IE and mime types - wha? IE cares about mime types now? I know in the past I've had to work around IE's lack of caring. But maybe enough exploits have come down the pipe that MS has learned... at least in some cases...


----------



## Clusty

larry99 said:


> See prior post in this thread (*DVR Version *, 3:39 PM Wednesday) for my software version. How do I know if its a beta?


I have the same version as you - is it a beta?


----------



## moyekj

ryanrk said:


> However the push didn't. I got this error for some reason. I can pull this file
> 
> 
> 
> Code:
> 
> 
> ----------------------------------------
> Exception happened during processing of request from ('192.168.0.185', 51127)
> Traceback (most recent call last):
> File "/usr/lib/python2.5/SocketServer.py", line 464, in process_request_thread
> self.finish_request(request, client_address)
> File "/usr/lib/python2.5/SocketServer.py", line 254, in finish_request
> self.RequestHandlerClass(request, client_address, self)
> File "/usr/lib/python2.5/SocketServer.py", line 522, in __init__
> self.handle()
> File "/usr/lib/python2.5/BaseHTTPServer.py", line 316, in handle
> self.handle_one_request()
> File "/usr/lib/python2.5/BaseHTTPServer.py", line 310, in handle_one_request
> method()
> File "/home/ryan/pyTivo/httpserver.py", line 103, in do_GET
> method(self, query)
> File "/home/ryan/pyTivo/plugins/video/video.py", line 288, in Push
> file_info['valid'] = transcode.supported_format(file_path)
> File "/home/ryan/pyTivo/plugins/video/transcode.py", line 707, in supported_format
> if video_info(inFile)['Supported']:
> File "/home/ryan/pyTivo/plugins/video/transcode.py", line 507, in video_info
> stdin=subprocess.PIPE)
> File "/usr/lib/python2.5/subprocess.py", line 594, in __init__
> errread, errwrite)
> File "/usr/lib/python2.5/subprocess.py", line 1091, in _execute_child
> raise child_exception
> OSError: [Errno 2] No such file or directory
> ----------------------------------------


This seems to happen for several valid mp4 files I have. I think the problem is that video_info procedure in transcode.py does not work properly for some mp4 videos. So looks like some more patching would be needed to make this more robust for mp4 pushes. I'm going to be out most of today so probably won't be looking into it further for a while...


----------



## ryanrk

moyekj said:


> I got that same error when trying to send a wmv file (I know that's not mp4 but I was trying it anyway). I bet you if you try with a known good file such as the ones posted earlier in this thread you will find those work OK.
> (Pull works because it transcodes to mpeg2 for an unmodified pyTivo installation)


That was a good file. It was a regular mpeg2 file that I can pull using the Tivo box. I never tried the push stuff before.


----------



## fyodor

moyekj said:


> ...
> 
> Still, it's nice to have the option to transfer mp4 files natively as well. Would be nice if it would work with the pull method instead of just push and hopefully something Tivo should be looking into for next TD update.
> Nice to have capability in pyTivo now and ironic that it can already do better than TD (supports mp4 with ac3 audio that doesn't work in TD) - I already uninstalled the Tivo Desktop monster.


Is it possible to configure Pytivo to check for MP4 files and push in response to a pull request for one of those files?

F


----------



## PaulS

fyodor said:


> Is it possible to configure Pytivo to check for MP4 files and push in response to a pull request for one of those files?
> 
> F


Now THAT would be slick. I would think that it's possible, but it might take some under-the-covers work to get done...


----------



## pmd

fyodor said:


> Is it possible to configure Pytivo to check for MP4 files and push in response to a pull request for one of those files?
> 
> F


Sounds cool, but please make that optional if it happens. I just tried some of the mind.py related push stuff in this thread on my Aussie TiVo HD running beta software and, rather unsurprisingly, it doesn't work (__pcBodyStore seems to be null, or empty). I think it's failing to login into mind.tivo.com - it seems that tivo.com and tivo.com.au accounts are not equivalent.

Is mind.tivo.com related to online scheduling, or is it some other service? We have online scheduling here, but it's done via a third party (yahoo7.com.au!).


----------



## moyekj

ryanrk said:


> That was a good file. It was a regular mpeg2 file that I can pull using the Tivo box. I never tried the push stuff before.


 I solved the problem by specifying same ffmpeg that is used by streambaby in my pyTivo.conf file. So check to make sure you are pointing to a recent complete version of ffmpeg in pyTivo to see if that helps.

EDIT: I'm pretty certain that your problem is incorrect or missing ffmpeg specification in your pyTivo.conf since you are getting an OS error when trying to spawn a child process:


> OSError: [Errno 2] No such file or directory


----------



## moyekj

fyodor said:


> Is it possible to configure Pytivo to check for MP4 files and push in response to a pull request for one of those files?
> 
> F


 That would be confusing as there is an inherent delay with a push (takes a few minutes to start) compared to a pull. The push mechanism could be made easier by having an auto push folder type setup like TD+ where you just drop files into the folder to initiate a push. But that's kind of a pain to code as it means setting things up properly so you don't push the same file over and over again among other things.
For instant gratification purposes I would just use streambaby for most mp4s and for the few longer mp4s the pyTivo push method is pretty easy as it is right now.


----------



## westside_guy

I know I'm probably stating the obvious but - rather than fake a pull front end I think it would make more sense to set up a Tivo front end of such a "push" app so it's more obvious exactly what you're initiating - which would be more of a HME style app, I'd guess.


----------



## PaulS

...or, pyTivo could allow you to specify what type of service is being provided by that share. Default would be "pull", but could be over-ridden to provide a "push" service.


----------



## ryanrk

moyekj said:


> I solved the problem by specifying same ffmpeg that is used by streambaby in my pyTivo.conf file. So check to make sure you are pointing to a recent complete version of ffmpeg in pyTivo to see if that helps.
> 
> EDIT: I'm pretty certain that your problem is incorrect or missing ffmpeg specification in your pyTivo.conf since you are getting an OS error when trying to spawn a child process:


Yea, that was it. Some how ffmpeg got uninstalled in the past day or two.


----------



## txporter

This is sort of a tangential question, but what are the Tivo limitations for mp4 as far as framerate, etc. The tivos don't like anything but 29.97 fps MPEG2, is the same true for mp4 file, or can tivo except 23.976 fps and other framerates?

Jason


----------



## wmcbrine

OK, I've integrated basic h.264 pushing into my pyTivo fork. It's a bit hackish yet, but you can run one pyTivo instance and serve both MPEG-2 and h.264. If you're pushing to an S3/HD, the push function now checks if the file has h.264 video and AAC or AC3 audio, and attempts to pass it through if so. Otherwise it behaves as before. This doesn't check things like the container type yet, so there could be a lot of false positives. If the push fails, you'll have to pull (and get it transcoded to MPEG-2). Still, it's pretty cool.


----------



## burnside

Great work man, can't wait to try this out!


----------



## moyekj

txporter said:


> This is sort of a tangential question, but what are the Tivo limitations for mp4 as far as framerate, etc. The tivos don't like anything but 29.97 fps MPEG2, is the same true for mp4 file, or can tivo except 23.976 fps and other framerates?
> 
> Jason


 That's news to me. I'm pretty sure I have a bunch of mpeg2 DVD rips with 23.97 fps that play fine on my S3 Tivos. Some of those rips I converted to mp4 (preserving frame rate) for mp4 testing purposes and they also play fine. Could it be perhaps S2s and earlier that didn't like alternate frame rates?
It looks to me like tivo decoding of mp4 container with h.264 video & aac or ac3 audio is pretty flexible as far as framerate, resolution, bit rate, etc. I believe level 4.1 or lower is still a requirement (i.e. level 5.1 for example doesn't work but I think encodings at that level are very rare anyway).


----------



## moyekj

wmcbrine said:


> OK, I've integrated basic h.264 pushing into my pyTivo fork. It's a bit hackish yet, but you can run one pyTivo instance and serve both MPEG-2 and h.264. If you're pushing to an S3/HD, the push function now checks if the file has h.264 video and AAC or AC3 audio, and attempts to pass it through if so. Otherwise it behaves as before. This doesn't check things like the container type yet, so there could be a lot of false positives. If the push fails, you'll have to pull (and get it transcoded to MPEG-2). Still, it's pretty cool.


 That was quick! Thanks for implementing the changes. Shall check it out tonight.


----------



## txporter

moyekj said:


> That's news to me. I'm pretty sure I have a bunch of mpeg2 DVD rips with 23.97 fps that play fine on my S3 Tivos. Some of those rips I converted to mp4 (preserving frame rate) for mp4 testing purposes and they also play fine. Could it be perhaps S2s and earlier that didn't like alternate frame rates?
> It looks to me like tivo decoding of mp4 container with h.264 video & aac or ac3 audio is pretty flexible as far as framerate, resolution, bit rate, etc. I believe level 4.1 or lower is still a requirement (i.e. level 5.1 for example doesn't work but I think encodings at that level are very rare anyway).


Hmm...well, I thought that I had tried this...but maybe not?? I have read a few posts from the folks at VideoRedo that seemed pretty certain about this and I thought I had followed up and tried it on my TivoHD, but maybe not. I would like to be able to take telecined DVDs and encode them as progressive 23.976 fps and avoid running pulldown on them again if it is possible. I suppose I will need to try it.

Thanks for the info on mp4 though. With the advent of being able to upload mp4s to the tivo, I might alter my plans to convert to MPEG2 and begin moving everything to mp4.

Also, I guess that I haven't seen a question or answer about this, but uploaded mp4s play and act the same as MPEG2 on the tivo?

Jason


----------



## euckersw

westside_guy said:


> One thing that messed me up at first - despite it having been mentioned more than once in this thread - is not making sure qt-faststart had been run on the video in question.
> 
> This is kinda cool!


Can I get some help in how to run qt-faststart? I see that there is a C and Python version - which should I use, and how do I execute it? I'm a bit lost...


----------



## moyekj

euckersw said:


> Can I get some help in how to run qt-faststart? I see that there is a C and Python version - which should I use, and how do I execute it? I'm a bit lost...


 Python version would make it more platform independent so would be the better choice. (I haven't tried Python version so can't confirm if it works properly or not). If you are on windows then there's compiled versions readily available already. I have a compiled version for windows as part of kmttg distribution (also includes the c code for a simple compile on other platforms). The syntax for the C-version is very simple:
*qt-faststart infile outfile*
If successful you will see some output messages about moov atom being moved to front of the file. The resulting output file will be identical in size to input file and function fine in all players so no point keeping the original version around once you are satisfied newly created file works.

P.S. Since a python version is available theoretically a nice touch to the pyTivo integration would be to check the mp4 file to see if moov atom is at the front of the file and if not spit out an error message accordingly... I suppose one could even make it automatically run the conversion if necessary but I think that's pushing things too far once you start messing with video files without user intervention since something could go wrong.

P.P.S. I suppose another possible tactic is pyTivo to take the streambaby approach and send the relevant section of the mp4 file containing moov atom ahead of the rest of the file (I'm not exactly sure how Keary is doing it in streambaby but I think that's the gist of it). That way no pre-processing of file would be needed just as it is in streambaby. However that would probably involve a lot of extra coding headaches and is probably not worthwhile.


----------



## euckersw

Thanks - I was looking for a compiled version and that helped greatly. So I tried using it on a .mkv file that I have, but it turns out it wasn't needed (I received an error when I tried to use it on the file) and the video transfered using pyTivo without transcoding - awesome! 

Now my question is what files require using qt-faststart before transfering, and do you receive some sort of error when pushing if a file needs qt-faststart? Thanks for the help.


----------



## moyekj

euckersw said:


> Thanks - I was looking for a compiled version and that helped greatly. So I tried using it on a .mkv file that I have, but it turns out it wasn't needed (I received an error when I tried to use it on the file) and the video transfered using pyTivo without transcoding - awesome!
> 
> Now my question is what files require using qt-faststart before transfering, and do you receive some sort of error when pushing if a file needs qt-faststart? Thanks for the help.


I posted about that here:
http://www.tivocommunity.com/tivo-vb/showthread.php?p=7084028#post7084028
i.e. As far as pyTivo is concerned there will be no problem, but transfer to Tivo will fail (and recordings history will show some catch all error about failed transfer due to incorrect file size).


----------



## euckersw

moyekj said:


> I posted about that here:
> http://www.tivocommunity.com/tivo-vb/showthread.php?p=7084028#post7084028
> i.e. As far as pyTivo is concerned there will be no problem, but transfer to Tivo will fail (and recordings history will show some catch all error about failed transfer due to incorrect file size).


So do .mkv files not need this extra step? As I said, I did not run qt-faststart on the .mkv file I pushed, and it worked like a charm. Thanks for your help.


----------



## burnside

Quick noob question. I'm getting ready to start converting my dvds to mp4. I had already started on a few conversions to mpeg2, but after finding this thread the other day, I'm excited to switch to mp4. Before I go overboard with converting my entire library, what is the best format for these conversions? I'm guessing h.264 video with aac/ac3, correct? If so, would this allow my tivo to play 5.1 audio?


----------



## moyekj

burnside said:


> Quick noob question. I'm getting ready to start converting my dvds to mp4. I had already started on a few conversions to mpeg2, but after finding this thread the other day, I'm excited to switch to mp4. Before I go overboard with converting my entire library, what is the best format for these conversions? I'm guessing h.264 video with aac/ac3, correct? If so, would this allow my tivo to play 5.1 audio?


 For Tivo purposes you can preserve ac3 5.1 audio as is and transcode to h.264 video. However be warned that since ac3 audio in mp4 container has only recently been ratified some players can't handle that yet. For example Quicktime and Windows Media Player won't play mp4 with ac3 audio but recent versions of VLC do. So if you are targeting other means to play back those files other than Tivo you should check whatever players you plan to use to see if they can handle it. If you use 6 channel aac audio instead that would make it compatible with most players but not sure how Tivo outputs 6 channel aac as far as surround sound goes.


----------



## moyekj

euckersw said:


> So do .mkv files not need this extra step? As I said, I did not run qt-faststart on the .mkv file I pushed, and it worked like a charm. Thanks for your help.


 Don't know much about Matroska container but probably is very different than mp4 container so doesn't have the same restriction. For mp4 depending on encoder you use it may not be necessary either. If you use ffmpeg to encode to mp4 container then qt-faststart is still necessary (ffmpeg ships with qt-faststart utility).


----------



## westside_guy

wmcbrine said:


> OK, I've integrated basic h.264 pushing into my pyTivo fork. It's a bit hackish yet, but you can run one pyTivo instance and serve both MPEG-2 and h.264. If you're pushing to an S3/HD, the push function now checks if the file has h.264 video and AAC or AC3 audio, and attempts to pass it through if so. Otherwise it behaves as before. This doesn't check things like the container type yet, so there could be a lot of false positives. If the push fails, you'll have to pull (and get it transcoded to MPEG-2). Still, it's pretty cool.


This is great! Thank you very much for making those changes, and for doing it so quickly to boot!


----------



## moyekj

euckersw said:


> So do .mkv files not need this extra step? As I said, I did not run qt-faststart on the .mkv file I pushed, and it worked like a charm. Thanks for your help.


 Can you post the details of a sample .mkv file that works with native push? I wasn't able to get a .mkv container with h.264 video & ac3 audio to push natively to Tivo (and it won't stream to Tivo natively either).


----------



## BobAd

My experiences with mp4 push. 

Videos were recoded into Digital AVC format, Main Profile. Max ref frames = 1, Max keyframes = 300, Max b frames = 3, CABAC and Weighted Prediction enabled.

Video Results: Choppy video motion at both 480p and 720p recodes. Files below approx. 1gb (around 1Mbs) showed lack of detail on 60" screen, and choppiness on high motion scenes. Files of approx. 2Gb size (around 3Mbs) had better detail but exhibited more choppiness on even slow motion playback. There were no observed problems on computer playback (24" monitor, WMP).

Audio Results: Files were transcoded into 5.1 AAC ( confirmed through examination of file properties on computer). However, my receiver indicated that only 2 channel was being passed from S3.

High Profile transcodes would not push.


----------



## wmcbrine

BobAd said:


> My experiences with mp4 push.


Mine look great so far.

You don't say what encoder you're using.



> _Audio Results: Files were transcoded into 5.1 AAC ( confirmed through examination of file properties on computer). However, my receiver indicated that only 2 channel was being passed from S3._


I think you have to use AC3 for 5.1 to work on the TiVo.


----------



## wmcbrine

Integrated qt-faststart into my pyTivo fork with a patch from Keary.


----------



## euckersw

moyekj said:


> Can you post the details of a sample .mkv file that works with native push? I wasn't able to get a .mkv container with h.264 video & ac3 audio to push natively to Tivo (and it won't stream to Tivo natively either).


Looks like we're having varying success (or limited success, for that matter) with pushing native .mkv files. Anybody else have any luck?


----------



## BobAd

wmcbrine said:


> Mine look great so far.
> 
> You don't say what encoder you're using.
> 
> I think you have to use AC3 for 5.1 to work on the TiVo.


Used NERO Recode (from version 8)- Could you be so kind as to share your encoder process/settings to allow me a little more success?

THANKS!


----------



## moyekj

wmcbrine said:


> Integrated qt-faststart into my pyTivo fork with a patch from Keary.


FYI looks to me like qt-faststart only works for mp4 container. I haven't yet found another container that works (I tried mkv and transport stream) so may not be a big deal, but probably the code could use a patch to restrict qt-faststart run for mp4 container type only just in case.


----------



## moyekj

BobAd said:


> Used NERO Recode (from version 8)- Could you be so kind as to share your encoder process/settings to allow me a little more success?
> 
> THANKS!


 Try ffmpeg with something like:
ffmpeg -y -i INPUT_FILE -vcodec libx264 -coder 0 -level 41 -sameq -g 300 -bufsize 14745k -b 5000k -maxrate 16000k -bug "+autodetect+ms" -me epzs -trellis 2 -mbd 1 -acodec copy -f mp4 OUTPUT_FILE
(You can adjust the bit rate to suit your size vs quality tradeoff with -b argument)

P.S. If you are looking for maximum quality then a multi-pass encode would probably be necessary. Nice thing about ffmpeg is you can google for recipes all over the place.


----------



## kearygriffin

moyekj said:


> FYI looks to me like qt-faststart only works for mp4 container. I haven't yet found another container that works (I tried mkv and transport stream) so may not be a big deal, but probably the code could use a patch to restrict qt-faststart run for mp4 container type only just in case.


The qt-faststart code is only run when the mime type is video/mp4 and the video has already been determined to be TiVo compatible. This should only occur when it is an mp4 container. (Because video/mp4 mime type implies an MP4 container).

Currently (as wmcbrine mentioned in an earlier post) only the codec is checked for h264, not the container, so more video is being marked as "no transcode needed" and marked with video/mp4, which will end up triggering the qt-faststart. Once we determine exactly what formats the TiVo supports as untranscoded (which I think is probably just mpeg-ps & mp4) and this is changed (at a higher level than the qt-faststart code) I think the issue you are mentioning will go away.


----------



## burnside

So Keary, does this mean that if our mp4s contain h264 then we should be fine?


----------



## moyekj

burnside said:


> does this mean that if our mp4s contain h264 then we should be fine?


 The current check is mp4 should contain h.264 video and aac or ac3 audio. With latest wmcbrine code now you no longer have to worry about potentially having to run qt-faststart on your mp4 since pyTivo will do it if necessary.


----------



## kearygriffin

burnside said:


> So Keary, does this mean that if our mp4s contain h264 then we should be fine?


I'm sure there are always going to be "exceptions", but an MP4 that has the following characteristics should work.

1. h264 video, high profile level 4.1 or below
2. AAC or AC3 audio3
3. Possibly needs to be well "interleaved"

The last one is the hardest to define, but it basically means that the audio and video within the file are interleaved correctly. This means that audio sample for a particular frame is somewhere "near" the video sample for that frame. (I don't know what the TiVo's definition of "near" is going to be...) The TiVo has problems streaming poorly interleaved files, I don't know how well it will do with downloading poorly interleaved files.

I noticed someone mentioned "choppy" video earlier, and for MP4 streaming to the TiVo this is sometimes caused by poorly interleaved files.


----------



## moyekj

Just to add to what Keary said, I haven't actually tried it but it's likely MP2 & MP3 audio would probably also work in addition to AAC and AC3 audio formats.


----------



## berkshires

kearygriffin said:


> 3. Possibly needs to be well "interleaved"
> 
> The last one is the hardest to define, but it basically means that the audio and video within the file are interleaved correctly. This means that audio sample for a particular frame is somewhere "near" the video sample for that frame. (I don't know what the TiVo's definition of "near" is going to be...) The TiVo has problems streaming poorly interleaved files, I don't know how well it will do with downloading poorly interleaved files.
> 
> I noticed someone mentioned "choppy" video earlier, and for MP4 streaming to the TiVo this is sometimes caused by poorly interleaved files.


This sounds like a fascinating topic all to itself. Would love to learn more.


----------



## burnside

Thanks for explaining all this to me. Never knew about interleaved either! Totally new to me.


----------



## tchescat2000

I have mkv files that are 1080p using DTS as audio. I know that the mkv is still up in the air for passthru, but how about the DTS audio? Is AC3 the only supported passthru audio type?


----------



## euckersw

tchescat2000 said:


> I have mkv files that are 1080p using DTS as audio. I know that the mkv is still up in the air for passthru, but how about the DTS audio? Is AC3 the only supported passthru audio type?


I have had success with 3 of 4 mkv files I have pushed using pyTivo, so while I'm still unsure as to why the 4th file did not work, I think the passthrough process is working (in general) for mkv files. You bring up a good question on the DTS audio. I have a file that I pushed via pyTivo this morning, which was originally a mkv file with DTS, however when I play the file my stereo shows Dolby Digital 5.1. However, when I stream the same file using my PS3, my stereo shows DTS. Any idea on what's happening to the DTS?


----------



## Dan203

What about integrating an MP4 muxer into pyTiVo so that any file with the proper audio and video type is remuxed into a proper MP4 container and that stream is push through to the TiVo?

What does pyTiVo do now if it finds a properly encoded MPEG-2 file in the wrong container? For example an MPEG-2 transport stream? Does it have to transcode? Or does it simply repackage the stream into a program stream and pass it over to the TiVo?

Dan


----------



## tchescat2000

Well I just downloaded the newest pytivo and tried to import one of my mkv files 1080p w/DTS and the debug log says: pyTivo.video.transcode: (False, 'TRANSCODE=YES, vCodec h264 not compatible.'), m:\movies\movie1080pDTS.mkv

Why vCodec h264 not compatible?


----------



## westside_guy

tchescat2000 said:


> Well I just downloaded the newest pytivo and tried to import one of my mkv files 1080p w/DTS and the debug log says: pyTivo.video.transcode: (False, 'TRANSCODE=YES, vCodec h264 not compatible.'), m:\movies\movie1080pDTS.mkv
> 
> Why vCodec h264 not compatible?


If you pull it down to the Tivo the usual way, it'll still get transcoded to mpeg2. You have to "push" it from your computer, using the web interface, to get the native mp4.


----------



## tchescat2000

I tried the push way of doing it and the debug log says the same thing, but it doesn't transfer at all that way. It shows up on the Tivo list as transfering, but nothing is really transfering.


----------



## moyekj

tchescat2000 said:


> I tried the push way of doing it and the debug log says the same thing, but it doesn't transfer at all that way. It shows up on the Tivo list as transfering, but nothing is really transfering.


 When you say you pulled down the latest version, did you get wmcbrine's latest zip file from here?
http://repo.or.cz/w/pyTivo/wmcbrine.git?a=snapshot;h=ed44e2300bccffa075afb422a00213b8275324de;sf=zip
i.e. You should get it from latest zip file here:
http://repo.or.cz/w/pyTivo/wmcbrine.git


----------



## tchescat2000

That is where I got it and that is what version I am running. Also:

I tried a mkv that is not using DTS and pushed it and got the following:

2009-02-24 18:27:42,707 DEBUG pyTivo.video.transcode: (True, 'TRANSCODE=NO, passing through mp4.'), m:\movies\\movie2.mkv
2009-02-24 18:27:42,707 DEBUG pyTivo.video.transcode: m:\movies\\movie2.mkv is tivo compatible
2009-02-24 18:27:42,707 DEBUG pyTivo.video.qt-faststart: ftyp atom not found, is this a valid MOV/MP4 file?

And nothing transfers. Doesn't even show up in the now playing.


----------



## kearygriffin

I just did some testing with MKV files, and could not get a push to work with any MKV's. The only pushes that worked were MKV files that were not h264 (so they transcoded).

I did make a mistake in the qtfaststart stuff that made testing this a little harder (it was throwing an exception when mkv files where put thru qtfaststart, I think moyekj pointed this out earlier) It was supposed to copy the file raw if it had problems parsing it. I am attaching a patch to fix this.

All of the MKV's I tried where h264/ac3 so someone else may want to try h264/aac, but I really have my doubts that TiVo is supporting MKV files natively at the moment.


----------



## moyekj

tchescat2000 said:


> That is where I got it and that is what version I am running. Also:
> 
> I tried a mkv that is not using DTS and pushed it and got the following:
> 
> 2009-02-24 18:27:42,707 DEBUG pyTivo.video.transcode: (True, 'TRANSCODE=NO, passing through mp4.'), m:\movies\\movie2.mkv
> 2009-02-24 18:27:42,707 DEBUG pyTivo.video.transcode: m:\movies\\movie2.mkv is tivo compatible
> 2009-02-24 18:27:42,707 DEBUG pyTivo.video.qt-faststart: ftyp atom not found, is this a valid MOV/MP4 file?
> 
> And nothing transfers. Doesn't even show up in the now playing.


That make sense now. With only DTS audio the mp4 compatible check fails and hence transcode to mpeg2 is enabled. With the other sample with non-DTS audio the mp4 compatible check passes but then qtfaststart code is trying to parse an mkv container which fails and throws exception.
I'm pretty convinced at this point that mkv container is NOT supported natively anyway so there is not much point trying to patch things to allow mkv containers to go through (bypassing qtfaststart).

EDIT: I see Keary beat me in posting but I'll leave my message anyway...
To clarify now with Keary's patch qtfaststart will be effectively bypassed so now pyTivo will pass the mkv file along to Tivo, but I'm pretty sure you will find the transfer will fail with an exception like "connection closed by peer "- peer being Tivo rejecting the stream in this case).


----------



## moyekj

Keary, did you ever experiment native streaming of mkv files with streambaby? From what I recall I believe streambaby currently transcodes mkv to mpeg2 right? I think I may have tried a long while ago with tivostream and found mkv didn't stream natively but I think I was trying to use video/mp4 mime and never tried other mime types such as video/mkv. I'm pretty sure it won't work but just curious if you experimented much at all...


----------



## kearygriffin

moyekj said:


> Keary, did you ever experiment native streaming of mkv files with streambaby? From what I recall I believe streambaby currently transcodes mkv to mpeg2 right? I think I may have tried a long while ago with tivostream and found mkv didn't stream natively but I think I was trying to use video/mp4 mime and never tried other mime types such as video/mkv. I'm pretty sure it won't work but just curious if you experimented much at all...


I don't think so, but I just gave it a shot with
video/x-matroska
And got an unsupported stream type error from the Tivo.

It's pretty easy to test out new formats with streambaby.
If you copy (or rename) the file to "filename.raw", and then create a text file called "filename.raw.fmt" with the first line the mimeType, and the second line the contentType: to send to the TiVo, streambaby will give it a whirl. I just tried an MKV with both mime and contentype set to video/x-matroska.

Edit: I am assuming video/x-matroska is correct, as it's what I see referenced most often on the web.


----------



## moyekj

OK thanks. So I think the bottom line is AFAWK mkv container is NOT supported. Dan's suggestion earlier on seems to be the next best plausible way of doing it:
use ffmpeg to change container to mp4 if h.264 video & compatible audio stream is detected. That would handle mkv and other non mp4 containers.

One potential complication with that approach however is that one can't just pipe the output of ffmpeg to qtfaststart since with ffmpeg remux I think the moov atom always goes at the end of the file. So one would have to wait until the ffmpeg remux finishes, then send it to qtfaststart and then initiate the push... not too pretty.


----------



## kearygriffin

moyekj said:


> OK thanks. So I think the bottom line is AFAWK mkv container is NOT supported. Dan's suggestion earlier on seems to be the next best plausible way of doing it:
> use ffmpeg to change container to mp4 if h.264 video & compatible audio stream is detected. That would handle mkv and other non mp4 containers.
> 
> One potential complication with that approach however is that one can't just pipe the output of ffmpeg to qtfaststart since with ffmpeg remux I think the moov atom always goes at the end of the file. So one would have to wait until the ffmpeg remux finishes, then send it to qtfaststart and then initiate the push... not too pretty.


The other complicating issue, and maybe it's just me, but I inevitably have audio/video sync issues when I try to do a straight demux/remux to another container format.


----------



## moyekj

True, so probably restricting native h.264 push to mp4 container only is the easiest way to go right now short of a reliable alternative. Everything else gets pushed as mpeg2. That's still a big leg up on Tivo Desktop Plus push which doesn't support AC3 audio with h.264.


----------



## euckersw

moyekj said:


> OK thanks. So I think the bottom line is AFAWK mkv container is NOT supported. Dan's suggestion earlier on seems to be the next best plausible way of doing it:
> use ffmpeg to change container to mp4 if h.264 video & compatible audio stream is detected. That would handle mkv and other non mp4 containers.


Ugh - I've been duped and I'm sorry. I thought I was able to stream mkv files without transcoding and in my excitement I never checked the log. Sure enough, the files were being transcoded. I'm sorry if I got anybody's hopes up. I'll go curl up under a rock again...


----------



## Rdian06

kearygriffin said:


> The other complicating issue, and maybe it's just me, but I inevitably have audio/video sync issues when I try to do a straight demux/remux to another container format.


Are you using ffmepg to remux? If so try adding the -copyts flag to preserve the timestamps. This is what fixed the sync issues for transcoding mkv to vob in the pyTivo code.

I've been trying to use ffmpeg to remux some of my mkvs to mp4s, but something with my ffmpeg build isn't quite working. It keeps complaining about a failure to set frame size for the ac3 audio stream. Have to try a newer ffmpeg SVN to see if that fixes it.


----------



## Rdian06

moyekj said:


> OK thanks. So I think the bottom line is AFAWK mkv container is NOT supported. Dan's suggestion earlier on seems to be the next best plausible way of doing it:
> use ffmpeg to change container to mp4 if h.264 video & compatible audio stream is detected. That would handle mkv and other non mp4 containers.
> 
> One potential complication with that approach however is that one can't just pipe the output of ffmpeg to qtfaststart since with ffmpeg remux I think the moov atom always goes at the end of the file. So one would have to wait until the ffmpeg remux finishes, then send it to qtfaststart and then initiate the push... not too pretty.


So I've been meaning to ask, exactly how does streambaby handle the moov atom issue? Can any of that work be leveraged to precompute the moov atom and send it before finishing the remux to mp4?


----------



## moyekj

Rdian06 said:


> So I've been meaning to ask, exactly how does streambaby handle the moov atom issue? Can any of that work be leveraged to precompute the moov atom and send it before finishing the remux to mp4?


 Keary should answer that question, but remember for streambaby case you are starting from a pre-existing mp4 so the moov atom is readily available to be extracted and sent at the head of the stream. In this case the moov atom is not available until the ffmpeg remux completes. (streambaby currently will transcode to mpeg any mkv file).

Good tip on the -copyts option. I wasn't aware of such a thing.


----------



## moyekj

I just confirmed that surprisingly Tivo rejects mp2 and mp3 audio with combination of h.264 video in mp4 container. So basically looks like AC3 & AAC audio only.


----------



## euckersw

kearygriffin said:


> I just did some testing with MKV files, and could not get a push to work with any MKV's. The only pushes that worked were MKV files that were not h264 (so they transcoded).
> 
> I did make a mistake in the qtfaststart stuff that made testing this a little harder (it was throwing an exception when mkv files where put thru qtfaststart, I think moyekj pointed this out earlier) It was supposed to copy the file raw if it had problems parsing it. I am attaching a patch to fix this.
> 
> All of the MKV's I tried where h264/ac3 so someone else may want to try h264/aac, but I really have my doubts that TiVo is supporting MKV files natively at the moment.


How do we apply this patch (not sure where to cut and paste the attached code)? Thanks.


----------



## wmcbrine

Don't bother, just get it from my fork.


----------



## txporter

wmcbrine said:


> Don't bother, just get it from my fork.


Sorry to be dense, but I have only installed pytivo from the windows installers. Do I just need to download your tar.gz or zip file and dump those files into the pytivo directory and restart the service?

Jason


----------



## euckersw

txporter said:


> Sorry to be dense, but I have only installed pytivo from the windows installers. Do I just need to download your tar.gz or zip file and dump those files into the pytivo directory and restart the service?
> 
> Jason


Yes, that's what I do. Just allow the tar.gz or zip file to overwrite everything as you dump it into your pyTivo directory. Obviously if you've made any changes to the files being overwritten, they will be eliminated, so make sure to backup anything you need to before dumping the new files.


----------



## burnside

Quick question. I have some mp4s on my HD and I have no idea what audio codec they're running. What do you guys use to see the properties of your mp4 files?


----------



## moyekj

burnside said:


> Quick question. I have some mp4s on my HD and I have no idea what audio codec they're running. What do you guys use to see the properties of your mp4 files?


 MediaInfo is pretty nice. You can also use the ffmpeg that comes with pyTivo: "ffmpeg_mp2 -i fileName" to get information in same way pyTivo is doing it.


----------



## burnside

Excellent. Thanks moyekj!


----------



## kearygriffin

Rdian06 said:


> So I've been meaning to ask, exactly how does streambaby handle the moov atom issue? Can any of that work be leveraged to precompute the moov atom and send it before finishing the remux to mp4?


Nothing in the current code would really help with this issue, but it is an issue I have given some thought to (mostly in the context of transcoding to MP4 instead of mpeg2).

The trick is that moov atom needs to contain offsets to the individual samples (set of frames) in the MP4. So, given the following it might be possible. (This is all theoretical, and I'm not advocating that any of this a good idea...)
1. If the original container format contains the needed offsets, a demuxing/remuxing program would need understand both containers, and read the offsets from the original container and translate it to an MP4 container format.
2. If we were talking about transcoding to MP4 on the fly, if we could use a true CBR audio/video we could precompute where each sample would begin/end and generate a moov atom based on that. (It doesn't have to be exact, because we could "pad" chunks that are not the correct size, but should be close and no sample could ever be longer than our pre-computed sample size)

I'm of the opinion that neither of the above options is worth the considerable effort of implementing.

What I hoping is that some future version of TiVo software (or possibly even the current software, I haven't had a chance to fool around as much as I would like to with it) supports h264 inside of a WMV container. From my limited knowledge WMV (asf) does not require a full fledged index (although it may contain one), so muxing into an WMV container on the fly is a much more reasonable project.

I've never actually gotten a succesfull WMV file to transfer to the TiVo however, mostly because my main OS is linux and creating such a TiVo compatible WMV is not easy (not possible?) on linux. And my limited efforts on windows were also unsuccessful (probably due to lack of persistence. I tried moyekj's instruction on the tivostream page, but didn't have the exact versions of the software he mentioned, and my guesses as to how to translate the instructions where probably wrong)


----------



## NA9D

Hey guys,

I'm trying to push a show and I get the following error:



> No option 'tivo_username' in section: 'Server'
> 
> Traceback (most recent call last):
> File "/mnt/disk1/Media/pyTivo/plugins/video/video.py", line 324, in Push
> m = mind.getMind()
> File "/mnt/disk1/Media/pyTivo/mind.py", line 272, in getMind
> username = config.getTivoUsername()
> File "/mnt/disk1/Media/pyTivo/config.py", line 42, in getTivoUsername
> return config.get('Server', 'tivo_username')
> File "/opt/lib/python2.5/ConfigParser.py", line 520, in get
> raise NoOptionError(option, section)
> NoOptionError: No option 'tivo_username' in section: 'Server'


Trying to set up a "Server" section gives me weird behavior on the part of the pyTivo webserver...

Edit: OK. Looking at the pyTivo.conf file, I see the "Server" section is the default general settings section. But now, what's this "TivoUsername" thing? My MAK is properly saved and all so I'm not sure what I need to have here...


----------



## euckersw

NA9D said:


> Hey guys,
> 
> I'm trying to push a show and I get the following error:
> 
> Trying to set up a "Server" section gives me weird behavior on the part of the pyTivo webserver...
> 
> Edit: OK. Looking at the pyTivo.conf file, I see the "Server" section is the default general settings section. But now, what's this "TivoUsername" thing? My MAK is properly saved and all so I'm not sure what I need to have here...


You need to add the following lines to your "Server" section:

tivo_username = <your user name> 
'tivo_password' <your password>

where the user name and password are the ones you use on tivo.com to access your account.


----------



## NA9D

Thanks for the user name thing. That worked.

Now, I am not sure why but it seems like FFMPEG is trying to transcode from a push. Here's my console log:



> 192.168.1.1 - - [25/Feb/2009 15:23:33] "GET /TiVoConnect?Command=Push&Container=LSL-MPEG4&File=%2FBarbie+Fairytopia.mp4&tsn=JOLivingRoom HTTP/1.1" 302 -
> 192.168.1.1 - - [25/Feb/2009 15:23:36] "GET /TiVoConnect?Command=QueryContainer&Container=LSL-MPEG4 HTTP/1.1" 200 -
> 192.168.1.1 - - [25/Feb/2009 15:23:42] "GET /TiVoConnect?Command=XSL&Container=LSL-MPEG4 HTTP/1.1" 200 -
> 192.168.1.58 - - [25/Feb/2009 15:23:47] "GET /LSL-MPEG4/Barbie%20Fairytopia.mp4 HTTP/1.1" 200 -
> FFmpeg version UNKNOWN, Copyright (c) 2000-2008 Fabrice Bellard, et al.
> configuration: --enable-cross-compile --cross-prefix=/home/slug/optware/cs05q3armel/toolchain/arm-none-linux-gnueabi/gcc-2005q3-glibc-2.3.6/bin/arm-none-l


Yes, I know it's a "Barbie" movie but when you have two little girls in the house who love fairies and princesses, ya gotta do what ya gotta do...

So I see the Tivo making the request to pull the show but then the transcoding starts. Is my MP4 not correct then?


----------



## euckersw

NA9D said:


> Thanks for the user name thing. That worked.
> 
> Now, I am not sure why but it seems like FFMPEG is trying to transcode from a push. Here's my console log:
> 
> Yes, I know it's a "Barbie" movie but when you have two little girls in the house who love fairies and princesses, ya gotta do what ya gotta do...
> 
> So I see the Tivo making the request to pull the show but then the transcoding starts. Is my MP4 not correct then?


Hrm - I'm certainly no expert at this point, but check the log for text similar to this to see if the file is being transcoded or not:

2009-02-24 18:27:42,707 DEBUG pyTivo.video.transcode: (True, 'TRANSCODE=NO, passing through mp4.'), m:\movies\\movie2.mkv


----------



## NA9D

euckersw said:


> Hrm - I'm certainly no expert at this point, but check the log for text similar to this to see if the file is being transcoded or not:
> 
> 2009-02-24 18:27:42,707 DEBUG pyTivo.video.transcode: (True, 'TRANSCODE=NO, passing through mp4.'), m:\movies\\movie2.mkv


OK. REALLY DUMB question that I should know better on, but where does pyTivo save the log file?


----------



## euckersw

NA9D said:


> OK. REALLY DUMB question that I should know better on, but where does pyTivo save the log file?


No problem - check debug.log in the pyTivo root directory.


----------



## NA9D

euckersw said:


> No problem - check debug.log in the pyTivo root directory.


OK. Maybe I have to start it with some flag or something. No debug.log in the root directory...

Guess I'll go RTFM on that... 

Edit: OK. I found out how to turn the log on. 

OK...Or so I thought. I set debug to True. But after resetting, I don't see the file in the pyTivo directory.

Now, I am running a startup script that sends console output to /dev/null but that should be an issue should it?

python /mnt/disk1/Media/pyTivo/pyTivo.py > /dev/null 2>&1 &

That's at least according to the script on Armooo's site..


----------



## NA9D

OK. I ended up directing my output to a file instead of /dev/null and that gets me the same thing basically! Not sure why the debug.txt file isn't being written out. But here's what I get when I try to push a file:

DEBUGyTivo.video.transcodeFalse, 'TRANSCODE=YES, vCodec h264 not compatible.'), /mnt/disk1/Media/Video/MPEG4-H.264/Battlestar Galactica - (HDTV) 404 - Escape Velocity.mp4

So why is an h.264 file not compatible? I thought it pushed w/o problem..


----------



## burnside

moyekj said:


> MediaInfo is pretty nice. You can also use the ffmpeg that comes with pyTivo: "ffmpeg_mp2 -i fileName" to get information in same way pyTivo is doing it.


Damn, one more question. What would be the ffmpeg command to encode an mpeg to an mp4 with ac3 (192kbps)? I'm using winff and see the preset for h264 but it uses aac instead of ac3.


----------



## moyekj

burnside said:


> Damn, one more question. What would be the ffmpeg command to encode an mpeg to an mp4 with ac3 (192kbps)? I'm using winff and see the preset for h264 but it uses aac instead of ac3.


 I created a streambaby Wiki entry on compatible video information which includes a sample ffmpeg recipe to create mp4 with h.264 video and ac3 audio (from mpeg2 source):
http://code.google.com/p/streambaby/wiki/video_compatibility


----------



## txporter

NA9D said:


> OK. I ended up directing my output to a file instead of /dev/null and that gets me the same thing basically! Not sure why the debug.txt file isn't being written out. But here's what I get when I try to push a file:
> 
> DEBUGyTivo.video.transcodeFalse, 'TRANSCODE=YES, vCodec h264 not compatible.'), /mnt/disk1/Media/Video/MPEG4-H.264/Battlestar Galactica - (HDTV) 404 - Escape Velocity.mp4
> 
> So why is an h.264 file not compatible? I thought it pushed w/o problem..


Are you using wmcbrine's latest pytivo fork from his git?

Jason


----------



## NA9D

txporter said:


> Are you using wmcbrine's latest pytivo fork from his git?
> 
> Jason


Yeah from earlier today (2/25)...


----------



## txporter

Tried pushing tonight for the first time with wmcbrine's latest git. My mp4 are also getting transcoded like NA9D. I will try to post some logs in the morning, but I created these files using the Constant Quality profile in Handbrake with http optimized checked. I thought I had read this moved the moov atom to the front of the file. Anyone else successfully push a handbrake encode? Or all using something else?

Jason


----------



## NA9D

FYI, I think most of my files were done using VisualHub on my Mac. I could try a file that I did using iTivo...Maybe my moov atom is fubar with the VisualHub files.

OK. Nope. A download converted using iTivo doesn't work either...


----------



## wmcbrine

txporter said:


> Anyone else successfully push a handbrake encode?


Yes, I have. BTW, if you have a version with Keary's qt-faststart patch, it no longer matters if the moov atom is at the start.

debug=True hasn't created a debug.txt file for a long time -- that was outdated documentation. (I've just corrected it.) Instead, it just adds more output, which still goes to the console, unless you redirect it or use the logging options.

It might be that your audio is being reported as libfaad instead of mpeg4aac. I've added that to the pass-through list this morning.


----------



## euckersw

So what if we were to use a program like mkv2vob to remux a mkv file with h264 to a mp4 file?:

http://www.videohelp.com/tools/mkv2vob

Would that be able to trick Tivo into accepting the remuxed mkv without transcoding?


----------



## wmcbrine

Sure. I wouldn't call that a trick, either.


----------



## NA9D

wmcbrine said:


> debug=True hasn't created a debug.txt file for a long time -- that was outdated documentation. (I've just corrected it.) Instead, it just adds more output, which still goes to the console, unless you redirect it or use the logging options.
> 
> It might be that your audio is being reported as libfaad instead of mpeg4aac. I've added that to the pass-through list this morning.


Thanks for the info. The latest release still isn't working for me. Just VPN'd into home and tried sending a show. Here's what my log file says:


Code:


DEBUG:pyTivo.video.transcode:(False, 'TRANSCODE=YES, vCodec h264 not compatible.'), /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4

So I guess I'll have to run MediaInfo or whatever on this and report it back to ya'll because something isn't right...


----------



## txporter

wmcbrine said:


> It might be that your audio is being reported as libfaad instead of mpeg4aac. I've added that to the pass-through list this morning.


Yes, my audio is being reported as libfaad. Will try with your newest version and report back.

Jason

update: push seems to be working now with new build. Thanks! Also, can i queue more than one show to push to Tivo, or do I need to do one at a time as they finish?


----------



## NA9D

OK. Here is the info on one of my files....Perhaps someone can tell me why it's trying to transcode?



> General
> 
> Complete name : /Volumes/Media-1/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> 
> Format : MPEG-4
> 
> Codec ID : M4V
> 
> File size : 293 MiB
> 
> Duration : 24mn 57s
> 
> Overall bit rate : 1 641 Kbps
> 
> Movie name : Darby Goes Woozle Sluethin'; How the Tigger Lost His
> 
> Album : My Friends Tigger & Pooh
> 
> Performer : My Friends Tigger & Pooh
> 
> Genre : Animated, Kids, Educational
> 
> Encoded date : 2008
> 
> Tagged date : UTC 2009-02-17 14:08:38
> 
> stik : 10
> 
> tvsh : My Friends Tigger & Pooh
> 
> tven : Darby Goes Woozle Sluethin'; How the Tigger Lost His
> 
> tves : 201
> 
> tvnn : DISNHD
> 
> Video
> 
> ID : 1
> 
> Format : AVC
> 
> Format/Info : Advanced Video Codec
> 
> Format profile : [email protected]
> 
> Format settings, CABAC : No
> 
> Format settings, ReFrames : 1 frame
> 
> Codec ID : avc1
> 
> Codec ID/Info : Advanced Video Coding
> 
> Duration : 24mn 57s
> 
> Bit rate mode : Variable
> 
> Bit rate : 1 506 Kbps
> 
> Width : 640 pixels
> 
> Height : 368 pixels
> 
> Display aspect ratio : 16/9
> 
> Frame rate mode : Constant
> 
> Frame rate : 59.940 fps
> 
> Resolution : 24 bits
> 
> Colorimetry : 4:2:0
> 
> Scan type : Progressive
> 
> Bits/(Pixel*Frame) : 0.107
> 
> Stream size : 269 MiB (92%)
> 
> Encoded date : UTC 1970-01-01 00:00:00
> 
> Tagged date : UTC 1970-01-01 00:00:00
> 
> Audio
> 
> ID : 2
> 
> Format : AAC
> 
> Format/Info : Advanced Audio Codec
> 
> Format version : Version 4
> 
> Format profile : LC
> 
> Format settings, SBR : No
> 
> Codec ID : 40
> 
> Duration : 24mn 57s
> 
> Bit rate mode : Variable
> 
> Bit rate : 128 Kbps
> 
> Channel(s) : 2 channels
> 
> Channel positions : L R
> 
> Sampling rate : 48.0 KHz
> 
> Resolution : 16 bits
> 
> Stream size : 22.9 MiB (8%)
> 
> Encoded date : UTC 1970-01-01 00:00:00
> 
> Tagged date : UTC 2009-02-17 14:08:38


----------



## NA9D

And here's one more that wouldn't transfer right:



> General
> 
> Complete name : /Volumes/Media-1/Video/MPEG4-H.264/Battlestar Galactica - 405 - The Road Less Traveled.mp4
> 
> Format : MPEG-4
> 
> Codec ID : M4V
> 
> File size : 701 MiB
> 
> Duration : 59mn 59s
> 
> Overall bit rate : 1 634 Kbps
> 
> Encoded date : UTC 2008-06-15 04:48:23
> 
> Tagged date : UTC 2008-06-15 13:44:46
> 
> Writing library : Apple QuickTime
> 
> tvsh : Battlestar Galactica
> 
> tven : The Road Less Traveled
> 
> tvsn : 4
> 
> tves : 5
> 
> Video
> 
> ID : 1
> 
> Format : AVC
> 
> Format/Info : Advanced Video Codec
> 
> Format profile : [email protected]
> 
> Format settings, CABAC : No
> 
> Format settings, ReFrames : 1 frame
> 
> Codec ID : avc1
> 
> Codec ID/Info : Advanced Video Coding
> 
> Duration : 59mn 59s
> 
> Bit rate mode : Variable
> 
> Bit rate : 1 499 Kbps
> 
> Width : 612 pixels
> 
> Height : 458 pixels
> 
> Display aspect ratio : 4/3
> 
> Frame rate mode  : Constant
> 
> Frame rate : 29.970 fps
> 
> Resolution : 24 bits
> 
> Colorimetry : 4:2:0
> 
> Scan type : Progressive
> 
> Bits/(Pixel*Frame) : 0.178
> 
> Stream size : 643 MiB (92%)
> 
> Encoded date : UTC 2008-06-15 04:48:23
> 
> Tagged date : UTC 2008-06-15 04:48:23
> 
> Audio
> 
> ID : 2
> 
> Format : AAC
> 
> Format/Info : Advanced Audio Codec
> 
> Format version : Version 4
> 
> Format profile : LC
> 
> Format settings, SBR : No
> 
> Codec ID : 40
> 
> Duration : 59mn 59s
> 
> Bit rate mode : Constant
> 
> Bit rate : 125 Kbps
> 
> Nominal bit rate : 128 Kbps
> 
> Channel(s) : 2 channels
> 
> Channel positions : L R
> 
> Sampling rate : 48.0 KHz
> 
> Resolution : 16 bits
> 
> Stream size : 53.7 MiB (8%)
> 
> Encoded date : UTC 2008-06-15 04:48:23
> 
> Tagged date : UTC 2008-06-15 13:44:46


----------



## kearygriffin

wmcbrine said:


> It might be that your audio is being reported as libfaad instead of mpeg4aac. I've added that to the pass-through list this morning.


Thanks, I hadn't seen libfaad before, so I just added it to streambaby's list of recognized AAC codecs. Streambaby also has the additional id's listed for AAC audio. (All from various versions/configurations of ffmpeg):

mp4a
aac

I think mp4a is what my stock Ubuntu 8.04 ffmpeg reported (i've since upgraded). The aac came from a user running a recent Mac version of ffmpeg.


----------



## CuriousMark

I don't know if this matters or not, but TiVo Desktop allows only one instance to be connected to mind at a time. So if users who are having trouble are also using TiVo Desktop Plus, perhaps the connection of pyTivo is being ignored. Just a thought from out in left field. I can't participate in any of this as a series 2 owner, so feel free to ignore this if it doesn't make sense.


----------



## jasonrn

Does this work for Series 2?


----------



## moyekj

NA9D, what is really needed is the output of:

ffmpeg -i "/Volumes/Media-1/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4"

ffmpeg -i "/Volumes/Media-1/Video/MPEG4-H.264/Battlestar Galactica - 405 - The Road Less Traveled.mp4"

The MediaInfo information seems to indicate these should work but perhaps the codec names don't match what is currently being checked in pyTivo.


----------



## moyekj

jasonrn said:


> Does this work for Series 2?


 S2 machines don't have mpeg4 decoding capabilities so native mpeg4 pushes are not possible.


----------



## NA9D

moyekj said:


> NA9D, what is really needed is the output of:
> 
> ffmpeg -i "/Volumes/Media-1/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4"
> 
> ffmpeg -i "/Volumes/Media-1/Video/MPEG4-H.264/Battlestar Galactica - 405 - The Road Less Traveled.mp4"
> 
> The MediaInfo information seems to indicate these should work but perhaps the codec names don't match what is currently being checked in pyTivo.


Now you've opened a can of worms!  There's something screwy with my FFMPEG install but I don't know what. I was hoping that these pushes wouldn't involve FFMPEG. I'll post some logs and maybe we can figure some of this out...

FYI: I'm running pyTivo in this instance on a Buffalo Linkstation Live. I don't expect it to do well for heavy on the fly transcoding but it serves up MPEG-2s to the Tivo just fine. There is something funky though that I've not been able to figure out as it won't even get to transcoding an MPEG-4 file but errors out. So I'm not even able to get to the point where transcoding starts but the NAS is swamped...


----------



## NA9D

Here you go. I just included basically all the output. It goes through this over and over and over again.

I was able to get one show yesterday to appear to push to the Tivo but FFMPEG got hung up and wouldn't transcode. Now it doesn't even get that far. Nothing ever shows up on the Tivo...



> 192.168.1.59 - - [26/Feb/2009 09:00:01] "GET /TiVoConnect?Command=QueryContainer&Container=%2F HTTP/1.0" 200 -
> 192.168.1.1 - - [26/Feb/2009 10:41:52] "GET /TiVoConnect?Command=QueryContainer&Container=LSL-iTivo/My%20Friends%20Tigger%20%26%20Pooh HTTP/1.1" 200 -
> 192.168.1.1 - - [26/Feb/2009 10:41:53] "GET /TiVoConnect?Command=XSL&Container=LSL-iTivo/My%20Friends%20Tigger%20%26%20Pooh HTTP/1.1" 200 -
> 192.168.1.1 - - [26/Feb/2009 10:41:53] "GET /favicon.ico HTTP/1.1" 200 -
> DEBUG:root:starting ffmpeg, will wait 30 seconds for it to complete
> DEBUG:root:ffmpeg output=FFmpeg version UNKNOWN, Copyright (c) 2000-2008 Fabrice Bellard, et al.
> configuration: --enable-cross-compile --cross-prefix=/home/slug/optware/cs05q3armel/toolchain/arm-none-linux-gnueabi/gcc-2005q3-glibc-2.3.6/bin/arm-none-linux-gnueabi- --arch=arm --disable-encoder=snow --disable-decoder=snow --enable-shared --disable-static --enable-gpl --enable-postproc --prefix=/opt
> libavutil version: 49.6.0
> libavcodec version: 51.54.0
> libavformat version: 52.13.0
> libavdevice version: 52.0.0
> built on Apr 15 2008 22:17:14, gcc: 3.4.4 (release) (CodeSourcery ARM 2005q3-2)
> Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4':
> Duration: 00:24:57.6, start: 0.000000, bitrate: 1641 kb/s
> Stream #0.0(und): Video: h264, yuv420p, 640x368, 59.94 tb(r)
> Stream #0.1(und): Audio: mp4a / 0x6134706D, 48000 Hz, stereo
> Must supply at least one output file
> 
> DEBUGyTivo.video.transcode:failed at aKbps
> DEBUGyTivo.video.transcode:aFreq=48000; vFps=59.94; kbps=1641; mapAudio=[('0.1', '(und)')]; vWidth=640; vCodec=h264; Supported=True; millisecs=1497600; par=None; aKbps=None; par2=None; par1=None; dar2=None; mapVideo=0.0; vHeight=368; dar1=None; aCodec=mp4a / 0x6134706D
> DEBUG:root:CACHE HIT! /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> DEBUGyTivo.video.transcodeFalse, 'TRANSCODE=YES, aCodec mp4a / 0x6134706D not compatible.'), /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> DEBUG:root:CACHE HIT! /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> DEBUG:root:CACHE HIT! /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> DEBUGyTivo.video.transcodeFalse, 'TRANSCODE=YES, vCodec h264 not compatible.'), /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> DEBUG:root:CACHE HIT! /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> DEBUG:root:CACHE HIT! /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> DEBUG:root:CACHE HIT! /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> DEBUG:root:CACHE HIT! /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> DEBUGyTivo.video.transcodeFalse, 'TRANSCODE=YES, vCodec h264 not compatible.'), /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> DEBUG:root:CACHE HIT! /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> DEBUG:root:CACHE HIT! /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> DEBUG:root:CACHE HIT! /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> DEBUG:root:tsn: 65200018039B973
> DEBUG:root:aspect169: True
> DEBUG:rootptres: False
> DEBUGyTivo.video.transcode:File=/mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4 vCodec=h264 vWidth=640 vHeight=368 vFps=59.94 millisecs=1497600 ratio=173 rheight=23 rwidth=40 TIVO_HEIGHT=720 TIVO_WIDTH=1280
> DEBUG:root:CACHE HIT! /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> DEBUG:root:CACHE HIT! /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> DEBUG:root:CACHE HIT! /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> DEBUG:root:CACHE HIT! /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> DEBUGyTivo.video.transcodeFalse, 'TRANSCODE=YES, vCodec h264 not compatible.'), /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> DEBUG:root:CACHE HIT! /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> DEBUGyTivo.mind:__login
> {'cams_security_domain': 'tivocom', 'cams_login_config': 'http', 'cams_cb_password': 'xxxxxxxx', 'cams_original_url': '/mind/mind7?type=infoGet', 'cams_cb_username': '[email protected]'}
> DEBUGyTivo.mindcBodySearch
> {}
> 
> <pcBodyList><isBottom>true</isBottom><isTop>true</isTop><pcBody><bucketNumber>2798</bucketNumber><levelOfDetail>low</levelOfDetail><name>NA9DSERVER</name><pcBodyId>tivoc.1000142011</pcBodyId><type>pcBody</type></pcBody></pcBodyList>g
> DEBUGyTivo.mindcBodySearch
> {}
> 
> <pcBodyList><isBottom>true</isBottom><isTop>true</isTop><pcBody><bucketNumber>2798</bucketNumber><levelOfDetail>low</levelOfDetail><name>NA9DSERVER</name><pcBodyId>tivoc.1000142011</pcBodyId><type>pcBody</type></pcBody></pcBodyList>g
> DEBUGyTivo.mind:bodyOfferModify&bodyId=tsn:65200018039B973
> {'subtitle': "Darby Goes Woozle Sluethin'; How the Tigger Lost His", 'description': '', 'title': 'My Friends Tigger & Pooh', 'url': 'http://192.168.1.20:9032/LSL-iTivo/My%20Friends%20Tigger%20%26%20Pooh/My%20Friends%20Tigger%20%26%20Pooh%20-%20Darby%20Goes%20Woozle%20Sluethin%27%3B%20How%20the%20Tigger%20Lost%20His.mp4', 'pcBodyId': 'tivoc.1000142011', 'bodyId': 'tsn:65200018039B973', 'publishDate': '2009-02-26 16:4204', 'source': 'My Friends Tigger & Pooh', 'state': 'complete', 'partnerId': 'tivot.3187', 'duration': 1497, 'encodingType': 'mpeg2ProgramStream', 'size': 3212681760L}
> 
> <bodyOffer><bodyId>tsn:65200018039B973</bodyId><bodyOfferId>tivo:bo.15000691</bodyOfferId><createDate>2009-02-26 16:42:06</createDate><description /><duration>1497</duration><encodingType>mpeg2ProgramStream</encodingType><levelOfDetail>high</levelOfDetail><offerId>tivof.bs.15000691</offerId><partnerId>tivot.3187</partnerId><pcBodyId>tivoc.1000142011</pcBodyId><publishDate>2009-03-01 14:04:00</publishDate><size>3212681760</size>My Friends Tigger & Pooh<state>complete</state><subtitle>Darby Goes Woozle Sluethin'; How the Tigger Lost His</subtitle>My Friends Tigger & Pooh<updateDate>2009-02-26 16:42:06</updateDate>http://192.168.1.20:9032/LSL-iTivo/...Woozle Sluethin'; How the Tigger Lost His.mp4</bodyOffer>g


----------



## wmcbrine

NA9D said:


> I was hoping that these pushes wouldn't involve FFMPEG.


Everything involves ffmpeg. Even when it's not used for transcoding, it's used to find out about the file -- codecs, bitrates, etc. The only exception is for .TiVo files.

Edit: In the log you just posted, the audio track is identified as "mp4a / 0x6134706D". That's why it doesn't want to pass it without transcoding. I just now added "mp4a" to the list of recognized IDs, but it still won't work for that, because of the extra bit at the end.

If you add that full string to the list in tivo_compatible_mp4() in plugins/video/transcode.py, it might work. Or you could upgrade your ffmpeg.


----------



## NA9D

wmcbrine said:


> If you add that full string to the list in tivo_compatible_mp4() in plugins/video/transcode.py, it might work. Or you could upgrade your ffmpeg.


This is what I'd like to do - upgrade my FFMPEG. Problem is, I'm not sure where to get a compiled version from. The version I am using is being supplied through iPKG but it does not seem to work well. So I'm not sure where to get another version compiled for the ARM processor. I'll admit I haven't looked real hard yet either...


----------



## NA9D

wmcbrine said:


> If you add that full string to the list in tivo_compatible_mp4() in plugins/video/transcode.py, it might work. Or you could upgrade your ffmpeg.


OK, looking at transcode.py, I see:



Code:


  if vInfo['vCodec'] != 'h264':
        message = (False, 'TRANSCODE=YES, vCodec %s not compatible.' %
                          vInfo['vCodec'])
    elif vInfo['aCodec'] not in ('mpeg4aac', 'libfaad', 'ac3', 'liba52'):
        message = (False, 'TRANSCODE=YES, aCodec %s not compatible.' %
                          vInfo['aCodec'])
    else:
        message = (True, 'TRANSCODE=NO, passing through mp4.')

    logger.debug('%s, %s' % (message, inFile))
    return message

I've never programmed in Python, but I do know a little and I see something that jumps out at me here - the first line...In English, I translate that to:

If the Video Codec is H.264 then force transcoding because it's not compatible...or am I reading that wrong?

I've got feeling I'm not because I changed the audio format and while that first appeared compatible, I'm still getting:

DEBUGyTivo.video.transcode:aFreq=48000; vFps=59.94; kbps=1641; mapAudio=[('0.1', '(und)')]; vWidth=640; vCodec=h264; Supported=True; millisecs=1497600; par=None; aKbps=None; par2=None; par1=None; dar2=None; mapVideo=0.0; vHeight=368; dar1=None; aCodec=mp4a / 0x6134706D
DEBUG:root:CACHE HIT! /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
DEBUGyTivo.video.transcodeTrue, 'TRANSCODE=NO, passing through mp4.'), /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4

.....

DEBUGyTivo.video.transcodeFalse, 'TRANSCODE=YES, vCodec h264 not compatible.'), /mnt/disk1/Media/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4

So it is failing on the H264 codec not compatible thing...


----------



## kearygriffin

NA9D said:


> This is what I'd like to do - upgrade my FFMPEG. Problem is, I'm not sure where to get a compiled version from. The version I am using is being supplied through iPKG but it does not seem to work well. So I'm not sure where to get another version compiled for the ARM processor. I'll admit I haven't looked real hard yet either...


Edit: I've removed this message because it was completely and utterly incorrect.


----------



## NA9D

kearygriffin said:


> Edit: I've removed this message because it was completely and utterly incorrect.


Damn! I was going to try that.


----------



## Yoav

NA9D said:


> If the Video Codec is H.264 then force transcoding because it's not compatible...or am I reading that wrong?


Not gonna comment about the rest, but
'!' is standard (from C) terminology for NOT.

So saying something like
if (A != B) is a test for A NOT equal to B.

So your whole interpretation of the rest is backwards.


----------



## kearygriffin

NA9D said:


> Damn! I was going to try that.


Sorry. But actually they way you fixed it accomplishes the same thing. (Putting the entire "mp4a / 0xblahblah" in the compatible audio codecs). My post was just changing the regular expression to accomplish the same thing.

I haven't looked into exactly how pyTivo tivo determines when to transcode, but what jumps out now that I am taking a closer look at both your ffmpeg -i output and the pyTivo log from above is:

vFps=59.94

Which may be causing pyTivo to force transcoding. (As it should if the frame rate is really 59.94 fps, because TiVo won't like that) Your mediainfo dump shows 29.97 as the fps, so it looks like ffmpeg is somehow getting an incorrect frame rate from that file.


----------



## NA9D

Yoav said:


> Not gonna comment about the rest, but
> '!' is standard (from C) terminology for NOT.
> 
> So saying something like
> if (A != B) is a test for A NOT equal to B.
> 
> So your whole interpretation of the rest is backwards.


OK. I never learned C (back in the day I taught myself Pascal but who uses that now!). I thought perhaps I might have it backwards. But then seeing the failure later on in the logs made me question it...

I would think files encoded from your build of iTivo should work and be pushable back but who knows at this point. And I haven't been able to get pyTivoX to configure properly to test this feature out with it...


----------



## NA9D

kearygriffin said:


> Sorry. But actually they way you fixed it accomplishes the same thing. (Putting the entire "mp4a / 0xblahblah" in the compatible audio codecs). My post was just changing the regular expression to accomplish the same thing.
> 
> I haven't looked into exactly how pyTivo tivo determines when to transcode, but what jumps out now that I am taking a closer look at both your ffmpeg -i output and the pyTivo log from above is:
> 
> vFps=59.94
> 
> Which may be causing pyTivo to force transcoding. (As it should if the frame rate is really 59.94 fps, because TiVo won't like that) Your mediainfo dump shows 29.97 as the fps, so it looks like ffmpeg is somehow getting an incorrect frame rate from that file.


Well, it looks like it all comes back to my install of ffmpeg. I really don't want to try compiling it for ARM9 (not sure I even have the compilers on the NAS) and wish I could find a useful version out there. I'll keep looking.

Rats...


----------



## Yoav

NA9D said:


> Well, it looks like it all comes back to my install of ffmpeg. I really don't want to try compiling it for ARM9 (not sure I even have the compilers on the NAS) and wish I could find a useful version out there. I'll keep looking.
> 
> Rats...


What OS is the NAS running? I know netbsd for ARM makes their packages easily available...


----------



## NA9D

Yoav said:


> What OS is the NAS running? I know netbsd for ARM makes their packages easily available...


It's whatever Linux distribution comes on the Buffalos.

I'm not sure if it's a variant of busy box or what. My unix isn't good enough to tell me how to know...


----------



## CuriousMark

NA9D said:


> This is what I'd like to do - upgrade my FFMPEG. Problem is, I'm not sure where to get a compiled version from. The version I am using is being supplied through iPKG but it does not seem to work well. So I'm not sure where to get another version compiled for the ARM processor. I'll admit I haven't looked real hard yet either...


You may have to set up the Precompiled C Cross Toolchain and compile it. I don't envy you that task.


----------



## moyekj

kearygriffin said:


> Which may be causing pyTivo to force transcoding. (As it should if the frame rate is really 59.94 fps, because TiVo won't like that) Your mediainfo dump shows 29.97 as the fps, so it looks like ffmpeg is somehow getting an incorrect frame rate from that file.


 Actually the mediainfo dump on the tigger show does show 59.94 fps so ffmpeg does appear to be reporting it correctly.


> Complete name : /Volumes/Media-1/Video/iTivoDownloads/My Friends Tigger & Pooh/My Friends Tigger & Pooh - Darby Goes Woozle Sluethin'; How the Tigger Lost His.mp4
> ...
> Frame rate : 59.940 fps
> ...


----------



## kearygriffin

moyekj said:


> Actually the mediainfo dump on the tigger show does show 59.94 fps so ffmpeg does appear to be reporting it correctly.


My bad. I was looking at the galactica episode mediainfo dump.

NA9D: You may want to see if the galactica episode pushes, now that the audio detection is probably OK.


----------



## wmcbrine

> If the Video Codec is H.264 then force transcoding because it's not compatible...or am I reading that wrong?


Backwards.



> DEBUGyTivo.video.transcodeTrue, 'TRANSCODE=NO, passing through mp4.')


This indicates that it passed the test. Period.



> DEBUGyTivo.video.transcodeFalse, 'TRANSCODE=YES, vCodec h264 not compatible.')


This is just standard output from tivo_compatible(). It's what _would_ happen if you did a pull. It doesn't indicate that the file is going to be transcoded... nothing in those log excerpts does.

If you're seeing the TRANSCODE=NO message, but not seeing a transfer, you may just need to wait. Push transfers don't start immediately. Or you may have to restart the TiVo... give it a few minutes, though.


----------



## Yoav

NA9D said:


> It's whatever Linux distribution comes on the Buffalos.
> 
> I'm not sure if it's a variant of busy box or what. My unix isn't good enough to tell me how to know...


I'm curious: what does


Code:


 uname -a

report?


----------



## NA9D

Yoav said:


> I'm curious: what does
> 
> 
> Code:
> 
> 
> uname -a
> 
> report?


Linux MEDIASERVER 2.6.16.16-arm1 #6 Fri Aug 31 13:07:15 JST 2007 armv5tejl unknown


----------



## CuriousMark

There are several projects out there for upgrading kernels and putting other distributions on it. foonas and debian are in various states of testing and availability. I have the same NAS, but am trying to stay as close to stock as I can since it has a lot of great functionality that would need to be recreated with quite a bit of effort if I replaced it with Debian. I am following this discussion with interest and plan on following in NA9D's footsteps this weekend.


----------



## Yoav

NA9D said:


> Linux MEDIASERVER 2.6.16.16-arm1 #6 Fri Aug 31 13:07:15 JST 2007 armv5tejl unknown


Ok I failed to find any dpkgs or rpms that would work. Welcome to the world of cross-compiling  Have fun!! (it's a pain in da butt).

*edit* I spoke too soon:

http://packages.debian.org/lenny/ffmpeg

you may need to install a bunch of dependencies (potentially really huge ones like libc), but I think that should work on your kernel... I see there is also an ipkg installation (ipkg install ffmpeg). I assume that's the one you have?

Or just use OpenLink? which has the compiler and tools installed...?
http://buffalo.nas-central.org/index.php/OpenLink
http://buffalo.nas-central.org/wiki/Precompiled_C_development_environment,_running_on_the_LS

http://buffalo.nas-central.org/wiki/Installing_pyTivo_on_Linkstation_Live


----------



## NA9D

Yoav said:


> Ok I failed to find any dpkgs or rpms that would work. Welcome to the world of cross-compiling  Have fun!! (it's a pain in da butt).
> 
> *edit* I spoke too soon:
> 
> http://packages.debian.org/lenny/ffmpeg
> 
> you may need to install a bunch of dependencies, but I think that should work on your kernel...


That was the one that I was looking at that I was thinking of doing. But it was too late last night. I downloaded the .deb file but then my lack of unix/linux skills failed me...

Not sure how to get the .deb file loaded properly onto the NAS...


----------



## NA9D

OK. I've got a push successfully going now! Thanks for the help.

My main Tivo is not starting pushes for some reason. I think I know why but I can't say for sure...

Anyhow, on my second Tivo, the Battlestar Galactica push started right away and it started playing which is very cool....

Now to figure out why Yoav's H.264 encodes from pyTivox have a 59 fps frame-rate. I wonder if I need to change the de-interlacing setting? Dunno....


----------



## txporter

According to the pytivo configure wiki, 59.94 fps should be ok.

Also, I answered my own question from earlier about pushing more than one show at a time. It seems to work fine. I clicked on one and then a few seconds later I pushed another and they both transferred up fine.

One question though, should I be able to watch a mp4 file that is being pushed up while it is actually being pushed? I was pushing 10min clips and wasn't able to view until they were finished. Was wondering if that is what I should expect or maybe just because they were short?

Jason

P.S. I am not sure what all is possible on the pytivo push screen, but is there a way to indicate if something is in the process of being pushed or that one is queued, or anything like that?


----------



## Yoav

NA9D said:


> OK. I've got a push successfully going now! Thanks for the help.
> 
> My main Tivo is not starting pushes for some reason. I think I know why but I can't say for sure...
> 
> Anyhow, on my second Tivo, the Battlestar Galactica push started right away and it started playing which is very cool....
> 
> Now to figure out why Yoav's H.264 encodes from pyTivox have a 59 fps frame-rate. I wonder if I need to change the de-interlacing setting? Dunno....


EDIT:
pyTivo doesn't encode to h.264

Can you explain what steps you took to end up with this framerate?


----------



## moyekj

txporter said:


> According to the pytivo configure wiki, 59.97 fps should be ok.


 Interlaced frames at that rate is fine, but for mp4 files we are talking about progressive frames. I don't think I've tried 60p framerate for mp4 files (30p or 24p is more common) so don't know if it works or not.


----------



## NA9D

Yoav said:


> EDIT:
> pyTivo doesn't encode to h.264
> 
> Can you explain what steps you took to end up with this framerate?


Don't know. I've used your standard mencoder settings for everything on these. The only change was to add an audio delay of 0.25 seconds.

I'll check my other files encoded by iTivo.


----------



## NA9D

txporter said:


> .
> 
> One question though, should I be able to watch a mp4 file that is being pushed up while it is actually being pushed? I was pushing 10min clips and wasn't able to view until they were finished. Was wondering if that is what I should expect or maybe just because they were short?


I was able to. Well, I wasn't supposed to since I'm at work (dang Slingboxes!) but I did check it and it was OK.


----------



## txporter

moyekj said:


> Interlaced frames at that rate is fine, but for mp4 files we are talking about progressive frames. I don't think I've tried 60p framerate for mp4 files (30p or 24p is more common) so don't know if it works or not.


Ok, I follow that. The mp4s that I have pushed so far have been 23.976 fps progressive and have played fine.

One question though, all of the 10min clips have had green blocky (stuff?) in the first few seconds of the video. I saw a post about this in the streambaby thread from westside_guy. I didn't see a solution for it. Does anyone know what is causing this or how to fix?

Jason


----------



## Yoav

NA9D said:


> Don't know. I've used your standard mencoder settings for everything on these. The only change was to add an audio delay of 0.25 seconds.
> 
> I'll check my other files encoded by iTivo.


oh iTiVo!! (sorry my bad I should have figured it out).

Do you know which 'format' you used as the basis? I'm looking at all the movies I created with iTiVo, and all the ones I've checked so far are mp4's with a framerate of 29.97

-- yoav


----------



## NA9D

Yoav said:


> oh iTiVo!! (sorry my bad I should have figured it out).
> 
> Do you know which 'format' you used as the basis? I'm looking at all the movies I created with iTiVo, and all the ones I've checked so far are mp4's with a framerate of 29.97
> 
> -- yoav


Oh, it's the iPod/iPhone Super-Res setting. Perhaps there's something in that?


----------



## Yoav

NA9D said:


> Oh, it's the iPod/iPhone Super-Res setting. Perhaps there's something in that?


Just tried using iphone super res on my computer:



Code:


Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/yoav/Desktop/tivo/The Daily Show With Jon Stewart - Tom Selleck.mp4':
  Duration: 00:00:29.01, start: 0.000000, bitrate: 1457 kb/s
    Stream #0.0(und): Video: h264, yuv420p, 640x480, 29.97 tb(r)
    Stream #0.1(und): Audio: aac, 48000 Hz, stereo, s16
At least one output file must be specified

So that's not the problem... (I also tried adding a -delay 0.25 to see if that changed anything.. nothing changed).


----------



## txporter

The seeking (FF/RW) within a mp4 is choppier or more discrete than the MPEG2 tivo recordings. Is this due to the decoding requirements of mp4 on the tivo, or is this something I can affect with min/max GOP (or something else) when encoding?

Jason


----------



## euckersw

euckersw said:


> So what if we were to use a program like mkv2vob to remux a mkv file with h264 to a mp4 file?:
> 
> http://www.videohelp.com/tools/mkv2vob
> 
> Would that be able to trick Tivo into accepting the remuxed mkv without transcoding?


Just in case anyone was going to try this, mkv2vob does not remux mkv files to mp4 files (in fact, it really only renames the resulting mpg to mp4). I've looked around for other good solutions for remuxing a mkv to mp4, but have yet to find one. Please let me know if you anybody else has any success...


----------



## moyekj

euckersw said:


> Just in case anyone was going to try this, mkv2vob does not remux mkv files to mp4 files (in fact, it really only renames the resulting mpg to mp4). I've looked around for other good solutions for remuxing a mkv to mp4, but have yet to find one. Please let me know if you anybody else has any success...


 There is the ffmpeg way, though it may lead to audio/video sync problems. Something like:
*ffmpeg -i "file.mkv" -acodec copy -vcodec copy -copyts -f mp4 "file.mp4"*

i.e. copy audio & video & timestamps and mux into mp4 container.


----------



## NA9D

Yoav said:


> Just tried using iphone super res on my computer:
> 
> 
> 
> Code:
> 
> 
> Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/yoav/Desktop/tivo/The Daily Show With Jon Stewart - Tom Selleck.mp4':
> Duration: 00:00:29.01, start: 0.000000, bitrate: 1457 kb/s
> Stream #0.0(und): Video: h264, yuv420p, 640x480, 29.97 tb(r)
> Stream #0.1(und): Audio: aac, 48000 Hz, stereo, s16
> At least one output file must be specified
> 
> So that's not the problem... (I also tried adding a -delay 0.25 to see if that changed anything.. nothing changed).


I'll check another encode with your software. Maybe this one was done with Visual Hub or something. I've used that in the past and thought this one had been done by iTivo but maybe not...


----------



## Rdian06

moyekj said:


> There is the ffmpeg way, though it may lead to audio/video sync problems. Something like:
> *ffmpeg -i "file.mkv" -acodec copy -vcodec copy -copyts -f mp4 "file.mp4"*
> 
> i.e. copy audio & video & timestamps and mux into mp4 container.


Technically that is the ffmpeg way, but -acodec copy into -f mp4 is horribly broken right now. The code fails to set the audio framesize (and apparently the bits per sample as well) for "copy" and so it bombs out immediately with an error.

And when you start with ac3 5.1 channels in the mkv and do:

*ffmpeg -i "file.mkv" -ab 448k -ar 48000 -acodec ac3 -vcodec copy -copyts -f mp4 "file.mp4"*

ffmpeg reports that it is transcoding from ac3 5.1 to ac3 5.1 in the console output, but ffmpeg -i on the resulting mp4 shows ac3 stereo. I think it's hard coding the header info somewhere 

All in all, ffmpeg's mp4 muxing isn't very reliable right now. I tested all this with an ffmpeg SVN snapshot from last night.


----------



## NA9D

Yoav said:


> Just tried using iphone super res on my computer:
> 
> 
> 
> Code:
> 
> 
> Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/yoav/Desktop/tivo/The Daily Show With Jon Stewart - Tom Selleck.mp4':
> Duration: 00:00:29.01, start: 0.000000, bitrate: 1457 kb/s
> Stream #0.0(und): Video: h264, yuv420p, 640x480, 29.97 tb(r)
> Stream #0.1(und): Audio: aac, 48000 Hz, stereo, s16
> At least one output file must be specified
> 
> So that's not the problem... (I also tried adding a -delay 0.25 to see if that changed anything.. nothing changed).


Yoav

That encode was done with iTivo, but I'm not sure how. All my other iTivo encodes show up fine. So not sure what happened with that one...


----------



## Yoav

NA9D said:


> Yoav
> 
> That encode was done with iTivo, but I'm not sure how. All my other iTivo encodes show up fine. So not sure what happened with that one...


If I understand you right, iTiVo made the 60fps video for sure, but that's the only one that seems to be 60.. all the others were 29.97?

I guess we need to figure out if there was somehting unique/special about the source movie that iTiVo encoded (the flags to mencoder do not specify a new framerate, so I'd guess that the source was 60fps... except broadcast TV doesn't do 60fps -- so I'm at a loss for now..)


----------



## wmcbrine

59.94 fps is standard for 720p. So yeah, broadcast TV does it.


----------



## Yoav

wmcbrine said:


> 59.94 fps is standard for 720p. So yeah, broadcast TV does it.


Whoa I thought 720p was 30fps (progressive, but 30fps anyways).

Ok, well we now know how he got the file then . He must have downloaded from a channel broadcasting in 720p. I'm gonna assume that the 60 fps is not a bug then...


----------



## jcthorne

euckersw said:


> Just in case anyone was going to try this, mkv2vob does not remux mkv files to mp4 files (in fact, it really only renames the resulting mpg to mp4). I've looked around for other good solutions for remuxing a mkv to mp4, but have yet to find one. Please let me know if you anybody else has any success...


I got megui to work. mp4 muxer under tools menu. Unfortunatly the output shows the ac3 audio as stereo not 5.1 like the original and MPC will not play the file as is. It did transfer to the tivo. But has aspect ratio and folderization problems.


----------



## jcthorne

NA9D said:


> OK. I've got a push successfully going now! Thanks for the help.
> 
> My main Tivo is not starting pushes for some reason. I think I know why but I can't say for sure...
> Dunno....


I also cannot say for sure but if your problem that you cannot talk about is anything similar to my problem that I could not talk about,

changing mind.tivo.com to stagingmind.tivo.com in the file mind.py may fix you up for now. Or not.


----------



## NA9D

Yoav said:


> If I understand you right, iTiVo made the 60fps video for sure, but that's the only one that seems to be 60.. all the others were 29.97?


I was going to make that comment that wmcbrine made that 720p is at 60 fps, but he beat me to it. Most progressive formats (480p, 720p, 1080p) are at 60fps. The only exception really are some movies encoded at the original 24 fps (Vudu does this with their 1080p movies for example). But no one uses 24 fps in broadcast. Keep in mind that the 30 fps for 480i and 1080i is really two interlaced frames so it still really is 60 fps...

Anyhow, it must have been recorded from an HD channel but I'm not sure why I would have recorded a cartoon from an HD source! And what's odd is that none of my other HD content is showing up this way...Most channels use 1080i and not 720p. Guess I'll have to record a definite 720p show and see. I think ABC uses 720p and PBS. I forget but it's not hard to find out.


----------



## wmcbrine

ABC and Fox are 720p. Also ESPN, National Geographic and DirecTV 101. PBS is 1080i at the network level, but some affiliates moved to 720p so they could squeeze more subchannels in. There's also a 1080i ABC affiliate out there.


----------



## NA9D

wmcbrine said:


> ABC and Fox are 720p. Also ESPN, National Geographic and DirecTV 101. PBS is 1080i at the network level, but some affiliates moved to 720p so they could squeeze more subchannels in. There's also a 1080i ABC affiliate out there.


I'm downloading an episode of Sarah Connor Chronicles now using iTivo. We'll see what we get!


----------



## txporter

I attempted to push a mp4 last night with AC3 audio and it forced a transcode. Not sure what the issue is there, but since I only listen in stereo, it's no biggie to convert the audio to 2ch AAC.

I am also fairly certain that pushing mp4 to my Tivos (well, I have only pushed to one of them) is somehow killing its ability to connect to the tivo servers for the service connection (even when not pushing video). Has anyone else seen this? I will check more into this over the weekend.

I am still seeing green blocky artifacts in the first ~5s of every video uploaded, XVID4PSP created, Handbrake w/ Web optimized and Handbrake w/o Web optimized. Am I the only one seeing this?

Jason


----------



## NA9D

txporter said:


> I am still seeing green blocky artifacts in the first ~5s of every video uploaded, XVID4PSP created, Handbrake w/ Web optimized and Handbrake w/o Web optimized. Am I the only one seeing this?


I see this quite often when streaming, etc. On my Roku Photobridge it would happen fairly frequently when doing FF/RW through files.

I think it's just that all the different patterns on the file get out of sync initially (not sure how to describe it better than that) and then it gets figured out and assembled correctly...


----------



## moyekj

I encoded to mp4+h.264+ac3 a 720p mpeg2 episode of Lost and sure enough both the source mpeg2 and destination mpeg4 are 59.94 fps and it streams and pushes fine to Tivo.


----------



## burnside

moyekj said:


> I encoded to mp4+h.264+ac3 a 720p mpeg2 episode of Lost and sure enough both the source mpeg2 and destination mpeg4 are 59.94 fps and it streams and pushes fine to Tivo.


What is the command you use to encode the mpeg2 to mp4+h.264+ac3 if I may ask?


----------



## moyekj

burnside said:


> What is the command you use to encode the mpeg2 to mp4+h.264+ac3 if I may ask?


ffmpeg -y -i "inputFile.mpg" -vcodec libx264 -coder 0 -level 41 -r 29.97 -sameq -g 300 -bufsize 14745k -b 5000k -maxrate 16000k -bug "+autodetect+ms" -me epzs -trellis 2 -mbd 1 -acodec copy -f mp4 "outputFile.mp4"

(P.S. You may want to remove the -r 29.97 option to use same frame rate as source and adjust the -b 5000k option for size vs. quality tradeoff)


----------



## txporter

Is there any way to make it possible for a pull to bring in a mp4 file without a transcode? I know my wife would prefer this...and I would too if the pushes are what are causing my Tivo's service connection calls to fail.

Jason


----------



## wmcbrine

txporter said:


> I am also fairly certain that pushing mp4 to my Tivos (well, I have only pushed to one of them) is somehow killing its ability to connect to the tivo servers for the service connection (even when not pushing video). Has anyone else seen this?


Absolutely not.


----------



## jg123

Rdian06 said:


> All in all, ffmpeg's mp4 muxing isn't very reliable right now. I tested all this with an ffmpeg SVN snapshot from last night.


If you have success transcoding mkv to mp4 using ffmpeg, I would love to hear it. I'm having the same issues with the latest builds of x264 and ffmpeg.


----------



## moyekj

A big downside to mpeg4 encodings natively on the Tivo is that FF/REW don't seem to work very well, especially FFx1.

This post seems to sum up pretty nicely why FF/REW with mpeg4 is not very good compared to mpeg2:
http://www.dbstalk.com/showthread.php?p=1999410#post1999410

In my small list of h.264 encodings there are some that FF better than others, but so far any h.264 clips I have produced with either ffmpeg or handbrake are not good for FFx1. I did some experimentation with ffmpeg altering GOP size from 30-300 range and reference frames from 1-6 range, etc. but they didn't really seem to have much effect.

If anyone here has h.264 encoding recipe that works well for FFx1 please post.

EDIT: Forgot to mention that wmv files with VC-1 video and WMA9 audio work pretty well with FFx1 when streaming natively to Tivos. (streambaby doesn't yet support native streaming of those files but I added that in tivostream a while back. There is a way to get streambaby to stream those natively if someone is interested)


----------



## txporter

There are a few different things that determine whether or not to place an I-frame (key frame) in h.264. One of them is the min/max keyint value (like GOP). The recommendations that I have seen is to use fps for min and 10xfps for max. That would give you key frames every 1 to 10s. The other thing that can add a key frame is scene cut. This is a function in h.264 (x.264) that attempts to determine when there is a scene change and will add a key frame. The number set determines sensitivity to a scene change. See this post for a decent summary of x.264 settings. So, it is possible that your videos that seek better have more scene changes and therefore more key frames.

My understanding of reference frames is that they are used to determine if the scene is static or moving and will allow better compression if you use a larger number. I don't think they should affect seeking (although I guess they could since I would expect that a higher number would require more processing power to decode).

Finally, here are couple of posts, one on mencoder settings and the other from doom9 forum about blu-ray. The mencoder post does say that keyint does affect seeking behavior. And the doom9 post lists the blu-ray values of -keyint as 24 and --min-keyint as 2. Based on that, it looks possible that in order to improve seeking in blu-ray they went with an extremely low value for min/max keyint. I don't have blu-ray, so I don't know what FF/RW is like, but I think we should be able to try some encodes with similar values to see if seeking is improved.

Jason


----------



## moyekj

x264 -> ffmpeg mappings can be found here:
http://ffmpeg.x264.googlepages.com/mapping

So x264->ffmpeg mappings for settings you mentioned would be:
--keyint maps to -g
--min-keyint maps to -keyint_min

I already experimented alot with -g argument (I tried as low as 12) as I mentioned in my post above and that alone didn't seem to help. I also adjusted -refs (reference frames) from 1-6 range and it didn't seem to help either. I really expected that combination of decreasing GOP size and increasing reference frames should have helped, but it didn't.
I don't see how -keyint_min is going to help but it's worth a try.

I suppose there is a risk that if you try and optimize too much for improved FF performance you may well end up sacrificing much of the efficiency/space savings compared to mpeg2.

I don't know offhand if there is a utility for viewing GOP structure of H.264 encodings but that sounds like it would be useful. (I think VideoRedo has such a utility for mpeg2 but not sure if it will work for mpeg4).


----------



## txporter

I think you can use MediaInfo to get the information that you are looking for.

I am not sure what the -keyint_min defaults to in ffmpeg, but I would guess that it could limit how low you could go with your -g setting (-keyint). And frankly, I guess I really don't know how often key frames are set in MPEG2? I assumed they are more often than H.264...but I never checked.

I will play around with some of these when I get a chance. Let me know if you find anything interesting. It is always possible that the seeking on the tivo is limited by the hardware decoding power, but it does sound like this is an overall problem with h.264.

Jason


----------



## moyekj

Thanks. For mpeg2 GOP size is typically pretty small (I think 18 frames/GOP is max for NTSC) and there is 1 I frame per GOP. I think a typical structure is something like: IBBPBBPBBPBB

According to the x264->ffmpeg reference I posted above looks like GOP size for h.264 encodings are typically much larger than that (250 is given as default).

To be honest though a lot of this is over my head and I don't really understand much about it which is why I post here in the hopes with others with a lot more understanding would be better suited to experiment and/or post here.


----------



## txporter

moyekj said:


> To be honest though a lot of this is over my head and I don't really understand much about it which is why I post here in the hopes with others with a lot more understanding would be better suited to experiment and/or post here.


Yeah, I am hacking my way through this. Trying to learn some as I go. At least it is relatively easy to test this now that I can push things up to the tivo as mp4. If I get a chance, I will run some test cases tonight.

Jason


----------



## Dan203

moyekj said:


> Thanks. For mpeg2 GOP size is typically pretty small (I think 18 frames/GOP is max for NTSC) and there is 1 I frame per GOP.


18 frames is max for an NTSC DVD, but the actual max for an MPEG-2 program stream is 1024 frames. It's very common for streams from DSS or digital cable to have really long GOPs (i.e. 30-50 frames). They do this because it increases compression and for a realtime playback system it doesn't effect quality too much. Although it does effect features like FF/RW, which is why on a S3 or DirecTiVo, which records the bitstream directly, some channels seem to FF/RW smoother then others.



moyekj said:


> According to the x264->ffmpeg reference I posted above looks like GOP size for h.264 encodings are typically much larger than that (250 is given as default).


H.264 videos do typically have longer GOPs in general, but most encoding programs try to keep to around 50 frames max so that features like seeking and FF/RW are more practical. The longer the GOP the harder it is too seek and the harder it is to create smooth, usable, FF/RW functionality.

Dan


----------



## moyekj

Thanks for your input Dan.

I experimented tonight again with GOP size and reference frames and got no improvement at all (-keyint_min=2 didn't have any noticeable impact either). The source clip is a recording from Tivo: a short 1 min 480i mpeg2 with Video CBR of 3Kbps and 29.97 fps. That clip has smooth FF/REW in mpeg2 format but various h.264 ffmpeg encodes with low GOP size and several reference frames failed to yield a good smooth FF/REW. No better with Handbrake and x264 encoder either. Could well be from that kind of source it's not possible to get a workable h.264.

The following h.264 encoding on the other hand behaves pretty nicely with FF (not very good with REW)
systm--0063--dolby--hd.h264.mp4 so I know the Tivo h.264 decoder is able to do smooth FF. The key specs according to MediaInfo on this are:
L3.1 AVC1 (H.264), 720p, 2 reference frames, 2Kbps CBR, 24 fps
MediaInfo doesn't give any other information on GOP structure AFAIK.

I also tried an mpeg2 720p clip (also from Tivo) as a starting point but had no better luck getting a smooth FF h.264 encoding from it either using ffmpeg.

From either of the 2 mpeg2 sources above on the other hand if I use Microsoft Media Encoder 9 or Microsoft Expression Encoder 2 (Silverlight encoder) to generate a wmv encoding (VC-1 AP video with wma9 audio) the result is a nice smooth FF in both cases when streaming to Tivo (REW is choppy as expected).

So not sure what else to try at this point. It's not a huge deal to me as I don't use FFx1 that much but still it bothers me and would be nice to get to the bottom of this.


----------



## txporter

I took a 10min clip at the beginning of Empire Strikes Back and encoded it 3 different ways in handbrake: my default profile (see P.S.), changed keyint to 24 (from 240) and keyint-min to 2 (from 24), and another with keyint=2/keyint-min=2 + no Pframe skip/DCT decimate. In all case, FFx1 looks fine. FFx2 (whatever speed of 2nd FF press) looks choppy in clip #1, but DOES look smoother in clips #2 & #3. I am not sure I can tell a difference between #2 and #3 though. It looks like each of those changes added another ~10% to the file size.

In MediaInfo, if you move to one of the other views (anything other than Basic), you will see what the encoding setting are for the mp4 that you are looking (at least I do with Handbrake).

Jason

P.S. Default profile (pulled from MediaInfo, text view):


Code:


cabac=1 / ref=3 / deblock=1:0:0 / analyse=0x3:0x133 / me=umh / subme=9 / psy_rd=1.0:0.0 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=0 / 8x8dct=1 / cqm=0 / deadzone=21,11 / chroma_qp_offset=-2 / threads=6 / nr=0 / decimate=1 / mbaff=0 / bframes=3 / b_pyramid=1 / b_adapt=1 / b_bias=0 / direct=3 / wpredb=1 / keyint=240 / keyint_min=24 / scenecut=40(pre) / rc=crf / crf=17.9 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / ip_ratio=1.40 / pb_ratio=1.30 / aq=1:1.00


----------



## txporter

txporter said:


> One question though, should I be able to watch a mp4 file that is being pushed up while it is actually being pushed? I was pushing 10min clips and wasn't able to view until they were finished. Was wondering if that is what I should expect or maybe just because they were short?


Ok, to answer my own question: The mp4 pushes seem to load in 10-15min chunks. The short clips were...shorter than that, so I was not able to watch as they loaded. If I push a longer video, then I can start watching after the first block of time is transferred up.

Jason


----------



## moyekj

txporter said:


> I took a 10min clip at the beginning of Empire Strikes Back and encoded it 3 different ways in handbrake: my default profile (see P.S.), changed keyint to 24 (from 240) and keyint-min to 2 (from 24), and another with keyint=2/keyint-min=2 + no Pframe skip/DCT decimate. In all case, FFx1 looks fine. FFx2 (whatever speed of 2nd FF press) looks choppy in clip #1, but DOES look smoother in clips #2 & #3. I am not sure I can tell a difference between #2 and #3 though. It looks like each of those changes added another ~10% to the file size.
> 
> In MediaInfo, if you move to one of the other views (anything other than Basic), you will see what the encoding setting are for the mp4 that you are looking (at least I do with Handbrake).
> 
> Jason
> 
> P.S. Default profile (pulled from MediaInfo, text view):
> 
> 
> Code:
> 
> 
> cabac=1 / ref=3 / deblock=1:0:0 / analyse=0x3:0x133 / me=umh / subme=9 / psy_rd=1.0:0.0 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=0 / 8x8dct=1 / cqm=0 / deadzone=21,11 / chroma_qp_offset=-2 / threads=6 / nr=0 / decimate=1 / mbaff=0 / bframes=3 / b_pyramid=1 / b_adapt=1 / b_bias=0 / direct=3 / wpredb=1 / keyint=240 / keyint_min=24 / scenecut=40(pre) / rc=crf / crf=17.9 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / ip_ratio=1.40 / pb_ratio=1.30 / aq=1:1.00


Thanks for the info. I'll have to try encoding from a DVD source to see if that's the difference. As Dan mentioned DVD sources have short GOP structure so are probably a better source candidate to begin with. Can you try the same profile on a Tivo SD recording to see how FFx1 looks? In my case FFx1 is not really choppy, but it doesn't appear to be any faster than normal play speed.


----------



## moyekj

BINGO! I just discovered that it is indeed the S3 H.264 decoder that seems to be the issue. When I play my same H.264 test clips on a Tivo HD unit FFx1 works pretty well. In comparison when playing on my S3s FFx1 doesn't really work at all - it's basically same as play speed.
So while I'm sure encoding parameters make a difference to some degree the basic problem is that S3 H.264 decoder is just not up to par with the Tivo HD H.264 decoder.

This holds true for S3 VC-1 decoder vs THD VC-1 decoder as well as there are Netflix streams that play fine on THD units but don't work properly on S3 units.


----------



## txporter

moyekj said:


> BINGO! I just discovered that it is indeed the S3 H.264 decoder that seems to be the issue.


That is an interesting data point. Hopefully that is something that Tivo is aware of and can fix with software.

Jason


----------



## Dan203

That is very strange.

FYI: The length of the GOPs in the source MPEG-2 file should not effect the output H.264 file in anyway. The encoder has to decode the source file to raw video frames before reencoding, so the structure of the source file never comes into play. However the framerate and whether or not the source is interlaced can effect the output file. So those are the things you should probably look at when trying to figure out why one file works OK and another does not.

Dan


----------



## steve614

moyekj said:


> BINGO! I just discovered that it is indeed the S3 H.264 decoder that seems to be the issue. When I play my same H.264 test clips on a Tivo HD unit FFx1 works pretty well. In comparison when playing on my S3s FFx1 doesn't really work at all - it's basically same as play speed.
> So while I'm sure encoding parameters make a difference to some degree the basic problem is that S3 H.264 decoder is just not up to par with the Tivo HD H.264 decoder.
> 
> This holds true for S3 VC-1 decoder vs THD VC-1 decoder as well as there are Netflix streams that play fine on THD units but don't work properly on S3 units.


This may be the reason why.



bkdtv said:


> (From this post in another thread elsewhere)
> 
> The *TiVo Series3* has a Broadcom SoC @ 300MHz with 128MB of DDR400 memory. It also has another 128MB of DDR400 (256MB total) devoted to a *separate MPEG-4 / VC-1 decoder*. The *TivoHD* is based on a newer Broadcom SoC @ 300MHz with* integrated MPEG-4 / VC-1 decoder*; it has direct access to 256MB of DDR400 memory.


----------



## moyekj

Yes, despite being an older hardware design the S3 has proved to be better than HD/HDXL in some aspects (faster TTG and MRV for sure, and faster GUI response in general from my experience). However it's becoming pretty clear from this FFx1 problem and certain Netflix titles that play fine on HDs but not on S3s that the S3 H.264/VC-1 decoder hardware and/or firmware are inferior to HD/HDXL. Could well be just a firmware issue but I'm not sure how much motivation Tivo has to optimize the phased out S3 model at this point.


----------



## txporter

Sort of off-topic, but how long have Amazon Unboxed downloads been VC-1 rather than being transcoded to MPEG2? My wife just downloaded a few episodes of The Office and they are ~0.48GB for 21mins and ~0.94GB for 42min episodes. Download speeds are ~7 mbps.

Jason


----------



## moyekj

txporter said:


> Sort of off-topic, but how long have Amazon Unboxed downloads been VC-1 rather than being transcoded to MPEG2? My wife just downloaded a few episodes of The Office and they are ~0.48GB for 21mins and ~0.94GB for 42min episodes. Download speeds are ~7 mbps.
> 
> Jason


 VC-1 or H.264? If it's VC-1 then sounds like more network sniffing is in order to determine how to store wmv natively on Tivo. I think the PC downloads version is VC-1 but that doesn't mean that the Tivo version needs to be VC-1 as well.

I guess if FFx1 on S3 doesn't work well then that would imply H.264. Another good test would be to see if you are getting multi-channel AC3 audio then that would imply H.264 as well.

Either way looks like the door is wide open for Amazon HD downloads now...


----------



## txporter

moyekj said:


> VC-1 or H.264? If it's VC-1 then sounds like more network sniffing is in order to determine how to store wmv natively on Tivo. I think the PC downloads version is VC-1 but that doesn't mean that the Tivo version needs to be VC-1 as well.
> 
> I guess if FFx1 on S3 doesn't work well then that would imply H.264. Another good test would be to see if you are getting multi-channel AC3 audio then that would imply H.264 as well.
> 
> Either way looks like the door is wide open for Amazon HD downloads now...


Well, I guess I don't know for sure what the format is. I did some googling and found that they were using VC-1 for other downloads and I figured they wouldn't be using both formats....but maybe I am wrong with that assumption? Is there any good way to determine the format on a TivoHD? Files are copyprotected, so can't move or transfer them.

Jason


----------



## moyekj

txporter said:


> Well, I guess I don't know for sure what the format is. I did some googling and found that they were using VC-1 for other downloads and I figured they wouldn't be using both formats....but maybe I am wrong with that assumption? Is there any good way to determine the format on a TivoHD? Files are copyprotected, so can't move or transfer them.
> 
> Jason


 Do you have surround sound system that indicates what kind of audio you are getting out of the TivoHD? If it's Dolby surround then I would lean to H.264.


----------



## fyodor

Someone posted in the Pytivo forums that amazon was going to start providing downloads to Tivos in MP4 format. The guy doesn't cite a source, but he posted it a few days before we realized that native MP4 transfers were available, so if he's making it up, his timing is really good.

http://pytivo.krkeegan.com/mpeg-4-for-hmo-very-near-t682.html

FWIW, I _think_ that if you look at the "ready to watch" times for different programs, you can figure out if they're available in MP4/WMV or MPEG-2. The MPEG-2 stuff (overwhelming majority) has a longer "ready to watch" time for the Tivo version. Whereas it looks like the Office, which txporter reports as being in a more efficient format, has the same download time.



txporter said:


> Well, I guess I don't know for sure what the format is. I did some googling and found that they were using VC-1 for other downloads and I figured they wouldn't be using both formats....but maybe I am wrong with that assumption? Is there any good way to determine the format on a TivoHD? Files are copyprotected, so can't move or transfer them.
> 
> Jason


----------



## fyodor

txporter said:


> Download speeds are ~7 mbps.
> 
> Jason


Interesting. Is this faster than what you normally get for unbox downloads? I ask because my Tivo unbox downloads have always been considerably slower (2.5-3 megabits) than my computer unbox downloads (6 megabits). I wonder if this has something to do with the general slower-performance of transferring MPEG-2 to Tivos.


----------



## txporter

moyekj said:


> Do you have surround sound system that indicates what kind of audio you are getting out of the TivoHD? If it's Dolby surround then I would lean to H.264.


No, I don't have my Tivos hooked up through a stereo system. Anyone else? I believe that there were some free episode from Amazon for Office Season 5 (previews for the season or something). Maybe those have the same format.

Jason


----------



## txporter

fyodor said:


> Interesting. Is this faster than what you normally get for unbox downloads? I ask because my Tivo unbox downloads have always been considerably slower (2.5-3 megabits) than my computer unbox downloads (6 megabits). I wonder if this has something to do with the general slower-performance of transferring MPEG-2 to Tivos.


Could be. Might it also be transcoding their other content TO MPEG2? But yes, I got numbers more inline with that you mention when I first used it. I think it might have actually been sub-2 mbps.

Jason


----------



## Dan203

TiVo obviously added the ability to play VC-1 content with MS DRM when it added Netflix. If Unbox uses the same format for their PC content then it was probably trivial for TiVo to add native Unbox support to the S3/HD platform. It seems unlikely that they would transcode to yet another format (H.264) just for TiVo. Although I guess they might if they really wanted DD5.1 audio, since WMV files don't support DD5.1 yet.

Dan


----------



## jcthorne

The amazon unbox downloads are stereo only, no 5.1 at this time. Might be VC-1 or h264, have no way to tell for sure. Should we try sending vc-1 content to the tivohd via pytivo push?

I was the one who posted over at pytivo forum, h264 was a guess, it could have been vc-1 all along without me knowing but it was known to be a higher order compression and I assumed h264 when I saw it.


----------



## wmcbrine

jcthorne said:


> Should we try sending vc-1 content to the tivohd via pytivo push?


No, we can't just try; we have to know the "encodingType" string to use, assuming that one exists. This would mean capturing the control data either from an Amazon push, or from a TiVo Desktop push of VC-1, if it can do that without reencoding.

Edit: Or, someone could look through tivoapp for the strings "mpeg2ProgramStream" and "avcL41MP4", and see if there are any likely candidates nearby.


----------



## fyodor

txporter said:


> Could be. Might it also be transcoding their other content TO MPEG2? But yes, I got numbers more inline with that you mention when I first used it. I think it might have actually been sub-2 mbps.
> 
> Jason


It's possible that it's transcoding in real-time, but I'd assume that it'd be much less resource intensive to just have an MPEG-2 stored somewhere.


----------



## moyekj

If sound is indeed stereo only then this suggests VC-1.

On the other hand I do have conforming wmv files with VC-1 video and WMA stereo audio that stream natively to Tivos, however with Tivo Desktop 2.7 auto push they are currently transcoded to mpeg2. So the "Keary spy gear trick" (MITM attack to intercept SSL communications between Tivo Desktop and mime.tivo.com) can't be applied the same way that was done for the H.264 discovery.

Sniffing out communication between Tivo and mime.tivo.com after an Amazon push is initiated could reveal something, but that means Tivo networking setup would have to be changed to go through a proxy server (perhaps using DNS spoofing and the MITM attack). DNS spoofing I've done before and is pretty well documented and pretty easy on Linux. The MITM attacks I've seen a few methods posted but again looks easiest on Linux. I don't have a Linux server at home to mess with and trying to do all that on Windows Vista seems too hard for me.
Plus, would be good to find an Amazon download (preferably free) that is obviously not an mpeg2 file. I tried a free 3 minute clip from The Office but that was 0.5GB 50MB for < 3 minutes so obviously must be mpeg2 going by size alone.

I like wmcbrine's suggestion of looking through tivoapp binary:
First see if the avcL41MP4 is found with something like (on Linux or Mac or Windows cygwin):
strings tivoapp | grep -i mp4

Then look for other possible strings such as wmv:
strings tivoapp | grep -i wmv

(Anyone have an 11.0b tivoapp binary handy?)


----------



## fyodor

moyekj said:


> If sound is indeed stereo only then this suggests VC-1.
> 
> Plus, would be good to find an Amazon download (preferably free) that is obviously not an mpeg2 file. I tried a free 3 minute clip from The Office but that was 0.5GB for 3 minutes so obviously must be mpeg2 going by size alone.


It looks like the free, beginning of the season clips of the office list a longer download time for Tivos than computers, so this may actually be a reliable indicator.

F


----------



## moyekj

OK, I found a tivoapp 11.0b binary and interestingly enough "strings" command does find avcL41MP4 among other strings containing mp4. But more interestingly:
> strings tivoapp | grep -i wmv
.wmv
WMV9
.wmv

EDIT: Actually this one looks more likely to work:
> strings tivoapp | grep -i vc1
vc1ApL3
(This is in same general location as avcL41MP4)

So looks like *vc1ApL3* could be the encodingType setting we are looking for. I'm not home to be able to make the necessary modifications to mind.py to try a native push of wmv. If someone is available to try it I have a compatible wmv encoding you can try with here:
http://tivostream.googlecode.com/files/test_vc1_wma.wmv
(P.S. You also probably need to use video/x-ms-wmv for Content-Type)

EDIT: This is off topic for this thread but an interesting find in tivoapp:
video/x-tivo-mpeg&System=ts
So maybe there is a way of streaming mpeg2 transport streams to Tivo after all...


----------



## wmcbrine

OK, I tried modifications to pyTivo along the same lines as for h264, using "vc1ApL3", but it didn't work -- the TiVo made the request, but then immediately dropped the connection.

Edit: I tried changing the string, and got "Bad value for encodingType in bodyOfferModify". So it looks like "vc1ApL3" is a _good_ value... it's just not working, at least not with that test file.


----------



## Rdian06

moyekj said:


> EDIT: Actually this one looks more likely to work:
> > strings tivoapp | grep -i vc1
> vc1ApL3
> (This is in same general location as avcL41MP4)
> 
> So looks like *vc1ApL3* could be the encodingType setting we are looking for. I'm not home to be able to make the necessary modifications to mind.py to try a native push of wmv. If someone is available to try it I have a compatible wmv encoding you can try with here:
> http://tivostream.googlecode.com/files/test_vc1_wma.wmv
> (P.S. You also probably need to use video/x-ms-wmv for Content-Type)
> 
> EDIT: This is off topic for this thread but an interesting find in tivoapp:
> video/x-tivo-mpeg&System=ts
> So maybe there is a way of streaming mpeg2 transport streams to Tivo after all...


Hrmm, googling vc1ApL3 turns up only this thread, BUT googling vc1 ApL3 turns up mentions of TS streams:

http://www.techgadgets.in/portable-audio/2008/05/popcorn-hour-a-100-digital-media-streaming-system-pre-order-begins/

So maybe try VC-1 in a TS container?


----------



## moyekj

wmcbrine said:


> OK, I tried modifications to pyTivo along the same lines as for h264, using "vc1ApL3", but it didn't work -- the TiVo made the request, but then immediately dropped the connection.
> 
> Edit: I tried changing the string, and got "Bad value for encodingType in bodyOfferModify". So it looks like "vc1ApL3" is a _good_ value... it's just not working, at least not with that test file.


 That's too bad and maybe explains why TD+ currently transcodes wmv files.
NOTE: That test clip does STREAM natively via HME so I don't think it's an encoding issue.

So now I'm back to leaning towards H.264 format being used for Amazon downloads


----------



## Rdian06

Also, there is a table at this page:

http://wiki.multimedia.cx/index.php?title=VC-1

I'm guessing ApL3 = Advanced Profile, Level 3

Max Bitrate = 45 Mbps

Representative Formats

1920 x 1080 @ 24 Hz (1080p)
1920 x 1080 @ 30 Hz (1080i)
1280 x 720 @ 60 Hz (720p)

Hopefully that's an indicator of maxes rather than requirements.

The clip moyekj linked has a format of [email protected] according to mediainfo.


----------



## moyekj

Yes it's likely an indication of max levels just like avcL41MP4 was a max indication for H.264 which we know works with levels <= 4.1 for example and at different resolutions, frame rates, etc. Plus as I mentioned above the clip streams natively via HME so it's a known good encoding that Tivo can decode.


----------



## wmcbrine

Sorted -- it was the MIME type it didn't like, oddly enough (since "video/x-ms-wmv" is the standard). The MIME type it wants, I found in tivoapp: "video/bif". WTF? Anyway, it works. Will be in my repo shortly.


----------



## Dan203

Could the MIME type also be why pull mode doesn't work with these new formats? I was looking at the code and it appears that you send everything as video/x-tivo-mpeg. Perhaps it needs be video/mp4 to work in pull mode?

Dan


----------



## moyekj

wmcbrine said:


> Sorted -- it was the MIME type it didn't like, oddly enough (since "video/x-ms-wmv" is the standard). The MIME type it wants, I found in tivoapp: "video/bif". WTF? Anyway, it works. Will be in my repo shortly.


 Cool, it's good that there is a limited number of types so trial and error works.

NOTE for those trying this, the decoder is currently very picky about the encoding so it's likely only a small subset of wmv files which actually work. For example using VBR for WMA audio doesn't work (it has to be CBR and 2 channel only, I don't think 6 channel worked). It's been a while since I experimented so maybe it's less restrictive now, but be warned. For those interested in exploring further I documented how to generate encodings that will work here:
http://code.google.com/p/streambaby/wiki/video_compatibility
(Unfortunately only for Windows platform since it's from Microsoft codecs)

Because of the restrictions and if you are choosing to encode your own videos then H.264 with AC3 audio is a much better choice.


----------



## wmcbrine

Actually, I take it back -- I wasn't looking at the TiVo, just the error messages. I believe it was working the whole time. But what happens is, the TiVo makes its first request, closes the socket, and then makes a second, successful request. (Kinda reminiscent of what happens when streaming... which is equally unexplained.) It doesn't do this for mp4 or mpeg2.

BTW, I didn't choose video/bif randomly. Here it is in context:



Code:


.mp3
audio/mpeg
audio/
.mpg
.mpeg
video/mpeg
video/mpg
.mp4
video/mp4
.asf
.wmv
.bif
video/bif
.png
.gif
.jpg
.jpeg
image/

which looks like lists of extensions followed by their associated MIME types.

Edit: Anyway, it's in my repo now. Hopefully we can figure out how to suppress the dual requests later.


----------



## moyekj

Interesting, just so I understand correctly you are saying that actually using "video/x-ms-wmv" mime was working, but it also works with "video/bif"?


----------



## Dan203

Perhaps the dual request has something to do with DRM. It may make the first request to see if it needs to authorize the video and the second one to actually play it.

Dan


----------



## moyekj

So since it's now been determined Tivo can store native VC-1 files it's very likely the Amazon non-mpeg2 downloads are in VC-1 format wrapped in MS DRM (just like they are if downloading to your PC). So I guess no DD 5.1 audio even if Amazon does make HD downloads available soon.
I suspect that WMA 5.1 CBR (constant bit rate) audio probably can be decoded by Tivo but probably like AAC 5.1 it spits out some kind of stereo like sound. Will have to try an encoding with WMA 5.1 CBR just to see if that's legal.


----------



## Dan203

That makes the most sense.

Now that TiVo has this whole WMV DRM scheme in place I wonder if they're working with any of the other download services that use it? Like Cinemanow or Blockbuster Online. I don't personally use either one, but I'd be willing to try either one if they had something I wanted to watch.

Dan


----------



## Dan203

moyekj said:


> OK, I found a tivoapp 11.0b binary and interestingly enough "strings" command does find avcL41MP4 among other strings containing mp4.


Can you open the binary in a hex editor and see if you can find any other strings around the vc1ApL3 and avcL41MP4 strings which might be of interest? Maybe they support some other format that we don't know about yet

Dan


----------



## moyekj

Dan203 said:


> Can you open the binary in a hex editor and see if you can find any other strings around the vc1ApL3 and avcL41MP4 strings which might be of interest? Maybe they support some other format that we don't know about yet
> 
> Dan


 The full list appears to be:
tivo, windowsMedia, mpeg2ProgramStream, mpeg2TransportStream, avcL41MP4, vc1ApL3

hmm. auto pushes of mpeg2 transport streams would be interesting...

P.S. Around that vicinity there are also some copy protection (CCI byte) related strings like:
copyFreely, copyNever, copyNoMore, copyOnce, none
Maybe there is hope to turn off copy protection when these are auto-pushed to Tivos...

wmcbrine's idea of searching through tivoapp binary looks like has opened up a bunch of mini-projects to explore new capabilities. Good stuff.


----------



## wmcbrine

moyekj said:


> The full list appears to be:
> tivo, windowsMedia, mpeg2ProgramStream, mpeg2TransportStream, avcL41MP4, vc1ApL3


There are two lists -- the first one, just after "supportedEncodingType", has only avcL41MP4, vc1ApL3 and mpeg2ProgramStream. I think those are the only valid ones... at least, "windowsMedia" doesn't work.



> _P.S. Around that vicinity there are also some copy protection (CCI byte) related strings like: copyFreely, copyNever, copyNoMore, copyOnce_


Yeah, those look like the right values, but what's the key name? I figured it was "cgms", but that gets rejected, and I don't see any other likely candidates around there.



Dan203 said:


> Perhaps the dual request has something to do with DRM.


I don't think that's it. The last time I tried, it actually took three requests. And one time, it went on the first try (just after I'd switched to video/bif, which is why I'd thought that was the fix). All with the same test file.



> _Could the MIME type also be why pull mode doesn't work with these new formats?_


I'm pretty sure I tested that unsuccessfully already, but that might've been before I realized that no MIME type was being sent at the start of a file, so I'll check again.

Edit: Yeah, reconfirming that it doesn't work.


----------



## moyekj

I confirmed last night that Tivo can't decode 5.1/7.1 channel WMA3 audio. So looks like still only VC-1 AP (<= L3) video and 2 channel CBR WMA2 audio is supported as documented in the Wiki page.


----------



## Rdian06

moyekj, can you try this clip?

http://forum.videohelp.com/images/guides/p1872213/wvc1%20+%205.1ac3.asf

It's converted from one of Apple's Dark Knight trailers. VC-1 video and AC3 5.1 audio in an ASF container. VLC seems to handle it fine on Windows.

Found it in this thread:

http://www.videohelp.com/forum/archive/mux-5-1-ac3-into-wmvhd-t354346.html

If that works, I might be able to rip a bunch of my old HD DVDs and just toss them at the Tivo. Would have to transcode the audio from EAC3 to AC3, but that shouldn't be too bad. Albeit they'd be rather huge files...


----------



## moyekj

I can try it but I doubt it will work. I already tried asf container with VC-1 video and AC3 audio (and AAC audio) and it didnt' work. If you convert to wma2 CBR 2 channel audio then it probably would work...


----------



## Rdian06

Ah, nevermind then.


----------



## moyekj

Confirmed. With AC3 audio the Tivo won't actually reject it for either streaming or pushing, but when you go play it will actually buffer on the Tivo but you get no video and super fast playback. WMA2 CBR 2 channel audio is required by the looks of it for now...

NOTE: After converting that clip's audio to WMA2 with ffmpeg it then did work:
ffmpeg.exe -i wvc1_51ac3.wmv -vcodec copy -acodec wmav2 -ac 2 -f asf test.wmv


----------



## Rdian06

Just a quick note to say I finally updated the Windows installer for wmcbrine's pyTivo fork over the weekend.

New version pyTivo-wmcbrine-2009.03.19-RC1 uses his near latest code with MP4 & WMV non-transcode Push, better Push metadata, and fixed web config browser compatibility.

On my side of things this new installer should play nicer on Vista (it avoids writing anything to the pyTivo program folder during runtime) and the Python detection issues for 64 bit OSes should be fixed.

NOTE: Because we moved a lot of things around, you should UNINSTALL past versions AND start over with the installer's freshly generated conf file (in the new conf location) and re-customize rather than reusing your old conf.

Download from:

http://pytivo.krkeegan.com/updated-windows-installer-2009-03-21-t512.html


----------



## wmcbrine

Rdian06 said:


> New version pyTivo-wmcbrine-2009.03.19-RC1 uses his near latest code


You probably should update again.


----------



## Rdian06

wmcbrine said:


> You probably should update again.


Hey, I said "near" didn't I 

I need to figure out how to script a few things on my end to make updating easier. The NSI builder script is somewhat painful to update when files are added or removed (like with the mutagen and eyeD3 stuff.) Synching changes to existing stuff isn't so bad. But you change stuff like hourly


----------



## bsnelson

OK, this isn't directly TiVo related, but it seems relevant: Is there a way to transcode only the audio portion of a MP4 file, leaving the video unmodified? I have an MP4 that was transcoded by Handbrake that has AC3 audio. I'd like it to have AAC audio instead; is there a way with Handbrake (or anything else, for that matter) to do this conversion?

Brad


----------



## moyekj

bsnelson said:


> OK, this isn't directly TiVo related, but it seems relevant: Is there a way to transcode only the audio portion of a MP4 file, leaving the video unmodified? I have an MP4 that was transcoded by Handbrake that has AC3 audio. I'd like it to have AAC audio instead; is there a way with Handbrake (or anything else, for that matter) to do this conversion?
> 
> Brad


 Something like this with ffmpeg may work:
ffmpeg -i INPUT.mp4 -vcodec copy -copyts -acodec libfaac -f mp4 OUTPUT.mp4
(This will only re-encode audio)

With the handbrake GUI you could do it by using your MP4 as source, but that will re-encode both video and audio thus losing quality and taking a long time.
Not sure if there is a way to do it with handbrakeCLI (the command line executable that handbrake calls).


----------



## bsnelson

Thank you very much, moyekj! The conversion took only a few minutes and I'm ten minutes into watching it and there are no sync issues or anything else funky as yet. 

Awesome!

Brad


----------



## DrewS3

How can I tell if a Tivo recording is in Mp4? All my HD recordings are about 6GB/hr. Does this mean they are all mpeg2?


----------



## wmcbrine

The TiVo doesn't (yet) make MP4 recordings. Only material transferred over the network can be in MP4; and, so far, all of it is marked as copy prohibited on the TiVo, so you can't extract it back to a .TiVo file. There doesn't seem to be any indication on the info screens, either.


----------



## DrewS3

OK, thanks. I misunderstood which direction these transfers were going.
Are all TivoHD recordings in MPEG2? Do cable companies have plans to broadcast other codecs?


----------



## morac

DrewS3 said:


> OK, thanks. I misunderstood which direction these transfers were going.
> Are all TivoHD recordings in MPEG2? Do cable companies have plans to broadcast other codecs?


The TiVoHD records the videos stream as sent by the cable company. All cable companies currently use MPEG2. Many do have plans for MPEG4 at some point in the future, but nothing that's been formally announced.


----------



## fyodor

DrewS3 said:


> OK, thanks. I misunderstood which direction these transfers were going.
> Are all TivoHD recordings in MPEG2? Do cable companies have plans to broadcast other codecs?


It's going to be a while-you need the whole pipeline to be upgraded (new set top boxes, switches, etc). Also, for reasons that would take a while to explain, the benefits of H.264 encoding are not as great for transmitted cable signals as they are for "local" video, so the benefits are going to only be moderate.

For now, you'll mostly see H.264 for satellite systems, where the simpler infrastructure (satellite to dish) makes it easier to upgrade.


----------



## Dan203

fyodor said:


> Also, for reasons that would take a while to explain, the benefits of H.264 encoding are not as great for transmitted cable signals as they are for "local" video, so the benefits are going to only be moderate.


I'd like to hear the explanation as to why this would be if you have time. I'm in the video industry (see sig) and have a good understanding of local TS streams so feel free to get technical. Is this a QAM issue? Or maybe since the cable companies would have to encode to H.264 on the fly they can't use some of the more advanced features of the codec that make it so efficient?

Just curious

Dan


----------



## fyodor

Well, I should clarify-a "locally" stored TS file streamed from the cable system would still have the same compression characteristics.

My basic understanding is that one of the main benefits of h.264 over mpeg-2 for say, a blu-ray disc is the lower number of keyframes. However, because of greater potential for lost packets and/or the need to keep channel change delay lower, h.264 transport streams used for satellite and cable have a higher number of keyframes and accordingly don't quite get as much extra benefit.



Dan203 said:


> I'd like to hear the explanation as to why this would be if you have time. I'm in the video industry (see sig) and have a good understanding of local TS streams so feel free to get technical. Is this a QAM issue? Or maybe since the cable companies would have to encode to H.264 on the fly they can't use some of the more advanced features of the codec that make it so efficient?
> 
> Just curious
> 
> Dan


----------



## morac

fyodor said:


> My basic understanding is that one of the main benefits of h.264 over mpeg-2 for say, a blu-ray disc is the lower number of keyframes. However, because of greater potential for lost packets and/or the need to keep channel change delay lower, h.264 transport streams used for satellite and cable have a higher number of keyframes and accordingly don't quite get as much extra benefit.


I'm not sure that's correct. It really doesn't make any sense since packet loss is more of an issue when streaming programming from the Internet, yet h.264 works very well there and the file sizes are significantly smaller than MPEG-2.

Also some channels are being distributed natively in h.264, HBO for example. HBO currently broadcasts using h.264 encoding at 8 mbps. Satellite providers just pass this on as is to their customers. Cable companies must convert the HBO channels to MPEG-2 which results in a loss of quality.


----------



## pmiranda

morac said:


> I'm not sure that's correct. It really doesn't make any sense since packet loss is more of an issue when streaming programming from the Internet, yet h.264 works very well there and the file sizes are significantly smaller than MPEG-2.


All internet streaming I've seen buffers a significant portion of the video before starting playback, so that there's time to re-fetch any dropped packets. With cable and even more for satellite, there's not enough time (and the infrastructure isn't there) to replace any dropped packets.


----------



## fyodor

morac said:


> I'm not sure that's correct. It really doesn't make any sense since packet loss is more of an issue when streaming programming from the Internet, yet h.264 works very well there and the file sizes are significantly smaller than MPEG-2.


My understanding of this is imperfect, but I believe that Internet streams can be buffered to re-request missing packets, etc, whereas for cable/satellite transmissions, it's basically a single broadcasted stream that goes to everyone. The cable company is basically sending the same flow of packets to everyone simultaneously.

Re HBO broadcasting in H.264, it doesn't really matter whether HBO is doing it or the satellite provider-they know that it's being provided for broadcast, so they encode the stream with more keyframes. It's not that there's anything structurally different about the encoding, but at the encoding stage they know that it will be used for transmission, so they build it with more redundancy/less efficiency.

All that being said, this is far from my area of expertise, so if someone has a more informed understanding, I'll defer to them.


----------



## morac

pmiranda said:


> All internet streaming I've seen buffers a significant portion of the video before starting playback, so that there's time to re-fetch any dropped packets. With cable and even more for satellite, there's not enough time (and the infrastructure isn't there) to replace any dropped packets.


Well that's where error correction kicks in.

Realistically though, there shouldn't be any packet loss on a cable system. Dropped packets would show up as macro-blocking on the TV since I'm pretty sure digital cable does not use TCP/IP, so there wouldn't be any way to re-fetch dropped packets.

Even if more key frames were needed, h.264 has some significant advantages over MPEG-2 which would still allow smaller bitrates without loss in quality.


----------



## Dan203

Most consumer H.264 encoders cap the GOP length at 50 frames. If cable companies used the same specs it would require up to 1.66 seconds to sync the stream and start decoding a channel change. That is a bit long, so I could see cable companies capping the GOP length at 25-30 frames to get it down to about 1 second.

However long GOPs are only part of the reason MPEG-4/H.264 is so efficient. Other things, such as the ability of B frames to reference multiple frames, variable macroblock sizes, multiple motion vectors and quarter pixel motion compensation can all still be used to lower the bitrate while retaining equivalent quality.

Like someone else pointed out above HBO uses an 8Mbps H.264 stream for transmission to providers. By contrast the major networks all use 15-18Mbps for the MPEG-2 HD streams. So by switching to H.264 cable companies could, at the very least, cram twice as many HD channels into their lineup.

I think the biggest problem is hardware. Most cable providers would have to upgrade a whole sloth of STBs and DVRs to support H.264, which would cost them some serious cash. There might also be some regulations due to the whole CableCARD thing. I don't H.264 decoding was ever part of CableCARD certification, so if the cable companies switched they'd limiting, or disabling, a lot of CableCARD enabled devices. Although they're still deploying SDV even though it breaks CableCARD, so maybe not.

Dan


----------



## TheAmigo

fyodor said:


> Is it possible to configure Pytivo to check for MP4 files and push in response to a pull request for one of those files?


I think this would be a very slick feature. Ideally, I'd like to see it as a special item when browsing a pyTivo share. For any folder that you browse into, pyTivo adds a special item as the first "file" called "[download this entire folder]". Selecting to transfer that item to the TiVo would trigger pyTivo to push all the individual files in there (while returning some dummy file to fulfill the pull request).

I'd love this for e.g. when returning from vacation and I put 10 short video clips from my digicam into a folder that pyTivo is sharing... then I wouldn't have to select to transfer each one, I could just click the [push all] option and wait a bit... then I'd get a folder called "My Vacation" (or whatever) on the TiVo that I could click Play on to see all and only those videos.


----------

