# Streaming TiVo Videos



## davidblackledge

I've got a surprise for you guys... well, the surprise is you guys give up too easily.

I coded this last weekend, but wanted to package it up and do some more testing...here it is...I even provided a .bat file to run it...woohoo!

Yes, Virginia, you CAN stream .tivo files!
Rather simple, actually. Just claim they're a supported video format. video/mpeg, and I even tried video/mp4 at one time and that seems to work as well. It works for files fed directly from a tivo, or files I downloaded to my desktop.

Attached is a little fun I put together with this feature, my VideoView (still needs work) and my hmo client library. It's still a little rough, but it works. (oh...not attached... too big for the forum... It's on my website at: http://David.Blackledge.com/tivo/NowPlaying.zip ) Unzip it, change the launch.txt to have your MAK, and run it. (it also contains the RSS-feed-based video listings, half of which don't work, but some do)

In and of itself, it's just a demo and/or silly, but it leaves the door open for creating one's own UI for now playing stuff, to a degree, and merging interfaces however you like.

It also notes when you've got a "resume" video (paused somewhere in the middle) but I've tried about 6 or 7 ways of asking the TiVo to send the video starting at that point and it hasn't cooperated... has anybody sniffed the network to see what one Tivo says to the other when he just wants to "transfer from paused point" video?

I also have one more trick up my sleeve that I want to add, but need some help. I really want somebody to just to point me to a jar file that will do the discovery stuff for me so I can see what is tivo-related on the network. Anybody got one of those? I started reading up on zeroconf and that's a bit confusing to me, plus it doesn't appear to address the rest of the service discovery (apps) that a TiVo does.

Enjoy, doubters ;]


----------



## Allanon

Good work figuring out how to stream .tivo files, I'm sure this will excite people that store their Tivo videos on their computer in order to free space on the Tivo. But I was a little disappointed that it didn't stream HD videos in real time without pauses, SD videos streamed in real time without pauses. It seems to stream at the same speed as it takes to transfer the video back to the Tivo using Tivo Desktop.


----------



## wmcbrine

Nope, I still can't stream .tivo files with HME/VLC, even marking them as video/mpeg. Perhaps it's some difference on the web server side, like the TiVo wants a Content-length header or something.


----------



## wmcbrine

OK, here's the winning combo: in the HME call to create the stream, use video/mpeg; _but_ in the Content-type header, use video/x-tivo-mpeg. Both must be present. Pretty messed up.

Anyway, thanks for finding this.


----------



## davidblackledge

wmcbrine said:


> OK, here's the winning combo: in the HME call to create the stream, use video/mpeg; _but_ in the Content-type header, use video/x-tivo-mpeg. Both must be present. Pretty messed up.
> 
> Anyway, thanks for finding this.


Wow... that's pretty weird... I didn't do anything special to stream the file from my computer, I just had my Factory find the file and send it along. Java must have taken care of the Content-type for me automatically in combination with whatever the TivoDesktop installation did to associate the .tivo files with that mime type.

Allanon: get me an HD TV and I'll test that out for you  - Not suprising that there was no speed difference, actually. Try getting your TiVo and PC hardwired into your network to boost your transfer speed.

----

Let me know if anybody finds out the key that TivoConnect is looking for to send a mid-stream file. That will make things much more interesting ;]


----------



## moyekj

What would be REALLY nice is if we can find a way to stream RAW transport stream variety of .TiVo files, (x-tivo-raw-tts) instead of the program stream variety.

As you guys probably know one can add *&Format=video/x-tivo-raw-tts* to TivoToGo transfer URLs to download the raw .TiVo file (which is what MRV transfers between S3 Tivos does). The advantage is that you get MRV type transfer speeds which are typically double TTG speeds since there is no conversion taking place. This would be especially useful to stream back HD mpeg2 to Tivos that would buffer fast enough for uninterrupted playback.

I tried various combinations of mime type during creation and Content-Type headers but couldn't get any of them to work. Some of the combos I tried that didn't work:
video/mpeg + video/x-tivo-raw-tts
video/m2ts + video/x-tivo-raw-tts
video/mpeg + video/x-tivo-raw
video/mpeg + video/x-tivo-mpeg
video/m2ts + video/x-tivo-tts
(Just in case I also added tsn as a header but it didn't make any difference)


----------



## davidblackledge

Wow, you totally lost me, moyekj.
Ok, I read it again, and I think I follow... let's see if I got this right:

So what I'm streaming is the link provided in the XML file... which I guess is the kind of link that lets you download a .tivo file that would play on your computer with the right codec installed. I assume that's also the same format as what TivoToGo would give you.

You're saying that if one supplies the special Format argument to the URL (no, I didn't know about that one for video... grr.. when are they gonna open that wiki up?) then you get another, faster format.
That faster format is what TiVos use when sharing with each other for Multi-room Viewing (MRV).



I see that the format argument is the same as the content-type that the XML claims the video is... which is interesting.

Your experiments have been unable to stream the faster format.

I'd like to recommend experimenting with using "video/mp4" for the protocol contentType argument, and probably video/x-tivo-raw-tts as the Content-Type header.
Oh... wait a minute... are you trying to stream raw files from your computer?? That confuses me (you downloaded the raw file to your computer then turned around and tried to feed it back to TiVo?)... But you must going from your computer since going from the TiVo directly doesn't give you the opportunity to control its headers which you say you're doing... unless you're proxying the connection for some reason. Surely the TiVo sends an appropriate Content-Type header when you download with that format. Whatever it sends is the right one... at least it better be, or we're up a creek unless we do the proxying, I guess.

If we can get that working, I strongly suspect that's a prerequisite for getting the start-in-the-middle transfer to work since that's the format the MRV is using in that circumstance.


By the way...hopefully keary is reading this:
I noticed when playing around that if I have a video running and try to start another video, The second one doesn't fail, but instead I think it stops at the "READY" state. What I think is important about this is: With that feature, surely one could pre-fetch the next 1GB section of the movie before the first section quits, then simply destroy the first section and play the second, already-queued section... I assume currently keary is just killing the first section and at that time starting to queue the second section. This other technique should reduce the pause between sections. (no, I haven't downloaded Keary's work yet... I think it's been less than 24 hours for goodness sakes)


Also, can anybody recommend a good, clean sequence for ending a streamed video so the next one won't break? I eventually figured out that sometimes when I can't stream it's because something broke the feature and all further videos that would have worked just get a "Connection Failed" error. I have occasionally seen the TiVo apparently continuing to show the video after I killed it a long time ago (e.g. when I go back to TiVo central the background is temporarily my video playing for a moment).
I was doing resource.remove() but I guess that's not enough to definitively clean up the streaming player inside the TiVo.
Perhaps it's just a fragile underlying system like the mp3 player.


----------



## moyekj

David, OK I'll break it down just to clarify:

For normal Tivo->PC TTG transfers you can get simple listing of programs available to TTG as follows:


Code:


https://<ip>/nowplaying/index.html

Then you have links to download that look like for example:


Code:


http://<ip>/download/Two&#37;20and%20a%20Half%20Men.TiVo?Container=%2FNowPlaying&id=1683865&Format=video/x-tivo-mpeg

Using that link you get a normal TTG transfer and an mpeg2 program stream .TiVo file. But if you substitute the Format option above as below then you get the raw Tivo mpeg2 transport stream file (which is what MRV transfers are doing which I discovered a long while ago doing network sniffing) and hence the download is quite a bit faster:


Code:


http://<ip>/download/Two%20and%20a%20Half%20Men.TiVo?Container=%2FNowPlaying&id=1683865&Format=video/x-tivo-raw-tts

Currently this raw .Tivo file is pretty useless on the PC since it's encrypted and and cannot be decoded like the normal program stream variety .TiVo.
It's this raw .TiVo file that I would like to be able to stream back to Tivo which supposedly should be a faster transfer back just as it is a faster transfer fom Tivo->PC.
i.e. The only potential value of getting this to work is to take advantage of faster transfers between PC & Tivo. TTG transfers can vary a lot but seem to cap out around 13 Mbps or so, while MRV transfer speeds can quite easily be 30 Mbps or better. 30 Mbps is plenty to stream HD better than real time.


----------



## kearygriffin

davidblackledge said:


> By the way...hopefully keary is reading this:
> I noticed when playing around that if I have a video running and try to start another video, The second one doesn't fail, but instead I think it stops at the "READY" state. What I think is important about this is: With that feature, surely one could pre-fetch the next 1GB section of the movie before the first section quits, then simply destroy the first section and play the second, already-queued section... I assume currently keary is just killing the first section and at that time starting to queue the second section. This other technique should reduce the pause between sections. (no, I haven't downloaded Keary's work yet... I think it's been less than 24 hours for goodness sakes)


I did play around with this for a while, and never could get it working quite right, so gave up. It was a few weeks ago, and streambaby itself wasn't really working very well yet, so its quite possible that I need to have another go at it. And like I think you said at the beginning of this thread, maybe I gave up too easily ;-)

Also about closing down the stream properly, I'm not really sure. It hasn't happened to me in a while, but I was definitely seeing Tivo refuse to stream anything till it had a chance to "cool down". I haven''t been banging my tivo as much as I was in earlier testing of streambaby, so I'm not sure if I close down the streams in a way that makes the problem not appear, or if I just need to try harder to make it happen.


----------



## moyekj

In the hope this may be of help to someone looking into raw tivo file transfers (via HMO or HME) here's a summary of packet sniffing of an MRV transfer. (Note that I intentionally replaced last 8 digits with xxxxxxxx at end of tsn numbers).

The trickiest part seems to be the authentication piece, namely how to come up with:
*response="4bd238b4ed6d8b3194fb224defc27f51"*

NOTE: In case it's not obvious, 192.168.1.107 is requesting a video from 192.168.1.101 in this example.

311	43.523098	192.168.1.107	192.168.1.101	HTTP
----------------------------------------------------
GET /download/Grey's%20Anatomy.TiVo?Container=%2FNowPlaying&id=1169277&Format=video%2Fx-tivo-raw-tts HTTP/1.1
Cookie: sid=45F68EE391E5B4D7
host: 192.168.1.101:80
Range: bytes=0-
User-Agent: TvHttpClient
tsn: 6480001xxxxxxxx
Connection: close

313	43.974932	192.168.1.101	192.168.1.107	HTTP
----------------------------------------------------
"HTTP/1.1 401 Authorization Required
Server: tivo-httpd-1:9.2a.B1-01-2:648
WWW-Authenticate: Digest realm="TiVo DVR", nonce="39EB249B42E7E598", qop="auth"
Content-Length: 31
Content-Type: text/html
Connection: close

*Authorization Required*

320	43.988079	192.168.1.107	192.168.1.101	HTTP
----------------------------------------------------
">GET /download/Grey's%20Anatomy.TiVo?Container=%2FNowPlaying&id=1169277&Format=video%2Fx-tivo-raw-tts HTTP/1.1
Authorization: Digest username="tivo", realm="TiVo DVR", nonce="39EB249B42E7E598", uri="/download/Grey's%20Anatomy.TiVo?Container=%2FNowPlaying&id=1169277&Format=video%2Fx-tivo-raw-tts", qop=auth, nc=00000001, cnonce="tivo tcd", response="4bd238b4ed6d8b3194fb224defc27f51"
Cookie: sid=45F68EE391E5B4D7
host: 192.168.1.101:80
Range: bytes=0-
User-Agent: TvHttpClient
tsn: 6480001xxxxxxxx
Connection: close

323	44.947812	192.168.1.101	192.168.1.107	TCP
---------------------------------------------------
">HTTP/1.1 200 File Follows
Server: tivo-httpd-1:9.2a.B1-01-2:648
Content-Type: video/x-tivo-raw-tts
TiVo-Estimated-Length: 4869586944
Connection: close


----------



## kearygriffin

moyekj said:


> In the hope this may be of help to someone looking into raw tivo file transfers (via HMO or HME) here's a summary of packet sniffing of an MRV transfer. (Note that I intentionally replaced last 8 digits with xxxxxxxx at end of tsn numbers).
> 
> The trickiest part seems to be the authentication piece, namely how to come up with:
> *response="4bd238b4ed6d8b3194fb224defc27f51"*
> 
> NOTE: In case it's not obvious, 192.168.1.107 is requesting a video from 192.168.1.101 in this example.


I haven't done a lot of playing with HMO, but this seems to be the wrong question. The question should be how to make the tivo request a raw video from a source other than another tivo.

The request= above is the requesting tivo "logging on" to the source TiVo using digest authentication (If I am following correctly). It is up to the source TiVo to decide if the response= key is correct. Since in our case "we" would be the source TiVo, so we could just respond in the affirmative (i.e. accept the digest authentication regardless of what it sends).
Edit: (Actually since we would be the source, we probably wouldn't even have to request authorization in the first place, unless for some reason the TiVo was unhappy without it)

Full Disclosure: I only have the single TiVoHD plus some disconnected Series 1s, so I've never actual seen how room-to-room transfers function.


----------



## davidblackledge

Interesting...thank you very much. I'm trying to play with a little bit of that right now. Would you mind doing the same thing with a transfer of a something from the "pause point" somewhere other than the beginning? I assume the "Range" header will change, but I want to know if the URL changes as well.
Thanks in advance.
(I'm getting slowed down by the fact that my Series 3 is refusing to respond to TivoConnect queries??? luckily I've got a HD that's not giving me any trouble)


----------



## moyekj

kearygriffin said:


> The question should be how to make the tivo request a raw video from a source other than another tivo.


 I know very little about HMO as well, but a server running Tivo Desktop or pyTivo does just that currently, at least for regular mpeg2 files.

David, I'm pretty certain only the Range field changes when starting from a pause point. This MRV capture I posted was actually from several months (maybe longer) ago and I'm not currently setup to capture more very easily but can do so if really necessary.


----------



## ggieseke

I wanted to look at the raw format, but changing the last part of the URL to x-tivo-raw-tts just gets me a standard .TiVo transfer. I have tried IE6 & IE7 to my S2DTs.

Am I missing something or have they disabled that capability recently?


----------



## moyekj

ggieseke said:


> I wanted to look at the raw format, but changing the last part of the URL to x-tivo-raw-tts just gets me a standard .TiVo transfer. I have tried IE6 & IE7 to my S2DTs.
> 
> Am I missing something or have they disabled that capability recently?


 I think this only applies to series 3 Tivos.


----------



## moyekj

A pretty interesting somewhat related thread in case you missed it on getting mp4 natively stored on a Tivo:
http://www.tivocommunity.com/tivo-vb/showthread.php?t=419427


----------



## ggieseke

Oh, well. From a few other posts I've seen I was thinking that 2.7 might be skipping the transcode for MP4 transfers to an S3 TiVo, but I don't have one to play with since I use DirecTV.

Thanks for the info.


----------



## davidblackledge

davidblackledge said:


> In and of itself, it's just a demo and/or silly, but it leaves the door open for creating one's own UI for now playing stuff, to a degree, and merging interfaces however you like.
> ...
> I also have one more trick up my sleeve that I want to add, but need some help. I really want somebody to just to point me to a jar file that will do the discovery stuff for me so I can see what is tivo-related on the network. Anybody got one of those? I started reading up on zeroconf and that's a bit confusing to me, plus it doesn't appear to address the rest of the service discovery (apps) that a TiVo does.


Ok, since I finally learned enough (on my own :/ ) about zeroconf/bonjour/jmdns, I've gotten some work done on that "one more trick" which I'm adding to the NowPlaying interface. It's really frickin cool ;] (but I'm easily amused). I need to make it prettier/friendlier, but it works without a major hitch which was a surprise.
Why am I not telling you what it is yet? Because as soon as I do somebody will go out and do a better one, and I can't compete with just a few hours a week. I'd at least like to get the first version. (and if I'm just too paranoid and nobody will write one, then it won't make a difference because you have to wait for me to finish it ;] )

Anyhow, a major stumbling block or two with the zeroconf stuff...I'm hoping somebody (preferably somebody from Galleon) can help me out on this.

1) Running NowPlaying+zeroconf stand-alone using the sample server it works perfectly except that the Galleon Video service presents itself with <full.numeric.ip>.local. as its zeroconf host address while other stuff either has a <name>.local. or <number>.local. (one number from ip address with no additional dots)... my stand-alone can't look up those full ip number versions correctly (for now I'm working around it by just removing the ".local." at the end if there is an UnknownHostException, then hoping for the best)... that's making it awkward to serve Galleon's videos as "another DVR" in the NowPlaying app.
That's the least of my problems, and I expect it to go away if I can run within Galleon and use whatever version of zeroconf it's using.

2) Attempting to run NowPlaying from within Galleon is just wacko. I was going to try it with an older version of Galleon and including the Bonjour jmdns jar that I'd downloaded/installed, but I saw in Galleon change log that they had updated its zeroconf stuff, so I thought I'd install the newest version and see if it magically worked. It didn't. In fact, it complained that it couldn't load the native library (dll). 
That makes no sense to me since Galleon itself is presumably SERVING zeroconf data using that same library. 
I found a copy if it as a dll in C:\windows\system32, I think, and copied it into Galleon's lib folder so it would be in the load library path. I honestly don't recall exactly what the result was after that, but it was different... it just didn't work still. I believe it broke and didn't load any of the hme apps for serving at all, at least once it hit the app that made use of it. In reality going this direction was silly anyhow... *I must be missing some point with Galleon itself*... perhaps I have an old Jar somewhere that includes jmdns which is somehow in my classpath but isn't in the main Galleon server's classpath.
I tested some stuff last night and found that Galleon was serving apps fine (with NowPlaying removed from its list) after a restart of the service, but in the middle of the night, it just stopped serving them...nothing in any logs. I had that dll file still in the lib folder. So I removed it and restarted the service this morning... haven't had a chance to see what the result was.

Wild guess is the .dll I had is different from what Galleon uses, but is overriding it... it serves fine, but doesn't send out reminders or doesn't respond to tivo "who's there?" requests after a while...or whatever it is that zeroconf does.

So does any of this mean anything to anybody? or am I still on my own?
I guess the short question is:
*How do I correctly use Zeroconf (DNSSD.browse, etc) from within Galleon's environment?*


----------



## Allanon

Ever since I saw your NowPlaying demo I thought someone would write an application that streams video from one Tivo on the home network to another. This would take the place of MRV allowing you to instantly stream a video instead of transferring it. Hopeful this is what you are working on.


----------



## moyekj

Allanon said:


> Ever since I saw your NowPlaying demo I thought someone would write an application that streams video from one Tivo on the home network to another. This would take the place of MRV allowing you to instantly stream a video instead of transferring it. Hopeful this is what you are working on.


 I thought about that quite a lot a while back, but one issue is that TTG protocol is being used instead of straight MRV which means maximum sustained transfer rates are approximately cut in half compared to MRV which makes it less than ideal and virtually impossible for real time HD mpeg2 streaming. Then of course is the nagging issue of the ~1.1GB streaming buffer limit... Both of those conspire to make MRV a much better choice, especially for HD transfers.


----------



## davidblackledge

Yes, because of moyekj's issues, this is largely a toy, but it does do exactly what you're saying Allanon (now that it has zeroconf to find other TiVo DVRs)... among other cute things ;]
It could be set up easily enough to merge the lists of all available video providers so you don't have to care what machine your show is stored on, just find it in the larger list and start streaming, although I'm not working on that feature right now.

Actually for *me* personally it's working out great since I only have SD and I record everything at basic quality, so the 1.1GB limit is only an issue for movies (if even then...I haven't really looked at the numbers)

Hmm... maybe I should abandon my zeroconf-in-Galleon problem for now given that I released it as a standalone before and probably will again... 
But I still want to know the answer if anybody has it!


----------



## davidblackledge

Ok, partial answer to my problem... I finally realized that Galleon was serving my apps fine (with our without the dll), then consistently just over an hour later it would log several "connection timed out" errors in a row over several minutes, and the apps are no longer served. Haven't had a chance to watch it happen live yet, but I'm hoping if I just re-install 2.5.1 (vs. the 2.5.5 upgrade I had installed) maybe the problem will go away.
Oddly enough the number of timeouts don't match the number of apps, so it's not a per-app thing. It also has a couple of the timeouts when it first starts up and one "connection refused" happens, too. It probably is related to my install of Bonjour, but I'm going to try downgrading Galleon first sometime this weekend.

If that fixes that problem, then I'll just concentrate on getting a standalone NowPlaying update working... hopefully that will work on other machines using only the sampleserver jar. Someday I might learn how to get NowPlaying with Bonjour served by Galleon *sigh*.


----------



## davidblackledge

In case anybody cares... I found the true cause of my Galleon problem.

Galleon is using JmDNS... JmDNS has a shortsighted section of code. When it tries to Renew the published information about services, it attempts to shove every service notification into a single data buffer which has a fixed maximum size.

Since Galleon is serving all of its stuff and all of my stuff (which is a lot... including the fact that I deploy my generic "Multitasker" app multiple times with different settings for e.g. icanhascheesburger.com and local news/weather as separate apps) that makes for a long list of services to renew.

I simply had hit the limit when I was trying to deploy my new NowPlaying version. 
JmDNS attempts to renew 30 minutes after starting - an error appears in wrapper.log at that time. I believe that also breaks the thread and no more attempts are made to renew the service listings.
Eventually the information, unrenewed, expires from the Tivo's and Simulator's listings and all apps seem to disappear.

I removed a couple of services from my list, and it works fine, now.

If Galleon were to use multiple instances of JmDNS that would work around this problem. But better, as I see has been mentioned on some JmDNS forums in the past, JmDNS should handle that buffer dynamically or at least allow the maximum message length, not just "typical" if the maximum is defined in the standard.

At an absolute minimum, the JmDNS renewer needs to catch the exception while adding service entries. Then it could back off the last attempt, and renew what it was able to add to the buffer... then you only lose the last few services, not ALL of them.

Hmm... Now how do I go about getting Galleon to use a custom version of jmdns.jar with one of these fixes? Obviously I could do it myself and replace the jar, but it'd be nicer to have everybody get the fix.

Anyhow... I just need to do some small enhancements then I'll be releasing a new NowPlaying zip with cool/fun enhancements... if I can ever get any time to do so.


----------

