# Java port of TivoDecode



## fflewddur

In case this is of any interest: I recently ported tivodecode 0.4.4 to Java and implemented support for Transport Stream files. Both the source code and precompiled JARs are available on GitHub. Any feedback/bug reports are welcome!

Update: You can download the latest stable version at https://github.com/fflewddur/tivolibre/releases.


----------



## telemark

Link:
https://github.com/fflewddur/tivolibre


----------



## fflewddur

Thanks for posting the link. I just updated the code with some fixes and a much-improved command-line application; usage instructions are included in the README file.


----------



## fflewddur

And as of today, TivoLibre also supports Program Stream files, bringing it up to feature parity with tivodecode. 

I also added support for PipedInputStream sources via an internal buffer, which allows TivoLibre to act as a pass-through while downloading. For example, you can setup a PipedInputStream and PipedOutputStream pair before downloading and create an instance of TivoDecoder with the PipedInputStream as its input and the destination file as its output. Each time you read a packet from the network, you write it to the PipedOutputStream. TivoLibre will decode it in real-time and write it to the output file, saving you the need to save the entire file to disk before beginning the decoding process.

As always, bug reports are welcome!


----------



## moyekj

Wow, thanks for sharing, this could be very useful.

I have a TS .TiVo file that doesn't decrypt either with tivodecode-ng or tivolibre. The tivolibre error is:
SEVERE: Error: No TransportStream exists with PID 0x0633

tivodecode-ng just gives a seg fault with no useful information.
NOTE: There are several other clips that tivodecode-0.44 and tivodecode-ng have trouble with as well. Haven't tried them all with tivolibre yet.
I did get tivolibre to work with at least 1 short H.264 TS .TiVo file.

DirectShowDump works without any issues at all.

I was able to make a very small 20 sec clip using VideoRedo that still has the problem and that you can download from here:
https://drive.google.com/file/d/0B0SMFC97ymdEdXdUaWw2eVRqdUE/view?usp=sharing

If you need my MAK for further debugging please send a PM, though you will probably need to get your post count up before you can do that.


----------



## fflewddur

Awesome, thanks for including a file so I can investigate. I'll shoot you a PM for the MAK.

EDIT: Or not; the forum won't let me send PMs without 10 posts. Could you send it to me instead?


----------



## moyekj

fflewddur said:


> Awesome, thanks for including a file so I can investigate. I'll shoot you a PM for the MAK.
> 
> EDIT: Or not; the forum won't let me send PMs without 10 posts. Could you send it to me instead?


 OK, PM sent. I assume you can at least read PMs sent to you.


----------



## fflewddur

Yep, got it. Thank you!

It looks like TivoLibre is also choking on a recording of Ken Burn's Civil War documentary from last night, so I've got a couple of test cases to fix now


----------



## wmcbrine

I get the following message when I try to clone the repo from the command line:



Code:


$ git clone https://github.com/fflewddur/tivolibre.git
Initialized empty Git repository in /Users/wmcbrine/tivolibre/.git/
Cannot get remote repository information.
Perhaps git-update-server-info needs to be run there?

I got the same thing on another GitHub repo a few days ago, so it may be something GitHub is doing wrong. "Download as .Zip" works.


----------



## fflewddur

Odd, it's working fine for me. Just ran this a second ago:



Code:


[email protected]:tmp$ git clone https://github.com/fflewddur/tivolibre.git
Cloning into 'tivolibre'...
remote: Counting objects: 121, done.
remote: Compressing objects: 100% (84/84), done.
remote: Total 121 (delta 44), reused 92 (delta 19), pack-reused 0
Receiving objects: 100% (121/121), 523.19 KiB | 0 bytes/s, done.
Resolving deltas: 100% (44/44), done.
Checking connectivity... done.
[email protected]:tmp$ git --version
git version 2.3.2 (Apple Git-55)
[email protected]:tmp$

Maybe it was a temporary issue with GitHub?


----------



## wmcbrine

Well, this seems to explain my problem:

https://code.djangoproject.com/ticket/17220

Sure enough, I tried it with "git://", and it worked fine. The odd thing is that I'm sure I cloned repos much more recently than the cut-off date they talked about there. Hmpf.

Sorry.


----------



## fflewddur

moyekj said:


> Wow, thanks for sharing, this could be very useful.
> 
> I have a TS .TiVo file that doesn't decrypt either with tivodecode-ng or tivolibre. The tivolibre error is:
> SEVERE: Error: No TransportStream exists with PID 0x0633
> 
> tivodecode-ng just gives a seg fault with no useful information.
> NOTE: There are several other clips that tivodecode-0.44 and tivodecode-ng have trouble with as well. Haven't tried them all with tivolibre yet.
> I did get tivolibre to work with at least 1 short H.264 TS .TiVo file.
> 
> DirectShowDump works without any issues at all.
> 
> I was able to make a very small 20 sec clip using VideoRedo that still has the problem and that you can download from here:
> https://drive.google.com/file/d/0B0SMFC97ymdEdXdUaWw2eVRqdUE/view?usp=sharing
> 
> If you need my MAK for further debugging please send a PM, though you will probably need to get your post count up before you can do that.


Thanks again for sending that file. I found a couple of obvious problems and fixed them, and decided to let the library discard packets for streams that weren't specified in the file's PMT. I'm not certain it's the right way to handle that situation--there are still a couple of fields in the PMT packet that I don't understand, and I'm guessing they describe the missing stream. Still, discarding the packets doesn't seem to hurt, and the resulting MPEG plays properly.

I'm still getting "Invalid TS header" errors on a separate file, so these fixes didn't take care of all of the issues.


----------



## moyekj

fflewddur said:


> Thanks again for sending that file. I found a couple of obvious problems and fixed them, and decided to let the library discard packets for streams that weren't specified in the file's PMT. I'm not certain it's the right way to handle that situation--there are still a couple of fields in the PMT packet that I don't understand, and I'm guessing they describe the missing stream. Still, discarding the packets doesn't seem to hurt, and the resulting MPEG plays properly.
> 
> I'm still getting "Invalid TS header" errors on a separate file, so these fixes didn't take care of all of the issues.


 Thanks for that fix and I confirmed my short clip now decrypts OK, though it comes out smaller file size than DSD decrypt and obviously diff shows a difference, but playing it there are no obvious glitches.

For the full clip decrypt I'm getting a different error now:
Sep 09, 2015 7:01:30 PM net.straylightlabs.tivolibre.TransportStreamDecoder process
SEVERE: Unsupported packet type: NONE

Looks like this happens at the very end of the decrypt since time wise the full decrypt does seem to happen. Again the full clip is smaller by several bytes from DSD version (2906039704 bytes vs 2909128544 bytes from DSD).

I'll try and see if I can duplicate the problem with a short clip of the end of the large clip so that I can upload it for you to take a look at.


----------



## moyekj

OK, I did manage to create short clip to reproduce the new problem:
https://drive.google.com/file/d/0B0SMFC97ymdESGtsNkprNHNhaUE/view?usp=sharing


----------



## fflewddur

moyekj said:


> For the full clip decrypt I'm getting a different error now:
> Sep 09, 2015 7:01:30 PM net.straylightlabs.tivolibre.TransportStreamDecoder process
> SEVERE: Unsupported packet type: NONE
> 
> Looks like this happens at the very end of the decrypt since time wise the full decrypt does seem to happen. Again the full clip is smaller by several bytes from DSD version (2906039704 bytes vs 2909128544 bytes from DSD).


This should be fixed now, too. That's from the same recording as the first clip you sent, right? I ask because it also has lots of packets associated with stream 0x0633, just like the first clip, but the header for the file doesn't setup a stream with that ID. I haven't seen that behavior on any files off of my TiVo, which is the only device I have to test with. If you have any thoughts on what the extra data might be (different language audio? Subtitles?), I'm all ears. But since they don't seem related to the main audio and video streams, there doesn't seem to be much harm in discarding those packets, hence the smaller file size.


----------



## moyekj

fflewddur said:


> This should be fixed now, too. That's from the same recording as the first clip you sent, right? I ask because it also has lots of packets associated with stream 0x0633, just like the first clip, but the header for the file doesn't setup a stream with that ID. I haven't seen that behavior on any files off of my TiVo, which is the only device I have to test with. If you have any thoughts on what the extra data might be (different language audio? Subtitles?), I'm all ears. But since they don't seem related to the main audio and video streams, there doesn't seem to be much harm in discarding those packets, hence the smaller file size.


 Yes, it's from same recording and confirm it's now fixed. I tried a couple more clips that I never got to work properly with tivodecode/tivodecode-ng and they are working now with tivolibre, so it looks like a big improvement already to me over tivodecode at least for TS TiVo files, so great job!


----------



## moyekj

This clip decrypts to exact same file size as DSD with tivolibre but playback is completely garbled while DSD generated file playback is fine:
https://drive.google.com/file/d/0B0SMFC97ymdEa3ZYQ0tlUUF2cVk/view?usp=sharing

Note that for this one, tivodecode-ng works fine (and gives same file size as DSD).


----------



## wmcbrine

moyekj said:


> This clip decrypts to exact same file size as DSD with tivolibre but playback is completely garbled while DSD generated file playback is fine:
> https://drive.google.com/file/d/0B0SMFC97ymdEa3ZYQ0tlUUF2cVk/view?usp=sharing
> 
> Note that for this one, tivodecode-ng works fine (and gives same file size as DSD).


Can't win, can we?

Meanwhile, I'm trying to figure out why your first two samples work better with tivolibre... The other day, I felt I was tantalizingly close to solving the whole thing -- my model "problem child" clip has large blocks that decode properly if I just shift the Turing data by one byte; the only trick is figuring out _when_ to do that -- but these clips are a completely different set of problems. Bleah.


----------



## fflewddur

wmcbrine said:


> Meanwhile, I'm trying to figure out why your first two samples work better with tivolibre... The other day, I felt I was tantalizingly close to solving the whole thing -- my model "problem child" clip has large blocks that decode properly if I just shift the Turing data by one byte; the only trick is figuring out _when_ to do that -- but these clips are a completely different set of problems. Bleah.


I think the first samples are working because 1) TivoLibre discards packets with an unknown PID (as suggested here) and 2) it fixes a problem handling what should be the 12 ES Info Length bits in the PMT packet, but the original tivodecode code treated them as 13 bits. That extra bit is almost always 0, so usually no harm done, but every now and then it's 1, and that screws up handling the rest of the PMT packet. I think you can fix that in tivo_decoder_ts.cxx by changing the line


Code:


es_info_length = pmt_field & 0x1fff;

to


Code:


es_info_length = pmt_field & 0x0fff;

Hope that helps!


----------



## wmcbrine

fflewddur said:


> Hope that helps!


It did indeed! Thanks. :up:


----------



## fflewddur

moyekj said:


> This clip decrypts to exact same file size as DSD with tivolibre but playback is completely garbled while DSD generated file playback is fine:
> https://drive.google.com/file/d/0B0SMFC97ymdEa3ZYQ0tlUUF2cVk/view?usp=sharing
> 
> Note that for this one, tivodecode-ng works fine (and gives same file size as DSD).


I think I fixed this inadvertently while getting another file to decode properly; at any rate, it decodes and plays fine on my computer (Mac OS X 10.10, VLC 2.21, Java 1.8.0_60) with today's 0.5.4 release of TivoLibre. Diff also doesn't show any differences between the file generated by tivodecode-ng and TivoLibre. Not sure what the problem was, though, and that's going to irk me. 

I really appreciate you testing this out on a wider selection of files than I have available. If you find any more that don't decode properly, please let me know.


----------



## moyekj

OK, here's an old one that gave the original author of TS tivodecode headaches trying to figure out (and I don't think ever did):
https://drive.google.com/file/d/0B0SMFC97ymdEanJua1Vad3ZuOHM/view?usp=sharing

tivolibre, tivodecode-ng and DSD all produce same file size.
tivolibre and tivodecode-ng files are identical (via diff) but there are several very obvious segments where it's not getting decrypted correctly - you will see square blocks appear all over the video.
The DSD produced file plays back perfectly.

From what I recall there were a few other clips such as this one that I had, but I can't seem to find them. Solving this one I think may solve the last type of problem that I recall seeing with tivodecode and TS files.


----------



## telemark

So glad to see revived activity.

VLC is known as a liberal decoder. If you want to track down unseen bugs, you might see if there's a mpeg validater that could be used.


----------



## PaulS

fflewddur said:


> Thanks again for sending that file. I found a couple of obvious problems and fixed them, and decided to let the library discard packets for streams that weren't specified in the file's PMT. I'm not certain it's the right way to handle that situation--there are still a couple of fields in the PMT packet that I don't understand, and I'm guessing they describe the missing stream. Still, discarding the packets doesn't seem to hurt, and the resulting MPEG plays properly.


In my experience previously noodling with tivodecode, I never saw streams that weren't part of the PMT. I can't see how that would impact the playout of the decoded video.


----------



## PaulS

wmcbrine said:


> Can't win, can we?
> 
> Meanwhile, I'm trying to figure out why your first two samples work better with tivolibre... The other day, I felt I was tantalizingly close to solving the whole thing -- my model "problem child" clip has large blocks that decode properly if I just shift the Turing data by one byte; the only trick is figuring out _when_ to do that -- but these clips are a completely different set of problems. Bleah.


Haha... Join the club! I saw the same thing when I was working on tivodecode. I'd have two sample clips that looked exactly the same at the same spot in the encryption process, excepting one had a packet with encrypted contents and the other did not. Never could figure that one out...


----------



## PaulS

As an aside : how performant do we think this port will be, as compared to the C/C++ implementations ? Quick enough for at least real-time decode ? Do we think that any/all of the fixes here can be backported to the C++ version ?

I'm thinking about the application of this new version, given the PipedInputStream/PipedOutputStream support. One could probably implement a streaming playback application using this.


----------



## fflewddur

PaulS said:


> As an aside : how performant do we think this port will be, as compared to the C/C++ implementations ? Quick enough for at least real-time decode ? Do we think that any/all of the fixes here can be backported to the C++ version ?
> 
> I'm thinking about the application of this new version, given the PipedInputStream/PipedOutputStream support. One could probably implement a streaming playback application using this.


It's noticeably faster than the C++ version. I just timed it on my machine (Core i7 2.6 Ghz), and a 2GB file took 16.8 seconds to decode; tivodecode took 22.8 seconds on the same file. Definitely fast enough for real-time playback.

As for back-porting fixes, I'm sure it's possible, but I'll leave that to someone else. I tried to clean up the code during the initial port, and I keep refactoring it into more module chunks as I gain a better understanding of the decoding processes, so there's not a one-to-one correspondence with tivodecode anymore.


----------



## wmcbrine

Well, so far, tivodecode-ng is 2x-3x faster than tivolibre for me, so that's interesting. Core 2 Duo (old MacBook Air).

I plan to backport anything applicable... of course, I still hope to be first to a complete solution, so the porting will have to go the other way.  Currently, I can cleanly decode every sample moyekj posted to this thread (except the last), with the benefit of the one-bit patch in #19. I still have my problem child, which both tivodecode-ng and tivolibre fail on -- seemingly in completely different ways, strangely enough. (They had produced identical output a few tivolibre versions ago.)


----------



## fflewddur

wmcbrine said:


> Well, so far, tivodecode-ng is 2x-3x faster than tivolibre for me, so that's interesting. Core 2 Duo (old MacBook Air).


I only tested one file, so my numbers may not be typical. I'm trying to get it correctly processing TS files before really looking into performance issues; just wanted to make it clear that it's fast enough for real-time use. 



wmcbrine said:


> I plan to backport anything applicable... of course, I still hope to be first to a complete solution, so the porting will have to go the other way.  Currently, I can cleanly decode every sample moyekj posted to this thread (except the last), with the benefit of the one-bit patch in #19. I still have my problem child, which both tivodecode-ng and tivolibre fail on -- seemingly in completely different ways, strangely enough. (They had produced identical output a few tivolibre versions ago.)


I'll keep you posted on any issues I find that were likely to come from the tivodecode codebase. In addition to the ES bit fix, tivodecode chokes on packets from streams that weren't specified in the PMT (for an example, see the first file moyekj posted), and if the transport stream gets out of sync, it can't recover. I've got a 10 GB file with this problem; if you're interested, I'll try to trim it down to a smaller size that still exhibits the issue and post it online for you. I fixed the later problem by scanning for TS sync bytes (0x47) 188 bytes apart; once TivoLibre finds a few of them, it resyncs at the first and starts reading packets again.


----------



## wmcbrine

fflewddur said:


> In addition to the ES bit fix, tivodecode chokes on packets from streams that weren't specified in the PMT (for an example, see the first file moyekj posted)


I'm not seeing the choking? 



> _and if the transport stream gets out of sync, it can't recover. I've got a 10 GB file with this problem; if you're interested, I'll try to trim it down to a smaller size that still exhibits the issue and post it online for you._


I welcome any test cases, and BTW, I'll take any size of file. As far as resync goes, tivodecode-ng already has a patch for that, but I'm not sure I've ever had a test case for it.


----------



## moyekj

fflewddur said:


> In addition to the ES bit fix, tivodecode chokes on packets from streams that weren't specified in the PMT (for an example, see the first file moyekj posted), and if the transport stream gets out of sync, it can't recover.





wmcbrine said:


> I'm not seeing the choking?


 From my testing last night with latest version of tivodecode-ng as wmcbrine stated all my testscases posted here so far save the last one are now working with tivodecode-ng. For this particular problem you mention above tivodecode-ng will spit out warnings to stderr about it but the decrypt completes anyway.


----------



## fflewddur

Ah, it outputs errors, but the decode works successfully. The decode finished so quickly, I assumed it was stopping after the error.

I'm uploading the sync test case to my web server right now, but it estimates another 6 hours till completion; I'll PM you the link once it's done.


----------



## moyekj

William, any idea how to generate static tivodecode-ng binary for Windows? I used to do it with cygwin back when they supported static libraries, but now on cygwin one can only compile using dynamic libraries, so generating an executable without any cygwin dll dependencies is not possible. I've been using cygwin to compile/test so far. A while back I briefly looked into using mingw but didn't get very far.


----------



## fflewddur

moyekj said:


> OK, here's an old one that gave the original author of TS tivodecode headaches trying to figure out (and I don't think ever did):
> https://drive.google.com/file/d/0B0SMFC97ymdEanJua1Vad3ZuOHM/view?usp=sharing.


Got it. This file has PES headers that extend across multiple TS frames, but the decoding processes wasn't accounting for that. We only want to decrypt stream data, not stream headers. The latest version in GitHub results in a file that plays properly.

I did notice that some TS frames are out of order in the resulting file, although that doesn't seem to impact playback. I'll try to get that fixed up this weekend, regardless. I also want to take another look at the first file you sent to see if we can figure out what those mystery streams are, or at least, how to ensure they end up in the right place in the output file

Any other files that still have playback issues?


----------



## moyekj

fflewddur said:


> Got it. This file has PES headers that extend across multiple TS frames, but the decoding processes wasn't accounting for that. We only want to decrypt stream data, not stream headers. The latest version in GitHub results in a file that plays properly.
> 
> I did notice that some TS frames are out of order in the resulting file, although that doesn't seem to impact playback. I'll try to get that fixed up this weekend, regardless. I also want to take another look at the first file you sent to see if we can figure out what those mystery streams are, or at least, how to ensure they end up in the right place in the output file
> 
> Any other files that still have playback issues?


Wow, that was qiuck! Well done! Binary diff vs DSD does show difference probably because as you said right now some TS frames are out of order.

I've got some longer clips that I'm getting messages like this:
Sep 11, 2015 7:37:38 PM net.straylightlabs.tivolibre.TransportStreamPacket checkHeader
SEVERE: Invalid TS packet header for packet 10896845

tivodecode-ng spits out messages to stderr for the above clip such as:
loss_of_sync
skipped 4 bytes, found a SYNC
found 3 syncs in a row, loss_of_sync = 0
globalBufferLen > TS_FRAME_SIZE, pulling packet from globalBuffer[]
globalBufferLen > TS_FRAME_SIZE, pulling packet from globalBuffer[]

These aren't fatal errors and the decrypt does complete generating a file of different size than DSD (normally larger). Just wondering if the above are expected and can be safely ignored? Since these are long clips it's difficult to watch the whole thing looking for problems.

FYI: tivodecode-ng and tivolibre for the above generate identical files (per diff).


----------



## fflewddur

moyekj said:


> Wow, that was qiuck! Well done! Binary diff vs DSD does show difference probably because as you said right now some TS frames are out of order.
> 
> I've got some longer clips that I'm getting messages like this:
> Sep 11, 2015 7:37:38 PM net.straylightlabs.tivolibre.TransportStreamPacket checkHeader
> SEVERE: Invalid TS packet header for packet 10896845
> 
> tivodecode-ng spits out messages to stderr for the above clip such as:
> loss_of_sync
> skipped 4 bytes, found a SYNC
> found 3 syncs in a row, loss_of_sync = 0
> globalBufferLen > TS_FRAME_SIZE, pulling packet from globalBuffer[]
> globalBufferLen > TS_FRAME_SIZE, pulling packet from globalBuffer[]
> 
> These aren't fatal errors and the decrypt does complete generating a file of different size than DSD (normally larger). Just wondering if the above are expected and can be safely ignored? Since these are long clips it's difficult to watch the whole thing looking for problems.
> 
> FYI: tivodecode-ng and tivolibre for the above generate identical files (per diff).


Cool, thanks for testing it! Yeah, that error probably doesn't need to be reported to the end user; it's in there right now because I wasn't positive I'd fixed the out-of-sync problem. I've got a file that throws those errors as well, so I'll take a look at why the file size is different than DSD's.


----------



## moyekj

Here's one that has the "blockiness" problem again during playback. Interestingly tivodecode-ng is fine for this one (as is DSD of course):
https://drive.google.com/file/d/0B0SMFC97ymdERnloOVNveUt5bFk/view?usp=sharing


----------



## PaulS

fflewddur said:


> Got it. This file has PES headers that extend across multiple TS frames, but the decoding processes wasn't accounting for that. We only want to decrypt stream data, not stream headers. The latest version in GitHub results in a file that plays properly.
> 
> I did notice that some TS frames are out of order in the resulting file, although that doesn't seem to impact playback. I'll try to get that fixed up this weekend, regardless. I also want to take another look at the first file you sent to see if we can figure out what those mystery streams are, or at least, how to ensure they end up in the right place in the output file
> 
> Any other files that still have playback issues?


WOW!!!! GREAT JOB!!! That was *EXACTLY* the problem that tormented me for so long. I could never figure out how to fix that one...

Congrats!

Just as long as the packets for each stream as still in the proper order, it shouldn't have any adverse effects on playback.

The development of this app has been going in spurts for a long while, and it's exciting that we're finally so close to the finish line.


----------



## fflewddur

PaulS said:


> WOW!!!! GREAT JOB!!! That was *EXACTLY* the problem that tormented me for so long. I could never figure out how to fix that one...
> 
> Congrats!
> 
> Just as long as the packets for each stream as still in the proper order, it shouldn't have any adverse effects on playback.
> 
> The development of this app has been going in spurts for a long while, and it's exciting that we're finally so close to the finish line.


Thanks, it's been really fun learning about MPEG streams and getting this to work 

I'm not sure my earlier fix is exactly right, though. The file Kevin posted works properly if I go back to the old way of ignoring PES header lengths when calculating where to start decrypting each packet. Right now my hope is that the MpegParser isn't calculating PES header length correctly on the latest file; if it's reporting a header that's slightly too long, it messes up the next packet's decryption. This could also explain the out-of-order packets I've seen in hex dumps; if the PES header is reported as the length of a packet's payload, it gets buffered so that we can keep reading the rest of the header in the next packet. If a second stream has a packet interleaved with the first streams' packets and it *doesn't* have a PES header >= the packet length, it would be output before the buffered packet. That shouldn't have any impact on playback (as long as the packet's aren't encrypted) because each stream is still in the same logical order; it only becomes a problem if one of the buffered packets is encrypted and we start decrypting it at the wrong offset. That seems to match up with the behavior I'm seeing, but it's still just a hypothesis.

Now it's time to see what the MPEG standard says about how PES header lengths should be calculated, and see if that matches what the code is currently doing.


----------



## wmcbrine

moyekj said:


> William, any idea how to generate static tivodecode-ng binary for Windows? I used to do it with cygwin back when they supported static libraries, but now on cygwin one can only compile using dynamic libraries, so generating an executable without any cygwin dll dependencies is not possible. I've been using cygwin to compile/test so far. A while back I briefly looked into using mingw but didn't get very far.


I'd definitely go with MinGW -- it does pretty much everything Cygwin does (more so now than ever), minus the problems. Only, I'm finding there are some current issues that keep me from building tivodecode-ng with it. Working on that now.

I also like OpenWatcom, although I haven't even tried tivodecode-ng with it yet. And of course there's a free subset of Visual C++, which used to be compatible with tivodecode, so I'll have to at least make sure that works again, even if I don't care much for it.


----------



## fflewddur

I just committed a large rewrite of the PES parser and packet handling code; now every file posted in this thread not only decrypts and plays properly, but is also binary identical to the output of the TiVo DirectShow filter (except the last 120 bytes of sample4.TiVo; they don't form a complete 188-byte TS packet so TivoLibre discards them).

The PES handling is a little tricky: the DirectShow filter doesn't seem to count all of the different types of PES headers when calculating decryption offsets, so without extensive testing it's impossible to say whether TivoLibre is handling the correct subset of PES headers now. I'm going to try running this on all of the files from my TiVo over the next few days to see if there are any other binary differences from the DirectShow filter's output. If anyone else has problematic files to test it on, I'd love to hear the results.


----------



## wmcbrine

Amazing work, fflewddur.


----------



## moyekj

Awesome progress fflewddur, a big step forwards! Being binary identical to DSD gives good assurances and makes it much easier to test. As you say, a couple of my samples are a few bytes shorter than DSD because of the end of the file.

Unfortunately now a couple of samples I was testing with that were working previously now do not with the new code:
1st one is:
The Neighbors - (s02e10) Supreme Like Me_ts.TiVo
https://drive.google.com/file/d/0B0SMFC97ymdEc0ZZbmh0Ny1CVlk/view?usp=sharing


Code:


Sep 13, 2015 5:52:28 PM net.straylightlabs.tivolibre.PesHeader parsePesHeaderExtension
SEVERE: PES header extension starts with invalid bits: 0x1
Sep 13, 2015 5:52:28 PM net.straylightlabs.tivolibre.TransportStream getPesHeaderLength
SEVERE: Exception: java.lang.RuntimeException: PES header extension start with invalid bits
java.lang.RuntimeException: PES header extension start with invalid bits
        at net.straylightlabs.tivolibre.PesHeader.parsePesHeaderExtension(PesHeader.java:225)
        at net.straylightlabs.tivolibre.PesHeader.parsePesHeader(PesHeader.java:217)
        at net.straylightlabs.tivolibre.PesHeader.parseBytes(PesHeader.java:81)
        at net.straylightlabs.tivolibre.PesHeader.<init>(PesHeader.java:45)
        at net.straylightlabs.tivolibre.PesHeader.createFrom(PesHeader.java:54)
        at net.straylightlabs.tivolibre.TransportStream.getPesHeaderLength(TransportStream.java:108)
        at net.straylightlabs.tivolibre.TransportStream.calculatePesHeaderOffset(TransportStream.java:95)
        at net.straylightlabs.tivolibre.TransportStream.writePacket(TransportStream.java:69)
        at net.straylightlabs.tivolibre.TransportStreamDecoder.addPacketToStream(TransportStreamDecoder.java:351)
        at net.straylightlabs.tivolibre.TransportStreamDecoder.process(TransportStreamDecoder.java:113)
        at net.straylightlabs.tivolibre.TivoStream.process(TivoStream.java:85)
        at net.straylightlabs.tivolibre.TivoDecoder.decode(TivoDecoder.java:58)
        at net.straylightlabs.tivolibre.DecoderApp.decode(DecoderApp.java:123)
        at net.straylightlabs.tivolibre.DecoderApp.run(DecoderApp.java:95)
        at net.straylightlabs.tivolibre.DecoderApp.main(DecoderApp.java:40)

2nd one is:
h264_tivo_sample2.TiVo
https://drive.google.com/file/d/0B0SMFC97ymdESnc0ci1QdTF6eDg/view?usp=sharing


Code:


Sep 13, 2015 5:54:09 PM net.straylightlabs.tivolibre.TransportStreamPacket readHeader
SEVERE: Found private adaptation field data; we don't know how to handle this
Exception in thread "main" java.lang.IndexOutOfBoundsException
        at java.nio.Buffer.checkIndex(Unknown Source)
        at java.nio.HeapByteBuffer.get(Unknown Source)
        at net.straylightlabs.tivolibre.TransportStreamDecoder.createPacketAtNextSyncByte(TransportStreamDecoder.java:154)
        at net.straylightlabs.tivolibre.TransportStreamDecoder.process(TransportStreamDecoder.java:63)
        at net.straylightlabs.tivolibre.TivoStream.process(TivoStream.java:85)
        at net.straylightlabs.tivolibre.TivoDecoder.decode(TivoDecoder.java:58)
        at net.straylightlabs.tivolibre.DecoderApp.decode(DecoderApp.java:123)
        at net.straylightlabs.tivolibre.DecoderApp.run(DecoderApp.java:95)
        at net.straylightlabs.tivolibre.DecoderApp.main(DecoderApp.java:40)

So currently I have 10 test clips total:
6 produce binary identical to DSD
2 are binary identical to DSD except at very end of file
2 posted above have problems.


----------



## fflewddur

moyekj said:


> Unfortunately now a couple of samples I was testing with that were working previously now do not with the new code


Yeah, I just noticed this on one my test cases as well; it's fixed in my local repo and should be committed later this evening. It looks like I missed at least one PES header because I'm also seeing some decryption problems in the same test case, and want to get that ironed out before committing.


----------



## fflewddur

moyekj said:


> So currently I have 10 test clips total:
> 6 produce binary identical to DSD
> 2 are binary identical to DSD except at very end of file
> 2 posted above have problems.


Thanks for posting those files, they're both decoding properly now


----------



## PaulS

Impressive work guys. Simply amazing.... Kudos!


----------



## moyekj

moyekj said:


> So currently I have 10 test clips total:
> 6 produce binary identical to DSD
> 2 are binary identical to DSD except at very end of file
> 2 posted above have problems.





fflewddur said:


> Thanks for posting those files, they're both decoding properly now


 Fantastic! Now 8 out of 10 are identical to DSD output and none have any playback issues. Will test some more over next few days. Thank you very much for all your work on this!


----------



## fflewddur

It's been my pleasure! Quick question: do you test with the DirectShow filter from the last free release of TiVo Desktop, or the current release of TiVo Desktop Plus (assuming that includes a different filter)? 

I ask because I've been investigating how the DirectShow filter handles out-of-sync streams, and from what I can tell it not only includes all of the unsynchronized packets in the output, it also stops decrypting the stream for quite a while, then jumps back and outputs decrypted packets again. On this test file I'm seeing about 24MB worth of junk data from just one loss of synchronization: 1 MB of out-of-sync frames, then 23 MB of encrypted MPEG data. VLC plays the file perfectly, but QuickTime keeps losing audio sync and ffmpeg cries bloody murder about the file not being a proper MPEG-2 stream.

I'm striving for binary compatibility, so I want to make sure this isn't just a bug in the DirectShow DLL I'm using (which is from TiVo Desktop 2.8.3). Is this the recommended source for the DirectShow filter?


----------



## moyekj

TiVo Desktop Plus uses same filter as regular TiVo Desktop. I'm using 2.8.3 as well. Generally speaking at least for me the .TiVo files (and resulting decrypted files) have lots of timestamp problems and have to be run through VRD QS Fix to fix them up before you can take them through encoding with something like ffmpeg. After fixing with VRD there are generally no problems for me.


----------



## fflewddur

Ah, good to know. I think a --compatibility flag might end up being necessary; it's really nice to have binary compatibility with the DirectShow filter (especially for testing), but if we can fix up the file before writing it to disk, we probably should. I started working on this code so that I wouldn't have to rely on a Windows box with the DirectShow filter to archive recordings off of my TiVo... if we can also remove the Video ReDo dependency, so much the better


----------



## wmcbrine

fflewddur said:


> Quick question: do you test with the DirectShow filter from the last free release of TiVo Desktop, or the current release of TiVo Desktop Plus (assuming that includes a different filter)?


I'm pretty sure those are the same thing (2.8.3). It hasn't been updated in a long time (except for the expired cookie patch, but they didn't even bump up the version for that).


----------



## fflewddur

More progress! I just committed some big improvements toward full binary compatibility with the DirectShow filter: every file posted in this thread is now a bit-for-bit replica of the filter's output.

I also put together a script to automate batch testing, it's located in the root of the Git repository. Given a directory full of .TiVo files and MPEGs decoded with the DirectShow filter, the script will decode each .TiVo file, compare its output with the reference MPEG, and print a nice list at the end of all of the files that didn't decode to a bit-for-bit copy of the reference MPEG. The test script requires Python 3.5 and a directory of consistently named files, where every TiVo file has the same extension and every reference file has the same name as the corresponding TiVo file, plus a consistent extension (I used ".ref.mpg", but you can specify your own). Run it with the -h parameter for a full list of configurable options.

I'm running the test script right now on every file from my TiVo, about 200GB worth. If anyone else cares to run it on a large directory, I'd love to know the results.


----------



## moyekj

Excellent!
Funny about your script. I setup a similar one to run DSD and tivolibre on a bunch of .TiVo files and then run a diff (I used Perl). Indeed all of my .TiVo clips decrypted by tivolibre are now identical to DSD ones so I'd say it's working perfectly now based on the small set of files I'm testing with so far. At some point I need to download and run many more just to check things out.

One question for you. Have you tested downloading from a TiVo and piping the inputStream to TiVoDecoder? I have no trouble when the inputStream comes from a .TiVo file, but so far when it's from inputStream of a URLConnection I'm getting very strange results - improperly decrypted and truncated resulting file. Could well be I'm just doing something wrong and I haven't actually looked at your source code yet to check how reads from inputStream are being done, but figure I would check with you to see if you have tested something along those lines.


----------



## fflewddur

moyekj said:


> One question for you. Have you tested downloading a .TiVo file and piping the inputStream to TiVoDecoder? I have no trouble when the inputStream comes from a .TiVo file, but so far when it's from inputStream of a URLConnection I'm getting very strange results - improperly decrypted and truncated resulting file. Could well be I'm just doing something wrong and I haven't actually looked at your source code yet to check how reads from inputStream are being done, but figure I would check with you to see if you have tested something along those lines.


This _should_ work, but if the InputStream in question is an instance of PipedInputStream, the current code will have problems dealing with it. The trouble is in the CountingDataInputStream class; for PipedInputStreams, it uses a variable-length buffer to prevent the pipe's circular buffer from filling up and overwriting itself. Something isn't right though, because sometimes memory usage shoots through the roof and the CPU gets pegged at 100% usage (for comparison, the command-line app never pushes my CPU higher than 25%; it's definitely IO-bound). I'm planning to look into it once these compatibility issues are all worked out. When the InputStream provided to TivoDecoder is not an instance of PipedInputStream, the CountingDataInputStream class just wraps the stream in a DataInputStream, which doesn't have the issue.

You can find an example of how TivoDecoder is supposed to work with a network connection at https://github.com/fflewddur/archiv...lightlabs/archivo/controller/ArchiveTask.java, particularly the way-too-long handleResponse() method. Again, it doesn't work properly at the moment, but it'll get there 

EDIT: Now that I think more about it, the problem might be with the DataInputStream; the read() methods don't verify that the requested number of bytes were returned, and with a network connection they might be returning early. That would definitely screw up the rest of the decoding process.


----------



## moyekj

I found 2 samples that give binary differences vs DSD. I've uploaded the short one. The other one is 7 GB so don't want to upload the whole thing. If needed I can try and determine where differences are exactly to try and make a small sample. The shorter one is here:
https://drive.google.com/file/d/0B0SMFC97ymdEUGNZMFZ1dE1KTTQ/view?usp=sharing

NOTE: I deleted most of the samples I'd uploaded previously to google drive so most previous links to my files will no longer work.


----------



## fflewddur

moyekj said:


> I found 2 samples that give binary differences vs DSD. I've uploaded the short one. The other one is 7 GB so don't want to upload the whole thing. If needed I can try and determine where differences are exactly to try and make a small sample. The shorter one is here:
> https://drive.google.com/file/d/0B0SMFC97ymdEUGNZMFZ1dE1KTTQ/view?usp=sharing
> 
> NOTE: I deleted most of the samples I'd uploaded previously to google drive so most previous links to my files will no longer work.


Great, thanks for the clip. Don't worry about the longer one right now; I'm still working on fixing all of the issues with my own collection of TiVo files, so let's test it again after these are working.


----------



## fflewddur

Woohoo! The current code successfully decodes every file on my TiVo (about 200 GB), all in TS format, with only three files that show 1-byte differences from the DirectShow filter's output. The last file moyekj posted triggered a bug in the PS decoding, which has also been fixed.

I'm having trouble tracking down exactly how to fix this (hopefully final) bit difference--the differing bits all follow a loss of TS sync but before decryption has kicked in again, so they appear to be placeholders of some sort. They're always 188 bytes after file positions evenly divisible by 0x100000, e.g., 0x2ee000bf, 0x697000bf, and 0x376000bf (but a different position in each file, and at different offsets from the loss of sync, re-establishment of sync, and offset for encryption to restart). The DirectShow filter applies a mask of 0xC0 to these bytes, resulting in the two highest bits being turned off. This is exactly what the filter does at *every* file position divisible by 0x100000 for some time after a loss of sync, but it doesn't do it consistently 188 bytes after that position. Anyone else have an idea of what may be going on here?

(Regardless, resolving this issue will only improve binary compatibility--the masked bits are in a region of the file that has no impact on playback, and in fact should be removed from the file in order to make it a standards-compliant MPEG.)


----------



## moyekj

Thanks for the update. The update makes the short one I posted above identical to DSD. The 7 GB clilp still has ~250MB bytes differences (cmp -l file1 file2 | wc -l). If you have suggestions on how to easily narrow in on where the differences are I can try and generate a short clilp around that point.


----------



## moyekj

Was trying to narrow in on where the difference may be in the large clip using VRD. This short clip generated by cutting out a 1 min segment using VRD now crashes tivolibre:
https://drive.google.com/file/d/0B0SMFC97ymdEd2JBSkFhYlpUM00/view?usp=sharing


Code:


Sep 17, 2015 7:52:48 AM net.straylightlabs.tivolibre.TransportStream processPacket
SEVERE: Exception while calculating PES header offset: Unknown PES extension header type
java.lang.RuntimeException: Unknown PES extension header type
        at net.straylightlabs.tivolibre.PesHeader.parseExtensionHeader(PesHeader.java:293)
        at net.straylightlabs.tivolibre.PesHeader.parseBytes(PesHeader.java:87)
        at net.straylightlabs.tivolibre.PesHeader.<init>(PesHeader.java:46)
        at net.straylightlabs.tivolibre.PesHeader.createFrom(PesHeader.java:55)
        at net.straylightlabs.tivolibre.TransportStream.getPesHeaderLength(TransportStream.java:149)
        at net.straylightlabs.tivolibre.TransportStream.calculatePesHeaderOffset(TransportStream.java:132)
        at net.straylightlabs.tivolibre.TransportStream.processPacket(TransportStream.java:87)
        at net.straylightlabs.tivolibre.TransportStreamDecoder.decryptAndWritePacket(TransportStreamDecoder.java:391)
        at net.straylightlabs.tivolibre.TransportStreamDecoder.process(TransportStreamDecoder.java:122)
        at net.straylightlabs.tivolibre.TivoStream.process(TivoStream.java:85)
        at net.straylightlabs.tivolibre.TivoDecoder.decode(TivoDecoder.java:58)
        at net.straylightlabs.tivolibre.DecoderApp.decode(DecoderApp.java:125)
        at net.straylightlabs.tivolibre.DecoderApp.run(DecoderApp.java:95)
        at net.straylightlabs.tivolibre.DecoderApp.main(DecoderApp.java:40)


----------



## fflewddur

Thanks for sending me that clip, it should decode properly now.

I use Hex Fiend to find the exact location of differences in binary files; it displays a side-by-side comparison of the files with differences highlighted and searchable. If you don't have access to a Mac, VBinDiff should work, or any hex editor that supports large files.

For what it's worth, if you have a place to store a 7 GB file on the Internet, I don't mind the large download and would be happy to test with it. If it's easier than you hunting through it for the problem area, we could even setup a Torrent to transfer it directly between our computers.


----------



## moyekj

OK thanks, that clip now works. This evening I found several places in the big file with binary differences, some right at the beginning of the file - so I simply made a quick cut using "head --bytes=60000K" to produce a clip and the resulting clip decrypted with DSD plays back fine, but has several blocky segments as decrypted by tivolibre:
https://drive.google.com/file/d/0B0SMFC97ymdEakstWEZ2YW9BOEk/view?usp=sharing


----------



## fflewddur

moyekj said:


> OK thanks, that clip now works. This evening I found several places in the big file with binary differences, some right at the beginning of the file - so I simply made a quick cut using "head --bytes=60000K" to produce a clip and the resulting clip decrypted with DSD plays back fine, but has several blocky segments as decrypted by tivolibre:
> https://drive.google.com/file/d/0B0SMFC97ymdEakstWEZ2YW9BOEk/view?usp=sharing


Thanks, that clip should work properly now. Does the rest of the file decode as well?


----------



## moyekj

fflewddur said:


> Thanks, that clip should work properly now. Does the rest of the file decode as well?


 Excellent! Yes that clip and now the whole 7 GB file decrypt identically to DSD. I just ran all my checks again and they are all binary identical to DSD now (I'm up to 18 test files now).

Do you have any clips of your own that are still problematic?


----------



## fflewddur

Awesome! No, I don't know of any files that are still problematic. I just finished a batch test of my own: 91 files, all of which are identical to the DirectShow filter's output. I'm starting to download all of my TiVo Suggestions just to have something else to throw at it (and... really TiVo? These are your suggestions? It's almost as if you don't know me at all...). I'll post the results tomorrow, but I think we're getting close to something release-worthy...


----------



## moyekj

fflewddur said:


> Awesome! No, I don't know of any files that are still problematic. I just finished a batch test of my own: 91 files, all of which are identical to the DirectShow filter's output. I'm starting to download all of my TiVo Suggestions just to have something else to throw at it (and... really TiVo? These are your suggestions? It's almost as if you don't know me at all...). I'll post the results tomorrow, but I think we're getting close to something release-worthy...


That's truly amazing, job well done!


----------



## moyekj

tivodecode has an option to just retrieve/dump the XML metadata from a .TiVo file. Is it possible to add a Java method for doing that to tivolibre?


----------



## HerronScott

moyekj said:


> That's truly amazing, job well done!


Congratulations and I agree with job well done! It's been fun lurking and watching the progress! 

Scott


----------



## fflewddur

Thanks, this has been a fun project!

I checked the output of the test I ran overnight... every file decrypted perfectly except the very last. Figures, right? Still not quite ready for prime-time, but getting close. 

And yeah, there's no reason we couldn't add a method for getting the XML data. I've added it to the list of issues blocking a public release.


----------



## fflewddur

moyekj said:


> tivodecode has an option to just retrieve/dump the XML metadata from a .TiVo file. Is it possible to add a Java method for doing that to tivolibre?


Just checked in code to do this: there's now a _getMetadata()_ method that returns a List of XML Document objects. You'll need to call either _decode()_ or the new method _decodeMetadata()_ before calling _getMetadata()_; the latter method will exit after processing the metadata, it won't decode the video itself.

I also added the same -D and -x command-line parameters tivodecode uses to dump the metadata files and disable video processing. And, the command-line app will automatically save the user's MAK now, so you only need to provide it the first time its executed.


----------



## moyekj

fflewddur said:


> Just checked in code to do this: there's now a _getMetadata()_ method that returns a List of XML Document objects. You'll need to call either _decode()_ or the new method _decodeMetadata()_ before calling _getMetadata()_; the latter method will exit after processing the metadata, it won't decode the video itself.
> 
> I also added the same -D and -x command-line parameters tivodecode uses to dump the metadata files and disable video processing. And, the command-line app will automatically save the user's MAK now, so you only need to provide it the first time its executed.


 Awesome, thanks! I've been working on integrating tivolibre into kmttg and just replaced use of tivodecode with tivolibre for metadata generation from .TiVo files as well as adding option to use tivolibre for decrypt and/or combined download/decrypt. Things are working very nicely.
I take it you solved the problem with your last file that didn't decrypt properly with this latest version?


----------



## fflewddur

moyekj said:


> Awesome, thanks! I've been working on integrating tivolibre into kmttg and just replaced use of tivodecode with tivolibre for metadata generation from .TiVo files as well as adding option to use tivolibre for decrypt and/or combined download/decrypt. Things are working very nicely.
> I take it you solved the problem with your last file that didn't decrypt properly with this latest version?


Great, can't wait to try it out! No, I haven't cracked that last file, but it plays properly; there's only one byte that's different from the reference file, and it's in that loss-of-sync area that gets skipped over during playback. Until I see the same problem in more files, it'll be hard to figure out what the pattern is that causes it.


----------



## fflewddur

Just a heads-up: I put together a new release today (version 0.7.0), and a couple of important things have changed in the API.

First, I added a compatibility-mode switch for maintaining binary compatibility with the TiVo DirectShow filter. Compatibility mode is disabled by default, which means TivoLibre will try to clean up obvious errors in TS files. Right now that means dropping out-of-sync frames, frames that even the DLL leaves encrypted, null frames, and orphaned packets. In the future, I'd also like to fix out-of-order packets and correct timestamps.

Second, I've moved to the Builder pattern for creating TivoDecoder objects and deprecated the public constructor. This makes adding options (like the compatibility-mode switch) easy to do without constantly adding newer, more complicated constructors. The API examples in the README file has been updated to illustrate the new usage.


----------



## moyekj

Correcting timestamp errors would be huge - many TiVo downloads for me have timestamp issues that require VideoRedo to clean them up. AFAIK fixing timestamp issues requires a remux doesn't it?


----------



## moyekj

Having trouble with this clip:
https://drive.google.com/file/d/0B0SMFC97ymdEdGVoTFFwdWF2cmM/view?usp=sharing


Code:


Sep 22, 2015 5:09:11 AM java.util.prefs.WindowsPreferences <init>
WARNING: Could not open/create prefs root node Software\JavaSoft\Prefs at root 0x80000002. Windows RegCreateKeyEx(...) returned error code 5.
Encryption by QUALCOMM
05:09:11.933 [main] ERROR c.s.tivolibre.TivoDecoder - Problem parsing Turing header
05:09:11.935 [main] ERROR c.s.tivolibre.TivoDecoder - Decrypting packet failed
Exception in thread "main" java.lang.RuntimeException: Decrypting packet failed
        at net.straylightlabs.tivolibre.TransportStream.decryptPacket(TransportStream.java:170)
        at net.straylightlabs.tivolibre.TransportStream.processPacket(TransportStream.java:98)
        at net.straylightlabs.tivolibre.TransportStreamDecoder.decryptAndWritePacket(TransportStreamDecoder.java:404)
        at net.straylightlabs.tivolibre.TransportStreamDecoder.process(TransportStreamDecoder.java:123)
        at net.straylightlabs.tivolibre.TivoStream.processVideo(TivoStream.java:131)
        at net.straylightlabs.tivolibre.TivoStream.process(TivoStream.java:77)
        at net.straylightlabs.tivolibre.TivoDecoder.decode(TivoDecoder.java:76)
        at net.straylightlabs.tivolibre.DecoderApp.decode(DecoderApp.java:168)
        at net.straylightlabs.tivolibre.DecoderApp.run(DecoderApp.java:131)
        at net.straylightlabs.tivolibre.DecoderApp.main(DecoderApp.java:52)

If it makes a difference, the above was uploaded to TiVo using pyTivo and then transferred back. The original clip was longer, but I managed to replicate problem on truncated smaller file.


----------



## fflewddur

Interesting, that file is the first example I've seen of decryption on a packet prior to reading an encryption key from the data stream, and it also sheds some light on what's happening inside the doHeader() method! The crash is fixed and the file plays back properly, though it's not a bit-for-bit copy with the DLL's output. I'll keep working on it to see if I can figure out how the DLL is setting up its decryption keys prior to the first private data packet.


----------



## fflewddur

moyekj said:


> Correcting timestamp errors would be huge - many TiVo downloads for me have timestamp issues that require VideoRedo to clean them up. AFAIK fixing timestamp issues requires a remux doesn't it?


I'm not sure; to be honest, this project is my first experience working with MPEG streams. But now that I've written an entire frame-handling pipeline, plus a good chunk of the necessary PES handling code, it seems like going a step further and correcting the data flowing through the pipeline should be feasible.

On the other hand, I've been playing around with recent ffmpeg builds and they seem to fix the timestamp issues too, so there may not be much need for TivoLibre to handle this.


----------



## moyekj

In latest version I'm not allowed to create a TivoDecoder instance with null output stream. I was doing that because for case when I'm only interested in getting metadata there's no need for an output stream, so I was using:


Code:


TivoDecoder decoder = new TivoDecoder.Builder().input(inputStream).output(null).mak(MAK).build();
if (decoder.decodeMetadata()) {
   docList = decoder.getMetadata();
...
...

Is there a better way to obtain just XML metadata without having to create a (temporary) output stream?


----------



## fflewddur

Ah, I hadn't thought of that. I fixed up the API so that it doesn't require an OutputStream when building a TivoDecoder. It'll still throw an exception if you try to decode a video without an OutputStream set, but it should let you decode the metadata without one.


----------



## moyekj

fflewddur said:


> Ah, I hadn't thought of that. I fixed up the API so that it doesn't require an OutputStream when building a TivoDecoder. It'll still throw an exception if you try to decode a video without an OutputStream set, but it should let you decode the metadata without one.


 Thanks, working well with those changes.


----------



## Allanon

Recently my cable company informed me that my Tivo HD will no longer support decoding some of the HD channels because they are changing the video compression format. The only option was to upgrade to a Tivo series 4 or above or use their cable box which is a Tivo Premiere. I opted not to upgrade because I'm moving soon and will just upgrade to their cable boxes when I do move.

But I do have one of their cable boxes now and I can see and record all HD channels with it. But when I download the .TiVo file from one of those newly encoded channels and try to decode it, all I get is audio. I tried both tivodecode and and tivo-libre but both just produce an audio only mpeg. Also when I transfer it from the cable box to my Tivo HD all I get is audio.

Do you think you could figure out how to decode one of these files? Below is a link to a .Tivo file with about 1 minute of recorded video. I will PM my MAK if needed.

test.zip (3.42 MB)


----------



## PaulS

Sounds like they converted to MPEG4/h.264 (from the former standard bandwidth hungry MPEG2) for the bandwidth savings. This will be a common thing in the near future. I believe Comcast has indicated that they will be doing this conversion this year. 

I had some Aussies/NZ folks give me samples of this stuff, and tivodecode 0.4.4 seemed to handle it without issue.


----------



## Allanon

PaulS said:


> I had some Aussies/NZ folks give me samples of this stuff, and tivodecode 0.4.4 seemed to handle it without issue.


I'm using tivodecode 0.3pre4, where can I get a Windows binary file of version 0.4.4? I don't have a compiler installed on my computer.


----------



## moyekj

Allanon said:


> Recently my cable company informed me that my Tivo HD will no longer support decoding some of the HD channels because they are changing the video compression format. The only option was to upgrade to a Tivo series 4 or above or use their cable box which is a Tivo Premiere. I opted not to upgrade because I'm moving soon and will just upgrade to their cable boxes when I do move.
> 
> But I do have one of their cable boxes now and I can see and record all HD channels with it. But when I download the .TiVo file from one of those newly encoded channels and try to decode it, all I get is audio. I tried both tivodecode and and tivo-libre but both just produce an audio only mpeg. Also when I transfer it from the cable box to my Tivo HD all I get is audio.
> 
> Do you think you could figure out how to decode one of these files? Below is a link to a .Tivo file with about 1 minute of recorded video. I will PM my MAK if needed.
> 
> test.zip (3.42 MB)


 Your problem is you are downloading in PS container. You need to switch to TS container downloads and then it should be fine using tivolibre to decrypt.


----------



## Allanon

moyekj said:


> Your problem is you are downloading in PS container. You need to switch to TS container downloads and then it should be fine using tivolibre to decrypt.


You were right, both tivodecode and tivo-libre decode the file when downloaded using TS container. Thanks!


----------



## PaulS

moyekj said:


> Your problem is you are downloading in PS container. You need to switch to TS container downloads and then it should be fine using tivolibre to decrypt.


Wait... tivolibre should be able to detect and decode both TS and PS. Am I incorrect in that assumption ?


----------



## moyekj

PaulS said:


> Wait... tivolibre should be able to detect and decode both TS and PS. Am I incorrect in that assumption ?


 tivolibre can decrypt either TS or PS containers. But recordings from H.264 channels from cable MUST be downloaded using TS container or you just get audio stream without video stream. That was Allanon's issue.


----------



## PaulS

moyekj said:


> tivolibre can decrypt either TS or PS containers. But recordings from H.264 channels from cable MUST be downloaded using TS container or you just get audio stream without video stream. That was Allanon's issue.


Of course you're right. I'd totally forgotten that.


----------



## gemann

I have an audio issue with decoding .TiVo files from certain TV channels only. It appears the audio is mixed down using only the back channels, the audio from the front and center channels is dropped. I downloaded the .TiVo file in the TS format and the PS format with the same results and decoded using the latest kmttg/tivolibre.

I have uploaded a sample .TiVo file for debugging purposes, but I can't post a link here yet since my post count is too low. Please PM me for the link to the file and the associated MAK.

I would really appreciate it if you could take a look at what is going on here!


----------



## fflewddur

gemann said:


> I have an audio issue with decoding .TiVo files from certain TV channels only. It appears the audio is mixed down using only the back channels, the audio from the front and center channels is dropped. I downloaded the .TiVo file in the TS format and the PS format with the same results and decoded using the latest kmttg/tivolibre.
> 
> I have uploaded a sample .TiVo file for debugging purposes, but I can't post a link here yet since my post count is too low. Please PM me for the link to the file and the associated MAK.
> 
> I would really appreciate it if you could take a look at what is going on here!


Sure thing, PM sent.


----------



## fflewddur

gemann said:


> I have an audio issue with decoding .TiVo files from certain TV channels only. It appears the audio is mixed down using only the back channels, the audio from the front and center channels is dropped. I downloaded the .TiVo file in the TS format and the PS format with the same results and decoded using the latest kmttg/tivolibre.
> 
> I have uploaded a sample .TiVo file for debugging purposes, but I can't post a link here yet since my post count is too low. Please PM me for the link to the file and the associated MAK.
> 
> I would really appreciate it if you could take a look at what is going on here!


I think the problem is with the file itself. I tested it with the TiVo-provided DLL, and it has the same audio issues. I also checked TivoLibre's output against the TiVo DLL, and they're bit-for-bit copies of one another. The file reports a single AC3 audio stream, exactly as it should, but it appears to be missing the front and center channels. Are those channels present when you play it on your TiVo? Does this happen with all recordings from certain channels, or do some recordings from those channels work as expected?


----------



## gemann

Yes those recordings play back fine on the TiVo itself. It appears to happen with all the recordings from this particular channel (I record three different shows). Recordings from other channels appear to convert ok. 

So this happens on the TiVo end during the copy?


----------



## fflewddur

gemann said:


> Yes those recordings play back fine on the TiVo itself. It appears to happen with all the recordings from this particular channel (I record three different shows). Recordings from other channels appear to convert ok.
> 
> So this happens on the TiVo end during the copy?


That's my guess at this point. The fact that TiVo Desktop decodes it exactly like TivoLibre suggests the problem isn't with the decoding process. It could be a problem with how the TiVo software encodes certain video formats, which your provider may only use on certain channels. I'm going to try recording the same channel as this clip to see if I can reproduce the problem with my own setup.


----------



## He'sDeadJim

I have a desktop and a laptop computer. Both have Win 7, kmttg v2.0u, and the current Java version. As far as I can tell kmttg has the same configuration on both.

I'm getting odd behavior out of tivolibre. I've downloaded (in .ts) the same programs to both computers, but the laptop generates .mpg files while the desktop gives .ts files. Since I use VLC I can watch either just as easily, so this is just curiosity I guess.

The only difference I've found is that when the download starts on the desktop the tivolibre line in the status window shows it will output an .mpg file. But after the download finishes and the decode begins the output switches to a .ts file.

I've been married long enough to know everything is my fault, so any tips on this would be appreciated.

Bob


----------



## moyekj

He'sDeadJim said:


> I have a desktop and a laptop computer. Both have Win 7, kmttg v2.0u, and the current Java version. As far as I can tell kmttg has the same configuration on both.
> 
> I'm getting odd behavior out of tivolibre. I've downloaded (in .ts) the same programs to both computers, but the laptop generates .mpg files while the desktop gives .ts files. Since I use VLC I can watch either just as easily, so this is just curiosity I guess.
> 
> The only difference I've found is that when the download starts on the desktop the tivolibre line in the status window shows it will output an .mpg file. But after the download finishes and the decode begins the output switches to a .ts file.
> 
> I've been married long enough to know everything is my fault, so any tips on this would be appreciated.
> 
> Bob


 You probably have TS downloads set for kmttg on one computer and not set on the other. The setting is under:
Configure--Program Options--Download TiVo files in Transport Stream format


----------



## He'sDeadJim

moyekj said:


> You probably have TS downloads set for kmttg on one computer and not set on the other. The setting is under:
> Configure--Program Options--Download TiVo files in Transport Stream format


They are both set for .ts format, and have been for some time.


----------



## mackworth

So, (he asked hesitantly, but hopefully) how can we get these fixes backported into tivodecode-ng?


----------



## PaulS

mackworth said:


> So, (he asked hesitantly, but hopefully) how can we get these fixes backported into tivodecode-ng?


That wouldn't be the first question I'd ask. I'd be asking if the Java implementation had comparable performance to the C++ tivodecode-ng codebase.

If the two implementations were comparable (or Java was more performant), then I really don't see the need for the backport. tivolibre removes a whole lot of platform specific (Windows, Linux, macOS) build and release headaches that Kevin and William have to deal with purely based on Java's portability.

Just asking... why is tivodecode-ng important to your workflow ? How does Java + tivolibre fall short ?


----------



## moyekj

Compiled C/C++ code is almost always faster than any scripting or semi-scripting language, so ported to tivodecode-ng would no doubt be faster than Java. Having said that the difference is not that dramatic from my experience and tivolibre is plenty fast enough as is and avoids the hassle of trying to compile for different platforms. William might find a port to python more useful than a port to C/C++. I don't think either way porting would be very trivial since I believe Todd used some 3rd party packages as part of tivolibre which means potentially porting more than just Todd's code.


----------



## wmcbrine

mackworth said:


> So, (he asked hesitantly, but hopefully) how can we get these fixes backported into tivodecode-ng?


I'm considering Adderall.


----------



## fflewddur

A C++ port would certainly be possible, but I refactored enough of the code that I'm not sure backporting to the existing tivodecode-ng framework would be particularly easy. That said, the only 3rd-party requirements in TivoLibre are for command-line parsing and logging; nothing related to the video processing/decoding depends on anything that C++ can't easily do.

The better question really is: why? The Java version is plenty fast enough to run in real-time. It uses about twice as much CPU time as the C++ version, but the total runtime is essentially identical because decoding large files is IO-bound, not CPU-bound. Even my SSD can't load a large video faster than my CPU can handle it. I've profiled it, and the only spot that would even benefit from further optimization is the Turing cypher implementation; nothing else in TivoLibre gets called enough to have a significant impact on the runtime. I've done what I can with that, but if there are any expert Java programmers reading, please, submit a PR on https://github.com/fflewddur/quick-turing and help us out


----------



## mackworth

Well, Java certainly makes the binary more portable, but it also adds a requirement to have the Java VM installed. My impression from the thread was that tivodecode-ng was a pretty complete implementation of the transport stream problem, but that there were a few intransigent bugs, which fflewddur manage to figure out and fix for tivolibre. I was hoping those changes could get ported back.


----------



## dougblair

Hello...

I have been a kmttg user for several years and have just discovered this thread. AMAZING work, all of you! It's a true delight to watch is kind of collaborative development work across several months, from different locations, and maybe some of you have real jobs . Your generosity with your time and skills and expertise is greatly appreciated. 

Please know that if I ever get away from my own TiVo downloading, decoding, transcoding and uploading to AWS addictive activities (resulting in some 60+ TB of old TV shows, now uploading to the Amazon Unlimited storage cloud, which is setting some kind of consumer account record) and find myself with enough time to actually watch some of the programs, I will be thinking about all your hard work!

Thank you!
Virtual beers! (Other suggestions will be accommodated!)

Doug Blair, Chicago


----------



## herbman

This may be old news, but I've been successful with the following pattern from a bash/cygwin shell:



Code:


wget -O - 'http://tivo:<mak>@....' | java -jar tivo-libre-0.7.2.jar -m <mak> | ffplay -

Thank you for this convenient way to watch items directly without steps!

2 tuner premiere on ethernet.


----------



## herbman

fflewddur said:


> I've profiled it, and the only spot that would even benefit from further optimization is the Turing cypher implementation; nothing else in TivoLibre gets called enough to have a significant impact on the runtime. I've done what I can with that, but if there are any expert Java programmers reading, please, submit a PR on https://github.com/fflewddur/quick-turing and help us out


Any chance you have a unit test which runs this a number of times to benchmark? I'm happy to play with the code but I'd love to have a straightfoward way to know if there's an impact.


----------



## fflewddur

herbman said:


> This may be old news, but I've been successful with the following pattern from a bash/cygwin shell:
> 
> 
> 
> Code:
> 
> 
> wget -O - 'http://tivo:<mak>@....' | java -jar tivo-libre-0.7.2.jar -m <mak> | ffplay -
> 
> Thank you for this convenient way to watch items directly without steps!
> 
> 2 tuner premiere on ethernet.


Nice! TivoLibre also caches your MAK, so if you're scripting this, you shouldn't even need the -m parameter after the first run.


----------



## fflewddur

herbman said:


> Any chance you have a unit test which runs this a number of times to benchmark? I'm happy to play with the code but I'd love to have a straightfoward way to know if there's an impact.


I don't; I use VisualVM to profile Java, so I manually run TivoLibre on a large file while capturing CPU and memory usage in VisualVM. The exact CPU time varies too much to be useful, but the percentage of time spent in each method is very stable across invocations, so I just pay attention to that. If you're optimizing the Turing code, you should see the percentage of time spent in the I/O methods increase while the time spent in the generateTuring() method decreases.


----------



## forumuser

I have several recordings on my tivo (premiere) which fail to fully download when retrieving in PS format, but download fully when downloaded in TS format.

However, the partial PS download (after decoding) plays back without hickups, but the TS download (after decoding) plays back with a ton of issues, even during the same spots where the PS file plays normally. The issues are primarily on the video: lots of artifacts.

The above occurs with both tivodecode (0.4.4) and tivolibre (latest).

Is this normal/expected behaviour? Analyzing this from a blackbox perspective, it would seem that the tivo's ts_to_ps converter on the device itself is able to fix issues with the TS file, but chokes when handling some of those issues (this is what I presume aborts the transfer of the PS formatted file). So, I am wondering if something similar could be done on the external decoding software (eg. tivolibre), but without the choking!


----------



## moyekj

forumuser said:


> I have several recordings on my tivo (premiere) which fail to fully download when retrieving in PS format, but download fully when downloaded in TS format.
> 
> However, the partial PS download (after decoding) plays back without hickups, but the TS download (after decoding) plays back with a ton of issues, even during the same spots where the PS file plays normally. The issues are primarily on the video: lots of artifacts.
> 
> The above occurs with both tivodecode (0.4.4) and tivolibre (latest).
> 
> Is this normal/expected behaviour? Analyzing this from a blackbox perspective, it would seem that the tivo ts_to_ps converter is able to fix issues with the TS file, but chokes on some (this is what I presume aborts the transfer of the PS formatted file). So, I am wondering if something similar could be done on the external decoding software (ie. tivolibre), but without the choking!


 You should check how TiVo DirectShow Dump decrypts it since that is THE standard to compare against. If it has issues as well then the problem is in the .TiVo file itself.


----------



## forumuser

The same problem occurs when I use the "directshow dump" utility on windows.

What is curious to me is that, as I understand it: Tivo stores the recordings in its harddrive in the the native format, which is "MPEG-2 transport stream" (aka TS). Note that MPEG-2 here has nothing to do with the video codec used for the video elementary stream within.

So, if one pulls the recording off the unit requesting it in TS format, then one is getting a copy of what is on the tivo's harddrive.

However, if one pulls the recording off the unit requesting it in PS format, then the tivo has to take what it has in its harddrive and convert it from TS to MPEG-2 PS format. Again, MPEG-2 here is not related to the elementary video stream within.

If the above is correct, then why is it that the "processed" file (i.e. the PS file) after being decoded plays back normally, while the source file (ie. TS file) after being decoded does not?

It has been almost a decade since I worked on MPEG-2 streams, but I think this has something to do with filler packets needed to maintain a bit rate or a clock or something like that. I can't recall right now.

Anyhow, I think we should be able to fix this in the post processing software (eg. tivolibre), just as the tivo's software does when it converts from TS to PS on the fly during the transfer.

I will dig in a bit into these streams.


----------



## Dan203

The TiVo does not store the file as a TS. They store the audio and video data separately as chunks and then links the chunks using a database. So regardless of which format you download the audio and video are being remuxed and encrypted on the fly. TS is a simpler format so it allows them to have better tollerance for stream errors, but the encryption process can still cause issues with corrupt streams.


----------



## forumuser

Thanks Dan203. That explains a lot.

Are there any known reasons as to why Tivo stores the recordings as you describe as opposed to using the source format?

It seems strange to do that, but I am sure they had good reasons to do it... I wonder what they gain by doing that. Anyone know? Just curious...


----------



## Dan203

The TiVo code started it's life in the SD world where they had to actually transcode analog audio/video to store it on the hard drive. I assume the way they store the streams was designed around that. When the S3 units came along and they started recording the digital streams directly it was probably easier to demux the incoming transport streams and store them the old way then it was to redesign the whole system to work around storing the transport streams directly.


----------



## forumuser

Cool. Thx for that insight. I forgot that this product has been around since before the HDTV changeover!


----------



## global_dev

great work guys! having some problems though

i just tried to change from tivodecode to tivolibre and i am getting some weird behavior, the log shows that it starts a job doenloading and decrypting to .mpg, but then it changes on the screen to .ts and i don't see the .mpg again (even in the log). perhaps i am doing something inappropriate. i've attached some screenshots and the log, maybe you guys can help.

I had no problems with the tivodecode + decrpyt prior. just looking to speed up the jobs.

Premiere
OSX El Cap
kmttg 2.1h ( i know the pics indicate an older version, but that's just the download spot w/upgrades)
tvlibre 0.7.2.




























log



Spoiler






Code:


>> DOWNLOADING/DECRYPTING TO /Users/j/TEST_VIDEO__2007.ts ...
Configuration saved to file: /Users/j/kmttg/config.ini

Refreshing encoding profiles
TEST_VIDEO__2007.ts: size=1682.38 MB elapsed=0:04:27 (52.86 Mbps)

---DONE--- 

job=tdownload_decrypt output=TEST_VIDEO__2007.ts

>> Running ccextractor on TEST_VIDEO__2007.ts ...
"ccextractor.0.63/ccextractor 0.6.3" "TEST_VIDEO__2007.ts" 

ccextractor job completed: 0:00:26

---DONE--- 

job=captions output=TEST_VIDEO__2007.srt





couldn't embed the vid, so the link is below of the behavior


----------



## global_dev

global_dev said:


> i just tried to change from tivodecode to tivolibre and i am getting some weird behavior, the log shows that it starts a job downloading and decrypting to .mpg, but then it changes on the screen to .ts and i don't see the .mpg again (even in the log).
> tvlibre 0.7.2.


so if i unmark download via TS, it all seems to work.. any reason i shouldn't try to get TS downloads working other than PS seems to work?


----------



## moyekj

global_dev, 2 things:
1. There is no need to replace tivodecode with tivolibre as you are doing. tivolibre jar is embedded in kmttg so as long as you turn on option in config as you did that's all you need.

2. When you switch to TS downloads the suffix changes from .mpg to .ts by design because .mpg implies mpeg2 program stream container, and .ts implies transport stream container. So I don't see any issues at all based on what you are showing.


----------



## moelleref

I'm using kmttg v2.1h and I've noticed that after downloading a number of files from my tivo and decrypting them into the mpeg ts format, they are highly pixilated. So much so they are unwatchable. I do not see any pixilation when I watch the recording on my tivo. 

Currently it seems to be happening mostly with recordings from FOX HD. 

I have a Tivo Premier XL, and I'm downloading the files in transport stream format. I'm using tivolibre to decrypt the file.

I've also re-downloaded the videos in PS format and decrypted them using tivolibre and they look ok.

A sample of the tivo file and the ts file can be found at the following location.

drive.google.com/folderview?id=0B1HsrgAVb3dxY0k3akdReEdBekk&usp=sharing


Any ideas what my be causing this issue when decrypting them into a ts file?

Thanks for any help.


----------



## moyekj

moelleref said:


> I'm using kmttg v2.1h and I've noticed that after downloading a number of files from my tivo and decrypting them into the mpeg ts format, they are highly pixilated. So much so they are unwatchable. I do not see any pixilation when I watch the recording on my tivo.
> 
> Currently it seems to be happening mostly with recordings from FOX HD.
> 
> I have a Tivo Premier XL, and I'm downloading the files in transport stream format. I'm using tivolibre to decrypt the file.
> 
> I've also re-downloaded the videos in PS format and decrypted them using tivolibre and they look ok.
> 
> A sample of the tivo file and the ts file can be found at the following location.
> 
> drive.google.com/folderview?id=0B1HsrgAVb3dxY0k3akdReEdBekk&usp=sharing
> 
> Any ideas what my be causing this issue when decrypting them into a ts file?
> 
> Thanks for any help.


fflewddur, I can confirm the above sample from moelleref seem to be a problem with tivolibre and TS decrypt. Decrypting with DSD results in a video that plays fine. (I obtained moelleref's 10 digit MAK via PM/email). Note that moelleref also confirmed that downloading in PS container and decrypting that way works fine, so it's TS specific decrypt issue.


----------



## fflewddur

moyekj said:


> fflewddur, I can confirm the above sample from moelleref seem to be a problem with tivolibre and TS decrypt. Decrypting with DSD results in a video that plays fine. (I obtained moelleref's 10 digit MAK via PM/email). Note that moelleref also confirmed that downloading in PS container and decrypting that way works fine, so it's TS specific decrypt issue.


Cool, thanks for the looking into this! Moelleref, can you PM me your MAK so I can investigate this myself and figure out where TivoLibre is going wrong?


----------



## moelleref

fflewddur said:


> Cool, thanks for the looking into this! Moelleref, can you PM me your MAK so I can investigate this myself and figure out where TivoLibre is going wrong?


fflewddur, I do not have enough posts yet to send PMs. However, if you PM me your email address (I can read PMs; however, I can't reply), I'll email my MAK to you.


----------



## moyekj

moelleref said:


> fflewddur, I do not have enough posts yet to send PMs. However, if you PM me your email address (I can read PMs; however, I can't reply), I'll email my MAK to you.


 I sent a PM to fflewddur with your MAK.


----------



## ejonesss

i have kmttg and just got the latest tivolibre jar how do i install the jar so i can use it with kmttg?


----------



## fflewddur

ejonesss said:


> i have kmttg and just got the latest tivolibre jar how do i install the jar so i can use it with kmttg?


TivoLibre is already included in the kmttg.jar, you don't need to download it separately.


----------



## ejonesss

does updating the tools and kmttg get the latest jar from https://github.com/fflewddur/tivolibre/releases ?


----------



## moyekj

ejonesss said:


> does updating the tools and kmttg get the latest jar from https://github.com/fflewddur/tivolibre/releases ?


 No.


----------



## Jayboy3

I just wanted to share that DSD works like a champ for me, gremlin free. Just curious, why isn't everybody using it? Does tivolibre or tivodecode fit some other circumspect better? 

Sent from my SM-N900V using Tapatalk


----------



## HerronScott

Jayboy3 said:


> I just wanted to share that DSD works like a champ for me, gremlin free. Just curious, why isn't everybody using it? Does tivolibre or tivodecode fit some other circumspect better?


From the kmttg page on descrypting TS TiVo files:

Windows users have option of using DirectShow Dump (DSD) program for decrypting .TiVo files instead of using tivodecode. The advantage of DSD is that it works with both PS and TS .TiVo files. The disadvantages are:

Only runs on Windows platform
Requires at least a partial install of TiVo Desktop, which is achieved by first fully installing TiVo Desktop and then uninstalling and choosing option to keep partial install.

http://sourceforge.net/p/kmttg/wiki/Decrypting_TS_TiVo_files/

This probably should be updated to include tivolibre as an option.

Scott


----------



## moyekj

HerronScott said:


> This probably should be updated to include tivolibre as an option.


 Done, thanks.


----------



## ejonesss

moyekj said:


> No.


then it is necessary to use the tivolibre jar and to add a way to use it in future version.


----------



## moyekj

ejonesss said:


> then it is necessary to use the tivolibre jar and to add a way to use it in future version.


 kmttg has regular updates, so any necessary tivolibre updates will be incorporated quickly, so no need.


----------



## ejonesss

ok is there source so i can make updates my self and what do i need to write and compile?


----------



## moyekj

ejonesss said:


> ok is there source so i can make updates my self and what do i need to write and compile?


https://sourceforge.net/p/kmttg/wiki/build_from_source/


----------



## fflewddur

moelleref said:


> I'm using kmttg v2.1h and I've noticed that after downloading a number of files from my tivo and decrypting them into the mpeg ts format, they are highly pixilated. So much so they are unwatchable. I do not see any pixilation when I watch the recording on my tivo.
> 
> Currently it seems to be happening mostly with recordings from FOX HD.
> 
> I have a Tivo Premier XL, and I'm downloading the files in transport stream format. I'm using tivolibre to decrypt the file.
> 
> I've also re-downloaded the videos in PS format and decrypted them using tivolibre and they look ok.
> 
> A sample of the tivo file and the ts file can be found at the following location.
> 
> drive.google.com/folderview?id=0B1HsrgAVb3dxY0k3akdReEdBekk&usp=sharing
> 
> Any ideas what my be causing this issue when decrypting them into a ts file?
> 
> Thanks for any help.


Just wanted to give a status update: I found the problem and think I've got it fixed, but I'm going to let it run through my entire test set before releasing. Hopefully I'll have the fix posted tomorrow night


----------



## ejonesss

http://workupload.com/file/FlvfmPxo sample of improper decrypt of ts format tivo files

fflewddur check your pm i sent the mak for you to check


----------



## fflewddur

ejonesss said:


> http://workupload.com/file/FlvfmPxo sample of improper decrypt of ts format tivo files
> 
> fflewddur check your pm i sent the mak for you to check


Got it, thanks for sending the file! I found the problem and am working on a fix...


----------



## HerronScott

fflewddur said:


> Got it, thanks for sending the file! I found the problem and am working on a fix...


Wow that was fast (in both cases)!

Scott


----------



## moyekj

Yes, great work in quickly finding fixes to remaining issues Todd! We appreciate your work.


----------



## fflewddur

moyekj said:


> Yes, great work in quickly finding fixes to remaining issues Todd! We appreciate your work.


Thanks, it's my pleasure! I'm going to let my test script run overnight, and if everything looks good, should have a release that fixes both of these issues out tomorrow after work.


----------



## HerronScott

fflewddur said:


> Thanks, it's my pleasure! I'm going to let my test script run overnight, and if everything looks good, should have a release that fixes both of these issues out tomorrow after work.


I'm curious what you found in each case?

Scott


----------



## fflewddur

HerronScott said:


> I'm curious what you found in each case?


Both issues were related to problems with how TivoLibre handles PES headers. Each TS packet begins with a series of PES headers which should not be decrypted, even if the TS packet itself says that it's encrypted. Sometimes these are split across multiple TS packets, so I had code that looked at the last PES header in each packet and calculated how far into the next packet it would extend, so decryption wouldn't start too soon (based on the structure of its output files, this seems to be the same thing TiVo Desktop does; tivodecode tries to concat all of the TS packets together, so it shouldn't have this issue, although it means tivodecode creates output files that aren't binary identical to TiVo Desktop. That makes it much harder to test whether tivodecode's output is correct, since a simple diff against the output of the same file run through TiVo Desktop will find differences in TS packet order).

Anyway, turns out there were a pair of PES header edge cases that never occurred in my test collection (so thanks for the bug reports!): 1) the PES header's start code can end in one packet but with the rest of the header in the next packet, or 2) the PES header's prefix (23 bits of 0 followed by a 1 bit) can be split across two TS packets. TivoLibre was treating both of these cases as if the PES headers had ended in the first packet, so if the next packet was encrypted, it would start decrypting it at the start of the TS packet, instead of skipping over the PES headers and starting decryption once they ended. Because TiVo files use a stream cypher, beginning the decryption even one byte too soon means that every byte after that point will be incorrectly decoded. TiVo files only encrypt a small percentage of their total packets, hence the largely normal video with blocks of pixelation: those pixelation artifacts were the improperly decrypted packets.

Tonight's release (0.7.3, available at https://github.com/fflewddur/tivolibre/releases) fixes both of these issues


----------



## moelleref

Thanks fflewddur for all your help fixing my tivolibre issue!


----------



## HerronScott

fflewddur said:


> Both issues were related to problems with how TivoLibre handles PES headers.)


Thanks for the explanation/education. 

Scott


----------



## mackworth

Hi @fflewddur. I was having a problem with certain shows and tivoLibre, and it took me quite a while to track down. These shows (e.g. Daily show on Comedy Central) sometimes download with a sync problem that causes tivoLibre 0.7.3 to lose track of the process, dropping about 43M before continuing, and the resulting mpg file is unplayable/unconvertible beyond that point. (That "sometimes" is why it was hard to track down. Most of the time, the file downloads fine from TiVo, but I caught it doing badly twice.)

When I run the same .tivo file through tivodecode-ng, it finds the same problem, but the recovery seems much better: the file are only missing 23K; while there's still a glitch in Quicktime player at that location (22:47), it can be played beyond that point and the file runs through ffmpeg without a problem.

There's a sample at this location: 
https://dl.dropboxusercontent.com/u/21507587/DailyShowBad.tivo
and a correctly downloaded version at 
https://dl.dropboxusercontent.com/u/21507587/DailyShowGood.tivo

I'll PM the MAK

Here's the debug output for 0.7.3:


> 10:48:18.175 [main] INFO c.s.tivolibre.TivoDecoder - dir = '/private/tmp/working', file = 'DailyShowBadDebug.mpg'
> 10:48:18.190 [main] DEBUG c.s.tivolibre.TivoDecoder - Header: TivoStreamHeader{ fileType=TiVo (54:69:56:6F), mpegOffset=0x4400, numChunks=3}
> 10:48:18.207 [main] DEBUG c.s.tivolibre.TivoDecoder - Chunk 0: TivoStreamChunk{chunkSize=1140, id=3, type=PLAINTEXT}
> 10:48:18.210 [main] DEBUG c.s.tivolibre.TivoDecoder - Chunk 1: TivoStreamChunk{chunkSize=7024, id=1, type=ENCRYPTED}
> 10:48:18.211 [main] DEBUG c.s.tivolibre.TivoDecoder - Chunk 2: TivoStreamChunk{chunkSize=7864, id=2, type=ENCRYPTED}
> 10:48:18.212 [main] DEBUG c.s.tivolibre.TivoDecoder - File format: TRANSPORT_STREAM
> 10:48:18.218 [main] DEBUG c.s.tivolibre.TivoDecoder - Starting TS processing at position 17408
> 10:48:18.220 [main] INFO c.s.tivolibre.TivoDecoder - Creating a new stream for PMT PID 0x0064
> 10:48:18.220 [main] DEBUG c.s.tivolibre.TivoDecoder - Creating a new VIDEO stream for PID 0x0f9a (0x1b)
> 10:48:18.220 [main] DEBUG c.s.tivolibre.TivoDecoder - Creating a new AUDIO stream for PID 0x0f9b (0x81)
> 10:48:18.220 [main] DEBUG c.s.tivolibre.TivoDecoder - Creating a new PRIVATE_DATA stream for PID 0x0f9d (0x97)
> 10:48:19.086 [main] DEBUG c.s.tivolibre.TivoDecoder - PacketId: 100,000 Type: AUDIO_VIDEO_PRIVATE_DATA PID: 0x0f9a Position after reading: 18,817,408
> ...
> 10:48:54.287 [main] DEBUG c.s.tivolibre.TivoDecoder - PacketId: 2,800,000 Type: AUDIO_VIDEO_PRIVATE_DATA PID: 0x0f9a Position after reading: 526,417,408
> 10:48:55.575 [main] DEBUG c.s.tivolibre.TivoDecoder - PacketId: 2,900,000 Type: AUDIO_VIDEO_PRIVATE_DATA PID: 0x0f9a Position after reading: 545,217,408
> 10:48:55.763 [main] WARN c.s.tivolibre.TivoDecoder - Invalid TS packet header for packet 2916787
> 10:48:55.763 [main] WARN c.s.tivolibre.TivoDecoder - TransportStream appears to be corrupt, cannot find sync bytes
> 10:48:55.765 [main] DEBUG c.s.tivolibre.TivoDecoder - Starting value for resumeDecryptionAtByte: 0x20af9850
> 10:48:55.767 [main] DEBUG c.s.tivolibre.TivoDecoder - Resume decryption at: 0x23400000
> 10:48:55.824 [main] INFO c.s.tivolibre.TivoDecoder - Re-synched at packet 2916788 (byte 0x20af9850)
> 10:48:55.857 [main] DEBUG c.s.tivolibre.TivoDecoder - PacketId: 3,000,000 Type: AUDIO_VIDEO_PRIVATE_DATA PID: 0x0f9a Position after reading: 564,040,156
> 10:48:55.893 [main] DEBUG c.s.tivolibre.TivoDecoder - PacketId: 3,100,000 Type: AUDIO_VIDEO_PRIVATE_DATA PID: 0x0f9a Position after reading: 582,840,156
> 10:48:55.907 [main] WARN c.s.tivolibre.TivoDecoder - Resuming decryption at 0x23400000, bytesWritten = 0x23400000
> 10:48:56.537 [main] DEBUG c.s.tivolibre.TivoDecoder - PacketId: 3,200,000 Type: AUDIO_VIDEO_PRIVATE_DATA PID: 0x0f9a Position after reading: 601,640,156
> 10:48:57.678 [main] DEBUG c.s.tivolibre.TivoDecoder - PacketId: 3,300,000 Type: AUDIO_VIDEO_PRIVATE_DATA PID: 0x0f9a Position after reading: 620,440,156
> ...


For your interest, here's the verbose output from tivodecode-ng:


> 548,373,176 : stream_id: bd, block_no: 1133 --FYI 0x20AF82B8
> loss_of_sync
> skipped 667 bytes, found a SYNC
> skipped 782 bytes, found a SYNC
> skipped 1633 bytes, found a SYNC
> skipped 1688 bytes, found a SYNC
> skipped 1927 bytes, found a SYNC
> skipped 3258 bytes, found a SYNC
> skipped 3385 bytes, found a SYNC
> skipped 3659 bytes, found a SYNC
> skipped 3746 bytes, found a SYNC
> skipped 4499 bytes, found a SYNC
> skipped 5004 bytes, found a SYNC
> skipped 5162 bytes, found a SYNC
> skipped 6007 bytes, found a SYNC
> skipped 6313 bytes, found a SYNC
> skipped 6456 bytes, found a SYNC
> skipped 6825 bytes, found a SYNC
> skipped 6889 bytes, found a SYNC
> skipped 7167 bytes, found a SYNC
> skipped 7503 bytes, found a SYNC
> skipped 7592 bytes, found a SYNC
> skipped 7783 bytes, found a SYNC
> skipped 7972 bytes, found a SYNC
> skipped 8438 bytes, found a SYNC
> skipped 8653 bytes, found a SYNC
> skipped 8939 bytes, found a SYNC
> skipped 8961 bytes, found a SYNC
> skipped 9154 bytes, found a SYNC
> skipped 9268 bytes, found a SYNC
> skipped 9316 bytes, found a SYNC
> skipped 9592 bytes, found a SYNC
> skipped 10057 bytes, found a SYNC
> skipped 10430 bytes, found a SYNC
> skipped 10682 bytes, found a SYNC
> skipped 10868 bytes, found a SYNC
> skipped 11003 bytes, found a SYNC
> skipped 11051 bytes, found a SYNC
> skipped 11371 bytes, found a SYNC
> skipped 11481 bytes, found a SYNC
> skipped 11507 bytes, found a SYNC
> skipped 11615 bytes, found a SYNC
> skipped 12121 bytes, found a SYNC
> skipped 12125 bytes, found a SYNC
> skipped 12222 bytes, found a SYNC
> skipped 12230 bytes, found a SYNC
> skipped 12862 bytes, found a SYNC
> skipped 12949 bytes, found a SYNC
> skipped 14278 bytes, found a SYNC
> skipped 14888 bytes, found a SYNC
> skipped 15562 bytes, found a SYNC
> skipped 16146 bytes, found a SYNC
> skipped 16434 bytes, found a SYNC
> skipped 16967 bytes, found a SYNC
> skipped 16992 bytes, found a SYNC
> skipped 17236 bytes, found a SYNC
> skipped 17353 bytes, found a SYNC
> skipped 17496 bytes, found a SYNC
> skipped 18423 bytes, found a SYNC
> skipped 19335 bytes, found a SYNC
> skipped 19567 bytes, found a SYNC
> skipped 19591 bytes, found a SYNC
> skipped 20133 bytes, found a SYNC
> skipped 20473 bytes, found a SYNC
> skipped 20521 bytes, found a SYNC
> skipped 20590 bytes, found a SYNC
> skipped 20763 bytes, found a SYNC
> skipped 21139 bytes, found a SYNC
> skipped 21296 bytes, found a SYNC
> skipped 21521 bytes, found a SYNC
> skipped 21567 bytes, found a SYNC
> skipped 21620 bytes, found a SYNC
> skipped 21627 bytes, found a SYNC
> skipped 22497 bytes, found a SYNC
> skipped 22587 bytes, found a SYNC
> skipped 22936 bytes, found a SYNC
> found 3 syncs in a row, loss_of_sync = 0
> globalBufferLen > TS_FRAME_SIZE, pulling packet from globalBuffer[]
> globalBufferLen > TS_FRAME_SIZE, pulling packet from globalBuffer[]
> 548,396,676 : stream_id: e0, block_no: 1133 --FYI 0x20AFDE84


----------



## fflewddur

Hi mackworth, thanks for reporting this and providing the sample files! I took a look at them tonight and compared the output of TivoLibre with TivoDesktop, and the output is binary identical between the two, both for the good and bad sample files. This is using the --compat-mode parameter with TivoLibre. Without --compat-mode, TivoLibre drops the 41 MB you mention, but this is by design; those packets are encrypted, and generally cause problems during playback. I'll investigate more this weekend to see if it's dropping some valid packets by mistake, but both files you sent me play back fine with VLC.

As an aside, QuickTime has problems playing both of them back, including the versions decoded with TivoDesktop; I haven't used QuickTime for video playback in years, so I'm not sure if this is a common problem anymore, but it used to have all sorts of issues playing .mpg files, particularly keeping the video and audio in sync.


----------



## mackworth

Thanks for looking into this. Although losing 40M isn't good, the main problem isn't QuickTime, it's that ffmpeg rejects the resulting file for further video processing. I'm curious; once you've found the sync again, why is it necessary to then skip to the "next position evenly divisible by the TS frame size"?


----------



## Dan203

Not trying to sell you anything here but have you tried running these files through VideoReDo Quick Stream Fix? It can probably fix up these bad files and get you something you can process with ffmpeg. (or you can process directly in VRD since it has a full set of transcoding capabilities)


----------



## mackworth

Sounds good; I look forward to trying out your Mac version!


----------



## Dan203

Didn't realize you were a Mac guy. (you spelled it wrong in your user name ) Although you could use BootCamp or Parallels.


----------



## mackworth

Well, having worked at Apple in the early 80's, I'm unlikely to change now . A little more seriously, I think you're correct; do you know any similar Mac programs to validate the streams?


----------



## Dan203

I am not aware of any, sorry. Based on the number of requests we get to port VideoReDo to Mac I'm not sure there are any. Unfortunately we're a small company (2 developers) so we just don't have the resources to port VRD to Mac.


----------



## fflewddur

mackworth said:


> I'm curious; once you've found the sync again, why is it necessary to then skip to the "next position evenly divisible by the TS frame size"?


After a sync loss, TiVo Desktop stops decrypting all of the packets until the next position in the byte stream that's evenly divisible by the TS frame size. To ensure binary compatibility, TivoLibre does the same thing. From my testing, dropping those packets completely seemed to improve the file's compatibility with ffmpeg and comskip (since they're still encrypted, they shouldn't have any bearing on the file's playback), but the file you provided shows that it doesn't always work that way.

I'm going to try testing a release that preserves these packets; if it works with the recent releases of ffmpeg and comskip, it should probably be the default behavior. In the meantime, you can pass the "--compat-mode" parameter to TivoLibre to prevent it from dropping these packets.


----------



## fflewddur

fflewddur said:


> I'm going to try testing a release that preserves these packets; if it works with the recent releases of ffmpeg and comskip, it should probably be the default behavior. In the meantime, you can pass the "--compat-mode" parameter to TivoLibre to prevent it from dropping these packets.


I tried this for the 1.0 release of Archivo, but the result was a slew of people reporting problems with audio/video synchronization. Reverting to dropping these packets in the 1.0.1 release solved all of those issues, so I think the smartest default behavior is to keep dropping the encrypted packets. Again, you can change this behavior by passing the --compat-mode argument to TivoLibre from the command-line.


----------



## fflewddur

Just a heads up that TivoLibre 0.7.4 is available at https://github.com/fflewddur/tivolibre/releases. This release fixes a rare decoding problem and, for anyone compiling from source, migrates to the Gradle build system. I also began releasing two JARs: tivo-libre.jar is intended for software developers to embed in their own projects, and doesn't include any dependencies. TivoDecoder.jar is an executable JAR that does include dependencies; it's intended for people who want to run TivoLibre directly from the command-line.


----------



## NMUser

Hello fflewddur,

I am having an issue running tivolibre 0.7.4. All of my programs download from the TiVo, but some of them do not decode correctly. When bringing in programs from CNBC I get the following debug output:

java -jar TivoDecoder.jar -d -i Fast* -o eric.mpg -m XXXXXXXXXX dir = 'null', file = 'eric.mpg'
Header: TivoStreamHeader{ fileType=TiVo (54:69:56:6F), mpegOffset=0x3400, numChunks=3}
Chunk 0: TivoStreamChunk{chunkSize=1140, id=3, type=PLAINTEXT}
Chunk 1: TivoStreamChunk{chunkSize=4768, id=1, type=ENCRYPTED}
Chunk 2: TivoStreamChunk{chunkSize=5640, id=2, type=ENCRYPTED}
File format: TRANSPORT_STREAM
Starting TS processing at position 13312
Creating a new stream for PMT PID 0x0064
Creating a new VIDEO stream for PID 0x0011 (type=0x1b)
Creating a new AUDIO stream for PID 0x0014 (type=0x81)
Creating a new PRIVATE_DATA stream for PID 0x0015 (type=0x97)
Unsupported packet type: RESERVED2
Closing CountingDataInputStream. Final read position: 13876


On the programs to decode correctly the output looks like:
java -jar TivoDecoder.jar -d -i The* -o eric.mpg -m XXXXXXXXXX dir = 'null', file = 'eric.mpg'
Header: TivoStreamHeader{ fileType=TiVo (54:69:56:6F), mpegOffset=0x5400, numChunks=3}
Chunk 0: TivoStreamChunk{chunkSize=1140, id=3, type=PLAINTEXT}
Chunk 1: TivoStreamChunk{chunkSize=9012, id=1, type=ENCRYPTED}
Chunk 2: TivoStreamChunk{chunkSize=9888, id=2, type=ENCRYPTED}
File format: TRANSPORT_STREAM
Starting TS processing at position 21504
Creating a new stream for PMT PID 0x0064
Creating a new VIDEO stream for PID 0x1291 (type=0x02)
Creating a new AUDIO stream for PID 0x1294 (type=0x81)
Creating a new PRIVATE_DATA stream for PID 0x1296 (type=0x97)
PacketId: 100,000 Type: AUDIO_VIDEO_PRIVATE_DATA PID: 0x1291 Position after read
ing: 18,821,504
PacketId: 200,000 Type: AUDIO_VIDEO_PRIVATE_DATA PID: 0x1291 Position after read
ing: 37,621,504
PacketId: 300,000 Type: AUDIO_VIDEO_PRIVATE_DATA PID: 0x1291 Position after read
ing: 56,421,504
PacketId: 400,000 Type: AUDIO_VIDEO_PRIVATE_DATA PID: 0x1291 Position after read
ing: 75,221,504
.
.
.

Please let me know if you need more info

Thanks


----------

