# Downloading to computer, files corrupted



## Roghet (Sep 19, 2017)

When I download shows from my Roamio to my computer, they get corrupted. In playing the shows (doesn't matter what player), there are places where the file hangs, and segments (minutes) can't play. And it will only download the video file in .ts format. If I set it to transcode to .mpg, (which I would rather have) it only downloads the audio.

I've tried to install older versions, because they have worked fine, the the program says "required tools not detected" but it seems that it's looking for files that don't exist.

Lil' help puhlease?


----------



## kpeters59 (Jun 19, 2007)

Do you mind telling us what software you're using to download files from your Roamio?

-KP


----------



## ThAbtO (Apr 6, 2000)

The show may be in h.264 if you can only download in TS format, and audio only in PS format. But if you only get audio while in TS, then try PS.

What are you using to download the shows? 

If you use KMTTG, you have the option to turn on/off TS downloads.

Also, we need more detailed information, or else all you may get is generic info which may not apply.


----------



## Roghet (Sep 19, 2017)

kpeters59 said:


> Do you mind telling us what software you're using to download files from your Roamio?
> 
> -KP


KMTTG


----------



## Roghet (Sep 19, 2017)

ThAbtO said:


> The show may be in h.264 if you can only download in TS format, and audio only in PS format. But if you only get audio while in TS, then try PS.
> 
> What are you using to download the shows?
> 
> ...


If I turn off TS, that's when it only downloads audio.


----------



## ThAbtO (Apr 6, 2000)

How much of the show were you able get, in show time wise?
Usually transfers stop when there are glitches in the download data.


----------



## Roghet (Sep 19, 2017)

ThAbtO said:


> The show may be in h.264 if you can only download in TS format, and audio only in PS format. But if you only get audio while in TS, then try PS.
> 
> What are you using to download the shows?
> 
> ...


----------



## Roghet (Sep 19, 2017)

Other info, I have a PC. It occurred to me that the files may have stopped downloading properly with a java update. Maybe there is a conflict, or I should try to reinstall java?

Please tell me what detailed info you need. And thanks in advance for your time!


----------



## ThAbtO (Apr 6, 2000)

ThAbtO said:


> How much of the show were you able get, in show time wise?
> Usually transfers stop when there are glitches in the download data.


----------



## Roghet (Sep 19, 2017)

I am able to record the whole show. however, when I play it (let's say is a 2 hr movie) there are 6 places where it hangs. And if I can get it to skip ahead, there are up to 5 minute that just aren't there. But, the movie plays fine on the TiVo.


----------



## ThAbtO (Apr 6, 2000)

That sounds like glitches (or hangups) in either the video or audio and most of the time glitches stop the download, either you can see the glitch or not.

If you want to download more of the show, you would need to locate where it stopped, pause the show (on the Tivo) at that location, bypass by a few frames (paused and press >> a couple of times), then in KMTTG, refresh your show list, under file menu, there is "Resume Download." You should be able to download more of it, but remember to rename the show's filename or it will be overwritten. You may need to repeat if there are more. (Pause Tivo at next location, refresh KMTTG show list, rename file, restart download)


----------



## kpeters59 (Jun 19, 2007)

It's fairly well known that sometimes TiVo recordings get a 'glitch' in the recording (that, for whatever reason, seems to often occur at commercial transitions). During normal playback, they mostly go unnoticed. But if you try to transfer them, they cause the transfer to fail. KMTTG has a feature to 'resume' download from the current 'paused' position. So you find the glitch, pause right after it and then re-start the transfer.

If that's what's going on with yours, you'd be able to skip up on the TiVo to the spot where the transfer had failed and see the glitch. 

Did I say that right?

-KP


----------



## Roghet (Sep 19, 2017)

I think I understand you. But I'm unsure if you understand me. It isn't hanging on the download. it's hanging when I try to play the downloaded file. And again, the whole movie downloads, however there are a number of glitches. (And if you are correct, I have a large number of programs and this becomes extremely time prohibitive.) And one other detail, I have had a Tivo for many years (this Roamio is my second) and up to a certain point every single program downloaded flawlessly. Now, NONE of them download without a glitch. This is why I asked about trying an older version of KMTTG and would very much appreciate help with that option.


----------



## ThAbtO (Apr 6, 2000)

Older versions of KMTTG does not work any more.


----------



## ThAbtO (Apr 6, 2000)

How much of the show were you actually able to download? 

I really mean actual, not "let's say 2 min" etc.


----------



## Roghet (Sep 19, 2017)

ThAbtO said:


> How much of the show were you actually able to download?
> 
> I really mean actual, not "let's say 2
> 
> ...


----------



## Roghet (Sep 19, 2017)

ThAbtO said:


> How much of the show were you actually able to download?
> 
> I really mean actual, not "let's say 2 min" etc.


Something went wrong in the reply. Again, I did not say "2 mins" I said 2 hrs. It can be two hours or 1 hour, it's irrelevant. The whole program downloads, but there are a number of glitches throughout. Always.


----------



## Roghet (Sep 19, 2017)

I'm going to try downloading as a .tivo file and see if that gets corrupted too. Will check back tomorrow. And thanks again


----------



## ThAbtO (Apr 6, 2000)

If you can download the entire show, what happens when you try to view it? And what player? Codec?


----------



## ThAbtO (Apr 6, 2000)

Roghet said:


> I'm going to try downloading as a .tivo file and see if that gets corrupted too. Will check back tomorrow. And thanks again


The download file is a .TiVo file, which is an encoded MPEG2 or TS.


----------



## Roghet (Sep 19, 2017)

I'm going to try downloading as a .tivo file and see if that gets corrupted too. Will check back tomorrow. And thanks again


ThAbtO said:


> The download file is a .TiVo file, which is an encoded MPEG2 or TS.


I'm sorry, but I don't know how to be any more clear. Please reread what i wrote. The programs fail to play at the point of file corruption in every player I have VLQ, Plex, etc.. They fail to play correctly because the file is bad. I don't know if the .tivo file is dowloading without flaw or not yet, (I need time to watch it fully) and to be useful to me it would have to be transcoded. The only format that works at all is the .TS. For 5 years I have had KMTTG transcode to .MPEG and it has worked fine. With the latest version of KMTTG, somehow the file is being partially corrupted. I have a fellow helping me that has worked in video for 30+ years. He can't figure out what the issue is either, so that is why I'm trying this forum to see if there are other ideas.


----------



## HerronScott (Jan 1, 2002)

Other users have seen this issue with Transport Stream downloads which unfortunately you need if you on Comcast and they have moved to MPEG4 in your area (for everything but local broadcast).

Take a look at pyTivo Desktop as I understand that it has an option to retry the download when it detects these glitches

Scott


----------



## Roghet (Sep 19, 2017)

Thanks Scott, good idea. I am on Comcast. 

I've never used pyTivo. Any chance I can ditch KMTTG and use that? If you know right where I can find info to install and set up, I'd be very appreciative.


----------



## dlfl (Jul 6, 2006)

Roghet said:


> Thanks Scott, good idea. I am on Comcast.
> 
> I've never used pyTivo. Any chance I can ditch KMTTG and use that? If you know right where I can find info to install and set up, I'd be very appreciative.


Here: Easier to use pyTivo

You don't need to "ditch" kmttg to use pyTivo Desktop.


----------



## mattack (Apr 9, 2001)

Actually, these don't sound like video glitches like others have suspected.

Even though in the past I swear I tried them and they wouldn't work at all, recently again I tried to play TS format downloads in VLC (on iPad), and I too ran into segments where it would just jump forward or had glitches. These were recordings that did NOT have problems on the original Tivo. (I was also able to play them in Quicktime Player, and get audio & video.. I haven't yet looked around long enough to see if I got glitches in the exact same places as in VLC.) Oh, and VLC has updated the iPad app, so my previous failures were with older versions.

These are of course the ones that only work downloaded in TS format. For the ones that work in non-TS format, I do that (since I know the regular mpeg2 files work fine in VLC)


----------



## Roghet (Sep 19, 2017)

mattack said:


> Actually, these don't sound like video glitches like others have suspected.
> 
> Even though in the past I swear I tried them and they wouldn't work at all, recently again I tried to play TS format downloads in VLC (on iPad), and I too ran into segments where it would just jump forward or had glitches. These were recordings that did NOT have problems on the original Tivo. (I was also able to play them in Quicktime Player, and get audio & video.. I haven't yet looked around long enough to see if I got glitches in the exact same places as in VLC.) Oh, and VLC has updated the iPad app, so my previous failures were with older versions.
> 
> These are of course the ones that only work downloaded in TS format. For the ones that work in non-TS format, I do that (since I know the regular mpeg2 files work fine in VLC)


This sounds like the closest understanding to what I am experiencing. I still have yet to disprove that the problem isn't the transcoding from .Tivo to .ts. It worked flawless when I could go to .mpeg previously.


----------



## Roghet (Sep 19, 2017)

mattack said:


> Actually, these don't sound like video glitches like others have suspected.
> 
> Even though in the past I swear I tried them and they wouldn't work at all, recently again I tried to play TS format downloads in VLC (on iPad), and I too ran into segments where it would just jump forward or had glitches. These were recordings that did NOT have problems on the original Tivo. (I was also able to play them in Quicktime Player, and get audio & video.. I haven't yet looked around long enough to see if I got glitches in the exact same places as in VLC.) Oh, and VLC has updated the iPad app, so my previous failures were with older versions.
> 
> These are of course the ones that only work downloaded in TS format. For the ones that work in non-TS format, I do that (since I know the regular mpeg2 files work fine in VLC)


This sounds like the closest understanding to what is going on with me. I still have yet to disprove that it isn't the transcoding from .tivo to .ts. (I plan to take it to a pro to use his software.) I have no explanation for what changed from when I was able to go from .tivo to .mpeg...


----------



## Roghet (Sep 19, 2017)

What other programs (besides KMTTG) are there to download programs off the TiVo and transcode then to .mpeg? Both free and paid options would be appreciated.


----------



## ThAbtO (Apr 6, 2000)

I doubt other programs will have a different result than what you had experienced.


----------



## HerronScott (Jan 1, 2002)

Roghet said:


> What other programs (besides KMTTG) are there to download programs off the TiVo and transcode then to .mpeg? Both free and paid options would be appreciated.


Archivo is another

Archivo

But I'm still guessing this is related to the issues with transport stream downloads and that you should try and verify with downloads using pyTivo Desktop to see if it is seeing the issues during the download and tries to download again (if you have it configured for that).

Scott


----------



## mattack (Apr 9, 2001)

Roghet said:


> What other programs (besides KMTTG) are there to download programs off the TiVo and transcode then to .mpeg? Both free and paid options would be appreciated.


AFAIK, the Tivo itself is giving you the transport stream data.. So I think TS file you get will be the same out of any program..
then in theory you could use things like ffmpeg, etc., to convert.. But I suspect the results will be the same.

I thought when I last looked at it that transport stream (hence the name) isn't really intended to be played directly.


----------



## lpwcomp (May 6, 2002)

The TS glitches are being created by the TiVo, not the PC.


----------



## Roghet (Sep 19, 2017)

lpwcomp said:


> The TS glitches are being created by the TiVo, not the PC.


So, there is no fix and something is wrong with the TiVo?

Is this why I can't get it to download as an .mpeg anymore?


----------



## Dan203 (Apr 17, 2000)

Roghet said:


> So, there is no fix and something is wrong with the TiVo?
> 
> Is this why I can't get it to download as an .mpeg anymore?


PyTivo has a retry option that could potentially get a clean copy if you set it high enough. Sometimes if you download the same file enough times you can get one without a glitch.

.mpg files still work, but only for MPEG-2 channels. If your cable company uses MPEG-4 then ts is the only option.


----------



## mattack (Apr 9, 2001)

Dan203 said:


> .mpg files still work, but only for MPEG-2 channels. If your cable company uses MPEG-4 then ts is the only option.


To clarify, for at least some of us (maybe everybody?), we have a combination of mpeg-2 and mpeg-4 channels. For me, as a guess, the broadcast channels (HD and SD downconversions) are mpeg-2, as well as many if not all SD variants of other channels. The "extended basic" type of HD channels are what seem to be mpeg-4 for me now.

Many of the things I want to download, I purposely still record in SD, since I want to either save them or copy them to my iPad, and the smaller file sizes are useful.

But conversely, the mpeg-4 channels now are SO much smaller than the mpeg-2 HD versions were (and still are for the broadcast channels), that an hour is around a gig in mpeg 4.. But ironically now the TS files have these glitches..

sigh..


----------



## ThAbtO (Apr 6, 2000)

mattack said:


> To clarify, for at least some of us (maybe everybody?), we have a combination of mpeg-2 and mpeg-4 channels. For me, as a guess, the broadcast channels (HD and SD downconversions) are mpeg-2, as well as many if not all SD variants of other channels. The "extended basic" type of HD channels are what seem to be mpeg-4 for me now.
> 
> Many of the things I want to download, I purposely still record in SD, since I want to either save them or copy them to my iPad, and the smaller file sizes are useful.
> 
> ...


I think the MPEG4/h2.64 would come from mostly the SDV channels.


----------



## mattack (Apr 9, 2001)

Maybe for others, not for me, I don't have any SDV channels.


----------



## Dan203 (Apr 17, 2000)

There isn't really a standard for storing H.264 data in a program stream, which is why TiVo only allows them in TS. In fact the reason they added the TS format at all was because they needed to support H.264 in New Zealand back in the S3 days.

What's worse is TS files didn't always have glitches. They introduced the bug somewhere after the Premiere was released and then never fixed it.


----------



## HerronScott (Jan 1, 2002)

ThAbtO said:


> think the MPEG4/h2.64 would come from mostly the SDV channels.


No, Comcast doesn't use SDV.

Scott


----------



## Diana Collins (Aug 21, 2002)

FiOS has had a number of MPEG-4 channels for years (but most are MPEG-2). I have used KMTTG to JUST download (I don't even decrypt in KMTTG). I then use VideoReDo to decrypt, transcode and edit out commercials (where applicable). I have dozens of MPEG-4 downloads that all downloaded and playback perfectly. I never knew some people have problems with MPEG-4 channels.


----------



## lpwcomp (May 6, 2002)

Diana Collins said:


> FiOS has had a number of MPEG-4 channels for years (but most are MPEG-2). I have used KMTTG to JUST download (I don't even decrypt in KMTTG). I then use VideoReDo to decrypt, transcode and edit out commercials (where applicable). I have dozens of MPEG-4 downloads that all downloaded and playback perfectly. I never knew some people have problems with MPEG-4 channels.


I just configure kmttg to use VRD to decrypt. And I quite often have to do multiple TS downloads of the same recording to get a decrypted file of a length even close to correct and there are some recordings that I am never able to get a good download.


----------



## ThAbtO (Apr 6, 2000)

lpwcomp said:


> I just configure kmttg to use VRD to decrypt. And I quite often have to do multiple TS downloads of the same recording to get a decrypted file of a length even close to correct and there are some recordings that I am never able to get a good download.


I had to download the finale of America's Got Talent about 5 times before it finally finished. Size was about 10.5 GB, which took about 40 min. Roamio was connected to my 5G while my PC was only able to connect on 2.4.


----------



## lpwcomp (May 6, 2002)

ThAbtO said:


> I had to download the finale of America's Got Talent about 5 times before it finally finished. Size was about 10.5 GB, which took about 40 min. Roamio was connected to my 5G while my PC was only able to connect on 2.4.


I'm hardwired Ethernet.


----------



## Roghet (Sep 19, 2017)

Diana Collins said:


> FiOS has had a number of MPEG-4 channels for years (but most are MPEG-2). I have used KMTTG to JUST download (I don't even decrypt in KMTTG). I then use VideoReDo to decrypt, transcode and edit out commercials (where applicable). I have dozens of MPEG-4 downloads that all downloaded and playback perfectly. I never knew some people have problems with MPEG-4 channels.


Thanks Diana, That was my next question. What 3rd party software will transcode .tivo files? The first one to show up on a search is EaseFab Video Converter. I'd love some user experiences and suggestions, please.


----------



## ThAbtO (Apr 6, 2000)

Its not transcode, but if you use KMTTG, you can decrypt with TivoLibre on the MP4/TS files. TivoDecode doesn't work well on TS.

Transcode is when the video is coding to or from the Tivo. Sort of like Translation + coding.


----------



## HerronScott (Jan 1, 2002)

Roghet said:


> Thanks Diana, That was my next question. What 3rd party software will transcode .tivo files? The first one to show up on a search is EaseFab Video Converter. I'd love some user experiences and suggestions, please


I use the same program that Diana uses which is VideoRedo.

Scott


----------



## ClearToLand (Jul 10, 2001)

Roghet said:


> *When I download shows from my Roamio to my computer, they get corrupted*. In playing the shows (doesn't matter what player), there are places where the file hangs, and segments (minutes) can't play. And it will only download the video file in .ts format. If I set it to transcode to .mpg, (which I would rather have) it only downloads the audio.
> 
> I've tried to install older versions, because they have worked fine, the the program says "required tools not detected" but it seems that it's looking for files that don't exist.
> 
> Lil' help puhlease?





Roghet said:


> Thanks Diana, That was my next question. *What 3rd party software will transcode .tivo files?* The first one to show up on a search is EaseFab Video Converter. I'd love some user experiences and suggestions, please.


I don't know if this is a taboo topic and my post will be deleted, or if folks honestly don't know / realize what's happening. But if YOU are interested in what's going on here, SEARCH TCF for posts from me, back around May / June 2017 with the words: *"TS 'Fast' Format Transfers" "PS 'Slow'Format Transfers"*

This 'problem' has existed for quite some time now and folks, like you, who don't SEARCH because they think they're the FIRST to discover it, become very frustrated, pointing the blame at the 'PC Download Program' (i.e. TiVo Desktop, kmttg, etc...) when the problem is TOTALLY originating from the TiVo unit itself.

*TL;DR* (regarding SEARCHing TCF)

During my first experience with kmttg / pyTiVo from @wmcbrine (Spring 2016), GT 50% of the files I offloaded to my PC to view later could not be viewed (Summer 2016). 
- I initially used the TS 'Fast' Format Transfer protocol in kmttg, since the majority of the posts I had read suggested it.
.
*TS 'Fast' Format Transfer protocol* has a HIGH probability (IMO) of introducing '*Sync Errors*' into the transfer.

You can address this problem either with:
.
*VideoRedo* ($$$) to edit out the 'Sync Errors' automatically in the kmttg task stream
- (Download / Metadata / Decrypt / QS Fix (Sync Errors) / Ad Detect / Ad Cut / Captions / Encode)
.
*pyTiVo Desktop* (FREE - from VideoReDo programmer @Dan203) to keep retrying the transfer until it's either error-free (preferred) or has a minimum number of errors (eh). This could take 10, 20, 30 or more retries! 
- *NOTE:* By downloading the available Python SOURCE code for pyTiVo Desktop (Thanks @Dan203!) and adding my own 'Debug / Display' statements, I was able to learn (Python and) that these 'Sync Errors' move around. You may get many near the end; next time you may get many near the beginning; next time you may get one; no rhyme or reason - it's just the way the TiVo unit (NOT the PC!) works. 
.
*PS 'Slow' Format Transfer protocol* has a HIGH probability of introducing 'Closed Caption' Errors into the transfer.
There's really nothing you can do about this *AND* you can't use PS 'Slow' Format Transfer protocol if the channel is using H.264 encoding FORCING you to use TS 'Fast' Format Transfer protocol and deal with 'Sync Errors'.
*Bottom Line: *If you don't want to be bothered dealing with 'Sync Errors', buy VideoReDo and add it to you kmttg task stream. That's what the author of kmttg, @moyekj, does.


----------



## lpwcomp (May 6, 2002)

Using VRD does not solve all of the problems. Yes, it gets rid of the sync errors but it often does this by removing bad frames which can result in loss of content.


----------



## lpwcomp (May 6, 2002)

ThAbtO said:


> Transcode is when the video is coding to or from the Tivo. Sort of like Translation + coding.


That is false. Transcoding is recoding to a different a/v coding, such as MPEG2 to H.264. There is no transcoding at all on a TiVo->PC transfer and not necessarily in the other direction.


----------



## Roghet (Sep 19, 2017)

And what is also false is placing blame on a person saying that they did not "SEARCH." I did. Nothing came up that helped me. Part of "the problem" is people "like (me)" don't always know the right keywords, or the right questions to ask, or terminology. I'm quite confident that as we each go thru our process of learning at our various levels, it is of very little help to be are called "b***Y little girls" by those that think so very highly of themselves. And it must be said that some of what has been offered, disagreed. What is helpful, is an understanding of *ontological humility.*


----------



## lpwcomp (May 6, 2002)

Roghet said:


> And what is also false is placing blame on a person saying that they did not "SEARCH." I did. Nothing came up that helped me. Part of "the problem" is people "like (me)" don't always know the right keywords, or the right questions to ask, or terminology. I'm quite confident that as we each go thru our process of learning at our various levels, it is of very little help to be are called "b***Y little girls" by those that think so very highly of themselves. And it must be said that some of what has been offered, disagreed. What is helpful, is an understanding of *ontological humility.*


I assume that since you quoted my sig line, at least part of your ire is aimed in my direction. I have never told anyone they should search this forum and wouldn't. As far as my sig line goes , *it's a joke! *I take it you never saw "Burn Notice".


----------



## Roghet (Sep 19, 2017)

lpwcomp said:


> I assume that since you quoted my sig line, at least part of your ire is aimed in my direction. I have never told anyone they should search this forum and wouldn't. As far as my sig line goes , *it's a joke! *I take it you never saw "Burn Notice".


 I searched a number of places before I came to this particular forum. I am not aware of "TCF," and no, I have never watched "Burn Notice." I am a college professor. And one of the first things that is helpful to know in teaching another is to know one's audience, and it is always a good practice to assume the person asking a question knows nothing. I am open to any and all accurate information, so if you have any, please continue. Let's not let the thread dwindle and get off track please. In a few suggestions, there is enough disagreement that I'm not sure which option to try first, and the Java info is over my pay grade!


----------



## lpwcomp (May 6, 2002)

TCF is "TiVo Community Forum", IOW, this place.


----------



## dlfl (Jul 6, 2006)

Roghet said:


> And what is also false is placing blame on a person saying that they did not "SEARCH." I did. Nothing came up that helped me. Part of "the problem" is people "like (me)" don't always know the right keywords, or the right questions to ask, or terminology. I'm quite confident that as we each go thru our process of learning at our various levels, it is of very little help to be are called "b***Y little girls" by those that think so very highly of themselves. And it must be said that some of what has been offered, disagreed. What is helpful, is an understanding of *ontological humility.*


Get off your high horse ... we don't need no stinking "ontological humility". We always know the one and only truth, even when we mutually disagree! I think it takes a lot of hubris to use a term like that in mixed company. 

Seriously, try being a little less brittle and you will get more value from the TCF. There are a lot of posters here who make a real effort to help you, including already in this thread -- and some who are not so helpful in various ways.

To address your question, I think @ClearToLand in post #47 gave a pretty comprehensive answer, despite his tone apparently irritating you.


----------



## mattack (Apr 9, 2001)

ClearToLand said:


> *pyTiVo Desktop* (FREE - from VideoReDo programmer @Dan203) to keep retrying the transfer until it's either error-free (preferred) or has a minimum number of errors (eh). This could take 10, 20, 30 or more retries!
> ...
> 
> *PS 'Slow' Format Transfer protocol* has a HIGH probability of introducing 'Closed Caption' Errors into the transfer.
> ...


ok, I see that there is pytivo desktop for the mac.. Just to be absolutely clear, this error free retrying stuff does NOT have any dependency on VideoReDo (which depends on Windows stuff), right?

Interesting, I have seen various closed captioning errors over the years, I had thought that was just a fluke, didn't realize it was actually the tivo causing it..

Though I sometimes have seen the closed captions screwed up EVEN WHEN DOWNLOADING TO A TIVO (i.e. via the iOS app). I guess I need to absolutely pay attention to the next time I see that, then write up a bug _on their site_, since this is an accessibility issue.


----------



## ClearToLand (Jul 10, 2001)

mattack said:


> ok, I see that there is pytivo desktop for the mac.. Just to be absolutely clear, *this error free retrying stuff does NOT have any dependency on VideoReDo* (which depends on Windows stuff), right?


Yep! :thumbsup:

*BUT*, I suggest that you start out with a short (30 min) H.264 recording and set:
togo_ts_error_mode = best
togo_ts_max_retries = 30 (or higher)
 and then sit back and watch what happens.

*NOTE:* Starting the transfer (in @Dan203's pyTiVo version) and then coming back when it's done is NOT going to 'teach / show' you anything interesting (i.e. it'll either be SUCCESS or FAILURE). I had to run the program via IDLE [Python Editor; included when you install Python on a Windows PC - I know NOTHING (Sgt Schultz!  ) about MACs] and even then I had to sit down for 24+ hours straight, learn some fundamental Python coding and add some 'Debug / Display' statements to @Dan203's code in order to see how the 'TS Sync' errors moved around.  {I've been teaching myself various coding languages since the late '70s when I built a Netronics Super Elf II computer kit [RCA COSMAC 1802 CPU (I discovered a decade later that it had the same register structure as an IBM 360!), 256 *BYTES* of static RAM, hex keypad, six 7-segment red LED displays for Address Bus (4) and Data Bus (2)].}

NOTES from @Dan203 that I saved in a .TXT file:

```
pyTivo_conf_Options.Txt - 20170523 @ 22:47
            @Dan203 - 2017/05/27 @ 01:42

TS SYNC LOSS:
=============
In browser:

Select Checkbox "Transfer as mpeg-ts"

In pyTivo.conf:

togo_ts_error_mode = ignore [best | reject]
togo_ts_max_retries = 5 [any positive number]

It has 3 options...

   1) IGNORE Errors - which works just like before
   2) REJECT Errors - which will abort the download as soon as an error is detected
   3) BEST File - which will keep the file with the least number of errors detected 

#2 and #3 have a user settable retry count setting which determines the number of times pyTivo (Desktop) will retry the download before giving up. If a clean file is downloaded before the retry count is hit, then it move on to the next file in the queue automatically.

ERROR REPORT:
=============

I believe that option #1 skips the check completely, so there is no record. But with option #3 the number of affected packets is appended to the file name. It will appear like:

   Show Name - ''Episode Title'' (Jan 01, 2017 KRST) (^56).tivo

The "^56" part is the number of affected packets, but it's cumulative, there is no record of exactly where the affected packets are or how many different sections are affected. I was thinking about adding an option to write that data out to a text file. I don't really have a way of reporting it as a time stamp though, only a byte offset. Although you can get a rough idea of where the error is by using the total file size and dividing it by the length of the recording.


ACCESSING PYTIVO DESKTOP FROM BROWSER:
======================================

http://localhost:port/Desktop

Error connecting to pyTivo!
Verify pyTivo is running and retry
```



mattack said:


> ...Interesting, I have seen various closed captioning errors over the years, I had thought that was just a fluke, didn't realize it was actually the tivo causing it..
> 
> Though I sometimes have seen the closed captions screwed up EVEN WHEN DOWNLOADING TO A TIVO (i.e. via the iOS app). I guess I need to absolutely pay attention to the next time I see that, then write up a bug _on their site_, since this is an accessibility issue.


There's a *LOT* going on here in TiVoLand that seems to only be mentioned '_here-and-there_' in random posts that you just have to have OCD to find.


----------



## Dan203 (Apr 17, 2000)

Man 30 retries is a lot. I’m not sure I have the patience for that. Although I guess if you’re queuing them up and just letting it run in the background it wouldn’t be that bad.


----------



## ClearToLand (Jul 10, 2001)

Dan203 said:


> *Man 30 retries is a lot*. I'm not sure I have the patience for that. Although I guess if you're queuing them up and just letting it run in the background it wouldn't be that bad.


After (unsuccessfully) bugging you to add additional 'Debug / Display' type statements  , I finally sat down and did it myself.  And it was certainly an 'EYE-OPENING' experience to say the least! 

These 'TS Sync' errors MOVE AROUND [Other TCF members (excluding @Dan203, of course; he already knew this) just look at the 'Minutes' below]!!! 

It was back in June 2017 that I had this marathon Python coding / learning experience. [IIRC, you won't see the 'Aborted' messages in @Dan203's log.]

Here, for everyone interested's viewing pleasure / entertainment, is one of my (work-in-progress / not-ready-for-prime-time) log files:

```
---SNIP--- (TCF limit of 10,000 characters per post)
Tivo.togo:[14/Jun/2017 01:27:08] Queued "http://192.168.0.141:80/download/Moms' Night Out.TiVo?Container=/NowPlaying&id=108946" for transfer to L:\TiVo\PyTiVoDesktop\Videos
INFO:pyTivo.togo:PC sleep has been disabled
INFO:pyTivo.togo:[14/Jun/2017 01:27:09] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 01:32:53] Done getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A, 2387096168 bytes, 52.92 Mb/s, 390 TS errors in Retry 0 taking 5.74 Minutes
INFO:pyTivo.togo:[14/Jun/2017 01:32:59] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 01:38:18] Done getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A, 2387096168 bytes, 57.13 Mb/s, 177 TS errors in Retry 1 taking 5.31 Minutes
INFO:pyTivo.togo:[14/Jun/2017 01:38:25] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 01:43:48] Done getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A, 2387096168 bytes, 56.35 Mb/s, 69 TS errors in Retry 2 taking 5.39 Minutes
INFO:pyTivo.togo:[14/Jun/2017 01:43:55] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 01:46:32] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A aborted with 101 TS errors in Retry 3 taking 2.61 Minutes
INFO:pyTivo.togo:[14/Jun/2017 01:46:33] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 01:49:59] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A aborted with 136 TS errors in Retry 4 taking 3.42 Minutes
INFO:pyTivo.togo:[14/Jun/2017 01:50:01] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 01:52:44] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A aborted with 102 TS errors in Retry 5 taking 2.72 Minutes
INFO:pyTivo.togo:[14/Jun/2017 01:52:46] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 01:56:11] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A aborted with 103 TS errors in Retry 6 taking 3.41 Minutes
INFO:pyTivo.togo:[14/Jun/2017 01:56:12] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 02:01:35] Done getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A, 2387096168 bytes, 56.45 Mb/s, 11 TS errors in Retry 7 taking 5.38 Minutes
INFO:pyTivo.togo:[14/Jun/2017 02:01:42] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 02:02:23] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A aborted with 31 TS errors in Retry 8 taking 0.69 Minutes
INFO:pyTivo.togo:[14/Jun/2017 02:02:25] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 02:03:48] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A aborted with 69 TS errors in Retry 9 taking 1.39 Minutes
INFO:pyTivo.togo:[14/Jun/2017 02:03:50] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 02:04:32] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A aborted with 31 TS errors in Retry 10 taking 0.71 Minutes
INFO:pyTivo.togo:[14/Jun/2017 02:04:34] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 02:09:16] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A aborted with 84 TS errors in Retry 11 taking 4.71 Minutes
INFO:pyTivo.togo:[14/Jun/2017 02:09:18] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 02:10:40] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A aborted with 69 TS errors in Retry 12 taking 1.37 Minutes
INFO:pyTivo.togo:[14/Jun/2017 02:10:42] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 02:15:24] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A aborted with 82 TS errors in Retry 13 taking 4.70 Minutes
INFO:pyTivo.togo:[14/Jun/2017 02:15:25] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 02:16:49] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A aborted with 69 TS errors in Retry 14 taking 1.39 Minutes
INFO:pyTivo.togo:[14/Jun/2017 02:16:51] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 02:21:43] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A aborted with 82 TS errors in Retry 15 taking 4.87 Minutes
INFO:pyTivo.togo:[14/Jun/2017 02:21:44] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 02:22:26] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A aborted with 31 TS errors in Retry 16 taking 0.69 Minutes
INFO:pyTivo.togo:[14/Jun/2017 02:22:27] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 02:25:12] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A aborted with 102 TS errors in Retry 17 taking 2.74 Minutes
INFO:pyTivo.togo:[14/Jun/2017 02:25:13] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 02:26:39] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A aborted with 69 TS errors in Retry 18 taking 1.42 Minutes
INFO:pyTivo.togo:[14/Jun/2017 02:26:40] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 02:27:22] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A aborted with 30 TS errors in Retry 19 taking 0.69 Minutes
INFO:pyTivo.togo:[14/Jun/2017 02:27:23] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A
INFO:pyTivo.togo:[14/Jun/2017 02:30:56] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Moms' Night Out - 2017-06-13 (FXMOVIEHD) (TS).tivo" from Roamio A aborted with 105 TS errors in Retry 20 taking 3.54 Minutes
INFO:pyTivo.togo:PC sleep has been enabled
```


----------



## Dan203 (Apr 17, 2000)

I've been meaning to add a report mode that would list more specific info about where the errors are specifically. Would be kind of cool if you could use multiple files to stitch together a clean copy.


----------



## ClearToLand (Jul 10, 2001)

Dan203 said:


> I've been meaning to add a report mode that would list more specific info about where the errors are specifically. *Would be kind of cool if you could use multiple files to stitch together a clean copy*.


Since they 'jump around' so much, YEAH! :handok:

If the first error didn't occur until 5.38 minutes into the file, on subsequent passes, could we IGNORE any errors before 5.38 minutes (and also ignore useless writing to HDD) and begin writing anew @ 5.38 minutes? Label each subsequent SUCCESSFUL transfer like a multi-file .rar file and then combine all the parts at the end...


----------



## ClearToLand (Jul 10, 2001)

[Another idea...]

If the 2nd pass gets past 5.38 minutes without error, REPLACE the first file segment with this file segment. And so on...

If it doesn't, discard everything before 5.38 minutes and begin writing a new second file segment.

It's been a LONG TIME since I coded something intricate, but, it seems like a table would be handy here - keeping track of the:
Segment number
First good block
Last good block
Total good blocks
(Man, it's so PAINFUL to try to think @ 1:34AM EDT!)

Anyway, the general idea is not to waste any 'Read from the start' attempts, just in case they get further along without errors. When an error occurs, you can refer to the table to see if there are any existing segments with good blocks overlapping that error.

Sound logical to you?


----------



## mattack (Apr 9, 2001)

Dan203 said:


> I've been meaning to add a report mode that would list more specific info about where the errors are specifically. Would be kind of cool if you could use multiple files to stitch together a clean copy.


Well, since Tivos allow download from pause-point, wouldn't it really allow you to download from ANY point, i.e. the pull-er can request the download start anywhere.. So after a bad patch, just start the new download before that, and stitch together the good pieces.

But I'm not sure if that requires deep knowledge of MPEG data/file formats/etc. (Probably does.)


----------



## lpwcomp (May 6, 2002)

mattack said:


> Well, since Tivos allow download from pause-point, wouldn't it really allow you to download from ANY point, i.e. the pull-er can request the download start anywhere.. So after a bad patch, just start the new download before that, and stitch together the good pieces.
> 
> But I'm not sure if that requires deep knowledge of MPEG data/file formats/etc. (Probably does.)


There are no "bad patches" to skip. They only exist in the transferred file. And they can move. What you end up with is a decrypted file with missing or corrupted frames. I've had the decrypted file length vary by 30 seconds or more.


----------



## mattack (Apr 9, 2001)

Yes, I realize that. But you know WHEN they came from, right? So you can re-download from before the bad spot *IN THE TRANSFERRED FILE*.


----------



## Dan203 (Apr 17, 2000)

mattack said:


> But I'm not sure if that requires deep knowledge of MPEG data/file formats/etc. (Probably does.)


Luckily I've managed to pick up a pretty deep knowledge of MPEG via my job. (see sig)


----------



## L David Matheny (Jan 29, 2011)

lpwcomp said:


> There are no "bad patches" to skip. They only exist in the transferred file. And they can move. What you end up with is a decrypted file with missing or corrupted frames. I've had the decrypted file length vary by 30 seconds or more.


So essentially they're bungling a data-file transfer between two computer devices? How is that reasonable in the 21st Century? Or what am I not understanding?


----------



## lpwcomp (May 6, 2002)

L David Matheny said:


> So essentially they're bungling a data-file transfer between two computer devices? How is that reasonable in the 21st Century? Or what am I not understanding?


It's not a simple file transfer. The data has to be encrypted and muxed into a TS container. That's where the errors are being created, not in the actual transfer. A SWAG is that there is a problem with the interrupt handler in that code.

Edit: Still unacceptable but since they don't officially support TiVo<->PC transfers anymore, not likely to be fixed.


----------



## lpwcomp (May 6, 2002)

mattack said:


> Yes, I realize that. But you know WHEN they came from, right? So you can re-download from before the bad spot *IN THE TRANSFERRED FILE*.


I admit that I misread your initial post on the subject. In any case, I tried doing this manually with kmttg and ended up with a relatively small (@ 10% what it should have been) .tivo file that could not be opened by VRD


----------



## mattack (Apr 9, 2001)

lpwcomp said:


> I admit that I misread your initial post on the subject. In any case, I tried doing this manually with kmttg and ended up with a relatively small (@ 10% what it should have been) .tivo file that could not be opened by VRD


Weird, that sounds like the "try to download in non TS format" problem...


----------



## moyekj (Jan 24, 2006)

Per the Wiki:
kmttg / Wiki / Resume_Downloads

Resume doesn't work with TS downloads, only with PS downloads, which would explain the above.


----------



## lpwcomp (May 6, 2002)

moyekj said:


> Per the Wiki:
> kmttg / Wiki / Resume_Downloads
> 
> Resume doesn't work with TS downloads, only with PS downloads, which would explain the above.


I thought that I remembered that as being the case but was unable to check because SF was having problems at the time.


----------



## ClearToLand (Jul 10, 2001)

After watching several dozen of my offloaded-to-PC (3TB USB Ext. HDD) TiVo recordings (via Streambaby) transferred in PS 'Slow' File Format (via kmttg), I miss '_ungarbled_' Closed Captions, so this time around I decided to revisit my experiment with @Dan203 's PyTivoDesktop and TS Sync Errors. IIRC, it was @reneg who noted that he got 10 times as many TS Sync Errors from his Bolt as compared to his Roamio (?), so, since I own several managed switches, I decided to lower the egress QoS Rate Limit on the port that my 'LR Overflow' BR Roamio OTA 1TB was connected to and compare the results.

The first test was at 'Unlimited' (i.e. no QoS Rate Limit with my 10yo desktop running as fast as it possibly could):

```
51:INFO:pyTivo.togo:[19/Apr/2018 16:50:00] Queued "http://192.168.0.143:80/download/Supergirl.TiVo?Container=/NowPlaying&id=472" for transfer to L:\TiVo\PyTiVoDesktop\Videos
52:INFO:pyTivo.togo:PC sleep has been disabled
55:INFO:pyTivo.togo:[19/Apr/2018 16:50:00] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR
60:INFO:pyTivo.togo:[19/Apr/2018 16:58:51] Done getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR, 4919411944 bytes, 70.69 Mb/s, 360 TS errors in Retry 0 taking 8.85 Minutes
62:INFO:pyTivo.togo:[19/Apr/2018 16:58:58] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR
66:INFO:pyTivo.togo:[19/Apr/2018 17:07:47] Done getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR, 4919411944 bytes, 70.90 Mb/s, 267 TS errors in Retry 1 taking 8.82 Minutes
68:INFO:pyTivo.togo:[19/Apr/2018 17:07:49] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR
73:INFO:pyTivo.togo:[19/Apr/2018 17:16:43] Done getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR, 4919411944 bytes, 70.27 Mb/s, 196 TS errors in Retry 2 taking 8.90 Minutes
75:INFO:pyTivo.togo:[19/Apr/2018 17:16:50] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR
78:INFO:pyTivo.togo:[19/Apr/2018 17:21:12] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR aborted with 258 TS errors in Retry 3 taking 4.37 Minutes
80:INFO:pyTivo.togo:[19/Apr/2018 17:21:14] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR
83:INFO:pyTivo.togo:[19/Apr/2018 17:25:38] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR aborted with 259 TS errors in Retry 4 taking 4.40 Minutes
85:INFO:pyTivo.togo:[19/Apr/2018 17:25:39] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR
90:INFO:pyTivo.togo:[19/Apr/2018 17:30:11] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR aborted with 259 TS errors in Retry 5 taking 4.52 Minutes
91:INFO:pyTivo.togo:PC sleep has been enabled
```
The second test was at '32Mbps'; (the choices on my Netgear switch are in a dropdown and go by powers of 2, so 16Mbps, 32Mbps, 64Mbps, 128Mbps, etc... are my only choices):

```
95:INFO:pyTivo.togo:[19/Apr/2018 17:38:10] Queued "http://192.168.0.143:80/download/Supergirl.TiVo?Container=/NowPlaying&id=472" for transfer to L:\TiVo\PyTiVoDesktop\Videos
96:INFO:pyTivo.togo:PC sleep has been disabled
99:INFO:pyTivo.togo:[19/Apr/2018 17:38:16] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR
109:INFO:pyTivo.togo:[19/Apr/2018 17:58:51] Done getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR, 4919411944 bytes, 30.39 Mb/s, 102 TS errors in Retry 0 taking 20.59 Minutes
111:INFO:pyTivo.togo:[19/Apr/2018 17:58:57] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR
114:INFO:pyTivo.togo:[19/Apr/2018 18:05:45] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR aborted with 142 TS errors in Retry 1 taking 6.79 Minutes
116:INFO:pyTivo.togo:[19/Apr/2018 18:05:46] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR
120:INFO:pyTivo.togo:[19/Apr/2018 18:12:34] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR aborted with 143 TS errors in Retry 2 taking 6.78 Minutes
122:INFO:pyTivo.togo:[19/Apr/2018 18:12:35] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR
126:INFO:pyTivo.togo:[19/Apr/2018 18:19:21] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR aborted with 143 TS errors in Retry 3 taking 6.76 Minutes
128:INFO:pyTivo.togo:[19/Apr/2018 18:19:22] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR
137:INFO:pyTivo.togo:[19/Apr/2018 18:39:55] Done getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-14 - ''Midvale'' (WPIXDT) (TS).tivo" from Roamio C BR, 4919411944 bytes, 30.44 Mb/s, 0 TS errors in Retry 4 taking 20.55 Minutes
138:INFO:pyTivo.togo:PC sleep has been enabled
```
It's now clear to me that giving the TiVo CPU / Interrupt Handler more '_time to think_' has a GREAT impact on the number of TS Sync Errors. As I previously posted (*Post #58*), whether you set Retries to 5, or 20, you're fighting an uphill battle.

But, limit the LAN speed to 32Mbps and, not only do the errors drop, SUCCESS arrives at Retry #4. :thumbsup:

I don't know how many TCF members, besides myself and @mlippert, are interested in this type of information but, with more and more channels switching to H.264, it's going to increase the usage of TS 'Fast' Format Transfers, with the resulting TS Sync Errors and dropped frames (even with VideoReDo).

@mlippert, would you please post some of YOUR results (using your Python 3x version with enhanced logging)? Thanks!


----------



## ClearToLand (Jul 10, 2001)

Dan203 said:


> *I've been meaning to add a report mode that would list more specific info about where the errors are specifically*. Would be kind of cool if you could use multiple files to stitch together a clean copy.


 @Dan203 ,

Have you given this any more thought?


----------



## Dan203 (Apr 17, 2000)

ClearToLand said:


> @Dan203 ,
> 
> Have you given this any more thought?


Honestly at this point I'm not sure I'm going to add any more features to pyTivo Desktop. TiVo has completely killed support for most of pyTivo in Hydra and I'm just not sure I want to put any more effort into it when it could be completely killed off at any moment. If TiVo would offer me some assurance that TTG isn't going to be pulled too then I'd reconsider, but I've asked and no one seems to be willing to guarantee me that.


----------



## mlippert (Apr 3, 2010)

@ClearToLand That's really interesting that limiting the network speed seemed to help the errors.

So just today I had a few shows to download and I can't attribute it to more than coincidence but I've found that my downloads of Krypton from the Syfy channel have consistently had errors and rarely get a clean copy in the 4 tries I allow, while Good Witch from the Hallmark channel has frequently had no errors on the 1st try. I downloaded 2 of each today, and that was the case, both Good Witch episodes had not errors on the 1st try, and both Krypton episodes had errors on all 4 tries. These downloads were all queued in the same run of pytivo and were all from my Bolt.

Here are the 2 Krypton download reports:

```
%YAML 1.2
---
fileName            : "Krypton - s0e00 - Savage Night (May_09_2018, SYFYHD-E).tivo"
fileSize            : 1773724800
downloadStarted     : 2018-05-14T19:30:32Z
attemptSaved        : 4
totalErrorPackets   : 58
downloadAttempts:
    - attemptNumber : 1
      status        : sync_errors_saved
      errorPackets:
          - { count:      7, start:   315139584, end:   315140900 }
          - { count:     52, start:   949666656, end:   949676432 }
          - { count:    122, start:  1562039056, end:  1562061992 }
    - attemptNumber : 2
      status        : sync_errors_saved
      errorPackets:
          - { count:    122, start:  1562039056, end:  1562061992 }
    - attemptNumber : 3
      status        : sync_errors_aborted
      errorPackets:
          - { count:      7, start:   315139584, end:   315140900 }
          - { count:    104, start:   633167904, end:   633187456 }
          - { count:     19, start:   633187456, end:   633191028 }
    - attemptNumber : 4
      status        : sync_errors_saved
      errorPackets:
          - { count:      7, start:   315139584, end:   315140900 }
          - { count:      3, start:   949666656, end:   949667220 }
          - { count:     48, start:   949667408, end:   949676432 }
```


```
%YAML 1.2
---
fileName            : "Krypton - s0e00 - Transformation (May_02_2018, SYFYHD-E).tivo"
fileSize            : 1862829280
downloadStarted     : 2018-05-14T19:50:57Z
attemptSaved        : 4
totalErrorPackets   : 218
downloadAttempts:
    - attemptNumber : 1
      status        : sync_errors_saved
      errorPackets:
          - { count:     80, start:   930685612, end:   930700652 }
          - { count:    155, start:  1236969196, end:  1236998336 }
    - attemptNumber : 2
      status        : sync_errors_aborted
      errorPackets:
          - { count:    138, start:   315455612, end:   315481556 }
          - { count:     23, start:   930685612, end:   930689936 }
          - { count:     56, start:   930690124, end:   930700652 }
          - { count:     41, start:  1236969196, end:  1236976904 }
          - { count:    113, start:  1236977092, end:  1236998336 }
    - attemptNumber : 3
      status        : sync_errors_aborted
      errorPackets:
          - { count:     15, start:   612673836, end:   612676656 }
          - { count:    123, start:   612676844, end:   612699968 }
          - { count:      8, start:   612700156, end:   612701660 }
          - { count:    155, start:  1236969196, end:  1236998336 }
    - attemptNumber : 4
      status        : sync_errors_saved
      errorPackets:
          - { count:    138, start:   315455612, end:   315481556 }
          - { count:     80, start:   930685612, end:   930700652 }
```


----------



## mlippert (Apr 3, 2010)

@ClearToLand I'm just curious are you using the commandline pytivo or Dan's pytivo Desktop?

If you're using the commandline pytivo you might want to see if you can get my fork working, it's got some enhancements that might help your workflow such as somewhat configurable file naming (some information I never figured out how to get such as season and episode numbers) such that I've been able to get the saved names very close to how I have kmttg name my files.
Also I'm guessing you'd like to see the download reports (like in the post above).

I've thought about trying to piece together good parts of multiple downloads, but each download has it's own encryption key so I can't just piece together segments of the various .tivo files, I'd probably have to decrypt them first and I'm not sure if the offsets would match after decrypting (I don't have enough knowledge about the various video and container encodings).

I'm doing slow downloads for all programs that still support them w/ kmttg, and only using pytivo to get the programs from channels that require TS downloads.

My process after that is much more manually intensive than yours, although I've been thinking about trying to automate more of it.
I copy the pytivo downloaded .tivo file to where kmttg sees it, name it to the exact name that kmttg would name it, then schedule kmttg: metadata, decrypt, qsfix, ad detect
kmttg recognizes that the .tivo file already exists, and runs VRD to decrypt and qsfix it.
kmttg also recognizes that the .tivo is a TS not PS and renames the metadata file.

I then manually open the .VPrj VRD file w/ ad cuts defined, and verify/correct all cuts. Then because I want the file saved as a .mp4, I just save the Video from VRD.

I then load up the .mp4 (or a bunch of them I've created) in kmttg FILES and extract captions. I also have to manually fix the metadata file from the .ts.txt name to .mp4.txt.

I don't know if any of the above is useful to you, but I figured you might appreciate me spelling all that out.


----------



## ClearToLand (Jul 10, 2001)

mlippert said:


> I'm just curious are you using the commandline pytivo or Dan's pytivo Desktop?


I run a 'customized' version of Dan's PyTivoDesktop from IDLE. IIRC, I described it earlier in this thread.


mlippert said:


> ...If you're using the commandline pytivo you might want to see if you can get my fork working...


My poor health limits what I can get done in a day. It's on my ToDo List though.


mlippert said:


> ...My process after that is much more manually intensive than yours, although I've been thinking about trying to automate more of it...


Probably confusing to others in that you're replying here to: *kmttg Post #11807*.


Spoiler



Per PM, talk more tomorrow...


----------



## reneg (Jun 19, 2002)

ClearToLand said:


> I don't know how many TCF members, besides myself and @mlippert, are interested in this type of information but, with more and more channels switching to H.264, it's going to increase the usage of TS 'Fast' Format Transfers, with the resulting TS Sync Errors and dropped frames (even with VideoReDo).


Following this thread. I don't have a managed switch, but seeing where this leads might change that.


----------



## reneg (Jun 19, 2002)

Since I don't have a managed switch, I decided to approach the throttling from the PC downloading the show.

Ran a test on my troublesome downloading cleanly Tivo Bolt tonight. When downloading through pytivo desktop, I get between 34-36MB/s network IO from the Bolt. I set up a 5.4GB file to retry 50 downloads at this speed. I did not get a clean download in 50 attempts.

Next, I installed NetLimiter 4 (NetLimiter , free trial, $25 home license) on my PC. I set up a rule on pytivo.exe executable to limit DL (to the PC) speed to 8MB/s (a swag). I transferred the 5.4GB cleanly on the first attempt. Repeated both with and without limiting the DL speed and got the same results a second time. Interesting.

I'm going to test my Roamio Pro which is where the bulk of my recording happens. Typical DL speed on my Roamio Pro is 15-16MB/s. Probably have to start lower than 8MB/s on the Roamio since it has less processing power than the Bolt.

Thanks for reviving this thread.


----------



## Phantom Gremlin (Jun 20, 2002)

Hi guys,

I'm reading this thread with concern. Please tell me that all these transfer problems only happen tivo->computer and not tivo<->tivo.

I just shuffled around a *bunch* of files from one Roamio to another. Most of them were my wife's shows. The transfers seemed to happen at 100 Mbps or faster over Moca.

I'll be in trouble if my wife finds "glitches" in all that stuff!!!


----------



## JoeKustra (Dec 7, 2012)

Phantom Gremlin said:


> Hi guys,
> I'm reading this thread with concern. Please tell me that all these transfer problems only happen tivo->computer and not tivo<->tivo.
> I just shuffled around a *bunch* of files from one Roamio to another. Most of them were my wife's shows. The transfers seemed to happen at 100 Mbps or faster over Moca.
> I'll be in trouble if my wife finds "glitches" in all that stuff!!!


I transfer files to and from various TiVo boxes on my network. With Ethernet and basic Roamio boxes I get 93Mbps. I have never had a file corruption problem.


----------



## reneg (Jun 19, 2002)

Phantom Gremlin said:


> Hi guys,
> 
> I'm reading this thread with concern. Please tell me that all these transfer problems only happen tivo->computer and not tivo<->tivo.
> 
> ...


These problems are tivo->computer related and not tivo <-> tivo. In the tivo -> computer case, the files are converted from a native Tivo format to either PS or TS format. In the tivo <-> tivo case, the files are transferred in the native Tivo format and should not experience transfer problems.


----------



## leswar (Apr 14, 2005)

I have my tivos on a Netgear ProSafe Plus GS-108E network switch but have no idea how I would set it up to manage the d/l speeds to 8MB/s. Also what happens when you use Amazon, Hulu, Youtube, etc. to watch content. Would that also limit their speeds to 8MB/s as well? Is it something that can be configured differently?


----------



## Phantom Gremlin (Jun 20, 2002)

It makes sense to find some sort of software solution to slow down transfers to avoid the corruption. Maybe the answer is as simple as configuring the Ethernet port on the PC to run at 10 mbits/second.

But here's an idea for a crazy hardware method, if you're a pack rat and you've got a bunch of old junk lying around. Do you have an old 10 mbit Ethernet hub gathering dust? Hook that up between the TiVo and the PC. Or maybe you can access your fancy new Ethernet switch via a web interface. Just set the port to 10 mbit/sec.

Voila ... slow transfers.

This is just a theoretical proposal for now. I haven't tried doing this.

And it probably would be marginal at best to try to watch Netflix or Hulu over such a slow connection.


----------



## reneg (Jun 19, 2002)

Ran some tests on my Roamio Pro varying (NetLimiter post above) the bandwidth available to download shows from my Tivo. I need to collect more data because I don't think there is enough data points to draw conclusions, but it does show an interesting trend for my Roamio Pro configuration.








To calculate the effective DL speed to get clean recordings, I summed the sizes of each recording I transferred. I then calculated the total time that it took to download the shows completely or whether it hit a retry limit and aborted starting from the first download attempt until end of the last one. In the data above, I had one recording abort after 50 retries. After that, I upped the retry attempts to 100 and all the shows cleanly transferred eventually (one after 99 retries). I then calculated the effective transfer speed by dividing the total size of the recordings over the total transfer duration









The graph above shows that I was getting better effective throughput when slowing the transfer speed to between 8-10MB/second from my Roamio Pro. When set to unlimited, I regularly see maximum transfer speeds around 16MB/second. I continue to collect data on this Tivo as it is my main recording device.

My Bolt is much faster, but I've just begun to collect data from the Bolt.


----------



## Sparky1234 (May 8, 2006)

Roghet said:


> I searched a number of places before I came to this particular forum. I am not aware of "TCF," and no, I have never watched "Burn Notice." I am a college professor. And one of the first things that is helpful to know in teaching another is to know one's audience, and it is always a good practice to assume the person asking a question knows nothing. I am open to any and all accurate information, so if you have any, please continue. Let's not let the thread dwindle and get off track please. In a few suggestions, there is enough disagreement that I'm not sure which option to try first, and the Java info is over my pay grade!


Can you provide more info on your home network setup and connections? Switches/hubs, router, computer connections, TiVo connection, etc..

I have a theory on patch cables going bad creating crosstalk or poorly made ethernet cable with split pairs causing network errors and interference.

Some switches and routers can read out the speed of the connected devices.

Noisy cables will pass simple continuity testers but fail with more expensive Fluke or similar testers.


----------



## JoeKustra (Dec 7, 2012)

Sparky1234 said:


> Can you provide more info on your home network setup and connections? Switches/hubs, router, computer connections, etc.


The post was from 2017. The person hasn't been here since January.


----------



## Sparky1234 (May 8, 2006)

JoeKustra said:


> The post was from 2017. The person hasn't been here since January.


Missed that time frame, sorry.

Trying to understand what the TiVo transfers rates should be. Premiere, Roamio, Bolt, to each other and to a PC....

Theoretical vice practical:

Bolt at 100 MBs 
Roamio at 10MBs
Premiere at 10MBs

But some have stated higher speeds.


----------



## Sparky1234 (May 8, 2006)

reneg said:


> Ran some tests on my Roamio Pro varying (NetLimiter post above) the bandwidth available to download shows from my Tivo. I need to collect more data because I don't think there is enough data points to draw conclusions, but it does show an interesting trend for my Roamio Pro configuration.
> View attachment 34474
> 
> To calculate the effective DL speed to get clean recordings, I summed the sizes of each recording I transferred. I then calculated the total time that it took to download the shows completely or whether it hit a retry limit and aborted starting from the first download attempt until end of the last one. In the data above, I had one recording abort after 50 retries. After that, I upped the retry attempts to 100 and all the shows cleanly transferred eventually (one after 99 retries). I then calculated the effective transfer speed by dividing the total size of the recordings over the total transfer duration
> ...


Looking forward to your Bolt data on charts and graphs.

What's your equipment?


----------



## JoeKustra (Dec 7, 2012)

Sparky1234 said:


> Trying to understand what the TiVo transfers rates should be. Premiere, Roamio, Bolt, to each other and to a PC....
> Theoretical vice practical:
> Bolt at 100 MBs
> Roamio at 10MBs
> ...


My basic Roamio to PC is about 100Mbs. Roamio to Roamio is 93Mbps. Premiere to Roamio is 65Mbps. Bolts have reports much higher speeds. No change with Hydra on my Roamio.


----------



## Sparky1234 (May 8, 2006)

Is that mbps or MB/s?

Bandwidth conversion calculator | toolstudio


----------



## leswar (Apr 14, 2005)

leswar said:


> I have my tivos on a Netgear ProSafe Plus GS-108E network switch but have no idea how I would set it up to manage the d/l speeds to 8MB/s. Also what happens when you use Amazon, Hulu, Youtube, etc. to watch content. Would that also limit their speeds to 8MB/s as well? Is it something that can be configured differently?


After researching my Netgear Switch I found two programs I could use: NETGEAR Switch Discovery Tool and ProSAFE Plus Utility. After limiting the speed to 8MB/sec I got a clean transfer on the first transfer to my pc using pyTiVo Desktop, but the 2 hr movie took over 3 hrs to d/l, yikes.

The program has a drop down box that only allows a few rate limit choices (4/8/16/32 etc..).
download | Support | NETGEAR to check out these programs for netgear switches.

At first I couldn't access my switch from my pc. Something to do with the switch being on 192.168.0.xxx and everything else on 192.168.1.xxx.
And since the switch is in a different room from the pc I wasn't able to do a direct pc to switch ethernet cable connection. But I was able to change the switch's ip address to a ...1... from my router. So it did work after all.

Might try 16 MB next.


----------



## JoeKustra (Dec 7, 2012)

I use the Network Troubleshooting screen on my TE3 Roamio. So b is bits, B is bytes. That display is missing on TE4.

Let me add this. I have two Roamio, one PC, Blu-ray and AVR on a wireless bridge. The other Roamio is CAT-5 to my Netgear R8000 router. Everything wireless is 802.11ac.


----------



## reneg (Jun 19, 2002)

Sparky1234 said:


> Looking forward to your Bolt data on charts and graphs.
> 
> What's your equipment?


I've been gathering more data points on the Roamio Pro. I'll running one set of transfers at a time between a Tivo & the PC as I don't want to skew the results by running multiple Tivos in parallel. Once I'm done with the Roamio Pro, I'll move over to the Bolt. As I continue to gather data points, the sweet spot has remained between 8-10MB/sec. I have seen the effective MB/sec come down from what I posted earlier, but the general trend in the graph remains close to what I posted.. I'll post updates for the Roamio.

As far as equipment, for Tivos, I've got a Tivo HD (can no longer xfer from Tivo->PC), Premiere, Roamio, & Bolt. All my Tivos are connected to a wired 1Gbps network. My PC is also wired to the same 1Gbps network.


----------



## Sparky1234 (May 8, 2006)

reneg said:


> I've been gathering more data points on the Roamio Pro. I'll running one set of transfers at a time between a Tivo & the PC as I don't want to skew the results by running multiple Tivos in parallel. Once I'm done with the Roamio Pro, I'll move over to the Bolt. As I continue to gather data points, the sweet spot has remained between 8-10MB/sec. I have seen the effective MB/sec come down from what I posted earlier, but the general trend in the graph remains close to what I posted.. I'll post updates for the Roamio.
> 
> As far as equipment, for Tivos, I've got a Tivo HD (can no longer xfer from Tivo->PC), Premiere, Roamio, & Bolt. All my Tivos are connected to a wired 1Gbps network. My PC is also wired to the same 1Gbps network.


Thanks!


----------



## aaronwt (Jan 31, 2002)

reneg said:


> These problems are tivo->computer related and not tivo <-> tivo. In the tivo -> computer case, the files are converted from a native Tivo format to either PS or TS format. In the tivo <-> tivo case, the files are transferred in the native Tivo format and should not experience transfer problems.


What is causing the issue? Since it seems to work without issue using kmttg in my setup at home. 94Mb/s from a Roamio and 300+Mb/s from Bolts to a PC.


----------



## ggieseke (May 30, 2008)

aaronwt said:


> What is causing the issue? Since it seems to work without issue using kmttg in my setup at home. 94Mb/s from a Roamio and 300+Mb/s from Bolts to a PC.


I have never had a TS transfer error out either. Just a data point FWIW.


----------



## leswar (Apr 14, 2005)

ggieseke said:


> I have never had a TS transfer error out either. Just a data point FWIW.


Could you help me understand your statement ( its above my head) 
like error out vs data point. ...Not getting it.?


----------



## ClearToLand (Jul 10, 2001)

mlippert said:


> @ClearToLand That's really interesting that *limiting the network speed seemed to help the errors*...


Your profile says you have a Roamio and 2 Bolts - which Roamio and have you tried PyTivoDesktop from it yet?

@reneg is getting better results from his Roamio Pro than his Bolt (he's willing to set 100 retries!) and even better with NetLimiter. Have you considered NetLimiter (free trial) or a managed switch with QoS Rate Limiting? For me, NetLimiter is $$$ per PC while the managed switch is controlling the TiVo unit(s).


mlippert said:


> ...So just today I had a few shows to download and I can't attribute it to more than coincidence but I've found that my *downloads of Krypton from the Syfy channel have consistently had errors and rarely get a clean copy in the 4 tries I allow*, while Good Witch from the Hallmark channel has frequently had no errors on the 1st try...
> Here are the 2 Krypton download reports:
> 
> ```
> ...


While my 'report' is no way near pretty, when I see results as for "Wake Up" from WPIXDT / FiOS as shown below, I get the impression that no matter how high I set the number of retries, something is 'corrupt' in the actual recording on the TiVo HDD and it's unable to get past it and create a good TS stream. kmttg was easily able to create a PS stream though, and 'garbled' Closed Captions is the way I had to go.

In your first example, "Savage Night", your 7 errors @ "start: 315139584" appear to be a similar situation - consistent throughout each pass. I'd be curious to see if reducing your bandwidth would get you past that error.

```
15/May/2018 21:17:23] Done getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-21 - ''Wake Up'' (WPIXDT) (TS).tivo" from Roamio C BR, 4682064848 bytes, 29.56 Mb/s, 38 TS errors in Retry 0 taking 20.14 Minutes
88:INFO:pyTivo.togo:[15/May/2018 21:17:30] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-21 - ''Wake Up'' (WPIXDT) (TS).tivo" from Roamio C BR
92:INFO:pyTivo.togo:[15/May/2018 21:20:59] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-21 - ''Wake Up'' (WPIXDT) (TS).tivo" from Roamio C BR aborted with 50 TS errors in Retry 1 taking 3.49 Minutes
94:INFO:pyTivo.togo:[15/May/2018 21:21:01] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-21 - ''Wake Up'' (WPIXDT) (TS).tivo" from Roamio C BR
100:INFO:pyTivo.togo:[15/May/2018 21:24:30] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-21 - ''Wake Up'' (WPIXDT) (TS).tivo" from Roamio C BR aborted with 50 TS errors in Retry 2 taking 3.48 Minutes
102:INFO:pyTivo.togo:[15/May/2018 21:24:32] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-21 - ''Wake Up'' (WPIXDT) (TS).tivo" from Roamio C BR
106:INFO:pyTivo.togo:[15/May/2018 21:28:02] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-21 - ''Wake Up'' (WPIXDT) (TS).tivo" from Roamio C BR aborted with 50 TS errors in Retry 3 taking 3.50 Minutes
108:INFO:pyTivo.togo:[15/May/2018 21:28:03] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-21 - ''Wake Up'' (WPIXDT) (TS).tivo" from Roamio C BR
121:INFO:pyTivo.togo:[15/May/2018 21:41:59] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-21 - ''Wake Up'' (WPIXDT) (TS).tivo" from Roamio C BR aborted with 38 TS errors in Retry 4 taking 13.92 Minutes
123:INFO:pyTivo.togo:[15/May/2018 21:42:00] Start getting "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-21 - ''Wake Up'' (WPIXDT) (TS).tivo" from Roamio C BR
129:INFO:pyTivo.togo:[15/May/2018 21:45:30] Transfer of "L:\TiVo\PyTiVoDesktop\Videos\Supergirl - 2017-11-21 - ''Wake Up'' (WPIXDT) (TS).tivo" from Roamio C BR aborted with 50 TS errors in Retry 5 taking 3.49 Minutes
```
BTW, I find elapsed time and bandwidth useful troubleshooting information. Have you considered them?


----------



## ClearToLand (Jul 10, 2001)

mlippert said:


> ...If you're using the commandline pytivo *you might want to see if you can get my fork working*...


I'm holding off on trying out your fork because the HDD in the Win7 desktop that I was building / setting up suffered a catastrophic failure - "stiction". It got into a condition where the HDD was thrashing for an extended period of time and Resource Monitor pointed to the swapfile. I figured that it was time for a reboot (I run my PCs 24x7) so after an hour or so of leaving it alone until the thrashing stopped 'gracefully', I powered it down (since most PCs don't have a reset button any more) to also allow the RAM to reset. Ten seconds later when I reapplied power, nada. I let it cool and days went by - nada. I pulled it and tried it in another Win7 desktop that I'm building - nada. I held it in my hand and tapped it gently on the side with a plastic handle of a Torx and it spun right up. Put it back into the original Win7 desktop and CrystalDiskInfo reported 8 SMART C5 Pending Sector errors. I immediately copied everything that was unique data off to a flash drive. After that finished, I had 1200+ C5 and C6 errors along with 8 05 errors. I have an image of the C: drive but it's also on the same HDD, different partition - lesson learned.

A few days ago, the 3TB USB Ext. HDD with my .tivo files (PS) that I've been watching got a similar C5 error, reported after RoboCopy failed during the transfer of #9 of 10 PyTivoDesktop .tivo files to it. I'll finally be purchasing a copy of SpinRite shortly. With their vast HDD experience, I plan to ask @ggieseke and @jmbach a few HDD theory questions.

Finally, back to your fork - I imagine the installation of Python 3 will change my path and probably add / change an environmental variable or two. With only one functioning desktop, I want to avoid rocking the boat until I have my Win7 desktop running again.


mlippert said:


> ...*I never figured out how to get such as season and episode numbers*) such that I've been able to get the saved names very close to how I have kmttg name my files...


I recently read that kmttg creates more metadata than pyTivo (and thus PyTivoDesktop). I imagine the information you're seeking will involve calling the Tivo Mind server. Take a look at the kmttg Java source code and see what @moyekj does.


mlippert said:


> ...*I've thought about trying to piece together good parts of multiple downloads, but each download has it's own encryption key* so I can't just piece together segments of the various .tivo files, I'd probably have to decrypt them first and I'm not sure if the offsets would match after decrypting (I don't have enough knowledge about the various video and container encodings).


@Dan203 mentioned this idea last year:


Dan203 said:


> I've been meaning to add a report mode that would list more specific info about where the errors are specifically. *Would be kind of cool if you could use multiple files to stitch together a clean copy.*


but deflected to a suggestion to purchase VideoReDo when I mentioned your fork and asked for ideas / tips in my current TiVo CPU/Processor thread.

Back in 2016, when I first dabbled with kmttg and pyTivo (@wmcbrine ), not much was written here on the subject of TS Sync errors. TS 'Fast' File Format transfers were the recommended way for kmttg and VideoReDo was recommended to 'fix' the errors. Since I haven't purchased VRD yet (I don't edit / keep much; more 'Watch & Delete'. When my Roamio got full, I offloaded via kmttg (TS) to a 3TB USB Ext. HDD; months later, when I attempted to pyTivo them back, I learned about TS Sync errors.), initially I thought VRD 'fixed' the errors. But, maybe it doesn't:


lpwcomp said:


> Using VRD does not solve all of the problems. *Yes, it gets rid of the sync errors but it often does this by removing bad frames which can result in loss of content*.


kmttg has a "QS Fix" option where ffmpeg 'fixes' the errors, so, in my mind, why would I need VRD? I still don't understand what "QS Fix" accomplishes, so since I found success with PS 'Slow' Format File transfers and Streambaby, I stuck with that (until I discovered 'garbled' Closed Captions).

A question I do have about VRD is how much CPU horsepower does it require? ComSkip Review on my i3-2350M in my Win7 laptop: Yes, I can move quickly with the mouse or PgUp/PgDn (600 frames), but using the arrow keys to move left or right one frame at a time is painful. Is VRD better?

For my current needs, kmttg produces .EDL files from SkipMode and "_way way back_" in around version 30 something, BEFORE SkipMode was even born, @kearygriffin added the ability to use .EDL files to Streambaby *SO* without further ado, Streambaby has AutoSkip (had it for years!). Don't see it discussed though. With "Watch & Delete", and the lowering price of GBs, it makes more sense to me to leave the commercials in.


mlippert said:


> ...I'm doing slow downloads for all programs that still support them w/ kmttg, and only using pytivo to get the programs from channels that require TS downloads...


I'd love to continue using PS for all the MPEG-2 channels that I record, but, as my age increases and my hearing decreases, I find that I enjoy Closed Captions and PS consistently 'garbles' Closed Captions. Now @reneg recently mentioned that TS with Sync Errors also 'garbles' his Closed Captions but I haven't researched that yet. One thing on my ToDo List is to see if a TS .TIVO d/l with minimal TS Sync errors will produce a .SRT file that can be used with a decrypted PS .MPG d/l (i.e. does the timing match up?).

A slight problem that I noticed with .TS files is that Streambaby, via ffmpeg, now has to 'trans-mux' the files instead of just 'copying' them to the TiVo unit IP. Buffering is MUCH slower than a .MPG file. This is from my workhorse E2200 desktop. I haven't tried it yet from my i3-2350M laptop.



mlippert said:


> ...*My process after that is much more manually intensive than yours*, although I've been thinking about trying to automate more of it...


My process feels very manually intensive, with the:

running, and checking, of kmttg to create the .EDL files
downloading via PyTivoDesktop (TS) to create the .TIVO files
renaming the .TIVO files to the kmttg-generated name on the .EDL files
adding all the .TIVO files to the FILE tab of kmttg
running Metadata, Decrypt, Captions in kmttg
using RoboCopy to copy the .EDL, .TXT, .SRT, .TS (and .TIVO just in case) files to my NAS
using FileSync (or FC /B) to verify the copy (since the SMART C5 errors recently appeared on TWO of my HDDs)



mlippert said:


> ...I don't know if any of the above is useful to you, but *I figured you might appreciate me spelling all that out*.




I think I'm at around the 2 hour point now in composing / proofreading this post. It has taken me almost 3 years of reading the TCF Archives to acquire this knowledge so I hope that others are interested in it and find it useful for their endeavors.


----------



## reneg (Jun 19, 2002)

Update of clean transfers using pyTivo Desktop on my Roamio Pro with retry limit set to 100. I am using a program called NetLimiter to vary the download speed from the Tivo to the PC.

This table shows a summary of the data I collected for my Roamio Pro:









Here is a chart of effective MB/sec (blue) for the various download speeds. I added a second axis (orange) showing the percentage of transfers which transferred cleanly at that speed. On my system, the sweet spot for downloading clean TS files has remained between 8-10MB/sec. As I've collected more data in the last week, the overall effective MB/Sec has dropped, but the sweet spot has remained the same.









I've added another chart which shows the number of retries attempted for each download speed. You can see a narrowing of the variability of the number of retries as the download speed is decreased. However, as you decrease the download speed, the file takes longer to transfer.









I'm going to start focusing on collecting data from my Bolt. It's going to take a bit longer to gather data from the Bolt as the Bolt has a higher max download speed (ie more download values to check) vs the Roamio, and my Bolt takes a lot more retries per show than the Roamio Pro to get a clean transfer.


----------



## ClearToLand (Jul 10, 2001)

leswar said:


> I have my tivos on a *Netgear ProSafe Plus GS-108E* network switch but have no idea how I would set it up to manage the d/l speeds to *8MB/s*. Also what happens when you use Amazon, Hulu, Youtube, etc. to watch content. Would that also limit their speeds to 8MB/s as well? Is it something that can be configured differently?


I have:

```
#NETGEAR ProSAFE Plus GS105Ev2
#NETGEAR ProSAFE Plus GS108Ev3
#NETGEAR ProSAFE Plus GS108E-300NAS
#NETGEAR ProSAFE Plus GS108T-200NAS
#NETGEAR ProSAFE Plus GS108T-200NAS
#TP-LINK Easy Smart TL-SG108E v2.0
#TP-LINK Easy Smart TL-SG108E v2.0
#TP-LINK Easy Smart TL-SG108E v2.0
#TP-LINK Easy Smart TL-SG108E v2.0
```
(copied from my LMHOSTS file).

When you use QoS Rate Limit on a port on the managed switch, you limit the bandwidth to the device(s) connected to that port. If the port is connected to the TiVo unit, it affects only the TiVo unit. If the port is an uplink to your router, or another switch, it affects all the ports on that switch.

From my research, ingress Rate Limit does not work properly on the Netgear switches so I was forced to use egress on the uplink port. Yes it affects everything but 32Mbps is plenty of bandwidth for streaming.


leswar said:


> After researching my Netgear Switch *I found two programs I could use*: NETGEAR Switch Discovery Tool and ProSAFE Plus Utility...


Older posts (and versions of the switch) stated that they required software to be installed in order to manage them. Linux users complained that the software was for Windows only. All of my current Netgear and TP-Link switches are accessible via a web browser.


leswar said:


> ... After limiting the speed to *8MB/sec* I got a clean transfer on the first transfer to my pc using pyTiVo Desktop, but the 2 hr movie took over 3 hrs to d/l, yikes.
> 
> The program has a drop down box that only allows a few rate limit choices (4/8/16/32 etc..).
> download | Support | NETGEAR to check out these programs for netgear switches...


I can't tell if you understand the difference between MB/sec and Mb/sec so I politely ask @reneg to post in Mb/sec in order to not confuse those not so technical. (If I'm wrong, I apologize in advance).

IME, ethernet equipment has always been described in terms of Mbps (Mb/sec; mega BITS per second) and HDD equipment in terms of MBps (MB/sec; mega BYTES per second). A BYTE consists of 8 BITS so 8MBps = 64Mbps. Windows Task Manager and Resource Monitor follow these conventions (my experience is with Vista and Win7). TiVo Transfer History is in Mbps. It would be great if everyone participating in this thread used Mbps, IMHO. Thanks! 

When @Dan203 first released PyTivoDesk with TS Retries, I found it difficult to comprehend what was actually going on. All I could do initially was to keep raising the Retry Limit in a (sometimes fruitless) attempt to get an error-free transfer. When I added my DEBUG statements, then I could see what the problem was. The server inside the TiVo unit has great difficulty, IME, generating a TS Stream without also creating (mostly) totally RANDOM TS Sync errors. Only by viewing the details that @mlippert and I added could one see where the errors were.

Here's an example based on my current settings:

The first pass is always a COMPLETE download attempt, ignoring any errors.
- Rounding numbers to make things easier to see, a QoS egress Rate Limit of 32Mbps results in a ~5GB file (60 minute CW show in MPEG-2; file sizes vary per network) transferring in ~22 minutes. If there were no errors, I just transferred a 60 minute show in 22 minutes. 
.
If there were errors, we enter Retry #1. During these Retry passes, the current error total is compared to the COMPLETE download error total from pass #1 (aka Retry #0 in my logs). As long as the current total is less, the transfer continues. If the COMPLETE errors are more toward the end of the file, the time will be greater. If the bulk of the errors are toward the beginning of the file, the transfer will abort and the time will be much less - maybe single digit minutes.
- Based on this, I find @reneg 's table difficult to interpret. If you're transferring ONE show and it transfers successfully during Retry #0, @ 32Mbps, it will take 22 minutes. If the errors are towards the end and it transfers successfully during Retry #1, it was take 44 minutes. Retry #3, 66 minutes, etc...

If the errors are towards the beginning of the file, Retry #1 may abort @ 5 minutes. Now, two passes took 27 minutes instead of 44 minutes; 3 passes would take 32 minutes instead of 66 minutes. So, unless you know how many Retries were required and how long each Retry took, it's difficult to form any conclusions on 'effective' bandwidth.
Regarding a 120 minute show taking 180 minutes to transfer successfully. I usually queue up 8 60 minute shows to transfer overnight. The last batch took 8 hours. Success varies from Retry #0 to Retry #5 so unless I do one show at a time, I can't say what the 'effective' bandwidth used was. 2 22 minute passes turn 32Mbps into 16Mbps; 4 passes turn it into 8Mbps. If the Retries are single digit minutes, the calculation gets too complicated for me.

If you don't already own a Managed Switch, NetLimiter appears to be a good choice for this application. From a brief read on the web site, it appears to have much greater, and finer, control over the specific process and the Rate Limit (i.e. pyTivo and NOT based on the powers of 2; 8, 16, 32,64, etc....).



leswar said:


> ...At first I couldn't access my switch from my pc. Something to do with the switch being on 192.168.0.xxx and everything else on 192.168.1.xxx.
> And since the switch is in a different room from the pc I wasn't able to do a direct pc to switch ethernet cable connection. But I was able to change the switch's ip address to a ...1... from my router...


The default IP of Netgear switches is 192.168.0.239. If your LAN is set up for 192.168.1.xxx, you could have temporarily changed your PC from 192.168.1.xxx to 192.168.0.xxx as long as xxx wasn't 239. Then you could have gotten into the Netgear config page with a web browser and set up a static IP. Personally, I use DHCP with reservations on my DD-WRT equipped router. Practically everything on my LAN has a fixed IP so that I can always know what I'm talking to just by its IP. I've used just about all of the available NVRAM in my router with almost 100 devices so it's time for a new router (or a new DHCP Server) but that's just how I like to run things.


leswar said:


> ...Might try 16 MB next.


So far, with just myself and @reneg comparing notes regarding bandwidth, what speed you try next is depending on what TiVo Unit you're using. The lower the TiVo CPU/Processor power, the lower the bandwidth you need to use.

And, the Rate Limit dropdown on the Netgear QoS page is in Mbps, not MBps, so, if you have a Roamio Basic / OTA, like me, you can try 32Mbps. With a Roamio Pro or Bolt, maybe 64Mbps.

Experiment and let us all know.


----------



## ClearToLand (Jul 10, 2001)

reneg said:


> Update of clean transfers using pyTivo Desktop on my Roamio Pro with retry limit set to 100... ...*I've added another chart which shows the number of retries attempted for each download speed*. You can see a narrowing of the variability of the number of retries as the download speed is decreased. However, as you decrease the download speed, the file takes longer to transfer.
> View attachment 34592


Thanks for the third chart.

I'm trying to correlate your 64Mbps (8MBps) results with 100 retries with @mlippert and my 5 and 4 retry limits respectively. I've successfully transferred 16 out of 18 ~5GB recordings @ 32Mbps within 5 retries. What are you using to create this chart and would it be possible to see the raw data.

I interpret very few successful transfers in 5 or less retries on your Roamio Pro @ 64Mbps. You have a spike @ ~67 and another @ ~23. Even at 32Mbps (4MBps) it appears that the majority of your recordings need at least 10 retries to be successful. But, with 100 retries as your limit, you state 100% success so are the 2 shows of my 18 that failed in 2-4 runs of 5 retries each chances of success better with even lower bandwidth or should I bump my retries up and just queue my 'problem children'? [I'll try 50 retries for those 2 shows tonight (or tomorrow; Wow! Where did this evening go?]

I've settled on 32Mbps and 5 retries to get the greatest number of successful transfers within a certain time period. 100 (or even 67) retries @ 22 minutes each (5GB @ 32Mbps; yours should be half of this) is 2200 minutes / 36.7 hours / 1.5 days for one show (unless it fails early most of the time and once it gets over that 'hump', it's home free.).

Are you running 24x7 to collect this amount of data?


----------



## Dan203 (Apr 17, 2000)

I could probably add throttling to pyTivo directly. When I was first adding the retry code I tried requesting various chunk sizes from TiVo to see if it made any difference in the error rates. It didn't seem to in my testing. But I could add a short sleep to each loop to artifically throttle the download. I don't think I ever tried that.

Another possibility is I could make the chunk size and sleep delay variables. Maybe the you guys could run some tests and figure out if there is some combo that results in error free downloads.


----------



## Dan203 (Apr 17, 2000)

ClearToLand said:


> @Dan203 mentioned this idea last year:but deflected to a suggestion to purchase VideoReDo when I mentioned your fork and asked for ideas / tips in my current TiVo CPU/Processor thread.


Real answer... let's assume that the byte positions in every attempt are exactly the same. (I have not tested this to confirm) Let's also assume that the corrupted bits contain valid TS packets just scrambled and that the clock continues to tick. (another assumption I have not confirmed) If those are both true and you have two files where the corruption does not over lap then you should be able to just concat the pieces together from the two, or more, files until you have a single file with no corruption.

This will only work if the files are decrypted because I'm pretty sure the encryption uses a rolling key and that wont line up if you just concat the bytes. There is also a big possibility that the timestamps/clock in each file wont line up when you concat this way. If that happens then using something like VideoReDo will be your only option.


----------



## Dan203 (Apr 17, 2000)

What's the GIT link to the v3 fork you're using?


----------



## ClearToLand (Jul 10, 2001)

Dan203 said:


> What's the GIT link to the v3 fork you're using?


mlippert/pytivo

pyTivo - Transcoding server


----------



## reneg (Jun 19, 2002)

ClearToLand said:


> ...
> 
> If there were errors, we enter Retry #1. During these Retry passes, the current error total is compared to the COMPLETE download error total from pass #1 (aka Retry #0 in my logs). As long as the current total is less, the transfer continues. If the COMPLETE errors are more toward the end of the file, the time will be greater. If the bulk of the errors are toward the beginning of the file, the transfer will abort and the time will be much less - maybe single digit minutes.
> - Based on this, I find @reneg 's table difficult to interpret. If you're transferring ONE show and it transfers successfully during Retry #0, @ 32Mbps, it will take 22 minutes. If the errors are towards the end and it transfers successfully during Retry #1, it was take 44 minutes. Retry #3, 66 minutes, etc...
> ...


Before I started these tests, I used the keep fewest errors within 50 retries settings in pyTivo deskop to do my overnight transfers. In the morning, I'd restart shows for another 50 retries that did not transfer cleanly overnight and repeat that during the day until I got a clean transfer. Eventually, I would get a clean transfer.

For these tests, I have been using the reject & retry setting in pyTivo Deskop. My goal is to get clean file transfers in the shortest amount of time while minimizing the number of times I have to manually restart transfers. If I have a high level of confidence that I can transfer 100% of my shows cleanly then I don't need to use keep fewest errors. If I do encounter a stubborn file that can't get a clean transfer after hundreds of attempts, I can always back off to the keep fewest error setting.

There is another aspect that doesn't come out in the data I've shared. I'm on Comcast which in my city doesn't compress local channels so files sizes are pretty much at the same as what you'd get via OTA, but the rest of the cable channels are aggressively compressed. The file size of a hour show on a local channel is larger than a hour show on a cable channel. Smaller files might have a higher probability of transferring cleanly as compared to larger files. I think this is true, but I don't have enough data to draw a conclusion.

Overall, it's difficult to form concrete conclusions on the data gathered so far. However, I feel like I've found a setting (8MB/sec) for my Roamio Pro that has a high probability of meeting my goal when I get up in the morning and see that all my shows transferred cleanly.


----------



## reneg (Jun 19, 2002)

ClearToLand said:


> Thanks for the third chart.
> 
> I'm trying to correlate your 64Mbps (8MBps) results with 100 retries with @mlippert and my 5 and 4 retry limits respectively. I've successfully transferred 16 out of 18 ~5GB recordings @ 32Mbps within 5 retries. What are you using to create this chart and would it be possible to see the raw data.
> 
> ...


I'm using Excel to create the charts. It's a little messy, but I'm happy to share this spreadsheet (attached). Remove the .txt extension from the attachment.

Your interpretation is correction. In my setup, my success percentage for 5 of less retries is lower than what you see. I'm sure I'll eventually run into a file that fails to cleanly transfer after 100 attempts at 8MB/sec.

I have been running tests pretty much 24x7 since you revived this thread and I found the NetLimiter program.


----------



## mlippert (Apr 3, 2010)

ClearToLand said:


> Your profile says you have a Roamio and 2 Bolts - which Roamio and have you tried PyTivoDesktop from it yet?


I prefer the control I get just running pytivo (my fork) and I run it from linux, so for both those reasons I haven't tried PyTivoDesktop.



ClearToLand said:


> In your first example, "Savage Night", your 7 errors @ "start: 315139584" appear to be a similar situation - consistent throughout each pass. I'd be curious to see if reducing your bandwidth would get you past that error.


I'm not sure how I'd go about limiting my bandwidth (and right now all TiVos are on my wireless network, and the system running pytivo is also)

Since my results have been tolerable, I don't have much personal bandwidth to devote to trying to get perfect downloads. 200 error packets is only 40KB, 40KB out of 1GB is 0.004 percent, if that 1GB is an hour then less than 1 second is bad (40KB/1GB * 1hr * 1GB/1000000KB * 3600sec/1hr = 0.144sec), if the 1hour of video is larger than 1G (mine are, that's even less bad in the video)

Even if you consider that percentage estimation to be underestimating the affect of the error, because It's possible that a bad packet could trash the entire frame it occurs in, or even span video frames, or maybe it's a reference frame and therefore affects a few more, but even so that's not a big glitch as long as the video player can handle it (or for me, as long as VRD can deal with it so I end up w/ a playable video).



ClearToLand said:


> BTW, I find elapsed time and bandwidth useful troubleshooting information. Have you considered them?


Do you mean putting them in my report file? I haven't, but it would be easy enough to put elapsed time in if someone else was actually using my fork.
I'm not sure what you mean by bandwidth in this instance, if you mean my network bandwidth, I'm not sure how I'd get that, if you mean transfer rate for an individual file (total size / elapsed time) that would also be easy to add.


----------



## mlippert (Apr 3, 2010)

ClearToLand said:


> mlippert/pytivo
> 
> pyTivo - Transcoding server


@Dan203 I've tried to be really good about the commit messages, so you can get a sense of what each commit does from the commit message.

On the other hand, the readme is sorely out of date and a shambles. It has useful information, but also outdated information and is poorly organized.

I did incorporate many of the changes from your fork (although I suspect you've made changes you haven't pushed) but many of your later changes were Windows/Mac centric so I didn't try to merge them.


----------



## mlippert (Apr 3, 2010)

Dan203 said:


> Real answer... let's assume that the bute positions in every attempt are exactly the same. (I have not tested this to confirm) Let's also assume that the corrupted bits contain valid TS packets just scrambled and that the clock continues to tick. (another assumption I have not confirmed) If those are both true and you have two files where the corruption does not over lap then you should be able to just concat the pieces together from the two, or more, files until you have a single file with no corruption.
> 
> This will only work if the files are decrypted because I'm pretty sure the encryption uses a rolling key and that wont line up if you just concat the bytes. There is also a big possibility that the timestamps/clock in each file wont line up when you concat this way. If that happens then using something like VideoReDo will be your only option.


OK 1st question, what's a bute position? Oh, nevermind, that's byte position, I thought it was some TS encoding field name at first.

I am definitely seeing corrupted packets at the same offset and for the same number of consecutive packets in multiple download attempts, that's one of the reasons added the report, to make it easier to see if that was the case, and if some attempts had some ranges while others didn't, which is also true.

Yes the encryption is definitely different in each download attempt (I tried the simple test of comparing 2 different downloads).

I was going to say that the decrypted file seems to always be a different size than the encrypted file which means that I wouldn't expect the offsets to line up anymore, but then I realized that I never just decrypt a .tivo, I always do it via VRD and qsfix it at the same time, so it would of course have more changes than just decrypting. Should the decrypted file be the same exact size as the encrypted file?

Thanks for the thoughts, there's so much I really don't understand about how videos are encoded and the video and audio streams are kept in sync in the various container formats. Your expertise is greatly appreciated.


----------



## Dan203 (Apr 17, 2000)

If you decrypt with tivolibre, and don't use the option that skips bad packets, then two decrypted files of the same show should be identical minus the corrupted parts.

In my tests the corruption was always in the same places. However occasionally I'd see a file where one of the corruption points was not corrupt. However unless you can cobble together enough files to where you have a good segment for every corruption point you wouldn't be able to create a clean file. This would likely take more retries then getting a clean file. 

And even if you did VideoReDo's joiner would work much better for this anyway. Then you could visually cut out the bad scenes from files and merge in the good scenes from other files. Trying to do this manually by bytes would be tedious, and still might not work if the clock is not identical between the two files. (I suspect it would be, but I haven't ever looked)


----------



## Dan203 (Apr 17, 2000)

mlippert said:


> @Dan203 I've tried to be really good about the commit messages, so you can get a sense of what each commit does from the commit message.
> 
> On the other hand, the readme is sorely out of date and a shambles. It has useful information, but also outdated information and is poorly organized.
> 
> I did incorporate many of the changes from your fork (although I suspect you've made changes you haven't pushed) but many of your later changes were Windows/Mac centric so I didn't try to merge them.


I haven't touched pyTivo in a while. I'm pretty sure my fork on GitHub is up to date. I'm not sure if I could even convert my fork to v3, since I'm using that pybuilder thing to make an exe. But I could play with the throttling stuff and you could port to your fork. I might also borrow your reporting stuff if it's v2 compatible.


----------



## mlippert (Apr 3, 2010)

Dan203 said:


> I haven't touched pyTivo in a while. I'm pretty sure my fork on GitHub is up to date. I'm not sure if I could even convert my fork to v3, since I'm using that pybuilder thing to make an exe. But I could play with the throttling stuff and you could port to your fork. I might also borrow your reporting stuff if it's v2 compatible.


Yep, I haven't done much w/ pyTivo in a while either. I've gotten busy with other stuff, so while I have some ideas for additional improvements, I haven't played with them.

I try to keep an eye on your fork (I've even got a older branch of your fork from long ago with a few minor changes (I tried to describe in the commit message what was going on, and I didn't want the mac and windows binaries in my repo, otherwise it's identical to your fork).

If you do play with the throttling, definitely let me know. I cleaned up the downloading in an attempt to improve thread safety, so my branch has deviated from yours there, but your basic algorithm is still there.


----------



## mlippert (Apr 3, 2010)

Dan203 said:


> If you decrypt with tivolibre, and don't use the option that skips bad packets, then two decrypted files of the same show should be identical minus the corrupted parts.


OK that's very good to know, thanks.



Dan203 said:


> In my tests the corruption was always in the same places. However occasionally I'd see a file where one of the corruption points was not corrupt. However unless you can cobble together enough files to where you have a good segment for every corruption point you wouldn't be able to create a clean file. This would likely take more retries then getting a clean file.


I've limited my attempts to 4, and I've seen several occasions where with those 4 attempts I could put together a single uncorrupted file, so I may give this a shot (although not in the near future, I'm pretty busy right now). If/when I do I'll post back.



Dan203 said:


> And even if you did VideoReDo's joiner would work much better for this anyway. Then you could visually cut out the bad scenes from files and merge in the good scenes from other files. Trying to do this manually by bytes would be tedious, and still might not work if the clock is not identical between the two files. (I suspect it would be, but I haven't ever looked)


I agree that using VRD's joiner for this would produce much better results, but that would require watching all copies to find the bad parts, setting up cut points and joining, an arduous process for restoring some small parts of a video.

Doing it by bytes I can make pytivo do as it downloads the files.


----------



## mlippert (Apr 3, 2010)

@Dan203 the one thing I'd really like to add to pytivo error detection if it is possible is the time offset where the error occurs so that I could go to that location say in VRD and see if it is a problem I care about. Because as you mentioned, 25+% of a TV show is commercials which means there's a 25% chance the errors are in commercials and I really don't care (in fact I'm likely to just be cutting it out anyway).

Is there a field in a TS packet that identifies its location in the stream?
And is that field encrypted, so it can only be interpreted after decryption?


----------



## Dan203 (Apr 17, 2000)

It's not in the individual TS packets. You'd basically have to demux the TS and then parse the audio or video stream to get the PTS values. The TS packets themselves do have a "clock" for transmission timing, but the values have nothing to do with the time you'd see in VideoReDo. 

One thing to keep in mind is that VideoReDo does show a byte offset when you seek to a certain point. It's not 100% accurate with .tivo files because of the header but it should get you close enough to find the point and see if it's in a commercial break.


----------



## BilliJoe (Oct 2, 2016)

mlippert said:


> I agree that using VRD's joiner for this would produce much better results, but that would require watching all copies to find the bad parts, setting up cut points and joining, an arduous process for restoring some small parts of a video.


You don't need to watch the copies. Just run each of them through VRD QSF and examine it's log files to see the time offset of the sync corrections it makes.
That marks the spots in each of the file that are corrupted.
That would require keeping all downloaded files and finding the ones that you could overlap.
It's still tedious though to piece together parts of the same file downloaded multiple times with corruption in different spots using VRD's joiner to create a good output file.
Spent a couple days doing that successfully and realized it's not worth the effort, YMMV.

I use my TiVo as a backup recording source now. My primary recordings don't have these issues since they're not from a TiVo anymore. When my primary source has an issue (rarely) I will attempt to download them from my TiVo and then I also encounter these issues, which has been two times in the last six months and I've been able to eventually get a good download using a 50 times retry.


----------



## mlippert (Apr 3, 2010)

@Dan203 and @BilliJoe thanks, very helpful information.


----------



## ClearToLand (Jul 10, 2001)

reneg said:


> Before I started these tests, I used the keep fewest errors within 50 retries settings in pyTivo deskop to do my overnight transfers. In the morning, I'd restart shows for another 50 retries that did not transfer cleanly overnight and repeat that during the day until I got a clean transfer. Eventually, I would get a clean transfer...


*4 x 5 != 1 x 20 !?!*​
I wasn't very happy with "Retries=30" when my gigabit LAN bandwidth was unlimited. It just seemed so fruitless to keep trying to 'force' something that didn't seem to want to happen. Then my light bulb went off and I remembered that I bought these various managed switches to experiment and learn about features that unmanaged switches didn't have (i.e. VLANs and QoS). While the Netgear brand only offers FIXED settings, based on a power of 2, the TP-LINK 'seems' to offer '_anything goes_' so soon I may be able to experiment with GT 32Mbps but LT 64Mbps and try some 'fine tuning'.

But, before that, I compared the raw numbers on your spreadsheet with my log files and, while we both have achieved much more success in fewer retries with lowered bandwidth limits, we both have also had some '_sticklers_' that just refused to transfer (in a reasonable number of retries). On my two stubborn Supergirl episodes (I had tried 4 runs of 5 retries each without success), I upped my retries to 50 and let them both lug. One completed @ 21 and the other @ 22.

Noting that you had similar problems, I came up with this new theory: FRAGMENTATION

As simple as it sounds, if the TiVo units are *SO* sensitive to timing with their interrupt handler during a TS 'Fast' Format File Transfer, what about the delay introduced when the HDD head has to seek GT XX tracks for the next sequential sector of data? One of the 'features' of SpinRite that I read about recently discussed repeated reads on a 'weak' sector before marking it bad like CHKDSK would do *BEFORE* CHKDSK recovered the data. Left to its own devices, Windows will gladly lose your data for you. Maybe this somehow relates to "4 x 5 != 1 x 20"? I was certainly surprised to see success on retries #21 and #22 but geez that sure sucks up wall time.

Instead, I calculated the storage requirement for the 2 Supergirl episodes and then PERMANENTLY DELETED (via kmttg) over 2X that amount of (attempted contiguous FIFO) space on a 2nd Roamio OTA. Then I transferred the shows to that Roamio OTA and reran PyTivo Desktop - SUCCESS on the *FIRST* PASS!

The big push behind all of this 'offloading to NAS' right now is because, like any good Pack Rat, all 3 of my currently in service TiVo units are full and I have to do something. The last one to be put into service was a 1TB OTA and it was pretty much filled from zero to 100% sequentially with minimal deletions (i.e. I've been saving the full seasons of Arrow, Gotham, Supergirl, The Flash for the summer doldrums) thus most of the recordings are contiguous and many of the oldest recordings are transferring in one pass.

While we both may still occasionally come across a 'glitched' recording that is actually 'glitched' on the original TiVo unit, you might consider this 'experiment' for the ones that still fail after ~20-30 retries. I did some research regarding Linux defragging and the consensus I gathered was that Linux, even more than NTFS, allocates more space to a file than it absolutely requires to allow for editing / growth without causing fragmentation compared to FAT (and possibly FAT32) which would butt files right next to one another. Relating this to a TiVo unit, I think of GC and Kickstart 57, IIRC.

I used to keep my TiVo unit HDDs full and allow them to permanently delete as required. I would change shows from KUID to 'Space Needed' as I watched them and allow 'Recently Deleted' to grow and shrink on its own. Now I'm 'attempting' to achieve a FIFO regiment, especially for the units that I will be using to transfer recordings off of. While 'Recently Deleted' appears to use FIFO according to how a show got there, I now think that keeping a bit more free space available, say 90% full, and then using kmttg to permanently delete the whole shebang in bigger chunks may promote more contiguous recording.


----------



## reneg (Jun 19, 2002)

ClearToLand said:


> *4 x 5 != 1 x 20 !?!*​
> I wasn't very happy with "Retries=30" when my gigabit LAN bandwidth was unlimited. It just seemed so fruitless to keep trying to 'force' something that didn't seem to want to happen. Then my light bulb went off and I remembered that I bought these various managed switches to experiment and learn about features that unmanaged switches didn't have (i.e. VLANs and QoS). While the Netgear brand only offers FIXED settings, based on a power of 2, the TP-LINK 'seems' to offer '_anything goes_' so soon I may be able to experiment with GT 32Mbps but LT 64Mbps and try some 'fine tuning'.
> 
> But, before that, I compared the raw numbers on your spreadsheet with my log files and, while we both have achieved much more success in fewer retries with lowered bandwidth limits, we both have also had some '_sticklers_' that just refused to transfer (in a reasonable number of retries). On my two stubborn Supergirl episodes (I had tried 4 runs of 5 retries each without success), I upped my retries to 50 and let them both lug. One completed @ 21 and the other @ 22.
> ...


Isn't the Roamio OTA networking limited to 100Mbs ethernet? TiVo® Roamio Comparison Chart - compare the TiVo Roamio Models

I can certainly try permanently deleting a bunch of recordings and see if there is drop in the number of retries on my Roamio Pro. I have all but given up on trying to collect data on my Bolt. I cannot reliably get clean transfers after 100 attempts. It's actually takes less time to transfer recordings from the Bolt to the Roamio Pro and then from the Roamio Pro to the PC for clean transfers.


----------



## mlippert (Apr 3, 2010)

@ClearToLand had suggested that it would be nice to see the download rate included in my log, @Dan203 mentioned how I could use VRD to see where the corruption was occurring by using the byte offset into the tivo file (actually the MB offset), and I realized that it would be useful to know which of my tivos the download was from.
Given all that I've enhanced the syncerr.yaml file I save with each download from my pytivo fork (v2.6.0). It now looks like this (note new tivoName, transfer and startMB property):

```
%YAML 1.2
---
fileName            : "Marvel's Cloak and Dagger - s0e00 - Suicide Sprints (Jun_07_2018, FREEFORMHD-E).tivo"
fileSize            : 2048898352
tivoName            : LivingRoomRoamio (192.168.2.81)
downloadStarted     : 2018-06-10T17:51:41Z
attemptSaved        : 2
totalErrorPackets   : 88
downloadAttempts:
    - attemptNumber : 1
      status        : sync_errors_saved
      transfer      : { bytes:  2054139792, seconds:  399.2, rate: " 39.26 Mb/s" }
      errorPackets:
          - { count:     88, start:   335333624, end:   335350168, startMB:   319.80 }
          - { count:     35, start:  1673818988, end:  1673825568, startMB:  1596.28 }
          - { count:     23, start:  1673825756, end:  1673830080, startMB:  1596.28 }
    - attemptNumber : 2
      status        : sync_errors_saved
      transfer      : { bytes:  2048898352, seconds:  377.2, rate: " 41.45 Mb/s" }
      errorPackets:
          - { count:     88, start:   335333624, end:   335350168, startMB:   319.80 }
    - attemptNumber : 3
      status        : sync_errors_aborted
      transfer      : { bytes:   331802608, seconds:   63.9, rate: " 39.63 Mb/s" }
      errorPackets:
          - { count:     88, start:   335333624, end:   335350168, startMB:   319.80 }
    - attemptNumber : 4
      status        : sync_errors_aborted
      transfer      : { bytes:   333899184, seconds:   67.1, rate: " 37.99 Mb/s" }
      errorPackets:
          - { count:     88, start:   335333624, end:   335350168, startMB:   319.80 }
...
```
I'm using yaml because it is very human readable, but also easily parsed by machine if someone later feels like using the info in some way.


----------



## BilliJoe (Oct 2, 2016)

I was seeing pretty much the same as you are, but your download attempts are way too small in number to get a good overview picture. If you expand your download attempts you'll see the TiVo induced TS TTG Sync Packet corruptions walk back and forth all over the place in the downloads (many times in the same place/places) and for me at least I was able to get a good download after many attempts (12 to 50). Don't know what the root cause of this issue is, but it was introduced a couple years ago (Early 2016) by a TiVo update and I'm sure TiVo's not interested in fixing it since saying TTG is no longer supported. That's what drove me to look for another solution and no longer depend on my TiVo for archiving. Good to see some people are still on TiVo and there's hope with your effort and Dan203's. After a long journey from TiVo-Land, I'm quite happy with my HDHOMERUN's and Channels DVR.


----------



## reneg (Jun 19, 2002)

Final update to my tests with limiting network speed to download clean files. I have run tests on both a Roamio Pro (1Gb network capable) and Roamio OTA or Basic (100Mb network capable). I also have a Bolt, but it was so unreliable at transferring clean recordings that I stopped collecting data on it. To get a clean download from my bolt, it's faster to transfer it Tivo to Tivo to a Roamio and then download the file from the Roamio. I am using a Windows program called NetLimiter 4 to vary the download speed from the Tivo to the PC.

This table shows a summary of the data I collected for my Roamio Pro:









Here is a chart of effective MB/sec (blue) for the various download speeds. I added a second axis (orange) showing the percentage of transfers which transferred cleanly at that speed within 100 retries.









Another chart which shows the number of retries attempted for each download speed.









Roamio OTA or Basic data:









The effective MB/sec (blue) for the various download speeds for the Roamio OTA:









And the number of retries attempted for each download speed:


----------



## mlippert (Apr 3, 2010)

reneg said:


> Final update to my tests with limiting network speed to download clean files.


Thanks, this is a lot of work and interesting data that it is too late right now for me to fully grok. I'll have to come back and review this again later.


----------



## BilliJoe (Oct 2, 2016)

reneg said:


> Final update to my tests with limiting network speed to download clean files. I have run tests on both a Roamio Pro (1Gb network capable) and Roamio OTA or Basic (100Mb network capable). I also have a Bolt, but it was so unreliable at transferring clean recordings that I stopped collecting data on it. To get a clean download from my bolt, it's faster to transfer it Tivo to Tivo to a Roamio and then download the file from the Roamio. I am using a Windows program called NetLimiter 4 to vary the download speed from the Tivo to the PC.
> 
> This table shows a summary of the data I collected for my Roamio Pro:
> View attachment 35367
> ...


Nice work! I think you're on to something.
I was able to reduce the TS sync erros in downloads by limiting the download rate also.
I'm experimenting with 10mb/s right now with 12 downloads on my Roamio base unit.
If that works, I'll try negotiating 10mb/s link speed to the Roamio.

Using a Ubiquity ER-X (very configurable and powerful for the price of $49 retail) between my Roamio and router as the rate controller.

Update: That worked, no TS sync errors in any of the 12 downloads where at least 2 failed at higher bitrates. After some shuteye going to try limiting the link speed to 10mb/s. Makes sense as the first TiVo to do TS downloads (because of H.264 TS streams) was the NZ/Australia version (series 3) that was limited to 10mb/s link speed.

Limiting the link speed between the Roamio and router to 10mb/s also worked.
Now trying to find the maximum error free download rate. Down to 15mb/s right now as 20 had one error.


----------



## BilliJoe (Oct 2, 2016)

```
Decimal notation
Mb - Megabit  - 10^6 bits
MB - MegaByte - 10^6 Bytes
                10^6 = 1,000,000
Binary notation
Mib - Mebibit  - 2^20 bits
MiB - Mebibyte - 2^20 Bytes
                 2^20 = 1,048,576
```
Much more limited than @reneg did, but testing TS downloads from Roamio Basic at different speeds using pyTivo Desktop (set to retry each download up to 50 times) and a Ubiquity EdgeRouter X ER-X configured as follows;

interface eth0 (IP 10.0.0.1) admin port -> admin PC (IP 10.0.0.2)
for *BANDWIDTH LIMITED TESTING*
interface eth1 (NO IP) -> Router (IP 192.168.1.1) -> PC doing downloads
interface eth2 (NO IP) -> Roamio (IP 192.168.1.5)
interface br0 (NO IP) bridging eth1 and eth2 interfaces
QoS smart-queue eth1 upload bandwidth = 12 Mb/sec (note: this results in effective TS file download bandwidth of 10-11Mb/s)
for *UNLIMITED BANDWIDTH TESTING*
interface eth3 (NO IP) -> Router (IP 192.168.1.1) -> PC doing downloads
interface eth4 (NO IP) -> Roamio (IP 192.168.1.5)
interface sw0 (NO IP) switch connecting eth3 and eth4 as a layer-2 switch

All 12 recordings (8 mpeg2, 4 H.264) on my Roamio consistently download clean on the first attempt when bandwidth is limited to 10Mb/s. Of the 12, these two H.264 recordings amost always have TS sync errors when downloaded at any bandwidth above 10Mb/s
1) Alone - 2018-07-20 - Of Mice and Men
2) Cooper's Treasure - 2018-07-21 - Shallow Water Blackout

_*edited to add: most mpeg2 TS downloads succeed at higher download speeds (all mpeg2 PS downloads succeed, of course) vs. H.264 TS downloads which have TS errors (TS sync loss) most of the time. In fact the 8 mpeg2 recordings mentioned succeeded in downloading as a TS file on the first attempt in the UNLIMITED BANDWIDTH test I did.*_

*BANDWIDTH LIMITED TESTING*

Recording 1 succeeds on first attempt, taking 1,271 seconds for an effective download speed of 10.47 Mib/s

```
INFO:pyTivo.togo:[22/Jul/2018 15:11:37] Start getting "T:\Alone - 2018-07-20 - ''Of Mice and Men'' (HISTHD-W) (TS).tivo" from Roamio
INFO:pyTivo.togo:[22/Jul/2018 15:32:48] Done getting "T:\Alone - 2018-07-20 - ''Of Mice and Men'' (HISTHD-W) (TS).tivo" from Roamio, 1745113016 bytes, 10.48 Mb/s
                                 21:11 = 21*60 + 11 = 1,271 secs
1,745,113,016 Bytes / 1,271 secs = 1,373,023 Bytes/sec (1.37 MB/s, 1.31 MiB/s)
1,745,113,016 Bytes  * 8 = 13,960,904,128 bits
13,960,904,128 bits / 1,271 secs = 10,984,189 bits/sec (10.98 Mb/s, 10.47 Mib/s)
```
Recording 2 succeeds on first attempt, taking 2,434 seconds for an effective download speed of 10.48 Mib/s

```
INFO:pyTivo.togo:[22/Jul/2018 15:32:54] Start getting "T:\Cooper's Treasure - 2018-07-21 - ''Shallow Water Blackout'' (TDCHD-W) (TS).tivo" from Roamio
INFO:pyTivo.togo:[22/Jul/2018 16:13:28] Done getting "T:\Cooper's Treasure - 2018-07-21 - ''Shallow Water Blackout'' (TDCHD-W) (TS).tivo" from Roamio, 3342460176 bytes, 10.48 Mb/s
                                 40:34 = 40*60 + 34 = 2,434 secs
3,342,460,176 Bytes / 2,434 secs = 1,373,237 Bytes/sec (1.37 MB/s, 1.31 MiB/s)
3,342,460,176 Bytes  * 8 = 26,739,681,408 bits
26,739,681,408 bits / 2,434 secs = 10,985,900 bits/sec (10.98 Mb/s, 10.48 Mib/s)
```
*UNLIMITED BANDWIDTH TESTING*

Recording 1 succeeds on 49th attempt, taking 2,142 seconds for an effective download speed of 6.22 Mib/s although each individual attempt is about 90Mib/s

```
INFO:pyTivo.togo:[22/Jul/2018 16:27:21] Start getting "T:\Alone - 2018-07-20 - ''Of Mice and Men'' (HISTHD-W) (TS).tivo" from Roamio
INFO:pyTivo.togo:TS sync losses detected, retrying download (1)
... retry (2) - (47)
INFO:pyTivo.togo:TS sync losses detected, retrying download (48)
INFO:pyTivo.togo:[22/Jul/2018 17:03:03] Done getting "T:\Alone - 2018-07-20 - ''Of Mice and Men'' (HISTHD-W) (TS).tivo" from Roamio, 1745113016 bytes, 87.12 Mb/s
                                 35:42 = 35*60 + 42 = 2,142 seconds
1,745,113,016 Bytes / 2,142 secs = 814,712 Bytes/sec (0.81 MB/s, 0.78 MiB/s)
1,745,113,016 Bytes  * 8 = 13,960,904,128 bits
13,960,904,128 bits / 2,142 secs = 6,517,696 bits/sec (6.51 Mb/s, 6.22 Mib/s)
```
Recording 2 fails all 51 attempts, taking 4,008 seconds for an effective download speed of 0 Mib/s although each individual attempt is about 90Mib/s

```
INFO:pyTivo.togo:[22/Jul/2018 17:23:52] Start getting "T:\Cooper's Treasure - 2018-07-21 - ''Shallow Water Blackout'' (TDCHD-W) (TS).tivo" from Roamio
INFO:pyTivo.togo:TS sync losses detected, retrying download (1)
    retry (2) - (49)
INFO:pyTivo.togo:TS sync losses detected, retrying download (50)
INFO:pyTivo.togo:TS sync loss detected: 286180460 - 286182716
INFO:pyTivo.togo:[22/Jul/2018 18:30:40] Transfer of "T:\Cooper's Treasure - 2018-07-21 - ''Shallow Water Blackout'' (TDCHD-W) (TS).tivo" from Roamio aborted
                               1:06:48 = 1*3600 + 6*60 + 48 = 4,008 seconds
```


----------



## BilliJoe (Oct 2, 2016)

Same result on a friend's Bolt VOX on Comcast cable that was downgraded to the previous software version (20.7.4.RC42), whatever that's called now.
Only difference is the Bolt transfers at 250+ mb/s vs. my slower Roamio.
Of the 14 recordings downloaded;
All (8) MPEG2 recordings transferred error free at full speed.
All (6) H.264 recordings couldn't transfer at full speed without TS Sync errors after 50 retrys.
All (6) H.264 recordings transferred error free on first attempt at 10mb/s.
Appears there's a speed limit involved if you want error free H.264 TS downloads?


----------



## Dan203 (Apr 17, 2000)

I get errors in MPEG-2 TS streams, so this isn't specific to H.264


----------



## BilliJoe (Oct 2, 2016)

Dan203 said:


> I get errors in MPEG-2 TS streams, so this isn't specific to H.264


Well, that blows that H.264 problem theory considering my limited testing.
Have you tried limiting the download bandwidth to around 10 mb/s to see if that helps?
I know it's slow, but it works for me every time, so far.

Not too concerned as my Roamio is only a backup DVR to my HD HOMERUN PRIME and Channels DVR recording to a NAS, but when I do have to rely on a TiVo recording it comes into play.


----------



## Dan203 (Apr 17, 2000)

What do you use to throttle bandwidth?


----------



## reneg (Jun 19, 2002)

Dan203 said:


> What do you use to throttle bandwidth?


I use NetLimiter 4 ( NetLimiter , free trial, $25 home license) on my PC. I set up a rule on pytivo.exe executable to limit DL (to the PC) speed to either 8MB/s or 10MB/s.


----------



## aaronwt (Jan 31, 2002)

I still don't get the error issue. I use kmttg to automatically transfer my One Passes to a PC from my Bolts and Roamio and then kmttg converts the .tivo files to .ts files. Then Plex adds them to it's folders for playback. And when I've played any of those back in Plex or from a PC, I've seen no issues during playback. Transfer speeds to the PC from my Bolts are at over 300Mb/s. And none of the transfers fail using kmttg whether they are MPEG-2 or h.264.

If I have actually had any errors, they have had no effect on playback or transfers.


----------



## ggieseke (May 30, 2008)

aaronwt said:


> If I have actually had any errors, they have had no effect on playback or transfers.


I use kmttg and VideoReDo to process MPEG-2 files from my (antenna) basic Roamio and my (Comcast) H.264 Roamio Pro. If there are any errors I'm not seeing them either.


----------



## HerronScott (Jan 1, 2002)

aaronwt said:


> still don't get the error issue. I use kmttg to automatically transfer my One Passes to a PC from my Bolts and Roamio and then kmttg converts the .tivo files to .ts files. Then Plex adds them to it's folders for playback. And when I've played any of those back in Plex or from a PC, I've seen no issues during playback. Transfer speeds to the PC from my Bolts are at over 300Mb/s. And none of the transfers fail using kmttg.


My understanding is the transfers won't fail on anything other than Dan's pyTivo Desktop if you have TS Sync detection enabled for an error-free download (Sorry Dan, still haven't installed this so not positive that I have the terminology correct). None of the other tools like kmttg or pyTivo or TiVo Desktop have this detection built-in (someone might have created a separate fork of pyTivo including it). This only impacts shows transferred in Transport Stream which hit most people when Comcast switched to MPEG4 (although Dan indicated above that he's seen this on MPEG2 shows also transferred via Transport Stream) since MPEG2 shows could be transferred in Program Stream. I think the end result is that you can get video issues where there are TS Sync errors so people are looking for ways to get clean downloads without TS Sync errors.

Since you indicated that kmttg converts the files to .ts files does that mean that you are downloading in Transport Stream and using TivoLibre for decryption?

Scott


----------



## reneg (Jun 19, 2002)

aaronwt said:


> I still don't get the error issue. I use kmttg to automatically transfer my One Passes to a PC from my Bolts and Roamio and then kmttg converts the .tivo files to .ts files. Then Plex adds them to it's folders for playback. And when I've played any of those back in Plex or from a PC, I've seen no issues during playback. Transfer speeds to the PC from my Bolts are at over 300Mb/s. And none of the transfers fail using kmttg whether they are MPEG-2 or h.264.
> 
> If I have actually had any errors, they have had no effect on playback or transfers.


For me, the issue manifests itself as garbled captions which I notice.


----------



## BilliJoe (Oct 2, 2016)

Dan203 said:


> What do you use to throttle bandwidth?


Ubiquity Edge Router X
See a few posts above yours
Although you could also use a smart switch that can limit the link speed to 10 mb/s.


----------



## BilliJoe (Oct 2, 2016)

Guess it's not a good fix for a Bolt, but works for my Roamio.
Lent my router to my friend with the Bolt and he was able to get a few troublesome H.264 recordings downloaded error free when bandwidth limited, but still has many that won't download error free, even at 10-12 mb/s. He went back to full bandwidth and is keeping downloads with the least errors after 5 retries.


----------



## BilliJoe (Oct 2, 2016)

Just wanted to add for those of you viewing this thread and shaking your head.

Originally a TiVo could only download MPEG2 programs as a PS (Program Stream).
When TiVo introduced TS (Transport Stream) downloads from a TiVo to PC it really sped up transfers quite a bit. Think this was NZ/Australian TiVo HD (Series 3).
An MPEG2 recording can be downloaded to a PC as either a PS or a TS.
An H.264 recording can be downloaded either way, but if PS it's only audio, to get both Video and Audio you have to download as a TS.

Sometime in early 2016 TiVo came out with a software update that caused TS (Transport Stream) downloads to be randomly corrupt, losing parts of the recording downloaded.
That's what we're talking about when we say TS Sync loss in the transfer.

A Tivo receives a broadcast from OTA or Cable as a TS (Transport Stream) and disassembles it to record in the proprietary TiVo way to it's hard drive.
When you ask the TiVo to transfer that recording from it's hard drive to your PC as a .TiVo TS file, it has to create a new TS (Transport Stream) file from what it has on its hard drive and send that over your network connection to your PC. That is where it's messing up.

If you don't care that your TiVo is dropping some of your program info randomly as it downloads the recording to your PC, than you have no need to be concerned.


----------



## mattack (Apr 9, 2001)

BilliJoe said:


> An H.264 recording can be downloaded either way, but if PS it's only audio, to get both Video and Audio you have to download as a TS.


Though I'm now truly seeing some shows downloaded in TS format show up as audio only too. I need to download and keep one of the .tivo files at some point to provide some info in the kmttg thread. But I've seen it many times.. seems to happen on the CNN documentary shows that I would otherwise download and watch at 1.5-2X in VLC.. (I'm instead watching in QuickMode at 30% faster.)


----------



## elprice7345 (Sep 28, 2009)

Another data point, I recently moved from Houston to Denver, keeping Comcast as my provider. I've noticed that I have TS download errors much more often with some channels than others.

The worst offenders (in both cities) were AMC, History, FX, USA, Natgeo, and Syfy. Other H264 channels don't give me many problems at all, e.g., Science Channel, Smithsonian, etc.

How could the feed from specific networks impact the generation of TS errors? Why would one H264 channel cause problems and not another?


----------



## BilliJoe (Oct 2, 2016)

elprice7345 said:


> Another data point, I recently moved from Houston to Denver, keeping Comcast as my provider. I've noticed that I have TS download errors much more often with some channels than others.
> 
> The worst offenders (in both cities) were AMC, History, FX, USA, Natgeo, and Syfy. Other H264 channels don't give me many problems at all, e.g., Science Channel, Smithsonian, etc.
> 
> How could the feed from specific networks impact the generation of TS errors? Why would one H264 channel cause problems and not another?


That's interesting. Do you know if there's anything in common with the trouble channels, like ownership or channel frequency?

I know some channels have more TS Sync errors than others on my Roamio, but never bothered to check which ones since I can always get a clean download from it and it's just a backup I hardly ever need to use.

My friends bolt is another issue. Seems he cannot get a clean download anymore from any channel, so he transfers those recordings he wants to keep to his Roamio and downloads from it.


----------



## Wil (Sep 27, 2002)

Given all this, should a purist go back to getting the shows from an enhanced Tivo model HD via tyftp or tyftpd and converting with 3tots (EDIT: there is a recent version that handles mpeg4s)? I used to have scripts to do all that pretty much hands-off but I abandoned it for the built-in modern methods. I have to say I'm not seeing any of the problems reported here, this is all new to me, but maybe I just don't have a critical eye anymore.


----------



## BilliJoe (Oct 2, 2016)

Wil said:


> I have to say I'm not seeing any of the problems reported here, this is all new to me, but maybe I just don't have a critical eye anymore.


That's where I noticed. I thought I saw him/her do/say this when I watched Live, but it's missing in the recording. The transferred recording was missing "CHUNKS" of the program material. Do I want to save that recording? UM, let me think for a minute... or two. Well it depends on the program and if I want to record it and save it offline for future viewing... If not, then I will just DELETE it!


----------



## elprice7345 (Sep 28, 2009)

When QSF runs, is it removing chunks of the file with TS errors, so that a file with fewer TS errors would more closely match the duration of the show as shown in pyTivo or kmttg? Would shows with more TS errors show a shorter duration after QSF processing?

Show durations after post-processing should only match if there are 0 TS errors?

Or asking the same question another way, if a show’s downloaded duration matches the TiVo duration, the show has no TS errors?

Is it a reasonable test to compare post-download, post-QSF show durations as a proxy for how many TS errors were encountered?


----------



## Soapm (May 9, 2007)

Wil said:


> Given all this, should a purist go back to getting the shows from an enhanced Tivo model HD via tyftp or tyftpd and converting with 3tots (EDIT: there is a recent version that handles mpeg4s)? I used to have scripts to do all that pretty much hands-off but I abandoned it for the built-in modern methods. I have to say I'm not seeing any of the problems reported here, this is all new to me, but maybe I just don't have a critical eye anymore.


Please share if you get those tools working, I need to get some shows off my TivoHD and nothing I've tried seems to work.


----------



## Wil (Sep 27, 2002)

Soapm said:


> Please share if you get those tools working, I need to get some shows off my TivoHD and nothing I've tried seems to work.


I'm away from home right now and I'm going from memory, but you need some files from "the other place." Assuming you still have ftp access to the HD (and other basics are still intact) just install tyftpd and its helper file (not the patchfile, that's just info). This is easier than msf_ftp.tcl because all the patches and support files are included and don't have to be dealt with manually. The default download port is 5013 but you can manually change it to the normal 3105 if you want with a parameter on running (or add the startup command to your rc.sysinit.author file). Then download your shows with a standard ftp client on your pc or Mac or whatever (using the appropriate port). Not all ftp clients work but most do. Settings have to be fiddled with to maximize speed but out-of-the-box settings should work well even if slowly, just to get started.

The resulting files are very different from series 2 ty, ty+ or mfs files and no video players around today are likely to play them directly. But that's OK because you want standard mpeg files anyway. Find 3tots at "the other place" and use it to convert (to transport streams, obviously). You'll probably want to set up a batch script rather than manually running it one file at a time. The most recent 3tots handles shows from mpeg4 channels as well as the standards.

If you're more traditionalist and want to stay with mfs_ftp I think a user on the other place went to the trouble of accumulating all the patches and utilities conveniently. You'll have to look.

EDIT: I just saw your reference to sapper (zapper I think you meant). So you probably haven't been keeping the "basics intact" and have to get back to a standard base configuration. I don't believe any of those really old scripts would work today except Irhorer (spelling?) did a nice script about 10 years ago that would probably still work but it needs to be run in a full Linux environment. I think he posts here occasionally. There is virtually no one active at the other place these days. I don't think I can reduce what is required to a concise straightforward set of instructions and even if I did it would be a dozen or so command line instructions. I could probably get it so it would work without understanding what is going on but if you typed something wrong troubleshooting would be a nightmare I don't want.

If you _*can*_ get your HD to a basic setup again, the above ftp process would get your files out just fine.


----------



## Soapm (May 9, 2007)

Wil said:


> EDIT: I just saw your reference to sapper (zapper I think you meant). So you probably haven't been keeping the "basics intact" and have to get back to a standard base configuration. I don't believe any of those really old scripts would work today except Irhorer (spelling?) did a nice script about 10 years ago that would probably still work but it needs to be run in a full Linux environment. I think he posts here occasionally. There is virtually no one active at the other place these days. I don't think I can reduce what is required to a concise straightforward set of instructions and even if I did it would be a dozen or so command line instructions. I could probably get it so it would work without understanding what is going on but if you typed something wrong troubleshooting would be a nightmare I don't want.


Now that you mention it, I think Irhorer (sp?) remoted in and hacked my Tivo using his scripts. I couldn't get my normal hacking script to work and didn't have enough Linux skill to work his work so he was kind enough to do it for me. Then someone or somehow I got a script I run after each update, all I do is type "backup.sh" before reboot and presto.

I guess what I'm saying is I'm inept when it comes to this stuff, but as it stands as long as I don't kill the TivoHD then I can't do any worse than my current situation. No need having it hacked and encryption disabled if I can't retrieve the shows. I will give it a try when I get some time. Thanks...


----------



## HerronScott (Jan 1, 2002)

Wil said:


> I don't believe any of those really old scripts would work today except Irhorer (spelling?) did a nice script about 10 years ago that would probably still work but it needs to be run in a full Linux environment. I think he posts here occasionally


I was wondering about @lrhorer the other day when I ran across an old post and found that he hasn't been online here since December 5, 2016. His last activity on that other forum was April 16, 2017.

Scott


----------



## ClearToLand (Jul 10, 2001)

ggieseke said:


> I use kmttg and VideoReDo to process MPEG-2 files from my (antenna) basic Roamio and my (Comcast) H.264 Roamio Pro. *If there are any errors I'm not seeing them either*.


As @HerronScott stated, PyTiVo Desktop is the ONLY currently available download method that even checks for TS Sync Errors. So, I've been meaning to ask you this '_for a long time now_' (just re-read your post while re-reading this thread looking for a LINK to include in a reply in the kmttg main thread), have you installed PyTiVo Desktop and done a TS transfer?

If not, when you get a chance, please try it and post your results, noting the model of TiVo and the reported transfer speed. [According to @reneg, his Bolt had more errors than his Roamio and, IIRC, his NetLimiter software program helped with the Roamio but not so much with the Bolt.] I have one Roamio Basic and three Roamio OTAs.

Thanks!


----------



## ggieseke (May 30, 2008)

ClearToLand said:


> As @HerronScott stated, PyTiVo Desktop is the ONLY currently available download method that even checks for TS Sync Errors. So, I've been meaning to ask you this '_for a long time now_' (just re-read your post while re-reading this thread looking for a LINK to include in a reply in the kmttg main thread), have you installed PyTiVo Desktop and done a TS transfer?
> 
> If not, when you get a chance, please try it and post your results, noting the model of TiVo and the reported transfer speed. [According to @reneg, his Bolt had more errors than his Roamio and, IIRC, his NetLimiter software program helped with the Roamio but not so much with the Bolt.] I have one Roamio Basic and three Roamio OTAs.
> 
> Thanks!


I have only compared about a dozen files since it has never been a problem for me but so far pyTivo Desktop, kmttg & direct downloads using a web browser are identical. The pyTivo Desktop downloads never report any errors on the first pass. Note that this is all on TE3 Roamios. My basic Roamio runs at about 95Mbps due to its 100Mbps ethernet port. The gigabit Pro runs at least twice as fast.

I have about 2,250 downloaded recordings using kmttg and I have never seen any visible corruption in them, although I obviously haven't watched them all. My gut says that it's a Bolt, TE4 or home network problem. Sorry to be the rare "it works fine" exception, but there you go.

P.S. All TS transfers


----------

