# Reliable transfer of .TS files?



## alleybj (Dec 6, 2000)

Has anyone found a reliable means to transfer Tivo files from Tivo to a pc without dropping video frames? I can't seem to find one. I never had any problems with non.TS files. Tivo Desktop and kmttg produce files that are inferior to what's on the Tivo. Thanks


----------



## MHunter1 (Oct 11, 2007)

pyTiVo?


----------



## alleybj (Dec 6, 2000)

MHunter1 said:


> pyTiVo?


Does it transfer from Tivo to pc? I thought it only went the other direction.


----------



## MHunter1 (Oct 11, 2007)

alleybj said:


> Does it transfer from TiVo to pc?


Sorry, I mis-read your question. After realizing you want to pull the .ts file from TiVo, I tried transferring a .ts file from my Premiere XL4 to a Windows PC using a lesser-known program called TiVoPlayList.

http://www.tvreviewer.com/tivoplaylist/

After scanning through the video to check for any problems, the show seemed to look and sound fine. But I also downloaded the same file using kmttg and found the size and quality to be identical. I'm not a hardcore videophile so maybe I'm just not seeing the dropped video frames you described. Are they obvious and constant or subtle and random?


----------



## gonzotek (Sep 24, 2004)

All of the available transfer utilities receive the same data from a TiVo when transferring, in either .PS or .TS format. If you're not getting everything(ie dropped frames), it's happening further up the pipeline than the transfer utility(inside the TiVo during the re-muxing process, or from the cable company). They just open an http connection and accept whatever data TiVo sends.

If there are very small glitches in the video signal coming from your cable co, I *believe* the ps format may be innately more tolerant of these issues than ts. Someone with more knowledge of video coding would have to confirm and/or expand on that though, it's beyond my ability of expertise.


----------



## alleybj (Dec 6, 2000)

gonzotek said:


> All of the available transfer utilities receive the same data from a TiVo when transferring, in either .PS or .TS format. If you're not getting everything(ie dropped frames), it's happening further up the pipeline than the transfer utility(inside the TiVo during the re-muxing process, or from the cable company). They just open an http connection and accept whatever data TiVo sends.
> 
> If there are very small glitches in the video signal coming from your cable co, I *believe* the ps format may be innately more tolerant of these issues than ts. Someone with more knowledge of video coding would have to confirm and/or expand on that though, it's beyond my ability of expertise.


Interesting. So, you think glitches in the video stream that aren't visible when watching on the Tivo are picked up and made visible in the transport stream?


----------



## alleybj (Dec 6, 2000)

gonzotek said:


> All of the available transfer utilities receive the same data from a TiVo when transferring, in either .PS or .TS format. If you're not getting everything(ie dropped frames), it's happening further up the pipeline than the transfer utility(inside the TiVo during the re-muxing process, or from the cable company). They just open an http connection and accept whatever data TiVo sends.
> 
> If there are very small glitches in the video signal coming from your cable co, I *believe* the ps format may be innately more tolerant of these issues than ts. Someone with more knowledge of video coding would have to confirm and/or expand on that though, it's beyond my ability of expertise.


You say that they receive the same data, but I notice different glitches with different programs. For example, using Tivo Desktop, there's a skipped frame at 30 minutes but with kmttg not until the one hour mark, with neither frame skipped on the Tivo. Thanks


----------



## wmcbrine (Aug 2, 2003)

alleybj said:


> Does [pyTivo] transfer from Tivo to pc? I thought it only went the other direction.


It does, but, it won't give different results.



alleybj said:


> You say that they receive the same data, but I notice different glitches with different programs.


Apart from the difference between PS and TS transfers, there should be no other differences.


----------



## alleybj (Dec 6, 2000)

wmcbrine said:


> Apart from the difference between PS and TS transfers, there should be no other differences.


And yet it does produce different results. Any thoughts as to why? thanks


----------



## nazopo (Dec 21, 2014)

alleybj said:


> And yet it does produce different results. Any thoughts as to why? thanks


I think the main reason why is because the TS files require less processing by the Tivo box since it's basically sending the original broadcast to the PC while the PS files are often remuxed by the Tivo and thus any errors in the stream are fixed or taken care of. TS streams basically have all the error correction bits and are thus more robust when being sent to a unit while PS streams don't have all the error correction bits and are not used for OTA or QAM broadcasts as a result. What's nice with using the TS option is that your recording is much less likely to cut off in the middle or anywhere in the stream when it's transferring to your PC which leaves you with an incomplete recording. Whereas the PS stream will sometimes do that. I often find certain times when it's more likely to affect the majority of all recordings I've done to when it only happens once every two months. Usually in the summer is when I've noticed it to happen much more frequently while in other parts of the year it rarely occurs. But with those recordings that don't fully transfer over PS, they will transfer completely with TS, and I don't recall the last time a situation where the TS stream wouldn't completely transfer. However, with the TS stream I sometimes find that in parts of a recording there may be glitches in the stream that aren't present at all in the PS stream, and these glitches occur at the exact same location if I try to transfer the recording again. But I have found that sometimes say if I used Tivo Desktop to transfer a recording in TS format and I transfer the same recording in kmttg in TS format then the glitch won't be there at all. Then there's times when no program seems to make a difference. So best practice is to use PS format when transferring recordings even if it's slower you will not run the chance of having glitches in your recordings that weren't present in the original recording. However, if a recording cannot transfer completely then use the TS format and try kmttg, Tivo Desktop, and Tivo Web Server(even though kmttg uses this method too) and see if there is a difference with each case.

In terms of finding a more reliable method of transferring TS files, I have found that wired or wireless doesn't really make a difference since TCP would handle any errors made across the network. But as I stated above some programs may transfer the recording better than others but it's not always the case.


----------



## nazopo (Dec 21, 2014)

Which Tivo model are you using by the way just out of curiosity.


----------



## alleybj (Dec 6, 2000)

nazopo said:


> Which Tivo model are you using by the way just out of curiosity.


I actually have three, RoamioPro, Premiere Elite XL, and a regular Premier. They all produce the same results.


----------



## lew (Mar 12, 2002)

alleybj said:


> And yet it does produce different results. Any thoughts as to why? thanks


What tool you use to download doesn't produce different results. Tivo does all the work. No different then saying who pushes the button to turn on a TV makes a difference.

The difference is what took you use to decrypt. Tivo desktop uses the direct show filter. KMTTG has an option to use direct show filter, or use videoredo to decrypt which is about the same. BT default KMTTG uses tivolibre.

A fair comparison requires using the same decryption tool


----------



## wuzznuubi (Jan 17, 2013)

alleybj said:


> I actually have three, RoamioPro, Premiere Elite XL, and a regular Premier. They all produce the same results.


Add my Roamio base model to the list. Try downloading the same recording from the TiVo a few times and you'll most likely get glitches in different parts of each download.

Here's a post I have on the suject at the VideoRedo support forum http://www.videoredo.net/msgBoard/showthread.php?35729-5-1-2-740-QSF-H-264-TS-logs-42-frames-removed-output-is-4-381-frames-2-26-shorter&p=122413#post122413


----------



## nazopo (Dec 21, 2014)

wuzznuubi said:


> Add my Roamio base model to the list. Try downloading the same recording from the TiVo a few times and you'll most likely get glitches in different parts of each download.
> 
> Here's a post I have on the suject at the VideoRedo support forum http://www.videoredo.net/msgBoard/showthread.php?35729-5-1-2-740-QSF-H-264-TS-logs-42-frames-removed-output-is-4-381-frames-2-26-shorter&p=122413#post122413


Do you notice the some of the glitches occur at the same locations in the downloaded file or does that never happen since for me it always seems to be the same glitches.


----------



## wuzznuubi (Jan 17, 2013)

Are we comparing apples to apples? Are your recordings broadcast H.264 video w/AC3 audio? Are you downloading them in TS format? What are you using to download them? 
Any glitch in the original recording may either transfer or prevent transfer.
A TS stream has to be intact in order to encapsulate the payload.
Glitch free recordings on the TiVo sometimes come over via TS ttg with their own TiVo made glitches (loss of sync and/or continuity in the ts stream delivered). I have one example recording left on my TiVo that I had to download 11 times to get a perfect transfer. In addition to discontinuities in the TS stream it would lose TS sync in one or more of 5 well defined areas of the TS stream. The last (11th) ttg TS download was sans glitches.
This has been my experience with my Roamio base unit and YMMV.

P.S. TiVo's official response is that TiVo Desktop is no longer available or supported. Doesn't work on the new Bolt, so it's antiquated tech that's been replaced by the new technology we all asked for called streaming! (You voted for it)


----------



## nazopo (Dec 21, 2014)

wuzznuubi said:


> Are we comparing apples to apples? Are your recordings broadcast H.264 video w/AC3 audio? Are you downloading them in TS format? What are you using to download them?
> Any glitch in the original recording may either transfer or prevent transfer.
> A TS stream has to be intact in order to encapsulate the payload.
> Glitch free recordings on the TiVo sometimes come over via TS ttg with their own TiVo made glitches (loss of sync and/or continuity in the ts stream delivered). I have one example recording left on my TiVo that I had to download 11 times to get a perfect transfer. In addition to discontinuities in the TS stream it would lose TS sync in one or more of 5 well defined areas of the TS stream. The last (11th) ttg TS download was sans glitches.
> ...


Are you determining whether there are any glitches using VideoRedo's QuickStream fix to see if there were any frames removed or are you using another possibly faster method? Also what setup on the tivo side do you use that you found ultimately reduces the glitches the most(i.e. not having any of the channels tuned to a working channels, having device on standby)?

I've read that Tivo Desktop does still work with the Bolt but that they don't officially support it so good luck talking to CSR about it (which usually doesn't make a difference anyway)


----------



## wuzznuubi (Jan 17, 2013)

nazopo said:


> Are you determining whether there are any glitches using VideoRedo's QuickStream fix to see if there were any frames removed or are you using another possibly faster method?


 Loading the unencrypted TiVo files in a hex editor and examining the transport stream looking for loss of sync. Easiest after setting the hex editor line length to a ts packet size of 188 bytes.



nazopo said:


> Also what setup on the tivo side do you use that you found ultimately reduces the glitches the most(i.e. not having any of the channels tuned to a working channels, having device on standby)?


 Haven't determined anything reliable, I just keep downloading a program until I get a clean transport stream with no sync loss.


----------



## nazopo (Dec 21, 2014)

wuzznuubi said:


> Haven't determined anything reliable, I just keep downloading a program until I get a clean transport stream with no sync loss.


I know on your VideoRedo posts that you said you tried so many different things that you don't really know what resulted in the Tivo file transferring without any glitches, but have you tried transferring the same recording multiple times with the same configuration and it produces different results(i.e. one the recording transfers glitch-free while another time it doesn't)?


----------



## wuzznuubi (Jan 17, 2013)

nazopo said:


> I know on your VideoRedo posts that you said you tried so many different things that you don't really know what resulted in the Tivo file transferring without any glitches, but have you tried transferring the same recording multiple times with the same configuration and it produces different results(i.e. one the recording transfers glitch-free while another time it doesn't)?


That's my next experiment. I have a one hour H.264 recording scheduled tonight. I'm going to download it a couple minutes after it's done recording, then wait until morning and download it again, then again tomorrow afternoon, etc. without messing with anything like delete/recover, standby, changing all channels, etc. I find if you look at the std (console) output of tivolibre using the -d debug flag it will tell you where it finds/recovers the TS packet sync loss which makes it easier to go to that location in a hex editor while examining the encrypted TiVo file you downloaded. If the TS sync loss in the encrypted TiVo files occur in different locations in different downloads of the same program from the TiVo, that pretty much points to the TiVo doing it as far as I'm concerned.


----------



## nazopo (Dec 21, 2014)

wuzznuubi said:


> That's my next experiment. I have a one hour H.264 recording scheduled tonight. I'm going to download it a couple minutes after it's done recording, then wait until morning and download it again, then again tomorrow afternoon, etc. without messing with anything like delete/recover, standby, changing all channels, etc. I find if you look at the std (console) output of tivolibre using the -d debug flag it will tell you where it finds/recovers the TS packet sync loss which makes it easier to go to that location in a hex editor while examining the encrypted TiVo file you downloaded. If the TS sync loss in the encrypted TiVo files occur in different locations in different downloads of the same program from the TiVo, that pretty much points to the TiVo doing it as far as I'm concerned.


This sounds like an interesting experiment. I'm eager to hear what the outcome will be.


----------



## wuzznuubi (Jan 17, 2013)

wuzznuubi said:


> That's my next experiment. I have a one hour H.264 recording scheduled tonight. I'm going to download it a couple minutes after it's done recording, then wait until morning and download it again, then again tomorrow afternoon, etc. without messing with anything like delete/recover, standby, changing all channels, etc. I find if you look at the std (console) output of tivolibre using the -d debug flag it will tell you where it finds/recovers the TS packet sync loss which makes it easier to go to that location in a hex editor while examining the encrypted TiVo file you downloaded. If the TS sync loss in the encrypted TiVo files occur in different locations in different downloads of the same program from the TiVo, that pretty much points to the TiVo doing it as far as I'm concerned.


As far as I'm concerned the Tivo is creating TS Sync loss at different points in the encrypted TS streams it delivers when sending those in request for a TS download of a program on the TiVo. Only the TiVo engineers know why (if they're still around). Different areas (I'll call TS zones) of the same program downloaded from the TiVo via TS ttg show up with TS packet loss due to TS sync lost in the different unencrypted TS TiVo downloads. Each of the downloads I made experienced TS sync loss at specific byte offsets in the TS stream I'm calling TS zones. Some of these downloads drop packets in zone 1, some in zone 2, some in zone 3, some in zone 1 & 2, 1 & 3, 1, 2 & 3, etc.

Latest program I downloaded from my TiVo, 18 times, represented all combinations of the three distinct TS zones of TS packet loss and one that was error free.

For me, case is closed after experimenting with 18 downloads of the same recorded program. Oh, wait, should I try again with the 19th download?


----------



## nazopo (Dec 21, 2014)

wuzznuubi said:


> As far as I'm concerned the Tivo is creating TS Sync loss at different points in the encrypted TS streams it delivers when sending those in request for a TS download of a program on the TiVo. Only the TiVo engineers know why (if they're still around). Different areas (I'll call TS zones) of the same program downloaded from the TiVo via TS ttg show up with TS packet loss due to TS sync lost in the different unencrypted TS TiVo downloads. Each of the downloads I made experienced TS sync loss at specific byte offsets in the TS stream I'm calling TS zones. Some of these downloads drop packets in zone 1, some in zone 2, some in zone 3, some in zone 1 & 2, 1 & 3, 1, 2 & 3, etc.
> 
> Latest program I downloaded from my TiVo, 18 times, represented all combinations of the three distinct TS zones of TS packet loss and one that was error free.
> 
> For me, case is closed after experimenting with 18 downloads of the same recorded program. Oh, wait, should I try again with the 19th download?


Interesting analysis. It seems perhaps that a sync loss-free download is something the user has very little control over. I know that since your provider seems to only use H.264 ts streams that you would not know the answer, but for the ps streams that quit transferring early is it possible that sync loss would always occur at those points where a ps transfer cuts off early in the ts transfer of the same recording or can it possible that no sync loss would occur at those ps stream cutoff points?


----------



## wuzznuubi (Jan 17, 2013)

Just finished downloading another H.264 recording. Took 21 downloads before I got a glitch free TS. The first 20 downloads had TS Sync loss in three different areas of the file, some downloads in any one of the areas, some in two of the three areas and others in all three areas. Only the 21st download had no TS Sync loss. Not sure about MPEG2 TS experiencing Sync loss as I only have one station I record that's still MPEG2 and I use PS downloads to avoid this issue.


----------



## nazopo (Dec 21, 2014)

wuzznuubi said:


> Just finished downloading another H.264 recording. Took 21 downloads before I got a glitch free TS. The first 20 downloads had TS Sync loss in three different areas of the file, some downloads in any one of the areas, some in two of the three areas and others in all three areas. Only the 21st download had no TS Sync loss. Not sure about MPEG2 TS experiencing Sync loss as I only have one station I record that's still MPEG2 and I use PS downloads to avoid this issue.


Luckily for me my provider still broadcasts in MPEG2 so I usually go the PS route, but I still occasionally get those incomplete downloads at which point I have to go the TS route to download the entire recording. I'll have to try to develop a program that will download a recording in TS and check for sync loss in the tivolibre output, and continue to download the same recording again until there are no issues. I may be going over my head with this. Let's hope not.


----------



## wuzznuubi (Jan 17, 2013)

wuzznuubi said:


> Just finished downloading another H.264 recording. Took 21 downloads before I got a glitch free TS. The first 20 downloads had TS Sync loss in three different areas of the file, some downloads in any one of the areas, some in two of the three areas and others in all three areas. Only the 21st download had no TS Sync loss. Not sure about MPEG2 TS experiencing Sync loss as I only have one station I record that's still MPEG2 and I use PS downloads to avoid this issue.


Problem appears to be fixed (crossing fingers). Not sure which one of the suggestions solved it, but don't care at this point. Friend suggested that I remove my Roamio hard drive and run it through SpinRite level two diagnostics and put it back which I did. Also that I shouldn't be recording/transferring/permanently deleting recordings so often and to just delete the recordings after successful recording/transfer and leave them on the TiVo for it to delete them as that would ensure the recordings were spread out on the hard drive and then would not keep writing to the same area on the HDD since little is known about the WD AV drives used and how the drive and the TiVo uses error correction/recovery (quoting her;-).

Happy to say I've had 23 straight H.264 recordings transferred as TS without a single sync loss since that time. I think she may have a point.


----------



## nazopo (Dec 21, 2014)

wuzznuubi said:


> Problem appears to be fixed (crossing fingers). Not sure which one of the suggestions solved it, but don't care at this point. Friend suggested that I remove my Roamio hard drive and run it through SpinRite level two diagnostics and put it back which I did. Also that I shouldn't be recording/transferring/permanently deleting recordings so often and to just delete the recordings after successful recording/transfer and leave them on the TiVo for it to delete them as that would ensure the recordings were spread out on the hard drive and then would not keep writing to the same area on the HDD since little is known about the WD AV drives used and how the drive and the TiVo uses error correction/recovery (quoting her;-).
> 
> Happy to say I've had 23 straight H.264 recordings transferred as TS without a single sync loss since that time. I think she may have a point.


So basically after running SpinRite on the drive, I should avoid permanently deleting recordings after transferring them? Basically the problem isn't transferring them so often but rather the fact that you were permanently deleting them after transferring them if I have interpreted that correctly. I never permanently delete recordings after transferring them so maybe I just need to run SpinRite to perhaps have the issue go away on my end?

Also how long would you say it would take a 2tb drive to run through SpinRite given how long it took to run what I presume is a 3tb WD AV drive in your Roamio?


----------



## nazopo (Dec 21, 2014)

wuzznuubi said:


> I find if you look at the std (console) output of tivolibre using the -d debug flag it will tell you where it finds/recovers the TS packet sync loss which makes it easier to go to that location in a hex editor while examining the encrypted TiVo file you downloaded.


This block from the console would be an example of tivolibre indicating sync loss from a missing TS packet header?

```
02:03:19.753 [main] WARN  c.s.tivolibre.TivoDecoder - Invalid TS packet header for packet 10428684
02:03:19.753 [main] WARN  c.s.tivolibre.TivoDecoder - TransportStream appears to be corrupt, cannot find sync bytes
02:03:19.763 [main] DEBUG c.s.tivolibre.TivoDecoder - Starting value for resumeDecryptionAtByte: 0x74dca738
02:03:19.773 [main] DEBUG c.s.tivolibre.TivoDecoder - Resume decryption at: 0x75800000
02:03:19.903 [main] INFO  c.s.tivolibre.TivoDecoder - Re-synched at packet 10428685 (byte 0x74dca738)
```


----------



## wuzznuubi (Jan 17, 2013)

nazopo said:


> Also how long would you say it would take a 2tb drive to run through SpinRite given how long it took to run what I presume is a 3tb WD AV drive in your Roamio?


Stock 500GB WD AV drive in my four tuner Roamio. Not sure how long it took as I started the SpinRite run late at night when I had no recordings scheduled and it was finished by morning. I'd guess no more than 8 hours, probably less. You could always contact their support and ask them.


----------



## wuzznuubi (Jan 17, 2013)

nazopo said:


> This block from the console would be an example of tivolibre indicating sync loss from a missing TS packet header?
> 
> ```
> 02:03:19.753 [main] WARN  c.s.tivolibre.TivoDecoder - Invalid TS packet header for packet 10428684
> ...


Yes. The byte value given above for "Re-synched at packet 10428685 (byte 0x74dca738)" is the byte offest within the Transport Stream (exclusive of the tivo header and three chunks) that the lost sync was re-established.
Once you get a download where tivolibre shows no sync loss you have a good download with no lost packets.


----------



## wuzznuubi (Jan 17, 2013)

wuzznuubi said:


> Problem appears to be fixed (crossing fingers). Not sure which one of the suggestions solved it, but don't care at this point. Friend suggested that I remove my Roamio hard drive and run it through SpinRite level two diagnostics and put it back which I did. Also that I shouldn't be recording/transferring/permanently deleting recordings so often and to just delete the recordings after successful recording/transfer and leave them on the TiVo for it to delete them as that would ensure the recordings were spread out on the hard drive and then would not keep writing to the same area on the HDD since little is known about the WD AV drives used and how the drive and the TiVo uses error correction/recovery (quoting her;-).
> 
> Happy to say I've had 23 straight H.264 recordings transferred as TS without a single sync loss since that time. I think she may have a point.


So far I've only had one recording that downloaded with a sync loss in one area and the second download of the same recording was without sync loss. I'm checking all of my H.264 TS downloads using tivolibre with the -d debug switch which reports if there are any sync losses. That's a lot easier than examining every one in a hex editor!


----------



## wuzznuubi (Jan 17, 2013)

Curious if anyone else is experiencing this who doesn't have a stock, out of the box TiVo drive (WD AV?) in a Roamio or Bolt? Like a WD Green or Blue drive, or whatever you are putting in them? Maybe something to do with how the TiVo and the original AV drive handle streaming protocols/remap of bad sectors/error recovery?

tivolibre is the java program which I use to see if the TiVo recording downloaded in Fast Transfer mode (a.k.a. TransportStream) has experienced sync loss which results in glitches/loss of program material.


----------



## lew (Mar 12, 2002)

wuzznuubi said:


> Curious if anyone else is experiencing this who doesn't have a stock, out of the box TiVo drive (WD AV?) in a Roamio or Bolt? Like a WD Green or Blue drive, or whatever you are putting in them? Maybe something to do with how the TiVo and the original AV drive handle streaming protocols/remap of bad sectors/error recovery?
> 
> tivolibre is the java program which I use to see if the TiVo recording downloaded in Fast Transfer mode (a.k.a. TransportStream) has experienced sync loss which results in glitches/loss of program material.


I have no problem with ts transfers. I'm using WD AV drives.

You might ask if posters having issues are using wireless, are using a laptop and if they have plenty if space on their computer.


----------



## wuzznuubi (Jan 17, 2013)

lew said:


> I have no problem with ts transfers.


Are you sure? If so, you're one of the lucky ones.


lew said:


> I'm using WD AV drives.


Which Tivo models do you have and are they original drives?


lew said:


> You might ask if posters having issues are using wireless, are using a laptop and if they have plenty if space on their computer.


Not sure why I would ask that as it has nothing to do with the TiVo sending you a TS file with TS Sync loss? With a https connection and using TCP protocol I doubt any of that matters. But just for grins - WIRED/PC/PLENTY. And the OP stated three different TiVo's.


----------



## nazopo (Dec 21, 2014)

wuzznuubi said:


> With a https connection and using TCP protocol I doubt any of that matters. But just for grins - WIRED/PC/PLENTY. And the OP stated three different TiVo's.


I was about to say the same thing about it not being seemingly plausible that the connection would be the issue given the TCP protocol was being used.


----------



## wuzznuubi (Jan 17, 2013)

wuzznuubi said:


> Problem appears to be fixed (crossing fingers). Not sure which one of the suggestions solved it, but don't care at this point. Friend suggested that I remove my Roamio hard drive and run it through SpinRite level two diagnostics and put it back which I did. Also that I shouldn't be recording/transferring/permanently deleting recordings so often and to just delete the recordings after successful recording/transfer and leave them on the TiVo for it to delete them as that would ensure the recordings were spread out on the hard drive and then would not keep writing to the same area on the HDD since little is known about the WD AV drives used and how the drive and the TiVo uses error correction/recovery (quoting her;-).
> 
> Happy to say I've had 23 straight H.264 recordings transferred as TS without a single sync loss since that time. I think she may have a point.


Thanks Jill for the advice, but it didn't last. After 12 days I find the problem has slowly returned. Today, some recordings download without issue, yet others take two to 18 download retries to resolve. I find it very strange that I'm the only one that has this issue. No other TiVo owner experiences this problem?


----------



## nazopo (Dec 21, 2014)

wuzznuubi said:


> Thanks Jill for the advice, but it didn't last. After 12 days I find the problem has slowly returned. Today, some recordings download without issue, yet others take two to 18 download retries to resolve. I find it very strange that I'm the only one that has this issue. No other TiVo owner experiences this problem? NOT FIXED YET TIVO!!!


Do you reschedule the download of a recording by hand or do you have some other way of doing so? If you're doing it by hand then I certainly commend you for being so persistent with a stubborn recording.


----------



## lpwcomp (May 6, 2002)

wuzznuubi said:


> Thanks Jill for the advice, but it didn't last. After 12 days I find the problem has slowly returned. Today, some recordings download without issue, yet others take two to 18 download retries to resolve. I find it very strange that I'm the only one that has this issue. No other TiVo owner experiences this problem? NOT FIXED YET TIVO!!!


I doubt TiVo is even working on a fix since it involves a feature that they are no longer actively supporting. If they ever do release a "fix", it may consist of a complete removal of TiVo to PC transfer capability.


----------



## wuzznuubi (Jan 17, 2013)

nazopo said:


> Do you reschedule the download of a recording by hand or do you have some other way of doing so? If you're doing it by hand then I certainly commend you for being so persistent with a stubborn recording.


Got to the point that archiving glitched TS streams was so unreliable I started doing it manually. Not worth the trouble unless you REALLY want to get that one and only recording that will never be shown again, but then you're at your TiVo's mercy. Time to move on, as TiVo says "Bah Humbug, who records anymore, enjoy the new Streaming features we gave you and be silently compliant"


----------



## wuzznuubi (Jan 17, 2013)

lpwcomp said:


> I doubt TiVo is even working on a fix since it involves a feature that they are no longer actively supporting. If they ever do release a "fix", it may consist of a complete removal of TiVo to PC transfer capability.


Came to the same conclusion. It's like investing in startups that never get off the ground and seem to lose sight of their original (kickstarter/etc.) goal/momentum and then succumb to the CORP AMERICA mantra of offshoring and...and..and. BYGONE, recycle, repeat AD NAUSEAM.
SIGH


----------



## wuzznuubi (Jan 17, 2013)

While waiting for TiVo to fix the .ts SYNC problem (yawn, if ever). If you really want to archive that recording that keeps giving you .ts SYNC errors, you can do a workaround. D/L the recording multiple times and stitch them together with something like VideoReDo using joiner.


----------



## lpwcomp (May 6, 2002)

wuzznuubi said:


> While waiting for TiVo to fix the .ts SYNC problem (yawn, if ever). If you really want to archive that recording that keeps giving you .ts SYNC errors, you can do a workaround. D/L the recording multiple times and stitch them together with something like VideoReDo using joiner.


You're assuming that multiple downloads will result in problems being in different locations in the recording.


----------



## nazopo (Dec 21, 2014)

lpwcomp said:


> You're assuming that multiple downloads will result in problems being in different locations in the recording.


wuzznuubi has found that to be the case before.


----------



## wuzznuubi (Jan 17, 2013)

lpwcomp said:


> You're assuming that multiple downloads will result in problems being in different locations in the recording.


I'm not assuming. It's a fact here with me. YMMV and you can take it or leave it as U C fit... was just trying to spread some knowledge, but some people just don't like that. So B IT


----------



## lpwcomp (May 6, 2002)

wuzznuubi said:


> I'm not assuming. It's a fact here with me. YMMV and you can take it or leave it as U C fit... was just trying to spread some knowledge, but some people just don't like that. So B IT


----------



## wuzznuubi (Jan 17, 2013)

Only thing in common is that both the Roamio and the Premiere are running Software v20.6.1.RC14.

Roamio (TCD846500) is Cable only.
Premiere (TCD746320) is OTA and Cable.

Duplicated the issue on the two tuner Premiere setup with both OTA and Cable and it happens with OTA and Cable recordings, mpeg2 or H.264, 720p or 1080i doesn't matter. Only thing I haven't tried is recording and downloading an SD program which I'll do tomorrow on both the Roamio and Premiere. - Update: Haven't been able to get any SD recordings to fail


----------



## wuzznuubi (Jan 17, 2013)

lpwcomp said:


> I doubt TiVo is even working on a fix since it involves a feature that they are no longer actively supporting. If they ever do release a "fix", it may consist of a complete removal of TiVo to PC transfer capability.


Since you claim TiVo Desktop is no longer being supported by TiVo, and I can't determine that you're an official TiVo rep, could you point us to that official announcement from TiVo, or provide that info that TiVo sent you claiming such? Strange that only a few TiVo owners have (or I think recognize) this issue. I'm sure most call in to the Friendly TiVo 800 Support# and get bamboozled by the L1 Support reps and just move on with their lives if they're not convinced into buying a BOLT. Thanks.


----------



## lpwcomp (May 6, 2002)

wuzznuubi said:


> Since you claim TiVo Desktop is no longer being supported by TiVo, and I can't determine that you're an official TiVo rep, could you point us to that official announcement from TiVo, or provide that info that TiVo sent you claiming such? Strange that only a few TiVo owners have (or I think recognize) this issue. I'm sure most call in to the Friendly TiVo 800 Support# and get bamboozled by the L1 Support reps and just move on with their lives if they're not convinced into buying a BOLT. Thanks.


I don't tend t0 deal with jerks who imply I'm a liar.


----------



## wuzznuubi (Jan 17, 2013)

lpwcomp said:


> I don't tend t0 deal with jerks who imply I'm a liar.





lpwcomp said:


> I doubt TiVo is even working on a fix since it involves a feature that they are no longer actively supporting. If they ever do release a "fix", it may consist of a complete removal of TiVo to PC transfer capability.


Wasn't implying that you're a liar. I was asking if you could point to anything on the TiVo website, or a letter or some kind of documentation that they're no longer supporting TiVo Desktop.

Per their support site, (this page was last updated on 06/14/2016)



> As of fall 2015, TiVo Desktop is no longer available for purchase. However, TiVo will continue to support your existing Desktop installation. If you still have your product key, you can use the installation instructions below to use Desktop.


----------



## lew (Mar 12, 2002)

Someone has issues reading. The questionus if tivo is * actively * supporting tivo desktop. Tivo released updated software so Tivo HD customers can record mp4 channels. Tivo didn't make the changes required to enable ttgo transfers for those channels. You can't easily find tivo desktop software on the site.

Does anyone have anything to suggest active support.

Edited to add on April 13 a thread was started on the tivo boards stating auto transfers to tivo HD no longer work. Last post from tivo was july 15. Still working on it.

In what world is that active support? Tivo has effectively discontinued tivo desktop.


----------



## wuzznuubi (Jan 17, 2013)

lew said:


> Someone has issues reading. The questionus if tivo is * actively * supporting tivo desktop. Tivo released updated software so Tivo HD customers can record mp4 channels. Tivo didn't make the changes required to enable ttgo transfers for those channels. You can't easily find tivo desktop software on the site.
> 
> Does anyone have anything to suggest active support.
> 
> ...


OK, given those facts the Roamio and Mini are no longer *actively* supported. Would you agree?


----------



## lpwcomp (May 6, 2002)

wuzznuubi said:


> OK, given those facts the Roamio and Mini are no longer *actively* supported. Would you agree?


That has got to be one of the most ridiculous statements ever posted on this board, and that is saying a lot.


----------



## lew (Mar 12, 2002)

lpwcomp said:


> that has got to be one of the most ridiculous statements ever posted on this board, and that is saying a lot.


+1


----------



## wuzznuubi (Jan 17, 2013)

lew said:


> Someone has issues reading. The questionus if tivo is * actively * supporting tivo desktop. Tivo released updated software so Tivo HD customers can record mp4 channels. Tivo didn't make the changes required to enable ttgo transfers for those channels. You can't easily find tivo desktop software on the site.
> 
> Does anyone have anything to suggest active support.
> 
> ...


Got it. A three month lack of resolution


> on the tivo boards


 means it's not actively being supported. How about this issue going back to February, it's only been five months, or doesn't that qualify as not being "actively supported", or the Comcast mp4 issues with Roamios and Minis, or...? I have TiVo case tickets that are still unresolved but are "being investigated". I guess what you're saying is TiVo is lying to me?


----------



## lpwcomp (May 6, 2002)

The last thing that would count as active support for TTG was the fix to the "expired cookie" problem. In contrast, the Mini, Premiere, Roamio, and Bolt all had their s/w updated a month ago.

They are actively working on a fix to the problem affecting _*some*_ H.264 channels in _*some*_ Comcast locales.

I don't think TiVo is exactly lying to you, just that "support" probably consists solely of TD setup and configuration help.

TTG is a dead product. The Stream and TiVo app are its replacement. My fear is that someday (and it may not be that far off), they will disable the TTG s/w on the TiVo side.


----------



## lew (Mar 12, 2002)

lpwcomp said:


> TTG is a dead product. The Stream and TiVo app are its replacement. My fear is that someday (and it may not be that far off), they will disable the TTG s/w on the TiVo side.


Western Digital used TTG on some of their cloud servers. WD (used?) to supply the external drive for tivos. I'm not sure when WD discontinued that feature. I don't know how long tivo is required, or thinks they should, continue TTG for those customers.

No question stream and the app are replacements. TTG wasn't realty marketed as an archival product. Customers were expected to download the show to their computer then burn on a DVD so it could be viewed on a portable DVD player when traveling. Also transcode to a portable media format.

edited to add My memory is the WD cloud servers were still supporting TTG at the time the expired cookie issue was fixed with a software update.


----------



## wuzznuubi (Jan 17, 2013)

Been investigating AND using three other DVR solutions SINCE TiVo-Rovi ditched us. They record CABLE and OTA with some issues, but they all let me archive my recordings glitch free and TiVo does not. I still have a couple TiVo's as backup, but since ROVI took over the Guide data, it has been abysmal to say the least in TiVoLand.
I wish you all the best and wish TiVo was once again what it was, but I'm not holding my breath.


----------



## yawitz (Apr 2, 2007)

Can you share the DVR solutions you found that support archiving shows? Also interested to know who provides the guide data for those solutions.


----------



## markmarz (Feb 3, 2002)

yawitz said:


> Can you share the DVR solutions you found that support archiving shows? Also interested to know who provides the guide data for those solutions.


I second that motion! I've been using TiVo since early days but I'm tired of the .ts glitches. I won't hold my breath for a reply.


----------



## alleybj (Dec 6, 2000)

Is it possible that Comcast's conversion to Mpeg4 has inadvertently "fixed" the issues with transferring .TS files? My rather unscientific random sampling makes it appear that movies are now transferring without the glitches I was previously seeing.


----------



## reneg (Jun 19, 2002)

alleybj said:


> Is it possible that Comcast's conversion to Mpeg4 has inadvertently "fixed" the issues with transferring .TS files? My rather unscientific random sampling makes it appear that movies are now transferring without the glitches I was previously seeing.


Anything is possible but I think it is more probably that your connection through the Comcast infrastructure has improved somehow.


----------



## wuzznuubi (Jan 17, 2013)

Just wanted to drop by and provide an update since I don't rely on TiVo much anymore, but still use one.

I've decided to use the Plex DVR solution with some SiliconDust HDHR Prime Network tuners and Motorola CableCards from XFINITY/Comcast.

It's not a panaceia as Plex has it's own issues which I'm still working through, but at least I get reliable recordings of HD MPEG2 and H.264 programs with no glitches other than those intoduced by my cable co. Plex DVR uses the old TiVo Gracenote/Tribune Media Company EPG.

I still use my Roamio as a backup for shows I don't want to miss, but have only relied on it a couple times in the last 3 months to catch those recordings that Plex DVR cut short because of a bug in their PMS which has since been fixed.

I can still download glitch free MPEG2 and H.264 recordings as Transport Streams from the Roamio without TiVo introduced ts sync errors and dropouts by doing a robo-download of the same program anywhere from 5 to 20 times back-2-back using kmttg. I only use PS downloads for the MPEG2 and TS downloads for the H.264 recordings.


----------



## ClearToLand (Jul 10, 2001)

wuzznuubi said:


> Just wanted to drop by and provide an update since I don't rely on TiVo much anymore, but still use one.
> 
> I've decided to use the Plex DVR solution with some SiliconDust HDHR Prime Network tuners and Motorola CableCards from XFINITY/Comcast.
> 
> ...



"*Robo-download*"? I've used RoboCopy - are you using a BATCH file or a VB script or? (i.e. Would you please post / attach it?)
What '_indicator(s)_' are you using to determine a SUCCESSFUL TS transfer (i.e. why sometimes only 5 tries, not always 20)?
Thanks!


----------



## wuzznuubi (Jan 17, 2013)

I would suggest you start reading this thread from post # 16.


ClearToLand said:


> "*Robo-download*"? I've used RoboCopy - are you using a BATCH file or a VB script or? (i.e. Would you please post / attach it?)
> What '_indicator(s)_' are you using to determine a SUCCESSFUL TS transfer (i.e. why sometimes only 5 tries, not always 20)?
> Thanks!


I would politely suggest that you start reading this thread from post #16. Like I said, I just dropped in as a courtesy, I no longer need to use this forum. Not meant as disrespect to anyone here, just that I no longer need to use a TiVo and have no interest in spending my time on it or here since Rovi killed it, IMHO...


----------



## Dan203 (Apr 17, 2000)

wuzznuubi said:


> I've decided to use the Plex DVR solution with some SiliconDust HDHR Prime Network tuners and Motorola CableCards from XFINITY/Comcast.


Does the Plex DVR support encrypted channels? I thought it only worked with the OTA HDHR tuners?


----------



## ClearToLand (Jul 10, 2001)

wuzznuubi said:


> *I would suggest you start reading this thread from post # 16.
> *
> 
> 
> ...


*Whoa!* Drop the attitude man.  [And you're stuttering...  ]

I completely understand your opinion of the POS "TS Transfer Format". I've been following *YOUR* thread since its inception. Thanks for sharing your success with SiliconDust HDHR tuners and PLEX. :thumbsup:

If *YOU* had read *MY* previous posts, in other threads (the TiVo World doesn't revolve around you  ), you would have seen that I too have been greatly disappointed with a 50% (probably closer to 66%) failure of TS transfers-to-PC to successfully completing a PyTiVo transfer-to-TiVo. I ended up decrypting them and viewing them WITH glitches, WITHOUT metadata. 

So, ignoring your initial rudeness, I reiterate my previous two questions, expecting, this time, a polite and logical reply...

Thanks in advance.


----------



## lpwcomp (May 6, 2002)

ClearToLand said:


> *Whoa!* Drop the attitude man.  [And you're stuttering...  ]
> 
> I completely understand your opinion of the POS "TS Transfer Format". I've been following *YOUR* thread since its inception. Thanks for sharing your success with SiliconDust HDHR tuners and PLEX. :thumbsup:
> 
> ...


There is absolutely no reason to view them "without metadata".


----------



## wuzznuubi (Jan 17, 2013)

OK. I guess this is as close as I'll ever get to posting a polite and logical reply according to ClearToLand here on this Forum.

Do your own homework as I have done mine to find what fits your situation. You are indivudually unique. There is not a one fits all solution. My solution fits me, but maybe not yours.

If you're interested in Plex, check it out, if not ignore... I'm not here to be a Plex fanboy, just wanted to be supportive of this forum and post an update to my situation *with a TiVo*.

I don't watch or record PROTECTED (DRM) CHANNELS and probably never will (that's my individual choice). I'm only interested in recording locally (to NAS) the recordings I want to archive, not the daily news or other shows that I want to watch and then delete.

TiVo was once GREAT, but is going downhill as far as I'm concerned and that's why I left them (ROVI). Everyone is entitled to their own opinion based on their situation.

Edited to remove erroneous report: Plex DVR does support recording PREMIUM CHANNELS (in some cases FOX and ABC where they're protected), BUT ONLY ON WINDOWS 10 PLATFORMS TODAY.
It supports recording encrypted content that is decrypted by the Cabe Card but does not yet support DRM Copy Once or Copy Never recording yet.


----------



## ClearToLand (Jul 10, 2001)

lpwcomp said:


> There is absolutely no reason to view them "without metadata".


You know, I haven't used PyTiVo for months, and I was a 'kmttg / PyTiVo novice / greenhorn' at that time. I probably could have used the FILES tab and also gotten the metadata out of the original TiVo file (I was just so surprised / disappointed / pissed off, that I didn't think of it back then - my *ONLY* thought was 'How can I view this show?" ) - next time!


----------



## ClearToLand (Jul 10, 2001)

wuzznuubi said:


> OK. I guess this is as close as I'll ever get to posting a polite and logical reply according to ClearToLand here on this Forum.
> 
> Do your own homework as I have done mine to find what fits your situation. You are indivudually unique. There is not a one fits all solution. My solution fits me, but maybe not yours.
> 
> ...


Thanks, once again, for sharing more details on your PLEX solution. But, I asked you about your "*Roamio / kmttg / 5-20 tries*" solution. Please re-read my previous post with the portion of your post QUOTE'd in Red.


----------



## wuzznuubi (Jan 17, 2013)

ClearToLand said:


> Thanks, once again, for sharing more details on your PLEX solution. But, I asked you about your "*Roamio / kmttg / 5-20 tries*" solution. Please re-read my previous post with the portion of your post QUOTE'd in Red.


Start Here;
Java port of TivoDecode
OR
GitHub - fflewddur/tivolibre: Java app and library for decoding TiVo files to standard MPEG files.
If you have problems using Java code or scripting it in your OS, ask questions or reread my posts.

BTW - I only use tivolibre to check that I get a pure TS stream from the TiVo file I downloaded from the TiVo without any TS Sync Loss, I don't use it to decode the TiVo file. It beats opening up the downloaded .TiVo file in a hex editor and manipulating/searching it to find the TS Sync loss. I use VRD TVS5 to decrypt/QSFix/Convert the .TiVo file.

Nite


----------



## ClearToLand (Jul 10, 2001)

wuzznuubi said:


> Just wanted to drop by and provide an update since I don't rely on TiVo much anymore, but still use one...
> 
> ...I still use my Roamio as a backup for shows I don't want to miss, but have only relied on it a couple times in the last 3 months to catch those recordings that Plex DVR cut short because of a bug in their PMS which has since been fixed.
> 
> *I can still download glitch free MPEG2 and H.264 recordings as Transport Streams from the Roamio without TiVo introduced ts sync errors and dropouts by doing a robo-download of the same program anywhere from 5 to 20 times back-2-back using kmttg*. I only use PS downloads for the MPEG2 and TS downloads for the H.264 recordings.





wuzznuubi said:


> Start Here;
> Java port of TivoDecode
> OR
> GitHub - fflewddur/tivolibre: Java app and library for decoding TiVo files to standard MPEG files.
> ...


You originally stated that you were doing 'robo-downloads... ...using kmttg' (be sure to expand the top QUOTE of your original post above so that you can see my highlighting in red) and now you're providing LINKs for TiVoDecode and tivolibre.

Where's your kmttg solution?


----------



## wuzznuubi (Jan 17, 2013)

ClearToLand said:


> You originally stated that you were doing 'robo-downloads... ...using kmttg' (be sure to expand the top QUOTE of your original post above so that you can see my highlighting in red) and now you're providing LINKs for TiVoDecode and tivolibre.
> 
> Where's your kmttg solution?


 Either ask the TiVoDesktop/kmttg/pytivo devs, or use a script of your choice depending on your OS of choice!

I got to the point where it was too manually intensive to do it for every recording I wanted to archive and that's why I moved to the Plex DVR with the SiliconDust-HDHR PRIME solution.

For my only remaining TIVO, a Roamio, I just do a *kmttg custom command as post processing* to rename the downloaded .TiVo files to unique names by appending _01 to _20 to the filenames while ROBODOWNLOAD the programs from the TiVo by selecting them in kmttg (ROBODOWNLOADING=keep manually selecting AND re-downloading them 20 times each since I don't know of any other way in kmttg), then drag-n-drop the downloaded .TiVo files to a custom cmd script I wrote that runs them through tivolibre with the debug option turned on looking for TS Sync Loss and if NONE that .Tivo file is good. Like I said, too labor intensive for normal archiving. If you have an easier way, go for it as I hardly use it anymore and I really think it's a losing battle with ROVI, KING of Copy Protection since the VCR days (if you research their history and previous names before they became TiVo).


----------



## Dan203 (Apr 17, 2000)

wuzznuubi said:


> Plex DVR does support recording PREMIUM CHANNELS (in some cases FOX and ABC where they're protected), BUT ONLY ON WINDOWS 10 PLATFORMS TODAY. If you're interested in Plex, check it out, if not ignore... I'm not here to be a Plex fanboy, just wanted to be supportive of this forum and post an update to my situation *with a TiVo*.
> 
> I don't watch or record PROTECTED (DRM) CHANNELS and probably never will (that's my individual choice). I'm only interested in recording locally (to NAS) the recordings I want to archive, not the daily news or other shows that I want to watch and then delete.


I was just curious. I thought I had read that it only worked with the OTA version of the HD HomeRun, but it's been a while since I looked into it.

Does it record just standard .ts files? Or do they have some sort of header with metadata? Any chance you could record a sample file for me and upload it to the VideoReDo FTP so we can see if it's a format we could support for editing?

Upload Files to VideoReDo


----------



## wuzznuubi (Jan 17, 2013)

Dan203 said:


> I was just curious. I thought I had read that it only worked with the OTA version of the HD HomeRun, but it's been a while since I looked into it.
> 
> Does it record just standard .ts files? Or do they have some sort of header with metadata? Any chance you could record a sample file for me and upload it to the VideoReDo FTP so we can see if it's a format we could support for editing?


If you send me a key to VRD TVS Pro, I'll tell 'ya:mask:
No, really, it's standard MPEG2 or H.264 TS delivered via http or udp direct from the SD HDHR Prime Network Tuner that can be viewed/recorded via commandline utility/VLC, etc. unless you select a DRM channel, then DTCP-IP, Protected (DRM) channel requirements: Xbox One, Windows 10 Phone, Windows 10 PCs (Desktops, Laptops, Tablets, HDMI-sticks, etc)


----------



## Dan203 (Apr 17, 2000)

So Plex doesn't attach any metadata to the file itself? I know the HDHR DVR does attach some sort of metadata to the file but it's still in beta (a year and a half later) so I haven't bothered to look into that yet. We like to make sure we support all the DVRs out there, since HTPC users and TiVo users make up the majority of or US based consumers.


----------



## wuzznuubi (Jan 17, 2013)

Dan203 said:


> So Plex doesn't attach any metadata to the file itself?


NO. All metadata, posters, etc. are kept in separate databases in Plex. The recording from the SD-HDHR Prime is a normal H.264 or MPEG2 transport stream which you could easily receive from the PRIME itself using a DLNA agent, a commandline utility, VLC media player, etc.


----------



## Dan203 (Apr 17, 2000)

Cool, then VideoReDo should work fine then.


----------



## Dan203 (Apr 17, 2000)

So are you guys actually seeing glitches in different places every time you download a show? I have a show which has no visible glitch on the TiVo itself, but when I download it I get an error at the same place every single time. I downloaded it 5 times and ran tivolibre with the debug option on all 5 and it reported the same two errors, at the exact same byte, in all 5 recordings. I was going to play around with it and see if I could patch together a clean file by merging multiple files, but I always get the glitch in the same two places so that doesn't even seem to be an option.

Has anyone else ran a similar test and found that the glitch(es) actually moved to a different location on subsequent downloads?


----------



## Dan203 (Apr 17, 2000)

I just tried transferring the same show to another TiVo, then I downloaded the show from that TiVo, the glitches are still in the exact same place. So whatever is causing this seems to be in the recording and not just being introduced during the PC transfer.


----------



## wuzznuubi (Jan 17, 2013)

@Dan203


Dan203 said:


> I just tried transferring the same show to another TiVo, then I downloaded the show from that TiVo, the glitches are still in the exact same place. So whatever is causing this seems to be in the recording and not just being introduced during the PC transfer.


If you still have MPEG2 recordings try one of those using both PS and TS downloads and compare the result. You may be able to figure out what is causing the glitch. I really think it's something in the TiVo units code that packs it as a TS TTG where maybe a glitch, dropout, timestamp issue, counter wrap or something messes them up. It was introduced aroung Feb., 2016 in both my Premiere and Roamio downloads using TS. I'll see if I can find the ouput from TiVoLibre on the last couple TiVo TS downloads I robodownloaded. It was always hit-or-miss. Some recordings downloaded without issues first time, others took many (5-35) retries downloading back-back to get one without transport stream sync loss (I know you know that means loss of TS packets/info like cutting out part of a program you're streaming and you can't rewind the stream, or a broadcast, so no way to recover what isn't there).


----------



## Dan203 (Apr 17, 2000)

I'm looking at the error. Multiple downloads have slightly different data in the damaged areas, but the damage always starts at the same byte. The damage is in the .tivo file, so it's not the decrypting causing the error. 

Does anyone know if the TiVo supports partial downloads with an offset byte? I'm wondering if I'll get the error to move if I start from a different point.


----------



## wuzznuubi (Jan 17, 2013)

Dan203 said:


> I'm looking at the error. Multiple downloads have slightly different data in the damaged areas, but the damage always starts at the same byte. The damage is in the .tivo file, so it's not the decrypting causing the error.
> 
> Does anyone know if the TiVo supports partial downloads with an offset byte? I'm wondering if I'll get the error to move if I start from a different point.


I think that it's only supported for PS downloads and not sure if or when that may have been deprecated if it was. Last time I used that function was too long ago to remember.


----------



## Dan203 (Apr 17, 2000)

Is there anything else that causes the error to move or stop? Like putting the tivo into standby? Or setting all the tuners to a channel you don't get?


----------



## wuzznuubi (Jan 17, 2013)

Dan203 said:


> Is there anything else that causes the error to move or stop? Like putting the tivo into standby? Or setting all the tuners to a channel you don't get?


Not that I have found. I will upload my results of a sixteen out of twenty download attempt with one recording where the TS Sync loss is all over the place and the sixteenth download produced a .TiVo TS file w/o sync loss. There is definitely a *moth* in the code that TiVo uses to give us a TTG TS download vs. their old PS download method.


----------



## Dan203 (Apr 17, 2000)

Are 1-15 all different? Or do most/all of then have the error in the same place?


----------



## ej42137 (Feb 16, 2014)

Dan203 said:


> Does anyone know if the TiVo supports partial downloads with an offset byte? I'm wondering if I'll get the error to move if I start from a different point.


You can transfer TiVo to TiVo starting from a paused point.


----------



## Dan203 (Apr 17, 2000)

ej42137 said:


> You can transfer TiVo to TiVo starting from a paused point.


I know that. I meant TiVo to PC. I wanted to see if I started the transfer in a different spot if the error section would change. I messed around with the HTTP headers but it didn't work. It still downloaded the entire file.


----------



## gonzotek (Sep 24, 2004)

kmttg has code to transfer from paused point, here's the wiki on the topic:
kmttg / Wiki / Resume_Downloads
Note, it says it's not available for TS transfers near the bottom of the page. But last time I tried it (I believe it was sometime last year), the process worked for an mpeg2 ps transfer. Not sure if that's helpful to the topic, but it's all I got


----------



## wuzznuubi (Jan 17, 2013)

Dan203 said:


> Are 1-15 all different? Or do most/all of then have the error in the same place?


Below is the output of my script last time I ran it that runs all 20 tivo ts downloads through tivo-libre looking for TS sync loss. Using grep or find I look for the phrase "Re-synched" in the output of tivo-libre and if it doesn't occur for that tivo file, then no TS sync loss. In this case it was the sixteenth download, filename TivoTSdownload_16.TiVo

```
TivoTSdownload_01.TiVo
Re-synched at packet 4568751 (byte 0x333293bc)
TivoTSdownload_02.TiVo
Re-synched at packet 4568751 (byte 0x333293bc)
Re-synched at packet 6091519 (byte 0x444356ec)
TivoTSdownload_03.TiVo
Re-synched at packet 1521712 (byte 0x110dad00)
Re-synched at packet 4568606 (byte 0x333293bc)
Re-synched at packet 7614637 (byte 0x5554f5b0)
TivoTSdownload_04.TiVo
Re-synched at packet 4568751 (byte 0x333293bc)
Re-synched at packet 6091519 (byte 0x444356ec)
TivoTSdownload_05.TiVo
Re-synched at packet 3045398 (byte 0x222047a8)
TivoTSdownload_06.TiVo
Re-synched at packet 1521712 (byte 0x110dad00)
Re-synched at packet 3045253 (byte 0x222047a8)
TivoTSdownload_07.TiVo
Re-synched at packet 3045398 (byte 0x22202eb4)
Re-synched at packet 3045433 (byte 0x22204864)
Re-synched at packet 7614930 (byte 0x5554f5b0)
TivoTSdownload_08.TiVo
Re-synched at packet 4568751 (byte 0x333293bc)
TivoTSdownload_09.TiVo
Re-synched at packet 1521712 (byte 0x110dad00)
Re-synched at packet 3045253 (byte 0x222047a8)
TivoTSdownload_10.TiVo
Re-synched at packet 1521712 (byte 0x110dad00)
Re-synched at packet 3045253 (byte 0x222047a8)
Re-synched at packet 6091488 (byte 0x444356ec)
TivoTSdownload_11.TiVo
Re-synched at packet 4568751 (byte 0x333293bc)
TivoTSdownload_12.TiVo
Re-synched at packet 1521712 (byte 0x110dad00)
Re-synched at packet 3045253 (byte 0x222047a8)
Re-synched at packet 4568573 (byte 0x333293bc)
Re-synched at packet 7614604 (byte 0x5554f5b0)
TivoTSdownload_13.TiVo
Re-synched at packet 6091666 (byte 0x444356ec)
TivoTSdownload_14.TiVo
Re-synched at packet 7614929 (byte 0x5554f5b0)
TivoTSdownload_15.TiVo
Re-synched at packet 3045398 (byte 0x222047a8)
Re-synched at packet 4568718 (byte 0x333293bc)
Re-synched at packet 6091486 (byte 0x444356ec)
Re-synched at packet 7614716 (byte 0x5554f5b0)
TivoTSdownload_16.TiVo
TivoTSdownload_17.TiVo
Re-synched at packet 4568751 (byte 0x333293bc)
TivoTSdownload_18.TiVo
Re-synched at packet 3045398 (byte 0x222047a8)
Re-synched at packet 6091633 (byte 0x444356ec)
TivoTSdownload_19.TiVo
Re-synched at packet 1521712 (byte 0x110dad00)
TivoTSdownload_20.TiVo
Re-synched at packet 3045398 (byte 0x22202eb4)
Re-synched at packet 3045433 (byte 0x22204864)
Re-synched at packet 4568752 (byte 0x33326cd0)
Re-synched at packet 4568758 (byte 0x333272b0)
Re-synched at packet 4568803 (byte 0x333293bc)
```


----------



## ClearToLand (Jul 10, 2001)

wuzznuubi said:


> Below is the output of my script last time I ran it that runs all 20 tivo ts downloads through tivo-libre looking for TS sync loss. Using grep or find I look for the phrase "Re-synched" in the output of tivo-libre and if it doesn't occur for that tivo file, then no TS sync loss. In this case it was the sixteenth download, filename TivoTSdownload_16.TiVo
> 
> ```
> TivoTSdownload_01.TiVo
> ...


*Congratulations on your FIRST LIKE!* :clapping: 

I appreciate all the help / information you've been giving / sharing with @Dan203 in the pursuit of solving this long-standing TiVo TS / 'Fast' Format transfer BUG. :thumbsup:


----------



## Dan203 (Apr 17, 2000)

I could actually make this a feature of pyTivo. I could scan the bytes as they're downloaded looking for ts sync loss. Even without decrypting. TS files are pretty simple. They should have a byte 0x47 every 188 bytes. If you skip ahead 188 bytes and it's not 0x47 then you've lost sync. If it's lost then I could stop the transfer immediately and restart it. Not sure if it would significantly slow down downloads though. I'd have to try it.


----------



## ClearToLand (Jul 10, 2001)

Dan203 said:


> I could actually make this a feature of pyTivo. I could scan the bytes as they're downloaded looking for ts sync loss. Even without decrypting. *TS files are pretty simple. They should have a byte 0x47 every 188 bytes. If you skip ahead 188 bytes and it's not 0x47 then you've lost sync. If it's lost then I could stop the transfer immediately and restart it. Not sure if it would significantly slow down downloads though.* I'd have to try it.


Well, one thing for sure is folks haven't been pleased with being '_surprised_' days / weeks / months later that their .TIVO TS / 'Fast' Format transferred files are corrupt (and have been since Day 1) and now they have to '_mess around_' with decrypting them into .TS files (or buy VideoReDo) in order to view them.

Regarding the potential slowdown, some ideas immediately come to mind.

Make the whole process optional (checkbox on entry line in NPL):
Option 1 (unchecked) - operate the 'old' way (ignore the errors; sounds dumb though...)
Option 2 (checked) - use the new 'Check for TS Sync Loss' routines
Enter a number, 01-99, for maximum retries
If maximum retries reached WITH success
- place the # of retries at the end of the field in NPL
(useful to gauge how much extra time / effort it is consuming)
If maximum retries reached WITHOUT success
- 'highlight' entry somehow in NPL (pick a character to place at the end of the field: !, @, #, etc...)
- post error to log

Just '_thinking out loud..._'


----------



## ClearToLand (Jul 10, 2001)

@Dan203,

BTW, has CNTL-CLICK/SPACE (individual) and / or SHFT-CLICK/SPACE (range) for selecting NPL entries to transfer been considered / implemented?


----------



## ClearToLand (Jul 10, 2001)

@Dan203

Have you begun creating formal 'Release Notes' yet?


----------



## Dan203 (Apr 17, 2000)

ClearToLand said:


> Option 2 (checked) - use the new 'Check for TS Sync Loss' routines
> Enter a number, 01-99, for maximum retries
> If maximum retries reached WITH success
> - place the # of retries at the end of the field in NPL
> ...


I've been thinking about adding some sort of transfer log to the desktop app, so I could just add the retry/sucess/fail stuff to that. The best part about doing it this way would be that I could kill the transfer as soon as a discontinuity was detected, so I could potentially save some time compared to downloading the full file and scanning with tivilibre.



ClearToLand said:


> @Dan203,
> 
> BTW, has CNTL-CLICK/SPACE (individual) and / or SHFT-CLICK/SPACE (range) for selecting NPL entries to transfer been considered / implemented?


I'm considering it, but that list I'm using doesn't support that functionality natively so it would require some extra work.


----------



## Dan203 (Apr 17, 2000)

ClearToLand said:


> @Dan203
> 
> Have you begun creating formal 'Release Notes' yet?


There are release notes on the website. That's really all I've got.


----------



## ClearToLand (Jul 10, 2001)

Dan203 said:


> *There are release notes on the website*. That's really all I've got.


Aha!

I never CLICK'd on 'Download' because I wasn't ready to d/l it yet so all I saw was the moving images.
Have you considered separating the 'Release Notes' from the 'Download' button?
.
Do you have 'Release Notes' prior to 1.5.9?
.
Is a Wiki 'How-To' set of pages being considered?


----------



## ClearToLand (Jul 10, 2001)

Dan203 said:


> I've been thinking about adding some sort of transfer log to the desktop app, so I could just add the retry/sucess/fail stuff to that. *The best part about doing it this way would be that I could kill the transfer as soon as a discontinuity was detected, so I could potentially save some time compared to downloading the full file and scanning with tivilibre*.
> 
> I'm considering it, but that list I'm using doesn't support that functionality natively so it would require some extra work.


Checking for 0x47 every 188 bytes is definitely the way to go - that's why I suggested adding a variable, user-selected 01-99 maximum retry count.

Now, getting all of this '_juicy stuff_' onto the NPL is another matter...


----------



## Dan203 (Apr 17, 2000)

ClearToLand said:


> Aha!
> 
> I never CLICK'd on 'Download' because I wasn't ready to d/l it yet so all I saw was the moving images.
> 
> ...


No I didn't start keeping track of changes until 1.5.9. But the vast majority of changes were bug fixes prior to that, so there wasn't much to report anyway. 1.6.1 is pretty stable. (I hope ) But I'm still engrossed in the project, so I've still got a lot of ideas for new features and improvements.

Although I'm thinking about trying to recruit some beta testers so I don't keep having these releases where I discover some bug the next day and have to annoy my users again with the update message.


----------



## reneg (Jun 19, 2002)

I've been playing with a Windows batch script which I plan to make available after the next kmttg update which is similar to the robo-download script that wuzznuubi posted about earlier in this thread. I've been testing my batch script on two Tivos; a 4 tuner Bolt & a 4 tuner Roamio. Both are connected OTA to the same amplified antenna with a splitter between the two Tivos. On the networking side, both Tivos are connected to the same switch which also connects my PC.

What I am seeing in my transfers is that I cannot get a clean download from the Bolt. After 180+ downloads of several different tivo files, I have yet to produce a clean transfer. I can get clean downloads from the Roamio, and often on the first attempt. The Bolt is transferring data about three times faster than the Roamio, I've tried several things to change the dynamics of the downloads with no effect.

I don't believe there is a known method of resuming TS downloads at this time.

With the results (or lack therefore of) that I've seen on my Bolt, I believe a faster method to a clean TS download may be to stitch together parts of two or more downloads to form a clean download. This method is not guaranteed to yield a clean download if the same packet fails to re-sync every transfer.

I've uploaded the download data for both my Tivos if someone wants to look at it.


----------



## ClearToLand (Jul 10, 2001)

reneg said:


> I've been playing with a Windows batch script which I plan to make available after the next kmttg update which is similar to the robo-download script that wuzznuubi posted about earlier in this thread. I've been testing my batch script on two Tivos; a 4 tuner Bolt & a 4 tuner Roamio. Both are connected OTA to the same amplified antenna with a splitter between the two Tivos. On the networking side, both Tivos are connected to the same switch which also connects my PC.
> 
> What I am seeing in my transfers is that *I cannot get a clean download from the Bolt*. After 180+ downloads of several different tivo files, I have yet to produce a clean transfer. *I can get clean downloads from the Roamio, and often on the first attempt*. *The Bolt is transferring data about three times faster than the Roamio*, I've tried several things to change the dynamics of the downloads with no effect.
> 
> ...


I'm in the midst of assembling another chrome storage rack so I just found your new (8 minutes old) update during a 'rest break'. The Bolt is 1000Mbps while the Roamio is 100Mbps (wired).

Two thoughts on the Bolt problem come to mind:

If you have an old 100Mbps switch, or router, laying around, run the Bolt through that to throttle its speed.
.
I've just gotten into 'Managed' switches since Newegg was practically giving them away at approximately the same price as 'Unmanaged' switches.
- Let me go upstairs and glance at the manual (in PDF) and / or log onto one of the several in service on my LAN and see if 'throttling' is an option.
- I know that QoS is since I have set QoS (lowest-to-highest priority) as:
Default
TiVo-to-TiVo and to kmttg / pyTiVo / Streambaby PC
Switch-to-Switch (i.e. UPLINKS)
Main Switch-to-Router (only *ONE* LAN port is in use on the router)


----------



## Dan203 (Apr 17, 2000)

reneg said:


> I've been playing with a Windows batch script which I plan to make available after the next kmttg update which is similar to the robo-download script that wuzznuubi posted about earlier in this thread. I've been testing my batch script on two Tivos; a 4 tuner Bolt & a 4 tuner Roamio. Both are connected OTA to the same amplified antenna with a splitter between the two Tivos. On the networking side, both Tivos are connected to the same switch which also connects my PC.
> 
> What I am seeing in my transfers is that I cannot get a clean download from the Bolt. After 180+ downloads of several different tivo files, I have yet to produce a clean transfer. I can get clean downloads from the Roamio, and often on the first attempt. The Bolt is transferring data about three times faster than the Roamio, I've tried several things to change the dynamics of the downloads with no effect.
> 
> ...


I was thinking about throwing together a program that would take 2+ .tivo recordings and try to stitch them together into a clean recording. But in my tests the files that fail always have the failures in the same place, so that wouldn't really work.

I'm going to try to add the sync scan to pyTivo soon, so we can try to replicate the robo download you're doing but perhaps with a little bit of saved time since I can stop the download as soon as I detect an error.


----------



## reneg (Jun 19, 2002)

ClearToLand said:


> I'm in the midst of assembling another chrome storage rack so I just found your new (8 minutes old) update during a 'rest break'. The Bolt is 1000Mbps while the Roamio is 100Mbps (wired).
> 
> Two thoughts on the Bolt problem come to mind:
> 
> ...


I got rid of all my old, slow switches when I moved. I changed where I ran the batch script from on the PC side. For my previous tests, I was running the batch file from an SSD on my PC. I switched to running the batch file from a hard drive and low and behold, I got a clean transfer from my Bolt on the very first try. The transfer was still about twice as fast as the Roamio. I'm going to retest the transfers that failed previously to see if this was an anomaly.


----------



## lpwcomp (May 6, 2002)

I'm seeing better TS transfers since I started idling everything as much as possible. No active recording on the TiVo and PC as idle as possible, especially making sure that nothing except kmttg is communicating with the TiVo from the PC.

I still get occasional glitches but they are less "intense".

I use VRD for everything except the actual transfer


----------



## Dan203 (Apr 17, 2000)

Does throttling the download help at all? I could probably put some artificial throttling into pyTivo if it would actually help.


----------



## reneg (Jun 19, 2002)

My successful transfer may have been an anomaly. I have not had another successful transfer from my Bolt in the last six hours. Nothing else going on with that Bolt at all. I'm not convinced throttling would help.


----------



## lpwcomp (May 6, 2002)

Dan203 said:


> Does throttling the download help at all? I could probably put some artificial throttling into pyTivo if it would actually help.


I haven't tried that but I doubt that it would help. I guess it's possible, but I believe it has been tested by others and had no effect.

My theory is that at least part of the problem is that the TiVo TS transfer code doesn't handle being interrupted very well and sometimes produces an incomplete block in the transfer buffer, but that is just a (S?)WAG.

Unfortunately, there is no way to force the TiVo to dedicate itself to the transfer, plus it seems likely that there is another bug whereby it has problems with some sequences, so that some glitches always appear at the same spot.


----------



## Dan203 (Apr 17, 2000)

Holy sh*t I think I just figured out how to fix this. I added code to pyTivo to download just the header first, then I setup the remaining loop to download the bytes in 188 byte aligned chunks and I got a clean download of a file that always has an error in the exact same spot.

Edit: False alarm. Second download produced an error at the same place, so apparently the clean one was just a fluke. 

Edit: 3rd attempt was clean again. This file consistently was not clean before, so this is an improvement. I'm going to keep playing with the buffer sizes and see if I can make it consistent.


----------



## ClearToLand (Jul 10, 2001)

*Rah! Rah! Sis Boom Bah!
"0x47 @ 188"
(Cheer)
:clapping: Go Dan Go! :clapping:
Touchdown! :handok:*​


----------



## Dan203 (Apr 17, 2000)

I've got the retry thing working. The downloads are just completely sporadic though. Even when I get past the point where the first error is in my test file there is one near the end that kills it. I've only gotten two clean downloads in dozens of attempts.


----------



## Dan203 (Apr 17, 2000)

I've got an idea for how to handle this that might be cool....

The simplest option will be to simply halt the download and start a new one as soon as a sync loss is detected. But that could result in dozens of attempts with no clean downloads. So my idea is to keep track of how many bytes are effected by the sync loss and try to keep the best copy of the show. This means that the first attempt will always download the full show. After that subsequent attempts will compare their number of effected bytes to that file. As soon as they match or exceed the number of effected bytes the download will abort. If they reach the end with less bytes effected, but still have errors, then the original file will be deleted and this will become the new benchmark that subsequent attempts are compared against. At the end of X iterations (set by user) you will be left with a complete download with the least number of bytes effected across all attempts. At the end I would rename the file so that the file name contained the number of bytes effected by sync loss. 

Sound good?


----------



## Dan203 (Apr 17, 2000)

OK I got this working. I'm going to have 3 options...

1) Ignore. Which will download the file once, but will still mark the file name with the number of effected packets
2) Best file. Which will work as outlined above retrying X times and keeping the file with the least number of errors
3) Fail. Which will retry X times but will abort the transfer as soon as it hits and error and if after X tries there isn't a clean copy it will leave nothing on disk and only a fail message in the queue.

In my tests some files could never be downloaded completely clean, but I could get versions with less errors then the initial download.


----------



## ClearToLand (Jul 10, 2001)

Wow Dan! 

I didn't want to monopolize this 'conversation', but it's been almost NINE hours and *NO ONE* else has replied to your questions?!? (I wanted to give other folks a chance to participate too.)

That is DEPRESSING, when you're hot into programming a solution to a problem, ask the 'Community' for some feedback and *NO ONE* replies - I apologize; I thought that I was giving the other forum members 'space'. I'm so sorry...



Dan203 said:


> I've got an idea for how to handle this that might be cool....
> 
> The simplest option will be to simply halt the download and start a new one as soon as a sync loss is detected. But that could result in dozens of attempts with no clean downloads.
> .
> ...


#2 :thumbsup:


Dan203 said:


> OK I got this working. I'm going to have 3 options...
> 
> *Ignore*. Which will download the file once, but will still mark the file name with the number of effected packets
> *Best file*. Which will work as outlined above retrying X times and keeping the file with the least number of errors
> ...


#2 again, but with a '_twist_'. 

OK. My original idea was based on using tivolibre, finding an error and then retrying 01-99 times based on the user selection PER SHOW on the NPL. You, @reneg, and @wuzznuubi have subsequently displayed that some shows will *NEVER* TS / 'Fast' Format transfer cleanly. So, what do we do now?

I appreciate your implementing the "*0x47 @ 188*" logic. Today (well, now it was yesterday), I read a post on the AVS Forum (in either the Hauppauge PVR-1212 thread or the PVR-1512 thread) where a fellow who couldn't tolerate *ANY* 'glitches' (in copies from his DVR to his PC) used VideoReDo to 'Cut-N-Paste' segments from MULTIPLE downloads to create one final 'clean' copy. Now, I certainly don't expect you to do that - I'm just presenting it as one person's solution to the 'glitch' problem.

Based on the facts that I currently have available to me, I vote for #2 "Best File" *WITH* a possible addendum. AFAIK, since this TS / 'Fast' Format transferred "Best File" *STILL* has 'glitches', neither pyTiVo (all versions) or Streambaby will be able to transfer / stream it back to a TiVo for viewing; i.e. it will need to be run through either VideoReDo or tivolibre, right? So, how about following up any unsuccessful TS / 'Fast' Format transfer with a PS / 'Slow' Format transfer (append "_PS" to the filename)? At the very least, we can be fairly confident that it will be viewable (transfer or stream).

Also, I'd like to see the "Total # of Retries Used" on the NPL on the detail line next to the "Total # of Retries Requested" and, for EACH retry, I'd like to see a "# of Sync Errors Detected" per retry attempt in a log file.

And this brings up a question that I intended to post in a separate thread:

*What are the PROs and CONs of TS vs PS Format transfers?*​
Back in the summer of 2016, I was *EXTREMELY* disappointed to discover that ~66% of the TS / 'Fast' Format transfers that I made (per the TCF recommendations) via kmttg and subsequently attempted to view via pyTiVo *FAILED*!  I was a Newbie - I'd didn't know better - I read LOTs of TCF posts; I read the kmttg and pyTiVo Wikis. And yet, I still got '_screwed_'. 

AFAIK, PS / 'Slow' Format transfers have the tendency to lose Closed Captions - are there any other known CONs?

*BOTTOM LINE:* Since we can be almost 100% confident that a PS / 'Slow' Format transfer TiVo-to-PC will be able to successfully transfer / stream PC-to-TiVo, I see no downside to using that as a 'fallback' to a TS / 'Fast' Format transfer failure.

Others are welcome to join in and comment.


----------



## Mikeguy (Jul 28, 2005)

That raises an interesting possibility--an option to fall back to a PS transfer if the TS doesn't work successfully. But then, I guess the question is, what would that point be (how many errors would that be/how big are those errors) and is it possible to set such a point? 

I'm seeing that the PS issues are transfer speed and possible closed caption loss--can something be done on the CC side to make PS a failsafe alternative/fallback? Perhaps a similar, try-again approach?


----------



## gonzotek (Sep 24, 2004)

PS transfers preclude h.264 video, which is becoming more and more prevalent in the field (Comcast is converting to it nationwide, excluding only locals, for now). For me that's the biggest con of a PS transfer - I can't even use it on a large portion of the channels I'd want to.
Some older threads on the topic (mostly rehashing things already discussed):
Difference between MPGEG-PS and MPEG-TS files?
Downloading in PS vs TS Format: Pros and Cons


----------



## Dan203 (Apr 17, 2000)

I was thinking about adding codec detection and automatic switch to PS for MPEG-2 as an option. For that I'd need to find and read the PMT packet, since there doesn't seem to be any indicator in any of the XML data or the header that denotes the video format.


----------



## Dan203 (Apr 17, 2000)

ClearToLand said:


> Back in the summer of 2016, I was *EXTREMELY* disappointed to discover that ~66% of the TS / 'Fast' Format transfers that I made (per the TCF recommendations) via kmttg and subsequently attempted to view via pyTiVo *FAILED*!


FYI my last release, 1.6.1, included a feature to get past these errors. You'll still have a glitch where the error is but it wont completely fail to transfer.


----------



## ClearToLand (Jul 10, 2001)

Dan203 said:


> FYI *my last release, 1.6.1, included a feature to get past these errors*. You'll still have a glitch where the error is but it wont completely fail to transfer.


To be clear, the *ORIGINAL* TiVo-to-PC TS / 'Fast' Format transfer would have had to be performed with pyTiVo Desktop v1.6.1 in order for the PC-to-TiVo '_glitch-free_' transfer back to occur, right?

.TIVO TS / 'Fast' Format files transferred TiVo-to-PC via kmttg from early 2016 are not going to now '_magically_' transfer PC-to-TiVo '_glitch-free_' with pyTiVo Desktop v1.6.1?


----------



## Dan203 (Apr 17, 2000)

They will not be "glitch free", but they will not just fail like they did before. I use tivolibre to skip over the bad parts so the file always completes the transfer. However there will be glitches where the bad parts are. How bad they are depends on how many packets were affected and what types of frames they encompassed. VideoReDo has better logic to repair these kinds of issues, so using it will produce better results, but with pyTivo Desktop 1.6.1 you can at least transfer your .tivo files without having to decrypt them first.

I should have the new version with the code to retry downloads available in a couple days.


----------



## ClearToLand (Jul 10, 2001)

Dan203 said:


> ClearToLand said:
> 
> 
> > To be clear, the *ORIGINAL* TiVo-to-PC TS / 'Fast' Format transfer would have had to be performed with pyTiVo Desktop v1.6.1 in order for the PC-to-TiVo '_glitch-free_' transfer back to occur, right?
> ...


Dan,

I trust that you already know how highly I think of you - I've applauded your efforts to (at least attempt to) solve *EVERY* '_known-to-man _' TiVo transfer problem (TiVo-to-PC and PC-to-TiVo) with pyTiVo Desktop *MANY* times via either LIKEs, posts or PMs. :clapping: (I pushed VERY hard to you @Dan203 , Kevin @moyekj , and William @wmcbrine via PMs.)

But... you need to learn how to answer a DIRECT question with a DIRECT answer. 

As an '_Old Fart / Baby Boomer / Grandfather / yada, yada, yada _', I've learned, through MUCH experience, to avoid being vague / stating an ABSOLUTE when it could really be a MAYBE. Remember, *EVERYTHING* entered onto the internet is *FOREVER* - there are no DELETEs; no changing your answer in a future post; whatever you say RIGHT NOW is *FACT* / "Cast-in-Stone".  

OK. Lightening up... pyTiVo Desktop v1.6.1. is no '_magic elixir _' - whatever shows that were previously transferred TiVo-to-PC in TS / 'Fast' Format via '_whatever _' tool other than pyTiVo Desktop v1.6.1 will **NOT** now be able to be transferred back PC-to-TiVo '_glitch-free _' with pyTiVo Desktop v1.6.1, or any other tool. (Let's use @oldradio99's grandchildren's shows as an example - sorry folks, you'll have to SEARCH for that thread on your own - too late / too tired).

IANAL, *BUT*, IMHO you HAVE to be more explicit in your statements. Otherwise '_someone _' is most certainly going to misinterpret '_what you said from what you meant to say _'.  :kissingheart:


----------



## Dan203 (Apr 17, 2000)

No. There is no way to magically fix the errors that were introduced into the original download by TiVo. VideoReDo can do a bit more then scanning for the missing TS packet headers though. It can look for audio and video frames in the garbage data and reconstruct them, to an extent, which is why when you fix one of these files with VideoReDo the glitch is usually less severe then what you get with tivolibre, which simply dumps all the bad packets. 

As to what I did in 1.6.1... prior to that version if you uploaded a file with a glitch there was a good chance the TiVo would simply stall at the glitch and never even receive the rest of the file. By skipping over those bad packets the TiVo is able to continue downloading the remainder of the file and you end up with something that is at least watchable. In my tests thw glitch was usually only about 2-3 seconda long so it didn't completely ruin the show as long as you're not too anal. But if I fixed the file with VideoReDo the glitch was usually only about 1/2 a second and was less likely to effect the audio specifically which makes the glitch less burdensome. 

Through all these replies have you even bothered to install pyTivo Desktop? You might have a better understanding of what it can and cannot do if you gave it a try.


----------



## ClearToLand (Jul 10, 2001)

Dan203 said:


> No. There is no way to magically fix the errors that were introduced into the original download by TiVo...
> 
> ...Through all these replies have you even bothered to install pyTivo Desktop? *You might have a better understanding of what it can and cannot do if you gave it a try*.


I believe that, just from reading your posts, that I already possess a good understanding of what pyTiVo Desktop v1.6.1 can, and cannot, do. I was simply pointing out to you that your reply to my question was ambiguous and that someone less technical than you or I might expect pyTiVo Desktop v1.6.1 to work '_magic _' on their old, corrupt '_glitched _' TS / 'Fast' Format transferred files (i.e. I used @oldradio99's problem with his TS / 'Fast' Format transferred, '_glitched _' grandchildren's shows as an example).

Give me some credit man  , I didn't just *fall of the back of the turnip truck*! 

Geez, after all of my support, ideas and encouragement, is that how little you think of me? Sad, so sad...  (and disappointing...) Wow...


----------



## Dan203 (Apr 17, 2000)

You just seem pretty invested in offering ideas for a piece of software you haven't even tried. You should install it. I promise it wont make your computer blow up.


----------



## reneg (Jun 19, 2002)

I've spent some time in HxD looking at ts files and think I possibly have a way to build a clean ts file out of two or more downloads of corrupted ts files. If the corruption repeatedly shows up in the same area of each downloaded file, this will not work. I won't have time to prototype this into code for a while, but thought I'd share my thoughts, so Dan or others can tell me that it won't work.

It's a bit sloppy, but for any of the following pseudo code to make sense, please reference the image I uploaded with this post.


```
Download Tivo File 1, convert to ts & scan with tivolibre, save output, if clean ts file (F1) then success & exit
Repeat
    Download Tivo File 2, convert to ts & scan with tivolibre, save output, if clean ts file (F2) then success & break

    Determine in file F1 both P1 & Q1, and in F2 both P2 & Q2
    P1, P2 – First Point of corruption (“Invalid TS packet header for packet xxxx”), in HxD, first corrupted packet was xxxx minus 1
    Q1, Q2 – Start of clean data (“Resume decryption at: xxxx”),  packet number is xxxx/188
    If overlap between corruption areas (green) P1, P2, Q1, or Q2 Then break

    //Favor longer gap until first corruption in F1
    If P2 > P1 then swap IDs for F1<->F2, P1<->P2, Q1<->Q2
   
    In F2, start at F2.Q2 packet, find matching F1.P1 packet (minus 2, ie. last known good packet in segment A1) in F2, matching all 188 bytes of the F1.P1 packet (minus 2)
    if no match is found then break
    In F2, starting at F2.P1 packet, find matching Q1 packet in F2 matching all 188 bytes of the F1.Q1 packet.
    if no match is found then break
    If there are any corrupted packets between F2.P1 through F2.Q1 then break
   
    //Build a better ts file into F3
    Copy segment F1.A1 to new file, F3
    Append onto F3, from F2.P1 through but not including F2.Q1
    Append onto F3, from F1.Q1 to EOF

    Scan file F3 for re-sync with tivolibre, save output
    If no resync errors found then F3 is a clean ts file, success & break Else Set F3 id as F1 file, close F1
Until Success or MaxAttempts reached
```


----------



## Dan203 (Apr 17, 2000)

In all my tests I find that the corruption is usually in the same spot. However I have found that with files that have two or more bits of corruption I will occasionally get a file that is missing one of them. So if you get lucky and and get two, or more, files without the corruption in each spot you could stitch them together and get a clean file. 

That being said I'm not sure tivolibre is going to do what you need. You can more easily do this in real code, even python. TS files have a sync byte every 188 bytes which is 0x47. So if you use buffers that are exactly 188 bytes aligned you can easily copy good chunk from one file over the bad part of another file. 

I think your best bet for logic would be to scan eash file as it downloads and keep a map of the bad chunks (location and length). If you get a duplicate of a file you already have delete it. Repeat as many times as needed until you have multiple files which can be merged into a clean file (or you simply get a clean file outright). When merging use the map to grab the chunks you need from each of the files and merge into one file.

If you want to do this in the .tivo file, without decrypting, you'll need to skip past the .tivo header. To do that read the first 16 bytes into a buffer. Bytes 10-13 (0 based index) is a 64bit number corresponding to the size of the entire header. Do a seek of that value, minus the 16 you already read, and you'll be at the start of the TS file. (this byte should be 0x47)


----------



## reneg (Jun 19, 2002)

Thanks for the feedback Dan. I went back and looked at the logs I uploaded earlier in this thread, and found that this would not have helped in any of the 8 downloads I performed on my Bolt. Multiple glitches per download and usually in the same spot. On my Roamio, it could have worked on the one file that couldn't get a clean download after 20 attempts. In fact, it would have been able to stitch together a clean file after only two downloads. If I do go forward with prototyping, maybe I'll just scale it back to take two glitched ts files as input and try and stitch them together into a clean ts file.


----------



## Dan203 (Apr 17, 2000)

I have one file on my Bolt that always has errors. It's a half hour and there is usually one at about the 10 minute mark and another around the 26 minute mark. I've downloaded this file hundreds of times now testing. Out of all those attempts I've only gotten a completely clean download twice. But I have had multiple occasions where I get a download with only one or the other and could have used those two files to create a clean download. The real question is how many times are you willing to try just to get a clean file?


----------



## Dan203 (Apr 17, 2000)

The new build of pyTivo Desktop with the TS error monitoring features has been posted


----------



## lew (Mar 12, 2002)

Dan203 said:


> No. There is no way to magically fix the errors that were introduced into the original download by TiVo. VideoReDo can do a bit more then scanning for the missing TS packet headers though. It can look for audio and video frames in the garbage data and reconstruct them, to an extent, which is why when you fix one of these files with VideoReDo the glitch is usually less severe then what you get with tivolibre, which simply dumps all the bad packets.
> 
> As to what I did in 1.6.1... prior to that version if you uploaded a file with a glitch there was a good chance the TiVo would simply stall at the glitch and never even receive the rest of the file. By skipping over those bad packets the TiVo is able to continue downloading the remainder of the file and you end up with something that is at least watchable. In my tests thw glitch was usually only about 2-3 seconda long so it didn't completely ruin the show as long as you're not too anal. But if I fixed the file with VideoReDo the glitch was usually only about 1/2 a second and was less likely to effect the audio specifically which makes the glitch less burdensome.
> 
> Through all these replies have you even bothered to install pyTivo Desktop? You might have a better understanding of what it can and cannot do if you gave it a try.


I haven't bother participating. Not really an issue for me. I don't think am getting many errors. The 1/2 second glitch, after using VRD, isn't noticeable.

Assume I don't bother with quick stream fix and directly edit the tivo file. Is the file similarly fixed when I use VRD to encode? Does it matter if I use the VRD software encoder vs Intel quick sync?

FWIW Kevin said he tried throttling downloads and the errors persisted.


----------



## Dan203 (Apr 17, 2000)

If you recode in VRD it's doing essentially the same things as QuickStream Fix so the glitch should be oughly the same. In fact it might be even less noticable because it would fix any errors that the playback decoder might have due to missing reference frames. Some players are more sensative to this then other, which is why some will only show a minor glitch while others will skip over a few seconds of the video. With it recoded you'll get as many frames in the output as VRD is able to decode.


----------



## mlippert (Apr 3, 2010)

Dan203 said:


> I was thinking about adding codec detection and automatic switch to PS for MPEG-2 as an option. For that I'd need to find and read the PMT packet, since there doesn't seem to be any indicator in any of the XML data or the header that denotes the video format.


I've just been catching up on this whole thread. And thanks Dan, I haven't tried your new pyTivo desktop yet (I've been pretty happy w/ kmttg) but I may give it a shot soon. Thanks for all your work on this and your posts have been super informative.

In regards to detecting the codec and using PS for mpeg2 and TS for h.264, I think that is a great idea, but I've got a suggestion that may be easier to implement (possibly).

I don't think any channel switches how it's shows are encoded, so if you just let the user specify whether to use PS or TS for a particular channel, then it's a simple lookup to decide which to use with no need to download anything first. Add a default setting so that the user can only specify whichever are the exceptional channels. Even if I had to specify for every channel I download shows from, that list is 10-30 channels not that many.

OR, if you want to do more work, build that channel list automatically, only checking the codec if the channel isn't in the list (or if it is PS and the download fails because the channel was recently changed to h.264. I don't think any channels are being changed from h.264 to mpeg2 so that isn't really worth checking) and then adding the codec you determined for that channel to the list.


----------



## lew (Mar 12, 2002)

Dan. I don't know if the "magic" is done by vrd or by your encoder. Does it matter if we use the software encoder or can we use Intel quick sync and still correct glitches


----------



## Dan203 (Apr 17, 2000)

lew said:


> Dan. I don't know if the "magic" is done by vrd or by your encoder. Does it matter if we use the software encoder or can we use Intel quick sync and still correct glitches


No. The encoder doesn't matter. The main difference between the way VRD fixes errors and the way tivolibre and pyTivo does is that when VRD detects an error it attempts to read the junk data and see if there is anything usable in there. tivolibre and pyTivo just skip all the junk data until the sync byte realigns.


----------



## Dan203 (Apr 17, 2000)

mlippert said:


> I've just been catching up on this whole thread. And thanks Dan, I haven't tried your new pyTivo desktop yet (I've been pretty happy w/ kmttg) but I may give it a shot soon. Thanks for all your work on this and your posts have been super informative.
> 
> In regards to detecting the codec and using PS for mpeg2 and TS for h.264, I think that is a great idea, but I've got a suggestion that may be easier to implement (possibly).
> 
> ...


I might consider this. In the mean time I did add a feature in 1.6.1 where you can pick yourself on a per-recording basis. It will put a little TS and PS button next to each show and you can click whichever one you want for that show. There is also a menu added to the Download All button that allows you to pick TS or PS for that operation as well.


----------



## tlp95129 (May 17, 2008)

Dan,

Love using your pyTivo Desktop. Great work.

Recent glitch however. I'm on v1.62 now, and when I try to transfer a TS .tivo file from from PC to Tivo Roamio. my AVG-Free virus program flags tivolibre.exe and quarantines it. So the transfer fails. I ran a separate scan of the tivolibre.exe file in your bin directory and it passed OK. Has anyone else reported this? It's most likely bogus, but thought I'd report anyway.

Link


----------



## Dan203 (Apr 17, 2000)

No one else reported. Just add the pyTivo install folder to its ignore list and you should be fine.


----------



## elprice7345 (Sep 28, 2009)

Not being that knowledgeable about video technical details, I'm having trouble following the TS error conversation.

Comcast has switched most of my channels to h264, so I had to switch to TS transfers.

I generally record shows, download them, and archive them until I pull them back to my TiVo to watch.

Since I've switched to TS containers, sometimes the downloaded file has a shorter duration than the TiVo file. If I repeatedly download the file I can generally get the duration to match within 10 seconds or less.

My current workflow:

Download in TS container using kmttg
Decrypt using VRD
Run VRD QSF
Check downloaded file duration vs. kmttg reported duration
Re-download/process if the TS duration < kmttg duration
Repeat until I get the file that most closely matches the kmttg duration
Run VRD AdScan
Cut commercials using VRD
I'm OK with the durations not quite matching, assuming I'm missing a few seconds. Nothing "fails" in my workflow, just shorter files than expected.

Do the failures only occur when downloading in TS containers, leaving in encrypted .tivo format and pulling back encrypted .tivo files?

Does my workflow prevent this issue because of the decryption and/or QSF processing?

What tool can I use to find TS sync errors?


----------



## Dan203 (Apr 17, 2000)

If you use my pyTivo Desktop to download them it will tell you if the file has TS sync errors. It also offers an option to retry download multiple times (user settable) until you get a clean copy or just keep the one with the least number of errors.


----------



## lpwcomp (May 6, 2002)

elprice7345 said:


> Decrypt using VRD
> Run VRD QSF


If you're using kmttg for this, then this is not actually 2 steps as VRD QSF is used for the decryption. If you're running QSF "manually", it is redundant.


----------



## elprice7345 (Sep 28, 2009)

Dan203 said:


> If you use my pyTivo Desktop to download them it will tell you if the file has TS sync errors. It also offers an option to retry download multiple times (user settable) until you get a clean copy or just keep the one with the least number of errors.


Do the failures only occur when downloading in TS containers, leaving in encrypted .tivo format and pulling back encrypted .tivo files?

Does my workflow prevent this issue because of the decryption and/or QSF processing?


----------



## ClearToLand (Jul 10, 2001)

elprice7345 said:


> Not being that knowledgeable about video technical details, I'm having trouble following the TS error conversation.
> 
> *Please see: Post #10927 re: TS vs PS*​
> Comcast has switched most of my channels to h264, so I had to switch to TS transfers.
> ...


*CLICK to Expand - comments in QUOTE*

You're following the 'Gold Standard' re: TS / 'Fast' Format transfers - I'm curious as to your statement re: 'Shorter Length' and asked about it in your QUOTE.


----------



## Dan203 (Apr 17, 2000)

elprice7345 said:


> Do the failures only occur when downloading in TS containers, leaving in encrypted .tivo format and pulling back encrypted .tivo files?
> 
> Does my workflow prevent this issue because of the decryption and/or QSF processing?


The errors only happen when downloading as TS. But if the channels are H.264 that's your only option.

The error is in the download, so it's bad from the moment it hits your disk. Some programs, like VRD, and deal with the error better then others but the error is still there and will still be there when you transfer it back.


----------



## ClearToLand (Jul 10, 2001)

Dan203 said:


> The errors only happen when downloading as TS. But if the channels are H.264 that's your only option.
> 
> The error is in the download, so it's bad from the moment it hits your disk. Some programs, like VRD, and deal with the error better then others but *the error is still there and will still be there when you transfer it back*.


To clarify, and this is where most folks stumble and get confused, a TiVo-to-PC transfer using TS / 'Fast' Format (without pyTiVo Desktop retrying until error-free - I'm talking the 'vanilla' (Fat, Dumb and Happy) TiVo Desktop or kmttg type of non-technical TiVo user transfer in TS mode) *MAY* (will HARDLY EVER, IME - see EDIT below) transfer back PC-to-TiVo until you decrypt it into a .TS file and run QS Fix (again, 'vanilla' as in no VideoReDo purchase involved yet...).

EDITed per input from Kevin.

I had an ~66% failure rate with 50+ files that I transferred TiVo-to-PC via kmttg TS / 'Fast' Format file transfer (Feb/Mar 2016) when I attempted to view them with pyTiVo (Summer doledrums 2016) so, like my distrust of Seagate HDDs, I'm wary of TS / 'Fast' Format TiVo-to-PC transfers. @Dan203 is changing the TiVo situation VASTLY with *pyTiVo Desktop v1.6.2+* and his new "*0x47 every 188*" logic (you don't need to know what it means - just know that it works). :thumbsup: What's even better is that he just stated that it will also work with existing 'glitched' TS / 'Fast' Format transfers back PC-to-TiVo. *Bravo Dan!* :handok:


----------



## Dan203 (Apr 17, 2000)

ClearToLand said:


> To clarify, and this is where most folks stumble and get confused, a TiVo-to-PC transfer using TS / 'Fast' Format (without pyTiVo Desktop retrying until error-free - I'm talking the 'vanilla' (Fat, Dumb and Happy) TiVo Desktop or kmttg type of non-technical TiVo user transfer in TS mode) will NEVER transfer back PC-to-TiVo until you decrypt it into a .TS file and run QS Fix (again, 'vanilla' as in no VideoReDo purchase involved yet...).


If you use pyTivo Desktop to transfer back it will work. I use similar logic on the return trip to skip over the errors so the transfer back will complete. It will have a glitch where the error was but it wont hang like it use to.


----------



## moyekj (Jan 24, 2006)

ClearToLand said:


> To clarify, and this is where most folks stumble and get confused, a TiVo-to-PC transfer using TS / 'Fast' Format (without pyTiVo Desktop retrying until error-free - I'm talking the 'vanilla' (Fat, Dumb and Happy) TiVo Desktop or kmttg type of non-technical TiVo user transfer in TS mode) will NEVER transfer back PC-to-TiVo until you decrypt it into a .TS file and run QS Fix (again, 'vanilla' as in no VideoReDo purchase involved yet...).


 Don't think it's that black and white. Returning a TS .TiVo file with download glitches directly without any tricks *MAY* fail to transfer back, but is not guaranteed to fail. I'm pretty sure I have TS .TiVo files with glitches that have transferred back fine using vanilla pyTivo.


----------



## ClearToLand (Jul 10, 2001)

Dan203 said:


> *If you use pyTivo Desktop to transfer back it will work*. I use similar logic on the return trip to skip over the errors so the transfer back will complete. It will have a glitch where the error was but it wont hang like it use to.


*@cmccarter1 DESPERATELY needs this info :handok:*​


----------



## elprice7345 (Sep 28, 2009)

To make sure I understand: with the pyTivo desktop download, I get a decrypted (assuming I have decrypt with tivolibre checked) TS file that I would then want to run QSF on before running VRD to find and remove commercials, correct?

One example:
I downloaded a show that is shown on my TiVo as duration = 28:54

Downloading using pyTivo desktop it has a duration = 28:59
After running QSF, duration = 28:47

Downloading using kmttg and running QSF, duration = 28:47

AFAICT, PyTivo desktop didn’t find any errors, but the post-QSF duration was shorter, leading me to believe it removed some glitches.

Is my conclusion correct?


----------



## Dan203 (Apr 17, 2000)

Don't decrypt if you have VideoReDo. VideoReDo will decrypt and fix all in one pass when you QSF. If you decrypt in pyTivo then it uses tivolibre which will remove all the error portions completely and VideoReDo wont have a chance to actually fix them.


----------



## elprice7345 (Sep 28, 2009)

Dan203 said:


> Don't decrypt if you have VideoReDo. VideoReDo will decrypt and fix all in one pass when you QSF. If you decrypt in pyTivo then it uses tivolibre which will remove all the error portions completely and VideoReDo wont have a chance to actually fix them.


Re-downloaded with decrypt turned off in pyTivo Desktop and get the same results:

Duration = 28:54 on TiVo
*.tivo duration = 28:59
Process .tivo file with QSF getting .ts file duration = 28:47
Same results. AFAICT, PyTivo desktop didn't find any errors, but the post-QSF duration was shorter, leading me to believe it removed some glitches.

Is my conclusion correct?


----------



## Dan203 (Apr 17, 2000)

Did VideoReDo say anything about resync frames removed?

If not the duration difference could be something else. TS files keep a running clock. Even if the signal cuts out the clock keeps ticking, so when it comes back the time stamps continue at the current clock positon. The way programs determine the duration of a video they simply look at the time stamps of the first frame and the last frame and calculate the difference. So if there is a big chunk missing in the middle the duration will still be calculated as the original scheduled recording. However when you run the file through QSF it recalculates the time stamps using only the frames that exist, so that can change the duration when you check the fixed files.


----------



## elprice7345 (Sep 28, 2009)

Dan203 said:


> Did VideoReDo say anything about resync frames removed?
> 
> If not the duration difference could be something else. TS files keep a running clock. Even if the signal cuts out the clock keeps ticking, so when it comes back the time stamps continue at the current clock positon. The way programs determine the duration of a video they simply look at the time stamps of the first frame and the last frame and calculate the difference. So if there is a big chunk missing in the middle the duration will still be calculated as the original scheduled recording. However when you run the file through QSF it recalculates the time stamps using only the frames that exist, so that can change the duration when you check the fixed files.


Thanks for the education Dan!

I always run QSF as part of my workflow in kmttg and therefore don't see the QSF message attached below:







I ran QSF in the VRD GUI and this particular video had:

3 Audio Frame errors
275 video resync frames removed
15 audio resync frames removed
To summarize in non-technical terms: I suppose this is the reason QSF is valuable, it removes the glitches in the video that could cause issues downstream. The net result is QSF removes the glitches, making the video a shorter duration.

Correct?


----------



## Dan203 (Apr 17, 2000)

Yep. 275 video frames at 59.94fps = 4.59 seconds, which most programs will round off to 5 seconds, which is exactly the duration difference you were seeing.


----------



## murgatroyd (Jan 6, 2002)

Hi guys -- I recently started transferring all of my files as TS files (using kmttg) because Comcast has switched several channels over to MPEG4. I have kmttg set to send the metadata over and to transfer just the file without decrypting or running QSFix because I plan to process the files later with VRD, per what Dan said earlier:



Dan203 said:


> Don't decrypt if you have VideoReDo. VideoReDo will decrypt and fix all in one pass when you QSF. If you decrypt in pyTivo then it uses tivolibre which will remove all the error portions completely and VideoReDo wont have a chance to actually fix them.


I have a Roamio Basic with the stock 500GB drive, and I have a wired connection between my Win 8.1 computer and the Roamio. I also have Tivo Desktop installed (not sure of what other codecs I have).

Usually after the transfer is complete, I do a quick check of the *.tivo file with Windows Media Player to see if WMP will play the recording all the way through, then I dump the files off to an external hard drive until I get back to them later. (I have VLC but maybe I'm missing some codec because it won't play the *.tivo files -- I thought it used to.)

Most of the transfers seem to be okay, but for the ones that aren't, I'm seeing two problems:

1) the file appears to be the right size, going by what Windows shows me in Explorer (that is, the files are more or less the same size as other episodes of the same TV show) but WMP's time indicator acts like they are truncated (i.e. an hour show will appear to be 15 minutes & change).

OR

2) the file length is fine, but WMP doesn't give me any sound.

I've had the "no sound" problem with some non-TiVo files, too, so it could it be a driver problem?

Any pointers towards "debugging for Dummies" welcome. I can understand a lot of the information that people post in the threads here, but since I don't use the information all the time, I don't always retain it well.


----------



## Dan203 (Apr 17, 2000)

1) VideoReDo QSF should fix that

2) Try installing AC3Filter....

Download AC3Filter - AC3Filter


----------



## murgatroyd (Jan 6, 2002)

Dan203 said:


> 1) VideoReDo QSF should fix that
> 
> 2) Try installing AC3Filter....
> 
> Download AC3Filter - AC3Filter


Thanks. I downloaded AC3Filter and I think it installed okay. If I see the no-audio problem again, I'll post again.

For the other problem with the mis-reported timings -- I just downloaded Stage 11 of the Tour de France; it was NBC SportsNet which is probably MPEG4 because the recording is tiny compared to the Stage 8 on NBC proper (recording of Stage 8 was 4:30 and was at least 32 GB.) kmttg reports the size for Stage 11 as 8.91 GB for 5 hours.

Windows Media Player acts like the recording is 4:01, but VRD TS sees all 5 hours. If VRD sees the whole program, I guess that's all I need to worry about for now. I don't have enough hard drive space on the Roamio to stockpile the recordings until later. (I look forward to playing with VRD TS later as I edit and do joins.)

Thanks again for the help.


----------



## reneg (Jun 19, 2002)

In the quest for the perfect tivo download, I'm making a Windows batch script available that attempts to download a pristine .tivo file using the custom job in kmttg. I recommend that if you use this script that the only jobs you check in kmttg are the metadata and custom jobs. Also, only run pristine script from the NPL of a tivo (not the Files tab) in kmttg. Kmttg initiates a download of the tivo file when you start a job and then calls the custom command pristine script. The pristine script uses that downloaded file if TS downloads are enabled in kmttg. If PS downloads are enabled, the pristine script will download the file again in TS format. If you try to run the pristine script from the files tab, you will see a kmttg null pointer exception error. After the download completes, if an error is detected in the downloaded TiVo file, the script will download the file again and check for errors. This repeats until either a downloaded files has no errors or the maximum number of download attempts are reached.

The name of the script, pristine, may be a misnomer. The script looks for re-synch errors from tivolibre, when it gets a download without re-synch errors, it's done. This doesn't necessarily mean that the downloaded file is free of errors. I've seen instances where even after I've downloaded a pristine tivo file, I still see input sequence errors and audio resync frames removed by Videoredo qsfix. I've also seen cases where a pristine tivo file runs cleanly through Videoredo qsfix. In any event, I decided to make this script available so others can play with it.

Tools:
The pristine script relies on two external programs. Make sure each of these are referenced on your path and invokable from a command prompt.
1) java - should already be installed if you're running kmttg. Needed by tivolibre
2) curl - I've used an old version of curl that kmttg made available with the kmttg tools. It's no longer distributed as part of the kmttg tools. I've also tested with the version of curl that is available with Tivo desktop. There are other versions of curl available for windows, but I have not tried them.

Also needed is the tivolibre tivo decoder jar file
1) tivolibre - latest tivodecoder jar - https://github.com/fflewddur/tivolibre/releases/download/v0.7.4/TivoDecoder.jar

Configure the pristine script:
Once you download the script, rename it pristine.cmd. Within the script, there are variables to edit:

@rem -------------------- begin - Edit these values as necessary in the script
@rem Tivo MAK number where <tivoMak> is your tivo Media Access Key (MAK)
set tivomak=<tivoMak>
@rem location of tivolibre
set "tivolibre=c:\ntbin\tivodecoder.jar"

@rem Number of attempts to download a pristine tivo file before giving up
set maxattempts=10
@rem Save the interim imperfect TS downloads; TRUE saves imperfect downloads, anything else causes overwrite
set saveTS=TRUE
@rem minimum filesize of a tivo file to be valid. In some error conditions, the error is embedded within the .tivo file
set mintivofilesize=50000
@rem -------------------- end - Edit these values as necessary

At the bottom of the script, once you've gained confidence in the script, you can uncomment some of the file clean-up commands by removing the "@rem ".

Configure kmttg:
To configure the custom command script in kmttg, go to File->Configure->Program->custom command, enter <path>\pristine.cmd [tivoFile] [downloadURL] where <path> is the directory where the pristine script is located.

Output:
Return codes set by the pristine script will be visible in the output log of kmttg. Messages and status are logged in a .out file which logs the output of the curl & tivolibre commands as well as some script status messages. If saveTS is TRUE, then each download attempt will be numbered and there will be a corresponding .out file with messages and status.

Return codes shown in kmttg output log:
0 - Success, pristine tivo file downloaded
1 - missing argument(s) passed into script
2 - Curl returned error
3 - Curl file transfer failed
4 - Transferred file too small. Open the .tivo file in text editor for more error information
5 - Tivolibre returned error
6 - maximum retry limit reached, unable to download a pristine tivo file
7 - expected 1st argument (tivofile) to have .tivo extension

Notes:
* If the pristine download is successful, the script attempts to rename the metadata file to a ts file if the metadata file exists.
* In my testing, I have had much better luck getting pristine .tivo files from my Roamios vs my Bolt. Don't know why, but it's just the way it goes in my configuration.
* For testing purposes, I recommend copying your kmttg directory to a test directory and playing with the pristine script there. Make sure that the kmttg service and skipmode services are disabled on your test version of kmttg.
* If you see the following in your kmttg output log, "The syntax of the command is incorrect.", check that you set your tivomak in the script
* I have not tested what happens under a disk full condition.
* I have not tested running the pristine script in the kmttg service.
* If you abort a job in kmttg and try to delete the files, sometimes it takes up to a couple minutes for windows to release all the files in the job.
* My test PC is a 64 bit Windows 7 Professional system.
* Use at your own risk. Your mileage may vary. Good Luck

Reneg


----------



## Dan203 (Apr 17, 2000)

You know I added an automated version of this to pyTivo desktop? In the download section of the options you can set it to retry downloads as many times as you want until it gets a prestine copy. There is even an option to keep the file with the least number of errors if a prestine copy can't be aquired. And now it has built in VideoReDo support, so you can QSF your files after they download to minimize the damage caused by the bad packets if there are any.


----------



## reneg (Jun 19, 2002)

Dan203 said:


> You know I added an automated version of this to pyTivo desktop? In the download section of the options you can set it to retry downloads as many times as you want until it gets a prestine copy. There is even an option to keep the file with the least number of errors if a prestine copy can't be aquired. And now it has built in VideoReDo support, so you can QSF your files after they download to minimize the damage caused by the bad packets if there are any.


I knew you added it to pyTivo desktop. I did it for myself for kmttg which I use extensively. I've found I use pyTivo less these days having installed Plex.


----------



## Dan203 (Apr 17, 2000)

Ahhh... ok. I just wanted to make sure you realized it was available.


----------



## ClearToLand (Jul 10, 2001)

reneg said:


> In the quest for the perfect tivo download, I'm making a Windows batch script available that attempts to download a pristine .tivo file using the custom job in kmttg. *I recommend that if you use this script that the only jobs you check in kmttg are the metadata and custom jobs*... ...After the download completes, if an error is detected in the downloaded TiVo file, *the script will download the file again and check for errors. This repeats until either a downloaded files has no errors or the maximum number of download attempts are reached*...
> Reneg





Dan203 said:


> *You know I added an automated version of this to pyTivo desktop?*





reneg said:


> I knew you added it to pyTivo desktop. *I did it for myself for kmttg which I use extensively. I've found I use pyTivo less these days having installed Plex*.


Hi @reneg ,

I remember your UserID from the "*PyTiVo Desktop / "0x47 every 188" / TS Sync Errors*" posts / threads where I relentlessly kept '_encouraging_' (aka nagging) @Dan203 to add the "*0x47 every 188*" logic to his PyTiVo Desktop program (@Dan203 - *THANK YOU!, THANK YOU!!, THANK YOU!!!*  ). He did, I tried it, it works! :thumbsup:

I applaud you for forging on, creating (and sharing) your BATCH file. I've written *MANY* BATCH files for personal projects and I enjoyed reading yours. While I understand your preference for kmttg (I too use it for the majority of my transfers), I've been debating with myself since Wednesday whether or not to ask you a few questions. With only the best of intentions, here goes:

Have you tried PyTiVo Desktop?
.
How often do you experience TS Sync Errors / need to run your BATCH file?
.
- Personally, after a BAD experience where over 50% of the shows that I had moved offline using TS / 'Fast' Format transfers via kmttg FAILED to transfer back (months later when ALL of the original recordings were long gone) with pyTiVo (@wmcbrine ), I've been using PS / 'Slow' Format transfers until several days ago when I had an *interesting discussion regarding 'Skip'* with @moyekj .

Sadly, IMO, similar to the shock many folks experience when they try to send a TS / 'Fast' Format transferred show back to their TiVos via pyTiVo (either version) and discover 'TS Sync Errors' for the first time, there has also been little discussion regarding '*Skip*' on transferred back shows. Much to my surprise, a .tivo file is NOT a complete representation of what came off your TiVo unit - specifically "*contentId*", which is CRUCIAL to getting your TiVo unit to request the '*Skip*' info from the TiVo servers, is missing. Fortunately, though, decrypting the file to a .mpg and saving the .mpg.txt (and the .srt and .edl for good measure) DOES work - I tested it tonight. :thumbsup:
.
AFAICT, your BATCH file needs to completely transfer the entire show TiVo-to-PC *BEFORE* tivolibre can check it for TS Sync Errors. :thumbsdown:
PyTiVo Desktop is checking EVERY 188 bytes. :thumbsup:
.
Back in June, I doggedly spent ~36 hours over the course of 2 days *creating a Python Development Environment and 'hacking' into the PyTiVo Desktop source code (v1.62)* that @Dan203 so graciously shared. As they say, "*A picture is worth a thousand words...*" and so, with additional debug / print statements, I too learned the '_nitty-gritty_' about TS Sync Errors (and why most folks just end up buying a copy of VideoReDo TV Suite).

*Short Answer:* PyTiVo Desktop will abort the transfer the instant it finds a TS Sync Error, i.e. it doesn't have to transfer the entire file first. AND, the TS Sync Errors *MOVE AROUND* (surprise!).

Maybe you (might have) thought that if the first error occurred @ 50%, the retry wouldn't have an error until (let's say) 75%, Nope! You could have an error @ 99% and the next run it will be @ 1%. So, I ask you, why bother transferring the other 99% of the file when you already know that there is a TS Sync Error?

Up until my '*Skip*' discussion with Kevin, I was using PS / 'Slow' Format transfers. I only *HAD* to use TS / 'Fast' Format transfers when the show that I wanted to archive was *H.264*. Now, armed with this new '*Skip*' information, I've changed my procedure (although currently I'm NOT deleting the .tivo when done). What I still need to check is how well a successful / pristine PyTiVo Desktop TS / 'Fast' Format transfer will decrypt (via the *Files Tab*) into a .mpg with '*Skip*' data available. Luckily, I'm on FiOS (with plans to eventually go OTA) and not one of the poor souls on Comcast who have many (ALL?) of their shows in H.264, forcing them to use TS / 'Fast' Format transfers.
.
After successfully transferring a H.264 show via PyTiVo Desktop, the only additional step I need to do is begin a transfer with kmttg and then cancel it once the .tivo file is created. I do this because, for future Plex compatibility, I've created a custom file naming 'template' in kmttg which PyTiVo Desktop is (currently) unable to replicate.
I hope that, after reading this, you at least give PyTiVo Desktop a 'Trial Run'. Regardless, best of luck with your TS Sync Errors.


----------



## reneg (Jun 19, 2002)

ClearToLand said:


> Have you tried PyTiVo Desktop?
> .
> How often do you experience TS Sync Errors / need to run your BATCH file?


1) I have not tried pyTivo Desktop, I may give it a try.
2) I see TS Sync Errors a lot more from the Bolt vs the Roamio. My Bolt is about 3x faster transferring than the Roamio but about 10x more errors.


----------



## murgatroyd (Jan 6, 2002)

Here's my new problem. The Roamio can't see my PC (it doesn't show up in Devices).

TiVoDesktopPlus (yeah, I know, ick phtooey) says:

```
There was an error while attempting to retrieve data from the selected DVR. The system cannot find the file specified
```
... and then doesn't tell me a filename. It sees the Roamio enough to count the recording, and then it spits out the error.


The Roamio and the TiVoHD see each other fine (most of the time).
TD+ can fetch the Now Playing list from the TiVoHD with no problem.
kmttg can see the Roamio and transfer recordings the Roamio just fine

We've rebooted the router, the Roamio, my PC. Sometimes the Roamio can see my PC, and sometimes not.

I'm installing pyTiVoDesktop to see what happens.

Update: okay, I appear to have installed PyTiVo Desktop correctly. Somewhere in the middle of that, the Roamio 'found' my PC again, but it only showed me the share that is in TiVoDesktopPlus, not the extra shares I added in pyTiVoDesktop.


----------



## Dan203 (Apr 17, 2000)

Those two can conflict. Try disabling TiVo Desktop and see if the pyTivo shares show up.


----------



## ClearToLand (Jul 10, 2001)

reneg said:


> ClearToLand said:
> 
> 
> > ...2. How often do you experience TS Sync Errors / need to run your BATCH file?
> ...


Let me rephrase question #2:

Do you run TS / 'Fast' Format transfers ALL of the time, or only when forced to (i.e. a H.264 show)?​
And a new question:

How often do you run your BATCH file and / or tivolibre to check a TiVo-to-PC TS / 'Fast' Format transfer for errors before deleting the original from your TiVo unit?​
What I'm attempting to determine is:
What percent of your TS / 'Fast' Format transfers are successful on the first attempt.
.
What's the average number of retries you see before you get a successful transfer?
If you're transferring everything via TS / 'Fast' Format transfers and you experience a 50% initial failure rate like I did, it would be simpler to switch to PS / 'Slow' Format transfers, like I did, and only use TS / 'Fast' Format transfers for H.264 material. Or purchase VideoReDo TV Suite v5 like @moyekj suggests.

Another recent discovery of mine, if you enjoy AutoSkip, is that .tivo files DO NOT contain "contentId" which the TiVo unit needs to contact the TiVo servers to restore the 'Skip' ability on the TiVo after you transfer your shows back. Kevin suggested decrypting to .mpg and saving the .mpg.txt file (which has the "contentId" field). I tested it once and it seemed to work. So now I have ~3TB of .tivo files offloaded for future 'Watch & Delete' WITHOUT AutoSkip  and 1 show with AutoSkip. If you decide to try this 'experiment', please let me know the results.

TTYL!


----------



## reneg (Jun 19, 2002)

ClearToLand said:


> Let me rephrase question #2:
> 
> Do you run TS / 'Fast' Format transfers ALL of the time, or only when forced to (i.e. a H.264 show)?​
> And a new question:
> ...


a) Comcast migration is complete for my area which means no PS on cable. I also have OTA setup for locals, but keep transfers in TS for simplicity. I've been a VideoRedo users for years.
b) Guessing roughly 75% on Roamio, 0% on Bolt on first attempt.
c) I don't really pay attention to number of retries. I run a two stage workflow. Stage 1: Transfers happen in the background. Stage 2: Skipmode/Review cuts, encode, & archive without commercials.


----------



## Dan203 (Apr 17, 2000)

FYI the new version of pyTivo Desktop includes built in VideoReDo post processing. So you can set the download retry to as many as you want, it'll keep going until it gets a clean copy or keep the one with the least number of errors and then it'll automatically QSF with VideoReDo to mitigate the errors if there are any. (or even transcode if you want)


----------



## murgatroyd (Jan 6, 2002)

Dan203 said:


> Those two can conflict. Try disabling TiVo Desktop and see if the pyTivo shares show up.


pyTiVo shares are showing fine now, thanks.


----------



## murgatroyd (Jan 6, 2002)

Okay, new problem. 

I had transferred a TV show marathon with kmttg, and am now transferring some of the episodes back to my Roamio basic for viewing. I copied the TiVo files and their corresponding metadata files from kmttg's output/tivo folder to the MyTiVoRecordings folder on my Win 8.1 desktop.

Now when I'm looking at Devices on the Roamio MyShows screen and asking the Roamio to transfer the episodes back, they are showing up in MyShows with the metadata for the next episode. 

I request S1 E10 from the list on my computer, and the TiVo file that transfers is S1 E10, but on MyShows, it appears as S1 E11, and so on. The Guide Data is always E+1 once it gets back to the Roamio. 

I did open copies of the *.txt files with Notepad to copy the metadata into my video cataloging software, but I didn't make any changes. If I want to read the metadata, is there a less destructive way to copy it?

I can live with this if it's not fixable, but it is annoying.


----------



## Dan203 (Apr 17, 2000)

Are you sure your txt files are named correctly?


----------



## mlippert (Apr 3, 2010)

murgatroyd said:


> Okay, new problem.
> 
> I had transferred a TV show marathon with kmttg, and am now transferring some of the episodes back to my Roamio basic for viewing. I copied the TiVo files and their corresponding metadata files from kmttg's output/tivo folder to the MyTiVoRecordings folder on my Win 8.1 desktop.
> 
> ...


First, opening the txt file and copying its contents and then closing it w/o saving will not have modified it in any way, and cannot be responsible for the behavior you are describing.

I'm assuming the tivo files in question were recorded since Sept 2016? (That was approximately the time of Tivo's program guide switch to Rovi and the txt metadata files downloaded before that time may have issues when uploaded back to the tivo.

In general the tivo no longer pays attention to the episode number uploaded. Instead it uses the programId and then looks up the season and episode number.

So I think it is more likely that for this series that Rovi's online guide data changed episode numbers after you downloaded them.

Now if this is happening to other series then I'm probably wrong and something else is going on.


----------



## murgatroyd (Jan 6, 2002)

mlippert said:


> First, opening the txt file and copying its contents and then closing it w/o saving will not have modified it in any way, and cannot be responsible for the behavior you are describing.
> 
> I'm assuming the tivo files in question were recorded since Sept 2016? (That was approximately the time of Tivo's program guide switch to Rovi and the txt metadata files downloaded before that time may have issues when uploaded back to the tivo.
> 
> ...


1. The marathon was on 6 Jun 2017, so after the switch to Rovi
2. I can't swear that I didn't re-save the *.txt files with Notepad but I was trying to just exit rather than re-saving them so they wouldn't get modified. 
3. I don't have other series to compare to yet, because I haven't been transferring shows back to the Roamio from the PC except for this one series. I haven't seen the behavior (at least not yet) while transferring shows to the Roamio from the TiVoHD.


----------



## mlippert (Apr 3, 2010)

murgatroyd said:


> 1. The marathon was on 6 Jun 2017, so after the switch to Rovi


I figured as much, but wanted to mention any remote possibility I could think of, just to rule them out.



murgatroyd said:


> 2. I can't swear that I didn't re-save the *.txt files with Notepad but I was trying to just exit rather than re-saving them so they wouldn't get modified.


Even if you re-saved them, the likelihood that you modified them in any meaning way is minuscule. Your method of copying the contents of the text files is not destructive.


murgatroyd said:


> 3. I don't have other series to compare to yet, because I haven't been transferring shows back to the Roamio from the PC except for this one series. I haven't seen the behavior (at least not yet) while transferring shows to the Roamio from the TiVoHD.


Transferring tivo to tivo would probably never exhibit this behavior because at the time of transfer they would both be getting the episode numbers from the same online source (I believe). I suspect there are other reasons as well in that the mechanism between TiVos is different from going from tivo to a PC and then back to a tivo.
I'm still curious for when you get the chance to try an episode from another show.


----------



## murgatroyd (Jan 6, 2002)

I still haven't had a chance to test transferring other shows, but for the marathon, the behavior has been consistent.

When I transferred S1E12, the Guide Data was correct on the screens where I chose the episode from the TiVoHD, but after the transfer started, the Roamio displayed the data for S1E13.

When I chose S1E13, the Roamio showed the Guide Data for the next episode, which was S2E01.


----------



## HerronScott (Jan 1, 2002)

We were just moved to MPEG4 so I tried downloading one of the newly recorded shows. It appeared to download fine with TiVo Desktop and I ran VideoRedo Quickstream Fix against it and it didn't report any issues. I'm guessing this was just luck of the draw?

I've got Dan's pyTivo Desktop installed so I'm ready. 

Scott


----------



## hjb47 (Jun 11, 2021)

Dan203 said:


> You know I added an automated version of this to pyTivo desktop? In the download section of the options you can set it to retry downloads as many times as you want until it gets a prestine copy. There is even an option to keep the file with the least number of errors if a prestine copy can't be aquired. And now it has built in VideoReDo support, so you can QSF your files after they download to minimize the damage caused by the bad packets if there are any.


Hi, is it possible you can also decrease the socket receive size on retry as well as disable the nagle algo? I'm unable to download some movies after 100 retries. I'm assuming you can get the socket from the object returned by tivo_opener.open(url).


----------



## Dan203 (Apr 17, 2000)

hjb47 said:


> Hi, is it possible you can also decrease the socket receive size on retry as well as disable the nagle algo? I'm unable to download some movies after 100 retries. I'm assuming you can get the socket from the object returned by tivo_opener.open(url).


Even with the slowdown you get errors?

I played with reducing the request size when I was writing this and it didn't seem to make a difference. A bigger request with a longer pause created the exact same result as a smaller request with a shorter pause.


----------



## hjb47 (Jun 11, 2021)

Dan203 said:


> Even with the slowdown you get errors?
> 
> I played with reducing the request size when I was writing this and it didn't seem to make a difference. A bigger request with a longer pause created the exact same result as a smaller request with a shorter pause.


Yes, I get errors with the slowdown. I set it to 100 retries


----------



## Dan203 (Apr 17, 2000)

hjb47 said:


> Yes, I get errors with the slowdown. I set it to 100 retries


Is it just one file? Could be an issue with the recording. Some people have had success transferring to another TiVo then transferring from there, if you have multiple TiVos.


----------



## hjb47 (Jun 11, 2021)

At least two for now. Always fails at the same spot. I do have multiple tivos so ill try the transfer to another tivo idea.


----------



## George John (Jun 19, 2021)

Hi, Dan203.

I'm new to pytivo as of yesterday, and like what I see so far! Although we have had a TiVo Bolt since 2018, the idea of copying files off it never crossed my mind, until the 3TB Western Digital Blue drive failed recently (losing about 225 hours of video. I repaired it myself with a 2TB Toshiba drive). I have searched for pytivo documentation, FAQs, and forums for my questions, with only partial success. So, I'm going to give it a try here. I apologize in advance if these questions have been answered many times before. Google is a good search engine but not perfect. And, my search skills definitely aren't perfect. 

No surprise, I sometimes get download errors (I'm using the TS option). My understanding is the original recording on the TiVo can have errors, and it does it's impossible to get an error free download. Is that correct? And, if so, any idea how often is this a problem? 

If the TiVo is recording a show and/or one is watching a show, does that increase the likelihood of errors? IOW, is it best to try to download when the TiVo is idle? I was hoping that a unit that can record six shows simultaneously would have enough horsepower to handle a download while viewing/recording. But, it also strikes me that TiVo did a really poor job providing a robust API, too. That's unfortunate. It makes me wonder how well their Android app does at downloading a file for offline viewing (I have never done this).

If repeated attempts at downloading a show have the exact number of errors, is it likely the errors are in the original TiVo recording? So far, that's what I'm seeing, and it's only with local OTA available channels that XFinity is transmitting through their cable system. Cable shows have been 'clean' so far.

Are the errors something I can possibly ignore? What's an acceptable error count for a one hour show? At a rate of roughly 60fps, a loss of say 300 frames is roughly 5 seconds, which may not be a big deal except for those who want a perfect recording. Note: I'm going to explore using VideoReDo to 'fix' the errors, and see how well it does at removing commercials (big plus if it does it well!!).

I have a NAS with PLEX. Having season and episode number metadata information is handy. I see from the documentation that's in the metafile spec (EpisodeNumber). But, the few files I have downloaded so far don't contain that information. And, the spec itself isn't totally clear to me. It looks like episode number is nn (reasonable). Is season number n or nn? Is Season 10 Episode 1, "1001" correct? I like how Plex recommends doing this with " (S10e01)" at the end of the file name. Seems to work well on my Plex server at least.

And, this is interesting. One recording on the TiVo is listed twice in your interface. And, when it's downloaded, it appears to be downloaded twice in parallel. 

About my system:

Hardwired 1000 mbps ethernet. Windows 10 PC with an Intel I9-11900k CPU and NVIDIA GeForce GTX 1080. The target drive is a Samsung Pro 980 Nvme running PCIe 4.0 (Z590 motherboard). So, I don't expect the computer/network to be the issue.

Finally, do you have a recommended donation amount? I wrote software for over 40 years (now retired). I like to support those who write quality software that performs a useful function. I'm currently not 100% sold on using the TiVo as my platform for long-term video. I'm going to give a SiliconDust HDHomeRun Prime a try soon (being shipped now). But, given TiVo's native lack of a backup option, and it appears to be hard on drives, I need an option to record and backup video that I really want, and not depend upon streaming it from somewhere in the future. If I do stick with the TiVo (likely because the wife will require it  ), I'll use your software to long-term archive to the NAS and offline backup the TiVo recordings. And, you will very likely get a donation.


----------



## Dan203 (Apr 17, 2000)

The errors reported by pyTivo are not errors in the video itself, they're errors in the TS structure. If there are errors in the video itself from recording those will not be reported by pyTivo.

There is a new feature in the latest version to slow down the transfer. This can help reduce the errors. Set the option to retry and slow the transfer on subsequent tries. Set the retry count to at least 15. That should get you a transfer with the fewest number of errors possible. But there is no way to guarantee they're error free.

The errors themselves are coming from the TiVo being over taxed. One thing that can help is to set all the tuners to a channel you don’t get so that it's not buffering and leave the TiVo itself alone. That should keep the CPU and drive usage to a minimum.

The bug with the double recording is interesting but likely coming from the TiVo itself. Behind the scenes I'm just querying XML data from the TiVo and converting it to JSON for the UI. I can’t think of anything in the UI itself that would cause duplicates. (but it's not impossible)

As for the donation it's completely up to you. Whatever you feel comfortable with. I've had them as low as a few dollars and as high as a hundred. All are appreciated as they help pay for the hosting and certificates.


----------



## George John (Jun 19, 2021)

Dan203 said:


> The errors reported by pyTivo are not errors in the video itself, they're errors in the TS structure. If there are errors in the video itself from recording those will not be reported by pyTivo.
> 
> There is a new feature in the latest version to slow down the transfer. This can help reduce the errors. Set the option to retry and slow the transfer on subsequent tries. Set the retry count to at least 15. That should get you a transfer with the fewest number of errors possible. But there is no way to guarantee they're error free.
> 
> ...


Thank you for your reply, and your suggestions.

I did try setting the retries to 30, and enabled the slowing option after each failure. By the 30th try it was quite a bit slower.  And, it failed. :-(

I'll find a time when the TiVo is inactive and test again. I'm uncertain how to set all six tuners to a channel we don't get, however. Unless you mean a premium channel we don't subscribe to, like HBO.

It's unfortunate you can't ask the TiVo box for specific packets that failed. It sounds like all you can do is request a file, and the process allows only so much time for any given packet or group of packets to arrive. If they don't, they are lost. OTOH, that makes no sense. I wonder what the issue actually is? If this were sending a file to the TiVo, it would make more sense. But, the PC should easily be fast enough to handle the data coming from the TiVo. Is is?

Maybe I'll download the code and take a look myself.  This has me puzzled.


----------



## Dan203 (Apr 17, 2000)

Yes set the tuners to any channel you don’t get, HBO will work.

It's actually the opposite. There isn’t a time limit for grabbing the packets, it fails when you attempt to grab them too quickly. Hence the option to slow it down. The amount of data you grab *might* also matter. I'm using a fixed size that balanced what worked and that roughly caused the speed to drop by 1/2 on each iteration. 

If you do decide to play with the code just keep in mind the chunks have to be a multiple of 188. That's the size of a TS packet and if you grab a partial packet the code that checks for the corruption will fail.


----------



## BilliJoe (Oct 2, 2016)

George John said:


> Thank you for your reply, and your suggestions.
> 
> I did try setting the retries to 30, and enabled the slowing option after each failure. By the 30th try it was quite a bit slower.  And, it failed. :-(
> 
> ...


Gave up doing that a long time ago since a TiVo corrupts most TS downloads.
Setting all tuners to a channel you don't receive doesn't help, that only sped up downloads on S3's.
TS downloads from earlier models like a Premier or Roamio are more likely to succeed with pyTiVoDesktop retries than a Bolt.
If you have a HDHR tuner and want to keep your recordings, checkout Channels - Live TV, everywhere DVR Server.
It will record Non-DRM channels directly to your hard drive.
It also supports TVE, m3u and Playon sources in addition to an HDHR source.

I went from ancient systems to TiVo's to Plex DVR to SiliconDust DVR to Channels DVR and am quite happy with my choice.


----------



## George John (Jun 19, 2021)

BilliJoe said:


> Gave up doing that a long time ago since a TiVo corrupts most TS downloads.
> Setting all tuners to a channel you don't receive doesn't help, that only sped up downloads on S3's.
> TS downloads from earlier models like a Premier or Roamio are more likely to succeed with pyTiVoDesktop retries than a Bolt.
> If you have a HDHR tuner and want to keep your recordings, checkout Channels - Live TV, everywhere DVR Server.
> ...


BilliJoe,

Thanks for your reply.

I guess great minds think alike. ;-) I am now a VERY happy owner of a SiliconDust HDHomeRun Prime. It works beautifully with my Synology NAS. So far I have tested Windows 10 PC, Android phone, and Amazon Fire TV clients using their software, and they all work very well. And, it is playing well with my PLEX and DS Video apps well. The miscellaneous other DLNA apps work selectively well, but are a very low priority.

So, the only thing the TiVo has going for it for *me* is the commercial skip. I gave VideoReDo a try for editing out commercials. That did not go well. :-(

Since I purchased an all-in subscription in 2018, the TiVo isn't costing me anything. And, the wife knows how to use it. So, as long as it continues to function (I have already replaced a dead hard drive myself for $60 and a loss of 1TB of storage), we'll continue to use it. But, for any recording that I *really* don't want to lose, I'll use the HDHomeRun Prime. And in general, anything that either I or the wife might want to watch more than once, I'll probably made a copy with the HDHomeRun Prime. The files on the Synology I can *easily* and *reliably* backup.


----------



## George John (Jun 19, 2021)

Dan203 said:


> Yes set the tuners to any channel you don't get, HBO will work.
> 
> It's actually the opposite. There isn't a time limit for grabbing the packets, it fails when you attempt to grab them too quickly. Hence the option to slow it down. The amount of data you grab *might* also matter. I'm using a fixed size that balanced what worked and that roughly caused the speed to drop by 1/2 on each iteration.
> 
> If you do decide to play with the code just keep in mind the chunks have to be a multiple of 188. That's the size of a TS packet and if you grab a partial packet the code that checks for the corruption will fail.


Dan,

Thanks for the tips.

I did attempt to compile the project, and failed after several attempts. I *did* read your README. And, while the statement that at least version 2.5 of Python is required is a factually correct statement , it doesn't say that a version less than 3.0 is necessary, too. The current version is 3.9, which of course fails. The 2.7 version is at "end of life". The Microsoft Visual Studio version of 2.7 has no security patches.

Note: I downloaded what I hope is the correct source from here: Dan203/pytivo

So, I *think* I have the latest build of yours, but who knows. The README and pytivo.py have no version or build number comments (possibly by design?). My attempts to run with Python 2.7 IDE from the URL you have in the README, and the Microsoft VS IDE 2.7 both fail early on. Note: 2.7 is the latest/only build that I can find that falls between 2.5 and 3.0.

While over my 40+ years as a software developer I worked with at least 22 different development languages, unfortunately PYTHON hasn't been one of them. So, while I expect I'm making a total n00b mistake, this isn't a high priority. The failure is in "import beacon" and on "import enum". This suggest to me that the installation software for 2.7 likely didn't set some PATH variable correctly. I didn't mess with any of the defaults. So, this is both a surprise, and not a surprise at all. As they say, you get what you pay for. 

I'll play with this a bit more because I'm stubborn. From a Google search, I can see I'm not the only person who has fallen down this rabbit hole. ;-) Hopefully, this is something simple. We will see.


----------



## George John (Jun 19, 2021)

Dan,

This is quite a mystery. Enum was implemented in version 3.4. How this works in version >= 2.5 and < 3.0 is beyond me. You are magical.  (Or, I'm an idiot. Probably the latter )

enum - Support for enumerations - Python 3.9.5 documentation


----------



## Dan203 (Apr 17, 2000)

I used 2.x because that's what the original pyTivo was written in. I didn’t write most of pyTivo. I mainly worked on the download part, and a few minor fixes, then all the stuff that allows the UI to function. The UI is actually in Angular, so it's basically just a fancy website that talks to the pyTivo server of http. 

There is someone in the underground who ported pyTivo to python 3.x. Not sure if it's up to date with my recent slowdown changes though, and if it's not the UI will fail because it wont be able to get/set all the options properly.


----------



## mlippert (Apr 3, 2010)

George John said:


> (I have already replaced a dead hard drive myself for $60 and a loss of 1TB of storage)


George, what model drive did you get to replace the dead one? Finding an appropriate drive to fix my dead Bolt has been difficult (it's my backup TiVo, but I'd still like it to be working again) since I've read you shouldn't use a drive which uses SMR, but it has been tough to find a 2-4GB CMR drive.


----------



## pl1 (Jan 18, 2007)

mlippert said:


> George, what model drive did you get to replace the dead one? Finding an appropriate drive to fix my dead Bolt has been difficult (it's my backup TiVo, but I'd still like it to be working again) since I've read you shouldn't use a drive which uses SMR, but it has been tough to find a 2-4GB CMR drive.


The Toshiba MQ03ABB200 2TB for $53.99 on Amazon is the recommended drive, or else you have to go with an external drive and external case with an internal SATA cable.

EDIT: For the External 3.5" Drives, the recommended ones are Western Digital Red Plus or Purple. (The Bolt uses a 2.5" laptop sized drive.)


----------



## mlippert (Apr 3, 2010)

Dan203 said:


> There is someone in the underground who ported pyTivo to python 3.x. Not sure if it's up to date with my recent slowdown changes though, and if it's not the UI will fail because it wont be able to get/set all the options properly.


I'm one of the people who ported pytivo to python 3.x. It doesn't have the very latest changes Dan made for slowing things down, but does have the changes to retry, along with other enhancements I wanted, such as configurable file naming. It also probably doesn't have all the support Dan's UI needs, so you only have the original web interface available.

My fork is available if you wanted to go that direction at: mlippert/pytivo
I am running a Kubuntu box as my main environment at home, and I use this code with Python 3.9 to download anything that requires a TS download from my TiVos. I haven't had a Windows box in a while, and I don't use Apple products. I'm willing to try to help you get it running on Windows if you wanted.

I know there is at least one other pytivo port to python 3.x which has different functionality than mine.


----------



## mlippert (Apr 3, 2010)

pl1 said:


> The Toshiba MQ03ABB200 2TB for $53.99 on Amazon is the recommended drive, or else you have to go with an external drive and external case with an internal SATA cable.
> 
> EDIT: For the External 3.5" Drives, the recommended ones are Western Digital Red Plus or Purple. (The Bolt uses a 2.5" laptop sized drive.)


That was all the information I needed. Thank you so much! I just ordered the Toshiba drive.


----------



## ClearToLand (Jul 10, 2001)

Hi George,

TS Sync Errors (*PyTiVo Desktop / "0x47 every 188" / TS Sync Errors: Do You Use It?*) is a subject near and dear to my heart every since I got back into using TiVos in 2015 with the purchase of a Roamio Basic w/Lifetime. You sound technical so I'll babble on a bit...



George John said:


> ...*I'm new to pytivo* as of yesterday, and like what I see so far! Although we have had a *TiVo Bolt* since 2018, the idea of copying files off it never crossed my mind, until the *3TB Western Digital Blue drive failed* recently (losing about 225 hours of video. I repaired it myself with a 2TB Toshiba drive). *I have searched for pytivo documentation, FAQs, and forums for my questions, with only partial success*. So, I'm going to give it a try here. I apologize in advance if these questions have been answered many times before. *Google is a good search engine but not perfect. And, my search skills definitely aren't perfect*.


First, *PyTiVo DESKTOP* is not *PyTiVo* - it's a fork from spring 2017 by @Dan203 (EXCELLENT addition to the TiVo Community! :thumbsup: ). PyTiVo (most popular fork by @wmcbrine ) is a Python program that uses Python 2.5 - 2.7 and requires a somewhat detailed manual installation by the user (read the Wiki). PyTiVo Desktop took that program and wrapped it up, with Python 2.7, into a self-installing package with GUI that just about any non-techie could use.

I was using PyTiVo before PyTiVo Desktop was created. Thus, I already had a Python 2.7 environment setup. Back in 2017 I was running all my TiVo 'accessories' (kmttg, PyTiVo, etc...) on a 32-bit Vista Desktop. PyTiVo Desktop didn't like Vista so since I was desperate to get the *"0x47 every 188"* logic running for my TS (Transport Stream) transfers (to get non-garbled closed captions), I needed to run it from the IDE. I suggest getting PyTiVo up and running first. Then using a dual-window editor, bring up the source code to pyTivo.py in both to see the differences, and make the (minimal) necessary changes to get PyTiVo Desktop to run. I also suggest putting PyTiVo on port 9033 and leaving PyTiVo Desktop on port 9032 so you can run them side-by-side.



George John said:


> No surprise, I sometimes get download errors (I'm using the TS option). My understanding is the original recording on the TiVo can have errors, and it does it's impossible to get an error free download. Is that correct? And, if so, any idea how often is this a problem?


I don't believe that the original mpeg-2 stream recorded on the TiVo unit has errors (unless your Signal sucks!  ). But I do theorize that the HDD in the TiVo unit could have bad sectors and / or fragmented files causing the TiVo OS to struggle when it's attempting to re-package the mpeg-2 stream into a Transport Stream container. Note, it appears to have 'different' problems when re-packaging the mpeg-2 stream into a PS (Program Stream) container - it garbles the closed captions.

As to how often do TS Sync Errors occur? Some folks say they have none while others have TONs, especially on Bolts. This post is already going to take a good hour (or TWO) to compose so just SEARCH TCF for '*TS Sync Errors*' and my UserID starting from 2017 and your find plenty of posts.



George John said:


> If the TiVo is recording a show and/or one is watching a show, does that increase the likelihood of errors? IOW, is it best to try to download when the TiVo is idle? I was hoping that a unit that can record six shows simultaneously would have enough horsepower to handle a download while viewing/recording. But, it also strikes me that TiVo did a really poor job providing a robust API, too. That's unfortunate. It makes me wonder how well their Android app does at downloading a file for offline viewing (I have never done this).


Many recommend having the TiVo unit idle when downloading. I don't see much of a difference. My best downloads come when I enable '*QoS Rate Limiting*' on my Managed Switches (Netgear and TP-Link). When that doesn't work, I clear off ~100GB on another TiVo unit (force erase via Permanently Delete in kmttg) to attempt to get contiguous space, copy the show there, and re-try. Most times it works - sometimes it never works (2 out of 100).



George John said:


> If repeated attempts at downloading a show have the exact number of errors, is it likely the errors are in the original TiVo recording? So far, that's what I'm seeing, and it's only with local OTA available channels that XFinity is transmitting through their cable system. Cable shows have been 'clean' so far.


See comment about sector errors and / fragmentation above. Also, I've had more problems with H.264 source than MPEG-2 - check what your tuners are recording.



George John said:


> Are the errors something I can possibly ignore? What's an acceptable error count for a one hour show? At a rate of roughly 60fps, a loss of say 300 frames is roughly 5 seconds, which may not be a big deal except for those who want a perfect recording. Note: I'm going to explore using VideoReDo to 'fix' the errors, and see how well it does at removing commercials (big plus if it does it well!!).


I'm OCD (and retired) so I spend *ENORMOUS* amounts of time trying to get an error-free download. The movie 2001 from TCM, SD MPEG-2, is killing me. I think at last check I'm down to 2 or 3 errors and just about ready to give up. I did a LOT of research into Transport Stream containers back in 2017, forgotten most of it by now, but what remains it how much overhead there is in every 188 bytes. When you discard a packet, you're not discarding 188 bytes of movie - subtract the headers / footers / preambles etc... Then, estimate when in time that error is, watch the show on your PC and see if the 'glitch' bothers you. It could even be in a commercial that you might be cutting out anyway.

VideoReDo, and other 'methods' (DirectShow vs tivolibre vs tivodecode), don't 'fix' errors, usually. Well, VideoReDo is said to try and it uses a proprietary algorithm in DirectShow to do its best to avoid dropping the packet but all of the actual 'decoders' just drop the packets, more packets as you move left to right for the group in parenthesis. This is all 'hidden' in this forum - knowledge freely shared once or twice or the decade(s) and you just have to be persistent enough in your reading to find it.  Many answers are too short and although good-intentioned, usually misleading (or sometimes plain wrong  ).



George John said:


> I have a NAS with PLEX. Having season and episode number metadata information is handy. I see from the documentation that's in the metafile spec (EpisodeNumber). But, the few files I have downloaded so far don't contain that information. And, the spec itself isn't totally clear to me. It looks like episode number is nn (reasonable). Is season number n or nn? Is Season 10 Episode 1, "1001" correct? I like how Plex recommends doing this with " (S10e01)" at the end of the file name. Seems to work well on my Plex server at least.


Use *kmttg* to get the metadata via an RPC call. Then use PyTiVo Desktop to download the TS file. Then rename the file with the kmttg-generated name and decode it in kmttg with DirectShow. I customized the kmttg file naming to match my Plex Server requirements and I also have kmttg extract a closed caption file. All of this is transferred to my NAS and plays just fine on my tablets.



George John said:


> ...I'm going to give a SiliconDust HDHomeRun Prime a try soon (being shipped now). But, given TiVo's native lack of a backup option, and it appears to be hard on drives, I need an option to record and backup video that I really want, and not depend upon streaming it from somewhere in the future. If I do stick with the TiVo (likely because the wife will require it  ), I'll use your software to long-term archive to the NAS and offline backup the TiVo recordings...


I have 5 SiliconDust units - 3 Duals, 1 Prime, 1 Quattro. I had 2 Duals running Windows Media Center on the Vista PC before I bought the Roamio Basic. I still haven't taken the Prime out of the box. I 'tested' the Quattro to stream SD live to my tablets. As soon as I get a '*Round Tuit*', I plan on trying something like NextPVR and since I have Plex Pass Lifetime, Plex DVR. If they work out, a trial run of Channels DVR $$$ is on the list.

TiVo units themselves are not hard on drives - putting a 3TB 2½" in a hot little box and running it 24x7 is hard on a HDD. The 3½" HDDs in many other TiVo units throughout the decades last, well, decades. I'm only now starting to see what I interpret as sector errors on my Living Room Roamio after 6 years of 24x7 with a lot of daily recording. I have a WD30EFRX ready to pop in. 

I hope you appreciate the effort involved in creating this post and that you (and others) benefit from the knowledge I've acquired from reading the TCF forums for hundreds of hours. 



Spoiler



Hope there aren't any typos - my eyes (and back) are tired...


----------



## ClearToLand (Jul 10, 2001)

George John said:


> ...I did try setting the retries to 30, and *enabled the slowing option* after each failure. By the 30th try it was quite a bit slower.  And, it failed. :-(


Do you own a Managed Switch?

Do you own another TiVo unit?



George John said:


> ...
> I'll find a time when the TiVo is inactive and test again. I'm uncertain *how to set all six tuners to a channel* we don't get, however. Unless you mean a premium channel we don't subscribe to, like HBO.


Press '1' on your TiVo Remote. That should set the active tuner to Channel 1 (usually doesn't exist). Reboot the TiVo. That should set all the tuners to the channel the active tuner was set to before the reboot (useful during Tuner Diagnostics to check if certain tuners are weaker / stronger than others).



George John said:


> ...
> It's unfortunate you can't ask the TiVo box for specific packets that failed. *It sounds like all you can do is request a file, and the process allows only so much time for any given packet or group of packets to arrive.* If they don't, they are lost. OTOH, that makes no sense. I wonder what the issue actually is? If this were sending a file to the TiVo, it would make more sense. But, the PC should easily be fast enough to handle the data coming from the TiVo. Is is?


The issue is the TiVo OS and how it handles 'stress' and errors (sector, fragmentation). AFAICT, it just 'gives up' and continues to march on. Slowing the data transfer rate on my Managed Switches gives it more time between packets. Moving the file to another TiVo unit 'defragments' the file (provided that you pre-cleared enough contiguous space). If it *STILL* has errors, I'm guessing that the original HDD had a bad sector and there's no good data there for the TiVo unit to package up for the transfer to PC.



George John said:


> ...
> Maybe I'll download the code and take a look myself.  This has me puzzled.


Guidance / suggestions given in earlier post...


----------



## ClearToLand (Jul 10, 2001)

George John said:


> ...So, the only thing the TiVo has going for it for *me* is the commercial skip. *I gave VideoReDo a try for editing out commercials. That did not go well.* :-(


What kind of problems did you have with VideoReDo?

The majority of folks here that buy it like it and recommend it to others. When folks have excessive TS Sync Errors and lose 5 minutes of a 60 minute show (i.e. kmttg TS users), they get VideoReDo to 'clean up' the file (as opposed to using PyTiVo Desktop in Retry Mode).

I was under the impression that VideoReDo did a much better job than ffmpeg / ComSkip for finding and cutting out commercials. The few times that I've used ffmpeg / ComSkip, I wasn't too excited on how the cut came out. IIRC, there was a sudden burst of audio at each cut. As I understand it, it has something to do with cutting at the proper 'Master' full frame (too lazy to Google the proper terms; GOP, B frame, I frame  ) and not one of the 15 or so partial frames.

@Dan203 - does VideoReDo help the users find this 'Master' full frame to cut at? During playback, are the cuts transparent / silent except for the 'jump' in the timeline?

I missed the one sale on VideoReDo a few years back and watch as I can, I don't believe there was ever another. I guess it's doing well. 



George John said:


> ...for any recording that I *really* don't want to lose, I'll use the HDHomeRun Prime. And in general, anything that either I or the wife might want to watch more than once, I'll probably made a copy with the HDHomeRun Prime. *The files on the Synology I can *easily* and *reliably* backup*.


Does ccextractor.exe (included with kmttg) work to extract closed captions from files downloaded from SiliconDust tuners?


----------



## ClearToLand (Jul 10, 2001)

George John said:


> ...*I did attempt to compile the project, and failed after several attempts.* I *did* read your README. And, while the statement that at least version 2.5 of Python is required is a factually correct statement , it doesn't say that a version less than 3.0 is necessary, too. The current version is 3.9, which of course fails. The 2.7 version is at "end of life". The Microsoft Visual Studio version of 2.7 has no security patches.
> 
> Note: I downloaded what I hope is the correct source from here: Dan203/pytivo
> 
> So, I *think* I have the latest build of yours, but who knows. The README and pytivo.py have no version or build number comments (possibly by design?). *My attempts to run with Python 2.7 IDE from the URL you have in the README, and the Microsoft VS IDE 2.7 both fail early on.* Note: 2.7 is the latest/only build that I can find that falls between 2.5 and 3.0.


As I suggested in my first post today, go to the PyTiVo Wiki and install the @wmcbrine version. That will help you get Python 2.7 installed properly. IIRC there were 4 or 5 'additions' that had to be added to the base install to eliminate run errors. Once that's running, you bring up the @Dan203 version in a dual-window difference editor and make the slight changes to get that running.

I'm not a Python or Linux or anything in the past 20 years programmer - Fortran, COBOL, IBM ASM (or whatever IBM 360 Assembly Language was called), JCL, SQL, Basic, DOS Batch Files  . Everything I learn now is via Google. 



George John said:


> ...
> While over my 40+ years as a software developer I worked with at least 22 different development languages, unfortunately PYTHON hasn't been one of them. So, while I expect I'm making a total n00b mistake, this isn't a high priority. *The failure is* in "import beacon" and *on "import enum"*. This suggest to me that the installation software for 2.7 likely didn't set some PATH variable correctly. I didn't mess with any of the defaults. So, this is both a surprise, and not a surprise at all. As they say, you get what you pay for.


*enum* rings a bell. IIRC, there were 4 or 5 'items' that needed to be manually added to the base Python 2.7 install. Now that he's appeared, @mlippert might be able to assist. I have the notes 'somewhere', but that scrap of paper was written in 2017. 



George John said:


> ...This is quite a mystery. *Enum was implemented in version 3.4. How this works in version >= 2.5 and < 3.0 is beyond me.* You are magical.  (Or, I'm an idiot. Probably the latter )
> 
> enum - Support for enumerations - Python 3.9.5 documentation


As I said, *enum* rang a bell. Beyond that, I'd get an error and then Google it...


----------



## George John (Jun 19, 2021)

mlippert said:


> George, what model drive did you get to replace the dead one? Finding an appropriate drive to fix my dead Bolt has been difficult (it's my backup TiVo, but I'd still like it to be working again) since I've read you shouldn't use a drive which uses SMR, but it has been tough to find a 2-4GB CMR drive.


I bought it from Amazon for $59.99 + tax. This is their description:

*Toshiba MQ03ABB200 2TB 5400RPM 16MB Cache SATA 6.0Gb/s 2.5in Hard Drive (15mm Thickness for TiVo or MiniPC only, not for Laptop) - 3 Year Warranty*


----------



## George John (Jun 19, 2021)

Dan203 said:


> I used 2.x because that's what the original pyTivo was written in. I didn't write most of pyTivo. I mainly worked on the download part, and a few minor fixes, then all the stuff that allows the UI to function. The UI is actually in Angular, so it's basically just a fancy website that talks to the pyTivo server of http.
> 
> There is someone in the underground who ported pyTivo to python 3.x. Not sure if it's up to date with my recent slowdown changes though, and if it's not the UI will fail because it wont be able to get/set all the options properly.


Dan,

I very much appreciate your responses. Thank you.

First, what still has got me scratching my head is how the enum feature, which was introduced in version 3.4, works with your 2.x development environment. That is truly magical to me.

Based on your and ClearToLand's comments, it sounds like the TiVo api and infrastructure are not robust. Life is too short for me.  Once upon a time, I was paid serious money to deal with headaches like this. Been there, done that. Too many scars. ;-)

The SiliconDust HDHomeRun Prime continues to work very well. The only possible gotcha I have found so far is a MPEG-2 that I converted to MPEG-4 using Handbrake ceased to work with their software. That's not a big deal at all. I will inquire into it on their forum. Note: the converted file does still playback using VLC and other media players.

Given that Python 2 is no longer supported as of January 1, 2020 ("end of life" ), *especially security patches*, I'm going to drop my investigation, which is motivated now out of curiosity only. I now have an alternative to the TiVo with a solidly reliable copy and backup option. That said, if you ever do migrate your code to the latest version (currently 3.9.5), please let me know. I am still curious about this lost packet issue.

Finally, interesting you mentioned Angular. What now seems like a long while ago, I investigated version 2. I was shocked to see that Google essentially did a complete redo from version 1, and huge rewrites were necessary. Compared to the professional languages that I was using, it struck me as a monumental pain to use. Although, maybe with a very good toolkit, it might have been okay. I now see it's a version 12. Wow! I expect (and certainly hope), it has improved a lot since version 2. Then again, I was primarily a database and desktop programmer. Web software development has always struck me as absurdly tedious and messy.


----------



## George John (Jun 19, 2021)

ClearToLand said:


> Hi George,
> 
> [SNIP]
> 
> I hope you appreciate the effort involved in creating this post and that you (and others) benefit from the knowledge I've acquired from reading the TCF forums for hundreds of hours.


I do appreciate it. Thanks!


----------



## George John (Jun 19, 2021)

mlippert said:


> I'm one of the people who ported pytivo to python 3.x. It doesn't have the very latest changes Dan made for slowing things down, but does have the changes to retry, along with other enhancements I wanted, such as configurable file naming. It also probably doesn't have all the support Dan's UI needs, so you only have the original web interface available.
> 
> My fork is available if you wanted to go that direction at: mlippert/pytivo
> I am running a Kubuntu box as my main environment at home, and I use this code with Python 3.9 to download anything that requires a TS download from my TiVos. I haven't had a Windows box in a while, and I don't use Apple products. I'm willing to try to help you get it running on Windows if you wanted.
> ...


That's great you upgraded it to 3.9! Does it have the issue with losing packets?

I'm going to download it and give it a try now. I'll let you know how it goes. Thanks!


----------



## George John (Jun 19, 2021)

ClearToLand said:


> What kind of problems did you have with VideoReDo?
> 
> [SNIP]
> 
> Does ccextractor.exe (included with kmttg) work to extract closed captions from files downloaded from SiliconDust tuners?


The only value VideoReDo would have for me is if it were really good at identifying commercials and allow me to easily verify what it did, and correct its error. I wasn't good enough *for me* at any of these tasks.

I know nothing about ccextractor, or kmttg. Sorry. Just curious, why do you extract closed captions? That would be an interesting way to get a transcript of something, say from a news show or speech.

And, from your other post's questions, I don't have a managed switch, and I only have one TiVo. I'm sure I have missed other questions, too.


----------



## Wil (Sep 27, 2002)

ClearToLand said:


> I spend *ENORMOUS* amounts of time trying to get an error-free download


Some of the old timers still use mfs_ftp.tcl or one of its variations (I use tyftpd.tcl) to download shows from modified Tivo s3's or the model HD (their "harvesting" Tivos), and then convert the resulting .ty or .ty+ files to .ts (via 3tots) and move the shows to their more modern Tivo networks for family consumption.

I was so concerned about these reported errors that I went back and started doing the old way again. But it was just too many steps and "hands-on" so I gave it up since I wasn't really seeing the errors to start with. I'm just so compulsive I wanted guaranteed perfect files.

Big problem in doing that from scratch is that the soldering required to adjust the s3/HD is really delicate. There are no modded machines on the used market (landfilled or being used by people who won't part with them).

Most of my shows are recorded on a Roamio so that may account for my not seeing errors. But in a vacation home we have a Bolt and while I do see errors and get files that fail to play well in certain non-Tivo environments, they are relatively rare and they seem isolated to a couple of specific channels.


----------



## George John (Jun 19, 2021)

mlippert said:


> I'm one of the people who ported pytivo to python 3.x.
> 
> [SNIP]
> 
> ...


FYI, I did give your code a try using the Windows version of Python 3.9.5. It's having issues (different), too. Sorry.


----------



## mlippert (Apr 3, 2010)

George John said:


> That's great you upgraded it to 3.9! Does it have the issue with losing packets?
> 
> I'm going to download it and give it a try now. I'll let you know how it goes. Thanks!


The losing packets issue is a TiVo problem. My port did include all of @Dan203 's code to recognize bad packets, and has additional reporting of what it found in a file.

As for those issues, yeah, I never wrote up robust instructions to get it running or really cleaned up the README I started with initially.

The issue you encountered is that you need to install the required python dependency packages. Usually you use pip to do this (pip install -r requirements.txt), and there is a requirements.txt file that specifies the necessary packages.

Dan's version is pre-packaged with everything needed, so it's actually user friendly for non-developers to get running.


----------



## mlippert (Apr 3, 2010)

@George John If you care here's an example report attempting to download a file from my TiVo which never got a clean copy, although I seem to mostly succeed in the 4 attempts of getting a copy with no packet errors. As people have said speed is a factor, I'm running pytivo on my machine connected to my LAN by 2.4GHz wifi channel, and my TiVo is connected to my LAN via a 5GHz wifi channel.

My version is set to abort an attempt as soon as the number of error packets is greater than the previous best attempt.
But you can see that while the errors are in somewhat consistent locations in the file, they do differ from one attempt to the next.


```
%YAML 1.2
---
fileName            : "Kim Possible Movie - So the Drama (2005) (Apr_26_2021, DISNEYHD-E).tivo"
fileSize            : 2898013680
tivoName            : LivingRoomRoamio (192.168.111.81)
downloadStarted     : 2021-05-01T17:00:26Z
attemptSaved        : 1
totalErrorPackets   : 103
downloadAttempts:
    - attemptNumber : 1
      status        : sync_errors_saved
      transfer      : { bytes:  2898013680, seconds: 3593.1, rate: "  6.15 Mb/s" }
      errorPackets:
          - { count:      2, start:  1616154480, end:  1616154856, startMB:  1541.29 }
          - { count:     51, start:  2018055992, end:  2018065580, startMB:  1924.57 }
          - { count:     50, start:  2418273212, end:  2418282612, startMB:  2306.25 }
    - attemptNumber : 2
      status        : sync_errors_aborted
      transfer      : { bytes:  2014306896, seconds:  517.7, rate: " 29.69 Mb/s" }
      errorPackets:
          - { count:     53, start:  1215169092, end:  1215179056, startMB:  1158.88 }
          - { count:     46, start:  1215179244, end:  1215187892, startMB:  1158.89 }
          - { count:     51, start:  2018055992, end:  2018065580, startMB:  1924.57 }
    - attemptNumber : 3
      status        : sync_errors_aborted
      transfer      : { bytes:  2417373632, seconds:  677.1, rate: " 27.24 Mb/s" }
      errorPackets:
          - { count:     53, start:  1215169092, end:  1215179056, startMB:  1158.88 }
          - { count:     46, start:  1215179244, end:  1215187892, startMB:  1158.89 }
          - { count:     50, start:  2418273212, end:  2418282612, startMB:  2306.25 }
    - attemptNumber : 4
      status        : sync_errors_aborted
      transfer      : { bytes:   802485968, seconds:  203.1, rate: " 30.15 Mb/s" }
      errorPackets:
          - { count:     65, start:   803791816, end:   803804036, startMB:   766.56 }
          - { count:     81, start:   803804224, end:   803819452, startMB:   766.57 }
...
```


----------



## George John (Jun 19, 2021)

mlippert said:


> [SNIP]
> 
> The issue you encountered is that you need to install the required python dependency packages. Usually you use pip to do this (pip install -r requirements.txt), and there is a requirements.txt file that specifies the necessary packages.
> 
> [SNIP]


Thanks for that insight. Having now read your README more carefully, I see that's in there, too.

As I said, I'm a total python n00b. The good news I installed the dependencies and your project now runs without error. The bad news is I need to, after all these years of avoiding it, finally learn python.

It feels like going back to my programming years in the 1980's, when my main development environment was UNIX and using the command line primarily. This feels very retro to me.  But, this is so popular. I guess I should finally learn why beyond it's free, open source, and multi-platform.

Thanks again for your help. It's very much appreciated. I would have responded earlier. But, I was in another deep rabbit hole. It involved MakeMKV and ripping Ultra HD Blu-ray discs. Good times. I so love the bleeding edge of technology. ;-) But, Intel dropping SGX support from the 11th generation finally forces my hand. But, that's another sad tale of woe. ;-)

And, in other unrelated news. Not that I need it, but last night I accidentally discovered first-hand that PLEX will use my SiliconDust HDHomeRun Prime. That was a cool to see.


----------

