# Clock off in Roamio by 2 minutes



## PSU_Sudzi (Jun 4, 2015)

I remember seeing a post about this with Bolts but not Roamios that I can recall. Just noticed tonight, anyone else? On version 20.7.4.RC18.


----------



## kpeters59 (Jun 19, 2007)

Force another connection to TiVo?

-KP


----------



## NorthAlabama (Apr 19, 2012)

PSU_Sudzi said:


> I remember seeing a post about this with Bolts but not Roamios that I can recall. Just noticed tonight, anyone else? On version 20.7.4.RC18.


the time is synchronized with tivo's servers during connections, so this issue isn't isolated to any one model or software version - it effects everyone who connects to the server(s) that's time is off.

the only solution is to force connections (like @kpeters59 suggested) until you connect to a server with accurate time, then report the issue, and wait until tivo adjusts the time on the affected server(s).

if you report it to tivo, don't get roped into any frustrating troubleshooting nonsense other than forcing connections - reboots, repeating guided setup, etc. won't help.


----------



## PSU_Sudzi (Jun 4, 2015)

NorthAlabama said:


> the time is synchronized with tivo's servers during connections, so this issue isn't isolated to any one model or software version - it effects everyone who connects to the server(s) that's time is off.
> 
> the only solution is to force connections (like @kpeters59 suggested) until you connect to a server with accurate time, then report the issue, and wait until tivo adjusts the time on the affected server(s).
> 
> if you report it to tivo, don't get roped into any frustrating troubleshooting nonsense other than forcing connections - reboots, repeating guided setup, etc. won't help.


Thanks for the tips, I don't think I ever noticed it happening before to me so I guess today I just hit a bad server at TiVo!


----------



## Razzer (Nov 5, 2015)

PSU_Sudzi said:


> Thanks for the tips, I don't think I ever noticed it happening before to me so I guess today I just hit a bad server at TiVo!


Happened to me today on Roamio OTA.

Sorry to say that a number of lengthy historical threads discuss customer frustration with this issue, spanning several years.


----------



## ClearToLand (Jul 10, 2001)

PSU_Sudzi said:


> I remember seeing a post about this with Bolts but not Roamios that I can recall. Just noticed tonight, anyone else? On version 20.7.4.RC18.


IIRC, this was a popular / hot topic several months ago and @sfhub (if you want to SEARCH TCF) did quite a bit of diagnosis on NTP servers, various time differences, how far TiVo would allow one of their servers to get out-of-sync before correcting it, etc... Also, by going into the TiVo logs, @JoeKustra found the IPs that were suspected of providing the incorrect time.

Just keep forcing connections until you hit a server with the CORRECT time.  I always use S-P-S-9-S to display the time; besides telling me how far into the show I am and the TiVo clock's time, if it disappears I know that my TiVo unit rebooted.


----------



## sfhub (Jan 6, 2007)

PSU_Sudzi said:


> I remember seeing a post about this with Bolts but not Roamios that I can recall. Just noticed tonight, anyone else? On version 20.7.4.RC18.


As of 11:58pm Eastern the 3 TiVo NTP servers (sjr1|sjr2|sjr3).tivo.com are within 1.5seconds of time.nist.gov

If you connect now your time should be fine.


----------



## kdmorse (Jan 29, 2001)

They look fine now, but sjr1's been ruining up to 120second off today. sjr2's been, meh. sjr3's been ok. The following pattern has repeated a few times today...



Spoiler: For length



*server 204.176.49.10, stratum 6, offset 118.620461, delay 0.11011*
server 204.176.49.11, stratum 6, offset 2.553283, delay 0.11194
server 204.176.49.12, stratum 6, offset 0.677905, delay 0.10966
*server 204.176.49.10, stratum 6, offset 118.855079, delay 0.11604*
server 204.176.49.11, stratum 6, offset 2.684828, delay 0.10983
server 204.176.49.12, stratum 6, offset 0.741389, delay 0.10977
*server 204.176.49.10, stratum 6, offset 119.060496, delay 0.10962*
server 204.176.49.11, stratum 6, offset 2.777870, delay 0.11188
server 204.176.49.12, stratum 6, offset 0.770458, delay 0.11217
*server 204.176.49.10, stratum 6, offset 119.350921, delay 0.11082*
server 204.176.49.11, stratum 6, offset 2.929371, delay 0.11023
server 204.176.49.12, stratum 6, offset 0.802268, delay 0.11133
*server 204.176.49.10, stratum 6, offset 119.580873, delay 0.11136*
server 204.176.49.11, stratum 6, offset 3.044121, delay 0.11066
server 204.176.49.12, stratum 6, offset 0.834583, delay 0.10994
*server 204.176.49.10, stratum 6, offset 119.808143, delay 0.11174*
server 204.176.49.11, stratum 6, offset 3.169599, delay 0.11139
server 204.176.49.12, stratum 6, offset 0.862006, delay 0.11023
*server 204.176.49.10, stratum 6, offset 120.083322, delay 0.11165*
server 204.176.49.11, stratum 6, offset 3.270567, delay 0.11308
server 204.176.49.12, stratum 6, offset 0.907657, delay 0.11015
*server 204.176.49.10, stratum 6, offset 120.285744, delay 0.11139*
server 204.176.49.11, stratum 6, offset 3.438764, delay 0.11127
server 204.176.49.12, stratum 16, offset 0.066923, delay 0.11032
*server 204.176.49.10, stratum 6, offset 120.487895, delay 0.11121*
server 204.176.49.11, stratum 6, offset 3.568935, delay 0.11328
server 204.176.49.12, stratum 16, offset 0.078701, delay 0.11360
*server 204.176.49.10, stratum 16, offset -0.778881, delay 0.10959*
server 204.176.49.11, stratum 6, offset 3.734676, delay 0.11063
server 204.176.49.12, stratum 16, offset 0.107089, delay 0.11049


----------



## velouria28 (Sep 23, 2008)

It's happened twice to one of my two premieres this week. A forced connection fixed it both times and thankfully I noticed it before it screwed up any recordings.


----------



## JoeKustra (Dec 7, 2012)

For the first time this morning's update put my Roamio 30 seconds fast. Probably unrelated: all tuners have their "Time since Tune Start" reset. The box did not use Standby or do a restart.


----------



## chiguy50 (Nov 9, 2009)

ClearToLand said:


> I always use S-P-S-9-S to display the time; besides telling me how far into the show I am and the TiVo clock's time, if it disappears I know that my TiVo unit rebooted.


Ditto here.

The user-enabled recording-marker/real-time clock is a feature I value and one of the reasons I am in no hurry to make the move to Hydra, where it is currently unavailable.


----------



## sfhub (Jan 6, 2007)

kdmorse said:


> They look fine now, but sjr1's been ruining up to 120second off today. sjr2's been, meh. sjr3's been ok. The following pattern has repeated a few times today...


sjr1 is now 1:32 fast. It sounds like what we suspected last time was correct and they didn't really fix the root cause of the drift, just masked it.

To summarize what we found before, all 3 NTP servers at TiVo sjr1|sjr2|sjr3.tivo.com drift. Usually one of them drifts quite a bit. This is mitigated by them resyncing with whatever source they use relatively frequently. Sometimes the servers will miss a few resyncs but normally they aren't off by more than 15-20 seconds. Once in a while they end up missing a lot of resyncs and they can be off by minutes. That looks like it may be happening again now.

Right now sjr1.tivo.com seems like the culprit. You can tell if there is a chance you'll get the wrong time by syncing your PC with time.nist.gov then running (the RefID of unknown for sjr1.tivo.com might be indicative of a sync problem with its source)


```
C:\>w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com

sjr1.tivo.com[204.176.49.10:123]:
    ICMP: 20ms delay
    NTP: +94.2005371s offset from local clock
        RefID: (unknown) [0x00017F7F]
        Stratum: 6
sjr2.tivo.com[204.176.49.11:123]:
    ICMP: 22ms delay
    NTP: +2.8702323s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 6
sjr3.tivo.com[204.176.49.12:123]:
    ICMP: 22ms delay
    NTP: +1.5761571s offset from local clock
        RefID: clock.isc.org [204.152.184.72]
        Stratum: 6
```
If sjr1 is off by a lot, there is probably 1 in 3 chance you'll get the wrong time. If all three are just a few seconds off, you can be assured you'll get usable time right away.

IMO this is probably what is going on:
1) all three TiVo NTP servers drift more than would be expected. One of them usually drifts a lot. Usually they drift fast.
2) TiVo compensates for this by syncing relatively frequently
3) every once in a while one or more of the time sources TiVo syncs with stops responding, and then we are exposed to the full drift of the TiVo NTP server, because many resyncs are missed
4) usually it is one server that is off, so when this happens you have 1 in 3 chance of getting time that is fast, so it will seem somewhat inconsistent with some people saying their time is fine but others saying their time is off by minutes.

We have theorized that one reason the TiVo NTP servers might be drifting a lot is they could be running in VMs or some sort of virtual hosting environment.

TiVo should really fix the drift problem on their NTP servers and they should monitor all 3 to see if they are getting out of sync with each other and if so alert some administrator. Lots of us depend on TiVo NTP servers to get our recordings started at the right time.


----------



## PSU_Sudzi (Jun 4, 2015)

After syncing back correctly last night mine is back off by 2 minutes again.


----------



## slowbiscuit (Sep 19, 2006)

sfhub said:


> sjr1 is now 1:32 fast. It sounds like what we suspected last time was correct and they didn't really fix the root cause of the drift, just masked it.
> 
> To summarize what we found before, all 3 NTP servers at TiVo sjr1|sjr2|sjr3.tivo.com drift. Usually one of them drifts quite a bit. This is mitigated by them resyncing with whatever source they use relatively frequently. Sometimes the servers will miss a few resyncs but normally they aren't off by more than 15-20 seconds. Once in a while they end up missing a lot of resyncs and they can be off by minutes. That looks like it may be happening again now.
> 
> ...


Can you PM Tivo_Ted with this info? He's apparently the only one here that can get anything done at Rivo when we all know that the issue is on their end.


----------



## GBL (Apr 20, 2000)

Looks like it's fixed now!


> sjr1.tivo.com[204.176.49.10:123]:
> ICMP: 54ms delay
> NTP: +1.3933455s offset from local clock
> RefID: time3.apple.com [17.254.0.31]
> ...


----------



## NorthAlabama (Apr 19, 2012)

GBL said:


> Looks like it's fixed now!


apparently the fix was short lived, i got hit tonight, just before prime time, all recordings started 2 minutes early, which means i've missed the end of my all prime time shows.

since there is no ffwding allowed with vod, i guess i'll just have to miss out, which stinks.


----------



## kdmorse (Jan 29, 2001)

NorthAlabama said:


> apparently the fix was short lived,.


There was no fix. The time server wanders back and forth, and sometimes it's right for a while. But until it is fixed, it will always go a wandering through time.

Let me be blunt. Their NTP server blows goats. It is misconfigured, unreliable, horribly architected, and the few times a day that it's accurate in no way excuses it's regular temporal wanderlust. It sucks... And the NTP admin should hang himself in shame.

(In fairness, I myself am a NTP NAZI from hell. Your clocks will be synced and correct, perfectly, and always! Failure to comply will result in merciless mocking, and the eventual extermination of your system. Or I may come at you from the other angle, for I am also the DNS NAZI. )

Since December 18, NTP server .10 and been off by over 10 seconds, for 4859 minutes (80 hours). That's an astoundingly bad record.

For the bored: Logfile attached...


----------



## samccfl99 (Sep 7, 2013)

I have been away from TCF for a bit. I got home Wed night and saw it was ahead. I went to connect and it showed my last connection was at 2:09 pm. It has been ok since and actually it is about 3 seconds ahead currently. Best I ever seen it. Got home before it killed the Survivor Finale next season preview...LOL.

*THEY ARE COMPLETE IDIOTS SOMETIMES...*


----------



## m.s (Mar 8, 2007)

Are sjr[1-3].tivo.com required for other than ntp? i.e. can I just use my dns server to send those to a real ntp server without screwing up other TiVo connections?


----------



## sfhub (Jan 6, 2007)

m.s said:


> Are sjr[1-3].tivo.com required for other than ntp? i.e. can I just use my dns server to send those to a real ntp server without screwing up other TiVo connections?


In my packet traces I didn't see them used for anything else. I didn't examine beyond the connection to the mothership (ie I didn't examine over several days.) TiVo does have an alternate way of setting time other than using those NTP servers, but that pathway hasn't been an issue.


----------



## sfhub (Jan 6, 2007)

Given how many people depend on TiVo's clock and how much it costs to field all the associated support calls, at $1500 a pop, perhaps TiVo could invest in a couple of these:

Now You Can Buy the Smallest Atomic Clock Ever Made


----------



## sfhub (Jan 6, 2007)

m.s said:


> Are sjr[1-3].tivo.com required for other than ntp? i.e. can I just use my dns server to send those to a real ntp server without screwing up other TiVo connections?


You can probably map sjr1.tivo.com to 204.176.49.11 (instead of current 204.176.49.10) and be 100% assured of zero impact to TiVo functions, though mapping to your choice of NTP servers should also work.

BTW you might need to do it via firewall rerouting vs DNS as there is a chance TiVo hard codes the NTP IP addresses.

From GBLs earlier post:


----------



## sfhub (Jan 6, 2007)

kdmorse said:


> For the bored: Logfile attached...


I was wondering why the data you listed always had sjr1 resetting at right around 120 seconds, very consistently.

I think I may have caught sjr1 in action resetting itself. I was monitoring sjr1 around +119sec and then it stopped responding briefly. Then after it started responding again, it listed RefID 'INIT'

Prior to that the RefID was (unknown)

So it seems there is some watchdog process that figures out sjr1 is screwed and isn't sync'ing time. That trigger is set to around 2 minutes difference from some other source. Once it triggers, it likely restarts the NTP daemon or server.

After the reset, it can sync with NTP servers again for a while, but something goes bad and it stops syncing, at which point we are exposed to the very bad drift for this server, until the watchdog timer kicks in at 2 minutes and restarts sjr1 again.


```
C:\>w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com
sjr1.tivo.com[204.176.49.10:123]:
    ICMP: 44ms delay
    NTP: +118.8151111s offset from local clock
        RefID: (unknown) [0x00017F7F]
        Stratum: 6
sjr2.tivo.com[204.176.49.11:123]:
    ICMP: 19ms delay
    NTP: +1.4753724s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 6
sjr3.tivo.com[204.176.49.12:123]:
    ICMP: 19ms delay
    NTP: -1.2707512s offset from local clock
        RefID: (unknown) [0x64F3A8C0]
        Stratum: 6
```


```
C:\>w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com
sjr1.tivo.com[204.176.49.10:123]:
    ICMP: 25ms delay
    NTP: -1.7977956s offset from local clock
        RefID: 'INIT' [0x54494E49]
        Stratum: 0
sjr2.tivo.com[204.176.49.11:123]:
    ICMP: 39ms delay
    NTP: -1.2588815s offset from local clock
        RefID: 'STEP' [0x50455453]
        Stratum: 0
sjr3.tivo.com[204.176.49.12:123]:
    ICMP: 25ms delay
    NTP: -1.1471614s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 6
```


```
C:\>w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com
sjr1.tivo.com[204.176.49.10:123]:
    ICMP: 17ms delay
    NTP: +1.3670662s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 6
sjr2.tivo.com[204.176.49.11:123]:
    ICMP: 37ms delay
    NTP: -0.0905585s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 6
sjr3.tivo.com[204.176.49.12:123]:
    ICMP: 17ms delay
    NTP: -1.4261377s offset from local clock
        RefID: 'STEP' [0x50455453]
        Stratum: 0
```


----------



## PSU_Sudzi (Jun 4, 2015)

sfhub said:


> As of 11:58pm Eastern the 3 TiVo NTP servers (sjr1|sjr2|sjr3).tivo.com are within 1.5seconds of time.nist.gov
> 
> If you connect now your time should be fine.


Their server(s) seem to keep losing their time every day by 2 minutes though. Since I posted this every day at some point I get 2 minutes behind I assume from my daily connection.


----------



## sfhub (Jan 6, 2007)

PSU_Sudzi said:


> Their server(s) seem to keep losing their time every day by 2 minutes though. Since I posted this every day at some point I get 2 minutes behind I assume from my daily connection.


sjr1 is constantly drifting, around +14s / hr right now.

The only thing that saves it is they run ntpdate a couple of times an hour to force it to sync with another NTP source.

What is happening is that forced sync is failing for sjr1 for an unknown reason, so you do the math +14s / hr means every 8.5hrs it will be off by 2 minutes. There is a separate process that notices an issue with sjr1 and when it is off 120s, it will reset sjr1, either the daemon or the server.


----------



## PSU_Sudzi (Jun 4, 2015)

sfhub said:


> sjr1 is constantly drifting, around +14s / hr right now.
> 
> The only thing that saves it is they run ntpdate a couple of times an hour to force it to sync with another NTP source.
> 
> What is happening is that forced sync is failing for sjr1 for an unknown reason, so you do the math +14s / hr means every 8.5hrs it will be off by 2 minutes. There is a separate process that notices an issue with sjr1 and when it is off 120s, it will reset sjr1, either the daemon or the server.


Is there away to blacklist the bad server within my router so it doesn't connect to that one?


----------



## kdmorse (Jan 29, 2001)

sfhub said:


> I was wondering why the data you listed always had sjr1 resetting at right around 120 seconds, very consistently.
> 
> I think I may have caught sjr1 in action resetting itself. I was monitoring sjr1 around +119sec and then it stopped responding briefly. Then after it started responding again, it listed RefID 'INIT'


The first part of your question could be simply NTPD doing it's job. It is supposed to monitor it's own drift related to it's peers, and if it gets bad enough start reporting itself as unsyncronized, or if it has no faith in itself, stop reporting at all. In all of the scenario's I've watched, it's clear that the servers are so wonky they know they cannot be trusted, demote themselves, and then somehow get back into sync on a regular basis.

But I've basically stopped trying to envision just what sort of screwed up circumstances might be the cause of the problem. They just need to get someone that knows how to run a damn NTP server into those servers with a hammer, and burn down what is likely a set of 10 year old ntp configs that have likely been copied from system to system over the years. They've probably got ntpd servers pointing at themselves, or each other in a loop, occasionally hammered with an external cron job running ntpdate, or a band-aid deamon that monitors and restarts the service. Probably running in a virtualized environment, under a hypervisor that points to a NTP server running under it causing a drift feedback loop.

Because, and I cannot stress this strongly enough, if your NTP Daemon cannot, on it's own, keep and serve accurate time, you have failed as a NTP administrator, *spectacularly*. I mean, it's a literal case of "You had one job!".


----------



## kdmorse (Jan 29, 2001)

I also want to add that sjr2 and sjr3 only look good in comparison to sjr1. sjr1 is completely off it's rocker. But both sjr2 and sjr3 are both horrible as ntp servers. By any objective measure, they blow goats, with drift and error rates 100 times what one would expect from an 'average' server. (In this case, 'average' being a old desktop you found in a corner, threw linux on, and used the default ntpd config).

That *any* of these are acting as time servers, controlling a world full of consumer devices that rally do care what time it is, is pretty sad.


----------



## kdmorse (Jan 29, 2001)

m.s said:


> Are sjr[1-3].tivo.com required for other than ntp? i.e. can I just use my dns server to send those to a real ntp server without screwing up other TiVo connections?


No, Yes, and no.

No, because their NTP script uses hardcoded IP addresses.

Yes, because if you redirect them externally (such as in your router), to known good NTP servers, it works fine.

No, because that's not the only source of time the Tivo uses. It also gets it's time source from something inside one of the HTTP/HTTPS transactions that occure outside the scope of daily calls (which use NTP). And when the NTP servers go sideways, that something does go sideways as well. So you'll still be exposed to half the problem potentially.

But it does make things slightly better. It just won't get you all the way to perfect.


----------



## sfhub (Jan 6, 2007)

kdmorse said:


> That *any* of these are acting as time servers, controlling a world full of consumer devices that rally do care what time it is, is pretty sad.


That is very true, the step tickers they have configured are never used because the skew is too great and the times get rejected. There is no way to slowly get the NTP server time back in check. If they didn't constantly force the time to set with ntpdate or equivalent we'd be off by minutes all the time, wait, that is where we are at now. +14s/hr is crazy.


----------



## m.s (Mar 8, 2007)

kdmorse said:


> Yes, because if you redirect them externally (such as in your router), to known good NTP servers, it works fine.
> 
> No, because that's not the only source of time the Tivo uses.


Thanks. I guess I should NAT the hardcoded addresses to one of my own stratum 1s to half fix the problem. It's more work to just NAT the ntp port, so that's good to know.

In the "old days" of my rooted S2/S3s, I had an hourly cron job which ntpdate sync'd locally, so was always within ms (except possibly for a short time after a connection to the mothership).


----------



## kdmorse (Jan 29, 2001)

For extra fun this time around, the current service 'issue' (widespread failed daily calls), is preventing me from correcting my unit's current incorrect clock (Running 73 seconds fast), so I've got plenty of screwy recordings. It's failing the call before it hits the 'set the clock' (ntpdate) part of the script, and whatever is broken is clearly also preventing its 'other' means of syncing as well.

But it's nice that I can look back in time through my ntp logs and correlate my last daily call to the middle of one of sjr1's drifting spikes. 

Edit: Daily call just went farther and the clock set. sjr1's at +3 seconds, and my tivo is now +3 seconds.


----------



## dlfl (Jul 6, 2006)

My tivo is +5 now -- had been perfect til today (vs. two WWV clocks and my iPad clock).


----------



## sfhub (Jan 6, 2007)

kdmorse said:


> whatever is broken is clearly also preventing its 'other' means of syncing as well.


The only time I've seen the other method of setting a clock used is during a reboot, some time right after the UI comes up. I forget now whether that was a warm or cold reboot, but I would hesitate to suggest anyone try that if they can't connect to the servers, because some of the new units don't have a battery backed clock, they just record the last known time and start from there, so you lose something like 1.5 - 2 minutes every reboot.


----------



## JoeKustra (Dec 7, 2012)

sfhub said:


> The only time I've seen the other method of setting a clock used is during a reboot, sometime right after the UI comes up. I forget now whether that was a warm or cold reboot, but I would hesitate to suggest anyone try that if they can't connect to the servers, because some of the new units don't have a battery backed clock, they just record the last known time and start from there, so you lose something like 1.5 - 2 minutes every reboot.


I see a clock setting during the network diagnostic test. But how can I know if it really tried to set the clock? I know how to access the log files. You could say I'm very skeptical of messages displayed by my TiVo.


----------



## sfhub (Jan 6, 2007)

JoeKustra said:


> I see a clock setting during the network diagnostic test. But how can I know if it really tried to set the clock? I know how to access the log files. You could say I'm very skeptical of messages displayed by my TiVo.


I capture the packet traces then look through what is going on.

The other way to sort of tell is if your clock is noticeably off, then run the test. If something changes right away, then it set something. If it doesn't then it still might have set something but to the bad NTP server.


----------



## NorthAlabama (Apr 19, 2012)

JoeKustra said:


> I see a clock setting during the network diagnostic test. But how can I know if it really tried to set the clock? I know how to access the log files. You could say I'm very skeptical of messages displayed by my TiVo.


it works, i've used it to set the clock before.


----------



## sharkster (Jul 3, 2004)

I wish they would stop messing with the damn clock! Mine, at least on my main Bolt, has been running just a smidge fast. That can make all the difference with the end of a show. I cannot see clear to pad everything and I don't think I should have to.


----------



## JoeKustra (Dec 7, 2012)

sharkster said:


> I wish they would stop messing with the damn clock! Mine, at least on my main Bolt, has been running just a smidge fast. That can make all the difference with the end of a show. I cannot see clear to pad everything and I don't think I should have to.


I can be hard to detect small changes. Most of my 1P have no padding. One show always needs it: The Daily Show. But watching CNBC, MSNBC and The Weather Channel I can check their clocks against my TiVo and two WWVB set clocks. Only constant is MSNBC, which has a 12 second delay all the time. Another constant is The Tonight Show, always 30 seconds late.

Earlier this year I went through all my 1P and noted which have a "Next Week On...". I stopped their padding. And how many begin with "Last Week On..."? But it all depends on your viewing habits. That's why there are 420 channels on my feed. Everybody gets something they like.


----------



## MScottC (Sep 11, 2004)

My Roamio Plus has drifted over 2 minutes several times in the past few weeks. Restarting it has solved the problem.


----------



## TonyD79 (Jan 4, 2002)

sharkster said:


> I wish they would stop messing with the damn clock! Mine, at least on my main Bolt, has been running just a smidge fast. That can make all the difference with the end of a show. I cannot see clear to pad everything and I don't think I should have to.


I default to 1 minute extension on my 1Ps. If there is a conflict, I clean it up. It's not TiVo's fault. A lot of networks and/or channels and/or shows are sloppy.


----------



## slowbiscuit (Sep 19, 2006)

MScottC said:


> My Roamio Plus has drifted over 2 minutes several times in the past few weeks. Restarting it has solved the problem.


It's not drifting, it's getting bad times from Tivo's server. Rebooting is just a temp workaround, as is forcing a net connect.

Note that you can do a 'Test network connection' from the Help->diagnostics menu that will also set the clock without checking for new guide data.


----------



## Mr Tony (Dec 20, 2012)

The last few days I've seen it off by a couple minutes. A refresh connection fixes that. Or I make sure (more like hope) that the next time it connects is after my morning shows record  so if it screws up I'm home to refresh the connection and get the clock back to the right time

It helps when I usually leave one of my tuners on Weather Nation (dont ask....) and they have the clock in the corner and also do weather on the 5's so I can usually match it up real close (within 10 seconds)


----------



## dlfl (Jul 6, 2006)

slowbiscuit said:


> It's not drifting, it's getting bad times from Tivo's server. Rebooting is just a temp workaround, as is forcing a net connect.
> 
> Note that you can do a 'Test network connection' from the Help->diagnostics menu that will also set the clock without checking for new guide data.


But that doesn't always correct the time. My Roamio was 16 secs fast just now and after the network connection test it was unchanged.


----------



## Rob Helmerichs (Oct 17, 2000)

dlfl said:


> But that doesn't always correct the time. My Roamio was 16 secs fast just now and after the network connection test it was unchanged.


Probably just hit the same server...you would probably have had the same result if you'd rebooted at that time. Either way, the TiVo is checking its clock against a server; it shouldn't matter which way you do it.


----------



## JYoung (Jan 16, 2002)

I got bit again by this yesterday with my Series 3 being two minutes off.
Fortunately, I wasn't recording anything on it last night.


----------



## TiVo_Ted (Oct 3, 2000)

At its heart, a TiVo is just a computer. Assuming you don't unplug your device, the clock is maintained by a crystal oscillator. Consumer devices don't usually use the worlds most accurate crystals, so a bit of "drift" is expected. A daily call to at time server usually keeps things in line. Unfortunately, crystals are subject to aging and the drift can get worse over time. Series 3 boxes could be as old as 10 years now. A drift of 1-2 minutes per day is a bit high, but not completely unexpected.


----------



## TiVo_Ted (Oct 3, 2000)

I just went back and re-read this whole thread. Your diagnostics are very helpful, and I'm going to open a ticket with our NOC. It sounds like there may be some small drift on the clients *and* some bad data coming from one of the NTP servers. I'll see what I can do.


----------



## sfhub (Jan 6, 2007)

TiVo_Ted said:


> At its heart, a TiVo is just a computer. Assuming you don't unplug your device, the clock is maintained by a crystal oscillator. Consumer devices don't usually use the worlds most accurate crystals, so a bit of "drift" is expected. A daily call to at time server usually keeps things in line. Unfortunately, crystals are subject to aging and the drift can get worse over time. Series 3 boxes could be as old as 10 years now. A drift of 1-2 minutes per day is a bit high, but not completely unexpected.


I don't believe Bolts have real time clocks anymore or if they do, they aren't using them.

The reason I say this is during debugging of this NTP server issue, at some point I was rebooting the Bolt with the network disconnected and everytime it would lose something like 90 seconds (which is around the amount of time for a reboot)

Suspecting the clock was not being maintained, I left the Bolt unplugged for 5 minutes. Upon booting it was then off an additional 6 minutes.

I concluded even if the Bolt has a real time clock maintained by battery, it isn't using it. Instead it is recording the last known time and picking up where it left off.


----------



## sfhub (Jan 6, 2007)

TiVo_Ted said:


> I just went back and re-read this whole thread. Your diagnostics are very helpful, and I'm going to open a ticket with our NOC. It sounds like there may be some small drift on the clients *and* some bad data coming from one of the NTP servers. I'll see what I can do.


Actually all 3 NTP servers sjr[1-3].tivo.com drift a LOT. To compensate for that they constantly force their time to set themselves to known NTP servers. This is not how NTP is supposed to work. Normally NTP expects your clock to mostly be able to maintain itself and it corrects for small drift like 1-2 seconds and it does it slowly. TiVo's bad NTP server drifts something like 14 seconds / hour. The other 2 might drift more like 7 seconds / hour but I didn't really pay attention to them.

What is happening on sjr1 (the really bad server) is the process to force time to be set to an NTP server is somehow failing repeatedly. This means around 8.5 hours, the time will be off 2 minutes and whomever connects at that time and is unlucky enough to choose sjr1 as the NTP server to use will be off 2 minutes.

The last time they "fixed" this problem, they didn't fix the horrible clock drift on all the NTP servers. They just fixed whatever was causing the forced clock sets to fail, leaving the drift alone. Then eventually the forced clock syncs failed again and we have a repeat of the problem.

The NOC team should really
1) fix the horrible clock drift
2) fix the failing forced clock set
3) add some script that lets them know automatically when the clocks on the NTP servers are drifting or off a lot

If they do #1, then NTP can do its job and correct for small drift and not fail all the time because the seconds that it needs to correct for are too great and it assumes something has gone terribly wrong and refuses to do any correction, awaiting the periodic forced clock sets to fix things.


----------



## dsando (Sep 12, 2015)

sfhub said:


> The NOC team should really
> 1) fix the horrible clock drift
> .


IIRC there was a glitch in VMware ESXi that would cause guest OS's clocks to drift fast, and was eventually patched.

May be worth looking into if the servers are VM's


----------



## dlfl (Jul 6, 2006)

dlfl said:


> But that doesn't always correct the time. My Roamio was 16 secs fast just now and after the network connection test it was unchanged.


And this morning, after a service connection in the wee hours, the time is back to correct.


----------



## lessd (Jan 23, 2005)

I checked last night after a service call and my clock was 30 Sec fast, did not check today.


----------



## TiVo_Ted (Oct 3, 2000)

The NOC team believes they have fixed the issue. I have also pointed them to this full thread. Is anyone still seeing any issues?


----------



## sfhub (Jan 6, 2007)

TiVo_Ted said:


> The NOC team believes they have fixed the issue. I have also pointed them to this full thread. Is anyone still seeing any issues?


I haven't monitored all day to confirm, but I can tell right away they didn't fix the horrible drift issue as sjr1 is currently +8sec. I know it doesn't sound like much, but for an NTP server it should be close to zero all the time, so +8sec is horrible and it will cause the NTP server to stop trying to fix itself because it thinks something has gone horribly wrong. That means it will depend on the fallback forced time set that happens around every 7 to 14 seconds, but what we have seen in the past is eventually after a month or two, those forced time sets start failing, and then we are exposed the very bad drift again.

The advice given in this post about a VMWare patch for clock drift might be pertinent if these servers are VMs:
Clock off in Roamio by 2 minutes


```
C:\>w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com
sjr1.tivo.com[204.176.49.10:123]:
    ICMP: 20ms delay
    NTP: +8.0521420s offset from local clock
        RefID: (unknown) [0x64B4A8C0]
        Stratum: 6
sjr2.tivo.com[204.176.49.11:123]:
    ICMP: 18ms delay
    NTP: +2.1364053s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 6
sjr3.tivo.com[204.176.49.12:123]:
    ICMP: 22ms delay
    NTP: +0.0441709s offset from local clock
        RefID: (unknown) [0x00017F7F]
        Stratum: 6
```


----------



## HerronScott (Jan 1, 2002)

I agree those offsets on the 2 are terrible and I'm curious why they are Stratum 6 and the one is syncing from Apple's time service (Mac server?).

Our lowest are Stratum 4 and that's only because we have an empty root domain and dc1, dc2 and dc3 are located in 3 different data centers around the world (not their real names).


```
C:\w32tm /monitor /computers:root1,dc1,dc2,dc3
root1
    ICMP: 87ms delay
    NTP: +0.4183905s offset from local clock
        RefID: time-c-g.nist.gov [129.6.15.30]
        Stratum: 2

dc1
    ICMP: 87ms delay
    NTP: +0.3382913s offset from local clock
        RefID: root1
        Stratum: 3

dc2
    ICMP: 63ms delay
    NTP: -0.2449238s offset from local clock
        RefID: (unknown) [0x05070B0A]
        Stratum: 4

dc3
    ICMP: 168ms delay
    NTP: +0.4700539s offset from local clock
        RefID: (unknown) [0x050A1FAC]
        Stratum: 4
```
Scott


----------



## sfhub (Jan 6, 2007)

HerronScott said:


> the one is syncing from Apple's time service (Mac server?).


I believe they have configured at least 3 NTP servers to do the NTP dance with. I've seen apple, clock.isc.org and I think a few others. However the drift is so bad that NTP won't normally correct for offsets so large as it tries to slowly adjust stuff back to 0 offset. The TiVo NTP servers depend on something like ntpdate running every 7 to 14 seconds to force the time to set. When that forced time set fails, that is when we get the 2 minute off (after around 8.5 hours of failure)


----------



## Mr Tony (Dec 20, 2012)

This morning around 5:20am CST I did a connect and bam the clock jumped more than a minute ahead. I was recording MeTv and they usually go 25 seconds or so past the top/bottom of hour but when the show got done my time said 5:32. Did a force connection right away and I went back in time to the correct time


----------



## GBL (Apr 20, 2000)

Not fixed yet.
Currently off by 36 seconds.


```
C:\>w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com
sjr1.tivo.com[204.176.49.10:123]:
    ICMP: 54ms delay
    NTP: +36.7374915s offset from local clock
        RefID: (unknown) [0x00017F7F]
        Stratum: 6
sjr2.tivo.com[204.176.49.11:123]:
    ICMP: 54ms delay
    NTP: +1.6824245s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 6
sjr3.tivo.com[204.176.49.12:123]:
    ICMP: 59ms delay
    NTP: +1.0476674s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 2
```


----------



## m.s (Mar 8, 2007)

GBL said:


> Not fixed yet.
> Currently off by 36 seconds.


They're all off by more than a second (assuming your baseline is accurate). 36 ms would be a poor ntp source.

I strongly suspect they're trying to run ntpd inside VMs (or worse, Windows Time Service). Bad, TiVo, bad. A couple thou, and they could be running physical stratum one servers. @TiVo_Ted, PM me, I can help.


----------



## kdmorse (Jan 29, 2001)

On 12/30/2017 at 19:32 eastern, someone kicked sjr1 (or it kicked itself). It came back 5 minutes later at +1.2s, and is now behaving like a 'normal' Tivo timeserver. ie, it drifts up to about +1.5, then corrects - sometimes drifting up to as high as +3.0. 

sjr3 has been almost behaving, spending most of its time less than 0.10s off. (Although as I type this it drifted up to +0.4s before drifting back down)

So right this minute, we're back to 'meh' ntp services. But that's still a vast improvements from when sjr1 went off on a persistent series of drunken ntp benders.


----------



## kdmorse (Jan 29, 2001)

Spoke too soon, someone's let sjr1 start drinking again. It's restarted it's inexorable march upwards at it's standard climb of +1s every 5 minutes. Made it up to +9s this instant, maybe it catches itself, maybe it walks all the way up to 120s, who knows, I'll check my logs in the morning.

It's almost fascinating. It was behaving. Then it marched up to +2s, then reset. Then +4s then reset. Then +6s then reset. Then +8s then reset. I expect it might reset this cycle at +10s. It manages to creep farther and farther every cycle. If I didn't have so much other work to do, I'd enjoy sitting down and making some graphs.


----------



## sfhub (Jan 6, 2007)

Someone needs to tell NOC that rebooting != fix


----------



## slowbiscuit (Sep 19, 2006)

Yep, my Roamio was off by 2 mins. again this morning after an early guide update.

Tivo_Ted - this is not something that a first-level NOC can fix, it needs to go to the sysadmin responsible for these servers to fix misconfiguration or patching. Please have them escalate the ticket.


----------



## Murf300 (Jun 2, 2007)

I'm off by two minutes on my Roamio today. Missing the end of several shows.


----------



## tim_m (Mar 8, 2017)

Tivo apparently says happy new year by putting their clock off by 1 minute screwing up every show last night. Thanks Tivo! @TiVo_Ted


----------



## V7Goose (May 28, 2005)

My Bolt is now 2 minutes fast this AM (for the first time). I had been ignoring this thread up until now just because it seemed pretty stupid to think that anything connected to the internet could not have the correct time (and I had not seen it on my boxes). Well, I have seen it now - so not stupid that it is being reported at all, but REALLY stupid that it is happening!

Here's how I noticed it today - morning network news program I always watch started with 2 minutes of commercials (something that never happens), so I hit the info button and compared the screen time with the Windoze computer screen that is sitting right next to the TV. The Bolt claimed the current time was 7:40am, while the computer reports 7:38 AM. Looks like I am going to be missing the ends of lots of programs until this idiocy is fixed.

For additional information, my Bolt last service connection was yesterday at 1:16 pm. I am about to do another connection right now to see if it changes the clock.
Update: After the connection just made at 7:58am, the clock on the Bolt seems to match the computer time again.


----------



## sharkster (Jul 3, 2004)

Mine seems fast again today (early am connection), but only by several seconds. Still - good grief! How hard is it to have the clock right? It never was that hard before Rovi. I still wish we could just set it ourselves, just like other clocks in the house. I'd have it stay around 4-5 seconds behind. Oh well.


----------



## GBL (Apr 20, 2000)

Currently almost 2 minutes off again - needs more work!


```
C:\>w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com
sjr1.tivo.com[204.176.49.10:123]:
    ICMP: 60ms delay
    NTP: +113.3741514s offset from local clock
        RefID: (unknown) [0x00017F7F]
        Stratum: 6
sjr2.tivo.com[204.176.49.11:123]:
    ICMP: 75ms delay
    NTP: +1.9407077s offset from local clock
        RefID: clock.isc.org [204.152.184.72]
        Stratum: 6
sjr3.tivo.com[204.176.49.12:123]:
    ICMP: 75ms delay
    NTP: +0.4923914s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 6
```


----------



## Rob Helmerichs (Oct 17, 2000)

Maybe Joe's hamsters are drunk..?


----------



## JoeKustra (Dec 7, 2012)

Rob Helmerichs said:


> Maybe Joe's hamsters are drunk..?


Really? Everybody knows hamsters can't tell time.


----------



## Rob Helmerichs (Oct 17, 2000)

JoeKustra said:


> Really? Everybody knows hamsters can't tell time.


Well, with all that drinking they do it's no wonder!


----------



## tim_m (Mar 8, 2017)

Mine was 3 minutes fast with this afternoons connection. It took two sequential forced connections to get it correct. This is totally unacceptable behavior.


----------



## NorthAlabama (Apr 19, 2012)

i've been really lucky, and have only synced to an off-time server once recently, but when i know it's happening, forget to check, and miss the endings of shows, it really burns me.


----------



## kdmorse (Jan 29, 2001)

Block 204.176.49.10 - UDP Port 123 at your firewall if you can. The ntpdate command used to sync the clock queries 204.176.49.10, 204.176.49.11, and 204.176.49.12 in parallel. 204.176.49.10 won't be missed if blocked.

(Or redirect .10 to .11, or to a real time server)

Right this minute sjr1 is back to just 'meh' (drifting to +2 or +3 seconds before resetting), but it was drunk most of the day, until at least 4pm eastern. Then a little tipsy for a while, but seems to have sobered up for the moment.

But it tends not to stay sober for long.


----------



## ggieseke (May 30, 2008)

sjr1 fell off the wagon again. What a cluster****.


----------



## JoeKustra (Dec 7, 2012)

kdmorse said:


> Block 204.176.49.10 - UDP Port 123 at your firewall if you can. The ntpdate command used to sync the clock queries 204.176.49.10, 204.176.49.11, and 204.176.49.12 in parallel. 204.176.49.10 won't be missed if blocked.


I'm feeling left out. Why am I never affected by this problem?


----------



## bigguy126 (Sep 4, 2007)

I am already monitoring my to do list daily to make sure everything records right, due to the data inconsistencies, now I also have to monitor the time.
So much for a stand alone service.


----------



## dlfl (Jul 6, 2006)

JoeKustra said:


> I'm feeling left out. Why am I never affected by this problem?


Contact Tivo support. Maybe they can get you into the mainstream .... nah, they probably couldn't even do that.


----------



## PSU_Sudzi (Jun 4, 2015)

JoeKustra said:


> I'm feeling left out. Why am I never affected by this problem?


I think it's just blind luck. I haven't had this issue at all until 2 weeks ago and it's happened twice since.


----------



## Jed1 (Jun 18, 2013)

I just got hit with this yesterday on my bedroom Roamio. This is the unit that records all my network shows. Went to watch NCIS, Bull, and NCIS NO and the clock was two minutes fast. Also missed the last two minutes of Major Crimes so now I have to pick up the repeat next week.
I have Hydra and I know some don't like it but if the guide data is going to be messed up and now the clock is messed up on top of this it does not matter what UI you have as these problems will make any DVR worthless. I have never had clock issues on any electronic device until now.
Is there any way to get these devices to use another time server? Does rebooting make it choose another server?


----------



## PSU_Sudzi (Jun 4, 2015)

Jed1 said:


> I just got hit with this yesterday on my bedroom Roamio. This is the unit that records all my network shows. Went to watch NCIS, Bull, and NCIS NO and the clock was two minutes fast. Also missed the last two minutes of Major Crimes so now I have to pick up the repeat next week.
> I have Hydra and I know some don't like it but if the guide data is going to be messed up and now the clock is messed up on top of this it does not matter what UI you have as these problems will make any DVR worthless. I have never had clock issues on any electronic device until now.
> Is there any way to get these devices to use another time server? Does rebooting make it choose another server?


Forcing a network connection will correct the time if you don't hit the bad server again (who's time seems to keep drifting).


----------



## das335 (Feb 8, 2006)

kdmorse said:


> Block 204.176.49.10 - UDP Port 123 at your firewall if you can. T


Thanks for the tip. It took me a while to figure out how to setup a firewall filter rule in my Actiontec router, but I finally succeeded.

Sadly, with my recent "customer support" experiences and several recent problems like the timeserver, I can no longer recommend TiVo for non-technical friends.

I have used TiVos since the early DirecTiVo days and I still enjoy using them but the vibe has really changed lately.

I am very grateful for all the support I get from users on the Tivocommunity site.


----------



## wish_bgr (Jul 19, 2014)

Had the +:02 clock problem here for the first time this morning. Was wondering why Home & Family was running behind, bleeding into the next hour’s show at 9:01/9:02a. I went and checked some morning news shows, which usually have a bleed-over segue into the next hour. I forced a connection while tuning into CNBC as they have a clock displayed in their crawl feed along the bottom. Time correctly synched after the connection was made, with both the TiVo and on-air time matching.


----------



## NorthAlabama (Apr 19, 2012)

setting an outlook reminder now to check the tivo clock daily before prime-time recordings begin...


----------



## Mr Tony (Dec 20, 2012)

had it happen this morning at 7:57 CST....was about a minute fast. I record "through the decades" and when its bang on the program starts with the ID. Today it was the last minute of the previous show

Its fixed now after a force connection


----------



## PSU_Sudzi (Jun 4, 2015)

JoeKustra said:


> Really? Everybody knows hamsters can't tell time.


Does a TiVo get time through other methods aside from a network connection or test? I did a manual update at lunch and time was ok but around 4 I turned the tv back on and checked the time and it was off by two minutes again. No additional calls were made by me or the system. I forced another connection and it corrected itself. But was wondering what caused it to go astray.


----------



## sharkster (Jul 3, 2004)

PSU_Sudzi said:


> Does a TiVo get time through other methods aside from a network connection or test? I did a manual update at lunch and time was ok but around 4 I turned the tv back on and checked the time and it was off by two minutes again. No additional calls were made by me or the system. I forced another connection and it corrected itself. But was wondering what caused it to go astray.


Well, that kinda sucks. I, too, am curious as to why the clock was affected with no connections made.


----------



## NorthAlabama (Apr 19, 2012)

i'm thinking it might be time to begin including whether or not an affected tivo clock is running fast or slow, along with the amount of time.

anyone else?


----------



## kdmorse (Jan 29, 2001)

Tivo's time servers are, well, one of three apparently has a drinking problem.

That said - in individual cases, Tivo_Ted's point that an individual unit's clock can drift (sometimes badly), is still valid. It's worth running the test of ensuring your unit has the right time, disconnecting the Ethernet cable, and over the next few days keep an eye on it to see if your unit has any personal drift of note.

Chances are, you're going to find that (absent a reboot), your unit keeps good time on it's own over a few days. But it's still worth running the experiment to see how your unit behaves.

That said, we suspect it has 'another' way of setting the clock that is invoked on boot, and there's some evidence that this 'other' way is invoked on it's own sometimes, but I don't think anyone's chased it down to be sure.


----------



## samccfl99 (Sep 7, 2013)

There is absolutely NO reason for this crap. I guess Ted needs to go back to someone who knows a simple thing like setting server(s) with the correct time or getting it from an internet clock somewhere.

*PATHETIC*


----------



## JoeKustra (Dec 7, 2012)

PSU_Sudzi said:


> Does a TiVo get time through other methods aside from a network connection or test? I did a manual update at lunch and time was ok but around 4 I turned the tv back on and checked the time and it was off by two minutes again. No additional calls were made by me or the system. I forced another connection and it corrected itself. But was wondering what caused it to go astray.


So many questions, so few answers. I'll make my plea again: I can access the logs, but I don't know what they mean. A lot of all caps, a lot of code, and mostly stuff that makes no sense. If TiVo wants us to be beta testers and run point on system failures, the least they can do is document the logs.

If they can.


----------



## tim_m (Mar 8, 2017)

NorthAlabama said:


> setting an outlook reminder now to check the tivo clock daily before prime-time recordings begin...


Did the same with my google calendar..


----------



## tim_m (Mar 8, 2017)

I'm wondering if this issue is in anyway related to the very slow responsiveness of the tivos the last few nights. Nothing but spinning blue circles in primetime.


----------



## slowbiscuit (Sep 19, 2006)

No, BSCs are a separate issue altogether. Tivo's servers are again (stupidly) the problem but not NTP-related.


----------



## JandS (Oct 1, 2010)

Following wish_bgr's suggestion of using the CNBC ticker to compare with the Tivo Bolt+ clock, I tuned to CNBC, watched for about 30 seconds to see how often the clock display appeared/disappeared. The CNBC ticker *seemed *~3-4 seconds behind the Tivo clock as shown by pressing Select x2. Even though I had just tuned to the channel and the green bar showed it as being current, I pushed 1x fast forward. The video sped up and bingo the clocks were exact.

So my point is that just tuning to a station didn't ensure that the Tivo was up to the exact current feed of the channel, it started off x seconds back.

I wonder if this delay is related to the delay that I was asking about a few weeks ago, that the speed of the video display seemed slower when switching tuners using the LiveTV button.

Tuner change speed using Live TV button


----------



## BobCamp1 (May 15, 2002)

kdmorse said:


> That said - in individual cases, Tivo_Ted's point that an individual unit's clock can drift (sometimes badly), is still valid. It's worth running the test of ensuring your unit has the right time, disconnecting the Ethernet cable, and over the next few days keep an eye on it to see if your unit has any personal drift of note.
> 
> Chances are, you're going to find that (absent a reboot), your unit keeps good time on it's own over a few days. But it's still worth running the experiment to see how your unit behaves.


A $15 battery operated clock keeps much better time than my $200 DVR? I have a $15 LCD clock that drifts less than a second before I change its battery once every 2 years. The reason the DVR syncs once a day is because it can -- it has to connect to download new guide data anyway. But it shouldn't be depending on that to keep accurate time day to day or even week to week.

As far as the NTP servers go, I could tie one to the back of my car bumper, drive home, plug it in, and it'll STILL keep way better time than Rovis'. At work, we have 20 year old time servers that are still accurate within 300 ns. Something is very clearly wrong with their setup.


----------



## kdmorse (Jan 29, 2001)

BobCamp1 said:


> As far as the NTP servers go, I could tie one to the back of my car bumper, drive home, plug it in, and it'll STILL keep way better time than Rovis'. At work, we have 20 year old time servers that are still accurate within 300 ns. Something is very clearly wrong with their setup.


sjr1 is +55 seconds this moment, so it's clearly mid bender. For giggles, sampled once a minute from January 1st to January 4th, offset rounded down to the nearest second (so +0.75s falls into the 0s catagory, 5.5s falls into 5s). Boundaries chosen simply because they were easy to grep.










So, sjr1 has spent 70% of it's time in 2018 at least 10 seconds fast (and usually much more).


----------



## NorthAlabama (Apr 19, 2012)

kdmorse said:


> sjr1 is +55 seconds this moment, so it's clearly mid bender. For giggles, sampled once a minute from January 1st to January 4th, offset rounded down to the nearest second (so +0.75s falls into the 0s catagory, 5.5s falls into 5s). Boundaries chosen simply because they were easy to grep.
> 
> So, sjr1 has spent 70% of it's time in 2018 at least 10 seconds fast (and usually much more).


my outlook reminder sounded, and i checked - i was about a minute fast. it took 3 test connections to sync it to only 15 sec fast, at which point i gave up - it'll have to be close enough.


----------



## kdmorse (Jan 29, 2001)

NorthAlabama said:


> my outlook reminder sounded, and i checked - i was about a minute fast. it took 3 test connections to sync it to only 15 sec fast, at which point i gave up - it'll have to be close enough.


Its full benders are staggeringly regular in length, lasting about 9 and a half hours each. So, you should try again at 12:30am eastern, maybe a little later. (You don't want to lock yourself in when it's on the tail end of a bender, and it's at +120s.)

Unfortunately, it sometimes goes on mini-benders, which throws the predictability off.

(Or block 204.176.49.10 at your router)


----------



## ClearToLand (Jul 10, 2001)

NorthAlabama said:


> my outlook reminder sounded, and i checked - i was about a minute fast. *it took 3 test connections to sync it to only 15 sec fast, at which point i gave up* - it'll have to be close enough.


IMO, many folks here are causing themselves unnecessary stress. 

We're lucky to have @kdmorse and @sfhub here to 'explain' the problem to us and @kdmorse has gone one step further and provided a (partial) workaround by suggesting folks block sjr1 in their router. Several folks have posted the Windows *w32tm /monitor* command, so by running that BEFORE doing anything with your TiVo unit, you could see that currently *your *ONLY* choices are +75, +16 and +16 - regardless of how many times you NET CONNECT*.

@TiVo_Ted has acknowledged the problem, but I wouldn't hold my breath waiting for daily updates and a quick fix. Just looking at the examples posted by @kdmorse leads me to believe that there is NO current NTP Admin working at TiVo and 'somebody' has to learn how to fix this (probably ancient code) while still doing their original job (i.e. +2 then reset, +4 then reset, +6 then reset, etc... looks like somebody 'experimenting' in order to learn). The relationship between port 123, port 37 and time also hints to me that the current code has been in place a LONG while... 

You know that saying that begins with "Give me the wisdom to..."


----------



## NorthAlabama (Apr 19, 2012)

ClearToLand said:


> IMO, many folks here are causing themselves unnecessary stress.


it's more stress to lose the end of my recordings, and easy to toggle between the connection and live tv. the stress would be padding every 1p when it shouldn't be necessary.

i kept connecting because 1 minute was unacceptable, but stopped because i can live with 15 sec. - not ideal, but close enough.

the only unnecessary stress is being created by tivo, not with me wanting my service to operate within norms.


----------



## ClearToLand (Jul 10, 2001)

kdmorse said:


> ...That said - *in individual cases, Tivo_Ted's point that an individual unit's clock can drift (sometimes badly), is still valid*. It's worth running the test of ensuring your unit has the right time, disconnecting the Ethernet cable, and over the next few days keep an eye on it to see if your unit has any personal drift of note...


This is a good experiment, especially for older (retail) equipment. i.e., Windows PCs can lose gobs of time if something has them running at 100% CPU (in Turbo Mode) for extended periods since, AFAICT, other processes do have priority over time keeping and the software clock could skip-a-beat. Components age, tolerances change, yada, yada, yada... As @kdmorse suggests, see how tight your unit holds time when disconnected from the Ethernet for a few days.



kdmorse said:


> ...That said, we suspect it has 'another' way of setting the clock that is invoked on boot, and *there's some evidence that this 'other' way is invoked on it's own sometimes*, but I don't think anyone's chased it down to be sure.


I don't usually check TiVo System Information (certainly not as often as some 'Old-Timers' here  ), but I did check it a few times recently and noticed that *VCT Connection* was a few hours (since Last Successful) one time and less than an hour the next. We're ~41 minutes into tonight's Prime Time as I type this, so I can only hope that my previously accurate time wasn't 'adjusted'...


----------



## ClearToLand (Jul 10, 2001)

NorthAlabama said:


> ...*i kept connecting because 1 minute was unacceptable*, but stopped because i can live with 15 sec. - not ideal, but close enough...


And I'm saying that your *ONLY* choices were +75, +16 and +16, so you had a 1 in 3 chance of getting +75 and ZERO chance of getting less than +16.


----------



## NorthAlabama (Apr 19, 2012)

ClearToLand said:


> And I'm saying that your *ONLY* choices were +75, +16 and +16, so you had a 1 in 3 chance of getting +75 and ZERO chance of getting less than +16.


yes, that makes sense, even though i wasn't making the effort to give it that deep of an analysis. after 5 attempts, i probably would have forgotten about it until the next day's reminder.

one advantage to using the test connection is that is quick, and i can toggle back and forth easily while watching live tv (the news). if i hadn't been successful after 5 attempts during the commercial breaks, and didn't forget, i would have reviewed my tdl, padded for the night where necessary, and tried again the next day. it's better for me in small doses than sitting down and padding all my 1p's - it's my preferred workaround for now, unless this drags out much longer.


----------



## PSU_Sudzi (Jun 4, 2015)

kdmorse said:


> Its full benders are staggeringly regular in length, lasting about 9 and a half hours each. So, you should try again at 12:30am eastern, maybe a little later. (You don't want to lock yourself in when it's on the tail end of a bender, and it's at +120s.)
> 
> Unfortunately, it sometimes goes on mini-benders, which throws the predictability off.
> 
> (Or block 204.176.49.10 at your router)


I have a Comcast XB3 modem and it does not seem to allow me to block IP addresses specifically, can I add an IP address into a URL field to block?

Edited: Just blocked *sjr1.tivo.com* which should work I hope.


----------



## m.s (Mar 8, 2007)

kdmorse said:


> (Or block 204.176.49.10 at your router)


I just DNAT'd sjr1 to sjr3.


----------



## tim_m (Mar 8, 2017)

PSU_Sudzi said:


> I have a Comcast XB3 modem and it does not seem to allow me to block IP addresses specifically, can I add an IP address into a URL field to block?
> 
> Edited: Just blocked *sjr1.tivo.com* which should work I hope.


I can't even do that with my Arris. It only allows for client blocking. Guess i am left babysitting my tivo clock.


----------



## JYoung (Jan 16, 2002)

kdmorse said:


> sjr1 is +55 seconds this moment, so it's clearly mid bender.


Yes, it is.
Grrr.

I got bit by this tonight on my Bolt this time, which connected at 18:08 PST and it's clock jumped ahead by close to two minutes.


----------



## tim_m (Mar 8, 2017)

I found a way to block that messed up serve using parental controls web filtering. After I did that I forced the the connection and the clock was 100% correct


----------



## kdmorse (Jan 29, 2001)

kdmorse said:


> Its full benders are staggeringly regular in length, lasting about 9 and a half hours each. So, you should try again at 12:30am eastern, maybe a little later. (You don't want to lock yourself in when it's on the tail end of a bender, and it's at +120s.)


Swing and a miss... (on the math)

Almost as if it knew I was making predictions about it, sjr1 changed the speed at which it drifts just to screw with me. At 22:30 (ish) eastern, it shifted from drifting about 1s per 5m, to 1s per 15m. But it's still wildly off, so it's just going to take longer to get to 120s and correct (if that's indeed what it's going to do this round). I genuinely don't know if this is better or worse, only time will tell.

It's only made it to +88s right now.


----------



## PSU_Sudzi (Jun 4, 2015)

tim_m said:


> I found a way to block that messed up serve using parental controls web filtering. After I did that I forced the the connection and the clock was 100% correct


I was just going to respond to say using parental controls was the only way I could block it in my modem too.


----------



## BobCamp1 (May 15, 2002)

m.s said:


> I just DNAT'd sjr1 to sjr3.


If you're going to do that, just DNAT it to a real NTP server on the Internet. There are hundreds of them.


----------



## sharkster (Jul 3, 2004)

Good grief! Mine was just fine earlier this morning and I just had a service connection (how I hate morning connections!) and now it's almost a minute fast.


----------



## NorthAlabama (Apr 19, 2012)

this lottery seems to be paying out more often, which isn't a good thing.


----------



## ClearToLand (Jul 10, 2001)

ClearToLand said:


> ...We're lucky to have @kdmorse and @sfhub here to 'explain' the problem to us and @kdmorse has gone one step further and provided a (partial) workaround by suggesting folks block sjr1 in their router. Several folks have posted the Windows *w32tm /monitor* command, so by running that BEFORE doing anything with your TiVo unit, you could see that currently *your *ONLY* choices are +75, +16 and +16 - regardless of how many times you NET CONNECT*...


w32tm /monitor works MUCH better if YOUR computer has the correct time.  I was (mistakenly) under the impression that once I went to:

Control Panel -> Date and Time -> Internet Time -> Change Settings -> Update Now​
and received an "Updated" message, that the operation completed successfully. Further GOOGLE research into w32tm parameters has informed me otherwise. In the first 'Update Now' below, although the screen said "Updated", w32tm said: "Last Sync Error: 1 (The computer did not resync because no time data was available.)". I only discovered this because "Time.Is" said my time was 15 seconds off. Since my first update was with time.nist.gov, I now tried time.windows.com and got: "Last Sync Error: 0 (The command completed successfully.)"

```
C:\Windows\system32>w32tm /query /status /verbose
Leap Indicator: 3(last minute has 61 seconds)
Stratum: 0 (unspecified)
Precision: -6 (15.625ms per tick)
Root Delay: 0.0552436s
Root Dispersion: 7.7760890s
ReferenceId: 0x00000000 (unspecified)
Last Successful Sync Time: 2018-01-05 16:20:57
Source: time.nist.gov,0x9
Poll Interval: 10 (1024s)

Phase Offset: 0.0000000s
ClockRate: 0.0156001s
State Machine: 0 (Unset)
Time Source Flags: 0 (None)
Server Role: 0 (None)
Last Sync Error: 1 (The computer did not resync because no time data was av
le.)
Time since Last Good Sync Time: 68.4140000s

C:\Windows\system32>w32tm /query /status /verbose
Leap Indicator: 0(no warning)
Stratum: 3 (secondary reference - syncd by (S)NTP)
Precision: -6 (15.625ms per tick)
Root Delay: 0.0383148s
Root Dispersion: 7.8115500s
ReferenceId: 0x34B31126 (source IP:  52.179.17.38)
Last Successful Sync Time: 2018-01-05 16:23:07
Source: time.windows.com,0x9
Poll Interval: 10 (1024s)

Phase Offset: 0.0023436s
ClockRate: 0.0156001s
State Machine: 1 (Hold)
Time Source Flags: 0 (None)
Server Role: 0 (None)
Last Sync Error: 0 (The command completed successfully.)
Time since Last Good Sync Time: 110.0410000s

C:\Windows\system32>w32tm /query /status /verbose
Leap Indicator: 0(no warning)
Stratum: 2 (secondary reference - syncd by (S)NTP)
Precision: -6 (15.625ms per tick)
Root Delay: 0.0314941s
Root Dispersion: 7.7760885s
ReferenceId: 0x81060F1E (source IP:  129.6.15.30)
Last Successful Sync Time: 2018-01-05 16:28:46
Source: time.nist.gov,0x9
Poll Interval: 10 (1024s)

Phase Offset: 0.0013057s
ClockRate: 0.0156001s
State Machine: 1 (Hold)
Time Source Flags: 0 (None)
Server Role: 0 (None)
Last Sync Error: 0 (The command completed successfully.)
Time since Last Good Sync Time: 5.3480000s
```
So my +75, +16, +16 post from last night was a bunch of hooey:

```
C:\Windows\system32>w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com
sjr1.tivo.com[204.176.49.10:123]:
    ICMP: 71ms delay
    NTP: +76.8957522s offset from local clock
        RefID: (unknown) [0x00017F7F]
        Stratum: 6
sjr2.tivo.com[204.176.49.11:123]:
    ICMP: 70ms delay
    NTP: +15.8330232s offset from local clock
        RefID: 'STEP' [0x50455453]
        Stratum: 0
sjr3.tivo.com[204.176.49.12:123]:
    ICMP: 82ms delay
    NTP: +15.9351843s offset from local clock
        RefID: timekeeper.isi.edu [128.9.176.30]
        Stratum: 6

C:\Windows\system32>w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com
sjr1.tivo.com[204.176.49.10:123]:
    ICMP: 69ms delay
    NTP: +109.4122088s offset from local clock
        RefID: (unknown) [0x00017F7F]
        Stratum: 6
sjr2.tivo.com[204.176.49.11:123]:
    ICMP: 68ms delay
    NTP: +16.6914488s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 6
sjr3.tivo.com[204.176.49.12:123]:
    ICMP: 68ms delay
    NTP: +16.8280332s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 6

C:\Windows\system32>w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com
sjr1.tivo.com[204.176.49.10:123]:
    ICMP: 70ms delay
    NTP: +95.3433151s offset from local clock
        RefID: (unknown) [0x00017F7F]
        Stratum: 6
sjr2.tivo.com[204.176.49.11:123]:
    ICMP: 70ms delay
    NTP: +0.4804667s offset from local clock
        RefID: 'STEP' [0x50455453]
        Stratum: 0
sjr3.tivo.com[204.176.49.12:123]:
    ICMP: 70ms delay
    NTP: +0.4917148s offset from local clock
        RefID: (unknown) [0x64F3A8C0]
        Stratum: 6

C:\Windows\system32>
```
sjr2 and sjr3 are performing as well as a retail PC can be expected while sjr1 is once again on its trip to Pluto.

I apologize for the confusion. [I'm running both my old Vista and newly re-imaged Win7 PCs through the same monitor, switching inputs, so maybe I only 'thought' that I updated the time on the Win7 PC before I ran w32tm /monitor.  ] Didn't want to introduce a false data point.

Regarding clock drift, time.is reports the old Vista PC is now 2.5 seconds behind, 33276 seconds after being sync'd.


----------



## m.s (Mar 8, 2007)

BobCamp1 said:


> If you're going to do that, just DNAT it to a real NTP server on the Internet. There are hundreds of them.


There are hundred/thousands of ntp pool servers available via dns, but it's considered bad form to point to a hard coded IP without explicit permission. I'd force the TiVo to my own stratum 1, except that would require double NAT to work, which is a PITA compared to simply redirecting it to a better TiVo server.


----------



## kdmorse (Jan 29, 2001)

Ran and traced a bunch of calls today for giggles, since sjr1 is hovering at +112s.

I started with a correct clock, and I could pretty much do all the daily calls I wanted, and it would stay that way. It simultaneously queried sjr1, sjr2, and sjr3. It liked sjr2 and sjr3, and discounted sjr1 as loopy.

If I blocked sjr2 and sjr3, on the first call it jumped ahead +112s, as it had no choice but to talk to sjr1.

If I then unblocked sjr1 and ran repeated calls, it stayed offset at +112s. It apparently liked sjr1, and discarded sjr2 and sjr3. This was a bit of a surprise at first, but the more I think about it, since sjr2 and sjr3 are reporting themselves as stratum 6, there's no real compelling reason for ntpdate to automatically assume sjr1 is the loopy one, as it's a tie, two local clocks to two local clocks.

I then blocked sjr1, and on the next call it snapped back to being correct as expected.

The moral of the story, despite sjr2 and sjr3 being sober, if a tivo latches on to sjr1, it will go out drinking with it, and ignore sjr2 and sjr3's voices of reason. Until it comes back hours later, hung over, and then gets to decide all over again if it's going to go out drinking again with sjr1 next time, or stay sober with 2 and 3.

Can't really fault the unit here, if ntpdate is fed bad information, sometimes it's going to make bad choices. It can only do so much when pointed at a ntp server that tells lies.

Now, starting with a correct clock, I've blocked all traffic from the unit. I'm curious how well the internal clock works. Although I may not be able to put up with the spinning blue circles this causes for long enough to collect any useful data.


----------



## PSU_Sudzi (Jun 4, 2015)

kdmorse said:


> Ran and traced a bunch of calls today for giggles, since sjr1 is hovering at +112s.
> 
> I started with a correct clock, and I could pretty much do all the daily calls I wanted, and it would stay that way. It simultaneously queried sjr1, sjr2, and sjr3. It liked sjr2 and sjr3, and discounted sjr1 as loopy.
> 
> ...


I don't know much about what these servers do aside from the time sync, are you saying they also are the same ones that provide guide updates, etc.? I blocked sjr1 and my time is correctly synced (or as correct as the other two servers are) so hopefully it will be the end of my time troubles.


----------



## ClearToLand (Jul 10, 2001)

kdmorse said:


> ...(Or block 204.176.49.10 at your router)


Is this the correct IPTABLES command (I'm running DD-WRT)?

iptables -I FORWARD -d 204.176.49.10 -j DROP​
*Reference: **DD-WRT and iptables - blocking outgoing access to specific IPs*


----------



## JoeKustra (Dec 7, 2012)

kdmorse said:


> Now, starting with a correct clock, I've blocked all traffic from the unit. I'm curious how well the internal clock works. Although I may not be able to put up with the spinning blue circles this causes for long enough to collect any useful data.


I've been thinking of this also. Tomorrow is my day to plug in my Premiere and Hydra Roamio (with a bad fan). I have removed their Ethernet connection, and the Roamio has never been configured for wireless. I'm curious if time is still kept with a box when its powered off. True, clock drift is still a factor, and I will test that as well. I have an old Magnavox DVDR that I had to put a capacitor across its crystal to get the time more accurate. Let you know what happens.

BTW, my Netgear router can block a URL with no problem. It's very easy. It can also log attempts at a URL's access.


----------



## kdmorse (Jan 29, 2001)

JoeKustra said:


> I'm curious if time is still kept with a box when its powered off. True, clock drift is still a factor, and I will test that as well.


I think this is model specific. I think Romaio's will keep their time when powered off. I think Bolts will lose time when powered off (or rebooting).


----------



## JoeKustra (Dec 7, 2012)

kdmorse said:


> I think this is model specific. I think Romaio's will keep their time when powered off. I think Bolts will lose time when powered off (or rebooting).


Makes sense. I've had the top off my Roamio and I've seen the battery.


----------



## Rob Helmerichs (Oct 17, 2000)

It looks like the only way to block sjr1.tivo.com on my Xfinity router is to do it through Parental Controls. But when I turn on Parental Controls, it has a list of every device that's ever been connected to the router with a yes/no toggle for "Trusted," and they're all set to No. Is there any downside to this?


----------



## PSU_Sudzi (Jun 4, 2015)

Rob Helmerichs said:


> It looks like the only way to block sjr1.tivo.com on my Xfinity router is to do it through Parental Controls. But when I turn on Parental Controls, it has a list of every device that's ever been connected to the router with a yes/no toggle for "Trusted," and they're all set to No. Is there any downside to this?


I wondered that too. By default all of my devices were set to NO but were still functioning online (only checked two of many). I changed them all to YES and today everything still seems to be working ok.

Edit: in reading the instructions further, you should set any devices you want to block as NO. So I updated that for all of my TiVos.


----------



## sharkster (Jul 3, 2004)

Wow, in a way I wish I understand one iota of the stuff some of you guys are discussing, for solutions, with all the sjr and w, etc, stuff. But, OTOH, I don't think my brain could handle it anymore. 

Carry on. So many here are so knowledgeable. I read a lot of stuff here, to learn more. Some of it just eludes me completely and - well, I guess that's ok.  Still in awe, though, and will read and learn as much as I can.


----------



## PSU_Sudzi (Jun 4, 2015)

sharkster said:


> Wow, in a way I wish I understand one iota of the stuff some of you guys are discussing, for solutions, with all the sjr and w, etc, stuff. But, OTOH, I don't think my brain could handle it anymore.
> 
> Carry on. So many here are so knowledgeable. I read a lot of stuff here, to learn more. Some of it just eludes me completely and - well, I guess that's ok.  Still in awe, though, and will read and learn as much as I can.


I don't understand a lot either. But I understand enough that there are 3 TiVo servers that set the time and one of them does a poor job at it jeopardizing my recording integrity. And whenever you connect to TiVo you could get the goat behind Door #3. So I blocked the address *sjr1.tivo.com* in my router so my TiVo can't connect to it any more. No more goat for me!


----------



## sharkster (Jul 3, 2004)

PSU_Sudzi said:


> I don't understand a lot either. But I understand enough that there are 3 TiVo servers that set the time and one of them does a poor job at it jeopardizing my recording integrity. And whenever you connect to TiVo you could get the goat behind Door #3. So I blocked the address *sjr1.tivo.com* in my router so my TiVo can't connect to it any more. No more goat for me!


Alrighty then!


----------



## NorthAlabama (Apr 19, 2012)

after 5 connections, one scheduled, 4 forced, i remain 45 sec fast - i give up (for tonight).


----------



## Mikeguy (Jul 28, 2005)

NorthAlabama said:


> after 5 connections, one scheduled, 4 forced, i remain 45 sec fast - i give up (for tonight).


Grab a glass of wine . . . .


----------



## NorthAlabama (Apr 19, 2012)

Mikeguy said:


> Grab a glass of wine . . . .


tonight - diet sundrop.  the occasional holiday wine was more regular than previous holidays, so i'm on self-imposed hiatus for a while...i'm such a cheap drunk.


----------



## Mikeguy (Jul 28, 2005)

NorthAlabama said:


> tonight - diet sundrop.  the occasional holiday wine was more regular than in previous holidays, so i'm on self-imposed hiatus for a while...i'm such a cheap drunk.


For me, can be as little as half a glass of wine. It can make Sunday brunches with free-flowing champagne a challenge, sometimes--remaining drink-free the last hour, reading a book at the table or in the car.


----------



## tim_m (Mar 8, 2017)

PSU_Sudzi said:


> I was just going to respond to say using parental controls was the only way I could block it in my modem too.


Thanks, i knew it had to be in there somewhere. It seems to be working too. Since i did that every connection has been almost correct give or take a few seconds. Not perfect but better then 1-3 minutes off.


----------



## tim_m (Mar 8, 2017)

Someone at tivo clearly needs to take the booze away from sjr1


----------



## DeltaOne (Sep 29, 2013)

PSU_Sudzi said:


> But I understand enough that there are 3 TiVo servers that set the time and one of them does a poor job at it jeopardizing my recording integrity. And whenever you connect to TiVo you could get the goat behind Door #3. So I blocked the address *sjr1.tivo.com* in my router so my TiVo can't connect to it any more. No more goat for me!


I've been following kdmorse and JoeKustra and love the work they're doing. Good stuff. I understand some of it, but not all.

A few months ago my Roamio was off by 2 or 3 minutes, and a network connection fixed it. But in this latest round of time problems my Roamio has been okay...or at least I haven't caught it with the clock being off.

I haven't blocked sjr1 yet, because I haven't seen the problem...and I'm hoping Tivo_Ted will get someone to fix the problem on TiVo's side.

New subject...does a Mini reach out to TiVo HQ for time settings...or does it ask the local host it's connected to?


----------



## JoeKustra (Dec 7, 2012)

DeltaOne said:


> New subject...does a Mini reach out to TiVo HQ for time settings...or does it ask the local host it's connected to?


While I haven't watched it, a Mini does have its own network service connection schedule and doesn't use its host for much more than the tuner. It will inherit apps, but unless you are as OCD as I am, the time on a Mini doesn't affect much anyhow.


----------



## DeltaOne (Sep 29, 2013)

JoeKustra said:


> While I haven't watched it, a Mini does have its own network service connection schedule and doesn't use its host for much more than the tuner. It will inherit apps, but unless you are as OCD as I am, the time on a Mini doesn't affect much anyhow.


Ah, good point about the time not being important on a Mini.


----------



## JoeKustra (Dec 7, 2012)

Applied power to an old Premiere and a new Roamio OTA. Both had no power for six days and no network connection when powered on. Both were 5 seconds fast. Not bad.

Premiere told me it had no internet access. Roamio (Hydra) just had blank thumbnails. No comment.


----------



## slowbiscuit (Sep 19, 2006)

Thought I had blocked NTP on sjr1 on my Asus RT-AC68U running Merlin firmware, but when my Roamio showed up 2 mins. fast again last night I had to do some digging. Found out that if you're running firmware prior to 380.65 there's a bug in the network services firewall that doesn't apply settings correctly. Upgraded this morning and now it's blocked, had to verify with w32tm as I long ago stopped running Linux here (I play too many games on Windows).

Sure wish Tivo_Ted would come back and update us, seems like he is a very sporadic poster here. Less often to respond than Margret?


----------



## moyekj (Jan 24, 2006)

slowbiscuit said:


> Thought I had blocked NTP on sjr1 on my Asus RT-AC86U running Merlin firmware, but when my Roamio showed up 2 mins. fast again last night I had to do some digging. Found out that if you're running firmware prior to 380.65 there's a bug in the network services firewall that doesn't apply settings correctly. Upgraded this morning and now it's blocked, had to verify with w32tm as I long ago stopped running Linux here (I play too many games on Windows).


I have Asus RT-N66R and I've been trying to figure out how to block a specific IP (204.176.49.10 in this case). But so far everything I've tried doesn't seem to work. I can still ping the above IP from my PC after applying changes to the router.
For my router the relevant section seems to be Firewall--Network Services Filter with "Filter table type" set to "Black List". This presents a table with:
Source IP, Port Range, Destination IP, Port Range, Protocol

According to router help leaving Source IP blank should apply to all LAN devices, so I left it blank and added 204.176.49.10 to Destination IP with empty Port Range and TCP protocol. But after doing that I can still ping that IP.

Can you share details of how you managed to block specific IP with your Asus router?


----------



## HerronScott (Jan 1, 2002)

moyekj said:


> I have Asus RT-N66R and I've been trying to figure out how to block a specific IP (204.176.49.10 in this case). But so far everything I've tried doesn't seem to work. I can still ping the above IP from my PC after applying changes to the router.
> For my router the relevant section seems to be Firewall--Network Services Filter with "Filter table type" set to "Black List". This presents a table with:
> Source IP, Port Range, Destination IP, Port Range, Protocol
> 
> ...


NTP is UDP (not TCP). Also, ping may not be affected since it's ICMP and not on a port.

Try testing with w32tm after blocking UDP traffic (assuming you are running Windows).

Scott


----------



## m.s (Mar 8, 2007)

moyekj said:


> According to router help leaving Source IP blank should apply to all LAN devices, so I left it blank and added 204.176.49.10 to Destination IP with empty Port Range and TCP protocol. But after doing that I can still ping that IP.
> 
> Can you share details of how you managed to block specific IP with your Asus router?


PING uses ICMP, not TCP (there's also a UDP PING, but that's less common). ntp uses UDP, not TCP.


----------



## PSU_Sudzi (Jun 4, 2015)

moyekj said:


> I have Asus RT-N66R and I've been trying to figure out how to block a specific IP (204.176.49.10 in this case). But so far everything I've tried doesn't seem to work. I can still ping the above IP from my PC after applying changes to the router.
> For my router the relevant section seems to be Firewall--Network Services Filter with "Filter table type" set to "Black List". This presents a table with:
> Source IP, Port Range, Destination IP, Port Range, Protocol
> 
> ...


Can you block a website on that router? If so block *sjr1.tivo.com* which is the domain name of 204.176.49.10. This should accomplish the same thing. Unless someone knows otherwise.


----------



## JoeKustra (Dec 7, 2012)

Just read the user manual for the Asus RT-N66R, and saw nothing the indicates it can block an IP address or URL.


----------



## moyekj (Jan 24, 2006)

Thanks guys. Yes the key was to block UDP Protocol, not TCP. So for my router:

Firewall--Network Services Filter with "Filter table type" set to "Black List".
This presents a table with:
Source IP, Port Range, Destination IP, Port Range, Protocol
I simply added Destination IP=204.176.49.10, Protocol=UDP as a Black List entry

Then checking with w32tm looks like sjr1.tivo.com is now successfully blocked since it returns a timeout error:

```
C:\WINDOWS\system32>w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com
sjr1.tivo.com[204.176.49.10:123]:
    ICMP: 34ms delay
    NTP: error ERROR_TIMEOUT - no response from server in 1000ms
sjr2.tivo.com[204.176.49.11:123]:
    ICMP: 32ms delay
    NTP: +2.2056805s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 6
sjr3.tivo.com[204.176.49.12:123]:
    ICMP: 31ms delay
    NTP: +1.1190138s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 6
```


----------



## TonyD79 (Jan 4, 2002)

PSU_Sudzi said:


> Can you block a website on that router? If so block *sjr1.tivo.com* which is the domain name of 204.176.49.10. This should accomplish the same thing. Unless someone knows otherwise.


So what happens when you do so? Does the connect fail or does it reroute on the TiVo side on connection?


----------



## PSU_Sudzi (Jun 4, 2015)

TonyD79 said:


> So what happens when you do so? Does the connect fail or does it reroute on the TiVo side on connection?


I don't know enough about networking to be 100% sure, but I'm guessing that it's rerouted. My connection today was successful. Maybe one of the guys here who know a lot about this can elaborate further.


----------



## TonyD79 (Jan 4, 2002)

PSU_Sudzi said:


> I don't know enough about networking to be 100% sure, but I'm guessing that it's rerouted. My connection today was successful. Maybe one of the guys here who know a lot about this can elaborate further.


Okay. I'm not sure anyone will know except by trail and error because it requires knowledge of internal servers.

I guess what I was asking was more along the lines of how long have you had this blocked and have you seen any failures?


----------



## NorthAlabama (Apr 19, 2012)

i'd love to hear the tape of someone attempting to explain this workaround to tivo tech support...with a strong accent...


----------



## PSU_Sudzi (Jun 4, 2015)

TonyD79 said:


> Okay. I'm not sure anyone will know except by trail and error because it requires knowledge of internal servers.
> 
> I guess what I was asking was more along the lines of how long have you had this blocked and have you seen any failures?


It's only been blocked since last night and one connection so far that was successful. I'll keep an eye on it and see if it fails and if so will let everyone know. But I think if you read back on this post there are others who have blocked the bad server for some time and I haven't seen any issues from them posted.

Edit: to clarify no connections have failed so far, 1 for 1 success.


----------



## TonyD79 (Jan 4, 2002)

PSU_Sudzi said:


> It's only been blocked since last night and one connection so far that was successful. I'll keep an eye on it and see if it fails and if so will let everyone know. But I think if you read back on this post there are others who have blocked the bad server for some time and I haven't seen any issues from them posted.
> 
> Edit: to clarify no connections have failed so far, 1 for 1 success.


Thanks. I will try it too.


----------



## BobCamp1 (May 15, 2002)

Note that when using parental controls, your Internet speed will probably get cut in half for the devices you apply it to. This is definitely true of the FIOS routers and most typical home routers. So be sure to only apply the rules just to the Tivos and not for your entire network.

I did this and my other devices don't seem to be affected. My Tivo is now getting 12 Mbps but I don't use it for streaming so I'll be fine.

An alternate solution is to use OpenDNS. You set up your router to use the DNS servers they give you, and you in turn can configure it to block the bad NTP Tivo server.


----------



## Rob Helmerichs (Oct 17, 2000)

Parental controls doesn't seem to work for me...I add sjr1.tivo.com, but I can still ping it.


----------



## PSU_Sudzi (Jun 4, 2015)

Rob Helmerichs said:


> Parental controls doesn't seem to work for me...I add sjr1.tivo.com, but I can still ping it.


From looking at the comments in this thread it seems pinging it is still possible but it's unclear to me if blocking it on the XB3 accomplishes what we want. I don't know enough to say I guess.


----------



## NorthAlabama (Apr 19, 2012)

Rob Helmerichs said:


> Parental controls doesn't seem to work for me...I add sjr1.tivo.com, but I can still ping it.


which model arris gateway with comcast are you using?


----------



## Rob Helmerichs (Oct 17, 2000)

NorthAlabama said:


> which model arris gateway with comcast are you using?


It's the Cisco DPC3941T.


----------



## NorthAlabama (Apr 19, 2012)

Rob Helmerichs said:


> It's the Cisco DPC3941T.


oh, i'd remembered (incorrectly) you said arris - i'm jealous, the cisco is a unicorn in our market, though the stories from outside forest are very positive.


----------



## m.s (Mar 8, 2007)

Trying to block sjr1.tivo.com may or may not work, depending on how filtering is implemented in your NAT gateway, but probably not. The IP addresses are hardcoded - the TiVo box doesn't do a DNS lookup, nor does it go to a URL to get to the ntp servers.


----------



## Rob Helmerichs (Oct 17, 2000)

m.s said:


> Trying to block sjr1.tivo.com may or may not work, depending on how filtering is implemented in your NAT gateway, but probably not. The IP addresses are hardcoded - the TiVo box doesn't do a DNS lookup, nor does it go to a URL to get to the ntp servers.


So I blocked the IP address, and that worked. Thanks!


----------



## PSU_Sudzi (Jun 4, 2015)

m.s said:


> Trying to block sjr1.tivo.com may or may not work, depending on how filtering is implemented in your NAT gateway, but probably not. The IP addresses are hardcoded - the TiVo box doesn't do a DNS lookup, nor does it go to a URL to get to the ntp servers.


Thanks for the input, I removed the block and will just keep checking the time nightly when I review the TDL, another manual process. Maybe I should just give up and see if Rovi/TiVo will start publishing VCRPlus codes again.


----------



## samccfl99 (Sep 7, 2013)

Sun 01/07 6:45, scheduled connection, woke up and it was 45+ seconds fast. Connected again at 9:45 am and almost perfect to the second.

Did a manual connection around 2:09 am earlier this morning and it was fairly close. It said next connection was supposed to be around 1800 tonite, so why did the stupid thing connect by itself at 6:45 am?

I use this to set MY clocks The Official NIST US Time:

*THIS **** MUST STOP*


----------



## JoeKustra (Dec 7, 2012)

samccfl99 said:


> Did a manual connection around 2:09 am earlier this morning and it was fairly close. It said next connection was supposed to be around 1800 tonite, so why did the stupid thing connect by itself at 6:45 am?


Manual service connections do not change scheduled connections by very much.


----------



## NorthAlabama (Apr 19, 2012)

JoeKustra said:


> Manual service connections do not change scheduled connections by very much.


test connections don't impact them at all.


----------



## JoeKustra (Dec 7, 2012)

NorthAlabama said:


> test connections don't impact them at all.


Yeah, but retraining can be hard. Test connections are much quicker also.


----------



## samccfl99 (Sep 7, 2013)

JoeKustra said:


> Manual service connections do not change scheduled connections by very much.


Yes, but I know what it said after I connected manually at 2:09 this morning and it said 8ish pm to connect again and it did change from what it was (6ish pm). DUH...LOL

Anyway, I got a Netgear WNDR4300 router and I cant figure out where to block it. I do have some Port Forwarding entries for my phone isp, and UDP is not an option, but I do not think that is the right place. No section for Blocked IPs. Anyone?

I also do not understand why we have to do this. For years there was no real problem with the Freaking Time. Our phones do it correctly all the time!


----------



## JoeKustra (Dec 7, 2012)

samccfl99 said:


> Yes, but I know what it said after I connected manually at 2:09 this morning and it said 8ish pm to connect again.
> Anyway, I got a Netgear WNDR4300 router and I cant figure out where to block it. I do have some Port Forwarding entries for my phone isp, and UDP is not an option, but I do not think that is the right place. No section for Blocked IPs. Anyone?
> I also do not understand why we have to do this. For years there was no real problem with the Freaking Time. Our phones do it correctly all the time!


You posted 1800, which is 6pm and within normal connection parameters -> Daily Guide Updates

My/Your Netgear router has a Security tab. There is a Block Sites tab. There is also a check box to log any access to a blocked site in Administration. Enter the above posted URL and check Always and Apply.

Page 69 -> http://www.downloads.netgear.com/files/GDC/WNDR4300/WNDR4300_UM_23Sept2014.pdf


----------



## japaget (Mar 12, 2007)

In the results below, note that the "RefID" for sjr1.tivo.com is unknown but the "RefID" for sjr2.tivo.com and sjr3.tivo.com is time3.apple.com, which is probably a valid NTP time server. Could this be the cause of the incorrect times on sjr1.tivo.com?


```
d:\tv>w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com
sjr1.tivo.com[204.176.49.10:123]:
    ICMP: 40ms delay
    NTP: +23.7247945s offset from local clock
        RefID: (unknown) [0x00017F7F]
        Stratum: 6
sjr2.tivo.com[204.176.49.11:123]:
    ICMP: 40ms delay
    NTP: +3.2238638s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 6
sjr3.tivo.com[204.176.49.12:123]:
    ICMP: 41ms delay
    NTP: +1.4959133s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 6

Warning:
Reverse name resolution is best effort. It may not be
correct since RefID field in time packets differs across
NTP implementations and may not be using IP addresses.

d:\tv>
```


----------



## samccfl99 (Sep 7, 2013)

JoeKustra said:


> Page 69 -> http://www.downloads.netgear.com/files/GDC/WNDR4300/WNDR4300_UM_23Sept2014.pdf


Thanks Professor! Of course I have the manual and looked at it. I just don't ask questions because I am lazy! I missed that one (why I do not know, I was looking at Block Sites for some stupid reason). I guess I am missing something. I thought the point was to block 204.176.49.10:123? In the IP sectoon, the first 3 octets are from my router address and they are grayed out only allowing the last octet. I am not totally up on my networking (but not a novice).


----------



## JoeKustra (Dec 7, 2012)

samccfl99 said:


> Thanks Professor! Of course I have the manual and looked at it. I just don't ask questions because I am lazy! I missed that one (why I do not know, I was looking at Block Sites for some stupid reason). I guess I am missing something. I thought the point was to block 204.176.49.10:123? In the IP sectoon, the first 3 octets are from my router address and they are grayed out only allowing the last octet. I am not totally up on my networking (but not a novice).


You want to block *sjr1.tivo.com. *If I'm wrong please correct me.

When I block that site, the command
w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com
no longer works.


----------



## kdmorse (Jan 29, 2001)

A large portion of "what you need to do", and "does it work for you" is completely router dependent - including even the most basic of questions.

If your router, when told to block something by DNS name, blocks DNS resolution for that name, then it won't help. DNS shenanigans don't help here. Similarly, if it's more of a 'web' blocker (as many parental controls are), it will block web requests to that name, which will be doubly useless (as the request is neither a web request, nor by name).

If your router, when told to block something by DNS name, looks up the name, and blocks the resulting IP address, it should work fine. iptables on the command line for example works that way, as do many router firewall interfaces.

As mentioned upthread, if you want to be specific, the port you want to block is UDP port 123. However there is no harm in simply blocking all access to 204.176.49.10. Its only use here is as a ntp server.

Realistically, a better test is:

*w32tm /monitor /computers:204.176.49.10,204.176.49.11,204.176.49.12*

If .10 doesn't respond, and .11 and .12 do, then you should be all set.


----------



## m.s (Mar 8, 2007)

JoeKustra said:


> You want to block *sjr1.tivo.com. *If I'm wrong please correct me.


The GP was correct. You want to block 204.176.49.10, which is hardcoded in the TiVo. Blocking sjr1.tivo.com will do nothing, unless your NAT gateway does a dns lookup then blocks the returned IP (very unlikely).


----------



## JoeKustra (Dec 7, 2012)

m.s said:


> The GP was correct. You want to block 204.176.49.10, which is hardcoded in the TiVo. Blocking sjr1.tivo.com will do nothing, unless your NAT gateway does a dns lookup then blocks the returned IP (very unlikely).


Interesting.

BTW, sjr1 has a 42 second error. sjr2 and sjr3 have a 3 and 1 second offset.


----------



## kdmorse (Jan 29, 2001)

japaget said:


> In the results below, note that the "RefID" for sjr1.tivo.com is unknown but the "RefID" for sjr2.tivo.com and sjr3.tivo.com is time3.apple.com, which is probably a valid NTP time server. Could this be the cause of the incorrect times on sjr1.tivo.com?


It's absolutely related. The RefID's on those servers are wildly inconsistent, frequently being lost entirety, or indicating that the NTP server itself no longer considers itself a reliable source. Null RefID's, and RefID's of 'LOCAL', 'STEP', and 'INIT' are common - which are all different ways of saying "Good lord man, don't ask *me* what time it is!" (for different reasons). As is the fact that they flap between stratum 2 (good), 6 (bad), and 16 (broken).

It's not the 'cause' per se, it's just another set of *very* obvious symptoms. Clearly these NTP servers have trouble talking to whatever they're supposed to consider their upstream servers, don't work well with each other as peers, and drift far worse than average when they do.

But given we can see all these things from our desktops, it's a little comedic when the Tivo noc comes back and says they don't see anything wrong.



JoeKustra said:


> BTW, sjr1 has a 42 second error. sjr2 and sjr3 have a 3 and 1 second offset.


Yup, they're all a little funkier than normal today.


----------



## m.s (Mar 8, 2007)

kdmorse said:


> it's a little comedic when the Tivo noc comes back and says they don't see anything wrong.


Bangalore is _hours_ different than your time, and you're complaining about a few minutes?

suggestion, for every post, add a shout out to @TiVo_Ted


----------



## reneg (Jun 19, 2002)

I have ATT fiber internet connection, blocking instructions for the Pace 5268AC (ATT gateway) router.

Connect to you router via a browser (192.168.1.254, ATT default). Navigate to Settings -> Firewall -> Firewall Rules

I could not enter a generic rule as the Pace router is looking for a source IP address and would not except "any", "all", etc... I had to add a rule for each Tivo, and I also added one for my computer so I could test the firewall rule.

Click the + sign to add a rule:
Type - Outbound
Source - <IP address of your Tivo>
Dest - 204.176.49.10
Port Range - 123 - 123
Protocol - all
Action - Deny

Click Save - you should be prompted for you access code (router password). If you've entered it correcting, you'll get a Configuration Successful message. Repeat if necessary for other Tivos.


----------



## PSU_Sudzi (Jun 4, 2015)

kdmorse said:


> A large portion of "what you need to do", and "does it work for you" is completely router dependent - including even the most basic of questions.
> 
> If your router, when told to block something by DNS name, blocks DNS resolution for that name, then it won't help. DNS shenanigans don't help here. Similarly, if it's more of a 'web' blocker (as many parental controls are), it will block web requests to that name, which will be doubly useless (as the request is neither a web request, nor by name).
> 
> ...


Thanks for explaining this all out, blocking the sj1 server web address on my Arris TG1682G (aka Comcast XB3) modem didn't do anything as I was able to get a response back for .10 so I guess I will just keep monitoring it.


----------



## Rob Helmerichs (Oct 17, 2000)

PSU_Sudzi said:


> Thanks for explaining this all out, blocking the sj1 server web address on my Arris TG1682G (aka Comcast XB3) modem didn't do anything as I was able to get a response back for .10 so I guess I will just keep monitoring it.


Did you try blocking the IP address? That worked for me where the URL didn't...


----------



## PSU_Sudzi (Jun 4, 2015)

Rob Helmerichs said:


> Did you try blocking the IP address? That worked for me where the URL didn't...


How did you block the IP address? If I recall from one of your previous posts I think we are both on Comcast but have different modems.


----------



## samccfl99 (Sep 7, 2013)

I do not know how to do it on my Netgear router and I would like TED to explain to us why we even have to worry about this CRAP and check on it CONSTANTLY...


----------



## Rob Helmerichs (Oct 17, 2000)

PSU_Sudzi said:


> How did you block the IP address? If I recall from one of your previous posts I think we are both on Comcast but have different modems.


I just put in the IP instead of the URL. http://204.176.49.10


----------



## PSU_Sudzi (Jun 4, 2015)

Rob Helmerichs said:


> I just put in the IP instead of the URL. http://204.176.49.10


I guess that doesn't work for me as I am still able to ping it and get a response. Oh well.


----------



## ClearToLand (Jul 10, 2001)

PSU_Sudzi said:


> I guess that doesn't work for me as I am still able to ping it and get a response. Oh well.


It was stated earlier in this thread that PING and NTP are different functions - try the w32tm /monitor command to be sure.


----------



## PSU_Sudzi (Jun 4, 2015)

ClearToLand said:


> It was stated earlier in this thread that PING and NTP are different functions - try the w32tm /monitor command to be sure.


Sorry, was using the word "ping" but had used the w32tm /monitor command suggested by @kdmorse and received back info on all three servers. I also made sure the PC I was using to run the command was also blocking the IP address (parental controls on my modem can be restricted to devices) and it seems like I could still get through.


----------



## powrcow (Sep 27, 2010)

I don't force network connections or updates or anything on my old Premiere. It "just works" (guide problems aside). Yesterday there was a power outage causing my TiVo to reboot a little before noon. Now my TiVo's clock is off by about one minute or so.

I was feeling left out of this conversation, but now I'm in!


----------



## NorthAlabama (Apr 19, 2012)

powrcow said:


> I was feeling left out of this conversation, but now I'm in!


welcome to the party?


----------



## astrohip (Jan 7, 2003)

I've never had this issue, but noticed yesterday my Premiere (Elite) was off by two minutes. A forced connection fixed it. My other two units did NOT have an issue.

I was feeling left out also, but now I'm part of the cool crowd too. I guess it's spreading.

I'm going to call TiVo CS, just for giggles...


----------



## V7Goose (May 28, 2005)

astrohip said:


> I'm going to call TiVo CS, just for giggles...


Some kind of strange masochist here . . .


----------



## astrohip (Jan 7, 2003)

Started with CHAT:

CSR Bernald Ian says
_I understand. *As of the moment we are not receiving any issues about the clock. *I highly suggest to contact our phone support so that we can do further investigation about this case._

Whaddaya know? No issues?!?


----------



## NorthAlabama (Apr 19, 2012)

astrohip said:


> Started with CHAT:
> 
> CSR Bernald Ian says
> _I understand. *As of the moment we are not receiving any issues about the clock. *I highly suggest to contact our phone support so that we can do further investigation about this case._
> ...


so, i've gotta ask - if they're not receiving any "issues", how'd he explain your report, and what did he offer as a solution (darfc)?


----------



## samccfl99 (Sep 7, 2013)

Just noticed. 1:07 PM today. Scheduled connection...*1:30 minutes AHEAD

where is the emoji for PISSED OFF????
*
Fixed after manual connection. They need an option to stop automatic connections.


----------



## NorthAlabama (Apr 19, 2012)

i'm only 20 sec fast after today's connection, not touchin' it.


----------



## dlfl (Jul 6, 2006)

About an hour ago my clock was 13 secs fast and I decided to block 204.176.49.10 in my router. During the 30 mins or so of fiddling around to get this done I noticed the clock had worsened to more than a minute fast!

I configured the router block and did a network test on the TiVo. Clock is now within 1 sec (limit of my eyeball measurement).


----------



## burdellgp (Mar 28, 2008)

m.s said:


> There are hundred/thousands of ntp pool servers available via dns, but it's considered bad form to point to a hard coded IP without explicit permission. I'd force the TiVo to my own stratum 1, except that would require double NAT to work, which is a PITA compared to simply redirecting it to a better TiVo server.


That's what I'm doing at my house:

$ ntpdate -qv sjr{1..3}.tivo.com
10 Jan 21:12:23 ntpdate[6142]: ntpdate [email protected] Fri Aug 4 04:54:25 UTC 2017 (1)
server 204.176.49.10, stratum 1, offset 0.000366, delay 0.02600
server 204.176.49.11, stratum 1, offset 0.000375, delay 0.02600
server 204.176.49.12, stratum 1, offset 0.000359, delay 0.02600

Perfect time, every time!


----------



## murgatroyd (Jan 6, 2002)

astrohip said:


> Started with CHAT:
> 
> CSR Bernald Ian says
> _I understand. *As of the moment we are not receiving any issues about the clock. *I highly suggest to contact our phone support so that we can do further investigation about this case._
> ...


Yeah, that's like going into a store and asking for [something] and the sales droid says "Oh, nobody ever asks for that!" I always wish I had some snappy comeback. Hello, there's a customer right in front of your face waving money at you, and the best you can say is "nobody ever?"


----------



## dlfl (Jul 6, 2006)

m.s said:


> There are hundred/thousands of ntp pool servers available via dns, but it's considered bad form to point to a hard coded IP without explicit permission. I'd force the TiVo to my own stratum 1, except that would require double NAT to work, which is a PITA compared to simply redirecting it to a better TiVo server.





burdellgp said:


> That's what I'm doing at my house:
> 
> $ ntpdate -qv sjr{1..3}.tivo.com
> 10 Jan 21:12:23 ntpdate[6142]: ntpdate [email protected] Fri Aug 4 04:54:25 UTC 2017 (1)
> ...


So you're redirecting (at your router) all three of those IP's to just one (that is not the ...49.10 one) ? Or perhaps to another NTP server that is stratum 1? Would appreciate more detail for those of us who are not networking experts.


----------



## m.s (Mar 8, 2007)

I only redirect the first one to point to the third, the other two seem to be close enough. You have to have a router or firewall which allows DNAT configuration. For iptables, it would be something like:

```
iptables -t nat -A PREROUTING -d 204.176.49.10 -j DNAT --to-destination 204.176.49.12
```
I use a Juniper SRX, where it's something like:

```
set security address-book global address sjr1-tivo-com 204.176.49.10/32
set security nat destination pool sjr3-tivo-com 204.176.49.12/32
set security nat destination rule-set HOME_OUT from zone HOME_NET
set security nat destination rule-set HOME_OUT rule BadTivoNtp1 match destination-address-name sjr1-tivo-com
set security nat destination rule-set HOME_OUT rule BadTivoNtp1 then destination-nat pool sjr3-tivo-com
```


----------



## jth tv (Nov 15, 2014)

Why would TiVo dvrs get time differently than the typical pc ? If they do it differently aren't they just asking for trouble ?


----------



## Rob Helmerichs (Oct 17, 2000)

jth tv said:


> Why would TiVo dvrs get time differently than the typical pc ? If they do it differently aren't they just asking for trouble ?


I don't think they do it any differently...it's just that Microsoft's time servers apparently never experience this kind of problem.

Although this very minute I'm looking at my work computer with my home computer in a window, and the times are ten seconds apart...


----------



## GBL (Apr 20, 2000)

Interestingly enough, currently sjr2 is on a bender ... 51 seconds fast. 

```
C:\>w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com
sjr1.tivo.com[204.176.49.10:123]:
    ICMP: 81ms delay
    NTP: +3.8278946s offset from local clock
        RefID: timekeeper.isi.edu [128.9.176.30]
        Stratum: 6
sjr2.tivo.com[204.176.49.11:123]:
    ICMP: 82ms delay
    NTP: +51.7164387s offset from local clock
        RefID: (unknown) [0x00017F7F]
        Stratum: 6
sjr3.tivo.com[204.176.49.12:123]:
    ICMP: 81ms delay
    NTP: +0.9183042s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 6
```


----------



## JoeKustra (Dec 7, 2012)

Rob Helmerichs said:


> I don't think they do it any differently...it's just that Microsoft's time servers apparently never experience this kind of problem.
> Although this very minute I'm looking at my work computer with my home computer in a window, and the times are ten seconds apart...


I've found even the .GOV servers can be off-line. I check my event log in the morning and sometime see a failure. As for windows time setting, that's a whole different thread.

So far, none of my TiVo boxes have had a problem.

w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com


----------



## dlfl (Jul 6, 2006)

GBL said:


> Interestingly enough, currently sjr2 is on a bender ... 51 seconds fast.
> . ............


Oh great! I've already blocked the sjr1 IP. If I block sjr2 that leaves only sjr3. Does it ever go on benders? And is just one server enough to update the time?

My router is a TP-Link Archer C7 (AC1750). I don't think it has double-NAT capability and doubt it can just redirect an IP to some NTP server IP other than the tivo.com ones. There is a feature called "static routing" but I don't understand enough about what it does to know if it would work or how to configure it (e.g., what submask?).


----------



## kdmorse (Jan 29, 2001)

GBL said:


> Interestingly enough, currently sjr2 is on a bender ... 51 seconds fast.


I worry that they "solved" the problem by swapping sjr1 and sjr2, which of course won't help, in many ways would make the problem worse, and is just the laziest of lazy approaches.

If so, there is no "Epic Facepalm" image that sufficiently encapsulates my response.

(Or, they somehow moved the problem from sjr1 to sjr2, which is a neat trick)


----------



## DBrunetti (Dec 6, 2016)

So I just checked and sjr2 is off about 72 seconds. Sjr1 and sjr3 are basically spot on and nearly identical. Maybe they did swapped them around.


----------



## slowbiscuit (Sep 19, 2006)

****, now I've blocked both 1 and 2 in the router. Hopefully 3 will stay alive until they fix this stupidity.

Yet another thread to monitor to deal with Tivo's crap.


----------



## SteveD (Oct 22, 2002)

I just checked my Tivos and one of them was almost 90 seconds fast even though I have sjr1 blocked. Checking my firewall logs, it made the connection around 2PM today.
I decided to block sjr2 for now and see how long that will last for.
Maybe we need a time server alert sticky thread?


----------



## dlfl (Jul 6, 2006)

**** Red said:


> ..........
> Maybe we need a time server alert sticky thread?


Great idea, then TiVo server ops team could monitor it. 

My time was fast by 1:20 mins just now after a 3 pm service connection with sjr1 blocked. Now I've also blocked sjr2, did an internet test and time is good.

Is there a time server monitoring app for iOS?


----------



## m.s (Mar 8, 2007)

Dam TiVo. Just went through the trouble of NAT'ing to my own server. It's off by about 12 microseconds. 

```
ntp3:~# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*GPS_NMEA(0)     .gPPS.           0 l    5   16  377    0.000   -0.012   0.015
+locke           .gPPS.           1 u    7   64  377    0.741    0.078   0.052
+ntp             .gPPS.           1 u   46   64  377    1.797    0.140   0.079
-ntp2            .gPPS.           1 u   49   64  377    1.848    0.179   0.114
 LOCAL(0)        .LOCL.          10 l   49   64  377    0.000    0.000   0.015
ntp3:~#
```


----------



## slowbiscuit (Sep 19, 2006)

Yeah that's my next step, DNAT'ing these IPs to some local NTP server. Hate to do that by IP instead of DNS but won't have much choice if sjr3 goes bad too.


----------



## sharkster (Jul 3, 2004)

I noticed last evening, just after 6:00, that my living room Bolt was fast again so I did a connection and it took an hour and half! I don't know what the heck it was 'loading' but that was way more than even an actual update would take. Clock got better, though. SMH


----------



## JoeKustra (Dec 7, 2012)

sharkster said:


> I noticed last evening, just after 6:00, that my living room Bolt was fast again so I did a connection and it took an hour and half! I don't know what the heck it was 'loading' but that was way more than even an actual update would take. Clock got better, though. SMH


Did your software change?


----------



## sharkster (Jul 3, 2004)

JoeKustra said:


> Did your software change?


Nope. I still have 7.4.RC18

I even looked when it was finished to see if it was 'pending restart', as I had recordings coming up for the evening so I knew it wouldn't restart during prime time and if it said that I'd know there was some kind of update. I have no idea. My internet is fast and my wireless is very good. I can transfer now a one-hour HD show in btwn 4.5 and 5.5 minutes.

Just weird, I guess. Maybe their servers are slower during that time period. (?) No harm done. I was just flummoxed by it.


----------



## moyekj (Jan 24, 2006)

No need to do a full net connect to set the clock. Simply running a network test is sufficient for setting the clock.


----------



## JoeKustra (Dec 7, 2012)

sharkster said:


> Nope. I still have 7.4.RC18
> I even looked when it was finished to see if it was 'pending restart', as I had recordings coming up for the evening so I knew it wouldn't restart during prime time and if it said that I'd know there was some kind of update. I have no idea. My internet is fast and my wireless is very good. I can transfer now a one-hour HD show in btwn 4.5 and 5.5 minutes.
> Just weird, I guess. Maybe their servers are slower during that time period. (?) No harm done. I was just flummoxed by it.


If you spent a long time on "Loading...XX%" then your box was working hard. If it spent a long time on "Downloading" then I would be surprised, since that should timeout. BTW, I'm still waiting for RC18.


----------



## sharkster (Jul 3, 2004)

Yeah, the loading percentage was what went super slowly. Hope that doesn't mean the box has problems. It's not even 2 years old and that would be horrible. Seems ok.


----------



## JoeKustra (Dec 7, 2012)

sharkster said:


> Yeah, the loading percentage was what went super slowly. Hope that doesn't mean the box has problems. It's not even 2 years old and that would be horrible. Seems ok.


Maybe tomorrow I will measure the difference between Hydra and classic UI. I have both boxes doing their two-day update tomorrow which should magnify any differences. I do feel the Hydra updates take longer, but it would be good to have numbers.

Maybe, like windows, every update gets bigger & slower?


----------



## sfhub (Jan 6, 2007)

sharkster said:


> Yeah, the loading percentage was what went super slowly. Hope that doesn't mean the box has problems. It's not even 2 years old and that would be horrible. Seems ok.


If you can, try rebooting and check the version. I noticed the recent updates have long loading phases and short boots (instead of the previous may take an hour page). Also they don't always say pending restart.


----------



## sharkster (Jul 3, 2004)

sfhub said:


> If you can, try rebooting and check the version. I noticed the recent updates have long loading phases and short boots (instead of the previous may take an hour page). Also they don't always say pending restart.


That's interesting. Thanks for the info and advice. Am recording on that box right now but will pocket that for a littler later this afternoon. 

I've never seen that before. Or, it's probably more likely that I was clueless to it.


----------



## GBL (Apr 20, 2000)

From my perspective it looks like TiVo finally may have their timer servers under control - I've only seen small variances of few seconds. Anybody else see something different?


```
>
>w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com
sjr1.tivo.com[204.176.49.10:123]:
    ICMP: 101ms delay
    NTP: +1.3095971s offset from local clock
        RefID: (unknown) [0x64F3960A]
        Stratum: 6
sjr2.tivo.com[204.176.49.11:123]:
    ICMP: 85ms delay
    NTP: +0.5298801s offset from local clock
        RefID: (unknown) [0x00017F7F]
        Stratum: 6
sjr3.tivo.com[204.176.49.12:123]:
    ICMP: 81ms delay
    NTP: +0.5973641s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 6
```


----------



## JoeKustra (Dec 7, 2012)

GBL said:


> From my perspective it looks like TiVo finally may have their timer servers under control - I've only seen small variances of few seconds. Anybody else see something different?


I see that also. But I'll give it a few hours. The day is young.


----------



## DBrunetti (Dec 6, 2016)

I've also been checking the servers for the last several hours and all three have been stable.


----------



## m.s (Mar 8, 2007)

GBL said:


> From my perspective it looks like TiVo finally may have their timer servers under control


No, they're still screwed up. Two of them have unknown refid, which probably means they're not sync'd to anything, just running on their local clock. When you checked, that they're all reporting stratum 6 just shows they're all misconfigured.
time3.apple.com is stratum 1, so sjr3 should be reporting itself as stratum 2. But right now, 1 and 2 are reporting stratum 16.

```
locke:~# ntpdate -q -p1 time3.apple.com
server 17.254.0.31, stratum 1, offset -0.000482, delay 0.09717
13 Jan 10:25:42 ntpdate[21349]: adjust time server 17.254.0.31 offset -0.000482 sec
locke:~# ntpdate -q -p1 sjr1.tivo.com
server 204.176.49.10, stratum 16, offset 0.828403, delay 0.09518
13 Jan 10:25:45 ntpdate[21350]: no server suitable for synchronization found
locke:~# ntpdate -q -p1 sjr2.tivo.com
server 204.176.49.11, stratum 16, offset 0.333987, delay 0.09563
13 Jan 10:25:48 ntpdate[21353]: no server suitable for synchronization found
locke:~# ntpdate -q -p1 sjr3.tivo.com
server 204.176.49.12, stratum 6, offset 0.971833, delay 0.09564
13 Jan 10:25:52 ntpdate[21359]: step time server 204.176.49.12 offset 0.971833 sec
```
I'm pretty sure they're running Windows time service, and not the canonical ntpd. Ugh.


----------



## dlfl (Jul 6, 2006)

Just got this:



> b'server 204.176.49.10, stratum 6, offset 1.705864, delay 0.11475\n13 Jan 10:36:29 ntpdate[3509]: step time server 204.176.49.10 offset 1.705864 sec\n'
> 
> b'server 204.176.49.11, stratum 6, offset 0.779760, delay 0.11456\n13 Jan 10:36:35 ntpdate[3510]: step time server 204.176.49.11 offset 0.779760 sec\n'
> 
> b'server 204.176.49.12, stratum 16, offset 0.247027, delay 0.11536\n'


Using ntpdate -q on my raspberry pi. Doesn't look good (i.e., stratum 16) for sjr3. I'm a noobie to this ntp stuff. I believe the offsets are relative to the RPI's clock, right? It is being set by a daemon that runs automatically and is always within a second of WWV ("atomic clock") time even when the RPI runs for days. Stratum 16 means the server is unsynchronized which can't be good, even though the offset is low.

EDIT: Now all three are stratum 6, with offsets ranging from 1 to 1.5 secs.


----------



## HerronScott (Jan 1, 2002)

m.s said:


> I'm pretty sure they're running Windows time service, and not the canonical ntpd. Ugh.


Windows time service works fine assuming your hardware and network infrastructure are configured correctly or at least we haven't seen any issues over the last 17 years here (1000 servers and 10,000 workstations) and yeah probably thus the refid as you mentioned, but stratum 6 or 16 is just wrong.

Scott


----------



## sharkster (Jul 3, 2004)

I wish some of you guys worked at Rivo! You all seem to know way more than the ones forking everything up there.


----------



## JoeKustra (Dec 7, 2012)

sharkster said:


> I wish some of you guys worked at Rivo! You all seem to know way more than the ones forking everything up there.


I would be happy if they would just document the logs. Plus, we work for free.


----------



## m.s (Mar 8, 2007)

HerronScott said:


> Windows time service works fine...


For the definition of "fine" which only includes what Windows needs to function. It's poorly documented (the disciplining algorithms are completely undocumented), doesn't support reference clocks per se, and has questionable accuracy. By Microsoft's own admission:


> Windows Server 2012 R2 and earlier versions of Windows cannot guarantee highly accurate time because the W32Time service was not a full-featured NTP solution. ... Computers running Windows Server 2016 and newer operating system versions can deliver 1 second, 50 ms second and 1 ms accuracy when specific OS version, network latency and other environmental requirements have been satisfied.


ntp or Chrony do a much better job.


----------



## dlfl (Jul 6, 2006)

Watching over the last hour or so, one of the servers will pop into stratum 16 for a little while then back to stratum 6. But the offsets all have been less than 2 secs.


----------



## burdellgp (Mar 28, 2008)

dlfl said:


> So you're redirecting (at your router) all three of those IP's to just one (that is not the ...49.10 one) ? Or perhaps to another NTP server that is stratum 1? Would appreciate more detail for those of us who are not networking experts.


In my case, I already have a stratum 1 server in the house (GPS receiver running in time-sync mode connected to an always-on server). I DNAT all UDP port 123 connections to the three TiVo IPs to my home server, then SNAT the source to the router (so the server replies to the router to reverse the NAT). This is on a router running LEDE (fork of OpenWRT).

There's really no reason for TiVo to even be running these bad NTP servers; they could work with the public NTP pool project and use hostnames like 0.tivo.pool.ntp.org, 1.tivo.pool.ntp.org, 2.tivo.pool.ntp.org (and TiVos setting the clock would get a variety of sources). Ideally then TiVo would set up some servers to join the pool and help with the load, but not if they can't run one any better than this.


----------



## HerronScott (Jan 1, 2002)

m.s said:


> For the definition of "fine" which only includes what Windows needs to function. It's poorly documented (the disciplining algorithms are completely undocumented), doesn't support reference clocks per se, and has questionable accuracy. By Microsoft's own admission:
> ntp or Chrony do a much better job.


Yes, I'm fully aware of Microsoft's statement and the details above (actually queried Microsoft last year on the algorithms used by the Windows Time Service to answer some internal queries on the time sync). The Kerberos requirement for time synchronization for authentication and other Windows requirements are pretty broad. My point is that we have never seen issues like the TiVo NTP servers are displaying.

Currently with 41 DC's in data centers and branch offices around the world, our greatest offset from our root is -0.0233637s (and our root's offset from the NIST NTP server is +0.0077669s) I'm sure ntp or Chony could do better but we're good with this level of accuracy and I'd be happy if TiVo's NTP servers were behaving this well. 

Scott


----------



## kdmorse (Jan 29, 2001)

I suspect someone kicked the environment, got them all back in sync, got 2 of the 3 reporting stratum 2, and decided to watch to see if they stay that way. And they did, for a while.

They all drifted back to stratum 6 (or 16/0) eventually, the major drifting started again, and sjr1 marched up to +8s before resetting. Now they're back to stratum 6/2/2, and mostly keeping sub 1s time.

sjr2 has occasionally been reporting stratum 3 since. This tells me a human changed something. I've not seen a stratum 3 response in the 3 months I've been watching.

But even so, it's clear they're doing minimal poking, and hoping that quick fixes make it all better. When in reality, if any of your servers are reporting stratum 4 or higher*, ever, you're not done fixing yet. They need to set up a cron script, once a minute, poll each server, and if any stratum is 6 or higher, email the admin...

(while writing this, they've lost their sync again, and are all back to stratum 6)

* excluding bizarre edge cases.


----------



## dlfl (Jul 6, 2006)

Here is what I just got:

server 17.254.0.31, stratum 1, offset 0.00177 (time3.apple.com)

server 204.176.49.10, stratum 2, offset 0.21881 (sjr1.tivo.com)

server 204.176.49.11, stratum 2, offset 0.19219 (sjr2.tivo.com)

server 204.176.49.12, stratum 16, offset 0.06090
ERROR: 13 Jan 21:43:54 ntpdate[5473]: no server suitable for synchronization found (sjr3.tivo.com)

I threw in the apple server, which indicates my local RPI clock is pretty close (This is usual).
Ironic that the tivo server at stratum 16 has the smallest offset. (Even a stopped clock is accurate once or twice a day )


----------



## m.s (Mar 8, 2007)

HerronScott said:


> Currently with 41 DC's in data centers and branch offices around the world, our greatest offset from our root is -0.0233637s


23 ms is very large in the ntp world. Do you have any metrics for the frequency domain?


----------



## sfhub (Jan 6, 2007)

It is silly this has gone on for so long. It is one thing if things didn't get reported to NOC because support didn't escalate, but this is after TiVo_Ted sent a note to them. Clearly TiVo NOC doesn't know how to configure NTP servers.

They've got some kludge in to get the offsets within 7 seconds again.

Since they didn't fix the core problem of drift, it will probably show up again whenever their kludge fails, but for now 99.9% of the people won't care if TiVo is within 7 seconds.


----------



## ggieseke (May 30, 2008)

sjr1 is way off again.


----------



## JoeKustra (Dec 7, 2012)

ggieseke said:


> sjr1 is way off again.


Almost 40 seconds.


----------



## tim1724 (Jul 3, 2007)

Over 51 seconds now.

sequoia:~ tim$ ntpdate -q sjr{1,2,3}.tivo.com time.apple.com
server 204.176.49.10, stratum 6, offset 51.154737, delay 0.04187
server 204.176.49.11, stratum 6, offset 2.413703, delay 0.04134
server 204.176.49.12, stratum 16, offset 0.195589, delay 0.04131
server 17.253.26.125, stratum 1, offset 0.000098, delay 0.02762
server 17.253.26.253, stratum 1, offset 0.000109, delay 0.02760
server 17.253.2.125, stratum 1, offset -0.000152, delay 0.05566
server 17.253.4.125, stratum 1, offset 0.000068, delay 0.03609
server 17.253.4.253, stratum 1, offset -0.000041, delay 0.03696
16 Jan 15:56:17 ntpdate[90631]: adjust time server 17.253.26.253 offset 0.000109 sec

(The 204.* entries are TiVo, 17.* is Apple.)


----------



## sharkster (Jul 3, 2004)

One of mine was almost a minute fast again this morning. Glad I caught it early. A re-connection seemed to fix it - for now. I'm getting to where I don't have the mental energy for some of this babysitting I have to do, now, with the Tivos.


----------



## tim1724 (Jul 3, 2007)

sjr1 is over a minute off now (69 seconds) … they clearly haven't fixed anything.


----------



## dlfl (Jul 6, 2006)

sjr1 at 88 sec now.


----------



## hapster85 (Sep 7, 2016)

I remember watching a recording, while I was off work over Christmas, that I thought the program had been delayed for whatever reason. Now I think it may have been this issue. Can't remember which recording now, though.


----------



## Mr Tony (Dec 20, 2012)

sharkster said:


> One of mine was almost a minute fast again this morning. Glad I caught it early. A re-connection seemed to fix it - for now. I'm getting to where I don't have the mental energy for some of this babysitting I have to do, now, with the Tivos.


agreed!


----------



## tivoknucklehead (Dec 12, 2002)

what is the fix for this nonsense?

tivo restart?
connect to service?


----------



## JoeKustra (Dec 7, 2012)

tivoknucklehead said:


> what is the fix for this nonsense?
> tivo restart?
> connect to service?


While you can do those things, it's faster to do a Network Diagnostic connection. A clock set is part of the test.


----------



## tivoknucklehead (Dec 12, 2002)

JoeKustra said:


> While you can do those things, it's faster to do a Network Diagnostic connection. A clock set is part of the test.


thanks, doing that got me within a second of an atomic clock I know is accurate


----------



## wish_bgr (Jul 19, 2014)

Off by 1min 30-ish seconds again...


----------



## dlfl (Jul 6, 2006)

I show sjr1 off 98+ secs now


----------



## NorthAlabama (Apr 19, 2012)

i just checked my clock (my new weeknight pre-primetime exercise), tivo was over 90 seconds fast, it took one quick test connection to resolve the issue.

my phones and computer are good, within 5 seconds of each other - something's wrong with this picture...

didn't @TiVo_Ted say this was being addressed? how long ago was his post? how old is this thread?


----------



## burdellgp (Mar 28, 2008)

NorthAlabama said:


> didn't @TiVo_Ted say this was being addressed? how long ago was his post? how old is this thread?


I think at this point, TiVo either doesn't care or is incapable of fixing this. If my boss had walked into my office and said "we need to fix this time thing", it would have taken at most a few days to a week or two to have it solved (and not just "worked around" where they seem to step the clock when they notice it is way off). It wouldn't even cost all that much money. My home stratum 1 NTP setup cost me $100 or less and probably has an accuracy of around ±5 µs (the GPS receiver I use says 1 µs, and I haven't spent a lot of effort beyond PPS on things like a temperature compensated clock crystal).

TiVo wouldn't need super high accuracy - any decent Linux/*BSD server running NTP against public servers on the Internet should be able to keep time within 10 ms and serve it out to a large number of clients. It should NOT be done in a virtual machine though (and I suspect that's what TiVo is doing that is causing the problem). The hardest thing to handle may be that they're doing time syncs by IP address, rather than DNS; if the TiVo OS has those IPs hard-coded, it may take a software update to fix this (because they may be using those IPs for other purposes that they can't/don't want to change). Never hard-code IP addresses!


----------



## dlfl (Jul 6, 2006)

NorthAlabama said:


> .......... didn't @TiVo_Ted say this was being addressed? how long ago was his post? how old is this thread?


Not to mention other threads including one started in 2015 when this same thing was happening.


----------



## dlfl (Jul 6, 2006)

sjr1 is off on a bender again -- 25 secs


----------



## Diana Collins (Aug 21, 2002)

tivoknucklehead said:


> what is the fix for this nonsense?
> 
> tivo restart?
> connect to service?


Extend all your recordings by 2 minutes.


----------



## ggieseke (May 30, 2008)

For what it's worth - my Roamios seem to update to the correct time shortly after I go to TiVo Central. If they're running a minute or two ahead (not uncommon lately) they always seem to reset to the correct time and I haven't missed the end of a recording yet. Just hit the TiVo button and wait a minute or two.

It's still a PITA to do that every day, and they really need to get their sh*t together on the sjr servers. NTP isn't even close to rocket science.


----------



## tim_m (Mar 8, 2017)

Diana Collins said:


> Extend all your recordings by 2 minutes.


Unfortunately that's not an option if you record a lot of shows. I tried that and created a ton of conflicts.


----------



## ClearToLand (Jul 10, 2001)

tivoknucklehead said:


> *what is the fix for this nonsense?*
> 
> tivo restart?
> connect to service?





Diana Collins said:


> *Extend all your recordings by 2 minutes.*





tim_m said:


> Unfortunately that's not an option if you record a lot of shows. I tried that and created a ton of conflicts.


It's important to notice the '_wink_' in @Diana Collins' answer. For the non-techie TiVo user, it is the ONLY available solution.

For the techie TiVo user, on the other hand, much has been written in this thread (and others) about what exactly is going on and why simply '*tivo restart?*' or '*connect to service?*' are invalid solutions.

In a nutshell:
There are 3 TiVo servers supplying time.
1 of those 3 has a tendency to '_drift_' up to 2 minutes before getting reset.
When you connect to the TiVo service, you have a 1 in 3 chance of getting that server.
If it is off in '_never never land_', you'll get the wrong time.
Running the suggested, i.e. Windows *w32tm /monitor*, command BEFORE connecting will show you your 'odds'.
NOTE: Make sure YOUR computer is correctly sync'd first. 
Since my return to TiVo in Sept 2015, I experienced the 'wrong time' problem once, just this past Friday. I noticed ABC World News began with ~90 seconds of Evening News. I didn't catch this until ~1959 EDT so I had to extend my 2000 show 2 minutes. Luckily, I got a good server on my manual connect and everything has been 'on-time' since.

The ULTIMATE solution is in TiVo's hands and, as we've all read, @TiVo_Ted is aware of it and says it was fixed:


TiVo_Ted said:


> *The NOC team believes they have fixed the issue*. I have also pointed them to this full thread. Is anyone still seeing any issues?


Since we know otherwise, many of us are waiting for his next response...


----------



## dlfl (Jul 6, 2006)

sjr1.tivo.com is the worst server but sjr2 has been known to wander badly also. For example:


GBL said:


> Interestingly enough, currently sjr2 is on a bender ... 51 seconds fast.
> 
> ```
> C:\>w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com
> ...


----------



## DBrunetti (Dec 6, 2016)

dlfl said:


> sjr1.tivo.com is the worst server but sjr2 has been known to wander badly also. For example:


And that is why I've been blocking sjr1 and sjr2 using my router.


----------



## GBL (Apr 20, 2000)

TiVoMargret said:


> Hi all,
> 
> First an apology -- I am REALLY sorry you experienced this issue. It both incredibly frustrating to miss parts of your shows, AND to feel that TiVo isn't being responsive to the issue.
> 
> ...


I do miss Margret.


----------



## dlfl (Jul 6, 2006)

Since this issue has been around since 2015, off and on, and TiVo has been made aware of it many times, I have to give TiVo the benefit of the doubt that if there were some easy way to fix it, they would just go ahead and do it. So the fix must be very difficult and/or very expensive, so they choose to just limp along without doing it. As usual, a public admission of such a shabby situation is very bad PR so they just don't provide the real story to us. Coverup and stonewall -- common CYA tactics of any person or organization, including corporations and government agencies. I guess we just keep complaining, hoping it will do some good.


----------



## BobCamp1 (May 15, 2002)

dlfl said:


> Since this issue has been around since 2015, off and on, and TiVo has been made aware of it many times, I have to give TiVo the benefit of the doubt that if there were some easy way to fix it, they would just go ahead and do it. So the fix must be very difficult and/or very expensive, so they choose to just limp along without doing it. As usual, a public admission of such a shabby situation is very bad PR so they just don't provide the real story to us. Coverup and stonewall -- common CYA tactics of any person or organization, including corporations and government agencies. I guess we just keep complaining, hoping it will do some good.


It's not difficult to fix with the proper hardware. A 16-year old can do it. I do it at work all the time.


----------



## dlfl (Jul 6, 2006)

BobCamp1 said:


> It's not difficult to fix with the proper hardware. A 16-year old can do it. I do it at work all the time.


And the cost of "proper hardware" ??


----------



## GBL (Apr 20, 2000)

in the last 24 hours, I haven't seen any drifts over 4 seconds on my random checks. Is it possible we nagged TiVo enough to fix it? Personally, a drift of up to 5 seconds fast is acceptable to me.



dlfl said:


> Since this issue has been around since 2015, off and on, and TiVo has been made aware of it many times, I have to give TiVo the benefit of the doubt that if there were some easy way to fix it, they would just go ahead and do it. So the fix must be very difficult and/or very expensive, so they choose to just limp along without doing it. As usual, a public admission of such a shabby situation is very bad PR so they just don't provide the real story to us. Coverup and stonewall -- common CYA tactics of any person or organization, including corporations and government agencies. I guess we just keep complaining, hoping it will do some good.


I don't think a fix is very difficult nor expensive for TiVo; I do believe they are in cost cutting mode and that's where their focus is.

TiVo apparently tolerates some clock variances[1] that affects a relatively small number of customers[2] - what's an acceptable clock variance for a DVR varies from customer to customer, as long as the clock isn't late and the beginning of a show is recorded most customers won't mind missing the last 30 seconds or so, usually commercials anyways (I know, I won't since I don't usually watch previews).

[1] Up to 2 minutes fast, apparently. 
[2] As long as it doesn't create a lot of support calls about it (to which they respond with canned answers like this un-helpful one I received


> I see you're having issues with recording programs since the server time is not sync in, please follow the steps below:
> 
> Force a service connection on the TiVo by going to *Settings & Messages > Network Settings > TiVo Service Connection*.
> Repeat the Guided Setup by going to *Settings & Messages > Help > Reset to Defaults > Repeat Guided Setup*.


----------



## m.s (Mar 8, 2007)

dlfl said:


> And the cost of "proper hardware" ??


Not much really. You can buy an off-the-shelf stratum 1 ntp server for $300 (see "cheap stratum 1", below). Unlikely to be able to handle the load of all the TiVos in the world, though. But, maybe they would - each box only polls a couple of times a day.

A robust turnkey stratum 1 server for <$10K (server itself, plus the required antenna and cabling) to receive time. Times 3, so maybe $30K.

But, TiVo doesn't need to be a stratum 1 source. A few physical Linux boxes (not VMs), sync'd to the ntp pool would suffice. Under $5K total (they don't need to be big Xeon powerhouses). Or, so they're not dependent on external servers, sync'd to an internal pool of cheap stratum 1s. Then, maybe $10K (proper installation would require better outdoor antennas and cabling, etc.), total.

Even the "expensive," $30K solution is cheap for a mission critical, enterprise solution.


----------



## tim1724 (Jul 3, 2007)

Here's one cheap and easy solution: They could get a vendor pool from ntp.org and not even need to run their own servers.

But really, replacing their current three servers with working NTP servers wouldn't cost much either. I'd buy five cheap machines (say $1000 each, but any five random old desktops from their office would most likely work fine) and two GPS receivers (about $100 each). I'd put Linux on the five computers and set up ntpd. I'd attach the GPS receivers to two of them, making them stratum 1 servers. Then I'd configure the other three to get time from the two stratum 1 servers (making them stratum 2) and make those the new sjr{1,2,3}.tivo.com &#8230; it would take one afternoon and a total of $5200 (plus sales tax, so call it $5700).

Or they could save a few bucks and just set up three servers, all with GPS receivers, and have three stratum 1 servers. ~$3300 plus tax.

But in any case, cost is certainly not the issue.


----------



## m.s (Mar 8, 2007)

GBL said:


> in the last 24 hours, I haven't seen any drifts over 4 seconds on my random checks. Is it possible we nagged TiVo enough to fix it? Personally, a drift of up to 5 seconds fast is acceptable to me.


That may be sufficient for the needs of a TiVo scheduling recordings, but 5 seconds off on an ntp server means something is still way not right, which means it's not to be trusted. Reasonable errors are measured in milliseconds. 100 milliseconds would be poor.


----------



## kdmorse (Jan 29, 2001)

It hasn't been on a major drinking binge since Sunday. But it has been up to +8 seconds today (5:43 PM EST) at it's worst.


----------



## dlfl (Jul 6, 2006)

I can believe that TiVo just doesn't consider this a big enough issue to spend thousands of dollars, plus the IT labor hours, fixing. To we few posting here it may seem like a huge issue, but we are only a few. I wonder if the MSO-supplied boxes use different time setting servers?


----------



## dlfl (Jul 6, 2006)

Looks like TiVo IT has bigger things on their plate -- check out the SNAFU being discussed on this thread:
TiVo.com Requiring Password Reset?


----------



## GBL (Apr 20, 2000)

Issue seems back; sjr1.tivo.com started to seriously drift again:


```
C:\>w32tm /stripchart /computer:204.176.49.10 /dataonly /samples:5
Tracking 204.176.49.10 [204.176.49.10:123].
Collecting 5 samples.
The current time is 1/24/2018 1:57:21 PM.
13:57:21, +16.5888564s
13:57:23, +16.5950555s
13:57:25, +16.6055395s
13:57:28, +16.6095739s
13:57:30, +16.6159664s

C:\>w32tm /stripchart /computer:204.176.49.10 /dataonly /samples:5
Tracking 204.176.49.10 [204.176.49.10:123].
Collecting 5 samples.
The current time is 1/24/2018 2:22:11 PM.
14:22:11, +20.0082241s
14:22:13, +20.0202901s
14:22:15, +20.0327721s
14:22:17, +20.0367169s
14:22:19, +20.0409065s
```


----------



## m.s (Mar 8, 2007)

dlfl said:


> To we few posting here it may seem like a huge issue, but we are only a few.


Yeah, and everyone who doesn't know the why is cursing their TiVo for cutting off the gripping ending of episodes two minutes early, on an irregular basis.


----------



## sfhub (Jan 6, 2007)

kdmorse said:


> It hasn't been on a major drinking binge since Sunday. But it has been up to +8 seconds today (5:43 PM EST) at it's worst.


Using that analogy, the servers are always drinking. Usually every 7-8 seconds they make it to the toilet to purge themselves, but for some unknown reason, sometimes the toilet is occupied for up to 2 minutes.


----------



## NorthAlabama (Apr 19, 2012)

anything less than 30sec doesn't seem to affect my recordings, and, honestly, that's my concern. i can see most people ignoring the worst case scenario if it only happened randomly and rarely, say once every few weeks, and never taking the time to call (or research). 

while i've only been burned once recently, since i began checking every weeknight and have caught a few serious offsets of 90sec+, it's now difficult for me to believe more customers aren't experiencing trouble with their recordings.


----------



## slowbiscuit (Sep 19, 2006)

dlfl said:


> I can believe that TiVo just doesn't consider this a big enough issue to spend thousands of dollars, plus the IT labor hours, fixing. To we few posting here it may seem like a huge issue, but we are only a few. I wonder if the MSO-supplied boxes use different time setting servers?


We're all paying for this service and being let down after Tivo has repeatedly been told it's in error, same as for guide data. Another dispute-ready complaint IMO, go get some free Slide Pro remotes when they settle with you for the trouble.


----------



## TiVo_Ted (Oct 3, 2000)

I worked with our network operations team earlier this week to fix this issue. They are designing a permanent fix to get past this problem. Meanwhile, they have created an automated check that warns them if the clocks drift by more than 5 seconds. At that point, they can manually restart the offending server. It's a hack while they work on a permanent fix, but it should be much better now.


----------



## NorthAlabama (Apr 19, 2012)

thanks for the update @TiVo_Ted, your follow-up post on the status is _*greatly*_ appreciated!


----------



## m.s (Mar 8, 2007)

TiVo_Ted said:


> Meanwhile, they have created an automated check that warns them if the clocks drift by more than 5 seconds. At that point, they can manually restart the offending server.


Maybe they can, but they don't.


----------



## JoeKustra (Dec 7, 2012)

m.s said:


> Maybe they can, but they don't.


Sounds like time to hire more hamsters.


----------



## kdmorse (Jan 29, 2001)

m.s said:


> Maybe they can, but they don't.


I'm willing to accept a definition of 'earlier this week' that just proceeds Ted's post here.

Any drift since since 4:12PM EST today however, is 100% fair game. (So far sjr1 has only stepped out of line by 3s since then)


----------



## NorthAlabama (Apr 19, 2012)

today's scheduled connection for my tivo was at 3pm central, the clock is the most accurate it's been in weeks (within 3 seconds).


----------



## m.s (Mar 8, 2007)

kdmorse said:


> I'm willing to accept a definition of 'earlier this week' that just proceeds Ted's post here.


If, on a Thursday, "earlier this week" means less than "yesterday", it's deliberately misleading. Especially when a counterexample preceded the post. This has been an ongoing, long term issue. TiVo doesn't get the benefit of the doubt in this matter.


----------



## kdmorse (Jan 29, 2001)

sjr1 has started going for a walk, let's see how far it gets....



Spoiler: Walkies



25 Jan 20:56:02 server 204.176.49.10 offset 1.033317 sec
25 Jan 20:57:02 server 204.176.49.10 offset 1.119248 sec
25 Jan 20:58:02 server 204.176.49.10 offset 1.186080 sec
25 Jan 20:59:02 server 204.176.49.10 offset 1.252553 sec
25 Jan 21:00:02 server 204.176.49.10 offset 1.306985 sec
25 Jan 21:01:02 server 204.176.49.10 offset 1.440197 sec
25 Jan 21:02:02 server 204.176.49.10 offset 1.572463 sec
25 Jan 21:03:01 server 204.176.49.10 offset 1.686438 sec
25 Jan 21:04:02 server 204.176.49.10 offset 1.785216 sec
25 Jan 21:05:01 server 204.176.49.10 offset 2.028144 sec
25 Jan 21:06:02 server 204.176.49.10 offset 2.206398 sec
25 Jan 21:07:02 server 204.176.49.10 offset 2.389540 sec
25 Jan 21:08:02 server 204.176.49.10 offset 2.469042 sec
25 Jan 21:09:02 server 204.176.49.10 offset 2.599691 sec
25 Jan 21:10:02 server 204.176.49.10 offset 2.753046 sec
25 Jan 21:11:02 server 204.176.49.10 offset 2.878543 sec
25 Jan 21:12:02 server 204.176.49.10 offset 2.997713 sec
25 Jan 21:13:02 server 204.176.49.10 offset 3.101249 sec
25 Jan 21:14:01 server 204.176.49.10 offset 3.221968 sec


----------



## tim1724 (Jul 3, 2007)

At the rate sjr1 is drifting it must be hitting that 5 second offset (and thus being "manually reset") every 45 minutes or so. I'd hate to have that job. (I really hope it's scripted rather than being done manually!)


----------



## sfhub (Jan 6, 2007)

tim1724 said:


> At the rate sjr1 is drifting it must be hitting that 5 second offset (and thus being "manually reset") every 45 minutes or so. I'd hate to have that job. (I really hope it's scripted rather than being done manually!)


It's been outsourced
https://www.amazon.com/Century-Heavy-Digital-Programmable-Timer/dp/B00MVF16JG


----------



## tim1724 (Jul 3, 2007)

I caught the restart in action, at about 7:40 PM PST. 

25 Jan 19:34:28 server 204.176.49.10, stratum 6, offset 3.596011, delay 0.04158
25 Jan 19:35:28 server 204.176.49.10, stratum 6, offset 3.737472, delay 0.04152
25 Jan 19:36:28 server 204.176.49.10, stratum 6, offset 3.964614, delay 0.04152
25 Jan 19:37:28 server 204.176.49.10, stratum 6, offset 4.089900, delay 0.04143
25 Jan 19:38:29 server 204.176.49.10, stratum 6, offset 4.247282, delay 0.04149
25 Jan 19:39:29 server 204.176.49.10, stratum 6, offset 4.424488, delay 0.04144
25 Jan 19:40:29 server 204.176.49.10, stratum 16, offset 0.563644, delay 0.04150
25 Jan 19:41:29 server 204.176.49.10, stratum 16, offset 0.757802, delay 0.04166
25 Jan 19:42:29 server 204.176.49.10, stratum 16, offset 0.951235, delay 0.04153
25 Jan 19:43:29 server 204.176.49.10, stratum 6, offset 1.141534, delay 0.04137
25 Jan 19:44:29 server 204.176.49.10, stratum 6, offset 1.296905, delay 0.04153​


----------



## GBL (Apr 20, 2000)

Someone's not paying attention at NOC:


```
C:\>w32tm /stripchart /computer:204.176.49.10 /dataonly /samples:5
Tracking 204.176.49.10 [204.176.49.10:123].
Collecting 5 samples.
The current time is 1/27/2018 9:17:01 AM.
09:17:01, +37.1068281s
09:17:03, +37.1109374s
09:17:05, +37.1211143s
09:17:07, +37.1179970s
09:17:10, +37.1206011s
C:\>w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com,time3.apple.com
sjr1.tivo.com[204.176.49.10:123]:
    ICMP: 81ms delay
    NTP: +37.1720157s offset from local clock
        RefID: (unknown) [0x00017F7F]
        Stratum: 6
sjr2.tivo.com[204.176.49.11:123]:
    ICMP: 84ms delay
    NTP: +1.8006677s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 6
sjr3.tivo.com[204.176.49.12:123]:
    ICMP: 81ms delay
    NTP: +0.6857078s offset from local clock
        RefID: clock.isc.org [204.152.184.72]
        Stratum: 6
time3.apple.com[17.254.0.31:123]:
    ICMP: 86ms delay
    NTP: +0.0027957s offset from local clock
        RefID: 'GPSs' [0x73535047]
        Stratum: 1
```


----------



## dlfl (Jul 6, 2006)

GBL said:


> Someone's not paying attention at NOC:


Yep, my results:

server 17.254.0.31, stratum 1, offset -0.00073 (time3.apple.com)

server 204.176.49.10, stratum 6, *offset* *38.21283 (sjr1.tivo.com)*

server 204.176.49.11, stratum 16, offset 0.42475
ERROR: 27 Jan 09:28:49 ntpdate[27334]: no server suitable for synchronization found (sjr2.tivo.com)

server 204.176.49.12, stratum 6, offset 1.42967 (sjr3.tivo.com)

And sjr2 is stratum 16, not suitable for synch.


----------



## PSU_Sudzi (Jun 4, 2015)

My TiVo clock had been good for a while (or I didn’t notice an offset) but just saw now it’s off by 2 minutes again.


----------



## NorthAlabama (Apr 19, 2012)

yes, sjr1 is now off by 75sec - my tivo is good, off by 2sec last night, now 3sec off after the overnight connection.


----------



## slowbiscuit (Sep 19, 2006)

LOL @TiVo_Ted, better follow up with the NOC again. And again. And again. Your new full-time job.

I repeat what I said earlier, this qualifies as a dispute-ready case with Tivo. Get some free stuff from them (monthly service credit, free Slides etc.) because they're screwing up recordings. They are not reliably delivering the service that we're paying for.


----------



## kdmorse (Jan 29, 2001)

Yup, that hypothetical alert should have gone off 12 hours ago.

27 Jan 04:42:02 time server 204.176.49.10 offset 5.217609 sec
27 Jan 04:43:02 time server 204.176.49.10 offset 5.462463 sec
27 Jan 04:44:02 time server 204.176.49.10 offset 5.707131 sec
27 Jan 04:45:02 time server 204.176.49.10 offset 5.891790 sec

.. snip ..*
*
27 Jan 14:30:02 time server 204.176.49.10 offset 94.045662 sec
27 Jan 14:31:02 time server 204.176.49.10 offset 94.146963 sec
27 Jan 14:32:02 time server 204.176.49.10 offset 94.269682 sec
27 Jan 14:33:01 time server 204.176.49.10 offset 94.348995 sec


----------



## NorthAlabama (Apr 19, 2012)

yeah, it's turned out to be quite a long stroll...


----------



## moyekj (Jan 24, 2006)

I blame kdmorse for querying sjr1 every 1 second and overloading it.


----------



## GBL (Apr 20, 2000)

We're over 100 seconds fast now, maybe it'll reset at 120?

```
C:\>w32tm /stripchart /computer:204.176.49.10 /dataonly /samples:5
Tracking 204.176.49.10 [204.176.49.10:123].
Collecting 5 samples.
The current time is 1/27/2018 3:34:48 PM.
15:34:48, +101.2646055s
15:34:50, +101.2701492s
15:34:52, +101.2904823s
15:34:55, +101.2964117s
15:34:57, +101.3027842s
```


----------



## NorthAlabama (Apr 19, 2012)

i'm all for sleeping in on a saturday, but this is a little extreme, even for me...did someone leave the cage door open last night?


----------



## kdmorse (Jan 29, 2001)

GBL said:


> We're over 100 seconds fast now, maybe it'll reset at 120?


That's one of it's most reliable traits. Once it passes 120, it goes offline. Then it comes back correct.

Ironically, you could set a clock by it...


----------



## burdellgp (Mar 28, 2008)

Up to almost 113 seconds, and the others are off by about 1.5 seconds. There's no excuse for an NTP server to ever be off by more than a few milliseconds. Stratum 6 is weird - random people on the Internet joining the free NTP server pools are typically at most stratum 3 - I've never seen a stratum 6 server in the wild.

$ ntpdate -qv sjr{1..3}.tivo.com
27 Jan 16:14:11 ntpdate[31735]: ntpdate [email protected] Thu Oct 26 10:57:33 UTC 2017 (1)
server 204.176.49.10, stratum 6, offset 112.828612, delay 0.09798
server 204.176.49.11, stratum 6, offset 1.469742, delay 0.09848
server 204.176.49.12, stratum 6, offset 1.537849, delay 0.09831


----------



## sharkster (Jul 3, 2004)

After the connection for today mine is 5 seconds behind. PERFECT! This is how I would set it if I had my druthers. 

Ok, don't worry - I'm not too excited because I do get that tomorrow will be another day. But it's fun for now.


----------



## kdmorse (Jan 29, 2001)

burdellgp said:


> Stratum 6 is weird - random people on the Internet joining the free NTP server pools are typically at most stratum 3 - I've never seen a stratum 6 server in the wild.


Stratum 5 is either the default for Undisciplined Local Clock, or Orphan mode, depending on how you have things configured. I'm sure their server contains the following statements (if ntpd - If it's something else, I'm sure the same theory applies)

server 127.127.1.0
fudge 127.127.1.0 stratum 5

The reasoning being, that if a NTP server completely loses connectivity to all servers, it will stop responding to clients by default. The above two lines tell it, (oversimplified), to talk to itself and consider itself stratum 5. Anyone that gets time from it would then be considered stratum 6. And that is clearly in play here.

*This configuration is never to be used on the internet. *

Why would you ever want to do such a thing? Think of a internal corporate network, especially back in the days before persistent internet access. You set up a NTP server, that server talks to the internet. This NTP server is pointed at stratum 2 servers, and is normally stratum 3. All your internal machines talk to that NTP server. If that NTP server loses it's connection to other time servers (external link goes down) - you have two choices.

1. The NTP server could stop responding to clients. It has lost sync, is floating on it's own undisciplined local clock, and is not configured to respond. Clients are on their own.

2. The NTP server could respond to clients, indicate it's a lower quality (stratum 5). It doesn't promise to be accurate, *but*, it will keep all the systems on the internal network set the same, and often that's important. It's the best it can do until connectivity is restored.

On an internet facing ntp server, #1 is clearly the right option. If you've lost faith in yourself, don't tell potential lies.

If you're internal, option #2 is often correct. Keep everyone internal synced together, and if there's gonna be drift, let's drift them all together so everyone's logs still line up.

The other option is they have deliberately configured orphan mode, but that would imply they deliberately made this mess, which is less likely than picking up bad defaults from a old ntpd.conf example without understanding it.

But all of this falls into the arena of 'how does sjr1 cope with being unable to sync with another ntp server'. Which is secondary to the question of 'why the hell is it not syncing with another ntp server'. They're little UDP packets, they're not that hard to move around. Hell, I'm sure they could do better just implementing rfc1149.

I'm sure at the end we'll discover that sjr1/2/3 are actually Tivo's (I mean, literally, Series 2s in a closet). And they only call home to sync their clocks once a day, or when someone pushes a button while watching TV in the office.


----------



## NorthAlabama (Apr 19, 2012)

and...we're back.


----------



## JoeKustra (Dec 7, 2012)

NorthAlabama said:


> and...we're back.


But moving in reverse quickly. I checked three hours ago and the offsets were under one second.


----------



## kdmorse (Jan 29, 2001)

I'm shocked, shocked I tell you, to see that sjr1 is drifting...

28 Jan 13:51:02 time server 204.176.49.10 offset 1.655606 sec
28 Jan 13:52:02 time server 204.176.49.10 offset 2.002196 sec
28 Jan 13:56:02 time server 204.176.49.10 offset 3.021568 sec
28 Jan 14:00:02 time server 204.176.49.10 offset 4.409867 sec
28 Jan 14:03:02 time server 204.176.49.10 offset 5.297646 sec
28 Jan 14:07:02 time server 204.176.49.10 offset 6.160549 sec
28 Jan 14:10:02 time server 204.176.49.10 offset 7.139999 sec
28 Jan 14:16:01 time server 204.176.49.10 offset 8.047975 sec
28 Jan 14:24:02 time server 204.176.49.10 offset 9.058988 sec
28 Jan 14:30:02 time server 204.176.49.10 offset 10.085154 sec
28 Jan 14:39:02 time server 204.176.49.10 offset 11.057472 sec
28 Jan 14:46:02 time server 204.176.49.10 offset 12.063765 sec
28 Jan 14:53:01 time server 204.176.49.10 offset 13.065876 sec


----------



## NorthAlabama (Apr 19, 2012)

yeah, it's been drifting all weekend. the weirdest part is my sat & sun connections are both overnight, and my tivo clock has remained within 2-4sec, so i'm wondering how i've avoided hitting sjr1.


----------



## sakaike (Jan 22, 2002)

Just out of curiosity, for those folks who are apparently very knowledgeable on this topic, is there a reason why the drift is always in the faster direction? Just wondering...


----------



## JoeKustra (Dec 7, 2012)

Curious. Just forced a service connection on two Roamio boxes. One Hydra. Both failed the first time with a C125 error. I hit select and it showed that ports 123 and 37 were blocked. As if. That happened during the clock set. I went to network troubleshooting, saw both passive tests pass and ran a connection diagnostic. No errors. Went back and reran the service connection and it passed. Perhaps someone is working on the time servers?

I also applied power to my Mini VOX but forgot to connect the internet. Time was off by 8 hours. Only three days without power.


----------



## burdellgp (Mar 28, 2008)

sakaike said:


> Just out of curiosity, for those folks who are apparently very knowledgeable on this topic, is there a reason why the drift is always in the faster direction? Just wondering...


Only way to know for sure is to know why they are not keeping in sync to begin with. In the past, certain combinations of virtualization servers and guest OSes would cause the clock to tick too often in VMs, so they'd always run fast. I don't know that that's the issue with TiVo's NTP servers (since I thought all such combinations were past end of life at this point). I can't say I've tried to run NTP servers for a large number of clients, but I can't think how I'd set up an NTP server that would act bad the same as TiVo's servers.

Right now, all three server are completely unsynced (stratum 16):

$ ntpdate -qv sjr{1..3}.tivo.com
2 Feb 18:08:28 ntpdate[16728]: ntpdate [email protected] Thu Oct 26 10:57:33 UTC 2017 (1)
server 204.176.49.10, stratum 16, offset 1.051599, delay 0.09825
server 204.176.49.11, stratum 16, offset 0.297541, delay 0.09828
server 204.176.49.12, stratum 16, offset 0.412517, delay 0.09866


----------



## morac (Mar 14, 2003)

I found this thread by accident as I haven't really noticed a problem on my boxes, but figured I'd try and block sjr1 on my ASUS router running Merlin, but apparently there is a bug in the network filtering service code which causes blacklisting to not work. I could manually add an iptable entry to block sjr1, but that's kind of a pain to do, so I just left it alone as I have never noticed a problem. Maybe a second or so of. I usually pad everything a minute which works unless I hit the long time padding bug where it doesn't pad back to back recordings.

My boxes tend to make connections early in the morning (EST) or early afternoon (like 1:34 pm today). I wonder if that's why. Hopefully this issue gets fixed though so I don't have to bother. For what it's worth sjr1 is _only_ 2.5 seconds off.

Is there any reason Tivo runs it's own ntp servers? Also hard coded ip addresses is a bad idea.


----------



## kdmorse (Jan 29, 2001)

morac said:


> My boxes tend to make connections early in the morning (EST) or early afternoon (like 1:34 pm today). I wonder if that's why. Hopefully this issue gets fixed though so I don't have to bother. For what it's worth sjr1 is _only_ 2.5 seconds off.


On the one hand, the band-aid measures they put in place to keep it from drifting beyond +5 seconds fast, have been working since Monday the 29th. So units shouldn't be ending up wildly off like they were.

On the other hard, the root problem clearly still remains. sjr1 still likes to march inexorably upwards, until something thumps it. Deviations beyond 1 second are the norm, as is responses of stratum 6 or 16. And all three of them continue to be dreadful ntp servers.



> Is there any reason Tivo runs it's own ntp servers?


I'm sure it made sense at the time. You release a product that is absolutely reliant on having a decent clock. It has no internet access, only having any sort of network access during a short daily call using a modem. Open NTP servers are not plentiful, the few that exist frequently change names, ip addresses, or access policies - are rate limited, frequently block abusive users, and specifically say something like "don't point all your users at me, you run a ntp server, you talk to me, point all your users at you". Pointing at other ntp servers also has a trust risk, as you trust them to keep your entire product base sane, and bear the brunt if something outside of your control goes wrong. (One popular ntp server decided it didn't want to foot the bill for the bandwith of serving everyone who didn't read it's terms of service, and just reset it's date to 1980 or somesuch one time). You only get one shot a day, so you have to talk to something you can trust. Finding a reliable open NTP server in 1999 was not an easy task.

So it made perfect sense to:

1. Run your own NTP servers
2. Care for them yourself, ensure that they're accurate.
3. Point your user base at them.

Somewhere along the line, they kinda lost track of step #2...

Now, the world has changed a lot in the past 18 years. But a lot of the daily call methodology hasn't, it still runs along pretty much the same script it has for the last 18 years, with a few changes here and there, but no major architecture changes...

And the basic idea is still a decent one. Assuming, you know, you actually "run" your ntp servers, not just shove them in the corner and neglect them for years.



> Also hard coded ip addresses is a bad idea.


I'm sure this too has legacy reasons. In order to keep it light, the original Tivo linux kernel had every function they didn't need ripped out - and this included name resolution. (This is the reason you had to put a -n on almost all commands, as any attempt to resolve a name would segfault. You had to netstat -r -n, because just netstat -r would crash). So *everything* the box did was by ip address. Much of that has changed over time, but since there was no reason to touch the clock set code, nobody did...

I'm not saying any of these are *good* reasons. But they're plausible legacy reasons. Were you designing a box today, you'd do things far differently across the board. The vast majority of the Daily Call process is exceedingly archaic by todays standards.


----------



## JoeKustra (Dec 7, 2012)

morac said:


> My boxes tend to make connections early in the morning (EST) or early afternoon (like 1:34 pm today). I wonder if that's why.


How guide updates are scheduled ->Daily Guide Updates


----------



## slowbiscuit (Sep 19, 2006)

morac said:


> I found this thread by accident as I haven't really noticed a problem on my boxes, but figured I'd try and block sjr1 on my ASUS router running Merlin, but apparently there is a bug in the network filtering service code which causes blacklisting to not work.


Yeah I posted this earlier, all you have to do is update Merlin to latest rev which fixes the filter bug. I have blocked both sjr1 and 2 in my router w/Merlin.


----------



## morac (Mar 14, 2003)

slowbiscuit said:


> Yeah I posted this earlier, all you have to do is update Merlin to latest rev which fixes the filter bug. I have blocked both sjr1 and 2 in my router w/Merlin.


I'm running 382.1_2, which is the latest non-Beta, and it's definitely not working on my AC88U. I confirmed this by checking the iptables and seeing that the rules being added by filtering are using a "RETURN" rule, rather than a "DROP" rule, which results in the connections being allowed.

I dug into the code base and found what I believe the offending code, but so far Merlin is ignoring my bug reports. Blacklisting via Network Services Filter isn't working

My only work around is to set up my own rule using scripting which is more trouble than it's worth at this point.


----------



## DBrunetti (Dec 6, 2016)

Having trouble completing a service connection. Fails on setting clock. First instance was during the night.


----------



## Rob Helmerichs (Oct 17, 2000)

Mine insists that it can't get through my firewall, even though I haven't changed anything.


----------



## DBrunetti (Dec 6, 2016)

I’m seeing the same message.


----------



## JoeKustra (Dec 7, 2012)

Oh thank God. I've just power cycled my entire network and it still dies on getting time. So it's not me or my router. I was just about to call my ISP.

BTW, I checked my event log which shows a good connection on 52:179:17:38:123 when I force a windows time update.

The Port test passes but the network diagnostics fail with a C125.


----------



## NorthAlabama (Apr 19, 2012)

my tivo successfully connected this morning about 8am (4 hours later than usual), tcp port and dns tests report "succeeded", but i'm not testing my connection for fear of a failure!


----------



## JoeKustra (Dec 7, 2012)

NorthAlabama said:


> my tivo successfully connected this morning about 8am (4 hours later than usual), tcp port and dns tests report "succeeded", but i'm not testing my connection for fear of a failure!


It won't hurt. Just run the Network Diagnostics. No need to do a Service Connection to get the error.

I think it's fixed.


----------



## NorthAlabama (Apr 19, 2012)

it appears so, test connection completed, no errors.


----------



## Rob Helmerichs (Oct 17, 2000)

I just successfully connected (and got another day's data).


----------



## DBrunetti (Dec 6, 2016)

Yes. Seems to be working again. Tried on Roamio and Mini. Both connected successfully.


----------



## slowbiscuit (Sep 19, 2006)

morac said:


> I'm running 382.1_2, which is the latest non-Beta, and it's definitely not working on my AC88U. I confirmed this by checking the iptables and seeing that the rules being added by filtering are using a "RETURN" rule, rather than a "DROP" rule, which results in the connections being allowed.
> 
> I dug into the code base and found what I believe the offending code, but so far Merlin is ignoring my bug reports. Blacklisting via Network Services Filter isn't working
> 
> My only work around is to set up my own rule using scripting which is more trouble than it's worth at this point.


*shrug* working fine on my RT-AC86U w/380.69. Confirmed by running w32tm to check NTP blocking per this thread. It did not work until I updated to this rev.


----------



## ClearToLand (Jul 10, 2001)

*Stratum 2* is something new. sjr1 *WAS* also Stratum 2 a few minutes ago and just changed to 'local clock':

```
C:\!Util\BAT>w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com

sjr1.tivo.com[204.176.49.10:123]:
    ICMP: 68ms delay
    NTP: +0.1731436s offset from local clock
        RefID: 'STEP' [0x50455453]
        Stratum: 0
sjr2.tivo.com[204.176.49.11:123]:
    ICMP: 67ms delay
    NTP: +0.0530219s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 2
sjr3.tivo.com[204.176.49.12:123]:
    ICMP: 68ms delay
    NTP: +0.1251091s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 2
```


----------



## PSU_Sudzi (Jun 4, 2015)

JoeKustra said:


> Oh thank God. I've just power cycled my entire network and it still dies on getting time. So it's not me or my router. I was just about to call my ISP.
> 
> BTW, I checked my event log which shows a good connection on 52:179:17:38:123 when I force a windows time update.
> 
> The Port test passes but the network diagnostics fail with a C125.


I've noticed a couple of failed connections when manually checking at lunch time over the past 3 or 4 days too.


----------



## ClearToLand (Jul 10, 2001)

DBrunetti said:


> Having trouble completing a service connection. Fails on setting clock. First instance was during the night.





Rob Helmerichs said:


> Mine insists that it can't get through my firewall, even though I haven't changed anything.


Maybe TiVo *finally* installed new hardware this weekend...


----------



## GBL (Apr 20, 2000)

Looks like we are finally seeing some better precision (and stratum 2 servers):

```
C:\>w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com,time3.apple.com
sjr1.tivo.com[204.176.49.10:123]:
    ICMP: 81ms delay
    NTP: +0.2183282s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 2
sjr2.tivo.com[204.176.49.11:123]:
    ICMP: 81ms delay
    NTP: +0.0304309s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 2
sjr3.tivo.com[204.176.49.12:123]:
    ICMP: 81ms delay
    NTP: +0.0170769s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 2
time3.apple.com[17.254.0.31:123]:
    ICMP: 84ms delay
    NTP: -0.0128145s offset from local clock
        RefID: 'GPSs' [0x73535047]
        Stratum: 1
```


----------



## dlfl (Jul 6, 2006)

GBL said:


> Looks like we are finally seeing some better precision (and stratum 2 servers):
> .............


I'm seeing stratum 2 on sjr1 and sjr2 but stratum 16 on sjr3, although the offset is only about 0.4 secs.

Question: I currently have sjr1 and sjr2 blocked in my router. If my Tivo tries to set time with only a stratum 16 server to connect to, will it succeed?


----------



## GBL (Apr 20, 2000)

dlfl said:


> I'm seeing stratum 2 on sjr1 and sjr2 but stratum 16 on sjr3, although the offset is only about 0.4 secs.
> 
> Question: I currently have sjr1 and sjr2 blocked in my router. If my Tivo tries to set time with only a stratum 16 server to connect to, will it succeed?


Currently sjr3 presents as stratum 0 to w32tm.

I'm not sure which algorithm ntpdate uses to pick one of the 3 NTP servers to set the time; if you look at your tivo's tcclient log after a test connection, you'll see which server it used or it may report no time service available.


```
C:\>w32tm /monitor /computers:sjr1.tivo.com,sjr2.tivo.com,sjr3.tivo.com,time3.apple.com
sjr1.tivo.com[204.176.49.10:123]:
    ICMP: 85ms delay
    NTP: -0.0185667s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 2
sjr2.tivo.com[204.176.49.11:123]:
    ICMP: 85ms delay
    NTP: +0.0610557s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 2
sjr3.tivo.com[204.176.49.12:123]:
    ICMP: 106ms delay
    NTP: +0.1334024s offset from local clock
        RefID: 'þ [0x1F00FE11]
        Stratum: 0
time3.apple.com[17.254.0.31:123]:
    ICMP: 84ms delay
    NTP: -0.0075909s offset from local clock
        RefID: 'GPSs' [0x73535047]
        Stratum: 1
```


----------



## m.s (Mar 8, 2007)

GBL said:


> Currently sjr3 presents as stratum 0 to w32tm.


Which is just plain wrong. There is no "stratum 0" ntp server. Stratum 0 is the (non-ntp) device a stratum 1 server syncs to, like an atomic clock, GPS, etc. When a server says it's stratum 0, it means "unspecified or invalid". The RefID is weird, too. It translates to - 31.0.254.17, which is an IP in Poland. It's the octets of time3.apple.com, reversed.


----------



## morac (Mar 14, 2003)

m.s said:


> Which is just plain wrong. There is no "stratum 0" ntp server. Stratum 0 is the (non-ntp) device a stratum 1 server syncs to, like an atomic clock, GPS, etc. When a server says it's stratum 0, it means "unspecified or invalid". The RefID is weird, too. It translates to - 31.0.254.17, which is an IP in Poland. It's the octets of time3.apple.com, reversed.


Both sjr1 and sjr2 are now showing as stratum 0:


```
sjr1.tivo.com[204.176.49.10:123]:
    ICMP: 85ms delay
    NTP: +0.2229983s offset from local clock
        RefID: '◄■ [0x1F00FE11]
        Stratum: 0
sjr2.tivo.com[204.176.49.11:123]:
    ICMP: 82ms delay
    NTP: +0.0865238s offset from local clock
        RefID: 'STEP' [0x50455453]
        Stratum: 0
sjr3.tivo.com[204.176.49.12:123]:
    ICMP: 85ms delay
    NTP: -0.0205171s offset from local clock
        RefID: time3.apple.com [17.254.0.31]
        Stratum: 2
```


----------



## clay.autery (Feb 3, 2018)

OK.... so their NTP servers appear to be behaving right now, but I am unwilling to rely on them.

Guess I have to go figure out how to route their NTP Server's Public IPs to my local NTP server on its Private IP.
First time I've even thought about routing on my newish Synology RT2600ac router.


----------



## JoeKustra (Dec 7, 2012)

clay.autery said:


> OK.... so their NTP servers appear to be behaving right now, but I am unwilling to rely on them.
> Guess I have to go figure out how to route their NTP Server's Public IPs to my local NTP server on its Private IP.
> First time I've even thought about routing on my newish Synology RT2600ac router.


Sorry Clay, but you may be lucky like me. I have never had a clock issue. I have two Roamio and two Mini boxes on-line right now. I record on both Roamio boxes daily. I don't know why I have never had a clock issue. Until then, I have enough things to keep me busy.


----------



## clay.autery (Feb 3, 2018)

JoeKustra said:


> Sorry Clay, but you may be lucky like me. I have never had a clock issue. I have two Roamio and two Mini boxes on-line right now. I record on both Roamio boxes daily. I don't know why I have never had a clock issue. Until then, I have enough things to keep me busy.


Heh, heh, heh... Yeah, I have a long list, too. But we've been bitten by the clock problem for sure! We had a Suddenlink supplied TiVo for 3-1/2 years, and my spouse has been pointing out to me for months (and months) that we frequently lose a couple minutes off programs at one end or the other. I never paid much attention before, but if I can do something like a little routing to prevent me from ever having to hear about missing minutes, or shifted programming start/stop times....

*I'm on it!*


----------



## BobCamp1 (May 15, 2002)

clay.autery said:


> Heh, heh, heh... Yeah, I have a long list, too. But we've been bitten by the clock problem for sure! We had a Suddenlink supplied TiVo for 3-1/2 years, and my spouse has been pointing out to me for months (and months) that we frequently lose a couple minutes off programs at one end or the other. I never paid much attention before, but if I can do something like a little routing to prevent me from ever having to hear about missing minutes, or shifted programming start/stop times....
> 
> *I'm on it!*


I've been experiencing the same thing. It's not a direct observation, but more like a feeling of, "that's weird, it barely caught the end even with the padding" or "why do I have to FF to the start of the show when there's no padding at the beginning?" or "why did I miss the first 20 seconds of the show?"


----------



## rdrrepair (Nov 24, 2006)

clay.autery said:


> ... we frequently lose a couple minutes off programs at one end or the other.





BobCamp1 said:


> ... or "why did I miss the first 20 seconds of the show?"


I don't think it's ever been slow. I've only see it clip the endings. I think all the reports here have shown the clock to be too fast by up to 2 minutes.


----------



## clay.autery (Feb 3, 2018)

BobCamp1 said:


> I've been experiencing the same thing. It's not a direct observation, but more like a feeling of, "that's weird, it barely caught the end even with the padding" or "why do I have to FF to the start of the show when there's no padding at the beginning?" or "why did I miss the first 20 seconds of the show?"


Yep... fortunately, it appears TiVo has fixed their NTP servers at least for now. In the last 24 hours or so, their servers have been pretty close. Not "Time Nuts" close, but plenty close for DVRing.

Regardless... I am going to figure out how to redirect the TiVo calls on Port 123 to my local Stratum 1 NTP server.


----------



## slowbiscuit (Sep 19, 2006)

Blocking sjr1 and 2 has been fine for me since 3 is not far off of reality (which could change any day, I know). Others have tried redirects and have found that Tivo service calls error out.


----------



## clay.autery (Feb 3, 2018)

Last two days, I've been seeing them ALL THREE close! To within a single millisecond.

And ALL three servers are reporting as Stratum 2 servers now.... meaning that ALL THREE are syncing with one or more Stratum 1 servers.


----------



## tim1724 (Jul 3, 2007)

clay.autery said:


> Last two days, I've been seeing them ALL THREE close! To within a single millisecond.
> 
> And ALL three servers are reporting as Stratum 2 servers now.... meaning that ALL THREE are syncing with one or more Stratum 1 servers.


I've seen them all be close, but I've also seen them be up to 0.7 seconds or so fast (in the last day or two). They're usually reporting stratum 2, but it's still pretty common to see one of them at 16.

Right now:
server 204.176.49.10, stratum 2, offset 0.062374, delay 0.04158
server 204.176.49.11, stratum 16, offset 0.000785, delay 0.04128
server 204.176.49.12, stratum 2, offset 0.392529, delay 0.04114

Although this is not great for an NTP server, it's good enough for a DVR. So I'm satisfied with it for now.


----------



## clay.autery (Feb 3, 2018)

tim1724 said:


> I've seen them all be close, but I've also seen them be up to 0.7 seconds or so fast (in the last day or two). They're usually reporting stratum 2, but it's still pretty common to see one of them at 16.
> 
> Right now:
> server 204.176.49.10, stratum 2, offset 0.062374, delay 0.04158
> ...


Ahhh.... I'm used to looking at Meinberg's NTP monitor on my laptop. w32tm returns offsets in SECONDS....

Yeah much better than many seconds to 2 minutes!!!

Question: What command line or program are you using to produce the output you have above?


----------



## dmurphy (Jan 17, 2002)

This is what they need ... Enterprise-Class SecureSync

plus a GPS antenna, and then we're all speaking the same language.

Great devices, these.


----------



## kdmorse (Jan 29, 2001)

Either I committed network shenanigans so subtle that I can't find them, *and* forgot I did it (Unlikely, but not outside the realm of possibility on either count), or they switched the script to no longer query sjr1/2/3, and directly query three nist timeservers instead.

Someone go trace a daily call and figure out if I'm imagining things...


----------



## tim1724 (Jul 3, 2007)

clay.autery said:


> Question: What command line or program are you using to produce the output you have above?



ntpdate -q sjr{1,2,3}.tivo.com


----------



## clay.autery (Feb 3, 2018)

tim1724 said:


> ntpdate -q sjr{1,2,3}.tivo.com


 Thank you, sir!


----------



## SteveD (Oct 22, 2002)

kdmorse said:


> Either I committed network shenanigans so subtle that I can't find them, *and* forgot I did it (Unlikely, but not outside the realm of possibility on either count), or they switched the script to no longer query sjr1/2/3, and directly query three nist timeservers instead.
> 
> Someone go trace a daily call and figure out if I'm imagining things...


The last time I saw one of my Tivo's query sjr1/2/3 was on Feb. 4. So maybe they did make a change?


----------



## clay.autery (Feb 3, 2018)

**** Red said:


> The last time I saw one of my Tivo's query sjr1/2/3 was on Feb. 4. So maybe they did make a change?


How are you monitoring your TiVo traffic? Are you logging from your router?


----------



## SteveD (Oct 22, 2002)

clay.autery said:


> How are you monitoring your TiVo traffic? Are you logging from your router?


Logging from my firewall. I just forced a call from one Tivo and it hit these four servers to set the time.

132.163.97.1 - time-a-wwv.nist.gov
132.163.96.4 - time-d-b.nist.gov
129.6.15.28 - time-a-g.nist.gov
129.6.15.27 - time-d-g.nist.gov

So it looks like Tivo has moved on from hosting their own NTP servers.


----------



## clay.autery (Feb 3, 2018)

**** Red said:


> Logging from my firewall. I just forced a call from one Tivo and it hit these four servers to set the time.
> 
> 132.163.97.1 - time-a-wwv.nist.gov
> 132.163.96.4 - time-d-b.nist.gov
> ...


Hah!!! That's weird! Reckon they also routed their NTP server IPs to query some other server to maintain the facade? IF not, then it seems weird to change how the "call home" time update works. Their server IPs are largely responding with pretty accurate times.... mostly less than 1/2 second offset, mostly much less.

Having EVERY TiVo in the US query those servers individually instead of querying the pool is, as I understand it, bad form.

That's gonna increase traffic unnecessarily on those 4 Stratum 1 servers.... 

Thanks!


----------



## kdmorse (Jan 29, 2001)

One possibility is that while making some changes to sjr1/2/3, they redirected all our units elsewhere in the meantime. As you know, a failed clock set can fail the daily call, so maybe this was a preventative measure to prevent failed calls. Or maybe this was a reactive measure, as a lot of people reported failed calls due to clock set failures. Maybe they get changed back, maybe not, we'll have to watch and see.


----------



## clay.autery (Feb 3, 2018)

kdmorse said:


> One possibility is that while making some changes to sjr1/2/3, they redirected all our units elsewhere in the meantime. As you know, a failed clock set can fail the daily call, so maybe this was a preventative measure to prevent failed calls. Or maybe this was a reactive measure, as a lot of people reported failed calls due to clock set failures. Maybe they get changed back, maybe not, we'll have to watch and see.


If they don't get swapped back in a reasonable amount of time, perhaps Ted might inquire. It really is poor form to clog up those Stratum 2 servers with unnecessary traffic.


----------



## morac (Mar 14, 2003)

clay.autery said:


> If they don't get swapped back in a reasonable amount of time, perhaps Ted might inquire. It really is poor form to clog up those Stratum 2 servers with unnecessary traffic.


How much traffic do you really think Tivo boxes generate? Each box hits the server once a day. While there are around 7 million subscribers, I would think MSO boxes would hit their own servers.

Even if they don't and all boxes hit the same time servers, that's only a few bytes per box per 26 hours each split across 4 servers. I'm pretty sure the servers can handle the traffic.


----------



## clay.autery (Feb 3, 2018)

morac said:


> How much traffic do you really think Tivo boxes generate? Each box hits the server once a day. While there are around 7 million subscribers, I would think MSO boxes would hit their own servers.
> 
> Even if they don't and all boxes hit the same time servers, that's only a few bytes per box per 26 hours each split across 4 servers. I'm pretty sure the servers can handle the traffic.


The point is that there are customs, courtesies, and rules for using the NTP system.... TiVo is not following them. 

TiVo should service its own machines.


----------



## morac (Mar 14, 2003)

clay.autery said:


> The point is that there are customs, courtesies, and rules for using the NTP system.... TiVo is not following them.
> 
> TiVo should service its own machines.


I'm pretty sure Microsoft Windows uses the nist.gov servers too or at least has an option to use them.


----------



## clay.autery (Feb 3, 2018)

morac said:


> I'm pretty sure Microsoft Windows uses the nist.gov servers too or at least has an option to use them.


Yes, but the option, which few people ever know exists OR switch to is a POOL/Global address..... NOT the direct servers listed above. Specifically, *time.nist.gov
*
*NIST Internet Time Servers*

_3. The generic name time.nist.gov will continue to point to all of our servers on a round-robin basis, and users are encouraged to access the service using this name.

(...)

The global address time.nist.gov is resolved to all of the server addresses below in a round-robin sequence to equalize the load across all of the servers.

Whether you connect to a server using the name or the IP address, it is a bad practice to "hard-code" a particular server name or address into a device so that these parameters cannot be changed by the end user if that becomes necessary at some future time._


----------



## slowbiscuit (Sep 19, 2006)

Yeah, they could've just used pool.ntp.org and been done with it like zillions of other devices out there. But that's Tivo, they apparently don't pay enough to get sysadmins that know their stuff. Even basic stuff like this, sadly.


----------



## HerronScott (Jan 1, 2002)

clay.autery said:


> Yes, but the option, which few people ever know exists OR switch to is a POOL/Global address..... NOT the direct servers listed above. Specifically, *time.nist.gov
> *
> *NIST Internet Time Servers*
> 
> ...


NIST isn't saying that it's bad practice for end devices to use these servers. They are saying the bad practice is to hard code it where the end user can't change it if they go away or have issues (and thus better to use the general dns time.nist.gov. In our case, TiVo is managing this for us as part of our service and should fix it if any of the ones they directly specify go away. 

Oh the important parts you left out from your quote of NIST's NTP page was where they *encourage* users to use the generic name (not require or insist) and the 4 second rule which our TiVo's are no where near causing an issue.

*"The generic name time.nist.gov will continue to point to all of our servers on a round-robin basis, and users are encouraged to access the service using this name."*

"All users should ensure that their software *NEVER* queries a server more frequently than once every 4 seconds. Systems that exceed this rate will be refused service. In extreme cases, systems that exceed this limit may be considered as attempting a denial-of-service attack."

Scott


----------



## clay.autery (Feb 3, 2018)

HerronScott said:


> NIST isn't saying that it's bad practice for end devices to use these servers. They are saying the bad practice is to hard code it where the end user can't change it if they go away or have issues (and thus better to use the general dns time.nist.gov. In our case, TiVo is managing this for us as part of our service and should fix it if any of the ones they directly specify go away.
> 
> Oh the important parts you left out from your quote of NIST's NTP page was where they *encourage* users to use the generic name (not require or insist) and the 4 second rule which our TiVo's are no where near causing an issue.
> 
> ...


We ARE the users, and the servers ARE effectively hard-coded, we DID have problems, and we DO NOT have any direct control over how time is served to OUR machines, UNLESS we perform routing trickery to fool the machines into getting time from functional servers.

Nevermind.... I'm not going to argue/explain the point with any of you any further. The quote from the NIST site is really only the tip of the iceberg.

Deal is... there doesn't always have to be a law or a set of ironclad regulations on how things should be done. Customs, practices, recommendations, and "just the right thing to do" are generally much better ways of handling things.

This will be a moot point for me soon anyway, as I will serve my own machines from my personal time server.

Have a great weekend!


----------



## kdmorse (Jan 29, 2001)

**** Red said:


> 132.163.97.1 - time-a-wwv.nist.gov
> 132.163.96.4 - time-d-b.nist.gov
> 129.6.15.28 - time-a-g.nist.gov
> 129.6.15.27 - time-d-g.nist.gov


Yup, didn't have time to look deeper until just now:

TIME_SVC=/bin/ntpdate -b 129.6.15.27 129.6.15.28 132.163.96.4 132.163.97.1


----------



## macwhizROC (Mar 14, 2007)

Wow, I was late finding this thread, but it explains some odd cut-off programs lately.

And it astounds me. I've maintained NTP servers at two Fortune 500 companies. I can't think of a good excuse for what I'm seeing here.

For a corporation the size of Rovi/TiVo, $3,000 is chump change. And that's what it costs to buy a basic radio clock appliance that can easily handle serving NTP to the entire user base (7,500 NTP packets per second). You can go cheap and get a CDMA-synced clock that uses signals from Verizon and Sprint cell towers; the towers need GPS sync because of how CDMA works. Cell signals usually penetrate buildings, so there's no install cost like there is with GPS (where you have to get Facilities to run wire up to the roof).

That appliance comes with a TCX0 oscillator, which drifts 10 milliseconds a day with no CDMA signal. That means it will continue to serve stratum-1 time for 24 hours. You can upgrade it with an OCX0 oscillator and get 80 microseconds a day, for 35 days stratum-1 with no CDMA signal.

Buy three of those, install them preferably at different data centers, configure them to peer with each other and perhaps the NIST pool, and you should never have NTP issues.

But let's say they're on a real shoestring budget. $70 will buy you a Garmin GPS 18X receiver that you can solder a serial connector onto, plug into your choice of UNIX computer, and get GPS-synced Stratum 0 time. I've got one. My home NTP server is currently reliably synced within 0.007 milliseconds of "real time."

And I can't figure out how their NTP setup was losing upstream connections. The golden rule of NTP setup is the "rule of threes." When you set up an NTP server, you point it to _at least_ three better time sources. That way, not only will one failure not take you out, but if one upstream is flaking out, the other two will "outvote" it. This is basic stuff!

Two _minutes_ off? An offset of 100 _milliseconds_ would be an issue anywhere I've worked. A full second would be a full red-alert, wake-everyone-up fire drill. _One_ minute would result in at least one firing!

Just... wow.

Somehow, this more than anything else tells me I need to make plans to move off the TiVo ecosystem after all these years, before it fails completely.


----------



## slowbiscuit (Sep 19, 2006)

Yep, this is certainly not the only case where stuff that should be taken for granted is a Rivo fail. Guide data is #1, obviously, but this one is #2 I'd say.


----------



## wish_bgr (Jul 19, 2014)

The hamsters are amok again! Or, I’m relegated to the TiVo.1 time server; have forced connection today to see if there was a fix for the missing program pictures and various icons for apps and also to keep up on syncing time. I’m on a wired connection, using an available Ethernet port on my switch to the router and have been ‘bonked’ out during the connection process when TiVo tried to set clock. Oh well, another business-as-usual day at Rivo/TiVo…


----------



## tim1724 (Jul 3, 2007)

Yuck, sjr2 is 21 seconds off. and it looks like sjr1 might have been reset just before I checked (as it's stratum 16 too)

$ ntpdate -q sjr{1,2,3}.tivo.com
server 204.176.49.10, stratum 16, offset 0.148524, delay 0.04129
server 204.176.49.11, stratum 16, offset 21.629903, delay 0.04140
server 204.176.49.12, stratum 2, offset 0.019779, delay 0.04121​


----------



## morac (Mar 14, 2003)

Is Tivo back to using their ntp servers again?


----------



## HerronScott (Jan 1, 2002)

wish_bgr said:


> The hamsters are amok again! Or, I'm relegated to the TiVo.1 time server; have forced connection today to see if there was a fix for the missing program pictures and various icons for apps and also to keep up on syncing time.


Fix should be coming for the missing images on Roamios (related to the RC30 update).

20.7.4.RC30 has arrived......

Scott


----------



## SteveD (Oct 22, 2002)

morac said:


> Is Tivo back to using their ntp servers again?


My Tivo's are still using the nist.gov time servers.


----------



## choco (Nov 3, 2000)

I just noticed our Roamio clock is 2 minutes fast once again. Not sure when this problem started. I tried testing the internet connection, it set the clock, but came back with the same incorrect time.


----------



## kdmorse (Jan 29, 2001)

choco said:


> I just noticed our Roamio clock is 2 minutes fast once again. Not sure when this problem started. I tried testing the internet connection, it set the clock, but came back with the same incorrect time.


Really, really???

.. checks logs ...

Yup. sjr2.tivo.com is clearly drunk off it's ass again. Wobbled around much of yesterday afternoon, and lost all semblance of sync around 7pm, and has been drifting linearly every since. It is, in fact, 2 minutes 13 seconds fast right now.

There was a time when all our units were redirected to nist servers instead so when their local timeservers got drunk, it didn't impact units. Your post strongly implies something's changed in that regard.


----------



## kdmorse (Jan 29, 2001)

kdmorse said:


> There was a time when all our units were redirected to nist servers instead so when the local timeservers got drunk, it didn't impact units. Your post strongly implies something's changed in that regard.


As part of a daily call, my unit just queried:

129.6.15.27 - time-d-g.nist.gov.
129.6.15.28 - time-a-g.nist.gov.
132.163.96.4 - time-d-b.nist.gov
132.163.97.1 - time-a-wwv.nist.gov.
*204.176.49.11 - sjr2.tivo.com.*

So, the drunk timeserver made it back into the list.


----------



## tim_m (Mar 8, 2017)

All correct here and i just did a check internet connection to check. The drunk one must've been removed again.


----------



## morac (Mar 14, 2003)

tim_m said:


> All correct here and i just did a check internet connection to check. The drunk one must've been removed again.


If only one is bad, there's a 4 out of 5 chance of getting a good time.


----------

