# Sharing commercial skip files



## moyekj (Jan 24, 2006)

With the introduction of kmttg AutoSkip which allows you to automatically skip commercials when viewing shows on series 4 or later TiVos the first step towards being able to share commercial skip files is in place.

Of course the problem is we have to currently manually generate our own .edl or .Vprj skip files for this to work. It would be nice to have a database of skip files available for download so that the community could share skip files with each other. That's what this thread is about - trying to come up with a viable scheme to allow sharing of commercial skip files.

TiVo with introduction of SkipMode is effectively doing that. They have a database of skip files. They generate a skip file on a specific TiVo and then came up with a way to share that data with other TiVos. Of course the problem is that the precise cut points can vary from TiVo to TiVo, so there needs to be a method to adjust cut points to a particular TiVo, otherwise it's worthless.

TiVo came up with a method to make those adjustments by using closed caption data which is assumed to be universally available among the TiVos sharing skip data. If you know the precise location in time of captions located close to skip points, you can adjust the skip points accordingly to account for timing differences between TiVos and thus effectively share skip data.

In light of the above, I was thinking we could mimic the TiVo mechanism to some degree. If we collect 2 sets of files for a particular TiVo recording of a show:
1. A .edl or .Vprj file with skip points
2. A .srt file containing caption data for that same recording

Then we can have some code that looks at the .srt file and the cut point file to determine time offsets of caption lines surrounding cut points. i.e. Attempting to depict this:

TIVO 1
captions_1-----cut point-----captions_2

TIVO 2
---------captions_1-----cut point------captions_2

If we can assume the time offset between captions_1 or captions_2 and cut point is the same for both TiVos then we can precisely adjust cut point for TIVO 2 using data from TIVO 1 and .srt data for TIVO 2. Multiple caption lines would have to be used to try and figure things out obviously since a single caption line by itself will not be unique for many cases.

Note that the above is just brainstorming. I have no idea how practical it is to implement. But I'd like to explore it further or perhaps get ideas from others on more effective ways of sharing skip data.

Perhaps as a 1st step assuming we pursue the above is to pick a certain show and generate skip files and .srt files for it and share the files with each other. Then just manually examine the files and see if we can manually generate an adjusted skip file that would work on our TiVo. Anybody up for doing such an experiment?


----------



## lew (Mar 12, 2002)

I'm not sure how often I'll be using kmttg autoskip however I'd be willing to participate in a test. Some people will be very precise with their cut points, others will be a second or two off.


----------



## moyekj (Jan 24, 2006)

We need to find a show we both have recordings for. Here's some recent ones I have:
Grimm [Ep 511] - Key Move
Elementary [Ep 415] - Up to Heaven and Down to Hell
Colony [Ep 108] - In From the Cold
The 100 [Ep 307] - Thirteen
The Family [Ep 101] - Pilot
Marvel's Agent Carter [Ep 210] - Hollywood Ending
Blindspot [Ep 111] - Cease Forcing Enemy
Hawaii Five-0 [Ep 616] - Ka Pohaku Kihi Pa'a
Marvel's Agent Carter [Ep 208] - The Edge of Mystery; A Little Song and Dance
Hawaii Five-0 [Ep 615] - Ke Koa Lokomaika'i
Grimm [Ep 510] - Map of the Seven Knights
Marvel's Agent Carter [Ep 206] - Life of the Party; Monsters
Grimm [Ep 509] - Star-Crossed


----------



## reneg (Jun 19, 2002)

If needed, I can help out. I have recordings of the following from your list:



moyekj said:


> Elementary [Ep 415] - Up to Heaven and Down to Hell
> Colony [Ep 108] - In From the Cold
> The 100 [Ep 307] - Thirteen
> The Family [Ep 101] - Pilot
> ...


----------



## moyekj (Jan 24, 2006)

OK, great. How about Blindspot Ep 111 then? I'll generate .Vprj file with my cut points and .srt file for it and put it in a zip file shortly.

NOTE: Use TS container when downloading so as to have less chance of captions in .srt file being corrupted.


----------



## moyekj (Jan 24, 2006)

OK here's my data for Blindspot. Note that I didn't mark the lead in commercial or the trailing commercial since I normally don't bother with those. From kmttg AutoSkip table these are the reported cut points:

AutoSkip data for 'Blindspot - Cease Forcing Enemy'
0: start=00:00 end=16:18
1: start=18:50 end=26:18
2: start=29:55 end=34:07
3: start=38:25 end=43:30
4: start=47:32 end=59:56

And more precisely from the AutoSkip.ini file in msecs:
0 978144
1130562 1578977
1795460 2047178
2305670 2610541
2852416 3596000

For the above points here's closest meaningful CC data from my .srt file:
Before "end=16:18"
343
00:16:12,072 --> 00:16:13,237
The passengers.

344
00:16:13,239 --> 00:16:14,772
They're still alive.

After "start=18:50"
347
00:18:54,634 --> 00:18:55,967
Do you speak English?

348
00:18:55,969 --> 00:18:58,302
I do. 
My name is Kurt Weller.

I think for now just focusing on those 2 points would be enough to get started with the analysis.

EDIT: Looking at above data with all timing converted to msecs here's what I get:
CC 343: 00:16:12,072 --> 00:16:13,237
972072 - 973237 => -6072 to -4907 away from my end point

CC 344: 00:16:13,239 --> 00:16:14,772
973239 - 974772 => -4905 to -3372 away from my end point

CC 347: 00:18:54,634 --> 00:18:55,967
1134634 - 1135967 => 4072 to 5405 away from my start point

CC 348: 00:18:55,969 --> 00:18:58,302
1135969 - 1138302 => 5407 to 7740 away from my start point

So if someone else posts their .srt file we can compute the location of end point and start point relative to their CC data and see if those points are close to good cut points for their recording.


----------



## reneg (Jun 19, 2002)

My data for Blindspot. I ran combined decrypt & qsfix on the file before running ad detect. My vPrj cut points are on I-frames. I extracted captions without running ad-cut.

Vprj Data:

```
<ProjectTime>25226226201</ProjectTime>
<CutList><cut Sequence="1" CutStart="00:00:00;00" CutEnd="00:00:09;15" Elapsed="00:00:00;00"><CutTimeStart>0</CutTimeStart>
<CutTimeEnd>95095112</CutTimeEnd>
<CutByteStart>0</CutByteStart>
<CutByteEnd>15373133</CutByteEnd>
</cut><cut Sequence="2" CutStart="00:16:20;18" CutEnd="00:18:52;04" Elapsed="00:16:11;03"><CutTimeStart>9805796112</CutTimeStart>
<CutTimeEnd>11321310112</CutTimeEnd>
<CutByteStart>1613090757</CutByteStart>
<CutByteEnd>1862538557</CutByteEnd>
</cut><cut Sequence="3" CutStart="00:26:21;21" CutEnd="00:29:57;03" Elapsed="00:23:40;18"><CutTimeStart>15816801112</CutTimeStart>
<CutTimeEnd>17970953112</CutTimeEnd>
<CutByteStart>2602087881</CutByteStart>
<CutByteEnd>2956524093</CutByteEnd>
</cut><cut Sequence="4" CutStart="00:34:09;26" CutEnd="00:38:27;19" Elapsed="00:27:53;11"><CutTimeStart>20498478112</CutTimeStart>
<CutTimeEnd>23076053112</CutTimeEnd>
<CutByteStart>3372372949</CutByteStart>
<CutByteEnd>3796463913</CutByteEnd>
</cut><cut Sequence="5" CutStart="00:43:32;06" CutEnd="00:47:34;02" Elapsed="00:32:57;28"><CutTimeStart>26122096112</CutTimeStart>
<CutTimeEnd>28540512112</CutTimeEnd>
<CutByteStart>4297689961</CutByteStart>
<CutByteEnd>4695676937</CutByteEnd>
</cut><cut Sequence="6" CutStart="00:56:02;12" CutEnd="00:59:04;03" Elapsed="00:41:26;08"><CutTimeStart>33623590112</CutTimeStart>
<CutTimeEnd>35440405112</CutTimeEnd>
<CutByteStart>5531852997</CutByteStart>
<CutByteEnd>5830770929</CutByteEnd>
</cut><cut Sequence="7" CutStart="00:59:40;12" CutEnd="01:00:00;27" Elapsed="00:42:02;19"><CutTimeStart>35803768112</CutTimeStart>
<CutTimeEnd>36008973112</CutTimeEnd>
<CutByteStart>5890619225</CutByteStart>
<CutByteEnd>5924652868</CutByteEnd>
</cut></CutList>
```
Autoskip.ini data:

```
<entry>
contentId=tivo:ct.340036130
offerId=tivo:of.ctd.3929992.612.cable.2016-03-01-03-00-00.3600
offset=0
tivoName=Living Room
title=Blindspot - Cease Forcing Enemy
0 0
9509 980579
1132131 1581680
1797095 2049847
2307605 2612209
2854051 3362359
3544040 3580376
3600897 3594000
```
SRT file near first commercial:

```
342
00:16:15,040 --> 00:16:16,207
        The passengers.         

343
00:16:16,209 --> 00:16:17,742
 They're still alive.           

344
00:18:55,400 --> 00:18:57,602
               ¶                

345
00:18:57,604 --> 00:18:58,936
     Do you speak English?
```


----------



## moyekj (Jan 24, 2006)

OK, so based on your srt data and applying my offsets I get the following assuming I did the math correctly:
343
00:16:16,209 --> 00:16:17,742
976209 - 977742 => end point = 977742 + 3372 = 981114 = 16:21 (980579) => 535 msec difference vs yours

345
00:18:57,604 --> 00:18:58,936
1137604 - 1138936 => start point = 1137604 - 4072 = 1133532 18:54 (1132131) => 1401 msec difference vs yours

So if you can check that the following would be viable for your Blindspot recording:
981114 msecs or ~ 16:21 = start of 1st commercial
1133532 msecs or ~ 18:54 = end of 1st commercial

EDIT: The reverse analysis using your points and applying your CC offsets to my CC data I get very close to what I had marked:

your end point: 980579
your closest cc point: 977742 => -2837 away from end point

your start point: 1132131
your closest cc point: 1137604 => 5473 away from start point

=> end point: 974772 + 2837 = 977609 =~ 16:18 (I marked ~ 16:18)

=> start point: 1134634 - 5473 = 1129161 =~ 18:49 (I marked ~ 18:50)


----------



## reneg (Jun 19, 2002)

moyekj said:


> OK, so based on your srt data and applying my offsets I get the following assuming I did the math correctly:
> 343
> 00:16:16,209 --> 00:16:17,742
> 976209 - 977742 => end point = 977742 + 3372 = 981114 = 16:21 (980579) => 535 msec difference vs yours
> ...


Using VideoRedo to check:
16:21:00 would be viable, It is a black screen.
I-frame
16:20.18 - Black video (My start of the 1st commercial cutpoint)
16:22.06 - Start of 1st commercial

At 18:54:00, it is into the show.
I- Frame 
18:52.04 - black screen (my end of 1st commercial cutpoint)
18:52.19 - Start of show after commercial
18:53.19
18:54.19 - Nearest I-frame to 18:54

Could the commercial on your system have run long? I have an additional caption at

```
344
00:18:55,400 --> 00:18:57,602
```
which was not in your SRT


----------



## moyekj (Jan 24, 2006)

reneg said:


> Could the commercial on your system have run long? I have an additional caption at
> 
> ```
> 344
> ...


I think mine is there but I didn't use it for the analysis since it's not text:
346
00:18:51,163 --> 00:18:52,430
♪♪

Let me double-check that my end of 1st commercial point is indeed valid for my recording. EDIT: Yup, my point is perfect for my recording - a black frame right before the show starts again.

Perhaps you can double-check my math as could be I screwed it up for that point. Note that it's 18:53.5 the actual point so probably about 1 second into your show.


----------



## moyekj (Jan 24, 2006)

We should probably look at the other cut points as well to get a complete picture how accurate this can be expected to be... Right now I'm doing all this manually which is pretty painful, but I don't want to spend time writing a program just yet if it turns out it's not going to work out very well. I will note that using TiVo SKIP data sometimes it skips about 1 second or so into the show for me so their methodology is not perfect either, but I don't expect it to be perfect. As long as it's within 1 or 2 seconds at the most I'm OK with it.


----------



## moyekj (Jan 24, 2006)

OK, did a full analysis with the help of some code using your data which I capture here:

```
9509 980579
1132131 1581680
1797095 2049847
2307605 2612209
2854051

00:09 16:20
18:52 26:21
29:57 34:09
38:27 43:32
47:34

16:20                               980579
YOURS
343
00:16:16,209 --> 00:16:17,742       976209 - x 977742 => 4370 before
 They're still alive.

MINE
344
00:16:13,239 --> 00:16:14,772       973239 - x 974772 => 974772 + 4370 = 979142 = 16:19
 They're still alive.           

18:52                               1132131
YOURS
345
00:18:57,604 --> 00:18:58,936       x 1137604 - 1138936 => 5473 after
     Do you speak English?      

MINE
347
00:18:54,634 --> 00:18:55,967       x 1134634 - 1135967 => 1134634 - 5473 = 1129161 = 18:49
     Do you speak English?      

26:21                               1581680
YOURS
521
00:26:17,442 --> 00:26:19,610       1577442 - x 1579610 => 2070 before
       (knife sharpening)
       
MINE
523
00:26:14,473 --> 00:26:16,640       1574473 - x 1576640 => 1576640 + 2070 = 1578710 = 26:18
       (knife sharpening)       

29:57                               1797095
YOURS
543
00:30:01,032 --> 00:30:03,400       x 1801032 - 1803400 => 3937 after
          (Mayfair)             
    So I need you to send       
     a team to the island       

MINE
539
00:29:58,063 --> 00:30:00,430       x 1798063 - 1800430 => 1798063 - 3937 = 1794126 = 29:54
    So I need you to send       
     a team to the island       

34:09                               2049847
YOURS
625
00:34:00,138 --> 00:34:01,371       2040138 - x 2041371 => 8476 before
              Agh!              

MINE
621
00:33:57,169 --> 00:33:58,402       2037169 - x 2038402 => 2038402 + 8476 = 2046878 = 34:06
              Agh!              

38:27                               2307605
YOURS
714
00:38:42,086 --> 00:38:43,320       x 2322086 - 2323320 => 14481 after
(man)                           
Why haWe're not high enough yet.

MINE
708
00:38:39,117 --> 00:38:40,350       x 2319117 - 2320350 => 2319117 - 14481 = 2304636 = 38:24
      We're not high enough yet.

43:32                               2612209
YOURS
831
00:43:30,275 --> 00:43:32,342       2610275 - x 2612342 => 133 before
      You're gonna what?!       

MINE
825
00:43:27,305 --> 00:43:29,372       2607305 - x 2609372 => 2609372 + 133 = 2609505 = 43:29
      You're gonna what?!       

47:34                               2854051
YOURS
868
00:47:37,521 --> 00:47:40,823       x 2857521 - 2860823 => 3470 after
    (fuselage shuddering,       
     instruments beeping))      

MINE
869
00:47:34,552 --> 00:47:37,854       x 2854552 - 2857854 => 2854552 - 3470 = 2851082 = 47:31
    (fuselage shuddering,       
     instruments beeping)
```
Bottom line of the above analysis is using your data would have worked very well for me:
MY ORIGINAL
0: start=00:00 end=16:18
1: start=18:50 end=26:18
2: start=29:55 end=34:07
3: start=38:25 end=43:30
4: start=47:32

USING YOUR DATA AND OFFSETS
0: start=00:00 end=16:19
1: start=18:49 end=26:18
2: start=29:54 end=34:06
3: start=38:24 end=43:29
4: start=47:31


----------



## reneg (Jun 19, 2002)

That's pretty close, looks approximately a second off. This is certainly cumbersome to analyze by hand. I was comparing with the help of Subtitle Edit & Excel. Subtitle Edit's Synchronization->Point Sync via other subtitle helped a bit (screenshot attached)


----------



## moyekj (Jan 24, 2006)

OK, now did the reverse. Used my data to calculate what it would end up for you:

```
0 978144
1130562 1578977
1795460 2047178
2305670 2610541
2852416

00:00 16:18
18:50 26:18
29:55 34:07
38:25 43:30
47:32

16:18                               978144
YOURS
343
00:16:16,209 --> 00:16:17,742       976209 - x 977742 => 977742 + 3372 = 981114 = 16:21
 They're still alive.

MINE
344
00:16:13,239 --> 00:16:14,772       973239 - x 974772 => 3372 before
 They're still alive.           

18:50                               1130562
YOURS
345
00:18:57,604 --> 00:18:58,936       x 1137604 - 1138936 => 1137604 - 4072 = 1133532 = 18:53
     Do you speak English?      

MINE
347
00:18:54,634 --> 00:18:55,967       x 1134634 - 1135967 => 4072 after
     Do you speak English?      

26:18                               1578977
YOURS
521
00:26:17,442 --> 00:26:19,610       1577442 - x 1579610 => 1579610 + 2337 = 1581947 = 26:21
       (knife sharpening)
       
MINE
523
00:26:14,473 --> 00:26:16,640       1574473 - x 1576640 => 2337 before
       (knife sharpening)       

29:55                               1795460
YOURS
543
00:30:01,032 --> 00:30:03,400       x 1801032 - 1803400 => 1801032 - 2603 = 1798429 = 29:58
          (Mayfair)             
    So I need you to send       
     a team to the island       

MINE
539
00:29:58,063 --> 00:30:00,430       x 1798063 - 1800430 => 2603 after
    So I need you to send       
     a team to the island       

34:07                               2047178
YOURS
625
00:34:00,138 --> 00:34:01,371       2040138 - x 2041371 => 2041371 + 8776 = 2050147 = 34:10
              Agh!              

MINE
621
00:33:57,169 --> 00:33:58,402       2037169 - x 2038402 => 8776 before
              Agh!              

38:25                               2305670
YOURS
714
00:38:42,086 --> 00:38:43,320       x 2322086 - 2323320 => 2322086 - 13447 = 2308639 = 38:28
(man)                           
Why haWe're not high enough yet.

MINE
708
00:38:39,117 --> 00:38:40,350       x 2319117 - 2320350 => 13447 after
      We're not high enough yet.

43:30                               2610541
YOURS
831
00:43:30,275 --> 00:43:32,342       2610275 - x 2612342 => 2612342 + 1169 = 2613511 = 43:33
      You're gonna what?!       

MINE
825
00:43:27,305 --> 00:43:29,372       2607305 - x 2609372 => 1169 before
      You're gonna what?!       

47:32                               2852416
YOURS
868
00:47:37,521 --> 00:47:40,823       x 2857521 - 2860823 => 2857521 - 2136 = 2855385 = 47:35
    (fuselage shuddering,       
     instruments beeping))      

MINE
869
00:47:34,552 --> 00:47:37,854       x 2854552 - 2857854 => 2136 after
    (fuselage shuddering,       
     instruments beeping)
```
YOUR ORIGINAL
0: start=00:00 end=16:20
1: start=18:52 end=26:21
2: start=29:57 end=34:09
3: start=38:27 end=43:32
4: start=47:34

USING MY DATA AND OFFSETS
0: start=00:00 end=16:21
1: start=18:53 end=26:21
2: start=29:58 end=34:10
3: start=38:28 end=43:33
4: start=47:35

So, seems to work pretty darn well to me (for sample size of 1 of course).

This may be worth pursuing after all. Would be nice to get the data from someone else for this show.


----------



## innocentfreak (Aug 25, 2001)

Downloading now to see what I show.

This is what my .edl file contains for Blindspot ep 111. Not sure if it is helpful or not. 

0.00 26.39	0
979.01	1131.00	0
1580.51	1795.96	0
2048.58	2306.44	0
2611.74	2852.88	0
3361.36	3599.70	0

This was in the other text document. 
FILE PROCESSING COMPLETE 107883 FRAMES AT 2997
-------------------
1	791
29341	33896
47368	53825
61396	69124
78274	85501
100740	107883


----------



## moyekj (Jan 24, 2006)

innocentfreak, we would need your .srt file as well.


----------



## innocentfreak (Aug 25, 2001)

I forgot to check captions. I wondered why I couldn't find it.


----------



## moyekj (Jan 24, 2006)

OK, I have code to automate things now which makes things much easier and more precise than previous postings. Here's my results using each other's data:

moyekj results
moyekj CUT DATA
1: start=00:00:00.000 end=00:16:18.144
2: start=00:18:50.562 end=00:26:18.977
3: start=00:29:55.460 end=00:34:07.178
4: start=00:38:25.670 end=00:43:30.541
5: start=00:47:32.416

USING reneg DATA moyekj WOULD GET THIS
1: start=00:00:07.674 end=00:16:17.609
2: start=00:18:49.161 end=00:26:18.710
3: start=00:29:54.125 end=00:34:06.878
4: start=00:38:24.636 end=00:43:29.239
5: start=00:47:31.082 end=00:55:59.389

USING innocentfreak DATA moyekj WOULD GET THIS
1: start=00:00:25.989 end=00:16:17.508
2: start=00:18:49.498 end=00:26:19.025
3: start=00:29:54.475 end=00:34:07.095
4: start=00:38:24.956 end=00:43:30.239
5: start=00:47:31.379 end=00:55:59.859

reneg results
reneg CUT DATA
1: start=00:00:09.509 end=00:16:20.579
2: start=00:18:52.131 end=00:26:21.680
3: start=00:29:57.095 end=00:34:09.847
4: start=00:38:27.605 end=00:43:32.209
5: start=00:47:34.051 end=00:56:02.359
6: start=00:59:04.040 end=00:59:40.376

USING moyekj DATA reneg WOULD GET THIS
1: start=00:00:00.000 end=00:16:21.114
2: start=00:18:53.532 end=00:26:21.947
3: start=00:29:58.430 end=00:34:10.147
4: start=00:38:28.639 end=00:43:33.511
5: start=00:47:35.385

USING innocentfreak DATA reneg WOULD GET THIS
1: start=00:00:27.824 end=00:16:20.478
2: start=00:18:52.468 end=00:26:21.995
3: start=00:29:57.428 end=00:34:10.064
4: start=00:38:27.925 end=00:43:33.209
5: start=00:47:34.349 end=00:56:02.829

innocentfreak results
innocentfreak CUT DATA
1: start=00:00:26.390 end=00:16:19.010
2: start=00:18:51.000 end=00:26:20.510
3: start=00:29:55.960 end=00:34:08.580
4: start=00:38:26.440 end=00:43:31.740
5: start=00:47:32.880 end=00:56:01.360
6: start=00:59:59.700

USING moyekj DATA innocentfreak WOULD GET THIS
1: start=00:00:00.000 end=00:16:19.646
2: start=00:18:52.064 end=00:26:20.462
3: start=00:29:56.945 end=00:34:08.663
4: start=00:38:27.154 end=00:43:32.042
5: start=00:47:33.917

USING reneg DATA innocentfreak WOULD GET THIS
1: start=00:00:08.074 end=00:16:19.111
2: start=00:18:50.663 end=00:26:20.195
3: start=00:29:55.627 end=00:34:08.363
4: start=00:38:26.120 end=00:43:30.724
5: start=00:47:32.582 end=00:56:00.890

From what I can see from above it seems to work quite remarkably well except using innocentfreak data results in wrong 1st start point which I will need to look into more.
EDIT: Looking into innocentfreak's 1st start point it appears he intentionally started program at seemingly arbitrary point into the show in his skip data: "You can't have this outside my apartment". So the above results are true to that location so there is no issue.

NOTE: As part of this system we could easily have a fixed user-defined offset in seconds to apply to all cut points to adjust start points backwards and end points forwards so that you could force partial viewing of commercial segments to avoid show segments being cut off.


----------



## innocentfreak (Aug 25, 2001)

All I did was download from my Bolt and run metadata, decrypt, qs fix and Ad detect. I forgot to run captions so I ran that separately on the TS file. Weird that it would give different results.

I can try another episode if you want to see if we get similar results or if it is specific to my recordings.


----------



## moyekj (Jan 24, 2006)

innocentfreak said:


> All I did was download from my Bolt and run metadata, decrypt, qs fix and Ad detect. I forgot to run captions so I ran that separately on the TS file. Weird that it would give different results.
> 
> I can try another episode if you want to see if we get similar results or if it is specific to my recordings.


 OK, so you used comskip to generate the cut file with no manual review. Note that reneg and I manually defined our skip points using VRD so they are expected to be more accurate.
I don't see anything weird. The differences in results are mostly because of different source cut points to begin with. I think this exercise has proven that syncing cut points by comparing caption files does actually work pretty well, and with the ability to pad either side of cut points we can pretty much guarantee not to miss any actual show assuming the source cut points are accurate to begin with.

It would be good to pick another show from my list of shows to compare now that I can easily run the comparison using code.

Setting up an actual online database for sharing these files is probably the next big challenge - I welcome ideas on how to do that and/or volunteers to house the data?


----------



## innocentfreak (Aug 25, 2001)

Yeah unfortunately I don't have VRD so I just used the comskip included with KMTTG. 

I am downloading Colony 108 and The Family 101 now.


----------



## innocentfreak (Aug 25, 2001)

Here are the zip files.


----------



## moyekj (Jan 24, 2006)

OK thanks. So you don't have a way of checking/reviewing the data generated by comskip? Because looks to me your Colony comskip points are quite a bit off for me. It's best to have manually verified data to do this analysis otherwise we don't know if it's the algorithm failing or if the cut points are wrong.

Note that for Windows there is a comskip GUI and a kmttg option to turn on comskip review with the GUI. config--Program Options--Use comskip GUI to review detected commercials. When GUI comes up press F1 to see instructions, and then you have to press Esc to close it (don't click on X).


----------



## innocentfreak (Aug 25, 2001)

I will try that. I never mess with comskip so wasn't aware of an easy way to check it.


----------



## moyekj (Jan 24, 2006)

FYI, I worked on this a little more with some more samples from reneg and looks like it can be pretty accurate in generating cuts accurate within 2-3 I-frames. However, given that this solution requires one to download a file in order to generate .srt captions file I think the appeal is very limited. If I have to take the time to download a show in order to generate .srt file anyway then I may as well spend the couple of minutes it takes to find and generate my own cut file using VideoRedo. So don't think this is worth pursuing much further given current limitations. If there was some other way to sync cut points somehow without requiring a download of anything then this would be a LOT more attractive.


----------



## davidblackledge (Sep 9, 2008)

moyekj said:


> FYI, I worked on this a little more with some more samples from reneg and looks like it can be pretty accurate in generating cuts accurate within 2-3 I-frames. However, given that this solution requires one to download a file in order to generate .srt captions file I think the appeal is very limited. If I have to take the time to download a show in order to generate .srt file anyway then I may as well spend the couple of minutes it takes to find and generate my own cut file using VideoRedo. So don't think this is worth pursuing much further given current limitations. If there was some other way to sync cut points somehow without requiring a download of anything then this would be a LOT more attractive.


Well, those of us that just want it to work and don't want to manually process anything, it's valuable. I'd rather have a background process that says "download TS, extract srt, (generate automatic ad skip), delete download. Then import community ad detect file when available (instead of automatic one) so AutoSkip works when I play this later."

In general, community ad vetting is worthwhile even without AutoSkip. It would be great to use that instead of automatic comskip to know your _cut file is reliable.

People like me are cheap and lazy. We'll use the free comskip and don't want to take the time to check the values (especially since comskip's GUI is reallllllllly bad IMO - I've been thinking about how to make an alternate external GUI). If we could get quality from the people who care enough to manually edit their files and simply automatically share the result, then you get a lot of happy people.

That said, there may be even more complexities to this process than you've found so far, so it might still be an awful lot of effort even beyond figuring out how to make this a service.


----------



## moyekj (Jan 24, 2006)

One thing I did learn is for TS downloads not to use VRD QS Fix for decrypting for the purpose of importing skip files because it can alter the timing in the decrypted .ts file vs the original .TiVo file and therefore also affect the timing in the .srt caption file. Instead I had to switch to using tivolibre with DirectShow compatibility turned on (or DSD would have worked too). That preserved the timing properly. TS downloads are needed in order to make sure you don't get corrupted captions.

Yeah, comskip GUI is essentially worthless. VRD is definitely the way to go for an editor and well worth the money.

I guess I'll keep poking this concept some more and perhaps take it to the point where there is some mostly automated way to import a skip share given a zip file that contains a cut file and srt file, and then see where it goes from there.


----------



## moyekj (Jan 24, 2006)

Version 2.1m of kmttg just released has some "hidden" functionality for importing skip data for someone that wants to try it.

See SkipShareImport Wiki.

For a sample skip share I've attached a zip file containing my .Vprj file and .srt file for season 1, episode 1 of the show "Of Kings and Prophets". If you don't have that recording then post here asking for a different one from the list of shows I posted earlier in this thread.

If this turns out to be usable then of course the next step would be defining a server to hold he zip files and a mechanism for kmttg to find and download them automatically. But for now we'll just have to rely on manual downloads of zip files for testing this out.

The GUI for doing the import is pretty crude right now but I think good enough for testing purposes.


----------



## moyekj (Jan 24, 2006)

Since Grimm season 5 ep 12 is missing TiVo SKIP I generated AutoSkip data for it. Find it attached to this post.


----------



## lew (Mar 12, 2002)

Kevin. I should be able to recover from my RDF Hawaii Five O Elementary, Agent Carter and Blindspot. If you get a chance post Blindspot.


----------



## moyekj (Jan 24, 2006)

lew said:


> Kevin. I should be able to recover from my RDF Hawaii Five O Elementary, Agent Carter and Blindspot. If you get a chance post Blindspot.


 Blindspot ep 11 is already posted in post #6 of this thread.


----------



## moyekj (Jan 24, 2006)

So anybody tried this yet? I see a few downloads of the zip files. Any feedback/problems or is this all too confusing?


----------



## keenanSR (Oct 13, 2004)

moyekj said:


> So anybody tried this yet? I see a few downloads of the zip files. Any feedback/problems or is this all too confusing?


I've only been following the "how-to" casually, so in my case, I need to do a deeper reading of how it's all done. So for me, yes, it's somewhat confusing.


----------



## lew (Mar 12, 2002)

moyekj said:


> So anybody tried this yet? I see a few downloads of the zip files. Any feedback/problems or is this all too confusing?


I didn't realize you had posted Blindspot. I wasn't interested until you updated kmttg. I gave it a go.

Worked great. Maybe a second or two of commercial. It can't be exact, I like to err on the side of not missing part of the program. From my point of view the actual commercial skip was perfect.

My bad in not reading documentation. I first just tried to get captions. Job bombed. I then decrypted the file then tried to extract captions. Bombed. Finally I converted the decrypted video from ts to mpg. Caption extraction worked.

Suggestion: If you think sharing commercial skip is viable set up a job which does whatever is necessary to extract the subtitles.

Observation--question. Downloading, decrypting, convert to mpeg file then extract captions takes time, I understand it can be automated. Suppose we skip the caption step but pad a second or two before and after the commercial break. Will that be close enough or will be looking for trouble.

What you're doing is impressive. I can see using it if the process can be automated. Set up an "auto transfer" job so the captions are extracted soon after the recording is completed.


----------



## moyekj (Jan 24, 2006)

lew said:


> Suggestion: If you think sharing commercial skip is viable set up a job which does whatever is necessary to extract the subtitles.


 There already is. If you press 'c' bindkey with an NPL show selected and the srt file is not already created, a dialog box comes up asking you if you want to download and create it. If you say OK then kmttg queues up all the necessary tasks to download and create it automatically (a combined download/decrypt followed by captions task). Then once the srt file is available press 'c' again and then it will be automatically set in the import form.

More refinement and automation is needed, but for now the focus in on testing functionality. An option to pad the skip points by user specified # seconds is something I've mentioned previously and can pretty easily be coded in. I'm looking to see if there are cases when there are gross inaccuracies in skip points of more than a couple of seconds.


----------



## lew (Mar 12, 2002)

moyekj said:


> There already is. If you press 'c' bindkey with an NPL show selected and the srt file is not already created, a dialog box comes up asking you if you want to download and create it. If you say OK then kmttg queues up all the necessary tasks to download and create it automatically (a combined download/decrypt followed by captions task). Then once the srt file is available press 'c' again and then it will be automatically set in the import form.
> 
> More refinement and automation is needed, but for now the focus in on testing functionality. An option to pad the skip points by user specified # seconds is something I've mentioned previously and can pretty easily be coded in. I'm looking to see if there are cases when there are gross inaccuracies in skip points of more than a couple of seconds.


My post wasn't very well written my bad.

I used the 'c' shortcut. The caption extraction bombed. I don't think the c job overrides the default configuration, if necessary. I'm set to download as ts streams, decrypt using VRD, and I don't have the tivolibre checked.

I know you talked about an option to pad the skip points. I'm wondering/speculating if that option will allow us to skip creating a srt file from our recording.

My other point/suggestion was to set up the caption extraction as an auto transfer job.


----------



## moyekj (Jan 24, 2006)

lew said:


> I used the 'c' shortcut. The caption extraction bombed. I don't think the c job overrides the default configuration, if necessary. I'm set to download as ts streams, decrypt using VRD, and I don't have the tivolibre checked.


 It does override any configuration by forcing use of TS download and tivolibre for decrypt. I'd be interested to see message window contents you get when it "bombs out". (Make sure your tools package is up to date since ccextractor was updated a while back).

EDIT: I seem to recall you're using a Mac though right? In that case the problem may be a too old version of ccextractor being used. Download latest version from here for your Mac:
https://sourceforge.net/projects/ccextractor/


----------



## lew (Mar 12, 2002)

failed exit code 

edited to add:

I'm running Windows 7. I'm using VRD, I don't think they have an Apple version.

The "c" job downloaded, didn't decrypt and extraction failed.

I think your logic assumes QS fix will be run to decrypt the tivo file but the c job doesn't bother with that step.

I don't know if I had checked the option to decrypt using QS fix. My normal work flow is to download as tivo ts (encrypted), edit with VRD then decrypt during the cut/encode step. Depending on my source program I might skip the QS fix step.

I think the options I'm selecting in configuration is affecting the way you process a c job.

My 2 cents. Relatively few of kmttg have a licensed copy of VRD. I know what I have to do to make it work. I don't care if it's fixed but I'm willing to help trouble shoot if you want.

I'm using current tools. I updated to the same version. I'll see if that makes a difference.


----------



## garryj (Sep 14, 2006)

I had used VRD to create my own skip points in Grimm(s05e12) prior to your release of v2.1n_beta, but after your release I cleaned out the target folder and ran through the "c" procedure to transfer the recording and create a local .srt caption file.

The recording transfer and .srt file creation went without a hitch. On re-viewing using your caption file "diff" process, the skip points ended up fairly well placed. In a couple of places the ad ran for a couple of seconds before the skip took place, but the entry back into the program looked to be pretty good.

The attached .zip has your original files with my local .srt added. Though it's likely not very useful, I included the .VProj file that has my manual edit skip points.


----------



## moyekj (Jan 24, 2006)

lew said:


> failed exit code
> 
> edited to add:
> 
> ...


 No, as I mentioned above, for captions generation I force use of "tdownload_decrypt" and "captions" jobs regardless of your VRD settings. So if you delete your existing srt file and use the 'c' bindkey that's what kmttg should run to generate captions, and it should not bomb out. If it is bombing out I'd like to see the messages.


----------



## moyekj (Jan 24, 2006)

garryj said:


> I had used VRD to create my own skip points in Grimm(s05e12) prior to your release of v2.1n_beta, but after your release I cleaned out the target folder and ran through the "c" procedure to transfer the recording and create a local .srt caption file.
> 
> The recording transfer and .srt file creation went without a hitch. On re-viewing using your caption file "diff" process, the skip points ended up fairly well placed. In a couple of places the ad ran for a couple of seconds before the skip took place, but the entry back into the program looked to be pretty good.
> 
> The attached .zip has your original files with my local .srt added. Though it's likely not very useful, I included the .VProj file that has my manual edit skip points.


 Great, thanks for the feedback. So it sounds like it worked OK for you and for reneg so far, for at least 2 different shows.


----------



## lew (Mar 12, 2002)

I deleted the captionextraction subdirectory (whatever it's called) and reinstalled tools.

I'm cable to get an srt file. Not sure if that solved it or if there was something corrupted in the download.

I'm running kmttg as a service. Does that affect the update process.


----------



## reneg (Jun 19, 2002)

lew said:


> I deleted the captionextraction subdirectory (whatever it's called) and reinstalled tools.
> 
> I'm cable to get an srt file. Not sure if that solved it or if there was something corrupted in the download.
> 
> I'm running kmttg as a service. Does that affect the update process.


When you update kmttg and tools, you should stop the service before updating.


----------



## moyekj (Jan 24, 2006)

FYI, with just released v2.1n version of kmttg you can now specify AutoSkip padding to use during AutoSkip playback which can be useful when importing external skip data which does not have 100% accurate cut points.

So I think most of the pieces are in place for skip data sharing if there is enough community interest to take this any further.


----------



## lew (Mar 12, 2002)

Makes sense for Premier owners.

Tivo skipmode doesn't work with mpeg4 channels. Do we know if autoskip works with mpeg4?

I don't think I'll use skipmode 30 second skip will take less time. 

Sometimes a feature (suggestion) we make is much harder then we think. Sometimes it's a lot easier and faster.

How hard would it be to use the skip file to auto edit. We could have a job type which will let us select a file on our NPL and wind up with an encoded, commercial free, video.

JMO but using auto skip to watch a current show seems like extra work but
using autoskip to create an edited program for viewing in the future makes a lot of sense. I'm thinking of a show I may wait and binge watch after all, or most, of the season has aired. Shows I may want to archive to watch sometime in the future.


----------



## moyekj (Jan 24, 2006)

lew said:


> Do we know if autoskip works with mpeg4?


 Yes, it should.

I didn't understand the rest of your questions.


----------



## minimeh (Jun 20, 2011)

moyekj said:


> So I think most of the pieces are in place for skip data sharing if there is enough community interest to take this any further.





lew said:


> Makes sense for Premier owners.


Absolutely. As a Premier owner, the auto skip functionality is a great feature.


lew said:


> I don't think I'll use skipmode 30 second skip will take less time.
> 
> JMO but using auto skip to watch a current show seems like extra work...


It is extra work, but most of the extra work is in the downloading, which does not call for human attention. So, for me, it comes down to: Do I want to identify and skip commercials in real time with 30 second skips as I watch a show, or do I want to invest the same effort in identifying the commercials prior to viewing with a tool (VideoRedo) which is much better at that? In my opinion, it is actually less work for me personally to identify the commercials and their durations prior to watching the video. The experience of watching a show without either working at skipping commercials or seeing commercials is the payoff.

I would gladly participate in data sharing, mostly providing my data to the community. I've not participated so far only because there hasn't been any overlap between what I record and what was needed to verify proof of concept.

The latest kmttg streamlines nicely some of the maintenance tasks for autoskip. Along those lines, when the method for data sharing is defined, it seems like a good idea to have kmttg provide a task to package and upload the autoskip data as a follow up to importing skip data.


----------



## moyekj (Jan 24, 2006)

Ironically since I got this prototype working TiVo SkipMode has been showing up on pretty much everything I record on my Roamio Pro so I haven't had to generate AutoSkip files much lately.

minimeh if you've got some skip files generated recently for some shows I'd be interested to download a couple to see how it works. So if you like please list a few shows for which you may have data for to see if there's any overlap with what I record.


----------



## minimeh (Jun 20, 2011)

Due to fastidious clean up, these are the only skip files I currently have:
American Crime.2016.03.09-Season Two Episode Ten
Baskets.2016.03.17-Picnic
Brooklyn Nine-Nine.2016.03.15-Terry Kitties
Madam Secretary.2016.03.20-Higher Learning
Modern Family.2016.03.16-The Cover-Up
Portlandia.2016.03.12-First Feminist City
Portlandia.2016.03.17-Lance Is Smart
The Americans.2016.03.16-The Glanders
Underground.2016.03.16-War Chest​I just went ahead and extracted the subtitles for these to ensure that I could provide both the .prj and .srt. So I figured I might as well zip everything up and attach it here in case it is of any use to anybody.

ETA: Skip data was faulty so removed the attachments.


----------



## minimeh (Jun 20, 2011)

minimeh said:


> Due to fastidious clean up, these are the only skip files I currently have:
> American Crime.2016.03.09-Season Two Episode Ten
> Baskets.2016.03.17-Picnic
> Brooklyn Nine-Nine.2016.03.15-Terry Kitties
> ...


Hmm, thought I could upload the others in more posts, but I'm being blocked currently. I'll try again later.

ETA: Here are shows on my to do list for which I will be creating skip data:
Bates Motel (Mon 3/21)
Blindspot Mon 3/21)
Underground (Wed 3/23)
Modern Family (Wed 3/23)
The Americans (Wed 3/23)
Portlandia (Thu 3/24)
The Blacklist (Thu 3/24)
Baskets (Thu 3/24)
Madam Secretary (Sun 3/27)
The Walking Dead (Sun 3/27)
Talking Dead (Sun 3/27)​


----------



## moyekj (Jan 24, 2006)

minimeh said:


> Due to fastidious clean up, these are the only skip files I currently have:
> American Crime.2016.03.09-Season Two Episode Ten
> Baskets.2016.03.17-Picnic
> Brooklyn Nine-Nine.2016.03.15-Terry Kitties
> ...


 Of that list only "Modern Family" and "The Americans" I have in common so if you can post those it would be good. Thanks.


----------



## minimeh (Jun 20, 2011)

I was able to attach Modern Family and The Americans before being locked out from a third attachment.

ETA: Skip data was faulty so removed the attachments.


----------



## moyekj (Jan 24, 2006)

minimeh said:


> I was able to attach Modern Family and The Americans before being locked out from a third attachment.


 OK thanks. I looked a little at Modern Family and things weren't good at all so I took a closer look at your .srt file and to me it seems to be way off compared to timing of your .TiVo file because it's not the .srt file I need:

For example take 1st end cut point in Vprj which is: 00:07:28.498
By your srt file that would correspond to caption already well into a commercial:
234
00:07:25,562 --> 00:07:28,430
"Hey, dumbbell, I can smell your
breath through the Internet."

But caption just before commercial point should be:
230
00:07:17,137 --> 00:07:19,621
_ And_ they validate parking.

Then take start point: 00:10:44.577
By your srt file the 1st caption right after that point would be roughly:
338
00:10:45,362 --> 00:10:47,162
Oh, just on my way 
to show more houses

But it should be the following:
231
00:07:21,258 --> 00:07:22,424
r one?

i.e. Looks like your srt file is generated after you took all the commercials out already.

*What I need are srt files that correspond to your original .TiVo file where you marked the cuts, and that is generated by making sure you don't already have a .srt file and then press 'c' bindkey with show highlighted in NPL table. *

This should generate a .srt that most closely matches the timing of the .TiVo file. i.e. We are trying to sync cuts in .TiVo file with corresponding captions close to them so it's vitally important the .srt file be based on timing of .TiVo file that the cuts are for.


----------



## minimeh (Jun 20, 2011)

Fresh off the presses, "Blindspot.2016.03.21-Rules in Defiance" skip data.


----------



## minimeh (Jun 20, 2011)

moyekj said:


> i.e. Looks like your srt file is generated after you took all the commercials out already.


Crap, I'm an idiot. Absolute idiot.

When I originally did the cuts I did not generate the srt files. So tonight I opened the VideoRedo projects with the original .tivo files and saved the video as .TS to decrypt the .tivo files. And for whatever reason didn't think about the fact that VideoRedo would not only decrypt but actually do the cuts. Idiot...

The "Blindspot" that I have just uploaded was generated entirely within the proscribed work flow. Please try that and I'll see about correcting the other two archives.


----------



## minimeh (Jun 20, 2011)

"Modern Family" and "The Americans", take two. Captions were extracted properly from unadulterated .ts files, checked and verified.


----------



## moyekj (Jan 24, 2006)

minimeh said:


> "Modern Family" and "The Americans", take two. Captions were extracted properly from unadulterated .ts files, checked and verified.


 Thanks. Here's my analysis on your data imports. I compared against what I would have used for my IFRAME cuts in VRD and then determined how much padding I would need for AutoSkip:

Modern Family (Note: AutoSkip 1 sec padding would have worked fine)
YOURS -- MY IFRAME
00:10.08 -- 00:10.47
07:28.28 -- 07:29.15
10:44.35 -- 10:45.18
16:36.02 -- 16:36.50
20:07.33 -- 20:08.09
27:15.37 -- 27:15.07
30:17.03 -- 30:15.35

The Americans (Note: AutoSkip no padding would have worked perfectly)
YOURS -- MY IFRAME
00:33.52 -- 00:33.50
08:09.46 -- 08:09.02
11:31.04 -- 11:30.38
19:21.03 -- 19:20.57
23:07.28 -- 23:07.20
32:31.17 -- 32:31.09
36:32.33 -- 36:32.28
41:46.22 -- 41:45.28
45:37.30 -- 45:37.23
55:36.36 -- 55:36.23
59:31.19 -- 59:30.52

(For Blindspot I gave up on that show so no longer have a SP for it).


----------



## bwperez (Jan 4, 2005)

moyekj said:


> Since Grimm season 5 ep 12 is missing TiVo SKIP I generated AutoSkip data for it. Find it attached to this post.


This is the first time I've used the AutoSkip feature and I am impressed. The first commercial skip came in about 3 seconds early and the last about 3 seconds late. All others were dead on perfect. This is a feature I'd use and possibly contribute to as I had time.


----------



## jdfs (Oct 21, 2002)

It would be very interesting if the skip files could be automatically crowd sourced, basically everyone running batch jobs for shows not already in the database. Check the database, if not there or already in en-process, volunteer to process. I think with just a handful of contributors it could be maintained nicely. It is also a great backup if Tivo ever decided to cancel the feature.


----------



## Mikeguy (Jul 28, 2005)

jdfs said:


> It would be very interesting if the skip files could be automatically crowd sourced, basically everyone running batch jobs for shows not already in the database. Check the database, if not there or already in en-process, volunteer to process. I think with just a handful of contributors it could be maintained nicely. It is also a great backup if Tivo ever decided to cancel the feature.


Concur 100%, although I'd like to see this adopted formally by TiVo itself into SkipMode. Kind of like what TomTom did (does?) with its GPS units, allowing users to suggest map corrections and (optionally) to download those from others. TiVo could make this entirely optional per box setup. It's this sort of option that could help set TiVo apart.

I do wonder whether there are any law considerations in this, though, as I recall such consideration by TiVo initially with SkipMode.


----------



## Hunny (May 29, 2018)

Is there any way to "download" the commerical/skip markers from TiVo?


----------



## Dan203 (Apr 17, 2000)

Hunny said:


> Is there any way to "download" the commerical/skip markers from TiVo?


There is but they're synced using a hash of the caption data and we couldn't reverse engineer the hash. (several of us tried)

I think kmttg has a way to sync them using an offset it can get by playing the show on one of your TiVos and sending the skip command and detecting the time code offsets.


----------



## moyekj (Jan 24, 2006)

kmttg can indirectly generate the skip points by running what's called "AutoSkip from SkipMode" despite the absence of hash data. It plays a video with SKIP data and scans backwards using "Channel Up" presses for show start points. Then SKIP data downloaded via RPC has the non commercial segment lengths. So based on that it builds the skip data specific to your TiVo's time points. So not a "download" but reconstruction of the skip points. Actually works out to be surprisingly accurate most of the time.


----------



## ClearToLand (Jul 10, 2001)

moyekj said:


> *kmttg can indirectly generate the skip points* by running what's called "AutoSkip from SkipMode" *despite the absence of hash data*. It plays a video with SKIP data and scans backwards using "Channel Up" presses for show start points. Then SKIP data downloaded via RPC has the non commercial segment lengths. So based on that it builds the skip data specific to your TiVo's time points...


I thought that I understood SkipMode theory, but this comment about *hash data vs SKIP data* has me confused.

As I understand it (from various posts including several recent detailed ones from @TiVo_Ted ):

Manually, a TiVo employee marks the beginning and end of the non-commercial portions of a SkipMode designated program.
.
Based on various markets, one show can have *up to 5 unique tag files per program*, i.e. complete sets of commercials.
(I'll SEARCH for the correct term per @TiVo_Ted after I post this.)
.
Usually within a few minutes of the show recording ending and while viewing the NPL, the user's TiVo unit checks to see IF the recorded program is a SkipMode designated program and then IF a hash (checksum, MD5,???) of the specific user's Closed Captions (closely) matches one of the up-to-five available 'tags'. 
- If YES, the SkipMode icon appears on the NPL.
- If NO, the SkipMode icon does NOT appear on the NPL *and* kmttg cannot generate the skip points in its AutoSkip Table (or create the .EDL file).
I just offloaded the complete last seasons of Arrow, The Flash, and Supergirl to my NAS (am working on the last 12 episodes of Gotham now) and of the ~80-90 recordings, less than ~10 don't have SkipMode icons and thus kmttg couldn't generate .EDL files. I created one with ComSkip, but my PCs lack the necessary horsepower to move about during ComSkip Review (I plan to post a list of .EDL requests shortly).

Kevin (and Dan), how can there be SKIP data without hash data? 

*ETA:* *SkipMode Insider Details*


----------



## Dan203 (Apr 17, 2000)

ClearToLand said:


> I thought that I understood SkipMode theory, but this comment about *hash data vs SKIP data* has me confused.
> 
> As I understand it (from various posts including several recent detailed ones from @TiVo_Ted ):
> 
> ...


The way skip mode works is like this... someone at TiVo records the program, on a TiVo, and marks the start and end of each segment. (show segments, not commercial blocks) These segments contain timecode values that are only relevant to the TiVo that recorded it. If you were to try and use these values directly then variations in padding, broadcasting delay, and your TiVo's clock would make it so they wouldn't line up. So what they do is they also create a hash table of every caption in the recording, along with it's timecode, and put that in the data as well. Then when you record the show your TiVo creates it's own hash table with timecodes that match your local recording. It then downloads the skip mode data from the TiVo servers and compares the two hash tables. Where it finds matches it compares the timecodes and uses the delta to offset the show segments.

This is how shows with completely different padding, in different timezones, recorded from different broadcasters can be aligned to the skip mode data TiVo created. It's also why if your recording is even a little corrupt it can cause skip mode not to work because the captions are also corrupt and mess up the hash table.


----------



## ClearToLand (Jul 10, 2001)

Dan203 said:


> ...what they do is they also create a *hash table of every caption in the recording*, along with it's timecode, and put that in the data as well. Then when you record the show your TiVo creates it's own hash table with timecodes that match your local recording. It then downloads the skip mode data from the TiVo servers and compares the two hash tables. Where it finds matches it compares the timecodes and uses the delta to offset the show segments...


From my initial reading of *Post #1*, I was under the impression that only the closed captions ADJACENT to the commercials were important. If ALL the closed captions are required, then a 'glitch' anywhere in the recording could knock the user's hash table below the required threshold of necessary matches for the SKIP icon to appear.



Dan203 said:


> ...This is how shows with completely different padding, in different timezones, recorded from different broadcasters can be aligned to the skip mode data TiVo created. *It's also why if your recording is even a little corrupt it can cause skip mode not to work because the captions are also corrupt and mess up the hash table*.


Since the sharing of commercial skip files (.EDL or .Vprj) also requires sharing an error-free .SRT file, this effectively removes the user's ability to use PS 'Slow' Format File Transfers since, IME, the majority of the .SRT files created from a PS transfer are corrupt (hindsight).

How does the format of a .Vprj file compare to a kmttg-generated .EDL file? (Would someone please attach a sample?) The few ComSkip-generated .EDL files that I've created are TAB delimited while the kmttg-generated ones are CRLF delimited but, IIRC, ComSkip and Streambaby accept both. If not the same format as a kmttg-generated .EDL, is the data in a .Vprj file readable / editable via NotePad in order to share Commercial Skip info with other TCF users?

TS 'Fast' Format File Transfers do introduce the possibility of TS Sync Errors though. The general consensus that I've gathered over the past few years here on TCF is that combining kmttg with VideoReDo will "FIX" TS Sync Errors, but I've also read a few posts stating that the Total Run Time of a VRD "FIXed" TS file is sometimes shorter. Does VRD "FIX" the 188 byte block, or does it DELETE it and recalculate the timecode (is this what "QS Fix" refers to? PTS?) ? If too many closed captions are deleted (or left as corrupted), say on an 'Unlimited Bandwidth' TS transfer via a gigabit LAN ala @reneg, the .SRT file will not be able to be used for sharing.

Which brings us to PyTivo Desktop and *My Successful Experiment with Bandwidth Limiting*. I wish that we had more volunteers willing to test; I use 32Mbps via a Netgear Managed Switch on a Roamio OTA and @reneg uses 88Mbps via NetLimiter software (on a PC) on a Roamio Pro. I transfer the majority of my shows in less than 5 retries (down from an original top stop of 30) while, IIRC, @reneg needs an average of ~22 retries using a top stop of 50.

Thanks again for helping me finally get readable closed captions!


----------



## Dan203 (Apr 17, 2000)

ClearToLand said:


> From my initial reading of *Post #1*, I was under the impression that only the closed captions ADJACENT to the commercials were important. If ALL the closed captions are required, then a 'glitch' anywhere in the recording could knock the user's hash table below the required threshold of necessary matches for the SKIP icon to appear.


This is where you'd need to see TiVo's code to know for sure. But I suspect that they only really use the captions closest to the start of each segment for alignment and don't really need them all. But they appear to dump them all into the data, so I can't say for sure what they're using. I did a line by line comparison of the skipmode data and the caption data dumped to a text file using a 3rd party app and they number of lines was identical. I was even able to line up some common phrases to their hashed values. I just couldn't figure out how to reverse the hash. (there is other info in the captions, besides just the text, that they could be using for the hash)


----------



## ClearToLand (Jul 10, 2001)

moyekj said:


> ...It would be nice to have a database of skip files available for download so that the community could share skip files with each other. That's what this thread is about - trying to come up with a viable scheme to allow sharing of commercial skip files...


It's been over two years since Kevin created this thread and, with a few volunteers, proved that exchanging .EDL and .SRT files is doable. But one month in, March 2016, and the thread died on the vine.

Now that pyTiVo Desktop allows me to download PERFECT closed captions via TS 'Fast' Format File transfer *without* TS Sync Errors, I have dozens of .EDL / .SRT pairs available to exchange. Recently I've been transferring the complete last season of Arrow, Gotham, Supergirl, and The Flash (saved for the summer doldrums) but I have a few 'gaps' in .EDL files.

Is it worth posting requests or is this idea truly dead?


----------



## ClearToLand (Jul 10, 2001)

moyekj said:


> kmttg can indirectly generate the skip points by running what's called "AutoSkip from SkipMode" despite the absence of hash data. It plays a video with SKIP data and scans backwards using "Channel Up" presses for show start points. Then SKIP data downloaded via RPC has the non commercial segment lengths. So based on that it builds the skip data specific to your TiVo's time points. So not a "download" but reconstruction of the skip points. Actually works out to be surprisingly accurate most of the time.


If a show on my TiVo unit does not have a SKIP icon and I transfer it to my PC via kmttg, decrypt, manually create an .EDL file (with VideoReDo or ComSkip) and then import that .EDL file into AutoSkip.ini, how does the AutoSkip Service know when to skip (since there is no 'Press D' on the screen)? Is the AutoSkip Service constantly monitoring the time position of the running show (via RPC)?

If that is how it works, then if someone has just the .EDL file for one of my shows that doesn't have the SKIP icon on my TiVo unit, I can just import it and have perfect (within a second or so) Commercial Skip?


----------



## moyekj (Jan 24, 2006)

ClearToLand said:


> If a show on my TiVo unit does not have a SKIP icon and I transfer it to my PC via kmttg, decrypt, manually create an .EDL file (with VideoReDo or ComSkip) and then import that .EDL file into AutoSkip.ini, how does the AutoSkip Service know when to skip (since there is no 'Press D' on the screen)? Is the AutoSkip Service constantly monitoring the time position of the running show (via RPC)?


 The "Press D" never had anything to do with it and is not something detectable via RPC. Yes the service is monitoring position every 1 second or so via RPC.



> If that is how it works, then if someone has just the .EDL file for one of my shows that doesn't have the SKIP icon on my TiVo unit, I can just import it and have perfect (within a second or so) Commercial Skip?


 That was the whole thrust of this thread that sputtered out. Though you need .edl or .vprj AND captions to share with someone else.

At least for me I don't watch so much TV anymore and TiVo is doing a good job with SkipMode for the few shows I do watch so lost interest in this too. If SkipMode goes away or gets so screwed up by the silly "interesting bookmarks" experiment then that could change.


----------

