# Editing .TiVo Metadata



## ggieseke (May 30, 2008)

I figured out a way to create .TiVo files that aren't encrypted so that you can edit the metadata. The actual metadata is in two XML blocks (hereafter referred to as chunks), that you can extract using tivodecode with the -D option. I will leave documenting the exact XML format to others. The basic format of the file header and chunk headers comes from the tivodecode source, and the rest from trial and error.

I'm attaching a very basic Windows command prompt exe for anyone that wants to try it. The rest of this post is probably of interest to developers only. The usage is as follows:

mktivo chunk1 chunk2 mpegfile tivofile

chunk1 & chunk2 are the XML files extracted by tivodecode which you can edit after extracting them, or create your own from scratch. mpegfile is the unencrypted MPEG from tivodecode, kmttg, VideoReDo etc, and tivofile is the output file that it will write.

The technical stuff:

TiVo files normally consist of the file header, followed by chunk #3, chunk #1, chunk #2, and the actual MPEG file. Chunk #3 is not encrypted and contains the disclaimer. Desktop could care less if it's there so I leave it out. The rest of the data is usually encrypted, but by changing a field in the chunk headers from 1 to 0 it works fine without the encryption. All numeric fields in the file and chunk headers use big-endian (or Motorola) format - Windows developers take note. Each chunk consists of a chunk header, the XML data, and some padding.

The file header is 16 bytes long:
Bytes 0-3: The text string "TiVo"
Bytes 4-5: (16-bit) Unknown - set it to 4
Bytes 6-7: (16-bit) Unknown - set it to 13
Bytes 8-9: (16-bit) Unknown - set it to zero
Bytes 10-13: (32-bit) The offset in bytes to the actual MPEG data
Bytes 14-15: (16-bit) The number of chunks - set it to 2

Each chunk header is 12 bytes long:
Bytes 0-3: (32-bit) The chunk size in bytes - more on that later
Bytes 4-7: (32-bit) The size of the XML file in bytes
Bytes 8-9: (16-bit) The chunk ID, which will be 1 or 2
Bytes 10-11: (16-bit) 0=unencrypted, 1=encrypted, so set it to 0

For some reason the first chunk has 4 zero bytes after it and subsequent chunks have 7 zero bytes of padding, so the size for chunk 1 is the header size+XML size+4 and the size for chunk 2 is the header size+XML size+7.

After writing the chunks, pad the output file with zero bytes until the file size is an even multiple of 1024 bytes, then pad it with another 1024 zero bytes. After that, append the MPEG file and you have a .TiVo file. I have no idea why any of the padding bytes are necessary but it seems to require them.

I have also noticed that if you transfer a show back to the TiVo that's in its current database it uses the some of the fields like description from the database, not the metadata once it's transferred. I'm pretty sure that's based on the unique episode ID, but more experimentation is probably needed.

I hope this helps somebody. I mostly use it to edit the airdate and duration on series shows like Good Eats so they show up in order by episode and have the correct duration after editing out the commercials, but a full-blown GUI program to edit or create metadata would be sweet if anyone wants to tackle it.


----------



## dlfl (Jul 6, 2006)

Nice work!

Am I correct that .TiVo files created this way can be shared among users since the MAK is no longer involved ?

I wonder if we now have a new MPEG-2-plus-metadata video file format standard? What are the competitors? Well I guess wtv and dvrms.


----------



## reneg (Jun 19, 2002)

Interesting. I wonder if this can be used to get episode number & genres metadata to display on files transferred back to the Tivo.


----------



## ggieseke (May 30, 2008)

dlfl said:


> Am I correct that .TiVo files created this way can be shared among users since the MAK is no longer involved ?


That would break TiVo's Terms & Conditions (which wasn't my intention), but yes. IMO it's hardly worth the effort since tivodecode, kmttg, VideoReDo & DSD could do that already. I just wanted to be able to organize my own files.

I'm kinda hoping that somebody with more .TiVo XML experience will run with it and come out with a really elegant metadata editor.


----------



## ggieseke (May 30, 2008)

reneg said:


> Interesting. I wonder if this can be used to get episode number & genres metadata to display on files transferred back to the Tivo.


Absolutely. I have 178 Iron Chef Japan episodes and 206 Good Eats episodes that are doing that right now, and I wrote my own XML files from scratch for those series.

I haven't tested it with pyTivo yet, but in my experience so far anything that transfers back using standard .TiVo files and Desktop transfers back using this method.


----------



## wmcbrine (Aug 2, 2003)

Here's some more information on the header, from Dan203:



Dan203 said:


> The byte where this information is stored is byte 7 (0 based index)
> 
> Bit 0 = Always 0
> Bit 1 = 0 for US TiVo files and 1 for AUS/NZ TiVo files
> ...


P.S. Dan's bit numbering here is high-to-low (backwards from what I'd use). Where he says "D = S3", this corresponds to what ggieseke is saying with "Bytes 6-7: (16-bit) Unknown - set it to 13".


----------



## ggieseke (May 30, 2008)

wmcbrine said:


> Here's some more information on the header, from Dan203:
> 
> P.S. Dan's bit numbering here is high-to-low (backwards from what I'd use). Where he says "D = S3", this corresponds to what ggieseke is saying with "Bytes 6-7: (16-bit) Unknown - set it to 13".


Thanks for the additional info! I get 13 (D in hexadecimal) from my S2DT TiVos, but the rest is interesting. I don't have a Series 3 - dunno what's up there.

4-13 is my birthday, so I thought it was just a cool coincidence.


----------



## Dan203 (Apr 17, 2000)

Excellent work! We might be able to do something with this in VideoReDo. Not exactly sure what yet, since it's still illegal for us to decrypt the original metadata, but perhaps we could use it to allow DVR-MS/WTV to TiVo conversions with intact metadata.



wmcbrine said:


> P.S. Dan's bit numbering here is high-to-low (backwards from what I'd use). Where he says "D = S3", this corresponds to what ggieseke is saying with "Bytes 6-7: (16-bit) Unknown - set it to 13".


You're right. When I wrote that I was reading the bits from a hex editor from left to right. My bad. 

Anyway the only important bit in that whole thing, at least for this program, is the SD/HD bit. If you set it to 0, which is SD, and the file is actually HD then S2 units will think that they are capable of transferring these files and will simply error out after the transfer starts. This may confuse users because the message provided by the TiVo is completely nondescript.

Dan


----------



## ggieseke (May 30, 2008)

Dan203 said:


> We might be able to do something with this in VideoReDo. Not exactly sure what yet, since it's still illegal for us to decrypt the original metadata, but perhaps we could use it to allow DVR-MS/WTV to TiVo conversions with intact metadata.Dan


Ya gotta love corporate rules.  It's legal to decrypt and edit the video, but not the metadata?

Maybe if you agree to NOT use the copyrighted TMS description it would be legal....


----------



## Dan203 (Apr 17, 2000)

It's more about the encryption then the actual data. We can access the unencrypted video using the DirectShow filter provided with TiVo Desktop. However they do not provide anything similar for accessing the metadata. We can't use tivodecode because it technically circumvents the encryption and as such is a violation of the DMCA.

Dan


----------



## reneg (Jun 19, 2002)

Played with your utility a bit and made myself a .tivo file from a small clip of an hour long show. First thing I ran into when transferring the file back was pyTivo crashing. Since it was a .tivo file that I was transferring, pyTivo tried to run Tdcat to get the metadata out of the file. I'm guessing Tdcat is assuming the .tivo metadata is encrypted and decrypted the clear text xml into garbage. [Edit] Ran tdcat on the .tivo file and it produced an empty file, not garbage.

I moved the tdcat utility so pytivo could not find it and transferred the file again. Second time it transferred. The duration was off but it played fine.

Next I changed the duration and recorded duration xml fields in the chunks and transferred a new .tivo file back. It worked exactly like you would expect which was cool.

Then I tried adding episodeNumber into the xml as the clip I was using didn't contain an episode number. Unfortunately, the episodeNumber did not show up on the Tivo. On the plus side, genres did display back on the Tivo.


----------



## reneg (Jun 19, 2002)

Following up after playing around some more. I got episode numbers to display back on the Tivo. I started with a recorded show on the Tivo that didn't have an episode number. I created the two xml chunks from the recorded program and created some tests.

First test was to add <episodeNumber> keyword to both xml chunks. There was no change in behavior on the transferred program. It did not display episode number.

Then I added <episodeNumber> keyword in both chunks and changed <uniqueId> keyword in chunk2 (not the seriesID uniqueId, but the longer one that has an additional two bytes at the end, I just changed these last two bytes). After these changes, the episode number displayed on the Tivo under both the program and info screen after the file was transferred back to the Tivo.

Furthermore, it appears that the Tivo is picking up data to display from the <showing> section of chunk 2.

I wonder if it would be possible for pyTivo to pull files as .tivo files and get complete metadata?


----------



## Dan203 (Apr 17, 2000)

pyTiVo should be able to be updated to support this new format. All they have to do is read that encryption bit and decide if the XML needs to be decrypted or not. However I'm not sure it's necessary. pyTiVo already has it's own external metadata format, so why would you want to go through the hassle of creating one of these modified .tivo files instead?

Dan


----------



## innocentfreak (Aug 25, 2001)

Dan203 said:


> Excellent work! We might be able to do something with this in VideoReDo. Not exactly sure what yet, since it's still illegal for us to decrypt the original metadata, but perhaps we could use it to allow DVR-MS/WTV to TiVo conversions with intact metadata.


I would definitely be interested in this. The transfer speeds of the Premiere have given me the itch to be able to transfer anything to the TiVo initiated from the TiVo. I looked briefly into it on the TiVo HD, but I couldn't handle the transfer speeds. Now I am just trying to figure out the best way to have my rips and that where I can pull them to the TiVo rather than streaming.

Does PyTiVo or VideoReDo currently have the ability to convert files in other formats/codecs to .TiVo files that the TiVo can see to pull preferably with guide data?


----------



## mattack (Apr 9, 2001)

I thought they did.. I've been playing with iTivo a bit lately, and it has the option to get metadata out. I just noticed something.. In the metadata for an episode of the tonight show, it ALSO put the metadata for the jimmy fallon show -- presumably because I padded the recording by one minute and so it technically now contains part of the jimmy fallon show.


----------



## reneg (Jun 19, 2002)

Dan203 said:


> pyTiVo should be able to be updated to support this new format. All they have to do is read that encryption bit and decide if the XML needs to be decrypted or not. However I'm not sure it's necessary. pyTiVo already has it's own external metadata format, so why would you want to go through the hassle of creating one of these modified .tivo files instead?
> 
> Dan


Because even with everything that pyTivo currently does, the metadata is incompletely displayed on the Tivo for files pulled back to the Tivo. Specifically, episode number and genres do not display.

I wasn't thinking of pyTivo creating a modified .tivo file, but rather feeding the Tivo a dynamically generated .tivo data stream on pulls of mpeg2's. pyTivo already deals with xml for metadata and transcoding files to mpeg2.


----------



## wmcbrine (Aug 2, 2003)

Yes, that's my plan. But I still have a few more things to figure out -- like, the bitstreamFormat field: What's _that_ about? Can we do without it?

I also suspect that this might be the only way to feed transport streams (hopefully to include H.264) to the TiVo, but I'm not sure yet.


----------



## ggieseke (May 30, 2008)

I leave out everything between </showing> and </TvBusMarshalledStruct:TvBusEnvelope> and it doesn't seem to affect anything, so I think we can do without bitstreamFormat. Some of the other fields like startTime, stopTime & expirationTime are ignored anyway on transfers. I suspect that the entire vActualShowing section may also be unnecessary since it has the wrong info if you pad a show early.

I'm attaching some typical chunks that I created from scratch. Episode number & genres display properly on my TiVos before and after transfer. I made up the episode number and the unique episode id since the data from TMS is incomplete for that series.

I _think_ that the unique episode id in chunk 2 may be the key to how TiVo determines if an episode is already in the NPL, but that's just a guess. From what I've seen it always starts with "EP", then the 6-digit series id, and 4 more digits. On some shows the TMS guide data seems to use the episode number for the last 4 digits (if it exists), and on other shows they just start at 1 and increment it with each new episode.


----------



## ggieseke (May 30, 2008)

Dan203 said:


> pyTiVo should be able to be updated to support this new format. All they have to do is read that encryption bit and decide if the XML needs to be decrypted or not.


A very minor patch to tivodecode would also fix it. Glancing at the source for tdcat, adding one 'if' near the bottom should work.

if ((o_chunk_1 && chunk->id == 1) || (o_chunk_2 && chunk->id == 2))
{
if(chunk->type == TIVO_CHUNK_BLOB)
{
prepare_frame(&metaturing, 0, 0);
skip_turing_data(&metaturing, (size_t)(chunk_start - current_meta_stream_pos));
decrypt_buffer(&metaturing, chunk->data, chunk->data_size);
current_meta_stream_pos = chunk_start + chunk->data_size;
}
if (fwrite (chunk->data, 1, chunk->data_size, ofh) != chunk->data_size)
{
perror("write chunk");
return 8;
}
}


----------



## reneg (Jun 19, 2002)

Just a bit of clarification on the 2nd <uniqueId> in chunk 2, I've seen it start with "EP", then the 6-digit series id, and 4 more digits and I've also seen it start with "EP", then contain 8-digit series id, and 4 more digits. At some point in the past, they ran out of unique 6-digit series id numbers.

An example from a recent episode of The Mentalist:

```
<uniqueId>EP010587140044</uniqueId>
```
Also, reformating the xml with whitespace doesn't seem to hurt anything, but helps readability quite a bit.


----------



## ggieseke (May 30, 2008)

reneg said:


> Just a bit of clarification on the 2nd <uniqueId> in chunk 2, I've seen it start with "EP", then the 6-digit series id, and 4 more digits and I've also seen it start with "EP", then contain 8-digit series id, and 4 more digits. At some point in the past, they ran out of unique 6-digit series id numbers.


Oops, forgot about that. I think that also explains why you had trouble getting the episode number to display until you changed the uniqueId. If that uniqueId was already in the database it would have pulled the data from its internal memory, not the metadata.


----------



## wmcbrine (Aug 2, 2003)

ggieseke said:


> I suspect that the entire vActualShowing section may also be unnecessary


Yeah, I already ditched that in pyTivo's TvBus responses (almost equivalent to .TiVo XML) -- I just send <vActualShowing/> (an empty section) now.


----------



## reneg (Jun 19, 2002)

ggieseke said:


> A very minor patch to tivodecode would also fix it. Glancing at the source for tdcat, adding one 'if' near the bottom should work.
> 
> if ((o_chunk_1 && chunk->id == 1) || (o_chunk_2 && chunk->id == 2))
> {
> ...


I needed another check to get this to work on the tivodecode-0.3pre source. Here is what I changed to get tdcat to work on encrypted & unencrypted tivo headers.

```
190c190
<         if (chunk->data_size && chunk->type == TIVO_CHUNK_PLAINTEXT_XML)
---
>         if (chunk->data_size && (chunk->type == TIVO_CHUNK_PLAINTEXT_XML) && (chunk->id == 3))
199a199,200
>             if(chunk->type == TIVO_CHUNK_ENCRYPTED_XML)
> {
203c205
< 
---
> }
```


----------



## Dan203 (Apr 17, 2000)

wmcbrine said:


> the bitstreamFormat field: What's _that_ about? Can we do without it?


I just decoded 3 different files. A regular PS file, a TS file from a Premier and an H.264 TS file from a NZ TiVo and all 3 had the exact same value for the bitstreamFormat. So I don't think it has anything to do with the video codec or the file type.

Dan


----------



## Dan203 (Apr 17, 2000)

FYI I had a sample TS .tivo file from a NZ user which was broadcast in H.264 with AAC LAOS audio. I decrypted the video manually using GraphEdit (similar technique to DSD) and then decrypted both chunks of metadata using tdcat. I then combined it all using the mktivo program from this thread and modified the TS and HD bits manually in a hex editor.

The resulting file played perfectly in WMP. So I tried to pull it to my S3 unit but it failed. I then tried pulling it to my Premier unit and it actually worked! Well kind of. I could play the file and see the video, but I could not hear the audio. Not sure why. The video does have a minor glitch at the beginning of the audio when played on a PC so perhaps that's screwing up the TiVo.

After that I tried to create a TS file myself and attach the same header, but that one crashed WMP and would not transfer to the TiVo. I even tried simply remuxing the NZ file and reattaching the header to that, but again it crashed WMP and refused to transfer. I'm guessing there is something in the TS stream itself that TiVo requires. I have DanR looking at the files to see if he can figure out what that might be. (he's the TS expert) If we can figure it out then maybe we can create H.264 TS files in VRD, attach one of these fake headers, and end up with an H.264 file that can be pulled to a Premier (and maybe a S3 after a software update) instead of having to use MP4s and pyTiVo push.

Dan


----------



## ggieseke (May 30, 2008)

Dan,

Sweet. Looking forward to any news even though I'm stuck in the S2 world.

P.S. I may have a way to get the metadata out of a .TiVo file for you without decrypting anything or breaking any DMCA rules. More on that later after some more programming and experimentation. VRD rocks!


----------



## reneg (Jun 19, 2002)

ggieseke said:


> Dan,
> 
> Sweet. Looking forward to any news even though I'm stuck in the S2 world.
> 
> P.S. I may have a way to get the metadata out of a .TiVo file for you without decrypting anything or breaking any DMCA rules. More on that later after some more programming and experimentation. VRD rocks!


Like allowing a user-defined pre-processing application to run?


----------



## ggieseke (May 30, 2008)

reneg said:


> Like allowing a user-defined pre-processing application to run?


I'm not sure if, or how it could be implemented in VRD - that's up to them.

I started with the assumption that if you could get the metadata out of Desktop itself then it would be legal, just like using the DirectShow filter to get the video. I took a program that I wrote a while back to dump the NPL of a TiVo and modified it to query TiVoServer. The basic HMO spec still applies, but the ports and connect strings are slightly different.

That got me the NPL from Desktop, and from its TiVoVideoDetails Url field you can get chunk 1, but the episode and series IDs needed for chunk 2 were still missing. Then I figured out the file format of Desktop's .cache files and that info was included, but there was no easy way to relate the physical file path stored in a cache file to a specific entry in the NPL.

I added code to list every file in the My TiVo Recordings directory including subdirectories and shortcuts, created a table of physical file paths and the details url that Desktop would use, then matched that to the NPL entries.

It's crude but effective, and it goes through nearly 500 recordings on my PC in less than a second. I still need to add the code to output the XML chunks for a particular file, but then I will have a way to get the metadata without decrypting anything or breaking any rules that I can think of. Of course, TiVoServer.exe has to be running and the file has to be in a directory that it can see, but IMHO that's a fairly minor limitation.

It's mostly just an exercise in curiosity at this point, but it could easily be rewritten into a DLL that other programs could use.


----------



## Dan203 (Apr 17, 2000)

I've actually investigated that path before (i.e. using TiVoServer) and found two problems with it...

1) TiVoServer had to be running, which is not a requirement for simply opening the file.

2) TiVoServer only contains data for .tivo files that are actually stored in the My TiVo Recordings folder. Again, not a requirement to simply open the file.

It's a neat solution for an indie project where most users will understand how it works and it's limitations, but for VideoReDo it would be a support nightmare.

As for the TS files... I think I figured out what's unique about them. However we need to make a change to our TS muxer to generate compatible files, and that's DanR's department, so it could be a few days before he can make the changes and I can test it out to see if it actually works. 

Dan


----------



## lucas_nz (Sep 27, 2010)

Dan203 said:


> As for the TS files... I think I figured out what's unique about them. However we need to make a change to our TS muxer to generate compatible files, and that's DanR's department, so it could be a few days before he can make the changes and I can test it out to see if it actually works.


I've been playing with creating mpeg-ts files with h264 video and encapsulating them in an unencrypted .TiVo file. This all works fine with files I download off the Tivo and decode. But haven't been able to remux a mp4 and successfully encapsulate it. Any pointers on the tivo mpeg-ts format or tools that I could use to remux and h264 mp4 into an mpeg-ts that the tivo will accept?

Cheers,

Luke


----------



## Dan203 (Apr 17, 2000)

I couldn't get it to work either. I could get it to work with one decrypted TiVo-TS file I had from an AU user, but I couldn't figure out how to make my own from other H.264 files. Eventually I gave up and moved on.

Dan


----------



## lucas_nz (Sep 27, 2010)

I have a hacked version of pyTivo that lets me transfer decrypted mpeg-ts files back to my TiVo. Ive been trying to figure out whats special about these files. I made some scripts to modify the files and see if it plays on the TiVo. I haven't had much success so far but here's what I've tried:
Took an mpeg-ts recording from the TiVo with h264 video and tried the following:
1. I noticed that the decrypted mpeg-ts files from the TiVo had an extra data stream. I removed all packets with this stream id and transferred the video back to the TiVo. The video successfully played on the TiVo.
2. As well as 1, I modified all the PMT packets so that only the video and audio streams were listed in the PMT table. The video successfully played on the TiVo.
So the data track is not required.

3. I used vlc to remux my sample recording from the TiVo. Video would not transfer to the TiVo.
4. I noticed that the remuxed video had adaptation field spacing on the PAT and PMT packets (moving the tables to the end of the packet). I removed the adaptation field from all the PAT and PMT packets (and moved the tables to the start of the packet, following the packet header). Video would now transfer to the TiVo but the TiVo displays a message waiting for the video to download even after it had finished transferring it  this was the same for all the following test cases.
5. I tried playing with the PIDs of the various packets changing them to match my sample recording (and updating PAT and PMT tables). The video would not play on my PC.
6. I noticed that the stream type in the PMT for the audio streams on my sample was 0x11 but my remuxed video had stream type 0xF. I changed this on all PMT packets. This resulted in no audio when played on the PC.
7. I noticed that the stream id for the audio stream in my sample (in PES header) was 0xC1 (instead of 0xC0 in the remuxed video). As well as 6, I changed the PES header on the audio packets. Still no audio on the PC
8. I noticed that the PES Packet length (in the PES header) of the video streams is set to 0 in my sample but has an actual value set in the remuxed video. I changed this to 0 in the PES headers of the video packets. This resulted in scrambled video displayed on PC.
My test files: 
ad = removed adaption fields from PAT and PMT packets
PMT100 = set PMT packet ID to 100
AUDID = set audio packet ID to same as TiVo sample 323
PES0 = set packet length in PES header to 0 in video packets
PMTx11 = set stream type to 0x11 in PMT packets from audio streams (from 0x0F)
xC1 = set stream id to 0xC1 in audio packets (from 0xC0)
VIDID = set video packet ID to same as TiVo Sample 273
remux.ts - stopes transffering after a second
The following all seem to transfer to the TiVo but will not play but display a message waiting for the video to download
remux_ad.ts	- plays ok on PC
remux_ad_PMT100.ts	- plays ok on PC
remux_ad_PMT100_AUDID.ts	- plays ok on PC
remux_ad_PMT100_PES0.ts	- video scrambled on PC
remux_ad_PMT100_PMTx11.ts	- no sound on PC
remux_ad_PMT100_PMTx11_xC1.ts - no sound on PC
remux_ad_PMT100_PMTx11_xC1_PES0.ts	- scrambled video and no sound on PC
remux_ad_PMT100_VIDID_AUDID.ts	- will not play on PC
remux_ad_PMT100_VIDID.ts	- will not play on PC

I also noticed that video stream occur before audio streams in my sample recording but dont know if this matters.
I also noticed that audio packets are spread out fairly regularly in my sample recording. packet #33, 80, 132, 183, 234, 285, 337, 390, 442, 494, 547...
With a PES header packet length value about 2000
vs:
3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 80, 151, 164, 172, 179, 186, 195, 211, 219, 269, 308
With a PES header packet length value about 350

Dont know if this is significant.

Anyone got any ideas??


----------



## sparks1 (Nov 17, 2011)

Hi,

I am new to this, and had a few questions about the use of this .exe. I am using Win 7.

First, is this the current version of mkTivo? I didn't know if commercial programs, such as VideoReDo, extracts the header info (title, summary, series number, etc.) from the .tivo file and places it back in the converted mpeg?

I was not able to use the tools, such as tivodecode, since only source code was available, and currently I cannot compile anything, until I upgrade my system.

Has anyone solved the problem with the new .tivo files being read in correctly after the conversion using your utility? == you state that you think it's possibly due to the episode number laying resident on the tivo.

I just need to find a way of safely deleting .tivo files, and making sure I have all the data. I want to make sure that the closed captioning and metadata is preserved. I did experiemnt with DirectShow Dump Utility, and I know that the closed captioning is preserved.

Any insight would be great to make sure all header info is saved using any of the utilities presented on the forum

Thanks!



ggieseke said:


> I figured out a way to create .TiVo files that aren't encrypted so that you can edit the metadata. The actual metadata is in two XML blocks (hereafter referred to as chunks), that you can extract using tivodecode with the -D option. I will leave documenting the exact XML format to others. The basic format of the file header and chunk headers comes from the tivodecode source, and the rest from trial and error.
> 
> I'm attaching a very basic Windows command prompt exe for anyone that wants to try it. The rest of this post is probably of interest to developers only. The usage is as follows:
> 
> ...


----------



## dlfl (Jul 6, 2006)

mpeg2 files cannot contain metadata, thus no program such as VideoReDo can save a mpeg2 file with metadata in it. Only if both your input and output files are .tivo files, can VideoReDo preserve the metadata in the output file. The reasons for this were explained by dan203 earlier in this thread.

You can extract and save the metadata from a TiVo file in a text file (pyTivo format) using pyTivoMetaGen, link in signature, for Windows only.


----------



## ggieseke (May 30, 2008)

sparks1 said:


> Hi,
> 
> I am new to this, and had a few questions about the use of this .exe. I am using Win 7.
> 
> ...


It's the current version but I'm probably the only person that actually uses it. Most people either extract the metadata into a text file for pyTivo or just keep it in TiVo format with VideoReDo.

The compiled versions of tivodecode and a lot of other good tools are downloaded automatically by kmttg, which is worth having around in any case.

Closed captioning is encoded within the video itself, not the metadata.


----------



## sparks1 (Nov 17, 2011)

Thanks, *dlfl* and *ggieseke* for the replies to this thread. I was able to use kmttg, and create the text files with the metadata. I thought it might be useful to insert this data into the MPEG-2 format header so that it's all in one place. I know that the MPEG2 has specific formatted fields, but it may be useful so that the MPEG-2 contains this info, instead of having a seperate text file.

I will look into this later to see if this is indeed possible - there must be some padding in the MPEG-2file, where I can insert this info. Will investiage this further.

Thanks for the info - I did not know the CC info was encoded in the MPEG stream. Thanks!



ggieseke said:


> It's the current version but I'm probably the only person that actually uses it. Most people either extract the metadata into a text file for pyTivo or just keep it in TiVo format with VideoReDo.
> 
> The compiled versions of tivodecode and a lot of other good tools are downloaded automatically by kmttg, which is worth having around in any case.
> 
> Closed captioning is encoded within the video itself, not the metadata.


----------



## ggieseke (May 30, 2008)

While MPEG-2 does support the use of private streams and a private data field within PES packets, other programs would just ignore it. You can probably bury the metadata in the MPEG-2 file but it would take a lot of work and I don't see any advantage unless you plan to write your own media server or player.


----------



## Jim-Ti.Vo (Jan 24, 2012)

In 2010 and 2011, in this thread I saw a lot of discussion re TiVo metadata, especially from *ggieseke* and *Dan203* .. but nothing since Nov 2011. Is massaging metadata still an active interest, or what?

I recently upgraded from a *Humax* S2 to a *TiVoHD* S3; all of the programs on the S2 had arrived there via my broadcast rooftop antenna .. I don't do cable. I transferred all my recorded programs from S2 to S3 via the wireless network adapters. Several of the Grouped programs didn't group correctly on the S3, so instead of a Bones group, and a Closer group, (and several others), I've got a dozen Bones episodes and a bunch of Closer episodes appearing in the Now Playing list .. which I have to wade through to get to other programs. Gone, too, from the transferred programs' Info are: list of characters, production year, story summary, and episode number, but each does include the TMS stamp (where it didn't on the S2). Going forward, episodes now downloading to the S3 are grouping correctly and do contain the expected Info.

How to get those like-named programs into their Group?

TiVo Support suggested making sure the Grouping option is set to Yes (it was), then rebooting, and hope they grouped upon restart .. they didn't.

OK, you say, just view them, delete them, and get over it! Several of the other ungrouped sets of programs are of an instructional/reference nature that I want to retain indefinitely, and I'd sure like to find a way to group them.

I've tried *WeaKnees*, *DVRStore* (where I bought a 2tb replacement internal HD), *iSkysoft* and the *TiVo* Support team for either a service or a product that would enable editing the metadata and enabling the grouping - all to no avail. This is all I want to do to/with the files .. I have no intention to redistribute any of these files, don't want to circumvent copy protection nor steal copyrighted materials, have no interest in viewing them on my PC or MAC, no ripping, no dumping to a DVD, and no mobile interests.

So, guys, are you still trying to figure out how to edit these things? Have you, or do you know of who might have, developed a utility that can massage these things?

My preferred home computer is an Intel-based iMAC desktop (OS X 10.6.8), but I also have a couple of PCs (XP, Win7); I could use either to edit the metadata.


----------



## wmcbrine (Aug 2, 2003)

Jim-Ti.Vo said:


> Is massaging metadata still an active interest, or what?


No, it was completely solved, long ago. It's only marginally relevant to your problem, though.

What I'd do in your situation is to transfer all the programs from the S2 to your computer, and then, from your computer to the S3. If that doesn't preserve the grouping, then you'll want to try "pushing" the recordings to the S3. You can do that via auto-transfer folders with TiVo Desktop (though probably not TiVo Desktop for Mac), or the push menu of pyTivo.

Note, none of this involves editing metadata. You already have all the right metadata. The issue is getting the S3 to recognize that.


----------



## Jim-Ti.Vo (Jan 24, 2012)

Thank you, *wmcbrine*! I'm about to pursue your suggs, and I'll report back with the outcome.

Uh, my S2 is now disconnected from my network (boxed, in the attic), but I do still have it, and the misbehaving programs are still on its HD.

But all of the programs in question are now on the S3, just not showing Info correctly and not grouping. Do you suppose the sought metadata is there, just not being processed correctly by the S3? It'd save me a lot of time if I could copy the programs S3-PC-S3. If not, I can hook up the S2 again and do the S2-PC-S3 dance.


----------



## wmcbrine (Aug 2, 2003)

I don't think it's there, no. Of course, you could extract the programs to the computer, manually add a seriesId, and transfer them back. To do that, you either have to know the correct seriesId, or use the "push" method.


----------



## ggieseke (May 30, 2008)

I still fool around with it in my spare time, but there wasn't much interest and pyTivo has its own solution.

Right now I'm working on a program similar to WinMFS for S4s.


----------



## Jim-Ti.Vo (Jan 24, 2012)

My main concern is to reinstate grouping on the TiVo HD box (which I'm going to refer to herein merely as an S3, for simplification); consistency of episode metadata is secondary. The S3 now has a 2TB WD internal drive .. whew!

*wmcbrine*, I followed your sugg to use TiVo Desktop for PC .. invoked it to pull all 10 members from a Group ("Jerry Yarnell School of Fine Art") on my S2. I've not used TVD before, so don't know whether the programs were supposed to group on the PC, which they did not. Then I had the S3 pull them from the PC, hoping they would group - they didn't. I rebooted the S3, hoping the grouping would take place upon coming back up - they didn't.

I decided to analyze just what/how the metadata is revealed for the shows on both of the DVRs now holding a copy of those 10 episodes, S2 and S3. I've removed all but the oldest and newest episodes from the spreadsheet I used to hold the what was revealed for the episodes.

-( go look at the attached spreadsheet now )-

OBSERVATIONS:
1. The series' last broadcast date was 3/3/2012.
2. The series didn't include a Production Date with any of its episodes.
3. The only place grouping was accomplished was on the Humax S2.
4. It doesn't show in the SS, but the episodes' Summary was consistent across all three instances.

I now have two sets of each of the 10 episodes on the S3. I'm able to distinguish one set from the other because I specified each episode of the 1st set (the set originally transferred S2>S3) as "Keep until I delete," which causes TiVo to display a green ball with each episode, prior to transferring the 2nd set. Episodes of the 2nd set (PC>S3) have not been so specified = no green ball.

You also suggested "pushing" the episodes from the PC to the S3 via the TiVo Desktop. I can't see how to make TVD do that. Everything I try accomplishes only transfers from either of the TiVo DVRs to the PC, but I don't see a way to push from the PC to either of those devices. The version of TVD I have (just downloaded) is v2.8.3 .. maybe the TiVo gods have removed the PUSH capability in hopes of getting users to purch the TiVo Desktop Plus version.

And you also suggested using the pyTiVo utility. As I began to look into that, it appeared that it assumed a software development environment, with commensurate skills, to install, build, configure that. I am a retired software engineer, with development skills that topped out at the C level, and I am somewhat comfortable mucking around in UNIX, but I don't have a development environment set up here at home. I'm not confident I can get pyTiVo working on my XP Pro PC.


----------



## lrhorer (Aug 31, 2003)

Jim-Ti.Vo said:


> You also suggested "pushing" the episodes from the PC to the S3 via the TiVo Desktop. I can't see how to make TVD do that.


You can't. It requires purchasing the + version. I wouldn't bother.



Jim-Ti.Vo said:


> The version of TVD I have (just downloaded) is v2.8.3 .. maybe the TiVo gods have removed the PUSH capability in hopes of getting users to purch the TiVo Desktop Plus version.


To my knowledge it was never there. I believe it has always been only in +.



Jim-Ti.Vo said:


> And you also suggested using the pyTiVo utility. As I began to look into that, it appeared that it assumed a software development environment, with commensurate skills, to install, build, configure that.


Oh, not even. It's just a set of Python scripts. It will run on virtually any old computer, as long as Python and ffmpeg are installed.



Jim-Ti.Vo said:


> I am a retired software engineer, with development skills that topped out at the C level, and I am somewhat comfortable mucking around in UNIX, but I don't have a development environment set up here at home. I'm not confident I can get pyTiVo working on my XP Pro PC.


If you even know what C is or what `vi` and `ls` do, then getting pyTivo installed on any system is not even a tiny challenge. Unzip the archive, install Python and ffmpeg (if they are not already installed) and edit the pyTivo.conf file to point to the appropriate directories for ffmpeg and wherever your videos are. Then run `python pytivo.py` from the pyTivo directory.


----------



## Jim-Ti.Vo (Jan 24, 2012)

Thanks, *irhorer*, for the encouragement to pursue pyTiVo .. sounded pretty straightforward.

I apparently went to the wrong place to download it; pytivo.sourceforge.net/wiki/index.php/Windows_Install took me on quite a ride. There, I saw the urging for newbies to follow the *wmcbrine* fork .. which I did. I clicked on Current Releases, which took me to a page offering the "wmcbrine fork." Clicking on that paragraph's Compressed Package and Changelog link led me to where I got balled up in MediaFinder and github, not willing to work with my IE8 browser (IE9, Chrome or FireFox preferred), wanting me to create/login to an account. I started all over again with Chrome, then github wanted me to create a repository.

GOOD GRIEF! Is there no simple way to just download the files, edit the configuration file as suggested, unzip/install/run 'em?

My PC is an XP Pro (v2002, SP3); I got Chrome (v22.0.1229.94 m).

Turned out to be not so straightforward, after all. Where have I taken the wrong fork?


----------



## lrhorer (Aug 31, 2003)

Jim-Ti.Vo said:


> Thanks, *irhorer*, for the encouragement to pursue pyTiVo .. sounded pretty straightforward.


That's lrhorer, if you please, not irhorer.



Jim-Ti.Vo said:


> GOOD GRIEF! Is there no simple way to just download the files, edit the configuration file as suggested, unzip/install/run 'em?


Surely. William's fork is at https://github.com/wmcbrine/pytivo. Near the top of the page, over to the left is a button labeled *ZIP*. Click on that to download the zipped version. (Or you could just enter the url https://github.com/wmcbrine/pytivo/archive/master.zip.) Unzip the file into a convenient directory. William packages his fork with a main directory inside the archive, so ordinarily after unzipping the file there will be a single new directory inside the directory where you unzipped the files. Change to that directory to access pyTivo.



Jim-Ti.Vo said:


> Turned out to be not so straightforward, after all. Where have I taken the wrong fork?


Honestly, probably by starting where you did. Once you have the program unzipped and loaded both Python 2.X (not 3.X) and ffmpeg on your PC, then go and edit the pyTivo.conf file. I have included mine below for an example:


```
[_tivo_00000DEADBEEF]

[Server]
tivo_username = [email protected]
tivo_password = XXXXXXXX
tivo_mak = 5555555555
ffmpeg = /usr/bin/ffmpeg
ffmpeg_tmpl=%(ffmpeg_pram)s %(video_codec)s %(video_fps)s %(video_br)s %(max_video_br)s %(buff_size)s %(aspect_ratio)s %(audio_br)s %(audio_fr)s %(audio_ch)s %(audio_codec)s %(ffmpeg_pram)s %(format)s
ffmpeg_pram = -threads 16
togo_path = /RAID/Server-Main/Movies/TiVo_TS
debug = True
port = 9032

[_tivo_HD]
audio_lang = en, eng, 0x80, 5.1, ac3

[_tivo_SD]
audio_lang = en, eng, 0x80, 5.1, ac3

[RAID Server]
force_alpha = True
type = video
path = /RAID/Recordings

[DVD]
path = /RAID/DVD
force_alpha = True
type = dvdvideo
fast_index = true

[TTG]
force_alpha = True
type = video
path = /RAID/Server-Main/Movies/TiVo_MPG

[pyMusic]
type = music
path = /RAID/Music

[Unverified]
type = video
path = /usr/share/pyTivo/Unverified
```
This is probably a lot more than you might need, but it illustrates some key points.

The tivo_username and tivo_password lines are only required if you wish to push files to your TiVos (including the use of vidmgr), rather than pulling from the NPL of the TiVo. Pushing transfers h.264/MP4 files much faster than pulling the same show coded as MPEG-II. OTOH, h.264 files cannot be pulled natively to the TiVo. PyTiVo has to have ffmpeg transcode them for pulling, and that takes some time.

The ffmpeg_tmpl and ffmpeg_pram lines are only required if the default ffmpeg parameters built in to pyTivo won't serve your needs. Chances are they will, but if not, you can customize ffmpeg's behavior.

The togo_path line is only required if you wishto use pyTivo's TTG capabilities. You may choose kmttg instead, although nothing prevents you from using both.

The debug = True line offers more info on stdout from pyTivo, as well as more information in the <Info> screen of your Tivo when listing a video to be potentially transferred, but only if you do not have a Premiere. This feature is broken on the Premiere.

The [_tivo_HD] and [_tivo_SD] lines set up how the program handles various parameters for HD and SD material, respectively. The lines shown specify audio track handling.

Below the system parameters you can see the individual shares. Each line with a pair of brackets followed by a line that says "type = XXXXX" is a share of one sort or another. As you can see, there is more than one "type = video" share. Each of these will show up as a separate folder in the NPL with the name inside the brackets. Thus, on my TiVos, at the very bottom of the Now Playing List (My Shows on the Premiere) are four folders named "RAID Server", "DVD", "TTG", and "Unverified".

Next you will notice a share with "type = DVD". This requires the DVD plug-in originally created by thebayer and now maintained by lucasnz. It allows the user to rip DVDs directly to the hard drive and play them on the TiVo. It's configuration is far beyond the scope of this message, but suffice to say any DVD shares also show up in the NPL.

The "type = music" share should be fairly obvious, as would a "type = photo" share, if I had one. The only thing of note is these two share types do not show up in the NPL. They show up under Music, Photos, and Showcases, oddly enough. Creating a photo share requires the Python Imaging Library, in addition to Python itself.

If you like, once you have a basic config file set up with pyTivo running, the rest of the configuration can be done from a web browser of your choice. Just point the browser to the server running pyTivo on the port specified by the "port =" line. For example, in my case that might be:

http://RAID-Server:9032

or

http://192.168.1.50:9032


----------



## Jim-Ti.Vo (Jan 24, 2012)

.
Thanks again, *lrhorer*. I did get Python and pyTiVo installed and running on my PC - with your and *wmcbrine*'s patient help - thanks, guys!

*BROWSER CONFIGURATION TOOL*
I used this Port:9032 tool to set values in Global Settings only .. any need for the individual TiVo box settings?

*GROUPING*
I deleted all of the series-that-won't-group's episodes from the S3. I then pulled a couple of the non-grouping shows from my S2 (where they are grouped) to my PC. Then I pushed those two to my S3, where they still refuse to Group. Is there perhaps a pyTiVo.conf file setting I need to make to have the Grouping on the S3 take place? Should I check the [ Decrypt with tivodecode ] box when pulling from the S2 (I did with these 2, but don't know what that accomplishes)?

*THE METADATA OF THE TWO FILES*
Maybe something in the files' metadata will help; the 1st group was contained in "Jerry Yarnell School of Fine Art - 2835210.mpg.txt;" the 2nd group was in "Jerry Yarnell School of Fine Art - 2835210.mpg.txt"

seriesId: SH313732
seriesTitle: Jerry Yarnell School of Fine Art
description: Snowdrifts; campfires; silhouette shapes; sleet.
showingBits: 512
displayMajorNumber: 30
vProgramGenre: Fine Arts
vProgramGenre: How To
partIndex: 4
isEpisode: true
callsign: WPBADT
episodeTitle: The Campfire
partCount: 4
episodeNumber: 504
title: Jerry Yarnell School of Fine Art
displayMinorNumber: 1
programId: EP3137320056
time: 2011-09-10T11:00:00Z

seriesId: SH313732
episodeNumber: 606
description: A picturesque scene on the Madison River.
title: Jerry Yarnell School of Fine Art
partIndex: 1
showingBits: 513
displayMajorNumber: 30
isEpisode: true
vProgramGenre: Fine Arts
vProgramGenre: How To
displayMinorNumber: 1
programId: EP3137320071
callsign: WPBADT
time: 2011-12-31T12:00:00Z
episodeTitle: Mist on the Madison
seriesTitle: Jerry Yarnell School of Fine Art
partCount: 4

*DOWNLOAD DATE STAMP, AND OTHER METADATA ISSUES*
When I transferred the errant programs directly from the S2 to the S3, the date stamp that displays in the S3's Now Playing List is the date of the original download to the S2. Examining the two metadata snippets above, using pyTiVo to pull programs from the S2 preserves the original metadata. ButUsing pyTiVo to push from my PC to the S3, _today's_ date is what appears in the S3's NPL.

When the programs had been transferred S2>S3, although the grouping didn't happen on the S3, much of the episodes' Info did appear. With the pyTiVo push from PC>S3, very little of the metadata appeared.

So, the questions for both paragraphs of this section are:
1. *Is there a way to preserve or edit the metadata before pushing it to the S3*?
2. *Do we have any control over which metadata fields TiVo picks out to show for program Info*?
.


----------

