# Backing up all those hacks



## PhilG (Jan 15, 2002)

Sorry it's taken a while but I did promise I'd report back on the steps I took to backup my Tivo hacks

For the uninitiated, Tivo will clear out /var/hack if it suspects that something "odd" is preventing it booting

This usually happens when the power is interrupted more than a couple of times, close enough together to prevent Tivo booting completely - which is what happened to me a few weeks ago 

SO

Now, I schedule THIS via cron - it runs every Sunday, early in the morning



> #! /bin/bash
> /var/hack/tar -cvf /var/backup/varhackbackup.tar /var/hack /etc/rc.d/rc.sysinit.author > /var/log/varhackbackup.log 2>&1
> cp /var/backup/varhackbackup.tar /var/backup/varhackbackup2.tar
> gzip -f1 /var/backup/varhackbackup.tar >> /var/log/varhackbackup.log 2>&1
> ...


Well, step by step here's what's going on



> /var/hack/tar -cvf /var/backup/varhackbackup.tar /var/hack /etc/rc.d/rc.sysinit.author > /var/log/varhackbackup.log 2>&1
> cp /var/backup/varhackbackup.tar /var/backup/varhackbackup2.tar
> gzip -f1 /var/backup/varhackbackup.tar >> /var/log/varhackbackup.log 2>&1


creates a tar file of /var/hack (including /etc/rc.d/sysinit.author, copies the tar file to a secondary backup tar file and then zips the original (the zipped version is MUCH handier to ftp around - see later



> mount -o remount,rw /
> 
> rm /var2/Ovarhackbackup.tar.gz-7
> mv /var2/Ovarhackbackup.tar.gz-6 /var2/Ovarhackbackup.tar.gz-7
> ...


mounts the filesystem read/write and creates SEVEN generations of the zipped tar file in /var2/ (this will NOT get wiped and seemed a pretty safe place to store the backup) - also moves the unzipped tar file to /var2/ as well



> rm /var/log/Ovarhackbackup.tar.gz-7
> mv /var/log/Ovarhackbackup.tar.gz-6 /var/log/Ovarhackbackup.tar.gz-7
> mv /var/log/Ovarhackbackup.tar.gz-5 /var/log/Ovarhackbackup.tar.gz-6
> mv /var/log/Ovarhackbackup.tar.gz-4 /var/log/Ovarhackbackup.tar.gz-5
> ...


belt & braces time - also puts seven generations of the zippedf file in /var/log/



> mount -o remount,ro /


remounts the file system read only

Lastly, as I could not make an automated ftp "put" from Tivo (the Tivo ftp client ALWAYS prompts for userid/password and doesn't use std input ) I run this occasionally from my pc



> echo
> ftp 192.168.0.222 < TivoBackup.inf


Where TivoBackup.inf contains


> me
> cd /var2
> lcd /!!installs/tivo/!backup
> bin
> ...


This copies the contents of /var2/ to c:/!!installs/tivo/!backup

SO, I now have seven generations of weekly backups in three different places

Let's see how long it is before I find a new way of losing all the data


----------



## PhilG (Jan 15, 2002)

PS Luckily when I did lose /var/hack, I did have a backup - it was 6 months old, but what the heck (I thought)

What I had forgotten is that I have now lost 6 months of tracker history 

This is why I thought a weekly backup was safe - even if no new hacks have been added

Phil G


----------



## mikerr (Jun 2, 2005)

Still don't know why you want SEVEN different backup sets though ?!


----------



## PhilG (Jan 15, 2002)

mikerr said:


> Still don't know why you want SEVEN different backup sets though ?!


Well, one is never enough and initially I was backing up daily, so seven was a weeks worth

Now I'm backing up weekly but didn't change the seven generations

I don't think you can ever have enough backups


----------



## nickf (Oct 12, 2001)

Nice work! I've just set this up.

Couple of extra bits of help for anyone else doing this.


my tar is in /var/hack/bin - not sure why there is a difference.
I had to create /var/backup, and /var2 - for the latter, I had to mount the filesystem as read-write.


----------



## Ian_m (Jan 9, 2001)

Thanks.

I modified it to not keep backups in log directory as these then appear under TiVoWeb log page.

I also placed the following for the last line.

date >> /var/log/varhackbackup.log 2>&1

to record the date the script completed.


----------



## PhilG (Jan 15, 2002)

Ian_m said:


> Thanks.
> 
> I modified it to not keep backups in log directory as these then appear under TiVoWeb log page.


Good spot - that's exactly why I put them there. So I could see it was backing itself up even when I am away from home (no ability to telnet or ftp in across the web, but I can always browse the logs  )

Actually, what I'd hope to be able to do was download those files by doing a right-click and save-as through Tivoweb - but I was forgetting that TW reads the file and returns it as a web page. Not so impressive when you're viewing a zipped tar file 

Ah well

Still glad to see folks are finding this a useful starting point for their own backups

Phil G


----------



## jp1p (Nov 6, 2008)

PhilG, hello I'm new to this site and new to Tivo hack, I just ran out of service and of course the economy is in shambles I have to resolve to hacking my system. I have a series 2 and would like to know if you could recommend or start me out on how to add or hack to my system to make it work better. your help is very much appreciated.


----------



## ColinYounger (Aug 9, 2006)

jp1p - you've strayed into the wrong area; we only have Series 1 boxes in the UK. You need the TiVo Underground section.


----------



## jp1p (Nov 6, 2008)

Thanks Colin


----------



## PhilG (Jan 15, 2002)

WARNING 

I just discovered that the /var/ partition seems to have a size limit

I have been accumulating backups weekly, heading for my (completely arbitrary) target of 7 backup generations, when odd things in Tivoweb started failing with "no room on device"

Then dailymail failed

This morning (early) my daily call failed 

All because I have been storing backups in the /var/log/ directory

I cleaned them out and now all is well

SO, the moral appears to be either

1. Keep only a few backup generations

or

2. Keep them somewhere other than /var/


----------



## ...coolstream (Dec 10, 2005)

What was the approximate size of the directory when things went wrong?


----------



## PhilG (Jan 15, 2002)

...coolstream said:


> What was the approximate size of the directory when things went wrong?


No idea, sorry - there are so many subdirectories under /var/ it'd be hard to calculate

The only thing I do know is that I had 5 or 6 5Mb backup tar files (on top of everything else) - and that was just about at the limit

Phil G


----------



## verses (Nov 6, 2002)

I've been playing with an automated hack backup system lately, along similar lines to what you've documented here. I have a server that is permanently on, so I just run a script on there, shortly after the backup one on TiVo runs, which copies the backup to a safe location.

As TiVo's ftp client won't accept parameters being passed to it in the traditional way you could possibly set up something using the 'expect' command. I've had a quick play and something along these lines should do the trick;


```
#!/bin/expect -f

### Variables ###
set host ftp_server              # IP address of FTP server
set name ftp_account_name        # FTP account username
set passwd ftp_account_passwd    # FTP account password
set backup name_of_file_to_ftp   # Full path and filename of backup file
set remote_dir backup_folder     # Location for backups on the ftp server

set timeout 20                   # Kills process if response takes too long
                                 #  may need to be greater value if backup
                                 #  file takes ages to ftp

### Main actions ###
spawn ftp $host

expect "Name"
send "$name\r"
expect "Password:"
send "$passwd\r"
expect "ftp>"
send "prompt\r"
expect "ftp>"
send "hash\r"
expect "ftp>"
send "bin\r"
expect "ftp>"
send "cd $remote_dir\r"
expect "ftp>"
send "put $backup\r"
expect "ftp>"
send "bye\r"
```
I would save the above as ftp_backup.exp (and make it executable - chmod 755 ftp_backup.exp) and replace the variables as appropriate.

Then at the bottom of the main backup script add a line that runs ftp_backup.exp.

DISCLAIMER: I don't have access to a convenient ftp server at the moment so this may or may not work. But the syntax is pretty straight-forward if you want to try it and iron out any issues 
Also different FTP servers may give slight variations in the prompts they use, so you may have to tell the script to expect "ftp%>" rather than "ftp>" for example or "Username" rather than "Name"

Cheers,

Ian


----------



## PhilG (Jan 15, 2002)

OK, so what does "expect" do?

and what are those "/r"s for at the end of each line??

Looks possibly interesting though

Phil G


----------



## verses (Nov 6, 2002)

The expect command basically pretends to be the end user. The basic commands are as follows;

The *spawn* command launches an application, in our case we want to ftp to our backup server. If you would normally type 'ftp 192.168.1.1' at a command prompt to manually connect to an ftp server - you use the spawn command to do that for you, eg 'spawn ftp 192.168.1.1'

When you manually run 'ftp 192.168.1.1' you get a load of text shown on the screen and then you are asked to enter a "Name" to log in that's how you know to type your username. So you use the *expect* command to tell the script to wait till the word "Name" appears on the screen before it does anything else, eg 'expect "Name"'

Once "Name" has appeared, expect needs to know what to do next so you tell it to *send* a response, i.e. what to write to the screen. In this case we want to write whatever your ftp account username is, eg 'send "username\r"'.

The '\r' tells expect that you want to send a carriage return. In other words when manually ftping you would type your username and then press enter, the '\r' is just telling expect to do that too.

After initially posting the expect script I've looked a little more at it and have a working solution. I need to tidy it all up, but once I've done that I'll post the details.

Cheers,

Ian


----------



## ColinYounger (Aug 9, 2006)

Ingenius, Ian!! :up:


----------



## verses (Nov 6, 2002)

ColinYounger said:


> Ingenius, Ian!! :up:




Right then I think I've got a 'definitive' solution (well, RC1 of a definitive solution )

Obviously PhilG's script is a VERY belt and braces approach; I personally just have a script that runs daily and keeps the last 7 backups. However for simplicity's sake I'll assume that everyone is using PhilG's script and base the 'expect' solution on the file and folder names shown in that.

I have two scripts (both need to be executable - chmod 755 _filename_), backup_hacks.sh and backup_ftp.exp, which I have located in /var/hack/scripts/; the first contains the normal backup steps (see PhilG's script), the second contains the 'automated' expect commands.

At the end of backup_hacks.sh I have added 2 lines, one that changes into the location where the backup is stored (it is easier to do this here than further down the process) and another that calls backup_ftp.exp and tells it the name of the backup file;

```
cd /var2/
/var/hack/scripts/backup_ftp.exp varhackbackup.tar.gz
```
I have then created backup_ftp.exp which contains the following lines;

```
/bin/expect -f

set host 192.168.1.200
set name username
set passwd password
set remote_dir backups

set backup [lindex $argv 0]

set timeout 90

spawn ftp $host
expect "Name"
send "$name\r"
expect "Password:"
send "$passwd\r"
expect "ftp>"
send "prompt\r"
expect "ftp>"
send "hash\r"
expect "ftp>"
send "bin\r"
expect "ftp>"
send "cd $remote_dir\r"
expect "ftp>"
send "mput $backup\r"
expect "ftp>"
send "bye\r"
```
You will need to change the following variables;
*set host* to the IP address of your FTP server
*set name* to the username you use to log into your FTP server
*set passwd* to the password you use to log into your FTP server
*set remote_dir* to the folder on your FTP server where you wish to save your backups, e.g. when you FTP to a server you may connect in C:\FTProot but actually want to save to C:\FTProot\tivo_backups, in that case just set the value to "tivo_backups" (without quotes).

NOTE: Not all FTP server software will give exactly the same prompts! Where mine shows "Name" when I need to enter my user name, another may show "Username" or "What's your name monkey-face?" or whatever the coder decided to put there. As such you may need to amend the various expect lines as appropriate.
NOTE: Expect doesn't have to match a full line. My login prompt shows "Name 192.168.1.1:" (where 192.168.1.1 is the IP address I'm connecting from) but expect will react to that when told to expect "Name".
NOTE: I've increased the timeout from my original example, as my backups are around 4MB and it was taking longer than 20s for the transfer to complete so expect was timing out too early. 90secs seems more than adequate for my needs but you may need to adapt for your own environments.

It's worth remembering that the ftp and tar commands are not on the TiVo by default and have to be sourced from elsewhere.

Right, that's enough waffle  Any probs, post back here and I'll try to help.

Cheers,

Ian

PS: When I said there was lots of tidying up to do, that was when I was planning on including my own backup script too. Obviously this doesn't look a lot different to what I posted yesterday


----------



## verses (Nov 6, 2002)

I've just realised that PhilG's script ftps all the contents of /var2/ across to his server, as such the line that calls the backup_ftp.exp script should be;

```
/var/hack/scripts/backup_ftp.exp *
```


----------



## PhilG (Jan 15, 2002)

Well, what do you know

Tivo just wiped /var/hack/ for some reason

Boy, was I glad of this backup script and schedule I implemented

Snag is, of course, I didn't include any instructions to perform a restore, so here they are

Assuming you follow my script, in /var2/ you will have a file caled varhackbackup2.tar

SO, to perform the restore you need to Telnet into Tivo, move that file back to the /var/ directory and "inflate" it (is that the correct term?)



> cd /
> cp /var2/varhackbackup2.tar /var/
> cpio -H tar -i < /var/varhackbackup2.tar


Following that a reboot puts everything back the way it was

Luckily (?) my Sunday backup was fine for a /var/hack/ loss on a Monday 

Hope this helps someone


----------



## PhilG (Jan 15, 2002)

One thing I would like to know though, is if I was stuck with just the zipped version of the backup, what are the switches to run gzip to get the .tar file back again?


----------



## PhilG (Jan 15, 2002)

PhilG said:


> Well, what do you know
> 
> Tivo just wiped /var/hack/ for some reason
> 
> ...


and just discovered, that as the /var/hack/ backup gets created into /var/backup/ if you forget to recreate /var/backup/ (as I did) then your backups will not restart


----------

