# Dumping mempool to /tmp/BlockFailure.258



## texster (Mar 11, 2005)

> Dumping mempool to /tmp/BlockFailure.258


This is the opening line of the serial log (attached) when my SDDVR80 reboots - which is now an hourly occurance.

Does this suggest a memory problem or hard drive failure?

This unit was zippered two years ago, and slicered to 6.2a last spring.
It is currently superpatched and bufferhacked. It was "enhanced" with TWP and Hackman, but I removed those in attempt to stop the reboots.

I have attached to 2 logs of 2 consecutive reboots.

Any help would be greatly appreciated.


----------



## BTUx9 (Nov 13, 2003)

Neither of the logs is showing which process is sending the "dumping mempools" message, but they both indicate TvRecorder strayed

Is there ANYTHING you've modified on the system, recently (coinciding with the reboots)?

Some ideas to try:

something may be running out of memory
try the command "free", and see if your swap is working properly

Extraneous IR noise can perturb 6.2x systems, so if you have a laptop or IR blaster, you may want to move it

you may also want to try returning to the stock tivoapp, to see if the reboots go away (though if it's been stable for many months, and you haven't made any changes, it's unlikely to be the cause -- barring corruption on the drive)

since you have multiple dtivos, you may want to try that drive in a different box, to narrow it down further


----------



## texster (Mar 11, 2005)

Thanks, Btux, for answering so quickly.



> Neither of the logs is showing which process is sending the "dumping mempools" message, but they both indicate TvRecorder strayed


Those logs were from the serial output. Which log file might show that? Something in /var/log?



> Is there ANYTHING you've modified on the system, recently (coinciding with the reboots)?


 About 2 weeks ago I did update to TWP to the latest 2.0 release. The previous TWP had been from whatever Russ used for his script, I think it was a modified mar'07 release. At that point I also added hackman. I did this on another of my units which is running fine. At the time, I was having problem with my Router's static ip reservations, so I also added the netconfig module. I uninstalled/deleted all those today.



> try the command "free", and see if your swap is working properly


What is the syntax for that command? Just typing "free" at bash?

Since my original post, I did a reset of thumbs and suggestions. I then immediately did a reset of program info and To Do, this sent the unit into permanent loop: I get the "powering up screen", then the "just a few minutes more minutes" screen, then the "clearing program info... this may take an hour" screen , then back to the "powering up" screen. I have attached the serial output from this loop. In terms of Serial Output, it "hangs" at "Checking for Database conversions"


----------



## BTUx9 (Nov 13, 2003)

looks like you may need to pulll the shows off that machine, then reimage

You'll need to connect via serial bash early enough to stop rc.sysinit before it gets to the point that it reboots (not sure if a zippered machine loads up serial bash early enough to allow that)


----------



## texster (Mar 11, 2005)

Does Kickstart 52 replace the kernel? Would rc.sysinit.author be overwritten? Would serial & telnet become disabled? Would it recognize the larger than stock drive?

If afterwords, it rebooted succesfully into a new (unhacked) 6.2a with recordings intact that actually wouldnt be too bad - then I could just rehack.

I fear the worst case where: it reinstalls, wipes the hacks - disabling serial and telnet, and still stuck in a reboot loop. Or does that not matter at this point anyway? Is there nothing else left to try?

(Have already tired kickstarts 57 & 58 without success)


----------



## texster (Mar 11, 2005)

Well, Kickstart 52 didn't work either. It seems the start up is alway crashing during the "deleting program info" phase - regardless of kickstarts. There was clear and delete tcl once apon a time for series 1, that allowed you to adjust the "zap level".

Is there something in rc.sysinit that i could edit in the pc to revert the "zap level" to 0, the bypassing the script that seems to crash the startup?


----------



## texster (Mar 11, 2005)

As an update, I pulled the drive, and replaced with the original unhacked drive (running 6.0), and the unit started right up and has had no problems (i.e. rooboots) for the past 12 hours.

My thought now is to try to edit the hd in the pc to alter the "zaprequest" setting (or whatever triggers the "clear guide info" script) so that it won't run at bootup. 

Failing that, in the event that it is a memory problem, would it be possible to edit the rc.sysinit to get serial bash early enough to kill "unnecessary" processes?

Any thoughts?


----------



## Da Goon (Oct 22, 2006)

If you're referring to the ZapRequest script I'm thinking of, don't bother. The mfs values it's looking for aren't present on your box. Create /test.conf with a call to bash in it, and bash will be started almost immediately. skill -9 rc.sysinit will kill rc.sysinit and leave you with bare bash. Once you have bash, you can insmod usb drivers, and set up your network with ifconfig to get ftp access,etc if you need it to fix something. As long as rc.sysinit and/or tivoapp aren't running, you should still be able to poke around a bit. Just be careful doing anything that accesses mfs (tivosh, etc).

ex test.conf :

```
#!/bin/bash
export PATH=$PATH:/busybox
/bin/bash </dev/ttyS2&> /dev/ttyS2&
```
as soon as bash is running :

```
skill -9 rc.sysinit
insmod /lib/modules/usbcore.o
insmod /lib/modules/usbnet.o
ifconfig eth0 192.168.1.50 netmask 255.255.255.0 metric 1
tnlited 23 /bin/bash -login &
tivoftpd
```


----------



## texster (Mar 11, 2005)

Thanks DaGoon.

The Hard Drive in question (Maxtor Quickview 160 GB) passed the manufacturers diagnostic utility (SeaTools for DOS v2.07).

Using MFSTools 2.0, I ran the free command as per Btux, and got the following:



> total used free shared buffers​Mem: 769148 29532 739616 0 16488
> Swap 0 0 0
> Total:769148 29532 739616


From the logs I see:



> Adding Swap: 130044k swap-space (priority -1)


And, from tvlog:



> fsmem: pool is out of space, requesting 52 bytes
> fsmem: DumpArenaAndBlocks: total pool size: 528128
> fsmem: BT: ....(backtrace info)...
> fsmem: Current (total pool size = 528128, unused = 20):
> ...


Does this shed more light on whether or not this is a memory and or swap file problem?

If so, can it be fixed/adjusted in the pc (w/ mfstools?) or must it be done in the unit (via test.conf)? Or am I outta luck (at least as far as things that can be talked about here)?


----------



## texster (Mar 11, 2005)

Using to mfstools 2.0 to view /proc/meminfo:



> total: used: free: shared: buffers: cached:
> Mem: 787607552 34025472 753582080 0 16896000 5824512
> Swap: 0 0 0
> MemTotal: 769148 kB
> ...


1. Does this suggest that THERE IS a problem with the Swap -despite the log:


> Adding Swap: 130044k swap-space (priority -1)


2. If there is indeed no Swap, then does it not make sense that this unit is crashing at the point of clearing guide info?

3. If this is the case, is there any way to fix the Swap AND preserve recordings?

4. Would a backup/restore do it?

5. If so, how much fat32 space would be needed?

6. Is there a way other than backup/restore (since I only have 1 160gb tivo hd)?


----------



## BTUx9 (Nov 13, 2003)

If I understand what you did, all you're seeing is that the mfstools CD is running without swap (which would make sense, for a live CD)


----------



## texster (Mar 11, 2005)

BTUx9 said:


> If I understand what you did


I used mfstools to boot the pc, mounted /dev/hda9 on /mnt/var; ran joe to /proc/meminfo.

I assumed the meminfo reflected the state of the hd when it was last in the unit in the reboot loop, and not when running in the pc.

(Note to self, never assume  )

Since the crashes always occured during the boot Stage E_Preapplication scripts, which I _assumed_ meant Tivoapp wasn't started, I _assumed_ that the problem might have more to do with the swapfile than memory.

Obviously, I am in way over my paygrade, but I really am eager to learn this stuff.

Thanks Btux and Dagoon.


----------



## BTUx9 (Nov 13, 2003)

As I understand it (and I'm fairly sure it's correct), the contents of /proc aren't truly files... they aren't stored on a disc anywhere, it's a pseudo-filesystem that exposes some of the running machine/OS's internals... by making it appear as a filesystem, it allows the use of tools (like grep, etc.) to access the data.


----------



## Da Goon (Oct 22, 2006)

Mount your alternate root, fill it with all stock files, and just insert /test.conf to get you bash access. If it still crashes with all stock files, then the issue may be in mfs, which is less than easy to repair. In that case, dumping the recordings and reimaging may be in order.

I attached a simple tcl script to dump a gziped cpio archive of a stock root fs from mfs. I assume you've got other 6.2a boxes due to your signature. Run the script on one of your other boxes and it'll dump the gz archive into /var/temp. You need mfs_uberexport in your PATH for the script to work. Unpack it with *gzip -d filename.gz* then *cpio -id < filename* and it'll extract the files into dirs, /tvbin /bin, etc... You'll get a crapload of errors during the cpio action but that's normal. You'll need to dos2unix the script and of course fix your bootpage to boot from the alternate root. HTH


----------



## Da Goon (Oct 22, 2006)

BTUx9 said:


> As I understand it (and I'm fairly sure it's correct), ....


Quite correct.


----------



## texster (Mar 11, 2005)

No success yet getting bash using test.conf. (Haven't tried alternate boot yet)
Does this indicate anything:


> Starting rc.sysinit
> Running boot Stage A_PreKickstart scripts
> Scanning for configuration files
> /etc/rc.d/rc.sysinit: /dev/ttys2: No such file or directory


----------



## BTUx9 (Nov 13, 2003)

capital S... /dev/ttyS2


----------



## texster (Mar 11, 2005)

So far so good. The test.conf worked. Entering the skill -9 command was a bit tricky trying to time the boot process, I was either to earely (before rc.sysinit had started) or to late (the Stage E_Preapplication Scripts had started).
I have written out what I believe to be all the steps below. As of 18 hours ago, I have serial bash,telnet and ftp working.The screen shows "Almost There", and there have been no reboots. I am now at "Step 4".

Sorry to ask for all the hand holding, but I am already in the doghouse with the wife. 

*0. Get stock root fileysytem from mfs of an identical unit*
- Check that mfs_uberexport exists, and is in PATH (busybox in my case)
- Ftp dump_files.tcl to working unit run dos2unix
- Ftp the resulting archive GZcore-[XXX=software version].gz from /var/temp to pc
- (In this case it was GZcore-6.2a-01-2-351.gz; file size = 11.55 mb)

*1. Put the rebooting drive in pc and create test.conf* (via joe or whatever):

```
#!/bin/bash
export PATH=$PATH:/busybox
/bin/bash </dev/ttyS2&> /dev/ttyS2&
```
*2. Put drive back in unit and power up. As soon as bash appears:*

```
skill -9 rc.sysinit
```
(this may take a few tries to time it right)

*3. Enable networking i.e. telnet,ftp,assign IP address:*

```
cd /sbin
insmod /lib/modules/usbcore.o
insmod /lib/modules/usbnet.o
ifconfig eth0 192.168.1.50 netmask 255.255.255.0 metric 1
tnlited 23 /bin/bash -login &
tivoftpd
```
(note - I got "SIOCSIFMETRIC: Operation not supported" until i deleted "metric 1" from above)

*4. Mount alternate root, remove all files, and add /proc and /var *

```
mount /dev/hda7 /mnt
rm -rf /mnt/*
mkdir -p /mnt/proc 
mkdir -p /mnt/var
mkdir -p /mnt/dev
```
*5. Fill alternate root with stock files*
ftp the GZcore-XXX.gz to /mnt

```
cd /mnt
gzip -d GZcore-XXX.gz
cpio -id < GZ-core-XXX
```
*6. Copy test.conf and /dev/* from root to alternate root*

```
cp test.conf /mnt
cp -xa /dev/* /mnt/dev
```
*7. Disable Netfilter*

```
mv /mnt/etc/netfilter-enable /mnt/etc/netfilter-notenabled
```
*8.Put a killhdinitrd'd kernel in /dev/hda3 and /dev/hda6*
(For this I put a fresh copy of the 3.1.5 kernel from the lba48 boot cd into /var)

```
dd if=/var/vmlinux.px of=/dev/hda3
dd if=/var/vmlinux.px of=/dev/hda6
```
 *9. Flip Bootpage*

```
bootpage -P "root=/dev/hda7 dsscon=true console=2,115200 upgradesoftware=false" /dev/hda
bootpage -B 6 /dev/hda
bootpage -A 3 /dev/hda
```
*10. Set prom password to "factory"*
With tivo running (tivoapp not necesssary)

```
/tvbin/crypto -u -srp factory
```
(I will edit this post with corrections and amendments)


----------



## Da Goon (Oct 22, 2006)

OK, you're close. In my examples I'm assuming your current root is 4 and alternate is 7, change that of course if I'm wrong. You can mount your alt root whenever you want, telnet, whatever. You'll want to unpack your virgin filesystem on your alt root. Ex:

```
mount /dev/hda7 /mnt
rm -rf /mnt/* (to remove all your old root fs junk)
mkdir -p /mnt/proc /mnt/var (proc and var aren't included but they'll need a mount point)
cp GZcore-whatever.gz /mnt
cd /mnt ; gzip -d *.gz ; cpio -id < GZcore
```
Then put in test.conf and rename /etc/netfilter-enable to /etc/netfilter-notenabled (probably unnecessary but just in case). Put a killhdinitrd'd kernel in /dev/hda3 and 6 just in case. To flip the bootpage params I do it one at a time just to be sure.

```
bootpage -P "root=/dev/hda7 dsscon=true console=2,115200 upgradesoftware=false" /dev/hda
bootpage -B 6 /dev/hda
bootpage -A 3 /dev/hda
```
Before doing all of this, set your prom password. This can only be done while the tivo is running. This sets your prom password to factory. */tvbin/crypto -u -srp factory* This is good in case you screwup, you can change root/kernel bootpage params without pulling the drive. Here's a cut and paste from my box. Hit enter twice as soon as the tivo starts to boot up and you'll get this prompt over a serial connection. I'd do this if I screwed up partition 3/4 and needed to boot from partition 6/7.

```
Output enabled
Ram size = 32
Service number is 264XXXXXXXXXXXX.
What is password? factory
Build target  = TiVo/mips/brcm/rel
Build version = 2.6
     reset -
color_bars -
       msw - [ -32 | -16 | -8 ] <address> <value>
       msr - [ -32 | -16 | -8 ] <address>
      help -
     param - [ <new boot args> ]
   netboot - [ -skip ] [ -f <file> ] [ <boot string> ]
      boot - [ -skip ] [ -3 | -6 ] [ <boot string> ]
  ememtest -
  identify -
What? boot -6 "root=/dev/hda7 dsscon=true console=2,115200 upgradesoftware=false"
```
A very good safeguard against future F*ck-ups is to dd a good root partition over to your alt root as well as a good kernel, so you can toggle back and forth in the future using the prom menu and not having to pull the drive.  We all make a mistake some time.


----------



## texster (Mar 11, 2005)

Thanks again Da Goon. I have edited my previous post to reflect your remarks.

Regarding the Kernel, you said to copy it into both 3 and 6 "just in case". You mean in case the current kernel is some how currupted? Otherwise I would have expected just to copy the current active kernel from 3 to 6.

Likewise you said disabling netfilter would be "probably unecessary", but to do it "just in case". Are you assuming using the 7.2.2-oth kernel?

And as far as that goes, what is the preference now: using the 3.1.5 or the 7.2.2-oth kernel? The last time I asked, there was no need/advantage for the 7.2.2 on an SD dtivo running 6.2a, but opinions seem to vary on this. 

As far as setting the prom password: what qualifies as a "running tivo"? The "lobotomized" version I have now, or one with tivoapp up and running?

And finally, once you reboot to the alternate partition, do you get dhcp, or do you ifconfig through serial again?


----------



## Da Goon (Oct 22, 2006)

texster said:


> Regarding the Kernel, you said to copy it into both 3 and 6 "just in case". You mean in case the current kernel is some how currupted? Otherwise I would have expected just to copy the current active kernel from 3 to 6.


However it gets there, just have a good kernel in both locations so if you change boot params you'll be ok.



> Likewise you said disabling netfilter would be "probably unecessary", but to do it "just in case". Are you assuming using the 7.2.2-oth kernel?
> 
> And as far as that goes, what is the preference now: using the 3.1.5 or the 7.2.2-oth kernel? The last time I asked, there was no need/advantage for the 7.2.2 on an SD dtivo running 6.2a, but opinions seem to vary on this.


Renaming netfilter disables the firewall. Kernel version doesn't matter.


> As far as setting the prom password: what qualifies as a "running tivo"? The "lobotomized" version I have now, or one with tivoapp up and running?


It'll work over the bash prompt, at any stage.


> And finally, once you reboot to the alternate partition, do you get dhcp, or do you ifconfig through serial again?


Use ifconfig to manually set network params.


----------



## texster (Mar 11, 2005)

Well, the first attempt didn't quite work, it is stuck at "welcome, powering up".
On the upside, there was serial output.

It seems start up got stuck right before I would have expected rc.sysinit to start. The last two lines of output were:


> Freeing unused kernel memory: 64k freed
> Warning: unable to open an initial console


Any thoughts?


----------



## texster (Mar 11, 2005)

For anyone whos still listening, here's the serial out put


```
CPU revision is: 00005430
FPU revision is: 00005410
Primary instruction cache 32kb, linesize 32 bytes.
Primary data cache 32kb, linesize 32 bytes.
Linux version 2.4.20 ([email protected]) (gcc version 3.0) #22 Fri Feb 20 18:19
:25 PST 2004
Determined physical RAM map:
 memory: 04000000 @ 00000000 (usable)
On node 0 totalpages: 16384
zone(0): 16384 pages.
zone(1): 0 pages.
zone(2): 0 pages.
Kernel command line: root=/dev/hda7 dsscon=true console=2,115200 upgradesoftware
=false
Monotonic time calibrated: 81.00 counts per usec
Calibrating delay loop... 161.79 BogoMIPS
Contiguous region 1: 8388608 bytes @ address 0x80d00000
Contiguous region 2: 1048576 bytes @ address 0x81500000
Contiguous region 8: 10485760 bytes @ address 0x81600000
Contiguous region of 19922944 bytes total reserved at 0x80d00000.
Memory: 43820k/65536k available (1222k kernel code, 21716k reserved, 81k data, 6
4k init, 0k highmem)
Dentry cache hash table entries: 8192 (order: 4, 65536 bytes)
Inode cache hash table entries: 4096 (order: 3, 32768 bytes)
Mount-cache hash table entries: 1024 (order: 1, 8192 bytes)
Buffer-cache hash table entries: 4096 (order: 2, 16384 bytes)
Page-cache hash table entries: 16384 (order: 4, 65536 bytes)
Checking for 'wait' instruction...  unavailable.
POSIX conformance testing by UNIFIX
PCI: Probing PCI hardware
ttyS00 at iomem 0xb4100100 (irq = 79) is a 16550A
ttyS00 at port 0xbc010000 (irq = 133) is a unknown
ttyS00 at iomem 0xb4100140 (irq = 81) is a 16550A
ttyS00 at iomem 0xb4100120 (irq = 80) is a 16550A
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
Starting kswapd
Serial driver version 5.05c (2001-07-08) with MANY_PORTS SHARE_IRQ SERIAL_PCI en
abled
ttyS00 at 0xb4100100 (irq = 79) is a 16550A
ttyS01 at 0xbc010000 (irq = 133) is a unknown
ttyS02 at 0xb4100140 (irq = 81) is a 16550A
ttyS03 at 0xb4100120 (irq = 80) is a 16550A
Uniform Multi-Platform E-IDE driver Revision: 6.31
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
hda: Maxtor 4R160L0, ATA DISK drive
ide0 at 0x400-0x407,0x438 on irq 87
hda: 320173056 sectors (163929 MB) w/2048KiB Cache, CHS=19929/255/63
Partition check:
 hda: [mac] hda1 hda2 hda3 hda4 hda5 hda6 hda7 hda8 hda9 hda10 hda11 hda12 hda13
 hda14 hda15 hda16
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
PPP generic driver version 2.4.2
PPP Deflate Compression module registered
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP
IP: routing cache hash table of 512 buckets, 4Kbytes
TCP: Hash tables configured (established 4096 bind 8192)
ip_conntrack version 2.1 (512 buckets, 4096 max) - 152 bytes per conntrack
ip_tables: (C) 2000-2002 Netfilter core team
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
VFS: Mounted root (ext2 filesystem) readonly.
Freeing unused kernel memory: 64k freed
Warning: unable to open an initial console.
```


----------



## texster (Mar 11, 2005)

I notice that there is no /dev file created from the core dump. Would this prevent rc sysinit from starting?

I tried mkdir -p /dev , just for laughs, but still no joy. 

Does the PATH in test.conf need something else added?

Or is there something in rc.sysinit itself that needs adjustment?


----------



## texster (Mar 11, 2005)

I wonder if the /etc/fstab setting had something to do with it?

The unit I copied from had boot 3 & root 4 active, the unit I am trying to fix has 6&7 as alternate boot & root. I am assuming (I know, again) that the fstab should be the same. So I changed it from 4 to 7 yet that didnt work. 

But it did get me thinking (I know, again), my third unit has 6 & 7 active so maybe that would be a better core dump. I am trying that now but.......

.... a new development, suddenly I can no longer skill -9 on boot up. (Command: skill not recognized). I don't know what may have caused this, although it happened right after i changed the fstab. I have also working the prom menu hard as well.

Can I kill rc.sysinit another way?

Also, is it possible to cpio id from the lba48 boot cd?


----------



## BTUx9 (Nov 13, 2003)

texster said:


> Can I kill rc.sysinit another way?


In order of difficulty:
1) you could remove the & at the end of the /bin/bash line in test.conf... Without the &, processing of rc.sysinit won't continue unless you exit bash. Unfortunately, this wouldn't mount var, and would leave several other things uninitialized

2) given the section that's aborting, add a .sh file that alphabetizes just before it, with the contents "exit" (the files are sourced, so it'll exit rc.sysinit)

3) move your test.conf file (without the &) to the .sh filename from #2... this gets your environment set up better, and allows you to exit bash to continue/test the startup


----------



## texster (Mar 11, 2005)

BTUx9 said:


> In order of difficulty:
> 2) given the section that's aborting, add a .sh file that alphabetizes just before it, with the contents "exit" (the files are sourced, so it'll exit rc.sysinit)
> 
> 3) move your test.conf file (without the &) to the .sh filename from #2


I'm not sure I understand the .sh file. What is doing the alphabetizing, and what is being alphabetized?


----------



## texster (Mar 11, 2005)

Actually I think I have bigger problem, no commands are found from bash, not even a simple ls.

Edit: some commands work some dont. ls doesnt. cd does. mount does. umount doesnt.


----------



## texster (Mar 11, 2005)

I can cd into busybox, and then the commands will work, however, I can't cd/ busybox and skill -9 before it's too late in the process.


----------



## texster (Mar 11, 2005)

Actually, I had accidently changed the PATH in test.conf. Changed it back to "export PATH=$PATH:/busybox" and regained the commands at serial bash.


----------



## Da Goon (Oct 22, 2006)

Argh...I forgot about /dev. Maybe mkdir -p /mnt/dev ; cp -xa /dev/* /mnt/dev would work. You definitely need it. I had to do something like this a while back, but I honestly don't remember exactly how I went about all of it.


----------



## texster (Mar 11, 2005)

Well, good news and bad news. Good news is that the dev file allowed the "alternate" (built from core_dump) partition to boot into rc.sysinit. The bad news, is that it crashed at almost exactly the same spot as the original active partition.

They both get to about the same spot: end of stage_D pre mfs scripts / beginning of stage_E preapplication scripts. The "original" gets a little further, not only according to the serial output - getting to stage_E db conversions but also getting an additional screen ("this may take an hour"). Whereas the "new" active partition craps out at the end of stage_D after starting tv launcher with the "almost there" screen still up.

Does it suggest anything that both active and alternate crap out at the same spot?

I have attached the latest serial output in case I missed something.


```
CPU revision is: 00005430
FPU revision is: 00005410
Primary instruction cache 32kb, linesize 32 bytes.
Primary data cache 32kb, linesize 32 bytes.
Linux version 2.4.20 ([email protected]) (gcc version 3.0) #22 Fri Feb 20 18:19
:25 PST 2004
Determined physical RAM map:
 memory: 04000000 @ 00000000 (usable)
On node 0 totalpages: 16384
zone(0): 16384 pages.
zone(1): 0 pages.
zone(2): 0 pages.
Kernel command line: root=/dev/hda7 dsscon=true console=2,115200 upgradesoftware
=false
Monotonic time calibrated: 81.00 counts per usec
Calibrating delay loop... 161.79 BogoMIPS
Contiguous region 1: 8388608 bytes @ address 0x80d00000
Contiguous region 2: 1048576 bytes @ address 0x81500000
Contiguous region 8: 10485760 bytes @ address 0x81600000
Contiguous region of 19922944 bytes total reserved at 0x80d00000.
Memory: 43820k/65536k available (1222k kernel code, 21716k reserved, 81k data, 6
4k init, 0k highmem)
Dentry cache hash table entries: 8192 (order: 4, 65536 bytes)
Inode cache hash table entries: 4096 (order: 3, 32768 bytes)
Mount-cache hash table entries: 1024 (order: 1, 8192 bytes)
Buffer-cache hash table entries: 4096 (order: 2, 16384 bytes)
Page-cache hash table entries: 16384 (order: 4, 65536 bytes)
Checking for 'wait' instruction...  unavailable.
POSIX conformance testing by UNIFIX
PCI: Probing PCI hardware
ttyS00 at iomem 0xb4100100 (irq = 79) is a 16550A
ttyS00 at port 0xbc010000 (irq = 133) is a unknown
ttyS00 at iomem 0xb4100140 (irq = 81) is a 16550A
ttyS00 at iomem 0xb4100120 (irq = 80) is a 16550A
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
Starting kswapd
Serial driver version 5.05c (2001-07-08) with MANY_PORTS SHARE_IRQ SERIAL_PCI en
abled
ttyS00 at 0xb4100100 (irq = 79) is a 16550A
ttyS01 at 0xbc010000 (irq = 133) is a unknown
ttyS02 at 0xb4100140 (irq = 81) is a 16550A
ttyS03 at 0xb4100120 (irq = 80) is a 16550A
Uniform Multi-Platform E-IDE driver Revision: 6.31
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
hda: Maxtor 4R160L0, ATA DISK drive
ide0 at 0x400-0x407,0x438 on irq 87
hda: 320173056 sectors (163929 MB) w/2048KiB Cache, CHS=19929/255/63
Partition check:
 hda: [mac] hda1 hda2 hda3 hda4 hda5 hda6 hda7 hda8 hda9 hda10 hda11 hda12 hda13
 hda14 hda15 hda16
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
PPP generic driver version 2.4.2
PPP Deflate Compression module registered
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP
IP: routing cache hash table of 512 buckets, 4Kbytes
TCP: Hash tables configured (established 4096 bind 8192)
ip_conntrack version 2.1 (512 buckets, 4096 max) - 152 bytes per conntrack
ip_tables: (C) 2000-2002 Netfilter core team
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
VFS: Mounted root (ext2 filesystem) readonly.
Freeing unused kernel memory: 64k freed
Starting rc.sysinit
Running boot Stage A_PreKickstart scripts
Scanning for configuration files
bash: no job control in this shell
bash-2.02# Invoking startup scripts for:
    platform 'trinity'
    implementation 'Series2'
    implementer 'TiVo'
Releasing /initrd and clearing ramdisk, if they exist
warning: can't open /var/mtab: No such file or directory
umount: /initrd: not found
Activating swap partitions
Adding Swap: 130044k swap-space (priority -1)
Loading core system drivers
Parallel modem driver loaded, 1 ports starting at /dev/ttyS1
Loading ircatch
Found rev 'F' Si2433 modem on /dev/cua1
Checking for Kickstart panic signal
Running boot Stage B_PostKickstart scripts
Cleanup /dev/hda9 pass 1
ext2fs_check_if_mount: No such file or directory while determining whether /dev/
hda9 is mounted.
/dev/hda9 was not cleanly unmounted, check forced.
Inode 6153, i_blocks wrong 1154 (counted=1142).  Set i_blocks to counted? yes

Inode 6146, i_blocks wrong 7474 (counted=7464).  Set i_blocks to counted? yes

Inode 6152, i_blocks wrong 3534 (counted=3520).  Set i_blocks to counted? yes

Fix summary information? yes

/dev/hda9: 65/32768 files (23.1% non-contiguous), 34529/131072 blocks
Cleanup /dev/hda9 pass 2
ext2fs_check_if_mount: No such file or directory while determining whether /dev/
hda9 is mounted.
/dev/hda9: clean, 65/32768 files, 34529/131072 blocks
/dev/hda9 is clean after pass 2
Mounting /var
/dev/hda9 on /var type ext2 (rw)
Cleaning up files in /var
Checking space in /var
Mounting initial environment
Starting logging daemons
Found Silicon Labs "Si2433" modem, rev F, skipping modem patch
Scanning for phase1 repair scripts
Running boot Stage C_MediaInitialization scripts
Loading input section drivers
cobra module was successfully installed, LNBPA 0x10 LNBPB 0x16
Loading output section drivers
Splash the screen
Running boot Stage D_PreMfs scripts
Remote control is  TIVO
MFS partition on  /dev/hda10
Loading Trinity dssapp
Look for debug board
/tvbin/dssappAV: DSS Interface Version 1.24, compiled on Aug 11 2006
Loading irblast
Loading ideturbo
Loading fan
Loading therm
Loading TvBus router
Updating system clock
Time set to: Fri Jan 25 04:58:03 2008
Enabling local route
Setting TCP keepalive parameters
Checking for additional disk
Start fan control
First temperature parameters set:
  Terminal temp: 71
  Critical temp: 62
  Logging temp: 60
  Target temp: 50
  Lowest fan speed: 7
/tvbin/fancontrol is running in the background.
Starting TvLauncher
Tmk Assertion Failure: 
    Foreground, line 512 ()
Tmk Fatal Error: Thread TvLauncher <84> strayed!
Paste the following into a shell to get a backtrace...

bt -t /tvbin/tivoapp <<END_OF_BT
  read 0x2aaa8000 /lib/ld.so.1
  read 0x2ab04000 /lib/libutil.so.1
  read 0x2ab48000 /lib/libdl.so.2
  read 0x2ab8c000 /lib/libpthread.so.0
  read 0x2abe8000 /lib/libm.so.6
  read 0x2acb0000 /lib/libc.so.6
  0x0075d588 0x0075fbf0 0x00612f18 0x00403090 0x2acc13fc 
END_OF_BT

Tmk Fatal Error: Thread TvLauncher <84>: assertion failure
flushing ide devices: hda 
Restarting system.
```


----------



## Da Goon (Oct 22, 2006)

Sh*t. It's crashing when tivoapp is trying to first access mfs. I'd guess by the "database conversions" and "only take an hour" clues that somehow your box tried to update to new software and mfs got seriously munged in the process, or it's trying to run a 6.2x root with a 6.3x mfs somehow.

At this point the most difficult question to answer would be : why/how did it happen? The easiest question to answer : How to fix? - Yank the recordings off with mfs-utils if you want them, reimage the drive, and reinsert. You can use mfs-utils quite easily with the disk in a pc, look around on dvrpedia, I think someone posted a bit of a "how-to" there.

I'd be interested to see the output of

```
mfs_ls /SwSystem
mfs_dumpobj -r /SwModule
mfs_dumpobj -r /State/ServiceConfig
```
though I probably couldn't offer much advice if the output resembled what I'm thinking. Sorry.  If you really want to save the drive/recordings, let me know via pm. It may not be too "TCF-friendly" from here on out. But, on a better note, the troubleshooting steps you've used can be quite invaluable at other times, when the issue may not be as bad.


----------



## texster (Mar 11, 2005)

I had a different interpretation of those "clues". I thought it was the unit doing a level 2 clear and delete ie "guide data" . Which is the thing I asked the tivo to do and what it says on the screen (see attached) before it reboots, which I "assumed" was the direct cause of the loop.

So tivoapp IS involved at this stage. That leads to more questions (sorrry): In the original active root, in /tvbin there are: tiovapp, tivoapp.bufferhack.backkup, and tivoapp.original. 
Maybe it's worth restoring the original. Is that as simple as:
mv /tvbin/tivoapp /tvbin/tivaopp.whatever 
mv /tvibin/tivoapp.original /tivbin tivoapp
sync 
reboot
or are there more steps involved?

As far as the core dump, it was pulling from a superpatched unit. In the the resulting tvbin there is just tivoapp. Would that be the superpatched tivoapp or the original? If it is superpatched, then would copying over the original tivoapp to the coredumped tvbin be worth a try?

Also, would e2fsck have any effect?

Here is the mfs output:


```
Main-TiVo# mfs_ls /SwSystem
dir: fsid=2997 count=2
      fsid   type     name
      -----------------------------------
   1838294   tyDb     6.2a-01-2-351
   1838294   tyDb     ACTIVE
Main-TiVo# mfs_dumpobj -r /SwModule
SwModule 1838287/11 PRIMARY {
        ServerVersion[9]=2
        ModuleFile[19]=1838297
        ModuleRelease[18]=1
        ModuleVersion[17]=6.2a-01-2
        Name[16]=GZcore
        ServerId[8]=89631619
        Version[1]=1
        IndexPath[4]=/SwModule/1838287 /Server/89631619
}
SwModule 1838288/11 PRIMARY {
        ServerVersion[9]=2
        ModuleFile[19]=1838298
        ModuleRelease[18]=1
        ModuleVersion[17]=6.2a-01-2
        Name[16]=GZkernel
        ServerId[8]=89631625
        Version[1]=1
        IndexPath[4]=/SwModule/1838288 /Server/89631625
}
SwModule 1838289/11 PRIMARY {
        ServerVersion[9]=2
        ModuleFile[19]=1838303
        ModuleRelease[18]=1
        ModuleVersion[17]=6.2a-01-2
        Name[16]=utils
        ServerId[8]=89631627
        Version[1]=1
        IndexPath[4]=/SwModule/1838289 /Server/89631627
}
Main-TiVo# mfs_dumpobj -r /State/ServiceConfig
ServiceConfig 3243/10 PRIMARY {
        Version[1]=2660
        DataGroupList[17]=SC_standard BS_standard BS_standard_351 CP_Standard CR
_combo CR_standard CR-Standard-v90 CR_Standard_v90 CR_Standard-v90 CR_Standard_3
51 CR_Standard_390 CR_DTV_Season_Pass SC_tvguide SC-tvguide SC_tvguide_e13010_r1
3003_v214 SC-tvguide-e13010-r13003-v214 AF_standard SC_unid SC-unid SC-unid-e130
13-r13006-v7 SC_unif SC-unif SC_unif_e13013_r13006_v5 CR_DTV_Season_Pass CR_TiVo
_Paid_Programming_Season_Pass CR_TiVo_Paid_Programming_Season_Pass_v4 CR_dragonf
ly CR_dragonfly_beta MI_4_Brothers MI_40_Year_Old_Virgin_Week_1_v7 MI_40_Year_Ol
d_Virgin_Week_2_ver4 MI_Skeleton_Key MI_Wynonna MI_Wynonna_20050811 MI_healthcas
tpilot MI_ditv.VideoStore MI-ditv.VideoStore MI_ditv MI_ditv_VideoStore MI_ifc_c
ds_preview MI_healthcasts_PCP MI_healthcasts MI_ifc_cds_preview MI_Showtime_Peek
 MI_ MI_Demo MI_intel_demo MI_seeshandtvg MI_seeshowcases MI_waldorf_test MI_dit
v.VideoStore SC_chevy SC_chevy_e13027_r12996_v24 SC-chevy-e13027-r12996-v24 SC_c
hevy-e13027-r12996-v24 SC_chevy_e13027_r12996 SC_directvb SC_directvb_e13026_r13
007_v2893 SC_directvc SC_directvc_e13010_r13006_v2608 SC_directvd SC_directvd_e1
3016_r13003_v2642 SC_directve SC_directve_e13010_r13003_v1841 SC_directvf SC_dir
ectvi SC_directvi_e13010_r13006_v844 SC_dr_e13027_r12997_v7 SC-dr-e13027-r12997-
v7 SC_dr-e13027-r12997-v7 SC_dr_e13027_r12997 SC_gmc_e13027_r12996_v96 SC_gmc-e1
3027-r12996-v96 SC_ifc_e13010_r13004_v9 SC-ifc-e13010-r13004-v9 SC_ifc-e13010-r1
3004-v9 SC_ifc_e13010_r13004 SC_igdtv_e13009_r13001_v406 SC_igdtv-e13016-r13008-
v408 SC_igdtv_e13016_r13008 SC_onstar_e13027_r12982_v5 SC-onstar-e13027-r12982-v
5 SC_onstar-e13027-r12982-v5 SC_onstar_e13027_r12982 SC_saturn_e13027_r12982_v6
SC-saturn-e13027-r12982-v6 SC_saturn-e13027-r12982-v6 SC_saturn_e13027_r12982 SC
_testk_e13148_r12900_v43 SC_dr SC_testk-e13148-r12900-v43 SC_testk_e13148_r12900
 SC_gmc SC-gmc-e13027-r12996-v96 SC_gmc_e13027_r12996 SC_ifc SC_igdtv SC-igdtv-e
13016-r13008-v408 SC_onstar SC_saturn SC_testk SC-testk-e13148-r12900-v43 SC_tes
tn SC-testn SC_testn_e13148_r12878_v97 SC-testn-e13148-r12878-v97 SC_testn-e1314
8-r12878-v97 SC_testn_e13148_r12878
        SequenceCookie[18]=12345678
        DataGroupIntList[28]=9 14 34 39 66538 66563 68173 68172 90 98
        SwSystemName[16]=6.2a-01-2-351
        LastCallId[22]=1200358500
        ServiceStateExpiration[21]=618694
        ServiceState[20]=3
        IndexPath[4]=/State/ServiceConfig
}
Main-TiVo#
```


----------



## Da Goon (Oct 22, 2006)

It doesn't matter what has been done to a box, if you pull the GZcore archive out of mfs, it will always contain 100% stock files, so your box is still crashing with a virgin tivoapp. At this point, it seems like some severe mfs corruption has occurred. In light of the steps you've taken so far, I really doubt there's any hope of saving the box at this point.


----------



## texster (Mar 11, 2005)

Well, certainly not the desired result, but many valuable lessons learned.

1. Enable serial bash before rc.sysinit -via test.conf or whatever
2. Enable access to prom for easy boot flipping
3. DD a stable root to alternate root (would this be dd if=/hda4/* of=/hda7/
4. skill -9 to abort rc.sysinit

Should anyone ever stumble on this in the future I have updated the steps on post #18.

Many, many thanks to Btux9 and Da Goon for their patience and genorosity.

Now it's off to find a big ol' hard drive so I can go do the umentionable things.


----------



## Da Goon (Oct 22, 2006)

texster said:


> 3. DD a stable root to alternate root (would this be dd if=/hda4/* of=/hda7/


Close. dd if=/dev/hda7 of=/dev/hda4


----------

