# Suggestions for the best way to align partitions on advanced format drives?



## CrashHD (Nov 10, 2006)

Any ideas for the most efficient way to align a S3 image to 4K sectors on an advanced format drive?

The current image has 512 byte bootstrap partitions. This is what throws it out of alignment. I tried manually building the partition map, putting in 4.0K bootstrap partitions, and dd'ing each partition over. I have not yet expanded, and the unit will boot up, but if I force a connection, or connect the antenna cable, it will reboot. I've tried a kickstart 57, no luck there.

There was an old beta version of MFSLive back in the day that had an option to do something about the bootstrap partition sizes, I think, but I don't know if that old beta could handle a S3.

Welcome any suggestions, Thanks.


----------



## ggieseke (May 30, 2008)

CrashHD said:


> Any ideas for the most efficient way to align a S3 image to 4K sectors on an advanced format drive?
> 
> The current image has 512 byte bootstrap partitions. This is what throws it out of alignment. I tried manually building the partition map, putting in 4.0K bootstrap partitions, and dd'ing each partition over. I have not yet expanded, and the unit will boot up, but if I force a connection, or connect the antenna cable, it will reboot. I've tried a kickstart 57, no luck there.
> 
> ...


Leave the first 64 sectors where they are. That's just Block0 and the Apple Partition Map, which is only accessed once during the boot process. If you mess with them it won't boot at all.

If you know how to edit the APM you can increase the size of the bootstrap partitions to 8 sectors and move the rest of them around to align the drive (assuming that there's enough space left at the end of the drive to do so).

For every change that you make you will have to copy the entire contents of that partition and all of the partitions that come after that physically to somewhere and restore the to their new locations. Not sure if it's worth the effort. If you move anything around I would focus on getting the MFS Media partitions aligned to 4K boundaries.

As jmbach so elegantly put it in his 3-4TB expansion thread, "it's not for the faint of heart".


----------



## jmbach (Jan 1, 2009)

Does WinMFS MFSInfo display a partition and zone map. If it does,can you post it. Might try to redo the whole procedure to make sure something did not go wrong with the copy procedure. If you think the change in size of the bootstrap partition is the problem, just change the size back to 512 in the APM and leave everything else where it is. Everything will still be 4k align.


----------



## CrashHD (Nov 10, 2006)

I'm not certain what even goes into the bootstrap partitions.

I've seen the old Series2 units not care whether that partition was 1 block or 8.










I manually partitioned the new 2TB disk, did not expand it, and only changed swap size, and bootstrap partition sizes. I used dd to copy every partition over from the original 250GB disc.

I have since tried kickstart 57, clear program info and ToDo list, and finally a Clear and Delete Everything. It still reboots if an update to tivo is attempted (which means it's presently stuck in guided setup).


----------



## CrashHD (Nov 10, 2006)

Whoops...that was the wrong image. I screenshot that one to use for partition sizes. I don't have one of the new disk until I put it back in the pc. Hopefully I'll find time tonight.


----------



## CrashHD (Nov 10, 2006)

I decided to just do this real quick. Here it is:


----------



## jmbach (Jan 1, 2009)

It looks fine. This is something I did with my OLED S3 and is still working just fine. I would run the manufacturers diagnostic on the drive. If it passes the long test, then recopy the partitions over. I assume the original drive works just fine in the unit.


----------



## CrashHD (Nov 10, 2006)

Original drive (250GB) works fine.

New drive (2TB) is a few weeks old, and had a week's runtime in a different tivo with a different image, without issues.

Checking the drive is a good idea. I have had bad drives before, right out of the brand new box.

This is something that should work in theory, is it not? Is there anything extra that needs to be done? With the new partitioning scheme, the MFS partitions are effectively 14 blocks further down than they were in the original. I don't know much of anything about dealing with MFS, short of manually manipulating partitions, and using tools such as winmfs or mfslive.


----------



## jmbach (Jan 1, 2009)

This should work in practice and in theory.


----------



## HerronScott (Jan 1, 2002)

I've got 2TB drives coming to replace the 1TB drives in my 2 S3 OLED TiVo's. Any simple instructions out there on doing this alignment or should I not worry about it and just use WinMFS to copy and expand? 

The original drive replacement was done with WinMFS about 6 years ago in case that makes a difference.

Mfsinfo (Drive 2)

Boot Page
Boot Page: root=/dev/hda7
Active Boot Partition: 6 Active Root Partition: 7
Backup Boot Partition: 3 Backup Root Partition: 4

MFS Super Header
state=0 magic=abbafeed
devlist=/dev/hda10 /dev/hda11 /dev/hda12 /dev/hda13 /dev/hda14 /dev/hda15
zonemap_ptr=1121 total_secs=1951665152

Zone Maps
Z0:	type=0
map_start=1121 map_size=1 backup_map_start=589822
next_map_start=263266 next_map_size=9 next_backup_map_start=589813
zone_first=1122 zone_last=263265 zone_size=262144 min(chunk)=262144
free=262144 checksum=c00f51ff logstamp=5475612 num_bitmap=1
Z1:	type=2
map_start=263266 map_size=9 backup_map_start=589813
next_map_start=263275 next_map_size=34 next_backup_map_start=589779
zone_first=589824 zone_last=217329663 zone_size=216739840 min(chunk)=20480
free=5672960 checksum=607382e logstamp=5556933 num_bitmap=15
Z2:	type=1
map_start=263275 map_size=34 backup_map_start=589779
next_map_start=217336832 next_map_size=1 next_backup_map_start=217926655
zone_first=263309 zone_last=589772 zone_size=326464 min(chunk)=8
free=139528 checksum=996a9dcd logstamp=5556933 num_bitmap=17
Z3:	type=0
map_start=217336832 map_size=1 backup_map_start=217926655
next_map_start=217598977 next_map_size=130 next_backup_map_start=217926525
zone_first=217336833 zone_last=217598976 zone_size=262144 min(chunk)=262144
free=262144 checksum=9bdc6cd1 logstamp=5475612 num_bitmap=1
Z4:	type=2
map_start=217598977 map_size=130 backup_map_start=217926525
next_map_start=217599107 next_map_size=34 next_backup_map_start=217926491
zone_first=217926656 zone_last=486544383 zone_size=268617728 min(chunk)=2048
free=8814592 checksum=e5a5cef logstamp=5556933 num_bitmap=19
Z5:	type=1
map_start=217599107 map_size=34 backup_map_start=217926491
next_map_start=486544384 next_map_size=66 next_backup_map_start=486546366
zone_first=217599141 zone_last=217926484 zone_size=327344 min(chunk)=8
free=290416 checksum=6d71ff58 logstamp=5556450 num_bitmap=17
Z6:	type=2
map_start=486544384 map_size=66 backup_map_start=486546366
next_map_start=0 next_map_size=0 next_backup_map_start=2863311530
zone_first=486546432 zone_last=1951665151 zone_size=1465118720 min(chunk)=20480
free=1465118720 checksum=7f3efd12 logstamp=0 num_bitmap=18

Partition Maps
#: type name length base ( size )
1 Apple_partition_map Apple [email protected] ( 31.5K)
2 Image Bootstrap 1 [email protected] ( 512.0 )
3 Image Kernel 1 [email protected] ( 4.0M)
4 Ext2 Root 1 [email protected] ( 256.0M)
5 Image Bootstrap 2 [email protected] ( 512.0 )
6 Image Kernel 2 [email protected] ( 4.0M)
7 Ext2 Root 2 [email protected] ( 256.0M)
8 Swap Linux swap [email protected] ( 128.0M)
9 Ext2 /var [email protected] ( 256.0M)
10 MFS MFS application region [email protected] ( 288.0M)
11 MFS MFS media region [email protected] ( 103.4G)
12 MFS MFS application region 2 [email protected] ( 288.0M)
13 MFS MFS media region 2 [email protected] ( 128.1G)
14 MFS MFS App by Winmfs [email protected] ( 1.0M)
15 MFS MFS Media by Winmfs [email protected] ( 698.6G)

Total SA SD Hours: 1040	Total DTV SD Hours: 908 76 % Free
Software: 9.4-01-2-648	Tivo Model: TCD648250B

Scott


----------



## CrashHD (Nov 10, 2006)

When I get a chance, I'll try again. It wouldn't be the first time that simply trying again made it work.


----------



## jmbach (Jan 1, 2009)

I think these shell scripts will work for you. The copy and rearrange script will copy as well as long as you type in the same order as you originally have.

The coalesce script probably will not work as it is for Premiere series and it will not touch partition 14.


----------



## CrashHD (Nov 10, 2006)

Thanks for those links. There's a lot of good info in those threads, I'll have to read them a few more times to let it all sink in.

I've gone over the new partition map, and even set up an excel spreadsheet to double check that each partition's lengths and base points lined up. I can't find an error or overlap in it, so I think my partition map must be good. I'm going to scrap it on the next try anyway. The first time I did this, I expanded the swap partition (2x). I should have left it the same. Then, if this works, I can copy back over to the original drive since it will still fit, and then that drive is my baseline for any future images I would need.


----------



## CrashHD (Nov 10, 2006)

I am barely shell script literate, but if I am reading this correctly, these scripts would automatically change an "Image" type partition of size "1" to a size of "8" (i.e. is exactly what I need) in the process of copying from to <dest device>. Is that correct?


----------



## jmbach (Jan 1, 2009)

Yes. It will copy, align, and trim the excess from the MFS partitions. (and rearrange as well if you desire)


----------



## CrashHD (Nov 10, 2006)

It appears to have worked. It was really pretty simple. I must have botched something when I manually copied it over the first time.

It's been running for an hour, has managed to successfully connect to tivo, hasn't crashed, so everything looks good. 

I'm going to test it for a few days, and then copy it back to the original drive (2TB drive has not yet been expanded). This will serve as my source drive for any future image needs, and also as a spare..

Thanks for the link to those scripts, it was exactly what I needed.


----------



## jmbach (Jan 1, 2009)

Congratulations on a job well done. Don't you love it when a plan comes together.


----------



## unitron (Apr 28, 2006)

Hey Crash, why are you not using the "optimized" partition layout that puts MFS Media partition #13 at base 64?


----------



## CrashHD (Nov 10, 2006)

I had several reasons at the time, although I'm not so solidly convinced it matters either way.

I think it might be irrelevant with drives of this size. On an original disk, it puts the OS partitions near the middle of the disk, with roughly 120 GB of MFS partitions before them, and roughly 120 GB of MFS partitions behind them.

With a 2TB drive, the OS partitions would have roughly 120GB of MFS partitions before them, and 1700GB of MFS partitions behind them. Even if I could get both original MFS partition pairs before OS partitions, they would still be closer to the beginning of the disk than the middle.

When I originally setup that image, I was considering pushing the dvr's limits of hard drive capacity, and anticipated a possible need to consolidate partitions.

I also think it's an antiquated practice. A dozen years ago, a dual tuner DVR with the slow hard drives of the time, and on an ATA33 interface no less, they needed all the help they could get to keep up, and every little bit mattered. With modern multi-terabyte SATA hard drives, 2 x a HD bitstream (19 megabits at max, IIRC) just doesn't pose the same relative load on the storage subsystem.


----------



## CrashHD (Nov 10, 2006)

jmbach said:


> Congratulations on a job well done. Don't you love it when a plan comes together.


Yes. Thanks for the plan!!


----------



## HerronScott (Jan 1, 2002)

jmbach said:


> I think these shell scripts will work for you. The copy and rearrange script will copy as well as long as you type in the same order as you originally have.
> 
> The coalesce script probably will not work as it is for Premiere series and it will not touch partition 14.


New 2TB drives came in last night and just finished running extended and short tests on both without any issues.

If I've sorted the mfsinfo.txt file correctly using the base column, the following should be the command that I plan to use unless someone indicates otherwise (need to confirm correct device names)

./tivo_copy_rearranged.sh /dev/xxx /dev/xxx 1 13 11 2 3 4 5 6 7 8 9 10 12 14 15

Scott


----------



## jmbach (Jan 1, 2009)

Looks like the current image has been expanded once already. So the question comes in on expanding further. Will expanding just add size to the existing partition or will it add another pair. It's been a long time since I upgraded my OLED S3 I cannot remember which happens. I think it will just add to the existing partition giving you more space. I am not sure if the unit will support another pair of partitions. 
If you have to coalesce partitions to add space without losing your recordings, then you will need a different order.


----------



## HerronScott (Jan 1, 2002)

I believe that WinMFS will expand it. I seem to recall that there were issues if there was additional space/partition beyond the 15th partition when expanded with other tools (MFSlive perhaps?). I'll confirm yay or nay when this is done.

I do have a question though as I've started the script and it indicated that the 2 Bootstrap partitions optimal size was 1 at the beginning when showing the source and destination APM layout. This doesn't mean that it's not going to resize them does it?

Scott


----------



## jmbach (Jan 1, 2009)

I believe it will resize them. We'll find out when it is done.


----------



## HerronScott (Jan 1, 2002)

Doesn't look it resized them. 










Am I miss-reading this? If not any thoughts as to why not and how I could get it to work? Or an alternative method for doing this?

Or should I just ignore it and expand the drive as is?

Scott


----------



## HerronScott (Jan 1, 2002)

I should mention that I used the ISO that cykotix created which already had the TiVoTools scripts on it. Not sure if that would make a difference.

CrashHD, you used the same script with your S3 and the bootstrap partitions were resized correctly?

Scott


----------



## HerronScott (Jan 1, 2002)

One more post. WinMFS was able expand the drive successfully and partition 15 is now showing as size of 1.6T.

Scott


----------



## jmbach (Jan 1, 2009)

You are correct in that it did not resize the partition. Did you change the order of the partitions or just copied over the same order.


----------



## HerronScott (Jan 1, 2002)

jmbach said:


> You are correct in that it did not resize the partition. Did you change the order of the partitions or just copied over the same order.


I copied them in the same order (or at least that's what I tried to do). If I did get the order wrong would that have prevented a resize? I had posted the partition map of the original drive in text form above.

Scott


----------



## jmbach (Jan 1, 2009)

I am wondering if by not changing the order at all it does not trigger a resize. I am not sure. I'll look at the scripts and see.
The other thing the script was supposed to do is round the partitions to a multiple of 1024 which it did not do either.


----------



## HerronScott (Jan 1, 2002)

Thanks for taking a look at it!

Scott


----------



## jmbach (Jan 1, 2009)

Not sure what happened with your copy. Looks like it just did a dd copy. So there are two things I would consider trying. If you used USB, try SATA. There other thing would be to try a slightly different order, say. 1 11 13 2 3 4 5 6 7 8 9 10 12 14 15. If all goes well you should see a size of 8 for the bootstrap partitions and the size of the MFS media partitions should be a multiple of 1024 which means they shrink a little. (Except for 15 which is already a multiple of 1024)


----------



## nooneuknow (Feb 5, 2011)

One thing I recall being a PITA about WinMFS was this:

Uncheck optimize = different layout/offsets than what the original (factory, or image from backup) was.

Check optimize = "Optimized" different layout/offsets.

WinMFS was made for the Series 2, and made assumptions/calculations/"optimizations" based on Series 2 TiVos, and the drive sizes at the time.

There's too much I just can't remember to say/contribute more. There was a point where I had to dig out Instant Cake, or image beg, for something WinMFS hadn't adulterated...

I'm just wondering if WinMFS automatically doing things the way it sees fit (without giving the option for "as-is") might have something to do with this...


----------



## HerronScott (Jan 1, 2002)

jmbach said:


> Not sure what happened with your copy. Looks like it just did a dd copy. So there are two things I would consider trying. If you used USB, try SATA. There other thing would be to try a slightly different order, say. 1 11 13 2 3 4 5 6 7 8 9 10 12 14 15. If all goes well you should see a size of 8 for the bootstrap partitions and the size of the MFS media partitions should be a multiple of 1024 which means they shrink a little. (Except for 15 which is already a multiple of 1024)


This was done using SATA connections for both drives (and they were the only 2 drives connected to the PC at the time).

I'll try again with the reorder you mention. I would expect the script to show the change in the destination APM structure information it displays before the copy starts since I believe that's the information used for the copy.

If that works, are there any negative impacts from having the partitions in a different order?

Scott


----------



## HerronScott (Jan 1, 2002)

nooneuknow said:


> One thing I recall being a PITA about WinMFS was this:


This copy/resize/alignment was not done with WinMFS but with a set of shell scripts written to try and automate this.

./tivo_copy_rearranged.sh /dev/xxx /dev/xxx 1 13 11 2 3 4 5 6 7 8 9 10 12 14 15

WinMFS only came into play to expand the new 2TB drive and I'm using it to report the MFSInfo information after the copy.

Scott


----------



## jmbach (Jan 1, 2009)

The destination APM should show the changes. And yes that is the APM that is used for the process. So if it is not different then let's rethink things before copying. There is no real difference in the order other than the order.


----------



## HerronScott (Jan 1, 2002)

OK, reran the script with a reorder of the partitions as recommended and it still reported the destination actual and optimal size as 1 for the bootstrap partitions as it did without the reordering the first time so I didn't continue as I'm pretty confident that the result would be the same.

When I did fdisk -l before running the script in order to positively identify the correct device names for source and destination drives, I noticed that it reported the following sector information for the 2TB drive.

Units = sectors of 1 * 512 = 512 bytes
Sector size (logical / physical) 512 bytes / 4096 bytes
I/O size (minimum / optimal) 4096 bytes / 4096 bytes

Does the logical sector size being 512 bytes cause any issues for the scripts?

Scott


----------



## jmbach (Jan 1, 2009)

No it does not. And I agree the result would be the same. Let me look at the scripts again a little more deeper to see where it is going astray. Could you send me or post the first 20 blocks of your drive. If you need some help with that then send me a PM. It would help figure things out as the scripts rely on certain things in the APM to make decisions off of.


----------



## HerronScott (Jan 1, 2002)

OK, trying to troubleshoot the scripts and in running the apm_get_optimal_size.sh script manually for partition 02, it returns 1. Looking at the script, it's going to do this if the partition type is not "MFS" or "Image".

From apm_get_optimal_size.sh

if [ $type == "MFS" ]
then
mod=$(($actual % 1024))
optimal=$(($actual - $mod))
printf "%i" $optimal
else
if [ $type == "Image" -a $actual -eq 1 ]
then
printf "%i" 8
else
printf "%i" $actual
fi
fi

When the TiVo_copy_rearranged.sh script returns the APM source and destination structures, it displays the types and on my drive it reports the following (I've included the partitions it lists as each type).

Apple_partition_map 01
Imagepartition_map 02 03 05 06
Ext2partition_map 04 07 09
Swappartition_map 08
MFSpartition_map 10 11 12 13
MFSe_Freetion_map 14
MFS 15

I assume in the If statement of apm_get_optimal_size.sh that == does mean it has to match exactly and not just contain either "MFS" or "Image" in which case only partition 15 is going to match and the rest are going continue to use the actual existing size.

Are these partition types something unique to the S3 OLED which is why his script didn't take them into account or something that was caused by the first copy and expansion with WinMFS from the original drive to the current 1TB drive (I suppose I could go pull the original drive out to check)?

Or is there an issue with his apm_get_partition_type.sh or the apm_read_string.sh that it calls (length is 32?)?

In any case, it doesn't look like the scripts will work for me without some modifications.

Scott


----------



## jmbach (Jan 1, 2009)

You are correct. If the partition type does not match exactly, the script copies as is. Sent you a PM


----------



## HerronScott (Jan 1, 2002)

jmbach said:


> You are correct. If the partition type does not match exactly, the script copies as is. Sent you a PM


Thanks for the PM on dumping those blocks. I sent them to you in e-mail.

Scott


----------



## jmbach (Jan 1, 2009)

It looks like when the drive was made the program had something in the buffer when writing the drive type label. Because of that, the program ignores adjusting the partition and just copies it intact. We need just zero out after the true drive type label. (Just don't go beyond the space allocated for drive type label) The scripts should then work as intended.


----------



## HerronScott (Jan 1, 2002)

It will be interesting to see if my second S3 OLED has the same issue. Both were upgraded at the same time with the same version of WinMFS (9.2 was the latest at that time).

And I didn't get your e-mail on how to clean these up.

Scott


----------



## HerronScott (Jan 1, 2002)

Success (or at least so I believe!).

The shell script did show the optimal size as 8 for the bootstrap partitions after the device type was cleaned up and here's the mfsinfo from the drive after it was copied with the shell script and then expanded it with WinMFS.










Drive is back in the TiVo and seems to be working fine so far.

Thank you so much for taking the time to identify the problem and basically fix it for me (while teaching me a little along the way)! 

I'll be doing my other S3 OLED in the next week and it will be interesting to see if it has the same problem.

Scott


----------



## jmbach (Jan 1, 2009)

It looks correct. The bootstrap partitions are now 8 blocks in size and the size of the MFS Media partitions have been trimmed of the excess blocks and now are a multiple of 1024 blocks. Should be smooth sailing from here on out.


----------



## CrashHD (Nov 10, 2006)

HerronScott said:


> I should mention that I used the ISO that cykotix created which already had the TiVoTools scripts on it. Not sure if that would make a difference.
> 
> CrashHD, you used the same script with your S3 and the bootstrap partitions were resized correctly?
> 
> Scott


My apologies for the late response.

Yes, it automatically adjusted the bootstrap partitions to 8 blocks for me.

I wasn't certain about the command syntax for reordering partitions, but they were already in the order I wanted them in on the source disk, so I ran the command in order
"./tivo_copy_rearranged.sh /dev/xxx /dev/xxx 1 2 3 4 5 6 7 8 9 10 11 12 13"
and that did what I needed to do.


----------



## HerronScott (Jan 1, 2002)

CrashHD said:


> My apologies for the late response.
> 
> Yes, it automatically adjusted the bootstrap partitions to 8 blocks for me.
> 
> ...


Thanks for replying! With Jim's help, we found it was some remnants of bad APM information on the drive type probably from the original WinMFS upgrade. Once that was fixed on the source drive, the script worked great.

Still waiting on downtime later this week to do the next one (I need to replace the caps in the power supply while I'm in there as a few are bulging but haven't impacted the P/S yet).

Scott


----------



## HerronScott (Jan 1, 2002)

HerronScott said:


> Still waiting on downtime later this week to do the next one (I need to replace the caps in the power supply while I'm in there as a few are bulging but haven't impacted the P/S yet).


OK, second S3 OLED had the exact same issue with the drive type values so I performed the same fix (exported first 20 blocks using iBored to file, made a backup copy, edited the file to eliminate the spurious characters in the drive type fields for partitions 2-14, wrote the file back to the drive).

Copy and 4K alignment from the 1TB to the 2tB using the TiVo_copy_rearranged.sh shell script worked fine then followed by expansion using WinMFS. TiVo is back up and running with lots of free disk space (and after getting the P/S capacitors replaced as well).

Thanks again Jim.

Scott


----------



## jmbach (Jan 1, 2009)

Most excellent. Since this might be an issue for more than just you, perhaps the scripts could be modified to work with issues as you had without modifying the APM.
Will see what the developer of the scripts thinks.


----------



## HerronScott (Jan 1, 2002)

jmbach said:


> Most excellent. Since this might be an issue for more than just you, perhaps the scripts could be modified to work with issues as you had without modifying the APM.


And I failed to mention this must have been a bug in WinMFS 9.2 which was used to copy and expand from the original 250GB drives to the 1TB drives for both S3 OLED's in 2008.

Scott


----------



## CrashHD (Nov 10, 2006)

I'm now trying to do this same process on another drive, and it is not resizing the bootstrap partitions this time.

I've attached a screencapture that I think confirms the theory. It looks like garbage in the "name" field of the partition table, not unlike when a text file has dos line endings, although not exactly the same thing, I suspect.


----------



## jmbach (Jan 1, 2009)

This is the same situation Scott had with his two drives. At this point, you will have to manually edit the partition entries to null out the extraneous characters in the partition type entries of each APM entry for the scripts to work.


----------



## CrashHD (Nov 10, 2006)

Yeah, that's what I did. It appears to be working.

The odd thing is, if I booted in mfslive and viewed the partition with pdisk, the info looked fine. Also if I viewed partition info in winmfs, it looked normal. Only the copy script showed the extra data.

Deleting a partition with pdisk, and then recreating the partition with the exact same parameters, removed the extra data.

I also had to shuffle some partition sizes. This drive had been previously expanded (all the way to the end), so there was not the unallocated 14 blocks I needed to extend the bootstrap partitions out to 8 blocks each. The drive had, however, been configured with 2GB swap space (not sure why). I reduced that to 1GB, then juggled to get free space at the end of the drive.


----------



## HerronScott (Jan 1, 2002)

CrashHD,

Thats's exactly the same thing that my 2 drives had! Do you know what version of WinMFS was used to copy/expand this drive?

Scott


----------



## jmbach (Jan 1, 2009)

Each of the initial partition type strings ends with a null before the extraneous characters start. Probably pdisk reads the string up to the first null and stops.  The scripts read the whole 32 bytes of data that is allocated to the partition type string and then does a compare.


----------



## CrashHD (Nov 10, 2006)

HerronScott said:


> CrashHD,
> 
> Thats's exactly the same thing that my 2 drives had! Do you know what version of WinMFS was used to copy/expand this drive?
> 
> Scott


I'm fairly certain it was the most recent version, as it was within the last year that I expanded that drive, and winmfs hasn't been updated in a long time.


----------



## marwatk (Apr 9, 2010)

Hi guys, just a quick note that I fixed the scripts to handle that case, it's a super simple manual change if you want to edit them live while doing it, otherwise you can redownload them from github.

In apm_read_string.sh, add this to the end of the last line (the one with the dd command):

```
| sed 's/\x0.*//'
```
Notice that the change starts with a pipe character. It should look like this when done:

```
dd if=$device bs=1 skip=$valuestart count=$length 2>/dev/null | sed 's/\x0.*//'
```
See the diff here:
https://github.com/marwatk/TivoScripts/commit/e6cfdd2915bb0e15a5e4489a231722bc0c7aabf2

Hope that helps!

-Marcus


----------



## CrashHD (Nov 10, 2006)

I've got another one giving trouble. This is the same problem I had the first time I tried to do this manually.

I copied the system over to the new drive. It will boot up just fine, but if the antenna is plugged in, it reboots. If I attempt to access the NPL, it reboots. The last time this occurred, it would reboot with the antenna connected, or when forcing a connection. The one thing common here is these things all need to read or write data into MFS. I'd see what happens when forcing a connection, but I'm tinkering with this while eating lunch, and will be back to work shortly.

I don't know that I'm going to chase down the issue on this one, as the recordings were copied off this box and it is not necessary to salvage them, but curiosity may drive me to poke into it a bit.


----------



## jmbach (Jan 1, 2009)

Does the original drive give it the same problems?


----------



## CrashHD (Nov 10, 2006)

That's where it gets dicey. 

I needed also to change the swap partition size, after copying (first copy) from driveA to driveB (realigning the partitions by changing the bootstrap partition sizes) I copied (second copy)driveB back to driveA, intending for the final (third) copy back again from driveA to driveB to be done with winMFS so as to create a custom swap size at that time. Something must have gone wrong on the initial copy from driveA to driveB, and because I did not boot test the drive before copying back, I've already overwritten my good source.

The recordings on that drive were expendable. I was going after this with the intent of preserving the recordings if it was possible with minor effort. This one is destined for ebay in the next month or two anyway.


----------



## jmbach (Jan 1, 2009)

To many steps before testing the results. I have been guilty of that before. 
If it is the same as your previous unit, use that original disk for the image and use the copy and rearrange scripts to place the image on the units drive. Then do a C&DE to marry the drive to the unit. If there is room left, then expand.


----------



## HerronScott (Jan 1, 2002)

Jim,

So I'm looking to go back and align my son's HD which I had upgraded from the original 160GB drive to a 2TB drive using comer's JMFS tool. I noticed that its partition order appears to be the default with just an added large 14th MFS media region 3 partition.

HD original 1,13,2,3,4,5,6,7,8,9,10,12,11
HD upgraded with JMFS 1,13,2,3,4,5,6,7,8,9,10,12,11,14

S3 OLED original 1,13,2,3,4,5,6,7,8,9,10,12,11
S3 OLED upgraded with WinMFS 1,13,11,2,3,4,5,6,7,8,9,10,12,14,15

I'm curious about 2 things when compared to the partitions on my 2 S3 OLED's and wasn't sure if you knew the answers. 

First would be it looks like the original S3 OLED partition order was the same as the HD so I'm assuming that when I used WinMFS to upgrade those to 1TB drives originally that it moved partition 11. I'm curious as to why it made the partition order change.

Second, I'm curious about WinMFS adding another MFS application region to go along with the added MFS media region when it expands a drive (although a small one) where JMFS did not. I'm curious about the reasons and wonder if there are any reasons with the expanded HD not having it.

Scott


----------



## jmbach (Jan 1, 2009)

Scott, 

Good questions. As far as WinMFS is concerned. It has an option when you copy the partitions to the new drive to optimize the copy. It has been a while since I played with WinMFS, but it moves the last partition to the front to try to keep the OS and MFS app partitions closer to the center of the drive for an optimized layout. At least that is the thought. 

JMFS actually creates two partitions, an app and a media partition and then coalesces it into one partition. This is because the Premiere units have a partition 14 already and since the MFS header only allows up to sda15 and any more are ignored, so JMFS has to coalesce the added partition pair so the unit will recognize and use the added space. 

Jim


----------



## nooneuknow (Feb 5, 2011)

WinMFS, if used to image from a file, will mess with the partition layout, regardless of the "optimize" option being checked. If actually uses a stock S2 layout if not optimized, and will make a S2 "optimized" layout if you check that option. This may not be 100% correct. It's what I was told when I inquired about why WinMFS wouldn't just restore the backup file to the drive as-is (as-was) for my TiVo HDs.

This is why I started only using WinMFS for copying from source to target drives, which IIRC doesn't mess with the drive layout. JMFS or any variant of dd or dd rescue will also leave the partitions in the layout they started out as. I would then use WinMFS only for operations that didn't have the the optimize function as an option (like to "supersize").

If all the images you have are molested by WinMFS, you either should get an unmolested image, or use instant cake to build a 1TB (max) drive, as a foundation to start with.

jmbach gave a good description of the specifics of what WinMFS tries to do, and why, as well as how JMFS operates.


----------



## HerronScott (Jan 1, 2002)

nooneuknow said:


> WinMFS, if used to image from a file, will mess with the partition layout, regardless of the "optimize" option being checked. If actually uses a stock S2 layout if not optimized, and will make a S2 "optimized" layout if you check that option. This may not be 100% correct. It's what I was told when I inquired about why WinMFS wouldn't just restore the backup file to the drive as-is (as-was) for my TiVo HDs.
> 
> This is why I started only using WinMFS for copying from source to target drives, which IIRC doesn't mess with the drive layout.


The 1TB drives were drive to drive copies from the original 250GB drives using WinMFS back in 2008 and I would not have selected optimized format unless it's the default.



nooneuknow said:


> If all the images you have are molested by WinMFS, you either should get an unmolested image, or use instant cake to build a 1TB (max) drive, as a foundation to start with.


Well given that the current layout on both S3 OLED's has functioned well since 2008 including a number of version upgrades from the 9.4 that was on them at the time of the upgrade to the current 11.0, I'm probably not going to worry about starting from scratch at this point.

Scott


----------



## HerronScott (Jan 1, 2002)

I completed the alignment on my son's HD today keeping the partition layout the same and using the script to first copy the drive to a spare 2TB drive which aligned and resized the partitions and then using the script to copy them back to the original drive (after verifying the temp drive booted and worked successfully).

It certainly appeared that the HD is noticeably more responsive to both my son and I after the alignment in displaying large lists of shows and in scrolling through upcoming shows using the channel up/down buttons while viewing the show information. I had really noticed this hesitation/slowness when I had originally upgraded it to the 2TB drive but wasn't sure if it was related to it being an HD (versus S3 OLED) or the larger drive or the advanced format.

Scott


----------



## nooneuknow (Feb 5, 2011)

HerronScott said:


> I would not have selected optimized format unless it's the default.


I think it is the default setting (checked).

But you seem to be done, and happy with your results. So, I'll leave the subject alone, until the next time it comes up.


----------



## telemark (Nov 12, 2013)

Blue pill folk might want to skip over this post, because there may be no way to fix.

Aligning the start of the partitions via the partition map is half the equation.
Better analogy, it's like the base of a pyramid. 
[every layer needs to use 4k blocks, but also depends on the layer below to be correctly aligned first]

For illustration, If you're building a drive image from scratch, you'd make sure every partition started on a even boundry. When you format the filesystems next, you would specify to the formatter the minimum block size should be 4k as well. Then for every application that managed it's own data structures on that filesystem, you'd have to leave hints to it, hey, this drive doesn't like <4k so keep your back and forth chatter to units of 4k.

Since we already have an image made for us from Tivo, we're inheriting and then propagating wherever that data is already within a partition since we're doing perfect bit/byte level copies.

This would probably be set "wrong" if the image was built before 4k drives existed.

Checking an S4 320GB image for example I get:


```
# dumpe2fs /dev/loop4  | grep "Block size"
Block size:               1024 (root)

# dumpe2fs /dev/loop7  | grep "Block size" 
Block size:               1024 (root)

# dumpe2fs /dev/loop9  | grep "Block size" 
Block size:               4096 (var)

# dumpe2fs /dev/loop14  | grep "Block size" 
Block size:               4096 (sql db)
```
That's the filesystem, then the database would need to be checked as well. Might be in here somewhere:
https://bugzilla.mozilla.org/show_bug.cgi?id=416330

Does any of this matter? Maybe not, I depends how often data is read / written off-alignment vs on-alignment, and disk I/O is dominated by MFS video anyway so that performance would eclipse Linux operations. The Linux bits were easier to check though and every little bit helps in the end.


----------



## nooneuknow (Feb 5, 2011)

telemark said:


> Does any of this matter? Maybe not, I depends how often data is read / written off-alignment vs on-alignment, and disk I/O is dominated by MFS video anyway so that performance would eclipse Linux operations. The Linux bits were easier to check though and every little bit helps in the end.


Even if it fails to produce a performance difference in use in a TiVo, being unaligned increases RMW (Read,Modify,Write) write operations. If everything about a drive, what is on it, and how the host device interacts with it made every read/write aligned, there would be no RMW write operations. Every unaligned write, resulting in it being a RMW, overwrites two whole adjacent underlying 4k physical sectors, if the write straddles two physical sectors, rather than one. It also increases the number of seek operations to complete the same operations.

If you are saying what I believe you are, you are correct. Aligning partition start boundaries is only half the equation, leaving the fact that TiVo writes in multiples of 512 byte sectors, rather than multiples of 4k.

The long writes, of long block sizes, of the type when writing the AV streams (which TiVo writes as regular data, rather than using the AV streaming feature), tends to only need the partition alignment, since it's running sequential long writes, that would straddle the physical adjacent consecutive sectors, anyway, as well as writing the whole 4k, anyway.

The partition alignment is even more critical for small writes, as in the things other than AV stream writes. Now factor in not writing in a minimum of 4k blocks, and things get worse.

The biggest inevitability many seem to completely overlook, is data fragmentation, and how it single-handedly creates an explosion of RMW operations, exponentially worsened if the boundary alignment is off.

Why do we care about how many RMW operations happen? It wears out the drive faster, even if your TiVo seems to be running at peak speed. Behind the scenes, inside the drive, the drive cache, with AF 512e (4k emulating 512) drives, is being used to hide the performance penalties. This is why drives with 32MB cache suddenly had 64MB cache. The extra was needed to mask the hobbled internal performance differences, caused by RWM operations. Masking them doesn't stop the drive from making excessive read/write/seek operations to deal with RMW operations.

There seems to be a consensus that modern drives are not holding up as well as many would expect. They should be getting more reliable, not just holding onto reliability, or losing it. RMW operations on platter drives have an effect similar to what happens inside SSDs when you run a disk defragmenter on them (which is a non-no, if you want them to last).

Now, take a standard "green" drive that comes with a built-in internal performance penalty (both in throughput and seeking performance), and use it in a DVR with 4 or 6 tuners. One might think that the aligned partitions in a Roamio will stop RMW operations from happening. This will only be true if/when TiVo stops treating AF 512e drives like native 512 byte sector drives. As soon as data fragmentation sets in, and writes can no longer be truly adjacent and sequential, the drive will start having to seek more, and this is what "wears out" a platter drive, more so than excessive rewrites, which also "wears out" sectors. Both types of "wear" are exacerbated by RMW operations.

I've posted this same info many times in the past, when AF drives hit the market. I was told a great many unkind things, and had to stop posting about it, unless somebody specifically asked about it. Even then, I was accused of over-thinking things, being overly-technical, being OCD, and inciting unfounded worries.

I've been watching for trends in drive image requests, especially those that were requested for use on a replacement drive. It seems like the 4 tuner Premieres have been "wearing out" at a pace which supports my early predictions of decreased drive life longevity.

Then again, when looking for patterns, you tend to find them, even if they aren't really there.

One thing I'd like to understand more about, is how, exactly, Seagate can claim their drives auto-align unaligned writes, eliminating the need to align them at all (partition-wise). I can't seem to find any published data to support this claim.

The other thing I'd like to understand better, is the first logical sector offset on drives, and if this has anything to do with AF, and what it's purpose is. I've googled to extremes on this matter, and can't find anything other than the fact that the offset exists. Does this mean a drive with a first logical sector offset of 16384 starts writing (block zero) at 16384 based on 512 byte logical units? Does it serve a purpose we should know about, and does the offset value mean anything to us when choosing a drive (one offset better than another)?


----------



## ggieseke (May 30, 2008)

On Series 4s the root partitions use 1K blocks. Series 3 and earlier models used 1K blocks on the /var partition as well. I don't think there's any practical way around it, but I doubt that it matters very much.

Aligning the partitions to 4K start addresses is probably as close as we're going to get to perfection.


----------



## nooneuknow (Feb 5, 2011)

ggieseke said:


> On Series 4s the root partitions use 1K blocks. Series 3 and earlier models used 1K blocks on the /var partition as well. I don't think there's any practical way around it, but I doubt that it matters very much.
> 
> Aligning the partitions to 4K start addresses is probably as close as we're going to get to perfection.


I'm inclined to agree. At minimum, I hope that some who figure aligning partitions is a waste of time (since they can't tell the difference, with the RMW penalties being masked by the drive), might see there's a bigger picture, and at least use aligned start boundaries, if they want to do what they can, to hopefully increase the drive lifespan, relative to what it would be without aligning them.

With the double-whammy of how much harder the drive has to work internally, just due to being an AF 512e drive, then compounded the additional TB/yr workload in, to support more tuners, it would seem entirely logical to say that those looking for a replacement drive, might want to think twice before picking a lowest-price drive, with a lesser warranty, especially when the difference can be as low as $5-$15.

As telemark has suggested (to me), it's probably best to lead with the warranty, then try to briefly explain why that warranty might pay-out, on drives that people often assume will only fail due to defects, or old-age (which is now more relative to other factors), when people ask "which drive?" or "will this drive work?". At least with the Roamio, this is as much as needs to be said, since manually aligning the partitions is not required, and the Idle Mode 3 Timer issue is a non-issue for Roamios.


----------



## HerronScott (Jan 1, 2002)

nooneuknow said:


> I'm inclined to agree. At minimum, I hope that some who figure aligning partitions is a waste of time (since they can't tell the difference, with the RMW penalties being masked by the drive), might see there's a bigger picture, and at least use aligned start boundaries, if they want to do what they can, to hopefully increase the drive lifespan, relative to what it would be without aligning them.


As I mentioned above, on the HD that we went back and aligned, it was noticeably faster in certain activities. This is certainly one of the reasons that I wanted to address this before I upgraded my 2 S3 OLEDs. The other reason was possible impacts of misalignment on drive longevity. The 1TB drives that I had initially upgraded to were 6 years old and still working fine (and still passed WD's tests) and I'd really like to see the 2TB drives last just as long. As an aside news that Comcast upgraded Augusta, GA to MPEG4 are disappointing for the life expectancy before we need to upgrade but I can only hope that since we're a small town of 25,000 that it will be a long while before that happens here.

Regarding early drive failures, it does tend to catch your eye when someone reports that their 2 year old 2TB drive has failed already.

Scott


----------



## nooneuknow (Feb 5, 2011)

HerronScott said:


> As I mentioned above, on the HD that we went back and aligned, it was noticeably faster in certain activities. This is certainly one of the reasons that I wanted to address this before I upgraded my 2 S3 OLEDs. The other reason was possible impacts of misalignment on drive longevity. The 1TB drives that I had initially upgraded to were 6 years old and still working fine (and still passed WD's tests) and I'd really like to see the 2TB drives last just as long. As an aside news that Comcast upgraded Augusta, GA to MPEG4 are disappointing for the life expectancy before we need to upgrade but I can only hope that since we're a small town of 25,000 that it will be a long while before that happens here.
> 
> Regarding early drive failures, it does tend to catch your eye when someone reports that their 2 year old 2TB drive has failed already.
> 
> Scott


I agree with you.

I was addressing the large number of pre-existing posts around the forum, where people report that their is no performance difference (even years later), sometimes complete with allegations that those wanting to align TiVo drives, or those considering buying a pre-imaged & aligned drive from WK or DVR_DUDE, are a bunch of OCD worry-warts, creating concern over things that they allege are of no concern, and writing alignment off as having no relevance for a TiVo drive, whatsoever.

Since I was one of the original ones to have concerns about this, I'm the one many of the aforementioned claims of having unfounded concerns are aimed at. So, I can't help but recall getting slammed and snarked at over the subject matter. This is why I've been watching what images are being requested, the specifics of why they need the image, what model TiVo it is for, and how long it took before they needed an image. The trends do seem to finally be revealing themselves. Premieres with 4 tuners, that have been in use for 2 years, seem to be eating drives, almost exactly as I predicted.

I usually prefer that I'm wrong, when I make such predictions, FWIW.


----------

