Saturday, October 22, 2011
Thursday, October 6, 2011
Run Log: 361.5 km in 51.9 weeks
Wednesday, October 5, 2011
Linux SW Raid - 2x WD20EARS = 2TB Raid1
I've been using Linux software raid now for at least 6 years. What started off as 3x200GB in Raid5, eventually grew to 4x500GB in Raid5. With storage becoming cheaper (and less reliable for that matter), I decided to shift to a Raid1 setup. At $70/disk, the WD20EARS seemed like cheap, safe storage with raid1.
What I guessed would be one afternoon of work for me, turned into 4 days of troubleshooting before I got it working properly. I'm using Ubuntu 10.10 userspace with Linux 3.0. Whether this combination of new kernel and old userspace (mdadm v2.6.7) was the reason or not, I still don't know.
This is how it went down (don't do this):
1) Made 'fd' type partitions sdb1 and sdc1 (both at 2048 block)
2) Created md0 with sdb1 + missing, level=1
3) Ran mkfs.ext4 -b 4096 on md0
At this point sdb1 disappeared! fdisk -lu reports that sdb hasn't even been initialised!
Anyways, I continue to mount md0, copy files, unmount, partprobe'd and remounted for luck and it all looked good.
4) Rebooted the machine
Now I couldn't mount md0 anymore. It couldn't find the ext4 superblocks. I tried a number of locations manually, but nothing seemed to work. I noticed the mdadm --detail md0 says that the array is using "sdb" now (instead of the sdb1 it was actually created with). But then again, sdb1 doesn't exist after mkfs md0. Arghh.
5) Created an '83' type partition sdb1 at the same 2048 block
Now out of nowhere, /dev/md0p1 appears! What the hell is this?
6) Ran fsck on md0 and it can't find the filesystem
7) Ran fsck on md0p1 and now it finds it, but it complains about some boundary sizes being wrong
8) Force fsck on md0p1. All OK.
9) Ran resize2fs to fix the boundary issue
Now I can mount md0. All my files are still there. fdisk -lu shows sdb1 as the linux partition, and shows disk "md0" as having "md0p1" as a partition of type linux. What???
10) mdadm /dev/md0 --add /dev/sdc1. It resyncs, and the array is all clean, non-degraded.
11) Reboot again
Can't access md0 anymore! WHAT IS GOING ON! mdadm --detail md0, shows clean, but it says its using "sdb" and "sdc". Obviously sdc1 doesn't exist anymore either.
I did all this twice to double check I didn't make any stupid mistakes, and I got the exact same result.
After much debugging, I decided to build the raid1 array again using the entire disk sdb and sdc, instead of the 'fd' partitions sdb1 and sdc1 I used first. This just worked! No problems at all. Rebooted a few times, and I can still see my ext4 fs on md0 and everything looks good. Why couldn't it deal with partition'd array members?
If anyone is interested, using an onboard Nvidia SATA II controller, my Raid1 re-sync speed is 100mb/s with the 2x WD20EARS. That's just under 6hrs for re-sync.
Copying from the 4x500GB Raid5 to the 2x2TB Raid1 was at about 90mb/s.
On more thing, before doing any of this, I added wdidle3.exe (v1.05) to UBCD and disabled the idle timer on both the green drives. My Load Cycle Count on two older 1TB WD10EARS drives hit 60,000+ after 1 year of usage. Although WD says these drives can handle up to 300,00 cycles, one of my old 1TB green drives has started reporting bad sectors. Whether this was because of the LCC or because these were just "Green" drives, I don't know.
With fewer drives in the chassis (3 now, was 5), disk temps are about 10C lower. And with the LCC disabled, I expect these drives to last at least 2 years. Only time will tell for sure.
What I guessed would be one afternoon of work for me, turned into 4 days of troubleshooting before I got it working properly. I'm using Ubuntu 10.10 userspace with Linux 3.0. Whether this combination of new kernel and old userspace (mdadm v2.6.7) was the reason or not, I still don't know.
This is how it went down (don't do this):
1) Made 'fd' type partitions sdb1 and sdc1 (both at 2048 block)
2) Created md0 with sdb1 + missing, level=1
3) Ran mkfs.ext4 -b 4096 on md0
At this point sdb1 disappeared! fdisk -lu reports that sdb hasn't even been initialised!
Anyways, I continue to mount md0, copy files, unmount, partprobe'd and remounted for luck and it all looked good.
4) Rebooted the machine
Now I couldn't mount md0 anymore. It couldn't find the ext4 superblocks. I tried a number of locations manually, but nothing seemed to work. I noticed the mdadm --detail md0 says that the array is using "sdb" now (instead of the sdb1 it was actually created with). But then again, sdb1 doesn't exist after mkfs md0. Arghh.
5) Created an '83' type partition sdb1 at the same 2048 block
Now out of nowhere, /dev/md0p1 appears! What the hell is this?
6) Ran fsck on md0 and it can't find the filesystem
7) Ran fsck on md0p1 and now it finds it, but it complains about some boundary sizes being wrong
8) Force fsck on md0p1. All OK.
9) Ran resize2fs to fix the boundary issue
Now I can mount md0. All my files are still there. fdisk -lu shows sdb1 as the linux partition, and shows disk "md0" as having "md0p1" as a partition of type linux. What???
10) mdadm /dev/md0 --add /dev/sdc1. It resyncs, and the array is all clean, non-degraded.
11) Reboot again
Can't access md0 anymore! WHAT IS GOING ON! mdadm --detail md0, shows clean, but it says its using "sdb" and "sdc". Obviously sdc1 doesn't exist anymore either.
I did all this twice to double check I didn't make any stupid mistakes, and I got the exact same result.
After much debugging, I decided to build the raid1 array again using the entire disk sdb and sdc, instead of the 'fd' partitions sdb1 and sdc1 I used first. This just worked! No problems at all. Rebooted a few times, and I can still see my ext4 fs on md0 and everything looks good. Why couldn't it deal with partition'd array members?
If anyone is interested, using an onboard Nvidia SATA II controller, my Raid1 re-sync speed is 100mb/s with the 2x WD20EARS. That's just under 6hrs for re-sync.
Copying from the 4x500GB Raid5 to the 2x2TB Raid1 was at about 90mb/s.
On more thing, before doing any of this, I added wdidle3.exe (v1.05) to UBCD and disabled the idle timer on both the green drives. My Load Cycle Count on two older 1TB WD10EARS drives hit 60,000+ after 1 year of usage. Although WD says these drives can handle up to 300,00 cycles, one of my old 1TB green drives has started reporting bad sectors. Whether this was because of the LCC or because these were just "Green" drives, I don't know.
With fewer drives in the chassis (3 now, was 5), disk temps are about 10C lower. And with the LCC disabled, I expect these drives to last at least 2 years. Only time will tell for sure.
Sunday, October 2, 2011
Hook #1: The Spinners - It's A Shame
hook: "A chorus or repeated instrumental passage in a piece of music, esp. a pop or rock song, that gives it immediate appeal and makes it easy to remember"
Once in a while, there's a song that I get hooked onto. I'd write it out for the sax or guitar and I'd probably have it on repeat for most of the week. I'm going to be starting a series of posts for my hooks.
Hook #1 is "It's a Shame" by "The Spinners" a.k.a "The Detroit Spinners"
Once in a while, there's a song that I get hooked onto. I'd write it out for the sax or guitar and I'd probably have it on repeat for most of the week. I'm going to be starting a series of posts for my hooks.
Hook #1 is "It's a Shame" by "The Spinners" a.k.a "The Detroit Spinners"
What's going on?
I may be a true gearhead but I'm also a keen technologist and jazz musician. I thought I'd use this space to post any of the other interesting things I'm doing, thinking or listening to.
Winter is almost here. Its 10degC in Ottawa right now. When the snow flies, both the XJ12 and the 560 will be tucked away in the garage and I start driving the 190E again. Without an insulated garage, any heavy wrenching has to be put aside for the summer.
In the winter, I spend more time playing music - the saxophone mostly these days and writing open source software (see my sourceforge page). And contrary to conventional wisdom, I actually go jogging/running more often in the winter.
But if you're here only for the car stuff, don't panic, I do have a few winter car projects lined up. On the 560, I will be changing the odometer gears, front brakes, front shocks+springs, and installing the original Becker radio. On the Jag, probably not much any time soon. I do need to do a valve adjustment and maybe even just rebuild the heads altogether, but that will definitely be a summer project.
Welcome to my blog 2.0.
Winter is almost here. Its 10degC in Ottawa right now. When the snow flies, both the XJ12 and the 560 will be tucked away in the garage and I start driving the 190E again. Without an insulated garage, any heavy wrenching has to be put aside for the summer.
In the winter, I spend more time playing music - the saxophone mostly these days and writing open source software (see my sourceforge page). And contrary to conventional wisdom, I actually go jogging/running more often in the winter.
But if you're here only for the car stuff, don't panic, I do have a few winter car projects lined up. On the 560, I will be changing the odometer gears, front brakes, front shocks+springs, and installing the original Becker radio. On the Jag, probably not much any time soon. I do need to do a valve adjustment and maybe even just rebuild the heads altogether, but that will definitely be a summer project.
Welcome to my blog 2.0.
Car blogs have moved
I decided to have the car blogs on their own addresses and maintain only one blog with my name in the address.
V12 Stealth Hunter is at http://v12stealthhunter.blogspot.com/
V8 Bruiser Benz is at http://v8bruiserbenz.blogspot.com/
V12 Stealth Hunter is at http://v12stealthhunter.blogspot.com/
V8 Bruiser Benz is at http://v8bruiserbenz.blogspot.com/
Subscribe to:
Posts (Atom)