iSCSI linux raid target failure file system corruption testing problems.

["Brian Kerhin" <brian@xxxxxxxxxxxxxxx>]

 

I’ve setup an iSCSI network using the initiator (cent os 4.2)

iscsi-initiator-utils-4.0.3.0-2

 

I have 4 targets and a RAID 6 array. (n-2 storage space, 2 targets can die before I loose data)

 

My problem is that I loose data when 1 target dies.

 

Here is my procedure for test:

 

1. Start iSCSI
2. Create raid 6 array.
3. format file system (ext3)
4. start 1.6GB copy of a usr directory. (/usr2 a copy of my /usr directory)
5. While it the middle of the copy I pull the Ethernet cord from one of my targets (simulate a failure J).
6. The copy locks up for a few seconds, the iscsi transfer processes all top showing up in my top monitor. Then after about 7 seconds they pop back up and my /proc/mdstat file shows my disk has failed.
7. This works as I would think it should, BUT when I do a du to look at the data I copied I find that a number of the copied files are crap:
1. lrwxrwxrwx    1 root root    17 Nov  3 12:35 libxslt.so.1 -> libxslt.so.1.1.11
2. -rwxr-xr-x    1 root root  189K Nov  1 18:05 libxslt.so.1.1.11
3. -rwxr-xr-x    1 root root   71K Nov  1 18:05 libz.a
4. lrwxrwxrwx    1 root root    15 Nov  3 12:32 libz.so -> libz.so.1.2.1.2
5. ?---------    ? ?    ?        ?            ? libz.so.1
6. -rwxr-xr-x    1 root root   63K Nov  1 18:05 libz.so.1.2.1.2
8. Note the libz.so.1 file is a mess. If I try and access messed up directories I get input/output error. If I umount the file system and run fsck on it I found I have a corrupt file system.

 

The goal of this project would be to have targets die and still have my array, filesystem and data survive the target crash(s).

 

My iscsi.conf

 

ConnFailTimeout=5

DiskCommandTimeout=5

 

Continuous=yes

 

DiscoveryAddress=10.10.10.34

DiscoveryAddress=10.10.10.30

DiscoveryAddress=10.10.10.26

DiscoveryAddress=10.10.10.36

 

 

Thanks,

Brian

 

 

BEGIN:VCARD
VERSION:2.1
N:Kerhin;Brian
FN:Brian Kerhin
ORG:Byte Harmony, Inc.
TITLE:President
TEL;WORK;VOICE:(414) 443-9594
TEL;HOME;VOICE:(414) 607-1743
TEL;CELL;VOICE:(414) 688-7458
TEL;WORK;FAX:(414) 476-7614
TEL;HOME;FAX:(206) 984-1182
ADR;WORK;ENCODING=QUOTED-PRINTABLE:;;Byte Harmony, Inc.=0D=0ABrian Kerhin=0D=0A1439 S 57th St;West Allis;WI;532=
14;United States of America
LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Byte Harmony, Inc.=0D=0ABrian Kerhin=0D=0A1439 S 57th St=0D=0AWest Allis, WI=
 53214=0D=0AUnited States of America
URL;WORK:http://www.byteharmony.com
EMAIL;PREF;INTERNET:brian@xxxxxxxxxxxxxxx
REV:20050121T232823Z
END:VCARD