Q: (2.6.16 & ext3) bad SMP load balancing when writing to ext3 on slow device

["Ulrich Windl" <ulrich.windl@xxxxxxxxxxxxxxxxxxxx>]

Hi,

while copying large remote files for an USB memory stick formatted with ext3 using 
scp, I noticed a stall in wrie speed. Looking at the system with top I saw:
top - 09:25:25 up 55 days, 23:49,  2 users,  load average: 11.09, 7.41, 4.43
Tasks: 128 total,   1 running, 127 sleeping,   0 stopped,   0 zombie
Cpu0  :  7.6%us,  0.3%sy,  0.0%ni,  0.0%id, 90.4%wa,  0.3%hi,  1.3%si,  0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,  0.0%id,100.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.0%us,  0.0%sy,  0.0%ni,  0.0%id,100.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  0.0%us,  1.7%sy,  0.0%ni,  0.0%id, 98.3%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1028044k total,  1017956k used,    10088k free,    34784k buffers
Swap:  2097140k total,      616k used,  2096524k free,   733100k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
11284 root      18   0 29168 1960 1504 D    2  0.2   0:11.81 scp
  137 root      15   0     0    0    0 D    0  0.0  14:16.59 pdflush
10865 root      15   0     0    0    0 D    0  0.0   0:00.50 kjournald
11355 root      15   0     0    0    0 D    0  0.0   0:00.09 pdflush
11396 root      15   0     0    0    0 D    0  0.0   0:00.12 pdflush
11397 root      15   0     0    0    0 D    0  0.0   0:00.06 pdflush
12007 root      15   0     0    0    0 D    0  0.0   0:00.02 pdflush
12070 root      16   0 23976 2376 1744 R    0  0.2   0:00.28 top
12294 root      15   0     0    0    0 D    0  0.0   0:00.00 pdflush
12295 root      15   0     0    0    0 D    0  0.0   0:00.02 pdflush
12296 root      15   0     0    0    0 D    0  0.0   0:00.02 pdflush
27490 root      10  -5     0    0    0 D    0  0.0   0:02.93 usb-storage

First, it's impressive that a singly copy job can raise the load to above 10, and 
the next thing is that writing to a slow device can make 4 CPUs (actually two with 
hyperthreading) busy. The pdflush daemons are expected to bring dirty blocks onto 
the device, I guess. Does it make any sense to make four CPUs busy with doing so?

Here's another snapshot showing the assigned CPU also:

top - 09:32:18 up 55 days, 23:56,  2 users,  load average: 10.63, 9.99, 6.78
Tasks: 127 total,   1 running, 126 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.0%us,  0.0%sy,  0.0%ni,  0.0%id,100.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,  0.0%id,100.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.0%us,  0.0%sy,  0.0%ni,  1.7%id, 98.3%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  0.0%us,  0.0%sy,  0.0%ni,  0.0%id,100.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1028044k total,  1017896k used,    10148k free,    18044k buffers
Swap:  2097140k total,      616k used,  2096524k free,   741616k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  P COMMAND
  137 root      15   0     0    0    0 D    0  0.0  14:16.71 1 pdflush
 4299 root      17   0  5860  752  596 D    0  0.1   9:36.19 1 syslogd
10865 root      15   0     0    0    0 D    0  0.0   0:00.62 1 kjournald
11284 root      18   0 29168 1960 1504 D    0  0.2   0:14.76 3 scp
11355 root      15   0     0    0    0 D    0  0.0   0:00.19 0 pdflush
11396 root      15   0     0    0    0 D    0  0.0   0:00.24 1 pdflush
11397 root      15   0     0    0    0 D    0  0.0   0:00.22 1 pdflush
12294 root      15   0     0    0    0 D    0  0.0   0:00.11 1 pdflush
12295 root      15   0     0    0    0 D    0  0.0   0:00.14 1 pdflush
12296 root      15   0     0    0    0 D    0  0.0   0:00.13 1 pdflush
12591 root      16   0 23976 2376 1744 R    0  0.2   0:00.07 3 top
27490 root      10  -5     0    0    0 D    0  0.0   0:03.13 3 usb-storage

At times like shown, the scp seems to come to a complete halt. (Previously I had 
been using VFAT filesystem on the stick, and copy went much smoother, but the 
filesystem was full, so I tried another filesystem.)

Would anybody bee so kind to explain why the system looks like that? I'm not 
subscribed, so please honor the CC:.

Regards,
Ulrich Windl

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/