Re: [RFC PATCH 0/6] Understanding delays due to throttling under very heavy write load
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
On 02/03/2012 05:03 PM, Yehuda Sadeh Weinraub wrote:
On Fri, Feb 3, 2012 at 3:33 PM, Jim Schutt<jaschut@xxxxxxxxxx> wrote:
You can try running 'iostat -t -kx -d 1' on the osds, and see whether %util
reaches 100%, and when it happens whether it's due to number of io
operations that are thrashing, or whether it's due to high amount of data.
FWIW, you may try setting 'filestore flusher = false', and set
/proc/sys/vm/dirty_background_ratio' to a small number (e.g., 1M).
Here's some iostat data from early in a run, when things are
running well:
02/02/2012 09:14:13 AM
avg-cpu: %user %nice %system %iowait %steal %idle
23.24 0.00 61.99 7.38 0.00 7.38
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sdc 0.00 0.00 0.00 206.00 0.00 101.57 1009.79 54.80 251.27 4.86 100.10
sdd 0.00 0.00 0.00 202.00 0.00 98.10 994.61 27.85 132.42 4.96 100.10
sde 0.00 4.00 0.00 212.00 0.00 105.09 1015.25 96.06 588.43 4.72 100.10
sdh 0.00 0.00 0.00 200.00 0.00 97.11 994.40 69.77 535.01 5.00 100.10
sdg 0.00 2.00 0.00 221.00 0.00 109.59 1015.60 82.05 298.71 4.53 100.10
sda 0.00 1.00 0.00 212.00 0.00 83.93 810.75 18.26 84.82 4.68 99.30
sdf 0.00 0.00 0.00 208.00 0.00 102.55 1009.73 77.23 383.19 4.50 93.70
sdb 0.00 0.00 0.00 205.00 0.00 98.66 985.68 19.97 133.98 4.84 99.20
sdj 0.00 0.00 0.00 202.00 0.00 99.59 1009.66 69.97 257.47 4.95 100.00
sdk 0.00 0.00 0.00 204.00 0.00 98.10 984.86 20.83 100.34 4.87 99.30
sdm 0.00 0.00 0.00 216.00 0.00 106.55 1010.22 77.73 268.67 4.63 100.00
sdn 0.00 0.00 0.00 205.00 0.00 98.60 985.05 19.33 95.88 4.81 98.60
sdo 0.00 0.00 0.00 232.00 0.00 106.25 937.93 23.26 82.19 4.29 99.50
sdl 0.00 0.00 0.00 181.00 0.00 85.12 963.09 24.73 131.71 4.80 86.80
sdp 0.00 4.00 0.00 207.00 0.00 87.41 864.77 37.01 111.13 4.49 93.00
sdi 0.00 0.00 0.00 208.00 0.00 103.04 1014.54 72.30 263.72 4.70 97.70
sdr 0.00 0.00 0.00 191.00 0.00 76.75 822.95 11.51 83.69 4.59 87.60
sds 0.00 0.00 0.00 209.00 0.00 101.91 998.58 49.95 278.08 4.70 98.20
sdt 0.00 0.00 0.00 209.00 0.00 99.57 975.69 27.31 157.44 4.79 100.10
sdu 0.00 0.00 0.00 216.00 0.00 107.09 1015.41 79.82 345.88 4.63 100.10
sdw 0.00 0.00 0.00 208.00 0.00 103.09 1015.00 74.55 308.15 4.81 100.10
sdv 0.00 0.00 0.00 201.00 0.00 98.05 999.08 76.87 265.88 4.98 100.10
sdx 0.00 0.00 0.00 202.00 0.00 100.50 1018.93 110.40 327.68 4.96 100.10
sdq 0.00 0.00 0.00 228.00 0.00 112.59 1011.30 54.84 281.04 4.39 100.10
02/02/2012 09:14:14 AM
avg-cpu: %user %nice %system %iowait %steal %idle
22.11 0.00 54.03 15.38 0.00 8.48
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sdc 0.00 0.00 0.00 233.00 0.00 99.68 876.15 95.98 384.42 4.29 100.00
sdd 0.00 0.00 0.00 205.00 0.00 96.64 965.46 20.37 108.51 4.84 99.30
sde 0.00 0.00 0.00 225.00 0.00 99.54 906.03 92.38 420.67 4.44 100.00
sdh 0.00 0.00 0.00 198.00 0.00 97.05 1003.84 79.39 410.56 5.05 100.00
sdg 0.00 0.00 0.00 245.00 0.00 108.38 905.99 84.40 385.47 4.08 100.00
sda 0.00 4.00 0.00 220.00 0.00 96.23 895.78 63.24 294.59 4.44 97.60
sdf 0.00 0.00 0.00 216.00 0.00 107.09 1015.41 87.67 399.14 4.57 98.80
sdb 0.00 0.00 0.00 156.00 0.00 72.05 945.95 11.61 58.94 4.84 75.50
sdj 0.00 0.00 0.00 199.00 0.00 95.41 981.95 56.28 366.11 4.84 96.40
sdk 0.00 0.00 0.00 206.00 0.00 100.14 995.57 54.69 241.41 4.86 100.10
sdm 0.00 0.00 0.00 200.00 0.00 99.09 1014.72 79.51 506.47 4.74 94.70
sdn 0.00 0.00 0.00 191.00 0.00 91.29 978.81 26.82 128.39 5.18 98.90
sdo 0.00 0.00 0.00 234.00 0.00 106.75 934.32 49.82 231.07 4.27 100.00
sdl 0.00 0.00 0.00 214.00 0.00 103.62 991.70 33.03 168.13 4.62 98.80
sdp 0.00 0.00 0.00 219.00 0.00 106.08 992.00 64.69 328.92 4.57 100.00
sdi 0.00 0.00 0.00 210.00 0.00 104.09 1015.09 100.98 421.01 4.76 100.00
sdr 0.00 0.00 0.00 180.00 0.00 81.66 929.07 10.31 63.59 5.12 92.20
sds 0.00 0.00 0.00 201.00 0.00 95.15 969.47 32.60 144.16 4.98 100.00
sdt 0.00 0.00 0.00 198.00 0.00 95.72 990.10 33.26 155.98 4.84 95.90
sdu 0.00 0.00 0.00 219.00 0.00 108.59 1015.53 66.10 347.91 4.57 100.00
sdw 0.00 0.00 0.00 204.00 0.00 100.75 1011.41 81.20 456.47 4.80 98.00
sdv 0.00 0.00 0.00 197.00 0.00 96.09 998.90 44.19 284.65 5.08 100.00
sdx 0.00 0.00 0.00 211.00 0.00 104.19 1011.26 84.87 542.85 4.69 99.00
sdq 0.00 0.00 0.00 216.00 0.00 105.10 996.52 36.63 134.40 4.63 100.00
This is later in the same run, when things are not going as well:
02/02/2012 09:21:52 AM
avg-cpu: %user %nice %system %iowait %steal %idle
5.13 0.00 13.31 8.52 0.00 73.04
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sdc 0.00 0.00 0.00 36.00 0.00 16.02 911.11 1.43 39.72 5.64 20.30
sdd 0.00 0.00 0.00 18.00 0.00 8.01 911.11 0.85 47.28 6.39 11.50
sde 0.00 0.00 0.00 4.00 0.00 0.01 6.00 0.08 20.00 13.00 5.20
sdh 0.00 0.00 0.00 20.00 0.00 8.01 820.40 0.65 32.40 5.30 10.60
sdg 0.00 0.00 0.00 19.00 0.00 8.01 863.58 0.60 31.63 4.63 8.80
sda 0.00 0.00 0.00 82.00 0.00 36.04 900.10 3.13 37.05 5.15 42.20
sdf 0.00 0.00 0.00 18.00 0.00 8.01 911.11 0.80 44.22 6.39 11.50
sdb 0.00 8.00 0.00 42.00 0.00 1.75 85.52 0.14 3.43 1.40 5.90
sdj 0.00 16.00 0.00 103.00 0.00 25.64 509.83 2.21 21.36 3.65 37.60
sdk 0.00 14.00 0.00 152.00 0.00 47.93 645.79 3.96 27.31 4.12 62.60
sdm 0.00 0.00 0.00 21.00 0.00 9.39 915.81 0.94 44.57 5.71 12.00
sdn 0.00 34.00 0.00 197.00 0.00 64.61 671.72 28.66 85.62 4.02 79.10
sdo 0.00 0.00 0.00 92.00 0.00 42.54 946.87 6.22 55.58 4.85 44.60
sdl 0.00 0.00 0.00 6.00 0.00 2.01 685.33 0.09 59.67 6.33 3.80
sdp 0.00 10.00 0.00 58.00 0.00 9.56 337.52 1.20 20.60 3.05 17.70
sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdr 0.00 0.00 0.00 37.00 0.00 16.02 886.92 1.19 32.27 5.11 18.90
sds 0.00 18.00 0.00 115.00 0.00 26.54 472.70 4.03 25.94 3.20 36.80
sdt 0.00 0.00 0.00 131.00 0.00 60.05 938.87 6.13 46.33 5.11 67.00
sdu 0.00 12.00 0.00 119.00 0.00 31.40 540.44 2.93 24.65 3.05 36.30
sdw 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdv 0.00 4.00 0.00 63.00 0.00 9.46 307.68 0.83 14.32 2.38 15.00
sdx 0.00 0.00 0.00 35.00 0.00 15.51 907.66 0.79 28.20 4.89 17.10
sdq 0.00 0.00 0.00 37.00 0.00 16.02 886.70 1.52 41.00 5.86 21.70
02/02/2012 09:21:53 AM
avg-cpu: %user %nice %system %iowait %steal %idle
3.74 0.00 8.75 6.60 0.00 80.90
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 0.00 0.00 18.00 0.00 8.01 911.11 0.88 48.94 6.83 12.30
sda 0.00 0.00 0.00 45.00 0.00 7.38 335.64 0.54 18.87 1.78 8.00
sdf 0.00 0.00 0.00 18.00 0.00 8.01 911.11 0.93 51.44 6.78 12.20
sdb 0.00 0.00 0.00 5.00 0.00 0.74 302.40 0.05 10.20 8.20 4.10
sdj 0.00 0.00 0.00 72.00 0.00 32.03 911.11 2.51 34.99 5.01 36.10
sdk 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdm 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdn 0.00 0.00 0.00 123.00 0.00 52.60 875.84 13.83 209.72 4.84 59.50
sdo 0.00 0.00 0.00 13.00 0.00 5.52 868.92 0.30 108.31 4.69 6.10
sdl 0.00 0.00 0.00 27.00 0.00 12.47 945.78 1.33 47.15 6.59 17.80
sdp 0.00 0.00 0.00 11.00 0.00 4.50 838.55 0.51 14.09 5.09 5.60
sdi 0.00 0.00 0.00 19.00 0.00 8.01 863.58 0.72 38.05 5.74 10.90
sdr 0.00 0.00 0.00 18.00 0.00 8.01 911.11 0.69 38.33 5.89 10.60
sds 0.00 0.00 0.00 56.00 0.00 19.66 718.86 1.31 39.16 5.11 28.60
sdt 0.00 0.00 0.00 161.00 0.00 72.57 923.18 6.97 37.39 5.07 81.70
sdu 0.00 0.00 0.00 66.00 0.00 30.02 931.64 2.77 39.85 5.09 33.60
sdw 0.00 0.00 0.00 20.00 0.00 8.51 871.60 1.47 27.80 4.85 9.70
sdv 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdx 0.00 0.00 0.00 36.00 0.00 16.02 911.11 1.37 38.08 5.72 20.60
sdq 0.00 0.00 0.00 44.00 0.00 19.46 906.00 1.15 26.02 4.50 19.80
And finally, this is still later, near the end of the run, when things have recovered
somewhat:
02/02/2012 09:22:34 AM
avg-cpu: %user %nice %system %iowait %steal %idle
15.25 0.00 52.27 20.88 0.00 11.60
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sdc 0.00 1.00 0.00 217.00 0.00 95.20 898.51 84.43 413.56 4.60 99.90
sdd 0.00 0.00 0.00 40.00 0.00 16.86 863.00 1.59 28.45 5.55 22.20
sde 0.00 0.00 0.00 206.00 0.00 99.27 986.95 89.64 452.92 4.85 99.90
sdh 0.00 0.00 0.00 51.00 0.00 22.53 904.63 2.02 35.45 5.47 27.90
sdg 0.00 0.00 0.00 230.00 0.00 112.49 1001.63 92.87 283.01 4.33 99.60
sda 0.00 0.00 0.00 215.00 0.00 106.10 1010.68 94.45 253.40 4.65 99.90
sdf 0.00 0.00 0.00 73.00 0.00 32.04 898.74 2.20 30.08 5.11 37.30
sdb 0.00 0.00 0.00 92.00 0.00 40.05 891.48 2.55 27.70 4.85 44.60
sdj 0.00 44.00 0.00 280.00 0.00 91.61 670.03 109.32 314.59 3.57 99.90
sdk 0.00 1.00 0.00 210.00 0.00 100.63 981.41 97.79 419.98 4.76 99.90
sdm 0.00 42.00 0.00 282.00 0.00 100.27 728.23 92.86 285.38 3.54 99.90
sdn 0.00 0.00 0.00 213.00 0.00 100.81 969.31 41.62 301.33 4.67 99.40
sdo 0.00 39.00 0.00 306.00 0.00 102.84 688.29 82.44 279.69 3.26 99.70
sdl 0.00 0.00 0.00 219.00 0.00 104.16 974.06 83.05 421.80 4.56 99.90
sdp 0.00 46.00 0.00 277.00 0.00 97.01 717.23 106.44 324.31 3.61 99.90
sdi 0.00 0.00 0.00 56.00 0.00 24.03 878.86 1.73 30.91 5.05 28.30
sdr 0.00 34.00 0.00 266.00 0.00 97.66 751.91 63.86 304.39 3.76 100.00
sds 0.00 18.00 0.00 67.00 0.00 17.41 532.18 1.68 25.03 3.79 25.40
sdt 0.00 0.00 0.00 130.00 0.00 64.01 1008.37 56.33 166.52 4.99 64.90
sdu 0.00 0.00 0.00 197.00 0.00 95.02 987.82 44.70 282.45 4.95 97.60
sdw 0.00 0.00 0.00 207.00 0.00 93.39 923.98 90.21 448.08 4.83 99.90
sdv 0.00 0.00 0.00 204.00 0.00 100.52 1009.14 84.16 425.70 4.85 98.90
sdx 0.00 0.00 0.00 203.00 0.00 88.75 895.33 87.10 475.92 4.92 99.90
sdq 0.00 0.00 0.00 18.00 0.00 8.01 911.11 0.52 28.83 4.83 8.70
02/02/2012 09:22:35 AM
avg-cpu: %user %nice %system %iowait %steal %idle
14.63 0.00 50.99 22.22 0.00 12.16
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
sdc 0.00 0.00 0.00 209.00 0.00 99.54 975.35 84.02 409.76 4.78 99.90
sdd 0.00 0.00 0.00 13.00 0.00 5.50 867.08 0.34 57.31 6.23 8.10
sde 0.00 0.00 0.00 204.00 0.00 98.12 985.06 87.28 418.62 4.88 99.50
sdh 0.00 0.00 0.00 78.00 0.00 34.12 895.79 2.15 30.26 5.37 41.90
sdg 0.00 0.00 0.00 226.00 0.00 108.48 983.04 93.54 336.46 4.42 99.80
sda 0.00 0.00 0.00 219.00 0.00 108.07 1010.63 80.90 510.96 4.53 99.20
sdf 0.00 6.00 0.00 81.00 0.00 21.20 535.90 1.99 24.47 3.59 29.10
sdb 0.00 0.00 0.00 71.00 0.00 32.03 923.94 2.46 34.63 4.65 33.00
sdj 0.00 0.00 0.00 192.00 0.00 83.87 894.62 83.33 459.53 5.21 100.10
sdk 0.00 41.00 0.00 285.00 0.00 94.12 676.32 104.34 310.17 3.51 100.10
sdm 0.00 0.00 0.00 202.00 0.00 90.44 916.91 86.45 506.52 4.96 100.10
sdn 0.00 0.00 0.00 208.00 0.00 101.48 999.23 87.79 323.35 4.79 99.70
sdo 0.00 1.00 0.00 228.00 0.00 108.63 975.75 89.79 327.24 4.38 99.80
sdl 0.00 28.00 0.00 270.00 0.00 97.64 740.65 52.06 281.67 3.54 95.60
sdp 0.00 0.00 0.00 195.00 0.00 85.65 899.57 92.28 453.54 5.14 100.20
sdi 0.00 14.00 0.00 31.00 0.00 9.02 595.61 0.96 30.94 4.77 14.80
sdr 0.00 0.00 0.00 192.00 0.00 83.11 886.46 14.22 142.39 5.06 97.10
sds 0.00 0.00 0.00 18.00 0.00 8.01 911.11 0.73 40.39 5.89 10.60
sdt 0.00 0.00 0.00 201.00 0.00 98.66 1005.29 65.87 425.37 4.89 98.30
sdu 0.00 0.00 0.00 209.00 0.00 103.01 1009.38 87.49 285.51 4.74 99.10
sdw 0.00 0.00 0.00 204.00 0.00 96.74 971.22 82.66 410.50 4.89 99.70
sdv 0.00 0.00 0.00 198.00 0.00 96.61 999.23 83.39 420.17 5.03 99.50
sdx 0.00 0.00 0.00 204.00 0.00 98.79 991.80 86.54 428.67 4.90 100.00
sdq 0.00 0.00 0.00 36.00 0.00 16.02 911.11 0.88 24.33 4.44 16.00
The above suggests to me that the slowdown is a result
of requests not getting submitted at the same rate as
when things are running well.
-- Jim
Yehuda
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
[CEPH Users]
[Information on CEPH]
[Linux USB Devel]
[Video for Linux]
[Linux Audio Users]
[Photo]
[Yosemite News]
[Yosemite Photos]
[Free Online Dating]
[Linux Kernel]
[Linux SCSI]
[XFree86]
