On Mon, May 04, 2020 at 05:24:11PM -0600, Chris Murphy wrote: > On Mon, May 4, 2020 at 5:09 PM Zygo Blaxell > <ce3g8jdj@xxxxxxxxxxxxxxxxxxxxx> wrote: > > > Some kinds of RAID rebuild don't provide sufficient idle time to complete > > the CMR-to-SMR writeback, so the host gets throttled. If the drive slows > > down too much, the kernel times out on IO, and reports that the drive > > has failed. The RAID system running on top thinks the drive is faulty > > (a false positive failure) and the fun begins (hope you don't have two > > of these drives in the same array!). > > This came up on linux-raid@ list today also, and someone posted this > smartmontools bug. > https://www.smartmontools.org/ticket/1313 > > It notes in part this error, which is not a time out. Uhhh...wow. If that's not an individual broken disk, but the programmed behavior of the firmware, that would mean the drive model is not usable at all. > [20809.396284] blk_update_request: I/O error, dev sdd, sector > 3484334688 op 0x1:(WRITE) flags 0x700 phys_seg 2 prio class 0 > > An explicit write error is a defective drive. But even slow downs > resulting in link resets is defective. The marketing of DM-SMR says > it's suitable without having to apply local customizations accounting > for the drive being SMR. > > > > Desktop CMR drives (which are not good in RAID arrays but people use > > them anyway) have firmware hardcoded to retry reads for about 120 > > seconds before giving up. To use desktop CMR drives in RAID arrays, > > you must increase the Linux kernel IO timeout to 180 seconds or risk > > false positive rejections (i.e. multi-disk failures) from RAID arrays. > > I think we're way past the time when all desktop oriented Linux > installations should have overridden the kernel default, using 180 > second timeouts instead. Even in the single disk case. The system is > better off failing safe to slow response, rather than link resets and > subsequent face plant. But these days most every laptop and desktop's > sysroot is on an SSD of some kind. > > > > Now here is the problem: DM-SMR drives have write latencies of up to 300 > > seconds in *non-error* cases. They are up to 10,000 times slower than > > CMR in the worst case. Assume that there's an additional 120 seconds > > for error recovery on top of the non-error write latency, and add the > > extra 50% for safety, and the SMR drive should be configured with a > > 630 second timeout (10.5 minutes) in the Linux kernel to avoid false > > positive failures. > > Incredible. > > > -- > Chris Murphy
