On Tue, Dec 17, 2019 at 07:06:00PM -0500, Dennis Zhou wrote: > > Regarding the slow io submission, I tried to increase the iops value, > > default was 10, but 100 and 1000 made no change. Increasing the maximum > > discard request size to 128M worked (when there was such long extent > > ready). I was expecting a burst of like 4 consecutive IOs after a 600MB > > file is deleted. I did not try to tweak bps_limit because there was > > nothing to limit. > > Ah so there's actually a max time between discards set to 10 seconds as > the maximum timeout is calculated over 6 hours. So if we only have 6 > extents, we'd discard 1 per hour(ish given it decays), but this is > clamped to 10 seconds. > > At least on our servers, we seem to discard at a reasonable rate to > prevent performance penalties during a large number of reads and writes > while maintaining reasonable write amplification performance. Also, > metadata blocks aren't tracked, so on freeing of a whole metadata block > group (minus relocation), we'll trickle discards slightly slower than > expected. So after watching the sysfs numbers, my observation is that the overall strategy of the async discard is to wait for larger ranges and discard one range every 10 seconds. This is a slow process, but this makes sense when there are reads or writes going on so the discard IO penalty is marginal. Running full fstrim will flush all the discardable extents so there's a way to reset the discardable queue. What I still don't see as optimal is the single discard request sent per one period. Namely because there's the iops_limit knob. My idea is that each timeout, 'iops_limit' times 'max_discard_size' is called, so the discard batches are large in total. However, this has impact on reads and writes and also on the device itself, I'm not sure if the too frequent discards are not making things worse (as this is a known problem). I'm interested in more strategies that you could have tested in your setups, either bps based or rate limited etc. The current one seems ok for first implementation but we might want to tune it once we get feedback from more users.
