Re: [RFC -mm] memcg: prevent from OOM with too many dirty pages

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Tue 29-05-12 15:51:01, Michal Hocko wrote:
[...]
> OK, I have tried it with a simpler approach:
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index c978ce4..e45cf2a 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -1294,8 +1294,12 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec,
>  	 *                     isolated page is PageWriteback
>  	 */
>  	if (nr_writeback && nr_writeback >=
> -			(nr_taken >> (DEF_PRIORITY - sc->priority)))
> -		wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
> +			(nr_taken >> (DEF_PRIORITY - sc->priority))) {
> +		if (global_reclaim(sc))
> +			wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
> +		else
> +			congestion_wait(BLK_RW_ASYNC, HZ/10);
> +	}
>  
>  	trace_mm_vmscan_lru_shrink_inactive(zone->zone_pgdat->node_id,
>  		zone_idx(zone),
> 
[...]
> As a conclusion congestion wait performs better (even though I haven't
> done repeated testing to see what is the deviation) when the
> reader/writer size doesn't fit into the memcg, while it performs much
> worse (at least for writer) if it does fit.
> 
> I will play with that some more

I have, yet again, updated the test. I am writing data to an USB stick
(with ext3, mounted in sync mode) and which writes 1G in 274.518s,
3.8MB/s so the storage is really slow. The parallel read is performed
from tmpfs and from a local ext3 partition (testing script is attached).
We start with writing so the LRUs will have some dirty pages when the
read starts and fill up the LRU with clean page cache.

congestion wait:
================
* ext3 (reader)                         avg      std/avg
** Write
5M	412.128	334.944	337.708	339.457	356.0593 [10.51%]
60M	566.652	321.607	492.025	317.942	424.5565 [29.39%]
300M	318.437	315.321	319.515	314.981	317.0635 [0.71%]
2G	317.777	314.8	318.657	319.409	317.6608 [0.64%]

** Read
5M	40.1829	40.8907	48.8362	40.0535	42.4908  [9.99%]
60M	15.4104	16.1693	18.9162	16.0049	16.6252  [9.39%]
300M	17.0376	15.6721	15.6137	15.756	16.0199  [4.25%]
2G	15.3718	17.3714	15.3873	15.4554	15.8965  [6.19%]

* Tmpfs (reader)
** Write
5M	324.425	327.395	573.688	314.884	385.0980 [32.68%]
60M	464.578	317.084	375.191	318.947	368.9500 [18.76%]
300M	316.885	323.759	317.212	318.149	319.0013 [1.01%]
2G	317.276	318.148	318.97	316.897	317.8228 [0.29%]

** Read
5M	0.9241	0.8620	0.9391	1.2922	1.0044   [19.39%]
60M	0.8753	0.8359	1.0072	1.3317	1.0125   [22.23%]
300M	0.9825	0.8143	0.9864	0.8692	0.9131   [9.35%]
2G	0.9990	0.8281	1.0312	0.9034	0.9404   [9.83%]


PageReclaim:
=============
* ext3 (reader)
** Write                                avg      std/avg  comparision 
                                                         (cong is 100%)
5M	313.08	319.924	325.206	325.149	320.8398 [1.79%]  90.11%
60M	314.135	415.245	502.157	313.776	386.3283 [23.50%] 91.00%
300M	313.718	320.448	315.663	316.714	316.6358 [0.89%]  99.87%
2G	317.591	316.67	316.285	316.624	316.7925 [0.18%]  99.73%

** Read
5M	19.0228	20.6743	17.2508	17.5946	18.6356	 [8.37%]  43.86%
60M	17.3657	15.6402	16.5168	15.5601	16.2707	 [5.22%]  97.87%
300M	17.1986	15.7616	19.5163	16.9544	17.3577	 [9.05%]  108.35%
2G	15.6696	15.5384	15.4381	15.2454	15.4729	 [1.16%]  97.34%

* Tmpfs (reader)
** Write
5M	317.303	314.366	316.508	318.883	316.7650 [0.59%]  82.26%
60M	579.952	666.606	660.021	655.346	640.4813 [6.34%]  173.60%
300M	318.494	318.64	319.516	316.79	318.3600 [0.36%]  99.80%
2G	315.935	318.069	321.097	320.329	318.8575 [0.73%]  100.33%

** Read  
5M	0.8415	0.8550	0.7892	0.8515	0.8343	 [3.67%]  83.07%
60M	0.8536	0.8685	0.8237	0.8805	0.8565	 [2.86%]  84.60%
300M	0.8309	0.8724	0.8553	0.8577	0.8541	 [2.01%]  93.53%
2G	0.8427	0.8468	0.8325	1.4658	0.9970	 [31.36%] 106.01%

Variance (std/avg) seems to be lower for both reads and writes with
PageReclaim approach and also if we compare the average numbers it seems
to be mostly better (especially for reads) or within the noise.
There are two "peaks" in numbers, though.
* 60M cgroup write performance when reading from tmpfs. While read
behaved well with PageReclaim patch (actually much better than
congwait), the writer stalled a lot.
* 5M cgroup read performance when reading from ext3 when congestion_wait
approach fall down flat while PageReclaim did better for both read and
write.

So I guess that the conclusion could be that the two approaches are
comparable. Both of them could lead to stalling but they are doing
mostly good which is much better than an OOM killer. We can do much
better but that would require conditional sleeping.

How do people feel about going with the simpler approach for now (even
for stable kernels as the problem is real and long term) and work on the
conditional part as a follow up?
Which way would be preferable? I can post a full patch for the
congestion wait approach if you are interested. I do not care much as
both of them fix the problem.
-- 
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic

Attachment: cgroup_cache_oom_test.sh
Description: Bourne shell script


[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]