Re: Very various speed of grep operation on btrfs partition

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 2015-12-06 22:32, Duncan wrote:
FWIW, I build kde without the semantic-desktop stuff even enabled at
build-time (gentoo offers that option) here.  All the kdepim stuff (kmail,
etc) uses it, so I dumped the several kdepim related apps (kmail,
akregator, kaddressbook) I used here and found alternatives.  I don't
normally need the indexing, which only takes space for the index and
lowers performance, so it's all turned off at build-time.
Personally, I just avoid both KDE and GNOME. For me efficiency is most important, and XFCE beats both at that (I've done testing between the three on my laptop with equivalent settings, and XFCE gives me about 50-70% more battery life than either KDE or GNOME, and a much smaller memory footprint), followed closely by lack of lock-in (both GNOME and KDE have a very 'all or nothing' feel to them, and in the case of GNOME, it really is all or nothing). I do have to say though, Okular is by far the best document viewer available for Linux.

A day later noticed that the effect of the cache is missing:
real 4m33.940s user 0m0.862s sys 0m1.711s

That's probably due to something knocking it out of cache overnite.  If
you have a cronjob running nitely to update the locate-variant database,
for example, as many distros do by default, that'd do it, as that scans
pretty much the entire filesystem, typically many times the size of RAM,
thus trashing cache.

The indexer could potentially wipe out cache too, particularly on lower
memory machines, if it's actively indexing files, as that would normally
pull what it's indexing into cache, throwing something else that hasn't
been used for awhile away, unless the indexer is smart enough to do
direct access and thus not disturb cache, since it's single-time access
and caching it isn't going to do anything but force stuff from cache you
use more frequently.
Somehow I doubt that the indexer is using O_DIRECT, while that bypasses cache, it also makes things really slow, which is directly counter to their intent to finish indexing as fast as possible (supposedly to minimize performance impact, but that's at odds with their behavior of trashing the cache). That, and the slowest part of indexing is calling stat() on everything (stat is one of the slowest filesystem related system calls, and is in general usually near the top of the list of calls to avoid when doing real-time work, or for that matter anything that needs to be fast).

As I understand to solve my problem just need to do the cache is always
effective, even if memory occupied by other applications.

Is possible to specify minimal size of disk cache?

AFAIK, not directly.  What happens is that rather than leave the memory
empty, the kernel caches stuff as it reads it.  If the memory is needed
for apps, it's reclaimed from cache and used for apps.  So Linux systems
tend to run close to zero really free memory, unless you just dropped
caches or rebooted, or you just used some memory hog and it's done and
just freed its memory, and you haven't read enough files since then to
fill that memory back up with cache.
Yep, there's no direct way to force it to a fixed size. Personally I would love to be able to reserve some fixed amount of RAM for disk caching, but I also usually run with _a lot_ of swap (usually on the order of 4 to 16 times physical RAM, I do work sometimes on really big images).

However, if you're running swap, there's an adjustment, file
/proc/sys/vm/swappiness, but would be set on most distros using the
sysctrl config (/etc/sysctl.conf and/or /etc/sysctl.d/*), 0-100, that
normally controls the balance preference between swapping apps out to
keep cache (nearer 100) vs. dumping cache to keep more apps in RAM
instead of swapped out (near 0). IIRC the default is 60.
You may also want to look into /proc/sys/vm/vfs_cache_pressure as well, the lower that is, the less likely it is that pages in the filesystem cache will be reclaimed. This tends to have a bigger impact if all you care about is the filesystem cache. Don't set it below about 50 though, otherwise it becomes very easy to run the system out of memory.

Obviously if you're not running swap, all app memory must be kept in
physical RAM as it can't be swapped out, and cache simply uses what's
left.

Pity that I can't do 'echo 3 > /proc/sys/vm/drop_caches' on Windows
machine. It be interesting how fast grep would be work without cache.

FWIW, I jumped off of MS when they started shipping malware[1] as part of
the OS, with eXPrivacy.  So I've no idea if they've something similar,
tho I'd be somewhat surprised if they didn't, at least as some obscure
and possibly undocumented system call, so you'd have to call it from a
program written for that purpose, instead of having it exposed such that
any admin with suitable privs can do it with a single line command using
only shell builtins, as Linux does.
Windows has no way that I know of outside of kernel mode to force the filesystem cache to be dropped. I'm not sure if installing Cygwin and running sync from that works or not, but that's really the only thing I know of that might do it.


Attachment: smime.p7s
Description: S/MIME Cryptographic Signature


[Index of Archives]     [Linux Filesystem Development]     [Linux NFS]     [Linux NILFS]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux