Re: fio memory usage

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]


On Wed, Apr 18, 2012 at 11:54 AM, Jens Axboe <axboe@xxxxxxxxx> wrote:
> On 2012-04-18 20:49, Jens Axboe wrote:
>> On 2012-04-18 17:41, Vikram Seth wrote:
>>> Hi Jens,
>>>
>>> What's the max memory used per job (per device) for fio? Is there a
>>> rule of thumb for min memory needed for fio?
>>> Like, if I am running N #threads with numjobs on M #devices in the
>>> system, then I'd like to know if I have enough memory before I start
>>> the test rather than I wait for it to crashes days later.
>>>
>>> Also, in case fio finds that it is running out of memory while
>>> running, then does it generate an OOM kind of message in the output
>>> file that can be used to track the reason for a crashed test.
>>
>> Generally, any memory that is being used is allocated before fio starts
>> running anything. That's not always strictly true. For verify workloads,
>> fio will store meta data for written blocks. So memory foot print could
>> grow for that. But that's the only case that isn't limited in that
>> sense. Fio will alloc small items while running, but now we are in the
>> sub-kb category. Things that should not fail. And they are continually
>> freed as well, so not persistent items.
>>
>> Usually IO buffers will take the most memory. You can easily calculate
>> that, that would be queue_depth * max_buffer_size * number_of_jobs.
>> Outside of that, fio sets up a shared memory segment. Default on Linux
>> is 32MB. If you use a random workload and don't set norandommap, fio
>> will allocate a device/file sized bitmap for tracking which blocks have
>> been written or not. That consumes 1 byte per block per not-shared file.
>> So for a 500gb drive using 4kb blocks as the minimum IO size, that'd be
>> 122070313 blocks or ~116MB of memory for that bitmap. That'd be your
>> biggest consumer of persistent memory, but one you can usually
>> eliminate.
>>
>> Hope that helps...
>
> Oh, and any sort of logging of IO will also continually allocate memory.
> That'd be options like write_*_log.
>
> And I forgot to touch on what fio does if memory allocations fail. It
> crashes. No attempts are being made at handling memory allocation
> failures. That is something that should most likely be improved. In
> reality most people run on Linux where allocations just don't fail with
> the default settings, so what would happen is that the OOM killer would
> terminate fio instead. I'm not sure what other platforms default to
> here, but this fact has made making handling allocation failures a
> smaller priority. In reality, what should be done, is have fio just exit
> and report status in case of allocation failures. It would not be hard
> to add.
>
> --
> Jens Axboe
>

Thanks Jens. That was really helpful.

OOM killer should output the details in syslog, about what process it
killed etc.
As long as that is true, we have a way to figure out after a crash
whether if fio ran out of memory so that we know where to look
further.

However, I think having fio report fail status in output file on
memory allocation failure, will remove any dependencies on OS and its
reporting of such a failure.

Vikram.
--
To unsubscribe from this list: send the line "unsubscribe fio" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[Home]     [Linux SCSI]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Photo]     [Yosemite News]    [Yosemite Photos]    [Video Projectors]     [Free Online Dating]     [Linux Kernel]     [Linux SCSI]     [XFree86]

Powered by Linux