Re: O_DIRECT and D-Cache issues on ARM

["Naval Saini" <navalnovel@xxxxxxxxx>]

Hi:

Why is this message not getting posted?

Regards,
Naval

On 10/23/08, Naval Saini <navalnovel@xxxxxxxxx> wrote:
> Hi:
>
>  We were facing some issues with O_DIRECT application on an arm 11
>  architecture / linux-2.6.18.5 / uClibc-0.9.28 . We debugged it to some
>  d-cache constraints with its VIPT cache. The arm technical ref manual
>  states the following :-
>
>
>  Problem :-
>
>  Since O_DIRECT on a raw device (such as mkfs.xfs) uses the same buffer
>  as allocated from user space, we have a situation where there are 2
>  VAs (user space and kernel map of RAM) for the same PA. For O_DIRECT
>  we use the API posix_memalign to allocate memory for buffer. While the
>  above condition wants first 14 bits of memory allocated to be same for
>  VA and PA; posix_memalign can only guarantee page_alignment (first 12
>  bits). As a result, we may have an allocation where O_DIRECT would
>  fail. For example, in our case mkfs.xfs utility would fail randomly,
>  and we would get magic number errors at mount time.
>
>  1. If multiple virtual addresses are mapped onto the same physical
>  address then for all mappings of bits [13:12] the virtual addresses
>  must be equal and the same as bits [13:12] of the physical address.
>  The same physical address can be mapped by TLB entries of different
>  page sizes, including page sizes over 4KB. Imposing this requirement
>  on the virtual address is called page coloring.
>  2. Alternatively, if all mappings to a physical address are of a page
>  size equal to 4KB, then the restriction that bits [13:12] of the
>  virtual address must equal bits [13:12] of the physical address is not
>  necessary. Bits [13:12] of all virtual address aliases must still be
>  equal.
>
>
>  Proposed Fix :-
>
>  To fix this allocation problem, we propose a patch for arm specific
>  parts of uclibc and linux kernel (particularly for
>  memalign/posix_memalign path).
>
>  What we need to do is that, when we call memalign (with an alignment
>  need of 16K or greater), we would like to allocate memory pages that
>  follow the above rule. Since, kernel allocates the physical pages, we
>  need to pass a bit from memalign to the linux kernel to indicate that
>  we want first 14 bits of VA and PA addresses to be aligned. We can use
>  the 'prot' flag of mmap (which has unsed bits) to tell so to the
>  kernel (in case of mmap); also, this mail does not cover sbrk.
>
>  Further, for larger than single page allocations, we can never be sure
>  that the memory is contiguous. So we need to check that each of the
>  contiguous parts are separately following the above (first 14 bits)
>  constraint.
>
>  Additionally, we will used some unused bit in page related data
>  structures to indicate that memory is being used by O_DIRECT. (for
>  kernel might want to swap with it, at some point of time)
>
>
>
>  Can people here suggest me if the patch based on above would be okey
>  to be selected into O_DIRECT kernels (for arm 11 or other processors
>  using VIPT cache with similar behaviour).
>
>
>  Regards,
>
> Naval Saini
>  NXP Semiconductors
>

-------------------------------------------------------------------
List admin: http://lists.arm.linux.org.uk/mailman/listinfo/linux-arm
FAQ:        http://www.arm.linux.org.uk/mailinglists/faq.php
Etiquette:  http://www.arm.linux.org.uk/mailinglists/etiquette.php