[PATCH v2 3/3] landlock: transpose the layer masks data structure

[Mickaël Salaün]

On Sun, Jan 25, 2026 at 08:58:53PM +0100, Günther Noack wrote:
> The layer masks data structure tracks the requested but unfulfilled
> access rights during an operation's security check.  It stores one bit
> for each combination of access right and layer index.  If the bit is
> set, that access right is not granted (yet) in the given layer and we
> have to traverse the path further upwards to grant it.
> 
> Previously, the layer masks were stored as arrays mapping from access
> right indices to layer_mask_t.  The layer_mask_t value then indicates
> all layers in which the given access right is still (tentatively)
> denied.
> 
> This patch introduces struct layer_access_masks instead: This struct
> contains an array with the access_mask_t of each (tentatively) denied
> access right in that layer.
> 
> The hypothesis of this patch is that this simplifies the code enough
> so that the resulting code will run faster:
> 
> * We can use bitwise operations in multiple places where we previously
>   looped over bits individually with macros.  (Should require less
>   branch speculation and lends itself to better loop unrolling.)
> 
> * Code is ~75 lines smaller.
> 
> Other noteworthy changes:
> 
> * In no_more_access(), call a new helper function may_refer(), which
>   only solves the asymmetric case.  Previously, the code interleaved
>   the checks for the two symmetric cases in RENAME_EXCHANGE.  It feels
>   that the code is clearer when renames without RENAME_EXCHANGE are
>   more obviously the normal case.
> 
> Tradeoffs:
> 
> This change improves performance, at a slight size increase to the
> layer masks data structure.
> 
> At the moment, for the filesystem access rights, the data structure
> has the same size as before, but once we introduce the 17th filesystem
> access right, it will double in size (from 32 to 64 bytes), as
> access_mask_t grows from 16 to 32 bit.  See the link below for
> measurements.
> 
> Link: https://lore.kernel.org/all/20260120.haeCh4li9Vae@digikod.net/

When adding extra links, please add a [1] reference at the end and use
this reference in the commit message.

> Signed-off-by: Günther Noack <gnoack3000 at gmail.com>
> ---
>  security/landlock/access.h  |  10 +-
>  security/landlock/audit.c   |  84 +++------
>  security/landlock/audit.h   |   3 +-
>  security/landlock/domain.c  |  45 +++--
>  security/landlock/domain.h  |   4 +-
>  security/landlock/fs.c      | 352 ++++++++++++++++--------------------
>  security/landlock/net.c     |  11 +-
>  security/landlock/ruleset.c |  88 ++++-----
>  security/landlock/ruleset.h |  21 ++-
>  9 files changed, 274 insertions(+), 344 deletions(-)