security_task_prctl: why -ENOSYS

[William Roberts]

On Tue, May 26, 2026 at 5:39 PM William Roberts
<bill.c.roberts at gmail.com> wrote:
>
> Hello,
>
> I am trying to understand the motivation behind having
> security_task_prctl only continue if the return value is -ENOSYS. This
> seems to be very different from other LSM hooks I have investigated.
> For example, in other hooks, the value from SE Linux avc_has_perms is
> used directly. This essentially means that a 0 will cause the check to
> pass, and anything < 0 usually an error.
>
> In commit:
> ----
> commit d84f4f992cbd76e8f39c488cf0c5d123843923b1 ("CRED: Inaugurate COW
> credentials")
>
> (8) security_task_prctl() and cap_task_prctl().
>
>          security_task_prctl() has been modified to return -ENOSYS if it doesn't
>          want to handle a function, or otherwise return the return
> value directly
>          rather than through an argument.
>
>          Additionally, cap_task_prctl() now prepares a new set of
> credentials, even
>          if it doesn't end up using it.
> ----
>
> The check in kernel/sys.c is currently:
>         error = security_task_prctl(option, arg2, arg3, arg4, arg5);
>         if (error != -ENOSYS)
>                 return error;
>
> Should this be something like, "error && error != -ENOSYS"?
>
> I ask because I am looking to leverage this hook in SE Linux, and it's
> annoying to have to coerce all 0 returns to -ENOSYS.

Of course after hours of banging my head and one email sent, it's more clear to
me now WHY. This hook isn't meant for making yes or no decisions on an operation
but rather to also handle special prctl flags for an LSM in question.

I guess with the said, do we want this interface to be used for both
a, let the lsm handle
this prctl flag directed to me, as well as a yes/no security decision
or do we want to split
this out into two hooks?

>
> Thanks,
> Bill