security_task_prctl: why -ENOSYS

[Casey Schaufler]

On 5/26/2026 4:21 PM, William Roberts wrote:
> On Tue, May 26, 2026 at 5:39 PM William Roberts
> <bill.c.roberts at gmail.com> wrote:
>> Hello,
>>
>> I am trying to understand the motivation behind having
>> security_task_prctl only continue if the return value is -ENOSYS. This
>> seems to be very different from other LSM hooks I have investigated.
>> For example, in other hooks, the value from SE Linux avc_has_perms is
>> used directly. This essentially means that a 0 will cause the check to
>> pass, and anything < 0 usually an error.
>>
>> In commit:
>> ----
>> commit d84f4f992cbd76e8f39c488cf0c5d123843923b1 ("CRED: Inaugurate COW
>> credentials")
>>
>> (8) security_task_prctl() and cap_task_prctl().
>>
>>          security_task_prctl() has been modified to return -ENOSYS if it doesn't
>>          want to handle a function, or otherwise return the return
>> value directly
>>          rather than through an argument.
>>
>>          Additionally, cap_task_prctl() now prepares a new set of
>> credentials, even
>>          if it doesn't end up using it.
>> ----
>>
>> The check in kernel/sys.c is currently:
>>         error = security_task_prctl(option, arg2, arg3, arg4, arg5);
>>         if (error != -ENOSYS)
>>                 return error;
>>
>> Should this be something like, "error && error != -ENOSYS"?
>>
>> I ask because I am looking to leverage this hook in SE Linux, and it's
>> annoying to have to coerce all 0 returns to -ENOSYS.
> Of course after hours of banging my head and one email sent, it's more clear to
> me now WHY. This hook isn't meant for making yes or no decisions on an operation
> but rather to also handle special prctl flags for an LSM in question.
>
> I guess with the said, do we want this interface to be used for both
> a, let the lsm handle
> this prctl flag directed to me, as well as a yes/no security decision
> or do we want to split
> this out into two hooks?

The task_prctl hook is used in capability and yama. It is only used to
provide a place to process LSM specific prctl options. It is not used to
make security decisions outside of processing the LSM's options. If you
want to make security decisions on general prctl options you will need
a new hook.

>
>> Thanks,
>> Bill