security_task_prctl: why -ENOSYS

[William Roberts]

On Tue, May 26, 2026 at 6:42 PM Casey Schaufler <casey at schaufler-ca.com> wrote:
>
> On 5/26/2026 4:21 PM, William Roberts wrote:
> > On Tue, May 26, 2026 at 5:39 PM William Roberts
> > <bill.c.roberts at gmail.com> wrote:
> >> Hello,
> >>
> >> I am trying to understand the motivation behind having
> >> security_task_prctl only continue if the return value is -ENOSYS. This
> >> seems to be very different from other LSM hooks I have investigated.
> >> For example, in other hooks, the value from SE Linux avc_has_perms is
> >> used directly. This essentially means that a 0 will cause the check to
> >> pass, and anything < 0 usually an error.
> >>
> >> In commit:
> >> ----
> >> commit d84f4f992cbd76e8f39c488cf0c5d123843923b1 ("CRED: Inaugurate COW
> >> credentials")
> >>
> >> (8) security_task_prctl() and cap_task_prctl().
> >>
> >>          security_task_prctl() has been modified to return -ENOSYS if it doesn't
> >>          want to handle a function, or otherwise return the return
> >> value directly
> >>          rather than through an argument.
> >>
> >>          Additionally, cap_task_prctl() now prepares a new set of
> >> credentials, even
> >>          if it doesn't end up using it.
> >> ----
> >>
> >> The check in kernel/sys.c is currently:
> >>         error = security_task_prctl(option, arg2, arg3, arg4, arg5);
> >>         if (error != -ENOSYS)
> >>                 return error;
> >>
> >> Should this be something like, "error && error != -ENOSYS"?
> >>
> >> I ask because I am looking to leverage this hook in SE Linux, and it's
> >> annoying to have to coerce all 0 returns to -ENOSYS.
> > Of course after hours of banging my head and one email sent, it's more clear to
> > me now WHY. This hook isn't meant for making yes or no decisions on an operation
> > but rather to also handle special prctl flags for an LSM in question.
> >
> > I guess with the said, do we want this interface to be used for both
> > a, let the lsm handle
> > this prctl flag directed to me, as well as a yes/no security decision
> > or do we want to split
> > this out into two hooks?
>
> The task_prctl hook is used in capability and yama. It is only used to
> provide a place to process LSM specific prctl options. It is not used to
> make security decisions outside of processing the LSM's options. If you
> want to make security decisions on general prctl options you will need
> a new hook.

Yeah I finally saw that after I sent the email, as always :-p, but thanks
Casey for confirming my understanding.

So now for naming... We have security_task_prctl for handling random
prctl interface calls into LSMs. So what should this hook be called?
Perhaps security_task_prctl_check? Should we rename the old hook to
something else?

For now, I will just pick a name, but suggestions are welcome.

As an aside, any lore as to why go through generic prctl vs an
LSM specific filesystem or an lsm specific prctl (we already
have arch_prctl why not lsm_prctl)?





>
> >
> >> Thanks,
> >> Bill