security_task_prctl: why -ENOSYS

[Serge E. Hallyn]

On Wed, May 27, 2026 at 11:05:56AM -0500, William Roberts wrote:
> On Tue, May 26, 2026 at 6:42 PM Casey Schaufler <casey at schaufler-ca.com> wrote:
> >
> > On 5/26/2026 4:21 PM, William Roberts wrote:
> > > On Tue, May 26, 2026 at 5:39 PM William Roberts
> > > <bill.c.roberts at gmail.com> wrote:
> > >> Hello,
> > >>
> > >> I am trying to understand the motivation behind having
> > >> security_task_prctl only continue if the return value is -ENOSYS. This
> > >> seems to be very different from other LSM hooks I have investigated.
> > >> For example, in other hooks, the value from SE Linux avc_has_perms is
> > >> used directly. This essentially means that a 0 will cause the check to
> > >> pass, and anything < 0 usually an error.
> > >>
> > >> In commit:
> > >> ----
> > >> commit d84f4f992cbd76e8f39c488cf0c5d123843923b1 ("CRED: Inaugurate COW
> > >> credentials")
> > >>
> > >> (8) security_task_prctl() and cap_task_prctl().
> > >>
> > >>          security_task_prctl() has been modified to return -ENOSYS if it doesn't
> > >>          want to handle a function, or otherwise return the return
> > >> value directly
> > >>          rather than through an argument.
> > >>
> > >>          Additionally, cap_task_prctl() now prepares a new set of
> > >> credentials, even
> > >>          if it doesn't end up using it.
> > >> ----
> > >>
> > >> The check in kernel/sys.c is currently:
> > >>         error = security_task_prctl(option, arg2, arg3, arg4, arg5);
> > >>         if (error != -ENOSYS)
> > >>                 return error;
> > >>
> > >> Should this be something like, "error && error != -ENOSYS"?
> > >>
> > >> I ask because I am looking to leverage this hook in SE Linux, and it's
> > >> annoying to have to coerce all 0 returns to -ENOSYS.
> > > Of course after hours of banging my head and one email sent, it's more clear to
> > > me now WHY. This hook isn't meant for making yes or no decisions on an operation
> > > but rather to also handle special prctl flags for an LSM in question.
> > >
> > > I guess with the said, do we want this interface to be used for both
> > > a, let the lsm handle
> > > this prctl flag directed to me, as well as a yes/no security decision
> > > or do we want to split
> > > this out into two hooks?
> >
> > The task_prctl hook is used in capability and yama. It is only used to
> > provide a place to process LSM specific prctl options. It is not used to
> > make security decisions outside of processing the LSM's options. If you
> > want to make security decisions on general prctl options you will need
> > a new hook.
> 
> Yeah I finally saw that after I sent the email, as always :-p, but thanks
> Casey for confirming my understanding.
> 
> So now for naming... We have security_task_prctl for handling random
> prctl interface calls into LSMs. So what should this hook be called?
> Perhaps security_task_prctl_check? Should we rename the old hook to

How about security_task_prctl_allowed()?  (Mirroring security_uring_*)

Renaming the existing hook security_task_prctl_handle() also wouldn't
be too bad, but that's probably more churn than it's worth.

> something else?
> 
> For now, I will just pick a name, but suggestions are welcome.
> 
> As an aside, any lore as to why go through generic prctl vs an
> LSM specific filesystem or an lsm specific prctl (we already
> have arch_prctl why not lsm_prctl)?
> 
> 
> 
> 
> 
> >
> > >
> > >> Thanks,
> > >> Bill