SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

[Andy Lutomirski]

On Sat, May 25, 2019 at 3:40 PM Xing, Cedric <cedric.xing at intel.com> wrote:
>
> > From: Andy Lutomirski [mailto:luto at amacapital.net]
> > Sent: Friday, May 24, 2019 4:42 PM
> >
> > > On May 24, 2019, at 3:41 PM, Sean Christopherson <sean.j.christopherson at intel.com> wrote:
> > >
> > >> On Fri, May 24, 2019 at 02:27:34PM -0700, Andy Lutomirski wrote:
> > >> On Fri, May 24, 2019 at 1:03 PM Sean Christopherson
> > >> <sean.j.christopherson at intel.com> wrote:
> > >>>
> > >>>> On Fri, May 24, 2019 at 12:37:44PM -0700, Andy Lutomirski wrote:
> > >>>>> On Fri, May 24, 2019 at 11:34 AM Xing, Cedric <cedric.xing at intel.com> wrote:
> > >>>>>
> > >>>>> If "initial permissions" for enclaves are less restrictive than
> > >>>>> shared objects, then it'd become a backdoor for circumventing LSM
> > >>>>> when enclave whitelisting is *not* in place. For example, an
> > >>>>> adversary may load a page, which would otherwise never be executable, as an executable
> > page in EPC.
> > >>>>>
> > >>>>> In the case a RWX page is needed, the calling process has to have
> > >>>>> a RWX page serving as the source for EADD so PROCESS__EXECMEM will
> > >>>>> have been checked. For SGX2, changing an EPC page to RWX is
> > >>>>> subject to FILE__EXECMEM on /dev/sgx/enclave, which I see as a
> > >>>>> security benefit because it only affects the enclave but not the whole process hosting
> > it.
> > >>>>
> > >>>> So the permission would be like FILE__EXECMOD on the source enclave
> > >>>> page, because it would be mapped MAP_ANONYMOUS, PROT_WRITE?
> > >>>> MAP_SHARED, PROT_WRITE isn't going to work because that means you
> > >>>> can modify the file.
> > >>>
> > >>> Was this in response to Cedric's comment, or to my comment?
> > >>
> > >> Yours.  I think that requiring source pages to be actually mapped W
> > >> is not such a great idea.
> > >
> > > I wasn't requiring source pages to be mapped W.  At least I didn't
> > > intend to require W.  What I was trying to say is that SGX could
> > > trigger an EXECMEM check if userspace attempted to EADD or EAUG an
> > > enclave page with RWX permissions, e.g.:
> > >
> > >  if ((SECINFO.PERMS & RWX) == RWX) {
> > >      ret = security_mmap_file(NULL, RWX, ???);
> > >      if (ret)
> > >          return ret;
> > >  }
> > >
> > > But that's a moot point if we add security_enclave_load() or whatever.
> > >
> > >>
> > >>>
> > >>>> I'm starting to think that looking at the source VMA permission
> > >>>> bits or source PTE permission bits is putting a bit too much policy
> > >>>> into the driver as opposed to the LSM.  How about delegating the
> > >>>> whole thing to an LSM hook?  The EADD operation would invoke a new
> > >>>> hook, something like:
> > >>>>
> > >>>> int security_enclave_load_bytes(void *source_addr, struct
> > >>>> vm_area_struct *source_vma, loff_t source_offset, unsigned int
> > >>>> maxperm);
> > >>>>
> > >>>> Then you don't have to muck with mapping anything PROT_EXEC.
> > >>>> Instead you load from a mapping of a file and the LSM applies
> > >>>> whatever policy it feels appropriate.  If the first pass gets
> > >>>> something wrong, the application or library authors can take it up
> > >>>> with the SELinux folks without breaking the whole ABI :)
> > >>>>
> > >>>> (I'm proposing passing in the source_vma because this hook would be
> > >>>> called with mmap_sem held for read to avoid a TOCTOU race.)
> > >>>>
> > >>>> If we go this route, the only substantial change to the existing
> > >>>> driver that's needed for an initial upstream merge is the maxperm
> > >>>> mechanism and whatever hopefully minimal API changes are needed to
> > >>>> allow users to conveniently set up the mappings.  And we don't need
> > >>>> to worry about how to hack around mprotect() calling into the LSM,
> > >>>> because the LSM will actually be aware of SGX and can just do the
> > >>>> right thing.
> > >>>
> > >>> This doesn't address restricting which processes can run which
> > >>> enclaves, it only allows restricting the build flow.  Or are you
> > >>> suggesting this be done in addition to whitelisting sigstructs?
> > >>
> > >> In addition.
> > >>
> > >> But I named the function badly and gave it a bad signature, which
> > >> confused you.  Let's try again:
> > >>
> > >> int security_enclave_load_from_memory(const struct vm_area_struct
> > >> *source, unsigned int maxperm);
> > >
> > > I prefer security_enclave_load(), "from_memory" seems redundant at best.
> >
> > Fine with me.
>
> If we think of EADD as a way of mmap()'ing an enclave file into memory, would this security_enclave_load() be the same as security_mmap_file(source_vma->vm_file, maxperm, MAP_PRIVATE), except that the target is now EPC instead of regular pages?

Hmm, that's clever.  Although it seems plausible that an LSM would
want to allow RX or RWX of a given file page but only in the context
of an approved enclave, so I think it should still be its own hook.

>
> >
> > >
> > >> Maybe some really fancy future LSM would also want loff_t
> > >> source_offset, but it's probably not terribly useful.  This same
> > >> callback would be used for EAUG.
>
> EAUG always zeroes the EPC page before making it available to an enclave. So I don't think there's anything needed to done here.

Duh.  So security_enclave_load_zeros() for EAUG.  See below.

>
> > >>
> > >> Following up on your discussion with Cedric about sigstruct, the
> > >> other callback would be something like:
> > >>
> > >> int security_enclave_init(struct file *sigstruct_file);
>
> I'd still insist in using a pointer rather than a file, for reasons that we've discussed before. For those who can't recall, the major reason is that most implementation would embed SIGSTRUCT into the same file as the enclave (or at least I don't want to prevent anyone from doing so), which could also be part of another file, such as a shared object or even the main executable itself. It could be difficult to obtain a fd in those cases. memfd won't work because it can't retain the same attributes of the original file containing the SIGSTRUCT.
>
> After all, what matters is the attributes associated with the backing file, which could be easily retrieve from vm_file of the covering VMA. So for the sake of flexibility, let's stay with what we've agreed before - a pointer to SIGSTRUCT.

I'm okay with this, except for one nastiness: there's a big difference
between a file that is just a sigstruct and a file that contains
essentially arbitrary data plus a sigstruct at an arbitrary offset.
We could do something tricky like saying that SIGSTRUCT can be in a
file that's just a SIGSTRUCT or it can be in a special SIGSTRUCT ELF
note in a file that isn't just a SIGSTRUCT, but that could be
annoyingly restrictive.

If it's going to be in an arbitrary file, then I think the signature
needs to be more like:

int security_enclave_init(struct vm_area_struct *sigstruct_vma, loff_t
sigstruct_offset, const sgx_sigstruct *sigstruct);

So that the LSM still has the opportunity to base its decision on the
contents of the SIGSTRUCT.  Actually, we need that change regardless.

>
> > >>
> > >> The main issue I see is that we also want to control the enclave's
> > >> ability to have RWX pages or to change a W page to X.  We might also
> > >> want:
> > >>
> > >> int security_enclave_load_zeros(unsigned int maxperm);
> > >
> > > What's the use case for this?  @maxperm will always be at least RW in
> > > this case, otherwise the page is useless to the enclave, and if the
> > > enclave can write the page, the fact that it started as zeros is
> > > irrelevant.
> >
> > This is how EAUG could ask if RWX is okay. If an enclave is internally doing dynamic loading,
> > the it will need a heap page with maxperm = RWX.  (If it’s well designed, it will make it RW
> > and then RX, either by changing SECINFO or by asking the host to mprotect() it, but it still
> > needs the overall RWX mask.).
>
> Any new page EAUG'ed will start in RW (as dictated by SGX ISA). EACCEPTCOPY will then change it to RX. RWX is never needed for all practical purposes. This in fact could be gated by mprotect() and the attributes associated with /dev/sgx/enclave. In the case of SELinux, FILE__EXECMOD is the right attribute and mprotect() will take care of all the rest. I don't see why the driver need a role here.

I find the SDM's discussion of EAUG, EACCEPT, and EACCEPTCOPY to be
extremely confusing.  My copy of the SDM has EACCEPT's SECINFO
argument as "Read access permitted by Non Enclave".  Is that an error?
 And is EACCEPTCOPY just EACCEPT + memcpy or is there some other
fundamental difference?  38.5.7 doesn't even mention EACCEPTCOPY.

Anyway, all my confusion aside, I was talking about the page table,
not the EPCM.  I think the enclave should need permission to write its
own content into a page that will ever become X, and the enclave's
untrusted host library would do this by adding the page with
MAXPERM=RWX and then mapping/mprotecting it as PROT_WRITE and then
(later or simultaneously) PROT_EXEC.

Since SGX2 doesn't seem to have a way to add an initialized page to
EPC after an enclave starts, I guess that it's impossible to have the
enclave do something like dlopen() without MAXPERM=RWX.  So be it.
Maybe someone will find this annoying someday and SGX3 will add
EAUG-but-don't-zero and EACCEPT-with-existing-contents.

>
> >
> > Also, do real SGX1 enclave formats have BSS? If so, then either we need an ioctl or load zeros
> > or user code is going to load from /dev/zero or just from the heap, but the LSM is going to
> > play better with an ioctl, I suspect :)
>
> Yes, it does. But an enclave would either measure BSS, in which case the initial bytes have to be zero or MRENCLAVE will change; or zero BSS explicitly in its initialization code.
>
> But from LSM's perspective it makes no difference than EADD'ing a page with non-zero content. And security_enclave_load(NULL, RW) would take care of it in exactly in the same way.

Sure, I suppose the same hook with NULL parameters would be equivalent.

>
> >
> > >
> > >> An enclave that's going to modify its own code will need memory with
> > >> maxperm = RWX or WX.
>
> With SGX2/EDMM, RWX is *never* needed for all practical purposes.
>
> In theory, in terms of security, no page shall be made executable while it is still being prepared. So W and X shall always be mutually exclusive, regardless it's in EPC or regular memory.
>
> RWX is only needed in SGX1, as a workaround for certain usages, because EPCM permissions can never change at runtime.

As above, I think I disagree.  MAXPERM is intended as an upper bound
on the permissions that a page can ever have, at least until it's
EREMOVEd and re-added.  Since there's no EAUG-but-don't-zero, EAUG
with MAXPERM.W=0 is basically useless because the page can never
contain anything other than zeros, so a dynamically allocated page
that is ever executed has to have MAXPERM=RWX or MAXPERM=WX.  And that
will need special permissions, which I think is consistent with your
recent emails on how this could all map to SELinux permissions.

>
> > >>
> > >> But this is a bit awkward if the LSM's decision depends on the
> > >> sigstruct.  We could get fancy and require that the sigstruct be
> > >> supplied before any EADD operations so that the maxperm decisions can
> > >> depend on the sigstruct.
> > >>
> > >> Am I making more sense now?
> > >
> > > Yep.  Requiring .sigstruct at ECREATE would be trivial.  If we wanted
> > > flexibility we could do:
> > >
> > >   int security_enclave_load(struct file *file, struct vm_area_struct *vma,
> > >                             unsigned long prot);
> > >
> > > And for ultimate flexibility we could pass both .sigstruct and the
> > > file pointer for /dev/sgx/enclave, but that seems a bit ridiculous.
> >
> > I agree.
>
> Loosely speaking, an enclave (including initial contents of all of its pages and their permissions) and its MRENCLAVE are a 1-to-1 correspondence (given the collision resistant property of SHA-2). So only one is needed for a decision, and either one would lead to the same decision. So I don't see anything making any sense here.
>
> Theoretically speaking, if LSM can make a decision at EINIT by means of security_enclave_load(), then security_enclave_load() is never needed.
>
> In practice, I support keeping both because security_enclave_load() can only approve an enumerable set while security_enclave_load() can approve a non-enumerable set of enclaves. Moreover, in order to determine the validity of a MRENCLAVE (as in development of a policy or in creation of a white/black list), system admins will need the audit log produced by security_enclave_load().

I'm confused.  Things like MRSIGNER aren't known until the SIGSTRUCT
shows up.  Also, security_enclave_load() provides no protection
against loading a mishmash of two different enclave files.  I see
security_enclave_init() as "verify this SIGSTRUCT against your policy
on who may sign enclaves and/or grant EXECMOD depending on SIGSTRUCT"
and security_enclave_load() as "implement your EXECMOD / EXECUTE /
WRITE / whatever policy and possibly check enclave files for some
label."

>
> >
> > >
> > > Passing both would allow tying EXECMOD to /dev/sgx/enclave as Cedric
> > > wanted (without having to play games and pass /dev/sgx/enclave to
> > > security_enclave_load()), but I don't think there's anything
> > > fundamentally broken with using .sigstruct for EXECMOD.  It requires
> > > more verbose labeling, but that's not a bad thing.
> >
> > The benefit of putting it on .sigstruct is that it can be per-enclave.
> >
> > As I understand it from Fedora packaging, the way this works on distros is generally that a
> > package will include some files and their associated labels, and, if the package needs EXECMOD,
> > then the files are labeled with EXECMOD and the author of the relevant code might get a dirty
> > look.
> >
> > This could translate to the author of an exclave that needs RWX regions getting a dirty look
> > without leaking this permission into other enclaves.
> >
> > (In my opinion, the dirty looks are actually the best security benefit of the entire concept
> > of LSMs making RWX difficult.  A sufficiently creative attacker can almost always bypass W^X
> > restrictions once they’ve pwned you, but W^X makes it harder to pwn you in the first place,
> > and SELinux makes it really obvious when packaging a program that doesn’t respect W^X.  The
> > upshot is that a lot of programs got fixed.)
>
> I'm lost here. Dynamically linked enclaves, if running on SGX2, would need RW->RX, i.e. FILE__EXECMOD on /dev/sgx/enclave. But they never need RWX, i.e. PROCESS__EXECMEM.

Hmm.  If we want to make this distinction, we need something a big
richer than my proposed callbacks.  A check of the actual mprotect() /
mmap() permissions would also be needed.  Specifically, allowing
MAXPERM=RWX wouldn't imply that PROT_WRITE | PROT_EXEC is allowed.