[PATCH] Makefile: Introduce CONFIG_ZERO_CALL_USED_REGS
Mark Rutland
mark.rutland at arm.com
Mon May 10 13:45:03 UTC 2021
On Thu, May 06, 2021 at 02:24:18PM -0700, Kees Cook wrote:
> On Thu, May 06, 2021 at 01:54:57PM +0100, Mark Rutland wrote:
> > Hi Kees,
> >
> > On Wed, May 05, 2021 at 12:18:04PM -0700, Kees Cook wrote:
> > > When CONFIG_ZERO_CALL_USED_REGS is enabled, build the kernel with
> > > "-fzero-call-used-regs=used-gpr" (in GCC 11). This option will zero any
> > > caller-used register contents just before returning from a function,
> > > ensuring that temporary values are not leaked beyond the function
> > > boundary. This means that register contents are less likely to be
> > > available for side channel attacks and information exposures.
> > >
> > > Additionally this helps reduce the number of useful ROP gadgets in the
> > > kernel image by about 20%:
> > >
> > > $ ROPgadget.py --nosys --nojop --binary vmlinux.stock | tail -n1
> > > Unique gadgets found: 337245
> > >
> > > $ ROPgadget.py --nosys --nojop --binary vmlinux.zero-call-regs | tail -n1
> > > Unique gadgets found: 267175
> > >
> > > and more notably removes simple "write-what-where" gadgets:
> > >
> > > $ ROPgadget.py --ropchain --binary vmlinux.stock | sed -n '/Step 1/,/Step 2/p'
> > > - Step 1 -- Write-what-where gadgets
> > >
> > > [+] Gadget found: 0xffffffff8102d76c mov qword ptr [rsi], rdx ; ret
> > > [+] Gadget found: 0xffffffff81000cf5 pop rsi ; ret
> > > [+] Gadget found: 0xffffffff8104d7c8 pop rdx ; ret
> > > [-] Can't find the 'xor rdx, rdx' gadget. Try with another 'mov [reg], reg'
> > >
> > > [+] Gadget found: 0xffffffff814c2b4c mov qword ptr [rsi], rdi ; ret
> > > [+] Gadget found: 0xffffffff81000cf5 pop rsi ; ret
> > > [+] Gadget found: 0xffffffff81001e51 pop rdi ; ret
> > > [-] Can't find the 'xor rdi, rdi' gadget. Try with another 'mov [reg], reg'
> > >
> > > [+] Gadget found: 0xffffffff81540d61 mov qword ptr [rsi], rdi ; pop rbx ; pop rbp ; ret
> > > [+] Gadget found: 0xffffffff81000cf5 pop rsi ; ret
> > > [+] Gadget found: 0xffffffff81001e51 pop rdi ; ret
> > > [-] Can't find the 'xor rdi, rdi' gadget. Try with another 'mov [reg], reg'
> > >
> > > [+] Gadget found: 0xffffffff8105341e mov qword ptr [rsi], rax ; ret
> > > [+] Gadget found: 0xffffffff81000cf5 pop rsi ; ret
> > > [+] Gadget found: 0xffffffff81029a11 pop rax ; ret
> > > [+] Gadget found: 0xffffffff811f1c3b xor rax, rax ; ret
> > >
> > > - Step 2 -- Init syscall number gadgets
> > >
> > > $ ROPgadget.py --ropchain --binary vmlinux.zero* | sed -n '/Step 1/,/Step 2/p'
> > > - Step 1 -- Write-what-where gadgets
> > >
> > > [-] Can't find the 'mov qword ptr [r64], r64' gadget
> > >
> > > In parallel build tests, this has a less than 1% performance impact,
> > > and grows the image size less than 1%:
> > >
> > > $ size vmlinux.stock vmlinux.zero-call-regs
> > > text data bss dec hex filename
> > > 22437676 8559152 14127340 45124168 2b08a48 vmlinux.stock
> > > 22453184 8563248 14110956 45127388 2b096dc vmlinux.zero-call-regs
> >
> > FWIW, I gave this a go on arm64, and the size increase is a fair bit
> > larger:
> >
> > | [mark at lakrids:~/src/linux]% ls -l Image*
> > | -rw-r--r-- 1 mark mark 31955456 May 6 13:36 Image.stock
> > | -rw-r--r-- 1 mark mark 33724928 May 6 13:23 Image.zero-call-regs
> >
> > | [mark at lakrids:~/src/linux]% size vmlinux.stock vmlinux.zero-call-regs
> > | text data bss dec hex filename
> > | 20728552 11086474 505540 32320566 1ed2c36 vmlinux.stock
> > | 22500688 11084298 505540 34090526 2082e1e vmlinux.zero-call-regs
> >
> > The Image is ~5.5% bigger, and the .text in the vmlinux is ~8.5% bigger
>
> Woo, that's quite a bit larger! So much so that I struggle to imagine
> the delta. That's almost 1 extra instruction for every 10.
About 31% of this seems to be due to GCC (almost) always clearing x16
and x17 (see further down for numbers). I suspect that's because GCC has
to assume that any (non-static) functions might be reached via a PLT
which would clobber x16 and x17 with specific values.
We also have a bunch of small functions with multiple returns, where
each return path gets the full complement of zeroing instructions, e.g.
Stock:
| <fpsimd_sync_to_sve>:
| d503245f bti c
| f9400001 ldr x1, [x0]
| 7209003f tst w1, #0x800000
| 54000040 b.eq ffff800010014cc4 <fpsimd_sync_to_sve+0x14> // b.none
| d65f03c0 ret
| d503233f paciasp
| a9bf7bfd stp x29, x30, [sp, #-16]!
| 910003fd mov x29, sp
| 97fffdac bl ffff800010014380 <fpsimd_to_sve>
| a8c17bfd ldp x29, x30, [sp], #16
| d50323bf autiasp
| d65f03c0 ret
With zero-call-regs:
| <fpsimd_sync_to_sve>:
| d503245f bti c
| f9400001 ldr x1, [x0]
| 7209003f tst w1, #0x800000
| 540000c0 b.eq ffff8000100152a8 <fpsimd_sync_to_sve+0x24> // b.none
| d2800000 mov x0, #0x0 // #0
| d2800001 mov x1, #0x0 // #0
| d2800010 mov x16, #0x0 // #0
| d2800011 mov x17, #0x0 // #0
| d65f03c0 ret
| d503233f paciasp
| a9bf7bfd stp x29, x30, [sp, #-16]!
| 910003fd mov x29, sp
| 97fffd17 bl ffff800010014710 <fpsimd_to_sve>
| a8c17bfd ldp x29, x30, [sp], #16
| d50323bf autiasp
| d2800000 mov x0, #0x0 // #0
| d2800001 mov x1, #0x0 // #0
| d2800010 mov x16, #0x0 // #0
| d2800011 mov x17, #0x0 // #0
| d65f03c0 ret
... where we go from 12 instructions to 20, which is a ~67% bloat.
> I don't imagine functions are that short. There seem to be only r9..r15 as
> call-used.
We have a bunch of cases like the above. Also note that per the AAPCS a
function can clobber x0-17 (and x18 if it's not reserved for something
like SCS), and I see a few places that clobber x1-x17.
> Even if every one was cleared at every function exit (28
> bytes), that implies 63,290 functions, with an average function size of
> 40 instructions?
I generated some (slightly dodgy) numbers by grepping the output of
objdump:
[mark at lakrids:~/src/linux]% usekorg 10.1.0 aarch64-linux-objdump -d vmlinux.stock | wc -l
3979677
[mark at lakrids:~/src/linux]% usekorg 10.1.0 aarch64-linux-objdump -d vmlinux.stock | grep 'mov\sx[0-9]\+, #0x0' | wc -l
50070
[mark at lakrids:~/src/linux]% usekorg 10.1.0 aarch64-linux-objdump -d vmlinux.stock | grep 'mov\sx1[67], #0x0' | wc -l
1
[mark at lakrids:~/src/linux]% usekorg 10.1.0 aarch64-linux-objdump -d vmlinux.zero-call-regs | wc -l
4422188
[mark at lakrids:~/src/linux]% usekorg 10.1.0 aarch64-linux-objdump -d vmlinux.zero-call-regs | grep 'mov\sx[0-9]\+, #0x0' | wc -l
491371
[mark at lakrids:~/src/linux]% usekorg 10.1.0 aarch64-linux-objdump -d vmlinux.zero-call-regs | grep 'mov\sx1[67], #0x0' | wc -l
135729
That's 441301 new MOVs, and the equivalent of 442511 new instructions
overall. There are 135728 new MOVs to x16 and x17 specifically, which
account for ~31% of that.
Overall we go from MOVs being ~1.3% of all instructions to 11%.
> > The resulting Image appears to work, but I haven't done anything beyond
> > booting, and I wasn't able to get ROPgadget.py going to quantify the
> > number of gadgets.
>
> Does it not like arm64 machine code? I can go check and see if I can get
> numbers...
It's supposed to, and I suspect it works fine, but I wasn't able to get
the tool running at all due to environment problems on my machine.
Thanks,
Mark.
More information about the Linux-security-module-archive
mailing list