[PATCH v4 00/12] selftests: kselftest_harness: support using xfail

[Mickaël Salaün]

On Thu, May 02, 2024 at 11:42:29AM GMT, Sean Christopherson wrote:
> +kvm
> 
> On Tue, Mar 05, 2024, Mickaël Salaün wrote:
> > On Tue, Mar 05, 2024 at 01:43:14AM -0800, Kees Cook wrote:
> > > On Mon, Mar 04, 2024 at 03:39:02PM -0800, Jakub Kicinski wrote:
> > > > On Mon, 4 Mar 2024 15:14:04 -0800 Kees Cook wrote:
> > > > > > Ugh, I'm guessing vfork() "eats" the signal, IOW grandchild signals,
> > > > > > child exits? vfork() and signals.. I'd rather leave to Kees || Mickael.  
> > > > > 
> > > > > Oh no, that does seem bad. Since Mickaël is also seeing weird issues,
> > > > > can we drop the vfork changes for now?
> > > > 
> > > > Seems doable, but won't be a simple revert. "drop" means we'd need 
> > > > to bring ->step back. More or less go back to v3.
> > > 
> > > I think we have to -- other CIs are now showing the most of seccomp
> > > failing now. (And I can confirm this now -- I had only tested seccomp
> > > on earlier versions of the series.)
> > 
> > Sorry for the trouble, I found and fixed the vfork issues.
> 
> Heh, you found and fixed _some of_ the vfork issues.  This whole mess completely
> breaks existing tests that use TEST_F() and exit() with non-zero values to
> indicate failure, including failures that occur during FIXTURE_SETUP().
> 
> E.g. all of the KVM selftests that use KVM_ONE_VCPU_TEST() are broken and will
> always show all tests as passing.
> 
> The below gets things working for KVM selftests again, but (a) I have no idea if
> it's a complete fix, (b) I don't know if it will break other users of the harness,
> and (c) I don't understand why spawning a grandchild is the default behavior, i.e.
> why usage that has zero need of separating teardown from setup+run is subjected to
> the complexity of the handful of tests that do.

Thanks for the fix.  I think it covers almost all cases.  I'd handle the
same way the remaining _exit() though.  The grandchild changes was a
long due patch from the time I added kselftest_harness.h and forked the
TEST_F() macro.  I'll send a new patch series with this fix.

> 
> diff --git a/tools/testing/selftests/kselftest_harness.h b/tools/testing/selftests/kselftest_harness.h
> index 4fd735e48ee7..24e95828976f 100644
> --- a/tools/testing/selftests/kselftest_harness.h
> +++ b/tools/testing/selftests/kselftest_harness.h
> @@ -391,7 +391,7 @@
>                                 fixture_name##_setup(_metadata, &self, variant->data); \
>                                 /* Let setup failure terminate early. */ \
>                                 if (_metadata->exit_code) \
> -                                       _exit(0); \
> +                                       _exit(_metadata->exit_code); \
>                                 _metadata->setup_completed = true; \
>                                 fixture_name##_##test_name(_metadata, &self, variant->data); \
>                         } else if (child < 0 || child != waitpid(child, &status, 0)) { \
> @@ -406,8 +406,10 @@
>                 } \
>                 if (_metadata->setup_completed && _metadata->teardown_parent) \
>                         fixture_name##_teardown(_metadata, &self, variant->data); \
> -               if (!WIFEXITED(status) && WIFSIGNALED(status)) \
> -                       /* Forward signal to __wait_for_test(). */ \
> +               /* Forward exit codes and signals to __wait_for_test(). */ \
> +               if (WIFEXITED(status)) \
> +                       _exit(WEXITSTATUS(status)); \
> +               else if (WIFSIGNALED(status)) \
>                         kill(getpid(), WTERMSIG(status)); \
>                 __test_check_assert(_metadata); \
>         } \
>