[PATCH v4 00/12] selftests: kselftest_harness: support using xfail

[Sean Christopherson]

+kvm

On Tue, Mar 05, 2024, Mickaël Salaün wrote:
> On Tue, Mar 05, 2024 at 01:43:14AM -0800, Kees Cook wrote:
> > On Mon, Mar 04, 2024 at 03:39:02PM -0800, Jakub Kicinski wrote:
> > > On Mon, 4 Mar 2024 15:14:04 -0800 Kees Cook wrote:
> > > > > Ugh, I'm guessing vfork() "eats" the signal, IOW grandchild signals,
> > > > > child exits? vfork() and signals.. I'd rather leave to Kees || Mickael.  
> > > > 
> > > > Oh no, that does seem bad. Since Mickaël is also seeing weird issues,
> > > > can we drop the vfork changes for now?
> > > 
> > > Seems doable, but won't be a simple revert. "drop" means we'd need 
> > > to bring ->step back. More or less go back to v3.
> > 
> > I think we have to -- other CIs are now showing the most of seccomp
> > failing now. (And I can confirm this now -- I had only tested seccomp
> > on earlier versions of the series.)
> 
> Sorry for the trouble, I found and fixed the vfork issues.

Heh, you found and fixed _some of_ the vfork issues.  This whole mess completely
breaks existing tests that use TEST_F() and exit() with non-zero values to
indicate failure, including failures that occur during FIXTURE_SETUP().

E.g. all of the KVM selftests that use KVM_ONE_VCPU_TEST() are broken and will
always show all tests as passing.

The below gets things working for KVM selftests again, but (a) I have no idea if
it's a complete fix, (b) I don't know if it will break other users of the harness,
and (c) I don't understand why spawning a grandchild is the default behavior, i.e.
why usage that has zero need of separating teardown from setup+run is subjected to
the complexity of the handful of tests that do.

diff --git a/tools/testing/selftests/kselftest_harness.h b/tools/testing/selftests/kselftest_harness.h
index 4fd735e48ee7..24e95828976f 100644
--- a/tools/testing/selftests/kselftest_harness.h
+++ b/tools/testing/selftests/kselftest_harness.h
@@ -391,7 +391,7 @@
                                fixture_name##_setup(_metadata, &self, variant->data); \
                                /* Let setup failure terminate early. */ \
                                if (_metadata->exit_code) \
-                                       _exit(0); \
+                                       _exit(_metadata->exit_code); \
                                _metadata->setup_completed = true; \
                                fixture_name##_##test_name(_metadata, &self, variant->data); \
                        } else if (child < 0 || child != waitpid(child, &status, 0)) { \
@@ -406,8 +406,10 @@
                } \
                if (_metadata->setup_completed && _metadata->teardown_parent) \
                        fixture_name##_teardown(_metadata, &self, variant->data); \
-               if (!WIFEXITED(status) && WIFSIGNALED(status)) \
-                       /* Forward signal to __wait_for_test(). */ \
+               /* Forward exit codes and signals to __wait_for_test(). */ \
+               if (WIFEXITED(status)) \
+                       _exit(WEXITSTATUS(status)); \
+               else if (WIFSIGNALED(status)) \
                        kill(getpid(), WTERMSIG(status)); \
                __test_check_assert(_metadata); \
        } \