Class RandomizedRunner
java.lang.Object
org.junit.runner.Runner
com.carrotsearch.randomizedtesting.RandomizedRunner
- All Implemented Interfaces:
org.junit.runner.Describable
,org.junit.runner.manipulation.Filterable
public final class RandomizedRunner
extends org.junit.runner.Runner
implements org.junit.runner.manipulation.Filterable
A
Runner
implementation for running randomized test cases with
predictable and repeatable randomness.
Supports the following JUnit4 features:
BeforeClass
-annotated methods (before all tests of a class/superclass),Before
-annotated methods (before each test),Test
-annotated methods,After
-annotated methods (after each test),AfterClass
-annotated methods (after all tests of a class/superclass),Rule
-annotated fields implementingMethodRule
andTestRule
.
Contracts:
BeforeClass
,Before
methods declared in superclasses are called before methods declared in subclasses,AfterClass
,After
methods declared in superclasses are called after methods declared in subclasses,BeforeClass
,Before
,AfterClass
,After
methods declared within the same class are called in randomized order derived from the master seed (repeatable with the same seed),
Deviations from "standard" JUnit:
- test methods are allowed to return values (the return value is ignored),
- hook methods need not be public; in fact, it is encouraged to make them private to avoid accidental shadowing which silently drops parent hooks from executing (applies to class hooks mostly, but also to instance hooks).
- all exceptions raised during hooks or test case execution are reported to the notifier, there is no suppression or chaining of exceptions,
- a test method must not leave behind any active threads; this is detected
using
ThreadGroup
active counts and is sometimes problematic (many classes in the standard library leave active threads behind without waiting for them to terminate). One can use theThreadLeakScope
,ThreadLeakAction
and other annotations to control how aggressive the detection strategy is and if it fails the test or not. - uncaught exceptions from any of children threads will cause the test to fail.
- See Also:
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionprivate static class
(package private) static class
Queue uncaught exceptions.private static class
(package private) class
Test candidate (model).private static class
Helper tuple (Method, instance params).(package private) static class
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate boolean
static final String
Fake package of a stack trace entry inserted into exceptions thrown by test methods.private final List
<org.junit.runner.notification.RunListener> private ClassModel
Class model.private RunnerContainer
The container we're running in.static final int
The default number of test repeat iterations.static final int
The default number of first interrupts, then Thread.stop attempts.static final int
Time in between interrupt retries or stop retries.static final int
Default timeout for a single test case.static final int
Default timeout for an entire suite.(package private) RandomizedRunner.QueueUncaughtExceptionsHandler
Thread.UncaughtExceptionHandler
for capturing uncaught exceptions from the test group and globally.private final Integer
The number of each test's randomized iterations.(package private) static final Logger
Package scope logger.(package private) static final ThreadGroup
The "main" thread group we will be tracking (including subgroups).private final RandomSupplier
Random class implementation supplier.(package private) final Randomness
The runner's seed (master).(package private) RunnerThreadGroup
All tests are executed under a specified thread group so that we can have some control over how many threads have been started/ stopped.private static final AtomicLong
A sequencer for affecting the initial seed in case of rapid succession of this class instance creations.private Map
<Class<? extends Annotation>, List<Method>> Methods cache.private final Class
<?> The class with test methods (suite).private org.junit.runner.Description
Class suite description.private List
<RandomizedRunner.TestCandidate> All test candidates, processed (seeds assigned) and flattened.private Randomness
Ifinvalid reference
#SYSPROP_RANDOM_SEED
private final TraceFormatting
Stack trace filtering/ dumping.(package private) static AtomicBoolean
A marker for flagging zombie threads (leaked threads that couldn't be killed). -
Constructor Summary
ConstructorsConstructorDescriptionRandomizedRunner
(Class<?> testClass) Creates a new runner for the given class. -
Method Summary
Modifier and TypeMethodDescriptionprivate static List
<RandomizedRunner.TestCandidate> applyFilters
(Class<?> suiteClass, List<RandomizedRunner.TestCandidate> testCandidates, Collection<org.junit.runner.manipulation.Filter> testFilters) (package private) static <T extends Throwable>
TaugmentStackTrace
(T e, Randomness... seeds) Augment stack trace of the given exception with seed infos.private List
<RandomizedRunner.TestCandidate> collectCandidatesForMethod
(Map<String, Integer> descriptionRepetitions, Constructor<?> constructor, RandomizedRunner.TestMethodExecution testCase) Collect test candidates for a single method and the given seed.collectMethodExecutions
(Constructor<?> constructor, List<Method> testMethods) Collect test method executions from list of test methods and potentially parameters from parameter factory methods.private List
<RandomizedRunner.TestCandidate> collectTestCandidates
(org.junit.runner.Description classDescription) Collect all test candidates, regardless if they will be executed or not.private RandomizedContext
Create randomized context for the run.private static String
createDefaultArgumentFormatting
(Constructor<?> constructor) Default formatting string for constructor arguments.private static RunnerContainer
Attempt to detect the container we're running under.private Map
<RandomizedRunner.TestCandidate, Boolean> Determine the set of ignored tests.private int
determineMethodIterationCount
(Method method) private long[]
determineMethodSeeds
(Method method) Determine a given method's initial random seed.private RandomSupplier
determineRandomSupplier
(Class<?> testClass) (package private) static String
emptyToNull
(String value) Normalize empty strings to nulls.void
filter
(org.junit.runner.manipulation.Filter filter) ImplementFilterable
because GUIs depend on it to run tests selectively.private void
fireTestFailure
(org.junit.runner.notification.RunNotifier notifier, org.junit.runner.Description description, Throwable t) private <T> List
<T> getAnnotatedFieldValues
(Object test, Class<? extends Annotation> annotationClass, Class<T> valueClass) private static <T extends Annotation>
List<T> getAnnotationsFromClassHierarchy
(Class<?> clazz, Class<T> annotation) Collect all annotations from a clazz hierarchy.org.junit.runner.Description
Return the current tree of test descriptions (filtered).private InstanceProvider
getInstanceProvider
(Constructor<?> constructor, Object[] args) Determine instance provider.getShuffledMethods
(Class<? extends Annotation> ann) Construct a list of ordered framework methods.Stack trace formatting utilities.private boolean
Returns true if we should ignore this test candidate.static boolean
Returns true if any previous (or current) suite marked withThreadLeakZombies.Consequence.IGNORE_REMAINING_TESTS
has left zombie threads.(package private) void
Invoke a given method on a suiteClass instance (can be null for static methods).private boolean
isConstantSeedForAllIterations
(Method method) Determine if a given method's iterations should run with a fixed seed or not.private boolean
static String
methodName
(org.junit.runner.Description description) Attempts to extract just the method name from parameterized notation.private void
private static void
(package private) void
reportAsIgnored
(org.junit.runner.notification.RunNotifier notifier, GroupEvaluator ge, RandomizedRunner.TestCandidate c) private void
void
run
(org.junit.runner.notification.RunNotifier notifier) Runs all tests and hooks.(package private) void
runSingleTest
(org.junit.runner.notification.RunNotifier notifier, RandomizedRunner.TestCandidate c, ThreadLeakControl threadLeakControl) Runs a single test in the "master" test thread.private void
runSuite
(RandomizedContext context, org.junit.runner.notification.RunNotifier notifier) Test execution logic for the entire suite, executing under designatedRunnerThreadGroup
.private void
runSuite
(org.junit.runner.notification.RunNotifier notifier) Test execution logic for the entire suite.private org.junit.runners.model.Statement
runTestsStatement
(org.junit.runner.notification.RunNotifier notifier, List<RandomizedRunner.TestCandidate> tests, Map<RandomizedRunner.TestCandidate, Boolean> ignored, ThreadLeakControl threadLeakControl) private long[]
seedFromAnnot
(AnnotatedElement element, long randomSeed) Get an annotated element'sSeed
annotation and determine if it's fixed or not.static String
RandomizedRunner
augments stack traces of test methods that ended in an exception and inserts a fake entry starting withAUGMENTED_SEED_PACKAGE
.private void
subscribeListeners
(org.junit.runner.notification.RunNotifier notifier) Subscribe annotation listeners to the notifier.private void
unsubscribeListeners
(org.junit.runner.notification.RunNotifier notifier) Unsubscribe listeners.private void
Validate methods and hooks in the suiteClass.private void
validateTestMethods
(List<Method> testMethods) Perform additional checks on methods returned from the providers.private org.junit.runners.model.Statement
withClassAfters
(org.junit.runners.model.Statement s) private org.junit.runners.model.Statement
withClassBefores
(org.junit.runners.model.Statement s) Decorate aStatement
withBeforeClass
hooks.private org.junit.runners.model.Statement
withClassRules
(org.junit.runners.model.Statement s) Wrap withClassRule
s.private static org.junit.runners.model.Statement
withCloseContextResources
(org.junit.runners.model.Statement s, LifecycleScope scope) Wrap with a rule to close context resources.private org.junit.runners.model.Statement
wrapBeforeAndAfters
(org.junit.runners.model.Statement s, RandomizedRunner.TestCandidate c, Object instance) Wrap before and after hooks.private org.junit.runners.model.Statement
wrapExpectedExceptions
(org.junit.runners.model.Statement s, RandomizedRunner.TestCandidate c) Wrap the given statement into another catching the expected exception, if declared.private org.junit.runners.model.Statement
wrapMethodRules
(org.junit.runners.model.Statement s, RandomizedRunner.TestCandidate c, Object instance) Wrap the given statement in any declared MethodRules (old style rules).private static int
Replace null with zero.Methods inherited from class org.junit.runner.Runner
testCount
-
Field Details
-
AUGMENTED_SEED_PACKAGE
Fake package of a stack trace entry inserted into exceptions thrown by test methods. These stack entries contain additional information about seeds used during execution.- See Also:
-
DEFAULT_TIMEOUT
public static final int DEFAULT_TIMEOUTDefault timeout for a single test case. By default the timeout is disabled. Use global system propertySysGlobals.SYSPROP_TIMEOUT
or an annotationTimeout
if you need to set timeouts or expect some test cases may hang. This will slightly slow down the tests because each test case is executed in a forked thread.- See Also:
-
DEFAULT_TIMEOUT_SUITE
public static final int DEFAULT_TIMEOUT_SUITEDefault timeout for an entire suite. By default the timeout is disabled. Use the global system propertySysGlobals.SYSPROP_TIMEOUT_SUITE
or an annotationTimeoutSuite
if you need to set timeouts or expect some tests (hooks) may hang.- See Also:
-
DEFAULT_KILLATTEMPTS
public static final int DEFAULT_KILLATTEMPTSThe default number of first interrupts, then Thread.stop attempts.- See Also:
-
DEFAULT_KILLWAIT
public static final int DEFAULT_KILLWAITTime in between interrupt retries or stop retries.- See Also:
-
DEFAULT_ITERATIONS
public static final int DEFAULT_ITERATIONSThe default number of test repeat iterations.- See Also:
-
logger
Package scope logger. -
sequencer
A sequencer for affecting the initial seed in case of rapid succession of this class instance creations. Not likely, but can happen two could get the same seed. -
DEFAULT_STACK_FILTERS
-
suiteClass
The class with test methods (suite). -
runnerRandomness
The runner's seed (master). -
testCaseRandomnessOverride
Ifinvalid reference
#SYSPROP_RANDOM_SEED
-
iterationsOverride
The number of each test's randomized iterations.- See Also:
-
testCandidates
All test candidates, processed (seeds assigned) and flattened. -
suiteDescription
private org.junit.runner.Description suiteDescriptionClass suite description. -
runnerThreadGroup
RunnerThreadGroup runnerThreadGroupAll tests are executed under a specified thread group so that we can have some control over how many threads have been started/ stopped. System daemons shouldn't be under this group. -
autoListeners
- See Also:
-
appendSeedParameter
private boolean appendSeedParameter- See Also:
-
traces
Stack trace filtering/ dumping. -
containerRunner
The container we're running in. -
handler
Thread.UncaughtExceptionHandler
for capturing uncaught exceptions from the test group and globally. -
classModel
Class model. -
randomSupplier
Random class implementation supplier. -
shuffledMethodsCache
Methods cache. -
zombieMarker
A marker for flagging zombie threads (leaked threads that couldn't be killed). -
mainThreadGroup
The "main" thread group we will be tracking (including subgroups). -
restoreProperties
-
groupEvaluator
-
-
Constructor Details
-
RandomizedRunner
Creates a new runner for the given class.- Throws:
org.junit.runners.model.InitializationError
-
-
Method Details
-
determineRandomSupplier
-
detectContainer
Attempt to detect the container we're running under. -
getDescription
public org.junit.runner.Description getDescription()Return the current tree of test descriptions (filtered).- Specified by:
getDescription
in interfaceorg.junit.runner.Describable
- Specified by:
getDescription
in classorg.junit.runner.Runner
-
filter
public void filter(org.junit.runner.manipulation.Filter filter) throws org.junit.runner.manipulation.NoTestsRemainException ImplementFilterable
because GUIs depend on it to run tests selectively.- Specified by:
filter
in interfaceorg.junit.runner.manipulation.Filterable
- Throws:
org.junit.runner.manipulation.NoTestsRemainException
-
prune
private static void prune(org.junit.runner.Description suite, Set<org.junit.runner.Description> permitted) -
run
public void run(org.junit.runner.notification.RunNotifier notifier) Runs all tests and hooks.- Specified by:
run
in classorg.junit.runner.Runner
-
restoreSystemProperties
private void restoreSystemProperties() -
processSystemProperties
private void processSystemProperties() -
runSuite
private void runSuite(org.junit.runner.notification.RunNotifier notifier) Test execution logic for the entire suite. -
runSuite
private void runSuite(RandomizedContext context, org.junit.runner.notification.RunNotifier notifier) Test execution logic for the entire suite, executing under designatedRunnerThreadGroup
. -
determineIgnoredTests
private Map<RandomizedRunner.TestCandidate,Boolean> determineIgnoredTests(List<RandomizedRunner.TestCandidate> tests) Determine the set of ignored tests. -
withCloseContextResources
private static org.junit.runners.model.Statement withCloseContextResources(org.junit.runners.model.Statement s, LifecycleScope scope) Wrap with a rule to close context resources. -
runTestsStatement
private org.junit.runners.model.Statement runTestsStatement(org.junit.runner.notification.RunNotifier notifier, List<RandomizedRunner.TestCandidate> tests, Map<RandomizedRunner.TestCandidate, Boolean> ignored, ThreadLeakControl threadLeakControl) -
reportAsIgnored
void reportAsIgnored(org.junit.runner.notification.RunNotifier notifier, GroupEvaluator ge, RandomizedRunner.TestCandidate c) -
fireTestFailure
private void fireTestFailure(org.junit.runner.notification.RunNotifier notifier, org.junit.runner.Description description, Throwable t) -
withClassBefores
private org.junit.runners.model.Statement withClassBefores(org.junit.runners.model.Statement s) Decorate aStatement
withBeforeClass
hooks. -
withClassAfters
private org.junit.runners.model.Statement withClassAfters(org.junit.runners.model.Statement s) -
withClassRules
private org.junit.runners.model.Statement withClassRules(org.junit.runners.model.Statement s) Wrap withClassRule
s. -
runSingleTest
void runSingleTest(org.junit.runner.notification.RunNotifier notifier, RandomizedRunner.TestCandidate c, ThreadLeakControl threadLeakControl) Runs a single test in the "master" test thread. -
wrapBeforeAndAfters
private org.junit.runners.model.Statement wrapBeforeAndAfters(org.junit.runners.model.Statement s, RandomizedRunner.TestCandidate c, Object instance) Wrap before and after hooks. -
wrapExpectedExceptions
private org.junit.runners.model.Statement wrapExpectedExceptions(org.junit.runners.model.Statement s, RandomizedRunner.TestCandidate c) Wrap the given statement into another catching the expected exception, if declared. -
wrapMethodRules
private org.junit.runners.model.Statement wrapMethodRules(org.junit.runners.model.Statement s, RandomizedRunner.TestCandidate c, Object instance) Wrap the given statement in any declared MethodRules (old style rules). -
getAnnotatedFieldValues
private <T> List<T> getAnnotatedFieldValues(Object test, Class<? extends Annotation> annotationClass, Class<T> valueClass) -
createContext
Create randomized context for the run. The context is shared by all threads in a given thread group (but the source ofRandomness
is assigned per-thread). -
subscribeListeners
private void subscribeListeners(org.junit.runner.notification.RunNotifier notifier) Subscribe annotation listeners to the notifier. -
unsubscribeListeners
private void unsubscribeListeners(org.junit.runner.notification.RunNotifier notifier) Unsubscribe listeners. -
applyFilters
private static List<RandomizedRunner.TestCandidate> applyFilters(Class<?> suiteClass, List<RandomizedRunner.TestCandidate> testCandidates, Collection<org.junit.runner.manipulation.Filter> testFilters) -
emptyToNull
Normalize empty strings to nulls. -
hasIgnoreAnnotation
Returns true if we should ignore this test candidate. -
isTestFiltered
-
getShuffledMethods
Construct a list of ordered framework methods. Minor tweaks are done depending on the annotation (reversing order, etc.). -
collectTestCandidates
private List<RandomizedRunner.TestCandidate> collectTestCandidates(org.junit.runner.Description classDescription) Collect all test candidates, regardless if they will be executed or not. At this point individual test methods are also expanded into multiple executions corresponding to the number of iterations (invalid reference
#SYSPROP_ITERATIONS
The order of test candidates is shuffled based on the runner's random.
- See Also:
-
collectCandidatesForMethod
private List<RandomizedRunner.TestCandidate> collectCandidatesForMethod(Map<String, Integer> descriptionRepetitions, Constructor<?> constructor, RandomizedRunner.TestMethodExecution testCase) Collect test candidates for a single method and the given seed. -
zeroForNull
Replace null with zero. -
collectMethodExecutions
public List<RandomizedRunner.TestMethodExecution> collectMethodExecutions(Constructor<?> constructor, List<Method> testMethods) Collect test method executions from list of test methods and potentially parameters from parameter factory methods. -
getInstanceProvider
Determine instance provider. -
createDefaultArgumentFormatting
Default formatting string for constructor arguments. -
isConstantSeedForAllIterations
Determine if a given method's iterations should run with a fixed seed or not. -
determineMethodIterationCount
-
determineMethodSeeds
Determine a given method's initial random seed.- See Also:
-
invoke
Invoke a given method on a suiteClass instance (can be null for static methods).- Throws:
Throwable
-
validateTestMethods
Perform additional checks on methods returned from the providers. -
validateTarget
private void validateTarget()Validate methods and hooks in the suiteClass. Follows "standard" JUnit rules, with some exceptions on return values and more rigorous checking of shadowed methods and fields. -
augmentStackTrace
Augment stack trace of the given exception with seed infos. -
getAnnotationsFromClassHierarchy
private static <T extends Annotation> List<T> getAnnotationsFromClassHierarchy(Class<?> clazz, Class<T> annotation) Collect all annotations from a clazz hierarchy. Superclass's annotations come first.Inherited
annotations are removed (hopefully, the spec. isn't clear on this whether the same object is returned or not for inherited annotations). -
seedFromAnnot
Get an annotated element'sSeed
annotation and determine if it's fixed or not. If it is fixed, return the seeds. Otherwise returnrandomSeed
. -
getTraceFormatting
Stack trace formatting utilities. These may be initialized to filter out certain packages. -
seedFromThrowable
RandomizedRunner
augments stack traces of test methods that ended in an exception and inserts a fake entry starting withAUGMENTED_SEED_PACKAGE
.- Returns:
- A string is returned with seeds combined, if any. Null is returned if no augmentation can be found.
-
methodName
Attempts to extract just the method name from parameterized notation. -
hasZombieThreads
public static boolean hasZombieThreads()Returns true if any previous (or current) suite marked withThreadLeakZombies.Consequence.IGNORE_REMAINING_TESTS
has left zombie threads.
-