Class RandomizedRunner

java.lang.Object
org.junit.runner.Runner
com.carrotsearch.randomizedtesting.RandomizedRunner
All Implemented Interfaces:
org.junit.runner.Describable, org.junit.runner.manipulation.Filterable

public final class RandomizedRunner extends org.junit.runner.Runner implements org.junit.runner.manipulation.Filterable
A Runner implementation for running randomized test cases with predictable and repeatable randomness.

Supports the following JUnit4 features:

  • BeforeClass-annotated methods (before all tests of a class/superclass),
  • Before-annotated methods (before each test),
  • Test-annotated methods,
  • After-annotated methods (after each test),
  • AfterClass-annotated methods (after all tests of a class/superclass),
  • Rule-annotated fields implementing MethodRule and TestRule.

Contracts:

  • BeforeClass, Before methods declared in superclasses are called before methods declared in subclasses,
  • AfterClass, After methods declared in superclasses are called after methods declared in subclasses,
  • BeforeClass, Before, AfterClass, After methods declared within the same class are called in randomized order derived from the master seed (repeatable with the same seed),

Deviations from "standard" JUnit:

  • test methods are allowed to return values (the return value is ignored),
  • hook methods need not be public; in fact, it is encouraged to make them private to avoid accidental shadowing which silently drops parent hooks from executing (applies to class hooks mostly, but also to instance hooks).
  • all exceptions raised during hooks or test case execution are reported to the notifier, there is no suppression or chaining of exceptions,
  • a test method must not leave behind any active threads; this is detected using ThreadGroup active counts and is sometimes problematic (many classes in the standard library leave active threads behind without waiting for them to terminate). One can use the ThreadLeakScope, ThreadLeakAction and other annotations to control how aggressive the detection strategy is and if it fails the test or not.
  • uncaught exceptions from any of children threads will cause the test to fail.
See Also:
  • Field Details

    • AUGMENTED_SEED_PACKAGE

      public static final String AUGMENTED_SEED_PACKAGE
      Fake package of a stack trace entry inserted into exceptions thrown by test methods. These stack entries contain additional information about seeds used during execution.
      See Also:
    • DEFAULT_TIMEOUT

      public static final int DEFAULT_TIMEOUT
      Default timeout for a single test case. By default the timeout is disabled. Use global system property SysGlobals.SYSPROP_TIMEOUT or an annotation Timeout if you need to set timeouts or expect some test cases may hang. This will slightly slow down the tests because each test case is executed in a forked thread.
      See Also:
    • DEFAULT_TIMEOUT_SUITE

      public static final int DEFAULT_TIMEOUT_SUITE
      Default timeout for an entire suite. By default the timeout is disabled. Use the global system property SysGlobals.SYSPROP_TIMEOUT_SUITE or an annotation TimeoutSuite if you need to set timeouts or expect some tests (hooks) may hang.
      See Also:
    • DEFAULT_KILLATTEMPTS

      public static final int DEFAULT_KILLATTEMPTS
      The default number of first interrupts, then Thread.stop attempts.
      See Also:
    • DEFAULT_KILLWAIT

      public static final int DEFAULT_KILLWAIT
      Time in between interrupt retries or stop retries.
      See Also:
    • DEFAULT_ITERATIONS

      public static final int DEFAULT_ITERATIONS
      The default number of test repeat iterations.
      See Also:
    • logger

      static final Logger logger
      Package scope logger.
    • sequencer

      private static final AtomicLong sequencer
      A sequencer for affecting the initial seed in case of rapid succession of this class instance creations. Not likely, but can happen two could get the same seed.
    • DEFAULT_STACK_FILTERS

      private static final List<String> DEFAULT_STACK_FILTERS
    • suiteClass

      private final Class<?> suiteClass
      The class with test methods (suite).
    • runnerRandomness

      final Randomness runnerRandomness
      The runner's seed (master).
    • testCaseRandomnessOverride

      private Randomness testCaseRandomnessOverride
      If
      invalid reference
      #SYSPROP_RANDOM_SEED
      property is used with two arguments (master:method) then this field contains method-level override.
    • iterationsOverride

      private final Integer iterationsOverride
      The number of each test's randomized iterations.
      See Also:
      • invalid reference
        #SYSPROP_ITERATIONS
    • testCandidates

      private List<RandomizedRunner.TestCandidate> testCandidates
      All test candidates, processed (seeds assigned) and flattened.
    • suiteDescription

      private org.junit.runner.Description suiteDescription
      Class suite description.
    • runnerThreadGroup

      RunnerThreadGroup runnerThreadGroup
      All tests are executed under a specified thread group so that we can have some control over how many threads have been started/ stopped. System daemons shouldn't be under this group.
    • autoListeners

      private final List<org.junit.runner.notification.RunListener> autoListeners
      See Also:
    • appendSeedParameter

      private boolean appendSeedParameter
      See Also:
      • invalid reference
        #SYSPROP_APPEND_SEED
    • traces

      private final TraceFormatting traces
      Stack trace filtering/ dumping.
    • containerRunner

      private RunnerContainer containerRunner
      The container we're running in.
    • handler

      Thread.UncaughtExceptionHandler for capturing uncaught exceptions from the test group and globally.
    • classModel

      private ClassModel classModel
      Class model.
    • randomSupplier

      private final RandomSupplier randomSupplier
      Random class implementation supplier.
    • shuffledMethodsCache

      private Map<Class<? extends Annotation>,List<Method>> shuffledMethodsCache
      Methods cache.
    • zombieMarker

      static AtomicBoolean zombieMarker
      A marker for flagging zombie threads (leaked threads that couldn't be killed).
    • mainThreadGroup

      static final ThreadGroup mainThreadGroup
      The "main" thread group we will be tracking (including subgroups).
    • restoreProperties

      private final Map<String,String> restoreProperties
    • groupEvaluator

      public GroupEvaluator groupEvaluator
  • Constructor Details

    • RandomizedRunner

      public RandomizedRunner(Class<?> testClass) throws org.junit.runners.model.InitializationError
      Creates a new runner for the given class.
      Throws:
      org.junit.runners.model.InitializationError
  • Method Details

    • determineRandomSupplier

      private RandomSupplier determineRandomSupplier(Class<?> testClass)
    • detectContainer

      private static RunnerContainer detectContainer()
      Attempt to detect the container we're running under.
    • getDescription

      public org.junit.runner.Description getDescription()
      Return the current tree of test descriptions (filtered).
      Specified by:
      getDescription in interface org.junit.runner.Describable
      Specified by:
      getDescription in class org.junit.runner.Runner
    • filter

      public void filter(org.junit.runner.manipulation.Filter filter) throws org.junit.runner.manipulation.NoTestsRemainException
      Implement Filterable because GUIs depend on it to run tests selectively.
      Specified by:
      filter in interface org.junit.runner.manipulation.Filterable
      Throws:
      org.junit.runner.manipulation.NoTestsRemainException
    • prune

      private static void prune(org.junit.runner.Description suite, Set<org.junit.runner.Description> permitted)
    • run

      public void run(org.junit.runner.notification.RunNotifier notifier)
      Runs all tests and hooks.
      Specified by:
      run in class org.junit.runner.Runner
    • restoreSystemProperties

      private void restoreSystemProperties()
    • processSystemProperties

      private void processSystemProperties()
    • runSuite

      private void runSuite(org.junit.runner.notification.RunNotifier notifier)
      Test execution logic for the entire suite.
    • runSuite

      private void runSuite(RandomizedContext context, org.junit.runner.notification.RunNotifier notifier)
      Test execution logic for the entire suite, executing under designated RunnerThreadGroup.
    • determineIgnoredTests

      Determine the set of ignored tests.
    • withCloseContextResources

      private static org.junit.runners.model.Statement withCloseContextResources(org.junit.runners.model.Statement s, LifecycleScope scope)
      Wrap with a rule to close context resources.
    • runTestsStatement

      private org.junit.runners.model.Statement runTestsStatement(org.junit.runner.notification.RunNotifier notifier, List<RandomizedRunner.TestCandidate> tests, Map<RandomizedRunner.TestCandidate,Boolean> ignored, ThreadLeakControl threadLeakControl)
    • reportAsIgnored

      void reportAsIgnored(org.junit.runner.notification.RunNotifier notifier, GroupEvaluator ge, RandomizedRunner.TestCandidate c)
    • fireTestFailure

      private void fireTestFailure(org.junit.runner.notification.RunNotifier notifier, org.junit.runner.Description description, Throwable t)
    • withClassBefores

      private org.junit.runners.model.Statement withClassBefores(org.junit.runners.model.Statement s)
      Decorate a Statement with BeforeClass hooks.
    • withClassAfters

      private org.junit.runners.model.Statement withClassAfters(org.junit.runners.model.Statement s)
    • withClassRules

      private org.junit.runners.model.Statement withClassRules(org.junit.runners.model.Statement s)
      Wrap with ClassRules.
    • runSingleTest

      void runSingleTest(org.junit.runner.notification.RunNotifier notifier, RandomizedRunner.TestCandidate c, ThreadLeakControl threadLeakControl)
      Runs a single test in the "master" test thread.
    • wrapBeforeAndAfters

      private org.junit.runners.model.Statement wrapBeforeAndAfters(org.junit.runners.model.Statement s, RandomizedRunner.TestCandidate c, Object instance)
      Wrap before and after hooks.
    • wrapExpectedExceptions

      private org.junit.runners.model.Statement wrapExpectedExceptions(org.junit.runners.model.Statement s, RandomizedRunner.TestCandidate c)
      Wrap the given statement into another catching the expected exception, if declared.
    • wrapMethodRules

      private org.junit.runners.model.Statement wrapMethodRules(org.junit.runners.model.Statement s, RandomizedRunner.TestCandidate c, Object instance)
      Wrap the given statement in any declared MethodRules (old style rules).
    • getAnnotatedFieldValues

      private <T> List<T> getAnnotatedFieldValues(Object test, Class<? extends Annotation> annotationClass, Class<T> valueClass)
    • createContext

      private RandomizedContext createContext(ThreadGroup tg)
      Create randomized context for the run. The context is shared by all threads in a given thread group (but the source of Randomness is assigned per-thread).
    • subscribeListeners

      private void subscribeListeners(org.junit.runner.notification.RunNotifier notifier)
      Subscribe annotation listeners to the notifier.
    • unsubscribeListeners

      private void unsubscribeListeners(org.junit.runner.notification.RunNotifier notifier)
      Unsubscribe listeners.
    • applyFilters

      private static List<RandomizedRunner.TestCandidate> applyFilters(Class<?> suiteClass, List<RandomizedRunner.TestCandidate> testCandidates, Collection<org.junit.runner.manipulation.Filter> testFilters)
    • emptyToNull

      static String emptyToNull(String value)
      Normalize empty strings to nulls.
    • hasIgnoreAnnotation

      private boolean hasIgnoreAnnotation(RandomizedRunner.TestCandidate c)
      Returns true if we should ignore this test candidate.
    • isTestFiltered

      private boolean isTestFiltered(GroupEvaluator ev, RandomizedRunner.TestCandidate c)
    • getShuffledMethods

      private List<Method> getShuffledMethods(Class<? extends Annotation> ann)
      Construct a list of ordered framework methods. Minor tweaks are done depending on the annotation (reversing order, etc.).
    • collectTestCandidates

      private List<RandomizedRunner.TestCandidate> collectTestCandidates(org.junit.runner.Description classDescription)
      Collect all test candidates, regardless if they will be executed or not. At this point individual test methods are also expanded into multiple executions corresponding to the number of iterations (
      invalid reference
      #SYSPROP_ITERATIONS
      ) and the initial method seed is preassigned.

      The order of test candidates is shuffled based on the runner's random.

      See Also:
    • collectCandidatesForMethod

      private List<RandomizedRunner.TestCandidate> collectCandidatesForMethod(Map<String,Integer> descriptionRepetitions, Constructor<?> constructor, RandomizedRunner.TestMethodExecution testCase)
      Collect test candidates for a single method and the given seed.
    • zeroForNull

      private static int zeroForNull(Integer v)
      Replace null with zero.
    • collectMethodExecutions

      public List<RandomizedRunner.TestMethodExecution> collectMethodExecutions(Constructor<?> constructor, List<Method> testMethods)
      Collect test method executions from list of test methods and potentially parameters from parameter factory methods.
    • getInstanceProvider

      private InstanceProvider getInstanceProvider(Constructor<?> constructor, Object[] args)
      Determine instance provider.
    • createDefaultArgumentFormatting

      private static String createDefaultArgumentFormatting(Constructor<?> constructor)
      Default formatting string for constructor arguments.
    • isConstantSeedForAllIterations

      private boolean isConstantSeedForAllIterations(Method method)
      Determine if a given method's iterations should run with a fixed seed or not.
    • determineMethodIterationCount

      private int determineMethodIterationCount(Method method)
      Determine method iteration count based on (first declaration order wins):
      • global property
        invalid reference
        #SYSPROP_ITERATIONS
        .
      • method annotation Repeat.
      • class annotation Repeat.
      • The default (1).
      • determineMethodSeeds

        private long[] determineMethodSeeds(Method method)
        Determine a given method's initial random seed.
        See Also:
      • invoke

        void invoke(Method m, Object instance, Object... args) throws Throwable
        Invoke a given method on a suiteClass instance (can be null for static methods).
        Throws:
        Throwable
      • validateTestMethods

        private void validateTestMethods(List<Method> testMethods)
        Perform additional checks on methods returned from the providers.
      • validateTarget

        private void validateTarget()
        Validate methods and hooks in the suiteClass. Follows "standard" JUnit rules, with some exceptions on return values and more rigorous checking of shadowed methods and fields.
      • augmentStackTrace

        static <T extends Throwable> T augmentStackTrace(T e, Randomness... seeds)
        Augment stack trace of the given exception with seed infos.
      • getAnnotationsFromClassHierarchy

        private static <T extends Annotation> List<T> getAnnotationsFromClassHierarchy(Class<?> clazz, Class<T> annotation)
        Collect all annotations from a clazz hierarchy. Superclass's annotations come first. Inherited annotations are removed (hopefully, the spec. isn't clear on this whether the same object is returned or not for inherited annotations).
      • seedFromAnnot

        private long[] seedFromAnnot(AnnotatedElement element, long randomSeed)
        Get an annotated element's Seed annotation and determine if it's fixed or not. If it is fixed, return the seeds. Otherwise return randomSeed.
      • getTraceFormatting

        public TraceFormatting getTraceFormatting()
        Stack trace formatting utilities. These may be initialized to filter out certain packages.
      • seedFromThrowable

        public static String seedFromThrowable(Throwable t)
        RandomizedRunner augments stack traces of test methods that ended in an exception and inserts a fake entry starting with AUGMENTED_SEED_PACKAGE.
        Returns:
        A string is returned with seeds combined, if any. Null is returned if no augmentation can be found.
      • methodName

        public static String methodName(org.junit.runner.Description description)
        Attempts to extract just the method name from parameterized notation.
      • hasZombieThreads

        public static boolean hasZombieThreads()
        Returns true if any previous (or current) suite marked with ThreadLeakZombies.Consequence.IGNORE_REMAINING_TESTS has left zombie threads.