1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
|
# JUnit Integration Implementation
Jazzer's JUnit integration starts from
the [`FuzzTest`](../src/main/java/com/code_intelligence/jazzer/junit/FuzzTest.java) annotation. As mentioned in the
annotation's javadoc, our integration runs in one of two modes: fuzzing and regression. Fuzzing mode will generate new
inputs to feed into the tests to find new issues and regression mode will run the tests against previous findings, no
fuzzing is done. The main entrypoints for the actual integration code are found in two of the annotations
on `FuzzTest`: `@ArgumentsSource(FuzzTestArgumentsProvider.class)` and `@ExtendsWith(FuzzTestExtensions.class)`.
Because these same files and functions are involved in two mostly separate sets of functionality, this will look at the
flow of the different methods involved in integrating with JUnit in fuzzing mode (when `JAZZER_FUZZ` is set to any
non-empty value) and in regression mode (when `JAZZER_FUZZ` is not set) separately.
# Fuzzing Flow
JUnit will call the following methods for each test marked with `FuzzTest`.
## `evaluateExecutionCondition`
The first call to this test will determine if the test should be run at all. In fuzzing mode, we only allow one test to
be run due to global state in libfuzzer that would mean multiple tests would interfere with each other. Jazzer will
accept the first fuzz test that is checked as the test to be run. It will cache which test it has seen first and
return that test as enabled.
If this returns that a test is disabled, JUnit will not run the rest of these methods for this test and instead skip
to the next one.
## `provideArguments`
This will configure the fuzzing agent to set up code instrumentation, instantiate a `FuzzTestExecutor` and put it into
JUnit's `extensionContext`, then create a stream of a single empty argument set. As the comment mentions, this is so
that JUnit will actually execute the test but the argument will not be used.
## `evaluateExecutionCondition`
This will be called for each argument set for the current test. In fuzzing mode, there will only be the single
empty argument set which will be enabled.
## `interceptTestTemplateMethod`
This will call `invocation.skip()` which prevents invoking the test function with the default set of
arguments `provideArguments` created. It will instead extract the `FuzzTestExecutor` instance from
the `extensionContext` and then calls `FuzzTestExecutor#execute` which creates a `FuzzTargetRunner` to run the actual
fuzzing.
Crashes are saved in `resources/<package>/<test file name>Inputs/<test method name>` and results that are interesting to
libfuzzer are saved in `.cifuzz-corpus`.
# Regression Flow
Similar to fuzzing mode, JUnit will call these methods for each test marked with `FuzzTest`.
## `evaluateExecutionCondition`
This checks if the given test should be run at all. In regression mode, all tests are run so this will always return
enabled.
## `provideArguments`
This will configure the fuzzing agent as in fuzzing mode, then gather test cases to run from the following sources:
1. A default argument set of just an empty input
2. A set of arguments from the associated resources directory
3. If a `.cifuzz-corpus` directory exists, relevant entries from that are added as well
Prior to returning, the stream of test cases is put through `adaptInputsForFuzzTest` to turn the raw bytes from the
files into the actual types to be given to the tested function.
### Resources Tests
The tests from the resources directory are gathered by `walkInputs`. This will look for inputs in two places:
- `resources/<package>/<test class name>Inputs` - files found directly within this directory will be used as inputs for
any tests within this class. This allows for easy sharing of corpus entries. Jazzer does not automatically put entries
here, instead a human will need to decide a finding should be shared and manually move it.
- `resources/<package>/<test class name>Inputs/<test method name>` - files found in this directory and any directory
under it are used as inputs for only the test of the same name.
JUnit will use the file's name as the name of the test case for its reporting. It also accepts .jar files where it will
search with the given directory in the jar.
### CIFuzz Corpus
The corpus kept in `.cifuzz-corpus/<test class name>/<test method name>` holds any inputs that libfuzzer found worth
saving and not necessarily just inputs that caused a crash. Jazzer is able to set the directory but the contents of
these directories are managed entirely by libfuzzer. Unlike with the resources test inputs above, this will not look
in `.cifuzz-corpus/<test class name>` for shared test cases. This is a limitation of libfuzzer.
## `evaluateExecutionCondition`
This will run once per argument set returned by `provideArguments` for this test. All argument sets will return as
enabled.
## `interceptTestTemplateMethod`
This will run for each individual test case for each fuzz test and will mostly just allow the test function to proceed
with the provided arguments. Prior to the call to the test, it will enable the agent's hooks and then disable them
afterward. It will also check for and report any findings from Jazzer to JUnit.
# Diagrams
Below are two sequence diagrams for how JUnit calls `evaluateExecutionConditions` and `provideArguments` in fuzzing and
regression mode. These diagrams ignore `interceptTestTemplateMethod` for brevity as its behavior and place in the
sequence is more clear.
## Fuzzing
## Regression
|
