Class ZestGuidance
- java.lang.Object
-
- edu.berkeley.cs.jqf.fuzz.ei.ZestGuidance
-
- All Implemented Interfaces:
Guidance
- Direct Known Subclasses:
ExecutionIndexingGuidance
public class ZestGuidance extends Object implements Guidance
A guidance that performs coverage-guided fuzzing using two coverage maps, one for all inputs and one for valid inputs only.- Author:
- Rohan Padhye
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
ZestGuidance.Input<K>
A candidate or saved test input that maps objects of type K to bytes.class
ZestGuidance.LinearInput
class
ZestGuidance.SeedInput
-
Field Summary
Fields Modifier and Type Field Description protected File
allInputsDirectory
The directory where all generated inputs are logged in sub-directories (if enabled).protected boolean
blind
Blind fuzzing -- if true then the queue is always empty.protected long
branchCount
Number of conditional jumps since last run was started.protected Console
console
A system console, which is non-null only if STDOUT is a console.protected File
coverageFile
The file contianing the coverage informationprotected ZestGuidance.Input<?>
currentInput
Current input that's running -- valid after getInput() and before handleResult().protected File
currentInputFile
The currently executing input (for debugging purposes).protected int
currentParentInputIdx
Index of currentInput in the savedInputs -- valid after seeds are processed (OK if this is inaccurate).protected int
cyclesCompleted
Number of cycles completed (i.e.protected boolean
DISABLE_SAVE_NEW_COUNTS
Whether to save inputs that only add new coverage bits (but no new responsibilities).protected String
EXACT_CRASH_PATH
save crash to specific location (should be used with EXIT_ON_CRASH)protected boolean
EXIT_ON_CRASH
Whether to stop/exit once a crash is found.protected Thread
firstThread
The first thread in the application, which usually runs the test method.protected boolean
GENERATE_EOF_WHEN_OUT
Whether to generate EOFs when we run out of bytes in the input, instead of randomly generating new bytes.protected long
lastNumTrials
Total execs at last stats refresh.protected Date
lastRefreshTime
Time at last stats refresh.protected boolean
LIBFUZZER_COMPAT_OUTPUT
Use libFuzzer like output instead of AFL like stats screen (https://llvm.org/docs/LibFuzzer.html#output)protected boolean
LOG_ALL_INPUTS
Whether to store all generated inputs to disk (can get slowww!)protected File
logFile
The file where log data is written.protected int
MAX_INPUT_SIZE
Max input size to generate.protected int
maxCoverage
The maximum number of keys covered by any single input found so far.protected long
maxDurationMillis
The max amount of time to run for, in milli-secondsprotected long
maxTrials
The max number of trials to runprotected double
MEAN_MUTATION_COUNT
Mean number of mutations to perform in each round.protected double
MEAN_MUTATION_SIZE
Mean number of contiguous bytes to mutate in each mutation.protected boolean
multiThreaded
Whether the application has more than one thread running coverage-instrumented codeprotected int
NUM_CHILDREN_BASELINE
Baseline number of mutated children to produce from a given parent input.protected int
NUM_CHILDREN_MULTIPLIER_FAVORED
Multiplication factor for number of children to produce for favored inputs.protected int
numChildrenGeneratedForCurrentParentInput
Number of mutated inputs generated from currentInput.protected int
numFavoredLastCycle
Number of favored inputs in the last cycle.protected int
numSavedInputs
Number of saved inputs.protected long
numTrials
The number of trials completed.protected long
numValid
The number of valid inputs.protected File
outputDirectory
The directory where fuzzing results are produced.protected boolean
QUIET_MODE
Whether to hide fuzzing statisticsprotected Random
random
A pseudo-random number generator for generating fresh values.protected Map<Object,ZestGuidance.Input>
responsibleInputs
A mapping of coverage keys to inputs that are responsible for them.protected ICoverage
runCoverage
Coverage statistics for a single run.protected Date
runStart
Date when last run was started.protected boolean
SAVE_ONLY_VALID
Whether to save only valid inputsprotected File
savedCorpusDirectory
The directory where interesting inputs are saved.protected File
savedFailuresDirectory
The directory where saved inputs are saved.protected ArrayList<ZestGuidance.Input>
savedInputs
Set of saved inputs to fuzz.protected Deque<ZestGuidance.Input>
seedInputs
Queue of seeds to fuzz.protected long
singleRunTimeoutMillis
Timeout for an individual run.protected Date
startTime
Time since this guidance instance was created.protected long
STATS_REFRESH_TIME_PERIOD
Minimum amount of time (in millis) between two stats refreshes.protected File
statsFile
The file where saved plot data is written.protected boolean
STEAL_RESPONSIBILITY
Whether to steal responsibility from old inputs (this increases computation cost).protected String
testName
The name of the test for display purposes.protected ICoverage
totalCoverage
Cumulative coverage statistics.protected Set<String>
uniqueFailures
The set of unique failures found so far.protected ICoverage
validCoverage
Cumulative coverage for valid inputs.protected boolean
validityFuzzing
Validity fuzzing -- if true then save valid inputs that increase valid coverageprotected boolean
verbose
Whether to print log statements to stderr (debug option; manually edit).
-
Constructor Summary
Constructors Constructor Description ZestGuidance(String testName, Duration duration, File outputDirectory)
Creates a new Zest guidance instance with seed inputs and optional duration.ZestGuidance(String testName, Duration duration, File outputDirectory, File seedInputDir)
Creates a new Zest guidance instance with seed inputs and optional duration.ZestGuidance(String testName, Duration duration, File outputDirectory, File[] seedFiles)
Creates a new Zest guidance instance with seed inputs and optional duration.ZestGuidance(String testName, Duration duration, Long trials, File outputDirectory, File[] seedInputFiles, Random sourceOfRandomness)
Creates a new Zest guidance instance with seed input files and optional duration, optional trial limit, an possibly deterministic PRNG.ZestGuidance(String testName, Duration duration, Long trials, File outputDirectory, File seedInputDir, Random sourceOfRandomness)
Creates a new Zest guidance instance with seed input directory and optional duration, optional trial limit, an possibly deterministic PRNG.ZestGuidance(String testName, Duration duration, Long trials, File outputDirectory, Random sourceOfRandomness)
Creates a new Zest guidance instance with optional duration, optional trial limit, and possibly deterministic PRNG.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected void
appendLineToFile(File file, String line)
protected List<String>
checkSavingCriteriaSatisfied(Result result)
protected void
completeCycle()
Handles the end of fuzzing cycle (i.e., having gone through the entire queue)protected org.eclipse.collections.impl.set.mutable.primitive.IntHashSet
computeResponsibilities(boolean valid)
protected void
conditionallySynchronize(boolean cond, Runnable task)
Conditionally run a method using synchronization.protected ZestGuidance.Input<?>
createFreshInput()
Spawns a new input from thin air (i.e., actually random)protected InputStream
createParameterStream()
Returns an InputStream that delivers parameters to the generators.protected void
displayStats(boolean force)
Consumer<TraceEvent>
generateCallBack(Thread thread)
Returns a callback generator for a thread's event trace.InputStream
getInput()
Returns a reference to a stream of values return from the pseudo-random number generator.protected String
getStatNames()
protected int
getTargetChildrenForParent(ZestGuidance.Input parentInput)
protected String
getTitle()
ICoverage
getTotalCoverage()
Returns a reference to the coverage statistics.protected void
handleEvent(TraceEvent e)
Handles a trace event generated during test execution.void
handleResult(Result result, Throwable error)
Handles the end of a fuzzing trial.boolean
hasInput()
Returns whether an input is ready for a new trial to be executed.protected void
infoLog(String str, Object... args)
protected String
millisToDuration(long millis)
protected void
saveCurrentInput(org.eclipse.collections.impl.set.mutable.primitive.IntHashSet responsibilities, String why)
void
setBlind(boolean blind)
protected void
updateCoverageFile()
Updates the data in the coverage fileprotected void
writeCurrentInputToFile(File saveFile)
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface edu.berkeley.cs.jqf.fuzz.guidance.Guidance
observeGeneratedArgs, run
-
-
-
-
Field Detail
-
random
protected Random random
A pseudo-random number generator for generating fresh values.
-
testName
protected final String testName
The name of the test for display purposes.
-
maxDurationMillis
protected final long maxDurationMillis
The max amount of time to run for, in milli-seconds
-
maxTrials
protected final long maxTrials
The max number of trials to run
-
numTrials
protected long numTrials
The number of trials completed.
-
numValid
protected long numValid
The number of valid inputs.
-
outputDirectory
protected final File outputDirectory
The directory where fuzzing results are produced.
-
savedCorpusDirectory
protected File savedCorpusDirectory
The directory where interesting inputs are saved.
-
savedFailuresDirectory
protected File savedFailuresDirectory
The directory where saved inputs are saved.
-
allInputsDirectory
protected File allInputsDirectory
The directory where all generated inputs are logged in sub-directories (if enabled).
-
savedInputs
protected ArrayList<ZestGuidance.Input> savedInputs
Set of saved inputs to fuzz.
-
seedInputs
protected Deque<ZestGuidance.Input> seedInputs
Queue of seeds to fuzz.
-
currentInput
protected ZestGuidance.Input<?> currentInput
Current input that's running -- valid after getInput() and before handleResult().
-
currentParentInputIdx
protected int currentParentInputIdx
Index of currentInput in the savedInputs -- valid after seeds are processed (OK if this is inaccurate).
-
numChildrenGeneratedForCurrentParentInput
protected int numChildrenGeneratedForCurrentParentInput
Number of mutated inputs generated from currentInput.
-
cyclesCompleted
protected int cyclesCompleted
Number of cycles completed (i.e. how many times we've reset currentParentInputIdx to 0.
-
numFavoredLastCycle
protected int numFavoredLastCycle
Number of favored inputs in the last cycle.
-
blind
protected boolean blind
Blind fuzzing -- if true then the queue is always empty.
-
validityFuzzing
protected boolean validityFuzzing
Validity fuzzing -- if true then save valid inputs that increase valid coverage
-
numSavedInputs
protected int numSavedInputs
Number of saved inputs. This is usually the same as savedInputs.size(), but we do not really save inputs in TOTALLY_RANDOM mode.
-
runCoverage
protected ICoverage runCoverage
Coverage statistics for a single run.
-
totalCoverage
protected ICoverage totalCoverage
Cumulative coverage statistics.
-
validCoverage
protected ICoverage validCoverage
Cumulative coverage for valid inputs.
-
maxCoverage
protected int maxCoverage
The maximum number of keys covered by any single input found so far.
-
responsibleInputs
protected Map<Object,ZestGuidance.Input> responsibleInputs
A mapping of coverage keys to inputs that are responsible for them.
-
EXACT_CRASH_PATH
protected final String EXACT_CRASH_PATH
save crash to specific location (should be used with EXIT_ON_CRASH)
-
verbose
protected final boolean verbose
Whether to print log statements to stderr (debug option; manually edit).- See Also:
- Constant Field Values
-
console
protected final Console console
A system console, which is non-null only if STDOUT is a console.
-
startTime
protected final Date startTime
Time since this guidance instance was created.
-
lastRefreshTime
protected Date lastRefreshTime
Time at last stats refresh.
-
lastNumTrials
protected long lastNumTrials
Total execs at last stats refresh.
-
STATS_REFRESH_TIME_PERIOD
protected final long STATS_REFRESH_TIME_PERIOD
Minimum amount of time (in millis) between two stats refreshes.- See Also:
- Constant Field Values
-
logFile
protected File logFile
The file where log data is written.
-
statsFile
protected File statsFile
The file where saved plot data is written.
-
currentInputFile
protected File currentInputFile
The currently executing input (for debugging purposes).
-
coverageFile
protected File coverageFile
The file contianing the coverage information
-
LIBFUZZER_COMPAT_OUTPUT
protected final boolean LIBFUZZER_COMPAT_OUTPUT
Use libFuzzer like output instead of AFL like stats screen (https://llvm.org/docs/LibFuzzer.html#output)
-
QUIET_MODE
protected final boolean QUIET_MODE
Whether to hide fuzzing statistics
-
LOG_ALL_INPUTS
protected final boolean LOG_ALL_INPUTS
Whether to store all generated inputs to disk (can get slowww!)
-
singleRunTimeoutMillis
protected long singleRunTimeoutMillis
Timeout for an individual run.
-
runStart
protected Date runStart
Date when last run was started.
-
branchCount
protected long branchCount
Number of conditional jumps since last run was started.
-
EXIT_ON_CRASH
protected final boolean EXIT_ON_CRASH
Whether to stop/exit once a crash is found.
-
firstThread
protected Thread firstThread
The first thread in the application, which usually runs the test method.
-
multiThreaded
protected boolean multiThreaded
Whether the application has more than one thread running coverage-instrumented code
-
SAVE_ONLY_VALID
protected final boolean SAVE_ONLY_VALID
Whether to save only valid inputs
-
MAX_INPUT_SIZE
protected final int MAX_INPUT_SIZE
Max input size to generate.
-
GENERATE_EOF_WHEN_OUT
protected final boolean GENERATE_EOF_WHEN_OUT
Whether to generate EOFs when we run out of bytes in the input, instead of randomly generating new bytes.
-
NUM_CHILDREN_BASELINE
protected final int NUM_CHILDREN_BASELINE
Baseline number of mutated children to produce from a given parent input.- See Also:
- Constant Field Values
-
NUM_CHILDREN_MULTIPLIER_FAVORED
protected final int NUM_CHILDREN_MULTIPLIER_FAVORED
Multiplication factor for number of children to produce for favored inputs.- See Also:
- Constant Field Values
-
MEAN_MUTATION_COUNT
protected final double MEAN_MUTATION_COUNT
Mean number of mutations to perform in each round.- See Also:
- Constant Field Values
-
MEAN_MUTATION_SIZE
protected final double MEAN_MUTATION_SIZE
Mean number of contiguous bytes to mutate in each mutation.- See Also:
- Constant Field Values
-
DISABLE_SAVE_NEW_COUNTS
protected final boolean DISABLE_SAVE_NEW_COUNTS
Whether to save inputs that only add new coverage bits (but no new responsibilities).
-
STEAL_RESPONSIBILITY
protected final boolean STEAL_RESPONSIBILITY
Whether to steal responsibility from old inputs (this increases computation cost).
-
-
Constructor Detail
-
ZestGuidance
public ZestGuidance(String testName, Duration duration, Long trials, File outputDirectory, Random sourceOfRandomness) throws IOException
Creates a new Zest guidance instance with optional duration, optional trial limit, and possibly deterministic PRNG.- Parameters:
testName
- the name of test to display on the status screenduration
- the amount of time to run fuzzing for, wherenull
indicates unlimited time.trials
- the number of trials for which to run fuzzing, wherenull
indicates unlimited trials.outputDirectory
- the directory where fuzzing results will be writtensourceOfRandomness
- a pseudo-random number generator- Throws:
IOException
- if the output directory could not be prepared
-
ZestGuidance
public ZestGuidance(String testName, Duration duration, Long trials, File outputDirectory, File[] seedInputFiles, Random sourceOfRandomness) throws IOException
Creates a new Zest guidance instance with seed input files and optional duration, optional trial limit, an possibly deterministic PRNG.- Parameters:
testName
- the name of test to display on the status screenduration
- the amount of time to run fuzzing for, wherenull
indicates unlimited time.trials
- the number of trials for which to run fuzzing, wherenull
indicates unlimited trials.outputDirectory
- the directory where fuzzing results will be writtenseedInputFiles
- one or more input files to be used as initial inputssourceOfRandomness
- a pseudo-random number generator- Throws:
IOException
- if the output directory could not be prepared
-
ZestGuidance
public ZestGuidance(String testName, Duration duration, Long trials, File outputDirectory, File seedInputDir, Random sourceOfRandomness) throws IOException
Creates a new Zest guidance instance with seed input directory and optional duration, optional trial limit, an possibly deterministic PRNG.- Parameters:
testName
- the name of test to display on the status screenduration
- the amount of time to run fuzzing for, wherenull
indicates unlimited time.trials
- the number of trials for which to run fuzzing, wherenull
indicates unlimited trials.outputDirectory
- the directory where fuzzing results will be writtenseedInputDir
- the directory containing one or more input files to be used as initial inputssourceOfRandomness
- a pseudo-random number generator- Throws:
IOException
- if the output directory could not be prepared
-
ZestGuidance
public ZestGuidance(String testName, Duration duration, File outputDirectory, File seedInputDir) throws IOException
Creates a new Zest guidance instance with seed inputs and optional duration.- Parameters:
testName
- the name of test to display on the status screenduration
- the amount of time to run fuzzing for, wherenull
indicates unlimited time.outputDirectory
- the directory where fuzzing results will be writtenseedInputDir
- the directory containing one or more input files to be used as initial inputs- Throws:
IOException
- if the output directory could not be prepared
-
ZestGuidance
public ZestGuidance(String testName, Duration duration, File outputDirectory) throws IOException
Creates a new Zest guidance instance with seed inputs and optional duration.- Parameters:
testName
- the name of test to display on the status screenduration
- the amount of time to run fuzzing for, wherenull
indicates unlimited time.outputDirectory
- the directory where fuzzing results will be written- Throws:
IOException
- if the output directory could not be prepared
-
ZestGuidance
public ZestGuidance(String testName, Duration duration, File outputDirectory, File[] seedFiles) throws IOException
Creates a new Zest guidance instance with seed inputs and optional duration.- Parameters:
testName
- the name of test to display on the status screenduration
- the amount of time to run fuzzing for, wherenull
indicates unlimited time.outputDirectory
- the directory where fuzzing results will be written- Throws:
IOException
- if the output directory could not be prepared
-
-
Method Detail
-
getStatNames
protected String getStatNames()
-
appendLineToFile
protected void appendLineToFile(File file, String line) throws GuidanceException
- Throws:
GuidanceException
-
millisToDuration
protected String millisToDuration(long millis)
-
displayStats
protected void displayStats(boolean force)
-
updateCoverageFile
protected void updateCoverageFile()
Updates the data in the coverage file
-
getTitle
protected String getTitle()
-
setBlind
public void setBlind(boolean blind)
-
getTargetChildrenForParent
protected int getTargetChildrenForParent(ZestGuidance.Input parentInput)
-
completeCycle
protected void completeCycle()
Handles the end of fuzzing cycle (i.e., having gone through the entire queue)
-
createFreshInput
protected ZestGuidance.Input<?> createFreshInput()
Spawns a new input from thin air (i.e., actually random)- Returns:
- a fresh input
-
createParameterStream
protected InputStream createParameterStream()
Returns an InputStream that delivers parameters to the generators. Note: The variable `currentInput` has been set to point to the input to mutate.- Returns:
- an InputStream that delivers parameters to the generators
-
getInput
public InputStream getInput() throws GuidanceException
Description copied from interface:Guidance
Returns a reference to a stream of values return from the pseudo-random number generator.This method is guaranteed to be invoked by JQF at most once after each invocation of
Guidance.hasInput()
that has returnedtrue
.If
Guidance.hasInput()
returnsfalse
or has not been invoked since the last call toGuidance.getInput()
, then invoking this method may throw an IllegalStateException.- Specified by:
getInput
in interfaceGuidance
- Returns:
- a stream of bytes to be used by the input generator(s)
- Throws:
GuidanceException
- if there was an I/O or other error in generating the input stream
-
hasInput
public boolean hasInput()
Description copied from interface:Guidance
Returns whether an input is ready for a new trial to be executed.This method may block until a new input stream is made ready. If this method returns
false
, then JQF stops fuzzing and this guidance will not be used further.
-
handleResult
public void handleResult(Result result, Throwable error) throws GuidanceException
Description copied from interface:Guidance
Handles the end of a fuzzing trial.This method is guaranteed to be invoked by JQF exactly once after each invocation of
Guidance.getInput()
. Therefore, it is safe to open resources such as files in a call toGuidance.getInput()
and only close them inside this method.If
result
isSUCCESS
, thenerror
is eithernull
or it is an instance of a throwable that is declared by the test method in itsthrows
clause.If
result
isINVALID
, thenerror
is either anAssumptionViolatedException
, if the argument of anassume()
statement wasfalse
, or it is aGuidanceException
, indicating that fuzzing was interrupted during the execution of this trial (and will not continue further).If
result
isFAILURE
, thenerror
is some other throwable that was thrown by the test execution but was not listed in itsthrows
clause. This is the only way to detect test errors. Assertion errors will typically fall into this category. So will other unlisted RuntimeExceptions such as NPE.- Specified by:
handleResult
in interfaceGuidance
- Parameters:
result
- the result of the fuzzing trialerror
- the error thrown during the trial, ornull
- Throws:
GuidanceException
- if there was an I/O or other error in handling the result
-
computeResponsibilities
protected org.eclipse.collections.impl.set.mutable.primitive.IntHashSet computeResponsibilities(boolean valid)
-
writeCurrentInputToFile
protected void writeCurrentInputToFile(File saveFile) throws IOException
- Throws:
IOException
-
saveCurrentInput
protected void saveCurrentInput(org.eclipse.collections.impl.set.mutable.primitive.IntHashSet responsibilities, String why) throws IOException
- Throws:
IOException
-
generateCallBack
public Consumer<TraceEvent> generateCallBack(Thread thread)
Description copied from interface:Guidance
Returns a callback generator for a thread's event trace.The application under test is instrumented such that each thread generates a sequence of
TraceEvent
s that may be handled by a separate callback method (though it may also be the same callback).The callback provided by this method will typically be used for collection execution information such as branch coverage, which in turn is used for constructing the next input stream.
This method is a supplier of event consumers. It is invoked once per new application thread spawned during fuzzing.
- Specified by:
generateCallBack
in interfaceGuidance
- Parameters:
thread
- the thread whose events to handle- Returns:
- a callback that handles trace events generated by that thread
-
handleEvent
protected void handleEvent(TraceEvent e)
Handles a trace event generated during test execution. Not used by FastNonCollidingCoverage, which does not allocate an instance of TraceEvent at each branch probe execution.- Parameters:
e
- the trace event to be handled
-
getTotalCoverage
public ICoverage getTotalCoverage()
Returns a reference to the coverage statistics.- Returns:
- a reference to the coverage statistics
-
conditionallySynchronize
protected void conditionallySynchronize(boolean cond, Runnable task)
Conditionally run a method using synchronization. This is used to handle multi-threaded fuzzing.
-
-