Class ZestGuidance
- java.lang.Object
-
- edu.berkeley.cs.jqf.fuzz.ei.ZestGuidance
-
- All Implemented Interfaces:
Guidance
- Direct Known Subclasses:
ExecutionIndexingGuidance
public class ZestGuidance extends Object implements Guidance
A guidance that performs coverage-guided fuzzing using two coverage maps, one for all inputs and one for valid inputs only.- Author:
- Rohan Padhye
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static classZestGuidance.Input<K>A candidate or saved test input that maps objects of type K to bytes.classZestGuidance.LinearInputclassZestGuidance.SeedInput
-
Field Summary
Fields Modifier and Type Field Description protected FileallInputsDirectoryThe directory where all generated inputs are logged in sub-directories (if enabled).protected booleanblindBlind fuzzing -- if true then the queue is always empty.protected longbranchCountNumber of conditional jumps since last run was started.protected ConsoleconsoleA system console, which is non-null only if STDOUT is a console.protected FilecoverageFileThe file contianing the coverage informationprotected ZestGuidance.Input<?>currentInputCurrent input that's running -- valid after getInput() and before handleResult().protected FilecurrentInputFileThe currently executing input (for debugging purposes).protected intcurrentParentInputIdxIndex of currentInput in the savedInputs -- valid after seeds are processed (OK if this is inaccurate).protected intcyclesCompletedNumber of cycles completed (i.e.protected booleanDISABLE_SAVE_NEW_COUNTSWhether to save inputs that only add new coverage bits (but no new responsibilities).protected StringEXACT_CRASH_PATHsave crash to specific location (should be used with EXIT_ON_CRASH)protected booleanEXIT_ON_CRASHWhether to stop/exit once a crash is found.protected ThreadfirstThreadThe first thread in the application, which usually runs the test method.protected booleanGENERATE_EOF_WHEN_OUTWhether to generate EOFs when we run out of bytes in the input, instead of randomly generating new bytes.protected longlastNumTrialsTotal execs at last stats refresh.protected DatelastRefreshTimeTime at last stats refresh.protected booleanLIBFUZZER_COMPAT_OUTPUTUse libFuzzer like output instead of AFL like stats screen (https://llvm.org/docs/LibFuzzer.html#output)protected booleanLOG_ALL_INPUTSWhether to store all generated inputs to disk (can get slowww!)protected FilelogFileThe file where log data is written.protected intMAX_INPUT_SIZEMax input size to generate.protected intmaxCoverageThe maximum number of keys covered by any single input found so far.protected longmaxDurationMillisThe max amount of time to run for, in milli-secondsprotected longmaxTrialsThe max number of trials to runprotected doubleMEAN_MUTATION_COUNTMean number of mutations to perform in each round.protected doubleMEAN_MUTATION_SIZEMean number of contiguous bytes to mutate in each mutation.protected booleanmultiThreadedWhether the application has more than one thread running coverage-instrumented codeprotected intNUM_CHILDREN_BASELINEBaseline number of mutated children to produce from a given parent input.protected intNUM_CHILDREN_MULTIPLIER_FAVOREDMultiplication factor for number of children to produce for favored inputs.protected intnumChildrenGeneratedForCurrentParentInputNumber of mutated inputs generated from currentInput.protected intnumFavoredLastCycleNumber of favored inputs in the last cycle.protected intnumSavedInputsNumber of saved inputs.protected longnumTrialsThe number of trials completed.protected longnumValidThe number of valid inputs.protected FileoutputDirectoryThe directory where fuzzing results are produced.protected booleanQUIET_MODEWhether to hide fuzzing statisticsprotected RandomrandomA pseudo-random number generator for generating fresh values.protected Map<Object,ZestGuidance.Input>responsibleInputsA mapping of coverage keys to inputs that are responsible for them.protected ICoveragerunCoverageCoverage statistics for a single run.protected DaterunStartDate when last run was started.protected booleanSAVE_ONLY_VALIDWhether to save only valid inputsprotected FilesavedCorpusDirectoryThe directory where interesting inputs are saved.protected FilesavedFailuresDirectoryThe directory where saved inputs are saved.protected ArrayList<ZestGuidance.Input>savedInputsSet of saved inputs to fuzz.protected Deque<ZestGuidance.Input>seedInputsQueue of seeds to fuzz.protected longsingleRunTimeoutMillisTimeout for an individual run.protected DatestartTimeTime since this guidance instance was created.protected longSTATS_REFRESH_TIME_PERIODMinimum amount of time (in millis) between two stats refreshes.protected FilestatsFileThe file where saved plot data is written.protected booleanSTEAL_RESPONSIBILITYWhether to steal responsibility from old inputs (this increases computation cost).protected StringtestNameThe name of the test for display purposes.protected ICoveragetotalCoverageCumulative coverage statistics.protected Set<String>uniqueFailuresThe set of unique failures found so far.protected ICoveragevalidCoverageCumulative coverage for valid inputs.protected booleanvalidityFuzzingValidity fuzzing -- if true then save valid inputs that increase valid coverageprotected booleanverboseWhether to print log statements to stderr (debug option; manually edit).
-
Constructor Summary
Constructors Constructor Description ZestGuidance(String testName, Duration duration, File outputDirectory)Creates a new Zest guidance instance with seed inputs and optional duration.ZestGuidance(String testName, Duration duration, File outputDirectory, File seedInputDir)Creates a new Zest guidance instance with seed inputs and optional duration.ZestGuidance(String testName, Duration duration, File outputDirectory, File[] seedFiles)Creates a new Zest guidance instance with seed inputs and optional duration.ZestGuidance(String testName, Duration duration, Long trials, File outputDirectory, File[] seedInputFiles, Random sourceOfRandomness)Creates a new Zest guidance instance with seed input files and optional duration, optional trial limit, an possibly deterministic PRNG.ZestGuidance(String testName, Duration duration, Long trials, File outputDirectory, File seedInputDir, Random sourceOfRandomness)Creates a new Zest guidance instance with seed input directory and optional duration, optional trial limit, an possibly deterministic PRNG.ZestGuidance(String testName, Duration duration, Long trials, File outputDirectory, Random sourceOfRandomness)Creates a new Zest guidance instance with optional duration, optional trial limit, and possibly deterministic PRNG.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected voidappendLineToFile(File file, String line)protected List<String>checkSavingCriteriaSatisfied(Result result)protected voidcompleteCycle()Handles the end of fuzzing cycle (i.e., having gone through the entire queue)protected org.eclipse.collections.impl.set.mutable.primitive.IntHashSetcomputeResponsibilities(boolean valid)protected voidconditionallySynchronize(boolean cond, Runnable task)Conditionally run a method using synchronization.protected ZestGuidance.Input<?>createFreshInput()Spawns a new input from thin air (i.e., actually random)protected InputStreamcreateParameterStream()Returns an InputStream that delivers parameters to the generators.protected voiddisplayStats(boolean force)Consumer<TraceEvent>generateCallBack(Thread thread)Returns a callback generator for a thread's event trace.InputStreamgetInput()Returns a reference to a stream of values return from the pseudo-random number generator.protected StringgetStatNames()protected intgetTargetChildrenForParent(ZestGuidance.Input parentInput)protected StringgetTitle()ICoveragegetTotalCoverage()Returns a reference to the coverage statistics.protected voidhandleEvent(TraceEvent e)Handles a trace event generated during test execution.voidhandleResult(Result result, Throwable error)Handles the end of a fuzzing trial.booleanhasInput()Returns whether an input is ready for a new trial to be executed.protected voidinfoLog(String str, Object... args)protected StringmillisToDuration(long millis)protected voidsaveCurrentInput(org.eclipse.collections.impl.set.mutable.primitive.IntHashSet responsibilities, String why)voidsetBlind(boolean blind)protected voidupdateCoverageFile()Updates the data in the coverage fileprotected voidwriteCurrentInputToFile(File saveFile)-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface edu.berkeley.cs.jqf.fuzz.guidance.Guidance
observeGeneratedArgs, run
-
-
-
-
Field Detail
-
random
protected Random random
A pseudo-random number generator for generating fresh values.
-
testName
protected final String testName
The name of the test for display purposes.
-
maxDurationMillis
protected final long maxDurationMillis
The max amount of time to run for, in milli-seconds
-
maxTrials
protected final long maxTrials
The max number of trials to run
-
numTrials
protected long numTrials
The number of trials completed.
-
numValid
protected long numValid
The number of valid inputs.
-
outputDirectory
protected final File outputDirectory
The directory where fuzzing results are produced.
-
savedCorpusDirectory
protected File savedCorpusDirectory
The directory where interesting inputs are saved.
-
savedFailuresDirectory
protected File savedFailuresDirectory
The directory where saved inputs are saved.
-
allInputsDirectory
protected File allInputsDirectory
The directory where all generated inputs are logged in sub-directories (if enabled).
-
savedInputs
protected ArrayList<ZestGuidance.Input> savedInputs
Set of saved inputs to fuzz.
-
seedInputs
protected Deque<ZestGuidance.Input> seedInputs
Queue of seeds to fuzz.
-
currentInput
protected ZestGuidance.Input<?> currentInput
Current input that's running -- valid after getInput() and before handleResult().
-
currentParentInputIdx
protected int currentParentInputIdx
Index of currentInput in the savedInputs -- valid after seeds are processed (OK if this is inaccurate).
-
numChildrenGeneratedForCurrentParentInput
protected int numChildrenGeneratedForCurrentParentInput
Number of mutated inputs generated from currentInput.
-
cyclesCompleted
protected int cyclesCompleted
Number of cycles completed (i.e. how many times we've reset currentParentInputIdx to 0.
-
numFavoredLastCycle
protected int numFavoredLastCycle
Number of favored inputs in the last cycle.
-
blind
protected boolean blind
Blind fuzzing -- if true then the queue is always empty.
-
validityFuzzing
protected boolean validityFuzzing
Validity fuzzing -- if true then save valid inputs that increase valid coverage
-
numSavedInputs
protected int numSavedInputs
Number of saved inputs. This is usually the same as savedInputs.size(), but we do not really save inputs in TOTALLY_RANDOM mode.
-
runCoverage
protected ICoverage runCoverage
Coverage statistics for a single run.
-
totalCoverage
protected ICoverage totalCoverage
Cumulative coverage statistics.
-
validCoverage
protected ICoverage validCoverage
Cumulative coverage for valid inputs.
-
maxCoverage
protected int maxCoverage
The maximum number of keys covered by any single input found so far.
-
responsibleInputs
protected Map<Object,ZestGuidance.Input> responsibleInputs
A mapping of coverage keys to inputs that are responsible for them.
-
EXACT_CRASH_PATH
protected final String EXACT_CRASH_PATH
save crash to specific location (should be used with EXIT_ON_CRASH)
-
verbose
protected final boolean verbose
Whether to print log statements to stderr (debug option; manually edit).- See Also:
- Constant Field Values
-
console
protected final Console console
A system console, which is non-null only if STDOUT is a console.
-
startTime
protected final Date startTime
Time since this guidance instance was created.
-
lastRefreshTime
protected Date lastRefreshTime
Time at last stats refresh.
-
lastNumTrials
protected long lastNumTrials
Total execs at last stats refresh.
-
STATS_REFRESH_TIME_PERIOD
protected final long STATS_REFRESH_TIME_PERIOD
Minimum amount of time (in millis) between two stats refreshes.- See Also:
- Constant Field Values
-
logFile
protected File logFile
The file where log data is written.
-
statsFile
protected File statsFile
The file where saved plot data is written.
-
currentInputFile
protected File currentInputFile
The currently executing input (for debugging purposes).
-
coverageFile
protected File coverageFile
The file contianing the coverage information
-
LIBFUZZER_COMPAT_OUTPUT
protected final boolean LIBFUZZER_COMPAT_OUTPUT
Use libFuzzer like output instead of AFL like stats screen (https://llvm.org/docs/LibFuzzer.html#output)
-
QUIET_MODE
protected final boolean QUIET_MODE
Whether to hide fuzzing statistics
-
LOG_ALL_INPUTS
protected final boolean LOG_ALL_INPUTS
Whether to store all generated inputs to disk (can get slowww!)
-
singleRunTimeoutMillis
protected long singleRunTimeoutMillis
Timeout for an individual run.
-
runStart
protected Date runStart
Date when last run was started.
-
branchCount
protected long branchCount
Number of conditional jumps since last run was started.
-
EXIT_ON_CRASH
protected final boolean EXIT_ON_CRASH
Whether to stop/exit once a crash is found.
-
firstThread
protected Thread firstThread
The first thread in the application, which usually runs the test method.
-
multiThreaded
protected boolean multiThreaded
Whether the application has more than one thread running coverage-instrumented code
-
SAVE_ONLY_VALID
protected final boolean SAVE_ONLY_VALID
Whether to save only valid inputs
-
MAX_INPUT_SIZE
protected final int MAX_INPUT_SIZE
Max input size to generate.
-
GENERATE_EOF_WHEN_OUT
protected final boolean GENERATE_EOF_WHEN_OUT
Whether to generate EOFs when we run out of bytes in the input, instead of randomly generating new bytes.
-
NUM_CHILDREN_BASELINE
protected final int NUM_CHILDREN_BASELINE
Baseline number of mutated children to produce from a given parent input.- See Also:
- Constant Field Values
-
NUM_CHILDREN_MULTIPLIER_FAVORED
protected final int NUM_CHILDREN_MULTIPLIER_FAVORED
Multiplication factor for number of children to produce for favored inputs.- See Also:
- Constant Field Values
-
MEAN_MUTATION_COUNT
protected final double MEAN_MUTATION_COUNT
Mean number of mutations to perform in each round.- See Also:
- Constant Field Values
-
MEAN_MUTATION_SIZE
protected final double MEAN_MUTATION_SIZE
Mean number of contiguous bytes to mutate in each mutation.- See Also:
- Constant Field Values
-
DISABLE_SAVE_NEW_COUNTS
protected final boolean DISABLE_SAVE_NEW_COUNTS
Whether to save inputs that only add new coverage bits (but no new responsibilities).
-
STEAL_RESPONSIBILITY
protected final boolean STEAL_RESPONSIBILITY
Whether to steal responsibility from old inputs (this increases computation cost).
-
-
Constructor Detail
-
ZestGuidance
public ZestGuidance(String testName, Duration duration, Long trials, File outputDirectory, Random sourceOfRandomness) throws IOException
Creates a new Zest guidance instance with optional duration, optional trial limit, and possibly deterministic PRNG.- Parameters:
testName- the name of test to display on the status screenduration- the amount of time to run fuzzing for, wherenullindicates unlimited time.trials- the number of trials for which to run fuzzing, wherenullindicates unlimited trials.outputDirectory- the directory where fuzzing results will be writtensourceOfRandomness- a pseudo-random number generator- Throws:
IOException- if the output directory could not be prepared
-
ZestGuidance
public ZestGuidance(String testName, Duration duration, Long trials, File outputDirectory, File[] seedInputFiles, Random sourceOfRandomness) throws IOException
Creates a new Zest guidance instance with seed input files and optional duration, optional trial limit, an possibly deterministic PRNG.- Parameters:
testName- the name of test to display on the status screenduration- the amount of time to run fuzzing for, wherenullindicates unlimited time.trials- the number of trials for which to run fuzzing, wherenullindicates unlimited trials.outputDirectory- the directory where fuzzing results will be writtenseedInputFiles- one or more input files to be used as initial inputssourceOfRandomness- a pseudo-random number generator- Throws:
IOException- if the output directory could not be prepared
-
ZestGuidance
public ZestGuidance(String testName, Duration duration, Long trials, File outputDirectory, File seedInputDir, Random sourceOfRandomness) throws IOException
Creates a new Zest guidance instance with seed input directory and optional duration, optional trial limit, an possibly deterministic PRNG.- Parameters:
testName- the name of test to display on the status screenduration- the amount of time to run fuzzing for, wherenullindicates unlimited time.trials- the number of trials for which to run fuzzing, wherenullindicates unlimited trials.outputDirectory- the directory where fuzzing results will be writtenseedInputDir- the directory containing one or more input files to be used as initial inputssourceOfRandomness- a pseudo-random number generator- Throws:
IOException- if the output directory could not be prepared
-
ZestGuidance
public ZestGuidance(String testName, Duration duration, File outputDirectory, File seedInputDir) throws IOException
Creates a new Zest guidance instance with seed inputs and optional duration.- Parameters:
testName- the name of test to display on the status screenduration- the amount of time to run fuzzing for, wherenullindicates unlimited time.outputDirectory- the directory where fuzzing results will be writtenseedInputDir- the directory containing one or more input files to be used as initial inputs- Throws:
IOException- if the output directory could not be prepared
-
ZestGuidance
public ZestGuidance(String testName, Duration duration, File outputDirectory) throws IOException
Creates a new Zest guidance instance with seed inputs and optional duration.- Parameters:
testName- the name of test to display on the status screenduration- the amount of time to run fuzzing for, wherenullindicates unlimited time.outputDirectory- the directory where fuzzing results will be written- Throws:
IOException- if the output directory could not be prepared
-
ZestGuidance
public ZestGuidance(String testName, Duration duration, File outputDirectory, File[] seedFiles) throws IOException
Creates a new Zest guidance instance with seed inputs and optional duration.- Parameters:
testName- the name of test to display on the status screenduration- the amount of time to run fuzzing for, wherenullindicates unlimited time.outputDirectory- the directory where fuzzing results will be written- Throws:
IOException- if the output directory could not be prepared
-
-
Method Detail
-
getStatNames
protected String getStatNames()
-
appendLineToFile
protected void appendLineToFile(File file, String line) throws GuidanceException
- Throws:
GuidanceException
-
millisToDuration
protected String millisToDuration(long millis)
-
displayStats
protected void displayStats(boolean force)
-
updateCoverageFile
protected void updateCoverageFile()
Updates the data in the coverage file
-
getTitle
protected String getTitle()
-
setBlind
public void setBlind(boolean blind)
-
getTargetChildrenForParent
protected int getTargetChildrenForParent(ZestGuidance.Input parentInput)
-
completeCycle
protected void completeCycle()
Handles the end of fuzzing cycle (i.e., having gone through the entire queue)
-
createFreshInput
protected ZestGuidance.Input<?> createFreshInput()
Spawns a new input from thin air (i.e., actually random)- Returns:
- a fresh input
-
createParameterStream
protected InputStream createParameterStream()
Returns an InputStream that delivers parameters to the generators. Note: The variable `currentInput` has been set to point to the input to mutate.- Returns:
- an InputStream that delivers parameters to the generators
-
getInput
public InputStream getInput() throws GuidanceException
Description copied from interface:GuidanceReturns a reference to a stream of values return from the pseudo-random number generator.This method is guaranteed to be invoked by JQF at most once after each invocation of
Guidance.hasInput()that has returnedtrue.If
Guidance.hasInput()returnsfalseor has not been invoked since the last call toGuidance.getInput(), then invoking this method may throw an IllegalStateException.- Specified by:
getInputin interfaceGuidance- Returns:
- a stream of bytes to be used by the input generator(s)
- Throws:
GuidanceException- if there was an I/O or other error in generating the input stream
-
hasInput
public boolean hasInput()
Description copied from interface:GuidanceReturns whether an input is ready for a new trial to be executed.This method may block until a new input stream is made ready. If this method returns
false, then JQF stops fuzzing and this guidance will not be used further.
-
handleResult
public void handleResult(Result result, Throwable error) throws GuidanceException
Description copied from interface:GuidanceHandles the end of a fuzzing trial.This method is guaranteed to be invoked by JQF exactly once after each invocation of
Guidance.getInput(). Therefore, it is safe to open resources such as files in a call toGuidance.getInput()and only close them inside this method.If
resultisSUCCESS, thenerroris eithernullor it is an instance of a throwable that is declared by the test method in itsthrowsclause.If
resultisINVALID, thenerroris either anAssumptionViolatedException, if the argument of anassume()statement wasfalse, or it is aGuidanceException, indicating that fuzzing was interrupted during the execution of this trial (and will not continue further).If
resultisFAILURE, thenerroris some other throwable that was thrown by the test execution but was not listed in itsthrowsclause. This is the only way to detect test errors. Assertion errors will typically fall into this category. So will other unlisted RuntimeExceptions such as NPE.- Specified by:
handleResultin interfaceGuidance- Parameters:
result- the result of the fuzzing trialerror- the error thrown during the trial, ornull- Throws:
GuidanceException- if there was an I/O or other error in handling the result
-
computeResponsibilities
protected org.eclipse.collections.impl.set.mutable.primitive.IntHashSet computeResponsibilities(boolean valid)
-
writeCurrentInputToFile
protected void writeCurrentInputToFile(File saveFile) throws IOException
- Throws:
IOException
-
saveCurrentInput
protected void saveCurrentInput(org.eclipse.collections.impl.set.mutable.primitive.IntHashSet responsibilities, String why) throws IOException- Throws:
IOException
-
generateCallBack
public Consumer<TraceEvent> generateCallBack(Thread thread)
Description copied from interface:GuidanceReturns a callback generator for a thread's event trace.The application under test is instrumented such that each thread generates a sequence of
TraceEvents that may be handled by a separate callback method (though it may also be the same callback).The callback provided by this method will typically be used for collection execution information such as branch coverage, which in turn is used for constructing the next input stream.
This method is a supplier of event consumers. It is invoked once per new application thread spawned during fuzzing.
- Specified by:
generateCallBackin interfaceGuidance- Parameters:
thread- the thread whose events to handle- Returns:
- a callback that handles trace events generated by that thread
-
handleEvent
protected void handleEvent(TraceEvent e)
Handles a trace event generated during test execution. Not used by FastNonCollidingCoverage, which does not allocate an instance of TraceEvent at each branch probe execution.- Parameters:
e- the trace event to be handled
-
getTotalCoverage
public ICoverage getTotalCoverage()
Returns a reference to the coverage statistics.- Returns:
- a reference to the coverage statistics
-
conditionallySynchronize
protected void conditionallySynchronize(boolean cond, Runnable task)Conditionally run a method using synchronization. This is used to handle multi-threaded fuzzing.
-
-