I am a lecturer at the University of Sydney, where I work on software reliability, including its security-critical cases. I received my Ph.D. in 2017 from the School of EECS at Oregon State University, and did my postdoc at CISPA Helmholtz Center for Information Security, Germany.
My research centers on a single question: how much confidence can we justify that a software system will not fail in operation?
Failures arrive by two routes. Most arrive by chance, through inputs nobody anticipated. Some arrive by design, through an adversary searching for the input that breaks you. The engineering problem is the same either way: find the inputs that provoke failure, measure whether the search was thorough, and estimate what it missed. This also means that the same techniques serve both reliability engineering and security. Fuzzing is the clearest case: it is automated test generation, and it is also how most modern vulnerabilities are found. I treat security as the adversarial corner of reliability, and the work below is organized accordingly.
The question is most urgent where failure is expensive. In high-consequence software such as industrial control, instrumentation, medical devices, and critical infrastructure, just finding bugs is not enough. We also need to have confidence in the reliability of the software, and hence we need to evaluate whether the evidence gathered justifies the confidence being claimed. That is a quantitative question, and it is largely unsolved.
Dependability engineering names four means of getting there: preventing faults, removing them, tolerating those that remain, and forecasting what is left. My work covers the last three, in five parts that build on one another:
- What remains? After a test campaign ends, how many faults are still there?
- Can we trust the measurement? Is our estimate of software test quality (and hence of the software it tests) sound?
- How do we provoke the failures? Reaching a fault requires inputs the system will accept, whether we are testing or attacking.
- How do we diagnose one? A failure report is only useful if we can act on it.
- What if the damage is already done? Corrupted data has to be recovered.
Residual risk
The oldest question in software reliability is when to stop. Classical software reliability growth models answer it by fitting a curve to the arrival of failures over time and extrapolating to the faults not yet seen. A more recent proposal borrows from ecology: species richness estimation, which infers how many species exist in a population from how often each one has been observed, treating coverage elements and killable faults as the species.
My interest is in whether these methods hold up. The field proposes metrics faster than it validates them, and a reliability estimate that is wrong does more damage than no estimate at all, because someone will act on it. Many of my findings here have been negative, and my focus has been on establishing the credibility, or lack of it, of these metrics.
The thread starts with a result on residual defects. We were the first, and to date the only ones, to find evidence that mutation score (injected fault detection score) and coverage are inversely related to the residual defect density of a program (FSE 2016): the number of live mutants remaining is related to the number of real bugs remaining.
We then asked whether richness estimators could count the killable mutants (injected faults) in a program directly. Across twelve frequency-based models and ten mature projects, they could not. The estimators lacked the predictive power to be useful (ESEM 2024). While the result is negative, it told us where the difficulty lies: in the sampling process.
Applied to coverage, the problem is harder still, because there is no ground truth to check an estimate against. We proposed an evaluation framework that synthesizes large programs with complex control flow and known reachability, paired with a reliability check that works on real programs without ground truth, by varying the size of the sampling unit (ICSME 2025). A further complication is that modern test generators use coverage feedback, which biases the sample adaptively. We are testing the hypothesis that this bias is minimized when singletons (coverage seen exactly once) equal doubletons (seen exactly twice), which would give a principled stopping criterion for a campaign (NDSS Workshop 2026). Most recently we asked whether parametric estimators would beat non-parametric ones by assuming a distribution for coverage discovery. Fitting Poisson, Exponential, Gamma, Gamma–Poisson, Negative Binomial, and Zipf–Mandelbrot models across seven benchmarks, we found that a better distributional fit does not yield better estimates (ISSRE 2026).
This bears directly on assurance. A safety case or an assurance argument must justify a claimed level of confidence with evidence, and the usual evidence is a test campaign that has ended. If we cannot say what that campaign missed, we cannot say what the claim is worth. There is a long-running argument in the software safety literature that software reliability cannot be quantified to the levels critical systems demand, and our results so far support the skeptical side of it: the estimators available today are not dependable enough to carry a confidence claim on their own. Establishing that clearly is more useful to a practitioner than another proposed metric with an untested claim attached. Whether any estimator can tell us when a campaign has genuinely saturated, and with what confidence, remains an open question, and one of my focus areas.
Measurement validity
Any claim about residual risk rests on a measurement of test quality, so the measurement has to be sound. Checking whether the field’s accepted measures survive examination has been a consistent thread in my work, and the answer has often been that they do not. Mutation analysis, which works by seeding artificial faults and counting how many the tests detect, is the best instrument we have. My Ph.D. was devoted to making it usable on real systems.
In dependability terms this is software fault injection, and fault injection does two jobs at once. It serves fault removal, by exposing tests that fail to detect what they should. It also serves fault forecasting: the estimation work above runs on mutation analysis, and the link between mutation score and residual defects is what makes that forecast possible at all. I treat the two separately here because an instrument has to be trusted before the estimates built on it mean anything.
I first asked whether seeded faults resemble real ones. Examining over 5,371 projects in four languages, we found the faults used by mutation analysis are simplistic compared to real-world bugs in terms of the size of the code change (ISSRE 2014). To reduce its cost I developed an algorithm exploiting execution redundancy between similar mutants (ICSE 2016), and showed how combinatorial evaluation can identify equivalent mutants (ISSRE 2015).
I then tested the prevailing belief that mutants should be selected rather than sampled. Comparing the theoretical best selection methods against random sampling, I found that even under oracular knowledge of test kills, selection can be at best less than 20% better than random sampling, and is often much worse (ICSE 2016). There is no such ceiling on the gains from adding operators, which says effort belongs in finding new operators rather than discarding existing ones. This settled a long-standing debate on mutation reduction strategies in favor of random sampling. Finally, we proved the coupling effect theoretically and quantified it empirically (ICST 2017), clarifying how the simple faults mutants represent relate to the higher order faults common in real programs.
Coverage is the other common instrument, and it is widely misread. Our work found that statement coverage, not branch or path coverage, is the better predictor of mutation score across more than 200 real-world projects (ICSE 2014), contradicting the prevailing wisdom of the time. We later settled how test suite size should be accounted for in empirical evaluations (ASE 2020).
These instruments now do work they were not built for. Automated test generators are judged almost entirely by coverage reached and crashes found, both of which saturate and invite overfitting. Mutation score is the better yardstick, but evaluating each mutant independently made it unaffordable. We set out the obstacles (arXiv 2022), then showed that pooling multiple mutations into a single execution brings the cost down far enough to compare generators by mutation score for the first time (Usenix Security 2023). Mutants also serve as intermediate targets: splitting a generation budget between a program and its mutants explores more behavior than spending all of it on the program (NDSS Workshop 2022).
Fuzzing and test generation
None of the above is measurable without inputs that reach the code. Fuzzing generates large volumes of unexpected and possibly invalid input and watches for anomalous behavior. It is the cheapest way to get them, and it is also the dominant technique in vulnerability discovery. A system that rejects every invalid input and behaves correctly on valid ones is robust under fuzzing, and fuzzing it before release finds the failures before users and attackers do.
This work produced the fuzzing book,
an open textbook now used by students and practitioners worldwide.
It takes a reader from simple random generators through fuzzers that analyze the
system under test to infer its expected inputs and use feedback from earlier runs
to steer later ones.
The hard part is reaching deep code. Most systems accept only structured input, and a generator that cannot produce valid structure never gets past the parser. Real systems compound this: an HTTP request wrapping a JSON object encoding an RPC call encoding a custom structure defeats coverage-guided fuzzing entirely, because the paths explored are identical for simple and complex inputs.
Our first approach generated valid inputs against an unmodified parser. Symbolic execution fails here through path explosion, so we built a lightweight alternative, Pygmalion, which iteratively corrects a generated prefix until it is accepted. It works for single pass parsers (PLDI 2019), for parsers with a lexical stage (ISSTA 2020), and even for systems that cannot be instrumented, such as embedded and remote systems. That constraint is common in security testing, where the target is frequently a binary nobody can recompile.
Correcting one input at a time is still expensive. So we built Mimid, which recovers the input structure a parser expects as a context-free grammar by dynamic analysis of program runs (FSE 2020), covering the full range from ad hoc handwritten parsers to parser combinators. With a grammar in hand the bottleneck moves to generation speed, so we adapted ideas from language implementation and virtual machine optimization to build the F1 fuzzer, which produces millions of inputs per second.
Since then we have pushed inference in several directions. Reimplementing the GLADE algorithm, we found its reported effectiveness overly optimistic and in some cases measured against the wrong language (PLDI 2022). Replication matters here, because grammar inference results are easy to overstate. CLIFuzzer mines the valid command-line invocations of a utility into a grammar (FSE 2022), and FormatFuzzer compiles a binary template into a parser, mutator, and generator for structured binary formats such as MP4 and ZIP, finding previously unknown memory errors in ffmpeg and timidity (TOSEM 2024).
The techniques hold up outside the lab. With an industrial partner we reverse-engineered the protocol accepted by a virtualized packet processing engine, with no access to source code or internal documentation, inferring its grammar at an F1 score of 0.94 and driving a full blackbox test campaign from it (ISSRE 2025). Blackbox conditions of this kind are the norm in industrial and security settings, where instrumentation is barred by legal, operational, or safety constraints.
Failure diagnosis
A detected failure is only useful if someone can act on it,
and generated inputs are typically enormous and unreadable.
Test case reduction shrinks them, but a minimal input still does not say what
went wrong, and casual inspection often suggests the wrong hypothesis.
We built DDSET,
which identifies the parts of an input responsible for the failure and abstracts
away the rest.
The resulting evocative patterns — for example ((<expr>)) when nested
parentheses are the cause — are precise and readable.
This work received the ACM SIGSOFT Distinguished Paper award
(ISSTA 2020).
An evocative pattern is a specialization of the input grammar. At ICSE 2021 we showed how to turn a base grammar and a pattern into a specialized grammar guaranteed to produce the evocative fragment in every input, and how to combine patterns under conjunction, disjunction, and negation to form evocative expressions.
The expression above specializes a JSON grammar so that every input has at least one empty key and no null key values, while still parsing any input meeting that specification. Patterns can be written by hand or mined from existing bugs with DDSET, and the expressions serve both as precise generators and as semantic pattern matchers in the spirit of Semgrep.
Reduction itself needed work. Delta debugging guarantees 1-minimality but pays quadratically for it, restarting at every partition level. Re-examining ddmin, we showed restarts are needed only at the single-element level to preserve 1-minimality, and that the quadratic worst case comes from causal chains rather than restarts. drdd is a drop-in replacement keeping the guarantee while dropping the redundant restarts, with a tunable restart budget trading minimality against linear worst-case behavior (ISSRE 2026).
Data repair
Not every fault can be removed before deployment, and not every damaged input is the program’s fault. Data arrives corrupted through entry error, truncated transmission, storage decay, inconsistent formatting, and specifications that changed underneath it. The usual response is to drop the affected records, which is a data loss decision dressed up as a correctness decision.
Where the data can be regenerated, that is merely wasteful. Where it cannot — a one-off experiment, a monitoring record, an instrument stream that will never be replayed — discarding is not an acceptable answer, and repair becomes a reliability requirement.
The obstacle is that established repair methods need a format specification, and frequently there is none to be had. Long-lived archives are the sharp case: formats drift across decades, tooling is retired, and the specification is often the first thing lost. εRepair works without one, using parser feedback alone to locate and correct inconsistencies. It produces repairs 2.6 times higher in quality than ddmax, measured by the edits needed to restore the data, while losing 2.8 times less of it, at 1.4 times the runtime (ISSRE 2025). Our follow-up generalizes this to maximal format-free repair, lifting the restrictions earlier methods imposed on repair operations, repair locations, and the parser properties they required (ASE 2026).
Repair of this kind is the fault tolerance half of reliability. It makes the consequence of corruption recoverable, which is the property that matters when the data is irreplaceable.
Practice
My interest in the reliability of programs is informed by a wealth of practical knowledge from industry. Before joining the Ph.D. program, I worked in the software industry as a developer for ten years, where I was part of the web and proxy server development teams at Quark Media House, and Sun Microsystems. My primary area of interest was web caches, particularly distributed caching systems and protocols. I participated in the OpenSolaris effort, where I was the maintainer of multiple open source packages. I have also contributed to the Apache HTTPD project, in core and mod_proxy modules. During my Ph.D., I worked at Puppet Labs where I contributed extensively towards the functionality of the Solaris operating system, and at Galois where I contributed to the visualization of effectiveness of one of the vulnerability mitigation approaches.
That experience continues to shape the work. The industrial protocol study above was run against a production system under real operational constraints, and the reduction and repair tools are built to be dropped into existing pipelines.
IMPORTANT: If you are my student, and facing any sort of difficulties, please do contact me. I will be happy to talk to you, and help you in any way.