Add sample-based program generation to the fuzzer

docs/fuzzing.md

@@ -64,6 +64,9 @@ flag `-f` to stop after the first error.
 | `--gen_bound` | Whether generated trees should try to reuse an existing bound name instead of a fresh one, so lookups get exercised too (default on). |
 | `--weights <tok> <n> [<tok> <n> ...]` | Bias generation towards or away from specific tokens; see [Token weights](#token-weights). |
 | `-l, --log_level` | `Trace`, `Debug`, `Info`, `Warning`, `Output`, `Error`, or `None`. `Trace` prints every generated tree and its output, not just failures. |
+| `--samples` | Files with sample programs to use in program generation. |
+| `--sampling-level` | Number of levels sliced from a sample tree in program generation. Defaults to 1. |
+| `--sampling-frequency` | Chance of preferring a sample node over random generation (0-100). Defaults to 50. |
 
 ## Reproducing a failure
 
@@ -107,6 +110,22 @@ This samples the random number generator itself across `seed_count` seeds and
 reports the entropy of the values produced. This is a sanity check on randomness,
 not on any WF spec so most users won't need it.
 
+## Using samples
+
+Using sample programs during fuzzing increases the likelihood of generating valid trees or valid subtrees. This is especially helpful when fuzzing early passes, which typically operate on less structured trees. Sampling has been shown to improve code coverage of fuzzing across several Trieste projects.
+
+To run the fuzzer with samples, provide one or more sample files:
+
+```
+my_tool test --samples examples/*.my_ext
+```
+
+When sampling is enabled, the fuzzer parses the sample files and may run them through all passes up to the selected starting pass. Each sample tree, and all of its subtrees, is stored in a map keyed by the type of its root node (token).
+
+Starting from the `Top` node, we generate the tree top-down: for each node, we choose its children by randomly picking one of the valid tokens (or token sequences) defined in the well-formedness specification. If sampling is enabled, we may instead copy a randomly chosen node of that type from the sample map, based on the sampling frequency (50% by default).
+
+The `--sampling-level` flag controls how much of a sampled subtree is copied. At the default level of 1, we copy the sampled node's children but not their descendants. At higher levels, we copy descendants down to that depth, or the whole subtree if it is shallower. A level of 0 copies the entire sampled subtre; to avoid reproducing complete sample programs, the `Top` node's children are then always generated from the well-formedness specification.
+
 ## Related: pattern checking
 
 The `check` subcommand performs a separate, static check: it looks for

include/trieste/driver.h

@@ -21,9 +21,8 @@ namespace trieste
     CLI::App app;
     Options* options;
 
-  public:
-    Driver(const Reader& reader_, Options* options_ = nullptr)
-    : reader(reader_), app(reader_.language_name()), options(options_)
+public : Driver(const Reader& reader_, Options* options_ = nullptr)
+: reader(reader_), app(reader_.language_name()), options(options_)
     {}
 
     Driver(
@@ -193,8 +192,26 @@ namespace trieste
       check->add_option(
         "-i,--ignore_token",
         ignored_tokens,
-        "Ignore this token when checking patterns against well-formedness "
-        "rules.");
+        "Ignore this token when checking patterns against well-formedness rules.");
+
+      std::vector<std::filesystem::path> sample_files;
+      test->add_option(
+        "--samples",
+        sample_files,
+        "Files to extract sample nodes from for fuzz testing");
+
+      size_t sampling_level = 1;
+      test->add_option(
+        "--sampling-level",
+        sampling_level,
+        "Level of sampling to use for selecting sample nodes");
+
+      size_t sampling_frequency = 50; // Default to 50% sampling frequency
+      test->add_option(
+        "--sampling-frequency",
+        sampling_frequency,
+        "Frequency of using sampled nodes over randomly generated nodes during "
+        "testing (0-100)");
 
       try
       {
@@ -263,6 +280,8 @@ namespace trieste
       }
       else if (*test)
       {
+        Nodes sample_trees;
+
         if (pass_names_no_parse.empty())
         {
           logging::Error() << "No passes available for testing." << std::endl;
@@ -298,6 +317,31 @@ namespace trieste
             test_end_pass = test_start_pass;
         }
 
+        // Sort the sample files so they are loaded in a deterministic
+        // order making fuzzing reproducible.
+        std::sort(sample_files.begin(), sample_files.end());
+
+        for (auto const& sample_path : sample_files)
+        {
+          if (!(std::filesystem::is_regular_file(sample_path) ||
+                std::filesystem::is_symlink(sample_path)))
+          {
+            logging::Output() << "Not sampling " << sample_path << std::endl;
+            continue;
+          }
+
+          Node sample_program = reader.parser().parse(sample_path);
+
+          if (!sample_program)
+          {
+            logging::Error() << "Failed to parse test program from "
+                             << sample_path << std::endl;
+            return 1;
+          }
+
+          sample_trees.push_back(sample_program);
+        }
+
         Fuzzer fuzzer =
           Fuzzer(reader)
             .max_retries(
@@ -311,7 +355,11 @@ namespace trieste
             .bound_vars(bound_vars)
             .token_weights(std::move(parsed_weights))
             .test_sequence(test_sequence)
-            .size_stats(test_size_stats);
+            .size_stats(test_size_stats)
+            .sample_trees(sample_trees)
+            .sampling_level(sampling_level)
+            .sampling_enabled(!sample_trees.empty())
+            .sampling_frequency(sampling_frequency);
 
         if (*entropy)
         {
@@ -322,6 +370,7 @@ namespace trieste
           return fuzzer.test();
         }
       }
+
       else if (*check)
       {
         if (pass_names_no_parse.empty())