Initial version of ODD monitoring code and examples by a-luque · Pull Request #60 · BerkeleyLearnVerify/VerifAI

a-luque · 2026-03-04T13:40:53Z

No description provided.

Minor fix

…Fork into DynamicSampling

…d corrected typo in README

dfremont · 2026-04-17T09:36:19Z

It's not ideal to add a 50 MB data file to the repository -- is there any way we can make this smaller? If not (e.g. it's not feasible to use a smaller model, or having users train their own would take a long time / a fancy GPU), I guess it's OK.

dfremont · 2026-04-17T09:37:41Z

+import torch
+import torchvision


Dependencies like these which are not already listed in pyproject.toml should be added there under the examples extra (so that someone running pip install verifai[examples] gets everything necessary to run the examples -- please test this in a fresh virtual environment).

dfremont · 2026-04-17T09:38:40Z

+from torch.utils.data import Dataset
+from torchvision.io import read_image


Please move to the top of the file with the rest of the imports so that they are more easily found (likewise for the other imports later on).

dfremont · 2026-04-17T09:40:58Z

+
+    import os
+
+    sys.setrecursionlimit(10000)


Is this really necessary? It's not a good sign to have to change the recursion limit, and nothing here looks recursive on a quick skim.

dfremont · 2026-04-17T09:42:33Z

@@ -0,0 +1,225 @@
+import pandas as pd


In all files, please separate imports into 3 sections separated by 1 blank line: Python stdlib imports, external packages (like pandas), and local imports from within VerifAI.

dfremont · 2026-04-17T09:45:14Z

+from dotmap import DotMap
+from PIL import Image
+import cv2
+
+import pandas as pd


These and some of the other imports don't appear to be used.

dfremont · 2026-04-17T09:47:20Z

+    if not os.path.exists(sim_dir):
+        os.mkdir(sim_dir)
+
+    flag_good = False


This is never used.

dfremont · 2026-04-17T09:48:39Z

+    flag_good = False
+    print(f"Executing sim {i}")
+    try:
+        os.system(f"scenic -S {scenic_path} --count 1 --time {num_of_steps} --2d --param recordFolder {sim_dir} --param timeout 30")


Using Scenic's Python API would probably be better here, but if you want to stick with the CLI, you should use subprocess.run with a list of arguments to make sure they are properly quoted/escaped on all platforms.

dfremont · 2026-04-17T09:50:18Z

+    try:
+        os.system(f"scenic -S {scenic_path} --count 1 --time {num_of_steps} --2d --param recordFolder {sim_dir} --param timeout 30")
+        flag_good = True
+    except: 


Never use a bare except:, which will catch KeyboardInterrupt and prevent people from stopping your script with Ctrl-C. You should use except Exception: if you have no more specific exception types in mind.

dfremont · 2026-04-17T09:50:41Z

+        flag_good = True
+    except: 
+        print(f"Simulation {j} failed.")
+        wait(5)


I don't think wait is defined.

dfremont · 2026-04-17T09:53:29Z

+    except: 
+        print(f"Simulation {j} failed.")
+        wait(5)
+        os.system(f"scenic -S {scenic_path} --count 1 --time {num_of_steps} --2d --param recordFolder {sim_dir} --param timeout 30")


If the first attempt failed, why would you expect it to work a second time after a 5-second delay? Scenic already retries the simulation if there was a SimulationCreationError, and any other ways Scenic might fail (e.g. if there is a syntax error in the program) are unlikely to fix themselves given a little time. Given that wait isn't defined, I suspect this code path was never run: in which case you might as well remove it.

dfremont · 2026-04-17T09:53:49Z

+
+
+
+
+
+


Please remove unnecessary blank lines (1 is OK).

dfremont · 2026-04-17T09:56:19Z

+import pickle
+import sklearn
+import numpy as np 
+import random


Scenic programs should not use the random module directly: use Scenic's own distribution facilities for generating random numbers (otherwise you will break serialization, for example).

dfremont · 2026-04-17T09:58:55Z

+EGO_BRAKING_THRESHOLD = 10
+
+
+behavior FollowLaneBehaviorModified(target_speed = 10, laneToFollow=None, is_oppositeTraffic=False, leaderCar=None):


It's unfortunate to have to duplicate most of the code of FollowLaneBehavior here. It's probably unavoidable, since it looks like you're adding noise to the actions the ordinary behavior would take, but in that case I would just put a short docstring explaining how this version differs from the ordinary one rather than copying the original docstring).

dfremont · 2026-04-17T10:00:09Z

+            with behavior FollowLaneBehaviorModified(target_speed=EGO_SPEED),
+            with color Color(0,0,0) 
+
+ego = new Car following roadDirection from leader.position for EGO_TO_LEADER,


Suggested change

ego = new Car following roadDirection from leader.position for EGO_TO_LEADER,

ego = new Car following roadDirection from leader for EGO_TO_LEADER,

dfremont · 2026-04-17T10:00:41Z

+            with sensors {"front_rgb": RGBSensor(offset=(0, 2, 1), attributes=attrs)
+                        } 


Suggested change

with sensors {"front_rgb": RGBSensor(offset=(0, 2, 1), attributes=attrs)

}

with sensors {"front_rgb": RGBSensor(offset=(0, 2, 1), attributes=attrs)}

dfremont · 2026-04-17T10:03:21Z

+
+
+
+require distance to intersection > 10 and distance from leader to intersection > 10 and ego can see leader


Better to write this as 3 separate require statements: it will help Scenic's pruning, and it's more obvious that you have 3 independent constraints.

Actually, for ego can see leader it would be better to just add with requireVisible True when defining leader (which will also help Scenic do pruning).

dfremont · 2026-04-17T10:05:06Z

@@ -0,0 +1,491 @@
+""" Scenario Description
+The ego car follows the leader car maintaining a normal distance while lane keeping.


This is the same description as the previous file.

dfremont · 2026-04-17T10:05:31Z

+import warnings
+warnings.filterwarnings("ignore") 


Why? If actually necessary, can you use a more specific filter?

dfremont · 2026-04-17T10:06:26Z

+if globalParameters.seed != "":
+    random.seed(globalParameters.seed)
+    np.random.seed(globalParameters.seed)


Should not need this: use Scenic's -s / --seed option.

dfremont · 2026-04-17T10:07:56Z

+
+torch.set_default_device('cuda')
+
+class resNet(torch.nn.Module):


Aren't this and the following class the same as in controller_training.py? Please consolidate if possible.

dfremont · 2026-04-17T10:23:08Z

+        ])
+
+    while True:
+        front_img = self.sensors["front_rgb"]._lastObservation


Any reason to use this internal, undocumented API? Why not self.observations["front_rgb"]?

dfremont · 2026-04-17T10:24:01Z

+
+
+        # Monitor trigger
+        if monitor_model:


The body of this conditional can probably be moved into a helper function used in 3 places as suggested above.

dfremont · 2026-04-17T10:24:40Z

+obstacle = new Car at 0 @ 25 relative to leader,
+            with heading leader 
+
+ego = new Car following roadDirection from leader.position for EGO_TO_LEADER,


Suggested change

ego = new Car following roadDirection from leader.position for EGO_TO_LEADER,

ego = new Car following roadDirection from leader for EGO_TO_LEADER,

dfremont · 2026-04-17T10:25:12Z

+            with distIntersec EGO_TO_LEADER,
+            with isSafe 1,
+            with sensors {"front_rgb": RGBSensor(offset=(0, 2, 1), width=640, height=320),
+                        "aerial_rgb": RGBSensor(offset=(0, -10, 4), width=1280, height=640)


This sensor doesn't appear to be used.

dfremont · 2026-04-17T10:25:31Z

+
+
+
+require ego can see leader


See note about requireVisible in previous file.

dfremont · 2026-04-17T10:26:25Z

@@ -0,0 +1,516 @@
+""" Scenario Description
+The ego car follows the leader car maintaining a normal distance while lane keeping.


Again, please distinguish from other scenarios.

dfremont · 2026-04-17T10:26:59Z

+import warnings
+warnings.filterwarnings("ignore") 


I won't keep repeating the comments -- please check all from the previous file in this one too.

dfremont · 2026-04-17T10:28:05Z

+
+
+
+behavior FollowLaneBehaviorModified(target_speed = 10, laneToFollow=None, is_oppositeTraffic=False, leaderCar=None):


This version looks almost identical to the one in the previous file -- please consolidate to make these scenarios easier to follow.

dfremont · 2026-04-17T10:31:02Z

+
+from scenic.core.sensors import ActiveSensor
+
+class CarlaOtherFeaturesSensor(ActiveSensor):


I don't think this is actually used anywhere. Maybe this file is left over from some other experiments?

dfremont · 2026-04-17T15:58:19Z

+#################### Sampling parameters ############################
+#####################################################################
+
+class SpecMonitor(specification_monitor):


Just as a heads-up, some minor changes will be needed to this code when the simplification of monitors is merged -- but I just realized while reviewing this that I forgot to make a PR for that last December. Once I've made that PR I can try fixing this up myself.

dfremont · 2026-04-17T16:09:06Z

+        try:
+            while True:
+                try:
+                    start = time.time()


Should use time.monotonic() for all timings so that they aren't messed up by changes to the system clock.

dfremont · 2026-04-17T16:10:22Z

+                    rho = self.sampling_params.server.lastValue
+                    self.total_sample_time += timings.sample_time
+                    self.total_simulate_time += timings.simulate_time
+                except:


Don't use bare except, and please distinguish the error and non-error paths (for the latter I assume you want TerminationException?).

dfremont · 2026-04-17T16:15:49Z

+                    sample = self.sampling_params.server.get_sample() 
+                    after_sampling = time.time()
+                    scene = self.sampling_params.server.sampler.lastScene
+                    assert scene
+                    result = self.sampling_params.server._simulate(scene)
+                    if result is None:
+                        print(self.sampling_params.server.rejectionFeedback)
+                        return self.sampling_params.server.rejectionFeedback
+                    print(f"Time steps run for this simulation: {len(result.trajectory)}")
+                    value = (0 if self.sampling_params.server.monitor is None
+                            else self.sampling_params.server.monitor.evaluate(result))
+                    self.sampling_params.server.lastValue = value


This block seems to replicate the logic of Server.run_server (and ScenicServer.evaluate_sample), including manipulating internal state of the server, which is fragile. Can't you just call run_server on the Server you have?

dfremont · 2026-04-17T16:16:45Z

+        return 0
+
+    def load_simulations(self, load_path):
+        import os


Unnecessary, and if it was needed it should be at the top of the file.

dfremont · 2026-04-17T16:18:21Z

+    def generate(self):
+        pass
+
+class GenericDataGenerator(DataGenerator):


There don't seem to be any other subclasses of DataGenerator, so why not just move all of this code into DataGenerator and drop this class?

dfremont · 2026-04-17T16:18:49Z

+
+
+
+class GenericEvaluator(Evaluator):


What's the need for this class vs. just moving everything into Evaluator?

dfremont · 2026-04-17T16:19:40Z

+        super().__init__(eval_params, sampling_params, global_params)
+
+
+    def sample_simulations(self, num_simulations, num_steps, save_datagen_path, save_scenes_path):


This seems very similar to DataGenerator.sample_simulations. Please consolidate the common code.

Actually, the entire class is almost the same as the data generator, so I would make a common superclass of those (which does not need to be public).

dfremont · 2026-04-17T16:22:06Z

+                if i == 1:
+                    t0 = time.time()
+                if i == num_simulations:
+                    filehandler = open(f"{save_path}training_{i}.pkl", 'wb') 


It doesn't look like this file will ever be closed, nor is it used anywhere else. I would use a with statement. (Ditto for the similar code a few lines down.)

dfremont · 2026-04-17T16:23:12Z

+        if save_path[-1] != "/":
+            save_path += "/"


This won't work on Windows. Use pathlib or os.path to construct paths instead.

dfremont · 2026-04-17T16:28:50Z

+        pass
+
+
+class MODDLearner(ODDLearner):


What is the difference supposed to be between these two classes? Unless you anticipate further subclasses, I would just have a single class.

dfremont · 2026-04-17T16:30:32Z

+            # collect losses
+            train_loss_all.append(train_loss)
+
+            # Print message


Can we drop all these obvious, uninformative comments? (presumably AI generated?)

dfremont · 2026-04-17T16:33:09Z

+        pass
+
+
+class GenericTrainer(Trainer):


I don't see the need for both of these classes.

dfremont · 2026-04-17T16:34:46Z

+
+
+
+
+
+
+
+
+


Please remove extraneous whitespace.

dfremont · 2026-04-17T16:42:44Z

+from verifai.modd.odd_monitor import Monitor
+from dotmap import DotMap
+
+class MODD():


Should this actually be a class? You only run it once, I think, so if there's no reason to subclass it, it might as well just be a function. A more informative name like generate_monitor might be better too.

Actually I now realize that MODDLearner is the top-level API. But it doesn't seem to do anything itself, just call this class. Why do you need both?

dfremont · 2026-04-17T16:44:11Z

+    def __init__(self, sampling_params):
+        self.sampling_params = sampling_params
+
+        if self.sampling_params.has_key("server_type"):


Can this condition ever be false?

dfremont · 2026-04-17T16:45:30Z

+        self.sampling_params = sampling_params
+
+        if self.sampling_params.has_key("server_type"):
+            if self.sampling_params.server_type == "scenic":


If I understand the code correctly, it only works with the Scenic server anyway, so why do this check? Raising an exception if the user tries to instantiate MODD with a different kind of server would make more sense. In any case, silently doing nothing when this condition is false seems like the wrong thing to do.

dfremont · 2026-04-17T16:47:24Z

+            sampling_data.sampler = ScenicSampler.fromScenario(self.sampling_params.path, mode2D=True, maxSteps=300, params={"monitor": "", "seed":"", "render" : 1, "verbosity": 3, "timeBound": 300, "controller": self.sampling_params.controller_path})
+
+        elif self.sampling_params.mode == "eval":
+            sampling_data.sampler = ScenicSampler.fromScenario(self.sampling_params.path, mode2D=True, maxSteps=300, params={"monitor": os.path.abspath(self.sampling_params.monitor), "seed": 42, "render" : 1, "verbosity": 3, "timeBound": 300, "controller": self.sampling_params.controller_path})


Do you really want to hard-code the seed and the number of steps/timeBound here?

Also, why force mode2D? I don't think anything you're doing requires that, and it's possible to run CARLA scenarios in 3D mode for example.

dfremont · 2026-04-17T16:49:39Z

+            sampling_data.sampler = ScenicSampler.fromScenario(self.sampling_params.path, mode2D=True, maxSteps=300, params={"monitor": os.path.abspath(self.sampling_params.monitor), "seed": 42, "render" : 1, "verbosity": 3, "timeBound": 300, "controller": self.sampling_params.controller_path})
+
+        elif self.sampling_params.mode == "eval_nomonitor":
+            sampling_data.sampler = ScenicSampler.fromScenario(self.sampling_params.path, mode2D=True, maxSteps=300, params={"monitor": "", "seed": 42, "render" : 1, "verbosity": 3, "timeBound": 300, "controller": self.sampling_params.controller_path})


The differences between these three cases would be more evident if you had them modify a common set of parameters, then call ScenicSampler.fromScenario once after the if statement.

Also you should have either an assertion or an exception to handle the case of an invalid mode (which one depends on whether the user can specify the mode or whether it's a purely internal thing).

dfremont · 2026-04-17T17:03:55Z

+from sklearn.pipeline import Pipeline
+from sklearn.linear_model import LogisticRegression
+from sklearn.preprocessing import StandardScaler
+from verifai.modd.odd_sampler import ODDSampler


dfremont

Hi Alejandro, thanks for putting this together! The code looks OK overall, but see the detailed comments above. A few bigger things to change/investigate:

I could not run the example on my laptop because it requires CARLA. I can try later on another machine, but it would be nice to have a more portable example. I don't think anything you're doing is CARLA-specific, so could you try making it work with MetaDrive too? I spent a few minutes on that myself but ran into this error:

  File "/Users/dfremont/Documents/Berkeley/repositories/BerkeleyLearnVerify/VerifAI/src/verifai/modd/odd_data_generator.py", line 48, in sample_simulations
    bar = progressbar.ProgressBar(max_value=num_simulations)
TypeError: __init__() got an unexpected keyword argument 'max_value'

Some unit tests for the new functionality should be added. Even a basic test (e.g. using the Newtonian simulator) would catch issues like forgetting to add a dependency to pyproject.toml. (For example, the code will currently not run because tqdm is missing.)
I think the MODDLearner class is the only one that's intended to be public (?), but it's not clear how to actually use it. The README in the example explains what it does mathematically, but there's no documentation of the actual API (e.g. how does one actually run the resulting monitor?). A new page should be added to the documentation explaining the monitor generation feature.
Even if all the other classes are internal, it would be good to at least have short docstrings so that people maintaining this code can understand what each of the components is supposed to do, what its arguments are, etc. Right now the internal architecture is not clear.

aluquecerp and others added 9 commits March 4, 2026 14:34

Initial version of ODD monitoring code and examples

508e838

Cleaning and expansion of the README file

bba415c

Paths fixes

ac66f0a

Minor fix

cabd349

Update followLeader_extracar_NN.scenic

9189a67

Minor fix

Removed comments and modifications added by mistake

263c151

Merge branch 'DynamicSampling' of https://github.com/a-luque/VerifAI_…

9dbf85f

…Fork into DynamicSampling

Removed unnecessary files (duplicated, error when commit was made) an…

cde394f

…d corrected typo in README

Merge branch 'tmp' into DynamicSampling

2c55eb1

dfremont reviewed Apr 17, 2026

View reviewed changes

Comment thread src/verifai/modd/pymodd.py

Comment on lines +60 to +68

Copy link
Copy Markdown

Collaborator

dfremont Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove extraneous whitespace.

dfremont reviewed Apr 17, 2026

View reviewed changes

dfremont requested changes Apr 17, 2026

View reviewed changes

		from torch.utils.data import Dataset
		from torchvision.io import read_image

		EGO_BRAKING_THRESHOLD = 10


		behavior FollowLaneBehaviorModified(target_speed = 10, laneToFollow=None, is_oppositeTraffic=False, leaderCar=None):

	ego = new Car following roadDirection from leader.position for EGO_TO_LEADER,
	ego = new Car following roadDirection from leader for EGO_TO_LEADER,

		with sensors {"front_rgb": RGBSensor(offset=(0, 2, 1), attributes=attrs)
		}




		require distance to intersection > 10 and distance from leader to intersection > 10 and ego can see leader

		@@ -0,0 +1,491 @@
		""" Scenario Description
		The ego car follows the leader car maintaining a normal distance while lane keeping.


		torch.set_default_device('cuda')

		class resNet(torch.nn.Module):

		@@ -0,0 +1,516 @@
		""" Scenario Description
		The ego car follows the leader car maintaining a normal distance while lane keeping.


		from scenic.core.sensors import ActiveSensor

		class CarlaOtherFeaturesSensor(ActiveSensor):

		super().__init__(eval_params, sampling_params, global_params)


		def sample_simulations(self, num_simulations, num_steps, save_datagen_path, save_scenes_path):

Conversation

a-luque commented Mar 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dfremont Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

dfremont Apr 17, 2026 •

edited

Loading