Check binary event time by the clock state, and an option for TB2017-003a

CHANGELOG.md

@@ -1,5 +1,14 @@
 # Changelog
 
+## 0.3.0
+
+- **Technical bulletins:** `outstation.bulletins` lists the bulletins a device implements, where they change what a revision's steps check; none by default. The first is TB2017-003a: with it, 2.7's binary and double-bit event tests with time (8.14.2.9 to 8.14.2.11, 8.22.2.9 to 8.22.2.11) accept events recorded while the clock is unsynchronized with relative time after g51v2, and require it.
+- **Clock state:** integration plugins can tell dnp3c whether the device's clock is synchronized (`IntegrationPlugin.timeQuality`), so 3.1 rev 1's binary and double-bit event tests, and 2.7's with TB2017-003a, check each event's variation against the state it was recorded in. A batch of events recorded while the clock changed may be reported each way in one response.
+
+### Upgrading from 0.2.0
+
+- Integration plugins: `IntegrationPlugin.timeQuality` has a default, so a plugin keeps compiling; without it, the checks accept either way of reporting the events, as before.
+
 ## 0.2.0
 
 - **DNP3 IED Certification Procedure 3.1 rev 1**, subset levels 1 to 3, alongside 2.7. Choose the revision with `-rev`; `-revs` lists them.

README.md

@@ -30,6 +30,8 @@ dnp3c implements two revisions of the DNP Users Group's *DNP3 IED Certification
 - **3.1 rev 1** (`3.1r1`, 13 September 2022) covers Subset Levels 1 to 3. Its tests have the numbers it gives them. One it leaves without a number has its section's number followed by a letter for its place among the section's unnumbered tests, e.g. 8.6.5b for the second; if it's the section's only test, it has the section's number and title, e.g. 8.6.1 Restart.
 - **2.7** (`2.7`, 2 March 2016) covers Subset Level 2. Where a technical bulletin replaces a 2.7 test outright, dnp3c runs the replacement: TB2017-001's 8.6.4.2 and 8.6.4.3, whose steps are numbered 1.a, 1.b, and so on.
 
+A device can declare the technical bulletins it implements, where they change what a revision's steps check, with `outstation.bulletins`, e.g. `outstation.bulletins = TB2017-003a`. It's empty by default, so a run checks each revision as published. The one bulletin it knows so far is TB2017-003a: a device that implements it reports binary and double-bit input events recorded while its clock is unsynchronized with relative time after an unsynchronized CTO (g51v2 and g2v3 or g4v3), and uses absolute time (g2v2, g4v2) only for events recorded while it's synchronized. 2.7's 8.14.2.9 to 8.14.2.11 and 8.22.2.9 to 8.22.2.11 accept only absolute time; for a device that lists the bulletin, they check its rule instead, and the result notes it. 3.1 rev 1 already has the rule (8.14.1.1). Both revisions learn whether the clock was synchronized when the events were generated from the integration plugin, where it knows (see [Automating the tests](docs/automation.md)); where it doesn't, they accept either way, as long as all the events are reported the same way.
+
 The section, procedure and step numbers in the tool, its logs and the report are those of the revision being run. Each revision is run as published, so a result under 2.7 can differ from one under 3.1 rev 1, which changes what some tests send or accept, often to apply technical bulletins published after 2.7.
 
 ### Where 3.1 rev 1 contradicts itself

docs/automation.md

@@ -136,6 +136,8 @@ To fail a step, throw an exception with a message saying what went wrong. The pr
 
 A few actions are used only by 3.1r1's tests, e.g. `setColdRestartSupport` (8.8.2). They have a default that throws `UnsupportedOperationException`, so a plugin written before them keeps compiling; override one if the device runs a test that uses it. `DefaultIntegrationPlugin` prompts for them like any other.
 
+`timeQuality` tells dnp3c whether the device's clock is synchronized, for the events it records now. dnp3c asks before and after each batch of binary and double-bit input events it has the plugin generate, and checks the variations the device reports them in against that: an event recorded while the clock was synchronized in the variation requested, one recorded while it was unsynchronized with relative time after an unsynchronized CTO (g51v2). 3.1r1's rules for event reporting (8.14.1.1) require that, and so does 2.7 for a device that lists TB2017-003a in `outstation.bulletins`. The default returns `None`, for a plugin that doesn't know: the checks then accept either way, as long as all the events are reported the same way.
+
 3.1r1's network tests (section 9) set up the device's network with `configureNetwork`, which gives the whole setup: TCP or UDP, the port to listen on, the masters to accept. Each test ends with `restoreNetwork`, which puts back the setup the device had before, even when the test fails.
 
 Every method gets a `PluginReporter` (verifications get a `CheckReporter`, which extends it). Anything you pass to its `log` method appears in the procedure's log, e.g. what the plugin did, or the values it compared.

src/main/scala/io/stepfunc/dnp3c/conformance/api/Bulletin.scala

@@ -0,0 +1,21 @@
+// SPDX-License-Identifier: GPL-3.0-only
+// Copyright Step Function I/O LLC
+
+package io.stepfunc.dnp3c.conformance.api
+
+/**
+ * A DNP Users Group technical bulletin that a device can declare it implements (outstation.bulletins), where it changes
+ * what a revision's steps check, or makes a requirement they leave implicit. A run checks a revision as published
+ * unless the configuration lists the bulletin, and each check it changes names it.
+ */
+enum Bulletin(val id: String) {
+
+  // TB2017-003a 5.1.3.1.2: binary and double-bit input events recorded while the clock is unsynchronized are reported
+  // with relative time after an unsynchronized CTO (g51v2), never with absolute time
+  case TB2017_003a extends Bulletin("TB2017-003a")
+}
+
+object Bulletin {
+
+  def fromId(id: String): Option[Bulletin] = values.find(_.id.equalsIgnoreCase(id.trim))
+}

src/main/scala/io/stepfunc/dnp3c/conformance/api/PluginSteps.scala

@@ -112,30 +112,54 @@ object PluginSteps {
   }
 
   def generateBinaryInputChangeEvents(): TestStep =
-    action("Change binary inputs several times, one or more of them, to create events") {
+    binaryEvents("Change binary inputs several times, one or more of them, to create events") {
       (plugin, reporter) =>
         plugin.generateInputChanges(reporter, PointKind.BinaryInput)
     }
 
   def generateExactBinaryInputChangeEvents(numEvents: Int): TestStep =
-    action(s"Generate ${numEvents} Binary Input Change Event(s)") { (plugin, reporter) =>
+    binaryEvents(s"Generate ${numEvents} Binary Input Change Event(s)") { (plugin, reporter) =>
       plugin.generateExactInputEvents(reporter, PointKind.BinaryInput, numEvents)
     }
 
+  // a step that generates binary or double-bit input events, recording the time quality the plugin reports for the
+  // DUT's clock before and after, for the checks of how the events are reported (IntegrationPlugin.timeQuality)
+  private def binaryEvents(desc: Comment*)(generate: (IntegrationPlugin, PluginReporter) => Unit): TestStep =
+    new TestStep {
+      override def description: List[Comment] = desc.toList
+
+      override def run(driver: TestDriver): List[DeferredCheck] = {
+        val plugin = driver.reporter.plugin
+        val before = plugin.timeQuality(driver.reporter.pluginReporter())
+        generate(plugin, driver.reporter.pluginReporter())
+        val after = plugin.timeQuality(driver.reporter.pluginReporter())
+        driver.reporter.recordEventClock(before, after)
+        (before, after) match {
+          case (Some(b), Some(a)) if a == b =>
+            driver.reporter.info(s"The plugin reports the DUT's clock ${b.toString.toLowerCase}")
+          case (Some(b), Some(a)) =>
+            driver.reporter.info(
+              s"The plugin reports the DUT's clock ${b.toString.toLowerCase} before the events and ${a.toString.toLowerCase} after")
+          case _ => ()
+        }
+        Nil
+      }
+    }
+
   // Double-bit Binary Inputs
   def generateDoubleBitBinaryInputPattern(): TestStep = action("Set the double-bit inputs to a known pattern") {
     (plugin, reporter) =>
       plugin.generatePattern(reporter, PointKind.DoubleBitInput)
   }
 
   def generateDoubleBitBinaryInputChangeEvents(): TestStep =
-    action("Change double-bit binary inputs several times, one or more of them, to create events") {
+    binaryEvents("Change double-bit binary inputs several times, one or more of them, to create events") {
       (plugin, reporter) =>
         plugin.generateInputChanges(reporter, PointKind.DoubleBitInput)
     }
 
   def generateExactDoubleBitBinaryInputChangeEvents(numEvents: Int): TestStep =
-    action(s"Generate ${numEvents} Double-bit Binary Input Change Event(s)") { (plugin, reporter) =>
+    binaryEvents(s"Generate ${numEvents} Double-bit Binary Input Change Event(s)") { (plugin, reporter) =>
       plugin.generateExactInputEvents(reporter, PointKind.DoubleBitInput, numEvents)
     }
 

src/main/scala/io/stepfunc/dnp3c/conformance/api/TestOption.scala

@@ -50,6 +50,8 @@ trait IndexListOption extends TypedTestOption[List[Index]]
 trait RequiredIntegerOption extends TypedTestOption[Option[Int]] {
   final override def required: Boolean = true
 }
+// the technical bulletins a device implements, by id
+trait BulletinsOption extends TypedTestOption[Set[Bulletin]]
 // procedures to leave out of a run, by revision and id, each with the reason: a family of properties, one per
 // procedure, under the option's id, e.g. "test.exclude.3.1r1/8.26.2 = <reason>". Each is keyed by the rest of its key,
 // e.g. "3.1r1/8.26.2", which the revisions check: the options don't know them
@@ -124,6 +126,9 @@ object OptionTypes {
   // one of the allowed words, in any case
   def choice(group: OptionGroup, name: String, default: String, desc: String, allowed: List[String]): StringOption =
     ChoiceOptionImpl(group, name, default, desc, allowed)
+  // a comma separated list of bulletin ids, none by default
+  def bulletins(group: OptionGroup, name: String, desc: String): BulletinsOption =
+    BulletinsOptionImpl(group, name, desc)
 
   private case class StringOptionImpl(group: OptionGroup, name: String, default: String, description: String)
       extends StringOption {
@@ -163,6 +168,22 @@ object OptionTypes {
           .mkString(", ")}"))
   }
 
+  private case class BulletinsOptionImpl(group: OptionGroup, name: String, desc: String) extends BulletinsOption {
+
+    private def known: String = Bulletin.values.map(_.id).mkString(", ")
+
+    override def description: String = s"$desc, of: $known"
+
+    override def default: Set[Bulletin] = Set.empty
+
+    override def defaultAsString: String = ""
+
+    override def parse(s: String): Set[Bulletin] =
+      s.split(",").map(_.trim).filter(_.nonEmpty).map { id =>
+        Bulletin.fromId(id).getOrElse(throw new Exception(s"'$id' isn't a bulletin dnp3c knows: $known"))
+      }.toSet
+  }
+
   private case class BooleanOptionImpl(group: OptionGroup, name: String, default: Boolean, description: String)
       extends BooleanOption {
     override def parse(s: String): Boolean = Parse.boolean(s)

src/main/scala/io/stepfunc/dnp3c/conformance/api/options/OutstationOptions.scala

@@ -3,7 +3,9 @@
 
 package io.stepfunc.dnp3c.conformance.api.options
 
-import io.stepfunc.dnp3c.conformance.api.{ IntegerOption, OptionGroup, OptionTypes, RequiredIntegerOption, TestOption }
+import io.stepfunc.dnp3c.conformance.api.{
+  BulletinsOption, IntegerOption, OptionGroup, OptionTypes, RequiredIntegerOption, TestOption
+}
 import io.stepfunc.dnp3c.dnp3.transport.Transport
 
 object OutstationOptions extends OptionGroup {
@@ -38,6 +40,13 @@ object OutstationOptions extends OptionGroup {
     List(1, 2, 3)
   )
 
+  val bulletins: BulletinsOption = OptionTypes.bulletins(
+    this,
+    "bulletins",
+    "Technical bulletins the outstation implements, comma separated, which change what some steps of a revision " +
+      "check: none by default, so a run checks each revision as published"
+  )
+
   def prefix: String = "outstation"
 
   def description: String = "Outstation application layer options"
@@ -47,5 +56,6 @@ object OutstationOptions extends OptionGroup {
     selectTimeout,
     responseTimeout,
     appConfirmTimeout,
-    configuredFragmentSize)
+    configuredFragmentSize,
+    bulletins)
 }

src/main/scala/io/stepfunc/dnp3c/conformance/common/EventTime.scala

@@ -0,0 +1,129 @@
+// SPDX-License-Identifier: GPL-3.0-only
+// Copyright Step Function I/O LLC
+
+package io.stepfunc.dnp3c.conformance.common
+
+import scala.util.Try
+
+import io.stepfunc.dnp3c.dnp3.{ Comment, DeferredCheck, MessageValidation, ProcedureReporter, TimeQuality }
+import io.stepfunc.dnp3c.dnp3.app.{ Message, Variation }
+import io.stepfunc.dnp3c.dnp3.app.objects.{ Group51Var1, Group51Var2 }
+
+/**
+ * How binary and double-bit input events recorded with time are reported, by the state of the device's clock when it
+ * recorded them: an event recorded while the clock was synchronized in the variation preferred; one recorded while it
+ * was unsynchronized with relative time, after an unsynchronized CTO (g51v2). That's TB2017-003a 5.1.3.1.2, which
+ * 3.1r1's 8.14.1.1 adopts, and which 2.7 applies only for a device that lists the bulletin.
+ */
+object EventTime {
+
+  /** how the device's clock stood when it recorded the events, as far as the procedure, or the plugin, knows */
+  enum Clock {
+    case Synchronized, Unsynchronized
+    // it changed while the events were recorded, so some may have been recorded each way
+    case Changed
+    case Unknown
+  }
+
+  /** which of the batches of events a procedure generated a response reports */
+  enum Batches {
+    // the latest batch only: the procedure confirmed the earlier ones
+    case Latest
+    // every batch since the procedure started
+    case All
+  }
+
+  /**
+   * How the clock stood while the batches were generated, by the time quality the plugin reported before and after each
+   * (IntegrationPlugin.timeQuality): unknown if it didn't say, every time
+   */
+  def recorded(reporter: ProcedureReporter, batches: Batches): Clock = {
+    val clocks = batches match {
+      case Batches.Latest => reporter.eventClocks.lastOption.toList
+      case Batches.All => reporter.eventClocks
+    }
+    val qualities = clocks.flatMap((before, after) => List(before, after))
+    if (qualities.isEmpty || qualities.contains(None)) Clock.Unknown
+    else
+      qualities.flatten.distinct match {
+        case List(TimeQuality.Synchronized) => Clock.Synchronized
+        case List(TimeQuality.Unsynchronized) => Clock.Unsynchronized
+        case _ => Clock.Changed
+      }
+  }
+
+  // a way to report the events: the variations they may use, and the CTOs that may come with them
+  final case class Form(name: String, events: List[Variation], ctos: Set[Variation]) {
+    def validation: MessageValidation = Validations.eventsWithCtos(events, ctos)
+
+    def describe: String =
+      s"$name (${events.mkString(" or ")}${if (ctos.isEmpty) "" else s", ${ctos.mkString(" or ")}"})"
+  }
+
+  /**
+   * The ways to report the input's events, recorded with time, for a read whose preferred variation has time
+   *
+   * @param relativeTime whether the device reports events with relative time: only one that does can report events
+   *   recorded unsynchronized as the rule requires, so where the clock is unknown, one that doesn't reports them as
+   *   synchronized
+   */
+  def withTime(input: InputType, preferred: Variation, relativeTime: Boolean, clock: Clock): List[Form] = {
+    val relative = input.relativeTimeVariation
+    val unsynchronized = Form("unsynchronized: relative time after g51v2", List(relative), Set(Group51Var2))
+    val synchronized =
+      if (preferred == relative) Form("synchronized: relative time after g51v1", List(relative), Set(Group51Var1))
+      else Form(s"synchronized: $preferred", List(preferred), Set.empty)
+    // each event as its own state requires, in any mix
+    val either = Form(
+      "synchronized and unsynchronized, each as its state requires",
+      (synchronized.events ++ unsynchronized.events).distinct,
+      synchronized.ctos ++ unsynchronized.ctos
+    )
+    clock match {
+      case Clock.Synchronized => List(synchronized)
+      case Clock.Unsynchronized => List(unsynchronized)
+      case Clock.Changed => List(either)
+      case Clock.Unknown => if (relativeTime) List(synchronized, unsynchronized) else List(synchronized)
+    }
+  }
+
+  /**
+   * For a device that implements TB2017-003a, a response to a read of `requested`, a variation with absolute time,
+   * reports the events as 5.1.3.1.2 requires, by the clock's state as the plugin reported it while generating them. 2.7
+   * checks this in place of the variation its steps require, and notes it on the result.
+   *
+   * @param relativeTime whether the device reports events with relative time (see withTime)
+   */
+  def tb2017003a(input: InputType, requested: Variation, relativeTime: Boolean): MessageValidation =
+    new MessageValidation {
+      private val bulletin = "TB2017-003a 5.1.3.1.2"
+
+      def description: List[Comment] = List(Comment(
+        s"Verify that the events are reported as $bulletin requires, which the DUT implements: $requested for events " +
+          s"recorded while the clock was synchronized, ${input.relativeTimeVariation} after g51v2 for those recorded " +
+          "while it was unsynchronized, by the clock's state as the plugin reports it"))
+
+      def validate(reporter: ProcedureReporter, message: Message): List[DeferredCheck] = {
+        // noted first, so the log of a check that fails says which rule it applied
+        val note = s"The events were checked as $bulletin requires, which the DUT implements, in place of the steps' " +
+          s"$requested only"
+        if (!reporter.notes.contains(note)) reporter.note(note)
+        val clock = recorded(reporter, Batches.Latest)
+        oneOf(message, reporter, withTime(input, requested, relativeTime, clock), s"$bulletin requires", clock)
+        Nil
+      }
+    }
+
+  /** the response reports the events in one of the forms; what for names the rules, for the failure */
+  def oneOf(message: Message, reporter: ProcedureReporter, forms: List[Form], what: String, clock: Clock): Unit =
+    if (!forms.exists(f => Try(f.validation.validate(reporter, message)).isSuccess))
+      throw new Exception(
+        s"The events aren't reported as $what${describe(clock)}; expected ${forms.map(_.describe).mkString(" or ")}")
+
+  private def describe(clock: Clock): String = clock match {
+    case Clock.Synchronized => ", for events recorded while the clock was synchronized"
+    case Clock.Unsynchronized => ", for events recorded while the clock was unsynchronized"
+    case Clock.Changed => ", for events recorded while the clock went from one state to the other"
+    case Clock.Unknown => ""
+  }
+}

src/main/scala/io/stepfunc/dnp3c/conformance/common/InputEventProcedures.scala

@@ -67,13 +67,13 @@ object InputEventProcedures {
   // runs whether or not the DUT reports events with time: a DUT without them must reject the request
   abstract class AllQualifierWithTime(input: InputType) extends EventProcedure(input) {
     override protected def steps(options: OptionSource): Iterable[LabeledStep] =
-      events.AllQualifierWithTimeSteps(options.get(input.withTime))
+      events.AllQualifierWithTimeSteps(options.get(input.withTime), options)
   }
 
   abstract class CountQualifierWithTime(input: InputType, twoByteCount: Boolean)
       extends EventProcedure(input, List(input.withTime)) {
     override protected def steps(options: OptionSource): Iterable[LabeledStep] =
-      events.CountQualifierWithTimeSteps(options.get(input.numEventsGenerated), twoByteCount)
+      events.CountQualifierWithTimeSteps(options.get(input.numEventsGenerated), twoByteCount, options)
   }
 
   // runs whether or not the DUT reports events with relative time: a DUT without them must reject the request