Let applications record feature flag exposures with ExposureHook

metadata/supported-configurations.json

@@ -1604,6 +1604,14 @@
         "aliases": []
       }
     ],
+    "DD_FEATURE_FLAGS_EXPOSURES_DATADOG_LOGGING_ENABLED": [
+      {
+        "version": "A",
+        "type": "boolean",
+        "default": "true",
+        "aliases": []
+      }
+    ],
     "DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED": [
       {
         "version": "A",

products/feature-flagging/feature-flagging-api/README.md

@@ -86,6 +86,103 @@ OTEL_EXPORTER_OTLP_PROTOCOL=grpc
 | `error.type` | Error code (lowercased, only on error) |
 | `feature_flag.result.allocation_key` | Allocation key (when present) |
 
+## Exposures in your own data store
+
+The provider sends exposures to Datadog. It also puts its exposure decision and the selected split's
+features into the flag metadata of every evaluation, so any OpenFeature hook can record exposures in
+your own data store (see [Flag metadata](#flag-metadata)).
+
+For convenience, `dd-openfeature` ships `ExposureHook`, which reads that metadata for you. It
+receives every completed evaluation, with the same exposure decision that the provider uses for
+Datadog.
+
+```java
+import datadog.trace.api.openfeature.ExposureHook;
+
+BlockingQueue<ExposureRecord> queue = new LinkedBlockingQueue<>(10_000);
+
+OpenFeatureAPI.getInstance().addHooks(new ExposureHook(evaluation -> {
+  if (!evaluation.shouldSend()) {
+    return;
+  }
+  Map<String, Object> features = evaluation.getFeatures();
+  queue.offer(new ExposureRecord(
+      evaluation.getDetails().getFlagKey(),
+      evaluation.getDetails().getVariant(),
+      evaluation.getContext().getTargetingKey(),
+      (String) features.get("holdout.key"),
+      (String) features.get("holdout.assignment_group")));
+}));
+// A background thread drains the queue and writes to the data store.
+```
+
+- The callback runs on the evaluation thread. Do not write to the data store from the callback.
+  Copy what you need and hand it to a background queue.
+- `shouldSend()` is true for a successful evaluation of an allocation that logs exposures, for a
+  subject not yet exposed to the same allocation, variant and serial id. The serial id counts only
+  with `dd-java-agent` 1.67.0 or later. These are exactly the exposures that the provider sends to
+  Datadog, as long as Datadog logging is on and the provider's own hooks run: evaluate through an
+  OpenFeature client, and inside a `MultiProvider` use OpenFeature SDK 1.23.0 or later. With an
+  older SDK, a provider inside a `MultiProvider` sends no exposures, and `shouldSend()` is true for
+  every exposure.
+- `isExposure()` is true for every successful evaluation of an allocation that logs exposures,
+  including repeats. `isCacheHit()` tells repeats apart.
+- An evaluation that fails, including one that a later `after` hook fails, is not an exposure.
+- An exception from the callback does not affect the evaluation. The first one is logged at warn
+  level, later ones at debug level.
+
+### Features
+
+`getFeatures()` returns the features of the selected split whose destinations include `HOOK`.
+Values are a `String`, `Double` or `Boolean`. Holdout splits carry:
+
+| Feature | Value |
+|---|---|
+| `holdout.key` | The holdout key |
+| `holdout.assignment_group` | `status_quo` or `all_shipped` |
+
+The map is empty when the split has no such features. Datadog can add destinations later; the
+provider ignores a destination it does not know and logs a warning once for each one.
+
+### Flag metadata
+
+The same data is in the evaluation's flag metadata, for hooks that use only the OpenFeature API.
+Read it in the hook's `finallyAfter` stage, where the details match what the application receives,
+and skip details that have an error code. `ExposureHook` does both.
+
+| Key | Type | Meaning |
+|---|---|---|
+| `allocationKey` | String | The selected allocation |
+| `__dd_do_log` | Boolean | Whether the allocation logs exposures. Present when the allocation logs exposures or span enrichment is on |
+| `__dd_exposure_cache_hit` | Boolean | The subject was already exposed. Present only when `__dd_do_log` is true |
+| `__dd_split_serial_id` | Integer | The split's serial id. Present when `__dd_do_log` is present, the split has a serial id, and the installed `dd-java-agent` supports serial ids |
+| `__dd_feature.<key>` | String, Double or Boolean | One entry for each feature of the selected split that lists the `HOOK` destination |
+
+### Agent versions
+
+`dd-openfeature` and `dd-java-agent` ship separately. The provider turns a capability off when the
+installed agent lacks it:
+
+| Capability | Needs `dd-java-agent` | Without it |
+|---|---|---|
+| Serial ids on splits (`__dd_split_serial_id`) | 1.65.0 | The key is absent |
+| Serial ids on exposures sent to Datadog | 1.67.0 | Exposures are sent without a serial id, so holdout attribution is lost |
+| Flag evaluation events sent to Datadog | 1.66.0 | No flag evaluation events are sent |
+| Split features (`getFeatures()`, `__dd_feature.<key>`) | 1.68.0 | `getFeatures()` is empty |
+
+### Stop sending exposures to Datadog
+
+```shell
+DD_FEATURE_FLAGS_EXPOSURES_DATADOG_LOGGING_ENABLED=false
+```
+
+The provider then sends no exposures to Datadog. `ExposureHook` callbacks still receive every
+evaluation, and `shouldSend()` gives the same result as before. The default is `true`.
+
+This setting stops exposures only. Flag evaluation events have their own setting,
+`DD_FLAGGING_EVALUATION_COUNTS_ENABLED`. The Java provider does not read dd-trace-py's
+`DD_FFE_INTAKE_ENABLED`.
+
 ## Requirements
 
 - Java 11+

products/feature-flagging/feature-flagging-api/src/main/java/datadog/trace/api/openfeature/DDEvaluator.java

@@ -8,6 +8,7 @@
 import datadog.trace.api.featureflag.ufc.v1.Allocation;
 import datadog.trace.api.featureflag.ufc.v1.ConditionConfiguration;
 import datadog.trace.api.featureflag.ufc.v1.ConditionOperator;
+import datadog.trace.api.featureflag.ufc.v1.Feature;
 import datadog.trace.api.featureflag.ufc.v1.Flag;
 import datadog.trace.api.featureflag.ufc.v1.ParsedSemver;
 import datadog.trace.api.featureflag.ufc.v1.Rule;
@@ -41,6 +42,7 @@
 import java.util.Map;
 import java.util.Objects;
 import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicBoolean;
@@ -63,6 +65,40 @@ class DDEvaluator implements Evaluator, FeatureFlaggingGateway.ConfigListener {
       new AtomicBoolean(
           SPLIT_SERIAL_ID_SUPPORTED.get() && exposureSerialIdSupported(ExposureEvent.class));
 
+  static final AtomicBoolean SPLIT_FEATURES_SUPPORTED =
+      new AtomicBoolean(splitFeaturesSupported(Split.class));
+
+  static final AtomicBoolean EXPOSURE_FEATURES_SUPPORTED =
+      new AtomicBoolean(exposureFeaturesSupported(ExposureEvent.class));
+
+  /** An older agent's exposure event has no features; its exposures are then sent without them. */
+  static boolean exposureFeaturesSupported(final Class<?> eventClass) {
+    try {
+      eventClass.getConstructor(
+          long.class,
+          datadog.trace.api.featureflag.exposure.Allocation.class,
+          datadog.trace.api.featureflag.exposure.Flag.class,
+          datadog.trace.api.featureflag.exposure.Variant.class,
+          Subject.class,
+          Integer.class,
+          Map.class);
+      return true;
+    } catch (final NoSuchMethodException | LinkageError | RuntimeException e) {
+      log.debug("The installed Datadog Java agent does not send features on exposures", e);
+      return false;
+    }
+  }
+
+  /** An older agent ships a Split without features; its evaluations then carry no features. */
+  static boolean splitFeaturesSupported(final Class<?> splitClass) {
+    try {
+      return splitClass.getField("features").getType() == List.class;
+    } catch (final NoSuchFieldException | LinkageError | RuntimeException e) {
+      log.debug("The installed Datadog Java agent does not carry split features", e);
+      return false;
+    }
+  }
+
   static boolean splitSerialIdSupported(final Class<?> splitClass) {
     try {
       return splitClass.getField("serialId").getType() == Integer.class;
@@ -149,6 +185,23 @@ static boolean exposureSerialIdSupported(final Class<?> eventClass) {
   // Emitted only when the allocation logs exposures: true when this subject's exposure was already
   // sent, so the exposure hook does not send it again.
   static final String METADATA_EXPOSURE_CACHE_HIT = "__dd_exposure_cache_hit";
+  // Each feature of the selected split is copied to the metadata once for each destination it
+  // lists, under that destination's prefix. Only the HOOK prefix is a public contract; the
+  // other two carry features to Datadog's own exposure and flag-evaluation hooks.
+  static final String METADATA_FEATURE_PREFIX = "__dd_feature.";
+  static final String METADATA_EXPOSURE_FEATURE_PREFIX = "__dd_exposure_feature.";
+  static final String METADATA_EVALUATION_FEATURE_PREFIX = "__dd_evaluation_feature.";
+  // Present only when the split routes a feature to EVALUATION, so FlagEvalLoggingHook can skip
+  // scanning the metadata on every other evaluation.
+  static final String METADATA_HAS_EVALUATION_FEATURES = "__dd_has_evaluation_features";
+
+  static final String FEATURE_DESTINATION_HOOK = "HOOK";
+  static final String FEATURE_DESTINATION_EXPOSURE = "EXPOSURE";
+  static final String FEATURE_DESTINATION_EVALUATION = "EVALUATION";
+  // A newer UFC can name destinations this SDK does not know; warn once per name, not per
+  // evaluation.
+  static final int MAX_WARNED_FEATURE_DESTINATIONS = 64;
+  static final Set<String> WARNED_FEATURE_DESTINATIONS = ConcurrentHashMap.newKeySet();
 
   // Stamped on every DD-produced evaluation (including PROVIDER_NOT_READY, with false). Missing
   // key = non-DD provider; the hook falls back to false (fail-closed).
@@ -611,6 +664,9 @@ private static <T> ProviderEvaluation<T> resolveVariant(
       }
       metadataBuilder.addBoolean(METADATA_DO_LOG, doLog);
     }
+    if (SPLIT_FEATURES_SUPPORTED.get()) {
+      addSplitFeatures(metadataBuilder, split);
+    }
     if (doLog) {
       metadataBuilder.addBoolean(
           METADATA_EXPOSURE_CACHE_HIT,
@@ -636,6 +692,69 @@ private static <T> ProviderEvaluation<T> resolveVariant(
     return result;
   }
 
+  private static void addSplitFeatures(
+      final ImmutableMetadata.ImmutableMetadataBuilder metadataBuilder, final Split split) {
+    if (split.features == null) {
+      return;
+    }
+    Set<String> seenKeys = null;
+    boolean evaluationFeatures = false;
+    for (final Object item : split.features) {
+      if (!(item instanceof Feature)) {
+        continue;
+      }
+      final Feature feature = (Feature) item;
+      if (feature.key == null || feature.destinations == null) {
+        continue;
+      }
+      if (seenKeys == null) {
+        seenKeys = new HashSet<>();
+      }
+      if (!seenKeys.add(feature.key)) {
+        continue;
+      }
+      for (final String destination : feature.destinations) {
+        final String prefix = metadataPrefix(destination);
+        if (prefix != null) {
+          addFeature(metadataBuilder, prefix + feature.key, feature.value);
+          evaluationFeatures |= prefix.equals(METADATA_EVALUATION_FEATURE_PREFIX);
+        } else if (WARNED_FEATURE_DESTINATIONS.size() < MAX_WARNED_FEATURE_DESTINATIONS
+            && WARNED_FEATURE_DESTINATIONS.add(destination)) {
+          log.warn("Ignoring unknown feature destination {}", destination);
+        }
+      }
+    }
+    if (evaluationFeatures) {
+      metadataBuilder.addBoolean(METADATA_HAS_EVALUATION_FEATURES, true);
+    }
+  }
+
+  private static String metadataPrefix(final String destination) {
+    if (FEATURE_DESTINATION_HOOK.equals(destination)) {
+      return METADATA_FEATURE_PREFIX;
+    }
+    if (FEATURE_DESTINATION_EXPOSURE.equals(destination)) {
+      return METADATA_EXPOSURE_FEATURE_PREFIX;
+    }
+    if (FEATURE_DESTINATION_EVALUATION.equals(destination)) {
+      return METADATA_EVALUATION_FEATURE_PREFIX;
+    }
+    return null;
+  }
+
+  private static void addFeature(
+      final ImmutableMetadata.ImmutableMetadataBuilder metadataBuilder,
+      final String key,
+      final Object value) {
+    if (value instanceof String) {
+      metadataBuilder.addString(key, (String) value);
+    } else if (value instanceof Boolean) {
+      metadataBuilder.addBoolean(key, (Boolean) value);
+    } else if (value instanceof Number) {
+      metadataBuilder.addDouble(key, ((Number) value).doubleValue());
+    }
+  }
+
   /** The serial id an exposure carries: none when the agent's exposure event predates it. */
   static Integer exposureSerialId(final Integer splitSerialId) {
     return EXPOSURE_SERIAL_ID_SUPPORTED.get() && SPLIT_SERIAL_ID_SUPPORTED.get()
@@ -715,7 +834,8 @@ static ExposureEvent exposureEvent(
       final String allocationKey,
       final String variantKey,
       final EvaluationContext context,
-      final Integer serialId) {
+      final Integer serialId,
+      final Map<String, Object> features) {
     final long timestamp = System.currentTimeMillis();
     // Exposure types share names with the imported UFC Allocation, Flag, and Variant types.
     final datadog.trace.api.featureflag.exposure.Allocation allocation =
@@ -726,9 +846,14 @@ static ExposureEvent exposureEvent(
         new datadog.trace.api.featureflag.exposure.Variant(variantKey);
     final Subject subject = new Subject(context.getTargetingKey(), flattenContext(context));
 
-    return EXPOSURE_SERIAL_ID_SUPPORTED.get()
-        ? new ExposureEvent(timestamp, allocation, exposureFlag, variant, subject, serialId)
-        : new ExposureEvent(timestamp, allocation, exposureFlag, variant, subject);
+    if (!EXPOSURE_SERIAL_ID_SUPPORTED.get()) {
+      return new ExposureEvent(timestamp, allocation, exposureFlag, variant, subject);
+    }
+    if (features.isEmpty() || !EXPOSURE_FEATURES_SUPPORTED.get()) {
+      return new ExposureEvent(timestamp, allocation, exposureFlag, variant, subject, serialId);
+    }
+    return new ExposureEvent(
+        timestamp, allocation, exposureFlag, variant, subject, serialId, features);
   }
 
   static AbstractMap<String, Object> flattenContext(final EvaluationContext context) {

products/feature-flagging/feature-flagging-api/src/main/java/datadog/trace/api/openfeature/DatadogExposureLoggingGate.java

@@ -0,0 +1,32 @@
+package datadog.trace.api.openfeature;
+
+import datadog.trace.api.featureflag.config.FeatureFlaggingConfig;
+import datadog.trace.bootstrap.config.provider.ConfigProvider;
+import java.util.function.BooleanSupplier;
+
+/**
+ * Reads {@code DD_FEATURE_FLAGS_EXPOSURES_DATADOG_LOGGING_ENABLED} with full {@link ConfigProvider}
+ * precedence. When false, exposures are not sent to Datadog, but the exposure cache and customer
+ * {@link ExposureHook}s keep working. On by default.
+ */
+final class DatadogExposureLoggingGate {
+
+  private DatadogExposureLoggingGate() {}
+
+  static boolean isEnabled() {
+    return isEnabled(
+        () ->
+            ConfigProvider.getInstance()
+                .getBoolean(
+                    FeatureFlaggingConfig.FEATURE_FLAGS_EXPOSURES_DATADOG_LOGGING_ENABLED, true));
+  }
+
+  static boolean isEnabled(final BooleanSupplier setting) {
+    try {
+      return setting.getAsBoolean();
+    } catch (final Throwable t) {
+      // Keep the default rather than silently stop sending exposures.
+      return true;
+    }
+  }
+}