Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions influxdata/library/plugin_library.json
Original file line number Diff line number Diff line change
Expand Up @@ -138,8 +138,8 @@
"path": "influxdata/notifier/notifier_plugin.py"
}
],
"required_libraries": ["requests"],
"last_update": "2025-06-16",
"required_libraries": ["influxdata-plugin-utils>=0.3.0", "requests"],
"last_update": "2026-08-03",
"trigger_types_supported": ["scheduler", "data_writes"]
},
{
Expand Down
51 changes: 31 additions & 20 deletions influxdata/threshold_deadman_checks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@ This plugin includes a JSON metadata schema in its docstring that defines suppor

### Required parameters

| Parameter | Type | Default | Description |
|---------------|--------|----------|---------------------------------------------------------------------------------------------------|
| `measurement` | string | required | Measurement to monitor for deadman alerts and aggregation-based conditions |
| `senders` | string | required | Dot-separated notification channels with multi-channel notification integration |
| `window` | string | required | Time window for periodic data presence checking |
| Parameter | Type | Default | Description |
|---------------|--------|----------|------------------------------------------------------------------------------------------------------------------------------------------|
| `measurement` | string | required | Measurement to monitor for deadman alerts and aggregation-based conditions |
| `senders` | string | required | Dot-separated notification channels with multi-channel notification integration |
| `window` | string | required | Time window for periodic data presence checking. Format: `<number><unit>`, units: `s`, `min`, `h`, `d`, `w`. Must be a positive duration |

### Data write trigger parameters

Expand All @@ -33,12 +33,12 @@ This plugin includes a JSON metadata schema in its docstring that defines suppor

### Threshold check parameters

| Parameter | Type | Default | Description |
|----------------------------|---------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `field_aggregation_values` | string | none | Multi-level aggregation conditions with aggregation support for avg, min, max, count, sum, median, stddev, first_value, last_value, var, and approx_median values |
| `deadman_check` | boolean | false | Enable deadman detection to monitor for data absence and missing data streams |
| `interval` | string | "5min" | Configurable aggregation time interval for batch processing with performance optimization |
| `trigger_count` | number | 1 | Configurable triggers requiring multiple consecutive failures before alerting |
| Parameter | Type | Default | Description |
|----------------------------|---------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `field_aggregation_values` | string | none | Multi-level aggregation conditions with aggregation support for avg, min, max, count, sum, median, stddev, first_value, last_value, var, and approx_median values |
| `deadman_check` | boolean | false | Enable deadman detection to monitor for data absence and missing data streams |
| `interval` | string | "5min" | Aggregation time interval used in `DATE_BIN`. Format: `<number><unit>`, units: `s`, `min`, `h`, `d`, `w` |
| `trigger_count` | number | 1 | Breaches required before alerting. Threshold checks count consecutive breaches per row identifier, including across the time bins of a single run; deadman checks count consecutive runs without data |

### Notification parameters

Expand All @@ -57,7 +57,11 @@ This plugin includes a JSON metadata schema in its docstring that defines suppor
|--------------------|--------|---------|----------------------------------------------------------------------------------|
| `config_file_path` | string | none | TOML config file path relative to `PLUGIN_DIR` (required for TOML configuration) |

*To use a TOML configuration file, set the `PLUGIN_DIR` environment variable and specify the `config_file_path` in the trigger arguments.* This is in addition to the `--plugin-dir` flag when starting InfluxDB 3.
*To use a TOML configuration file, set the `PLUGIN_DIR` environment variable and specify the `config_file_path` in the trigger arguments.* This is in addition to the `--plugin-dir` flag when starting InfluxDB 3. Relative paths are resolved against the first directory that is set: `PLUGIN_DIR`, then `INFLUXDB3_PLUGIN_DIR`, then the parent of `VIRTUAL_ENV`. Only that directory is used — the file is not looked up in the remaining ones.

When `config_file_path` is set, the TOML file provides the whole configuration and inline trigger arguments are ignored. `INFLUXDB3_AUTH_TOKEN` from the environment still applies when `influxdb3_auth_token` is not set in the file. In TOML, `senders`, `field_conditions`, and `field_aggregation_values` use native structures instead of the inline string formats.

Data write triggers cache the loaded configuration for 10 minutes to keep the write path fast, so configuration changes take effect within that window.

Example TOML configuration files provided:

Expand All @@ -77,6 +81,7 @@ The plugin assumes that the table schema is already defined in the database, as
## Software requirements

- **InfluxDB v3 Core/Enterprise**: with the Processing Engine enabled.
- **Python packages**: `influxdata-plugin-utils>=0.3.0`, `requests`
- **Notification Sender Plugin for InfluxDB 3**: This plugin is required for sending notifications. See the [influxdata/notifier plugin](../notifier/README.md).

## Installation steps
Expand All @@ -94,6 +99,7 @@ The plugin assumes that the table schema is already defined in the database, as
2. **Install required Python packages**:

```bash
influxdb3 install package "influxdata-plugin-utils>=0.3.0"
influxdb3 install package requests
```

Expand All @@ -110,7 +116,7 @@ influxdb3 create trigger \
--database mydb \
--path "gh:influxdata/threshold_deadman_checks/threshold_deadman_checks_plugin.py" \
--trigger-spec "every:10m" \
--trigger-arguments "measurement=cpu,senders=slack,field_aggregation_values=temp:avg@>=30-ERROR,window=10m,trigger_count=3,deadman_check=true,slack_webhook_url=$SLACK_WEBHOOK_URL" \
--trigger-arguments "measurement=cpu,senders=slack,field_aggregation_values=temp:avg@>=30-ERROR,window=10min,trigger_count=3,deadman_check=true,slack_webhook_url=$SLACK_WEBHOOK_URL" \
threshold_scheduler
```

Expand Down Expand Up @@ -159,7 +165,7 @@ influxdb3 create trigger \
--database sensors \
--path "gh:influxdata/threshold_deadman_checks/threshold_deadman_checks_plugin.py" \
--trigger-spec "every:5m" \
--trigger-arguments "measurement=heartbeat,senders=slack,window=5m,deadman_check=true,slack_webhook_url=$SLACK_WEBHOOK_URL" \
--trigger-arguments "measurement=heartbeat,senders=slack,window=5min,deadman_check=true,slack_webhook_url=$SLACK_WEBHOOK_URL" \
heartbeat_monitor

influxdb3 enable trigger --database sensors heartbeat_monitor
Expand All @@ -185,7 +191,7 @@ influxdb3 create trigger \
--database sensors \
--path "gh:influxdata/threshold_deadman_checks/threshold_deadman_checks_plugin.py" \
--trigger-spec "every:15m" \
--trigger-arguments "measurement=heartbeat,senders=sms,window=10m,deadman_check=true,trigger_count=2,twilio_from_number=+1234567890,twilio_to_number=+0987654321,notification_deadman_text=CRITICAL: No heartbeat data from \$table between \$time_from and \$time_to" \
--trigger-arguments "measurement=heartbeat,senders=sms,window=10min,deadman_check=true,trigger_count=2,twilio_from_number=+1234567890,twilio_to_number=+0987654321,notification_deadman_text=CRITICAL: No heartbeat data from \$table between \$time_from and \$time_to" \
heartbeat_monitor
```

Expand All @@ -198,7 +204,7 @@ influxdb3 create trigger \
--database monitoring \
--path "gh:influxdata/threshold_deadman_checks/threshold_deadman_checks_plugin.py" \
--trigger-spec "every:5m" \
--trigger-arguments "measurement=system_metrics,senders=slack.discord,field_aggregation_values='cpu_usage:avg@>=80-WARN cpu_usage:avg@>=95-ERROR memory_usage:max@>=90-WARN',window=5m,interval=1min,trigger_count=3,slack_webhook_url=$SLACK_WEBHOOK_URL,discord_webhook_url=$DISCORD_WEBHOOK_URL" \
--trigger-arguments "measurement=system_metrics,senders=slack.discord,field_aggregation_values='cpu_usage:avg@>=80-WARN cpu_usage:avg@>=95-ERROR memory_usage:max@>=90-WARN',window=5min,interval=1min,trigger_count=3,slack_webhook_url=$SLACK_WEBHOOK_URL,discord_webhook_url=$DISCORD_WEBHOOK_URL" \
system_threshold_monitor
```

Expand Down Expand Up @@ -228,7 +234,7 @@ influxdb3 create trigger \
--database comprehensive \
--path "gh:influxdata/threshold_deadman_checks/threshold_deadman_checks_plugin.py" \
--trigger-spec "every:10m" \
--trigger-arguments "measurement=temperature_sensors,senders=whatsapp,field_aggregation_values='temperature:avg@>=35-WARN temperature:max@>=40-ERROR',window=15m,deadman_check=true,trigger_count=2,twilio_from_number=+1234567890,twilio_to_number=+0987654321" \
--trigger-arguments "measurement=temperature_sensors,senders=whatsapp,field_aggregation_values='temperature:avg@>=35-WARN temperature:max@>=40-ERROR',window=15min,deadman_check=true,trigger_count=2,twilio_from_number=+1234567890,twilio_to_number=+0987654321" \
comprehensive_sensor_monitor
```

Expand All @@ -240,6 +246,9 @@ influxdb3 create trigger \
- `threshold_deadman_checks_plugin.py`: The main plugin code containing handlers for scheduled and data write triggers
- `threshold_deadman_config_scheduler.toml`: Example TOML configuration for scheduled triggers
- `threshold_deadman_config_data_writes.toml`: Example TOML configuration for data write triggers
- `test_threshold_deadman_checks.py`: Pytest suite (59 tests, runs without a live InfluxDB 3 server)
- `requirements.txt`: Runtime dependencies (`influxdata-plugin-utils>=0.3.0`, `requests`)
- `requirements-dev.txt`: Development dependencies (`pytest`)

### Logging

Expand Down Expand Up @@ -269,7 +278,7 @@ Handles real-time threshold monitoring on data writes. Evaluates incoming data a

#### Issue: False positive alerts

**Solution**: Increase `trigger_count` to require more consecutive failures. Adjust threshold values to be less sensitive. Consider longer aggregation intervals for noisy data.
**Solution**: Increase `trigger_count` to require more consecutive breaches. In scheduled mode every `DATE_BIN` bin of the window counts as a breach, so keep `window`, `interval`, and `trigger_count` aligned. Adjust threshold values to be less sensitive. Consider longer aggregation intervals for noisy data.

#### Issue: Missing deadman alerts

Expand Down Expand Up @@ -333,12 +342,14 @@ Handles real-time threshold monitoring on data writes. Evaluates incoming data a
- `$op_sym`: Operator symbol
- `$compare_val`: Threshold value
- `$actual`: Actual field value
- `$trigger_count`: Consecutive matches required before alerting
- `$row`: Unique identifier

### Row identification

The `row` variable uniquely identifies alert contexts using format: `measurement:level:tag1=value1:tag2=value2`
The `row` variable uniquely identifies alert contexts using format: `measurement:field[:aggregation]:level:tag1=value1:tag2=value2` (`aggregation` is present for scheduled threshold checks only). Tags without a value are omitted.

This ensures trigger counts are maintained independently for each unique combination of measurement, severity level, and tag values.
Trigger counts are maintained independently for each unique combination of measurement, field, aggregation, severity level, tag values, **and the condition's operator and threshold** — two conditions that differ only by threshold never share a count.

## Questions/Comments

Expand Down
4 changes: 2 additions & 2 deletions influxdata/threshold_deadman_checks/manifest.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ manifest_schema_version = "1.3"

[plugin]
name = "threshold_deadman_checks"
version = "1.2.0"
version = "1.3.0"
description = "Provides comprehensive monitoring capabilities including deadman alerts and aggregation-based threshold checks. Supports both scheduler and data write triggers with multi-channel notifications."
triggers = ["process_writes", "process_scheduled_call"]
homepage = "https://www.influxdata.com/"
Expand All @@ -16,7 +16,7 @@ exclude = [

[dependencies]
database_version = ">=3.0.0"
python = ["requests"]
python = ["influxdata-plugin-utils>=0.3.0", "requests"]

[[dependencies.plugins]]
index_url = "https://github.com/influxdata/influxdb3_plugins/releases/download/registry/index.json"
Expand Down
3 changes: 3 additions & 0 deletions influxdata/threshold_deadman_checks/requirements-dev.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
pytest
influxdata-plugin-utils>=0.3.0
requests
1 change: 1 addition & 0 deletions influxdata/threshold_deadman_checks/requirements.txt
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
influxdata-plugin-utils>=0.3.0
requests
Loading