Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 26 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# dau utils

Host-side operational utilities for DAU bench work
Host utilities for FPGA bench machines

[![Build Status](https://github.com/dau-dev/dau-utils/actions/workflows/build.yaml/badge.svg?branch=main&event=push)](https://github.com/dau-dev/dau-utils/actions/workflows/build.yaml)
[![codecov](https://codecov.io/gh/dau-dev/dau-utils/branch/main/graph/badge.svg)](https://codecov.io/gh/dau-dev/dau-utils)
Expand All @@ -9,48 +9,49 @@ Host-side operational utilities for DAU bench work

## Overview

Small, package-neutral utilities for operating FPGA bench hosts. Each owns
one mechanism; policy (which devices, when to arm) belongs to the caller —
higher layers such as `dau-build` compose these into their hardware
command plans.
Two small command-line tools for the Linux machine an FPGA card is plugged
into. Each one does a single thing and leaves the decisions (which devices,
when to arm) to the caller. Higher layers such as `dau-build` run them
from their hardware command plans; you can also run them by hand.

## Deadman: survive a wedged PCIe device
## Deadman: recover from a hung PCIe device

A driver/PCIe hang can leave the kernel and systemd alive — the hardware
watchdog keeps getting petted, yet a clean `systemctl reboot` blocks on
the stuck device, and the box needs a walk to the power switch. The
deadman arms a forced reboot *before* the risky operation and cancels it
on success:
A driver or PCIe hang can leave the kernel and systemd running while the
device itself is stuck. The hardware watchdog keeps getting fed, but a
`systemctl reboot` blocks on the hung device, and someone has to walk over
and pull the power. The deadman schedules a forced reboot before you do the
risky thing, and you cancel it once the operation comes back cleanly:

```bash
dau-utils-deadman arm --timeout 300 # transient systemd timer; survives SSH loss
dau-utils-deadman arm --timeout 300 # transient systemd timer; outlasts a lost SSH session
# ... rescan / flash / probe ...
dau-utils-deadman disarm # clean return: cancel the reset
dau-utils-deadman disarm # operation returned: cancel the reboot
dau-utils-deadman status
```

If the host wedges before `disarm`, the timer fires `sysrq-b`
(`emergency_restart()`), which resets past the wedged driver without
touching it. The same operations are available as a Python API
(`dau_utils.deadman.arm/disarm/is_armed` and an `armed()` context
manager); failures raise `DeadmanError` — treat the host as unsafe.
If the host hangs before `disarm`, the timer fires `sysrq-b`
(`emergency_restart()`), which resets the machine without going through the
stuck driver. The same operations are available from Python:
`dau_utils.deadman.arm`, `disarm`, `is_armed`, and an `armed()` context
manager. Failures raise `DeadmanError`; if you see one, assume the host is
not protected.

Arm it before every PCIe rescan, flash, or register probe against a device
that has previously wedged. Synthesis and other non-PCIe work does not
need it.
Arm it before every PCIe rescan, flash or register probe against a device
that has hung before. Synthesis and other work that does not touch PCIe does
not need it.

## Runtime PM: hold and release PCIe power management

`dau-utils-pci-runtime-pm` holds or releases Linux PCI runtime PM for
devices matched from `lspci -Dnn`. The caller decides which patterns
matter; this utility owns the sysfs writes:
`dau-utils-pci-runtime-pm` holds or releases Linux PCI runtime power
management for devices matched against `lspci -Dnn`. You choose the patterns;
the tool does the sysfs writes:

```bash
dau-utils-pci-runtime-pm hold --pattern Thunderbolt --pattern 10ee:7011
dau-utils-pci-runtime-pm release --pattern Thunderbolt --pattern 10ee:7011
```

Use `--dry-run` to print the planned writes without touching sysfs.
`--dry-run` prints the writes it would make without touching sysfs.

## Development

Expand Down