parsec is a DHT lookup performance measurement tool. It specifically measures the PUT and GET performance of the
IPFS Amino DHT but could also be configured to measure
other libp2p-kad-dht networks.
The setup is split into two components: a scheduler and a server.
The server is just a normal libp2p peer that supports and participates in the public IPFS DHT and exposes a lean HTTP API that allows the scheduler to issue publication and retrieval operations. Currently, in ProbeLab's deployment, the scheduler goes around all deployed server nodes, instructs one to publish provider records for a random data blob and asks the others' to look them up. All servers take timing measurements about the publication or retrieval latencies and report back the results to the scheduler. The scheduler then tracks this information in a database for later analysis.
Next to the concept of servers and schedulers there's the concept of a fleet. A fleet is a set of server nodes that
have a common configuration. For example, we are running three different fleets with multiple nodes each (in different regions): 1) default 2) fullrt 3) cid.contact
Each of these three fleets are configured differently. The default fleet uses the default configuration in the go-libp2p-kad-dht repository, the optprov fleet uses the optimistic provide configuration to publish data into the DHT, the fullrt fleet uses the accelerated DHT client, and the cid.contact fleet probes the InterPlanetary Network Index which hasn't been mentioned yet.
Schedulers are then configured to interface with any combination of fleets. Right now, we have one scheduler for each fleet. As said above, it asks one node to publish content, then instructs the others to find the provider records, and then repeats the process with the next peer. However,
we could configure a scheduler that does the same thing but with nodes from multiple fleets e.g., default+fullrt to check if content that's published with one implementation is reachable with another one.
docker compose upstarts two servers, one scheduler and a PostgreSQL database and lets them interact. To run the binary on its own, start
a database with just database and just db-reset, then:
just build
./dist/parsec server dht --fleet=local
./dist/parsec scheduler --fleets=localEvery flag has an environment variable. .env.example lists all of them with their defaults.
Run parsec --help, parsec server --help and parsec scheduler --help for the descriptions.
| Command | Environment prefix | Examples |
|---|---|---|
| all | PARSEC_ |
PARSEC_DEBUG, PARSEC_LOG_LEVEL (default 4), PARSEC_TELEMETRY_HOST (default localhost), PARSEC_TELEMETRY_PORT (default 6666), PARSEC_DATABASE_* |
scheduler |
PARSEC_SCHEDULER_ |
FLEETS, INTERVAL (default 1m), PROBES (default 1) |
server |
PARSEC_SERVER_ |
SERVER_HOST/SERVER_PORT (default localhost:7070), PEER_PORT (default 4001), FLEET, LEVELDB, FIREHOSE_* |
server dht |
PARSEC_SERVER_DHT_ |
PROJECT (default ipfs), NETWORK (default amino), FULLRT, OPTPROV, SERVER_MODE, ALPHA (default 10) |
server ipni |
PARSEC_SERVER_IPNI_ |
INDEXER_HOST, INDEXER_INTERVAL, PUBLISHER_HOST, PUBLISHER_PORT |
On AWS ECS, ECS_CONTAINER_METADATA_URI_V4 and AWS_REGION give the server its CPU, memory, IP address and region.
Parsec serves Prometheus metrics, and pprof, on PARSEC_TELEMETRY_HOST:PARSEC_TELEMETRY_PORT (default
localhost:6666). It uses the Prometheus client directly, so names carry a parsec_ prefix and do not yet follow
the ProbeLab naming rules. A server implementation in another language needs no metrics.
| Metric | Type | Unit | Labels | Exported by |
|---|---|---|---|---|
parsec_durations |
summary (p50, p90, p99; max age 24h) | seconds | type (retrieval_ttfpr, provide_duration), success, scheduler |
server |
parsec_http_requests_total |
counter | requests | method, path, scheduler |
server |
parsec_active_nodes |
gauge | nodes | none | scheduler |
parsec_issued_provides |
counter | operations | success |
scheduler |
parsec_issued_retrievals |
counter | operations | success |
scheduler |
parsec_datastore_disk_usage |
gauge | bytes | none | scheduler |
The Grafana dashboard link is not yet documented here.
| Command | Does |
|---|---|
just |
Lists the recipes |
just build |
Builds dist/parsec |
just test |
Runs the tests |
just lint |
Checks formatting, go fix, go vet and staticcheck |
just format |
Formats the code with gofumpt and applies go fix |
just ci |
Runs lint, build and test, as CI does |
just tools, just database, just db-reset, just models |
Install the database tools, start PostgreSQL, rebuild the schema, regenerate pkg/models |
Copy .env.example to .env for local settings. Run just ci before you open a pull request.
ProbeLab runs the scheduler and the servers as AWS ECS services in several regions, with one scheduler per fleet. The
probelab-infra repository holds the Terraform
(tf/parsec_*.tf, tf/modules/parsec_server, tf/modules/parsec_scheduler). Every merge to main builds the
image probelab:parsec-sha<commit> in ProbeLab's ECR. A pull request with the label push-image pushes its
image too. To deploy, set the image tag in probelab-infra.
Right now, the server component is implemented in Go and uses the go-libp2p-kad-dht implementation. It consequently measures the Go implementations performance. However, other implementations exist that support the DHT protocol. These implementations can be easily integrated with this measurement infrastructure. They just need to behave as a parsec server. The existing schedulers can then be reused.
Here are the things that a new implementation would need to do:
- Expose an HTTP interface with three endpoints
- Upon startup write general information about the node configuration to a postgres database.
- Regularly refresh the heartbeat field in the database.
You can find the OpenAPI specification in the ./server.yaml.
The new server would need to initialize a postgres client. The default environment variables to configure the client are as follows:
PARSEC_DATABASE_HOST
PARSEC_DATABASE_PORT
PARSEC_DATABASE_NAME
PARSEC_DATABASE_PASSWORD
PARSEC_DATABASE_USER
PARSEC_DATABASE_SSL_MODEThen upon startup, the server needs to write a row into the nodes_ecs table. The definition looks like this:
CREATE TABLE nodes_ecs
(
-- auto generated, doesn't need to be set manually
id INT GENERATED ALWAYS AS IDENTITY,
-- available CPUs
cpu INT NOT NULL,
-- available memory rounded to the nearest MB
memory INT NOT NULL,
-- the peer ID of the libp2p host
peer_id TEXT NOT NULL,
-- in which region does this server/node run? Given via the AWS_REGION environment var
region TEXT NOT NULL,
-- os.Args - with which arguments was this server run?
cmd TEXT NOT NULL,
-- a fleet identifier (see section `Concepts` above)
fleet TEXT NOT NULL,
-- a JSON document with no enforced schema. Could be anything really. It's intended
-- to give information about the exact dependencies, and especially kad-dht implementation
-- that the server uses.
dependencies JSONB NOT NULL,
-- the private IP address of the server. The scheduler will query this table and use this
-- ip address to contact the HTTP API. In the ECS context it's provided via an environment
-- variable called `ECS_CONTAINER_METADATA_URI_V4`. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v4.html
-- see below for the JSON structure
ip_address INET NOT NULL,
-- under which port is the HTTP server reachable
server_port SMALLINT NOT NULL,
-- which port does the libp2p host use
peer_port SMALLINT NOT NULL,
-- a timestamp of the last heartbeat
last_heartbeat TIMESTAMPTZ,
-- a timestamp since when the node is offline (set from the scheduler if the node is unreachable, e.g., crashed)
-- but should also be set from server when shutdown gracefully.
offline_since TIMESTAMPTZ,
-- when was this node row created.
created_at TIMESTAMPTZ NOT NULL,
PRIMARY KEY (id)
);This row serves two purposes: 1) it tracks the exact configuration and dependencies of the server and 2) is used for service discovery.
The scheduler will query this table for all nodes that matches the fleet that the scheduler is after where the offline_since field is null and
the last_heartbeat is not null. In AWS, we're using VPC peering between different AWS regions, so we can use private IP addresses for connectivity between the scheduler, servers, and database.
The server must update the node_ecs row every minute to indicate it's still alive and happy to accept requests.
The server can extract the available CPU and Memory from Limits.CPU and Limits.Memory. Further,
its own private IP address is in the Networks array. Look for the first entry with NetworkMode == awsvpc
and then just take the first entry of IPv4Addresses - that's a good enough heuristic so far.
{
"DockerId": "ea32192c8553fbff06c9340478a2ff089b2bb5646fb718b4ee206641c9086d66",
"Name": "curl",
"DockerName": "ecs-curltest-24-curl-cca48e8dcadd97805600",
"Image": "111122223333.dkr.ecr.us-west-2.amazonaws.com/curltest:latest",
"ImageID": "sha256:d691691e9652791a60114e67b365688d20d19940dde7c4736ea30e660d8d3553",
"Labels": {
"com.amazonaws.ecs.cluster": "default",
"com.amazonaws.ecs.container-name": "curl",
"com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-west-2:111122223333:task/default/8f03e41243824aea923aca126495f665",
"com.amazonaws.ecs.task-definition-family": "curltest",
"com.amazonaws.ecs.task-definition-version": "24"
},
"DesiredStatus": "RUNNING",
"KnownStatus": "RUNNING",
"Limits": {
"CPU": 10,
"Memory": 128
},
"CreatedAt": "2020-10-02T00:15:07.620912337Z",
"StartedAt": "2020-10-02T00:15:08.062559351Z",
"Type": "NORMAL",
"LogDriver": "awslogs",
"LogOptions": {
"awslogs-create-group": "true",
"awslogs-group": "/ecs/metadata",
"awslogs-region": "us-west-2",
"awslogs-stream": "ecs/curl/8f03e41243824aea923aca126495f665"
},
"ContainerARN": "arn:aws:ecs:us-west-2:111122223333:container/0206b271-b33f-47ab-86c6-a0ba208a70a9",
"Networks": [
{
"NetworkMode": "awsvpc",
"IPv4Addresses": [
"10.0.2.100"
],
"AttachmentIndex": 0,
"MACAddress": "0e:9e:32:c7:48:85",
"IPv4SubnetCIDRBlock": "10.0.2.0/24",
"PrivateDNSName": "ip-10-0-2-100.us-west-2.compute.internal",
"SubnetGatewayIpv4Address": "10.0.2.1/24"
}
]
}Feel free to dive in! Open an issue or submit PRs.