Add atlas-metrics: metrics-driven sizing collector for MongoDB -> DocumentDB

[Hanganalyze]

What this adds

A new tool at migration/atlas-metrics/: a metrics-driven sizing collector for MongoDB -> Amazon DocumentDB migrations. It collects a multi-day workload metric series (MongoDB Atlas Admin API, or Amazon EC2 + CloudWatch) and produces a sizing package - the Cost Estimator CSV plus a sizing report with an instance recommendation.

Why (relationship to the existing sizing-tool)

This is not a replacement for migration/sizing-tool - it is the measured, deeper-analysis counterpart, and it reuses the same compression-review plumbing. The existing sizing-tool README notes that its workload columns (Data_Working_Set, Inserts/Updates/Deletes/Reads_Per_Day) are placeholder defaults the user must edit by hand, because "these statistics cannot be calculated automatically from database metadata." This tool measures those from a real metric window:

	sizing-tool (today)	atlas-metrics
Source	one --uri, single-shot	Atlas Admin API, or EC2 + CloudWatch, ~14-day P95
Compression sampling	compression-review.py (zstd-3-dict)	same
Collection stats	doc counts, sizes, indexes	same
Workload metrics	manual placeholders	measured (ops/sec, working set, IOPS, CPU, connections)
Operator compatibility	not covered	delegates to compat-tool
Index-type compatibility	not covered	delegates to index-tool
Output	CSV for the Cost Estimator	CSV plus a sizing report with an instance recommendation

The two serve different users: sizing-tool is the quick self-service CSV; atlas-metrics is the measured deep-dive for a hands-on assessment.

What it does

• Collects a multi-day, per-node metric series and reports P95 for ops, connections, IOPS, CPU, memory (Atlas Admin API on the Atlas path; CloudWatch on the EC2 path).
• Runs the same compression sampling and collection-stats collection as sizing-tool.
• Runs an operator-compatibility scan (via compat-tool) and an index-type-compatibility scan (via index-tool), separating real application issues from monitoring-only artifacts.
• Flags unused/redundant indexes and index types DocumentDB does not support.
• Emits the Cost Estimator CSV plus a structured sizing report (instance recommendation, cost breakdown, pre-migration actions).
• Handles replica-set and sharded (mongos) topologies.

Single file, Apache-2.0 (inherits the repo LICENSE), with its own README, CHANGELOG, and pinned requirements.txt. Validated against replica-set and sharded Atlas topologies and against self-managed MongoDB on EC2.

Known follow-ups (kept out of this PR to keep it focused)

1. Sibling-tool resolution. The tool calls compat-tool / index-tool / compression-review; today it locates them relative to itself and clones amazon-documentdb-tools if not found. Now that it lives inside the repo, a follow-up should resolve those siblings via the repo root instead of cloning. It is functional as-is (the clone is the fallback).
2. Name. It began Atlas-specific but now also has the EC2 / CloudWatch source, so a source-neutral name may be preferable - happy to rename if you'd like.

A note on process

I went straight to a PR rather than opening an issue first. Fully happy to convert this into an issue/discussion, reshape it (for example, fold just the metric-collection piece into sizing-tool), or adjust placement and naming - whatever fits how you want the repo organized. Also glad to sign a CLA if a contribution this size needs one.

[dbonser]

Python script

1. Inconsistent default compression ratio (3.5 vs 5). Three different fallback ratios are used when real zstd sampling is unavailable:
   Line 1895 (JSON report estimated_zstd_gb): _sampled_zstd_ratio or 5
   Line 2069 (MD summary zr_div): zr or 5
   Line 2068 (MD label) & 2304 (CSV path): advertises ~3.5:1
   Line ~2380 (CSV ratio): defaults to 3.5
   Pick one documented default.
2. period_to_start() parses years wrong and duplicates _period_days()
   Line 286: it runs the P(\d+)D regex, then unconditionally runs the P(\d+)Y regex, overwriting days. re.match is anchored at start, so P2D won't match P(\d+)Y, and it silently defaults anything it can't parse (e.g. P14D works, but PT12H, P2W) to 30 days.
   Line 205: _period_days() handles D/H/M/W but not Y.
3. PROCESS_CPU_KERNEL etc. requested but never collected
   generate_report calls fm("PROCESS_CPU_KERNEL"), fm("MEMORY_VIRTUAL"), fm("SWAP_USAGE_USED"), fm("DISK_PARTITION_IOPS_READ"), fm("CACHE_FILL_RATIO"), fm("TICKETS_AVAILABLE_READ"), etc., but PROCESS_METRIC_BATCHES never requests these metric names from the API.
   Disk metrics come in via the separate /disks call, but CPU kernel/iowait, virtual memory, swap, cache-fill-ratio, and tickets will always resolve to N/A. Either add them to the batch definitions or drop the report fields.

README

Since the tool supports Atlas and self-managed MongoDB, should it be called mongodb-metrics?

There are 4 items that need to be addressed in the README.

1. Version mismatch in Known Limitations heading The header says v2.3.1, but the Known Limitations section is labeled (v2.0.0). Either update that section's heading to reflect the current version or add a note like "originally documented at v2.0.0, still applicable in 2.3.1."
2. --source, --aws-region, --samples missing from Options table These EC2-specific flags are documented in prose but absent from the Options reference table. Readers who use that table as the definitive flag reference will miss them. Add them (with an "EC2 only" note) or create a separate EC2-Specific Options table.
3. --all override behavior never stated explicitly Add a one-liner: "--all implicitly sets --granularity PT5M --period P14D; explicit values for those flags are ignored when --all is present."
4. PT10S M40+ restriction missing from Options table The Retention table notes PT10S is "M40+ only," but the Options table lists it as a valid granularity value without caveats. Add a parenthetical or footnote.

I also recommend reorganizing the content into 2 main section (Atlas and Self-managed MongoDB) so the reader knows where to find everything. Right now it feels like the two are combined and it's confusing.

MongoDB Metrics Collector -- User Guide

Sources (was Which source?)

Sharded Cluster Support

What Gets Collected

From Atlas Admin API (automatic)

From Direct MongoDB Connection

From Compat-Tool (requires --compat)

Atlas

Prerequisites

Installation (was Step 5: Install the Tool)

Step 1: ...

...

Preflight Checks

Standard sizing run (recommended)

Debug / quick check

Options

Output

...

Self-managed MongoDB

Prerequisites

Installation (was Step 5: Install the Tool)

Step 1: ...

...

Preflight Checks

Standard sizing run (recommended)

Debug / quick check

Options

Output

...

Next Steps After Running

Troubleshooting

Known Limitations

License