Skip to content

Commit ad185ad

Browse files
webstackdevwebstackdev
committed
Add comp for dsar email template
1 parent 003bfbe commit ad185ad

4 files changed

Lines changed: 333 additions & 2 deletions

File tree

‎_TODO.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,3 +30,8 @@ https://aws.plainenglish.io/how-to-build-a-chatbot-using-aws-lex-and-lambda-in-2
3030
## Contact Form
3131

3232
- `0/2000` characters should show number of characters left instead
33+
34+
I put our default data deletion request email template in src/pages/testing/comps/scratchpad.astro
35+
I'd like to get some design ideas for making this page nicer. You're free to generate any idea you like - add images, add sections, change the copy, anything to improve the page for its intended purpose. Add the comp as a block in a div below the existing "Default Layout" section. Since this is an email, we use mjml to generate them. That means we should restrict our CSS to styles that mjml supports / are generally supported by a wide range of email clients (but don't change it to mjml markup, we'll do that for the winning comp).
36+
37+
I want to add several comps, but let's do them one at a time since it's a fairly large task.

‎docs/newsletter/example-issue-copy.md‎

Lines changed: 119 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,120 @@
1-
# Sample Issue Final Copy
1+
# The Kubernetes DNS Bug That Wasted 40 Engineering Hours
2+
3+
*Monthly dispatch from Webstack Builders — platform engineering, DevOps, and cloud infrastructure*
4+
5+
---
6+
7+
## The Deep Dive — Debugging ndots and Why Your DNS Is Slower Than You Think
8+
9+
Three weeks ago, a client's platform team started seeing intermittent 5xx errors from a handful of microservices. Latency percentiles looked normal. CPU and memory were fine. The on-call engineer checked the usual suspects — upstream dependencies, recent deploys, connection pool exhaustion — and found nothing.
10+
11+
Forty engineering hours later, the root cause turned out to be DNS.
12+
13+
**The symptoms were misleading.** Under moderate load, roughly 2% of HTTP requests to internal services would time out. The timeouts were evenly distributed across services, which made it look like a network issue rather than a resolution issue. The team spent a full day chasing a phantom connectivity problem between nodes before someone finally ran `tcpdump` on a pod's network namespace and noticed something odd: every single DNS lookup was generating five queries instead of one.
14+
15+
**The culprit: `ndots:5`.** Kubernetes sets `ndots:5` in every pod's `/etc/resolv.conf` by default. This means that any hostname with fewer than five dots gets treated as a relative name, and the resolver appends each search domain before trying the name as-is. A lookup for `auth-service.production.svc.cluster.local` (four dots) would first try `auth-service.production.svc.cluster.local.production.svc.cluster.local`, then three more permutations, before finally resolving correctly on the fifth attempt.
16+
17+
Under normal load, this is invisible. Under sustained traffic, it was multiplying DNS query volume by 5x and saturating CoreDNS pods that were sized for the expected query rate — not five times the expected query rate.
18+
19+
**What made this hard to find.** The standard CoreDNS metrics — `coredns_dns_requests_total` and `coredns_dns_responses_total` — were elevated but didn't trigger alerts because the team's thresholds were based on historical averages that had gradually crept up. The metrics told the truth; nobody was looking at the right graph.
20+
21+
**The fix was two lines.** In the pod spec's `dnsConfig`:
22+
23+
```yaml
24+
dnsConfig:
25+
options:
26+
- name: ndots
27+
value: "2"
28+
```
29+
30+
This tells the resolver to treat any name with two or more dots as fully qualified, skipping the search domain dance. For internal service names that use the full `<service>.<namespace>.svc.cluster.local` format, this eliminates four unnecessary queries per lookup.
31+
32+
The team also added explicit search domain entries to avoid breaking short names used in legacy configuration:
33+
34+
```yaml
35+
dnsConfig:
36+
searches:
37+
- production.svc.cluster.local
38+
- svc.cluster.local
39+
```
40+
41+
**What should have caught this earlier.** Two monitoring changes went in immediately after the fix:
42+
43+
- A Prometheus alert on `coredns_dns_requests_total` rate-of-change, not just absolute value. A 5x query spike in an hour is never normal.
44+
- A dashboard panel showing cache hit ratio alongside query volume. During the incident, cache hit rate had dropped to 31% — a clear signal that pods were hammering CoreDNS with queries that could never be cached because they were for nonexistent names.
45+
46+
The takeaway isn't "change your ndots setting." It's that default configurations optimized for convenience can become performance landmines at scale, and the monitoring that catches them is rarely the monitoring you set up on day one.
47+
48+
---
49+
50+
## Quick Wins
51+
52+
- **Terraform state backup before every apply.** Add this to your CI pipeline or local workflow. One line, zero regret when someone applies against the wrong workspace:
53+
54+
```bash
55+
terraform state pull > "tfstate-backup-$(date +%Y%m%d-%H%M%S).json" && terraform apply
56+
```
57+
58+
- **Find abandoned Grafana dashboards.** This PromQL query surfaces dashboards with zero views in the last 90 days. Clean them out before your Grafana instance becomes a graveyard of dashboards nobody trusts:
59+
60+
```text
61+
grafana_db_dashboard_last_viewed_at < (time() - 86400 * 90)
62+
```
63+
64+
Run it against your Grafana metrics endpoint, or use the Grafana API: `GET /api/search?query=&sort=viewed-asc` and filter by `meta.lastViewedAt`.
65+
66+
- **Pre-commit secrets scanning that works with monorepos.** Most `gitleaks` setups choke on monorepos because they scan the entire history on every commit. This `.pre-commit-config.yaml` entry scans only staged changes:
67+
68+
```yaml
69+
- repo: https://github.com/gitleaks/gitleaks
70+
rev: v8.18.0
71+
hooks:
72+
- id: gitleaks
73+
args: ["protect", "--staged"]
74+
```
75+
76+
---
77+
78+
## From the Blog
79+
80+
Our latest article covers **structured logging with correlation IDs** — how to thread a single request identifier through every service in a call chain so that debugging distributed failures doesn't require cross-referencing timestamps across six different log streams.
81+
82+
If that DNS investigation above had used correlation IDs, the team could have traced a single failing request from the API gateway through to the DNS timeout in minutes instead of hours. The post walks through implementation patterns for Node.js and Go services, with examples using OpenTelemetry's trace context propagation.
83+
84+
[Read the full article →](#)
85+
86+
---
87+
88+
## What We're Reading
89+
90+
- **Cloudflare's routing incident post-mortem** — A BGP misconfiguration took down a significant chunk of their network for 17 minutes. The post-mortem is worth reading for the timeline alone: how a change that passed validation in staging behaved differently in production because of a subtle difference in route map evaluation order. [Read it →](#)
91+
92+
- **OpenTelemetry Collector tail-sampling processor** — The new tail-sampling processor lets you make sampling decisions after a trace is complete, which means you can keep 100% of error traces and slow traces while sampling routine ones aggressively. If you're spending too much on trace storage, this is the feature to evaluate. [Read it →](#)
93+
94+
- **Google SRE: A practical guide to SLO-based alerting** — Moves past the theory and into implementation. The section on multi-window, multi-burn-rate alerts is the clearest explanation of the concept available. If your alerts still fire on static thresholds, start here. [Read it →](#)
95+
96+
---
97+
98+
## One Thing to Try This Month
99+
100+
Check your CoreDNS cache hit rate. If it's below 80%, you're probably hammering upstream resolvers unnecessarily — and you might be one traffic spike away from the exact scenario described in this issue's deep dive.
101+
102+
Here's the Prometheus query:
103+
104+
```text
105+
sum(rate(coredns_cache_hits_total[5m])) /
106+
(sum(rate(coredns_cache_hits_total[5m])) + sum(rate(coredns_cache_misses_total[5m])))
107+
```
108+
109+
If the number is low, two things to check: your `ndots` setting (see above) and your CoreDNS Corefile's cache TTL. The default cache block caches positive responses for 30 seconds, which is usually too short for internal service names that rarely change. Bumping it to 300 seconds is safe for most clusters:
110+
111+
```text
112+
cache 300
113+
```
114+
115+
Add that line to the `Corefile` ConfigMap and roll the CoreDNS pods. Measure again the next day.
116+
117+
---
118+
119+
*Webstack Builders, Inc. — You're receiving this because you subscribed at webstackbuilders.com.*
2120

‎src/pages/testing/comps/_scratchpad.css‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
#outlook a { padding:0; }
2+
3+
body { margin:0;padding:0;-webkit-text-size-adjust:100%;-ms-text-size-adjust:100%; }
4+
5+
table, td { border-collapse:collapse;mso-table-lspace:0pt;mso-table-rspace:0pt; }
6+
7+
img { border:0;height:auto;line-height:100%; outline:none;text-decoration:none;-ms-interpolation-mode:bicubic; }
8+
9+
p { display:block;margin:13px 0; }
10+
11+
@media only screen and (min-width:480px) {
12+
.mj-column-per-25 { width:25% !important; max-width: 25%; }
13+
.mj-column-per-75 { width:75% !important; max-width: 75%; }
14+
.mj-column-per-100 { width:100% !important; max-width: 100%; }
15+
}
16+
17+
.footer-mobile {
18+
display: none !important;
19+
mso-hide: all !important;
20+
max-height: 0 !important;
21+
overflow: hidden !important;
22+
}
23+
24+
.footer-desktop {
25+
display: block !important;
26+
max-height: none !important;
27+
overflow: visible !important;
28+
}
29+
30+
@media only screen and (max-width: 480px) {
31+
div.footer-desktop,
32+
table.footer-desktop,
33+
tbody.footer-desktop,
34+
tr.footer-desktop,
35+
td.footer-desktop {
36+
display: none !important;
37+
mso-hide: all !important;
38+
max-height: 0 !important;
39+
overflow: hidden !important;
40+
}
41+
42+
div.footer-mobile,
43+
table.footer-mobile,
44+
tbody.footer-mobile,
45+
tr.footer-mobile,
46+
td.footer-mobile {
47+
display: block !important;
48+
width: 100% !important;
49+
max-height: none !important;
50+
overflow: visible !important;
51+
}
52+
53+
td.footer-mobile {
54+
box-sizing: border-box !important;
55+
}
56+
}

‎src/pages/testing/comps/scratchpad.astro‎

Lines changed: 153 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,163 @@
11
---
22
import BaseLayout from '@layouts/BaseLayout.astro'
3+
import './_scratchpad.css'
34
45
const pageTitle = 'Scratchpad'
56
const pageDescription = 'Troubleshooting Component Variants'
67
const path = '/testing/comps/scratchpad'
78
---
89

910
<BaseLayout pageDescription={pageDescription} noindex={true} pageTitle={pageTitle} path={path}>
10-
<h1>Empty</h1>
11+
<div style="margin-top: 64px;">
12+
<h1>Modern Card Layout</h1>
13+
<div aria-label="Verify Your Data Deletion Request" aria-roledescription="email" style="background-color:#f3f4f6; padding: 40px 20px;" role="article" lang="und" dir="auto">
14+
<div style="background:#ffffff; margin:0px auto; max-width:600px; border-radius: 8px; overflow: hidden; box-shadow: 0 4px 6px -1px rgba(0, 0, 0, 0.1);">
15+
<table align="center" border="0" cellpadding="0" cellspacing="0" role="presentation" style="background:#ffffff; width:100%;">
16+
<tbody>
17+
<!-- Header (Logo Centered) -->
18+
<tr>
19+
<td style="padding: 16px 16px 8px 16px; text-align: left; background: #e5e7eb; border-bottom: 1px solid #d1d5db;">
20+
<table align="left" border="0" cellpadding="0" cellspacing="0" role="presentation" style="margin: 0;">
21+
<tr>
22+
<td>
23+
<a href="https://www.webstackbuilders.com" style="color:#001A39; text-decoration:none; display:inline-block;">
24+
<table align="center" cellpadding="0" cellspacing="0" role="presentation" style="width:49px; height:48px; background-color:#ffffff; border:3px solid #ffffff;">
25+
<tbody>
26+
<tr>
27+
<td style="height:17px;">
28+
<table cellpadding="0" cellspacing="0" role="presentation" style="width:100%; height:100%; background-color:#ffffff; border:0;">
29+
<tbody>
30+
<tr>
31+
<td style="width:10px; height:17px; background-color:#001A39;"></td>
32+
<td style="width:3px; height:17px; background-color:#ffffff;"></td>
33+
<td style="width:23px; height:17px; background-color:#001A39;"></td>
34+
<td style="width:3px; height:17px; background-color:#ffffff;"></td>
35+
<td style="width:10px; height:17px; background-color:#001A39;"></td>
36+
</tr>
37+
</tbody>
38+
</table>
39+
</td>
40+
</tr>
41+
<tr>
42+
<td style="height:3px; background-color:#ffffff; line-height:3px; font-size:3px;">&nbsp;</td>
43+
</tr>
44+
<tr>
45+
<td style="height:9px;">
46+
<table cellpadding="0" cellspacing="0" role="presentation" style="width:100%; height:100%; background-color:#ffffff; border:0;">
47+
<tbody>
48+
<tr>
49+
<td style="width:23px; height:9px; background-color:#006DCA;"></td>
50+
<td style="width:3px; height:9px; background-color:#ffffff;"></td>
51+
<td style="width:23px; height:9px; background-color:#006DCA;"></td>
52+
</tr>
53+
</tbody>
54+
</table>
55+
</td>
56+
</tr>
57+
<tr>
58+
<td style="height:2px; background-color:#ffffff; line-height:2px; font-size:2px;">&nbsp;</td>
59+
</tr>
60+
<tr>
61+
<td style="height:17px;">
62+
<table cellpadding="0" cellspacing="0" role="presentation" style="width:100%; height:100%; background-color:#ffffff; border:0;">
63+
<tbody>
64+
<tr>
65+
<td style="width:10px; height:17px; background-color:#001A39;"></td>
66+
<td style="width:3px; height:17px; background-color:#ffffff;"></td>
67+
<td style="width:23px; height:17px; background-color:#001A39;"></td>
68+
<td style="width:3px; height:17px; background-color:#ffffff;"></td>
69+
<td style="width:10px; height:17px; background-color:#001A39;"></td>
70+
</tr>
71+
</tbody>
72+
</table>
73+
</td>
74+
</tr>
75+
</tbody>
76+
</table>
77+
</a>
78+
</td>
79+
<td style="padding-left: 14px; vertical-align: middle;">
80+
<a href="https://www.webstackbuilders.com" style="color:#001A39; text-decoration:none; display:inline-block; font-family:Arial, sans-serif; font-size:24px; font-weight:700; line-height:26px;">
81+
Webstack<br>
82+
Builders
83+
</a>
84+
</td>
85+
</tr>
86+
</table>
87+
</td>
88+
</tr>
89+
<!-- Hero Icon & Title -->
90+
<tr>
91+
<td style="padding: 40px 24px 20px; text-align: center;">
92+
<div style="font-size: 56px; line-height: 56px; margin-bottom: 20px;">👋</div>
93+
<h2 style="margin: 0; font-family: Arial, sans-serif; font-size: 24px; font-weight: 700; color: #001A39; line-height: 32px;">
94+
We received a request to<br>delete your data.
95+
</h2>
96+
</td>
97+
</tr>
98+
<!-- Main Content -->
99+
<tr>
100+
<td style="padding: 0 40px 32px; text-align: left;">
101+
<p style="margin: 0 0 16px; font-family: Arial, sans-serif; font-size: 16px; line-height: 26px; color: #333333;">
102+
Hello,
103+
</p>
104+
<p style="margin: 0 0 24px; font-family: Arial, sans-serif; font-size: 16px; line-height: 26px; color: #333333;">
105+
To complete this request and permanently remove your account, please verify your email address. <strong>This link is valid for 24 hours.</strong>
106+
</p>
107+
108+
<!-- Danger Box -->
109+
<table border="0" cellpadding="0" cellspacing="0" role="presentation" style="width: 100%; background: #fef2f2; border-left: 4px solid #ef4444; border-radius: 4px; margin-bottom: 28px;">
110+
<tr>
111+
<td style="padding: 20px;">
112+
<p style="margin: 0 0 12px; font-family: Arial, sans-serif; font-size: 15px; font-weight: 700; color: #991b1b;">What happens next?</p>
113+
<ul style="margin: 0; padding-left: 20px; font-family: Arial, sans-serif; font-size: 14px; line-height: 24px; color: #7f1d1d;">
114+
<li style="margin-bottom: 4px;">Your account profile will be permanently deleted.</li>
115+
<li style="margin-bottom: 4px;">All usage history and subscriptions will be canceled.</li>
116+
<li>You will immediately be unsubscribed from all communications.</li>
117+
</ul>
118+
</td>
119+
</tr>
120+
</table>
121+
122+
<!-- Primary Action -->
123+
<table border="0" cellpadding="0" cellspacing="0" role="presentation" style="width: 100%; margin-bottom: 24px;">
124+
<tr>
125+
<td align="center">
126+
<a href="https://www.webstackbuilders.com/privacy/my-data?token=mock-dsar-token-123" style="display:inline-block; background:#dc2626; color:#ffffff; font-family:Arial, sans-serif; font-size:16px; font-weight:600; line-height:120%; margin:0; text-decoration:none; text-transform:none; padding:16px 40px; border-radius:6px;" target="_blank">
127+
Yes, Delete My Data
128+
</a>
129+
</td>
130+
</tr>
131+
</table>
132+
133+
<p style="margin: 0 0 16px; font-family: Arial, sans-serif; font-size: 14px; line-height: 22px; color: #666666; text-align: center;">
134+
If the button doesn't work, copy and paste this link:<br>
135+
<a href="https://www.webstackbuilders.com/privacy/my-data?token=mock-dsar-token-123" style="color: #0066cc; word-break: break-all; display: inline-block; margin-top: 4px;">https://www.webstackbuilders.com/privacy/my-data?token=mock-dsar-token-123</a>
136+
</p>
137+
138+
<hr style="border: none; border-top: 1px solid #f0f0f0; margin: 32px 0;">
139+
140+
<p style="margin: 0; font-family: Arial, sans-serif; font-size: 14px; line-height: 22px; color: #666666; text-align: center;">
141+
<strong>Didn't request this?</strong> You can safely ignore this email.<br>No action will be taken without your verification.
142+
</p>
143+
</td>
144+
</tr>
145+
<!-- Minimal Footer -->
146+
<tr>
147+
<td style="padding: 24px 40px; background: #f8fafc; text-align: center; border-top: 1px solid #f0f0f0;">
148+
<p style="margin: 0 0 12px; font-family: Arial, sans-serif; font-size: 13px; line-height: 18px; color: #64748b;">
149+
Questions? Contact us at <a href="mailto:privacy@webstackbuilders.com" style="color: #475569; text-decoration: underline;">privacy@webstackbuilders.com</a>
150+
</p>
151+
<p style="margin: 0; font-family: Arial, sans-serif; font-size: 12px; line-height: 18px; color: #94a3b8;">
152+
© 2026 Webstack Builders, Inc.<br>
153+
1032 E. Brandon Boulevard, Suite 5230, Brandon, FL 33511<br>
154+
1 (888) 987 1881 • 1 (302) 608 6864
155+
</p>
156+
</td>
157+
</tr>
158+
</tbody>
159+
</table>
160+
</div>
161+
</div>
162+
</div>
11163
</BaseLayout>

0 commit comments

Comments
 (0)