Accelerate PD jobs with tmpfs + rsync (working scripts + demo) - #5
Draft
sethupathib wants to merge 13 commits into
Draft
Accelerate PD jobs with tmpfs + rsync (working scripts + demo)#5sethupathib wants to merge 13 commits into
sethupathib wants to merge 13 commits into
Conversation
Stage Physical Design workspaces into RAM-backed tmpfs, run tools against the hot tree, and rsync checkpoints back to durable storage. Includes a license-free demo, I/O microbench, and LinkedIn post draft. Co-authored-by: Sethupathi Balakrishnan <b.sethupathi@gmail.com>
When design DBs exceed available RAM, redirect only chatty paths (logs, tmp, TMPDIR) into a small /dev/shm scratch via symlinks, then materialize them back to durable storage on teardown. Co-authored-by: Sethupathi Balakrishnan <b.sethupathi@gmail.com>
PD tool logs are often 20GB+ and must not consume limited RAM. Default PD_RAM_PATHS is now tmp; docs steer fat logs to local SSD. Co-authored-by: Sethupathi Balakrishnan <b.sethupathi@gmail.com>
Lead with limited-RAM hybrid (tmp/TMPDIR only). Explicitly reject putting 20GB+ logs in tmpfs. Fix diagram, benefits table, and LinkedIn draft. Co-authored-by: Sethupathi Balakrishnan <b.sethupathi@gmail.com>
Replace the outdated "logs in RAM" framing with Mode B hybrid as the default and Mode A full-workspace only when the tree fits. Co-authored-by: Sethupathi Balakrishnan <b.sethupathi@gmail.com>
- Root README no longer says "stage everything into RAM" - Mode A: PD_KEEP_LOGS_ON_DISK=1 rewires logs/ to durable disk - Fix rsync exclude (logs not logs/) so symlink cannot overwrite durable - Align Mode A/B docs, LinkedIn draft, and script comments Co-authored-by: Sethupathi Balakrishnan <b.sethupathi@gmail.com>
No metal-fill project exists on main; use RC Extraction as a real PD binary. Wrapper runs Mode B/A with logs on disk, tmp/TMPDIR in RAM (or workspace in tmpfs), and durable SPEF outputs. Co-authored-by: Sethupathi Balakrishnan <b.sethupathi@gmail.com>
Benchmark baseline disk job against Mode B/A, reporting tool-only and end-to-end wall time plus SPEF checksum equality. Co-authored-by: Sethupathi Balakrishnan <b.sethupathi@gmail.com>
Bring BEOL metal-fill engine from beol-metal-fill-partitioning branch and benchmark baseline disk vs Mode B/A tmpfs acceleration on the GPU-block fill. Co-authored-by: Sethupathi Balakrishnan <b.sethupathi@gmail.com>
Liberty-vault random lookups + checkpoint/report I/O workload, benchmarked baseline vs Mode B/A. Avoid flushing regenerable scratch so e2e reflects the win. Co-authored-by: Sethupathi Balakrishnan <b.sethupathi@gmail.com>
Multi-phase liberty/SPEF/ECO/report workload. Baseline emulates NFS per-op RTT; Mode B/A run identical work on tmpfs for ~40-50x speedups. Co-authored-by: Sethupathi Balakrishnan <b.sethupathi@gmail.com>
Fourteen-section design/ops white paper covering Mode A/B architecture, log policy, safety model, experiments, and interview defense, with a generated PDF under PD Job Acceleration/docs/. Co-authored-by: Sethupathi Balakrishnan <b.sethupathi@gmail.com>
Wire shell sync after finalize/flush so rsync checkpoints are writeback-hardened, and add pd_perf_profile.sh (perf + GNU time) to classify I/O vs CPU before placing work on tmpfs. Document knobs in README/white paper and regenerate PDF. Co-authored-by: Sethupathi Balakrishnan <b.sethupathi@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Practical tmpfs + rsync toolkit for accelerating Physical Design (PD) jobs, plus a white paper PDF,
perfprofiling, and durablesyncafter rsync.White paper
PD Job Acceleration/WHITEPAPER.mdPD Job Acceleration/docs/pd_tmpfs_rsync_whitepaper.pdf(14 pages)PD Job Acceleration/docs/build_whitepaper_pdf.pyModes
scripts/ram_scratch.shtmp) in/dev/shmscripts/run_pd_job.shFat logs stay on disk
PD logs are often 20GB+. Neither mode puts them in RAM by default.
New:
perf+ shellsyncscripts/pd_perf_profile.sh—perf stat+ GNUtime -v→ I/O-vs-CPU classification and Mode A/B advice. Enable withPD_PERF=1on Mode A/B.pd_durable_sync— after finalize/flush (defaultPD_SYNC_MODE=fs), run shellsyncso rsync success actually means durable writeback. Optional per-checkpoint viaPD_SYNC_AFTER_CHECKPOINT=1.examples/demo_perf_and_sync.shImportant:
sync(1) ≠ rsync. rsync copies;syncflushes page cache to media.Workloads + compare harnesses
Test plan
bash -non updated scriptsexamples/demo_perf_and_sync.sh(perf SUMMARY +.pd_job_status/sync.log)