Measure the performance numbers of the article, and build its table - #12
Merged
Merged
Conversation
The article this repository reproduces is first of all a performance paper, and
its tables are made of three numbers this solver did not measure: the sparsity
index of the adapted mesh, the cell updates per second, and the time to
solution. Without them the repository reproduces pictures and compares nothing.
euler/metrics.hpp measures them where they happen rather than leaving them to
be reconstructed from a log, and the conventions are the article's:
sparsity index cells of the adapted mesh over the cells of the uniform
mesh at max-level, reported at the initial and at the final
time. A Riemann problem fills its mesh up as the waves
spread, and one number taken at one end would flatter or
damn it.
cell updates one cell advanced by one time step, counted once per cell
per step whatever the integrator does inside. SSP-RK2
evaluates the flux twice and Strang sweeps 2*dim - 1 times;
counting those would make the more expensive integrator
look faster.
time to solution the time loop. The mesh adaptation is part of it and is
reported apart, which is the number the comparison of a
multiresolution against a gradient criterion turns on.
Writing files is not part of it: --nfiles is a choice of
whoever runs the solver, and a table of times that moved
with it would compare nothing.
Everything is accumulated on the local subdomain and reduced once, at report
time, which is a collective call: cells and updates summed over the ranks,
times taken at their maximum.
--metrics-file writes the same numbers as JSON, and python/performance.py makes
the table out of one run per resolution. On the reference case of the article,
configuration 3 of Lax & Liu to t_f = 0.8 at second order, on one core:
l_min l_max resolution Mcu/s time (s) AMR cells ti/tf sparsity ti/tf
3 6 64^2 2.2 0.39 25.1% 1720 / 3784 42.0% / 92.4%
3 7 128^2 2.4 2.49 29.6% 3916 / 13522 23.9% / 82.5%
3 8 256^2 2.5 15.41 28.1% 8416 / 45394 12.8% / 69.3%
3 9 512^2 2.5 102.92 28.4% 17524 / 151480 6.7% / 57.8%
9 9 512^2 4.7 147.64 0.0% 262144 / 262144 100.0% / 100.0%
The last row is the uniform mesh at the same resolution, the reference the
article puts at the bottom of its own table: the adapted run does 2.7 times
fewer cell updates and is 1.4 times faster, running at a little more than half
the throughput of the uniform one. About a third of what is lost is the
adaptation itself; the rest is what a cell of an adapted mesh costs.
The sparsity column is where a comparison with the article goes wrong most
easily. What the multiresolution keeps is a neighbourhood of the
discontinuities, which are curves in a plane, so their share of the mesh falls
with the resolution -- 92%, 83%, 69%, 58% over the four rows above. A sparsity
index quoted without the max-level it was measured at compares nothing. The
threshold is the other half: --mr-eps takes several values and measures each,
and two decades of it take the final sparsity at level 9 from 58% to 16% and
the time to solution from 115 s to 59 s, with the initial mesh unmoved, the
details of a piecewise constant state being of order one at a discontinuity and
far above every threshold in that range. Those rows say nothing about accuracy,
which is what python/error_analysis.py measures.
The metrics are held against what the run computed rather than against
themselves: the sparsity index is 100% on a uniform mesh whose cell count is
arithmetic on a unit domain, the final cell count is the cell count of the HDF5
file the run wrote, and the time to solution holds the adaptation and not the
output. One slow test holds the fall of the sparsity index with the resolution,
which is the reading every comparison with the article depends on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The three numbers the tables of the article are made of, measured by the solver
instead of reconstructed from a log, and the script that turns them into the
table.
What is measured
euler/metrics.hpp, wired into the three binaries:--max-level, at the initial and at the final time. A Riemann problem fills its mesh up as the waves spread, and one number taken at one end would flatter or damn it.2*dim - 1sweeps would read as more work done.--nfilesbeing a choice of whoever runs the solver.Reduced over the ranks once, in a collective call: cells and updates summed,
times taken at their maximum.
--metrics-filewrites the same numbers as JSON.The table
python python/performance.py --levels 6 7 8 9 --uniform, on configuration 3of Lax & Liu to
t_f = 0.8at second order, one core:The last row is the uniform mesh at the same resolution, the reference the
article puts at the bottom of its own table: the adapted run does 2.7 times
fewer cell updates and is 1.4 times faster, running at a little more than half
the throughput of the uniform one. About a third of what is lost is the
adaptation itself; the rest is what a cell of an adapted mesh costs.
--mr-epstakes several values and measures each. At level 9, two decades ofthreshold take the final sparsity from 58% to 16% and the time to solution from
115 s to 59 s, with the initial mesh unmoved — the details of a piecewise
constant state are of order one at a discontinuity, far above every threshold in
that range. Those rows say nothing about accuracy, which is what
python/error_analysis.pymeasures.Reading it against the article
The readme says it at length, and the short version is that the sparsity column
is where the comparison goes wrong most easily. What the multiresolution keeps
is a neighbourhood of the discontinuities, which are curves in a plane, so their
share of the mesh falls with the resolution: 92%, 83%, 69%, 58% over the four
rows. A sparsity index quoted without the max-level it was measured at compares
nothing.
Tests
Held against what the run computed, not against themselves: 100% sparsity on a
uniform mesh whose cell count is arithmetic on a unit domain, final cell count
equal to the cell count of the HDF5 file the run wrote, time to solution holding
the adaptation and not the output. One slow test holds the fall of the sparsity
index with the resolution.
136 fast tests, 23 slow.