-
Notifications
You must be signed in to change notification settings - Fork 1
Add outline for bioimage format skew project #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,2 +1,58 @@ | ||||||
| # tesserift | ||||||
| Measure how bioimaging storage layouts shape performance, analysis, and biological discovery. | ||||||
|
|
||||||
| Bioimaging data can follow a format specification and still perform poorly. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. i recommend adding a bit more specificity to what you mean by "format specification". Also, what do you mean by "perform" and "poorly"? Perform what? Poorly measured how? |
||||||
|
|
||||||
| We will study around **six public datasets from three sources**, with two datasets from each source: | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm wondering how you decided six datasets - is there a reason not to scan through every dataset from these three sources? |
||||||
|
|
||||||
| * **Image Data Resource (IDR):** established OME bioimaging datasets with different image shapes and workloads. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Consider defining OME before first use. Also, what is an "image shape" and what is an "image workload"? |
||||||
| * **SSBD:** multidimensional OME-Zarr data, including 3D and time-series images. | ||||||
| * **Cell Painting Gallery / JUMP:** large-scale imaging data with associated metadata and feature tables. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| Two datasets from each source will help us measure variation both **within** and **between** public data services. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Two is not sufficient to measure these things - how tough would it be to measure all datasets? (i imagine what we're measuring will answer this question!) |
||||||
|
|
||||||
| Unlike general-purpose Zarr/Parquet benchmarks, we tie storage properties to bioimaging-specific workloads and test causality directly, not just correlation. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. too terse. What is a "general purpose Zarr/Parquet benchmark"? What do you mean by "tie" - what is a workload (same question as prior)? Causality of what? Poor performance? (again, going back to defining poor performance) |
||||||
|
|
||||||
| The work will focus on four goals: | ||||||
|
|
||||||
| * **Measure format skew.** Compare chunking, compression, file counts, metadata, schemas, row groups, and related storage properties. | ||||||
| * **Find costly patterns.** Identify storage properties that predict slow access or analysis. | ||||||
| * **Measure real workloads.** Test image reads, table scans, filtering, joins, and related bioimaging tasks. | ||||||
| * **Turn evidence into rules.** Add tested performance rules to `zarr-lint` and `arrow-lint`. | ||||||
|
Comment on lines
+15
to
+20
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this is effective! I recommend moving this up a section or two. Starting with goals is usually helpful! It would also be helpful to add a "motivation" answering "why we're focusing on these goals"? Was there an observation that triggered this research? Add a brief description of what these linting tools are and what they do |
||||||
|
|
||||||
| Cost means wall-clock latency and throughput, measured on a fixed compute and network setup. | ||||||
|
|
||||||
| We will not search for many examples of the same problem. | ||||||
| When we find an important pattern, we will create controlled versions of a dataset that change that property. | ||||||
|
d33bs marked this conversation as resolved.
|
||||||
|
|
||||||
| This lets us test whether the storage difference actually causes the observed performance change. | ||||||
|
d33bs marked this conversation as resolved.
|
||||||
|
|
||||||
| The main question is: | ||||||
|
|
||||||
| **How much do storage layouts vary across public bioimaging resources, and can that variation predict the cost of accessing and analyzing the data?** | ||||||
|
|
||||||
| The manuscript will also examine how these costs affect interactive exploration and large-scale biological analysis. | ||||||
|
d33bs marked this conversation as resolved.
|
||||||
|
|
||||||
| ## Operational outline | ||||||
|
|
||||||
| Keep the complete study in one public GitHub repository. | ||||||
|
|
||||||
| * Write the manuscript with **Quarto Manuscripts**. | ||||||
| * Keep analysis and benchmark code beside the manuscript. | ||||||
| * Store generated figures and tables in the repository. | ||||||
| * Make benchmarks reproducible from scripts or workflows. | ||||||
| * Record dataset versions, inputs, and benchmark settings. | ||||||
| * Develop the manuscript and analysis together. | ||||||
| * Publish the completed manuscript as a **bioRxiv preprint**. | ||||||
| * Select a **right-size peer-reviewed journal** after the results and scope are clear. | ||||||
|
d33bs marked this conversation as resolved.
|
||||||
|
|
||||||
| The repository becomes the manuscript, analysis, figures, and reproducibility record. | ||||||
|
|
||||||
| ## Timeline | ||||||
|
d33bs marked this conversation as resolved.
|
||||||
|
|
||||||
| Rough estimate, ~9 months total: | ||||||
|
|
||||||
| * Months 1-2: select datasets, set up benchmarking infrastructure. | ||||||
| * Months 3-4: measure format skew and workload cost. | ||||||
| * Months 5-6: build controlled dataset variants, test causality. | ||||||
| * Months 7-8: draft the manuscript, publish the bioRxiv preprint. | ||||||
| * Month 9: submit to a right-size journal. | ||||||
Uh oh!
There was an error while loading. Please reload this page.