Read #subsystem#/MCOS property values on demand - #256
Open
Thenewmanator15 wants to merge 1 commit into
Open
Thenewmanator15 wants to merge 1 commit into
Thenewmanator15 wants to merge 1 commit into
Conversation
The MCOS cell (class FileWrapper__) holds the property values of every object in the
file, and matopen read all of them. A read usually needs only the objects its
variables reach, and acquisition code often saves many unreferenced snapshots, so a
LazyCell now reads an element the first time it is indexed, and load_subsys! keeps
views into it rather than copies. Opens for writing ("r+") still read it in full,
since saving objects appends to it.
On a 1.6 GB acquisition file (7,940 objects), reading its one variable touches 718 of
11,764 values, with identical results. New test: nothing is read at matopen, and
reading one variable reads only part of the cell.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
matopenreads the whole#subsystem#/MCOScell (classFileWrapper__), which holds the property values of every object in the file, before any variable is asked for. A read usually needs only the objects its variables reach, and acquisition code often saves many unreferenced snapshots, so this reads the cell on demand:LazyCell, anAbstractArray{Any}over the cell's references, reads an element (and keeps it) the first time it is indexed;load_subsys!keeps views into it (@view mcos_data[3:(end - 1), 1], …) rather than copies, which would read everything;"r+") still read it in full, since saving objects appends toprop_vals_saved;prop_vals_savedbecomesAbstractVector{Any}to hold the view; v7 (MAT_v5) files, whose subsystem is one decompressed blob, are unchanged.On a 1.6 GB acquisition file (7,940 MATLAB objects: 2,058 timetables, 5,770 datetimes, 30 of the acquisition class), reading its one variable now touches 718 of 11,764 property values, and a digest of everything
matreadreturns is identical tomaster's. New test: aftermatopennothing in the cell has been read, and reading one variable ofuser_defined_classdefs.matreads only part of it;"r+"still reads it all. Full suite passes (586).To be plain about the benefit: on those files I measured no speed difference, warm or cold (
matopen0.62 s onmastervs 0.64 s with this, first touch of an unread file) — their property values are small, and a cold open is dominated by reading/#refs#metadata off disk, whichever step does it first. It should matter for files whose unreferenced objects hold large property values, but I have no such file to show, so please close it if you'd rather not take it without that. Related: #247 (partial reading).🤖 Generated with Claude Code