This scalability approach still assumes that we have the full edge_index in memory, however, perhaps we can leverage zarr to read/write the embeddings from disk as we are iterating over our batches. This will keep the memory requirement low as we would not need to allocate space for the feature matrix. Simply read by indices needed per batch.
The question, however, is how we will write the nth-layer embeddins back to the
This scalability approach still assumes that we have the full edge_index in memory, however, perhaps we can leverage zarr to read/write the embeddings from disk as we are iterating over our batches. This will keep the memory requirement low as we would not need to allocate space for the feature matrix. Simply read by indices needed per batch.
The question, however, is how we will write the nth-layer embeddins back to the