How to support Matplotlib's gigantic API?

[nvaytet]

Supporting all of Matplotlib will never happen, it is too large.

But it would be good to find a way where we could support a lot of it without duplicating everything.
It would also be good if it could be done in a way (either patch mpl functions or use mpl under-the-hood) that would mean updates to the mpl API would automatically propagate here.

The way https://github.com/scipp/mplcanvas did this was by calling all methods on underlying mpl axes, and rendering with draw calls to a canvas.

In Matplotgl, we decided to clone the artists into their own implementation, because it meant that updating one line does not require re-drawing all the other lines on the plot, leading to significant performance gains.

[nvaytet]

One thing to think about is whether we would like matplotgl to exactly match the mpl API, or should it be a mpl-like library.

Having it be a mpl-like library could allow us to make useful additions, such as adding a set_xdata and set_ydata on the object returned by ax.scatter() to update the positions of the dots, instead of having to call set_offsets which has a somewhat obscure/unintuitive name and is also error-prone as the shape of the array that has to be passed can be a source of errors/annoyance.

[asteppke]

Thanks Neil for this project to make interactive plotting of much more convenient!

From a humble user perspective it would be advantageous if matplotgl would exactly match the mpl API. There are just so vast resources out there and if plots fail to run under matplotlib that would be a source of pain.

I can see that for some, very limited areas, such as animations or live changes to datasets matplotlib was not really designed for. Here extending the API slightly would be a possibility but please remain as compatible as possible. Ideally this would also include some slow 'fallback' where matplotgl is forwarding responsibility to matplotlib as you mentioned for more obscure features and accelerates or improves upon the 10% of functions that are commonly used.

Maybe some inspiration can also be obtained from https://github.com/mpld3/mpld3, which does not seem to be showing much activity anymore.

[nvaytet]

Thanks, I think all of this makes sense. I guess by suggesting having easier ways to update some plots, I was saying we could have additional methods, but still keep all the original MPL methods as well.

I agree that I only see this project being successful if it works on at least 90% of all the examples/code that people have.

The difficulty I see with the fallback mechanism is how do you know when you need to fallback, and what do you do if you need to fallback half way through, after you've already created your figure?
For example, let say we don't initially support something like a SymLog colormap. I begin by making my axes, adding the scatter points. All this is supported by native matplotgl. Then I create the SymLog normalization. Now matplotgl recognises that it's not supported. Does it switch out the figure for an ipympl one? What if the figure was already displayed before the SymLog was added?

Or maybe I misunderstood?

[nvaytet]

Looking again at mpld3 and mplcanvas, they were basically just doing plain matplotlib and then had a method to render/update the figure.

Maybe we could envisage something similar after all?
Say we have a function that inspects a figure, finds all the artists in the figure and draws them on the scene.
Then, every time something is updated, we patch the set_SOMETHING method with a call that marks the object that was modified as 'out-of-date'.
Then, require a manual draw call on the rendered figure (that should be ok, users are already used to fig.canvas.draw_idle(), which is similar), and then only update the objects that are out-of-date instead of everything.
The out-of-date objects would get new data from the underlying matplotlib artists (e.g. get_xdata()) and update the Matplotgl representation.

We could either just mark an entire artist as 'out-of-date' (simpler), or even store which were the properties that changed (e.g. xdata and color, but not the width) which is more complicated but potentially faster.

The question is: would e.g. updating the positions of scatter points first on the matplotlib object, and then on the pythreejs object make things much slower? We would have to benchmark...

[asteppke]

Yes, I see the issue with the fallback when creating a figure and then only afterwards using the SymLog colormap.

One possibility would then indeed to switch back to a normal ipympl figure and issue a warning that things will be slower from here onwards. If that can be achieved seamless that sounds like a good user experience without the need for a different API.

Regarding the implementation I am not familiar with the matplotlib code base to be able to comment on that. An update of positions, i.e. the underlying data, is at least rare compared to the zoom, pan and rotate (in case of 3d plots) operations. Here the pronounced difference between ipympl and bokeh/plotly in interactivity becomes quite apparent and even a few fps can often not be achieved in ipympl with the full roundtrip.

I would assume that updating the matplotlib object first and then the pythreejs object would be significantly slower but the advantage of keeping the matplotlib API weighs heavily. Otherwise one can consider for example fastplotlib, which already provides a high-speed interface but requires rewriting every plot.

[nvaytet]

I didn't know about fastplotlib, thanks for sharing.

An update of positions, i.e. the underlying data, is at least rare compared to the zoom, pan and rotate (in case of 3d plots) operations

This is not always true. If you are using widgets like a slider to update line or image data (like navigating through time slices), then you are updating very frequently while not using zoom nor pan...

I will have a go with a prototype of what I outlined, and see where that takes us.