Repository navigation
[Python] Type checking support #32609
Description
mypy and static type checking
As of Python3.6, it has been possible to introduce typing information in the code. This became immensely popular in a short period of time. Shortly after, the tool mypy arrived and this has become the industry standard for static type checking inside Python. It is able to check very quickly for invalid types which makes it possible to serve as a pre-commit. It has raised many bugs that I did not see myself and has been a very valuable tool.
Now what does this mean for PyArrow?
When we run mypy on code that uses PyArrow, you will get error message as follows:
some_util_using_pyarrow/hdfs_utils.py:5: error: Skipping analyzing "pyarrow": module is installed, but missing library stubs or py.typed marker
some_util_using_pyarrow/hdfs_utils.py:9: error: Skipping analyzing "pyarrow": module is installed, but missing library stubs or py.typed marker
some_util_using_pyarrow/hdfs_utils.py:11: error: Skipping analyzing "pyarrow.fs": module is installed, but missing library stubs or py.typed marker
More information is available here: https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-library-stubs-or-py-typed-marker
You can solve this in three ways:
-
Ignore the message. This, however, will put all types from PyArrow to
Any, making it unable to find user errors with the PyArrow library -
Create a Python stub file. This is what previously used to be the standard, however, it no longer a popular option. This is because stubs are extra, next to the source code, while you can also inline the code with type hints, which brings me to our third option.
-
Create a
py.typedfile and use inline type hints. This is the most popular option today because it requires no extra files (except for the py.typed file), allows all the type hints to be with the code (like now in the documentation) and not only provides your users but also the developers of the library themselves with type hints (and hinting of issues inside your IDE).My personal opinion already shines through the options, it is 3 as this has shortly become the industry standard since the introduction.
What should we do?
I'd very much like to work on this, however, I don't feel like wasting time. Therefore, I am raising this ticket to see if this had been considered before or if we just didn't get to this yet.
I'd like to open the discussion here:
-
Do you agree with number ARROW-10: Fix mismatch of javadoc names and method parameters #3 as type hints.
-
Should we remove the documentation annotations for the type hints given they will be inside the functions? Or should we keep it and specify it in the code? Which would make it double.
Reporter: Jorrick Sleijster
Note: This issue was originally created as ARROW-17335. Please see the migration documentation for further details.
Activity
asfimport commentedon Aug 8, 2022
Antoine Pitrou / @pitrou:
Much of our code is in Cython. If you put the type annotations inline, any change or addition to typing info will require a recompile.
(also I'm assuming that Cython is compatible with type annotations, but I'm not 100% sure)
asfimport commentedon Aug 8, 2022
Antoine Pitrou / @pitrou:
cc @xhochy @jorisvandenbossche
asfimport commentedon Aug 8, 2022
Jorrick Sleijster:
Hi @pitrou,
Thanks for reaching out. I checked out the code indeed and I have to say, it's super clean and I was overwhelmed.
I have no worked with Cython code bases before but if we just changed the
\*.pyfiles with type annotations, that would that also require a recompilation?Let's take this line for example:
arrow/python/pyarrow/hdfs.py
Line 96 in 9391951
That would become
Would that change require recompilation?
asfimport commentedon Aug 8, 2022
Antoine Pitrou / @pitrou:
No, that wouldn't require recompilation, but some APIs are implemented in Cython.
asfimport commentedon Aug 8, 2022
David Li / @lidavidm:
Arguably also duplicates ARROW-8175 (#24376)
asfimport commentedon Aug 8, 2022
David Li / @lidavidm:
And cython/cython#3818 is relevant
asfimport commentedon Aug 8, 2022
Jorrick Sleijster:
Well it's not really a duplicate of ARROW-8175.
The difference lies in the fact that that ticket is focused perform type checking on the PyArrow code base and ensuring all the types are valid inside the library.
My ticket is about using the PyArrow code base as a library and ensuring we can type check projects that are using PyArrow by using type annotations on functions specified inside the PyArrow codebase.
I think PySpark 3.2.2 was a nice example of having stubs: https://github.com/apache/spark/tree/v3.2.2/python/pyspark
I'm pretty sure they created them manually though (and note: this is a Java bindings and not C, but I don't think that's a lot of difference in terms of stubs).
However, they changed it in their latest version to ditch the pyi files. I think this is because they have a lot more percentage of the code in Python compared to Java.
asfimport commentedon Aug 8, 2022
Jorrick Sleijster:
I suppose we can just give it a start and see how far we get?
asfimport commentedon Aug 8, 2022
Antoine Pitrou / @pitrou:
Yes, we can probably give it a start. Probably start with the most basic utilities, for example in
memory.pxi.asfimport commentedon Aug 9, 2022
Joris Van den Bossche / @jorisvandenbossche:
AFAIK it is not (yet) possible to do inline type annotations in cython code (for type checking purposes, see the links in #6676 as well), so I think that basically means we need to use the stub file approach?
(but I certainly agree it's fine to give this a go with a small subset, see how that looks, and discuss further from there)
asfimport commentedon Aug 9, 2022
Joris Van den Bossche / @jorisvandenbossche:
It's indeed not exactly the same. But in practice, I think both aspects are very much related and we could (should?) do those at the same time. If we start adding type annotations so that pyarrow can used by other projects that are type-checked, it would be good that at the same time we also check that those type annotations we are adding are correct (although, based on my limited experience with this, just running mypy on the code base is always a bit limited I suppose, as it doesn't guarantee the type checks are actually correct? (it only might find some incorrect ones))
asfimport commentedon Aug 9, 2022
Jorrick Sleijster:
I think you make a good point Joris but as you mention, I don't think we can use inline type annotations :'(. Therefore, we'd have to use generated stubs, which we can't use for checking whether the underlying code actually has the right types.
I think we will therefore have to wait (or take action ourselves upstream) until mypy or cython implements decent support for Python stub generation.
Hence, I think it's better to threat them separate for now and start of with stub generation, which can then later be replaced by a better implementation once available.
asfimport commentedon Aug 9, 2022
Joris Van den Bossche / @jorisvandenbossche:
Mypy doesn't use pyi files when eg doing
mypy pyarrow?asfimport commentedon Aug 10, 2022
Jorrick Sleijster:
It does, in fact, the pyi files are the python-interfaces / python stubs as mentioned here: https://mypy.readthedocs.io/en/stable/stubs.html.
As far as I can see from this page, these are only used for external project checking it's types against this project and not for internal project checking.
asfimport commentedon Aug 10, 2022
Uwe Korn / @xhochy:
My initial efforts with regards to typing the code base stopped as the inline type annotations (and their automatic extraction into pyi) is the crucial component here. All the important data structures of pyarrow are implemented in Cython, only a very small fraction of the externally visible API is in plain Python. Thus as long as the referenced Cython issue isn't solved, I don't think it makes sense to progress on the Arrow side.
45 remaining items
Task #0: Add ORC column statistics APIs (#2)
apacheGH-49102: [CI] Add type checking infrastructure and CI workflow…
Apply dependabot dependency updates (#1217)
github-actions commentedon Sep 22, 2026
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.
A5rocks commentedon Oct 1, 2026
Uh, this is not stale!
rok commentedon Oct 1, 2026
@A5rocks indeed! Work incoming in a week or so.
jorisvandenbossche commentedon Oct 9, 2026
Seeing rok#62, some general questions:
python/pyarrow-stubs) Or rather alongside the source?(personally, I am in favor of putting it as close to the actual source as possible, which means inline when possible (most python files) and the stub files alongside the code; but I don't know if this was discussed before or if there are specific reasons for the current setup)
sakgoyal commentedon Oct 9, 2026
along side source is better imho. I think adding adding them inline is a good idea for pure python. it's less messy for everyone and easier to find things compared to having things spread out. It would also ensure things stay synced up.
The bare minimum would be to have fully strict types for everything public facing. Internal would be good too of course, but not required for the initial stage.
Probably a big ask, but making sure the types are supported across all major type checkers would also be greatly appreciated (mypy, pyright/basedpyright, ruff/ty, pyrefly)
rok commentedon Oct 10, 2026
@jorisvandenbossche
I would slightly prefer
python/pyarrow-stubsas we as we already have some machinery for this in place. I find the .pyi files in the regular tree distracting for working on stabs and editors allow ctrl + click navigation through from stubs to source, so this is not a huge issue. I'd suggest we keep them separate for duration of the work and if we so decide move them into the "regular python tree" when we're done.From my experience working on the entire pyarrow-stubs import PRs, I found stubs separate from the code much more readable. The absence of logic makes it really easy to parse and batch edit them. I would poropose that if we do move them inline we do it after we've created them as stubs first. I believe we also discussed this here or somewhere else.
I would too. Not sure what you mean by separate stubs.
This was probably discussed before here or on the discussion.
As proposed above - moving annotations inline should be automatable. Let's open an issue for that specifically and decide on it after we have well type annotated stubs.
@sakgoyal
See my reply above.
Agreed.
Existing PRs porting pyarrow-stubs passed mypy/pyright/ty checks. I would strive for somesort of reasonable set that would keep maintenance reasonable.
A5rocks commentedon Oct 10, 2026
I also agree that in-source type hints would be ideal. I don't know pyarrow's particulars, but I've looked at e.g. pandas stubs and was surprised about the policies there (e.g. removing deprecated functions, having some functions take in the wrong parameter -- e.g. be documented as taking in a date-like object then being typed as taking a
datetime.date, when anythingpd.Timestampwould accept would work). I think having types next to the code in question helps keep them sync with the docs and generally more accurate.I also think that libraries should not change their types to support more typecheckers, assuming they're unambiguously correct -- for a library I maintain, I've simply created upstream issues and linked to them.