Skip to content

[Python][Packaging] Strip unnecessary symbols from libarrow.so to reduce wheel package size #40749

Closed
Closed
[Python][Packaging] Strip unnecessary symbols from libarrow.so to reduce wheel package size#40749
Assignees
raulcd
Labels
Component: PackagingComponent: PythonType: enhancement
Milestone
17.0.0

Description

@raulcd
raulcd
opened on Mar 22, 2024

Describe the enhancement requested

There has been some effort in order to reduce pyarrow size and there are some issues opened in order to split pyarrow wheels in order to have pyarrow-core, pyarrow-all, etcetera.

There seems to be the possibility to strip unnecessary symbols via strip --discard-all. The libarrow.so file seems to be reduced from 61MB to 45 MB.

Another example:

-rwxrwxr-x 1 user group  49M Feb 22 18:55 libarrow.so.1600.0.0

$ strip --strip-unneeded libarrow.so.1600.0.0

-rwxrwxr-x 1 user group  33M Mar 12 15:27 libarrow.so.1600.0.0

This issue is to investigate the possibility of using strip and the possible disadvantages.

edited to fix typo

Component(s)

Packaging, Python

Activity

  1. raulcd
    added
    Type: enhancement
    on Mar 22, 2024
  2. github-actions
    added
    Component: Python
    Component: Packaging
    on Mar 22, 2024

  3. kou commented on Mar 25, 2024

    @kou
    kou
    on Mar 25, 2024
    Member

    Can we check whether backtrace on crash is still available with strip --discard-all/--strip-unneeded? (We may need to use gdb for it.)

  4. jorisvandenbossche commented on Apr 9, 2024

    @jorisvandenbossche
    jorisvandenbossche
    on Apr 9, 2024
    Member

    Potentially related issue with some useful content (but specifically for the wheels, so maybe mostly relevant for our cython code?): pypa/cibuildwheel#331

    That issue mentions numpy used --strip-all though multibuild, although nowadays the wheel building infrastructure is different (using cibuildwheel), and I don't directly see any mention of that in the numpy sources.
    Pandas is using -Wl,--strip-all: pandas: https://github.com/pandas-dev/pandas/blob/8da8b54412020a9478d7b6b0fccde9f9bbf3d8ba/pyproject.toml#L151

  5. pitrou commented on Apr 9, 2024

    @pitrou
    pitrou
    on Apr 9, 2024
    Member

    I've tried on the reproducer in #38770 and only --strip-debug preserves the full backtrace including non-public functions:

    [...]
#7  0x00007ffff310b277 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff310b4d8 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#9  0x00007ffff373737b in std::__throw_bad_variant_access(char const*) () from /home/antoine/t/venv-3.10/lib/python3.10/site-packages/pyarrow/libarrow.so.1500
#10 0x00007ffff3737399 in std::__throw_bad_variant_access(bool) () from /home/antoine/t/venv-3.10/lib/python3.10/site-packages/pyarrow/libarrow.so.1500
#11 0x00007ffff37960e7 in arrow::compute::internal::(anonymous namespace)::FilterMetaFunction::ExecuteImpl(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const [clone .cold] ()
   from /home/antoine/t/venv-3.10/lib/python3.10/site-packages/pyarrow/libarrow.so.1500
#12 0x00007ffff42cc3e0 in arrow::compute::MetaFunction::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const () from /home/antoine/t/venv-3.10/lib/python3.10/site-packages/pyarrow/libarrow.so.1500
[...]

    More powerful strip options make the backtrace less useful because the symbols of non-public functions are removed:

    [...]
#7  0x00007ffff310b277 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff310b4d8 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#9  0x00007ffff373737b in ?? () from /home/antoine/t/venv-3.10/lib/python3.10/site-packages/pyarrow/libarrow.so.1500
#10 0x00007ffff3737399 in ?? () from /home/antoine/t/venv-3.10/lib/python3.10/site-packages/pyarrow/libarrow.so.1500
#11 0x00007ffff37960e7 in ?? () from /home/antoine/t/venv-3.10/lib/python3.10/site-packages/pyarrow/libarrow.so.1500
#12 0x00007ffff42cc3e0 in arrow::compute::MetaFunction::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const () from /home/antoine/t/venv-3.10/lib/python3.10/site-packages/pyarrow/libarrow.so.1500
[...]

    This is on the PyArrow 15.0.2 wheels. The savings are still significant: from 61MB (original) to 54MB (stripped) for libarrow.so.

  6. paleolimbot commented on Apr 9, 2024

    @paleolimbot
    paleolimbot
    on Apr 9, 2024
    Member

    For pyarrow, there are also nightly wheel builds as well, correct? Might it also be an option to keep the debug symbols in those (so that there is a route to getting a user-reported stack trace) but strip them from the wheel most users get with a default pip?

  7. pitrou commented on Apr 9, 2024

    @pitrou
    pitrou
    on Apr 9, 2024
    Member

    A user-reported stack trace would be possible with --strip-debug.

  8. raulcd
    added this to the 17.0.0 milestone on May 7, 2024
  9. raulcd
    self-assigned this
    on May 7, 2024

  10. raulcd commented on Jun 7, 2024

    @raulcd
    raulcd
    on Jun 7, 2024
    MemberAuthor

    For macOS on pandas it seems that -g0 was used:

    And the fix

  11. raulcd
    added 2 commits that reference this issue on Jun 10, 2024

    apacheGH-40749: [Python][Packaging] Strip unnecessary symbols when bu…

    de3fa6b

    GH-40749: [Python][Packaging] Strip unnecessary symbols when building…

    baf4089

  12. raulcd commented on Jun 11, 2024

    @raulcd
    raulcd
    on Jun 11, 2024
    MemberAuthor

    Issue resolved by pull request 42028
    #42028

  13. raulcd
    closed this as completedon Jun 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Component: PackagingComponent: PythonType: enhancement

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions