jemalloc: upgrade from 3.6.0 to 5.3.1

[kenhys]

In the historical reasons, there are memory consumption issue between 4.x and 5.2.x for fluent-package.

Since 5.3.x, the situation seems changed about that issue, upgrade it from 3.6.0.

There are trade of about memory and performance, adopted already tuned MALLOC_CONF parameters from here:

https://gist.github.com/jjb/9ff0d3f622c8bbe904fe7a82e35152fc

[kenhys]

Recently checked memory usage trends: RSS and PSS

[kenhys]

Another aspects : ref fluent/fluentd#1657

• Executed on single node.
• Recorded sampling RSS during processing out_forward => in_forward => flush to file
• Applied export MALLOC_CONF="dirty_decay_ms:0,muzzy_decay_ms:0,narenas:2,background_thread:true,thp:never,abort_conf:true" for jemalloc 5.3.1-tuned
• Some performance degration was observed for jemalloc-5.3.1-tuned.

[kenhys]

in_tail with 1 GB file (rake benchmark:run:in_tail with fluent-package-builder master)

It seems that total execution time takes a bit longer for jemalloc 5.3.1+MALLOC_CONF.

jemalloc 3.6.0

       user     system      total        real
  11.471232   1.479785  12.951017 ( 13.461987)

jemalloc 5.3.1 vanilla

       user     system      total        real
  11.465409   1.506601  12.972010 ( 12.855354)

jemalloc 5.3.1 + MALLOC_CONF tuning

       user     system      total        real
  12.204533   4.434312  16.638845 ( 15.979180)

[kenhys]

With dirty_decay_ms:0, it behaves similar to 3.6.0, but you must pay
performance penalty.
dirty_decay_ms:1000 might be conservative approach for performance and
memory usage, but it tends to use more memory in contrast to 3.6.0.

[Watson1978]

fluent-package-builder/fluent-package/Rakefile

Lines 703 to 710 in d0be2a0

	if JEMALLOC_VERSION.split('.')[0].to_i >= 4
	if ENV["FLUENT_PACKAGE_STAGING_PATH"] and
	(ENV["FLUENT_PACKAGE_STAGING_PATH"].end_with?("el8.aarch64") or
	ENV["FLUENT_PACKAGE_STAGING_PATH"].end_with?("el7.aarch64"))
	# NOTE: There is a case that PAGE_SIZE detection on
	# CentOS 7 CentOS 8 with aarch64 AWS ARM instance.
	# So, explicitly set PAGE_SIZE by with-lg-page 16 (2^16 = 65536)
	configure_opts.concat(["--with-lg-page=16"])

It looks like --with-lg-page=16 is needed for all aarch64 builds, not just el7/el8.

[Watson1978]

jemalloc 5.3 adds an unnecessary libstdc++ dependency to the package.

Comparing the released 6.0.4 rpm with a 6.1.0 build:

[root@alma9 ~]# rpm -qpR fluent-package-6.0.4-2.el9.x86_64.rpm > 6.0.4.txt
[root@alma9 ~]# rpm -qpR fluent-package-6.1.0-1.el9.x86_64.rpm > 6.1.0.txt
[root@alma9 ~]# diff 6.0.4.txt 6.1.0.txt
12c12
< config(fluent-package) = 6.0.4-2.el9
---
> config(fluent-package) = 6.1.0-1.el9
58a59,62
> libstdc++.so.6()(64bit)
> libstdc++.so.6(CXXABI_1.3)(64bit)
> libstdc++.so.6(GLIBCXX_3.4)(64bit)
> libstdc++.so.6(GLIBCXX_3.4.11)(64bit)

libjemalloc.so.2 is the only object in the package that gains a libstdc++.so.6 NEEDED entry.
jemalloc 5 enables its C++ shim by default (jemalloc 3.6.0 had no C++ source at all), which links the DSO with $(CXX) and pulls in -lstdc++.

The shim only replaces the global operator new / operator delete. Disabling it does not take C++ allocations away from jemalloc —
libstdc++'s default operator new calls malloc(), whices to jemalloc. So this is a dependency we gain fornothing.

It seems we need to add the --disable-cxx option when configuring jemalloc:

    configure_opts = [
      "--prefix=#{install_prefix}",
      "--disable-cxx",
    ]

fluent-package-builder/fluent-package/Rakefile

Lines 699 to 701 in d0be2a0

	configure_opts = [
	"--prefix=#{install_prefix}",
	]

[kenhys]

That situation was fixed:

diff -u /tmp/604.txt /tmp/610.txt
--- /tmp/604.txt        2026-08-07 18:07:28.217364776 +0900
+++ /tmp/610.txt        2026-08-07 18:12:06.502130940 +0900
@@ -4,7 +4,7 @@
 /usr/bin/env
 /usr/bin/getent
 /usr/sbin/adduser
-config(fluent-package) = 6.0.4-1.el8
+config(fluent-package) = 6.1.0-1.el8
 ld-linux-x86-64.so.2()(64bit)
 ld-linux-x86-64.so.2(GLIBC_2.2.5)(64bit)
 ld-linux-x86-64.so.2(GLIBC_2.3)(64bit)

[Watson1978]

Suggested change

	Environment=MALLOC_CONF="dirty_decay_ms:400,muzzy_decay_ms:0,narenas:2,background_thread:true,thp:never,abort_conf:true"
	Environment=MALLOC_CONF="dirty_decay_ms:400,muzzy_decay_ms:0,narenas:2,background_thread:true,thp:never"

Could we drop abort_conf:true from the shipped MALLOC_CONF?

jemalloc's default is to warn and ignore a bad conf entry; abort_conf:true turns it into abort(). And "bad" is broader than a misspelled option — a stray trailing comma is enough:

$ MALLOC_CONF="abort_conf:true,narenas:2," LD_PRELOAD=./libjemalloc.so.2 /bin/echo ok
<jemalloc>: Conf string ends with comma -- narenas:2,
<jemalloc>: Abort (abort_conf:true) on invalid conf value (see above).
exit=134                       # SIGABRT, /bin/echo never runs

That matters here because EnvironmentFile=-/etc/sysconfig/fluentd comes after Environment=MALLOC_CONF=..., so overriding it is the intended path for operators. A small typo there would now kill fluentd during malloc init — before it opens its log file, so nothing appears in fluentd.log. LD_PRELOAD also applies to ExecStop=/bin/kill, so systemctl stop would fail as well.

Dropping it doesn't change any of the other five settings.

[kenhys]

I believe it is better not to allow configuration errors about MALLOC_CONF.
For example, if a typo exists about dirty_decay_ms, it fallback to default.
Thus it changes the behavior about memory usage a lot.

Of course, there is a merit not to abort, but it might be critical in production.

[Watson1978]

Thanks.
OK, I see.