Replace fluentd with fluent-bit in the operator

[hjiawei]

Description

New feature (Calico Enterprise): the LogCollector controller now deploys fluent-bit (calico-fluent-bit in calico-system) in place of fluentd, completing the log-collector migration on the operator side.

• Resource identity migration: namespace tigera-fluentd → calico-system; DaemonSet/ServiceAccount fluentd-node → calico-fluent-bit; TLS secret → calico-fluent-bit-tls; metrics port 9081 → 2020 (fluent-bit's built-in HTTP server).
• Configuration is rendered in fluent-bit's YAML schema into per-OS ConfigMaps (calico-fluent-bit-conf/-windows), subPath-mounted on Linux and directory-mounted on Windows, started with -c. The render loads the Go plugins via plugins_file, defines parsers inline, applies the record-transform Lua filter, and inlines user-provided fluent-bit YAML filter lists. A rendered-config hash annotation rolls the pods on config-only changes.
• Tail inputs use the producing components' real log paths, with SQLite offset DBs and filesystem buffering under /var/log/calico/calico-fluent-bit; a pos-migrator init container (Linux and Windows) seeds offsets from the legacy fluentd .pos files and pre-creates the tailed directories. Windows tails the same log types the fluentd Windows variant shipped.
• The linseed output matches only Linseed-bound tags, authenticates with mTLS + the pod's ServiceAccount token, and retries without limit against bounded filesystem storage. S3/Splunk/Syslog outputs mirror fluentd's per-type fan-out (standard AWS credential env vars, endpoint scheme honored, syslog ships the whole record as JSON via a per-output Lua processor with TLS properly enabled).
• NonClusterHost renders the :9880 HTTP input with client-certificate verification; the input Service is cleaned up when the resource is removed.
• eks-log-forwarder runs the fluent-bit image with a rendered in_eks → linseed pipeline and health probes (no startup init container; the input plugin resolves its resume point from Linseed).
• Probes hit :2020/api/v1/health; the ServiceMonitor scrapes plain HTTP (fluent-bit's monitoring server has no TLS) with access restricted by the component NetworkPolicy, and legacy fluentd monitors are deleted.
• The LogCollector controller no longer creates/owns the calico-system namespace (deleting the LogCollector must not garbage-collect it); the deprecated fluentdDaemonSet override is honored as an alias of the new calicoFluentBitDaemonSet field (with container-name translation); legacy tigera-fluentd resources — the namespace last — are cleaned up idempotently.

Testing: render, controller, and monitor unit suites updated/extended (ConfigMap-content assertions replacing the env-var assertions); the rendered configuration was validated against the real fluent-bit binary; the full migration was validated end-to-end on a test cluster — all log types flowing to Linseed/Elasticsearch, fluentd resources fully removed, tail-offset handover without re-shipping, NonClusterHost ingestion with client-certificate enforcement, and EKS/Windows render shapes verified.

Release Note

The LogCollector now deploys fluent-bit (calico-fluent-bit in calico-system) in place of fluentd, with operator-rendered configuration and automatic migration of fluentd tail positions.

For PR author

• Tests for change.
• If changing pkg/apis/, run make gen-files
• If changing versions, run make gen-versions

For PR reviewers

A note for code reviewers - all pull requests must have the following:

• Milestone set according to targeted release.
• Appropriate labels:
  • kind/bug if this is a bugfix.
  • kind/enhancement if this is a a new feature.
  • enterprise if this PR applies to Calico Enterprise only.