Lookup replication issue with DA-ESS-ContentUpdate on distributed search

[Masoud00013]

Hi,

I am facing an issue with one of the lookups provided by DA-ESS-ContentUpdate in a distributed Splunk Enterprise Security environment.

Environment
Splunk Enterprise: 10.4
Splunk Enterprise Security: 8.x
Search Head Cluster: 3 members
Multiple Indexers
Linux
The lookup is a CSV-based lookup provided by DA-ESS-ContentUpdate

The lookup is available and works correctly on the Search Head, but it is not available on the Search Peers/Indexers when the search is distributed.

Lookup

The lookup definition and CSV are both located under:

/opt/splunk/etc/apps/DA-ESS-ContentUpdate/

The lookup CSV is under:

/opt/splunk/etc/apps/DA-ESS-ContentUpdate/lookups/

and its definition is in the corresponding transforms.conf.

The lookup works correctly when executed locally on the Search Head, but distributed searches fail because the Search Peer does not have the lookup.

Investigation

We checked the effective distsearch.conf configuration using:

/opt/splunk/bin/splunk btool distsearch list replicationDenylist --debug

The following entry was found:

/opt/splunk/etc/apps/DA-ESS-ContentUpdate/default/distsearch.conf
[replicationDenylist]
excludeESCU = apps[/\]DA-ESS-ContentUpdate[/\]lookups[/\]...

This appears to exclude all lookup files under DA-ESS-ContentUpdate/lookups/ from Knowledge Bundle replication.

At the same time, the global configuration contains:

replicate.lookups = true

and the effective replication allowlist contains:

[replicationAllowlist]
other = (system|(apps/(?!pdfserver))|users(/_reserved)?//*)/(bin|lookups)/...

Therefore, lookups are generally allowed to be replicated, but the ESCU-specific denylist appears to explicitly exclude the entire DA-ESS-ContentUpdate/lookups/ directory.

We also checked:

/opt/splunk/bin/splunk btool distsearch list replicationSettings:refineConf --debug

which confirms:

replicate.lookups = true
replicate.transforms = true

However, the replicationDenylist entry above prevents the ESCU lookup from being included in the Knowledge Bundle.

Knowledge Bundle verification

We inspected the generated Knowledge Bundle and confirmed that the relevant lookup is not present in the bundle.

This explains why:

The lookup exists on the Search Head.
The lookup definition exists.
replicate.lookups = true.
The lookup still does not exist on the Search Peer.
Distributed searches cannot use the lookup.
Question

Is this exclusion intentional for ESCU lookups?

If so, what is the recommended/supported way to replicate a specific ESCU lookup to Search Peers while keeping the other ESCU lookup files excluded?

We specifically want to achieve the following:

DA-ESS-ContentUpdate/lookups/

lookup-A.csv                         -> NOT replicated
lookup-B.csv                         -> NOT replicated
...
cisco_secure_firewall_filetype_lookup.csv
                                      -> SHOULD be replicated

We do not want to replicate all ESCU lookups because of the potential Knowledge Bundle size and performance impact.

We also do not want to copy the lookup into a custom App, because doing so would cause the custom copy to become independent from future ESCU Content Updates. If Splunk updates this lookup in a future Content Update, we would want the updated official lookup to continue being used automatically.

Expected behavior

Ideally, we would like to keep the lookup managed by:

DA-ESS-ContentUpdate

so that it continues to receive future Content Updates, while allowing only this specific lookup to be included in Knowledge Bundle replication.

Is there a supported configuration or recommended approach for creating an exception to:

excludeESCU = apps[/\]DA-ESS-ContentUpdate[/\]lookups[/\]...

for a single lookup?

Thanks.

[nasbench]

cc @pyth0n1c @cmcginley-splunk - You might wanna take a look at this.