`map_extract` fails to match equal struct keys with differently encoded dictionary nulls

[neilconway]

Describe the bug

map_extract can fail to find a non-null struct key when a nullable dictionary-encoded field uses a different representation of null in the query key.

Both keys have type Struct<d: Dictionary<Int8, Utf8>> and display as {d: NULL}:

• In the stored key, a valid dictionary index points to a null dictionary entry.
• In the query key, the dictionary index itself is null.

DataFusion reports that the structs are equal using both = and IS NOT DISTINCT FROM. It also reports that both d fields are null. Nevertheless, lookup in a single-entry map returns \ [42] for the stored key and [NULL] for the equivalent query key.

To Reproduce

CREATE TABLE dictionary_structs AS
SELECT arrow_cast(
    named_struct('d', CAST(NULL AS VARCHAR)),
    'Dictionary(Int8, Struct("d": Utf8))'
) AS s;

WITH keys AS (
    SELECT named_struct('d', s['d']) AS stored_key,
           named_struct('d', arrow_cast(NULL, 'Dictionary(Int8, Utf8)')) AS query_key
    FROM dictionary_structs
)
SELECT stored_key = query_key AS keys_equal,
       stored_key IS NOT DISTINCT FROM query_key AS keys_not_distinct,
       map_extract(MAP {stored_key: 42}, stored_key) AS self_lookup,
       map_extract(MAP {stored_key: 42}, query_key) AS equivalent_key_lookup
FROM keys;

Actual result:

keys_equal	keys_not_distinct	self_lookup	equivalent_key_lookup
true	true	[42]	[NULL]

Expected behavior

equivalent_key_lookup should return [42], matching self_lookup.

Additional context

No response

[neilconway]

take