Turning on kerberos on existing Ozone cluster

[mark-bb]

Hello.

Ozone 2.0.0.

I want to turn on kerberos authentication on a cluster successfully initialized previously w/o kerberos.
I installed, configured and tested kerberos client on all hosts with keytabs & principals specified in the following properties:

core-site.xml:

<property><name>hadoop.security.authentication</name><value>kerberos</value></property>

ozone-site.xml:

<property><name>ozone.security.enabled</name><value>true</value></property>

<property><name>hdds.scm.kerberos.principal</name><value>{{ ozone_scm_kerberos_principal }}</value></property>
<property><name>hdds.scm.kerberos.keytab.file</name><value>{{ ozone_scm_kerberos_keytab }}</value></property>

<property><name>ozone.om.kerberos.principal</name><value>{{ ozone_om_kerberos_principal }}</value></property>
<property><name>ozone.om.kerberos.keytab.file</name><value>{{ ozone_om_kerberos_keytab }}</value></property>

<property><name>ozone.s3g.kerberos.keytab.file</name><value>{{ ozone_s3g_kerberos_keytab }}</value></property>
<property><name>ozone.s3g.kerberos.principal</name><value>{{ ozone_s3g_kerberos_principal }}</value></property>

No other changes were done.
But I get the following error in the SCM log after restart of all cluster components:

2025-09-12 11:52:49,139 [main] INFO org.apache.hadoop.hdds.security.x509.keys.KeyStorage: Reading public key from /ozone/data/metadata/ozone-metadata/scm/sub-ca/keys/public.pem.
2025-09-12 11:52:49,139 [main] ERROR org.apache.hadoop.hdds.security.x509.keys.KeyStorage: Failed to read the public key.
java.nio.file.NoSuchFileException: /ozone/data/metadata/ozone-metadata/scm/sub-ca/keys/public.pem
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)

There is no such a file indeed.

Questions:

1. Is it possible to turn on kerberos authentication on non-kerberized Ozone cluster?
2. What needs to be down if so?

[mark-bb]

I was not able to make it work w/o reinitialization.
But after a reinitialization attempt I see the following in the SCM log:

Failed on local exception: java.io.IOException: javax.security.sasl.SaslException:
Bad Kerberos server principal configuration [Caused by java.lang.IllegalArgumentException: 
Server has invalid Kerberos principal: SCM/<FQDN>@REALM, expecting: SCM/<host_short_name>@REALM]

I use the following config:

<property><name>hdds.scm.kerberos.principal</name><value>SCM/_HOST@REALM</value></property>

, the hostname and hostname -f OS commands return <FQDN> on my server, nslookup <ip> & dig -x <ip> +short return expected output: <FQDN>..
I'm curious why Ozone reports as invalid my SPN with a fully qualified domain name of my server and requires me to use its short host name instead.

What's the reason of such an error?
Is it possible to make it work with some Ozone configuration changes?

[mark-bb]

Seems, that "_HOST" is not resolved properly on my debian 12. Found similar complaints on Ubuntu.
I use OpenJDK 17.
But when I started to use exact SPNs instead, I get the following in the SCM log:

2025-09-15 19:02:04,904 [97ce0d65-557a-4501-b313-3d18afe4a9d1@group-6AF01C4F40EA-StateMachineUpdater] INFO org.apache.hadoop.hdds.scm.safemode.SCMSafeModeManager: HealthyPipelineSafeModeRule rule is successfully validated
2025-09-15 19:02:04,904 [97ce0d65-557a-4501-b313-3d18afe4a9d1@group-6AF01C4F40EA-StateMachineUpdater] INFO org.apache.hadoop.hdds.scm.safemode.SCMSafeModeManager: ContainerSafeModeRule rule is successfully validated
2025-09-15 19:02:04,904 [97ce0d65-557a-4501-b313-3d18afe4a9d1@group-6AF01C4F40EA-StateMachineUpdater] INFO org.apache.hadoop.hdds.scm.safemode.SCMSafeModeManager: AtleastOneDatanodeReportedRule rule is successfully validated

2025-09-15 19:02:07,412 [Socket Reader #1 for port 9961] WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for a.b.c.d:38599:null (GSS initiate failed) with true cause: (GSS initiate failed)
2025-09-15 19:02:07,412 [97ce0d65-557a-4501-b313-3d18afe4a9d1-scm/sub-ca-refreshCACertificates] WARN org.apache.hadoop.security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1757952118276
2025-09-15 19:02:08,046 [Socket Reader #1 for port 9961] WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for a.b.c.d:32941:null (GSS initiate failed) with true cause: (GSS initiate failed)
2025-09-15 19:02:08,047 [97ce0d65-557a-4501-b313-3d18afe4a9d1-scm/sub-ca-refreshCACertificates] WARN org.apache.hadoop.security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1757952118276
2025-09-15 19:02:11,081 [Socket Reader #1 for port 9961] WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for a.b.c.d:44021:null (GSS initiate failed) with true cause: (GSS initiate failed)
2025-09-15 19:02:11,081 [97ce0d65-557a-4501-b313-3d18afe4a9d1-scm/sub-ca-refreshCACertificates] WARN org.apache.hadoop.security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1757952118276
2025-09-15 19:02:12,013 [Socket Reader #1 for port 9961] WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for a.b.c.d:44691:null (GSS initiate failed) with true cause: (GSS initiate failed)
2025-09-15 19:02:12,014 [97ce0d65-557a-4501-b313-3d18afe4a9d1-scm/sub-ca-refreshCACertificates] WARN org.apache.hadoop.ipc.Client: Couldn't setup connection for SCM/<FQDN>@<REALM> to <FQDN>/a.b.c.d:9961
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:383)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:578)
        at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:364)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:799)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:795)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795)
        at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:364)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1649)
        at org.apache.hadoop.ipc.Client.call(Client.java:1473)
        at org.apache.hadoop.ipc.Client.call(Client.java:1426)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:250)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:132)
        at jdk.proxy2/jdk.proxy2.$Proxy25.submitRequest(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:437)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:170)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:162)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:100)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:366)
        at jdk.proxy2/jdk.proxy2.$Proxy25.submitRequest(Unknown Source)
        at org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB.submitRequest(SCMSecurityProtocolClientSideTranslatorPB.java:93)
        at org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB.getAllRootCaCertificates(SCMSecurityProtocolClientSideTranslatorPB.java:381)
        at org.apache.hadoop.hdds.security.x509.certificate.client.SCMCertificateClient$RefreshCACertificates.run(SCMCertificateClient.java:238)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

No errors in the out file (export KRB5_TRACE=/dev/stdout) - just info on working with tickets, creating InitSecContextToken, etc.
MSAD security log contains info on successful operations with tickets as well.
Nothing additional is printed in the SCM log & out files on:

export HADOOP_OPTS="-Dsun.security.krb5.debug=true"
export OZONE_OPTS="-Dsun.security.krb5.debug=true"
export HADOOP_JAAS_DEBUG=true

The following command returns 0 with no errors on this scm primordial node as user running ozone scm & ozone --daemon commands:

hadoop kdiag \
--secure \
--keytab /etc/security/keytabs/SCM.keytab --principal SCM/<FQDN>@<REALM> \
--out ${out?}

Kerberized SSH on all hosts works.

The situation is nearly similar to this one:
#5012

So, the question is: how to debug this strange "GSS initiate failed"?

[mark-bb]

With log4j.logger.org.apache.hadoop.security=DEBUG in ${OZONE_CONF_DIR}/log4j.properties:

2025-09-17 17:00:26,180 [main] INFO org.apache.hadoop.hdds.server.http.BaseHttpServer: HTTP server of scm listening at http://0.0.0.0:9876
2025-09-17 17:00:26,201 [8b303da3-5768-41af-bf6a-ca5099f05d15-scm/sub-ca-refreshCACertificates] DEBUG org.apache.hadoop.security.UserGroupInformation: PrivilegedAction [as: SCM/<FQDN>@<REALM> (auth:KERBEROS)][action: org.apache.hadoop.ipc.Client$Connection$2@6d619171]
java.lang.Exception
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1950)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795)
        at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:364)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1649)
        at org.apache.hadoop.ipc.Client.call(Client.java:1473)
        at org.apache.hadoop.ipc.Client.call(Client.java:1426)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:250)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:132)
        at jdk.proxy2/jdk.proxy2.$Proxy25.submitRequest(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:437)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:170)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:162)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:100)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:366)
        at jdk.proxy2/jdk.proxy2.$Proxy25.submitRequest(Unknown Source)
        at org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB.submitRequest(SCMSecurityProtocolClientSideTranslatorPB.java:93)
        at org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB.getAllRootCaCertificates(SCMSecurityProtocolClientSideTranslatorPB.java:381)
        at org.apache.hadoop.hdds.security.x509.certificate.client.SCMCertificateClient$RefreshCACertificates.run(SCMCertificateClient.java:238)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
2025-09-17 17:00:26,242 [8b303da3-5768-41af-bf6a-ca5099f05d15-scm/sub-ca-refreshCACertificates] DEBUG org.apache.hadoop.security.SaslRpcClient: Sending sasl message state: NEGOTIATE

...


2025-09-17 17:00:27,163 [Socket Reader #1 for port 9961] DEBUG org.apache.hadoop.security.UserGroupInformation: PrivilegedAction [as: SCM/<FQDN>@<REALM> (auth:KERBEROS)][action: org.apache.hadoop.security.SaslRpcServer$1@24328b34]
java.lang.Exception
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1950)
        at org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:153)
        at org.apache.hadoop.ipc.Server$Connection.createSaslServer(Server.java:2625)
        at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:2372)
        at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:2252)
        at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:2194)
        at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:3004)
        at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2802)
        at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2555)
        at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1650)
        at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1505)
        at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1476)
2025-09-17 17:00:27,164 [Socket Reader #1 for port 9961] DEBUG org.apache.hadoop.security.SaslRpcServer: Created SASL server with mechanism = GSSAPI
2025-09-17 17:00:27,164 [Socket Reader #1 for port 9961] WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for a.b.c.d:35991:null (GSS initiate failed) with true cause: (GSS initiate failed)
2025-09-17 17:00:27,164 [8b303da3-5768-41af-bf6a-ca5099f05d15-scm/sub-ca-refreshCACertificates] DEBUG org.apache.hadoop.security.UserGroupInformation: PrivilegedActionException as: SCM/<FQDN>@<REALM> (auth:KERBEROS)
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:383)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:578)
        at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:364)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:799)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:795)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795)
        at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:364)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1649)
        at org.apache.hadoop.ipc.Client.call(Client.java:1473)
        at org.apache.hadoop.ipc.Client.call(Client.java:1426)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:250)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:132)
        at jdk.proxy2/jdk.proxy2.$Proxy25.submitRequest(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:437)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:170)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:162)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:100)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:366)
        at jdk.proxy2/jdk.proxy2.$Proxy25.submitRequest(Unknown Source)
        at org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB.submitRequest(SCMSecurityProtocolClientSideTranslatorPB.java:93)
        at org.apache.hadoop.hdds.protocolPB.SCMSecurityProtocolClientSideTranslatorPB.getAllRootCaCertificates(SCMSecurityProtocolClientSideTranslatorPB.java:381)
...

[greenwich]

@mark-bb, I faced the same problem with the certificates earlier.
The following trick made me upgrade from a non-Kerberos to a Kerberos cluster:

1. Shut down the whole thing, update the configs to enable Kerberos and have the correct keytabs in place.
2. Start primordial SCM first -> it generates all required CA and sub-CA certs in its metadata folders.
3. Copy them to the corresponding places for non-primordial SCMs
4. Start other SCMs as usual.

[jojochuang]

opened a doc jira to include this solution: https://issues.apache.org/jira/browse/HDDS-14489

[greenwich]

Yeah, unfortunately, I don't 100% remember all the steps applied. There were a few important tricks.
Probably, I need to test it again.