HA Ozone Deployment on Kubernetes does not work with bootstrap flag for SCM #5019

dantalian-pv · 2023-07-03T13:00:29Z

dantalian-pv
Jul 3, 2023

I am trying to make a HA deployment of Apache Ozone version 1.3.0 on Kubernetes cluster.
For that I've written a Helm chart (see the attachment below) and was following instructions on https://ozone.apache.org/docs/1.3.0/feature/om-ha.html and https://ozone.apache.org/docs/1.3.0/feature/scm-ha.html

OM looks like is working in HA mode, but not SCM.
The issue is that scm-0 when seeing OZONE-SITE.XML_ozone.scm.primordial.node.id = scm-0 stops working with output SCM bootstrap command can only be executed in non-Primordial SCM ozz-ozone-scm-0, self id ozz-ozone-scm-0 Ignoring it, however in the documentation is stated that:

This can be changed with using ozone.scm.primordial.node.id. You can define the primordial node. After setting this node, you should execute both scm --init and scm --bootstrap on all nodes.
Based on the ozone.scm.primordial.node.id, the init process will be ignored on the second/third nodes and bootstrap process will be ignored on all nodes except the primordial one.

From that point the documentation contradicts the log output.

Am I missing something in my configuration? Is there anybody who made HA on Kubernetes for Ozone successfully?

Helm chart:
ozone-010.zip, please, rename the file after downloading to ozone-0.1.0.tgz, github just was blocking file upload with original name.
Values for Helm chart:
ozone-ha.zip

Command:

helm install ozz ./ozone-0.1.0.tgz --values ./values/ozone-ha.yaml

Log output from scm-0:
scm-0.log

Just in case all logs for om and scm pods:
ozone-ha-logs.zip

P.S. Surprisingly, docker-compose example does not have --init or --bootstrap flags: https://github.com/apache/ozone/blob/master/hadoop-ozone/dist/src/main/compose/ozone-ha/docker-compose.yaml . Is that a misconfiguration, or I am missing some points here? Because this file does not follow the documentation.

Answered by adoroszlai

Jul 3, 2023

@dantalian-pv

Sorry if the instructions in the doc are a bit confusing. Based on your Helm chart I think the doc should be clarified to indicate the following SCM HA startup order for all 3 nodes:

ozone scm --init
ozone scm --bootstrap
ozone scm

One of --init and --bootstrap will succeed for each SCM, depending on node ID and primordial node ID. These are just one-off commands that initialize metadata storage. ozone scm starts the actual server process, it should be run for all three nodes.

In Kubernetes, ozone scm --bootstrap should be executed in an initContainer, similarly to ozone scm --init.

Hope that helps.

Ozone has a small set of sample Kubernetes environments where basic tes…

View full answer

adoroszlai · 2023-07-03T16:41:58Z

adoroszlai
Jul 3, 2023
Collaborator

@dantalian-pv

Sorry if the instructions in the doc are a bit confusing. Based on your Helm chart I think the doc should be clarified to indicate the following SCM HA startup order for all 3 nodes:

ozone scm --init
ozone scm --bootstrap
ozone scm

One of --init and --bootstrap will succeed for each SCM, depending on node ID and primordial node ID. These are just one-off commands that initialize metadata storage. ozone scm starts the actual server process, it should be run for all three nodes.

In Kubernetes, ozone scm --bootstrap should be executed in an initContainer, similarly to ozone scm --init.

Hope that helps.

Ozone has a small set of sample Kubernetes environments where basic tests are run for each commit. Currently none of them have SCM HA, and so far we have not been able to complete enabling SCM HA in the ozone-ha environment (HDDS-5471. Enable SCM HA in ozone-ha Kubernetes example #2447). Our problem is that after successful --bootstrap, the non-primordial SCM nodes cannot start due to uninitialized storage. This is just a heads-up, I hope you won't run into the same problem.
HA docker-compose examples set the following environment variables:

ozone/hadoop-ozone/dist/src/main/compose/ozone-ha/docker-compose.yaml

Line 79 in dd25740

ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION

ozone/hadoop-ozone/dist/src/main/compose/ozone-ha/docker-compose.yaml

Line 89 in dd25740

ENSURE_SCM_BOOTSTRAPPED: /data/metadata/scm/current/VERSION

The entrypoint script looks for these and runs the corresponding command:

ozone/hadoop-ozone/dist/src/main/dockerlibexec/entrypoint.sh

Lines 113 to 123 in dd25740

    
           if [ -n "$ENSURE_SCM_INITIALIZED" ]; then 
        
             if [ ! -f "$ENSURE_SCM_INITIALIZED" ]; then 
        
               # Improve om and scm start up options 
        
               /opt/hadoop/bin/ozone scm --init || /opt/hadoop/bin/ozone scm -init 
        
             fi 
        
           elif [ -n "$ENSURE_SCM_BOOTSTRAPPED" ]; then 
        
             if [ ! -f "$ENSURE_SCM_BOOTSTRAPPED" ]; then 
        
               # Improve om and scm start up options 
        
               /opt/hadoop/bin/ozone scm --bootstrap 
        
             fi 
        
           fi

It could be safely adapted to the same model recommended for Kubernetes, I've just tried it now locally.

1 reply

adoroszlai Jul 4, 2023
Collaborator

@dantalian-pv

2. Our problem is that after successful --bootstrap, the non-primordial SCM nodes cannot start due to uninitialized storage.

This is now fixed, turned out to be specific to the tool we use instead of Helm.

See complete example at https://github.com/apache/ozone/tree/0c6a2e7644a062d9106d061786b4cd2ee8c01717/hadoop-ozone/dist/src/main/k8s/examples/ozone-ha

dantalian-pv · 2023-07-04T14:49:53Z

dantalian-pv
Jul 4, 2023
Author

Hello @adoroszlai ,
thank you very much for the fast and detailed reply.

I have tried to follow your offer having ozone scm --init and ozone scm --bootstrap as initContainers. The problem is here, that on non-primordial nodes (which are scm-1 and scm-2 in my case), the command with --bootstrap is not shut down, and Kubernetes waits forever for this initContainer. That means the chain ozone scm --init, ozone scm --bootstrap, ozone scm should be executed only on primordial node, and without the last command on others. I have circumvented it by having initContaner only with --init and in runtime container the next configuration:

          command:
            - bash
            - -c
          args:
            - if [[ "$OZONE-SITE.XML_ozone.scm.primordial.node.id" == "$OZONE-SITE.XML_ozone.scm.node.id" ]]; then env; ozone scm --bootstrap && ozone scm; else env; ozone scm --bootstrap; fi

But possibly I am doing something wrong here, because surprisingly it tries to connect to localhost:

com.google.protobuf.ServiceException: java.net.ConnectException: Call From ozz-ozone-scm-0/10.42.7.33 to localhost:9863 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused, while invoking $Proxy14.send over nodeId=scmNodeId,nodeAddress=localhost/127.0.0.1:9863 after 1 failover attempts. Trying to failover after sleeping for 2000ms.

Or maybe this issue is related to the ticket, you have mentioned before. Could you confirm that? And if so, that means this issue still exists in version 1.3.0?

P.S. I also enclose logs from all containers and the statefulset configuration together with configmap in one archive, if it can help:
ozone-logs-and-config.zip

Thank you in advance.

3 replies

adoroszlai Jul 5, 2023
Collaborator

Ozone reads configuration from ozone-site.xml. The apache/ozone image's entrypoint creates the config file based on environment variables named OZONE-SITE.XML_... (e.g. OZONE-SITE.XML_ozone.scm.service.ids=ozz-ozone-scm).

I think this pre-defined entrypoint may not get run if you explicitly specify the command for the container:

      containers:
      - args:
        - if [[ "$OZONE-SITE.XML_ozone.scm.primordial.node.id" == "$OZONE-SITE.XML_ozone.scm.node.id"
          ]]; then env; ozone scm --bootstrap && ozone scm; else env; ozone scm --bootstrap;
          fi
        command:
        - bash
        - -c

Hence Ozone does not see the intended configuration, even with correct environment variable definitions.

This may explain why SCM finds HA service in --init, but not in --bootstrap.

STARTUP_MSG: Starting StorageContainerManager
STARTUP_MSG:   host = ozz-ozone-scm-0/10.42.7.33
STARTUP_MSG:   args = [--init]
...
2023-07-04 14:34:40 INFO  SCMHANodeDetails:307 - Found matching SCM address with SCMServiceId: ozz-ozone-scm, SCMNodeId: ozz-ozone-scm-0, RPC Address: ozz-ozone-scm-0.ozz-ozone-scm.test-namespace.svc.cluster.local:9894 and Ratis port: 9894

vs.

STARTUP_MSG: Starting StorageContainerManager
STARTUP_MSG:   host = ozz-ozone-scm-0/10.42.7.33
STARTUP_MSG:   args = [--bootstrap]
...
2023-07-04 14:34:44 INFO  SCMBlockLocationFailoverProxyProvider:114 - Created block location fail-over proxy with 1 nodes: [nodeId=scmNodeId,nodeAddress=localhost/127.0.0.1:9863]

(Here scmNodeId is a "dummy" node ID, used when HA config is not defined.)

I think that's the reason for --bootstrap trying to connect to localhost.

I have tried to follow your offer having ozone scm --init and ozone scm --bootstrap as initContainers. The problem is here, that on non-primordial nodes (which are scm-1 and scm-2 in my case), the command with --bootstrap is not shut down, and Kubernetes waits forever for this initContainer.

Can you please share the definitions and logs for this attempt? I think it's a step in the right direction and we should investigate it further.

dantalian-pv Jul 6, 2023
Author

Hello again @adoroszlai,
I have tried the configuration with 2 initContainers and 1 runtime container again.
And, surprisingly, it is working now.
This time I've deleted exiting PVCs before creating a deployment, so the deployment was created from scratch.

With that I enclose the logs, the helm chart and the generated statefulset and the configmap.
Please, confirm that HA was initialized correctly for scm, so I can mark this discussion as solved afterwards.

csm-logs-config_attempt-3.zip

adoroszlai Jul 6, 2023
Collaborator

it is working now.

Glad to hear that.

confirm that HA was initialized correctly for scm

Confirmed. Some relevant log lines:

scm-0:
2023-07-06 11:23:29 INFO  SCMRatisServerImpl:309 - Successfully added new SCM: 7ee4884e-f8bd-462f-b3be-1333bc2da668.
2023-07-06 11:23:48 INFO  SCMRatisServerImpl:309 - Successfully added new SCM: 6e3638c1-64a3-4017-bc72-63ab7f97f3e0.
2023-07-06 11:23:19 INFO  RaftServer$Division:321 - ee12f12c-5669-4698-a1cd-8146b3b37013@group-9601E0FFCD5D: changes role from CANDIDATE to LEADER at term 2 for changeToLeader
2023-07-06 11:23:19 INFO  SCMStateMachine:259 - current SCM becomes leader of term 2.

scm-1:
2023-07-06 11:23:29 INFO  SCMHAManagerImpl:117 - Successfully added SCM ozz-ozone-scm-1 to group group-9601E0FFCD5D:[ee12f12c-5669-4698-a1cd-8146b3b37013|rpc:ozz-ozone-scm-0.ozz-ozone-scm.test-namespace.svc.cluster.local:9894|admin:|client:|dataStream:|priority:0|startupRole:FOLLOWER, 7ee4884e-f8bd-462f-b3be-1333bc2da668|rpc:ozz-ozone-scm-1.ozz-ozone-scm.test-namespace.svc.cluster.local:9894|admin:|client:|dataStream:|priority:0|startupRole:FOLLOWER]

scm-2:
2023-07-06 11:23:48 INFO  SCMHAManagerImpl:117 - Successfully added SCM ozz-ozone-scm-2 to group group-9601E0FFCD5D:[ee12f12c-5669-4698-a1cd-8146b3b37013|rpc:ozz-ozone-scm-0.ozz-ozone-scm.test-namespace.svc.cluster.local:9894|admin:|client:|dataStream:|priority:0|startupRole:FOLLOWER, 6e3638c1-64a3-4017-bc72-63ab7f97f3e0|rpc:ozz-ozone-scm-2.ozz-ozone-scm.test-namespace.svc.cluster.local:9894|admin:|client:|dataStream:|priority:0|startupRole:FOLLOWER, 7ee4884e-f8bd-462f-b3be-1333bc2da668|rpc:ozz-ozone-scm-1.ozz-ozone-scm.test-namespace.svc.cluster.local:9894|admin:|client:|dataStream:|priority:0|startupRole:FOLLOWER]

dantalian-pv · 2023-07-06T12:05:21Z

dantalian-pv
Jul 6, 2023
Author

BTW, is there any documentation, where all configuration parameters are present with some description, similar like it's done for Apache Flink (See: https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/config/)?

I am asking, because I have seen OZONE-SITE.XML_ozone.replication somewhere, but do not understand the meaning.

1 reply

adoroszlai Jul 6, 2023
Collaborator

Each component (SCM, OM, etc.) has a web server which lists all its configurations with description. I realize this works only for current users. We should provide similar list at https://ozone.apache.org/docs

dantalian-pv · 2023-07-07T08:36:34Z

dantalian-pv
Jul 7, 2023
Author

To sum it up, I'll put here the Kubernetes configuration for SCM deployment as a hint for others.
kubernetes.zip

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HA Ozone Deployment on Kubernetes does not work with bootstrap flag for SCM #5019

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 5 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

HA Ozone Deployment on Kubernetes does not work with bootstrap flag for SCM #5019

dantalian-pv Jul 3, 2023

Replies: 4 comments · 5 replies

adoroszlai Jul 3, 2023 Collaborator

adoroszlai Jul 4, 2023 Collaborator

dantalian-pv Jul 4, 2023 Author

adoroszlai Jul 5, 2023 Collaborator

dantalian-pv Jul 6, 2023 Author

adoroszlai Jul 6, 2023 Collaborator

dantalian-pv Jul 6, 2023 Author

adoroszlai Jul 6, 2023 Collaborator

dantalian-pv Jul 7, 2023 Author

dantalian-pv
Jul 3, 2023

Replies: 4 comments 5 replies

adoroszlai
Jul 3, 2023
Collaborator

adoroszlai Jul 4, 2023
Collaborator

dantalian-pv
Jul 4, 2023
Author

adoroszlai Jul 5, 2023
Collaborator

dantalian-pv Jul 6, 2023
Author

adoroszlai Jul 6, 2023
Collaborator

dantalian-pv
Jul 6, 2023
Author

adoroszlai Jul 6, 2023
Collaborator

dantalian-pv
Jul 7, 2023
Author