[OCTRL-1081] Wrap kubectl into Mesos executor task#805
[OCTRL-1081] Wrap kubectl into Mesos executor task#805justonedev1 wants to merge 3 commits intomasterfrom
Conversation
user infor setup properly... kubectl passes arguments properly to the kubernetes attempt for fairmq bla
| // aux.Control.Mode = "direct" | ||
| // } | ||
|
|
||
| aux.Control.Mode = c.Control.Mode.String() |
There was a problem hiding this comment.
There is a change in behaviour for hooks though, no? Before they were getting direct instead of hook, which actually smells like a bug, but perhaps something is relying on it?
There was a problem hiding this comment.
This is probably my misunderstanding, as I thought that it is a bug to implicitly change hook to direct.. especially when we have hooktask that is created only if controlmode.HOOK is present.
see
| While prototyping we used many Kubernetes clusters, namely [`kind`](https://kind.sigs.k8s.io/), [`minikube`](https://minikube.sigs.k8s.io/docs/) and [`k3s`](https://k3s.io/) | ||
| in both local and remote cluster deployment. We used Openstack for remote deployment. | ||
| Follow the guides at the individual distributions in order to create the desired cluster setup. | ||
| For now we chose `k3s` for most of the activities performed because it is lightweight |
There was a problem hiding this comment.
That might give an impression that there was some kind of decision process in ALICE or FLP. Perhaps rephrase to:
k3sis recommended to run this prototype, as it is a lightweight and...
| All settings of `k3s` were used as default except one: locked-in-memory size. Use `ulimit -l` to learn | ||
| what is the limit for the current user and `LimitMEMLOCK` inside the k3s systemd service config | ||
| to set it for correct value. Right now the `flp` user has unlimited size (`LimitMEMLOCK=infinity`). | ||
| This config is necessary because even if you are running PODs with the privileged security context |
There was a problem hiding this comment.
| This config is necessary because even if you are running PODs with the privileged security context | |
| This config is necessary because even if you are running Pods with the privileged security context |
It's a detail, but "Pod" and "pod" are likely the correct spellings, since it's a proper English word, not an abbreviation, like "Plain Old Data". I mention this, so that we avoid having someone in ALICE read our documentation and then learn to read it "pee-o-dee".
|
|
||
| * add path to the kubectl manifest as the first argument in `.command.arguments` field | ||
| * change `.control.mode` to either `kubectl_direct` or `kubectl_fairmq` | ||
| You can find working template inside `control-operator/ecs-manifests/control-workflows/*_kube.yaml` |
There was a problem hiding this comment.
| You can find working template inside `control-operator/ecs-manifests/control-workflows/*_kube.yaml` | |
| You can find working template inside `control-operator/ecs-manifests/control-workflows/*-kube.yaml` |
| log.WithFields(logrus.Fields{ | ||
| "controlmode": task.Tci.ControlMode, | ||
| "name": task.ti.Name, | ||
| }).WithError(err).Errorf("kubectl apply failed stderr: %s , stdin: %s", stderrBuf.String(), stdoutBuf.String()) |
There was a problem hiding this comment.
| }).WithError(err).Errorf("kubectl apply failed stderr: %s , stdin: %s", stderrBuf.String(), stdoutBuf.String()) | |
| }).WithError(err).Errorf("kubectl apply failed stderr: %s , stdout: %s", stderrBuf.String(), stdoutBuf.String()) |
| } | ||
| supplementalString := "[" + strings.Join(strIds, ", ") + "]" | ||
|
|
||
| os.Setenv("FLP_SUPPLEMENTAL_GORUPS", supplementalString) |
There was a problem hiding this comment.
| os.Setenv("FLP_SUPPLEMENTAL_GORUPS", supplementalString) | |
| os.Setenv("FLP_SUPPLEMENTAL_GROUPS", supplementalString) |
typo?
| > ⚠️ **Warning** | ||
| > All Kubernetes work done is in a stage of prototype. | ||
|
|
||
| ## Kubernetes Cluster |
There was a problem hiding this comment.
My notes on what was generally missing here (or I just didn't see it):
make installto register the task CRD- make sure executor has access to a
~/.kube/config - kubectl does not react well to stfsender env var
http_proxy="", which is surrounded by additional quotes in its manifest, makingkubectl applyfail. Easy fix -http_proxy=.
There was a problem hiding this comment.
ah, and one has to remember to create an image pull secret for the controller after make deploy, because only then the namespace already exists.
2 participants
The task description is in the documentation
docs/kubernetes_ecs.md