Cortex Production Deployment using Tanka

[ChinmayMishra]

I did a production deployment for Loki using Tanka and it was smooth. But I am still struggling with Cortex deployment using Tanka into Kubernetes cluster. I am trying to deploy the block Storage using MinIO. Any pointer will be helpful.

[pracucci]

Are you using cortex-jsonnet? Are you getting any specific issue or error? Could you share more details about your issue, please?

[ChinmayMishra]

Thanks for the replay. I tried cortex-jsonnet, but this is missing the compactor and store-gateway services.
I add those services manually, but I cannot make cortex work.
For Write Path:
It seems like distributor are not communicating with ingesters. I can see the push request in distributors logs but after than I don't see it in ingester. There is no errors in the logs.
distributor.cortex-ns/ingester/ring : Showing me all the 3 active ingester
distributor.cortex-ns/distributor/all_user_stats. : Not showing any stats
I can see error in querier:
level=error ts=2021-01-07T15:34:16.0361289Z caller=pool.go:161 msg="error removing stale clients" err="too many failed ingesters"

[pracucci]

I tried cortex-jsonnet, but this is missing the compactor and store-gateway services.

You need to include tsdb.libsonnet as well when setting up a Cortex cluster running with the blocks storage.

[pracucci]

distributor.cortex-ns/ingester/ring : Showing me all the 3 active ingester

Please take a look at each ingester in the table. Is the IP address correct? The distributor needs to be able to directly connect to each ingester's IP.

[ChinmayMishra]

Thanks a lot for the guidance. The issue was with cloud agent deployment. The issue is fixed now.