Issue #4193 : Define lifecycle environments in the project configuration

[mattcasters]

Lifecycle environment definitions can be stored in project-config.json and checked in with the project. Each definition has a name, a description, and three lists of variables (optional, mandatory, and secret). The default value is a placeholder, not a live value or a secret.

Adding a project from a folder or from Git asks whether to create those environments on this computer when the project already defines them. Only mandatory variables and secrets are written to one configuration file per environment. The folder comes from ${ENVIRONMENTS_FOLDER} when that project variable is set, and otherwise Hop asks for a folder outside the project. An environment that already exists is left unchanged, and an existing configuration file is not overwritten.

The environment dialog links to an embedded definition by name. Enabling that environment applies the placeholders, then the configuration file. A project variable with the same name still wins.

fixes #4193

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Run mvn clean install apache-rat:check to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
• If you have a group of commits related to the same change, please squash your commits into one and force push your branch using git rebase -i.
• Mention the appropriate issue in your description (for example: addresses #123), if applicable.

To make clear that you license your contribution under the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

• I hereby declare this contribution to be licensed under the Apache License Version 2.0, January 2004
• In any other case, please file an Apache Individual Contributor License Agreement.

[dsanderbi]

Hi @mattcasters ,

thanks for this PR. I tested it locally and there are a few things I'd like to point out / discuss.

My setup

• Project test-project with one embedded environment project-environment (variables DB_NAME, DB_PASS, VAR1, VAR2).
• The configuration file of the lifecycle environment is stored in the project under .local/environments/. .local is in .gitignore, so it is not committed.
• The project has one configuration file per stage, config/dev/project-environment-config.json and config/prod/test-project-prod.json. Both are committed, just like project-config.json.

test-project
├── .gitignore                                # contains ".local"
├── .local                                    # not committed
│   └── environments
│       └── project-environment.json          # local lifecycle environment config (created by Hop)
├── config                                    # could be a git-submodule
│   ├── dev
│   │   └── project-environment-config.json   # committed, stage values for dev
│   └── prod
│       └── test-project-prod.json            # committed, stage values for prod
├── main.hwf
└── project-config.json                       # committed, contains embeddedEnvironments

1. Mandatory/secret defaults end up in the repository and in the local file

When I set the variables with their defaults under Project → Environments, the values are stored in project-config.json and in the local environment file. For a secret this means the value is always committed:

{ "name": "DB_PASS", "defaultValue": "password", "mandatory": true, "secret": true }

This doesn't match the message Hop shows when the environments are created:

I read "Mandatory variables and secrets are written to a configuration file … Other variables stay in the project" as: mandatory variables and secrets move to the local file, and only the other variables stay in the project.

Right now the secret flag doesn't change how the value is handled. It behaves like mandatory: the value is committed, stored in clear text locally, and not masked in the dialog.

Suggestion: for secret variables, keep only the name and description in project-config.json. Ask for the value when the environment is created and store it only in the local file, ideally with the existing Encrypted … mechanism.

2. hop-run doesn't use the defaults from project-config.json

Changes to the defaults under Project → Environments are not copied to an existing local file. I assume that's intended.

What I didn't expect: when I run the project from the command line, the defaults from project-config.json are not used at all. After a deployment there is no .local file. I expected Hop to take the defaults from project-config.json and only replace the values that come from the stage configuration file, here config/prod/test-project-prod.json with DB_NAME and VAR3.

Repro with a fresh client (fresh hop-config.json):

./hop-run.sh \
  --project-locations test-project=/path/to/test-project \
  --environments 'prod=test-project:${PROJECT_HOME}/config/prod/test-project-prod.json' \
  -e prod \
  -f '${PROJECT_HOME}/main.hwf' \
  -r local

Variable	Default in project-config.json	config/prod/…	Result	Expected
DB_NAME	localhost	prod-db-server	prod-db-server ✅	prod-db-server
VAR1	var-1-value	–	${VAR1} ❌	var-1-value
VAR3	–	prod-var-3	prod-var-3 ✅	prod-var-3

For comparison, in Hop GUI:

With hop-run:

The values from the configuration file are applied, but none of the defaults from project-config.json are.

As far as I can see, Project.applyEmbeddedEnvironmentDefaults() only applies the defaults when the lifecycle environment has embeddedEnvironmentName set. That field is only set from the GUI. hop-conf --environment-create and hop-run --environments have no option for it, and an environment with the same name as the embedded definition is not linked automatically either.

Question: Is this intentional? If so, what's the idea behind it? Maybe the intention is that in production every value is set explicitly, through the environment's configuration files or through environment variables, and the embedded definition is only a template for creating environments in the GUI? In that case it would help to mention this in the documentation, because I expected the defaults to be applied everywhere.

If it's not intentional, I'd prefer an option on hop-run to link an environment to an embedded definition. Then a project can be run without any registration in hop-config.json, which is what I want for a deployment (e.g. Docker or CI). Another option would be to link automatically when the environment name matches an embedded definition.

3. Minor: the name "Mandatory"

The dialog explains what happens to mandatory variables (they are written to the local configuration file). Still, when I first looked at the flag, the name was misleading for me. I read Mandatory as "the project needs this variable", not as "has to be set on each computer". A name like Local or Per computer would describe the behavior more directly. Alternatively, secret alone could decide where the value is stored, and mandatory would only check that the value is not empty.

4. Question: development and deployment workflow

More generally, I'd like to understand how you see the workflow from development to deployment. Do you expect the environments to be registered in hop-config.json on the target system (e.g. with hop-conf) as part of every deployment, or should a deployment work with hop-run alone? With the first approach, every deployment needs an extra, often manual, step to create or update the environment. With hop-run alone, a single command is enough to run the project with the right stage configuration.

---

I attached the test-project.zip. Like after a deployment, it doesn't contain the .local folder.

[mattcasters]

I know it's possible to create a lot of text with LLMs, but please understand I have to read all this. I disagree with a number of points raised so all you're doing is stalling this project.

[mattcasters]

We already went through this. Yes you need to do manual steps to enter passwords, configure things. That's the general consensus. Most environments have every data engineer in a team authenticate with their own username and password, for example. Surely that can't be too much of an issue?

When I set the variables with their defaults under Project → Environments, the values are stored in project-config.json and in the local environment file. For a secret this means the value is always committed:

How can it be otherwise? Should the project be changed after an environment is implemented? That doesn't seem to make sense. Obviously it's clear to never store actual secrets in the project configuration nor in git. The notable exception there are passwords stored in a key-vault. I assume that will more and more become the standard.

Suggestion: for secret variables, keep only the name and description in project-config.json. Ask for the value when the environment is created and store it only in the local file, ideally with the existing Encrypted … mechanism.

I considered it useful to have things like Fill in the token you get from x as hints for the person that needs to fill in the details.

I'll look at the other comments when I find the time.

[dsanderbi]

Hi Matt, thank you for your input, you are right, maybe I want too much in this PR.
The implementation solves the feature request, so using this option in the GUI works as requested.

However, to keep things short: I'm wondering if hop-run could get an option like --embedded-environment=project-environment.

[mattcasters]

I'm sorry if I came across a bit direct. I really like the direction this is going in. It will be quite useful. I will give this the time it deserves. It's just that I have a number of other priorities right now and time often feels limited.