Add agent group names and TCP port configuration to enable multi-server training

[stefanfausser]

What this PR does:

• Agent group names can now be set both in 2D / 3D AI controller and sync nodes. This is by default "AGENT" (as it was before)
• TCP override mechanism added for the sync node. When this is enabled then both the default TCP port and the TCP port from command line arguments are overridden.

Now multi-server + multi-agent can be done by having multiple AI controllers (e.g. one for the player and another for a movable platform as done in the MultiAgentSimple example) and one sync node per AI controller. The sync nodes find their corresponding AI-controllers by the agent group names. In addition to this, every sync node must have a distinct TCP port, which can now be overridden.

Then multiple of the Python server examples, e.g. stable_baselines3_example.py, can be started in parallel, each with a different TCP port that match the TCP port of one of the sync nodes (it even could be two different Python server examples or two times the same server example but with different learning rates, n_steps etc.).

How I tested it:

The current version of the MultiAgentSimple example with the current version of godot_rl_agents_plugin + this PR merged, and then a second sync node added, works out-of-the-box for training, Python inference and Onnx inference (with the two Onnx files from the two stable_baselines3_example.py servers).

So for the MultiAgentSimple example, I started stable_baselines3_example.py two times, one with TCP port 11008 (for the player, agent group name AGENT_PLAYER) and the other with TCP port 11009 (for the platform, agent group name AGENT_PLATFORM). For this I had to slightly modify stable_baselines3_example.py by handing over the port argument to StableBaselinesGodotEnv() call.

If this PR is considered to be merged, then I can add a port argument to SB3 Python example to finish this line of feature.

[Ivan-267]

This comment is a discussion on multiagent training including this approach/workflow.
It could also be done separately from this PR if needed (an issue would also work), but this method is a part of the broader topic (multiagent training with GDRL).

I have used this approach for testing previously, and it works (it can also work with the same port so multi-port is not a strict requirement).
There are interesting advantages of this too, you can e.g. use different action repeat rates for each sync node.
However, even though this is a valid workaround, there is one downside to it, and that's an issue with n_parallel > 1 without a more complex implementation.

For now, for multiagent training we have:

SB3

Multi-agent training with shared policy and critic:

This works for simple multiagent training with same obs/action space (some examples envs use it already).

For special cases like training agents with different obs/action spaces, there are some non-optimal workarounds (results not guaranteed):

• You can mask extra observations, and ignore certain actions (I tested something like this with the multiagent example, and it trained successfully, although the test env is made to be very simple to train)
• You can segment actions/observations by agent type (an agent type only uses its own "part" of the observation/action space)

Obviously, both are non optimal, but are potential workarounds for using SB3 (or e.g. CleanRL single policy scripts) directly.

Multi-agent training with multiple servers (this approach):

It allows using multiple policies.

There are those unique advantages as mentioned such as complete flexibility of what script is used on the server side (as you mentioned),
ability to use different action_repeat rates.

For disadvantages:
It has a limit of n_parallel = 1 (or in-editor training). Some envs with PPO benefit from having more agent instances for faster training. This is not an issue for simple, fast training envs.
It enables only independent training, e.g. shared critic approaches won't work with this method.

Rllib:

It supports multiple policies, shared critic (not implemented in the example, but supported),
and multiple processes running.

This approach should also support extending to some other RL libs that support multi-agent training natively.

For disadvantages:
The current implementation has a compromise with episode dones mentioned in the docs (that was made to keep core GDRL edits minimal),
although (not sure without checking), a potential workaround might be using multiple processes and
just one set of AIControllers (one for each policy) in each Godot process.

It hasn't been updated very recently (however version should be limited to the last version that was tested to work fine).

This PR and method specifically:

The main question is:

• Do we integrate this method into the core, or document it as a possible workaround?
  The question is mostly do we consider it one of the supported ways of doing multi-policy training in GDRL,
  or a useful workaround outside the core.

The code in this PR itself is written well and a small change either way, it shouldn't create compatibility issues.
(as a note I haven't fully reviewed it for edge cases, for now I wanted to start a discussion on the method)

The only potential, yet small, UX cost I see is having multiple paths to multi-agent training,
i.e.:

• What agent group and policy name should I set, and how do they interact?
• Should I/when should I set the TCP port. If I leave it blank, will training work?

These are thoughts for brainstorming only, not anything that prevents merging itself,
and docs will help greatly with those.

Your description helps with these already, at best it could be slightly more clear that setting the TCP port is entirely optional, and that setting a port here could break the standard workflow with SB3 script with n_parallel > 1 (which sets the ports automatically and that would be ignored when a custom port is set in Godot), this could be partially solved with an implementation warning not to use it unless you mean to do n_parallel = 1.

Also, I made some assumptions here based on previous experience with my tests without a detailed test of your PR implementation. Feel free to add info/correct anything.

Not directly related to PR:

Another interesting workaround I consider similar in spirit (have tried and thought about, but haven't yet integrated into GDRL main due to lack of
comprehensive testing and potential limitations):

Stepping can be done manually instead of at every action repeat. It's a small tweak/easy to test per env when needed,
and it's one potential way to handle envs that need manual stepping.

One limitation are potential connection timeouts (can be adjusted),
another larger one is with either n_parallel or multiple envs in a single Godot process,
the other envs have to wait for the current one to step (if the delay between steps can be huge, this is a bottleneck).

Still, it's an interesting potential tweak I mention to users and thought I'd share it here as well.
I might consider documenting it if/when I have an example that works well, but it's something that for now I think
doesn't need to be built into GDRL (being a small change). For such tweaks, there might be alternative ways to
describe them in docs/use in examples without necessarily adding them to the core yet (by that I don't necessarily mean this PR is the same case at all).

[stefanfausser]

Thank you, for your very good thoughts @Ivan-267.

Let me elaborate a bit more about this PR.

The main idea is to allow multi-server training by running multiple Python servers independently where each Python server corresponds to a neural network (actor-critic architecture) that could be exported to an ONNX model (when ONNX export is supported) after the training.

This way, even with basic/unchanged SB3-Python-server, it is possible to have multiple agents without sharing the same obs/action space or workarounds (as you said). Later, one (or all) of the multiple agents could be reused in another environment / game via "Onnx inference" training mode.

The Rllib multi-policy mode would still work but I don't think that it is possible this way to have multiple ONNX models (one per policy) as an outcome.

The code in this PR itself is written well and a small change either way, it shouldn't create compatibility issues.
(as a note I haven't fully reviewed it for edge cases, for now I wanted to start a discussion on the method)

This PR should not break compatibility (as you said) but introduces two issues:

For disadvantages:
It has a limit of n_parallel = 1 (or in-editor training). Some envs with PPO benefit from having more agent instances for faster training. This is not an issue for simple, fast training envs.
It enables only independent training, e.g. shared critic approaches won't work with this method.

1. When setting n_parallel > 1 in a Python server, then multiple TCP ports are utilized:

class StableBaselinesGodotEnv(VecEnv):
...
# Create a list of GodotEnv instances
        self.envs = [
            GodotEnv(
                env_path=env_path,
                convert_action_space=True,
                port=port + p,
                seed=seed + p,
                **kwargs,
            )
            for p in range(n_parallel)
        ]

As can be seen in above code (stable_baselines_wrapper.py), with e.g. 3 environments in parallel and the standard TCP port, the following ports are used: 11008, 11009, 11010.

Understanding this pattern, multiple-servers + n_parallel > 1 is actually possible but would require a careful documentation of it and users reading the documentation.

Example with two sync nodes and 3environments in parallel:

Sync node a: 11008, 11009, 11010
Sync node b: 11108, 11109, 11110

2. The Godot env arguments are applied to all sync nodes with port as an exception.

This includes the following arguments: speedup, env_seed, action_repeat.

I think that this is not a large issue as all of this could be pre-set in the nodes and does not require it to be set via arguments. However, this behaviour should be documented.

The only potential, yet small, UX cost I see is having multiple paths to multi-agent training,
i.e.:

• What agent group and policy name should I set, and how do they interact?
• Should I/when should I set the TCP port. If I leave it blank, will training work?

These are thoughts for brainstorming only, not anything that prevents merging itself,
and docs will help greatly with those.

I fully agree on this. The configurable group names and TCP ports would be needed to be documented carefully should this PR be accepted. However, with the defaults I set, everything works as it did before.

[stefanfausser]

Understanding this pattern, multiple-servers + n_parallel > 1 is actually possible but would require a careful documentation of it and users reading the documentation.

Example with two sync nodes and 3environments in parallel:

Sync node a: 11008, 11009, 11010 Sync node b: 11108, 11109, 11110

I have to correct myself. While theoretically the TCP ports could be distributed this way, currently this does not work.

In fact, an exported executable of a Godot environment with two or more sync nodes, cannot be executed error-free via a Python server with the env_path=path-to-executable argument because at least another Python server with a TCP port of the second node would be needed. And this is even for the case of n_parallel = 1.

So when this PR would be accepted then multi-server would only work along with Godot in-editor environments (at least this works fine for me). And exported executables with two or more sync nodes could not be started via a Python server.

Maybe there could be a workaround but this would involve to start multiple Python servers manually where only the first is starting the exported executable of a Godot env and the others are connecting to it (this would involve setting the right TCP ports and n_parallel > 1 without providing an `env_path'). @Ivan-267: What are your thoughts about it / any ideas?

[Ivan-267]

Thanks for the detailed response. This comment is only about Rllib, the other part on SB3 I'm still considering (it takes a bit more time).

The Rllib multi-policy mode would still work but I don't think that it is possible this way to have multiple ONNX models (one per policy) as an outcome.

If you mean the original implementation, Rllib multiagent training can produce an .onnx model for each trained policy.

You'll find two .onnx models in the example, each exported via Rllib multiagent training:
https://github.com/edbeeching/godot_rl_agents_examples/tree/main/examples/MultiAgentSimple/onnx

With script changes, as mentioned previously, you could also utilize shared critics or other more advanced methods for multiagent training.

Another nice feature is that the Rllib PPO implementation allows using hybrid actions (discrete + continuous),
not related to multiagent of course, and not required for the examples.

Note that this was some years ago already. I did set the maximum Rllib version to a tested one, so hopefully it's still functional. I'm aware some restructure of original code is needed for a full update, but if the set version works fine, it should be good enough for now. I may revisit it at some point depending on time and how frequently multi-policy (not just multiagent) training gets used.

[Ivan-267]

Note that we already reached some of the same conclusions, but I was already writing this reply at the same time, I'll keep some parts to keep things chronological.

The main idea is to allow multi-server training by running multiple Python servers independently where each Python server corresponds to a neural network (actor-critic architecture) that could be exported to an ONNX model (when ONNX export is supported) after the training.
This way, even with basic/unchanged SB3-Python-server, it is possible to have multiple agents without sharing the same obs/action space or workarounds (as you said). Later, one (or all) of the multiple agents could be reused in another environment / game via "Onnx inference" training mode.

I understand, I briefly tried using this method before as well (like you, within in-editor, I didn't deeply enough analyze the n_parallel case workarounds as we have the Rllib feature as the main option).

1. When setting n_parallel > 1 in a Python server, then multiple TCP ports are utilized:

class StableBaselinesGodotEnv(VecEnv):
...
# Create a list of GodotEnv instances
        self.envs = [
            GodotEnv(
                env_path=env_path,
                convert_action_space=True,
                port=port + p,
                seed=seed + p,
                **kwargs,
            )
            for p in range(n_parallel)
        ]

As can be seen in above code (stable_baselines_wrapper.py), with e.g. 3 environments in parallel and the standard TCP port, the following ports are used: 11008, 11009, 11010.

Understanding this pattern, multiple-servers + n_parallel > 1 is actually possible but would require a careful documentation of it and users reading the documentation.

Example with two sync nodes and 3environments in parallel:

Sync node a: 11008, 11009, 11010 Sync node b: 11108, 11109, 11110

You are correct it can work, it does need a small modification.
Let's go through a few scenarios below.

Let's take an example of 2 sync nodes (an env with 2 different agent types, like that platform example env).

Scenario 1:

Step 1:
Start 2 SB3 example training scripts, do not include env_path so they wait for the env.

Step 2:
E.g. in Godot editor, launch the env with 2 sync nodes, each using their "group" of AIControllers.

Step 3:
Each sync node should connect to a SB3 training scripts as expected.

It should work, you can train, export to onnx, etc.

Scenario 2:

Step 1:
Start 2 SB3 example training scripts, but use env_path for exported env.

Step 2:
Each SB3 example script will start 1 Godot env. So we have 2 Godot executables launched (4 sync nodes total).

What do we want to happen?
One Godot executable, each sync node linking to its own SB3 training script.

Solution:

• Do not start the executable from the training script (remove env path),
  either launch executable separately or from Godot editor.

Scenario 3:

Step 1:
Start 2 SB3 example training scripts, but use env_path for exported env and n_parallel = 2 (n_parallel requires env_path).

Step 2:
Each SB3 example script will start 2 Godot envs. We have 4 Godot envs instead of 2.

The previous solution won't work as n_parallel requires env_path. However, it's something that can be changed.

It's this check in sb3 wrapper:

        if env_path is None and n_parallel > 1:
            raise ValueError("You must provide the path to a exported game executable if n_parallel > 1")

So, it should work if we modify this check and then launch 2 executables manually.
The check is valuable so I wouldn't directly remove it. If we proceed with this, I would consider an override instead.

UX things left to consider/address

1. env_path for multi-server training won't work (with any n_parallel)

Potential solutions:

• The easiest initial solution is to document the manual process well for a first iteration of this feature.
• Alternative (could also be done later) is to write a custom script that starts SB3 training server(s), then starts the executables.

Note: As this option is flexible (SB3, CleanRL, or anything), users might also want to write their own scripts per use case.

2. setting a port override in sync node will break n_parallel > 1 + env_path for standard single policy training

Potential solutions:

• Make it a bit more hidden/warn users that changing this is for custom use cases only. Maybe a group like @export_group("Optional settings") (not sure about the name yet),
  and a warning in tooltip for that option. Still some design questions remain. Would we put the agent group name there too, and if so, whether we do it also for AIController.

Also, we have a Multi-policy mode options group in AIController that is used for Rllib (with possible extension to other frameworks, but not SB3 with the current implementation),
and as this PR adds agent group name as another method of doing a similar feature, we need to consider the best design solution to keep the UI simple.

It would be simpler if one name is used only for both features, but they work quite differently (Godot group vs policy name that is sent to rllib), so we need to also avoid
confusion (as much as feasible).

3. The Godot env arguments are applied to all sync nodes with port as an exception.

This includes the following arguments: speedup, env_seed, action_repeat.

I think that this is not a large issue as all of this could be pre-set in the nodes and does not require it to be set via arguments. However, this behaviour should be documented.

Yes, I think the most common use case would be to use the same arguments anyway for most of those so it would not be a huge downside.
When we launch the executables manually (env_path won't work for reasons mentioned above), they will only get the arguments pre-set in nodes (so they can be unique per sync node).

Update:
It seems it can work, but with both port being fixed disabled and env_path not being required.

Test implementation:
https://github.com/Ivan-267/godot_rl_agents/blob/test_multiple_python_servers/godot_rl/wrappers/stable_baselines_wrapper.py
(this is just for test, not for the final implementation)

I tried this on the multiagent example with two sync nodes at different ports, your PR, this change above, and from Godot Editor.
I used --timesteps=15_000_000 --port=11010 --n_parallel=2 for one script instance, and 11009 for the other (I also added the port argument to the SB3 example locally).

New Godot has Debug > Customize Run Instances > Enable Multiple Instances > 2. First time I tried this feature, seems quite useful for this test.

Note that we're using the same port twice. But, as the listening socket isn't open the entire time (it closes after each Godot process connects), it doesn't result in an error. We could also consider whether this can cause any other issues more. Not commenting out the port increment makes the TCP override not work with n_parallel 2. The same "agent group" sync node should use 2 ports then in the 2 Godot processes (one for each server). There might be ways to implement this, or the single port solution with TCP override might be OK if it works well.

It would be good to test more to confirm.

We can potentially collaborate on the implementation too if time allows. Let me know your thoughts on these details.

Update 2:
A small addition to the above: If we merge this as a core feature (one alternative would be a documented workaround that doesn't require future maintenance with perhaps a specific branch for the feature), for future updates, we also have another test case to consider (does it break the multi-sync-node functionality). Whether this is at all an issue or not is just an additional design/maintenance decision to consider at this point.

[stefanfausser]

Also, we have a Multi-policy mode options group in AIController that is used for Rllib (with possible extension to other frameworks, but not SB3 with the current implementation), and as this PR adds agent group name as another method of doing a similar feature, we need to consider the best design solution to keep the UI simple.

It would be simpler if one name is used only for both features, but they work quite differently (Godot group vs policy name that is sent to rllib), so we need to also avoid confusion (as much as feasible).

I think I addressed this part by my very recent commit. Please check.

[stefanfausser]

Scenario 3:

Step 1: Start 2 SB3 example training scripts, but use env_path for exported env and n_parallel = 2 (n_parallel requires env_path).

Step 2: Each SB3 example script will start 2 Godot envs. We have 4 Godot envs instead of 2.

Yes, this would not work as you said. However, how about this scenario (I have not yet tested it)?

Scenario 4:

Step 1: Start 1 SB3 example training script with n_parallel = 2 (or larger), port = 11108' and without providing the env_path`

Port 11108 + 11109 could be for the platform sync node.

Step 2: Start 1 SB3 example training script with n_parallel = 2' (must be the same number as above), port = 11008and usingenv_path` for exported envs

This should start two executables
Port 11008 + 11009 could be for the player sync node.

From Godot plugin perspective, this could work when instead of the port argument an port_offset argument is provided / handed over from the Python example. So the TCP ports are kept hard-coded (11108 and 11008) and internally (in both sync nodes), the port_offset could be used by adding the value to the hard-coded TCP ports.

From the Python server SB3 examples perspective, it must be possible to use n_parallel > 1without providingenv_path`.

[stefanfausser]

We can potentially collaborate on the implementation too if time allows. Let me know your thoughts on these details.

I checked your responses and can confirm. I am happy to collaborate with you. Please have a look at my previous comments. Maybe you could work on the SB3 Python example? (Just a suggestion)

[Ivan-267]

Scenario 3:
Step 1: Start 2 SB3 example training scripts, but use env_path for exported env and n_parallel = 2 (n_parallel requires env_path).
Step 2: Each SB3 example script will start 2 Godot envs. We have 4 Godot envs instead of 2.

Yes, this would not work as you said. However, how about this scenario (I have not yet tested it)?

Scenario 4:

Step 1: Start 1 SB3 example training script with n_parallel = 2 (or larger), port = 11108' and without providing the env_path`

Port 11108 + 11109 could be for the platform sync node.

Step 2: Start 1 SB3 example training script with n_parallel = 2' (must be the same number as above), port = 11008and usingenv_path` for exported envs

This should start two executables Port 11008 + 11009 could be for the player sync node.

From Godot plugin perspective, this could work when instead of the port argument an port_offset argument is provided / handed over from the Python example. So the TCP ports are kept hard-coded (11108 and 11008) and internally (in both sync nodes), the port_offset could be used by adding the value to the hard-coded TCP ports.

From the Python server SB3 examples perspective, it must be possible to use n_parallel > 1without providingenv_path`.

There was a scenario that did work in testing, under update:

Update:
It seems it can work, but with both port being fixed disabled and env_path not being required.

Will think about your scenario to compare when I have more time, just wanted to remind it can/did work under at least one setting so far.

We can potentially collaborate on the implementation too if time allows. Let me know your thoughts on these details.

I checked your responses and can confirm. I am happy to collaborate. Please have a look at my previous comments. Maybe > you could work on the SB3 Python example? (Just a suggestion)

Yes, I think that makes sense, I can work on some of the changes on the main repo (we'll see which are needed exactly after working out the details) + add suggestions here as before, as time allows.

[stefanfausser]

Will think about your scenario to compare when I have more time, just wanted to remind it can/did work under at least one setting so far.

I understood. I brought Scenario 4 up because I currently have no idea what problem could occur when TCP ports are reused. With Scenario 4, there would be no port reuse.

[Ivan-267]

Will think about your scenario to compare when I have more time, just wanted to remind it can/did work under at least one setting so far.

I understood. I brought Scenario 4 up because I currently have no idea what problem could occur when TCP ports are reused. With Scenario 4, there would be no port reuse.

I'd have to check the details more, but as is, the issue is with having multiple listening sockets on the same port. What is done with our n_parallel approach with SB3 is sequential. Start listening, connect one env, stop listening, start listening (from another Python Godot env instance), connect another... So two scripts using the same port started in parallel would cause the error.

To be more robust, we'll prefer a multi-port solution.

[stefanfausser]

To be more robust, we'll prefer a multi-port solution.

Agreed. So Scenario 4 (see above) could help. I'll try to test this hypothetical example as soon as possible (maybe this evening, maybe tomorrow)

[Ivan-267]

To be more robust, we'll prefer a multi-port solution.

Agreed. So Scenario 4 (see above) could help. I'll try to test this hypothetical example as soon as possible (maybe this evening, maybe tomorrow)

Yes, I see you've pushed some commits.

P.S. After considering this further, I am going to try locally a slight modification to the plugin approach that builds on the PR.

I am thinking of trying a MultiSync node. I think it could make the UX more seamless and intentional (also we can keep the sync node user interface exactly the same as it was before), but it's just an alternative I'm testing at this point. I'll make it locally and if it seems to work OK and not too much work/big of a change, I'll make a PR for you to consider.

This shouldn't affect your testing, it's just an alternate approach at this point which may or may not be used, we'll see.

Edit: Already I see some things to work out with this approach (e.g. when it comes to inference/human mode settings) and whether it adds more complexity than it removes. For start I'll check only the training part to see if it's useful there.

[stefanfausser]

With the last commit I now added the port_offset argument (to the Godot side). This can be used for the exported executables. When the argument is not provided then port_offset = 0 by default for compatibility reasons.

Now with this PR, the Godot plugin side is prepared for Scenario 4.

From my current understanding, on the Python side the following modifications would need to be done:

• Add port_offset in class GodotEnv (see godot_env.py). Basically port_offset needs to be provided to the called executables
• Add port_offset in class StableBaselinesGodotEnv(VecEnv) (see stable_baselines_wrapper.py). Basically port_offset needs to be provided to GodotEnv()
• The condition for the ValueError (as you said) needs to be changed

[stefanfausser]

Scenario 4 works. In addition to this PR, the only necessary changes must be done in stable_baselines_wrapper.py:

Here:

        # Create a list of GodotEnv instances
        self.envs = [
            GodotEnv(
                env_path=env_path,
                convert_action_space=True,
                port=port + p,
                port_offset=p, # adding this line will add port_offset to **kwargs
                seed=seed + p,
                **kwargs,
            )
            for p in range(n_parallel)
        ]

and here (as you @Ivan-267 pointed out):

        # If we are doing editor training, n_parallel must be 1
#        if env_path is None and n_parallel > 1:
#            raise ValueError("You must provide the path to a exported game executable if n_parallel > 1")

In particular the last change is not "nice" but enough for testing Scenario 4.

There is no need to change GodotEnv() (in file godot_rl.py).

However, the port argument needs to be added to the Python SB3 example.

Steps for Scenario 4 with the MultiAgentSimple example:

1. Start Python server for policy player (sync node has port 11008):
   python examples/godot_rl_agents/examples/stable_baselines3_example.py --timesteps 50000 --save_model_path=platform-model.zip --onnx_export_path=platform-model.onnx --port=11008 --n_parallel=2

Please note that this will use two TCP ports (because of `n_parallel = 2'): 11008 and 11009

2. Start Python server for policy agent and run the exported executables (sync node has port 11108):
   python examples/godot_rl_agents/examples/stable_baselines3_example.py --timesteps 50000 --save_model_path=platform-model.zip --onnx_export_path=platform-model.onnx --port=11108 --env_path=examples/godot_rl_agents_examples/examples/MultiAgentSimple/MultiAgentSimple.x86_64 --viz --n_parallel=2

Please note that this will use two TCP ports (because of `n_parallel = 2'): 11108 and 11109

And two exported executables will run.

Edit: The agents learned nicely and the model zips and onnx were saved. No errors

[Ivan-267]

Thanks, it's really good to have n_parallel support verified to make the feature more complete/competitive.

I was able to briefly test it and it seems to work for me as well. Some onnx errors, but it's likely due to versions of things, I'll do a fresh latest version install for a final onnx check later.

In the following days, I'd like to try it a bit more.

[stefanfausser]

Some onnx errors, but it's likely due to versions of things, I'll do a fresh latest version install for a final onnx check later.

No error visible on my setup. Btw. I use ONNX version 1.20.1 for C# / Godot side and a newer version for Python.

[Ivan-267]

Just as a brief update, now that we have a baseline that worked in our tests, my plan is to try a few small modifications in the next few days or so as mentioned before. Python side implementation also depends on that.

I will focus on that first, and if the results are interesting - then I will propose the modifications and we can see whether to merge them or keep as is.

The ONNX issue is something I'll check after the implementation is fully done, I don't think it's related to this PR specifically.

[Ivan-267]

I still have to test more, but the basic concept of the additions I'm working on (built upon the base idea and functionality of the PR with modifications explained below).

Python side:

• For starting training, a Python launcher script that takes arguments for SB3 training example, and starts the scripts with automatically assigning ports (there's a new n_policies argument to enable that).
• A new SBGMultiSyncEnv wrapper is introduced (inherits from StableBaselinesGodotEnv) which handles modified port assignment, and implements a method to read the policy name from Godot.
• Modifications of the SB3 training example introducing port and n_policies arguments. It will use the new wrapper env if n_policies > 1, as well as use the policy name for log name and saving/onnx export names (still in progress).

Godot side:

• A new MultiServerSyncManager node that creates MultiServerSync nodes.

To use it, we add it to the scene instead of the sync node, and type in the policy names to train:

It will create new MultiServerSync nodes for each policy with corresponding names (platform 2 is just a third policy I've temporarily added to the env for testing):

MultiServerSyncManager reads the base port from the command-line and handles port assignment to each sync node. It is meant as the node to use when using this functionality, while the "old" sync node is used for the standard workflow (so settings like policy_name are removed).

MultiServerSync is an extension of the sync node that only changes the functionality needed for multi-server training, such as loading only the agents that have the same policy name as set for the sync. I changed this from group matching to policy name matching, and returned the default group name to solve an incompatibility we have right now with how RLLib multi-policy training and testing (onnx inference) scenes expect all agents to be in the AGENT group to work.

The main idea is that the process is automated regarding port settings, relatively simple to use, while kept separate from the old functionality (as much as possible), as conceptually it is a different approach to both multi-policy training and even inference (with this, we'll have two possible ways to handle multi-policy inference, both approaches should work regardless of the training method).

Now, this is still WIP, and might have further issues to address. Once I finish it and if it's working well enough with my basic tests, I'll upload it to my fork of the repositories so you can test it, and we can then make further adjustments as needed.