Skip to content

feat: add an orchestration layer for testing packages - #10727

Merged
paulbalandan merged 1 commit into
codeigniter4:4.8from
paulbalandan:package-orchestrator
Nov 15, 2026
Merged

paulbalandan merged 1 commit into
codeigniter4:4.8from
paulbalandan:package-orchestrator

Conversation

@paulbalandan

Copy link
Copy Markdown
Member

Description
Libraries that supplement CodeIgniter have no standard way to replicate an appstarter environment in their tests. Each package copies the same <const> block into phpunit.xml.dist, points CONFIGPATH at vendor/codeigniter4/framework/app/Config, and fakes everything else in tests/_support. A customised app/, a .env file, or a real spark run cannot be tested.

This adds CodeIgniter\Test\Orchestrator\AbstractOrchestrator. A package extends it, implements getPackageRoot(), and gets a throwaway skeleton built from the installed framework:

  • build() copies app, public, writable and spark from the framework, points Paths.php at its system directory, writes a .env, shares the package's autoloader through a vendor/autoload.php shim, mirrors the fixture directories, then calls afterBuild().
  • boot() builds the skeleton, defines the path constants against it, and loads system/Test/bootstrap.php, so CIUnitTestCase tests run unchanged against it.
  • spark() runs the skeleton's spark in a separate process and returns a SparkResult with the exit code, stdout, stderr and timedOut. It accepts stdin, environment overrides and a timeout.
  • write(), delete() and clean() change or remove the skeleton. They refuse paths that leave it, including through .. and symlinks.
  • Subclasses can override getFixtures(), getEnv(), getPath(), getFrameworkPath() and afterBuild().

Parallel runs get separate skeletons from ORCHESTRATOR_TEST_TOKEN, or from TEST_TOKEN as set by ParaTest.

Not covered: starting FrankenPHP for Worker Mode. The user guide shows how to publish the worker files with worker:install from afterBuild().

Checklist:

  • Securely signed commits
  • Component(s) with PHPDoc blocks, only if necessary or adds value (without duplication)
  • Unit testing, with >80% coverage
  • User guide updated
  • Conforms to style guide

@carson-codeigniter4 carson-codeigniter4 Bot added the 4.8 PRs that target the `4.8` branch. label Nov 5, 2026
@neznaika0

Copy link
Copy Markdown
Contributor

Have you looked at the forum? This is a very useful feature; I haven’t added it to the framework yet, as such innovations aren’t always added to production.
I’ll take a closer look later.

@neznaika0 neznaika0 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t fully understand the entire process, but I grasped the main idea.
All my tests passed on Arch.

  1. The reason for using two vendor/autoload.php files is unclear.
  2. The documentation should describe the limitations — the current version specified in the project is being tested. That to test another version, you need a copy of the framework in some folder.
  3. I tried to launch spark directly after creation, but nothing happens. Is this correct?
orchestrator/$ ORCHESTRATOR_FLOOD=1 php spark orchestrator:pro
be 

CodeIgniter v4.8.0-dev Command Line Tool - Server Time: 2026-11-05 19:26:58 UTC+00:00

That’s all for now; we need to try it with a real package.

Comment thread docs/testing/packages.md Resolved
@paulbalandan

Copy link
Copy Markdown
Member Author
  1. The reason for using two vendor/autoload.php files is unclear.

It's a shim to the package/library's own vendor/autoload.php. Instead of creating the skeleton's own autoload.php, it instead reuses the library's own autoload.php, which then boots the orchestrator.

  1. The documentation should describe the limitations — the current version specified in the project is being tested. That to test another version, you need a copy of the framework in some folder.

Added a note admonition in the docs.

  1. I tried to launch spark directly after creation, but nothing happens. Is this correct?

If you have not called clean() on the skeleton, running spark from the library root is php build/orchestrator/spark <command_name>. On the orchestrator:probe command, it's supposed to hang, since it's meant to run in a non-TTY stdin.

@paulbalandan

Copy link
Copy Markdown
Member Author

Shield can be a first-party consumer to this feature. I tried this on my clone of codeigniter4/shield (develop, 2f4b0a4) with vendor/codeigniter4/framework pointed at this branch. Adopting it takes three small changes and no changes to Shield's own code.

1. The orchestrator

// tests/_support/ShieldOrchestrator.php

namespace Tests\Support;

use CodeIgniter\Test\Orchestrator\AbstractOrchestrator;

final class ShieldOrchestrator extends AbstractOrchestrator
{
    protected function getPackageRoot(): string
    {
        return dirname(__DIR__, 2);
    }

    protected function getEnv(): array
    {
        return [
            'app.baseURL'   => 'https://example.com/',
            'auth.hashCost' => '4',
        ];
    }
}

2. The bootstrap

// tests/bootstrap.php

require_once __DIR__ . '/../vendor/autoload.php';
require_once __DIR__ . '/_support/ShieldOrchestrator.php';

(new Tests\Support\ShieldOrchestrator())->boot();

3. phpunit.xml.dist

-    bootstrap="vendor/codeigniter4/framework/system/Test/bootstrap.php"
+    bootstrap="tests/bootstrap.php"
...
-        <const name="HOMEPATH" value="./"/>
-        <const name="CONFIGPATH" value="vendor/codeigniter4/framework/app/Config/"/>
-        <const name="PUBLICPATH" value="./public/"/>

The <server name="app.baseURL"> and <env name="auth.hashCost"> entries can move into getEnv() or stay where they are. The database.tests.* variables keep coming from PHPUnit's environment, which the spark() child inherits.

What I ran

A probe test, extending Shield's existing TestCase:

public function testShieldConfigIsDiscoveredInProcess(): void
{
    $this->assertInstanceOf(Auth::class, config('Auth'));
    $this->assertStringContainsString('/build/orchestrator/app/', APPPATH);
    $this->assertSame('https://example.com/', config('App')->baseURL);
}

public function testShieldCommandsAreDiscoveredBySpark(): void
{
    $result = (new ShieldOrchestrator())->spark(['list']);

    $this->assertSame(0, $result->exitCode);
    $this->assertStringContainsString('shield:setup', $result->stdout);
}

Both pass. The package namespace is found with no fixture or Registrar, both in the PHPUnit process and by spark, because the skeleton's vendor/autoload.php requires the package's autoloader.

I also ran these existing Shield tests under the new bootstrap: AuthRoutesTest, FilterInCliTest, PasswordsTest, CompositionValidatorTest and GenerateDummyPasswordHashTest. All pass. SetupTest::testRun fails, and it fails the same way under Shield's original config, so it is not caused by the orchestrator.

What this does not cover

  • Shield supports framework versions older than 4.8, which do not have the orchestrator. Adopting it would need a 4.8-only branch or two PHPUnit configs.
  • Shield's TestCase and DatabaseTestCase stay as they are. The orchestrator replaces the path constants and the skeleton, not the in-process test cases.

What it adds is spark(), so commands such as shield:setup and shield:user can be tested end-to-end instead of only through mocks.

@neznaika0 neznaika0 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The choice of var_export() for strings is a bit unclear.
Overall, I like it.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A symlinked ancestor can redirect clean() and build() into recursively deleting an external directory, leaving a critical filesystem-safety flaw.

3 open issues
What changed in this PR

Adds a package-testing orchestration layer that builds disposable CodeIgniter applications, allowing libraries to test against real framework configuration and commands.

Changes:

  • Introduces skeleton creation, PHPUnit bootstrapping, fixture overlays, and guarded file operations.
  • Adds subprocess spark() execution with input, environment overrides, timeout handling, and structured results.
  • Adds tests, fixtures, and user-guide documentation.

Validation was limited to source inspection; automated tests and full validation remain for CI.

File Description
tests/​system/​Test/​Orchestrator/​AbstractOrchestratorTest.php Tests skeleton operations, path safety, environment handling, and subprocess execution.
tests/​_support/​Orchestrator/​OrchestratorStub.php Provides a configurable orchestrator test double.
tests/​_support/​Orchestrator/​Fixtures/​override.txt Supplies the initial fixture-overlay value.
tests/​_support/​Orchestrator/​Fixtures/​app/​Commands/​OrchestratorProbe.php Exercises subprocess input, output, failures, and timeouts.
tests/​_support/​Orchestrator/​ExtraFixtures/​override.txt Verifies later fixtures overwrite earlier files.
tests/​_support/​Orchestrator/​ExtraFixtures/​extra.txt Verifies additional fixture files are copied.
system/​Test/​Orchestrator/​SparkResult.php Holds process output, exit code, and timeout status.
system/​Test/​Orchestrator/​AbstractOrchestrator.php Implements application orchestration and subprocess execution.
docs/​testing/​packages.md Documents setup, customization, testing, and Worker Mode limitations.
docs/​testing/​index.md Links the package-testing guide.
docs/​index.md Adds the guide to the main documentation index.
docs/​changelogs/​v4.8.0.md Announces the new testing API.
docs/​.nav.yml Adds the guide to documentation navigation.

Review effort: Balanced

@paulbalandan

Copy link
Copy Markdown
Member Author

The choice of var_export() for strings is a bit unclear. Overall, I like it.

When you var_export a string, it produces a single-quoted string literal that is valid as PHP code.

@michalsn michalsn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good on paper. I guess we will have to test it out with one of our packages to catch any issues.

@paulbalandan
paulbalandan merged commit 0b38173 into codeigniter4:4.8 Nov 15, 2026
59 checks passed
@paulbalandan
paulbalandan deleted the package-orchestrator branch November 15, 2026 12:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

4.8 PRs that target the `4.8` branch.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants