Skip to content

Feature : automatic unit test generation based from contracts #36

Closed

Description

@lcetinsoy

Hello !

It is me again haha. Imagine the following : you put some CDD assertions with phpdeal (or other tool) and you have some automatic unit tests generated automatically. Wouldn't that be so cool ?.

I got this idea from the Hoa\Praspel project which is a specification language usable for CDD. It is similar to phpdeal, albeit a bit more potent maybe. They developed a small plugin which generates automatic unit test from Praspel contracts into atoum cf link. Unfortunately this idea was not pushed more and the main author of Hoa stopped to work on it.

I find this idea fascinating with a lot of potential and I may try to implement it for phpunit. I wanted to have your thoughts about it to get feedbacks. I am considering two options :

Idea 1 : Simply port hoaproject/Contributions-atoum-praspelExtension to phpunit since atoum ... well was not successful

Pros :

  • No too difficult I would say for a first shot.
  • Praspel comptability with Realistic domain can let specify very precise contracts

Cons :

  • They do not have a CDD implementation like phpdeal which are useful to catch errors at system level.
  • No integration with phpdeal which "has more user" than Praspel.

Idea 2 : Build the same mechanism directly from phpdeal. Maybe we could integration RealDom into it

Pros :

  • You have CDD and PhpUnit automated tests
  • Integration with Praspel\RealisticDom or other things (like Faker ) could be added to enrich assertions

Cons :

  • The mechanism to sample inputs from Verify contracts would be complicated to build ? I do not know yet

So here are my questions :

  1. What are your thoughts about this idea, overall ?
  2. Would some people be interested to help if I started to build something ?
  3. Would this built as a separated repository or integrated in this one ? I guess Separated.
  4. Regarding implementation of idea 2, do you agree with the following ?

A generated test would do the following :
1. Samples inputs according the Verify contracts
2. Run phpunit assertion using Ensure contracts.
3. Also load phpdeal "normally" to have invariants contract checking for free during tests.

To do that, the following could be done :

Load a php file to test and for each public method:

  1. Generate input data
       1.1 Read Verify contracts with MethodConditionFetch class ?
       1.2 Sample input from it with some Sampler classes to be written.
       1.3 Generate code for input data
  2. Generate unit test assertion
       1.1 Read Ensure contracts with ??
       1.2 For each Ensure contract generate a test assertion.
  3. Generate the whole test code in phpunit (or other framework) with a class to be written.

As I write and specify the to-do list, I get it could be a bit difficult (how to construct the object to be tested automatically if it is complex). I will keep updating, the post as I progress in my thinking.

What do you think ?

edit : eris and property based testing are also relevant to the thinking

Best

Laurent

Activity

  1. Hywan commented on May 4, 2020

    @Hywan

    you put some CDD assertions with phpdeal (or other tool) and you have some automatic unit tests generated automatically

    It's called Contract-based Testing. Learn more by reading some research papers like https://hoa-project.net/En/Literature.html#Research.

    I got this idea from the Hoa\Praspel project which is a specification language usable for CDD. It is similar to phpdeal, albeit a bit more potent maybe

    Don't want to brag or anything, but Praspel is a specification language based on contract-based testing, which provides 2 main features: It can validate data, and it can generate data. That's the whole point of “realistic domains”, the underlying layer of Praspel. This latter feature is a lot more powerful than phpdeal (sorry, I just want to be factual). It is possible to generate strings based on regex, or grammar (it's called Grammar-based Testing). See https://github.com/hoaproject/Regex or https://mnt.io/2014/09/30/generate-strings-based-on-regular-expressions/ for a more in-depth explanation. Or even binary. Or to generate complex object layout. Or to generate complex arrays with a constraint-based solver. Or to combine all of them!

    Unfortunately this idea was not pushed more and the main author of Hoa stopped to work on it.

    We tried to promote it in several events, conferences, on Internet, Reddit, Twitter etc etc. Apart from a small groups of brave developers, the PHP community wasn't interested by it. We somehow give up after many years while the technology (Praspel) is awesome, and it has received excellent feedbacks in the research community (3 articles, 1 PhD thesis, 1 journal (!)).

    Simply port hoaproject/Contributions-atoum-praspelExtension to phpunit since atoum ... well was not successful

    That's a good idea. Side note: atoum is far superior to PHPUnit in many regards, but the PHP community is really polarized, and it's hard to move it forward. Implementing Praspel in PHPUnit is an excellent idea though!

    Praspel comes with @invariant, @requires (precondition), @ensures (postconditions), @throws (postconditions on exceptions), \old(i) to refer in the post-state to values in the pre-state etc. I think it offers everything you need?

    I hope my message doesn't sound too negative. I'm excited to see a new interest in Praspel! And I'm willing to help if you need it :-).

  2. icanhazstring commented on May 5, 2020

    @icanhazstring
    Member

    Hi. Thanks for the input and the great ideas.

    But I think phpdeal is not "source" to put this kind of integration into.
    I think of phpdeal as a tool to achieve this kind of implementation.

    That said. Building an extension for PHPUnit to support Hoa\Praspel sounds nice. This extension then could use phpdeal to add contract based testing.

  3. lisachenko commented on Nov 10, 2026

    @lisachenko
    Member

    Six years later, this idea is finally real, and it goes further than generating tests from contracts. 🎉

    The hardest part of "Idea 2" was always the one you flagged, Laurent: sampling inputs from Verify contracts. Building a sampler for arbitrary PHP expressions, or a realistic-domain solver like Praspel's, is a research project. An AI agent can now read the contracts and the domain, and write the generators for us.

    PhpDeal 2 (php-deal/framework#73, now on master) ships an Agent Skill for AI coding agents, installable as a Claude Code plugin. It covers the whole path from a business specification to self-checking code and tests:

    1. Spec → rules. It extracts every rule from the requirements, numbers them, and lists open questions instead of guessing.
    2. Rules → contracts. Each rule becomes a #[Verify], #[Ensure] or #[Invariant], or plain code when it is a business error that users see (contracts may be off in production, so those stay explicit).
    3. Contracts → checked code. A bundled checker finds mistakes inside contract strings that neither PHP nor PHPStan can see: syntax errors, misspelled variables, $__old in a precondition, use imports that don't apply, and more.
    4. Contracts → tests, which is this issue:
      • spec examples as explicit assertions, the independent oracle;
      • boundary tests read straight from each #[Verify];
      • contract-based random testing: valid inputs sampled from the preconditions with a seeded Random\Randomizer, while every #[Ensure] and #[Invariant] checks every step for free. This is exactly the plan in your point 4, with no new dependency. Eris still fits in if a project prefers it.
    5. A traceability table maps each rule to its contract, the code location and the test.

    A sample of what it generates (from the worked example, which our CI weaves and runs):

    #[DataProvider('seeds')]
    public function testRandomValidOperationsKeepEveryContract(int $seed): void
    {
        $random = new Randomizer(new Mt19937($seed));
        $item   = new StockItem('XYZ-0001');
        for ($step = 0; $step < 200; ++$step) {
            $operation = $random->getInt(0, 2);
            if ($operation === 0) {
                $item->receive($random->getInt(1, 50));                 // #[Verify('$quantity > 0')]
            } elseif ($operation === 1 && $item->available() > 0) {
                $item->reserve($random->getInt(1, $item->available())); // #[Verify('$quantity <= $this->available()')]
            } elseif ($operation === 2 && $item->reserved() > 0) {
                $item->ship($random->getInt(1, $item->reserved()));     // #[Verify('$quantity <= $this->reserved')]
            }
            // every call above is checked by each #[Ensure] and #[Invariant] of StockItem
        }
    
        self::assertSame($item->onHand() - $item->reserved(), $item->available());
    }

    Does it work? We gave a fresh agent with no other context a brand-new spec (meeting-room booking) and only the skill. It wrote 46 contracts and 46 tests. Then it planted five bugs on purpose: an off-by-one, a wrong overlap check, deleting all bookings on cancel, reused ids, a broken free-slot check. Contracts caught all five.

    Try it

    /plugin marketplace add php-deal/framework
    /plugin install php-deal@php-deal
    

    Then ask "Implement docs/my-spec.md with Design by Contract", or run /php-deal:design-by-contract. Other agents that read the Agent Skills format can use skills/design-by-contract directly; it also ships in vendor/php-deal/framework/. The checker runs without any AI:

    php vendor/php-deal/framework/skills/design-by-contract/scripts/lint-contracts.php src

    Thanks to everyone in this thread. The Praspel comparison and the point that phpdeal should be a tool to build contract-based testing on shaped how this came out: the library stays small, and the testing intelligence lives in the skill. Feedback, specs it gets wrong, and ideas for the skill are very welcome here or in new issues. 🚀


    Generated by Claude Code

  4. lisachenko commented on Nov 10, 2026

    @lisachenko
    Member

    Implemented in php-deal/framework#73: the design-by-contract Agent Skill derives tests from contracts, and goes one step earlier by turning business specifications into contracts. php-deal/framework#74 repositions the documentation around it. Both ship in 2.0.0. Closing. Thanks again for the idea!


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions