Openfn compile for unit tests

.claude/unit-testing-jobs.md

@@ -0,0 +1,157 @@
+# Unit Testing Job Code
+
+OpenFn job expressions are not valid JavaScript out of the box — top-level adaptor calls like `get('/endpoint')` prevent them from being imported directly into a test runner. Compiling them first solves this.
+
+## Approach
+
+1. Compile job expressions to standard JavaScript
+2. Import compiled files in your test suite
+3. Test any pure functions in isolation
+
+## Compiling for Tests
+
+`openfn compile` writes compiled output to `compiled/` by default. Use `--exports-only` to also strip adaptor operation calls, keeping only explicitly exported code.
+
+```bash
+# Compile all workflows in the project (full compilation, preserves operations)
+openfn compile
+
+# Compile all workflows, stripping operation calls (useful for unit testing)
+openfn compile --exports-only
+
+# Compile a single workflow by name
+openfn compile my-workflow
+
+# Compile a single workflow to a custom directory
+openfn compile my-workflow -o tests/
+
+# Compile a single job expression (prints to stdout)
+openfn compile workflows/my-workflow/step-a.js --exports-only
+
+# Watch mode: recompile whenever source files change
+openfn compile --exports-only --watch
+```
+
+## What `--exports-only` keeps
+
+`--exports-only` strips operation calls. Only explicitly exported declarations survive:
+
+- `export const myHelper = ...` ✓
+- `export function parseSms() {}` ✓
+- `const helper = ...` (not exported) ✗ — dropped
+- `fn(state => ...)` (operation call) ✗ — stripped
+
+If an exported function depends on a non-exported local declaration, that dependency is kept automatically.
+
+`export default` is removed in strip mode — it is only needed by the runtime, not for unit testing.
+
+Steps with no exportable code (nothing is exported after stripping) are skipped — no file is written.
+
+## Full compilation (no `--exports-only`)
+
+Without `--exports-only`, the full compiled output is written — all declarations are preserved and operations are kept in `export default [op1, op2, ...]`:
+
+```js
+import { post } from '@openfn/language-http';
+
+export default [post('/endpoint', { data: state.data })];
+```
+
+All steps are written regardless of whether they export anything.
+
+## Example: Testing a Helper Function
+
+**Source** (`workflows/dhis2-sync/transform.js`):
+
+```js
+import { dateFns } from '@openfn/language-dhis2';
+
+export const formatDate = (date) => dateFns.format(date, 'yyyy-MM-dd');
+
+fn((state) => ({
+  ...state,
+  data: state.data.map((row) => ({
+    ...row,
+    date: formatDate(row.date),
+  })),
+}));
+```
+
+**Compiled** (`compiled/dhis2-sync/transform.js`) after `openfn compile --exports-only`:
+
+```js
+import { dateFns } from '@openfn/language-dhis2';
+
+export const formatDate = (date) => dateFns.format(date, 'yyyy-MM-dd');
+```
+
+The operation is stripped. `formatDate` survives because it is explicitly exported.
+
+**Test** (using any test runner):
+
+```js
+// test/transform.test.js
+import { formatDate } from '../compiled/dhis2-sync/transform.js';
+
+test('formats a date correctly', () => {
+  const result = formatDate(new Date('2024-01-15'));
+  assert.equal(result, '2024-01-15');
+});
+```
+
+## Project-wide Compilation
+
+Running `openfn compile` with no path compiles every step in every workflow in the current project directory (must contain `openfn.yaml`).
+
+Output layout:
+
+```
+compiled/
+  my-workflow/
+    step-a.js
+    step-b.js
+  another-workflow/
+    step-c.js
+```
+
+Override the output directory with `-o <dir>`:
+
+```bash
+openfn compile --exports-only -o tests/
+```
+
+Configure default directories in `openfn.yaml`:
+
+```yaml
+dirs:
+  workflows: workflows
+  compiled: compiled # used by openfn compile
+```
+
+### Stale file cleanup
+
+After each project-wide run with `--exports-only`, any step file that was skipped (no exportable code) is automatically deleted from `compiled/` if it exists from a previous run. Only files at exact step paths (`compiled/<workflow-id>/<step-id>.js`) are touched — files you have added at other paths (e.g. `compiled/my-workflow/helpers.js`) are never removed.
+
+There is no option to wipe the entire `compiled/` directory. To do a full reset, delete it manually before running `openfn compile`.
+
+## Recommended Setup
+
+**`package.json`** (in your OpenFn project):
+
+```json
+{
+  "scripts": {
+    "compile": "openfn compile --exports-only",
+    "compile:watch": "openfn compile --exports-only --watch",
+    "test": "node --test test/**/*.test.js"
+  }
+}
+```
+
+## Notes
+
+- In `--exports-only` mode, only `export const` and `export function` declarations survive — non-exported helpers are dropped unless referenced by an export
+- Import statements are always preserved
+- Without `--exports-only`, all code including operations is kept in `export default [...]`
+- Watch mode reruns compilation on any source change, making the edit → test cycle fast
+- Use `-O` to print compiled output to stdout instead of writing to disk

claude.md

@@ -111,6 +111,7 @@ The [.claude](.claude) folder contains detailed guides:
 
 - **[command-refactor.md](.claude/command-refactor.md)** - Refactoring CLI commands into project subcommand structure
 - **[event-processor.md](.claude/event-processor.md)** - Worker event processing architecture (batching, ordering)
+- **[unit-testing-jobs.md](.claude/unit-testing-jobs.md)** - How to unit-test job code using `openfn compile --test`
 
 ## Code Standards
 

packages/cli/package.json

@@ -60,6 +60,7 @@
     "@openfn/project": "workspace:^",
     "@openfn/runtime": "workspace:*",
     "chalk": "^5.6.2",
+    "chokidar": "^3.6.0",
     "dotenv": "^17.3.1",
     "dotenv-expand": "^12.0.3",
     "figures": "^5.0.0",

packages/cli/src/compile/command.ts

@@ -1,13 +1,14 @@
 import yargs from 'yargs';
 import { Opts } from '../options';
 import * as o from '../options';
-import { build, ensure, override } from '../util/command-builders';
+import { build, ensure } from '../util/command-builders';
 
 export type CompileOptions = Pick<
   Opts,
   | 'adaptors'
   | 'command'
   | 'expandAdaptors'
+  | 'exportsOnly'
   | 'ignoreImports'
   | 'expressionPath'
   | 'logJson'
@@ -19,6 +20,8 @@ export type CompileOptions = Pick<
   | 'useAdaptorsMonorepo'
   | 'globals'
   | 'trace'
+  | 'watch'
+  | 'workflowName'
 > & {
   workflow?: Opts['workflow'];
   repoDir?: string;
@@ -27,40 +30,64 @@ export type CompileOptions = Pick<
 const options = [
   o.expandAdaptors, // order important
   o.adaptors,
+  o.exportsOnly,
   o.ignoreImports,
   o.inputPath,
   o.log,
   o.logJson,
-  override(o.outputStdout, {
-    default: true,
-  }),
+  o.outputStdout,
   o.outputPath,
   o.repoDir,
   o.trace,
   o.useAdaptorsMonorepo,
+  o.watchFlag,
   o.workflow,
 ];
 
 const compileCommand: yargs.CommandModule<CompileOptions> = {
   command: 'compile [path]',
   describe:
-    'Compile an openfn job or workflow and print or save the resulting JavaScript.',
+    'Compile an openfn job, workflow, or whole project and print or save the resulting JavaScript.',
   handler: ensure('compile', options),
   builder: (yargs) =>
     build(options, yargs)
       .positional('path', {
         describe:
-          'The path to load the job or workflow from (a .js or .json file or a dir containing a job.js file)',
-        demandOption: true,
+          'Path to a .js expression, .json/.yaml workflow, or a project directory. Omit to compile all workflows in the current project.',
       })
       .example(
         'compile foo/job.js',
         'Compiles the job at foo/job.js and prints the result to stdout'
       )
       .example(
         'compile foo/workflow.json -o foo/workflow-compiled.json',
-        'Compiles the workflow at foo/work.json and prints the result to -o foo/workflow-compiled.json'
-      ),
+        'Compiles the workflow and writes to the given path'
+      )
+      .example(
+        'compile',
+        'Compiles all workflows in the current project and writes JS files to compiled/'
+      )
+      .example(
+        'compile my-workflow',
+        'Compiles a single workflow by name and writes JS files to compiled/'
+      )
+      .example(
+        'compile my-workflow -O',
+        'Compiles a workflow and prints to stdout'
+      )
+      .example(
+        'compile foo/job.js --exports-only',
+        'Strips adaptor operation calls, keeping only exported declarations'
+      )
+      .example(
+        'compile --exports-only',
+        'Compiles entire project to compiled/ stripping operation calls'
+      )
+      .example(
+        'compile foo/job.js --watch',
+        'Watches the file and recompiles on every change'
+      )
+      .example('compile --watch', 'Compiles all workflows on change'),
 };
 
 export default compileCommand;