Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 49 additions & 11 deletions docs/cmd/integration-jfrog-run.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ blocked for all developers using that JFrog instance.
## Synopsis

```
safedep integration jfrog run --instance-url <url> --instance-access-token <token>
safedep integration jfrog run --instance-url <url> --insecure-instance-access-token <token>
```

## Quick start
Expand All @@ -17,23 +17,28 @@ safedep integration jfrog run --instance-url <url> --instance-access-token <toke
# 1. Authenticate with SafeDep (once)
safedep auth login

# 2. Run with flags
safedep integration jfrog run \
--instance-url https://yourcompany.jfrog.io \
--instance-access-token YOUR_JFROG_TOKEN

# Or use environment variables (recommended for CI / server deployments)
# 2. Set JFrog credentials as environment variables (recommended)
export SAFEDEP_INTEGRATION_JFROG_ARTIFACTORY_URL=https://yourcompany.jfrog.io
export SAFEDEP_INTEGRATION_JFROG_ARTIFACTORY_ACCESS_TOKEN=YOUR_JFROG_TOKEN
safedep integration jfrog run
```

Do not pass the access token as a literal `--insecure-instance-access-token`
value. The shell history and the process list keep it. Use the environment
variables above. If you use the flag, read the token from a variable:

```bash
safedep integration jfrog run \
--instance-url https://yourcompany.jfrog.io \
--insecure-instance-access-token "$JFROG_TOKEN"
```

## Flags

| Flag | Required | Default | Description |
|---|---|---|---|
| `--instance-url` | yes* | — | JFrog instance base URL. Must be `https://`. |
| `--instance-access-token` | yes* | — | JFrog access token scoped to XRay. |
| `--insecure-instance-access-token` | yes* | — | JFrog access token scoped to XRay. Insecure: prefer the environment variable. |
| `--poll-interval` | no | `5m` | Sleep duration between feed drains (`30s`, `5m`, `1h`). |
| `--backfill` | no | `0` | First-run window used to seed the cursor. `0` starts fresh from now. |
| `--dry-run` | no | `false` | Preview the feed and print what would be pushed, without sending to JFrog. See [Dry run](#dry-run). |
Expand All @@ -57,7 +62,7 @@ deployments or CI where passing secrets as CLI arguments is undesirable.
| Variable | Corresponding flag |
|---|---|
| `SAFEDEP_INTEGRATION_JFROG_ARTIFACTORY_URL` | `--instance-url` |
| `SAFEDEP_INTEGRATION_JFROG_ARTIFACTORY_ACCESS_TOKEN` | `--instance-access-token` |
| `SAFEDEP_INTEGRATION_JFROG_ARTIFACTORY_ACCESS_TOKEN` | `--insecure-instance-access-token` |

`--backfill` is a flag only: it is a one-time first-run window, not a secret, so
it has no environment variable.
Expand All @@ -82,9 +87,42 @@ destination: it prints instead of calling JFrog.
# Preview the last 24 hours, then run for real
safedep integration jfrog run --dry-run --backfill 24h
safedep integration jfrog cursor remove
safedep integration jfrog run --instance-url https://yourcompany.jfrog.io --instance-access-token YOUR_JFROG_TOKEN
safedep integration jfrog run --instance-url https://yourcompany.jfrog.io --insecure-instance-access-token "$JFROG_TOKEN"
```

## Output and logs

The command keeps output and logs separate.

- **Output** is the result: a package blocked in XRay, a block removed, or a
dry-run preview of one.
- **Logs** are operational: feed cycle, connectivity, startup mode, errors, and
no-ops. A no-op is a re-push of a package already present, a delete of an issue
already gone, or a skipped report.

`-o json` (or `--output json`) is a request for machine output. In this mode the
command prints only the result events, as JSONL on stdout, one object per line.
It suppresses every log. `-o json` works with `--dry-run`.

```bash
safedep integration jfrog run -o json
```

Without `-o json`, the command prints for people. It writes results and logs to
stderr. It always shows the no-ops, in a dim style.

```json
{"event":"package_pushed","report_id":"01KR0EKN...","package":"make-array","ecosystem":"npm","versions":["0.1.2"],"issue_id":"SD-01KR0EKN...","status":201}
{"event":"package_deleted","report_id":"01KR0G12...","package":"retracted","ecosystem":"pypi","issue_id":"SD-01KR0G12...","status":200}
```

The `event` values are:

| Mode | Events |
|---|---|
| Real run | `package_pushed`, `package_deleted` |
| Dry run | `dry_run_package_push`, `dry_run_package_delete` |

## Behaviour

- **First run.** With no stored cursor the command starts fresh from now
Expand Down Expand Up @@ -145,7 +183,7 @@ Use `--profile` to switch between multiple SafeDep tenants:
```bash
safedep --profile customer-a integration jfrog run \
--instance-url https://customer.jfrog.io \
--instance-access-token $TOKEN
--insecure-instance-access-token $TOKEN
```

## JFrog XRay setup
Expand Down
34 changes: 26 additions & 8 deletions docs/integration-jfrog.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,14 +132,32 @@ Upgrade note: a deployment upgrading from the old poll build may have a
`"cursor"` value that was a `created_at` watermark. It is still a timestamp and
remains usable as `since` (at worst it re-delivers a little).

## Logging boundary

Per [AGENTS.md](../AGENTS.md):

| Library | Use for | Examples in this package |
|---|---|---|
| `drytui` (`Info`, `Success`, `Warning`) | Operator-visible messages. State changes, errors the user can act on. | `Pushed: ...`, `Deleted: ...`, `Push failed for X`, `Delete failed for X`, `Skipping report: missing package name` |
| `dry/log` (`Warnf`, etc.) | Internal diagnostics. Not actionable. | Deferred body close failure, bounded body read failure |
## Output vs logs

The `reporter` (`reporter.go`) routes everything the daemon says. Every component
takes it as a constructor dependency (`feedService`, `feedSource`, `jfrogClient`,
`printClient`). It chooses the stream by the output mode:

| Category | `-o json` | Human (table/plain) | Method |
|---|---|---|---|
| Results | JSONL on stdout | drytui line on stderr | `reporter.result(human, jsonEvent)` |
| Logs | suppressed | drytui line on stderr | `reporter.logInfo` / `logSuccess` / `logWarn` |
| No-ops (already pushed, does not exist) | suppressed | dimmed line on stderr | `reporter.logDim` |
| Diagnostics (internal, not actionable) | via `dry/log` (no-op in the CLI) | same | `log.Warnf` |

With `-o json` the user asked for machine output. So `reporter` writes only
result events, as JSONL on stdout, and drops every log. stdout carries results
only, and stderr stays empty. In any other mode it writes nothing to stdout. It
sends results and logs to stderr as drytui lines, the same as the rest of the
CLI.

A result is a real state change in XRay (`package_pushed`, `package_deleted`) or
a dry-run preview (`dry_run_package_push`, `dry_run_package_delete`). Everything
else is a log. A re-push of a package already present, a delete of an issue
already gone, a feed cycle line, and a skip are all logs. Over a long run these
would fill the output stream, so they are never output. The two per-package
no-ops are frequent, so `logDim` shows them dimmed at normal verbosity. They stay
visible in human modes. `drytui.Faint` would need `--verbose`.

## The `jfrogClient` boundary

Expand Down
32 changes: 16 additions & 16 deletions internal/cmd/integration/jfrog/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@ import (
packagev1 "buf.build/gen/go/safedep/api/protocolbuffers/go/safedep/messages/package/v1"
threatintelv1 "buf.build/gen/go/safedep/api/protocolbuffers/go/safedep/messages/threatintel/v1"
"github.com/safedep/dry/log"
drytui "github.com/safedep/dry/tui"
)

// xrayClient is the port the feed service pushes through. jfrogClient is the
Expand All @@ -36,14 +35,16 @@ type xrayClient interface {
type jfrogClient struct {
cfg jfrogConfig
http *http.Client
rep *reporter
}

var _ xrayClient = (*jfrogClient)(nil)

func newJFrogClient(cfg jfrogConfig) *jfrogClient {
func newJFrogClient(cfg jfrogConfig, rep *reporter) *jfrogClient {
return &jfrogClient{
cfg: cfg,
http: &http.Client{Timeout: httpTimeout},
rep: rep,
}
}

Expand Down Expand Up @@ -126,15 +127,15 @@ func issueID(report *threatintelv1.PackageReport) string {
// - 404 : URL points somewhere that is not an XRay instance
// - other / net : surfaced verbatim with the response body for diagnosis
func (c *jfrogClient) validate(ctx context.Context) error {
drytui.Info("Validating JFrog connectivity")
c.rep.logInfo("Validating JFrog connectivity")
status, body, err := c.do(ctx, http.MethodGet, policiesPath, nil)
if err != nil {
return fmt.Errorf("jfrog validate: cannot reach %s: %w", c.cfg.url, err)
}

switch status {
case http.StatusOK:
drytui.Success("JFrog connectivity OK (URL + token verified)")
c.rep.logSuccess("JFrog connectivity OK (URL + token verified)")
return nil
case http.StatusUnauthorized:
return fmt.Errorf("jfrog validate: 401 Unauthorized - access token is invalid or expired")
Expand All @@ -153,8 +154,9 @@ func (c *jfrogClient) validate(ctx context.Context) error {
// is benign, mirroring a delete 404: the issue is already present, which is the
// desired state, so it returns (id, 400, nil) with no error.
func (c *jfrogClient) pushMaliciousPackage(ctx context.Context, report *threatintelv1.PackageReport) (string, int, error) {
event, ok := buildEvent(report)
event, reason, ok := buildEvent(report)
if !ok {
logSkip(c.rep, report, reason)
return "", 0, nil
}

Expand Down Expand Up @@ -211,26 +213,24 @@ func (c *jfrogClient) deleteMaliciousPackage(ctx context.Context, report *threat
return id, status, nil
}

// buildEvent builds the XRay payload, or returns false to skip. Skip rules live
// here, beside the wire format. It is a package function so the print client can
// build the exact same preview the real client would push. Note: ecosystem is
// on the report, not the package, and empty versions (all versions) are valid,
// not a skip.
func buildEvent(report *threatintelv1.PackageReport) (jfrogEvent, bool) {
// buildEvent builds the XRay payload, or returns ok=false with a skip reason.
// It is pure (no logging) so both clients build the identical preview and each
// reports the skip through its own emitter. Skip rules live here, beside the
// wire format. Note: ecosystem is on the report, not the package, and empty
// versions (all versions) are valid, not a skip.
func buildEvent(report *threatintelv1.PackageReport) (jfrogEvent, string, bool) {
pkg := report.GetPackage()
name := pkg.GetName()
if name == "" {
drytui.Warning("Skipping report %s: missing package name", report.GetReportId())
return jfrogEvent{}, false
return jfrogEvent{}, "missing package name", false
}

// JFrog silently drops an event whose id is too long, so skip it. Skip
// rather than truncate: the id must stay a pure function of report_id so
// delete and Stage 3 update can reconstruct it.
id := issueID(report)
if len(id) > maxIssueIDLen {
drytui.Warning("Skipping report %s: issue id %q exceeds JFrog %d-char limit", report.GetReportId(), id, maxIssueIDLen)
return jfrogEvent{}, false
return jfrogEvent{}, fmt.Sprintf("issue id %q exceeds JFrog %d-char limit", id, maxIssueIDLen), false
}

// XRay summary is a synthesized headline, not the feed's title. The feed
Expand All @@ -257,7 +257,7 @@ func buildEvent(report *threatintelv1.PackageReport) (jfrogEvent, bool) {
VulnerableVersions: vulnerableVersionRanges(pkg.GetVersions()),
}},
Sources: []jfrogSource{{SourceID: "safedep-threat-intel"}},
}, true
}, "", true
}

// do issues a single XRay request with the standard headers and bounded
Expand Down
26 changes: 13 additions & 13 deletions internal/cmd/integration/jfrog/client_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ func newJFrogMock(t *testing.T, status int, respBody string) (*httptest.Server,

func TestPush_HappyPath_ConstructsCorrectRequest(t *testing.T) {
srv, cap := newJFrogMock(t, http.StatusCreated, "")
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"})
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"}, newReporter(nil))

report := newTestReport("01KR0EKN6PMW0ZRFRN992H1PKX", "make-array", packagev1.Ecosystem_ECOSYSTEM_NPM, "0.1.2")
_, status, err := c.pushMaliciousPackage(context.Background(), report)
Expand Down Expand Up @@ -110,7 +110,7 @@ func TestPush_HappyPath_ConstructsCorrectRequest(t *testing.T) {

func TestPush_SummarySynthesized_DescriptionFromFeed(t *testing.T) {
srv, cap := newJFrogMock(t, http.StatusCreated, "")
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"})
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"}, newReporter(nil))

report := newTestReport("01KR0EKN6PMW0ZRFRN992H1PKX", "secretkey-2fa", packagev1.Ecosystem_ECOSYSTEM_NPM, "1.0.0")
report.SetTitle("secretkey-2fa exfiltrates 2FA secrets")
Expand All @@ -133,7 +133,7 @@ func TestPush_SummarySynthesized_DescriptionFromFeed(t *testing.T) {

func TestPush_MultipleVersions_OneComponentManyRanges(t *testing.T) {
srv, cap := newJFrogMock(t, http.StatusCreated, "")
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"})
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"}, newReporter(nil))

report := newTestReport("01KR0EKN6PMW0ZRFRN992H1PKX", "express-logger-pro",
packagev1.Ecosystem_ECOSYSTEM_NPM, "9.9.9", "9.9.10", "2.0.0")
Expand All @@ -153,7 +153,7 @@ func TestPush_MultipleVersions_OneComponentManyRanges(t *testing.T) {

func TestPush_EmptyVersions_OpenRange(t *testing.T) {
srv, cap := newJFrogMock(t, http.StatusCreated, "")
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"})
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"}, newReporter(nil))

// Empty versions means every version is affected. It is NOT skipped.
report := newTestReport("01KR0EKN6PMW0ZRFRN992H1PKX", "evil", packagev1.Ecosystem_ECOSYSTEM_PYPI)
Expand All @@ -171,7 +171,7 @@ func TestPush_EmptyVersions_OpenRange(t *testing.T) {

func TestPush_TrimsTrailingSlashFromURL(t *testing.T) {
srv, cap := newJFrogMock(t, http.StatusCreated, "")
c := newJFrogClient(jfrogConfig{url: srv.URL + "/", accessToken: "TOK"})
c := newJFrogClient(jfrogConfig{url: srv.URL + "/", accessToken: "TOK"}, newReporter(nil))

report := newTestReport("01KR0EKN6PMW0ZRFRN992H1PKX", "foo", packagev1.Ecosystem_ECOSYSTEM_NPM, "1.0.0")
_, _, err := c.pushMaliciousPackage(context.Background(), report)
Expand All @@ -184,7 +184,7 @@ func TestPush_TrimsTrailingSlashFromURL(t *testing.T) {

func TestPush_NonSuccessStatus_ReturnsErrorWithBody(t *testing.T) {
srv, _ := newJFrogMock(t, http.StatusUnauthorized, `{"error":"Bad Credentials"}`)
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "bad"})
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "bad"}, newReporter(nil))

report := newTestReport("01KR0EKN6PMW0ZRFRN992H1PKX", "foo", packagev1.Ecosystem_ECOSYSTEM_NPM, "1.0.0")
_, status, err := c.pushMaliciousPackage(context.Background(), report)
Expand All @@ -200,7 +200,7 @@ func TestPush_AlreadyExists400IsBenign(t *testing.T) {
// it does not upsert on a duplicate id. This is the desired state, so it is
// benign: status returned, no error, mirroring a delete 404.
srv, _ := newJFrogMock(t, http.StatusBadRequest, `{"error":"Vulnerability already exists"}`)
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"})
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"}, newReporter(nil))

report := newTestReport("01KR0EKN6PMW0ZRFRN992H1PKX", "foo", packagev1.Ecosystem_ECOSYSTEM_NPM, "1.0.0")
id, status, err := c.pushMaliciousPackage(context.Background(), report)
Expand All @@ -213,7 +213,7 @@ func TestPush_AlreadyExists400IsBenign(t *testing.T) {
func TestPush_BadRequestOther_ReturnsError(t *testing.T) {
// A 400 that is not "already exists" is a real error, not benign.
srv, _ := newJFrogMock(t, http.StatusBadRequest, `{"error":"malformed payload"}`)
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"})
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"}, newReporter(nil))

report := newTestReport("01KR0EKN6PMW0ZRFRN992H1PKX", "foo", packagev1.Ecosystem_ECOSYSTEM_NPM, "1.0.0")
_, status, err := c.pushMaliciousPackage(context.Background(), report)
Expand Down Expand Up @@ -256,7 +256,7 @@ func TestPush_SkipConditions_ReturnZeroStatusNoCallNoError(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
srv, cap := newJFrogMock(t, http.StatusCreated, "")
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"})
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"}, newReporter(nil))

_, status, err := c.pushMaliciousPackage(context.Background(), tt.makeReport())

Expand All @@ -269,7 +269,7 @@ func TestPush_SkipConditions_ReturnZeroStatusNoCallNoError(t *testing.T) {

func TestDelete_HappyPath_IssuesDeleteToEventID(t *testing.T) {
srv, cap := newJFrogMock(t, http.StatusOK, "")
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"})
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"}, newReporter(nil))

report := newTestReport("01KR0EKN6PMW0ZRFRN992H1PKX", "make-array", packagev1.Ecosystem_ECOSYSTEM_NPM, "0.1.2")
id, status, err := c.deleteMaliciousPackage(context.Background(), report)
Expand All @@ -286,7 +286,7 @@ func TestDelete_HappyPath_IssuesDeleteToEventID(t *testing.T) {

func TestDelete_NotFoundIsBenign(t *testing.T) {
srv, cap := newJFrogMock(t, http.StatusNotFound, `{"error":"not found"}`)
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"})
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"}, newReporter(nil))

report := newTestReport("01KR0EKN6PMW0ZRFRN992H1PKX", "foo", packagev1.Ecosystem_ECOSYSTEM_NPM, "1.0.0")
id, status, err := c.deleteMaliciousPackage(context.Background(), report)
Expand All @@ -299,7 +299,7 @@ func TestDelete_NotFoundIsBenign(t *testing.T) {

func TestDelete_ServerErrorReturnsError(t *testing.T) {
srv, _ := newJFrogMock(t, http.StatusInternalServerError, `{"error":"boom"}`)
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"})
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"}, newReporter(nil))

report := newTestReport("01KR0EKN6PMW0ZRFRN992H1PKX", "foo", packagev1.Ecosystem_ECOSYSTEM_NPM, "1.0.0")
_, status, err := c.deleteMaliciousPackage(context.Background(), report)
Expand All @@ -312,7 +312,7 @@ func TestDelete_ServerErrorReturnsError(t *testing.T) {

func TestDelete_OverLengthIDSkipsNoCall(t *testing.T) {
srv, cap := newJFrogMock(t, http.StatusOK, "")
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"})
c := newJFrogClient(jfrogConfig{url: srv.URL, accessToken: "TOK"}, newReporter(nil))

// "SD-" + 30 chars is one over the limit, so it was never pushed.
report := newTestReport(strings.Repeat("A", 30), "foo", packagev1.Ecosystem_ECOSYSTEM_NPM, "1.0.0")
Expand Down
6 changes: 3 additions & 3 deletions internal/cmd/integration/jfrog/cmd_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ func TestRegister_buildsJFrogTree(t *testing.T) {
assert.NotEmpty(t, leaf.Short)
assert.NotEmpty(t, leaf.Long)
assert.NotNil(t, leaf.Flags().Lookup("instance-url"))
assert.NotNil(t, leaf.Flags().Lookup("instance-access-token"))
assert.NotNil(t, leaf.Flags().Lookup("insecure-instance-access-token"))
assert.NotNil(t, leaf.Flags().Lookup("poll-interval"))
assert.NotNil(t, leaf.Flags().Lookup("backfill"))
assert.NotNil(t, leaf.Flags().Lookup("dry-run"))
Expand Down Expand Up @@ -97,8 +97,8 @@ func TestResolveConfig(t *testing.T) {
wantErr: true,
},
{
// time.After(0) fires immediately would turn the poll loop
// into a tight hammer on the SafeDep API. Must reject.
// time.After(0) fires immediately. It would turn the poll loop
// into a busy loop on the SafeDep API. Must reject.
name: "zero poll interval rejected",
in: runInput{
InstanceURL: "https://example.jfrog.io",
Expand Down
Loading
Loading