Repository navigation
Conversation
| // its raw path to contain UTF-8 percent escapes; Windows happens to accept the Unicode form. | ||
| // VFS can leave these filename characters unescaped; they are illegal in a URI path. | ||
| URI uri = | ||
| URI.create( |
There was a problem hiding this comment.
[bug] localPath is on the listing hot path (isLink calls it for every local file and directory). Commons VFS 2.10 only percent-encodes #, space, and % in getURI(). This method then encodes []{}|^`` and still calls URI.create. A legal Unix name containing ?, ", <, >, `, or a newline is left raw. URI.create throws IllegalArgumentException for most of those, and Path.of throws for ? (URI has a query component) and for # if it were unescaped. tick only catches IOException, and processRow turns that runtime exception into a transform failure, so one such file stops the watcher. The same URI.create pattern is repeated in resolveFile for explicit file: paths.
Suggestion: Percent-encode every character that is illegal in a URI path (at least ?, ", <, >, \, and controls) before URI.create, or build the Path from the decoded VFS path instead of round-tripping the URI. Add a regression with a name such as report?.csv and a<b.txt.
| // A key already removed from the service must be processed on the next scan. | ||
| if (key != null) { | ||
| key.pollEvents(); | ||
| key.reset(); |
There was a problem hiding this comment.
[suggestion] When the hint cap stops the drain, this already-dequeued key is discarded with pollEvents() and reset() without using the return value. The in-loop path removes keys and registered when reset() is false; this path does not. If the directory is gone, it stays in registered, so register will not watch it again after it is recreated, and it still counts toward the directory limit. Reconciliation can still see the files, but native delivery for that directory stays off for the rest of the run.
Suggestion: Use the same reset() handling as the loop: on false, drop the key and its path from keys and registered before setting reconcile.
| *** xref:pipeline/transforms/webservices.adoc[Web services lookup] | ||
| *** xref:pipeline/transforms/workflow-executor.adoc[Workflow Executor] | ||
| *** xref:pipeline/transforms/workflow-logging.adoc[Workflow logging] | ||
| *** xref:pipeline/transforms/watchfiles.adoc[Watch Files] |
There was a problem hiding this comment.
[nit] The transform list is alphabetical by title. Watch Files is inserted after Workflow logging and before Write to log, so it sits past Web services lookup and Workflow Executor.
Suggestion: Move this xref to just before Web services lookup.
539f1d9 to
6fd8ccc
Compare
2 participants
Adds a Watch Files source for directory arrivals and changes, with AUTO/NATIVE/POLLING detection, stable-file checks, persistent checkpoints and reconciliation. It emits CREATED/MODIFIED/DELETED metadata for existing Hop readers. The GUI has General, Advanced and Maintenance tabs, wildcard filters, automatic state identity and an optional runtime in minutes/hours; blank keeps monitoring continuously.
Implemented with Codex (high reasoning setting). Extensive tests cover Windows/Linux, real pipeline restarts, overflow, checkpoint/crash recovery, SFTP, ENOSPC and GUI behavior. State represents source observations, independently of downstream completion; support is limited to the normal Local Hop Engine.
General (optional timeout):
Maintenance: