diff --git a/.github/workflows/platforms.yml b/.github/workflows/platforms.yml index c00fdbf9..e24abf60 100644 --- a/.github/workflows/platforms.yml +++ b/.github/workflows/platforms.yml @@ -198,7 +198,7 @@ jobs: # whole run, which is worse than not running it: a queued job # keeps its run open and holds concurrency. If Intel coverage is # wanted back it needs a runner that actually exists. - - { runner: macos-14, arch: "arm64" } + - { runner: macos-15, arch: "arm64" } steps: - uses: actions/checkout@v4 with: diff --git a/RELEASE.md b/RELEASE.md index 2eea7adc..576409ff 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -9,6 +9,44 @@ shell. The welcome banner links to the guide for the latest one: --- +## v3.3.0 — *the hint knows what you run most* + +**The guide: [What's new in 3.3](https://univers42.github.io/hellish/whats-new/)**. + +- **Hints are ranked by frecency.** With `HELLISH_AUTOSUGGEST=1`, the dim + hint is the matching history line you run most, weighted toward recent + runs, and no longer simply the newest one. Run `make re` three times and + `make clean` once more recently, and `make ` hints `re`. Alt-N / Alt-P + step through the matches in the same order (#140). +- **`make ` lists the targets you run first**, most-run first, then + the rest alphabetically (#140). +- `HELLISH_SUGGEST_STRATEGY=recent` brings back newest-first for both. It + is read at every prompt. +- `HISTCONTROL=ignoredups` or `ignoreboth` stores a back-to-back repeat + once, so those repeats rank as one run. + +Fixed on the way to the release: + +- **A `$(...)` left as literal text.** A double-quoted word inside it that + held a nested substitution with an odd number of quote bytes stopped the + expansion: `x=$(echo "$(echo "a'b")")` assigned its own source text, with + status 0 and no error. All four reported forms now match bash 5.3.9 + (#190). +- **A `case` inside `"$(...)"`.** `x="$(case a in a) echo "q'r";; esac)"` + was a syntax error, because the pattern's `)` closed the substitution + (#191). +- **`^C` inside `^R`, Alt-P or Alt-X on GNU readline 8.3.** Readline 8.3 + frees a search's state on SIGINT and expects the application to longjmp + out. hellish's reader returns instead, so the search freed it a second + time: a double free in a sanitizer build, and a crash on `^R` in a + release build. A filter in front of readline's handler now holds the + signal while a search is up and delivers it at the main loop (#194). + +Every change: +[v3.2.0...v3.3.0](https://github.com/Univers42/hellish/compare/v3.2.0...v3.3.0). + +--- + ## v3.2.0 — *Up finds what you typed, a hint is one key away, and `bind`* **The guide: [What's new in 3.2](https://univers42.github.io/hellish/whats-new/)**. diff --git a/USER_DOC.md b/USER_DOC.md index 63ff105c..d7c12a3b 100644 --- a/USER_DOC.md +++ b/USER_DOC.md @@ -169,6 +169,10 @@ escape hatches above get you back to your previous shell at any time. current directory (also `gmake`, `/usr/bin/make`, and behind `sudo`, `env X=1`, `&&`, `(`; a wrapper option that takes an argument, as in `sudo -u root make`, falls back to filenames). The file is parsed in the shell itself; make is never run. +Targets you have run (history lines starting with `make` or `gmake`) come first, +most-run (weighted toward recent runs) first; +the rest follow alphabetically. With `HELLISH_SUGGEST_STRATEGY=recent` the +run ones are ordered by how recently they ran. - Files read: the first regular file of `GNUmakefile`, `makefile`, `Makefile` (GNU make's order, symlinks followed), at most 4 MiB. No such file, a @@ -194,8 +198,9 @@ falls back to filenames). The file is parsed in the shell itself; make is never |---|---| | `HELLISH_BANNER=0\|1` | welcome panel off / on | | `HELLISH_HISTORY_PREFIX=0\|off` | Up/Down plain history; default is prefix search, read at every prompt | -| `HELLISH_AUTOSUGGEST=1\|on` | inline suggestion, off by default: the newest history line that starts with what you typed, shown dim after the cursor (see below); read at every prompt | +| `HELLISH_AUTOSUGGEST=1\|on` | inline suggestion, off by default: the best-ranked history line (see `HELLISH_SUGGEST_STRATEGY`) that starts with what you typed, shown dim after the cursor (see below); read at every prompt | | `HELLISH_SUGGEST_MAX=N` | how many of the newest history lines the suggestion looks at (default 1000; not a number, or below 1, is the default); read at every prompt | +| `HELLISH_SUGGEST_STRATEGY=recent` | rank suggestions newest first; by default (unset or any other value) they are ranked by frecency: how often a line was run, weighted toward recent runs. Also orders `make `. Read at every prompt | | `HELLISH_NO_UPDATE_CHECK=1` | never check for updates, no badge | | `HELLISH_NO_ANIM=1` | no startup animation | | `HELLISH_NO_EXEC=1` | user-install hook: skip for this one session | @@ -219,10 +224,10 @@ typed, and the hint is wiped from the screen. A word is the blanks at the front of the hint plus the run of characters up to the next space or tab (so `-m`, `a/b`, `x=1` are one word each). Alt-N and Alt-P step through distinct lines (same full text offered once), look -at no more than `HELLISH_SUGGEST_MAX` entries, ring the bell at the oldest -and the newest end instead of wrapping, and start over from the newest +at no more than `HELLISH_SUGGEST_MAX` entries, go in ranking order, ring the bell at the last and +the first instead of wrapping, and start over from the best match as soon as the line is edited. Right after Alt-N inserts the line -shown, not the newest. A hint a zle widget set as `POSTDISPLAY` is not +shown, not the best-ranked one. A hint a zle widget set as `POSTDISPLAY` is not cycled or taken by words. In vi insert mode Alt-F / Alt-N / Alt-P work the same (no Ctrl-Right). `unset HELLISH_AUTOSUGGEST` (or `=0`) turns it off at the next prompt. A key you bound yourself diff --git a/backlog.md b/backlog.md index 8fddf2a4..b419a455 100644 --- a/backlog.md +++ b/backlog.md @@ -9,6 +9,54 @@ pass counts in `bench/baseline/`). Performance claims come from ## Done (this program) +- [x] **`prompt_row_writes_test.py` counts rows, not write()s (#194).** Before + readline83-004 (Debian's 8.3-4 stops at 003) a multibyte prompt is + reprinted from column 0 at once, so on `ubuntu-26.04` one write() held + the row twice and the test reported 7 of 13 rows as split. None was: the + whole row was inside one write() every time, drawn twice. The redraw is + readline's and 004 ends it; the test now counts rows, so a row that is + really split still fails it. +- [x] **^C inside ^R, Alt-P or M-x no longer kills the shell on readline 8.3 + (#194).** 8.3's `rl_free_line_state` frees those readers' contexts on + SIGINT, on the bet that the application's handler longjmps. bash does, + and readline83-003 taught readline's own `rl_getc` to abort to the top + level for handlers that return; `rl_getc_hook` replaces that `rl_getc` + and hands the reader EOF, so the reader unwound through a freed `cxt`: + a double free under ASan, a SIGSEGV on a release build (3 of 3 on ^R), + and every PR red when `ubuntu-latest` moved to 26.04 (readline 8.3-4) + on an unchanged tree. Guarding only `rl_getc_hook`'s own look was tried + first and is not enough: readline also looks at its deferred signal at + the end of every `rl_redisplay`, and a ^C sent with the keys of a ^R + came through there. `rl_intr.c` now puts a filter in front of + readline's SIGINT handler: while such a reader is up the signal is + parked, the reader unwinds on EOF with its context intact, and the ^C + is replayed in the main loop. Outside those readers the filter only + forwards, so ^C anywhere else, a slow completion included, is unchanged. + `prompt_interrupt_test.py` gains nsearch, mx and redraw, + `rl_read_race_test.py` gains nsearch and M-x. 48 and 34 ok on 8.1.2, + 8.2.13 and 8.3 at patch levels 0, 3 and 6, and under a readline built + with ASan, where the old code reported a heap-use-after-free WRITE in + `_rl_search_getchar` for all three readers. +- [x] **A case pattern's `)` inside a quoted substitution is no longer a syntax + error (#191).** `advance_cmdsub` counted parens, so in + `x="$(case a in a) echo "q'r";; esac)"` the pattern's `)` closed the + substitution and the rest of the word was read out of step. It delegates to + `span_skip`, which walks these through `casescan` (#95), so the lexer and the + expander no longer hold two walkers that can disagree. +- [x] **A nested substitution with an odd count of quotes no longer leaves the + outer `$(...)` as text (#190).** `sh_skip_quoted` read a double-quoted span + flat, to the next `"`, so in `$(a "$(tr -d '"')" b)` it stopped on the inner + quote, the `'` after it opened a span with no end, and + `find_cmd_sub_end` reported the substitution as incomplete. The word kept + `$(a "..." b)` as literal text: no error, exit 0, the command never ran, and + `hellish -n` passed it. An even count balanced by accident, so + `$(a "$(b "$c")")` always worked and hid it. Found when a born2root + provisioner failed every first boot (hellish) and passed every retry (bash). + `sh_skip_quoted` now delegates to `span_skip`, which also fixes the `...` and + `${...}` neighbours, and `tests/cmdsub_nested_quote` compares 18 cases + against bash; 13 fail on 3.2.0. + A green `hellish -n` says nothing about this class: it is an expansion-time + scan, and it fails silently. - [x] **`"$VAR"` reaches the expander fast path: −19.5% loop instructions** (callgrind, loop_arith: 1,266.9M → 1,019.2M Ir). `try_simple_envvar` (`src/expander/expand_word2.c`) gated on `t->tt != TT_ENVVAR`, so the diff --git a/docker/Dockerfile b/docker/Dockerfile index 8fea9287..f7af3293 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -74,7 +74,7 @@ RUN set -eux; \ done; \ rm -rf /var/lib/apt/lists/*; \ elif command -v pacman >/dev/null 2>&1; then \ - pacman -Sy --noconfirm --needed base-devel readline git bash \ + pacman -Sy --noconfirm --needed --overwrite '*' base-devel readline git bash \ python clang; \ elif command -v dnf >/dev/null 2>&1; then \ dnf -y install gcc clang make readline-devel ncurses-devel git \ diff --git a/incs/history.h b/incs/history.h index 790b8a92..994fb8b8 100644 --- a/incs/history.h +++ b/incs/history.h @@ -15,6 +15,7 @@ # include "shell.h" # include +# include /* hist_cmds is the truth; readline's own list is a mirror kept in step so the arrow keys agree with what `history` prints. @@ -67,5 +68,10 @@ t_string encode_cmd_hist(char *cmd); void manage_history(t_shell *state); bool worthy_of_being_remembered(t_shell *state); void add_history_line(t_shell *state, const char *cmd); +/* Frecency (history_frecency.c). */ +uint32_t hist_weight(size_t age); +uint64_t hist_score_add(uint64_t score, size_t len, size_t idx, + bool recent); +size_t hist_window_floor(size_t len, int max); #endif \ No newline at end of file diff --git a/incs/prompt.h b/incs/prompt.h index 8eafbe4d..802621bc 100644 --- a/incs/prompt.h +++ b/incs/prompt.h @@ -108,8 +108,10 @@ typedef struct s_rl char *sg_post; /* POSTDISPLAY a widget set (zle_rl5.c) */ char *sg_post_at; /* the line it was set for */ char *sg_pub; /* what a widget was handed as POSTDISPLAY */ - t_vec sg_cyc; /* history indices offered (size_t) */ - size_t sg_cyc_k; /* which of them is on screen */ + bool sg_recent; /* HELLISH_SUGGEST_STRATEGY=recent */ + t_vec sg_rank; /* ranked history indices (size_t) */ + t_vec sg_scr; /* ranking scratch (t_sg_ent), reused */ + size_t sg_cyc_k; /* which of sg_rank is on screen */ char *sg_cyc_line; /* line the cycle is for; NULL: none */ bool sg_painted; /* a ghost is on screen right now */ int sg_col; /* column it starts at */ diff --git a/incs/shell.h b/incs/shell.h index 29186cb2..1e2d2d84 100644 --- a/incs/shell.h +++ b/incs/shell.h @@ -332,6 +332,7 @@ typedef struct s_make_cache long long mtime_s; long long mtime_ns; t_vec targets; /* char *, xmalloc'd */ + t_vec order; /* const char *, targets ranked for this TAB */ } t_make_cache; typedef struct s_shell diff --git a/incs/version.h b/incs/version.h index 9a6bf8a5..aa881816 100644 --- a/incs/version.h +++ b/incs/version.h @@ -15,7 +15,7 @@ /* The single source of truth for the running shell's version. Bumped in step with the git tag / GitHub release / npm package / docker image. */ -# define HELLISH_VERSION "3.2.0" +# define HELLISH_VERSION "3.3.0" /* Where releases live; used by the `update` builtin and the daily check. */ # define HELLISH_REPO "Univers42/hellish" @@ -30,6 +30,6 @@ /* Bumped whenever the welcome header itself changes what it says. The header is shown once a day, but a new revision is new information, so it is shown again immediately rather than waiting for tomorrow. */ -# define HELLISH_HEADER_REV 4 +# define HELLISH_HEADER_REV 5 #endif diff --git a/mkdocs.yml b/mkdocs.yml index b264d002..7c2d7f7c 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -55,7 +55,7 @@ markdown_extensions: nav: - Home: index.md - - What's new in 3.2: whats-new.md + - What's new in 3.3: whats-new.md - What it is + Install: product.md - The manual: manual.md - Builtins reference: diff --git a/npm/package.json b/npm/package.json index 7421118f..08bfaa1d 100644 --- a/npm/package.json +++ b/npm/package.json @@ -1,6 +1,6 @@ { "name": "hellish-shell", - "version": "3.2.0", + "version": "3.3.0", "description": "hellish — a fast, POSIX-compliant shell, with horns", "bin": { "hellish": "bin/hellish.js" diff --git a/src/completion/complete_make.c b/src/completion/complete_make.c index c9601da2..f0f7cd32 100644 --- a/src/completion/complete_make.c +++ b/src/completion/complete_make.c @@ -34,6 +34,9 @@ ** TAB. LIMITATION: a change to an INCLUDED file alone does not invalidate ** it -- touch the Makefile. ** +** ORDER. Targets used in history come first (frecency, see +** complete_make_rank.c), then the rest alphabetically. +** ** WHEN IT STAYS OUT. A word starting with '-' or holding '=' or '/' is not ** a target; and if an earlier argument is -C, --directory*, -f, --file* or ** --makefile*, the Makefile is not the one in the cwd, so filenames win. @@ -54,12 +57,12 @@ static char *mk_generator(const char *text, int state) t_vec *v; char *s; - v = &(*zle_state_cell())->make_cache.targets; + v = &(*zle_state_cell())->make_cache.order; if (!state) i = 0; while (i < v->len) { - s = ((char **)v->ctx)[i++]; + s = (char *)((const char **)v->ctx)[i++]; if (!ft_strncmp(s, text, ft_strlen(text))) return (rl_dup(s)); } @@ -90,11 +93,21 @@ static int mk_diverts(t_pv_line *a) return (0); } +/* The matches come out of mk_generator already in the order worth + showing (mk_order), so readline must not sort them again: it does that + AFTER this returns, so the flag is cleared here and cmd_completion sets + it back at the start of every attempt (completion.c). */ char **comp_make(t_shell *st, const char *text, t_pv_line *a) { + char **m; + if (text[0] == '-' || ft_strchr(text, '=') || ft_strchr(text, '/')) return (NULL); if (mk_diverts(a) || !mk_load(st)) return (NULL); - return (rl_completion_matches(text, mk_generator)); + mk_order(st); + m = rl_completion_matches(text, mk_generator); + if (m) + rl_sort_completion_matches = 0; + return (m); } diff --git a/src/completion/complete_make.h b/src/completion/complete_make.h index b4cf7865..53aa0109 100644 --- a/src/completion/complete_make.h +++ b/src/completion/complete_make.h @@ -49,6 +49,25 @@ typedef struct s_mk_scan int def; } t_mk_scan; +/* A target while ranking by history: how much it was used, and where. */ +typedef struct s_mk_use +{ + const char *name; + uint64_t score; + size_t newest; + size_t seen; +} t_mk_use; + +/* What counting needs besides the targets: the history length (for age) + and the strategy. */ +typedef struct s_mk_rank +{ + t_mk_use *u; + size_t cnt; + size_t len; + bool recent; +} t_mk_rank; + # define MK_MAX 4194304 /* The nanosecond mtime field is st_mtim on POSIX 2008, st_mtimespec on @@ -62,11 +81,17 @@ typedef struct s_mk_scan size_t pc_cmd_off(int start); size_t prov_word(t_pv_line *p, const char **w); size_t prov_base(const char *w, size_t n); +bool prov_is_make(const char *w, size_t n); char **comp_provider_try(const char *text, int start, int end); char **comp_make(t_shell *st, const char *text, t_pv_line *a); char *rl_dup(const char *s); t_vec *mk_load(t_shell *st); void mk_cache_free(t_shell *st); +void mk_order(t_shell *st); +int mk_cmp_use(const void *x, const void *y); +int mk_cmp_name(const void *x, const void *y); +void mk_count_entry(t_mk_rank *r, const char *e, size_t idx); +bool mk_is_make_entry(const char *e); char *mk_slurp(const char *path, size_t *len); void mk_scan(t_mk_scan *m, const char *buf, size_t len); void mk_line(t_mk_scan *m, char *l); diff --git a/src/completion/complete_make_cache.c b/src/completion/complete_make_cache.c index 26cc3dc7..2def35de 100644 --- a/src/completion/complete_make_cache.c +++ b/src/completion/complete_make_cache.c @@ -44,6 +44,7 @@ void mk_clear(t_make_cache *c) while (i < c->targets.len) xfree(((char **)c->targets.ctx)[i++]); xfree(c->targets.ctx); + xfree(c->order.ctx); xfree(c->name); ft_bzero(c, sizeof(*c)); } diff --git a/src/completion/complete_make_rank.c b/src/completion/complete_make_rank.c new file mode 100644 index 00000000..67d93c43 --- /dev/null +++ b/src/completion/complete_make_rank.c @@ -0,0 +1,130 @@ +/* ************************************************************************** */ +/* */ +/* ::: :::::::: */ +/* complete_make_rank.c :+: :+: :+: */ +/* +:+ +:+ +:+ */ +/* By: dlesieur +#+ +:+ +#+ */ +/* +#+#+#+#+#+ +#+ */ +/* Created: 2026/10/17 12:00:00 by dlesieur #+# #+# */ +/* Updated: 2026/10/17 12:00:00 by dlesieur ### ########.fr */ +/* */ +/* ************************************************************************** */ + +#include "complete_make.h" +#include + +/* make ordering: the targets the history shows being used come +** first, best frecency first (history_frecency.c, the ranking the inline +** suggestion uses); the rest follow alphabetically. +** +** An entry counts for a target when its FIRST word is a command the +** provider table maps to comp_make (make, gmake, any path to them) and +** the target is one of its whitespace-separated words, once per entry. +** +** Caveat: this reads words, not make's command line. `make -C re test` +** credits a target named `re` for the directory, and in `make a && make b` +** the `b` counts because the entry starts with make: a target can rank +** higher than it deserves, but is never hidden. `sudo make x` is not +** counted, and a word with `;`, a quote or a backslash glued on never +** equals a target name, so those under-credit. HELLISH_SUGGEST_STRATEGY= +** recent, or an alphabetical list, is the way out. */ + +/* Alphabetical, the way readline sorts when it is allowed to (strcoll), + so an unused-only list looks exactly as it always did. */ +int mk_cmp_name(const void *x, const void *y) +{ + const t_mk_use *a; + const t_mk_use *b; + + a = x; + b = y; + return (strcoll(a->name, b->name)); +} + +/* Used targets first: best score, then newest; ties, and every unused + target (score 0), by name. */ +int mk_cmp_use(const void *x, const void *y) +{ + const t_mk_use *a; + const t_mk_use *b; + + a = x; + b = y; + if (a->score != b->score && a->score < b->score) + return (1); + if (a->score != b->score) + return (-1); + if (a->newest != b->newest && a->newest < b->newest) + return (1); + if (a->newest != b->newest) + return (-1); + return (mk_cmp_name(x, y)); +} + +/* Is the entry a make invocation: its first word names the make provider + (the same table row that decides `make ` itself). */ +bool mk_is_make_entry(const char *e) +{ + size_t n; + size_t b; + + while (*e == ' ' || *e == '\t') + e++; + n = 0; + while (e[n] && e[n] != ' ' && e[n] != '\t') + n++; + b = prov_base(e, n); + return (prov_is_make(e + b, n - b)); +} + +/* Binary search of the name-sorted `u` for the word w[0..n); NULL if it + is not a target. */ +static t_mk_use *mk_find(t_mk_use *u, size_t cnt, const char *w, size_t n) +{ + size_t lo; + size_t hi; + size_t mid; + int c; + + lo = 0; + hi = cnt; + while (lo < hi) + { + mid = lo + (hi - lo) / 2; + c = ft_strncmp(u[mid].name, w, n); + if (!c && u[mid].name[n]) + c = 1; + if (!c) + return (&u[mid]); + if (c < 0) + lo = mid + 1; + else + hi = mid; + } + return (NULL); +} + +/* Credit entry `idx` (text `e`) to every distinct target among its words. + `seen` stops a word repeated in one entry from counting twice. */ +void mk_count_entry(t_mk_rank *r, const char *e, size_t idx) +{ + t_mk_use *t; + size_t n; + + while (*e) + { + while (*e == ' ' || *e == '\t') + e++; + n = 0; + while (e[n] && e[n] != ' ' && e[n] != '\t') + n++; + t = mk_find(r->u, r->cnt, e, n); + if (t && n && t->seen != idx + 1) + { + t->seen = idx + 1; + t->score = hist_score_add(t->score, r->len, idx, r->recent); + t->newest = idx; + } + e += n; + } +} diff --git a/src/completion/complete_make_rank2.c b/src/completion/complete_make_rank2.c new file mode 100644 index 00000000..7e8cb98b --- /dev/null +++ b/src/completion/complete_make_rank2.c @@ -0,0 +1,80 @@ +/* ************************************************************************** */ +/* */ +/* ::: :::::::: */ +/* complete_make_rank2.c :+: :+: :+: */ +/* +:+ +:+ +:+ */ +/* By: dlesieur +#+ +:+ +#+ */ +/* +#+#+#+#+#+ +#+ */ +/* Created: 2026/10/17 12:00:00 by dlesieur #+# #+# */ +/* Updated: 2026/10/17 12:00:00 by dlesieur ### ########.fr */ +/* */ +/* ************************************************************************** */ + +#include "complete_make.h" +#include +#include + +/* Fill st->make_cache.order with the cached targets in the order + `make ` offers them (complete_make_rank.c): used first, by + frecency, then the rest alphabetically. */ + +/* One record per target, sorted by name so that words can be looked up + by binary search. Returns NULL for no targets or no memory. */ +static t_mk_use *mk_records(t_vec *targets) +{ + t_mk_use *u; + size_t i; + + if (!targets->len) + return (NULL); + u = xmalloc(targets->len * sizeof(*u)); + i = 0; + while (i < targets->len) + { + u[i] = (t_mk_use){((char **)targets->ctx)[i], 0, 0, 0}; + i++; + } + qsort(u, targets->len, sizeof(*u), mk_cmp_name); + return (u); +} + +/* Walk the window of the newest HELLISH_SUGGEST_MAX history entries, the + same one the inline suggestion looks at, crediting make invocations. + Cost: O(window * words * log targets) per TAB. */ +static void mk_credit(t_shell *st, t_mk_rank *r) +{ + char **a; + size_t i; + + a = (char **)st->hist.hist_cmds.ctx; + i = hist_window_floor(r->len, st->rl.sg_max); + while (i < r->len) + { + if (a[i] && mk_is_make_entry(a[i])) + mk_count_entry(r, a[i], i); + i++; + } +} + +/* The ranked order, as borrowed pointers into the target list. It is + rebuilt on every TAB: the history moves on, the targets rarely do. */ +void mk_order(t_shell *st) +{ + t_mk_rank r; + size_t i; + + st->make_cache.order.len = 0; + st->make_cache.order.elem_size = sizeof(const char *); + r.cnt = st->make_cache.targets.len; + r.u = mk_records(&st->make_cache.targets); + r.len = st->hist.hist_cmds.len; + r.recent = st->rl.sg_recent; + if (!r.u) + return ; + mk_credit(st, &r); + qsort(r.u, r.cnt, sizeof(*r.u), mk_cmp_use); + i = 0; + while (i < r.cnt) + vec_push(&st->make_cache.order, &r.u[i++].name); + xfree(r.u); +} diff --git a/src/completion/complete_provider.c b/src/completion/complete_provider.c index 270b7e89..8c57493d 100644 --- a/src/completion/complete_provider.c +++ b/src/completion/complete_provider.c @@ -75,6 +75,14 @@ static t_prov_fn prov_find(const char *w, size_t n) return (NULL); } +/* Does the command named by w[0..n) belong to the make provider (make, + gmake)? The table is the one place that says so; the history ranking in + complete_make_rank.c asks instead of keeping a list of its own. */ +bool prov_is_make(const char *w, size_t n) +{ + return (prov_find(w, n) == comp_make); +} + /* Dispatch an argument word to the provider of its command, or NULL. */ char **comp_provider_try(const char *text, int start, int end) { diff --git a/src/completion/completion.c b/src/completion/completion.c index f9025ba4..d285bed4 100644 --- a/src/completion/completion.c +++ b/src/completion/completion.c @@ -101,13 +101,19 @@ char *rl_dup_dollar(const char *name, size_t len) Only when no spec exists does a built-in provider get a turn (make targets, complete_make.c), so a user's `complete` always wins; a provider - that has nothing to offer answers NULL and the filenames come back. */ + that has nothing to offer answers NULL and the filenames come back. + + readline sorts the matches after this returns (postprocess_matches), so + a provider that returns an order of its own (make) can only switch the + sort off for that attempt; the flag is therefore set back on here, for + every attempt, or a later filename or command TAB would stay unsorted. */ static char **cmd_completion(const char *text, int start, int end) { char **spec; rl_completion_append_character = ' '; rl_completion_suppress_append = 0; + rl_sort_completion_matches = 1; if (text[0] == '$' && !ft_strchr(text, '/')) return (rl_attempted_completion_over = 1, complete_variables(text, start, end)); diff --git a/src/helpers/free_utils.c b/src/helpers/free_utils.c index 8953f77a..eac090bc 100644 --- a/src/helpers/free_utils.c +++ b/src/helpers/free_utils.c @@ -112,6 +112,10 @@ static void free_session_strings(t_shell *state) state->dft_ctx = 0; xfree(state->rl.buff.ctx); xfree(state->rl.rp_txt.ctx); + xfree(state->rl.sg_rank.ctx); + xfree(state->rl.sg_scr.ctx); + state->rl.sg_rank = (t_vec){}; + state->rl.sg_scr = (t_vec){}; xfree(state->hist.file); state->hist.file = NULL; if (state->path_dirs) diff --git a/src/helpers/var_name.c b/src/helpers/var_name.c index c0ff955f..8156acaf 100644 --- a/src/helpers/var_name.c +++ b/src/helpers/var_name.c @@ -1,57 +1,57 @@ -/* ************************************************************************** */ -/* */ -/* ::: :::::::: */ -/* var_name.c :+: :+: :+: */ -/* +:+ +:+ +:+ */ -/* By: dlesieur +#+ +:+ +#+ */ -/* +#+#+#+#+#+ +#+ */ -/* Created: 2026/01/20 14:25:44 by dlesieur #+# #+# */ -/* Updated: 2026/01/20 14:25:44 by dlesieur ### ########.fr */ -/* */ -/* ************************************************************************** */ - -#include -#include - -/* POSIX says a variable name starts with [a-zA-Z_]. The unsigned char cast - is required by the C standard: isalpha on a plain char would be UB for - chars with the high bit set (e.g. UTF-8 bytes). */ -bool is_var_name_p1(char c) -{ - return (isalpha((unsigned char)c) || c == '_'); -} - -/* Subsequent characters extend to [a-zA-Z0-9_]. Same unsigned cast for the - same reason. The two-function split (p1 / p2) lets callers cheaply detect - whether they're looking at a valid identifier start vs continuation without - duplicating the is-digit check. */ -bool is_var_name_p2(char c) -{ - return (isalnum((unsigned char)c) || c == '_'); -} - -/* Advance past a quoted region or a backslash escape starting at s[i], - so paren-matching scans (the $(...) boundary hunt in both the reparser - and the expander) never count parens that live inside quotes — a sed - script like 's,[-(],,' must not derail the depth. Inside double quotes - a backslash still escapes the next character. Returns the index after - the skipped span, or i unchanged when s[i] starts no quoted span. */ -int sh_skip_quoted(const char *s, int len, int i) -{ - char q; - - if (s[i] == '\\' && i + 1 < len) - return (i + 2); - if (s[i] != '\'' && s[i] != '"') - return (i); - q = s[i++]; - while (i < len && s[i] != q) - { - if (q == '"' && s[i] == '\\' && i + 1 < len) - i++; - i++; - } - if (i < len) - i++; - return (i); -} +/* ************************************************************************** */ +/* */ +/* ::: :::::::: */ +/* var_name.c :+: :+: :+: */ +/* +:+ +:+ +:+ */ +/* By: dlesieur +#+ +:+ +#+ */ +/* +#+#+#+#+#+ +#+ */ +/* Created: 2026/01/20 14:25:44 by dlesieur #+# #+# */ +/* Updated: 2026/01/20 14:25:44 by dlesieur ### ########.fr */ +/* */ +/* ************************************************************************** */ + +#include +#include +#include "helpers.h" + +/* POSIX says a variable name starts with [a-zA-Z_]. The unsigned char cast + is required by the C standard: isalpha on a plain char would be UB for + chars with the high bit set (e.g. UTF-8 bytes). */ +bool is_var_name_p1(char c) +{ + return (isalpha((unsigned char)c) || c == '_'); +} + +/* Subsequent characters extend to [a-zA-Z0-9_]. Same unsigned cast for the + same reason. The two-function split (p1 / p2) lets callers cheaply detect + whether they're looking at a valid identifier start vs continuation without + duplicating the is-digit check. */ +bool is_var_name_p2(char c) +{ + return (isalnum((unsigned char)c) || c == '_'); +} + +/* Advance past a quoted region or a backslash escape starting at s[i], + so paren-matching scans (the $(...) boundary hunt in both the reparser + and the expander) never count parens that live inside quotes -- a sed + script like 's,[-(],,' must not derail the depth. span_skip owns the + walk, because a double-quoted span is not flat: its $(...), ${...} and + `...` hold quotes of their own. Read flat, "$(tr -d '"')" ended at the + inner `"`, the `'` after it opened a span with no end, and the whole + outer substitution stayed in the word as literal text, with no error. + An even count of quote bytes inside balanced by accident and hid it. + Returns the index after the skipped span, or i unchanged when s[i] + starts no quoted span. An unterminated span ends the scan at len. */ +int sh_skip_quoted(const char *s, int len, int i) +{ + int end; + + if (s[i] == '\\' && i + 1 < len) + return (i + 2); + if (s[i] != '\'' && s[i] != '"') + return (i); + end = span_skip(s, i, len); + if (end < 0) + return (len); + return (end); +} diff --git a/src/infrastructure/README.md b/src/infrastructure/README.md index 22ac9e3d..9c092cc6 100644 --- a/src/infrastructure/README.md +++ b/src/infrastructure/README.md @@ -132,6 +132,11 @@ used to contain is now handled in place: tick on its timeout. A `SIGINT` readline handled during the read makes it return `READERR`, readline's way to end its main loop, so `readline()` returns `NULL` and the read reports "interrupted" (`status == 2`). +- A `SIGINT` while `^R`, `Alt-P` or `M-x` is up never reaches readline: + 8.3 frees those readers' contexts on `SIGINT`, expecting the handler to + longjmp, and this reader returns instead. A filter in front of readline's + handler parks the signal, the reader unwinds on `EOF`, and the `^C` is + replayed in the main loop (`rl_intr.c`). - Shell code run from the editor -- widgets, completion functions -- goes through `rl_shell_exec` (`rl_editor.c`): readline's signal handlers come down for the call, the terminal settings and `$?` are put back, and diff --git a/src/infrastructure/banner.c b/src/infrastructure/banner.c index dc8148a6..9b54426a 100644 --- a/src/infrastructure/banner.c +++ b/src/infrastructure/banner.c @@ -98,7 +98,7 @@ static void build_left(const char **l) " \xc2\xb7 a POSIX shell with bite"; l[i + 4] = cwd; l[i + 5] = ""; - l[i + 6] = "\033[38;5;245mthe 3.2 guide, with examples\033[0m"; + l[i + 6] = "\033[38;5;245mthe 3.3 guide, with examples\033[0m"; l[i + 7] = guide_link(); l[i + 8] = NULL; } diff --git a/src/infrastructure/banner2.c b/src/infrastructure/banner2.c index ece2aad6..5c1e0069 100644 --- a/src/infrastructure/banner2.c +++ b/src/infrastructure/banner2.c @@ -13,7 +13,7 @@ #include "banner_private.h" #include "shell.h" -/* One row of "What's new in 3.2": the keys that reach a new feature, in +/* One row of "What's new in 3.3": the keys that reach a new feature, in code-yellow, then what it does, in muted grey -- the shape of "Getting started", so each can be tried before the prompt even appears. Every row fits the 32 columns the right column has on an 80-column terminal. @@ -26,16 +26,12 @@ Returns 0 at the end of a group. */ static int news_row(char *dst, int n) { - static const char *utf8[] = {"\xe2\x86\x91 \xe2\x86\x93 ", - "make ", "bind -x ", "bindkey ", "${var@P} ", "$SRANDOM ", - NULL, "\xe2\x86\x92 ^F End ", "Alt-F ", "Alt-N/P ", NULL}; - static const char *ascii[] = {"Up Down ", "make ", "bind -x ", - "bindkey ", "${var@P} ", "$SRANDOM ", NULL, "Right ^F ", - "Alt-F ", "Alt-N/P ", NULL}; - static const char *what[] = {"history by prefix", "Makefile targets", - "key runs a command", "zsh binds now work", "render as a prompt", - "and $EPOCHREALTIME", NULL, "take the dim hint", "take one word", - "older/newer hint"}; + static const char *utf8[] = {"make ", NULL, + "\xe2\x86\x92 ^F End ", "Alt-N/P ", NULL}; + static const char *ascii[] = {"make ", NULL, "Right ^F ", + "Alt-N/P ", NULL}; + static const char *what[] = {"most-run first", NULL, + "take your most-run", "the next most-run"}; const char **keys; keys = ascii; @@ -57,7 +53,7 @@ static void news_block(const char **r, int *i, char nw[][192]) int m; r[(*i)++] = "\033[1;48;5;209;38;5;236m NEW \033[0m " - "\033[1;38;5;81mWhat's new in 3.2\033[0m"; + "\033[1;38;5;81mWhat's new in 3.3\033[0m"; r[(*i)++] = ""; k = 0; m = 0; @@ -74,7 +70,7 @@ static void news_block(const char **r, int *i, char nw[][192]) (shown once). */ void build_right(const char **r) { - static char nw[9][192]; + static char nw[3][192]; static char status[192]; int i; diff --git a/src/infrastructure/history_frecency.c b/src/infrastructure/history_frecency.c new file mode 100644 index 00000000..4cbaa227 --- /dev/null +++ b/src/infrastructure/history_frecency.c @@ -0,0 +1,67 @@ +/* ************************************************************************** */ +/* */ +/* ::: :::::::: */ +/* history_frecency.c :+: :+: :+: */ +/* +:+ +:+ +:+ */ +/* By: dlesieur +#+ +:+ +#+ */ +/* +#+#+#+#+#+ +#+ */ +/* Created: 2026/10/17 12:00:00 by dlesieur #+# #+# */ +/* Updated: 2026/10/17 12:00:00 by dlesieur ### ########.fr */ +/* */ +/* ************************************************************************** */ + +#include "shell.h" + +/* Frecency: how much a history entry is worth as a suggestion, from how +** often AND how recently its text was run. Shared by the inline ghost +** and Alt-N/P (platform/posix/rl_suggest8.c) and by `make ` +** (completion/complete_make_rank.c), so both rank the same way. +** +** An occurrence's weight starts at 2^16 and halves every FRE_HALF newer +** entries, down to 1 once it is 16 halvings old; a text scores the sum of +** its occurrences. The score does not depend on what is being typed, so +** the best match for a prefix stays the best for every longer prefix it +** still matches, which is what lets the ghost skip rescanning per key. +** +** Caveat: age is a count of entries, not time. A burst of 64 unrelated +** commands halves a habit's weight, while a command last used a year ago +** in a quiet shell still weighs as much as one used yesterday; and +** FRE_HALF = 64 is a guess, not a measurement. HELLISH_SUGGEST_MAX bounds +** the window, so nothing older than it counts at all. Escape hatch: +** HELLISH_SUGGEST_STRATEGY=recent ignores frequency entirely. */ +#define FRE_HALF 64 +#define FRE_STEPS 16 + +/* Weight of an entry `age` entries older than the newest. */ +uint32_t hist_weight(size_t age) +{ + size_t steps; + + steps = age / FRE_HALF; + if (steps > FRE_STEPS) + steps = FRE_STEPS; + return ((uint32_t)(1u << FRE_STEPS) >> steps); +} + +/* Fold the occurrence at index `idx` (of `len` entries) into `score`. + `recent` makes the score the newest index (+1, so 0 stays "unused"), + which orders newest first and collapses repeats. */ +uint64_t hist_score_add(uint64_t score, size_t len, size_t idx, + bool recent) +{ + if (recent) + { + if ((uint64_t)idx + 1 > score) + return ((uint64_t)idx + 1); + return (score); + } + return (score + hist_weight(len - 1 - idx)); +} + +/* First index of the window of the newest `max` entries. */ +size_t hist_window_floor(size_t len, int max) +{ + if (max > 0 && len > (size_t)max) + return (len - (size_t)max); + return (0); +} diff --git a/src/infrastructure/rl_private.h b/src/infrastructure/rl_private.h index ccece657..e0ed3783 100644 --- a/src/infrastructure/rl_private.h +++ b/src/infrastructure/rl_private.h @@ -97,6 +97,15 @@ typedef struct s_frame char *rl_suggest_update(t_shell *state, int cols); char *rl_suggest_live(t_rl *l); +/* One candidate while ranking: the text points into hist_cmds and is only + good for the call that filled it. */ +typedef struct s_sg_ent +{ + const char *text; + size_t idx; + uint64_t score; +} t_sg_ent; + char *rl_suggest_find(t_shell *state, t_rl *l); int rl_suggest_accept(int count, int key); t_shell *rl_suggest_ghost_state(void); @@ -104,7 +113,8 @@ int rl_suggest_bell(void); int rl_suggest_word(int count, int key); int rl_suggest_older(int count, int key); int rl_suggest_newer(int count, int key); -ssize_t rl_suggest_scan(t_shell *state, t_rl *l, size_t from); +size_t rl_suggest_cands(t_shell *state, t_rl *l); +size_t rl_suggest_rank(t_shell *state, t_rl *l); void rl_suggest_cyc_clear(t_rl *l); bool rl_suggest_cyc_sync(t_shell *state, t_rl *l); void rl_suggest_end(t_shell *state); @@ -135,6 +145,20 @@ bool rl_push_col(t_string *f, int n); # define RL_INPROC_OK 0 # endif +/* The readers that keep a heap context on their stack, the ones readline + 8.3 frees on SIGINT (rl_intr.c). The bit is spelled out for older + headers: a shared build runs on whatever libreadline.so.8 the machine + has, and before 8.3 the bit means nothing. */ +# ifndef RL_STATE_READSTR +# define RL_STATE_READSTR 0x10000000 +# endif +# define RL_CXT_READERS 0x10000180 + +/* rl_intr_cell: a ^C the filter kept from readline, not yet replayed. */ +# define RL_INTR_PARKED 2 + +typedef volatile sig_atomic_t t_intr; + /* SIGWINCH during a read (rl_winch.c, rl_winch2.c). readline's own resize path calls rl_forced_update_display() as soon as a redisplay function is installed -- and the RPROMPT painter installs one -- which appended a @@ -165,12 +189,13 @@ int rl_shell_leave(t_shell *state, t_rl_bracket *b); void rl_prerow_arm(t_string *rows); FILE *rl_out_open(void); int rl_getc_hook(FILE *stream); +void rl_take_signals(void); int rl_nsearch_reclaim(int c); int rl_idle_timeout(void); int rl_idle_fd(void); void rl_idle_event(int r); void rl_block(sigset_t *old); -int *rl_intr_cell(void); +t_intr *rl_intr_cell(void); int *rl_editor_abort_cell(void); bool rl_should_abort(void); int rl_abort_value(void); diff --git a/src/lexer/lexer_advance.c b/src/lexer/lexer_advance.c index 8d3ac087..441daa2f 100644 --- a/src/lexer/lexer_advance.c +++ b/src/lexer/lexer_advance.c @@ -1,146 +1,140 @@ -/* ************************************************************************** */ -/* */ -/* ::: :::::::: */ -/* lexer.c :+: :+: :+: */ -/* +:+ +:+ +:+ */ -/* By: marvin +#+ +:+ +#+ */ -/* +#+#+#+#+#+ +#+ */ -/* Created: 2026/01/09 23:33:55 by marvin #+# #+# */ -/* Updated: 2026/01/09 23:33:55 by marvin ### ########.fr */ -/* */ -/* ************************************************************************** */ - -#include "lexer.h" -#include - -static void advance_cmdsub(char **str); - -/* Skip a $(...) command substitution, honouring nested parens and quoted - segments (whose quotes are independent of any enclosing quote). Used so a - '"' inside $(...) does not prematurely close an outer double quote. */ -static void advance_cmdsub(char **str) -{ - int depth; - - *str += 2; - depth = 1; - while (**str && depth > 0) - { - if (**str == '\\') - advance_bs(str); - else if (**str == '\'') - advance_squoted(str); - else if (**str == '\"') - advance_dquoted(str); - else - { - depth += (**str == '('); - depth -= (**str == ')'); - (*str)++; - } - } -} - -/* Advance past a double-quoted span. Double quotes still expand $(...), - ${...} and `...`, so we must recurse into each of them — otherwise a `"` - that belongs to the nested construct is read as the outer quote's end - (autoconf: x="`... "" ...`"). Missing the ${...} case split - "${u:-"a b"}" at the space, because the quote after :- looked like the - closing one; the word came apart into two tokens and the reparser then - tripped its own closing-quote assertion. advance_brace_param recurses - back into here for a quote inside the braces, so the two stay in step. - The prev_bs flag handles `\"` escapes while avoiding a double-count - on `\\`. */ -int advance_dquoted(char **str) -{ - bool prev_bs; - - ft_assert(**str == '\"'); - (*str)++; - prev_bs = false; - while (**str && (**str != '\"' || prev_bs)) - { - if (!prev_bs && **str == '$' && (*str)[1] == '(') - advance_cmdsub(str); - else if (!prev_bs && **str == '$' && (*str)[1] == '{') - advance_brace_param(str, 1); - else if (!prev_bs && **str == '`') - advance_backtick(str); - else - { - prev_bs = **str == '\\' && !prev_bs; - (*str)++; - } - } - if (**str != '\"') - return (1); - (*str)++; - return (0); -} - -/* One unit of a ${...} body: an escaped byte, a quoted span, a nested - $( ) or `...`, a nested ${ (depth + 1), or one plain byte (a `}` is - depth - 1). The nested command substitutions are spans of their own: - bash prints } for ${x:-$(echo })}, and an unterminated one is an - unterminated ${...} -- ${x%$(} used to be accepted as literal text. */ -static void brace_step(char **str, int sq, int *depth) -{ - if (**str == '\\' && (*str)[1]) - (*str) += 2; - else if (**str == '\'' && sq) - advance_squoted(str); - else if (**str == '"') - advance_dquoted(str); - else if (**str == '`') - advance_backtick(str); - else if (**str == '$' && (*str)[1] == '(') - advance_cmdsub(str); - else if (**str == '$' && (*str)[1] == '{') - { - (*depth)++; - (*str) += 2; - } - else - { - *depth -= (**str == '}'); - (*str)++; - } -} - -/* Scan a ${...} parameter expansion as one span so spaces inside (e.g. - ${x:-a b c}) do not break the surrounding word. Honours nested braces and - quoted segments. in_dq says the expansion sits inside a double-quoted - word, where a ' is an ordinary character for the word operators -- bash - prints a'b for "${u:-a'b}" but reports an unterminated quote for - ${u:-a'b} -- and still a quote for the pattern ones: brace_sq_live. - A backslash escapes the next byte, so ${u-\"} does - not open a quoted section and ${u-\}} does not close the brace - early — the same rule the reparser's scan_brace_depth applies. It - has to be here too now that advance_dquoted recurses in: without it - the \" of "${u-\"}" read as the start of a nested quote and - swallowed the rest of the line. - - ONLY `${` NESTS. A bare `{` inside the expansion is an ordinary - character to bash, and counting it as an opener meant the closing `}` - paid off a brace that was never open, so the scan ran to end of input: - - echo ${x:-a{b} unexpected EOF, matching `}' - echo ${o%%[<{().[]*} the same, on a bracket expression - - The second is not a curiosity -- it is bash-completion's - _comp_compgen_help__parse, which is why 3800 lines of the most widely - installed completion framework in existence failed to parse at line 1757 - and defined nothing. `${x:-${y}}` still nests, because that is a real - opener. */ -int advance_brace_param(char **str, int in_dq) -{ - int depth; - int sq; - - *str += 2; - sq = brace_sq_live(*str, INT_MAX, in_dq); - depth = 1; - while (**str && depth > 0) - brace_step(str, sq, &depth); - return (depth != 0); -} +/* ************************************************************************** */ +/* */ +/* ::: :::::::: */ +/* lexer.c :+: :+: :+: */ +/* +:+ +:+ +:+ */ +/* By: marvin +#+ +:+ +#+ */ +/* +#+#+#+#+#+ +#+ */ +/* Created: 2026/01/09 23:33:55 by marvin #+# #+# */ +/* Updated: 2026/01/09 23:33:55 by marvin ### ########.fr */ +/* */ +/* ************************************************************************** */ + +#include "helpers.h" +#include "lexer.h" +#include + +static void advance_cmdsub(char **str); + +/* Skip a $(...) command substitution, whose quotes are independent of any + enclosing quote, so a '"' inside does not close an outer double quote. + span_skip owns the walk: it knows a case pattern's unbalanced `)` does not + end the span, which a plain paren count here did not, and + x="$(case a in a) echo "q'r";; esac)" was refused as a syntax error while + the same text with no quotes around it worked. An unterminated span stops + at the end of the input, for the prompt to ask for more. */ +static void advance_cmdsub(char **str) +{ + int len; + int end; + + len = (int)ft_strlen(*str); + end = span_skip(*str, 0, len); + if (end < 0) + end = len; + *str += end; +} + +/* Advance past a double-quoted span. Double quotes still expand $(...), + ${...} and `...`, so we must recurse into each of them — otherwise a `"` + that belongs to the nested construct is read as the outer quote's end + (autoconf: x="`... "" ...`"). Missing the ${...} case split + "${u:-"a b"}" at the space, because the quote after :- looked like the + closing one; the word came apart into two tokens and the reparser then + tripped its own closing-quote assertion. advance_brace_param recurses + back into here for a quote inside the braces, so the two stay in step. + The prev_bs flag handles `\"` escapes while avoiding a double-count + on `\\`. */ +int advance_dquoted(char **str) +{ + bool prev_bs; + + ft_assert(**str == '\"'); + (*str)++; + prev_bs = false; + while (**str && (**str != '\"' || prev_bs)) + { + if (!prev_bs && **str == '$' && (*str)[1] == '(') + advance_cmdsub(str); + else if (!prev_bs && **str == '$' && (*str)[1] == '{') + advance_brace_param(str, 1); + else if (!prev_bs && **str == '`') + advance_backtick(str); + else + { + prev_bs = **str == '\\' && !prev_bs; + (*str)++; + } + } + if (**str != '\"') + return (1); + (*str)++; + return (0); +} + +/* One unit of a ${...} body: an escaped byte, a quoted span, a nested + $( ) or `...`, a nested ${ (depth + 1), or one plain byte (a `}` is + depth - 1). The nested command substitutions are spans of their own: + bash prints } for ${x:-$(echo })}, and an unterminated one is an + unterminated ${...} -- ${x%$(} used to be accepted as literal text. */ +static void brace_step(char **str, int sq, int *depth) +{ + if (**str == '\\' && (*str)[1]) + (*str) += 2; + else if (**str == '\'' && sq) + advance_squoted(str); + else if (**str == '"') + advance_dquoted(str); + else if (**str == '`') + advance_backtick(str); + else if (**str == '$' && (*str)[1] == '(') + advance_cmdsub(str); + else if (**str == '$' && (*str)[1] == '{') + { + (*depth)++; + (*str) += 2; + } + else + { + *depth -= (**str == '}'); + (*str)++; + } +} + +/* Scan a ${...} parameter expansion as one span so spaces inside (e.g. + ${x:-a b c}) do not break the surrounding word. Honours nested braces and + quoted segments. in_dq says the expansion sits inside a double-quoted + word, where a ' is an ordinary character for the word operators -- bash + prints a'b for "${u:-a'b}" but reports an unterminated quote for + ${u:-a'b} -- and still a quote for the pattern ones: brace_sq_live. + A backslash escapes the next byte, so ${u-\"} does + not open a quoted section and ${u-\}} does not close the brace + early — the same rule the reparser's scan_brace_depth applies. It + has to be here too now that advance_dquoted recurses in: without it + the \" of "${u-\"}" read as the start of a nested quote and + swallowed the rest of the line. + + ONLY `${` NESTS. A bare `{` inside the expansion is an ordinary + character to bash, and counting it as an opener meant the closing `}` + paid off a brace that was never open, so the scan ran to end of input: + + echo ${x:-a{b} unexpected EOF, matching `}' + echo ${o%%[<{().[]*} the same, on a bracket expression + + The second is not a curiosity -- it is bash-completion's + _comp_compgen_help__parse, which is why 3800 lines of the most widely + installed completion framework in existence failed to parse at line 1757 + and defined nothing. `${x:-${y}}` still nests, because that is a real + opener. */ +int advance_brace_param(char **str, int in_dq) +{ + int depth; + int sq; + + *str += 2; + sq = brace_sq_live(*str, INT_MAX, in_dq); + depth = 1; + while (**str && depth > 0) + brace_step(str, sq, &depth); + return (depth != 0); +} diff --git a/src/platform/posix/rl_getc.c b/src/platform/posix/rl_getc.c index 18e4e886..5f159c24 100644 --- a/src/platform/posix/rl_getc.c +++ b/src/platform/posix/rl_getc.c @@ -25,7 +25,7 @@ ** is what bash does. This one can: a SIGINT that readline handled during ** the read, or any other reason to stop (rl_should_abort), makes it ** return READERR, readline's own "stop reading" answer (readline.c), and -** readline() then returns NULL. +** readline() then returns NULL. rl_intr.c is where the signal is taken. ** ** It is also where the prompt's idle work runs -- the mascot animation ** ticks on a timeout -- since an rl_event_hook would make readline stop @@ -37,24 +37,6 @@ ** second, non-blocking descriptor where there is one (rl_keyfd.c), so the ** only place this ever waits is pselect. */ -/* A signal readline caught since the last look: let readline handle it - now -- cleanup, the application's handler, re-prep -- and remember an - interrupt, which ends this read. */ -static void rl_take_signals(void) -{ - int sig; - - sig = rl_pending_signal(); - if (!sig) - return ; - if (sig == SIGINT) - { - *rl_intr_cell() = 1; - rl_suggest_unpaint(); - } - rl_check_signals(); -} - static struct timespec *rl_timeout(struct timespec *ts) { int ms; diff --git a/src/platform/posix/rl_intr.c b/src/platform/posix/rl_intr.c new file mode 100644 index 00000000..9f3d612a --- /dev/null +++ b/src/platform/posix/rl_intr.c @@ -0,0 +1,109 @@ +/* ************************************************************************** */ +/* */ +/* ::: :::::::: */ +/* rl_intr.c :+: :+: :+: */ +/* +:+ +:+ +:+ */ +/* By: dlesieur +#+ +:+ +#+ */ +/* +#+#+#+#+#+ +#+ */ +/* Created: 2026/11/04 00:00:00 by dlesieur #+# #+# */ +/* Updated: 2026/11/04 00:00:00 by dlesieur ### ########.fr */ +/* */ +/* ************************************************************************** */ + +#include "rl_private.h" +#include + +/* What a ^C does to a read, in one place. +** +** readline 8.3 made rl_free_line_state free the context of ^R, Alt-P and +** M-x, on the bet that an application's SIGINT handler longjmps and that +** reader never runs again. bash does; readline's own rl_getc, since +** readline83-003, aborts to the top level for the handlers that return. +** This read does neither, by design (rl_getc.c): the reader is handed EOF +** and unwinds itself, through a `cxt` local that now points at freed +** memory. Issue #194, from CI on readline 8.3-4: +** +** rl_getc_hook -> rl_check_signals -> _rl_handle_signal +** -> rl_free_line_state -> _rl_isearch_cleanup -> free(cxt) +** rl_search_history -> _rl_isearch_cleanup(cxt) -> free(cxt) +** +** A double free under ASan, a use after free under an instrumented +** readline for all three readers, and on a release build a SIGSEGV that +** takes the login shell, 3 runs of 3 on ^R. +** +** readline runs that cleanup wherever it next looks at its deferred +** signal, and the look at the end of every rl_redisplay is inside the +** search, so guarding this file's own look is not enough: a ^C sent while +** ^R is drawing its keys came through there, and the test that caught it +** sends one. So readline must not hear of a ^C while such a reader is up. +** A filter goes in front of its handler and parks the signal; the reader +** gets EOF and unwinds with its context intact, the path every readline +** before 8.3 took, and back in the main loop the ^C is replayed into +** readline, which handles it as one that arrived there. Outside these +** readers the filter only forwards, so ^C anywhere else is what it always +** was, a slow completion included. +** +** Caveat: the filter reads the state bit, and the bit goes up only once +** the reader's command runs. A ^C landing after readline's last look at +** the signal and before that (a few hundred instructions, no system call +** in them) reaches readline as usual and frees the context at the next +** look. Blocking SIGINT from the key's return until the next wait would +** close it, but the block would need lifting around every widget and +** completion, and a child forked on a path that misses it inherits it. A +** context a later readline adds to _rl_state_sigcleanup is not in +** RL_CXT_READERS until someone puts it there. */ + +static struct sigaction *rl_readline_sigint(void) +{ + static struct sigaction act; + + return (&act); +} + +static void rl_sigint_filter(int sig) +{ + if (rl_readline_state & RL_CXT_READERS) + *rl_intr_cell() = RL_INTR_PARKED; + else + rl_readline_sigint()->sa_handler(sig); +} + +/* rl_set_signals installs readline's handler at the top of readline() and + again after a signal it handled or a widget's bracket (rl_editor.c), so + this looks every time. Its restore puts back what it saved then, never + the filter. A SIG_DFL here is the forked reader's child in its cleanup: + nothing to stand in front of. */ +static void rl_sigint_arm(void) +{ + struct sigaction act; + + if (sigaction(SIGINT, NULL, &act) != 0 + || act.sa_handler == rl_sigint_filter + || act.sa_handler == SIG_DFL || act.sa_handler == SIG_IGN) + return ; + *rl_readline_sigint() = act; + act.sa_handler = rl_sigint_filter; + sigaction(SIGINT, &act, NULL); +} + +/* A signal since the last look: let readline handle it now -- cleanup, + the application's handler, re-prep -- and remember an interrupt, which + ends this read. A parked ^C waits for its reader to be gone, and while + it waits rl_should_abort is what makes the reader go. */ +void rl_take_signals(void) +{ + int sig; + + if (*rl_intr_cell() == RL_INTR_PARKED + && !(rl_readline_state & RL_CXT_READERS)) + rl_readline_sigint()->sa_handler(SIGINT); + sig = rl_pending_signal(); + if (sig == SIGINT) + { + *rl_intr_cell() = 1; + rl_suggest_unpaint(); + } + if (sig) + rl_check_signals(); + rl_sigint_arm(); +} diff --git a/src/platform/posix/rl_sig.c b/src/platform/posix/rl_sig.c index a4b22b69..749a4fa7 100644 --- a/src/platform/posix/rl_sig.c +++ b/src/platform/posix/rl_sig.c @@ -15,12 +15,12 @@ /* Why a read ends early. */ -/* Set when readline handled a SIGINT during the current read. Written - from the reader, never from a signal handler: readline defers the - signal and our reader is where it is handled (rl_take_signals). */ -int *rl_intr_cell(void) +/* Set when a SIGINT reached this read. Written from the reader, which is + where readline's deferred signal is handled, and from the one handler + of ours in front of readline's (rl_intr.c). */ +t_intr *rl_intr_cell(void) { - static int intr; + static t_intr intr; return (&intr); } diff --git a/src/platform/posix/rl_slot_env.c b/src/platform/posix/rl_slot_env.c index e1375d37..2cfc6a6b 100644 --- a/src/platform/posix/rl_slot_env.c +++ b/src/platform/posix/rl_slot_env.c @@ -64,6 +64,18 @@ static int sg_limit(t_shell *state) return (ft_atoi(e->value)); } +/* HELLISH_SUGGEST_STRATEGY: how suggestions are ranked. Exactly `recent` + is newest first; unset, empty or anything else (typos included) is the + default, frecency, so a misspelt value never silently changes order to + the old one. */ +static bool sg_wants_recent(t_shell *state) +{ + t_env *e; + + e = env_get(&state->env, "HELLISH_SUGGEST_STRATEGY"); + return (e && e->value && !ft_strcmp(e->value, "recent")); +} + /* Interactive reads only, right before rl_preinit: note what is wanted and ** switch the slot hooks on. Doing it inside rl_preinit's caller keeps ** `bind -p` (which also runs rl_preinit, in -c mode) exactly as bash's. @@ -76,4 +88,5 @@ void rl_slot_sync(t_shell *state) state->rl.feat_want[FEAT_HIST] = hp_wanted(state); state->rl.feat_want[FEAT_ACCEPT] = as_wanted(state); state->rl.sg_max = sg_limit(state); + state->rl.sg_recent = sg_wants_recent(state); } diff --git a/src/platform/posix/rl_suggest.c b/src/platform/posix/rl_suggest.c index c74a0a5d..44f61c48 100644 --- a/src/platform/posix/rl_suggest.c +++ b/src/platform/posix/rl_suggest.c @@ -12,7 +12,8 @@ #include "rl_private.h" -/* The inline suggestion: the newest history entry that starts with what +/* The inline suggestion: the best-ranked history entry (frecency, or the +** newest with HELLISH_SUGGEST_STRATEGY=recent) that starts with what ** is typed, shown dim after the cursor and never part of the line. This ** file decides WHAT may be suggested and WHEN; rl_suggest2.c turns it into ** the text and rl_rprompt.c paints it. @@ -23,16 +24,16 @@ ** is a C1 control in its UTF-8 form: running `printf '\e[31m'` once must ** not repaint every later prompt. */ -/* The newest of the last sg_max entries that starts with the first `n` - bytes of the line, is longer than it and is clean (rl_suggest_scan). */ +/* The best-ranked of the last sg_max entries that start with the line, are + longer than it and are clean (rl_suggest_rank), or NULL. */ static char *sg_scan(t_shell *state, t_rl *l) { - ssize_t i; + char **a; - i = rl_suggest_scan(state, l, state->hist.hist_cmds.len); - if (i < 0) + if (!rl_suggest_rank(state, l)) return (NULL); - return (((char **)state->hist.hist_cmds.ctx)[i]); + a = (char **)state->hist.hist_cmds.ctx; + return (a[*(size_t *)l->sg_rank.ctx]); } /* True when the last scan still answers for the line: the line extends the @@ -54,9 +55,12 @@ static bool sg_cached(t_rl *l, size_t n) ** Typing extends the line, and an entry that matches the longer line ** matched the shorter one too -- so when the line still starts with the one ** the last scan was for, the answer is the same entry as long as it still -** matches (or "nothing" for good). That makes a key O(line) with no scan, -** and a paste of 4 KB one scan instead of 4096. Anything else (a -** Backspace, a recalled line) scans again. The history cannot change under +** matches (or "nothing" for good). That holds for the frecency ranking as +** well: an entry's score does not depend on the line, so the winner for +** the shorter line, still matching, is the best of the smaller set too. +** That makes a key O(line) with no scan, and a paste of 4 KB one scan +** instead of 4096. Anything else (a Backspace, a recalled line) scans +** again. The history cannot change under ** it: it only grows between reads, and the cache is dropped per read. */ char *rl_suggest_find(t_shell *state, t_rl *l) { diff --git a/src/platform/posix/rl_suggest4.c b/src/platform/posix/rl_suggest4.c index 1dd645ab..5703af41 100644 --- a/src/platform/posix/rl_suggest4.c +++ b/src/platform/posix/rl_suggest4.c @@ -53,7 +53,9 @@ void rl_suggest_stale(void) /* The read is over (or about to start): nothing of the ghost, of the search it was found with or of a widget's POSTDISPLAY outlives it. The - history may have changed since, so the cache cannot either. */ + history may have changed since, so the cache cannot either. The ranking + buffers are freed here too: they are reused by every key of one read, + never kept between reads. */ void rl_suggest_end(t_shell *state) { t_rl *l; @@ -65,6 +67,10 @@ void rl_suggest_end(t_shell *state) xfree(l->sg_post_at); xfree(l->sg_pub); rl_suggest_cyc_clear(l); + xfree(l->sg_rank.ctx); + xfree(l->sg_scr.ctx); + l->sg_rank = (t_vec){}; + l->sg_scr = (t_vec){}; l->sg_ent = NULL; l->sg_prev = NULL; l->sg_post = NULL; diff --git a/src/platform/posix/rl_suggest5.c b/src/platform/posix/rl_suggest5.c index 484134c1..d59204ee 100644 --- a/src/platform/posix/rl_suggest5.c +++ b/src/platform/posix/rl_suggest5.c @@ -12,11 +12,10 @@ #include "rl_private.h" -/* Scanning the history for suggestions, and the state of Alt-N / Alt-P -** cycling. A cycle remembers the history INDICES it has shown (sg_cyc, a -** stack: Alt-N pushes the next older distinct one, Alt-P pops back), so -** going back to a newer alternative costs nothing and a revisit never -** re-scans. */ +/* Collecting suggestion candidates from the history, and the state of +** Alt-N / Alt-P cycling. A cycle is the ranking (rl_suggest_rank) made +** once, as history INDICES in sg_rank, and a position in it: Alt-N and +** Alt-P only move the position, so nothing is ever scanned again. */ /* True when `s` holds nothing a terminal would interpret. 0x7f is DEL; the pair 0xC2 0x80..0x9F is U+0080..U+009F, which some terminals read as a @@ -37,63 +36,43 @@ static bool sg_clean(const unsigned char *s) return (true); } -/* True when an entry with the same full text was already offered in this - cycle. Distinct means distinct text: history keeps duplicates, and - stepping through three copies of one command is not a choice. */ -static bool sg_offered(t_shell *state, t_rl *l, const char *text) -{ - char **a; - size_t *v; - size_t i; - - a = (char **)state->hist.hist_cmds.ctx; - v = (size_t *)l->sg_cyc.ctx; - i = 0; - while (i < l->sg_cyc.len) - { - if (!ft_strcmp(a[v[i]], text)) - return (true); - i++; - } - return (false); -} - -/* The index of the next older entry below `from` that starts with the -** line, is longer than it, is clean and was not offered yet; -1 if none. -** Only the newest sg_max entries are looked at (HELLISH_SUGGEST_MAX), so a -** huge history cannot slow a key down. +/* Fill the scratch with every candidate of the window of the newest +** sg_max entries: starts with the line, is longer than it, is clean. +** Returns how many. The scratch and its capacity are kept for the next +** call (a key never allocates once it has grown), and the texts point into +** the history, so they are good until the history changes. ** -** Cost: O(sg_max * (line + offered)) per call, offered being the entries -** shown so far in this cycle -- a handful in practice. */ -ssize_t rl_suggest_scan(t_shell *state, t_rl *l, size_t from) +** Cost: O(sg_max * line). */ +size_t rl_suggest_cands(t_shell *state, t_rl *l) { - char **a; - size_t floor; - size_t n; + char **a; + size_t i; + size_t n; + t_sg_ent e; a = (char **)state->hist.hist_cmds.ctx; n = (size_t)rl_end; - floor = 0; - if (state->hist.hist_cmds.len > (size_t)l->sg_max) - floor = state->hist.hist_cmds.len - (size_t)l->sg_max; - while (from > floor) + l->sg_scr.elem_size = sizeof(t_sg_ent); + l->sg_scr.len = 0; + i = hist_window_floor(state->hist.hist_cmds.len, l->sg_max); + while (i < state->hist.hist_cmds.len) { - from--; - if (a[from] && !ft_strncmp(a[from], rl_line_buffer, n) - && ft_strlen(a[from]) > n && sg_clean((unsigned char *)a[from]) - && !sg_offered(state, l, a[from])) - return ((ssize_t)from); + if (a[i] && !ft_strncmp(a[i], rl_line_buffer, n) + && ft_strlen(a[i]) > n && sg_clean((unsigned char *)a[i])) + { + e = (t_sg_ent){a[i], i, 0}; + vec_push(&l->sg_scr, &e); + } + i++; } - return (-1); + return (l->sg_scr.len); } -/* Forget the cycle (not the entry on screen). */ +/* Forget the cycle (not the entry on screen). The ranking buffers keep + their capacity: they are freed with the read (rl_suggest_end). */ void rl_suggest_cyc_clear(t_rl *l) { - xfree(l->sg_cyc.ctx); - l->sg_cyc.ctx = NULL; - l->sg_cyc.len = 0; - l->sg_cyc.cap = 0; + l->sg_rank.len = 0; l->sg_cyc_k = 0; xfree(l->sg_cyc_line); l->sg_cyc_line = NULL; diff --git a/src/platform/posix/rl_suggest7.c b/src/platform/posix/rl_suggest7.c index 139f4933..24fae904 100644 --- a/src/platform/posix/rl_suggest7.c +++ b/src/platform/posix/rl_suggest7.c @@ -27,41 +27,19 @@ static void sg_show(t_shell *state, size_t idx) l->sg_txt = l->sg_ent + rl_end; } -/* Start a cycle on the ghost on screen: its index is found again (it is - the newest match, see rl_suggest_find) so that only indices are kept. */ +/* Start a cycle on the ghost on screen: the ranking is made once, and + its first entry is that ghost (the history has not changed, see + rl_suggest_cyc_sync). */ static bool sg_cyc_begin(t_shell *state, t_rl *l) { - ssize_t i; - size_t idx; - - i = rl_suggest_scan(state, l, state->hist.hist_cmds.len); - if (i < 0) + if (!rl_suggest_rank(state, l)) return (false); - idx = (size_t)i; - l->sg_cyc.elem_size = sizeof(size_t); - vec_push(&l->sg_cyc, &idx); l->sg_cyc_k = 0; l->sg_cyc_line = ft_strdup(rl_line_buffer); return (true); } -/* The next older distinct match, pushed on the cycle. False at the oldest - one: there is nothing more. */ -static bool sg_push_older(t_shell *state, t_rl *l) -{ - ssize_t i; - size_t idx; - - idx = ((size_t *)l->sg_cyc.ctx)[l->sg_cyc.len - 1]; - i = rl_suggest_scan(state, l, idx); - if (i < 0) - return (false); - idx = (size_t)i; - vec_push(&l->sg_cyc, &idx); - return (true); -} - -/* Alt-N: the next older distinct match. At the oldest one there is +/* Alt-N: the next lower-ranked distinct match. At the last one there is nothing more: bell, and the ghost stays (no wrap). */ int rl_suggest_older(int count, int key) { @@ -75,15 +53,15 @@ int rl_suggest_older(int count, int key) rl_suggest_cyc_sync(state, l); if (!l->sg_cyc_line && !sg_cyc_begin(state, l)) return (rl_suggest_bell()); - if (l->sg_cyc_k + 1 >= l->sg_cyc.len && !sg_push_older(state, l)) + if (l->sg_cyc_k + 1 >= l->sg_rank.len) return (rl_suggest_bell()); l->sg_cyc_k++; - sg_show(state, ((size_t *)l->sg_cyc.ctx)[l->sg_cyc_k]); + sg_show(state, ((size_t *)l->sg_rank.ctx)[l->sg_cyc_k]); return (0); } -/* Alt-P: back to the newer alternative shown before. At the newest match - there is nothing newer: bell. */ +/* Alt-P: back to the better-ranked alternative shown before. At the best + match there is nothing better: bell. */ int rl_suggest_newer(int count, int key) { t_shell *state; @@ -97,6 +75,6 @@ int rl_suggest_newer(int count, int key) if (!l->sg_cyc_line || l->sg_cyc_k == 0) return (rl_suggest_bell()); l->sg_cyc_k--; - sg_show(state, ((size_t *)l->sg_cyc.ctx)[l->sg_cyc_k]); + sg_show(state, ((size_t *)l->sg_rank.ctx)[l->sg_cyc_k]); return (0); } diff --git a/src/platform/posix/rl_suggest8.c b/src/platform/posix/rl_suggest8.c new file mode 100644 index 00000000..a2db82b4 --- /dev/null +++ b/src/platform/posix/rl_suggest8.c @@ -0,0 +1,118 @@ +/* ************************************************************************** */ +/* */ +/* ::: :::::::: */ +/* rl_suggest8.c :+: :+: :+: */ +/* +:+ +:+ +:+ */ +/* By: dlesieur +#+ +:+ +#+ */ +/* +#+#+#+#+#+ +#+ */ +/* Created: 2026/10/17 12:00:00 by dlesieur #+# #+# */ +/* Updated: 2026/10/17 12:00:00 by dlesieur ### ########.fr */ +/* */ +/* ************************************************************************** */ + +#include "rl_private.h" +#include + +/* libc qsort, not ft_qsort: ft_qsort pivots on the last element, so a run +** of identical entries -- already in ranking order -- costs O(n^2), 6.8 s +** for one key over 20000 copies (autosuggest_test F13). libc is +** O(n log n) on glibc, musl and macOS, and both comparators here are +** total orders, so every libc gives the same ranking. */ + +/* The ranking behind the ghost and Alt-N / Alt-P: the distinct texts of +** the candidates (rl_suggest_cands), best first. No hash table: libft's +** hash dups every key, which is one malloc per candidate per key. Sorting +** the reused scratch by text puts equal texts side by side, so one walk +** scores each group; a second sort orders the groups. */ + +/* Descending order for a comparator: the larger sorts first. */ +static int sg_three(uint64_t x, uint64_t y) +{ + if (x == y) + return (0); + if (x < y) + return (1); + return (-1); +} + +/* Text first, and within one text the newest occurrence first, so a group + starts with the entry that will stand for it. */ +static int sg_cmp_text(const void *x, const void *y) +{ + const t_sg_ent *a; + const t_sg_ent *b; + int c; + + a = x; + b = y; + c = ft_strcmp(a->text, b->text); + if (c) + return (c); + return (sg_three(a->idx, b->idx)); +} + +/* Best score first; on a tie the newest. The representatives are distinct + entries, so this is a strict total order and the sort needs no + stability. */ +static int sg_cmp_rank(const void *x, const void *y) +{ + const t_sg_ent *a; + const t_sg_ent *b; + + a = x; + b = y; + if (a->score != b->score) + return (sg_three(a->score, b->score)); + return (sg_three(a->idx, b->idx)); +} + +/* Collapse each run of equal texts to its first entry (the newest) holding + the run's score, in place: the write cursor never passes the read one. + Returns the number of distinct texts. */ +static size_t sg_fold(t_sg_ent *s, size_t n, size_t len, bool recent) +{ + size_t r; + size_t w; + + r = 0; + w = 0; + while (r < n) + { + s[w] = s[r]; + s[w].score = 0; + while (r < n && !ft_strcmp(s[r].text, s[w].text)) + s[w].score = hist_score_add(s[w].score, len, s[r++].idx, + recent); + w++; + } + return (w); +} + +/* Rank the history for the line: sg_rank gets the history index of each +** distinct candidate text, best first (score, then newest). The newest +** occurrence stands for its text. Returns how many. +** +** `recent` is today's order exactly: newest first, repeats collapsed. +** Only indices are kept: the scratch holds pointers into the history +** that a later `history -c` would leave dangling. +** +** Cost: O(m log m) for m candidates; no allocation once the two vectors +** have grown to the window. */ +size_t rl_suggest_rank(t_shell *state, t_rl *l) +{ + t_sg_ent *s; + size_t n; + size_t i; + + rl_suggest_cyc_clear(l); + l->sg_rank.elem_size = sizeof(size_t); + n = rl_suggest_cands(state, l); + s = (t_sg_ent *)l->sg_scr.ctx; + qsort(s, n, sizeof(*s), sg_cmp_text); + n = sg_fold(s, n, state->hist.hist_cmds.len, l->sg_recent); + qsort(s, n, sizeof(*s), sg_cmp_rank); + i = 0; + while (i < n) + vec_push(&l->sg_rank, &s[i++].idx); + return (l->sg_rank.len); +} diff --git a/tests/autosuggest_test.py b/tests/autosuggest_test.py index 5add7bf8..1e2fa94e 100644 --- a/tests/autosuggest_test.py +++ b/tests/autosuggest_test.py @@ -16,6 +16,13 @@ matching entries, no wrap (bell at either end), reset by any edit. With no history ghost these keys are their stock readline functions. +Ranking (HELLISH_SUGGEST_STRATEGY): by default the entry with the best +frecency (how often and how recently its text was run; weight 2^16 halved +per 64 newer entries, summed per distinct text, ties to the newest) is the +ghost and the order Alt-N / Alt-P walk; `recent` is newest first. `model()` +below is the same rule written again in Python, and the cases check the +shell against it as well as against hand-computed answers. + Usage: python3 autosuggest_test.py /path/to/hellish [case-name-part [inproc]] """ import fcntl @@ -47,6 +54,7 @@ "\x06", "\x15", "\x03", "\x04", "\x12", "\x07") BKSP = "\x7f" ON = "HELLISH_AUTOSUGGEST=1" +STRAT = "HELLISH_SUGGEST_STRATEGY" try: import pyte @@ -717,17 +725,32 @@ def case_cycle(fork, tag): def case_cycle_dupes(fork, tag): - s = shell(fork, ["echo a1", "echo a2", "echo a1", "echo a3"]) + """The pre-frecency rule, pinned: HELLISH_SUGGEST_STRATEGY=recent.""" + s = shell(fork, ["echo a1", "echo a2", "echo a1", "echo a3"], + env={STRAT: "recent"}) s.keys("echo a") seen = [s.ghost()] for _ in range(4): s.keys(ALT_N) seen.append(s.ghost()) - check(tag + "30 C2 distinct entries, once each, then the end", + check(tag + "30 C2 recent: distinct entries, once each, then the end", seen == ["3", "1", "2", "2", "2"], repr(seen)) s.close() +def case_cycle_dupes_frecent(fork, tag): + """Same history, default strategy: a1 was run twice, so it leads.""" + s = shell(fork, ["echo a1", "echo a2", "echo a1", "echo a3"]) + s.keys("echo a") + seen = [s.ghost()] + for _ in range(4): + s.keys(ALT_N) + seen.append(s.ghost()) + check(tag + "30 C2 frecent: twice-run a1 first, once each, the end", + seen == ["1", "3", "2", "2", "2"], repr(seen)) + s.close() + + def case_cycle_reset(fork, tag): s = three(fork, ["echo ab9"]) s.keys("echo a") @@ -945,6 +968,295 @@ def case_cycle_history_shrinks(fork, tag): +def clean_text(t): + """The ghost's filter: no control byte, no DEL, no C1 (as UTF-8).""" + return not any(ord(c) < 0x20 or ord(c) == 0x7f or 0x80 <= ord(c) < 0xa0 + for c in t) + + +def model(lines, prefix, strategy="frecent", mx=1000): + """The ranking, written again: distinct texts, best first.""" + n = len(lines) + score, newest = {}, {} + for i in range(max(0, n - mx), n): + t = lines[i] + if not (t.startswith(prefix) and len(t) > len(prefix) + and clean_text(t)): + continue + if strategy == "recent": + score[t] = i + 1 + else: + score[t] = score.get(t, 0) + (65536 >> min((n - 1 - i) // 64, 16)) + newest[t] = i + return sorted(score, key=lambda t: (-score[t], -newest[t])) + + +def seeded(fork, lines, env=None, **kw): + """A shell whose history is exactly `lines` (oldest first): nothing is + run, so no other entry is in it, and autosuggest is on from the start.""" + home = tempfile.mkdtemp(prefix="autosuggest_") + with open(os.path.join(home, "hist"), "w") as f: + f.write("".join(line + "\n" for line in lines)) + e = {"HELLISH_AUTOSUGGEST": "1"} + e.update(env or {}) + return Session(home, fork=fork, env=e, **kw) + + +def walk(s, typed, steps): + """The ghost after `typed`, then after each of `steps` Alt-N.""" + s.keys(typed) + seen = [s.ghost()] + for _ in range(steps): + s.keys(ALT_N) + seen.append(s.ghost()) + return seen + + +# re is run three times, clean twice and newest, test once: frecency says +# re, the newest entry says clean. +ISSUE = ["make re", "make test", "make re", "make clean", "make re", + "make clean"] + + +def case_frecent_issue(fork, tag): + s = seeded(fork, ISSUE) + got = walk(s, "make ", 3) + check(tag + "37 F1 frecent: the most-run command, not the newest", + got == ["re", "clean", "test", "test"], repr(got)) + s.close() + s = seeded(fork, ISSUE, env={STRAT: "recent"}) + got = walk(s, "make ", 3) + check(tag + "37 F1 recent: the newest, repeats collapsed", + got == ["clean", "re", "test", "test"], repr(got)) + s.close() + + +def case_frecent_recent(fork, tag): + lines = ["echo a1", "echo b1", "echo a2", "echo b2", "echo a3"] + s = seeded(fork, lines, env={STRAT: "recent"}) + got = walk(s, "echo ", 4) + check(tag + "38 F2 recent is newest first, as before frecency", + got == ["a3", "b2", "a2", "b1", "a1"], repr(got)) + s.close() + + +def case_frecent_default(fork, tag): + for val in ("", "bogus", "RECENT", "recent "): + s = seeded(fork, ISSUE, env={STRAT: val}) + got = walk(s, "make ", 0) + check(tag + "39 F3 %r is frecent" % val, got == ["re"], repr(got)) + s.close() + s = seeded(fork, ISSUE) + got = walk(s, "make ", 0) + check(tag + "39 F3 unset is frecent", got == ["re"], repr(got)) + s.close() + + +def case_frecent_live(fork, tag): + s = seeded(fork, ISSUE) + steps = (("", "re"), (STRAT + "=recent", "clean"), + (STRAT + "=bogus", "re"), (STRAT + "=recent", "clean"), + ("unset " + STRAT, "re")) + for cmd, want in steps: + if cmd: + s.run(cmd) + got = walk(s, "make ", 0) + check(tag + "40 F4 after %r the ghost is %r" % (cmd, want), + got == [want], repr(got)) + s.keys(CTRL_U) + s.close() + + +def case_frecent_tie(fork, tag): + lines = ["echo x2", "echo x1", "echo x2", "echo x1"] + s = seeded(fork, lines) + got = walk(s, "echo x", 2) + check(tag + "41 F5 equal scores: the newest first", + got == ["1", "2", "2"], repr(got)) + s.close() + s = seeded(fork, lines[1:] + ["echo x2"]) + got = walk(s, "echo x", 2) + check(tag + "41 F5 equal scores, other way round", + got == ["2", "1", "1"], repr(got)) + s.close() + + +def decay_lines(total): + """fa twice at the very start, filler, fb newest: `total` entries.""" + lines = ["echo fa", "true 0", "echo fa"] + lines += ["true %d" % n for n in range(1, total - 3)] + return lines + ["echo fb"] + + +def case_frecent_decay(fork, tag): + # Two fa at ages total-1 and total-3. Weight 2^16 halves each 64 newer + # entries: at 66 entries (ages 65, 63) 2^15 + 2^16 beats fb's 2^16; at + # 67 (ages 66, 64) both have halved, 2^16 ties, and the newest wins. + for total, want in ((20, "fa"), (66, "fa"), (67, "fb"), (300, "fb")): + lines = decay_lines(total) + s = seeded(fork, lines) + got = walk(s, "echo f", 0) + check(tag + "42 F6 %d entries: %s wins" % (total, want), + got == [want[1]] and model(lines, "echo f")[0] == "echo " + want, + repr(got)) + s.close() + + +def case_frecent_window(fork, tag): + lines = ["echo wa", "echo wb", "echo wa", "echo wb", "echo wa", + "echo wc", "echo wb"] + s = seeded(fork, lines) + got = walk(s, "echo w", 0) + check(tag + "43 F7 whole history: wa 3, wb 3, wc 1: the tie goes to " + "the newest", got == ["b"], repr(got)) + s.close() + s = seeded(fork, lines, env={"HELLISH_SUGGEST_MAX": "3"}) + got = walk(s, "echo w", 3) + check(tag + "43 F7 MAX=3: wa, wc, wb once each, newest first", + got == ["b", "c", "a", "a"], repr(got)) + s.close() + lines = ["echo wa", "echo wa", "echo wb", "echo wa", "echo wb", + "echo wc"] + s = seeded(fork, lines) + got = walk(s, "echo w", 0) + check(tag + "43 F7 MAX unset: wa x3 wins", got == ["a"], repr(got)) + s.close() + s = seeded(fork, lines, env={"HELLISH_SUGGEST_MAX": "4"}) + got = walk(s, "echo w", 2) + check(tag + "43 F7 MAX=4: the two older wa do not count, wb x2 wins", + got == ["b", "c", "a"], repr(got)) + s.close() + + +def case_frecent_cycle(fork, tag): + s = seeded(fork, ISSUE) + s.keys("make ") + order = [] + for _ in range(3): + order.append(s.ghost()) + s.keys(ALT_N) + check(tag + "44 F8 Alt-N follows the score", order == ["re", "clean", + "test"] and s.ghost() == "test", repr(order)) + s.keys(ALT_N) + check(tag + "44 F8 bell at the last, ghost stays", + b"\a" in s.since() and s.ghost() == "test", repr(s.since())) + s.keys(ALT_P + ALT_P) + check(tag + "44 F8 Alt-P walks back to the best", + s.ghost() == "re" and s.row() == PROMPT + "make re", + repr(s.row())) + s.keys(ALT_P) + check(tag + "44 F8 bell at the best, no wrap", + b"\a" in s.since() and s.ghost() == "re", repr(s.since())) + s.close() + + +def case_frecent_dirty(fork, tag): + lines = ["echo ok", "echo \x1b[31mred", "echo \x1b[31mred", + "echo \x1b[31mred", "echo ok2"] + s = seeded(fork, lines) + got = walk(s, "echo ", 3) + check(tag + "45 F9 a frequent entry holding ESC is never offered", + got == ["ok2", "ok", "ok", "ok"] and b"\x1b[31m" not in s.buf, + repr(got)) + s.close() + + +def case_frecent_history_shrinks(fork, tag): + s = seeded(fork, ISSUE) + s.run("bind -x '\"\\C-t\": history -c'") + s.keys("make ") + s.keys(ALT_N) + check(tag + "46 F10 cycled before the clear", s.ghost() == "clean", + repr(s.ghost())) + s.keys("\x14") + s.keys(ALT_N) + check(tag + "46 F10 no ghost survives a cleared history (Alt-N)", + s.ghost() == "" and s.alive(), repr(s.ghost())) + s.keys(CTRL_U + "make ") + s.keys(ALT_N + ALT_P) + check(tag + "46 F10 nor a cycle begun afterwards", + s.ghost() == "" and s.alive(), repr(s.ghost())) + s.keys("\x07\x15") + s.run("echo ok") + check(tag + "46 F10 the shell still runs lines", + "ok" in s.screen_text(), repr(s.screen_text()[-200:])) + s.close() + + +def case_frecent_typing(fork, tag): + long_a = "echo abcdefghijklmnopqrstuvwxyz0123456789" + long_b = "echo abcdefgXYZ0123456789abcdefghij" + lines = [long_a, long_b, long_a, "echo abc", long_a] + s = seeded(fork, lines) + bad = [] + for path in (long_a, long_b): + s.keys(CTRL_U + path[:4]) + for k in range(5, len(path) + 1): + s.keys(path[k - 1]) + want = model(lines, path[:k]) + want = want[0][k:] if want else "" + if s.ghost() != want: + bad.append((path[:k], s.ghost(), want)) + check(tag + "47 F11 typing forward: every ghost is the best match for " + "what is typed (%d keys)" % (len(long_a) + len(long_b) - 10), + not bad, repr(bad[:3])) + s.close() + + +def random_history(rng): + words = ["alpha", "alps", "alt", "amber", "ant"] + n = rng.choice([8, 40, 70, 130, 200]) + lines = [] + for i in range(n): + w = rng.choice(words) + if rng.random() < 0.5: + lines.append("echo %s" % w) + else: + lines.append("echo %s %d" % (w, rng.randrange(3))) + if i and lines[-1] == lines[-2]: + lines[-1] += " x" + return lines + + +def case_frecent_model(fork, tag): + import random + rng = random.Random(140 + int(fork)) + for k in range(5): + lines = random_history(rng) + mx = rng.choice([1000, 1000, 50]) + prefix = rng.choice(["echo ", "echo al", "echo a"]) + s = seeded(fork, lines, env={"HELLISH_SUGGEST_MAX": str(mx)}) + want = model(lines, prefix, mx=mx) + got = walk(s, prefix, len(want) + 1) + exp = [t[len(prefix):] for t in want] + exp = (exp + exp[-1:] * 2)[:len(want) + 2] if exp else [""] + exp = exp[:len(got)] if exp != [""] else [""] * len(got) + check(tag + "48 F12 random history %d (%d entries, max %d, %r): " + "the cycle equals the model" % (k, len(lines), mx, prefix), + got == exp, "got %r\nwant %r" % (got, exp)) + s.close() + + + +def case_frecent_sorted_worst(fork, tag): + """20000 copies of one entry arrive already in ranking order, the worst + case of a last-element-pivot quicksort (O(n^2)): the first key must + still paint its ghost quickly.""" + n = 20000 + s = seeded(fork, ["ls -la"] * n + ["pwd"], + env={"HELLISH_SUGGEST_MAX": str(n + 1), + "HISTSIZE": str(n + 10), "HISTFILESIZE": str(n + 10)}) + s.mark = len(s.buf) + t0 = time.time() + os.write(s.fd, b"l") + s.until(lambda: s.ghost() == "s -la", 20.0) + dt = time.time() - t0 + check(tag + "37 F13 sorted worst case paints within 1.5 s", + s.ghost() == "s -la" and dt < 1.5, "%r %.2fs" % (s.ghost(), dt)) + s.close() + + + CASES = (case_off_by_default, case_on, case_accept, case_enter_plain, case_newest, case_edit, case_cursor, case_interrupt, case_history_keys, case_search, case_vi, case_escape, @@ -952,10 +1264,16 @@ def case_cycle_history_shrinks(fork, tag): case_paste, case_max, case_unset, case_user_bind, case_postdisplay, case_resize, case_dumb, case_word, case_word_ctrl_right, case_word_blanks, case_word_stock, - case_word_utf8, case_cycle, case_cycle_dupes, case_cycle_reset, + case_word_utf8, case_cycle, case_cycle_dupes, + case_cycle_dupes_frecent, case_cycle_reset, case_cycle_accept, case_cycle_max, case_cycle_stock, case_cycle_post, case_cycle_user_bind, case_cycle_off, - case_cycle_vi, case_cycle_history_shrinks) + case_cycle_vi, case_cycle_history_shrinks, case_frecent_issue, + case_frecent_recent, case_frecent_default, case_frecent_live, + case_frecent_tie, case_frecent_decay, case_frecent_window, + case_frecent_cycle, case_frecent_dirty, + case_frecent_history_shrinks, case_frecent_typing, + case_frecent_model, case_frecent_sorted_worst) def main(): diff --git a/tests/cmdsub_nested_quote b/tests/cmdsub_nested_quote new file mode 100644 index 00000000..d8bc763d --- /dev/null +++ b/tests/cmdsub_nested_quote @@ -0,0 +1,18 @@ +x=$(echo "$(echo "a'b")"); echo "[$x]" +x=$(echo "$(echo '"')"); echo "[$x]" +x=$(printf %s "$(echo b | tr -d '"')" c); echo "[$x]" +x=$(printf %s "k: $(printf %s 'A"B' | tr -d '"')" tail); echo "[$x]" +x=$(echo "`echo "a'b"`"); echo "[$x]" +x=$(echo "`echo '"'`"); echo "[$x]" +x=$(echo "${u:-"a'b"}"); echo "[$x]" +x=$(echo "${u:-$(echo "a'b")}"); echo "[$x]" +x=$(echo "$(echo "$(echo "a'b")")"); echo "[$x]" +x=$(echo "$(( 1 + 1 ))$(echo "it's")"); echo "[$x]" +x=$(echo "a\"$(echo "b'c")"); echo "[$x]" +x=$(echo "$(echo "it's)")"); echo "[$x]" +x=$(echo "$(echo ')')"); echo "[$x]" +x=$(echo "$(echo 'hi')"); echo "[$x]" +x=$(echo "$(echo "hi")"); echo "[$x]" +y=v; x=$(printf %s "$(printf %s "$y'")"); echo "[$x]" +[ "$(printf %s "a$(printf %s 'b"' | tr -d '"')")" = ab ]; echo "rc=$?" +f() { [ "$(printf %s "$(printf 200 | tr -d '"')")" = 200 ]; }; f; echo "rc=$?" diff --git a/tests/complete_make_test.py b/tests/complete_make_test.py index 236da2a9..53fab79b 100755 --- a/tests/complete_make_test.py +++ b/tests/complete_make_test.py @@ -96,7 +96,7 @@ def drain_idle(fd, chunks, quiet=0.35, cap=8.0): return chunks -def spawn(cwd, home): +def spawn(cwd, home, cols=200, extra=None): env = { "HOME": home, "PATH": "/usr/bin:/bin", "TERM": "xterm-256color", "LANG": "C.UTF-8", "PS1": "$ ", @@ -106,6 +106,7 @@ def spawn(cwd, home): "ASAN_OPTIONS": "detect_leaks=%d" % (1 if ASAN else 0), } env.update(EXTRA) + env.update(extra or {}) pid, fd = pty.fork() if pid == 0: # nothing may leave this block except _exit (see @@ -118,16 +119,22 @@ def spawn(cwd, home): except BaseException: pass os._exit(127) - fcntl.ioctl(fd, termios.TIOCSWINSZ, struct.pack("HHHH", 60, 200, 0, 0)) + fcntl.ioctl(fd, termios.TIOCSWINSZ, struct.pack("HHHH", 60, cols, 0, 0)) return pid, fd -def session(cwd, steps, settle=0.6): +def session(cwd, steps, settle=0.6, hist=None, env=None, cols=200): """Run `steps` (bytes to type, or a callable run between keystrokes) - in a fresh shell; return the output of each bytes step, in order.""" + in a fresh shell; return the output of each bytes step, in order. + `hist` is the history the shell starts with, oldest first.""" home = tempfile.mkdtemp(prefix="hellish_mk_home_") TMP.append(home) - pid, fd = spawn(cwd, home) + extra = dict(env or {}) + if hist is not None: + extra["HISTFILE"] = os.path.join(home, "hist") + with open(extra["HISTFILE"], "w") as f: + f.write("".join(h + "\n" for h in hist)) + pid, fd = spawn(cwd, home, cols, extra) chunks = [] drain_idle(fd, chunks, quiet=0.4, cap=6.0) outs = [] @@ -177,6 +184,19 @@ def listing(out): return set(ANSI.sub("", block).split()) +def ordered(out): + """Like listing(), but in the order printed. Needs a terminal so + narrow (session(..., cols=10)) that readline prints one name a row.""" + i = out.find("\a\r\n") + if i < 0: + return [] + block = out[i + 3:] + j = block.find("\r\n$ ") + if j >= 0: + block = block[:j] + return ANSI.sub("", block).split() + + def line_is(out, want): return want in ANSI.sub("", out).replace("\a", "") @@ -453,6 +473,86 @@ def noperm(d): b.stdout, b.returncode)) +# `re` is run three times, `clean` twice and last, `test` once: frecency +# says re, clean, test; the newest entry says clean, re, test. +HIST = ["make re", "make test", "make re", "make clean", "make re", + "make clean"] +RANKED = "all clean fclean lint re test".split() +UNUSED = ["all", "fclean", "lint"] + + +def ranked_listing(hist, typed=b"make ", env=None, files=None): + d = fixture(dict({mk: "".join("%s:\n" % t for t in RANKED)}, + **(files or {}))) + out = session(d, [typed + b"\t\t\t"], hist=hist, env=env, cols=10)[0] + return ordered(out) + + +def ranked_cases(): + # 25 + got = ranked_listing(HIST) + check("25 used targets by frecency, then the rest alphabetically", + got == ["re", "clean", "test"] + UNUSED, str(got)) + got = ranked_listing(HIST, env={"HELLISH_SUGGEST_STRATEGY": "recent"}) + check("25 STRATEGY=recent: used targets newest first", + got == ["clean", "re", "test"] + UNUSED, str(got)) + got = ranked_listing([]) + check("25 no history: alphabetical", got == RANKED, str(got)) + got = ranked_listing(["echo re", "ls test", "cmake lint", "make"]) + check("25 only invocations of make count", got == RANKED, str(got)) + got = ranked_listing(["gmake lint", "/usr/bin/make test", "make lint"]) + check("25 gmake and a path to make count", + got == ["lint", "test", "all", "clean", "fclean", "re"], str(got)) + got = ranked_listing(["make re re re", "make lint test", "make test"]) + check("25 a word repeated inside one entry counts once", + got == ["test", "lint", "re", "all", "clean", "fclean"], str(got)) + out = comp_hist(HIST, b"make cl") + check("25 make cl still completes", line_is(out, "make clean "), + repr(out[-80:])) + + # 26 + got = ranked_listing(["make re", "make re", "make re", "make fclean", + "make test"], env={"HELLISH_SUGGEST_MAX": "2"}) + check("26 the window bound applies to make too (MAX=2)", + got == ["test", "fclean", "all", "clean", "lint", "re"], str(got)) + + # 27 + many = {"f%02d" % n: "x" for n in range(30)} + d = fixture(dict({mk: "".join("%s:\n" % t for t in RANKED)}, **many)) + outs = session(d, [b"make \t\t\t", KILL, b"ls f\t\t\t", KILL, + b"bas\t\t\t"], hist=HIST, cols=10) + mk_order = ordered(outs[0]) + files = ordered(outs[2]) + cmds = ordered(outs[4]) + check("27 make TAB ranked", mk_order == ["re", "clean", "test"] + + UNUSED, str(mk_order)) + check("27 then a filename TAB: sorted again, not left unsorted", + len(files) >= 30 and files == sorted(files), str(files[:8])) + check("27 then a command TAB: sorted again", + len(cmds) >= 2 and cmds == sorted(cmds), str(cmds[:8])) + + # 28 + d = fixture({mk: "all:\nclean:\n"}) + outs = session(d, [b"shopt -s progcomp\n", + b"complete -W 'zz yy' make\n", b"make \t\t\t"], + hist=["make clean", "make all", "make clean"]) + check("28 a user complete -W still wins over the ranking", + listing(outs[2]) == {"zz", "yy"}, str(sorted(listing(outs[2])))) + + # 29 + files = {"sub/Makefile": "s1:\n", "other.mk": "o1:\n"} + for typed in (b"make -C sub ", b"make -f other.mk "): + got = ranked_listing(HIST, typed, files=files) + check("29 %s keeps filenames with a history" % typed.decode(), + set(got) == {mk, "sub/", "other.mk"} + and got == sorted(got), str(got)) + + +def comp_hist(hist, typed): + d = fixture({mk: "".join("%s:\n" % t for t in RANKED)}) + return session(d, [typed + b"\t"], hist=hist)[0] + + def main(): if not os.access(SHELL, os.X_OK): sys.exit("complete_make_test: not executable: %s" % SHELL) @@ -465,6 +565,8 @@ def main(): EXTRA.clear() EXTRA.update(env) cases() + if not ASAN: + ranked_cases() finally: for d in TMP: shutil.rmtree(d, ignore_errors=True) diff --git a/tests/frontend_budget_test.py b/tests/frontend_budget_test.py index 39918ef5..f0563182 100755 --- a/tests/frontend_budget_test.py +++ b/tests/frontend_budget_test.py @@ -46,6 +46,9 @@ MARK = b"#> " ENTERS = 20 KEYS = 40 +ROUNDS = 12 +DIM = b"\x1b[2m" +HIST_N = 1000 TRACE_LINE = re.compile(r"^(\d+)\s+(\d+\.\d+)\s+([a-z_0-9]+)") # name -> (per Enter, per key, idle total, writes per resize phase) @@ -63,6 +66,11 @@ BUDGET["in-process/as"] = dict(BUDGET["in-process"], nomatch_write=1.2) BUDGET["forked/as+rp"] = dict(BUDGET["forked"], nomatch_write=2.2) BUDGET["forked/as"] = dict(BUDGET["forked"], nomatch_write=1.2) +# Frecency (#140): a history of 1000 distinct entries, all matching the +# first key, so every Backspace ranks them all. Same per-key budget: the +# ranking is CPU, and reuses its buffers instead of asking the kernel. +BUDGET["in-process/frecent"] = dict(BUDGET["in-process"], nomatch_write=1.2) +BUDGET["forked/frecent"] = dict(BUDGET["forked"], nomatch_write=1.2) SEED = "x" * 60 AS_ON = {"HELLISH_AUTOSUGGEST": "1"} FORKING = ("clone", "clone3", "fork", "vfork", "pipe2", "pipe", "wait4", @@ -77,7 +85,7 @@ def check(name, ok, detail=""): class Traced: - def __init__(self, home, trace, extra, seed=()): + def __init__(self, home, trace, extra, seed=(), hist=()): env = { "HOME": home, "PATH": os.environ.get("PATH", "/usr/bin:/bin"), "TERM": "xterm-256color", "LANG": "C.UTF-8", "LC_ALL": "C.UTF-8", @@ -87,6 +95,10 @@ def __init__(self, home, trace, extra, seed=()): "HISTFILE": "/dev/null", "ASAN_OPTIONS": "detect_leaks=0", } env.update(extra) + if hist: + env["HISTFILE"] = os.path.join(home, "hist") + with open(env["HISTFILE"], "w") as f: + f.write("".join(h + "\n" for h in hist)) self.buf = bytearray() self.phases = [] self.pid, self.fd = pty.fork() @@ -148,6 +160,19 @@ def typing(self, ch=b"x"): os.write(self.fd, b"\x15") self.settle() + def scanning(self): + """Forward keys and Backspace over a long history: each Backspace + ends a cached answer, so the ranking runs; forward keys must not.""" + self.mark = len(self.buf) + for _ in range(ROUNDS): + for key in (b"x", b"x", b"x", b"x", b"\x7f"): + os.write(self.fd, key) + self.pump(0.03) + self.settle() + self.dim = DIM in bytes(self.buf[self.mark:]) + os.write(self.fd, b"\x15") + self.settle() + def idle(self): time.sleep(2.0) @@ -184,14 +209,16 @@ def counts(lines, t0, t1): return c -def cell(label, extra, base, seed=()): +def cell(label, extra, base, seed=(), hist=()): trace = os.path.join(base, "trace-" + label.replace("/", "_")) home = tempfile.mkdtemp(dir=base) - s = Traced(home, trace, extra, seed) + s = Traced(home, trace, extra, seed, hist) s.phase("enter", s.enters) s.phase("idle", s.idle) s.phase("key", s.typing) s.phase("nomatch", lambda: s.typing(b"q")) + if hist: + s.phase("scan", s.scanning) s.phase("winch", s.winch_same) s.phase("rows", s.rows_only) s.close() @@ -234,6 +261,19 @@ def cell(label, extra, base, seed=()): check("%s: a key with nothing to paint does at most %.1f writes" % (label, b["nomatch_write"]), n <= b["nomatch_write"], "%.1f per key" % n) + if hist: + n = sum(by["scan"].values()) / float(ROUNDS * 5) + w = by["scan"]["write"] / float(ROUNDS * 5) + print("# %s: %d distinct entries, per key %.1f syscalls, " + "%.1f writes" % (label, len(hist), n, w)) + check("%s: a key over %d distinct entries, Backspace included, " + "costs at most %d syscalls" % (label, len(hist), b["key"]), + n <= b["key"], "%.1f per key: %s" % ( + n, dict(by["scan"].most_common(6)))) + check("%s: and at most %d writes" % (label, b["key_write"]), + w <= b["key_write"], "%.1f per key" % w) + check("%s: the ghost was painted over that history" % label, + s.dim, "no dim ghost in the output") check("%s: an idle prompt costs nothing" % label, sum(by["idle"].values()) == 0, "%d syscalls while idle: %s" % (sum(by["idle"].values()), @@ -258,6 +298,10 @@ def main(): cell("forked/as+rp", dict(AS_ON, HELLISH_RL_FORK="1"), base, seed) cell("forked/as", dict(AS_ON, RPROMPT="", HELLISH_RL_FORK="1"), base, seed) + hist = ["xxxx%04d" % n for n in range(HIST_N)] + cell("in-process/frecent", dict(AS_ON, RPROMPT=""), base, (), hist) + cell("forked/frecent", dict(AS_ON, RPROMPT="", HELLISH_RL_FORK="1"), + base, (), hist) finally: shutil.rmtree(base, ignore_errors=True) if FAILS: diff --git a/tests/issue95_case_cmdsub b/tests/issue95_case_cmdsub index 4f3f6f74..cda3a2b2 100644 --- a/tests/issue95_case_cmdsub +++ b/tests/issue95_case_cmdsub @@ -11,3 +11,8 @@ p=$(case x in y) echo no ;; esac; echo done); echo "p=$p" q=$(v=case; echo "$v"); echo "q=$q" r=$(echo case; echo esac); echo "r=$r" s=$(case m in m) echo one; echo two ;; esac); echo "s=$s" +x="$(case a in a) echo "q'r";; esac)"; echo "[$x]" +x="$(case a in a) echo 'q"r';; esac)"; echo "[$x]" +echo "$(case a in (a) echo "b'c";; esac)" +x="$(case a in a) echo "$(case b in b) echo "n'd";; esac)";; esac)"; echo "[$x]" +x="$(case 'a)' in 'a)') echo y;; esac)"; echo "[$x]" diff --git a/tests/prompt_interrupt_test.py b/tests/prompt_interrupt_test.py index ff9d987a..e6410a5e 100755 --- a/tests/prompt_interrupt_test.py +++ b/tests/prompt_interrupt_test.py @@ -28,6 +28,11 @@ burst three ^C in a row, then a command runs intact command ^C interrupts a foreground `sleep` isearch ^C inside ^R reverse search + nsearch ^C inside Alt-P non-incremental search + mx ^C inside M-x execute-named-command + redraw ^C while ^R is still drawing the keys sent with it, and a key + and ^C in one write, several times over: readline looks at the + signal at the end of every redraw, not only in the read ps2 ^C at a continuation prompt returns to PS1 with $? = 130 eof ^D on an empty line still exits @@ -181,6 +186,28 @@ def cell(label, home, extra): check("%s: isearch: ^C inside ^R, then a command runs" % label, s.expect(b"ISR_2"), repr(s.tail())) + s.send(b"\x1bpech") + s.until(lambda: b"ech" in s.buf[s.at:], 3.0) + interrupt(s) + s.send(b"echo NSR_$((2+1))\r") + check("%s: nsearch: ^C inside Alt-P, then a command runs" % label, + s.expect(b"NSR_3"), repr(s.tail())) + + s.send(b"\x1bxbeg") + s.until(lambda: b"beg" in s.buf[s.at:], 3.0) + interrupt(s) + s.send(b"echo MX_$((2+2))\r") + check("%s: mx: ^C inside M-x, then a command runs" % label, + s.expect(b"MX_4"), repr(s.tail())) + + for keys in (b"\x12e\x03", b"\x12\x03", b"\x12ec\x03") * 3: + s.send(keys) + s.until(lambda: b"^C" in s.buf[s.at:], 3.0) + s.prompt(5.0) + s.send(b"echo RD_$((4+4))\r") + check("%s: redraw: ^C sent with the keys of a ^R, nine times" % label, + s.expect(b"RD_8"), repr(s.tail())) + s.send(b"echo \"unterminated\r") s.until(lambda: b"> " in s.buf[s.at:], 3.0) s.pump(0.2) diff --git a/tests/prompt_row_writes_test.py b/tests/prompt_row_writes_test.py index 2686cb30..9924f713 100644 --- a/tests/prompt_row_writes_test.py +++ b/tests/prompt_row_writes_test.py @@ -207,7 +207,10 @@ def main(): rows = stream.count(GREEN.encode()) check("the trace holds the prompt's last row", rows >= 6, "found it %d times in %d writes" % (rows, len(writes))) - whole = sum(1 for w in writes if GREEN.encode() + MARK in w) + # Rows, not writes: readline 8.3 before readline83-004 (Debian's 8.3-4) + # reprints a multibyte prompt from column 0 at once, so one write() holds + # the row twice, and counting writes reported that as 7 of 13 "split". + whole = sum(w.count(GREEN.encode() + MARK) for w in writes) check("readline's row reaches the tty in one write() each time", whole >= rows > 0, "%d of %d rows were whole inside a single write()" % (whole, rows)) diff --git a/tests/rl_read_race_test.py b/tests/rl_read_race_test.py index a6e06447..ab118152 100644 --- a/tests/rl_read_race_test.py +++ b/tests/rl_read_race_test.py @@ -25,6 +25,8 @@ prompt a half-typed line, ^C in the gap, then a command isearch ^C in the gap inside a ^R search (the reported failure) + nsearch the same inside Alt-P, whose context readline 8.3 frees too + M-x the same inside execute-named-command Each runs with the in-process reader and with the forked one (HELLISH_RL_FORK=1), whose child inherits the same read. @@ -279,7 +281,9 @@ def main(): for label, extra in cells: for trigger, keys, what in ( (ord("x"), b"x; echo LEAK_$((6*7))", "prompt"), - (0x12, b"\x12ech", "isearch")): + (0x12, b"\x12ech", "isearch"), + (ord("p"), b"\x1bpech", "nsearch"), + (ord("x"), b"\x1bxbeg", "M-x")): r = scenario(label, shim, base, extra, trigger, keys, what) if r is None: print("skip: the shell never stalled in the shim -- its own " diff --git a/tests/tester b/tests/tester index ffea6c3a..da42b959 100755 --- a/tests/tester +++ b/tests/tester @@ -68,6 +68,7 @@ if [[ ${#test_lists[@]} -eq 0 ]]; then "array_subscript" "declare_quoting" "procsub_assign" "longtail" "issue95_case_cmdsub" + "cmdsub_nested_quote" "issue96_trap_func" "issue105_expansions" "issue121_errexit_andor" diff --git a/wiki/interactive.md b/wiki/interactive.md index 0a73a670..1a609580 100644 --- a/wiki/interactive.md +++ b/wiki/interactive.md @@ -171,11 +171,12 @@ $ git commit -m "fix: close [[ ]] on metachar-adjacent ]]" ### Autosuggestions ✅ *(3.2)* -Shipped in 3.2, opt-in with `export HELLISH_AUTOSUGGEST=1`. As you type, the newest history line -that starts with the typed text is shown dimmed after the cursor. **→ / End / Ctrl-F** take it, -**Alt-F** one word of it, and **Alt-N / Alt-P** show other matches. In the same release, Up and -Down search the history by the text before the cursor. See -[What's new in 3.2](whats-new.md#inline-hints). +Shipped in 3.2, opt-in with `export HELLISH_AUTOSUGGEST=1`. As you type, the history line you run +most that starts with the typed text is shown dimmed after the cursor; before 3.3 it was the newest +one, which `HELLISH_SUGGEST_STRATEGY=recent` brings back. **→ / End / Ctrl-F** take it, **Alt-F** +one word of it, and **Alt-N / Alt-P** show other matches. In 3.2, Up and Down also began searching +the history by the text before the cursor. See [the hint keys](whats-new.md#inline-hints) and +[how 3.3 ranks them](whats-new.md#hints-rank-what-you-run-most). ### Completion polish 🚧 *(Week 1, time-boxed)* diff --git a/wiki/manual.md b/wiki/manual.md index 6a6d6592..2fb57448 100644 --- a/wiki/manual.md +++ b/wiki/manual.md @@ -175,6 +175,7 @@ inside the shell: `help` for the grouped list, `help NAME` for one, | `HELLISH_HISTORY_PREFIX=0\|off` | Up/Down walk plain history; default on = search by the text before the cursor; read at every prompt | | `HELLISH_AUTOSUGGEST=1\|on` | inline suggestion (off by default): the newest history line starting with the typed text, dim after the cursor; Right / End / Ctrl-F insert it, Alt-F / Ctrl-Right insert its next word (leading blanks + the run up to the next space or tab), Alt-N / Alt-P cycle through older / newer distinct matches (bell at either end, no wrap; any edit resets); never a line with control characters or a newline; read at every prompt | | `HELLISH_SUGGEST_MAX=N` | history lines the suggestion scans, newest first (default 1000; non-numeric or below 1 = default); read at every prompt | +| `HELLISH_SUGGEST_STRATEGY=recent` | newest-first suggestions; default (unset/other) = frecency (run count weighted toward recent runs), also ordering `make `; read at every prompt | | `POSTDISPLAY` | zle widgets only: the text painted after the cursor, i.e. the suggestion; assign it to show your own until the buffer next changes | | `HELLISH_NO_UPDATE_CHECK=1` | never check for releases; no badge | | `HELLISH_NO_ANIM=1` | skip the startup animation | diff --git a/wiki/whats-new.md b/wiki/whats-new.md index 6add2881..b9205bc2 100644 --- a/wiki/whats-new.md +++ b/wiki/whats-new.md @@ -1,10 +1,9 @@ -# What's new in 3.2 +# What's new in 3.3 -hellish 3.2 is about the prompt. The up arrow finds what you started typing. -A dim hint from your history can complete the line, and you take it with one -key. `make ` knows your targets, and `bind` puts a shell command on a -key. This page shows how to use each one. Every example on it was run on a -3.2 build. +hellish 3.3 ranks by what you run, not by what you ran last. The dim hint and +`make ` put your most-run command first. This page shows how that works +and which settings change it. Every example on it was run on a 3.3 build. +[What came in 3.2](#what-came-in-32) follows further down. !!! tip "Get it" Already running hellish? Type `update --now`. Otherwise install it: @@ -17,6 +16,83 @@ key. This page shows how to use each one. Every example on it was run on a This is the table from the welcome banner, explained. +| Keys | What it does | Needs | +|---|---|---| +| `make ` | [most-run targets first](#make-targets-you-run-first) | | +| `→ ^F End` | [take the hint, now your most-run match](#hints-rank-what-you-run-most) | `export HELLISH_AUTOSUGGEST=1` | +| `Alt-N/P` | [step through the matches in the same order](#hints-rank-what-you-run-most) | `export HELLISH_AUTOSUGGEST=1` | + +## Hints rank what you run most + +With hints on, typing `make ` used to show the newest line that starts with +it. It now shows the line you run most, weighted toward recent runs: + +```console +$ export HELLISH_AUTOSUGGEST=1 +$ make re +$ make re +$ make re +$ make clean +$ make re +``` + +On the last line you typed `make `; `re` is the dim hint. Alt-N +moves to the next match, `clean`. A run's weight halves for every 64 newer +history lines, so a habit you dropped fades out. + +Two settings change the order: + +- `HELLISH_SUGGEST_STRATEGY=recent` brings back newest first: the history + above hints `clean`. It is read at every prompt. +- `HISTCONTROL=ignoredups` or `ignoreboth` stores a back-to-back repeat once, + so the three `make re` above count as one run and the hint is `clean`. + Repeats with another command between them still count. + +## make targets you run first + +`make ` still reads the Makefile itself and never runs make. Targets you +have run now come first, most-run first, and the rest follow alphabetically: + +```console +$ make +re clean all fclean test +``` + +`HELLISH_SUGGEST_STRATEGY=recent` orders them newest first here too. + +## Fixed + +In scripts: + +- A `$(...)` no longer stays as literal text when a double-quoted word inside + it holds a nested substitution with an odd number of quote bytes. + `x=$(echo "$(echo "a'b")"); echo "[$x]"` prints `[a'b]`, as in bash + ([#190](https://github.com/Univers42/hellish/issues/190)). +- A `case` inside a double-quoted `$(...)` parses. + `x="$(case a in a) echo "q'r";; esac)"; echo "[$x]"` prints `[q'r]` + ([#191](https://github.com/Univers42/hellish/issues/191)). + +At the prompt: + +- On a build linked against GNU readline 8.3, Ctrl-C inside + Ctrl-R, Alt-P or Alt-X freed the search + state twice. It now cancels the search and returns to a clean prompt + ([#194](https://github.com/Univers42/hellish/issues/194)). + +Every change since 3.2.0: +[v3.2.0...v3.3.0](https://github.com/Univers42/hellish/compare/v3.2.0...v3.3.0). + +## What came in 3.2 + +hellish 3.2 is about the prompt. The up arrow finds what you started typing. +A dim hint from your history can complete the line, and you take it with one +key. `make ` knows your targets, and `bind` puts a shell command on a +key. Every example in this section was run on a 3.2 build. + +### At a glance + +This is the table from the welcome banner, explained. + | Keys | What it does | Needs | |---|---|---| | `↑ ↓` | [history by prefix](#history-that-matches-what-you-typed) | on by default | @@ -29,7 +105,7 @@ This is the table from the welcome banner, explained. | `Alt-F` | [take one word](#inline-hints) | `HELLISH_AUTOSUGGEST=1` | | `Alt-N/P` | [older/newer hint](#inline-hints) | `HELLISH_AUTOSUGGEST=1` | -## History that matches what you typed +### History that matches what you typed Type the start of a command, then press ↑. Up now steps only through history lines that begin with the text before the cursor, and @@ -52,7 +128,7 @@ On an empty line, Up and Down walk the whole history as before. dead. - A key you bound yourself keeps your binding. -## Inline hints +### Inline hints Hints are off until you ask for them. Add this line to `~/.hellishrc`, or type it at the prompt to try it: @@ -92,7 +168,7 @@ $ ech|o hello you typed "ech", the cursor is at |, "o hello" is dim searched (default 1000). - In zsh mode, widgets see the hint as `POSTDISPLAY`, and can set their own. -## Completing make targets +### Completing make targets Tab after `make` offers the targets of the Makefile in the current directory. With this Makefile: @@ -133,7 +209,7 @@ would type, so they are not offered. The full rules are in the manual's [make targets](manual.md#make-targets) section. -## Key bindings with bind +### Key bindings with bind hellish now has bash's `bind` builtin, so the lines you would put in `~/.inputrc` work in `~/.hellishrc`: @@ -162,7 +238,7 @@ To see what is bound: `bind -p` lists key bindings, `bind -v` the settings, `bind -X` the `-x` commands and `bind -l` every command name. `help bind` shows every option. -## bindkey in zsh configs +### bindkey in zsh configs In zsh mode (`set -o zsh`, `emulate zsh`, or a sourced `.zsh` file), `bindkey` now understands zsh's key notation (`^[[A`, `^R`, `^[b`, `\E`) @@ -174,7 +250,7 @@ bindkey '^[[A' history-beginning-search-backward bindkey '^[[B' history-beginning-search-forward ``` -## Rendering text like the prompt +### Rendering text like the prompt `${var@P}` expands a value the way the prompt does. It uses the same renderer as `PS1`, so the result always matches your prompt: @@ -188,7 +264,7 @@ ada in share $ Both escape languages work, bash's `\u \w` and zsh's `%n %~`. `prompt preview` uses it to draw the themes. -## Two new variables +### Two new variables ```text $ echo $SRANDOM @@ -201,7 +277,7 @@ $ echo $EPOCHREALTIME `$EPOCHREALTIME` is the time in seconds since the epoch, with microseconds. Both come from bash: `$EPOCHREALTIME` from 5.0, `$SRANDOM` from 5.1. -## Fixed +### Fixed At the prompt: