From b8647b4c40f71f141308759c6698181fb0b0d70b Mon Sep 17 00:00:00 2001 From: qwO-Owp <332893456+qwO-Owp@users.noreply.github.com> Date: Sun, 18 Oct 2026 20:39:02 +0200 Subject: [PATCH 01/10] feat(editor): rank suggestions and make targets by frecency (#140) The ghost, Alt-N/Alt-P and `make ` now prefer what is run often and lately: each distinct line scores the sum of 2^16 >> (age / 64) over its occurrences in the HELLISH_SUGGEST_MAX window, ties to the newest. HELLISH_SUGGEST_STRATEGY=recent restores newest-first; read every prompt. One ranking primitive (rl_suggest_rank) feeds the ghost and the cycle; it groups by sort over a reused scratch buffer instead of libft's t_hash, which strdups every key (per-key malloc churn). The fast path is kept: a score does not depend on the prefix, so a winner that still matches a longer line stays the best of the smaller set. make TAB turns readline's match sort off only for its own answer; every attempt starts with it on. libc qsort, not ft_qsort: ft_qsort pivots on the last element and is O(n^2) on runs of identical entries, which arrive already in ranking order: 6.76 s for one key over 20000 copies (F13 red), under 1.5 s with qsort. Tests: autosuggest F1-F13 plus a randomized model-vs-shell cycle check, case_cycle_dupes pinned to `recent` with a frecent twin (the only existing case whose order changes), make cases 25-29 (order, sort-flag restore, recent, user complete spec, -C/-f), frontend budget with 1000 distinct entries and Backspace: 6.0 syscalls / 1.1 writes per key. Mutants MF1-MF11 (age, frequency, tie, direction, strategy x2, window, sort flag, make history, cycle order, clean filter) each turn cases red. --- USER_DOC.md | 13 +- incs/history.h | 6 + incs/prompt.h | 6 +- incs/shell.h | 1 + src/completion/complete_make.c | 19 +- src/completion/complete_make.h | 25 ++ src/completion/complete_make_cache.c | 1 + src/completion/complete_make_rank.c | 130 ++++++++++ src/completion/complete_make_rank2.c | 80 +++++++ src/completion/complete_provider.c | 8 + src/completion/completion.c | 8 +- src/helpers/free_utils.c | 4 + src/infrastructure/history_frecency.c | 67 ++++++ src/infrastructure/rl_private.h | 12 +- src/platform/posix/rl_slot_env.c | 13 + src/platform/posix/rl_suggest.c | 24 +- src/platform/posix/rl_suggest4.c | 8 +- src/platform/posix/rl_suggest5.c | 81 +++---- src/platform/posix/rl_suggest7.c | 42 +--- src/platform/posix/rl_suggest8.c | 118 ++++++++++ tests/autosuggest_test.py | 326 +++++++++++++++++++++++++- tests/complete_make_test.py | 112 ++++++++- tests/frontend_budget_test.py | 50 +++- wiki/manual.md | 1 + 24 files changed, 1038 insertions(+), 117 deletions(-) create mode 100644 src/completion/complete_make_rank.c create mode 100644 src/completion/complete_make_rank2.c create mode 100644 src/infrastructure/history_frecency.c create mode 100644 src/platform/posix/rl_suggest8.c diff --git a/USER_DOC.md b/USER_DOC.md index 63ff105c..d7c12a3b 100644 --- a/USER_DOC.md +++ b/USER_DOC.md @@ -169,6 +169,10 @@ escape hatches above get you back to your previous shell at any time. current directory (also `gmake`, `/usr/bin/make`, and behind `sudo`, `env X=1`, `&&`, `(`; a wrapper option that takes an argument, as in `sudo -u root make`, falls back to filenames). The file is parsed in the shell itself; make is never run. +Targets you have run (history lines starting with `make` or `gmake`) come first, +most-run (weighted toward recent runs) first; +the rest follow alphabetically. With `HELLISH_SUGGEST_STRATEGY=recent` the +run ones are ordered by how recently they ran. - Files read: the first regular file of `GNUmakefile`, `makefile`, `Makefile` (GNU make's order, symlinks followed), at most 4 MiB. No such file, a @@ -194,8 +198,9 @@ falls back to filenames). The file is parsed in the shell itself; make is never |---|---| | `HELLISH_BANNER=0\|1` | welcome panel off / on | | `HELLISH_HISTORY_PREFIX=0\|off` | Up/Down plain history; default is prefix search, read at every prompt | -| `HELLISH_AUTOSUGGEST=1\|on` | inline suggestion, off by default: the newest history line that starts with what you typed, shown dim after the cursor (see below); read at every prompt | +| `HELLISH_AUTOSUGGEST=1\|on` | inline suggestion, off by default: the best-ranked history line (see `HELLISH_SUGGEST_STRATEGY`) that starts with what you typed, shown dim after the cursor (see below); read at every prompt | | `HELLISH_SUGGEST_MAX=N` | how many of the newest history lines the suggestion looks at (default 1000; not a number, or below 1, is the default); read at every prompt | +| `HELLISH_SUGGEST_STRATEGY=recent` | rank suggestions newest first; by default (unset or any other value) they are ranked by frecency: how often a line was run, weighted toward recent runs. Also orders `make `. Read at every prompt | | `HELLISH_NO_UPDATE_CHECK=1` | never check for updates, no badge | | `HELLISH_NO_ANIM=1` | no startup animation | | `HELLISH_NO_EXEC=1` | user-install hook: skip for this one session | @@ -219,10 +224,10 @@ typed, and the hint is wiped from the screen. A word is the blanks at the front of the hint plus the run of characters up to the next space or tab (so `-m`, `a/b`, `x=1` are one word each). Alt-N and Alt-P step through distinct lines (same full text offered once), look -at no more than `HELLISH_SUGGEST_MAX` entries, ring the bell at the oldest -and the newest end instead of wrapping, and start over from the newest +at no more than `HELLISH_SUGGEST_MAX` entries, go in ranking order, ring the bell at the last and +the first instead of wrapping, and start over from the best match as soon as the line is edited. Right after Alt-N inserts the line -shown, not the newest. A hint a zle widget set as `POSTDISPLAY` is not +shown, not the best-ranked one. A hint a zle widget set as `POSTDISPLAY` is not cycled or taken by words. In vi insert mode Alt-F / Alt-N / Alt-P work the same (no Ctrl-Right). `unset HELLISH_AUTOSUGGEST` (or `=0`) turns it off at the next prompt. A key you bound yourself diff --git a/incs/history.h b/incs/history.h index 790b8a92..994fb8b8 100644 --- a/incs/history.h +++ b/incs/history.h @@ -15,6 +15,7 @@ # include "shell.h" # include +# include /* hist_cmds is the truth; readline's own list is a mirror kept in step so the arrow keys agree with what `history` prints. @@ -67,5 +68,10 @@ t_string encode_cmd_hist(char *cmd); void manage_history(t_shell *state); bool worthy_of_being_remembered(t_shell *state); void add_history_line(t_shell *state, const char *cmd); +/* Frecency (history_frecency.c). */ +uint32_t hist_weight(size_t age); +uint64_t hist_score_add(uint64_t score, size_t len, size_t idx, + bool recent); +size_t hist_window_floor(size_t len, int max); #endif \ No newline at end of file diff --git a/incs/prompt.h b/incs/prompt.h index 8eafbe4d..802621bc 100644 --- a/incs/prompt.h +++ b/incs/prompt.h @@ -108,8 +108,10 @@ typedef struct s_rl char *sg_post; /* POSTDISPLAY a widget set (zle_rl5.c) */ char *sg_post_at; /* the line it was set for */ char *sg_pub; /* what a widget was handed as POSTDISPLAY */ - t_vec sg_cyc; /* history indices offered (size_t) */ - size_t sg_cyc_k; /* which of them is on screen */ + bool sg_recent; /* HELLISH_SUGGEST_STRATEGY=recent */ + t_vec sg_rank; /* ranked history indices (size_t) */ + t_vec sg_scr; /* ranking scratch (t_sg_ent), reused */ + size_t sg_cyc_k; /* which of sg_rank is on screen */ char *sg_cyc_line; /* line the cycle is for; NULL: none */ bool sg_painted; /* a ghost is on screen right now */ int sg_col; /* column it starts at */ diff --git a/incs/shell.h b/incs/shell.h index 29186cb2..1e2d2d84 100644 --- a/incs/shell.h +++ b/incs/shell.h @@ -332,6 +332,7 @@ typedef struct s_make_cache long long mtime_s; long long mtime_ns; t_vec targets; /* char *, xmalloc'd */ + t_vec order; /* const char *, targets ranked for this TAB */ } t_make_cache; typedef struct s_shell diff --git a/src/completion/complete_make.c b/src/completion/complete_make.c index c9601da2..f0f7cd32 100644 --- a/src/completion/complete_make.c +++ b/src/completion/complete_make.c @@ -34,6 +34,9 @@ ** TAB. LIMITATION: a change to an INCLUDED file alone does not invalidate ** it -- touch the Makefile. ** +** ORDER. Targets used in history come first (frecency, see +** complete_make_rank.c), then the rest alphabetically. +** ** WHEN IT STAYS OUT. A word starting with '-' or holding '=' or '/' is not ** a target; and if an earlier argument is -C, --directory*, -f, --file* or ** --makefile*, the Makefile is not the one in the cwd, so filenames win. @@ -54,12 +57,12 @@ static char *mk_generator(const char *text, int state) t_vec *v; char *s; - v = &(*zle_state_cell())->make_cache.targets; + v = &(*zle_state_cell())->make_cache.order; if (!state) i = 0; while (i < v->len) { - s = ((char **)v->ctx)[i++]; + s = (char *)((const char **)v->ctx)[i++]; if (!ft_strncmp(s, text, ft_strlen(text))) return (rl_dup(s)); } @@ -90,11 +93,21 @@ static int mk_diverts(t_pv_line *a) return (0); } +/* The matches come out of mk_generator already in the order worth + showing (mk_order), so readline must not sort them again: it does that + AFTER this returns, so the flag is cleared here and cmd_completion sets + it back at the start of every attempt (completion.c). */ char **comp_make(t_shell *st, const char *text, t_pv_line *a) { + char **m; + if (text[0] == '-' || ft_strchr(text, '=') || ft_strchr(text, '/')) return (NULL); if (mk_diverts(a) || !mk_load(st)) return (NULL); - return (rl_completion_matches(text, mk_generator)); + mk_order(st); + m = rl_completion_matches(text, mk_generator); + if (m) + rl_sort_completion_matches = 0; + return (m); } diff --git a/src/completion/complete_make.h b/src/completion/complete_make.h index b4cf7865..53aa0109 100644 --- a/src/completion/complete_make.h +++ b/src/completion/complete_make.h @@ -49,6 +49,25 @@ typedef struct s_mk_scan int def; } t_mk_scan; +/* A target while ranking by history: how much it was used, and where. */ +typedef struct s_mk_use +{ + const char *name; + uint64_t score; + size_t newest; + size_t seen; +} t_mk_use; + +/* What counting needs besides the targets: the history length (for age) + and the strategy. */ +typedef struct s_mk_rank +{ + t_mk_use *u; + size_t cnt; + size_t len; + bool recent; +} t_mk_rank; + # define MK_MAX 4194304 /* The nanosecond mtime field is st_mtim on POSIX 2008, st_mtimespec on @@ -62,11 +81,17 @@ typedef struct s_mk_scan size_t pc_cmd_off(int start); size_t prov_word(t_pv_line *p, const char **w); size_t prov_base(const char *w, size_t n); +bool prov_is_make(const char *w, size_t n); char **comp_provider_try(const char *text, int start, int end); char **comp_make(t_shell *st, const char *text, t_pv_line *a); char *rl_dup(const char *s); t_vec *mk_load(t_shell *st); void mk_cache_free(t_shell *st); +void mk_order(t_shell *st); +int mk_cmp_use(const void *x, const void *y); +int mk_cmp_name(const void *x, const void *y); +void mk_count_entry(t_mk_rank *r, const char *e, size_t idx); +bool mk_is_make_entry(const char *e); char *mk_slurp(const char *path, size_t *len); void mk_scan(t_mk_scan *m, const char *buf, size_t len); void mk_line(t_mk_scan *m, char *l); diff --git a/src/completion/complete_make_cache.c b/src/completion/complete_make_cache.c index 26cc3dc7..2def35de 100644 --- a/src/completion/complete_make_cache.c +++ b/src/completion/complete_make_cache.c @@ -44,6 +44,7 @@ void mk_clear(t_make_cache *c) while (i < c->targets.len) xfree(((char **)c->targets.ctx)[i++]); xfree(c->targets.ctx); + xfree(c->order.ctx); xfree(c->name); ft_bzero(c, sizeof(*c)); } diff --git a/src/completion/complete_make_rank.c b/src/completion/complete_make_rank.c new file mode 100644 index 00000000..67d93c43 --- /dev/null +++ b/src/completion/complete_make_rank.c @@ -0,0 +1,130 @@ +/* ************************************************************************** */ +/* */ +/* ::: :::::::: */ +/* complete_make_rank.c :+: :+: :+: */ +/* +:+ +:+ +:+ */ +/* By: dlesieur +#+ +:+ +#+ */ +/* +#+#+#+#+#+ +#+ */ +/* Created: 2026/10/17 12:00:00 by dlesieur #+# #+# */ +/* Updated: 2026/10/17 12:00:00 by dlesieur ### ########.fr */ +/* */ +/* ************************************************************************** */ + +#include "complete_make.h" +#include + +/* make ordering: the targets the history shows being used come +** first, best frecency first (history_frecency.c, the ranking the inline +** suggestion uses); the rest follow alphabetically. +** +** An entry counts for a target when its FIRST word is a command the +** provider table maps to comp_make (make, gmake, any path to them) and +** the target is one of its whitespace-separated words, once per entry. +** +** Caveat: this reads words, not make's command line. `make -C re test` +** credits a target named `re` for the directory, and in `make a && make b` +** the `b` counts because the entry starts with make: a target can rank +** higher than it deserves, but is never hidden. `sudo make x` is not +** counted, and a word with `;`, a quote or a backslash glued on never +** equals a target name, so those under-credit. HELLISH_SUGGEST_STRATEGY= +** recent, or an alphabetical list, is the way out. */ + +/* Alphabetical, the way readline sorts when it is allowed to (strcoll), + so an unused-only list looks exactly as it always did. */ +int mk_cmp_name(const void *x, const void *y) +{ + const t_mk_use *a; + const t_mk_use *b; + + a = x; + b = y; + return (strcoll(a->name, b->name)); +} + +/* Used targets first: best score, then newest; ties, and every unused + target (score 0), by name. */ +int mk_cmp_use(const void *x, const void *y) +{ + const t_mk_use *a; + const t_mk_use *b; + + a = x; + b = y; + if (a->score != b->score && a->score < b->score) + return (1); + if (a->score != b->score) + return (-1); + if (a->newest != b->newest && a->newest < b->newest) + return (1); + if (a->newest != b->newest) + return (-1); + return (mk_cmp_name(x, y)); +} + +/* Is the entry a make invocation: its first word names the make provider + (the same table row that decides `make ` itself). */ +bool mk_is_make_entry(const char *e) +{ + size_t n; + size_t b; + + while (*e == ' ' || *e == '\t') + e++; + n = 0; + while (e[n] && e[n] != ' ' && e[n] != '\t') + n++; + b = prov_base(e, n); + return (prov_is_make(e + b, n - b)); +} + +/* Binary search of the name-sorted `u` for the word w[0..n); NULL if it + is not a target. */ +static t_mk_use *mk_find(t_mk_use *u, size_t cnt, const char *w, size_t n) +{ + size_t lo; + size_t hi; + size_t mid; + int c; + + lo = 0; + hi = cnt; + while (lo < hi) + { + mid = lo + (hi - lo) / 2; + c = ft_strncmp(u[mid].name, w, n); + if (!c && u[mid].name[n]) + c = 1; + if (!c) + return (&u[mid]); + if (c < 0) + lo = mid + 1; + else + hi = mid; + } + return (NULL); +} + +/* Credit entry `idx` (text `e`) to every distinct target among its words. + `seen` stops a word repeated in one entry from counting twice. */ +void mk_count_entry(t_mk_rank *r, const char *e, size_t idx) +{ + t_mk_use *t; + size_t n; + + while (*e) + { + while (*e == ' ' || *e == '\t') + e++; + n = 0; + while (e[n] && e[n] != ' ' && e[n] != '\t') + n++; + t = mk_find(r->u, r->cnt, e, n); + if (t && n && t->seen != idx + 1) + { + t->seen = idx + 1; + t->score = hist_score_add(t->score, r->len, idx, r->recent); + t->newest = idx; + } + e += n; + } +} diff --git a/src/completion/complete_make_rank2.c b/src/completion/complete_make_rank2.c new file mode 100644 index 00000000..7e8cb98b --- /dev/null +++ b/src/completion/complete_make_rank2.c @@ -0,0 +1,80 @@ +/* ************************************************************************** */ +/* */ +/* ::: :::::::: */ +/* complete_make_rank2.c :+: :+: :+: */ +/* +:+ +:+ +:+ */ +/* By: dlesieur +#+ +:+ +#+ */ +/* +#+#+#+#+#+ +#+ */ +/* Created: 2026/10/17 12:00:00 by dlesieur #+# #+# */ +/* Updated: 2026/10/17 12:00:00 by dlesieur ### ########.fr */ +/* */ +/* ************************************************************************** */ + +#include "complete_make.h" +#include +#include + +/* Fill st->make_cache.order with the cached targets in the order + `make ` offers them (complete_make_rank.c): used first, by + frecency, then the rest alphabetically. */ + +/* One record per target, sorted by name so that words can be looked up + by binary search. Returns NULL for no targets or no memory. */ +static t_mk_use *mk_records(t_vec *targets) +{ + t_mk_use *u; + size_t i; + + if (!targets->len) + return (NULL); + u = xmalloc(targets->len * sizeof(*u)); + i = 0; + while (i < targets->len) + { + u[i] = (t_mk_use){((char **)targets->ctx)[i], 0, 0, 0}; + i++; + } + qsort(u, targets->len, sizeof(*u), mk_cmp_name); + return (u); +} + +/* Walk the window of the newest HELLISH_SUGGEST_MAX history entries, the + same one the inline suggestion looks at, crediting make invocations. + Cost: O(window * words * log targets) per TAB. */ +static void mk_credit(t_shell *st, t_mk_rank *r) +{ + char **a; + size_t i; + + a = (char **)st->hist.hist_cmds.ctx; + i = hist_window_floor(r->len, st->rl.sg_max); + while (i < r->len) + { + if (a[i] && mk_is_make_entry(a[i])) + mk_count_entry(r, a[i], i); + i++; + } +} + +/* The ranked order, as borrowed pointers into the target list. It is + rebuilt on every TAB: the history moves on, the targets rarely do. */ +void mk_order(t_shell *st) +{ + t_mk_rank r; + size_t i; + + st->make_cache.order.len = 0; + st->make_cache.order.elem_size = sizeof(const char *); + r.cnt = st->make_cache.targets.len; + r.u = mk_records(&st->make_cache.targets); + r.len = st->hist.hist_cmds.len; + r.recent = st->rl.sg_recent; + if (!r.u) + return ; + mk_credit(st, &r); + qsort(r.u, r.cnt, sizeof(*r.u), mk_cmp_use); + i = 0; + while (i < r.cnt) + vec_push(&st->make_cache.order, &r.u[i++].name); + xfree(r.u); +} diff --git a/src/completion/complete_provider.c b/src/completion/complete_provider.c index 270b7e89..8c57493d 100644 --- a/src/completion/complete_provider.c +++ b/src/completion/complete_provider.c @@ -75,6 +75,14 @@ static t_prov_fn prov_find(const char *w, size_t n) return (NULL); } +/* Does the command named by w[0..n) belong to the make provider (make, + gmake)? The table is the one place that says so; the history ranking in + complete_make_rank.c asks instead of keeping a list of its own. */ +bool prov_is_make(const char *w, size_t n) +{ + return (prov_find(w, n) == comp_make); +} + /* Dispatch an argument word to the provider of its command, or NULL. */ char **comp_provider_try(const char *text, int start, int end) { diff --git a/src/completion/completion.c b/src/completion/completion.c index f9025ba4..d285bed4 100644 --- a/src/completion/completion.c +++ b/src/completion/completion.c @@ -101,13 +101,19 @@ char *rl_dup_dollar(const char *name, size_t len) Only when no spec exists does a built-in provider get a turn (make targets, complete_make.c), so a user's `complete` always wins; a provider - that has nothing to offer answers NULL and the filenames come back. */ + that has nothing to offer answers NULL and the filenames come back. + + readline sorts the matches after this returns (postprocess_matches), so + a provider that returns an order of its own (make) can only switch the + sort off for that attempt; the flag is therefore set back on here, for + every attempt, or a later filename or command TAB would stay unsorted. */ static char **cmd_completion(const char *text, int start, int end) { char **spec; rl_completion_append_character = ' '; rl_completion_suppress_append = 0; + rl_sort_completion_matches = 1; if (text[0] == '$' && !ft_strchr(text, '/')) return (rl_attempted_completion_over = 1, complete_variables(text, start, end)); diff --git a/src/helpers/free_utils.c b/src/helpers/free_utils.c index 8953f77a..eac090bc 100644 --- a/src/helpers/free_utils.c +++ b/src/helpers/free_utils.c @@ -112,6 +112,10 @@ static void free_session_strings(t_shell *state) state->dft_ctx = 0; xfree(state->rl.buff.ctx); xfree(state->rl.rp_txt.ctx); + xfree(state->rl.sg_rank.ctx); + xfree(state->rl.sg_scr.ctx); + state->rl.sg_rank = (t_vec){}; + state->rl.sg_scr = (t_vec){}; xfree(state->hist.file); state->hist.file = NULL; if (state->path_dirs) diff --git a/src/infrastructure/history_frecency.c b/src/infrastructure/history_frecency.c new file mode 100644 index 00000000..4cbaa227 --- /dev/null +++ b/src/infrastructure/history_frecency.c @@ -0,0 +1,67 @@ +/* ************************************************************************** */ +/* */ +/* ::: :::::::: */ +/* history_frecency.c :+: :+: :+: */ +/* +:+ +:+ +:+ */ +/* By: dlesieur +#+ +:+ +#+ */ +/* +#+#+#+#+#+ +#+ */ +/* Created: 2026/10/17 12:00:00 by dlesieur #+# #+# */ +/* Updated: 2026/10/17 12:00:00 by dlesieur ### ########.fr */ +/* */ +/* ************************************************************************** */ + +#include "shell.h" + +/* Frecency: how much a history entry is worth as a suggestion, from how +** often AND how recently its text was run. Shared by the inline ghost +** and Alt-N/P (platform/posix/rl_suggest8.c) and by `make ` +** (completion/complete_make_rank.c), so both rank the same way. +** +** An occurrence's weight starts at 2^16 and halves every FRE_HALF newer +** entries, down to 1 once it is 16 halvings old; a text scores the sum of +** its occurrences. The score does not depend on what is being typed, so +** the best match for a prefix stays the best for every longer prefix it +** still matches, which is what lets the ghost skip rescanning per key. +** +** Caveat: age is a count of entries, not time. A burst of 64 unrelated +** commands halves a habit's weight, while a command last used a year ago +** in a quiet shell still weighs as much as one used yesterday; and +** FRE_HALF = 64 is a guess, not a measurement. HELLISH_SUGGEST_MAX bounds +** the window, so nothing older than it counts at all. Escape hatch: +** HELLISH_SUGGEST_STRATEGY=recent ignores frequency entirely. */ +#define FRE_HALF 64 +#define FRE_STEPS 16 + +/* Weight of an entry `age` entries older than the newest. */ +uint32_t hist_weight(size_t age) +{ + size_t steps; + + steps = age / FRE_HALF; + if (steps > FRE_STEPS) + steps = FRE_STEPS; + return ((uint32_t)(1u << FRE_STEPS) >> steps); +} + +/* Fold the occurrence at index `idx` (of `len` entries) into `score`. + `recent` makes the score the newest index (+1, so 0 stays "unused"), + which orders newest first and collapses repeats. */ +uint64_t hist_score_add(uint64_t score, size_t len, size_t idx, + bool recent) +{ + if (recent) + { + if ((uint64_t)idx + 1 > score) + return ((uint64_t)idx + 1); + return (score); + } + return (score + hist_weight(len - 1 - idx)); +} + +/* First index of the window of the newest `max` entries. */ +size_t hist_window_floor(size_t len, int max) +{ + if (max > 0 && len > (size_t)max) + return (len - (size_t)max); + return (0); +} diff --git a/src/infrastructure/rl_private.h b/src/infrastructure/rl_private.h index ccece657..7fe01086 100644 --- a/src/infrastructure/rl_private.h +++ b/src/infrastructure/rl_private.h @@ -97,6 +97,15 @@ typedef struct s_frame char *rl_suggest_update(t_shell *state, int cols); char *rl_suggest_live(t_rl *l); +/* One candidate while ranking: the text points into hist_cmds and is only + good for the call that filled it. */ +typedef struct s_sg_ent +{ + const char *text; + size_t idx; + uint64_t score; +} t_sg_ent; + char *rl_suggest_find(t_shell *state, t_rl *l); int rl_suggest_accept(int count, int key); t_shell *rl_suggest_ghost_state(void); @@ -104,7 +113,8 @@ int rl_suggest_bell(void); int rl_suggest_word(int count, int key); int rl_suggest_older(int count, int key); int rl_suggest_newer(int count, int key); -ssize_t rl_suggest_scan(t_shell *state, t_rl *l, size_t from); +size_t rl_suggest_cands(t_shell *state, t_rl *l); +size_t rl_suggest_rank(t_shell *state, t_rl *l); void rl_suggest_cyc_clear(t_rl *l); bool rl_suggest_cyc_sync(t_shell *state, t_rl *l); void rl_suggest_end(t_shell *state); diff --git a/src/platform/posix/rl_slot_env.c b/src/platform/posix/rl_slot_env.c index e1375d37..2cfc6a6b 100644 --- a/src/platform/posix/rl_slot_env.c +++ b/src/platform/posix/rl_slot_env.c @@ -64,6 +64,18 @@ static int sg_limit(t_shell *state) return (ft_atoi(e->value)); } +/* HELLISH_SUGGEST_STRATEGY: how suggestions are ranked. Exactly `recent` + is newest first; unset, empty or anything else (typos included) is the + default, frecency, so a misspelt value never silently changes order to + the old one. */ +static bool sg_wants_recent(t_shell *state) +{ + t_env *e; + + e = env_get(&state->env, "HELLISH_SUGGEST_STRATEGY"); + return (e && e->value && !ft_strcmp(e->value, "recent")); +} + /* Interactive reads only, right before rl_preinit: note what is wanted and ** switch the slot hooks on. Doing it inside rl_preinit's caller keeps ** `bind -p` (which also runs rl_preinit, in -c mode) exactly as bash's. @@ -76,4 +88,5 @@ void rl_slot_sync(t_shell *state) state->rl.feat_want[FEAT_HIST] = hp_wanted(state); state->rl.feat_want[FEAT_ACCEPT] = as_wanted(state); state->rl.sg_max = sg_limit(state); + state->rl.sg_recent = sg_wants_recent(state); } diff --git a/src/platform/posix/rl_suggest.c b/src/platform/posix/rl_suggest.c index c74a0a5d..44f61c48 100644 --- a/src/platform/posix/rl_suggest.c +++ b/src/platform/posix/rl_suggest.c @@ -12,7 +12,8 @@ #include "rl_private.h" -/* The inline suggestion: the newest history entry that starts with what +/* The inline suggestion: the best-ranked history entry (frecency, or the +** newest with HELLISH_SUGGEST_STRATEGY=recent) that starts with what ** is typed, shown dim after the cursor and never part of the line. This ** file decides WHAT may be suggested and WHEN; rl_suggest2.c turns it into ** the text and rl_rprompt.c paints it. @@ -23,16 +24,16 @@ ** is a C1 control in its UTF-8 form: running `printf '\e[31m'` once must ** not repaint every later prompt. */ -/* The newest of the last sg_max entries that starts with the first `n` - bytes of the line, is longer than it and is clean (rl_suggest_scan). */ +/* The best-ranked of the last sg_max entries that start with the line, are + longer than it and are clean (rl_suggest_rank), or NULL. */ static char *sg_scan(t_shell *state, t_rl *l) { - ssize_t i; + char **a; - i = rl_suggest_scan(state, l, state->hist.hist_cmds.len); - if (i < 0) + if (!rl_suggest_rank(state, l)) return (NULL); - return (((char **)state->hist.hist_cmds.ctx)[i]); + a = (char **)state->hist.hist_cmds.ctx; + return (a[*(size_t *)l->sg_rank.ctx]); } /* True when the last scan still answers for the line: the line extends the @@ -54,9 +55,12 @@ static bool sg_cached(t_rl *l, size_t n) ** Typing extends the line, and an entry that matches the longer line ** matched the shorter one too -- so when the line still starts with the one ** the last scan was for, the answer is the same entry as long as it still -** matches (or "nothing" for good). That makes a key O(line) with no scan, -** and a paste of 4 KB one scan instead of 4096. Anything else (a -** Backspace, a recalled line) scans again. The history cannot change under +** matches (or "nothing" for good). That holds for the frecency ranking as +** well: an entry's score does not depend on the line, so the winner for +** the shorter line, still matching, is the best of the smaller set too. +** That makes a key O(line) with no scan, and a paste of 4 KB one scan +** instead of 4096. Anything else (a Backspace, a recalled line) scans +** again. The history cannot change under ** it: it only grows between reads, and the cache is dropped per read. */ char *rl_suggest_find(t_shell *state, t_rl *l) { diff --git a/src/platform/posix/rl_suggest4.c b/src/platform/posix/rl_suggest4.c index 1dd645ab..5703af41 100644 --- a/src/platform/posix/rl_suggest4.c +++ b/src/platform/posix/rl_suggest4.c @@ -53,7 +53,9 @@ void rl_suggest_stale(void) /* The read is over (or about to start): nothing of the ghost, of the search it was found with or of a widget's POSTDISPLAY outlives it. The - history may have changed since, so the cache cannot either. */ + history may have changed since, so the cache cannot either. The ranking + buffers are freed here too: they are reused by every key of one read, + never kept between reads. */ void rl_suggest_end(t_shell *state) { t_rl *l; @@ -65,6 +67,10 @@ void rl_suggest_end(t_shell *state) xfree(l->sg_post_at); xfree(l->sg_pub); rl_suggest_cyc_clear(l); + xfree(l->sg_rank.ctx); + xfree(l->sg_scr.ctx); + l->sg_rank = (t_vec){}; + l->sg_scr = (t_vec){}; l->sg_ent = NULL; l->sg_prev = NULL; l->sg_post = NULL; diff --git a/src/platform/posix/rl_suggest5.c b/src/platform/posix/rl_suggest5.c index 484134c1..d59204ee 100644 --- a/src/platform/posix/rl_suggest5.c +++ b/src/platform/posix/rl_suggest5.c @@ -12,11 +12,10 @@ #include "rl_private.h" -/* Scanning the history for suggestions, and the state of Alt-N / Alt-P -** cycling. A cycle remembers the history INDICES it has shown (sg_cyc, a -** stack: Alt-N pushes the next older distinct one, Alt-P pops back), so -** going back to a newer alternative costs nothing and a revisit never -** re-scans. */ +/* Collecting suggestion candidates from the history, and the state of +** Alt-N / Alt-P cycling. A cycle is the ranking (rl_suggest_rank) made +** once, as history INDICES in sg_rank, and a position in it: Alt-N and +** Alt-P only move the position, so nothing is ever scanned again. */ /* True when `s` holds nothing a terminal would interpret. 0x7f is DEL; the pair 0xC2 0x80..0x9F is U+0080..U+009F, which some terminals read as a @@ -37,63 +36,43 @@ static bool sg_clean(const unsigned char *s) return (true); } -/* True when an entry with the same full text was already offered in this - cycle. Distinct means distinct text: history keeps duplicates, and - stepping through three copies of one command is not a choice. */ -static bool sg_offered(t_shell *state, t_rl *l, const char *text) -{ - char **a; - size_t *v; - size_t i; - - a = (char **)state->hist.hist_cmds.ctx; - v = (size_t *)l->sg_cyc.ctx; - i = 0; - while (i < l->sg_cyc.len) - { - if (!ft_strcmp(a[v[i]], text)) - return (true); - i++; - } - return (false); -} - -/* The index of the next older entry below `from` that starts with the -** line, is longer than it, is clean and was not offered yet; -1 if none. -** Only the newest sg_max entries are looked at (HELLISH_SUGGEST_MAX), so a -** huge history cannot slow a key down. +/* Fill the scratch with every candidate of the window of the newest +** sg_max entries: starts with the line, is longer than it, is clean. +** Returns how many. The scratch and its capacity are kept for the next +** call (a key never allocates once it has grown), and the texts point into +** the history, so they are good until the history changes. ** -** Cost: O(sg_max * (line + offered)) per call, offered being the entries -** shown so far in this cycle -- a handful in practice. */ -ssize_t rl_suggest_scan(t_shell *state, t_rl *l, size_t from) +** Cost: O(sg_max * line). */ +size_t rl_suggest_cands(t_shell *state, t_rl *l) { - char **a; - size_t floor; - size_t n; + char **a; + size_t i; + size_t n; + t_sg_ent e; a = (char **)state->hist.hist_cmds.ctx; n = (size_t)rl_end; - floor = 0; - if (state->hist.hist_cmds.len > (size_t)l->sg_max) - floor = state->hist.hist_cmds.len - (size_t)l->sg_max; - while (from > floor) + l->sg_scr.elem_size = sizeof(t_sg_ent); + l->sg_scr.len = 0; + i = hist_window_floor(state->hist.hist_cmds.len, l->sg_max); + while (i < state->hist.hist_cmds.len) { - from--; - if (a[from] && !ft_strncmp(a[from], rl_line_buffer, n) - && ft_strlen(a[from]) > n && sg_clean((unsigned char *)a[from]) - && !sg_offered(state, l, a[from])) - return ((ssize_t)from); + if (a[i] && !ft_strncmp(a[i], rl_line_buffer, n) + && ft_strlen(a[i]) > n && sg_clean((unsigned char *)a[i])) + { + e = (t_sg_ent){a[i], i, 0}; + vec_push(&l->sg_scr, &e); + } + i++; } - return (-1); + return (l->sg_scr.len); } -/* Forget the cycle (not the entry on screen). */ +/* Forget the cycle (not the entry on screen). The ranking buffers keep + their capacity: they are freed with the read (rl_suggest_end). */ void rl_suggest_cyc_clear(t_rl *l) { - xfree(l->sg_cyc.ctx); - l->sg_cyc.ctx = NULL; - l->sg_cyc.len = 0; - l->sg_cyc.cap = 0; + l->sg_rank.len = 0; l->sg_cyc_k = 0; xfree(l->sg_cyc_line); l->sg_cyc_line = NULL; diff --git a/src/platform/posix/rl_suggest7.c b/src/platform/posix/rl_suggest7.c index 139f4933..24fae904 100644 --- a/src/platform/posix/rl_suggest7.c +++ b/src/platform/posix/rl_suggest7.c @@ -27,41 +27,19 @@ static void sg_show(t_shell *state, size_t idx) l->sg_txt = l->sg_ent + rl_end; } -/* Start a cycle on the ghost on screen: its index is found again (it is - the newest match, see rl_suggest_find) so that only indices are kept. */ +/* Start a cycle on the ghost on screen: the ranking is made once, and + its first entry is that ghost (the history has not changed, see + rl_suggest_cyc_sync). */ static bool sg_cyc_begin(t_shell *state, t_rl *l) { - ssize_t i; - size_t idx; - - i = rl_suggest_scan(state, l, state->hist.hist_cmds.len); - if (i < 0) + if (!rl_suggest_rank(state, l)) return (false); - idx = (size_t)i; - l->sg_cyc.elem_size = sizeof(size_t); - vec_push(&l->sg_cyc, &idx); l->sg_cyc_k = 0; l->sg_cyc_line = ft_strdup(rl_line_buffer); return (true); } -/* The next older distinct match, pushed on the cycle. False at the oldest - one: there is nothing more. */ -static bool sg_push_older(t_shell *state, t_rl *l) -{ - ssize_t i; - size_t idx; - - idx = ((size_t *)l->sg_cyc.ctx)[l->sg_cyc.len - 1]; - i = rl_suggest_scan(state, l, idx); - if (i < 0) - return (false); - idx = (size_t)i; - vec_push(&l->sg_cyc, &idx); - return (true); -} - -/* Alt-N: the next older distinct match. At the oldest one there is +/* Alt-N: the next lower-ranked distinct match. At the last one there is nothing more: bell, and the ghost stays (no wrap). */ int rl_suggest_older(int count, int key) { @@ -75,15 +53,15 @@ int rl_suggest_older(int count, int key) rl_suggest_cyc_sync(state, l); if (!l->sg_cyc_line && !sg_cyc_begin(state, l)) return (rl_suggest_bell()); - if (l->sg_cyc_k + 1 >= l->sg_cyc.len && !sg_push_older(state, l)) + if (l->sg_cyc_k + 1 >= l->sg_rank.len) return (rl_suggest_bell()); l->sg_cyc_k++; - sg_show(state, ((size_t *)l->sg_cyc.ctx)[l->sg_cyc_k]); + sg_show(state, ((size_t *)l->sg_rank.ctx)[l->sg_cyc_k]); return (0); } -/* Alt-P: back to the newer alternative shown before. At the newest match - there is nothing newer: bell. */ +/* Alt-P: back to the better-ranked alternative shown before. At the best + match there is nothing better: bell. */ int rl_suggest_newer(int count, int key) { t_shell *state; @@ -97,6 +75,6 @@ int rl_suggest_newer(int count, int key) if (!l->sg_cyc_line || l->sg_cyc_k == 0) return (rl_suggest_bell()); l->sg_cyc_k--; - sg_show(state, ((size_t *)l->sg_cyc.ctx)[l->sg_cyc_k]); + sg_show(state, ((size_t *)l->sg_rank.ctx)[l->sg_cyc_k]); return (0); } diff --git a/src/platform/posix/rl_suggest8.c b/src/platform/posix/rl_suggest8.c new file mode 100644 index 00000000..a2db82b4 --- /dev/null +++ b/src/platform/posix/rl_suggest8.c @@ -0,0 +1,118 @@ +/* ************************************************************************** */ +/* */ +/* ::: :::::::: */ +/* rl_suggest8.c :+: :+: :+: */ +/* +:+ +:+ +:+ */ +/* By: dlesieur +#+ +:+ +#+ */ +/* +#+#+#+#+#+ +#+ */ +/* Created: 2026/10/17 12:00:00 by dlesieur #+# #+# */ +/* Updated: 2026/10/17 12:00:00 by dlesieur ### ########.fr */ +/* */ +/* ************************************************************************** */ + +#include "rl_private.h" +#include + +/* libc qsort, not ft_qsort: ft_qsort pivots on the last element, so a run +** of identical entries -- already in ranking order -- costs O(n^2), 6.8 s +** for one key over 20000 copies (autosuggest_test F13). libc is +** O(n log n) on glibc, musl and macOS, and both comparators here are +** total orders, so every libc gives the same ranking. */ + +/* The ranking behind the ghost and Alt-N / Alt-P: the distinct texts of +** the candidates (rl_suggest_cands), best first. No hash table: libft's +** hash dups every key, which is one malloc per candidate per key. Sorting +** the reused scratch by text puts equal texts side by side, so one walk +** scores each group; a second sort orders the groups. */ + +/* Descending order for a comparator: the larger sorts first. */ +static int sg_three(uint64_t x, uint64_t y) +{ + if (x == y) + return (0); + if (x < y) + return (1); + return (-1); +} + +/* Text first, and within one text the newest occurrence first, so a group + starts with the entry that will stand for it. */ +static int sg_cmp_text(const void *x, const void *y) +{ + const t_sg_ent *a; + const t_sg_ent *b; + int c; + + a = x; + b = y; + c = ft_strcmp(a->text, b->text); + if (c) + return (c); + return (sg_three(a->idx, b->idx)); +} + +/* Best score first; on a tie the newest. The representatives are distinct + entries, so this is a strict total order and the sort needs no + stability. */ +static int sg_cmp_rank(const void *x, const void *y) +{ + const t_sg_ent *a; + const t_sg_ent *b; + + a = x; + b = y; + if (a->score != b->score) + return (sg_three(a->score, b->score)); + return (sg_three(a->idx, b->idx)); +} + +/* Collapse each run of equal texts to its first entry (the newest) holding + the run's score, in place: the write cursor never passes the read one. + Returns the number of distinct texts. */ +static size_t sg_fold(t_sg_ent *s, size_t n, size_t len, bool recent) +{ + size_t r; + size_t w; + + r = 0; + w = 0; + while (r < n) + { + s[w] = s[r]; + s[w].score = 0; + while (r < n && !ft_strcmp(s[r].text, s[w].text)) + s[w].score = hist_score_add(s[w].score, len, s[r++].idx, + recent); + w++; + } + return (w); +} + +/* Rank the history for the line: sg_rank gets the history index of each +** distinct candidate text, best first (score, then newest). The newest +** occurrence stands for its text. Returns how many. +** +** `recent` is today's order exactly: newest first, repeats collapsed. +** Only indices are kept: the scratch holds pointers into the history +** that a later `history -c` would leave dangling. +** +** Cost: O(m log m) for m candidates; no allocation once the two vectors +** have grown to the window. */ +size_t rl_suggest_rank(t_shell *state, t_rl *l) +{ + t_sg_ent *s; + size_t n; + size_t i; + + rl_suggest_cyc_clear(l); + l->sg_rank.elem_size = sizeof(size_t); + n = rl_suggest_cands(state, l); + s = (t_sg_ent *)l->sg_scr.ctx; + qsort(s, n, sizeof(*s), sg_cmp_text); + n = sg_fold(s, n, state->hist.hist_cmds.len, l->sg_recent); + qsort(s, n, sizeof(*s), sg_cmp_rank); + i = 0; + while (i < n) + vec_push(&l->sg_rank, &s[i++].idx); + return (l->sg_rank.len); +} diff --git a/tests/autosuggest_test.py b/tests/autosuggest_test.py index 5add7bf8..1e2fa94e 100644 --- a/tests/autosuggest_test.py +++ b/tests/autosuggest_test.py @@ -16,6 +16,13 @@ matching entries, no wrap (bell at either end), reset by any edit. With no history ghost these keys are their stock readline functions. +Ranking (HELLISH_SUGGEST_STRATEGY): by default the entry with the best +frecency (how often and how recently its text was run; weight 2^16 halved +per 64 newer entries, summed per distinct text, ties to the newest) is the +ghost and the order Alt-N / Alt-P walk; `recent` is newest first. `model()` +below is the same rule written again in Python, and the cases check the +shell against it as well as against hand-computed answers. + Usage: python3 autosuggest_test.py /path/to/hellish [case-name-part [inproc]] """ import fcntl @@ -47,6 +54,7 @@ "\x06", "\x15", "\x03", "\x04", "\x12", "\x07") BKSP = "\x7f" ON = "HELLISH_AUTOSUGGEST=1" +STRAT = "HELLISH_SUGGEST_STRATEGY" try: import pyte @@ -717,17 +725,32 @@ def case_cycle(fork, tag): def case_cycle_dupes(fork, tag): - s = shell(fork, ["echo a1", "echo a2", "echo a1", "echo a3"]) + """The pre-frecency rule, pinned: HELLISH_SUGGEST_STRATEGY=recent.""" + s = shell(fork, ["echo a1", "echo a2", "echo a1", "echo a3"], + env={STRAT: "recent"}) s.keys("echo a") seen = [s.ghost()] for _ in range(4): s.keys(ALT_N) seen.append(s.ghost()) - check(tag + "30 C2 distinct entries, once each, then the end", + check(tag + "30 C2 recent: distinct entries, once each, then the end", seen == ["3", "1", "2", "2", "2"], repr(seen)) s.close() +def case_cycle_dupes_frecent(fork, tag): + """Same history, default strategy: a1 was run twice, so it leads.""" + s = shell(fork, ["echo a1", "echo a2", "echo a1", "echo a3"]) + s.keys("echo a") + seen = [s.ghost()] + for _ in range(4): + s.keys(ALT_N) + seen.append(s.ghost()) + check(tag + "30 C2 frecent: twice-run a1 first, once each, the end", + seen == ["1", "3", "2", "2", "2"], repr(seen)) + s.close() + + def case_cycle_reset(fork, tag): s = three(fork, ["echo ab9"]) s.keys("echo a") @@ -945,6 +968,295 @@ def case_cycle_history_shrinks(fork, tag): +def clean_text(t): + """The ghost's filter: no control byte, no DEL, no C1 (as UTF-8).""" + return not any(ord(c) < 0x20 or ord(c) == 0x7f or 0x80 <= ord(c) < 0xa0 + for c in t) + + +def model(lines, prefix, strategy="frecent", mx=1000): + """The ranking, written again: distinct texts, best first.""" + n = len(lines) + score, newest = {}, {} + for i in range(max(0, n - mx), n): + t = lines[i] + if not (t.startswith(prefix) and len(t) > len(prefix) + and clean_text(t)): + continue + if strategy == "recent": + score[t] = i + 1 + else: + score[t] = score.get(t, 0) + (65536 >> min((n - 1 - i) // 64, 16)) + newest[t] = i + return sorted(score, key=lambda t: (-score[t], -newest[t])) + + +def seeded(fork, lines, env=None, **kw): + """A shell whose history is exactly `lines` (oldest first): nothing is + run, so no other entry is in it, and autosuggest is on from the start.""" + home = tempfile.mkdtemp(prefix="autosuggest_") + with open(os.path.join(home, "hist"), "w") as f: + f.write("".join(line + "\n" for line in lines)) + e = {"HELLISH_AUTOSUGGEST": "1"} + e.update(env or {}) + return Session(home, fork=fork, env=e, **kw) + + +def walk(s, typed, steps): + """The ghost after `typed`, then after each of `steps` Alt-N.""" + s.keys(typed) + seen = [s.ghost()] + for _ in range(steps): + s.keys(ALT_N) + seen.append(s.ghost()) + return seen + + +# re is run three times, clean twice and newest, test once: frecency says +# re, the newest entry says clean. +ISSUE = ["make re", "make test", "make re", "make clean", "make re", + "make clean"] + + +def case_frecent_issue(fork, tag): + s = seeded(fork, ISSUE) + got = walk(s, "make ", 3) + check(tag + "37 F1 frecent: the most-run command, not the newest", + got == ["re", "clean", "test", "test"], repr(got)) + s.close() + s = seeded(fork, ISSUE, env={STRAT: "recent"}) + got = walk(s, "make ", 3) + check(tag + "37 F1 recent: the newest, repeats collapsed", + got == ["clean", "re", "test", "test"], repr(got)) + s.close() + + +def case_frecent_recent(fork, tag): + lines = ["echo a1", "echo b1", "echo a2", "echo b2", "echo a3"] + s = seeded(fork, lines, env={STRAT: "recent"}) + got = walk(s, "echo ", 4) + check(tag + "38 F2 recent is newest first, as before frecency", + got == ["a3", "b2", "a2", "b1", "a1"], repr(got)) + s.close() + + +def case_frecent_default(fork, tag): + for val in ("", "bogus", "RECENT", "recent "): + s = seeded(fork, ISSUE, env={STRAT: val}) + got = walk(s, "make ", 0) + check(tag + "39 F3 %r is frecent" % val, got == ["re"], repr(got)) + s.close() + s = seeded(fork, ISSUE) + got = walk(s, "make ", 0) + check(tag + "39 F3 unset is frecent", got == ["re"], repr(got)) + s.close() + + +def case_frecent_live(fork, tag): + s = seeded(fork, ISSUE) + steps = (("", "re"), (STRAT + "=recent", "clean"), + (STRAT + "=bogus", "re"), (STRAT + "=recent", "clean"), + ("unset " + STRAT, "re")) + for cmd, want in steps: + if cmd: + s.run(cmd) + got = walk(s, "make ", 0) + check(tag + "40 F4 after %r the ghost is %r" % (cmd, want), + got == [want], repr(got)) + s.keys(CTRL_U) + s.close() + + +def case_frecent_tie(fork, tag): + lines = ["echo x2", "echo x1", "echo x2", "echo x1"] + s = seeded(fork, lines) + got = walk(s, "echo x", 2) + check(tag + "41 F5 equal scores: the newest first", + got == ["1", "2", "2"], repr(got)) + s.close() + s = seeded(fork, lines[1:] + ["echo x2"]) + got = walk(s, "echo x", 2) + check(tag + "41 F5 equal scores, other way round", + got == ["2", "1", "1"], repr(got)) + s.close() + + +def decay_lines(total): + """fa twice at the very start, filler, fb newest: `total` entries.""" + lines = ["echo fa", "true 0", "echo fa"] + lines += ["true %d" % n for n in range(1, total - 3)] + return lines + ["echo fb"] + + +def case_frecent_decay(fork, tag): + # Two fa at ages total-1 and total-3. Weight 2^16 halves each 64 newer + # entries: at 66 entries (ages 65, 63) 2^15 + 2^16 beats fb's 2^16; at + # 67 (ages 66, 64) both have halved, 2^16 ties, and the newest wins. + for total, want in ((20, "fa"), (66, "fa"), (67, "fb"), (300, "fb")): + lines = decay_lines(total) + s = seeded(fork, lines) + got = walk(s, "echo f", 0) + check(tag + "42 F6 %d entries: %s wins" % (total, want), + got == [want[1]] and model(lines, "echo f")[0] == "echo " + want, + repr(got)) + s.close() + + +def case_frecent_window(fork, tag): + lines = ["echo wa", "echo wb", "echo wa", "echo wb", "echo wa", + "echo wc", "echo wb"] + s = seeded(fork, lines) + got = walk(s, "echo w", 0) + check(tag + "43 F7 whole history: wa 3, wb 3, wc 1: the tie goes to " + "the newest", got == ["b"], repr(got)) + s.close() + s = seeded(fork, lines, env={"HELLISH_SUGGEST_MAX": "3"}) + got = walk(s, "echo w", 3) + check(tag + "43 F7 MAX=3: wa, wc, wb once each, newest first", + got == ["b", "c", "a", "a"], repr(got)) + s.close() + lines = ["echo wa", "echo wa", "echo wb", "echo wa", "echo wb", + "echo wc"] + s = seeded(fork, lines) + got = walk(s, "echo w", 0) + check(tag + "43 F7 MAX unset: wa x3 wins", got == ["a"], repr(got)) + s.close() + s = seeded(fork, lines, env={"HELLISH_SUGGEST_MAX": "4"}) + got = walk(s, "echo w", 2) + check(tag + "43 F7 MAX=4: the two older wa do not count, wb x2 wins", + got == ["b", "c", "a"], repr(got)) + s.close() + + +def case_frecent_cycle(fork, tag): + s = seeded(fork, ISSUE) + s.keys("make ") + order = [] + for _ in range(3): + order.append(s.ghost()) + s.keys(ALT_N) + check(tag + "44 F8 Alt-N follows the score", order == ["re", "clean", + "test"] and s.ghost() == "test", repr(order)) + s.keys(ALT_N) + check(tag + "44 F8 bell at the last, ghost stays", + b"\a" in s.since() and s.ghost() == "test", repr(s.since())) + s.keys(ALT_P + ALT_P) + check(tag + "44 F8 Alt-P walks back to the best", + s.ghost() == "re" and s.row() == PROMPT + "make re", + repr(s.row())) + s.keys(ALT_P) + check(tag + "44 F8 bell at the best, no wrap", + b"\a" in s.since() and s.ghost() == "re", repr(s.since())) + s.close() + + +def case_frecent_dirty(fork, tag): + lines = ["echo ok", "echo \x1b[31mred", "echo \x1b[31mred", + "echo \x1b[31mred", "echo ok2"] + s = seeded(fork, lines) + got = walk(s, "echo ", 3) + check(tag + "45 F9 a frequent entry holding ESC is never offered", + got == ["ok2", "ok", "ok", "ok"] and b"\x1b[31m" not in s.buf, + repr(got)) + s.close() + + +def case_frecent_history_shrinks(fork, tag): + s = seeded(fork, ISSUE) + s.run("bind -x '\"\\C-t\": history -c'") + s.keys("make ") + s.keys(ALT_N) + check(tag + "46 F10 cycled before the clear", s.ghost() == "clean", + repr(s.ghost())) + s.keys("\x14") + s.keys(ALT_N) + check(tag + "46 F10 no ghost survives a cleared history (Alt-N)", + s.ghost() == "" and s.alive(), repr(s.ghost())) + s.keys(CTRL_U + "make ") + s.keys(ALT_N + ALT_P) + check(tag + "46 F10 nor a cycle begun afterwards", + s.ghost() == "" and s.alive(), repr(s.ghost())) + s.keys("\x07\x15") + s.run("echo ok") + check(tag + "46 F10 the shell still runs lines", + "ok" in s.screen_text(), repr(s.screen_text()[-200:])) + s.close() + + +def case_frecent_typing(fork, tag): + long_a = "echo abcdefghijklmnopqrstuvwxyz0123456789" + long_b = "echo abcdefgXYZ0123456789abcdefghij" + lines = [long_a, long_b, long_a, "echo abc", long_a] + s = seeded(fork, lines) + bad = [] + for path in (long_a, long_b): + s.keys(CTRL_U + path[:4]) + for k in range(5, len(path) + 1): + s.keys(path[k - 1]) + want = model(lines, path[:k]) + want = want[0][k:] if want else "" + if s.ghost() != want: + bad.append((path[:k], s.ghost(), want)) + check(tag + "47 F11 typing forward: every ghost is the best match for " + "what is typed (%d keys)" % (len(long_a) + len(long_b) - 10), + not bad, repr(bad[:3])) + s.close() + + +def random_history(rng): + words = ["alpha", "alps", "alt", "amber", "ant"] + n = rng.choice([8, 40, 70, 130, 200]) + lines = [] + for i in range(n): + w = rng.choice(words) + if rng.random() < 0.5: + lines.append("echo %s" % w) + else: + lines.append("echo %s %d" % (w, rng.randrange(3))) + if i and lines[-1] == lines[-2]: + lines[-1] += " x" + return lines + + +def case_frecent_model(fork, tag): + import random + rng = random.Random(140 + int(fork)) + for k in range(5): + lines = random_history(rng) + mx = rng.choice([1000, 1000, 50]) + prefix = rng.choice(["echo ", "echo al", "echo a"]) + s = seeded(fork, lines, env={"HELLISH_SUGGEST_MAX": str(mx)}) + want = model(lines, prefix, mx=mx) + got = walk(s, prefix, len(want) + 1) + exp = [t[len(prefix):] for t in want] + exp = (exp + exp[-1:] * 2)[:len(want) + 2] if exp else [""] + exp = exp[:len(got)] if exp != [""] else [""] * len(got) + check(tag + "48 F12 random history %d (%d entries, max %d, %r): " + "the cycle equals the model" % (k, len(lines), mx, prefix), + got == exp, "got %r\nwant %r" % (got, exp)) + s.close() + + + +def case_frecent_sorted_worst(fork, tag): + """20000 copies of one entry arrive already in ranking order, the worst + case of a last-element-pivot quicksort (O(n^2)): the first key must + still paint its ghost quickly.""" + n = 20000 + s = seeded(fork, ["ls -la"] * n + ["pwd"], + env={"HELLISH_SUGGEST_MAX": str(n + 1), + "HISTSIZE": str(n + 10), "HISTFILESIZE": str(n + 10)}) + s.mark = len(s.buf) + t0 = time.time() + os.write(s.fd, b"l") + s.until(lambda: s.ghost() == "s -la", 20.0) + dt = time.time() - t0 + check(tag + "37 F13 sorted worst case paints within 1.5 s", + s.ghost() == "s -la" and dt < 1.5, "%r %.2fs" % (s.ghost(), dt)) + s.close() + + + CASES = (case_off_by_default, case_on, case_accept, case_enter_plain, case_newest, case_edit, case_cursor, case_interrupt, case_history_keys, case_search, case_vi, case_escape, @@ -952,10 +1264,16 @@ def case_cycle_history_shrinks(fork, tag): case_paste, case_max, case_unset, case_user_bind, case_postdisplay, case_resize, case_dumb, case_word, case_word_ctrl_right, case_word_blanks, case_word_stock, - case_word_utf8, case_cycle, case_cycle_dupes, case_cycle_reset, + case_word_utf8, case_cycle, case_cycle_dupes, + case_cycle_dupes_frecent, case_cycle_reset, case_cycle_accept, case_cycle_max, case_cycle_stock, case_cycle_post, case_cycle_user_bind, case_cycle_off, - case_cycle_vi, case_cycle_history_shrinks) + case_cycle_vi, case_cycle_history_shrinks, case_frecent_issue, + case_frecent_recent, case_frecent_default, case_frecent_live, + case_frecent_tie, case_frecent_decay, case_frecent_window, + case_frecent_cycle, case_frecent_dirty, + case_frecent_history_shrinks, case_frecent_typing, + case_frecent_model, case_frecent_sorted_worst) def main(): diff --git a/tests/complete_make_test.py b/tests/complete_make_test.py index 236da2a9..53fab79b 100755 --- a/tests/complete_make_test.py +++ b/tests/complete_make_test.py @@ -96,7 +96,7 @@ def drain_idle(fd, chunks, quiet=0.35, cap=8.0): return chunks -def spawn(cwd, home): +def spawn(cwd, home, cols=200, extra=None): env = { "HOME": home, "PATH": "/usr/bin:/bin", "TERM": "xterm-256color", "LANG": "C.UTF-8", "PS1": "$ ", @@ -106,6 +106,7 @@ def spawn(cwd, home): "ASAN_OPTIONS": "detect_leaks=%d" % (1 if ASAN else 0), } env.update(EXTRA) + env.update(extra or {}) pid, fd = pty.fork() if pid == 0: # nothing may leave this block except _exit (see @@ -118,16 +119,22 @@ def spawn(cwd, home): except BaseException: pass os._exit(127) - fcntl.ioctl(fd, termios.TIOCSWINSZ, struct.pack("HHHH", 60, 200, 0, 0)) + fcntl.ioctl(fd, termios.TIOCSWINSZ, struct.pack("HHHH", 60, cols, 0, 0)) return pid, fd -def session(cwd, steps, settle=0.6): +def session(cwd, steps, settle=0.6, hist=None, env=None, cols=200): """Run `steps` (bytes to type, or a callable run between keystrokes) - in a fresh shell; return the output of each bytes step, in order.""" + in a fresh shell; return the output of each bytes step, in order. + `hist` is the history the shell starts with, oldest first.""" home = tempfile.mkdtemp(prefix="hellish_mk_home_") TMP.append(home) - pid, fd = spawn(cwd, home) + extra = dict(env or {}) + if hist is not None: + extra["HISTFILE"] = os.path.join(home, "hist") + with open(extra["HISTFILE"], "w") as f: + f.write("".join(h + "\n" for h in hist)) + pid, fd = spawn(cwd, home, cols, extra) chunks = [] drain_idle(fd, chunks, quiet=0.4, cap=6.0) outs = [] @@ -177,6 +184,19 @@ def listing(out): return set(ANSI.sub("", block).split()) +def ordered(out): + """Like listing(), but in the order printed. Needs a terminal so + narrow (session(..., cols=10)) that readline prints one name a row.""" + i = out.find("\a\r\n") + if i < 0: + return [] + block = out[i + 3:] + j = block.find("\r\n$ ") + if j >= 0: + block = block[:j] + return ANSI.sub("", block).split() + + def line_is(out, want): return want in ANSI.sub("", out).replace("\a", "") @@ -453,6 +473,86 @@ def noperm(d): b.stdout, b.returncode)) +# `re` is run three times, `clean` twice and last, `test` once: frecency +# says re, clean, test; the newest entry says clean, re, test. +HIST = ["make re", "make test", "make re", "make clean", "make re", + "make clean"] +RANKED = "all clean fclean lint re test".split() +UNUSED = ["all", "fclean", "lint"] + + +def ranked_listing(hist, typed=b"make ", env=None, files=None): + d = fixture(dict({mk: "".join("%s:\n" % t for t in RANKED)}, + **(files or {}))) + out = session(d, [typed + b"\t\t\t"], hist=hist, env=env, cols=10)[0] + return ordered(out) + + +def ranked_cases(): + # 25 + got = ranked_listing(HIST) + check("25 used targets by frecency, then the rest alphabetically", + got == ["re", "clean", "test"] + UNUSED, str(got)) + got = ranked_listing(HIST, env={"HELLISH_SUGGEST_STRATEGY": "recent"}) + check("25 STRATEGY=recent: used targets newest first", + got == ["clean", "re", "test"] + UNUSED, str(got)) + got = ranked_listing([]) + check("25 no history: alphabetical", got == RANKED, str(got)) + got = ranked_listing(["echo re", "ls test", "cmake lint", "make"]) + check("25 only invocations of make count", got == RANKED, str(got)) + got = ranked_listing(["gmake lint", "/usr/bin/make test", "make lint"]) + check("25 gmake and a path to make count", + got == ["lint", "test", "all", "clean", "fclean", "re"], str(got)) + got = ranked_listing(["make re re re", "make lint test", "make test"]) + check("25 a word repeated inside one entry counts once", + got == ["test", "lint", "re", "all", "clean", "fclean"], str(got)) + out = comp_hist(HIST, b"make cl") + check("25 make cl still completes", line_is(out, "make clean "), + repr(out[-80:])) + + # 26 + got = ranked_listing(["make re", "make re", "make re", "make fclean", + "make test"], env={"HELLISH_SUGGEST_MAX": "2"}) + check("26 the window bound applies to make too (MAX=2)", + got == ["test", "fclean", "all", "clean", "lint", "re"], str(got)) + + # 27 + many = {"f%02d" % n: "x" for n in range(30)} + d = fixture(dict({mk: "".join("%s:\n" % t for t in RANKED)}, **many)) + outs = session(d, [b"make \t\t\t", KILL, b"ls f\t\t\t", KILL, + b"bas\t\t\t"], hist=HIST, cols=10) + mk_order = ordered(outs[0]) + files = ordered(outs[2]) + cmds = ordered(outs[4]) + check("27 make TAB ranked", mk_order == ["re", "clean", "test"] + + UNUSED, str(mk_order)) + check("27 then a filename TAB: sorted again, not left unsorted", + len(files) >= 30 and files == sorted(files), str(files[:8])) + check("27 then a command TAB: sorted again", + len(cmds) >= 2 and cmds == sorted(cmds), str(cmds[:8])) + + # 28 + d = fixture({mk: "all:\nclean:\n"}) + outs = session(d, [b"shopt -s progcomp\n", + b"complete -W 'zz yy' make\n", b"make \t\t\t"], + hist=["make clean", "make all", "make clean"]) + check("28 a user complete -W still wins over the ranking", + listing(outs[2]) == {"zz", "yy"}, str(sorted(listing(outs[2])))) + + # 29 + files = {"sub/Makefile": "s1:\n", "other.mk": "o1:\n"} + for typed in (b"make -C sub ", b"make -f other.mk "): + got = ranked_listing(HIST, typed, files=files) + check("29 %s keeps filenames with a history" % typed.decode(), + set(got) == {mk, "sub/", "other.mk"} + and got == sorted(got), str(got)) + + +def comp_hist(hist, typed): + d = fixture({mk: "".join("%s:\n" % t for t in RANKED)}) + return session(d, [typed + b"\t"], hist=hist)[0] + + def main(): if not os.access(SHELL, os.X_OK): sys.exit("complete_make_test: not executable: %s" % SHELL) @@ -465,6 +565,8 @@ def main(): EXTRA.clear() EXTRA.update(env) cases() + if not ASAN: + ranked_cases() finally: for d in TMP: shutil.rmtree(d, ignore_errors=True) diff --git a/tests/frontend_budget_test.py b/tests/frontend_budget_test.py index 39918ef5..f0563182 100755 --- a/tests/frontend_budget_test.py +++ b/tests/frontend_budget_test.py @@ -46,6 +46,9 @@ MARK = b"#> " ENTERS = 20 KEYS = 40 +ROUNDS = 12 +DIM = b"\x1b[2m" +HIST_N = 1000 TRACE_LINE = re.compile(r"^(\d+)\s+(\d+\.\d+)\s+([a-z_0-9]+)") # name -> (per Enter, per key, idle total, writes per resize phase) @@ -63,6 +66,11 @@ BUDGET["in-process/as"] = dict(BUDGET["in-process"], nomatch_write=1.2) BUDGET["forked/as+rp"] = dict(BUDGET["forked"], nomatch_write=2.2) BUDGET["forked/as"] = dict(BUDGET["forked"], nomatch_write=1.2) +# Frecency (#140): a history of 1000 distinct entries, all matching the +# first key, so every Backspace ranks them all. Same per-key budget: the +# ranking is CPU, and reuses its buffers instead of asking the kernel. +BUDGET["in-process/frecent"] = dict(BUDGET["in-process"], nomatch_write=1.2) +BUDGET["forked/frecent"] = dict(BUDGET["forked"], nomatch_write=1.2) SEED = "x" * 60 AS_ON = {"HELLISH_AUTOSUGGEST": "1"} FORKING = ("clone", "clone3", "fork", "vfork", "pipe2", "pipe", "wait4", @@ -77,7 +85,7 @@ def check(name, ok, detail=""): class Traced: - def __init__(self, home, trace, extra, seed=()): + def __init__(self, home, trace, extra, seed=(), hist=()): env = { "HOME": home, "PATH": os.environ.get("PATH", "/usr/bin:/bin"), "TERM": "xterm-256color", "LANG": "C.UTF-8", "LC_ALL": "C.UTF-8", @@ -87,6 +95,10 @@ def __init__(self, home, trace, extra, seed=()): "HISTFILE": "/dev/null", "ASAN_OPTIONS": "detect_leaks=0", } env.update(extra) + if hist: + env["HISTFILE"] = os.path.join(home, "hist") + with open(env["HISTFILE"], "w") as f: + f.write("".join(h + "\n" for h in hist)) self.buf = bytearray() self.phases = [] self.pid, self.fd = pty.fork() @@ -148,6 +160,19 @@ def typing(self, ch=b"x"): os.write(self.fd, b"\x15") self.settle() + def scanning(self): + """Forward keys and Backspace over a long history: each Backspace + ends a cached answer, so the ranking runs; forward keys must not.""" + self.mark = len(self.buf) + for _ in range(ROUNDS): + for key in (b"x", b"x", b"x", b"x", b"\x7f"): + os.write(self.fd, key) + self.pump(0.03) + self.settle() + self.dim = DIM in bytes(self.buf[self.mark:]) + os.write(self.fd, b"\x15") + self.settle() + def idle(self): time.sleep(2.0) @@ -184,14 +209,16 @@ def counts(lines, t0, t1): return c -def cell(label, extra, base, seed=()): +def cell(label, extra, base, seed=(), hist=()): trace = os.path.join(base, "trace-" + label.replace("/", "_")) home = tempfile.mkdtemp(dir=base) - s = Traced(home, trace, extra, seed) + s = Traced(home, trace, extra, seed, hist) s.phase("enter", s.enters) s.phase("idle", s.idle) s.phase("key", s.typing) s.phase("nomatch", lambda: s.typing(b"q")) + if hist: + s.phase("scan", s.scanning) s.phase("winch", s.winch_same) s.phase("rows", s.rows_only) s.close() @@ -234,6 +261,19 @@ def cell(label, extra, base, seed=()): check("%s: a key with nothing to paint does at most %.1f writes" % (label, b["nomatch_write"]), n <= b["nomatch_write"], "%.1f per key" % n) + if hist: + n = sum(by["scan"].values()) / float(ROUNDS * 5) + w = by["scan"]["write"] / float(ROUNDS * 5) + print("# %s: %d distinct entries, per key %.1f syscalls, " + "%.1f writes" % (label, len(hist), n, w)) + check("%s: a key over %d distinct entries, Backspace included, " + "costs at most %d syscalls" % (label, len(hist), b["key"]), + n <= b["key"], "%.1f per key: %s" % ( + n, dict(by["scan"].most_common(6)))) + check("%s: and at most %d writes" % (label, b["key_write"]), + w <= b["key_write"], "%.1f per key" % w) + check("%s: the ghost was painted over that history" % label, + s.dim, "no dim ghost in the output") check("%s: an idle prompt costs nothing" % label, sum(by["idle"].values()) == 0, "%d syscalls while idle: %s" % (sum(by["idle"].values()), @@ -258,6 +298,10 @@ def main(): cell("forked/as+rp", dict(AS_ON, HELLISH_RL_FORK="1"), base, seed) cell("forked/as", dict(AS_ON, RPROMPT="", HELLISH_RL_FORK="1"), base, seed) + hist = ["xxxx%04d" % n for n in range(HIST_N)] + cell("in-process/frecent", dict(AS_ON, RPROMPT=""), base, (), hist) + cell("forked/frecent", dict(AS_ON, RPROMPT="", HELLISH_RL_FORK="1"), + base, (), hist) finally: shutil.rmtree(base, ignore_errors=True) if FAILS: diff --git a/wiki/manual.md b/wiki/manual.md index 6a6d6592..2fb57448 100644 --- a/wiki/manual.md +++ b/wiki/manual.md @@ -175,6 +175,7 @@ inside the shell: `help` for the grouped list, `help NAME` for one, | `HELLISH_HISTORY_PREFIX=0\|off` | Up/Down walk plain history; default on = search by the text before the cursor; read at every prompt | | `HELLISH_AUTOSUGGEST=1\|on` | inline suggestion (off by default): the newest history line starting with the typed text, dim after the cursor; Right / End / Ctrl-F insert it, Alt-F / Ctrl-Right insert its next word (leading blanks + the run up to the next space or tab), Alt-N / Alt-P cycle through older / newer distinct matches (bell at either end, no wrap; any edit resets); never a line with control characters or a newline; read at every prompt | | `HELLISH_SUGGEST_MAX=N` | history lines the suggestion scans, newest first (default 1000; non-numeric or below 1 = default); read at every prompt | +| `HELLISH_SUGGEST_STRATEGY=recent` | newest-first suggestions; default (unset/other) = frecency (run count weighted toward recent runs), also ordering `make `; read at every prompt | | `POSTDISPLAY` | zle widgets only: the text painted after the cursor, i.e. the suggestion; assign it to show your own until the buffer next changes | | `HELLISH_NO_UPDATE_CHECK=1` | never check for releases; no badge | | `HELLISH_NO_ANIM=1` | skip the startup animation | From 3432f8ad6fc43146399824f1392b12802c91bb1f Mon Sep 17 00:00:00 2001 From: qwO-Owp <332893456+qwO-Owp@users.noreply.github.com> Date: Sun, 18 Oct 2026 21:35:24 +0200 Subject: [PATCH 02/10] docs(release): draft the v3.3.0 notes (frecency, #140) Unreleased: develop carries the change, main does not yet. The notes are ready for whoever cuts the release. --- RELEASE.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/RELEASE.md b/RELEASE.md index 2eea7adc..9becb480 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -9,6 +9,20 @@ shell. The welcome banner links to the guide for the latest one: --- +## v3.3.0 — *the hint knows what you run most* (unreleased) + +- **Hints are ranked by frecency.** With `HELLISH_AUTOSUGGEST=1`, the dim + hint is the matching history line you run most, weighted toward recent + runs, and no longer simply the newest one. Run `make re` three times and + `make clean` once more recently, and `make ` hints `re`. Alt-N / Alt-P + step through the matches in the same order (#140). +- **`make ` lists the targets you run first**, most-run first, then + the rest alphabetically (#140). +- `HELLISH_SUGGEST_STRATEGY=recent` brings back newest-first for both. It + is read at every prompt. + +--- + ## v3.2.0 — *Up finds what you typed, a hint is one key away, and `bind`* **The guide: [What's new in 3.2](https://univers42.github.io/hellish/whats-new/)**. From 397a151aa7f015bac9f18774f7fc20ed6ba60cb9 Mon Sep 17 00:00:00 2001 From: LESdylan Date: Wed, 4 Nov 2026 15:44:37 +0100 Subject: [PATCH 03/10] fix(readline): Keep a ^C from readline while ^R, Alt-P or M-x is up readline 8.3 made rl_free_line_state free the context of those three readers on SIGINT, on the bet that an application's handler longjmps and the reader never runs again. bash does, and readline83-003 taught readline's own rl_getc to abort to the top level for handlers that return. rl_getc_hook replaces that rl_getc and hands the reader EOF, so the reader unwound through a freed `cxt`. On readline 8.3 at Debian's patch level: freed by: rl_getc_hook -> rl_take_signals -> rl_check_signals -> _rl_handle_signal -> rl_free_line_state -> _rl_state_sigcleanup -> _rl_isearch_cleanup -> _rl_scxt_dispose double free: rl_search_history -> _rl_isearch_cleanup -> _rl_scxt_dispose ASan reports the double free for ^R and M-x. Under a readline built with ASan all three readers report a heap-use-after-free WRITE first, in _rl_search_getchar. A release build takes a SIGSEGV on ^R, 3 runs of 3, which is the login shell gone. The `ps2` failures in CI were the same crash: that scenario shares the session ^R killed. Guarding only rl_take_signals' own look is not enough, and the tests say so: readline also looks at its deferred signal at the end of every rl_redisplay, inside the search, and a ^C sent with the keys of a ^R came through there. rl_intr.c puts a filter in front of readline's SIGINT handler instead. While such a reader is up the signal is parked, the reader gets EOF and unwinds with its context intact, the path every readline before 8.3 took, and back in the main loop the ^C is replayed into readline. Outside those readers the filter only forwards, so ^C anywhere else is unchanged. Blocking SIGINT from the key's return to the next wait would also work, but needs lifting around every widget and completion, and a child forked on a path that misses it inherits it. prompt_interrupt_test.py gains nsearch, mx and redraw; rl_read_race_test.py gains nsearch and M-x. 48 and 34 ok on readline 8.1.2, 8.2.13 and 8.3 at patch levels 0, 3 and 6, and against a readline built with ASan. On the old code the same files fail on 8.3.3, the redraw scenario included. Closes #194 Co-Authored-By: Claude Opus 5.5 --- backlog.md | 21 ++++++ src/infrastructure/README.md | 5 ++ src/infrastructure/rl_private.h | 17 ++++- src/platform/posix/rl_getc.c | 20 +----- src/platform/posix/rl_intr.c | 109 ++++++++++++++++++++++++++++++++ src/platform/posix/rl_sig.c | 10 +-- tests/prompt_interrupt_test.py | 27 ++++++++ tests/rl_read_race_test.py | 6 +- 8 files changed, 189 insertions(+), 26 deletions(-) create mode 100644 src/platform/posix/rl_intr.c diff --git a/backlog.md b/backlog.md index 8fddf2a4..268bb4c7 100644 --- a/backlog.md +++ b/backlog.md @@ -9,6 +9,27 @@ pass counts in `bench/baseline/`). Performance claims come from ## Done (this program) +- [x] **^C inside ^R, Alt-P or M-x no longer kills the shell on readline 8.3 + (#194).** 8.3's `rl_free_line_state` frees those readers' contexts on + SIGINT, on the bet that the application's handler longjmps. bash does, + and readline83-003 taught readline's own `rl_getc` to abort to the top + level for handlers that return; `rl_getc_hook` replaces that `rl_getc` + and hands the reader EOF, so the reader unwound through a freed `cxt`: + a double free under ASan, a SIGSEGV on a release build (3 of 3 on ^R), + and every PR red when `ubuntu-latest` moved to 26.04 (readline 8.3-4) + on an unchanged tree. Guarding only `rl_getc_hook`'s own look was tried + first and is not enough: readline also looks at its deferred signal at + the end of every `rl_redisplay`, and a ^C sent with the keys of a ^R + came through there. `rl_intr.c` now puts a filter in front of + readline's SIGINT handler: while such a reader is up the signal is + parked, the reader unwinds on EOF with its context intact, and the ^C + is replayed in the main loop. Outside those readers the filter only + forwards, so ^C anywhere else, a slow completion included, is unchanged. + `prompt_interrupt_test.py` gains nsearch, mx and redraw, + `rl_read_race_test.py` gains nsearch and M-x. 48 and 34 ok on 8.1.2, + 8.2.13 and 8.3 at patch levels 0, 3 and 6, and under a readline built + with ASan, where the old code reported a heap-use-after-free WRITE in + `_rl_search_getchar` for all three readers. - [x] **`"$VAR"` reaches the expander fast path: −19.5% loop instructions** (callgrind, loop_arith: 1,266.9M → 1,019.2M Ir). `try_simple_envvar` (`src/expander/expand_word2.c`) gated on `t->tt != TT_ENVVAR`, so the diff --git a/src/infrastructure/README.md b/src/infrastructure/README.md index 22ac9e3d..9c092cc6 100644 --- a/src/infrastructure/README.md +++ b/src/infrastructure/README.md @@ -132,6 +132,11 @@ used to contain is now handled in place: tick on its timeout. A `SIGINT` readline handled during the read makes it return `READERR`, readline's way to end its main loop, so `readline()` returns `NULL` and the read reports "interrupted" (`status == 2`). +- A `SIGINT` while `^R`, `Alt-P` or `M-x` is up never reaches readline: + 8.3 frees those readers' contexts on `SIGINT`, expecting the handler to + longjmp, and this reader returns instead. A filter in front of readline's + handler parks the signal, the reader unwinds on `EOF`, and the `^C` is + replayed in the main loop (`rl_intr.c`). - Shell code run from the editor -- widgets, completion functions -- goes through `rl_shell_exec` (`rl_editor.c`): readline's signal handlers come down for the call, the terminal settings and `$?` are put back, and diff --git a/src/infrastructure/rl_private.h b/src/infrastructure/rl_private.h index 7fe01086..e0ed3783 100644 --- a/src/infrastructure/rl_private.h +++ b/src/infrastructure/rl_private.h @@ -145,6 +145,20 @@ bool rl_push_col(t_string *f, int n); # define RL_INPROC_OK 0 # endif +/* The readers that keep a heap context on their stack, the ones readline + 8.3 frees on SIGINT (rl_intr.c). The bit is spelled out for older + headers: a shared build runs on whatever libreadline.so.8 the machine + has, and before 8.3 the bit means nothing. */ +# ifndef RL_STATE_READSTR +# define RL_STATE_READSTR 0x10000000 +# endif +# define RL_CXT_READERS 0x10000180 + +/* rl_intr_cell: a ^C the filter kept from readline, not yet replayed. */ +# define RL_INTR_PARKED 2 + +typedef volatile sig_atomic_t t_intr; + /* SIGWINCH during a read (rl_winch.c, rl_winch2.c). readline's own resize path calls rl_forced_update_display() as soon as a redisplay function is installed -- and the RPROMPT painter installs one -- which appended a @@ -175,12 +189,13 @@ int rl_shell_leave(t_shell *state, t_rl_bracket *b); void rl_prerow_arm(t_string *rows); FILE *rl_out_open(void); int rl_getc_hook(FILE *stream); +void rl_take_signals(void); int rl_nsearch_reclaim(int c); int rl_idle_timeout(void); int rl_idle_fd(void); void rl_idle_event(int r); void rl_block(sigset_t *old); -int *rl_intr_cell(void); +t_intr *rl_intr_cell(void); int *rl_editor_abort_cell(void); bool rl_should_abort(void); int rl_abort_value(void); diff --git a/src/platform/posix/rl_getc.c b/src/platform/posix/rl_getc.c index 18e4e886..5f159c24 100644 --- a/src/platform/posix/rl_getc.c +++ b/src/platform/posix/rl_getc.c @@ -25,7 +25,7 @@ ** is what bash does. This one can: a SIGINT that readline handled during ** the read, or any other reason to stop (rl_should_abort), makes it ** return READERR, readline's own "stop reading" answer (readline.c), and -** readline() then returns NULL. +** readline() then returns NULL. rl_intr.c is where the signal is taken. ** ** It is also where the prompt's idle work runs -- the mascot animation ** ticks on a timeout -- since an rl_event_hook would make readline stop @@ -37,24 +37,6 @@ ** second, non-blocking descriptor where there is one (rl_keyfd.c), so the ** only place this ever waits is pselect. */ -/* A signal readline caught since the last look: let readline handle it - now -- cleanup, the application's handler, re-prep -- and remember an - interrupt, which ends this read. */ -static void rl_take_signals(void) -{ - int sig; - - sig = rl_pending_signal(); - if (!sig) - return ; - if (sig == SIGINT) - { - *rl_intr_cell() = 1; - rl_suggest_unpaint(); - } - rl_check_signals(); -} - static struct timespec *rl_timeout(struct timespec *ts) { int ms; diff --git a/src/platform/posix/rl_intr.c b/src/platform/posix/rl_intr.c new file mode 100644 index 00000000..9f3d612a --- /dev/null +++ b/src/platform/posix/rl_intr.c @@ -0,0 +1,109 @@ +/* ************************************************************************** */ +/* */ +/* ::: :::::::: */ +/* rl_intr.c :+: :+: :+: */ +/* +:+ +:+ +:+ */ +/* By: dlesieur +#+ +:+ +#+ */ +/* +#+#+#+#+#+ +#+ */ +/* Created: 2026/11/04 00:00:00 by dlesieur #+# #+# */ +/* Updated: 2026/11/04 00:00:00 by dlesieur ### ########.fr */ +/* */ +/* ************************************************************************** */ + +#include "rl_private.h" +#include + +/* What a ^C does to a read, in one place. +** +** readline 8.3 made rl_free_line_state free the context of ^R, Alt-P and +** M-x, on the bet that an application's SIGINT handler longjmps and that +** reader never runs again. bash does; readline's own rl_getc, since +** readline83-003, aborts to the top level for the handlers that return. +** This read does neither, by design (rl_getc.c): the reader is handed EOF +** and unwinds itself, through a `cxt` local that now points at freed +** memory. Issue #194, from CI on readline 8.3-4: +** +** rl_getc_hook -> rl_check_signals -> _rl_handle_signal +** -> rl_free_line_state -> _rl_isearch_cleanup -> free(cxt) +** rl_search_history -> _rl_isearch_cleanup(cxt) -> free(cxt) +** +** A double free under ASan, a use after free under an instrumented +** readline for all three readers, and on a release build a SIGSEGV that +** takes the login shell, 3 runs of 3 on ^R. +** +** readline runs that cleanup wherever it next looks at its deferred +** signal, and the look at the end of every rl_redisplay is inside the +** search, so guarding this file's own look is not enough: a ^C sent while +** ^R is drawing its keys came through there, and the test that caught it +** sends one. So readline must not hear of a ^C while such a reader is up. +** A filter goes in front of its handler and parks the signal; the reader +** gets EOF and unwinds with its context intact, the path every readline +** before 8.3 took, and back in the main loop the ^C is replayed into +** readline, which handles it as one that arrived there. Outside these +** readers the filter only forwards, so ^C anywhere else is what it always +** was, a slow completion included. +** +** Caveat: the filter reads the state bit, and the bit goes up only once +** the reader's command runs. A ^C landing after readline's last look at +** the signal and before that (a few hundred instructions, no system call +** in them) reaches readline as usual and frees the context at the next +** look. Blocking SIGINT from the key's return until the next wait would +** close it, but the block would need lifting around every widget and +** completion, and a child forked on a path that misses it inherits it. A +** context a later readline adds to _rl_state_sigcleanup is not in +** RL_CXT_READERS until someone puts it there. */ + +static struct sigaction *rl_readline_sigint(void) +{ + static struct sigaction act; + + return (&act); +} + +static void rl_sigint_filter(int sig) +{ + if (rl_readline_state & RL_CXT_READERS) + *rl_intr_cell() = RL_INTR_PARKED; + else + rl_readline_sigint()->sa_handler(sig); +} + +/* rl_set_signals installs readline's handler at the top of readline() and + again after a signal it handled or a widget's bracket (rl_editor.c), so + this looks every time. Its restore puts back what it saved then, never + the filter. A SIG_DFL here is the forked reader's child in its cleanup: + nothing to stand in front of. */ +static void rl_sigint_arm(void) +{ + struct sigaction act; + + if (sigaction(SIGINT, NULL, &act) != 0 + || act.sa_handler == rl_sigint_filter + || act.sa_handler == SIG_DFL || act.sa_handler == SIG_IGN) + return ; + *rl_readline_sigint() = act; + act.sa_handler = rl_sigint_filter; + sigaction(SIGINT, &act, NULL); +} + +/* A signal since the last look: let readline handle it now -- cleanup, + the application's handler, re-prep -- and remember an interrupt, which + ends this read. A parked ^C waits for its reader to be gone, and while + it waits rl_should_abort is what makes the reader go. */ +void rl_take_signals(void) +{ + int sig; + + if (*rl_intr_cell() == RL_INTR_PARKED + && !(rl_readline_state & RL_CXT_READERS)) + rl_readline_sigint()->sa_handler(SIGINT); + sig = rl_pending_signal(); + if (sig == SIGINT) + { + *rl_intr_cell() = 1; + rl_suggest_unpaint(); + } + if (sig) + rl_check_signals(); + rl_sigint_arm(); +} diff --git a/src/platform/posix/rl_sig.c b/src/platform/posix/rl_sig.c index a4b22b69..749a4fa7 100644 --- a/src/platform/posix/rl_sig.c +++ b/src/platform/posix/rl_sig.c @@ -15,12 +15,12 @@ /* Why a read ends early. */ -/* Set when readline handled a SIGINT during the current read. Written - from the reader, never from a signal handler: readline defers the - signal and our reader is where it is handled (rl_take_signals). */ -int *rl_intr_cell(void) +/* Set when a SIGINT reached this read. Written from the reader, which is + where readline's deferred signal is handled, and from the one handler + of ours in front of readline's (rl_intr.c). */ +t_intr *rl_intr_cell(void) { - static int intr; + static t_intr intr; return (&intr); } diff --git a/tests/prompt_interrupt_test.py b/tests/prompt_interrupt_test.py index ff9d987a..e6410a5e 100755 --- a/tests/prompt_interrupt_test.py +++ b/tests/prompt_interrupt_test.py @@ -28,6 +28,11 @@ burst three ^C in a row, then a command runs intact command ^C interrupts a foreground `sleep` isearch ^C inside ^R reverse search + nsearch ^C inside Alt-P non-incremental search + mx ^C inside M-x execute-named-command + redraw ^C while ^R is still drawing the keys sent with it, and a key + and ^C in one write, several times over: readline looks at the + signal at the end of every redraw, not only in the read ps2 ^C at a continuation prompt returns to PS1 with $? = 130 eof ^D on an empty line still exits @@ -181,6 +186,28 @@ def cell(label, home, extra): check("%s: isearch: ^C inside ^R, then a command runs" % label, s.expect(b"ISR_2"), repr(s.tail())) + s.send(b"\x1bpech") + s.until(lambda: b"ech" in s.buf[s.at:], 3.0) + interrupt(s) + s.send(b"echo NSR_$((2+1))\r") + check("%s: nsearch: ^C inside Alt-P, then a command runs" % label, + s.expect(b"NSR_3"), repr(s.tail())) + + s.send(b"\x1bxbeg") + s.until(lambda: b"beg" in s.buf[s.at:], 3.0) + interrupt(s) + s.send(b"echo MX_$((2+2))\r") + check("%s: mx: ^C inside M-x, then a command runs" % label, + s.expect(b"MX_4"), repr(s.tail())) + + for keys in (b"\x12e\x03", b"\x12\x03", b"\x12ec\x03") * 3: + s.send(keys) + s.until(lambda: b"^C" in s.buf[s.at:], 3.0) + s.prompt(5.0) + s.send(b"echo RD_$((4+4))\r") + check("%s: redraw: ^C sent with the keys of a ^R, nine times" % label, + s.expect(b"RD_8"), repr(s.tail())) + s.send(b"echo \"unterminated\r") s.until(lambda: b"> " in s.buf[s.at:], 3.0) s.pump(0.2) diff --git a/tests/rl_read_race_test.py b/tests/rl_read_race_test.py index a6e06447..ab118152 100644 --- a/tests/rl_read_race_test.py +++ b/tests/rl_read_race_test.py @@ -25,6 +25,8 @@ prompt a half-typed line, ^C in the gap, then a command isearch ^C in the gap inside a ^R search (the reported failure) + nsearch the same inside Alt-P, whose context readline 8.3 frees too + M-x the same inside execute-named-command Each runs with the in-process reader and with the forked one (HELLISH_RL_FORK=1), whose child inherits the same read. @@ -279,7 +281,9 @@ def main(): for label, extra in cells: for trigger, keys, what in ( (ord("x"), b"x; echo LEAK_$((6*7))", "prompt"), - (0x12, b"\x12ech", "isearch")): + (0x12, b"\x12ech", "isearch"), + (ord("p"), b"\x1bpech", "nsearch"), + (ord("x"), b"\x1bxbeg", "M-x")): r = scenario(label, shim, base, extra, trigger, keys, what) if r is None: print("skip: the shell never stalled in the shim -- its own " From 8fc4bcf04ee2c8674fb371f34bfd8e81f458ab98 Mon Sep 17 00:00:00 2001 From: LESdylan Date: Wed, 4 Nov 2026 15:44:37 +0100 Subject: [PATCH 04/10] test(prompt): Count rows, not write()s, in the row-writes gate prompt_row_writes_test.py failed on ubuntu-26.04 with "7 of 13 rows were whole inside a single write()". No row was split. The trace has `GREEN RESET \r GREEN RESET` in one write(), every prompt: readline 8.3 before readline83-004 reprints a prompt holding multibyte characters from column 0, deciding on a screen position where it needs a buffer offset. Debian's 8.3-4 carries patches 001 to 003. Built here with 006 the second copy is gone and the old test passes; built at 003 it fails the way CI does. The invariant is that every row is whole in one write(). The test counted the writes holding a row and compared them with the rows, which assumes one row per write. It counts the rows now, so one really cut in two is still a failure, and it is not a change in what is pinned. The redraw itself is readline's, costs 28 bytes a prompt, lands on the same cells, and ends with 004. Refs #194 Co-Authored-By: Claude Opus 5.5 --- backlog.md | 7 +++++++ tests/prompt_row_writes_test.py | 5 ++++- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/backlog.md b/backlog.md index 268bb4c7..f97e4e00 100644 --- a/backlog.md +++ b/backlog.md @@ -9,6 +9,13 @@ pass counts in `bench/baseline/`). Performance claims come from ## Done (this program) +- [x] **`prompt_row_writes_test.py` counts rows, not write()s (#194).** Before + readline83-004 (Debian's 8.3-4 stops at 003) a multibyte prompt is + reprinted from column 0 at once, so on `ubuntu-26.04` one write() held + the row twice and the test reported 7 of 13 rows as split. None was: the + whole row was inside one write() every time, drawn twice. The redraw is + readline's and 004 ends it; the test now counts rows, so a row that is + really split still fails it. - [x] **^C inside ^R, Alt-P or M-x no longer kills the shell on readline 8.3 (#194).** 8.3's `rl_free_line_state` frees those readers' contexts on SIGINT, on the bet that the application's handler longjmps. bash does, diff --git a/tests/prompt_row_writes_test.py b/tests/prompt_row_writes_test.py index 2686cb30..9924f713 100644 --- a/tests/prompt_row_writes_test.py +++ b/tests/prompt_row_writes_test.py @@ -207,7 +207,10 @@ def main(): rows = stream.count(GREEN.encode()) check("the trace holds the prompt's last row", rows >= 6, "found it %d times in %d writes" % (rows, len(writes))) - whole = sum(1 for w in writes if GREEN.encode() + MARK in w) + # Rows, not writes: readline 8.3 before readline83-004 (Debian's 8.3-4) + # reprints a multibyte prompt from column 0 at once, so one write() holds + # the row twice, and counting writes reported that as 7 of 13 "split". + whole = sum(w.count(GREEN.encode() + MARK) for w in writes) check("readline's row reaches the tty in one write() each time", whole >= rows > 0, "%d of %d rows were whole inside a single write()" % (whole, rows)) From 28203ebc60393d532257c3b49393edde75dc6f1b Mon Sep 17 00:00:00 2001 From: LESdylan Date: Wed, 4 Nov 2026 13:39:43 +0100 Subject: [PATCH 05/10] fix(expander): walk nested spans inside a double-quoted word sh_skip_quoted read a double-quoted span flat, to the next `"`. Inside a $(...), that span can hold a nested $(...), ${...} or `...` with quotes of its own, so find_cmd_sub_end stopped on the inner quote: x=$(printf %s "$(echo b | tr -d '"')" c) # bash: bc # hellish: $(printf %s "b" c) The `'` after that wrong stop opened a span with no end, depth never reached zero, and the outer substitution stayed in the word as literal text. No error, exit 0, the command never ran, and `hellish -n` passes it. Only an odd count of quote bytes inside shows it: an even one balances by accident, which is why $(a "$(b "$c")") always worked. Found by a born2root provisioner: a readiness check written this way failed on every first boot, which runs it under hellish, and passed on every retry, which runs it under bash. sh_skip_quoted now delegates to span_skip, the scanner that already walks these spans for the word expander, so no new walker is added. tests/cmdsub_nested_quote has 18 cases compared against bash; 13 of them fail on 3.2.0. Left alone, as a separate concern: a `case` with its `)` inside a nested substitution, x=$(echo "$(case a in a) echo "q'r";; esac)"), is refused earlier by the lexer's advance_cmdsub, which has no case handling. Co-Authored-By: Claude Opus 5.5 --- backlog.md | 14 +++++ src/helpers/var_name.c | 114 +++++++++++++++++++------------------- tests/cmdsub_nested_quote | 18 ++++++ tests/tester | 1 + 4 files changed, 90 insertions(+), 57 deletions(-) create mode 100644 tests/cmdsub_nested_quote diff --git a/backlog.md b/backlog.md index f97e4e00..eb271109 100644 --- a/backlog.md +++ b/backlog.md @@ -37,6 +37,20 @@ pass counts in `bench/baseline/`). Performance claims come from 8.2.13 and 8.3 at patch levels 0, 3 and 6, and under a readline built with ASan, where the old code reported a heap-use-after-free WRITE in `_rl_search_getchar` for all three readers. +- [x] **A nested substitution with an odd count of quotes no longer leaves the + outer `$(...)` as text (#190).** `sh_skip_quoted` read a double-quoted span + flat, to the next `"`, so in `$(a "$(tr -d '"')" b)` it stopped on the inner + quote, the `'` after it opened a span with no end, and + `find_cmd_sub_end` reported the substitution as incomplete. The word kept + `$(a "..." b)` as literal text: no error, exit 0, the command never ran, and + `hellish -n` passed it. An even count balanced by accident, so + `$(a "$(b "$c")")` always worked and hid it. Found when a born2root + provisioner failed every first boot (hellish) and passed every retry (bash). + `sh_skip_quoted` now delegates to `span_skip`, which also fixes the `...` and + `${...}` neighbours, and `tests/cmdsub_nested_quote` compares 18 cases + against bash; 13 fail on 3.2.0. + A green `hellish -n` says nothing about this class: it is an expansion-time + scan, and it fails silently. - [x] **`"$VAR"` reaches the expander fast path: −19.5% loop instructions** (callgrind, loop_arith: 1,266.9M → 1,019.2M Ir). `try_simple_envvar` (`src/expander/expand_word2.c`) gated on `t->tt != TT_ENVVAR`, so the diff --git a/src/helpers/var_name.c b/src/helpers/var_name.c index c0ff955f..8156acaf 100644 --- a/src/helpers/var_name.c +++ b/src/helpers/var_name.c @@ -1,57 +1,57 @@ -/* ************************************************************************** */ -/* */ -/* ::: :::::::: */ -/* var_name.c :+: :+: :+: */ -/* +:+ +:+ +:+ */ -/* By: dlesieur +#+ +:+ +#+ */ -/* +#+#+#+#+#+ +#+ */ -/* Created: 2026/01/20 14:25:44 by dlesieur #+# #+# */ -/* Updated: 2026/01/20 14:25:44 by dlesieur ### ########.fr */ -/* */ -/* ************************************************************************** */ - -#include -#include - -/* POSIX says a variable name starts with [a-zA-Z_]. The unsigned char cast - is required by the C standard: isalpha on a plain char would be UB for - chars with the high bit set (e.g. UTF-8 bytes). */ -bool is_var_name_p1(char c) -{ - return (isalpha((unsigned char)c) || c == '_'); -} - -/* Subsequent characters extend to [a-zA-Z0-9_]. Same unsigned cast for the - same reason. The two-function split (p1 / p2) lets callers cheaply detect - whether they're looking at a valid identifier start vs continuation without - duplicating the is-digit check. */ -bool is_var_name_p2(char c) -{ - return (isalnum((unsigned char)c) || c == '_'); -} - -/* Advance past a quoted region or a backslash escape starting at s[i], - so paren-matching scans (the $(...) boundary hunt in both the reparser - and the expander) never count parens that live inside quotes — a sed - script like 's,[-(],,' must not derail the depth. Inside double quotes - a backslash still escapes the next character. Returns the index after - the skipped span, or i unchanged when s[i] starts no quoted span. */ -int sh_skip_quoted(const char *s, int len, int i) -{ - char q; - - if (s[i] == '\\' && i + 1 < len) - return (i + 2); - if (s[i] != '\'' && s[i] != '"') - return (i); - q = s[i++]; - while (i < len && s[i] != q) - { - if (q == '"' && s[i] == '\\' && i + 1 < len) - i++; - i++; - } - if (i < len) - i++; - return (i); -} +/* ************************************************************************** */ +/* */ +/* ::: :::::::: */ +/* var_name.c :+: :+: :+: */ +/* +:+ +:+ +:+ */ +/* By: dlesieur +#+ +:+ +#+ */ +/* +#+#+#+#+#+ +#+ */ +/* Created: 2026/01/20 14:25:44 by dlesieur #+# #+# */ +/* Updated: 2026/01/20 14:25:44 by dlesieur ### ########.fr */ +/* */ +/* ************************************************************************** */ + +#include +#include +#include "helpers.h" + +/* POSIX says a variable name starts with [a-zA-Z_]. The unsigned char cast + is required by the C standard: isalpha on a plain char would be UB for + chars with the high bit set (e.g. UTF-8 bytes). */ +bool is_var_name_p1(char c) +{ + return (isalpha((unsigned char)c) || c == '_'); +} + +/* Subsequent characters extend to [a-zA-Z0-9_]. Same unsigned cast for the + same reason. The two-function split (p1 / p2) lets callers cheaply detect + whether they're looking at a valid identifier start vs continuation without + duplicating the is-digit check. */ +bool is_var_name_p2(char c) +{ + return (isalnum((unsigned char)c) || c == '_'); +} + +/* Advance past a quoted region or a backslash escape starting at s[i], + so paren-matching scans (the $(...) boundary hunt in both the reparser + and the expander) never count parens that live inside quotes -- a sed + script like 's,[-(],,' must not derail the depth. span_skip owns the + walk, because a double-quoted span is not flat: its $(...), ${...} and + `...` hold quotes of their own. Read flat, "$(tr -d '"')" ended at the + inner `"`, the `'` after it opened a span with no end, and the whole + outer substitution stayed in the word as literal text, with no error. + An even count of quote bytes inside balanced by accident and hid it. + Returns the index after the skipped span, or i unchanged when s[i] + starts no quoted span. An unterminated span ends the scan at len. */ +int sh_skip_quoted(const char *s, int len, int i) +{ + int end; + + if (s[i] == '\\' && i + 1 < len) + return (i + 2); + if (s[i] != '\'' && s[i] != '"') + return (i); + end = span_skip(s, i, len); + if (end < 0) + return (len); + return (end); +} diff --git a/tests/cmdsub_nested_quote b/tests/cmdsub_nested_quote new file mode 100644 index 00000000..d8bc763d --- /dev/null +++ b/tests/cmdsub_nested_quote @@ -0,0 +1,18 @@ +x=$(echo "$(echo "a'b")"); echo "[$x]" +x=$(echo "$(echo '"')"); echo "[$x]" +x=$(printf %s "$(echo b | tr -d '"')" c); echo "[$x]" +x=$(printf %s "k: $(printf %s 'A"B' | tr -d '"')" tail); echo "[$x]" +x=$(echo "`echo "a'b"`"); echo "[$x]" +x=$(echo "`echo '"'`"); echo "[$x]" +x=$(echo "${u:-"a'b"}"); echo "[$x]" +x=$(echo "${u:-$(echo "a'b")}"); echo "[$x]" +x=$(echo "$(echo "$(echo "a'b")")"); echo "[$x]" +x=$(echo "$(( 1 + 1 ))$(echo "it's")"); echo "[$x]" +x=$(echo "a\"$(echo "b'c")"); echo "[$x]" +x=$(echo "$(echo "it's)")"); echo "[$x]" +x=$(echo "$(echo ')')"); echo "[$x]" +x=$(echo "$(echo 'hi')"); echo "[$x]" +x=$(echo "$(echo "hi")"); echo "[$x]" +y=v; x=$(printf %s "$(printf %s "$y'")"); echo "[$x]" +[ "$(printf %s "a$(printf %s 'b"' | tr -d '"')")" = ab ]; echo "rc=$?" +f() { [ "$(printf %s "$(printf 200 | tr -d '"')")" = 200 ]; }; f; echo "rc=$?" diff --git a/tests/tester b/tests/tester index ffea6c3a..da42b959 100755 --- a/tests/tester +++ b/tests/tester @@ -68,6 +68,7 @@ if [[ ${#test_lists[@]} -eq 0 ]]; then "array_subscript" "declare_quoting" "procsub_assign" "longtail" "issue95_case_cmdsub" + "cmdsub_nested_quote" "issue96_trap_func" "issue105_expansions" "issue121_errexit_andor" From acaa05f1c8ff5c8524306c403f5348a69ad34e9f Mon Sep 17 00:00:00 2001 From: LESdylan Date: Wed, 4 Nov 2026 13:53:55 +0100 Subject: [PATCH 06/10] fix(lexer): a case pattern's `)` inside a quoted substitution advance_cmdsub counted parens, so the `)` of a case pattern closed a $(...) that sits inside a double-quoted word. The scan went on with the word's quote out of step, and a valid command was refused: x="$(case a in a) echo "q'r";; esac)" # bash: q'r # hellish: syntax error span_skip already walks these spans through casescan, the automaton #95 added for the same `)`, so advance_cmdsub delegates to it and a second walker goes. Stacked on fix/nested-cmdsub-quote: a case nested in a case needs that half too. Five cases join tests/issue95_case_cmdsub. Closes #191 Co-Authored-By: Claude Opus 5.5 --- backlog.md | 6 + src/lexer/lexer_advance.c | 286 +++++++++++++++++++------------------- tests/issue95_case_cmdsub | 5 + 3 files changed, 151 insertions(+), 146 deletions(-) diff --git a/backlog.md b/backlog.md index eb271109..b419a455 100644 --- a/backlog.md +++ b/backlog.md @@ -37,6 +37,12 @@ pass counts in `bench/baseline/`). Performance claims come from 8.2.13 and 8.3 at patch levels 0, 3 and 6, and under a readline built with ASan, where the old code reported a heap-use-after-free WRITE in `_rl_search_getchar` for all three readers. +- [x] **A case pattern's `)` inside a quoted substitution is no longer a syntax + error (#191).** `advance_cmdsub` counted parens, so in + `x="$(case a in a) echo "q'r";; esac)"` the pattern's `)` closed the + substitution and the rest of the word was read out of step. It delegates to + `span_skip`, which walks these through `casescan` (#95), so the lexer and the + expander no longer hold two walkers that can disagree. - [x] **A nested substitution with an odd count of quotes no longer leaves the outer `$(...)` as text (#190).** `sh_skip_quoted` read a double-quoted span flat, to the next `"`, so in `$(a "$(tr -d '"')" b)` it stopped on the inner diff --git a/src/lexer/lexer_advance.c b/src/lexer/lexer_advance.c index 8d3ac087..441daa2f 100644 --- a/src/lexer/lexer_advance.c +++ b/src/lexer/lexer_advance.c @@ -1,146 +1,140 @@ -/* ************************************************************************** */ -/* */ -/* ::: :::::::: */ -/* lexer.c :+: :+: :+: */ -/* +:+ +:+ +:+ */ -/* By: marvin +#+ +:+ +#+ */ -/* +#+#+#+#+#+ +#+ */ -/* Created: 2026/01/09 23:33:55 by marvin #+# #+# */ -/* Updated: 2026/01/09 23:33:55 by marvin ### ########.fr */ -/* */ -/* ************************************************************************** */ - -#include "lexer.h" -#include - -static void advance_cmdsub(char **str); - -/* Skip a $(...) command substitution, honouring nested parens and quoted - segments (whose quotes are independent of any enclosing quote). Used so a - '"' inside $(...) does not prematurely close an outer double quote. */ -static void advance_cmdsub(char **str) -{ - int depth; - - *str += 2; - depth = 1; - while (**str && depth > 0) - { - if (**str == '\\') - advance_bs(str); - else if (**str == '\'') - advance_squoted(str); - else if (**str == '\"') - advance_dquoted(str); - else - { - depth += (**str == '('); - depth -= (**str == ')'); - (*str)++; - } - } -} - -/* Advance past a double-quoted span. Double quotes still expand $(...), - ${...} and `...`, so we must recurse into each of them — otherwise a `"` - that belongs to the nested construct is read as the outer quote's end - (autoconf: x="`... "" ...`"). Missing the ${...} case split - "${u:-"a b"}" at the space, because the quote after :- looked like the - closing one; the word came apart into two tokens and the reparser then - tripped its own closing-quote assertion. advance_brace_param recurses - back into here for a quote inside the braces, so the two stay in step. - The prev_bs flag handles `\"` escapes while avoiding a double-count - on `\\`. */ -int advance_dquoted(char **str) -{ - bool prev_bs; - - ft_assert(**str == '\"'); - (*str)++; - prev_bs = false; - while (**str && (**str != '\"' || prev_bs)) - { - if (!prev_bs && **str == '$' && (*str)[1] == '(') - advance_cmdsub(str); - else if (!prev_bs && **str == '$' && (*str)[1] == '{') - advance_brace_param(str, 1); - else if (!prev_bs && **str == '`') - advance_backtick(str); - else - { - prev_bs = **str == '\\' && !prev_bs; - (*str)++; - } - } - if (**str != '\"') - return (1); - (*str)++; - return (0); -} - -/* One unit of a ${...} body: an escaped byte, a quoted span, a nested - $( ) or `...`, a nested ${ (depth + 1), or one plain byte (a `}` is - depth - 1). The nested command substitutions are spans of their own: - bash prints } for ${x:-$(echo })}, and an unterminated one is an - unterminated ${...} -- ${x%$(} used to be accepted as literal text. */ -static void brace_step(char **str, int sq, int *depth) -{ - if (**str == '\\' && (*str)[1]) - (*str) += 2; - else if (**str == '\'' && sq) - advance_squoted(str); - else if (**str == '"') - advance_dquoted(str); - else if (**str == '`') - advance_backtick(str); - else if (**str == '$' && (*str)[1] == '(') - advance_cmdsub(str); - else if (**str == '$' && (*str)[1] == '{') - { - (*depth)++; - (*str) += 2; - } - else - { - *depth -= (**str == '}'); - (*str)++; - } -} - -/* Scan a ${...} parameter expansion as one span so spaces inside (e.g. - ${x:-a b c}) do not break the surrounding word. Honours nested braces and - quoted segments. in_dq says the expansion sits inside a double-quoted - word, where a ' is an ordinary character for the word operators -- bash - prints a'b for "${u:-a'b}" but reports an unterminated quote for - ${u:-a'b} -- and still a quote for the pattern ones: brace_sq_live. - A backslash escapes the next byte, so ${u-\"} does - not open a quoted section and ${u-\}} does not close the brace - early — the same rule the reparser's scan_brace_depth applies. It - has to be here too now that advance_dquoted recurses in: without it - the \" of "${u-\"}" read as the start of a nested quote and - swallowed the rest of the line. - - ONLY `${` NESTS. A bare `{` inside the expansion is an ordinary - character to bash, and counting it as an opener meant the closing `}` - paid off a brace that was never open, so the scan ran to end of input: - - echo ${x:-a{b} unexpected EOF, matching `}' - echo ${o%%[<{().[]*} the same, on a bracket expression - - The second is not a curiosity -- it is bash-completion's - _comp_compgen_help__parse, which is why 3800 lines of the most widely - installed completion framework in existence failed to parse at line 1757 - and defined nothing. `${x:-${y}}` still nests, because that is a real - opener. */ -int advance_brace_param(char **str, int in_dq) -{ - int depth; - int sq; - - *str += 2; - sq = brace_sq_live(*str, INT_MAX, in_dq); - depth = 1; - while (**str && depth > 0) - brace_step(str, sq, &depth); - return (depth != 0); -} +/* ************************************************************************** */ +/* */ +/* ::: :::::::: */ +/* lexer.c :+: :+: :+: */ +/* +:+ +:+ +:+ */ +/* By: marvin +#+ +:+ +#+ */ +/* +#+#+#+#+#+ +#+ */ +/* Created: 2026/01/09 23:33:55 by marvin #+# #+# */ +/* Updated: 2026/01/09 23:33:55 by marvin ### ########.fr */ +/* */ +/* ************************************************************************** */ + +#include "helpers.h" +#include "lexer.h" +#include + +static void advance_cmdsub(char **str); + +/* Skip a $(...) command substitution, whose quotes are independent of any + enclosing quote, so a '"' inside does not close an outer double quote. + span_skip owns the walk: it knows a case pattern's unbalanced `)` does not + end the span, which a plain paren count here did not, and + x="$(case a in a) echo "q'r";; esac)" was refused as a syntax error while + the same text with no quotes around it worked. An unterminated span stops + at the end of the input, for the prompt to ask for more. */ +static void advance_cmdsub(char **str) +{ + int len; + int end; + + len = (int)ft_strlen(*str); + end = span_skip(*str, 0, len); + if (end < 0) + end = len; + *str += end; +} + +/* Advance past a double-quoted span. Double quotes still expand $(...), + ${...} and `...`, so we must recurse into each of them — otherwise a `"` + that belongs to the nested construct is read as the outer quote's end + (autoconf: x="`... "" ...`"). Missing the ${...} case split + "${u:-"a b"}" at the space, because the quote after :- looked like the + closing one; the word came apart into two tokens and the reparser then + tripped its own closing-quote assertion. advance_brace_param recurses + back into here for a quote inside the braces, so the two stay in step. + The prev_bs flag handles `\"` escapes while avoiding a double-count + on `\\`. */ +int advance_dquoted(char **str) +{ + bool prev_bs; + + ft_assert(**str == '\"'); + (*str)++; + prev_bs = false; + while (**str && (**str != '\"' || prev_bs)) + { + if (!prev_bs && **str == '$' && (*str)[1] == '(') + advance_cmdsub(str); + else if (!prev_bs && **str == '$' && (*str)[1] == '{') + advance_brace_param(str, 1); + else if (!prev_bs && **str == '`') + advance_backtick(str); + else + { + prev_bs = **str == '\\' && !prev_bs; + (*str)++; + } + } + if (**str != '\"') + return (1); + (*str)++; + return (0); +} + +/* One unit of a ${...} body: an escaped byte, a quoted span, a nested + $( ) or `...`, a nested ${ (depth + 1), or one plain byte (a `}` is + depth - 1). The nested command substitutions are spans of their own: + bash prints } for ${x:-$(echo })}, and an unterminated one is an + unterminated ${...} -- ${x%$(} used to be accepted as literal text. */ +static void brace_step(char **str, int sq, int *depth) +{ + if (**str == '\\' && (*str)[1]) + (*str) += 2; + else if (**str == '\'' && sq) + advance_squoted(str); + else if (**str == '"') + advance_dquoted(str); + else if (**str == '`') + advance_backtick(str); + else if (**str == '$' && (*str)[1] == '(') + advance_cmdsub(str); + else if (**str == '$' && (*str)[1] == '{') + { + (*depth)++; + (*str) += 2; + } + else + { + *depth -= (**str == '}'); + (*str)++; + } +} + +/* Scan a ${...} parameter expansion as one span so spaces inside (e.g. + ${x:-a b c}) do not break the surrounding word. Honours nested braces and + quoted segments. in_dq says the expansion sits inside a double-quoted + word, where a ' is an ordinary character for the word operators -- bash + prints a'b for "${u:-a'b}" but reports an unterminated quote for + ${u:-a'b} -- and still a quote for the pattern ones: brace_sq_live. + A backslash escapes the next byte, so ${u-\"} does + not open a quoted section and ${u-\}} does not close the brace + early — the same rule the reparser's scan_brace_depth applies. It + has to be here too now that advance_dquoted recurses in: without it + the \" of "${u-\"}" read as the start of a nested quote and + swallowed the rest of the line. + + ONLY `${` NESTS. A bare `{` inside the expansion is an ordinary + character to bash, and counting it as an opener meant the closing `}` + paid off a brace that was never open, so the scan ran to end of input: + + echo ${x:-a{b} unexpected EOF, matching `}' + echo ${o%%[<{().[]*} the same, on a bracket expression + + The second is not a curiosity -- it is bash-completion's + _comp_compgen_help__parse, which is why 3800 lines of the most widely + installed completion framework in existence failed to parse at line 1757 + and defined nothing. `${x:-${y}}` still nests, because that is a real + opener. */ +int advance_brace_param(char **str, int in_dq) +{ + int depth; + int sq; + + *str += 2; + sq = brace_sq_live(*str, INT_MAX, in_dq); + depth = 1; + while (**str && depth > 0) + brace_step(str, sq, &depth); + return (depth != 0); +} diff --git a/tests/issue95_case_cmdsub b/tests/issue95_case_cmdsub index 4f3f6f74..cda3a2b2 100644 --- a/tests/issue95_case_cmdsub +++ b/tests/issue95_case_cmdsub @@ -11,3 +11,8 @@ p=$(case x in y) echo no ;; esac; echo done); echo "p=$p" q=$(v=case; echo "$v"); echo "q=$q" r=$(echo case; echo esac); echo "r=$r" s=$(case m in m) echo one; echo two ;; esac); echo "s=$s" +x="$(case a in a) echo "q'r";; esac)"; echo "[$x]" +x="$(case a in a) echo 'q"r';; esac)"; echo "[$x]" +echo "$(case a in (a) echo "b'c";; esac)" +x="$(case a in a) echo "$(case b in b) echo "n'd";; esac)";; esac)"; echo "[$x]" +x="$(case 'a)' in 'a)') echo y;; esac)"; echo "[$x]" From 0eea026c116e12d7f1daba4d9f0b26c99305db26 Mon Sep 17 00:00:00 2001 From: LESdylan Date: Wed, 4 Nov 2026 22:36:48 +0100 Subject: [PATCH 07/10] feat(banner): Show what is new in 3.3 in the welcome panel banner2.c asks each release to replace the news table and wiki/whats-new.md together and to bump HELLISH_HEADER_REV, so the banner is shown again after the update rather than the next day. The table now holds what 3.3 adds: make with most-run targets first, and the hint keys, which take the most-run match. The switch HELLISH_SUGGEST_STRATEGY=recent is 33 columns wide against the 32 the right column has at 80, so it stays on the page. The 3.2 guide stays below the 3.3 one, demoted one level: interactive.md links its #inline-hints anchor, and nothing else documents those keys. The page records one limit found while running its examples: HISTCONTROL=ignoredups or ignoreboth stores a back-to-back repeat once, so three `make re` in a row rank as one run and the newer `make clean` wins the tie. Co-Authored-By: Claude Opus 5.5 --- incs/version.h | 2 +- mkdocs.yml | 2 +- src/infrastructure/banner.c | 2 +- src/infrastructure/banner2.c | 22 +++----- wiki/interactive.md | 11 ++-- wiki/whats-new.md | 104 ++++++++++++++++++++++++++++++----- 6 files changed, 108 insertions(+), 35 deletions(-) diff --git a/incs/version.h b/incs/version.h index 9a6bf8a5..e83c6a87 100644 --- a/incs/version.h +++ b/incs/version.h @@ -30,6 +30,6 @@ /* Bumped whenever the welcome header itself changes what it says. The header is shown once a day, but a new revision is new information, so it is shown again immediately rather than waiting for tomorrow. */ -# define HELLISH_HEADER_REV 4 +# define HELLISH_HEADER_REV 5 #endif diff --git a/mkdocs.yml b/mkdocs.yml index b264d002..7c2d7f7c 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -55,7 +55,7 @@ markdown_extensions: nav: - Home: index.md - - What's new in 3.2: whats-new.md + - What's new in 3.3: whats-new.md - What it is + Install: product.md - The manual: manual.md - Builtins reference: diff --git a/src/infrastructure/banner.c b/src/infrastructure/banner.c index dc8148a6..9b54426a 100644 --- a/src/infrastructure/banner.c +++ b/src/infrastructure/banner.c @@ -98,7 +98,7 @@ static void build_left(const char **l) " \xc2\xb7 a POSIX shell with bite"; l[i + 4] = cwd; l[i + 5] = ""; - l[i + 6] = "\033[38;5;245mthe 3.2 guide, with examples\033[0m"; + l[i + 6] = "\033[38;5;245mthe 3.3 guide, with examples\033[0m"; l[i + 7] = guide_link(); l[i + 8] = NULL; } diff --git a/src/infrastructure/banner2.c b/src/infrastructure/banner2.c index ece2aad6..5c1e0069 100644 --- a/src/infrastructure/banner2.c +++ b/src/infrastructure/banner2.c @@ -13,7 +13,7 @@ #include "banner_private.h" #include "shell.h" -/* One row of "What's new in 3.2": the keys that reach a new feature, in +/* One row of "What's new in 3.3": the keys that reach a new feature, in code-yellow, then what it does, in muted grey -- the shape of "Getting started", so each can be tried before the prompt even appears. Every row fits the 32 columns the right column has on an 80-column terminal. @@ -26,16 +26,12 @@ Returns 0 at the end of a group. */ static int news_row(char *dst, int n) { - static const char *utf8[] = {"\xe2\x86\x91 \xe2\x86\x93 ", - "make ", "bind -x ", "bindkey ", "${var@P} ", "$SRANDOM ", - NULL, "\xe2\x86\x92 ^F End ", "Alt-F ", "Alt-N/P ", NULL}; - static const char *ascii[] = {"Up Down ", "make ", "bind -x ", - "bindkey ", "${var@P} ", "$SRANDOM ", NULL, "Right ^F ", - "Alt-F ", "Alt-N/P ", NULL}; - static const char *what[] = {"history by prefix", "Makefile targets", - "key runs a command", "zsh binds now work", "render as a prompt", - "and $EPOCHREALTIME", NULL, "take the dim hint", "take one word", - "older/newer hint"}; + static const char *utf8[] = {"make ", NULL, + "\xe2\x86\x92 ^F End ", "Alt-N/P ", NULL}; + static const char *ascii[] = {"make ", NULL, "Right ^F ", + "Alt-N/P ", NULL}; + static const char *what[] = {"most-run first", NULL, + "take your most-run", "the next most-run"}; const char **keys; keys = ascii; @@ -57,7 +53,7 @@ static void news_block(const char **r, int *i, char nw[][192]) int m; r[(*i)++] = "\033[1;48;5;209;38;5;236m NEW \033[0m " - "\033[1;38;5;81mWhat's new in 3.2\033[0m"; + "\033[1;38;5;81mWhat's new in 3.3\033[0m"; r[(*i)++] = ""; k = 0; m = 0; @@ -74,7 +70,7 @@ static void news_block(const char **r, int *i, char nw[][192]) (shown once). */ void build_right(const char **r) { - static char nw[9][192]; + static char nw[3][192]; static char status[192]; int i; diff --git a/wiki/interactive.md b/wiki/interactive.md index 0a73a670..1a609580 100644 --- a/wiki/interactive.md +++ b/wiki/interactive.md @@ -171,11 +171,12 @@ $ git commit -m "fix: close [[ ]] on metachar-adjacent ]]" ### Autosuggestions ✅ *(3.2)* -Shipped in 3.2, opt-in with `export HELLISH_AUTOSUGGEST=1`. As you type, the newest history line -that starts with the typed text is shown dimmed after the cursor. **→ / End / Ctrl-F** take it, -**Alt-F** one word of it, and **Alt-N / Alt-P** show other matches. In the same release, Up and -Down search the history by the text before the cursor. See -[What's new in 3.2](whats-new.md#inline-hints). +Shipped in 3.2, opt-in with `export HELLISH_AUTOSUGGEST=1`. As you type, the history line you run +most that starts with the typed text is shown dimmed after the cursor; before 3.3 it was the newest +one, which `HELLISH_SUGGEST_STRATEGY=recent` brings back. **→ / End / Ctrl-F** take it, **Alt-F** +one word of it, and **Alt-N / Alt-P** show other matches. In 3.2, Up and Down also began searching +the history by the text before the cursor. See [the hint keys](whats-new.md#inline-hints) and +[how 3.3 ranks them](whats-new.md#hints-rank-what-you-run-most). ### Completion polish 🚧 *(Week 1, time-boxed)* diff --git a/wiki/whats-new.md b/wiki/whats-new.md index 6add2881..b9205bc2 100644 --- a/wiki/whats-new.md +++ b/wiki/whats-new.md @@ -1,10 +1,9 @@ -# What's new in 3.2 +# What's new in 3.3 -hellish 3.2 is about the prompt. The up arrow finds what you started typing. -A dim hint from your history can complete the line, and you take it with one -key. `make ` knows your targets, and `bind` puts a shell command on a -key. This page shows how to use each one. Every example on it was run on a -3.2 build. +hellish 3.3 ranks by what you run, not by what you ran last. The dim hint and +`make ` put your most-run command first. This page shows how that works +and which settings change it. Every example on it was run on a 3.3 build. +[What came in 3.2](#what-came-in-32) follows further down. !!! tip "Get it" Already running hellish? Type `update --now`. Otherwise install it: @@ -17,6 +16,83 @@ key. This page shows how to use each one. Every example on it was run on a This is the table from the welcome banner, explained. +| Keys | What it does | Needs | +|---|---|---| +| `make ` | [most-run targets first](#make-targets-you-run-first) | | +| `→ ^F End` | [take the hint, now your most-run match](#hints-rank-what-you-run-most) | `export HELLISH_AUTOSUGGEST=1` | +| `Alt-N/P` | [step through the matches in the same order](#hints-rank-what-you-run-most) | `export HELLISH_AUTOSUGGEST=1` | + +## Hints rank what you run most + +With hints on, typing `make ` used to show the newest line that starts with +it. It now shows the line you run most, weighted toward recent runs: + +```console +$ export HELLISH_AUTOSUGGEST=1 +$ make re +$ make re +$ make re +$ make clean +$ make re +``` + +On the last line you typed `make `; `re` is the dim hint. Alt-N +moves to the next match, `clean`. A run's weight halves for every 64 newer +history lines, so a habit you dropped fades out. + +Two settings change the order: + +- `HELLISH_SUGGEST_STRATEGY=recent` brings back newest first: the history + above hints `clean`. It is read at every prompt. +- `HISTCONTROL=ignoredups` or `ignoreboth` stores a back-to-back repeat once, + so the three `make re` above count as one run and the hint is `clean`. + Repeats with another command between them still count. + +## make targets you run first + +`make ` still reads the Makefile itself and never runs make. Targets you +have run now come first, most-run first, and the rest follow alphabetically: + +```console +$ make +re clean all fclean test +``` + +`HELLISH_SUGGEST_STRATEGY=recent` orders them newest first here too. + +## Fixed + +In scripts: + +- A `$(...)` no longer stays as literal text when a double-quoted word inside + it holds a nested substitution with an odd number of quote bytes. + `x=$(echo "$(echo "a'b")"); echo "[$x]"` prints `[a'b]`, as in bash + ([#190](https://github.com/Univers42/hellish/issues/190)). +- A `case` inside a double-quoted `$(...)` parses. + `x="$(case a in a) echo "q'r";; esac)"; echo "[$x]"` prints `[q'r]` + ([#191](https://github.com/Univers42/hellish/issues/191)). + +At the prompt: + +- On a build linked against GNU readline 8.3, Ctrl-C inside + Ctrl-R, Alt-P or Alt-X freed the search + state twice. It now cancels the search and returns to a clean prompt + ([#194](https://github.com/Univers42/hellish/issues/194)). + +Every change since 3.2.0: +[v3.2.0...v3.3.0](https://github.com/Univers42/hellish/compare/v3.2.0...v3.3.0). + +## What came in 3.2 + +hellish 3.2 is about the prompt. The up arrow finds what you started typing. +A dim hint from your history can complete the line, and you take it with one +key. `make ` knows your targets, and `bind` puts a shell command on a +key. Every example in this section was run on a 3.2 build. + +### At a glance + +This is the table from the welcome banner, explained. + | Keys | What it does | Needs | |---|---|---| | `↑ ↓` | [history by prefix](#history-that-matches-what-you-typed) | on by default | @@ -29,7 +105,7 @@ This is the table from the welcome banner, explained. | `Alt-F` | [take one word](#inline-hints) | `HELLISH_AUTOSUGGEST=1` | | `Alt-N/P` | [older/newer hint](#inline-hints) | `HELLISH_AUTOSUGGEST=1` | -## History that matches what you typed +### History that matches what you typed Type the start of a command, then press ↑. Up now steps only through history lines that begin with the text before the cursor, and @@ -52,7 +128,7 @@ On an empty line, Up and Down walk the whole history as before. dead. - A key you bound yourself keeps your binding. -## Inline hints +### Inline hints Hints are off until you ask for them. Add this line to `~/.hellishrc`, or type it at the prompt to try it: @@ -92,7 +168,7 @@ $ ech|o hello you typed "ech", the cursor is at |, "o hello" is dim searched (default 1000). - In zsh mode, widgets see the hint as `POSTDISPLAY`, and can set their own. -## Completing make targets +### Completing make targets Tab after `make` offers the targets of the Makefile in the current directory. With this Makefile: @@ -133,7 +209,7 @@ would type, so they are not offered. The full rules are in the manual's [make targets](manual.md#make-targets) section. -## Key bindings with bind +### Key bindings with bind hellish now has bash's `bind` builtin, so the lines you would put in `~/.inputrc` work in `~/.hellishrc`: @@ -162,7 +238,7 @@ To see what is bound: `bind -p` lists key bindings, `bind -v` the settings, `bind -X` the `-x` commands and `bind -l` every command name. `help bind` shows every option. -## bindkey in zsh configs +### bindkey in zsh configs In zsh mode (`set -o zsh`, `emulate zsh`, or a sourced `.zsh` file), `bindkey` now understands zsh's key notation (`^[[A`, `^R`, `^[b`, `\E`) @@ -174,7 +250,7 @@ bindkey '^[[A' history-beginning-search-backward bindkey '^[[B' history-beginning-search-forward ``` -## Rendering text like the prompt +### Rendering text like the prompt `${var@P}` expands a value the way the prompt does. It uses the same renderer as `PS1`, so the result always matches your prompt: @@ -188,7 +264,7 @@ ada in share $ Both escape languages work, bash's `\u \w` and zsh's `%n %~`. `prompt preview` uses it to draw the themes. -## Two new variables +### Two new variables ```text $ echo $SRANDOM @@ -201,7 +277,7 @@ $ echo $EPOCHREALTIME `$EPOCHREALTIME` is the time in seconds since the epoch, with microseconds. Both come from bash: `$EPOCHREALTIME` from 5.0, `$SRANDOM` from 5.1. -## Fixed +### Fixed At the prompt: From aa08aadfb3065b46616b51f68c89457fafbe3216 Mon Sep 17 00:00:00 2001 From: LESdylan Date: Wed, 4 Nov 2026 22:36:48 +0100 Subject: [PATCH 08/10] chore(release): Cut 3.3.0, the hint knows what you run most Frecency ranking for hints and make (#140), the two parser fixes (#190, #191) and the readline 8.3 ^C fix (#194). RELEASE.md has the summary; wiki/whats-new.md, which the banner links to, has the long form. Uses chore(release) because the commit-msg hook has no `release` type, which the 3.1.1 to 3.2.0 subjects used. Co-Authored-By: Claude Opus 5.5 --- RELEASE.md | 26 +++++++++++++++++++++++++- incs/version.h | 2 +- npm/package.json | 2 +- 3 files changed, 27 insertions(+), 3 deletions(-) diff --git a/RELEASE.md b/RELEASE.md index 9becb480..576409ff 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -9,7 +9,9 @@ shell. The welcome banner links to the guide for the latest one: --- -## v3.3.0 — *the hint knows what you run most* (unreleased) +## v3.3.0 — *the hint knows what you run most* + +**The guide: [What's new in 3.3](https://univers42.github.io/hellish/whats-new/)**. - **Hints are ranked by frecency.** With `HELLISH_AUTOSUGGEST=1`, the dim hint is the matching history line you run most, weighted toward recent @@ -20,6 +22,28 @@ shell. The welcome banner links to the guide for the latest one: the rest alphabetically (#140). - `HELLISH_SUGGEST_STRATEGY=recent` brings back newest-first for both. It is read at every prompt. +- `HISTCONTROL=ignoredups` or `ignoreboth` stores a back-to-back repeat + once, so those repeats rank as one run. + +Fixed on the way to the release: + +- **A `$(...)` left as literal text.** A double-quoted word inside it that + held a nested substitution with an odd number of quote bytes stopped the + expansion: `x=$(echo "$(echo "a'b")")` assigned its own source text, with + status 0 and no error. All four reported forms now match bash 5.3.9 + (#190). +- **A `case` inside `"$(...)"`.** `x="$(case a in a) echo "q'r";; esac)"` + was a syntax error, because the pattern's `)` closed the substitution + (#191). +- **`^C` inside `^R`, Alt-P or Alt-X on GNU readline 8.3.** Readline 8.3 + frees a search's state on SIGINT and expects the application to longjmp + out. hellish's reader returns instead, so the search freed it a second + time: a double free in a sanitizer build, and a crash on `^R` in a + release build. A filter in front of readline's handler now holds the + signal while a search is up and delivers it at the main loop (#194). + +Every change: +[v3.2.0...v3.3.0](https://github.com/Univers42/hellish/compare/v3.2.0...v3.3.0). --- diff --git a/incs/version.h b/incs/version.h index e83c6a87..aa881816 100644 --- a/incs/version.h +++ b/incs/version.h @@ -15,7 +15,7 @@ /* The single source of truth for the running shell's version. Bumped in step with the git tag / GitHub release / npm package / docker image. */ -# define HELLISH_VERSION "3.2.0" +# define HELLISH_VERSION "3.3.0" /* Where releases live; used by the `update` builtin and the daily check. */ # define HELLISH_REPO "Univers42/hellish" diff --git a/npm/package.json b/npm/package.json index 7421118f..08bfaa1d 100644 --- a/npm/package.json +++ b/npm/package.json @@ -1,6 +1,6 @@ { "name": "hellish-shell", - "version": "3.2.0", + "version": "3.3.0", "description": "hellish — a fast, POSIX-compliant shell, with horns", "bin": { "hellish": "bin/hellish.js" From e616f2dd14fa464d6f5af83c7e21cea9f797a00f Mon Sep 17 00:00:00 2001 From: La Cebolla Date: Wed, 11 Nov 2026 02:19:46 +0100 Subject: [PATCH 09/10] ci(docker): pass --overwrite '*' to pacman on arch linux Arch Linux containers can fail package upgrades due to file conflicts (e.g. openssl-3 split /usr/lib/libssl.so.3). Overwrite prevents pacman from aborting in archlinux:base-devel. --- docker/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker/Dockerfile b/docker/Dockerfile index 8fea9287..f7af3293 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -74,7 +74,7 @@ RUN set -eux; \ done; \ rm -rf /var/lib/apt/lists/*; \ elif command -v pacman >/dev/null 2>&1; then \ - pacman -Sy --noconfirm --needed base-devel readline git bash \ + pacman -Sy --noconfirm --needed --overwrite '*' base-devel readline git bash \ python clang; \ elif command -v dnf >/dev/null 2>&1; then \ dnf -y install gcc clang make readline-devel ncurses-devel git \ From 040fd4b8d37a1e3716f3b0ef62ba6564799bd083 Mon Sep 17 00:00:00 2001 From: La Cebolla Date: Wed, 11 Nov 2026 02:41:27 +0100 Subject: [PATCH 10/10] ci(workflow): bump macos runner from macos-14 to macos-15 --- .github/workflows/platforms.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/platforms.yml b/.github/workflows/platforms.yml index c00fdbf9..e24abf60 100644 --- a/.github/workflows/platforms.yml +++ b/.github/workflows/platforms.yml @@ -198,7 +198,7 @@ jobs: # whole run, which is worse than not running it: a queued job # keeps its run open and holds concurrency. If Intel coverage is # wanted back it needs a runner that actually exists. - - { runner: macos-14, arch: "arm64" } + - { runner: macos-15, arch: "arm64" } steps: - uses: actions/checkout@v4 with: