release: 3.3.0 — frecency suggestions, Makefile completion and quoted cmdsub fixes

.github/workflows/platforms.yml

@@ -198,7 +198,7 @@ jobs:
           # whole run, which is worse than not running it: a queued job
           # keeps its run open and holds concurrency. If Intel coverage is
           # wanted back it needs a runner that actually exists.
-          - { runner: macos-14, arch: "arm64" }
+          - { runner: macos-15, arch: "arm64" }
     steps:
       - uses: actions/checkout@v4
         with:

RELEASE.md

@@ -9,6 +9,44 @@ shell. The welcome banner links to the guide for the latest one:
 
 ---
 
+## v3.3.0 — *the hint knows what you run most*
+
+**The guide: [What's new in 3.3](https://univers42.github.io/hellish/whats-new/)**.
+
+- **Hints are ranked by frecency.** With `HELLISH_AUTOSUGGEST=1`, the dim
+  hint is the matching history line you run most, weighted toward recent
+  runs, and no longer simply the newest one. Run `make re` three times and
+  `make clean` once more recently, and `make ` hints `re`. Alt-N / Alt-P
+  step through the matches in the same order (#140).
+- **`make <TAB>` lists the targets you run first**, most-run first, then
+  the rest alphabetically (#140).
+- `HELLISH_SUGGEST_STRATEGY=recent` brings back newest-first for both. It
+  is read at every prompt.
+- `HISTCONTROL=ignoredups` or `ignoreboth` stores a back-to-back repeat
+  once, so those repeats rank as one run.
+
+Fixed on the way to the release:
+
+- **A `$(...)` left as literal text.** A double-quoted word inside it that
+  held a nested substitution with an odd number of quote bytes stopped the
+  expansion: `x=$(echo "$(echo "a'b")")` assigned its own source text, with
+  status 0 and no error. All four reported forms now match bash 5.3.9
+  (#190).
+- **A `case` inside `"$(...)"`.** `x="$(case a in a) echo "q'r";; esac)"`
+  was a syntax error, because the pattern's `)` closed the substitution
+  (#191).
+- **`^C` inside `^R`, Alt-P or Alt-X on GNU readline 8.3.** Readline 8.3
+  frees a search's state on SIGINT and expects the application to longjmp
+  out. hellish's reader returns instead, so the search freed it a second
+  time: a double free in a sanitizer build, and a crash on `^R` in a
+  release build. A filter in front of readline's handler now holds the
+  signal while a search is up and delivers it at the main loop (#194).
+
+Every change:
+[v3.2.0...v3.3.0](https://github.com/Univers42/hellish/compare/v3.2.0...v3.3.0).
+
+---
+
 ## v3.2.0 — *Up finds what you typed, a hint is one key away, and `bind`*
 
 **The guide: [What's new in 3.2](https://univers42.github.io/hellish/whats-new/)**.

USER_DOC.md

@@ -169,6 +169,10 @@ escape hatches above get you back to your previous shell at any time.
 current directory (also `gmake`, `/usr/bin/make`, and behind `sudo`, `env X=1`,
 `&&`, `(`; a wrapper option that takes an argument, as in `sudo -u root make`,
 falls back to filenames). The file is parsed in the shell itself; make is never run.
+Targets you have run (history lines starting with `make` or `gmake`) come first,
+most-run (weighted toward recent runs) first;
+the rest follow alphabetically. With `HELLISH_SUGGEST_STRATEGY=recent` the
+run ones are ordered by how recently they ran.
 
 - Files read: the first regular file of `GNUmakefile`, `makefile`, `Makefile`
   (GNU make's order, symlinks followed), at most 4 MiB. No such file, a
@@ -194,8 +198,9 @@ falls back to filenames). The file is parsed in the shell itself; make is never
 |---|---|
 | `HELLISH_BANNER=0\|1` | welcome panel off / on |
 | `HELLISH_HISTORY_PREFIX=0\|off` | Up/Down plain history; default is prefix search, read at every prompt |
-| `HELLISH_AUTOSUGGEST=1\|on` | inline suggestion, off by default: the newest history line that starts with what you typed, shown dim after the cursor (see below); read at every prompt |
+| `HELLISH_AUTOSUGGEST=1\|on` | inline suggestion, off by default: the best-ranked history line (see `HELLISH_SUGGEST_STRATEGY`) that starts with what you typed, shown dim after the cursor (see below); read at every prompt |
 | `HELLISH_SUGGEST_MAX=N` | how many of the newest history lines the suggestion looks at (default 1000; not a number, or below 1, is the default); read at every prompt |
+| `HELLISH_SUGGEST_STRATEGY=recent` | rank suggestions newest first; by default (unset or any other value) they are ranked by frecency: how often a line was run, weighted toward recent runs. Also orders `make <TAB>`. Read at every prompt |
 | `HELLISH_NO_UPDATE_CHECK=1` | never check for updates, no badge |
 | `HELLISH_NO_ANIM=1` | no startup animation |
 | `HELLISH_NO_EXEC=1` | user-install hook: skip for this one session |
@@ -219,10 +224,10 @@ typed, and the hint is wiped from the screen.
 A word is the blanks at the front of the hint plus the run of characters up
 to the next space or tab (so `-m`, `a/b`, `x=1` are one word each). Alt-N
 and Alt-P step through distinct lines (same full text offered once), look
-at no more than `HELLISH_SUGGEST_MAX` entries, ring the bell at the oldest
-and the newest end instead of wrapping, and start over from the newest
+at no more than `HELLISH_SUGGEST_MAX` entries, go in ranking order, ring the bell at the last and
+the first instead of wrapping, and start over from the best
 match as soon as the line is edited. Right after Alt-N inserts the line
-shown, not the newest. A hint a zle widget set as `POSTDISPLAY` is not
+shown, not the best-ranked one. A hint a zle widget set as `POSTDISPLAY` is not
 cycled or taken by words. In vi insert mode Alt-F / Alt-N / Alt-P work the
 same (no Ctrl-Right). `unset HELLISH_AUTOSUGGEST`
 (or `=0`) turns it off at the next prompt. A key you bound yourself

backlog.md

@@ -9,6 +9,54 @@ pass counts in `bench/baseline/`). Performance claims come from
 
 ## Done (this program)
 
+- [x] **`prompt_row_writes_test.py` counts rows, not write()s (#194).** Before
+      readline83-004 (Debian's 8.3-4 stops at 003) a multibyte prompt is
+      reprinted from column 0 at once, so on `ubuntu-26.04` one write() held
+      the row twice and the test reported 7 of 13 rows as split. None was: the
+      whole row was inside one write() every time, drawn twice. The redraw is
+      readline's and 004 ends it; the test now counts rows, so a row that is
+      really split still fails it.
+- [x] **^C inside ^R, Alt-P or M-x no longer kills the shell on readline 8.3
+      (#194).** 8.3's `rl_free_line_state` frees those readers' contexts on
+      SIGINT, on the bet that the application's handler longjmps. bash does,
+      and readline83-003 taught readline's own `rl_getc` to abort to the top
+      level for handlers that return; `rl_getc_hook` replaces that `rl_getc`
+      and hands the reader EOF, so the reader unwound through a freed `cxt`:
+      a double free under ASan, a SIGSEGV on a release build (3 of 3 on ^R),
+      and every PR red when `ubuntu-latest` moved to 26.04 (readline 8.3-4)
+      on an unchanged tree. Guarding only `rl_getc_hook`'s own look was tried
+      first and is not enough: readline also looks at its deferred signal at
+      the end of every `rl_redisplay`, and a ^C sent with the keys of a ^R
+      came through there. `rl_intr.c` now puts a filter in front of
+      readline's SIGINT handler: while such a reader is up the signal is
+      parked, the reader unwinds on EOF with its context intact, and the ^C
+      is replayed in the main loop. Outside those readers the filter only
+      forwards, so ^C anywhere else, a slow completion included, is unchanged.
+      `prompt_interrupt_test.py` gains nsearch, mx and redraw,
+      `rl_read_race_test.py` gains nsearch and M-x. 48 and 34 ok on 8.1.2,
+      8.2.13 and 8.3 at patch levels 0, 3 and 6, and under a readline built
+      with ASan, where the old code reported a heap-use-after-free WRITE in
+      `_rl_search_getchar` for all three readers.
+- [x] **A case pattern's `)` inside a quoted substitution is no longer a syntax
+      error (#191).** `advance_cmdsub` counted parens, so in
+      `x="$(case a in a) echo "q'r";; esac)"` the pattern's `)` closed the
+      substitution and the rest of the word was read out of step. It delegates to
+      `span_skip`, which walks these through `casescan` (#95), so the lexer and the
+      expander no longer hold two walkers that can disagree.
+- [x] **A nested substitution with an odd count of quotes no longer leaves the
+      outer `$(...)` as text (#190).** `sh_skip_quoted` read a double-quoted span
+      flat, to the next `"`, so in `$(a "$(tr -d '"')" b)` it stopped on the inner
+      quote, the `'` after it opened a span with no end, and
+      `find_cmd_sub_end` reported the substitution as incomplete. The word kept
+      `$(a "..." b)` as literal text: no error, exit 0, the command never ran, and
+      `hellish -n` passed it. An even count balanced by accident, so
+      `$(a "$(b "$c")")` always worked and hid it. Found when a born2root
+      provisioner failed every first boot (hellish) and passed every retry (bash).
+      `sh_skip_quoted` now delegates to `span_skip`, which also fixes the `...` and
+      `${...}` neighbours, and `tests/cmdsub_nested_quote` compares 18 cases
+      against bash; 13 fail on 3.2.0.
+      A green `hellish -n` says nothing about this class: it is an expansion-time
+      scan, and it fails silently.
 - [x] **`"$VAR"` reaches the expander fast path: −19.5% loop instructions**
       (callgrind, loop_arith: 1,266.9M → 1,019.2M Ir). `try_simple_envvar`
       (`src/expander/expand_word2.c`) gated on `t->tt != TT_ENVVAR`, so the

docker/Dockerfile

@@ -74,7 +74,7 @@ RUN set -eux; \
 		done; \
 		rm -rf /var/lib/apt/lists/*; \
 	elif command -v pacman >/dev/null 2>&1; then \
-		pacman -Sy --noconfirm --needed base-devel readline git bash \
+		pacman -Sy --noconfirm --needed --overwrite '*' base-devel readline git bash \
 			python clang; \
 	elif command -v dnf >/dev/null 2>&1; then \
 		dnf -y install gcc clang make readline-devel ncurses-devel git \

incs/history.h

@@ -15,6 +15,7 @@
 
 # include "shell.h"
 # include <sys/types.h>
+# include <stdint.h>
 
 /* hist_cmds is the truth; readline's own list is a mirror kept in step so
    the arrow keys agree with what `history` prints.
@@ -67,5 +68,10 @@ t_string	encode_cmd_hist(char *cmd);
 void		manage_history(t_shell *state);
 bool		worthy_of_being_remembered(t_shell *state);
 void		add_history_line(t_shell *state, const char *cmd);
+/* Frecency (history_frecency.c). */
+uint32_t	hist_weight(size_t age);
+uint64_t	hist_score_add(uint64_t score, size_t len, size_t idx,
+				bool recent);
+size_t		hist_window_floor(size_t len, int max);
 
 #endif

incs/prompt.h

@@ -108,8 +108,10 @@ typedef struct s_rl
 	char		*sg_post; /* POSTDISPLAY a widget set (zle_rl5.c) */
 	char		*sg_post_at; /* the line it was set for */
 	char		*sg_pub; /* what a widget was handed as POSTDISPLAY */
-	t_vec		sg_cyc; /* history indices offered (size_t) */
-	size_t		sg_cyc_k; /* which of them is on screen */
+	bool		sg_recent; /* HELLISH_SUGGEST_STRATEGY=recent */
+	t_vec		sg_rank; /* ranked history indices (size_t) */
+	t_vec		sg_scr; /* ranking scratch (t_sg_ent), reused */
+	size_t		sg_cyc_k; /* which of sg_rank is on screen */
 	char		*sg_cyc_line; /* line the cycle is for; NULL: none */
 	bool		sg_painted; /* a ghost is on screen right now */
 	int			sg_col; /* column it starts at */

incs/shell.h

@@ -332,6 +332,7 @@ typedef struct s_make_cache
 	long long	mtime_s;
 	long long	mtime_ns;
 	t_vec		targets; /* char *, xmalloc'd */
+	t_vec		order; /* const char *, targets ranked for this TAB */
 }	t_make_cache;
 
 typedef struct s_shell

incs/version.h

@@ -15,7 +15,7 @@
 
 /* The single source of truth for the running shell's version. Bumped in step
    with the git tag / GitHub release / npm package / docker image. */
-# define HELLISH_VERSION "3.2.0"
+# define HELLISH_VERSION "3.3.0"
 
 /* Where releases live; used by the `update` builtin and the daily check. */
 # define HELLISH_REPO "Univers42/hellish"
@@ -30,6 +30,6 @@
 /* Bumped whenever the welcome header itself changes what it says. The
    header is shown once a day, but a new revision is new information, so
    it is shown again immediately rather than waiting for tomorrow. */
-# define HELLISH_HEADER_REV 4
+# define HELLISH_HEADER_REV 5
 
 #endif

mkdocs.yml

@@ -55,7 +55,7 @@ markdown_extensions:
 
 nav:
   - Home: index.md
-  - What's new in 3.2: whats-new.md
+  - What's new in 3.3: whats-new.md
   - What it is + Install: product.md
   - The manual: manual.md
   - Builtins reference:

npm/package.json

@@ -1,6 +1,6 @@
 {
   "name": "hellish-shell",
-  "version": "3.2.0",
+  "version": "3.3.0",
   "description": "hellish — a fast, POSIX-compliant shell, with horns",
   "bin": {
     "hellish": "bin/hellish.js"

src/completion/complete_make.c

@@ -34,6 +34,9 @@
 ** TAB.  LIMITATION: a change to an INCLUDED file alone does not invalidate
 ** it -- touch the Makefile.
 **
+** ORDER.  Targets used in history come first (frecency, see
+** complete_make_rank.c), then the rest alphabetically.
+**
 ** WHEN IT STAYS OUT.  A word starting with '-' or holding '=' or '/' is not
 ** a target; and if an earlier argument is -C, --directory*, -f, --file* or
 ** --makefile*, the Makefile is not the one in the cwd, so filenames win.
@@ -54,12 +57,12 @@ static char	*mk_generator(const char *text, int state)
 	t_vec			*v;
 	char			*s;
 
-	v = &(*zle_state_cell())->make_cache.targets;
+	v = &(*zle_state_cell())->make_cache.order;
 	if (!state)
 		i = 0;
 	while (i < v->len)
 	{
-		s = ((char **)v->ctx)[i++];
+		s = (char *)((const char **)v->ctx)[i++];
 		if (!ft_strncmp(s, text, ft_strlen(text)))
 			return (rl_dup(s));
 	}
@@ -90,11 +93,21 @@ static int	mk_diverts(t_pv_line *a)
 	return (0);
 }
 
+/* The matches come out of mk_generator already in the order worth
+   showing (mk_order), so readline must not sort them again: it does that
+   AFTER this returns, so the flag is cleared here and cmd_completion sets
+   it back at the start of every attempt (completion.c). */
 char	**comp_make(t_shell *st, const char *text, t_pv_line *a)
 {
+	char	**m;
+
 	if (text[0] == '-' || ft_strchr(text, '=') || ft_strchr(text, '/'))
 		return (NULL);
 	if (mk_diverts(a) || !mk_load(st))
 		return (NULL);
-	return (rl_completion_matches(text, mk_generator));
+	mk_order(st);
+	m = rl_completion_matches(text, mk_generator);
+	if (m)
+		rl_sort_completion_matches = 0;
+	return (m);
 }

src/completion/complete_make.h

@@ -49,6 +49,25 @@ typedef struct s_mk_scan
 	int		def;
 }	t_mk_scan;
 
+/* A target while ranking by history: how much it was used, and where. */
+typedef struct s_mk_use
+{
+	const char	*name;
+	uint64_t	score;
+	size_t		newest;
+	size_t		seen;
+}	t_mk_use;
+
+/* What counting needs besides the targets: the history length (for age)
+   and the strategy. */
+typedef struct s_mk_rank
+{
+	t_mk_use	*u;
+	size_t		cnt;
+	size_t		len;
+	bool		recent;
+}	t_mk_rank;
+
 # define MK_MAX 4194304
 
 /* The nanosecond mtime field is st_mtim on POSIX 2008, st_mtimespec on
@@ -62,11 +81,17 @@ typedef struct s_mk_scan
 size_t	pc_cmd_off(int start);
 size_t	prov_word(t_pv_line *p, const char **w);
 size_t	prov_base(const char *w, size_t n);
+bool	prov_is_make(const char *w, size_t n);
 char	**comp_provider_try(const char *text, int start, int end);
 char	**comp_make(t_shell *st, const char *text, t_pv_line *a);
 char	*rl_dup(const char *s);
 t_vec	*mk_load(t_shell *st);
 void	mk_cache_free(t_shell *st);
+void	mk_order(t_shell *st);
+int		mk_cmp_use(const void *x, const void *y);
+int		mk_cmp_name(const void *x, const void *y);
+void	mk_count_entry(t_mk_rank *r, const char *e, size_t idx);
+bool	mk_is_make_entry(const char *e);
 char	*mk_slurp(const char *path, size_t *len);
 void	mk_scan(t_mk_scan *m, const char *buf, size_t len);
 void	mk_line(t_mk_scan *m, char *l);

src/completion/complete_make_cache.c

@@ -44,6 +44,7 @@ void	mk_clear(t_make_cache *c)
 	while (i < c->targets.len)
 		xfree(((char **)c->targets.ctx)[i++]);
 	xfree(c->targets.ctx);
+	xfree(c->order.ctx);
 	xfree(c->name);
 	ft_bzero(c, sizeof(*c));
 }