Skip to content

Appropriately deal with files that don't end with a line terminator - #5058

Merged
Aster89 merged 10 commits into
haskell:masterfrom
Aster89:win
Sep 15, 2026
Merged

Aster89 merged 10 commits into
haskell:masterfrom
Aster89:win

Conversation

@Aster89

@Aster89 Aster89 commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Both the given and expected files don't have a line terminator at EOF.

The new test does fail, demonstrating there is a bug, just like I observe in VSCode, see GIF attached to #5059.

However, Vim+YCM and Neovim seem to be immune to it. (As far as Vim+YCM goes, I know why it's immune because I fixed another bug, ycm-core/YouCompleteMe#4311.)

I think that the test failing shows that the bug is in HLS, not in VSCode. Vim+YCM and Neovim are probably just being smart and sidestepping the bug entirely.


Fixes #5059.

@Aster89 Aster89 changed the title Window-generated file for class-plugin tests Appropriately deal with files that don't end with a line terminator Aug 27, 2026
@Aster89

Aster89 commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator Author

A small experiment that I should probably turn into a test:

$ cabal  repl /home/enrico/haskell-language-server/hls-plugin-api/src/Ide/PluginUtils.hs
λ> import Language.LSP.Protocol.Types
λ> import Ide.PluginUtils
λ> :set -XOverloadedStrings
λ> uri = Uri {getUri = "file:///home/enrico/haskell-language-server/plugins/hls-class-plugin/test/testdata/T1W.hs"}
λ> verTxtDocId = VersionedTextDocumentIdentifier uri 0
λ> old = "module T1 where\n\ndata X = X\n\ninstance Eq X where\n"
λ> new = "module T1 where\n\ndata X = X\n\ninstance Eq X where\n  (==) = _\n"
λ> e1 = diffText' True (verTxtDocId, old) new IncludeDeletions
λ> e1
WorkspaceEdit {_changes = Nothing, _documentChanges = Just [InL (TextDocumentEdit {_textDocument = OptionalVersionedTextDocumentIdentifier {_uri = Uri {getUri = "file:///home/enrico/haskell-language-server/plugins/hls-class-plugin/test/tes
tdata/T1W.hs"}, _version = InL 0}, _edits = [InL (TextEdit {_range = Range {_start = Position {_line = 5, _character = 0}, _end = Position {_line = 5, _character = 0}}, _newText = "  (==) = _\n"})]})], _changeAnnotations = Nothing}

See that the WorkspaceEdit contains _newText = " (==) = _\n".

This is correct. But look what happens if we remove the trailing \n to both old and new:

λ> old = "module T1 where\n\ndata X = X\n\ninstance Eq X where"
λ> new = "module T1 where\n\ndata X = X\n\ninstance Eq X where\n  (==) = _"
λ> e2 = diffText' True (verTxtDocId, old) new IncludeDeletions
λ> e1 == e2
True

which is wrong! The _newText should be "\n (==) = _", not " (==) = _\n".

These are probably the shortest reproduction steps (but diffTextEdit is not currently exported):

λ> d1 = diffTextEdit "foo" "foo\nbar" IncludeDeletions 
λ> d2 = diffTextEdit "foo\n" "foo\nbar\n" IncludeDeletions 
λ> d1 == d2
True
λ> d1
[TextEdit {_range = Range {_start = Position {_line = 1, _character = 0}, _end = Position {_line = 1, _character = 0}}, _newText = "bar\n"}]

@Aster89

Aster89 commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator Author

Well, the bug is clearly on this line:

d = getGroupedDiff (lines $ T.unpack fText) (lines $ T.unpack f2Text)

I mean, once you've done lines on the input texts, linebreaks are long gone. Unless you re-inspect the input texts to see whether they ended with a line terminator, but that's not done, as fText and f2Text are only used on this line.

@Aster89

Aster89 commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator Author

Building on my earlier experience with text diff tools, the solution I've attempted consisted of

  1. swapping Prelude's lines/unlines for these
    lines = split (dropFinalBlank $ keepDelimsR $ whenElt (== '\n'))
    unlines = concat
  2. swapping getGroupedDiff for getGroupedDiffBy ((==) `on` takeWhile (/= '\n'))

The idea is that point 1 allows us not to throw away the line terminators, and point 2 preserves the equality used so far.

The drawback is that the resulting [Diff [String]] for foo vs foo\nbar is the following

[Both ["foo"] ["foo\n"],Second ["bar"]]

from which the current code deduces that "bar" is the only thing to be added to the Second side, forgetting entirely that there was a \n difference between the two first lines.

In the context of a full-fledged text diff tool, the direction I'd take is to perform a sub-comparison between the 2 sides of the Boths; eventually we'd get something like this,

[(Both ["foo"] ["foo\n"], Just (NonEmpty [Both "foo" "foo", Second "\n"])]),(Second ["bar"], Nothing)]

where the Maybe wraps the possibly absent/empty subcomparison.

If we were to diff foo\n vs foo\nbar\n then the above "diff+subdiff" would look like this:

[(Both ["foo\n"] ["foo\n"], Nothing]),(Second ["bar\n"], Nothing)]

Anyway, I've taken note of the above to avoid forgetting, but it sounds too much of a complication considering that the only time that Just would ever materialize is when we're adding a line at the end of a Windows/VSCode/windows-like-thingy--generated file.

And maybe it would break several tests.

Probably a simple hack is a better approach. Looking into it. But I also have to check what happens in case an action removes the last line of a file.

@Aster89
Aster89 marked this pull request as ready for review August 28, 2026 17:08
@Aster89

Aster89 commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator Author

This PR, in its current state, is

What do I mean?

  1. From the perspective of text comparison, the solution is wrong, as demonstrated by the two tests that fail;

    • incidentally, it doesn't even update the the TextEdit's _range field, only the _newText field;
  2. from the perspective of our usage of it, which is from call sites that guarantee (or don't they?) that we'll never get those inputs like in the files that cause those failures, it's good enough.

The point is that we never use this diffTextEdit function on two independent Text inputs. Those two inputs are

  • the content of the source file on which HLS wants to do the change,
  • the content after the change as computed via GHC's API.

As long as GHC (well, and ghc-exactprint after it) guarantees to honor the line-ending policy of a file, the scenario of the tests at point 1 above should never materialize.

@Aster89
Aster89 requested a review from MangoIV August 28, 2026 17:23
@Aster89
Aster89 force-pushed the win branch 3 times, most recently from c7f5e54 to 010d923 Compare September 2, 2026 17:33
@Aster89 Aster89 self-assigned this Sep 2, 2026
@Aster89

Aster89 commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

All tests that fail are failing the same way.

Here's an example failure:

  TSimpleDecl (golden):          FAIL (6.45s)
    Test output was different from 'plugins/hls-splice-plugin/test/testdata/TSimpleDecl.expected.hs'. Output of ["git","-c","core.fileMode=false","diff","--no-index","--text","--exit-code","plugins/hls-splice-plugin/test/testdata/TSimpleDecl.expected.hs","/tmp/TSimpleDecl.expected1941492-18.actual"]:
    diff --git a/plugins/hls-splice-plugin/test/testdata/TSimpleDecl.expected.hs b/tmp/TSimpleDecl.expected1941492-18.actual
    index 90c2bf1b0..d0178c7b0 100644
    --- a/plugins/hls-splice-plugin/test/testdata/TSimpleDecl.expected.hs
    +++ b/tmp/TSimpleDecl.expected1941492-18.actual
    @@ -7,6 +7,7 @@ import Language.Haskell.TH ( mkName, clause, normalB, funD, sigD )
     --  Bar
     foo :: Int
     foo = 42
    +
     -- Bar
     -- ee
     -- dddd

    Use -p '/TSimpleDecl (golden)/' to rerun this test only.

The given file is this

{-# LANGUAGE TemplateHaskell #-}
{-# LANGUAGE QuasiQuotes #-}
module TSimpleDecl where
import Language.Haskell.TH ( mkName, clause, normalB, funD, sigD )

-- Foo
--  Bar
$(sequence
    [sigD (mkName "foo") [t|Int|]
    ,funD (mkName "foo") [clause [] (normalB [|42|]) []]
    ]
    )
-- Bar
-- ee
-- dddd

and the expected is this

{-# LANGUAGE TemplateHaskell #-}
{-# LANGUAGE QuasiQuotes #-}
module TSimpleDecl where
import Language.Haskell.TH ( mkName, clause, normalB, funD, sigD )

-- Foo
--  Bar
foo :: Int
foo = 42
-- Bar
-- ee
-- dddd

I've logged the TextEdit inside the WorkspaceEdit:

TextEdit {_range = Range {_start = Position {_line = 7, _character = 0}, _end = Position {_line = 11, _character = 5}}, _newText = "foo :: Int\nfoo = 42\n"}

Its _range goes from the beginning of the line containing the $(, to right after the last character of the line containing the matching ).

Since Range represets a half-open range, that _range is representing the text from $( to the matching ), without the line break character after it. In other words, that range is representing the following bytes

  • if the file was saved on Linux:
    $(sequence\n    [sigD (mkName "foo") [t|Int|]\n    ,funD (mkName "foo") [clause [] (normalB [|42|]) []]\n    ]\n    )
    
  • if the file was saved on Windows:
    $(sequence\r\n    [sigD (mkName "foo") [t|Int|]\r\n    ,funD (mkName "foo") [clause [] (normalB [|42|]) []]\r\n    ]\r\n    )
    

Notice that there isn't a line break at the end, because a Range is half open (and, to be precise, the LSP doesn't even talk about line breaks as proper characters/bytes).

If we change that text for one that does have a line break at the end (see the _newText from the TextEdit above), we'll get an empty line in excess.

I'm now trying to understand where that \n comes from, and what breaks if I change the code that inserts it not to insert it.

@Aster89

Aster89 commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

This snippet,

transform
dflags
clientCapabilities
verTxtDocId
(graftDecls (RealSrcSpan spliceSpan Nothing) expanded)
ps
<&>
-- FIXME: Why ghc-exactprint sweeps preceding comments?
adjustToRange (verTxtDocId ^. J.uri) range

is where the WorkspaceEdit is generated by feeding the document and a function of type Graft (Either String) ParsedSource to the transform function below:

transform ::
DynFlags ->
ClientCapabilities ->
VersionedTextDocumentIdentifier ->
Graft (Either String) ParsedSource ->
ParsedSource ->
Either String WorkspaceEdit
transform dflags ccs verTxtDocId f a = do
let src = printA a
a' <- transformA a $ runGraft f dflags
let res = printA a'
pure $ diffText ccs (verTxtDocId, T.pack src) (T.pack res) IncludeDeletions

But transform seems to do a pretty simple job, and it uses the very diffText of which I'm alterning the implementation, so I would first investigate graftDecls, as maybe that's the one that is doing something weird with line breaks.


For the very example in a previous message, of which I copy-and-paste the input file,

{-# LANGUAGE TemplateHaskell #-}
{-# LANGUAGE QuasiQuotes #-}
module TSimpleDecl where
import Language.Haskell.TH ( mkName, clause, normalB, funD, sigD )

-- Foo
--  Bar
$(sequence
    [sigD (mkName "foo") [t|Int|]
    ,funD (mkName "foo") [clause [] (normalB [|42|]) []]
    ]
    )
-- Bar
-- ee
-- dddd

I've printed the src and res passed to diffText, together with a comment marking the start of each 0-based line:

   "\n\nmodule TSimpleDecl where\nimport Language.Haskell.TH ( mkName, clause, normalB, funD, sigD )\n\n\n\n$(sequence\n    [sigD (mkName \"foo\") [t|Int|]\n    ,funD (mkName \"foo\") [clause [] (normalB [|42|]) []]\n    ]\n    )\n\n\n\n"
--  0 1 2                         3                                                                   4 5 6 7           8                                    9                                                           10     11     12131415
   "\n\nmodule TSimpleDecl where\nimport Language.Haskell.TH ( mkName, clause, normalB, funD, sigD )\nfoo :: Int\nfoo = 42\n\n\n\n"
--  0 1 2                         3                                                                   4           5         6 7 8 9

A couple of observations:

  • I see that the the leading \n\n, i.e. the two leading empty lines, are what remains of the two pragmas {-# LANGUAGE TemplateHaskell #-} and {-# LANGUAGE QuasiQuotes #-} taken away by some pre-processing phase;

  • I see that the \n\n\n\n in the first string correspond to the fact that the line where $( is 4 lines after the import line, so 3 intercurring lines, and again, they are what remains of (one line that was already empty, and) two lines with just comments (-- Foo, and -- Bar);

  • I don't see why in the result there's only one \n between the import line and the signature of foo, i.e. no empty line in between, considering that the generated file still has the correct 3 lines in between.

  • After all, the TextEdit

    TextEdit {_range = Range {_start = Position {_line = 4, _character = 0}, _end = Position {_line = 11, _character = 6}}, _newText = "foo :: Int\nfoo = 42\n"}

    is correct (notice that _end is after the \n that's after the ) corresponding to $(, which is consistent with _newText providing its own trailing \n)…

  • … but despite that TextEdit has the same _newText as that in a previous message, it has, crucially, a different _start and _end:

    TextEdit {_range = Range {_start = Position {_line = 7, _character = 0}, _end = Position {_line = 11, _character = 5}}, _newText = "foo :: Int\nfoo = 42\n"}

    This above is the value of edits (wrongly called with plural) of the following line:

  • Maybe, the _start = Position {_line = 7, _character = 0} that replaces the _start = Position {_line = 4, _character = 0} is precisely to take into account those 3 lines (1 empty + 2 just comments) that have to be preserved in the output;

  • however replacing _end = Position {_line = 11, _character = 6} with _end = Position {_line = 11, _character = 5} is what causes the printing of the empty line, because it doesn't match the \n that the other TextEdit matches, but it substitutes this range with the same _newText, which comes with a trailing \n.

Maybe I'm getting somewhere.

@Aster89

Aster89 commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

Oh, that's the culprit!

<&>
-- FIXME: Why ghc-exactprint sweeps preceding comments?
adjustToRange (verTxtDocId ^. J.uri) range

Eh... Not sure what to do yet. Surely removing it does not good.

Next to investigate: adjustToRange.

@Aster89
Aster89 requested a review from konn as a code owner September 9, 2026 11:06
@fendor
fendor requested a review from crtschin September 9, 2026 12:23
As described in the PR, the issue with the existing `diffTextEdit`
algorithm is that it entirely throws away line breaks, so it can't
really make a difference between a file that ends with a line break and
a file that doesn't.

The road to the solution, in hindsight, was pretty simple:

  1. Trust that, other than the issue described above, the algorithm is
     sound.

  2. Override the "classic" `lines` and `unlines` with lossless
     counterparts (in other words, when splitting, don't throw away the
     separators).

     This was as easy as
     ```
     lines = split (dropFinalBlank $ keepDelimsR $ whenElt (== '\n'))
     unlines = concat
     ```

  3. See what breaks and fix it.

     This boiled down to just removing a call to `init` that was applied
     to the result of `unlines`.

The ad-hoc tests I've written for the class- and case-split- plugins
both pass, but the tests I've added for `diffTextEdit`, and more
specifically the `diffTextEditComplete` helper function, deserve an
explanation:

  - (All tests' `Text` triples (left, right, and expected
    deleted/inserted text) are carefully aligned to help the eye detect
    how they relate to each other.)

  - When both inputs `Text`s to `diffTextEditComplete` end with `'\n'`,
    the expected edit should not surprise, both in the tests that insert
    something at EOF and in those that delete something at EOF.

  - In all other cases, the expected edit might catch you off guard; at
    least it did in my case.

    Here follows one of those tests, together with an explanatory
    commentary:
    ```haskell
         …
           $ diffTextEditComplete "foo"
                                  "foo\nbar\n"
                    @?= [textEdit "foo\nbar\n"
                                  (mkRange 0 0 0 3)]
    ```
    The line-based tokenization will result in `["foo"]` for the left
    file and in `["foo\n", "bar\n"]` for the second file. As you can
    see, there's no entry in common between these two lists, because
    `"foo" /= "foo\n"` (yes, we still use `getGroupedDiff`, i.e.
    `getGroupedDiffBy (==)`). This translates to the fact that the
    algorithm, rather than detecting that `"\nbar\n"` was inserted,
    detects that `"foo"` was deleted, and `"foo\nbar\n"` was inserted,
    which boils down to the same result.
@Aster89

Aster89 commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

In the usecase I'm referring to (see #5058 (comment)), the relevant line of adjustToRange is this

but what adjustLine does

adjustLine :: Range -> TextEdit -> TextEdit
adjustLine bad =
J.range %~ \r ->
if r == bad then ran else bad

is simply picking one between bad and ran, where ran is actually an input to adjustToRange, which is passed at this line:

adjustToRange (verTxtDocId ^. J.uri) range

range is defined here:

and spliceSpan is brought into scope by .. here

expandTHSplice _eStyle ideState _ params@ExpandSpliceParams {..} = ExceptT $ do

and it's one of ExpandSpliceParams's fields:

-- | Parameter for the addMethods PluginCommand.
data ExpandSpliceParams = ExpandSpliceParams
{ verTxtDocId :: VersionedTextDocumentIdentifier
, spliceSpan :: RealSrcSpan
, spliceContext :: SpliceContext
}
deriving (Show, Eq, Generic)
deriving anyclass (ToJSON, FromJSON)

@Aster89

Aster89 commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

A short-ish recap:

  1. The spliceSpan passed to the plugin through ExpandSpliceParams's:

    SrcSpanMultiLine "/home/enrico/haskell-language-server/plugins/hls-splice-plugin/test/testdata/TSimpleDecl.hs" 8 1 12 6

    represents the sequence of bytes

    $(sequence\n    [sigD (mkName "foo") [t|Int|]\n    ,funD (mkName "foo") [clause [] (normalB [|42|]) []]\n    ]\n    )
    

    which does not include a trailing \n,

  2. a Graft (Either String) ParsedSource is created via graftDecls

    (graftDecls (RealSrcSpan spliceSpan Nothing) expanded)

  3. the whole call to transform

    transform
    dflags
    clientCapabilities
    verTxtDocId
    (graftDecls (RealSrcSpan spliceSpan Nothing) expanded)
    ps

    results in this TextEdit:

    TextEdit {_range = Range {_start = Position {_line = 4, _character = 0}, _end = Position {_line = 11, _character = 6}}, _newText = "foo :: Int\nfoo = 42\n"}

    which is:

    • good, because it is self-consistent in that

      • it covers the \n after the ) corresponding to the opening $(,

      • has a _newText coming with its own trailing \n;

    • bad, because it starts too early, covering the lines with the comments.

  4. That TextEdit is then "post-processed" here:

    <&>
    -- FIXME: Why ghc-exactprint sweeps preceding comments?
    adjustToRange (verTxtDocId ^. J.uri) range

    which is:

    • good, because it works around the bad point above;

    • bad, because it exchanges the computed TextEdit's _range with the Range corresponding to spliceSpan, which does not cover the trailing \n, and is therefore inconsistent with the computed TextEdit's _newText.


Now, I think that the call to adjustToRange is a hack, in line with the comment accompanying it, to workaround a buggy (or not well understood) behavior of ghc-exactprint. As a hack, it is ad-hoc, and it's falling apart upon my attempt to fix diffTextEdit.

So what's adjustToRange really doing?

I've rewritten it below, but with some slight change that is really just cosmetic

adjustToRange :: Uri -> Range -> WorkspaceEdit -> WorkspaceEdit
adjustToRange uri ran wsEdit@WorkspaceEdit{..} =
    wsEdit { _changes = adjustWS <$> _changes
           , _documentChanges = fmap adjustDoc <$> _documentChanges }
    where
        adjustTextEdits :: Traversable f => f TextEdit -> f TextEdit
        adjustTextEdits eds =
            let minStart =
                    case L.fold (L.premap (view J.range) L.minimum) eds of
                        Nothing -> error "impossible"
                        Just v  -> v
            in adjustLine ran minStart <$> eds

        adjustATextEdits :: Traversable f => f (TextEdit |? AnnotatedTextEdit) -> f (TextEdit |? AnnotatedTextEdit)
        adjustATextEdits = fmap $ \case
          InL t -> InL $ runIdentity $ adjustTextEdits (Identity t)
          InR ate@AnnotatedTextEdit{ _annotationId } ->
               InR $ annotate (runIdentity $ adjustTextEdits $ Identity $ unannotate ate) _annotationId
          where
            unannotate AnnotatedTextEdit{..} = TextEdit _range _newText
            annotate TextEdit{..} anno = AnnotatedTextEdit _range _newText anno

        adjustWS = ix uri %~ adjustTextEdits

        adjustDoc :: DocumentChange -> DocumentChange
        adjustDoc (InR es) = InR es
        adjustDoc (InL es)
            | es ^. J.textDocument . J.uri == uri =
                InL $ es & J.edits %~ adjustATextEdits
            | otherwise = InL es

adjustLine :: Range -> Range -> TextEdit -> TextEdit
adjustLine good bad =
    J.range %~ \r ->
        if r == bad then good else bad

adjustWS and adjustDoc do very similar things, one using adjustTextEdits and the other using adjustATextEdits, but in very similar ways; furthermore, adjustATextEdits simply offloads the job to adjustTextEdits, and the crucial bit is what this latter function does:

  1. it extracts the Range of the left-most (or shortest when ties) TextEdit from the Traversable argument;

  2. then for each TextEdit in the Traversable, it calls adjustLine with

    • the ran :: Range that turns out to correspond to spliceSpan

    • the minStart :: Range computed at step 1

    • the TextEdit.

I need to understand why adjustTextEdits and adjustLine do what they do.

I haven't investigated why adjustLine, and the whole adjustToRange, really, do what they do, but they were part of a hack in the first place, according to the comment accompanying the invocation of adjustToRange:

<&>
-- FIXME: Why ghc-exactprint sweeps preceding comments?
adjustToRange (verTxtDocId ^. J.uri) range

With my change, I'm really just changing adjustLine:

  • before my change, it swaps bad :: Range for ran :: Range;

  • after my change, it limits such a replacement to the _start of
    bad, leaving _end unchanged.

So mine is just another hack on top of that existing hack.

Here I'm really just chainging `adjustLine`:

  - before my change, it swaps `bad :: Range` for `ran :: Range`

  - after my change, it limits such a replacement to the `_start` of
    `bad`, leaving `_end` unchanged.

I haven't investigated why `adjustLine`, and the whole `adjustToRange`,
really, do what they do, but they were part of a hack in the first
place, according to the comment accompaigning the _invocation_ of
`adjustToRange`:

https://github.com/haskell/haskell-language-server/blob/410747c50f3eb6e53d7891ffbae59050670eb382/plugins/hls-splice-plugin/src/Ide/Plugin/Splice.hs#L172-L174

So mine is just another hack on top fo that hack.

@crtschin crtschin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice investigative work! I think there might still be a positional bug here.

Comment thread hls-plugin-api/hls-plugin-api.cabal Outdated
Comment thread hls-plugin-api/src/Ide/PluginUtils.hs
Comment thread hls-plugin-api/src/Ide/PluginUtils.hs
Comment thread hls-plugin-api/test/Ide/PluginUtilsTest.hs Outdated
@Aster89
Aster89 requested a review from crtschin September 14, 2026 17:24

@crtschin crtschin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job fixing this! 🚀

I'm not going to be insistent about the version bounds, so approving! Feel free to loosen or delete the version bound on split if you still feel like it.

Keeping as-is is also okay, split is stable enough where it probably doesn't matter.

@Aster89
Aster89 enabled auto-merge (squash) September 15, 2026 08:47
@Aster89
Aster89 merged commit 01a25d4 into haskell:master Sep 15, 2026
57 of 59 checks passed
@Aster89
Aster89 deleted the win branch September 15, 2026 11:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants