Text missing from HTML diff

When using the following test HTML files as input...

$ cat old.html
<html>
        <body>
                some <div>text and more</div> text
        </body>
</html>

$ cat new.html
<html>
        <body>
                some <div class='red'>text</div> and more <strong>text</strong>
        </body>
</html>

$ graphtage old.html new.html
<html>
        <body>
                some <̟d̟i̟v̟ ̟c̟l̟a̟s̟s̟=̟"̟r̟e̟d̟"̟>̟t̟e̟x̟t̟<̟/̟d̟i̟v̟>̟
        <̟s̟t̟r̟o̟n̟g̟>̟t̟e̟x̟t̟<̟/̟s̟t̟r̟o̟n̟g̟>̟
        <̶d̶i̶v̶>̶t̶e̶x̶t̶ ̶a̶n̶d̶ ̶m̶o̶r̶e̶<̶/̶d̶i̶v̶>̶
    </body>
</html>

+ screenshot:

..., as you can see, the text and more is missing from the diff generated by graphtage.

I've tried some other diff tools and it seems and none of them had any success with correctly processing these two files for some reason (many are using the same core algorithm I suppose). Is there some kind of general issue with processing text not enclosed in tags (as in, and more is between two elements, but not enclosed in any tag (apart from the parent <body> tag) itself)?

I have also tried surrounding and more in a <p> tag in new.html, which resulted in this mess:

$ graphtage old.html new.html
<html>
        <body>
                some <̟d̟i̟v̟ ̟c̟l̟a̟s̟s̟=̟"̟r̟e̟d̟"̟>̟t̟e̟x̟t̟<̟/̟d̟i̟v̟>̟
        <p̟d̶i̶v̶>t̶e̶x̶t̶ ̶and more</p̟d̶i̶v̶>
        <̟s̟t̟r̟o̟n̟g̟>̟t̟e̟x̟t̟<̟/̟s̟t̟r̟o̟n̟g̟>̟
    </body>
</html>

+ screenshot:

What's happening?

[ESultanik]

Still open, and I can now say why. Short version: Graphtage discards that text at parse time, before any diffing happens.

ElementTree splits an element's character data in two. Text before the first child is element.text; text after an element's closing tag, up to the next sibling, is element.tail. Your and more sits between </div> and <strong>, so it is the div's tail. Graphtage reads .text and never reads .tail:

>>> import graphtage.xml as gx
>>> gx.build_tree("new.html")
XMLElement(tag=StringNode('html'), attrib=DictNode([]), text=StringNode('\n\t'),
  children=XMLElementChildren((
    XMLElement(tag=StringNode('body'), attrib=DictNode([]), text=StringNode('\n\t\tsome '),
      children=XMLElementChildren((
        XMLElement(tag=StringNode('div'), attrib=..., text=StringNode('text'), children=...),
        XMLElement(tag=StringNode('strong'), attrib=DictNode([]), text=StringNode('text'), children=...)))),)))

There is no tail anywhere in that tree. grep -rn "\.tail" graphtage/ returns a single hit, matching.py:363, which is an unrelated linked-list accessor in the bipartite matcher; nothing in the package reads an ElementTree tail. XMLElement.__init__ (graphtage/xml.py:161-200) has slots for tag, attrib, text, and _children only, and build_tree (xml.py:290-303) reads only those. The text is gone before the differ sees it, which is why no diff tool setting can bring it back and why your <p>-wrapped attempt produced a scramble instead.

Two consequences worth recording:

This is not an HTML problem. It affects XML identically, and it is independent of which parser you use. XMLElementObj.__str__ (xml.py:125-138) reconstructs <tag attrs>text{children}</tag>, so tail text has nowhere to go on round-trip either.

PR #106 does not fix it, despite adding test_text_between_elements_issue_80. That test's assertion is

self.assertTrue('and more' in all_edits_str or 'and' in all_edits_str, ...)

The or 'and' makes it pass on almost any input, and the first clause passes too: your old document contains text and more as the div's ordinary .text, so 'and more' appears in the edit strings whether or not tails are handled. I ran that exact assertion against current master, where tails are provably dropped, and it passes:

'and more' in edits -> True
assertion would PASS -> True

So the test cannot detect the bug it is named for. Switching parsers is still worth doing for #25, but this issue needs its own fix.

A real fix is a model change, not a parser change: add a tail slot to XMLElement and XMLElementObj, include it in children(), __eq__, and calculate_total_size, handle it in XMLElementEdit, and render it in both the XML and JSON formatters. A regression test should assert on the rendered diff containing the tail text, not on a substring of str(edit).