[Bug]: Text after an inline <link>, <meta>, <style> or <noscript> is dropped from cleaned_html and markdown #2416

crawl4ai version

0.9.4 (reproduced). The same code is in v0.9.5 and on main and develop (checked 2026-10-19).

Expected Behavior

When the scraper strips <style>, <link>, <meta> and <noscript> from the body, the text that follows each of them stays in cleaned_html and in the markdown.

Current Behavior

The text that follows each removed element is lost. LXMLWebScrapingStrategy._scrap removes these tags with lxml's remove() (crawl4ai/content_scraping_strategy.py, lines 827-830 in v0.9.5):

for tag in ["style", "link", "meta", "noscript"]:
    for element in body.xpath(f".//{tag}"):
        if element.getparent() is not None:
            element.getparent().remove(element)

lxml stores the text after an element as that element's .tail, and remove() discards it together with the element.

Wikipedia articles are hit in particular, because MediaWiki puts <link rel="mw:PageProp/Category"> inside paragraphs. https://en.wikipedia.org/wiki/Tokyo has 18 such <link> elements followed by text inside a <p>; one of them is followed by ", Tokyo is home to 26 of the world's 500 largest companies, ...", which never reaches the output. The crawl still reports success, so nothing signals the loss.

Is this reproducible?

Yes

Inputs Causing the Bug

Any HTML where <link>, <meta>, <style> or <noscript> is followed by text in the same parent, for example https://en.wikipedia.org/wiki/Tokyo.

Steps to Reproduce

1. Run the snippet below.
2. Compare the printed markdown with the paragraphs in HTML.

Code snippets

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

HTML = """<html><body><article>
<p>The port grew quickly, and as of 2020<link rel="mw:PageProp/Category" href="/x">, it handles more cargo than any other port.</p>
<p>The old warehouses stood on the quay<meta property="mw:PageProp/toc"> until the great fire destroyed them.</p>
<p>Styled text<style>.a{color:red}</style> continues after a style element.</p>
<p>Plain text<noscript>Enable JavaScript</noscript> continues after a noscript element.</p>
</article></body></html>"""

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("raw:" + HTML, config=CrawlerRunConfig(cache_mode=CacheMode.BYPASS))
        print(result.markdown.raw_markdown)

asyncio.run(main())

Output on 0.9.4:

The port grew quickly, and as of 2020
The old warehouses stood on the quay
Styled text
Plain text

Expected:

The port grew quickly, and as of 2020, it handles more cargo than any other port.
The old warehouses stood on the quay until the great fire destroyed them.
Styled text continues after a style element.
Plain text continues after a noscript element.

Suggested fix

Replace element.getparent().remove(element) with element.drop_tree(). drop_tree() comes with lxml.html and moves the element's tail text onto the previous sibling or the parent. With this one-line change the snippet prints the expected output above.

Related:

• [Bug]: remove_empty_elements_fast() drops trailing text when removing empty elements with non-empty .tail #1938 / fix: preserve .tail text when removing empty elements (#1938) #1939 fixed the same loss in remove_empty_elements_fast() by moving .tail by hand.
• Fix: preserve trailing prose when excluding links #2377 (open) fixes it for links removed by exclude_external_links and similar options, using drop_tree().

This loop is a third place with the same pattern.

OS

Debian GNU/Linux 12 (bookworm), in the official unclecode/crawl4ai:0.9.4 Docker image (lxml 6.1.3)

Python version

3.12.14

Error logs & Screenshots (if applicable)

None. The crawl succeeds and the text is silently missing.