[BUG] org.apache.lucene.analysis.TokenStream.end() can slow down elemental

[tohidemyname]

Describe the bug

Lucene fixed a performance bug: https://issues.apache.org/jira/browse/LUCENE-7419

It complains that TokenStream.end() is quite costly. This bug is marked as a blocker bug. According to the bug report, it affects 5.5.5, 6.2, and 7.0. The reporter complains that TokenStream.end() wrongly calls getAttribute().The buggy end() method is as follows:

public void end() throws IOException {
clearAttributes(); // LUCENE-3849: don't consume dirty atts
PositionIncrementAttribute posIncAtt = getAttribute(PositionIncrementAttribute.class);
if (posIncAtt != null) {
posIncAtt.setPositionIncrement(0);
}
}

Elemental uses lucene 4.10.4. I checked the source code of lucene 4.10.4. Its code is identical to the buggy code:

public void end() throws IOException {
clearAttributes(); // LUCENE-3849: don't consume dirty atts
PositionIncrementAttribute posIncAtt = getAttribute(PositionIncrementAttribute.class);
if (posIncAtt != null) {
posIncAtt.setPositionIncrement(0);
}
}

As a result, this bug should also affect 4.10.4.

To Reproduce

In the lucene bug report (LUCENE-7419), Michael McCandless mentioned that this bug was found by elasticsearch:

"This is the apparent source of the very unexpected slowdown here: elastic/elasticsearch#19867 (comment)"

He also explained how to reproduce such a bug.

Elemental calls the buggy method at the following locations:

<--XMLToQuery.phraseQuery
<--XMLToQuery.nearQuery
<--XMLToQuery.getTerm
<--MarkableTokenFilter.incrementToken
<--MarkableTokenFilter.incrementToken
<--RangeIndexWorker.analyzeContent

The lucene bug is fixed in 6.2.

[adamretter]

this bug should also affect 4.10.4.

@tohidemyname Can you confirm that this is a problem in 4.10.4, or do you only suspect that it could be?

[tohidemyname]

The bug is confirmed by Lucene and elasticsearch. However, when lucene fix this bug, it does not provide a test case to reproduce the bug:
apache/lucene-solr@f8536ce

I have checked the source files of lucene 4.10.4 and the buggy code of https://issues.apache.org/jira/browse/LUCENE-7419. They are identical. This is a performance bug. It could be difficult to produce. As it is fixed, it could be safe to update lucene?