Repository navigation
Add Support for none UTF-8 texts & Performance improvements - #2637
Merged
Merged
Conversation
Remove not needed decoder on stream byte sources removing the unwritten contract that each source of them must stop on line breaks. This changes is sound after the previous changes on producer loop to ensure that parsers can communicate their state and what they need. The serial port is no longer split into halves: reading and writing both go through `&mut self`, so the lock behind the split paid for concurrency that can't happen. Also fixes the intermittently flappy TCP reconnect tests, which drove real sockets under paused time and asserted on attempt counts read from a `watch` channel that keeps the latest value only.
Text files were linked as the session file, so consumers read the original bytes while the display path decoded them lossily. They now go through the producer and StringTokenizer like every other source, which makes the session file valid UTF-8 by construction. - Drop SessionFileOrigin, the linked flow in observe_file and the origin checks in observe.rs; the session file is always generated. - Delete dead code uncovered by this: grabber::factory, TextFileSource::contains_text, file_tools::is_binary, and the UpdateSession/GetSessionFileOrigin state APIs.
Use spawn_blocking instead of block_in_place as the move cost is small compared with forcing a multi threaded tokio runtime as hidden contract
* Add parser for text encoded with UTF-16 format. * Determine which text parser to use by sniffing the first few bytes of the source. * Sniffing respects SDE income messages. * String encoding is respected in raw exports too. * Add files for testing and snapshot tests.
Replace logs general collector with special collector for benchmarks which will avoid allocation on message, instead it'll write them into a sink and clear it out once full to avoid measuring none-related memory allocations
+ Add validation for texts benches to ensure the provided text input matches the bench's parser
Attachment can take much more space compared with messages. This is the case for string parsers where we gain up to 15% performance improvements with this changes. It was also tested with attachments-heavy DLT files without any noticeable regressions there.
Scan the whole provided bytes slice once for line breaks instead of implementing single parser which will repeated memchar setup for each line. Changes improved performance about 5%
Performance improvements for UTF-16 parser by: * Use memchar to find line breaks instead of the normal linear search with no SIMD. * Scan & Parse all available bytes at once by implementing Parser trait instead of SingleParser Changes ended up with 12% performance gain
Pre-allocate the output string instead of collecting. This small change improved performance by about 30% + Small code style refactoring
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR closed #1921
It adds support for string encoding other that UTF-8 and also improve parsing performance for text sources to up 1.35 x and for attachments heavy dlt files with about 1.20 x
It includes:
Here is a table for performance changes: