Improve faulty scanning on comments #12892
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Comment was the only (remaining) scanned token to possibly signal a syntax error notification (and possibly launch a recovery editor in the UI) instead of producing an error token. This PR change that.
The issue was that comments are stripped from the scanner flow, but AFTER the next token is scanned (so that we can attach the comments on the token returned by
next
).The proposal is if one of the comment is a bad comment (the only current case is an unterminated comment), we have to rewind the stream position (and the scanned state) just before the faulty token so that the nextnext
will scan and return this faulty token.The proposal is if one of the comment is a bad comment (the only current case is an unterminated comment), we just remember it, so it will be the next token (returned by
next
).Note that EFFormatter have some issues with comments, so tests are let as it but will catch on future improvements.