Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up results display #66

Merged
merged 3 commits into from
Dec 29, 2022
Merged

Speed up results display #66

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3120046
Speed up results display
Dec 28, 2022
b1cea64
Add RESULT_DISPLAY_SLEEP_STEP
Dec 28, 2022
f05c62f
Rename to display_results_batch_size
Dec 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
### Fixed

- Fix execution of SHOW, EXPLAIN and DESCRIBE commands.
- Speed up results display.

## [0.12.0] - 2022-11-30

Expand Down
6 changes: 5 additions & 1 deletion streaming_jupyter_integrations/magics.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@

@magics_class
class Integrations(Magics):

def __init__(self, shell: Any):
super(Integrations, self).__init__(shell)
self._secrets: Dict[str, str] = {}
Expand All @@ -55,6 +56,7 @@ def __init__(self, shell: Any):
self.wait_timeout_ms = 60 * 60 * 1000 # 1H
# 20ms
self.async_wait_s = 2e-2
self.display_results_batch_size = 25
Integrations.__enable_sql_syntax_highlighting()
self.deployment_bar = DeploymentBar(interrupt_callback=self.__interrupt_execute)
# Indicates whether a job is executing on the Flink cluster in the background
Expand Down Expand Up @@ -405,7 +407,9 @@ async def display_execution_result(self, execution_result: TableResult, display_
display_handle = None
for result in results:
# Explicit await for the same reason as in `__internal_execute_sql`
await asyncio.sleep(self.async_wait_s)
if rows_counter.value % self.display_results_batch_size == 0:
# Sleeping for each row slows down showing the results significantly.
await asyncio.sleep(self.async_wait_s)
res = list(result)
if display_row_kind:
res = [result.get_row_kind()] + res
Expand Down