Refresh button for source rows in Status page #6175

lovincyrus · 2024-11-26T21:38:57Z

address #6154

invalidate resources data when refresh source is clicked
unable to invalidate individual source resource

ericpgreen2 · 2024-11-27T03:25:13Z

Note: we should also include the refresh button for Models because 1) a Model can be a root node of the DAG, 2) an intermediate model could still have logic that needs a refresh independent from an upstream node, and 3) we're soon deprecating the Source concept anyways.

And actually, if it's not too much a scope expansion, it'd be nice to be able to refresh Alerts & Reports too. See:

lovincyrus · 2024-12-04T01:30:06Z

Note: we should also include the refresh button for Models because 1) a Model can be a root node of the DAG, 2) an intermediate model could still have logic that needs a refresh independent from an upstream node, and 3) we're soon deprecating the Source concept anyways.

And actually, if it's not too much a scope expansion, it'd be nice to be able to refresh Alerts & Reports too. See:

Added row click refresh button to Source and Model resource kind. I also added a tooltip to indicate that a resource trigger will update all dependent resources.

Latest demo:

CleanShot.2024-12-03.at.17.27.54.mp4

ericpgreen2

There's a lot of action when clicking the refresh button: flickering, jumpiness, table resizing. Some things I noticed:

The Resources table unmounts temporarily. See Jam and screenshot:
The page gets auto-scrolled to the top (Jam), probably due to point 1.
A RefreshTrigger resource is shown temporarily. Can we hide it from the list? Here's a screenshot from your video:

lovincyrus · 2024-12-04T20:28:05Z

Latest demo:

CleanShot.2024-12-04.at.12.26.40.mp4

resolved

ericpgreen2

Mostly looks good, but I ran into a weird state: ~50% of the time when refreshing a Source with an error, the source is marked "Idle", yet for about 30s the "Refresh all sources and models" button is disabled and I see polling ListResources network requests. Here's a Jam.

I also hit this 500 error, but I think it's been fixed on main, right? So it'd help if you could merge main into this branch.

scripts/tsc-with-whitelist.sh

lovincyrus · 2024-12-10T02:37:35Z

Mostly looks good, but I ran into a weird state: ~50% of the time when refreshing a Source with an error, the source is marked "Idle", yet for about 30s the "Refresh all sources and models" button is disabled and I see polling ListResources network requests. Here's a Jam.

I also hit this 500 error, but I think it's been fixed on main, right? So it'd help if you could merge main into this branch.

I can see from the Jam that the specific resource didn't get refreshed properly due to the "failed to ingest source" error. Here's a video of a happy path:

CleanShot.2024-12-10.at.10.32.21.mp4

Is it expected to disable the refresh icon when a resource errors out? I can see in the Jam, the resource continues to error out even after a refresh..

Update:

Gracefully stop polling after detecting reconcile error

CleanShot.2024-12-10.at.14.36.28.mp4

lovincyrus · 2024-12-10T06:04:41Z

I also hit this 500 error, but I think it's been fixed on main, right? So it'd help if you could merge main into this branch.

After a rebase, I still run into the 500 error occasionally when I fire a $createTrigger call from the individual rows. This doesn't need to block this PR. Would you have more context? @begelundmuller

Error message — "grpc: error while marshaling: marshaling rill.runtime.v1.ListResourcesResponse: size mismatch (see golang/protobuf#1609): calculated=229, measured=177"

begelundmuller · 2024-12-10T12:56:02Z

After a rebase, I still run into the 500 error occasionally when I fire a $createTrigger call from the individual rows. This doesn't need to block this PR. Would you have more context? @begelundmuller

Yeah that should get fixed by this PR: #6233

ericpgreen2

UXQA: I've triggered a Source refresh, yet I don't see any polling of the ListResource API.

Kapture.2024-12-16.at.14.40.05.mp4

web-admin/src/features/projects/status/ProjectResources.svelte

lovincyrus · 2024-12-18T15:57:15Z

UXQA: I've triggered a Source refresh, yet I don't see any polling of the ListResource API.

Kapture.2024-12-16.at.14.40.05.mp4

In this PR, I've used the ListResources API to refresh a resource.

Taking another look at this,

Specifically, the Refresh a model command.

I see runtimev1.RefreshModelTrigger and runtimev1.CreateTriggerRequest from the refresh.go. Any reason why we don't have RefreshModelTrigger runtime service? @ericpgreen2

begelundmuller · 2024-12-27T15:17:23Z

@lovincyrus @ericpgreen2 FYI RefreshModelTrigger is used in CreateTriggerRequest.models (link) to enable fine-grained refreshes of models (incremental/full refreshes, partition refreshes, retries of errored partitions, etc.). You can still use specify a model in CreateTriggerRequest.resources (link), but then it will always be a "normal" model refresh (which is a full refresh for non-incremental models and an incremental refresh for incremental models).

ericokuma · 2025-01-17T23:00:00Z

I'm getting a 400 error when trying to access this PR

ericokuma · 2025-01-22T17:47:36Z

from @jkhwu: After they click refresh we can display a temp toast “🔘 Refreshing ” Followed by another toast “✔️ Successfully refreshed ”

web-admin/src/features/projects/status/selectors.ts

web-admin/src/features/projects/status/ProjectResources.svelte

web-common/src/lib/event-bus/events.ts

web-common/src/features/file-explorer/new-files.ts

ericpgreen2

When the user navigates away from the status page, we'll have to get rid of this toast message, else it'll spin forever.

Tangentially, @ericokuma / @jkhwu, I'd propose we remove this toast message altogether, since it's redundant to the spinner in the table (pictured below). And, in its current form, it's not always accurately describing what's happening. For example, when the initial resource has refreshed successfully, and a downstream resource (or multiple) are refreshing, the spinner will incorrectly say that the initial resource is currently refreshing.

web-admin/src/features/projects/status/ProjectResources.svelte

web-admin/src/features/projects/status/ActionsCell.svelte

web-common/src/lib/event-bus/event-bus.ts

resolved

This reverts commit b0811b8.

web-admin/src/features/projects/status/ProjectResources.svelte

ericpgreen2

I just left a comment that the backoff function isn't currently working, but, after resolving that, this looks good to go!

* new cell to refresh source in status tab * move refresh sources to higher order * wip * add refresh confirm dialog * wip, fixme * refetch allResources on row reload click * force a new query using refetchKey for clean data * add tooltip to menu item * lint * feedback, custom polling to avoid stale data * error handling * tsc whitelist * Revert "tsc whitelist" This reverts commit 792276e. * add refresh trigger to new files * stop polling when reconcile error * only stop polling when all resources are in idle status * rebase fix * reserved space for actions in table * color loading spinner, loading type in notification * confirmation dialog for single resource * improve notifications based on diff resource status * lint * refactor onError, feedback * message fix * customizable timeout notification, feedback * use persisted, remove x from loading notification * proper notification clean up with id * hoist timeout to constants * max backoff * use resources, fetch interval * remove timestamp * polling for refresh all * lint * readd loading notification, type check * type check * use is fetching for refreshing, max poll * handle navigate away case * fix * optional type, clean up * fix * polish navigate away and back notification * add refresh trigger to user facing resource kinds * add clear-all-notifications event * feedback * lint * reset * make refresh call async * just right * backoff strat, error handling * move to pure function for refetch interval * clean up, clean up, clean up * lint * reverts * Revert "reverts" This reverts commit b0811b8. * update refetch interval * revalidate in refresh all * fix action cell sizing, compact

lovincyrus marked this pull request as ready for review December 4, 2024 01:30

lovincyrus requested a review from ericpgreen2 December 4, 2024 01:30

ericpgreen2 previously requested changes Dec 4, 2024

View reviewed changes

lovincyrus requested a review from ericpgreen2 December 4, 2024 20:28

lovincyrus force-pushed the cyrus/trigger-individual-resources branch 2 times, most recently from d78933b to 22f6d01 Compare December 6, 2024 19:23

ericpgreen2 reviewed Dec 9, 2024

View reviewed changes

scripts/tsc-with-whitelist.sh Outdated Show resolved Hide resolved

lovincyrus force-pushed the cyrus/trigger-individual-resources branch from d27ca47 to 792276e Compare December 10, 2024 02:11

lovincyrus requested a review from ericpgreen2 December 10, 2024 02:38

ericpgreen2 requested changes Dec 16, 2024

View reviewed changes

ericpgreen2 reviewed Dec 16, 2024

View reviewed changes

web-admin/src/features/projects/status/ProjectResources.svelte Outdated Show resolved Hide resolved

lovincyrus mentioned this pull request Dec 19, 2024

Refresh all resources confirm dialog #6309

Merged

ericpgreen2 requested a review from ericokuma January 17, 2025 20:14

ericpgreen2 assigned lovincyrus Jan 21, 2025

lovincyrus added 5 commits January 24, 2025 15:02

new cell to refresh source in status tab

2a8574f

move refresh sources to higher order

11378f3

wip

8babbc3

add refresh confirm dialog

263604a

wip, fixme

b878b9d

ericpgreen2 requested changes Jan 30, 2025

View reviewed changes

lovincyrus added 8 commits January 30, 2025 08:21

add refresh trigger to user facing resource kinds

0c85b01

add clear-all-notifications event

2e0c8cf

feedback

c00fcb3

lint

f660ac3

reset

8609468

make refresh call async

25527d4

just right

a73d269

backoff strat, error handling

08406c7

lovincyrus requested a review from ericpgreen2 January 30, 2025 23:10

ericpgreen2 previously requested changes Jan 31, 2025

View reviewed changes

web-admin/src/features/projects/status/ProjectResources.svelte Outdated Show resolved Hide resolved

web-admin/src/features/projects/status/ActionsCell.svelte Outdated Show resolved Hide resolved

lovincyrus added 3 commits January 31, 2025 09:07

move to pure function for refetch interval

6bbf94c

clean up, clean up, clean up

4321669

lint

78d9b08

lovincyrus requested review from ericpgreen2 and removed request for briangregoryholmes January 31, 2025 19:50

lovincyrus commented Feb 3, 2025

View reviewed changes

web-common/src/lib/event-bus/event-bus.ts Show resolved Hide resolved

lovincyrus added 2 commits February 3, 2025 14:31

reverts

b0811b8

Revert "reverts"

3ef4c92

This reverts commit b0811b8.

ericpgreen2 reviewed Feb 4, 2025

View reviewed changes

web-admin/src/features/projects/status/ProjectResources.svelte Show resolved Hide resolved

ericpgreen2 approved these changes Feb 4, 2025

View reviewed changes

lovincyrus added 3 commits February 4, 2025 08:16

update refetch interval

40f2da7

revalidate in refresh all

18e4b93

fix action cell sizing, compact

2a5f7d0

lovincyrus merged commit bc9f20b into main Feb 4, 2025
7 checks passed

lovincyrus deleted the cyrus/trigger-individual-resources branch February 4, 2025 17:15

lovincyrus mentioned this pull request Feb 7, 2025

Project Status: Ability to refresh individual sources/models #6154

Closed

ericpgreen2 mentioned this pull request Feb 12, 2025

Cloud UI: apply refetchInterval function to all resource tables #6667

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refresh button for source rows in Status page #6175

Refresh button for source rows in Status page #6175

lovincyrus commented Nov 26, 2024 •

edited

Loading

ericpgreen2 commented Nov 27, 2024 •

edited

Loading

lovincyrus commented Dec 4, 2024

ericpgreen2 left a comment

lovincyrus commented Dec 4, 2024

ericpgreen2 left a comment

lovincyrus commented Dec 10, 2024 •

edited

Loading

lovincyrus commented Dec 10, 2024

begelundmuller commented Dec 10, 2024 •

edited

Loading

ericpgreen2 left a comment

lovincyrus commented Dec 18, 2024

begelundmuller commented Dec 27, 2024 •

edited

Loading

ericokuma commented Jan 17, 2025

ericokuma commented Jan 22, 2025

ericpgreen2 left a comment •

edited

Loading

ericpgreen2 left a comment

Refresh button for source rows in Status page #6175

Refresh button for source rows in Status page #6175

Conversation

lovincyrus commented Nov 26, 2024 • edited Loading

ericpgreen2 commented Nov 27, 2024 • edited Loading

lovincyrus commented Dec 4, 2024

ericpgreen2 left a comment

Choose a reason for hiding this comment

lovincyrus commented Dec 4, 2024

ericpgreen2 left a comment

Choose a reason for hiding this comment

lovincyrus commented Dec 10, 2024 • edited Loading

lovincyrus commented Dec 10, 2024

begelundmuller commented Dec 10, 2024 • edited Loading

ericpgreen2 left a comment

Choose a reason for hiding this comment

lovincyrus commented Dec 18, 2024

begelundmuller commented Dec 27, 2024 • edited Loading

ericokuma commented Jan 17, 2025

ericokuma commented Jan 22, 2025

ericpgreen2 left a comment • edited Loading

Choose a reason for hiding this comment

ericpgreen2 left a comment

Choose a reason for hiding this comment

lovincyrus commented Nov 26, 2024 •

edited

Loading

ericpgreen2 commented Nov 27, 2024 •

edited

Loading

lovincyrus commented Dec 10, 2024 •

edited

Loading

begelundmuller commented Dec 10, 2024 •

edited

Loading

begelundmuller commented Dec 27, 2024 •

edited

Loading

ericpgreen2 left a comment •

edited

Loading