Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for cloud tasks within the emulator. #7475

Merged
merged 32 commits into from
Aug 27, 2024

Conversation

GarrettBurroughs
Copy link
Contributor

Support for cloud tasks within the emulator

src/emulator/functionsEmulatorShared.ts Show resolved Hide resolved
src/emulator/taskQueue.ts Show resolved Hide resolved
src/emulator/taskQueue.ts Show resolved Hide resolved
src/emulator/taskQueue.ts Outdated Show resolved Hide resolved
src/emulator/tasksEmulator.ts Outdated Show resolved Hide resolved
src/emulator/tasksEmulator.ts Outdated Show resolved Hide resolved
src/emulator/tasksEmulator.ts Outdated Show resolved Hide resolved
src/emulator/tasksEmulator.ts Outdated Show resolved Hide resolved
src/emulator/tasksEmulator.ts Outdated Show resolved Hide resolved
src/emulator/tasksEmulator.ts Fixed Show fixed Hide fixed
src/emulator/tasksEmulator.ts Fixed Show fixed Hide fixed
src/emulator/taskQueue.ts Dismissed Show resolved Hide resolved
src/emulator/taskQueue.ts Show resolved Hide resolved
src/emulator/taskQueue.ts Dismissed Show resolved Hide resolved
src/emulator/taskQueue.ts Outdated Show resolved Hide resolved
src/emulator/tasksEmulator.spec.ts Outdated Show resolved Hide resolved
defaultUri: string;

// The configurations below this point do not currently have any effect on how the task queues are handled within the emulator
secrets?: string[]; // TODO(gburroughs): look into how we can handle this
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Secrets should be handled by the functions emulator at startup, but it would be good to confirm this.

src/emulator/tasksEmulator.ts Show resolved Hide resolved
src/emulator/tasksEmulator.ts Show resolved Hide resolved
src/emulator/tasksEmulator.ts Fixed Show fixed Hide fixed
src/emulator/tasksEmulator.ts Fixed Show fixed Hide fixed
Comment on lines +596 to +602
const tasksAddr = legacyGetFirstAddr(Emulators.TASKS);
const tasksEmulator = new TasksEmulator({
host: tasksAddr.host,
port: tasksAddr.port,
});

await startEmulator(tasksEmulator);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: does this mean we always run the task emulator whenever we run the functions emulator?

I think this is fine for now, but I think an ideal solution would run the task emulator only if a task queue function is defined.

I don't know whether something like that is possible, but if you have time would love for you to see if that's even feasible.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the task queue emulator is run whenever the functions emulator is run.

There is the code here:

if (!this.controller.isRunning()) {

which only starts the listening logic of the task queue emulator once an actual task has been enqueued to avoid doing unnecessary updates while the user isn't doing anything with task queues in their functions code. The express server is still spun up every time the functions emulator starts though.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm I wonder if we should instead have Task Queue Emulators be explicitly started, just like the Pub/sub emulator.

@blidd-google thoughts?

src/emulator/tasksEmulator.ts Show resolved Hide resolved
src/emulator/functionsEmulator.ts Outdated Show resolved Hide resolved
src/emulator/functionsEmulator.ts Outdated Show resolved Hide resolved
src/emulator/tasksEmulator.ts Outdated Show resolved Hide resolved
src/emulator/taskQueue.ts Outdated Show resolved Hide resolved
src/emulator/taskQueue.ts Show resolved Hide resolved
src/emulator/taskQueue.ts Outdated Show resolved Hide resolved
src/emulator/tasksEmulator.ts Show resolved Hide resolved
src/emulator/taskQueue.ts Outdated Show resolved Hide resolved
@GarrettBurroughs GarrettBurroughs marked this pull request as ready for review August 14, 2024 20:40
@blidd-google blidd-google enabled auto-merge (squash) August 27, 2024 17:32
@joehan joehan disabled auto-merge August 27, 2024 20:40
@joehan joehan merged commit 2b1a314 into master Aug 27, 2024
40 of 41 checks passed
@joehan
Copy link
Contributor

joehan commented Aug 27, 2024

Overriding the CLA check here as Garret worked on this during his time at Google.

@joehan joehan deleted the gburroughs-taskqueue-emulator branch August 27, 2024 21:02
hlshen pushed a commit that referenced this pull request Sep 3, 2024
* register cloud task triggers

* Create and register task queue emulator

* Add basic features of task queue emulator

* Implement a queue to improve performance of tasks emulator

* Remove unused import

* Register task queues from functions emulator

* change task queue endpoint to accept IDs in body instead of query params

* set enviornment variable when task queue emulator is running

* Remove bug where tasks were getting dispatched too fast

* Rework task enqueueing API and support task deletion

* properly support task deletion

* Refactor Task Dispatching Algorithm

* Add task deduplication for all tasks that have ever been added

* Refactor Task Queue

* Add tests for task queue emulator

* respond to PR review

* update firebase-config.json

* use node-fetch

* fix tests

* Make tasks emulator start up whenever functions emulator starts

* fix error message

* Add statistics endpoint

* Add in global task deduplication

* Add headers to request

* Review Changes

* Remove console logs

* Update backoff calculations

* add changelog

---------

Co-authored-by: blidd-google <[email protected]>
Co-authored-by: joehan <[email protected]>
Co-authored-by: Brian Li <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants