Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data dispatcher implementation #13

Open
wants to merge 22 commits into
base: main
Choose a base branch
from

Conversation

LTrestka
Copy link
Collaborator

@LTrestka LTrestka commented Sep 2, 2023

Introducing a Massive update for POMS, which includes:

Fully integrated support for data dispatcher and metacat:

  • Experimenters can now create/manage their campaigns, using data dispatcher as the data handler instead of SAM.

  • Full support for recovery, dependency, and split-type campaign and campaign stage submissions.

  • An updated Submission broker for handling dd/mc submissions.

    • Optimized queries also enable much faster performance, this makes submission updates available significantly faster than before, this holds true for SAM submissions too.
  • Real-time updates are available on data dispatcher projects once submitted.

  • Intuitive UI elements which allow experimenters to view details, file statistics, and real time updates on their submitted data dispatcher campaigns and stages.

    • A new data dispatcher overview page, which shows all data dispatcher projects available within a given experiment (filtered by role)...AND:
    • Project details are available within a universal window on each relevant page, which displays only the relevant projects submissions for that page
      • ie:
        • campaign details shows all submissions within that campaign
        • campaign stage details shows submissions within that campaign stage
        • etc..
    • Output files link to metacat server for detailed file information.
    • In the event that many submissions are made, a Search feature is available within the UI. This can help users search by project id, owner, project status, files, and more.
    • Full history of submissions is available, experimenters may also relaunch any existing project (by setting 'data_dispatcher_project_id' in the campaign stage editor , a quick launch feature may be added in the future)
  • Simplified controls allow experimenters to set either a project id, or an mql query as a campaign stage dataset

    • This allows for highly customizable submissions
    • May make certain split types obsolete
    • Enables a quick turnaround time, without the need for creating entirely new campaign stages.

-All aspects of this update were designed with the experimenter in mind, experimenters can rest easy knowing that POMS behaves and feels exactly the same as it currently does with SAM - the only thing to do is tell POMS that you want to use data dispatcher instead of SAM, and enter your data dispatcher/metacat information in the appropriate configuration files.

Full Documentation will be available soon.

LTrestka and others added 22 commits May 30, 2023 15:53
…/password, vaulttokens (if stored on poms filesystem, and enabled enabled on metacat server), and existing metacat tokens.

Data dispatcher sessions are acquired via context (experiment, role, user) and automatically logged in if a metacat token exists.

The page has the ability to load and view rse's, projects, and project handles.

All UI related stuff is scalable and will be separated into several Jinja2 templates for easy insertion into other pages as development progresses. I have also created separate files for all js and css related to data dispatcher. This will allow easy imports on pages where data_dispatcher views are prefered over our classic views.
…ermitools/poms into data_dispatcher_implementation

Conflicts:
	webservice/DataDispatcherService.py
	webservice/poms_service.py
* Monitoring system in place
* added several templates to quickly add dd ui to pages
* gui editor includes dd related fields
* database and model updated to support dd
* added testing methods for tracking simulated dd progress live
…_dispatcher handling.

Displays information about dd project submissions on relevant pages, generates queries for relevant files.

Renamed DataDispatcherService to DMRService because it makes more sense.

Fetches jobsub id from submission table if it exists (captured using standard submission method).

DMRService added to cherrypy.request rather than as a ctx field. This makes it easier and faster to acquire.

Standard login is as poms by an existing token, otherwise it automatically attempts to log in via x509 certs, which refreshes the token for 7 days.
…deals with the results.

incorporated data dispatcher projects into submission agent.

Need to create "update_submissions" function to handle multiple updates at once.

Other things
submission agent now gets and updates files in bulk. May need to check for limits on this.
Average seems to only come back with 10 results (with taskid)
Updated gui editor to support proper split type editors, and utilize the dd_split_types where applicable.

Added limitn, drainingn, mod, and list split types for data_dispatcher

Began working on new, multiparam, and stagedfiles split types. (stagedfiles may not be applicable to data dispatcher)
…lly set for certain split types and defaults to false.

This allows us to send just a dataset rather than creating/using a dd project for each submission (gen case)

Finished remaining split types. Still need to test and debug if necessary
… param launch, rather than project launch)

Modifications to UI
Fixed submission agent to handle dd_task_id
Added archived option in db for dd submissions (hides old dd_submissions, mostly for dev purposes)
submission_details.html shows metacat query, and lets user copy it to clipboard

resetting cs_last_split on split types that utilize the idx
prevents any previous submissions from effecting dataset

Stability improvements
… which are formatted when creating a snapshot
* Fixes duplicate dd_project retrieval error when launching split types
* fixes "dataset" undefined error during submission.
* Updated poms_client to use token auth (9443)
@LTrestka LTrestka self-assigned this Jan 23, 2024
@LTrestka LTrestka added the v5_0_0 Tasks to be include in the Upcoming production release for poms v5 label Jan 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
v5_0_0 Tasks to be include in the Upcoming production release for poms v5
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant