-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Data dispatcher implementation #13
Open
LTrestka
wants to merge
22
commits into
main
Choose a base branch
from
data_dispatcher_implementation
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…/password, vaulttokens (if stored on poms filesystem, and enabled enabled on metacat server), and existing metacat tokens. Data dispatcher sessions are acquired via context (experiment, role, user) and automatically logged in if a metacat token exists. The page has the ability to load and view rse's, projects, and project handles. All UI related stuff is scalable and will be separated into several Jinja2 templates for easy insertion into other pages as development progresses. I have also created separate files for all js and css related to data dispatcher. This will allow easy imports on pages where data_dispatcher views are prefered over our classic views.
…ermitools/poms into data_dispatcher_implementation Conflicts: webservice/DataDispatcherService.py webservice/poms_service.py
* Monitoring system in place * added several templates to quickly add dd ui to pages * gui editor includes dd related fields * database and model updated to support dd * added testing methods for tracking simulated dd progress live
…_dispatcher handling. Displays information about dd project submissions on relevant pages, generates queries for relevant files. Renamed DataDispatcherService to DMRService because it makes more sense. Fetches jobsub id from submission table if it exists (captured using standard submission method). DMRService added to cherrypy.request rather than as a ctx field. This makes it easier and faster to acquire. Standard login is as poms by an existing token, otherwise it automatically attempts to log in via x509 certs, which refreshes the token for 7 days.
…deals with the results. incorporated data dispatcher projects into submission agent. Need to create "update_submissions" function to handle multiple updates at once. Other things
submission agent now gets and updates files in bulk. May need to check for limits on this. Average seems to only come back with 10 results (with taskid)
Updated gui editor to support proper split type editors, and utilize the dd_split_types where applicable. Added limitn, drainingn, mod, and list split types for data_dispatcher Began working on new, multiparam, and stagedfiles split types. (stagedfiles may not be applicable to data dispatcher)
…lly set for certain split types and defaults to false. This allows us to send just a dataset rather than creating/using a dd project for each submission (gen case) Finished remaining split types. Still need to test and debug if necessary
… param launch, rather than project launch) Modifications to UI
Fixed submission agent to handle dd_task_id Added archived option in db for dd submissions (hides old dd_submissions, mostly for dev purposes) submission_details.html shows metacat query, and lets user copy it to clipboard resetting cs_last_split on split types that utilize the idx prevents any previous submissions from effecting dataset Stability improvements
… which are formatted when creating a snapshot
* Fixes duplicate dd_project retrieval error when launching split types * fixes "dataset" undefined error during submission. * Updated poms_client to use token auth (9443)
LTrestka
added
the
v5_0_0
Tasks to be include in the Upcoming production release for poms v5
label
Jan 23, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introducing a Massive update for POMS, which includes:
Fully integrated support for data dispatcher and metacat:
Experimenters can now create/manage their campaigns, using data dispatcher as the data handler instead of SAM.
Full support for recovery, dependency, and split-type campaign and campaign stage submissions.
An updated Submission broker for handling dd/mc submissions.
Real-time updates are available on data dispatcher projects once submitted.
Intuitive UI elements which allow experimenters to view details, file statistics, and real time updates on their submitted data dispatcher campaigns and stages.
Simplified controls allow experimenters to set either a project id, or an mql query as a campaign stage dataset
-All aspects of this update were designed with the experimenter in mind, experimenters can rest easy knowing that POMS behaves and feels exactly the same as it currently does with SAM - the only thing to do is tell POMS that you want to use data dispatcher instead of SAM, and enter your data dispatcher/metacat information in the appropriate configuration files.
Full Documentation will be available soon.