Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

template_tags error seen for DUNE #42

Closed
dougbenjamin opened this issue Oct 22, 2024 · 12 comments
Closed

template_tags error seen for DUNE #42

dougbenjamin opened this issue Oct 22, 2024 · 12 comments

Comments

@dougbenjamin
Copy link

Hi,

See this error below - (template_tags function not defined in dune.py file)

10/21/2024 22:33:27.629: MoverTask[fardet-vd_fd_mc_2023a_reco2_reco2_dunevd10kt_nu_1x8x6_3view_30deg_geov3.fcl_reco2_v09_81_00d02_merged_skip000000_lim000020_stage_makecaf_20241009T023641.root]: data file already exists at the destination and has correct size. Not overwriting
10/21/2024 22:33:27.641: Mover:
Mover failed: hd-protodune_detector_run0000028005_physics_standard_reco_stage2_calibration_protodunehd_keepup_root-tuple_merged_skip000050_lim000014_testing_20241009T023519.root status: queued error:
Traceback (most recent call last):
File "/exp/dune/app/home/dunepro/venvs/declad/lib64/python3.11/site-packages/pythreader/task_queue.py", line 210, in run
result = task.run()
^^^^^^^^^^
File "/exp/dune/app/home/dunepro/2x2_Minerva/declad_merging/declad/declad/mover.py", line 389, in run
metacat_dataset_did = custom.metacat_dataset( self.FileDesc, metadata, self.Config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/exp/dune/app/home/dunepro/2x2_Minerva/declad_merging/declad/declad/custom/init.py", line 163, in metacat_dataset
return rucio_dataset(desc, metadata, config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/exp/dune/app/home/dunepro/2x2_Minerva/declad_merging/declad/declad/custom/init.py", line 155, in rucio_dataset
meta.update(template_tags(metadata))
^^^^^^^^^^^^^
NameError: name 'template_tags' is not defined

@marcmengel
Copy link
Collaborator

Closing this as fixed in #44 , which was mistagged as fixing #41...

@dougbenjamin
Copy link
Author

After fetching latest tag 2.3.1

dunepro@dunedecladgpvm02 declad]$ git status
HEAD detached at 2.3.1
Untracked files:
(use "git add ..." to include in what will be committed)
declad/custom/init.py

nothing added to commit but untracked files present (use "git add" to track)

10/22/2024 13:06:26.596: Mover:
Mover failed: fardet-vd_fd_mc_2023a_reco2_reco2_dunevd10kt_nu_1x8x6_3view_30deg_geov3.fcl_reco2_v09_81_00d02_merged_skip000020_lim000020_stage_makecaf_20241009T024029.root status: downloading metadata error:
Traceback (most recent call last):
File "/exp/dune/app/home/dunepro/venvs/declad/lib64/python3.11/site-packages/pythreader/task_queue.py", line 210, in run
result = task.run()
^^^^^^^^^^
File "/exp/dune/app/home/dunepro/2x2_Minerva/declad_merging/declad/declad/mover.py", line 399, in run
metacat_dataset_did = custom.metacat_dataset( self.FileDesc, metadata, self.Config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/exp/dune/app/home/dunepro/2x2_Minerva/declad_merging/declad/declad/custom/init.py", line 163, in metacat_dataset
return rucio_dataset(desc, metadata, config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/exp/dune/app/home/dunepro/2x2_Minerva/declad_merging/declad/declad/custom/init.py", line 155, in rucio_dataset
meta.update(template_tags(metadata))
^^^^^^^^^^^^^
NameError: name 'template_tags' is not defined

Should this routine be added to dune.py in custom area -

        def template_tags(metadata):
            return {}

adding the dummy routine to dune.py appears to have fixed the issue.

@marcmengel
Copy link
Collaborator

That should work but you're not supposed to have to add that; it should have defined it that way if the import didn't work...

@dougbenjamin
Copy link
Author

but it did not. so ... how do we get the import to work.

here is the Run script for starting declad

#!/usr/bin/env bash
#Note this script is located same directory as declad.py
. $HOME/venvs/declad/bin/activate
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
cd $SCRIPT_DIR
source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh
#setup rucio
#setup metacat
setup sam_web_client
export SAM_EXPERIMENT=dune
export METACAT_SERVER_URL=https://metacat.fnal.gov:9443/dune_meta_prod/app
export METACAT_AUTH_SERVER_URL=https://metacat.fnal.gov:8143/auth/dune
export PATH="/dune/app/home/dunepro/miniconda3/bin:$PATH"
env | sort
which python3
python3 -c "import sys; print('\n'.join(sys.path))"
exec python3 -u product/declad.py -d -c config.yaml -p declad.pid $@

@dougbenjamin
Copy link
Author

I am likely doing something wrong but can not see it. -
here is PYTHONPATH -
PYTHONPATH=/cvmfs/fermilab.opensciencegrid.org/products/common/db//../prd/sam_web_client/v3_6/NULL/python

@marcmengel
Copy link
Collaborator

marcmengel commented Oct 22, 2024 via email

@dougbenjamin
Copy link
Author

yes hence the hack

@marcmengel
Copy link
Collaborator

Aaargh.. I'm not reading the stack trace right; you're getting that failure in "/exp/dune/app/home/dunepro/2x2_Minerva/declad_merging/declad/declad/custom/init.py"
which is the symlink to the custom/dune.py ...

@marcmengel
Copy link
Collaborator

Yes we need to add one there in the custom module, if we're calling it in the custom module... sigh.

@marcmengel
Copy link
Collaborator

It's the mover.py code that's protected against it not being in the custom module; but when I moved that rucio_dataset() code into the custom module, it then needs to be in there... Sigh.

@marcmengel
Copy link
Collaborator

Tagged 2.3.2

@dougbenjamin
Copy link
Author

It works thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants