Summary

I'm porting a big project from CKAN 2.8 to 2.9. My plugin overrides a template from ckanext-scheming to customize the form. After upgrade, my changes weren't reflected because my custom template wasn't loaded.
Only after changing the ordering of the plugins in ckan.plugins it was picked up.

So on CKAN <= 2.8, in order for plugin abc to override scheming_datasets you needed:

ckan.plugins = abc scheming_datasets 

In CKAN 2.9, you need:

ckan.plugins = scheming_datasets abc

Why it is important

This is pretty significant change, which AFAICT wasn't mentioned in the changelog.

After initial investigation it looks like the issue is not how we parse the config option or load the plugins, but how the PluginImplementations iterator returns them. We use them in all places where we let plugins integrate with CKAN core. For instance in environment.py we call:

      for plugin in p.PluginImplementations(p.IConfigurer):                                                                                                                                                                                                                          
          plugin.update_config(config) 

This is one is relevant to my issue, as it registers template directories from plugins and stores them on a list in config['extra_template_paths']. Order is important, as the first template path found will be used to render.

At this point we get the following behaviour:

• On CKAN 2.8:

[plugin for plugin in p.PluginImplementations(p.IConfigurer)]

# [<Plugin AbcPlugin 'abc'>, <Plugin SchemingDatasetsPlugin 'scheming_datasets'>]

config['extra_template_paths'].split(',')

# [
#  u'/home/adria/dev/pyenvs/ckan/src/ckanext-abc/ckanext/abc/templates',
#  u'/home/adria/dev/pyenvs/ckan/src/ckanext-scheming/ckanext/scheming/templates',
# ]

• On CKAN 2.9:

[plugin for plugin in p.PluginImplementations(p.IConfigurer)]

# [<Plugin SchemingDatasetsPlugin 'scheming_datasets'>, <Plugin AbcPlugin 'abc'>]

config['extra_template_paths'].split(',')

# [
#  u'/home/adria/dev/pyenvs/ckan/src/ckanext-scheming/ckanext/scheming/templates',
#  u'/home/adria/dev/pyenvs/ckan/src/ckanext-abc/ckanext/abc/templates',
# ]

Apart from template loading issues this is likely to affect everywhere where the order of plugins is important, eg chained actions, chained auth functions.

Root cause

After looking at ckan/plugins/core.py my current thinking is that this is not related to the loading of the plugins. AFAICT we´ve always loaded them in the order that they are defined in ckan.plugins. It´s the actual iterator returned by PluginImplementations that changed the order of the returned plugins at some point between the two versions (pyutilib.component.core==4.6.4 in CKAN 2.8, PyUtilib==5.7.1 in CKAN 2.9). We are importing this class directly from Pyutillib. The only work done on this code between these two versions was #4886, and I don´t think it should affect the ordering (apart from upgrading the library of course)

What should we do?

My ideas so far:

1. Change nothing and assume this is the new behaviour, but documenting it in the relevant places (2.9 Changelog, plugins docs, mail to ckan-dev). I don´t think we can leave a change like this undocumented
2. Create our own PluginImplementations wrapper that restores the old ordering (maybe optionally based on a config option). We would need to override the __iter__() method, not sure how easy that is

Any thoughts or other ideas on what to do? @ckan/core