ACC-349 Bulk element operation tracking

I had some objections, but in thinking it through, if I just listen on Elements::EVENT_AFTER_BULK_OP then I can "do the thing" then because it will only be triggered at the end of the save (whether it's one element or thousands).

So I don't need to have an event before the bulk operation starts (though I suspect someone, somewhere may want an Elements::EVENT_BEFORE_BULK_OP too).

Obviously, this would only be possible when refactoring things for Craft 5, but it looks great to me, thanks!

(he says, in a vacuum, having never used it in the real world)

Yeah exactly. Keeping track of elements as things happen felt like the right approach for a couple reasons:

• That will include any nested elements that are modified as a result of the primary elements being saved, which wouldn’t have been known at the outset.
• If there are thousands, keeping a list of modified element IDs wouldn’t really be helpful, since querying for elements by thousands of IDs would be horrible for query performance (and possibly beyond resource limits).

So I don't need to have an event before the bulk operation starts (though I suspect someone, somewhere may want an Elements::EVENT_BEFORE_BULK_OP too).

Yeah maybe. Added it just in case. But in general I suspect ElementQuery::inBulkOp() will suffice.

I assume Craft is then going to move to use this for the resave elements stuff, propagation method changes, etc., and in FeedMe, and other bulk element operations ya?

Yes, everything in core already uses it in this PR, and we would start using it in Feed Me when porting it to Craft 5.

I wonder if this can be leveraged by the template {% cache %} mechanism, too, to coalesce the cache breaking that happens on element changes. Or maybe it already does in some manner?

Anyway, nifty, well done!

This looks great, will definitely be taking advantage of it in Blitz 5, thanks @brandonkelly and @khalwat for raising this!

Thanks for taking a look @khalwat and @bencroker! Merged 🚀

This is a great addition, something I can use in the v5 version of Scout too 👍

plugins could potentially stop listening to Elements::EVENT_AFTER_SAVE_ELEMENT/Element::EVENT_AFTER_SAVE altogether, and could go all-in on EVENT_AFTER_BULK_OP.

This sounds like a nice direction, but I'm confused on the event naming. It sounds like these events are going to be a wrapper for all element saves, not just bulk ones.

If I'm reading this right, this is Craft's way of saying:

• "I'm about to save some number of elements."
• "I'm done saving some number of elements; You can use this key to find the elements that changed."

Eh, I think I misunderstood. This is only for bulk operations. i.e. This functionality does not cover normal run-of-the-mill atomic element saves/deletes.

So then, is there a way for me to tell, from Elements::EVENT_AFTER_SAVE_ELEMENT or Elements::EVENT_AFTER_PROPAGATE, whether an individual save/propagate action is part of a bulk operation?

Also, curious how this would work in multi-site use cases...

Currently, we can listen for Element::EVENT_AFTER_PROPAGATE to do third-party stuff once an element is fully saved and propagated to other sites. Importantly, we can tell which site is relevant for this operation, because the initiating element is accessible directly from the event.

By contrast, these new events don't have the actual elements attached; Instead, they carry a key for querying those elements after-the-fact. We presumably need that key to facilitate querying all the site-elements that changed during an operation. Capturing element ID alone would not be sufficient... right?

Keeping track of elements as things happen felt like the right approach for a couple reasons:

• That will include any nested elements that are modified as a result of the primary elements being saved, which wouldn’t have been known at the outset.

This feels like it might get tricky. A batch operation might save elements of many types, some of which may be third-party... So, how do I know, listening to Elements::EVENT_AFTER_BULK_OP, what types of elements changed?

For example:

Event::on(Elements::class, Elements::EVENT_AFTER_BULK_OP, function(BulkOpEvent $event) {
  // Fetch all the entries that were affected
  $entries = Entry::find()
    ->inBulkOp($event->key)
    ->status(null)
    ->all();
});

How do I know, from the event, that I need to build an Entry query here?

Eh, I think I misunderstood. This is only for bulk operations. i.e. This functionality does not cover normal run-of-the-mill atomic element saves/deletes.

@michaelrog Your first take was correct. As I said:

All built-in element (multi-)save and deletion actions will ensure that a bulk operation is active, even if only impacting a single element. So plugins could potentially stop listening to Elements::EVENT_AFTER_SAVE_ELEMENT/Element::EVENT_AFTER_SAVE altogether, and could go all-in on EVENT_AFTER_BULK_OP.

By contrast, these new events don't have the actual elements attached; Instead, they carry a key for querying those elements after-the-fact. We presumably need that key to facilitate querying all the site-elements that changed during an operation. Capturing element ID alone would not be sufficient... right?

Correct, this is only concerned with which elements were updated, regardless of site. I considered tracking the site, but decided against it since most element saves will propagate across all sites anyway.

How do I know, from the event, that I need to build an Entry query here?

You would just need to check each individual element type you care about. Or continue listening for EVENT_AFTER_ELEMENT_SAVE and track which element types are being touched from there – but considering you would need to do that potentially across multiple requests (in the case of batched queue jobs), that may be more trouble than it’s worth.

Your first take was correct.

In that case, naming these as "bulk" does feel a bit too specific... It feels conceptually weird that every element save/delete action is definitionally a bulk action, whether or not multiple elements are involved. This feels more analogous to "transactions" and I wish I could think of term that felt more precise, to make documentation/teaching about this a bit less cumbersome.

...

I see where bulk queue jobs are amended to beginBulkOp() and endBulkOp(), but I don't see where this plays into ordinary element saving. i.e. Where is the service actually tracking the individual elements that change?

Edit: I am a silly goose, and Large diffs are not rendered by default.

...

You would just need to check each individual element type you care about.

The challenge is, I may not know ahead-of-time which element types I care about, especially if 3p plugins are involved. So then do I have to check every possible element type?

I guess I could try to query the elements_bulkops table using the key, join it into the elements table to get type, and select distinct types from that. That seems like a hoop to jump... but I guess I don't have a good suggestion for an alternative... 🤔

So if we're hung up on naming... transaction might work? Sort of mirroring Yii2's yii\db\Connection::beginTransaction() / yii\db\Connection::endTransaction() so instead we'd have:

• Events - Elements::EVENT_BEFORE_ELEMENT_TRANSACTION & Elements::EVENT_AFTER_ELEMENT_TRANSACTION
• Methods - $transactionKey = Craft::$app->elements->beginElementTransaction(); & Craft::$app->elements->endElementTransaction($transactionKey);
• Query param - ->inElementTransaction($event->key)

It probably more accurately describes what's going on this way... but honestly, the naming doesn't bother me terribly, as long as the functionality is there.

Trust me, 90% of the time I spent on this change was trying to think of a better name for it than “bulk operation”.

“Transaction” is definitely the closest thing conceptually. But I don’t want to overload that word.

It occurs to me that deleted elements may be difficult to work with in this context:

Soft-deleted elements can still be queried using inBulkOp()... But hard-deleted elements aren't registered at all. (The elementId fk prevents them from existing in elements_bulkops in the first place.)

e.g. If I hard-delete a bunch of entries, and now need to remove those entries from an external search index, there's no native mechanism for me to batch that, right?

(I could see this being especially frustrating since Delete For Site behavior for multi-site entries is inconsistent between hard-/soft-delete.)

So it seems like the best practice will still be to listen to EVENT_AFTER_DELETE_ELEMENT for elements being deleted, rather than trying to handle them in bulk... Am I understanding this correctly?

@michaelrog yeah, no way to keep track of hard deletions after-the-fact, including site-specific hard-deletions, so if you care about those, you would still want to listen to EVENT_AFTER_DELETE and EVENT_AFTER_DELETE_FOR_SITE.