Skip to content

feat: Postgres intelligent updates #18643

@duncanmapes

Description

@duncanmapes

Which problem is this feature request solving?

I have workflows that pickup data that goes into a postgres database (using the destination plugin).

But I currently don't have a way to know what rows have changed. The postgres plugin updates the "cq-sync" time whether the actual values changed or not. So I have to process EVERY row against downstream data sources to see if there is a delta. It's very wasteful and slow.

Describe the solution you'd like

A way to know the true "last updated" time for a row. Not the last time a sync was run.

This could take the form of upsert logic doesn't execute an insert if the values haven't changed (obviously excluding things like cq-sync), or an additional field like "cq-updated".

It might also be nice to configure this on a table by table basis for fine tuning.

Pull request (optional)

  • I can submit a pull request

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions