Introducing Job Hooks in Neosync

Introduction

We're excited to announce support for pre and post job sync hooks in Neosync! This new feature gives you more control and flexibility over your data syncing workflows by allowing you to execute custom SQL actions before and after a sync job runs.

What are Pre and Post Sync Hooks?

Pre and post sync hooks are custom SQL actions that you can configure to run automatically at specific points in your job's lifecycle:

• Pre-sync hooks run before the main sync job starts. This includes table truncation and schema initialization. Thus letting you prep your source or destination databases
• Post-sync hooks run after the sync completes or after the last table syncs, enabling cleanup tasks or data normalization

These hooks give you a way to automate setup and cleanup tasks that previously had to be done manually or through separate automation.

Why did we build this?

Many of our customers have workflows that require some preparation before syncing data or cleanup afterward. For example:

• Truncating specific tables before syncing instead of truncating all tables
• Running schema migrations
• Normalizing data after it lands in the destination
• Validating data integrity
• Cleaning up temporary tables or artifacts

Rather than requiring separate automation scripts or manual intervention, we wanted to make it easy to configure these common tasks right within your Neosync jobs.

Use Cases

Here are some specific examples of how you can use pre and post sync hooks:

Pre-sync hooks:

• Run schema migrations to prep destination database
• Clear out staging tables before fresh sync
• Set up temporary tables or indexes
• Validate source data meets requirements
• Enable/disable triggers or foreign key constraints

Post-sync hooks:

• Data integrity checks and validation
• Cleanup of temporary objects
• Re-enable constraints or triggers
• Generate reports or metrics
• Run data transformations or normalization

How to Use Hooks

Configuring hooks is straightforward. When creating or editing a job, you'll see new sections for pre and post sync hooks. For each hook you can specify:

• SQL commands to run
• The database connection to use
• Order of execution if you have multiple hooks
• Whether the job should continue if the hook fails

Here's a simple example of configuring a pre-sync hook to truncate a staging table:

TRUNCATE TABLE staging.customer_data CASCADE;

And a post-sync hook to validate row counts:

SELECT COUNT(*) FROM production.users =
(SELECT COUNT(*) FROM staging.users);

Best Practices

As you start using hooks, here are some recommendations:

1. Keep hooks focused - Each hook should do one specific task for clarity
2. Test thoroughly - Validate hooks in development before production
3. Document purpose - Add comments explaining what each hook does
4. Monitor execution - Check hook logs to ensure they're running as expected

What's Next?

This is just the beginning for hooks in Neosync. We're already working on additional capabilities like:

• Support for more complex scripting
• Integration with external tools and services
• Enhanced logging and monitoring
• More granular control over hook execution
• More timing options
• Additional hook types just as REST API calls

Try It Out

Check out our documentation to learn more about configuring hooks for your jobs and try it in our free trial.

We're excited to see how you'll use hooks to automate your data workflows! Let us know what you build or if you have feature requests by joining our Discord.

The power of hooks is that they let you automate entire workflows end-to-end within Neosync. No more cobbling together multiple tools or manual steps - you can define everything in one place. This makes your data pipelines more reliable and easier to maintain.

Let us know what you think!