feat(cli) Changes rollback behaviour to apply soft deletes by default #4358
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
metadata-io/src/main/java/com/linkedin/metadata/entity/ebean/EbeanEntityService.java
Outdated
Show resolved
Hide resolved
metadata-io/src/main/java/com/linkedin/metadata/entity/EntityService.java
Outdated
Show resolved
Hide resolved
metadata-io/src/main/java/com/linkedin/metadata/entity/ebean/EbeanAspectDao.java
Show resolved
Hide resolved
metadata-io/src/main/java/com/linkedin/metadata/entity/ebean/EbeanEntityService.java
Outdated
Show resolved
Hide resolved
metadata-io/src/main/java/com/linkedin/metadata/systemmetadata/ESSystemMetadataDAO.java
Outdated
Show resolved
Hide resolved
...o/src/main/java/com/linkedin/metadata/systemmetadata/ElasticSearchSystemMetadataService.java
Show resolved
Hide resolved
Summary: Addresses feature request: Flag in delete command to only delete aspects touched by an ingestion run; add flag to nuke everything by modifying the default behaviour of a rollback operation which will not by default delete an entity if a keyAspect is being rolled-back. Instead the key aspect is kept and a StatusAspect is upserted with removed=true, effectively making a soft delete. Another PR will follow to perform garbage collection on these soft deleted entities. To keep old behaviour, a new parameter to the cli ingest rollback endpoint: --hard-delete was added.
4 tasks
dexter-mh-lee
approved these changes
Mar 15, 2022
maggiehays
pushed a commit
to maggiehays/datahub
that referenced
this pull request
Aug 1, 2022
…datahub-project#4358) * Changes rollback behaviour to apply soft deletes by default Summary: Addresses feature request: Flag in delete command to only delete aspects touched by an ingestion run; add flag to nuke everything by modifying the default behaviour of a rollback operation which will not by default delete an entity if a keyAspect is being rolled-back. Instead the key aspect is kept and a StatusAspect is upserted with removed=true, effectively making a soft delete. Another PR will follow to perform garbage collection on these soft deleted entities. To keep old behaviour, a new parameter to the cli ingest rollback endpoint: --hard-delete was added. * Adds restli specs * Fixes deleteAspect endpoint & adds support for nested transactions * Enable regression test & fix docker-compose for local development * Add generated quickstart * Fix quickstart generation script * Adds missing var env to docker-compose-without-neo4j * Sets status removed=true when ingesting resources * Adds soft deletes for ElasticSearch + soft delete flags across ingestion sub-commands * Makes elastic search consistent * Update tests with new behaviour * apply review comments * apply review comment * Forces Elastic search to add documents with status removed false when ingesting * Reset gradle properties to default * Fix tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Addresses feature request: Flag in delete command to only delete aspects touched by an ingestion run; add flag to nuke everything.
Adresses feature request by modifying the default behaviour of a rollback operation.
By default rollback will not delete an entity if a keyAspect is being rolled-back.
Instead, the key aspect is kept and a StatusAspect is upserted with
removed=true, effectively implementing a soft delete.To make rollback consistent & symmetric, this PR also adds a Status Aspect with
removed=falsewhenever ingesting an Entity if status is not present.Another PR will follow to perform garbage collection on these soft deleted entities.
To keep old behaviour, a new parameter to the cli ingest rollback endpoint: --hard-delete was added.
To keep cli commands like
list-runsconsistent, this PR also modifies MCL processor'sUpdateIndiceshook to upset Elasticsearch documents with a new propertyremoved.This property is used to indicate whether a given elastic document has been soft deleted or not so that we may exclude soft deleted aspects from ingestion run history.
Checklist