Skip to content

fix(restore indices): Add RESTATE ChangeType to MCL / MCP to permit restore indices #5022

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shirshanka merged 3 commits into from
May 30, 2022
Merged

fix(restore indices): Add RESTATE ChangeType to MCL / MCP to permit restore indices #5022

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
98875d1
Finalizing restore indices operation
jjoyce0510 May 26, 2022
5ffa3e5
Update comment
jjoyce0510 May 26, 2022
346882c
Renaming restore to restate
jjoyce0510 May 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion datahub-upgrade/src/main/java/com/linkedin/datahub/upgrade/restoreindices/SendMAEStep.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ public Function<UpgradeContext, UpgradeStepResult> executable() {
_entityService.produceMetadataChangeLog(urn, entityName, aspectName, aspectSpec, null, aspectRecord, null,
latestSystemMetadata,
new AuditStamp().setActor(UrnUtils.getUrn(SYSTEM_ACTOR)).setTime(System.currentTimeMillis()),
ChangeType.UPSERT);
ChangeType.RESTATE);

totalRowsMigrated++;
}
Expand Down
2 changes: 1 addition & 1 deletion ...a-jobs/mae-consumer/src/main/java/com/linkedin/metadata/kafka/hook/UpdateIndicesHook.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ public void invoke(@Nonnull MetadataChangeLog event) {
}
Urn urn = EntityKeyUtils.getUrnFromLog(event, entitySpec.getKeyAspectSpec());

if (event.getChangeType() == ChangeType.UPSERT) {
if (event.getChangeType() == ChangeType.UPSERT || event.getChangeType() == ChangeType.RESTATE) {

if (!event.hasAspectName() || !event.hasAspect()) {
log.error("Aspect or aspect name is missing");
Expand Down
10 changes: 9 additions & 1 deletion .../src/main/java/com/linkedin/metadata/kafka/hook/event/EntityChangeEventGeneratorHook.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,14 @@ public class EntityChangeEventGeneratorHook implements MetadataChangeLogHook {
Constants.TAG_KEY_ASPECT_NAME,
Constants.STATUS_ASPECT_NAME
);
/**
* The list of change types that are supported for generating semantic change events.
*/
private static final Set<String> SUPPORTED_OPERATIONS = ImmutableSet.of(
"CREATE",
"UPSERT",
"DELETE"
);
private final AspectDifferRegistry _aspectDifferRegistry;
private final EntityClient _entityClient;
private final Authentication _systemAuthentication;
Expand Down Expand Up @@ -167,7 +175,7 @@ private <T extends RecordTemplate> List<ChangeEvent> generateChangeEvents(
}

private boolean isEligibleForProcessing(final MetadataChangeLog log) {
return SUPPORTED_ASPECT_NAMES.contains(log.getAspectName());
return SUPPORTED_OPERATIONS.contains(log.getChangeType().toString()) && SUPPORTED_ASPECT_NAMES.contains(log.getAspectName());
}

private void emitPlatformEvent(@Nonnull final PlatformEvent event, @Nonnull final String partitioningKey) throws Exception {
Expand Down
5 changes: 5 additions & 0 deletions metadata-models/src/main/pegasus/com/linkedin/events/metadata/ChangeType.pdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,4 +32,9 @@ enum ChangeType {
* patch the changes instead of full replace
*/
PATCH

/**
* Restate an aspect, eg. in a index refresh.
*/
RESTATE
}
3 changes: 2 additions & 1 deletion metadata-service/restli-api/src/main/snapshot/com.linkedin.entity.aspects.snapshot.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1679,11 +1679,12 @@
"name" : "ChangeType",
"namespace" : "com.linkedin.events.metadata",
"doc" : "Descriptor for a change action",
"symbols" : [ "UPSERT", "CREATE", "UPDATE", "DELETE", "PATCH" ],
"symbols" : [ "UPSERT", "CREATE", "UPDATE", "DELETE", "PATCH", "RESTATE" ],
"symbolDocs" : {
"CREATE" : "NOT SUPPORTED YET\ninsert if not exists. otherwise fail",
"DELETE" : "NOT SUPPORTED YET\ndelete action",
"PATCH" : "NOT SUPPORTED YET\npatch the changes instead of full replace",
"RESTATE" : "Restate an aspect, eg. in a index refresh.",
"UPDATE" : "NOT SUPPORTED YET\nupdate if exists. otherwise fail",
"UPSERT" : "insert if not exists. otherwise update"
}
Expand Down