Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Confluent CLI migration #91

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions docs/confluent-cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,15 @@ This tool was designed to work with Confluent Cloud. It can manage service accou

Ensure you have installed `kafka-gitops` or are using the `kafka-gitops` docker image as described in the [installation][installation] instructions.

You must have the `ccloud` command line tools installed if you wish to auto-populate the `principal` fields on services.
You must have the `confluent` command line tool installed if you wish to auto-populate the `principal` fields on services.

## Desired State File

Create a basic desired state file, `state.yaml`, such as:

```yaml
settings:
ccloud:
confluent:
enabled: true

topics:
Expand Down Expand Up @@ -46,18 +46,18 @@ To use `kafka-gitops` with Confluent Cloud, you'll need to set a few environment
* `KAFKA_SASL_MECHANISM`: `PLAIN`
* `KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM`: `HTTPS`

Additionally, you'll need to login to the `ccloud` tool. You can automate this by setting the following environment variables:
Additionally, you'll need to login to the `confluent` tool. You can automate this by setting the following environment variables:

* `XX_CCLOUD_EMAIL`: Your Confluent Cloud administrator email
* `XX_CCLOUD_PASSWORD`: Your Confluent Cloud administrator password
* `CONFLUENT_CLOUD_EMAIL`: Your Confluent Cloud administrator email
* `CONFLUENT_CLOUD_PASSWORD`: Your Confluent Cloud administrator password

Then, you can run `ccloud login` and it will run without a prompt. This is great for CI builds.
Then, you can run `confluent login` and it will run without a prompt. This is great for CI builds.

You can optionally specify a path to a `ccloud` executable:
You can optionally specify a path to a `confluent` executable:

* `CCLOUD_EXECUTABLE_PATH`: `/full/path/to/ccloud`
* `CONFLUENT_EXECUTABLE_PATH`: `/full/path/to/confluent`

Otherwise, `ccloud` must be on your path.
Otherwise, `confluent` must be on your path.

## Validate

Expand Down
4 changes: 2 additions & 2 deletions docs/quick-start.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Getting started with `kafka-gitops` is simple. For this tutorial, we will assume
- You have installed the `kafka-gitops` command as [described here](/installation.md).
- You have a kafka cluster running on `localhost:9092`.

!> **NOTE**: If you desire to use this with Confluent Cloud, read our [Confluent Cloud page][ccloud].
!> **NOTE**: If you desire to use this with Confluent Cloud, read our [Confluent Cloud page][confluent].

## Desired State File

Expand Down Expand Up @@ -147,7 +147,7 @@ org.apache.kafka.common.errors.PolicyViolationException: Topic replication facto

Congrats! You've successfully started using GitOps strategies to manage your cluster. If you have security on your cluster, read the [services][services] page to start defining services.

[ccloud]: /confluent-cloud.md
[confluent]: /confluent-cloud.md
[permissions]: /permissions.md
[specification]: /specification.md
[services]: /services.md
4 changes: 2 additions & 2 deletions docs/specification.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,15 +19,15 @@ The desired state file consists of:

**Options**:

- **ccloud** [Optional]: An object which contains an `enabled` field. Set this to true if using a Confluent Cloud cluster.
- **confluent** [Optional]: An object which contains an `enabled` field. Set this to true if using a Confluent Cloud cluster.
- **topics** [Optional]:
- **defaults** [Optional]: Specify topic defaults so you don't need to specify them for every topic in the state file. Currently, only replication is supported.
- **blacklist** [Optional]: Add a prefixed topic blacklist for ignoring specific topics when using `kafka-gitops`. This allows topics to be ignored from being deleted if they are not defined in the desired state file.

**Example**:
```yaml
settings:
ccloud:
confluent:
enabled: true
topics:
defaults:
Expand Down
6 changes: 3 additions & 3 deletions examples/confluent-cloud/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ The following environment variables need to be set locally or in the build job:

```bash
# For service account creation / listing
export XX_CCLOUD_EMAIL="Your Confluent Cloud email address"
export XX_CCLOUD_PASSWORD="Your Confluent Cloud password"
export CONFLUENT_CLOUD_EMAIL="Your Confluent Cloud email address"
export CONFLUENT_CLOUD_PASSWORD="Your Confluent Cloud password"

# For executing against the cluster
export KAFKA_BOOTSTRAP_SERVERS="Your Confluent Cloud cluster URL"
Expand All @@ -32,7 +32,7 @@ export CLICOLOR_FORCE="true"

Once defining services and users, you can generate service accounts.

**NOTE**: Before running `accounts` or `plan`, ensure you are logged in to Confluent Cloud. Use `ccloud login`.
**NOTE**: Before running `accounts` or `plan`, ensure you are logged in to Confluent Cloud. Use `confluent login`.

Create accounts:

Expand Down
2 changes: 1 addition & 1 deletion examples/confluent-cloud/state.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
settings:
ccloud:
confluent:
enabled: true
files:
topics: topics.yaml
Expand Down
4 changes: 2 additions & 2 deletions src/main/java/com/devshawn/kafka/gitops/StateManager.java
Original file line number Diff line number Diff line change
Expand Up @@ -322,8 +322,8 @@ private void validateTopics(DesiredStateFile desiredStateFile) {
}

private boolean isConfluentCloudEnabled(DesiredStateFile desiredStateFile) {
if (desiredStateFile.getSettings().isPresent() && desiredStateFile.getSettings().get().getCcloud().isPresent()) {
return desiredStateFile.getSettings().get().getCcloud().get().isEnabled();
if (desiredStateFile.getSettings().isPresent() && desiredStateFile.getSettings().get().getConfluent().isPresent()) {
return desiredStateFile.getSettings().get().getConfluent().get().isEnabled();
}
return false;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
@JsonDeserialize(builder = Settings.Builder.class)
public interface Settings {

Optional<SettingsCCloud> getCcloud();
Optional<SettingsConfluent> getConfluent();

Optional<SettingsTopics> getTopics();

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@
import org.inferred.freebuilder.FreeBuilder;

@FreeBuilder
@JsonDeserialize(builder = SettingsCCloud.Builder.class)
public interface SettingsCCloud {
@JsonDeserialize(builder = SettingsConfluent.Builder.class)
public interface SettingsConfluent {

boolean isEnabled();

class Builder extends SettingsCCloud_Builder {
class Builder extends SettingsConfluent_Builder {
}
}
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package com.devshawn.kafka.gitops.manager;

import com.devshawn.kafka.gitops.config.ManagerConfig;
import com.devshawn.kafka.gitops.domain.plan.AclPlan;
import com.devshawn.kafka.gitops.domain.plan.DesiredPlan;
import com.devshawn.kafka.gitops.domain.plan.TopicConfigPlan;
import com.devshawn.kafka.gitops.domain.plan.TopicPlan;
Expand Down Expand Up @@ -63,7 +64,9 @@ private void applyTopicConfiguration(TopicPlan topicPlan, TopicConfigPlan topicC
}

public void applyAcls(DesiredPlan desiredPlan) {
desiredPlan.getAclPlans().forEach(aclPlan -> {
List<AclPlan> modifiableAclPlan = new ArrayList<>(desiredPlan.getAclPlans());
modifiableAclPlan.sort(Comparator.comparing(AclPlan::getAction));
modifiableAclPlan.forEach(aclPlan -> {
if (aclPlan.getAction() == PlanAction.ADD) {
LogUtil.printAclPreApply(aclPlan);
kafkaService.createAcl(aclPlan.getAclDetails().toAclBinding());
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,15 @@ public class ConfluentCloudService {
private static org.slf4j.Logger log = LoggerFactory.getLogger(ConfluentCloudService.class);

private final ObjectMapper objectMapper;
private static final String confluentExecutable;
private static final String ccloudExecutable;

public ConfluentCloudService(ObjectMapper objectMapper) {
this.objectMapper = objectMapper;
}

public List<ServiceAccount> getServiceAccounts() {
log.info("Fetching service account list from Confluent Cloud via ccloud tool.");
log.info("Fetching service account list from Confluent Cloud via confluent tool.");
try {
String result = execCmd(new String[]{ccloudExecutable, "service-account", "list", "-o", "json"});
return objectMapper.readValue(result, new TypeReference<List<ServiceAccount>>() {
Expand All @@ -33,11 +34,11 @@ public List<ServiceAccount> getServiceAccounts() {
}

public ServiceAccount createServiceAccount(String name, boolean isUser) {
log.info("Creating service account {} in Confluent Cloud via ccloud tool.", name);
log.info("Creating service account {} in Confluent Cloud via confluent tool.", name);
try {
String serviceName = isUser ? String.format("user-%s", name) : name;
String description = isUser ? String.format("User: %s", name) : String.format("Service account: %s", name);
String result = execCmd(new String[]{ccloudExecutable, "service-account", "create", serviceName, "--description", description, "-o", "json"});
String result = execCmd(new String[]{confluentExecutable, "iam", "service-account", "create", serviceName, "--description", description, "-o", "json"});
return objectMapper.readValue(result, ServiceAccount.class);
} catch (IOException ex) {
throw new ConfluentCloudException(String.format("There was an error creating Confluent Cloud service account: %s.", name));
Expand All @@ -50,7 +51,9 @@ public static String execCmd(String[] cmd) throws java.io.IOException {
}

static {
confluentExecutable = System.getenv("CONFLUENT_EXECUTABLE_PATH") != null ? System.getenv("CONFLUENT_EXECUTABLE_PATH") : "confluent";
log.info("Using confluent executable at: {}", confluentExecutable);
ccloudExecutable = System.getenv("CCLOUD_EXECUTABLE_PATH") != null ? System.getenv("CCLOUD_EXECUTABLE_PATH") : "ccloud";
log.info("Using ccloud executable at: {}", ccloudExecutable);
log.info("Using ccloud executable at: {}", confluentExecutable);
}
}