Apache Kafka® with custom configurations
========================================

This example deploys an `Aiven for Apache Kafka® <https://aiven.io/kafka>`_ service with some custom configurations, as well as a Kafka topic, a Kafka user, and an access control list (ACL) to allow fine-grained permissions about which topic that user can access, using the `Aiven Terraform Provider <https://registry.terraform.io/providers/aiven/aiven/latest/docs>`_.

Before looking at the Terraform script, let's visualize the resources:

.. mermaid::

   flowchart LR
      subgraph Aiven-for-Apache-Kafka
      id4[[Aiven Kafka User]]
      id5[[Aiven Kafka User ACL]]
      end
      Producer --> Aiven-for-Apache-Kafka --> Consumer

Let's cook!
------------

Be sure to check out the :doc:`getting started guide <../../get-started>` to learn about the common files required to execute the following recipe. For example, you'll need to declare the variables for ``project_name``, ``api_token``, and ``kafka_user_name``.

.. dropdown:: Expand to check out the relevant common files needed for this recipe.

    Navigate to a new folder and add the following files.

    1. Add the following to a new ``provider.tf`` file:

    .. code:: terraform

       terraform {
         required_providers {
           aiven = {
             source  = "aiven/aiven"
             version = "~> 3.10.0"
           }
         }
       }
   
       provider "aiven" {
         api_token = var.aiven_api_token
       }
   
    You can also set the environment variable ``TF_VAR_aiven_api_token`` for the ``api_token`` property. With this, you don't need to pass the ``-var-file`` flag when executing Terraform commands.
 
    2. To avoid including sensitive information in source control, the variables are defined here in the ``variables.tf`` file. You can then use a ``*.tfvars`` file with the actual values so that Terraform receives the values during runtime, and exclude it.

    The ``variables.tf`` file defines the API token, the project name to use, and the prefix for the service name:

    .. code:: terraform

       variable "aiven_api_token" {
         description = "Aiven console API token"
         type        = string
       }
   
       variable "project_name" {
         description = "Aiven console project name"
         type        = string
       }
   
       variable "kafka_user_name" {
         description = "Username for Aiven for Apache Kafka user"
         type        = string
       } 
    
    3. The ``var-values.tfvars`` file holds the actual values and is passed to Terraform using the ``-var-file=`` flag.

    ``var-values.tfvars`` file:

    .. code:: terraform

       aiven_api_token     = "<YOUR-AIVEN-AUTHENTICATION-TOKEN-GOES-HERE>"
       project_name        = "<YOUR-AIVEN-CONSOLE-PROJECT-NAME-GOES-HERE>"
       kafka_user_name     = "<A-SAMPLE-USERNAME>"

Here is the sample Terraform script to stand-up Aiven for Apache Kafka and related resources. The script also performs some custom configurations on these resources.

``services.tf`` file:

.. code:: terraform
  
  resource "aiven_kafka" "demo-kafka" {
    project                 = var.project_name
    cloud_name              = "google-europe-west1"
    plan                    = "business-4"
    service_name            = "demo-kafka"
    maintenance_window_dow  = "sunday"
    maintenance_window_time = "01:00:00"
    default_acl             = false
  
    kafka_user_config {
      kafka_rest      = true
      kafka_connect   = false
      schema_registry = true
      kafka_version   = "3.2"
  
      kafka {
        auto_create_topics_enable  = true
        num_partitions             = 3
        default_replication_factor = 2
        min_insync_replicas        = 2
      }
  
      kafka_authentication_methods {
        certificate = true
      }
  
      public_access {
        kafka_rest = true
      }
    }
  }
  
  resource "aiven_kafka_topic" "demo-kafka-topic" {
    project      = var.project_name
    service_name = aiven_kafka.demo-kafka.service_name
    topic_name   = "demo-kafka-topic"
    partitions   = 5
    replication  = 3
  }
  
  resource "aiven_kafka_user" "demo-kafka-user" {
    project      = var.project_name
    service_name = aiven_kafka.demo-kafka.service_name
    username     = var.kafka_user_name
  }
  
  resource "aiven_kafka_acl" "demo-kafka-user-acl" {
    project      = var.project_name
    service_name = aiven_kafka.demo-kafka.service_name
    username     = var.kafka_user_name
    permission   = "read"
    topic        = aiven_kafka_topic.demo-kafka-topic.topic_name
  }
  
.. dropdown:: Expand to check out how to execute the Terraform files.

    The ``init`` command performs several different initialization steps in order to prepare the current working directory for use with Terraform. In our case, this command automatically finds, downloads, and installs the necessary Aiven Terraform provider plugins.
    
    .. code:: shell

       terraform init

    The ``plan`` command creates an execution plan and shows you the resources that will be created (or modified) for you. This command does not actually create any resource; this is more like a preview.

    .. code:: bash

       terraform plan -var-file=var-values.tfvars

    If you're satisfied with the output of ``terraform plan``, go ahead and run the ``terraform apply`` command which actually does the task or creating (or modifying) your infrastructure resources. 

    .. code:: bash

       terraform apply -var-file=var-values.tfvars

Let's go over a few of these configurations and understand their functions:

``aiven_kafka`` resource configurations:

- ``default_acl`` parameter, when set to **true**, creates default wildcard Kafka ACL. This example sets this parameter to **false** and prevents the default wildcard ACL for resources.

- For ``kafka_user_config``, ``schema_registry`` is set to **true**, which enables the `Karapace Schema Registry <https://aiven.io/blog/what-is-karapace>`_ and ``kafka_rest`` allows you to view the messages in the topics from the Aiven web console when set to **true**.

- ``auto_create_topics_enable`` under ``kafka`` nested configurations enables the auto creation of topics when set to **true**. This means that a topic doesn't need to exist before sending a message.

- ``num_partitions`` will set the number of partitions for the automatically created topics.

- By default, the replication factor is 1. This example sets ``default_replication_factor`` to 2 and thus requires a minimum of two brokers. For production environments, a replication factor of 3 is recommended. 

- ``min_insync_replicas`` indicates that at least 2 replicas (brokers) should respond back if all replicas(brokers) are not functioning properly. When all replicas are functioning properly, this setting has no effect. 

- The ``certificate`` parameter under the ``kafka_authentication_methods`` nested configurations, when set to **true**, enables certificate/SSL authentication.

``aiven_kafka_topic`` resource configurations:

- ``partitions`` denotes the number of partitions to create in the topic, and ``replication`` sets the replication factor for the topic.

``aiven_kafka_user`` resource configurations:

- We are passing a preset username using ``var.kafka_user_name``

``aiven_kafka_acl`` resource configurations:

- This ACL allows **read** access to the ``demo-kafka-topic`` topic for the ``var.kafka_user_name`` user. 

.. Warning:: 

  By default, Aiven adds an ``avnadmin`` account to every new service and adds `admin` permission for all topics to that user. When you create your own ACLs to restrict access, you probably want to remove this ACL entry.

.. Note::

  When using the Aiven Terraform Provider, you can add the ``default_acl`` key to your ``resource`` and set it to ``false`` if you do not want to create the admin user with wildcard permissions.

More resources
--------------

Keep in mind that some parameters and configurations will vary for your case. Some related resources are provided below:

- `Configuration options for Aiven for Apache Kafka <https://docs.aiven.io/docs/products/kafka/reference/advanced-params.html>`_
- `Aiven for Apache Kafka access control lists permission mapping <https://docs.aiven.io/docs/products/kafka/concepts/acl.html>`_
- `How to Manage Aiven for Apache Kafka Parameters <https://www.youtube.com/watch?v=pXQZWI0ddLg>`_
- `Set up your first Aiven Terraform project <https://docs.aiven.io/docs/tools/terraform/get-started.html>`_