Install Streams

Learn how to install Streams on-premise or deploy it in your private cloud, configure a helm chart, and validate the installation.

This section covers recommended steps to install Streams both on development environment and production environment.

Prerequisites

Kubernetes 1.19+
Kubernetes Secrets
Helm 3.2.0+
RBAC enabled
PersistentVolumes and LoadBalancer provisioner supported by the underlying infrastructure
Resources:
- Minimal non-HA configuration: 9 CPUs and 10 GB RAM dedicated to the platform.
- Minimal HA configuration: 34 CPUs and 51 GB RAM dedicated to the platform.

For more information, see Reference Architecture.

Prepare your environment

After you have been onboarded on Amplify Platform, you will be able to download our latest Helm chart from the Downloads section of the Axway Support portal. Ensure to download the correct version of the Streams Helm chart corresponding to the release you wish to install.

To prepare your environment, extract the Helm chart and open a terminal from the extracted directory.

Note

You can find others resources in the Axway Support portal, for example, Postman collections, OpenAPI, Streams RBAC Policies, and Docker Compose files, which can help you to configure your environment or test Streams.

Prepare customizations

You must use Helm parameters to customize Streams during its installation. Any customization must be done before you install Helm.

To customize your installation, we recommend you to create a custom values file (for example, my-values.yaml) where you can overwrite parameters and pass on the file to the installation or upgrade commands by using the -f option. For example:

helm install -f my-values.yaml

For alternate methods to customize your installation, see Customize installation.

Accept general conditions for license and subscription services

To proceed with the installation, you must accept Axway General Terms and Conditions:

“You hereby accept that the Axway Products and/or Services shall be governed exclusively by the Axway General Terms and Conditions, unless an agreement has been signed with Axway in which case such agreement shall apply.”

To accept the conditions. you must set the helm value acceptGeneralConditions to "yes". Ensure to add the double quotation around the yes flag.

Kubernetes namespace

We recommend you deploy Streams components inside a dedicated namespace. To create a namespace, run the following command:

export NAMESPACE="my-namespace"
kubectl create namespace "${NAMESPACE}"

Use Amplify Platform as your Docker registry

Docker images must be hosted in a docker registry accessible from your Kubernetes (K8s) cluster. We recommend that you use the Amplify Platform repository for a custom docker registry. Alternatively, you can use your own custom Docker registry.

Before you start using Amplify Platform as your Docker registry, ensure the following:

Your Streams artifacts are listed on the Amplify repository under your Amplify Central organization.
You must have a service account in Amplify Central with the Authentication method set to Client Secret.

Create Kubernetes secret

Streams needs the credentials of your service account to pull images from the repository. The following is an example of how to create a secret containing these credentials:

export NAMESPACE="my-namespace"
export REGISTRY_USERNAME="<my-service-account-client-id>"
export REGISTRY_PASSWORD="<my-service-account-client-secret>"
export REGISTRY_SERVER="docker.repository.axway.com"

kubectl create secret docker-registry streams-docker-registry-secret --docker-server="${REGISTRY_SERVER}"  --docker-username="${REGISTRY_USERNAME}" --docker-password="${REGISTRY_PASSWORD}" -n "${NAMESPACE}"

Configuration for development environment

For a quick installation of Streams, the default settings target a development environment. For production environment settings, see Configuration for production environment section.

Embedded MariaDB settings

By default, an embedded MariaDB database is installed on your K8s cluster next to Streams.

Set MariaDB credentials

Passwords are required for Streams microservices to securely connect to an embedded MariaDB. To set your database passwords, you need to create a secret including your passwords, for example:

export NAMESPACE="my-namespace"
export MARIADB_ROOT_PASSWORD="my-mariadb-root-password"
export MARIADB_PASSWORD="my-mariadb-user-password"
export MARIADB_REPLICATION_PASSWORD="my-mariadb-replication-password"

kubectl create secret generic streams-database-passwords-secret --from-literal=mariadb-root-password=${MARIADB_ROOT_PASSWORD} --from-literal=mariadb-password=${MARIADB_PASSWORD}  --from-literal=mariadb-replication-password=${MARIADB_REPLICATION_PASSWORD} -n ${NAMESPACE}

Enable MariaDB security

By default, MariaDB is configured with TLS communication and Transparent Data Encryption enabled. To customize your MariaDB security settings, see Custom embedded MariaDB security settings.

TLS communication

To configure the TLS communication between MariaDB and Streams microservices, provide a CA certificate, a server certificate, and a server key.

For more information on how to generate a self-signed certificate, see Certificate Creation with OpenSSL.

Note

The server certificate’s Common Name must be set up with streams-database.

Transparent Data Encryption (TDE)

To configure the MariaDB data-at-rest encryption, you must provide a keyfile. The keyfile must contain a 32-bit integer identifier followed by the hex-encoded encryption key separated by semicolon. For example, <encryption_key_id>;<hex-encoded_encryption_key>.

To generate the keyfile, run the following command:

echo "1;$(openssl rand -hex 32)" > keyfile

MariaDB security secret

To enable TLS and TDE on MariaDB, you must create a k8s secret. For example:

export NAMESPACE="my-namespace"
kubectl create secret generic streams-database-secret --from-file=CA_PEM=ca.pem --from-file=SERVER_CERT_PEM=server-cert.pem --from-file=SERVER_KEY_PEM=server-key.pem --from-file=KEYFILE=keyfile -n ${NAMESPACE}

Performance tuning

You can update the following embedded MariaDB configuration values:

wait-timeout - update by setting the Helm parameter embeddedMariadb.waitTimeout.
max-connections - updated by setting the Helm parameter embeddedMariadb.maxConnections.

For more information, see database considerations.

Embedded Kafka settings

By default, an embedded Kafka cluster is installed on your K8s cluster next to Streams.

For security reasons, we strongly recommend to enable SASL authentication and TLS encryption for Kafka clients and brokers. For alternate security settings, see Custom embedded kafka security settings.

SASL and TLS are enabled by default. Because there is no sensitive data in Zookeeper, the communications with Zookeeper are in plaintext without authentication.

To configure SASL authentication, you must create a K8s secret with your kafka credentials, for example:

export NAMESPACE="my-namespace"
export KAFKA_CLIENT_PASSWORD="my-kakfa-client-password"
export KAFKA_INTERBROKER_PASSWORD="my-kakfa-interbroker-password"

kubectl -n ${NAMESPACE} create secret generic streams-kafka-passwords-secret --from-literal="client-passwords=${KAFKA_CLIENT_PASSWORD}" --from-literal="inter-broker-password=${KAFKA_INTERBROKER_PASSWORD}"

To configure TLS encryption, you must have a valid truststore and one certificate per broker. They must all be integrated into Java Key Stores (JKS) files. Note that each broker needs its own keystore and a dedicated CN name matching the Kafka pod hostname, as described in the bitnami documentation.

To facilitate the TLS configuration, we provide you with a script to help with truststore and keystore generation (based on bitnami’s script that properly handles Kubernetes deployment). For example:

cd tools
./kafka-generate-ssl.sh

After you keystore and certificates are created, you must create a secret, which contains all the previously generated files:

export NAMESPACE="my-namespace"
export KAFKA_SECRET_PASSWORD="my-kakfa-secret-password"
kubectl -n ${NAMESPACE} create secret generic streams-kafka-client-jks-secret --from-file="./truststore/kafka.truststore.jks" --from-file=./keystore/kafka-0.keystore.jks --from-file=./keystore/kafka-1.keystore.jks --from-file=./keystore/kafka-2.keystore.jks --from-literal="jks-password=${KAFKA_SECRET_PASSWORD}"

Configuration for production environment

This section covers recommended steps to install Streams in a production environment.

MariaDB settings

By default, an embedded MariaDB database is installed on your K8s cluster next to Streams. For production, we recommend that you use an externalized database instead.

To disable embedded MariaDB installation, set embeddedMariadb.enabled to false.

Externalized MariaDB configuration

To configure an external MariaDB database:

Connect to you MariaDB and create a database, which will be used by Streams:

export DB_HOST="my-db-host"
export DB_PORT="my-db-port"
export DB_USER="my-db-user"
export DB_NAME="streams"

mysql -h "${DB_HOST}" -P "${DB_PORT}" -u "${DB_USER}" -p -e "CREATE DATABASE ${DB_NAME};"

Create a dedicated streams user:

export DB_HOST="my-db-host"
export DB_PORT="my-db-port"
export DB_USER="my-db-user"
export DB_STREAMS_USER="streams"
export DB_STREAMS_PASS="my-streams-db-password"

mysql -h "${DB_HOST}" -P "${DB_PORT}" -u "${DB_USER}" -p -e "CREATE USER IF NOT EXISTS '${DB_STREAMS_USER}'@'%' IDENTIFIED BY '${DB_STREAMS_PASS}';"

Provide the new user with rights to select, insert, update, and delete on Streams database. It is also recommended to force the TLS authentication for this user. You can grant these privileges by running the following:

export DB_HOST="my-db-host"
export DB_PORT="my-db-port"
export DB_USER="my-db-user"
export DB_NAME="streams"
export DB_STREAMS_USER="streams"

mysql -h "${DB_HOST}" -P "${DB_PORT}" -u "${DB_USER}" -p -e "GRANT SELECT, INSERT, UPDATE, DELETE ON ${DB_NAME}.* TO ${DB_STREAMS_USER} REQUIRE SSL;"

Provide the connectivity information to the Streams installation by setting the following Helm parameters:
- externalizedMariadb.host
- externalizedMariadb.port
- externalizedMariadb.rootUsername
Set the Helm parameter streams.serviceArgs.spring.datasource.hikari.maxLifetime to a value (in seconds) according to the wait-timeout value of your MariaDB database. For more information, see Database considerations.

Set external MariaDB credentials

Passwords are required for Streams microservices to securely connect to an embedded MariaDB. To set your database passwords, you need to create a secret including your passwords, for example:

export NAMESPACE="my-namespace"
export MARIADB_ROOT_PASSWORD="my-mariadb-root-password"
export MARIADB_PASSWORD="my-mariadb-user-password"

kubectl create secret generic streams-database-passwords-secret --from-literal=mariadb-root-password=${MARIADB_ROOT_PASSWORD} --from-literal=mariadb-password=${MARIADB_PASSWORD} -n ${NAMESPACE}

Externalized MariaDB TLS

For security reasons, we strongly recommend to enable TLS communication between your database and Streams microservices. The One-Way TLS is the most used mode, and it can be enabled as follows:

Provide a CA certificate by creating a secret:

export NAMESPACE="my-namespace"
kubectl create secret generic streams-database-secret --from-file=CA_PEM=ca.pem -n ${NAMESPACE}

Set the Helm parameters externalizedMariadb.tls.twoWay to false.

For more information on how to generate a self-signed certificate, see Certificate Creation with OpenSSL.

For alternate security settings, see Custom externalized MariaDB security settings.

Externalized Kafka settings

By default, an embedded Kafka cluster is installed on your K8s cluster next to Streams. For production environments, we recommend that you use an externalized Kafka instead.

To disable the embedded Kafka installation and use a externalized kafka cluster, follow these steps::

Set the Helm parameter embeddedKafka.enabled to false.
Provide connectivity information to the Streams installation by specifying the Helm parameter externalizedKafka.bootstrapServers.
Ensure that delete.topic.enable is set to true in your Kafka installation.

Externalized Kafka security settings

For security reasons, we strongly recommend to enable SASL authentication and TLS encryption for Kafka clients and brokers.

To configure SASL authentication, follow these steps:

Create a K8s secret with your kafka credentials:

export NAMESPACE="my-namespace"
export KAFKA_USER_PASSWORD="my-kafka-password"

kubectl create secret generic streams-kafka-passwords-secret --from-literal=client-passwords=${KAFKA_USER_PASSWORD} -n ${NAMESPACE}

Provide a JKS containing the Kafka TLS truststore and its password:

export NAMESPACE="my-namespace"
export KAFKA_JKS_PASSWORD="my-kafka-jks-password"
export KAFKA_JKS_PATH="my-kafka-jks-path"

kubectl create secret generic streams-kafka-client-jks-secret --from-file=kafka.truststore.jks=${KAFKA_JKS_PATH} --from-literal=jks-password=${KAFKA_JKS_PASSWORD} -n ${NAMESPACE}

Set the Helm parameter externalizedKafka.auth.clientUsername with your Kafka username.

To disable security settings, see Disable externalized kafka security settings.

Ingress settings

An ingress controller is installed on your K8s cluster next to Streams.

Load Balancer settings

Depending on your cloud provider, deploying a load balancer might require additional parameters. For example, if you are using AWS, you must define the load balancer by setting the Helm parameter nginx-ingress-controller.service.annotations."service.beta.kubernetes.io/aws-load-balancer-type" to nlb.

Refer to your own cloud provider for further details.

For more information on the load balancer types, see Reference Architecture.

Ingress hostname

You must specify a hostname for the ingress installed with Streams helm chart. To do so, you must set the Helm parameter ingress.host parameter to specify the hostname.

Because this is a mandatory setting, if you do not have a hostname yet, use a temporary value and edit it later. In most case a DNS name will be generated after the installation by your cloud provider load balancer.

To update the ingress host, run the following command:

export NAMESPACE="my-namespace"
kubectl get svc -o=jsonpath='{.items[?(streams-*)].status.loadBalancer.ingress[0].hostname}' -n "${NAMESPACE}"

Then upgrade your Streams installation with the Helm parameter ingress.host set with the DNS name retrieved previously. For more information, see Helm upgrade.

Note

k8s.yourdomain.tld is used throughout this documentation as an example hostname value.

Ingress TLS

SSL/TLS is enabled by default on the embedded Ingress controller with a auto-generated certificate. To use your own controller, see Custom ingress controller security settings.

Ingress CORS

Cross-Origin Resource Sharing (CORS) is disabled by default. To enable it, see Custom ingress controller CORS.

Monitor your installation

Streams ships with traffic monitoring. You can activate metrics with the parameters listed in Monitoring parameters, which will open endpoints designed to be scrapped by Prometheus.

Note

Enabling monitoring may increase CPU and memory loads.

Add self-signed TLS certificates

TLS endpoints which Streams services connect to must have a valid TLS certificate. If your endpoints use self-signed certificates, you must add them to Streams services as trusted certificates as follows:

Ensure that your certificates are in PEM format.

Create one or several secrets containing your PEM files, for example:

export NAMESPACE="my-namespace"
export SECRET_NAME="my-secret"
export PEM_PATH="my-pem-path"

kubectl create secret generic "${SECRET_NAME}" -n "${NAMESPACE}" --from-file="${PEM_PATH}" [--from-file=<other-pem-path>]

Set the Helm parameter streams.extraCertificatesSecrets to your $SECRET_NAME. If you have more than one secrets, they must be separated by a comma.

Integrate with Amplify Central

Streams can connect to Amplify Central and expose assets to leverage tools like the Amplify Marketplace. This integration is disabled by default. For more information on how to enable it, see Amplify Central Integration.

Customize your installation

You can specify optional Helm parameters to customize your installation.

You can also find examples of customization at Customize installation.

Helm install command

You can deploy Streams in non-HA or HA configurations.

Non-High availability configuration

The following command deploys Streams components in a non-HA configuration with one replica per microservices. This might take a few minutes.

Note

This is not recommended for production environments.

export NAMESPACE="my-namespace"
export HELM_RELEASE_NAME="my-release"

helm install "${HELM_RELEASE_NAME}" . \
  -n "${NAMESPACE}"

High availability configuration

The following command deploys Streams on the Kubernetes cluster in high availability. The deploy might take a few minutes.

Note

This configuration is recommended for production environments.

export NAMESPACE="my-namespace"
export HELM_RELEASE_NAME="my-release"

helm install "${HELM_RELEASE_NAME}" . \
  -f values-ha.yaml \
  -n "${NAMESPACE}"

Validate the installation

If Streams is successfully installed, the output of the helm install command should look like the following (for non-HA configuration):

NAME: my-release
LAST DEPLOYED: Wed Mar 25 15:18:45 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTE: ...

After a few minutes, all the pods should be running properly:

export NAMESPACE="my-namespace"
kubectl get po -n "${NAMESPACE}"

NAME                                                           READY   STATUS    RESTARTS   AGE
streams-kafka-0                                                1/1     Running   0          116s
streams-mariadb-master-0                                       1/1     Running   0          116s
streams-zookeeper-0                                            1/1     Running   0          116s
my-release-hub-675c6f9f6-8gplz                                 1/1     Running   0          116s
my-release-nginx-ingress-controller-58bfd85658-6plf5           1/1     Running   0          116s
my-release-publisher-http-poller-6cc5cd9fc6-q564p              1/1     Running   0          116s
my-release-publisher-http-post-5b745f864-dpws8                 1/1     Running   0          116s
my-release-subscriber-sse-7fd8c56f48-wvgtq                     1/1     Running   0          116s
my-release-subscriber-webhook-84469bd68f-lqxgk                 1/1     Running   0          116s
[...]

Run smoke tests to verify Streams configuration

To verify Streams is properly configured, launch the automated smoke tests provided by the Helm Chart. The tests perform the following:

Test the Hub API by creating a topic named smoke-test-topic using our public test API https://stockmarket.streamdata.io/prices and the http-poller publisher.
Test the SSE subscription API by starting a subscription to the topic previously created.

Run the following command to start the smoke tests:

export NAMESPACE="my-namespace"
export HELM_RELEASE_NAME="my-release"
helm test "${HELM_RELEASE_NAME}" -n "${NAMESPACE}"

The following shows the output in case of success:

NAME: my-namespace
LAST DEPLOYED: Fri Dec  3 12:30:58 2021
NAMESPACE: test
STATUS: deployed
REVISION: 1
TEST SUITE:     my-release-smoke-tests
Last Started:   Fri Dec  3 12:32:32 2021
Last Completed: Fri Dec  3 12:34:04 2021
Phase:          Succeeded
NOTES:
Validate your installation following the documentation (https://streams-open-docs.netlify.app/docs/install/#validate-the-installation).

In case of failure (Phase : Failed), check the logs of the smoke-tests pod as follows:

export NAMESPACE="my-namespace"
export HELM_RELEASE_NAME="my-release"
kubectl logs "pod/${HELM_RELEASE_NAME}-smoke-tests" -n "${NAMESPACE}"

Finally, verify the SSE subscription API is accessible through the Load Balancer provided during installation. Run the following command to start the subscription:

export NAMESPACE="my-namespace"
INGRESS_ADDRESS=$(kubectl get ingress -o=jsonpath='{.items[?(@.metadata.name=="streams-subscriber-sse")].status.loadBalancer.ingress[0].hostname}' -n ${NAMESPACE}) | echo ${INGRESS_ADDRESS}
curl -v "https://${INGRESS_ADDRESS}/streams/subscribers/sse/api/v1/topics/smoke-test-topic"

Note

The default configuration only accepts incoming HTTP/HTTPS requests to k8s.yourdomain.tld. For more information, see Helm parameters.

Last modified June 24, 2022: Create APIM integration page (#163) (ba76e99)