Operational architecture

Operational architecture overview

HeartAI operational environments are cloud-native and compose containerised and/or virtualised components. These approaches allow a natively distributed deployment of operational components, where these components may be dynamically scaled to meet resource demand. Current orchestration software includes Red Hat OpenShift, Kubernetes, and Docker Compose, with container technologies provided by Docker and Podman. The primary system deployment implements Microsoft Azure Red Hat OpenShift, which provides a fully-managed of instance of Red Hat OpenShift on Microsoft Azure. OpenShift natively supports many popular technology frameworks.

For operational implementations, HeartAI endeavours to provide modern and robust security profiles. Operational orchestration with Microsoft Azure Red Hat OpenShift provides an increased ability to meet demanding security requirements. As a platform extension to Red Hat OpenShift instances, Red Hat Advanced Cluster Security (RHACS) is a Kubernetes-native security platform that provides best-in-class security integrations and solutions for container-based environments. RHACS security capabilities ensure that Kubernetes-based infrastructure is continuously protected and secure.

HeartAI instances of Red Hat OpenShift are integrated with cloud-native operational logging, monitoring, and observability software. Prometheus provides monitoring of systems and services, and Alertmanager implements event-triggered system behaviour. Grafana provides a real-time observability solution, with various adaptors to interface with data and metrics providers, allowing this data to be processed and visualised through reporting and user application dashboards.

Red Hat OpenShift implementation

Red Hat OpenShift is an enterprise-grade implementation of Kubernetes, providing a modern and secure platform for the orchestration of container-based solutions. OpenShift provides general platform-level capabilities, including:

• Abstractions for container-level deployments.
• Software orchestration that is natively cloud-based and distributable.
• Secure implementations of software-defined networking.
• Real-time and aggregated logging, monitoring, and observability.
• Frameworks for eventing and alerting.
• Controls for resource management.
• Access-level controls based around service accounts, roles, groups, and role bindings.
• Graphical user interfaces for both administrators and developers.

HeartAI deploys OpenShift with Microsoft Azure Red Hat OpenShift (ARO), a fully-managed implementation of OpenShift within Microsoft Azure. ARO is deployed to instances of Microsoft Azure cloud resources, which are fully-managed by Microsoft Azure, including operational lifecycle management, patching and updating, logging, monitoring, and security hardening.

OpenShift identity provision and access control

HeartAI instances of OpenShift integrate natively with the HeartAI implementation of Keycloak. This allows users to authenticate with OpenShift through the OpenShift OAuth 2.0 and OpenID Connect framework. With support for federated identity, users will also be able to authenticate with their Microsoft Azure Active Directory identity.

The OpenShift identity provider allows role-based access control (RBAC) at the following levels:

RBAC level	Description
Cluster RBAC	Roles and bindings that are applicable across all projects. Cluster roles exist cluster-wide, and cluster role bindings can reference only cluster roles.
Local RBAC	Roles and bindings that are scoped to a given project. While local roles exist only in a single project, local role bindings can reference both cluster and local roles.

Access control is managed with the following authorisation types:

Authorisation type	Description
Rules	Sets of permitted verbs on a set of objects. For example, whether a user or service account can create pods.
Roles	Collections of rules. You can associate, or bind, users and groups to multiple roles.
Bindings	Associations between users and/or groups with a role.

The following general roles exist within an OpenShift instance:

Role	Description
admin	A project manager. If used in a local binding, an admin has rights to view any resource in the project and modify any resource in the project except for quota.
basic-user	A user that can get basic information about projects and users.
cluster-admin	A super-user that can perform any action in any project. When bound to a user with a local binding, they have full control over quota and every action on every resource in the project.
cluster-status	A user that can get basic cluster status information.
edit	A user that can modify most objects in a project but does not have the power to view or modify roles or bindings.
self-provisioner	A user that can create their own projects.
view	A user who cannot make any modifications, but can see most objects in a project. They cannot view or modify roles or bindings.

Corresponding roles are applied to HeartAI administrators and developers in relation to their access to the OpenShift environment.

OpenShift console

OpenShift provides a comprehensive graphical user interface console for both administrator and developer environments. The following image shows the OpenShift console overview, describing various cluster components, including:

• General cluster information.
• Cluster status and availability.
• Cluster resource deployment metrics.
• Real-time monitoring of resource usage, events, and alerting.

OpenShift Namespaces

OpenShift Namespaces provides cluster scoping that logical separates individual namespaces within corresponding overlay VXLAN networks. Through this approach distinct namespaces are effectively compartmentalised from each other.

Example: OpenShift declaration file for Namespace

The following example shows an OpenShift Namespace declaration file for the HeartAI HelloWorldService production environment:

copyapiVersion: v1
kind: Namespace
metadata:
  name: heartai-hello-world-prod

OpenShift Pods

OpenShift Pod resources are atomic and composable components of OpenShift clusters, representing the smallest logical orchestration unit. Pods are deployed to corresponding virtual IPs on the residing host network, with these networks being logically separated by VXLAN at the level of OpenShift Namespaces. Pod resource declarations may specify zero-or-more corresponding containers with associated configurations, through which the OpenShift cluster maintains the declared number of containers within the Pod host environment.

Example: OpenShift console for namespace Pods

The following image shows the OpenShift console for Pods within the HeartAI Red Hat Advanced Cluster Security production environment namespace:

Example: OpenShift console for Pod

The following image shows the Red Hat OpenShift web interface console for a Pod resource instance. The Pod web interface console provides information and functionality to support OpenShift hosting of Pod resources, including:

• The managing Namespace of the Pod resource.
• The Pod name.
• Monitoring metrics of the Pod, including:
  • Memory utilisation.
  • CPU utilisation.
  • Filesystem utilisation.
  • Network inbound bandwidth.
  • Network outbound bandwidth.
• Assigned labels of the Pod.
• Pod health status.
• Pod virtual IP address assignment.
• The hosting Node of the Pod.
• The Pod creation timestamp.
• The owning resource.
• Pod-hosted containers.
• Pod-hosted volumes.
• The event history of the Pod.

OpenShift Deployments

OpenShift Deployments allow the declaration of Pod deployments and higher-level configuration for how these Pods should be orchestrated and managed within the OpenShift environment. The OpenShift Deployment Controller synchronises the actual state of the cluster to the declared state with configurable controls for how this synchronisation operates.

Example: OpenShift declaration file for Deployment

The following example shows an OpenShift Deployment declaration file for the HeartAI HelloWorldService production environment namespace:

copyapiVersion: "apps/v1"
kind: Deployment
metadata:
  name: heartai-service-hello-world-prod
  namespace: heartai-hello-world-prod
  labels:
    app: heartai-service-hello-world-prod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: heartai-service-hello-world-prod
  strategy:
    rollingUpdate:
      maxSurge: 3
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: heartai-service-hello-world-prod
        version: v0.33.56
        actorSystemName: heartai-service-hello-world-prod
      annotations:
        sidecar.istio.io/inject: "true"
        traffic.sidecar.istio.io/includeInboundPorts: "2552,8558,14000,14020"
        traffic.sidecar.istio.io/excludeOutboundPorts: "2552,8558"
    spec:
      containers:
        - name: heartai-service-hello-world-prod
          image: "quay.io/heartai/heartai-hello-world:0.33.56"
          imagePullPolicy: Always
          livenessProbe:
            httpGet:
              path: /alive
              port: management
            initialDelaySeconds: 20
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /ready
              port: management
            initialDelaySeconds: 20
            periodSeconds: 10
          ports:
            - name: remoting
              containerPort: 2552
              protocol: TCP
            - name: management
              containerPort: 8558
              protocol: TCP
            - name: http
              containerPort: 14000
              protocol: TCP
            - name: https
              containerPort: 14020
              protocol: TCP
          env:
            - name: OS_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: JAVA_OPTS
              value: "-Xms1024m -Xmx1024m -Dconfig.resource=production.conf"
            - name: APPLICATION_SECRET
              valueFrom:
                secretKeyRef:
                  name: heartai-play-secret
                  key: secret
            - name: AKKA_DISCOVERY_SERVICE_NAME
              value: "heartai-service-hello-world-prod"
            - name: AKKA_CLUSTER_BOOTSTRAP_SERVICE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: "metadata.labels['app']"
            - name: REQUIRED_CONTACT_POINT_NR
              value: "1"
            #            - name: AKKA_REMOTING_HOST
            #              value: "heartai-service-hello-world-prod.heartai-hello-world-prod.svc.cluster.local"
            - name: AKKA_REMOTING_PORT
              value: "2552"
            - name: AKKA_MANAGEMENT_PORT
              value: "8558"
            - name: HTTP_BIND_ADDRESS
              value: "0.0.0.0"
            - name: HTTP_PORT
              value: "14000"
            - name: HTTPS_PORT
              value: "14020"
            - name: KAFKA_SERVICE_NAME
              value: "strimzi-kafka-kafka-brokers.heartai-strimzi.svc.cluster.local:9092"
            - name: KAFKA_BROKERS_SERVICE_URL
              value: "strimzi-kafka-kafka-bootstrap.heartai-strimzi.svc.cluster.local:9092"
            - name: POSTGRESQL_CONTACT_POINT
              valueFrom:
                secretKeyRef:
                  name: postgres-url
                  key: secret
            - name: POSTGRESQL_USERNAME
              valueFrom:
                secretKeyRef:
                  name: postgres-id
                  key: secret
            - name: POSTGRESQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-key
                  key: secret
          resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 200m
              memory: 1048Mi

Example: OpenShift console for namespace Deployments

The following image shows the OpenShift console for Deployments within the HeartAI Red Hat Advanced Cluster Security production environment namespace:

Example: OpenShift console for Deployment

The following image shows the OpenShift console for the sensor Deployment within the HeartAI Red Hat Advanced Cluster Security production environment namespace:

OpenShift ReplicaSets

Example: OpenShift console for namespace ReplicaSets

The following image shows the OpenShift console for ReplicaSets within the HeartAI Red Hat Advanced Cluster Security production environment namespace:

Example: OpenShift console for ReplicaSet

The following image shows the OpenShift console for the sensor ReplicaSet within the HeartAI Red Hat Advanced Cluster Security production environment namespace:

OpenShift Secrets

Encrypted sensitive data store that is injectable at initiation / run-time to corresponding system components.

Example: OpenShift console for namespace Secrets

The following image shows the OpenShift console for Secrets within the HeartAI Red Hat Advanced Cluster Security production environment namespace:

OpenShift Services

OpenShift Services provide a cluster-internal access point to corresponding overlay network address spaces. For network routing to OpenShift Pods, Services allow consistent resolution to the virtual IP address space of one-or-more Pods, noting that such Pods may generally be transient. Access to deployment Pods through a Service is location transparent, scalable, and tolerant to Pod failure. Services may also be configured with load balancing, port-forwarding capability, and session affinity.

Example: OpenShift declaration file for Service

The following example shows an OpenShift Service declaration file for the HeartAI HelloWorldService production environment namespace:

copyapiVersion: v1
kind: Service
metadata:
  name: heartai-service-hello-world-prod
  namespace: heartai-hello-world-prod
spec:
  ports:
    - name: http
      port: 80
      targetPort: 14000
  selector:
    app: heartai-service-hello-world-prod
  type: LoadBalancer

Example: OpenShift console for namespace Services

The following image shows the OpenShift console for Services within the HeartAI Red Hat Advanced Cluster Security production environment namespace:

Example: OpenShift console for Service

The following image shows the OpenShift console for the sensor Service within the HeartAI Red Hat Advanced Cluster Security production environment namespace:

OpenShift Routes

OpenShift Routes provide ingress points to access cluster services. A typical route refers to a cluster-internal Service that resolves to corresponding Pods. Routes may be accessed through public or private networks where appropriate.

Routes support the establishment of TLS-encrypted connections. The system OpenShift implementation natively supports the following methods of TLS connection:

TLS connection type	Description
Passthrough	Server certificates from downstream server passed through the edge router and presented to requesting agent. TLS encryption occurs end-to-end from requesting agent to downstream server.
Re-encryption	The edge router presents its server certificate to the requesting agent, and the edge router itself requests a server certificate from the downstream server. TLS encryption occurs both between the requesting agent and the edge router, and the edge router and the downstream server.
Edge termination	The edge router presents its server certificate to the the requesting agent, but the edge router does not request a server certificate from the downstream server. TLS encryption occurs between the requesting agent and the edge router, but not between the edge router and the downstream server.

For purposes of security, HeartAI only implements the passthrough and re-encryption methods of TLS connections.

Example: OpenShift console for namespace Routes

The following image shows the OpenShift console for Routes within the HeartAI Red Hat Advanced Cluster Security production environment namespace:

Example: OpenShift console for Route

The following image shows the OpenShift console for the sensor Service within the HeartAI Red Hat Advanced Cluster Security production environment namespace:

OpenShift PersistentVolumes

OpenShift PersistentVolumes provide methods to abstract storage provision and consumption. PersistentVolume resources are units of storage that are manually provisioned by a cluster administrator or are dynamically generated by a corresponding StorageClass. PersistentVolumes may be consumed by cluster Resources, such as Pods, but also have a lifecycle that is independent of the consumer.

For HeartAI instances of Microsoft Azure Red Hat OpenShift, PersistentVolume resources are provided by Azure Managed Disks.

Example: OpenShift console for cluster Persistent Volumes

The following image shows the Red Hat OpenShift web interface console for cluster-level PersistentVolumes. The PersistentVolumes web interface console provides:

• An overview of cluster-level PersistentVolumes, including:
  • PersistentVolume names.
  • The operation status of PersistentVolumes.
  • The associated PersistentVolumeClaims.
  • The allocated storage capacities.
  • Assigned metadata labels.
  • Creation timestamps.

OpenShift PrometheusRules

OpenShift PrometheusRules provide generic and extensible alerting in response to system state or event behaviour. OpenShift alerting functionality is integrated with the Prometheus monitoring solutions. Alerts are transmissible to various end-points, such as email and SMS, and forwarding functionality exists with native adapters, such as for the Splunk data-observability platform.

Example: OpenShift console for cluster PrometheusRules

The following image shows the OpenShift console for PrometheusRules within the HeartAI OpenShift cluster:

Example: OpenShift console for PrometheusRule

The following image shows the OpenShift console for the PrometheusRule NodeFileSystemAlmostOutOfSpace:

OpenShift ServiceAccounts

OpenShift ServiceAccounts provide resources for abstracting service principles. ServiceAccounts may specify cluster- and namespace-level access control with corresponding Roles and RoleBindings.

Example: OpenShift console for namespace ServiceAccounts

The following image shows the OpenShift console for ServiceAccounts within the HeartAI Red Hat Advanced Cluster Security production environment namespace:

OpenShift Roles

OpenShift Roles provide specifications to configure access control at the cluster- or namespace-level.

Example: OpenShift declaration file for Role

The following example shows a Role declaration from the HeartAI HelloWorldService production environment. This particular Role declaration provides the permission required for the HelloWorldService service Deployment to be able to locate corresponding Pods of the service, fulfilling the ability of individual Pod resources to provide location transparent service discovery to other Pod resources.

copyapiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: heartai-pod-reader
  namespace: heartai-hello-world-prod
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "watch", "list"]

Example: OpenShift console for namespace Roles

The following image shows the OpenShift console for Roles within the HeartAI Red Hat Advanced Cluster Security production environment namespace:

OpenShift RoleBindings

OpenShift Role Bindings allow the association of Roles to corresponding ServiceAccounts.

Example: OpenShift declaration file for RoleBinding

The following example shows a Role Binding declaration from the HeartAI HelloWorldService production environment.

copyapiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: heartai-hello-world-prod
subjects:
  - kind: ServiceAccount
    name: default
roleRef:
  kind: Role
  name: heartai-pod-reader
  apiGroup: rbac.authorization.k8s.io

Example: OpenShift console for namespace RoleBindings

The following image shows the OpenShift console for RolesBindings within the HeartAI Red Hat Advanced Cluster Security production environment namespace:

OpenShift Nodes

OpenShift Nodes provide resource abstractions for the underlying physical or virtual machines of the cluster. For the HeartAI instances of Microsoft Azure Red Hat OpenShift, Node resources are implemented by instances of Microsoft Azure Virtual Machines.

Example: OpenShift console for cluster Nodes

The following image shows the OpenShift console for cluster Nodes:

OpenShift Machines

OpenShift Machines provide additional abstractions to Node resources specifically for OpenShift-based clusters.

Example: OpenShift console for cluster Machines

The following image shows the OpenShift console for cluster Machines:

OpenShift Operators

The following image shows the OpenShift console for cluster Machines: OpenShift Operators are extendable compositions of OpenShift resources generally, and provide a declarative framework for the composition of these resources. The Red Hat Marketplace provides a variety of Operators that are readily deployable to OpenShift clusters.

Example: OpenShift console for Operator Hub

The following image shows the OpenShift console for the Operator Hub, with filtering applied to show security-based Operators:

OpenShift Topology

OpenShift provides capabilities that are tailored for developer experiences. In addition to the administrator console, OpenShift provides a dedicated developer portal.

Example: OpenShift console for namespace topology overview

The following image shows the OpenShift developer console topology overview for the HeartAI Red Hat Advanced Cluster Security namespace:

Example: OpenShift console for namespace topology graph

The following image shows the OpenShift web interface console for a namespace topology graph. This web interface is specifically available through the developer console and provides information about an application-level Deployment. The topology graph web interface provides:

• The managing project of the Deployment.
• An overview of an application-level Deployment, including:
  • A topological visualisation of the resource components that compose the Deployment.
  • Information about Deployment resources:
    • Deployment Pods.
    • Deployment Services.
    • Deployment Routes.