Operational architecture
Operational architecture overview
HeartAI operational environments are cloud-native and compose containerised and/or virtualised components. These approaches allow a natively distributed deployment of operational components, where these components may be dynamically scaled to meet resource demand. Current orchestration software includes Red Hat OpenShift, Kubernetes, and Docker Compose, with container technologies provided by Docker and Podman. The primary system deployment implements Microsoft Azure Red Hat OpenShift, which provides a fully-managed of instance of Red Hat OpenShift on Microsoft Azure. OpenShift natively supports many popular technology frameworks.
For operational implementations, HeartAI endeavours to provide modern and robust security profiles. Operational orchestration with Microsoft Azure Red Hat OpenShift provides an increased ability to meet demanding security requirements. As a platform extension to Red Hat OpenShift instances, Red Hat Advanced Cluster Security (RHACS) is a Kubernetes-native security platform that provides best-in-class security integrations and solutions for container-based environments. RHACS security capabilities ensure that Kubernetes-based infrastructure is continuously protected and secure.
HeartAI instances of Red Hat OpenShift are integrated with cloud-native operational logging, monitoring, and observability software. Prometheus provides monitoring of systems and services, and Alertmanager implements event-triggered system behaviour. Grafana provides a real-time observability solution, with various adaptors to interface with data and metrics providers, allowing this data to be processed and visualised through reporting and user application dashboards.
Red Hat OpenShift implementation
Red Hat OpenShift is an enterprise-grade implementation of Kubernetes, providing a modern and secure platform for the orchestration of container-based solutions. OpenShift provides general platform-level capabilities, including:
- Abstractions for container-level deployments.
- Software orchestration that is natively cloud-based and distributable.
- Secure implementations of software-defined networking.
- Real-time and aggregated logging, monitoring, and observability.
- Frameworks for eventing and alerting.
- Controls for resource management.
- Access-level controls based around service accounts, roles, groups, and role bindings.
- Graphical user interfaces for both administrators and developers.
HeartAI deploys OpenShift with Microsoft Azure Red Hat OpenShift (ARO), a fully-managed implementation of OpenShift within Microsoft Azure. ARO is deployed to instances of Microsoft Azure cloud resources, which are fully-managed by Microsoft Azure, including operational lifecycle management, patching and updating, logging, monitoring, and security hardening.
HeartAI orchestrates system services with the Kubernetes-based Red Hat OpenShift container platform. Further information about the HeartAI implementation of Red Hat OpenShift may be found with the following documentation:
OpenShift identity provision and access control
HeartAI instances of OpenShift integrate natively with the HeartAI implementation of Keycloak. This allows users to authenticate with OpenShift through the OpenShift OAuth 2.0 and OpenID Connect framework. With support for federated identity, users will also be able to authenticate with their Microsoft Azure Active Directory identity.
Further information about the HeartAI identity and access implementation may be found with the following documentation:
The OpenShift identity provider allows role-based access control (RBAC) at the following levels:
RBAC level | Description |
---|---|
Cluster RBAC | Roles and bindings that are applicable across all projects. Cluster roles exist cluster-wide, and cluster role bindings can reference only cluster roles. |
Local RBAC | Roles and bindings that are scoped to a given project. While local roles exist only in a single project, local role bindings can reference both cluster and local roles. |
Access control is managed with the following authorisation types:
Authorisation type | Description |
---|---|
Rules | Sets of permitted verbs on a set of objects. For example, whether a user or service account can create pods. |
Roles | Collections of rules. You can associate, or bind, users and groups to multiple roles. |
Bindings | Associations between users and/or groups with a role. |
The following general roles exist within an OpenShift instance:
Role | Description |
---|---|
admin | A project manager. If used in a local binding, an admin has rights to view any resource in the project and modify any resource in the project except for quota. |
basic-user | A user that can get basic information about projects and users. |
cluster-admin | A super-user that can perform any action in any project. When bound to a user with a local binding, they have full control over quota and every action on every resource in the project. |
cluster-status | A user that can get basic cluster status information. |
edit | A user that can modify most objects in a project but does not have the power to view or modify roles or bindings. |
self-provisioner | A user that can create their own projects. |
view | A user who cannot make any modifications, but can see most objects in a project. They cannot view or modify roles or bindings. |
Corresponding roles are applied to HeartAI administrators and developers in relation to their access to the OpenShift environment.
Further information about these approaches may be found with the following external references:
OpenShift console
OpenShift provides a comprehensive graphical user interface console for both administrator and developer environments. The following image shows the OpenShift console overview, describing various cluster components, including:
- General cluster information.
- Cluster status and availability.
- Cluster resource deployment metrics.
- Real-time monitoring of resource usage, events, and alerting.
OpenShift Namespaces
OpenShift Namespaces provides cluster scoping that logical separates individual namespaces within corresponding overlay VXLAN networks. Through this approach distinct namespaces are effectively compartmentalised from each other.
Example: OpenShift declaration file for Namespace
The following example shows an OpenShift Namespace declaration file for the HeartAI HelloWorldService
production environment:
apiVersion: v1
kind: Namespace
metadata:
name: heartai-hello-world-prod
OpenShift Pods
OpenShift Pod resources are atomic and composable components of OpenShift clusters, representing the smallest logical orchestration unit. Pods are deployed to corresponding virtual IPs on the residing host network, with these networks being logically separated by VXLAN at the level of OpenShift Namespaces. Pod resource declarations may specify zero-or-more corresponding containers with associated configurations, through which the OpenShift cluster maintains the declared number of containers within the Pod host environment.
Example: OpenShift console for namespace Pods
The following image shows the OpenShift console for Pods within the HeartAI Red Hat Advanced Cluster Security production environment namespace:
Example: OpenShift console for Pod
The following image shows the Red Hat OpenShift web interface console for a Pod resource instance. The Pod web interface console provides information and functionality to support OpenShift hosting of Pod resources, including:
- The managing Namespace of the Pod resource.
- The Pod name.
- Monitoring metrics of the Pod, including:
- Memory utilisation.
- CPU utilisation.
- Filesystem utilisation.
- Network inbound bandwidth.
- Network outbound bandwidth.
- Assigned labels of the Pod.
- Pod health status.
- Pod virtual IP address assignment.
- The hosting Node of the Pod.
- The Pod creation timestamp.
- The owning resource.
- Pod-hosted containers.
- Pod-hosted volumes.
- The event history of the Pod.
OpenShift Deployments
OpenShift Deployments allow the declaration of Pod deployments and higher-level configuration for how these Pods should be orchestrated and managed within the OpenShift environment. The OpenShift Deployment Controller synchronises the actual state of the cluster to the declared state with configurable controls for how this synchronisation operates.
Example: OpenShift declaration file for Deployment
The following example shows an OpenShift Deployment declaration file for the HeartAI HelloWorldService
production environment namespace:
apiVersion: "apps/v1"
kind: Deployment
metadata:
name: heartai-service-hello-world-prod
namespace: heartai-hello-world-prod
labels:
app: heartai-service-hello-world-prod
spec:
replicas: 1
selector:
matchLabels:
app: heartai-service-hello-world-prod
strategy:
rollingUpdate:
maxSurge: 3
maxUnavailable: 0
template:
metadata:
labels:
app: heartai-service-hello-world-prod
version: v0.33.56
actorSystemName: heartai-service-hello-world-prod
annotations:
sidecar.istio.io/inject: "true"
traffic.sidecar.istio.io/includeInboundPorts: "2552,8558,14000,14020"
traffic.sidecar.istio.io/excludeOutboundPorts: "2552,8558"
spec:
containers:
- name: heartai-service-hello-world-prod
image: "quay.io/heartai/heartai-hello-world:0.33.56"
imagePullPolicy: Always
livenessProbe:
httpGet:
path: /alive
port: management
initialDelaySeconds: 20
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: management
initialDelaySeconds: 20
periodSeconds: 10
ports:
- name: remoting
containerPort: 2552
protocol: TCP
- name: management
containerPort: 8558
protocol: TCP
- name: http
containerPort: 14000
protocol: TCP
- name: https
containerPort: 14020
protocol: TCP
env:
- name: OS_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: JAVA_OPTS
value: "-Xms1024m -Xmx1024m -Dconfig.resource=production.conf"
- name: APPLICATION_SECRET
valueFrom:
secretKeyRef:
name: heartai-play-secret
key: secret
- name: AKKA_DISCOVERY_SERVICE_NAME
value: "heartai-service-hello-world-prod"
- name: AKKA_CLUSTER_BOOTSTRAP_SERVICE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: "metadata.labels['app']"
- name: REQUIRED_CONTACT_POINT_NR
value: "1"
# - name: AKKA_REMOTING_HOST
# value: "heartai-service-hello-world-prod.heartai-hello-world-prod.svc.cluster.local"
- name: AKKA_REMOTING_PORT
value: "2552"
- name: AKKA_MANAGEMENT_PORT
value: "8558"
- name: HTTP_BIND_ADDRESS
value: "0.0.0.0"
- name: HTTP_PORT
value: "14000"
- name: HTTPS_PORT
value: "14020"
- name: KAFKA_SERVICE_NAME
value: "strimzi-kafka-kafka-brokers.heartai-strimzi.svc.cluster.local:9092"
- name: KAFKA_BROKERS_SERVICE_URL
value: "strimzi-kafka-kafka-bootstrap.heartai-strimzi.svc.cluster.local:9092"
- name: POSTGRESQL_CONTACT_POINT
valueFrom:
secretKeyRef:
name: postgres-url
key: secret
- name: POSTGRESQL_USERNAME
valueFrom:
secretKeyRef:
name: postgres-id
key: secret
- name: POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-key
key: secret
resources:
limits:
cpu: 1000m
memory: 2048Mi
requests:
cpu: 200m
memory: 1048Mi
Example: OpenShift console for namespace Deployments
The following image shows the OpenShift console for Deployments within the HeartAI Red Hat Advanced Cluster Security production environment namespace:
Example: OpenShift console for Deployment
The following image shows the OpenShift console for the sensor
Deployment within the HeartAI Red Hat Advanced Cluster Security production environment namespace:
OpenShift ReplicaSets
Example: OpenShift console for namespace ReplicaSets
The following image shows the OpenShift console for ReplicaSets within the HeartAI Red Hat Advanced Cluster Security production environment namespace:
Example: OpenShift console for ReplicaSet
The following image shows the OpenShift console for the sensor
ReplicaSet within the HeartAI Red Hat Advanced Cluster Security production environment namespace:
OpenShift Secrets
Encrypted sensitive data store that is injectable at initiation / run-time to corresponding system components.
Example: OpenShift console for namespace Secrets
The following image shows the OpenShift console for Secrets within the HeartAI Red Hat Advanced Cluster Security production environment namespace:
OpenShift Services
OpenShift Services provide a cluster-internal access point to corresponding overlay network address spaces. For network routing to OpenShift Pods, Services allow consistent resolution to the virtual IP address space of one-or-more Pods, noting that such Pods may generally be transient. Access to deployment Pods through a Service is location transparent, scalable, and tolerant to Pod failure. Services may also be configured with load balancing, port-forwarding capability, and session affinity.
HeartAI services provide approaches for location transparent service discovery within corresponding HeartAI Red Hat OpenShift container platform instances. Further information about HeartAI service discovery may be found with the following documentation:
Example: OpenShift declaration file for Service
The following example shows an OpenShift Service declaration file for the HeartAI HelloWorldService
production environment namespace:
apiVersion: v1
kind: Service
metadata:
name: heartai-service-hello-world-prod
namespace: heartai-hello-world-prod
spec:
ports:
- name: http
port: 80
targetPort: 14000
selector:
app: heartai-service-hello-world-prod
type: LoadBalancer
Example: OpenShift console for namespace Services
The following image shows the OpenShift console for Services within the HeartAI Red Hat Advanced Cluster Security production environment namespace:
Example: OpenShift console for Service
The following image shows the OpenShift console for the sensor
Service within the HeartAI Red Hat Advanced Cluster Security production environment namespace:
OpenShift Routes
OpenShift Routes provide ingress points to access cluster services. A typical route refers to a cluster-internal Service that resolves to corresponding Pods. Routes may be accessed through public or private networks where appropriate.
Routes support the establishment of TLS-encrypted connections. The system OpenShift implementation natively supports the following methods of TLS connection:
TLS connection type | Description |
---|---|
Passthrough | Server certificates from downstream server passed through the edge router and presented to requesting agent. TLS encryption occurs end-to-end from requesting agent to downstream server. |
Re-encryption | The edge router presents its server certificate to the requesting agent, and the edge router itself requests a server certificate from the downstream server. TLS encryption occurs both between the requesting agent and the edge router, and the edge router and the downstream server. |
Edge termination | The edge router presents its server certificate to the the requesting agent, but the edge router does not request a server certificate from the downstream server. TLS encryption occurs between the requesting agent and the edge router, but not between the edge router and the downstream server. |
For purposes of security, HeartAI only implements the passthrough and re-encryption methods of TLS connections.
Example: OpenShift console for namespace Routes
The following image shows the OpenShift console for Routes within the HeartAI Red Hat Advanced Cluster Security production environment namespace:
Example: OpenShift console for Route
The following image shows the OpenShift console for the sensor
Service within the HeartAI Red Hat Advanced Cluster Security production environment namespace:
OpenShift PersistentVolumes
OpenShift PersistentVolumes provide methods to abstract storage provision and consumption. PersistentVolume resources are units of storage that are manually provisioned by a cluster administrator or are dynamically generated by a corresponding StorageClass. PersistentVolumes may be consumed by cluster Resources, such as Pods, but also have a lifecycle that is independent of the consumer.
For HeartAI instances of Microsoft Azure Red Hat OpenShift, PersistentVolume resources are provided by Azure Managed Disks.
Example: OpenShift console for cluster Persistent Volumes
The following image shows the Red Hat OpenShift web interface console for cluster-level PersistentVolumes. The PersistentVolumes web interface console provides:
- An overview of cluster-level PersistentVolumes, including:
- PersistentVolume names.
- The operation status of PersistentVolumes.
- The associated PersistentVolumeClaims.
- The allocated storage capacities.
- Assigned metadata labels.
- Creation timestamps.
OpenShift PrometheusRules
OpenShift PrometheusRules provide generic and extensible alerting in response to system state or event behaviour. OpenShift alerting functionality is integrated with the Prometheus monitoring solutions. Alerts are transmissible to various end-points, such as email and SMS, and forwarding functionality exists with native adapters, such as for the Splunk data-observability platform.
Example: OpenShift console for cluster PrometheusRules
The following image shows the OpenShift console for PrometheusRules within the HeartAI OpenShift cluster:
Example: OpenShift console for PrometheusRule
The following image shows the OpenShift console for the PrometheusRule NodeFileSystemAlmostOutOfSpace
:
OpenShift ServiceAccounts
OpenShift ServiceAccounts provide resources for abstracting service principles. ServiceAccounts may specify cluster- and namespace-level access control with corresponding Roles and RoleBindings.
Example: OpenShift console for namespace ServiceAccounts
The following image shows the OpenShift console for ServiceAccounts within the HeartAI Red Hat Advanced Cluster Security production environment namespace:
OpenShift Roles
OpenShift Roles provide specifications to configure access control at the cluster- or namespace-level.
Example: OpenShift declaration file for Role
The following example shows a Role declaration from the HeartAI HelloWorldService
production environment. This particular Role declaration provides the permission required for the HelloWorldService
service Deployment to be able to locate corresponding Pods of the service, fulfilling the ability of individual Pod resources to provide location transparent service discovery to other Pod resources.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: heartai-pod-reader
namespace: heartai-hello-world-prod
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "watch", "list"]
Example: OpenShift console for namespace Roles
The following image shows the OpenShift console for Roles within the HeartAI Red Hat Advanced Cluster Security production environment namespace:
OpenShift RoleBindings
OpenShift Role Bindings allow the association of Roles to corresponding ServiceAccounts.
Example: OpenShift declaration file for RoleBinding
The following example shows a Role Binding declaration from the HeartAI HelloWorldService
production environment.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-pods
namespace: heartai-hello-world-prod
subjects:
- kind: ServiceAccount
name: default
roleRef:
kind: Role
name: heartai-pod-reader
apiGroup: rbac.authorization.k8s.io
Example: OpenShift console for namespace RoleBindings
The following image shows the OpenShift console for RolesBindings within the HeartAI Red Hat Advanced Cluster Security production environment namespace:
OpenShift Nodes
OpenShift Nodes provide resource abstractions for the underlying physical or virtual machines of the cluster. For the HeartAI instances of Microsoft Azure Red Hat OpenShift, Node resources are implemented by instances of Microsoft Azure Virtual Machines.
Example: OpenShift console for cluster Nodes
The following image shows the OpenShift console for cluster Nodes:
OpenShift Machines
OpenShift Machines provide additional abstractions to Node resources specifically for OpenShift-based clusters.
Example: OpenShift console for cluster Machines
The following image shows the OpenShift console for cluster Machines:
OpenShift Operators
The following image shows the OpenShift console for cluster Machines: OpenShift Operators are extendable compositions of OpenShift resources generally, and provide a declarative framework for the composition of these resources. The Red Hat Marketplace provides a variety of Operators that are readily deployable to OpenShift clusters.
Example: OpenShift console for Operator Hub
The following image shows the OpenShift console for the Operator Hub, with filtering applied to show security-based Operators:
OpenShift Topology
OpenShift provides capabilities that are tailored for developer experiences. In addition to the administrator console, OpenShift provides a dedicated developer portal.
Example: OpenShift console for namespace topology overview
The following image shows the OpenShift developer console topology overview for the HeartAI Red Hat Advanced Cluster Security namespace:
Example: OpenShift console for namespace topology graph
The following image shows the OpenShift web interface console for a namespace topology graph. This web interface is specifically available through the developer console and provides information about an application-level Deployment. The topology graph web interface provides:
- The managing project of the Deployment.
- An overview of an application-level Deployment, including: