MCM - Netcool Ops Manager (NOM) - Installation Guide

Solution Overview

Netcool Ops Manager (NOM) is an optional component of Cloud Pak for Multicloud Management. Netcool Ops Manager consists of multiple parts. The components are:

IBM Netcool Operations Insight (NOI)
IBM Agile Service Manager (ASM)
IBM Cloud Event Management (CEM)
IBM Predictive Insights (PI)

The components can be installed on Prem, on the cloud, or in a hybrid installation. This playbook is written for the Cloud Pak for Multicloud Management (MCM), so the focus is on installing Netcool Ops Manager on OpenShift Container Platform (OCP), which can run on Prem or in the cloud.

The targetted audience for this section of the playbook is technical sales or technical services engineers who need a unified guide to install Netcool Ops Manager.

The current version 1.6.1 of NOI changes the installation method to use the Operator. It no longer requires the IBM Common Platform (ICP) Common Service. NOI 1.6.0.3 uses ICP. Because of this, you can not perform an upgrade from 1.6.0.3 to 1.6.1.

Previous Version

The previous version installation instruction is desribed in The installation instruction for NOI 1.6.0.3 including ASM 1.1.7 section.

The Operator Lifecycle Management

The NOI 1.6.1 can be installed online by using the Operator Lifecycle Management(OLM) or offline by downloading the container software first and then pushing it to the OpenShift Cluster. The latter method is typically referred to as the CLI (Command Line Interface) method, as you need to use the CLI to perform the installation.

Installation Steps.

The following are the steps to install NOI 1.6.1:

Obtain your ICR entitlement key
Prepare the Openshift cluster
Prepare the installation workstation
Prepare the LDAP server
Create the Openshift resources
Create the application secret
Install the Operator
Create the NOI Instance
Post installation steps

Each step is detailed as follows:

Obtain your ICR entitlement key

When installing through the Operator Lifecycle Management, the container will be downloaded directly from the cp.icr.io (IBM Cloud Container Registry). You first need to get your entitlement key from the IBM Cloud Container Registry. Once you got your entitlement key, you can create a Kubernetes secret. You specify the secret name in the Operator. This step is described later.

For IBM Staff, please go to My IBM Entitlement site to get your internal entitlement key.

Preparing the Openshift cluster

The Official documentation of NOI 1.6.1 mentioned that NOI 1.6.1 is supported to run on:

OpenShift Container Platform version 4.3
OpenShift Container Platform version 4.4.3

A later version of 4.4.x seems to work; in fact, Specifying Custom Resource using a form is only supported on 4.4.6 or newer.

If you install on OCP 4.3 or 4.4.5 (or earlier), you need to specify the Custom Resource (CR) using the YAML. On OCP 4.4.6 or above, you have the option to specify the CR using a form. In this playbook, we will describe the custom resource using YAML.

Ensuring your ingress controller allows traffic.

The Red Hat OpenShift ingress controller’s endpointPublishingStrategy is used to publish the Ingress controller endpoints to other networks. Depending on your OpenShift cluster host, the value of the endpointPublishingStrategy will be different, and this value can not be updated. Please see the OpenShift 4.4 documentation for more information.

If your host infrastructure is vCentre, then the endpointPublishingStrategy is HostNetwork. You can check the value by executing the following command:

oc get ingress controller default -n openshift-ingress-operator -o YAML

With endpointPublishingStrategy set to HostNetwork, to make the network policy route to work, you need to assign a selector label to the default namespace.

You can do this by executing the following command.

oc patch namespace default --type=json -p '[{"op":"add","path":"/metadata/labels","value":{"network.openshift.io/policy-group":"ingress"}}]'

Suppose you do not do the above step. In that case, the network policy may block your network traffic. You may see messages such as the 503 Application Not Available error when you access your Netcool operation insight services after the installation is completed.

Sizing

The online documentation (links provided below) provides sizing guidelines. Separate sizing guidelines are available depending on whether you are installing for a trial (PoC) or for a production environment.

CPU and Memory

Recommended minimal worker nodes sizing:

Description	Quantity
Number of worker nodes	5
Number of vCPUs per worker node	16
Minimum memory per worker node	32 GB

Information on sizing can be found at the following sites: Sizing Guidelines

Storage Capacity

If you are installing into OCP, then Rook/Ceph or Openshift Container Storage (OCS), with RADOS Block Device (RBD) storage class, is the default supported OCP Storage solution. We recommend Rook/Ceph as the dynamic storage solution for Netcool Ops Manager.

Persistent Volume Claim: If you are deploying the Openshift Container Storage (OCS), then OCS creates a default 2 TB Rook/Ceph/RDB block storage. For an initial production installation of Netcool Ops Manager, you need about 800 GB of storage (PVC) space and image-registry storage. Please take note of the storage class name. You need this later during the installation.

If you need help with installing your OCP environment, please see this playbook’s section on installing OpenShift.

If other team members configured your OCP cluster, then please ensure that they provide you with an account with a cluster administrator role.

Preparing the installation workstation

You will need to use your local workstation web browser to access the OCP Web Interface, install the Operator, and create the NOI Custom Resource.

You will also need the OCP client to help you with the installation.

Getting the oc and kubectl command lines

You download oc and kubectl from your OCP cluster. The kubectl executable is a symbolic link of the oc executable. The following documentation from Red Hat describes the steps to get started with the oc and kubectl command line interface.

Preparing the LDAP server

You need to provide details of your LDAP server for the following components:

OCP Cluster
NOI Proxy configuration.

Setting up your LDAP server is a common requirement across all Cloud Paks, so it is not detailed here.

During the installation, you will need to specify the following information, so get the information before you start the helm chart configuration:

Your Base Distinguished name.
Your LDAP URL.
Your LDAP Bind User Name and Password.

One of the pods deployed by the NOI Helm Chart is an OpenLDAP pod. You can choose to set up the OpenLDAP as a standalone repository or as a proxy to an external LDAP server.

If you choose to use the OpenLDAP pod as a proxy, then Netcool Ops Manager expects the external LDAP server to support the hierarchical LDAP structure. In particular, the NOI LDAP configuration wants to use ou (Organizational Unit).

More information on the NOI Proxy LDAP requirement can be found in the IBM Knowledge Center.

Note that since RHEL 7.4, the RHEL repository no longer distributes the OpenLDAP server. The default LDAP server for RHEL 7.5 onwards is the IPA (Identity, Policy, Audit) server. The IPA server only supports a flat LDIF structure. It does not support ou, so you can not use the IPA server as the external LDAP server for NOI. This information can be found in the Red Hat Solutions documentation.

User in LDAP.

If you are using an external LDAP server, then create the following user in the external LDAP:

smadmin - The administrative user for the dashboard.
impactadmin - The administrative user for Netcool/Impact.
icpadmin - The default ICP admin user
icpuser - The default ICP standard user

Create the Openshift resources

You need to create the following Openshift resource for the Operator.

namespace
Custom Resource name
Service Account Registry Secret.
Service Account

namespace

Create the namespace for the NOI installation. If you decide your namespace to be noi161ns then perform the following:

oc new-project noi161ns

Custom Resource name

All your pods will be prefixed with your custom resource name, so choose something short. For example, noicr.

Service Account Registry Secret.

You create a secret containing your entitlement key described earlier.

oc create secret docker-registry noi-registry-secret --docker-server=cp.icr.io --docker-username=cp --docker-password="your entitlement key from the first step above"

You specify the noi-registry-secret into the service account (next step) and into a custom resource (later).

Service Account

It is recommended to use the suggested noi-service-account, perform the following:

oc create serviceaccount noi-service-account -n noi161ns
oc adm policy add-scc-to-user privileged system:serviceaccount:noi161ns:noi-service-account
oc patch serviceaccount default -p '{"imagePullSecrets": [{"name": "noi-registry-secret"}]}'

Create the application secret

If you are using the internal OpenLDAP, then this step is optional. If you do not specify the secret, then the password will be created for you. You can get the password post-install from the Kubernetes secret.

If you are going to use the external LDAP, then you need to specify the password for:

LDAP user
smadmin
impactadmin
icpadmin

The password should match each user’s password in the external LDAP.

If you want to use a friendly password, you can either create the secret before installing or change the password after the installation.

The details of specifying the password through Kubernetes secret are described in the Configuring Authentication section of the online document.

For your convenience, the following list the command to specify the password Netcool2020 to everything, including the internal LDAP pod. Note there is an additional line for impact as the secret name should be custom-resource-impact-secret rather than the documented custom-resource-impactadmin-secret. Copy and paste this snippet and change the password, custom resource name, and namespace to your preferred value.

oc create secret generic noicr-icpadmin-secret --from-literal=ICP_ADMIN_PASSWORD=Netcool2020 --namespace noi161ns
oc create secret generic noicr-impactadmin-secret --from-literal=IMPACT_ADMIN_PASSWORD=Netcool2020 --namespace noi161ns
oc create secret generic noicr-ldap-secret --from-literal=LDAP_BIND_PASSWORD=Netcool2020 --namespace noi161ns
oc create secret generic noicr-omni-secret --from-literal=OMNIBUS_ROOT_PASSWORD=Netcool2020 --namespace noi161ns
oc create secret generic noicr-was-secret --from-literal=WAS_PASSWORD=Netcool2020 --namespace noi161ns
oc create secret generic noicr-couchdb-secret --from-literal=password=Netcool2020 --from-literal=secret=couchdb --from-literal=username=root --namespace noi161ns
oc create secret generic noicr-systemauth-secret --from-literal=password=Netcool2020 --from-literal=username=system --namespace noi161ns
oc create secret generic noicr-ibm-hdm-common-ui-session-secret --from-literal=session=Netcool2020 --namespace noi161ns
oc create secret generic noicr-cassandra-auth-secret --from-literal=username=hdm --from-literal=password=Netcool2020 --namespace noi161ns

Install the Operator

Using a browser login to the OCP Web Interface as a user with a cluster-admin role.
choose the following menu / sub-menu: Administration > Cluster Settings > Global Configurations > OperatorHub > Sources.
Click Create Catalog Source.
Specify the image URL as docker.io/ibmcom/noi-operator-catalog:1.0.0-20200620093846. Specify other details to your preference.
Click Create
After a few minutes in the Sources tab, you should see that the # of Operators should turn to 1, as per the diagram below

Go to the Main Menu, and select Operators > OperatorHub.
In the search text box, enter Netcool, and the NOI Operator should be listed, click on it and select Install.
Select the namespace that you have created earlier, do not specify the Approval Strategy > Manual, and select subscribe.
From the main menu, select Operators > Installed Operators, wait until the status says Succeeded.
You can use your workstation, perform an oc login, ensure that you are in the correct namespace perform the oc project noi161ns otherwise, do the oc get pods and you should see the noi-operator pods is running.

Create the NOI instances.

Continue from the Operators > Installed Operators select the Netcool Operations Insight Operator. You should see the following screen:

Select the Create Instances under the Cloud Deployment.
You will be presented with a YAML editor. An example of the YAML file is provided, in the example, the following are the options that had been selected:
- Custom Resource name: noicr
- Namespace: noi161ns
- antiAffinity: true
- clusterDomain: apps.yourdomain.com
- deploymenttype: trial (Enter production for production use)
- entitlementSecret: noi-registry-secret
- for internal LDAP, do not change any of the LDAP entry.
- storageClass: rook-ceph-block (the name of your ceph storage class) There are multiple locations where the storage class information is required; in the example, all pods are assigned the same storage class.
- Enable ASM
- Enable a selection of ASM Observer: Kubernetes, Docker, REST, File, vCentre.
- Disable the Topology netDisco and appDisco

Note that at any time after the installation, you can change most of the configuration by editing the custom resource noicr.

The following is the example YAML specification:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving, this file will be
# reopened with the relevant failures.
#
apiVersion: noi.ibm.com/v1beta1
kind: NOI
metadata:
  creationTimestamp: "2020-07-18T20:25:52Z"
  generation: 2

If you are installing on an Open Shift version 4.4.6 or later, you might specify the YAML content through a form that will look like the following:

Once you are ready to initialize, select the Create button.

The Operator starts by running the pod noicr-verifysecrets-* you can check using the oc get pods commands. If the verifysecrets do not complete, then you have some authorization configuration errors; otherwise, the Operator starts deploying the pods in stages.

As the pods are started, the container images will be downloaded directly from the IBM Cloud Container Registry.

You should watch for any pods that failed to load and perform an oc describe pods *failed_to_load pods* . At the pods’ initial loading, a common issue is that the cluster is running out CPU or Memory resources.

If everything is running successfully you should be able to see the list of pods similar to the following:

[jwahidin@workstation noi-operator-1.0.0]$ oc get pods
NAME                                                              READY   STATUS      RESTARTS   AGE
asm-operator-86d7867886-jtqrf                                     1/1     Running     0          37h
cem-operator-bc5bb4ff9-6jhcv                                      1/1     Running     0          37h
noi-operator-6d5786bcf4-55xw2                                     1/1     Running     0          38h
noicr-alert-action-service-alertactionservice-b988bcb76-mwb66     1/1     Running     0          37h
noicr-alert-trigger-service-alerttriggerservice-68c9f56d-5jv8k    1/1     Running     0          37h
noicr-cassandra-0                                                 1/1     Running     0          38h
noicr-common-dash-auth-im-repo-dashauth-5785bff598-p2kl7          1/1     Running     0          37h

Post-installation steps

If you check on the status of the noicr (custom resource), you can see the next steps that you can perform. It is listed here for convenience.

status:
  message: >-
    This deployment of Netcool Operation Insight is now complete. You can now
    access to the following services:
    Identify the public IP of the cluster:
      export NODE_IP=<Public IP of the ICp cluster>

Assigning roles

More information on administering users can be found in the IBM Knowledge Center

Installation instruction for NOI 1.6.0.3 including ASM 1.1.7

Edit this page on GitHub

Cloud Pak for Multicloud Management: Edge Computing Manager - Installation Guide

Cloud Pak for Multicloud Management: NOM - Day2