ICAM - Post Install Setup
Now that you have installed your IBM Cloud App Management (ICAM) server, you are ready to start instrumenting the monitoring sources. ICAM has the ability to:
- Monitor your traditional resources with monitoring agents
- Monitor the kubernetes cluster to see the performance, status, and topology of the kubernetes environment
- Setup Data Collectors to instrument application runtimes like Node.js, Liberty, J2SE, etc. or monitor your HMC
- Setup Synthetic PoP (Point of Presence) Agent to synthetically test your websites and APIs
- Setup Unified Agent plugins to monitor Nginx, Redis, API Connect, and more
- Setup the Unified Agent pluins to monitor transactional traffic via zipkin, jaeger, or istio
The following document will provide detailed information on why you would use each of the monitoring agents and data collectors, the value they provide, and how to set them up.
Before you begin installing Agents and Data Collectors, you should understand the general process and how the config packs play into the installation process. The config packs are downloaded from your ICAM server and contain communications information that allow your Agents and Data Collectors communicate with the ICAM server. This includes the Hostname/IP Address of the ICAM server, ssl keys, and the tenant ID of your ICAM tenant. Since ICAM is a multi-tenant solution, the tenant ID is an important piece of information used during the Agent to Server communications. You need to download and use the config packs as part of the installation process when installing Agents and Data Collectors.
The first step is to download the config packs. To download the config packs, go to the Administration tab within your ICAM server and click on Integrations. Within the Integrations tab, you will see Incoming and Outgoing Integrations. We’re going focus on Incoming Integrations because the Agents and Data Collectors are going to be sending data to the ICAM server via HTTPS. Click on the “New Integration button”. At the top of the screen you will see 4 tiles. The role of each one is explained below:
APM V8 Agents: This config pack is used when customers want to upgrade their IBM Application Performance Management (APM) v8 Agents and connect them into the ICAM server. This is used as a upgrade/migration path for our existing APM v8 customers. Customers download the config pack and run it on their APM v8 Agents and it reconfigures the Agents so that they have all of the correct information to communicate with the ICAM server. More information can be found here: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/integ_apm_agents_intro.html
ITM/ITCAM Agents: This config pack is similar to the APM v8 config pack, but is used with customers running IBM Tivoli Monitoring (ITM) v6. You follow a similar approach to APM v8, but you must also apply a TEMA patch to update the core Agent framework prior to configuring the ITM v6 Agents to connect to the ICAM server. In addition, the config pack for ITM v6 Agents have dual path communications. This allows the ITM v6 Agent to stay connected to the ITM v6 TEMS/TEPS while simultaneously sending data to ICAM. More information can be found here: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/integ_itm_agents_intro.html
ICAM Data Collectors: The config pack for ICAM Data Collectors provides the correct ssl keys and communications information so that the Data Collectors can communicate with the ICAM server. This config pack is used for both the ICAM Data Collectors and for the Unified Agent. Download this file from the ICAM server by clicking on the “Configure” button on the tile. Each Data Collector and Unified Agent has slightly different steps for using the config pack. Follow the instructions that are documented for the Data Collector or Unified Agent.
ICAM Agents: The ICAM Agents use a slightly different approach than the Data Collectors and Unified Agent. Instead of using the config pack as part of the installation process, you reconfigure the Agent media so that includes the information from the config pack. Follow athe steps below to use the config pack:
Download the config pack by clicking “Configure” on the ICAM Agents tile
Download the Agent media from Passport Advantage
Extract the config pack tar file
Within the config pack directory, run the following command: **pre_config.sh -s <source_agent_images_dir> -d <destination_directory> -e <env_profile_dir>
source_agent_images_dir is the directory containing the source images that you want to configure. These are the Agent media packages that you downloaded from Passport Advantage
Destination_directory is the directory where you want to place the configured Agent media. The configured Agent media in the destination directory is the media that you want to install
env_profile_dir is the directory containing the env.properties file. This is the top level directory that was extracted from the config pack.
Once the Agent media has been configured, you install the Agents following the instructions in the product documetation. Then, some of the Agents must be configured per the documentation.
Our recommendation is to start by instrumenting your Kubernetes clusters with the Kubernetes Data Collector. The Kubernetest Data Collector will gather detailed information about the performance of the components of the kubernetes cluster and will generate alerts on abnormal behavior. You’ll see performance and topology information about the cluster, namespaces, nodes, pods, containers, and more.
For detailed steps on setting up the Kubernetes Data Collector, go to the section labeled [Setting up the Kubernetes Data Collector] (#setting-up-the-kubernetes-data-collector)
The next recommendation is to setup synthetic monitoring. We recommend it for two reasons. First, it is easy and you can get value very quickly. Second, it is a powerful way to ensure your applications and microservices are up and running within the expected response times. There are two types of synthetics provided by ICAM. You can test your webpages. We use Selenium to test a single webpage or to exercise a web based application. By recording a script with the selenium IDE, you can exercise a sequence of steps through the browser and simulate real interactions with the application. In addition, you can proactively test your REST APIs either with single REST requests (GET, POST, PUT, DELETE) or via a scripted sequence of API calls. You begin by setting up a synthetic PoP (Point of Presence) Agent. The Agent is installed on any machine running Docker and most customers install two or more PoP servers so that they can compare the performance from different locations.
See the section titled [Setting up Synthetics] (#setting-up-synthetics) for detailed instructions.
The next step is to begin instrumenting the workloads that are running within the kubernetes environment. Ideally, you want to gain visibility into the application and how it is performing as well as assess the transactional flows within the application. By instrumenting the application runtime with the lightweight Data Collectors, we give visibility into the Golden Signals to help you understand and analyze the performance characteristics. The runtime Data Collector collectors can also provide visibility into the topology and transactional flows within the microservices. In additional, once you start getting visibility into the topologies, you can start to analyze that topolgy for change and understand how change might be affecting your application or service.
There are a few different Data Collectors for instrumenting node.js, Liberty, Springboot, Go, and more. In addition to the topology data and Golden Signals, these data collectors allow you to drill down into detailed performance KPIs related to the app server performance characteristics. If you want additional information on instrumenting the application runtimes, see the [Lightweight Data Collectors] (#lightweight-data-collectors) section below
If it is not possible to get access to the DevOps pipeline to instrument the runtimes, there are other ways to get visibility into the topology and transactional flows. Some customers instrument their applications with zipkin or jaeger. If zipkin or jaeger are in place, then you can deploy the Unified Agent plugin for zipkin/jaeger and it will begin to track the transactions. You won’t the deep App Server metric KPIs, but you will get very detailed information about the transactions. For customers that have setup istio for their application, the istio service mesh provides a wealth of data related to the topology and transactional flows. The Unified Agent has a plugin for istio that will provide valuable transactional/topology data. The Unified Agent has additional plugins to monitor NGINX traffic, Redis, API Connect, and more. See the section on the [Unified Agent] (#unified-agent) below for more information.
ICAM ships with a large number of traditional monitoring agents that can help you monitor your on-premise applications and infrastructure. In addition, some of the monitoring agents are designed to monitor cloud infrastructure like Office 365, Amazon EC2, and Azure IaaS services. There are Agents to monitor:
- Application Servers
- 3rd party Applications
- and more
For more information on setting up Monitoring Agents, see the section below called [Traditional Monitoring Agents] (#traditional-monitoring-agents).
You will setup 1 Kubernetes Data Collector on each of the clusters that you are managing. The Data Collector is sometimes referred to as k8monitor and gets deployed as a pod within the cluster. By running inside the cluster, it is able to pull in detailed performance KPIs and topology information about the entire cluster. And, it is able to detect and alert on problems being reported by kubernetes. As long as it is a recent version of kubernetes, the Data Collector will work to gather data from OCP, ICP, open source kubernetes, and even public cloud kubernetes services such as IKS and GKS.
The Data Collector needs to be tuned based on the size of the environment being monitored. The more nodes and pods that are being monitored, the more resources the Kubernetes Data Collector will require. Rough guideslines are provided in the documentation and you can choose a Size 0 or Size 1 deployment. Sometimes it is necessary to do further tuning by editing the deployment and giving the data collector more CPU and/or Memory resources.
The Kubernetes Data Collector can be installed in multiple ways. In this section, we’ll help you choose the best options based on your environment. In addition, you’ll notice that there are multiple media options depending on your installation method.
The easiest and preferred method is to install the ICAM klusterlet on the managed cluster. This method requires that the ICP core services be running in the cluster and that helm is available. Your MCM cluster will have these services. For the klusterlet install, you will be using the media labeled as “IBM Cloud App Management V2019.4.0 for Eventing Klusterlet Config…“. Find the part number that corresponds to the platform where the worker nodes are running (AMD64 for Intel). The steps to setup the klusterlet with helm can be found here: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/install_mcm_klusterlet.html
One of the advantages of using the klusterlet is that it is tied into your MCM server. If anything changes such as the certificates, the Data Collector will continue to function properly because the certs will automatically be updated.
In an MCM environment where help and the ICP core services are not available, the recommended option is to deploy the klusterlet without helm. These steps are a bit more complicated, but you gain the advantage that certs would be updated automatically. Use the same Klusterlet media identified in the previous option. You can find the installation instructions here: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/install_mcm_klusterlet_no_helm.html
There are additional options for stand-alone ICAM environments or environments where you can’t use the klusterlet. Normally, you want to use the klusterlet installs, but if you run into trouble you can use the other installation options. When you do the Kubernetes Data Collector install, you will use different Agent media. The media will be labeled as “IBM Cloud App Management V2019.4.0 Data Collectors Install” followed by the platform.
The Kubernetes Data Collector actually has 3 different options for the installation. Each of the options is listed in priority order starting with the easiest and working down to the most difficult option. The easiest option is to use the option labeled simply as “Configuring Kubernetes Monitoring”. This option requires that both helm and ansible be available to perform the install. If helm and ansible are avialable or the customer is willing to install the, use this option because it is the easiest. Instructions can be found here: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/config_cloudresource.html
The next option is for customers that have help, but don’t have ansible. This is a little more complicated, but still relatively easy to install. Instructions can be found here: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/config_cloudresource_alt.html
The most complicated option is to manually deploy the Data Collector without helm or ansible. The option is titled “Configuring Kubernetes monitoring without Helm”. Choose this option as your last resort. Instructions can be found here: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/dc_cloudresource_nohelm.html
As mentioned previously, setting up Synthetic monitoring is extremely powerful and easy to get up and running. You will get quick time to value. Follow the instructions found here to setup the Synthetic PoP (Point of Presence) Agents: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/dc_synthetics_intro.html.
Most customers will setup at least 2 Synthetic PoP Agents. There are definitely considerations as to where to place the PoPs. Ultimately, your goal is to do 2 things:
Ensure that the application or service up and running by proactively testing it periodically. This is particularly important for applications where this is no workload during off-hours. You want to ensure that the application is performing well first thing in the morning when people start using the application.
You want to try to quickly isolate whether the performance problems are related to the backend servers or the network and which network zone(s). Choose different network subnets and locations based on where the network gear, load balancers, and HTTP servers are located.
The most expensive synthetic tests are the scripted selenium based tests. From a capacity planning perspective, you want to ensure that the PoP can execute the tests. Synthetic tests with selenium are a little bit heavy weight because they actually launch a Firefox browser to execute the test. It takes time and processing power to launch Firefox and then execute the website test. You can run 10 concurrent tests on a single PoP. If each test takes 10 seconds to execute (including the time to launch Firefox), you can execute approximately 60 tests per minute. So, pay attention to how many tests are executing and how long they take to execute. Single web page tests are less taxing. And, REST API tests are very quick and efficient. You can many more REST API requests.
To record the web based scripted tests, you download the Selenium IDE. It is an open source plugin into your browser. There are good plugin versions available for either Firefox or Chrome. There is some really goo information here about the recent Selenium IDE: https://www.ibm.com/developerworks/community/blogs/0587adbc-8477-431f-8c68-9226adea11ed/entry/Best_Practice_for_Selenium_IDE_3_2_X_3_3_X_and_Geckodriver?lang=en
Here are some Best Practices for using APM version 8 and Selenium. A lot of the content will apply to ICAM: https://developer.ibm.com/apm/docs/website-monitoring-best-practices/
Finally, a cookbook document that was written for APM v8, but again a lot of the content is applicable since it is written for Selenium: http://ibm.biz/BdHswx
After you have written your synthetic test, there are two different places within ICAM related to Synthetics. To access the Administrative tools to setup a synthetic playback, click on the “Administration” tab. Within the Administration tab, you’ll see a tile labeled “Synthetics Configure”. Click on that tile and launch the administrative tools. They will guide you through the process of creating a synthetic test. To view the results of your synthetic tests, you’ll see there is a tab at the top of the ICAM screen titled “Synthetic results”. Click on the Synethetic results tab to view the detailed results of your synthetic tests.
There are a few different types of data collectors. All of the Data Collectors are documented here: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/deploy_dc_intro.html. You have already installed the synthetic PoP. You have also installed and configured the Kubernetes Data Collector. In addition to those Data Collectors, you’ll see a Data Collector for the HMC. This Data Collector monitors the IBM Power HMC and all of the frames and LPARs managed by that HMC or SDMC.
The remaining Data Collectors are referred to as the Runtime Data Collectors. These Data Collectors get instrumented into the Application Server runtimes. See information here: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/dc_runtime_intro.html. There are Data Collectors for Liberty, node.js, Go, J2SE, etc. By instrumenting the Application Server runtimes, you will be able to see the Golden Signals, track transactions through the Application Servers, and start to visualize the topology data in the application.
There are three general approaches and installation processes for the Runtime Data Collectors.
There is an installation process for a Data Collector being installed into a non-containerized application server. These are labeled in the documentation as “on-premise” installations. The term on-premise is a little bit misleading. These Data Collector installation steps can be used in public cloud environments as well as on-premise. The important thing is that these instructions are intended for static app server environments that are not orchestrated with tools like kubernetes.
There is an installation process for a Data Collector that is running in a Kubernetes environment. You may even see separate steps for installing the Data Collector into an IBM Cloud Private environment. The steps are slightly different from a generic, open source, kubernetes environment. What is important about these step is that they must be done early in the Development cycle or built into the DevOps pipeline. You can’t instrument a kubernetes runtime after the pod has been deployed. The instrumentation must happen before the runtime is deployed into the kubernetes environment.
Finally, there are steps for customers that develop their applications with Microclimate. When using Microclimate the data collector is automatically coonfigured into to application runtime.
Each Runtime Data Collector has slightly different installation steps. The Data Collectors were designed with the domain expert/developer in mind. So, the installation steps are consistent with the domain expert’s world. For example, the Node.js Data Collector uses NPM commands as part of the installation process. Follow the installation instructions documented in the Runtime Data Collector section of the documentation for that specific domain: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/dc_runtime_intro.html
The idea behind the Unified Agent is that it is a single Agent that gets deployed, but it has plugins that can be configured to monitor multiple different monitoring domains. More and more of the new Monitoring tooling will be delivered as a plugin into the Unified Agent. Under the covers, the Unified Agent leverage telegraf and the plugins that are available in the open source telegraf community. In the future, you’ll be able to use any of the telegraf plugins. But, today, you can only use. the ones where we have developed and packaged up a supported plugin that is delivered with the Unified Agent. The overall instructions for setting up the Unified Agent can be found here: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/unifiedagent_intro.html
Currently, we have Unified Agent plugins to monitor. NGINX, Redis, API Connect, App Connect and more. Most of these Unified Agent plugins are pretty intuitive. NGINX plugin is used to monitor the NGINX server and web traffic; the API Connect plugin is designed to monitor the containerized version of API Connect; the redis plugin monitors the Redis server, etc. But, there are two plugins that should be. discussed separately because they are a little unique.
The istio plugin is used to monitor the traffic flows of microservices based applications that are configured to use istio. By tapping into the fabric mesh, we can extract traffic flow and topology data from istio. But, this only works if that applications have been configured to use istio
The zipkin and jaeger plugin is designed for customers that have. instrumented their applications with either zipkin or jaeger. Zipkin and jaeger are open source tracing standards that allow customers to add in valuable monitoring instrumentation. If the runtime is instrumented with either zipkin or jaeger, this plugin and gather that performance data and display it in a meaningful way in the ICAM user interface.
ICAM has many different monitoring Agents that can be installed and used to monitor your application servers, databases, intrastructure, hypervisors and more. The majority of the Agents are designed for monitoring traditional on-premise resources. But, there are a few that are designed to monitor cloud workloads such as Office 365, Azure, Amazon EC2, etc. When installing Agents, all of the Agents get installed into the same directory. The default directory is /opt/ibm/apm/agent on UNIX/Linux and c:\IBM\APM on Windows. As described earlier, the first step is to configure your agent media. After downloading the Agent media from Passport Advantage, process the media with the config pack and use the configured media to do the install. If you forget to configure the Agent media, the. installer will fail and tell you that the media hasn’t been configured.
There are multiple Agent packages in Passport Advantage. The core monitoring Agents like WebSphere, Oracle, MQ, OS monitoring, etc. are part of the main Agent media. You’ll. see Agent package names like “IBM Cloud App Management V2019.4.0 Agents Install xLinux English (CC4KPEN )“. You’ll notice that there is a unique package for each platform (xLinux, zLinux, pLinux LE, pLinux BE, AIX, Solaris, Windows, etc.). Download the packages that you need based on the platforms you are trying to monitor. In addition, you’ll see that there are Extension Packs for some additional specialized monitoring domains. These Extension Packs are licensed separately. Specialized domains include SAP HANA, SAP Netweaver, InfoSphere DataStage, Hadoop, etc. If you can’t find the Agent you are looking for in the main package, look at the Extension Pack to see if you agent is part of the Extension Pack.
The general process is to install the Agent. Then, some of the monitoring Agents require configuration. The configuration steps allow the Agent to communicate with and gather data from the Application, middleware, hypervisor, etc. The OS Agents do not require any configuration steps. After you install an Agent like the OS Agent that does not require configuration, the Agent will start automatically and will begin gathering performance KPIs. If the Agent requires configuration settings, then you must configure the Agent and then manually start the Agent before it will gather performance KPIs. After you configure an Agent, it will be setup to Autostart the next time the server boots, but you must manually start the Agent the first time you want it to run.
The installation process for Agents is pretty straightforward. You take the configured Agent media and extract the TAR/ZIP file. CD into the media directory. Run the installAPMAgents.sh. The installer will guide you through the processes of installing the Agents. You can select more than one Agent at a time during the install process. For example, you would probably install the OS Agent and WebSphere Agent on a WebSphere Application Server.
There a some additional options when doing the installs. The product does support Silent Installs. This allows you to automate the deployment of large numbers of Agents. Steps for performing Silent Installs can be found here: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/install_agent_silent.html
The installs can be performed as root and non-root. root installs are simpler to perform. For information on non-root installs, look at this section of the documentation: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/install_linuxaix_agent_nonroot.html
The overall installation process can be found here. Notice that there are specific installation instructions for Linux, UNIX, and Windows: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/deploy_icam_agent_intro.html
Sometimes, when performing an Agent install, the prerequisite scanner will fail and tell you that your Agent does not meet the prereqs. Usually, it means that the Agent won’t install and run properly. However, there are cases where it is simply a matter of the prerequisite checks not being updated to keep up with the product releases. For example, RedHat 8 ships and the prerequisite checker only checks for versions 6.x and 7.x. It is possible to bypass the prerequisite checker by setting an environment variable prior to runnning the install.
- UNIX/Linux: export SKIP_PRECHECK=1
- Windows: set SKIP_PRECHECK=1
Once the Agents are installed, many of them require configuration. On Windows servers, there is a GUI editor where you can configure the Agents. You can get to the GUI editor by launching the “IBM Performance Management”
Configure the Agents per the product documentation. Each Agent that requires configuration has documented steps in this portion of the documentation: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/config_agents_intro.html
Agent do have options for Silent Configuration, so that you can automate the process of setting up large numbers of Agents.
After setting up your Agents, Data Collectors, and Unified Agent, you are ready to start using the Monitoring tools. To get any value out of ICAM, you must install at least one Agent or Data Collector or Unified Agent. Otherwise, you won’t see anything useful in the user interface.
If you have setup synthetic tests, you can view the test results via the “Synthetic results” tab. If you still need to setup synthetic tests, go to the Administration tab and select the “Synthetics” tile. You can find more information on synthetics here: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/dc_synthetics_intro.html
If you want to visualize the performance KPIs for the Agents/DCs that you have setup, go to the “Resources” tab within ICAM and you can see the list of Resources. There is a filter in the upper right hand corner so that you can quickly search for specific types of resources. In the example below, the view is filtered to the Kubernetes resources. If you want, you can click on the heart next to the resource type or Group and it will show up at the top of the screen as a favorite. Click on various Resources and drill down to investigate the performance KPIs.
For more information on viewing the various types of data that are returned by the Monitoring Agents and Data Collectors, review this part of the product documentation: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/ui_monitor_intro.html
If you want, under the Admnistration tab, you can. click on the “Users and Groups” tab and add users and groups into the system. When adding Users and Groups, you assign roles to those users depending on what permissions you want them to have. The highlight level permissions is an “Operations Lead” which give you access to the Administratiive tools. An “Operator” has the fewest permissions in the tool and can primarly do read-only operations. For more information in configuring Users and Groups: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/admin_manage_users.html
If you want to organize your monitored resources into Resource Groups so that you can group related resources and perform administrative tasks against those groups, click on the “Resource Groups” tile within the Administrative tab. More information on Resource Groups can be found here: https://www.ibm.com/support/knowledgecenter/en/SSHLNR_8.1.4/com.ibm.pm.doc/install/admin_thresholds_intro.html
By default, there are hundreds of thresholds that are setup when you install the product. This will automatically detect problems in the environment by leveraging the Best Practices as defined by our domain experts. If you want to disable any of the out of the box thresholds or define your own, you can native to the Threshold Editor by clicking on the Administration tab and then selecting the “Thresholds Configured” tile. Note: The kubernetes data collector get notified by kubernetes related to problems. You will receive alerts on those problems even if you don’t have a threshold defined. More information on thresholds can be found here: https://www.ibm.com/support/knowledgecenter/en/SSHLNR_8.1.4/com.ibm.pm.doc/install/admin_thresholds_intro.html
You can explore the problems that are being detected by these thresholds by clicking on the “Incidents” tab in the ICAM user interface. Incidents can be assigned to individual users or groups and the assigned users/groups can work on those problems until they are resolved. Incidents can be resolved manually via the user interface or they will automatically close after all of the underlying Events have been fixed. For more information on Incidents review the product documentation: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/em_managingincidents.html
In some environments, you want to setup additional integrations within the ICAM tool. For example, you might want to forward the Events or Incendents to Netcool or slack. Or, you might want to configure 3rd party tools to send Events into ICAM so that they can be correlated with the Events being detected by ICAM. Inbound and Outbound integrations can be setup by clicking on the Administration tab and then selecting the “Integrations” tile. See the product documentation for more information on Integrations: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/integ_intro.html
Policies are way of customizing the way that Events and Incidents are processed. As new Events and Incidents are detected, you may want to customize the event details, send out a notification, suppress specific events and more. All of this can be achieved by selecting the “Policies” tile within the Administration tab. Find information on Event Policies here https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/em_setupeventpolicies.html and on Incident Policies here https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/em_overview.html.
In many cases, customers want improve their operational efficiency by using Runbooks. Runbooks can be a prescriptive set of steps to tell the operations team or SREs how to resolve a problem. They can be semi-automated where they include scripted automations to save time and avoid mistakes. Or, they can be fully automated so that when a problem is detected, the runback automatically corrects that problem. Runbook automations can be executed via ssh, Ansible Tower, Big Fix, or via REST API calls. Find more information on runbooks here: https://www.ibm.com/support/knowledgecenter/en/SS8G7U_19.4.0/com.ibm.app.mgmt.doc/content/em_rba.html