Skip to main contentCloud Paks At Work 

Configuring Incident Similarity for Watson AIOps AI Manager

In this section we describe the configuration of Incident Similarity for Watson AIOps AI Manager on IBM RedHat OpenShift Kubernetes Service (ROKS). This document was written based on our experience configuring Incident Similarity for AI Manager v2.1 on Red Hat OpenShift 4.5.

Exporting Incidents from ServiceNow

This section will describe how to manually export incident data from ServiceNow.

Log in to ServiceNow and using the Filter Navigator search for Incidents. Select All under Incident to bring up a list of all the incidents.

sn all

Filter the list to include only records where State is Closed. Incidents records where State is Resolved are skipped/discarded by AI Manager consumption process. If we want to include resolved incidents, we can include them in the filter here, but we will need to massage the records and set the state field to Closed before loading them in to AI Manager.

Keep in mind that most ServiceNow instances will put an upper limit on the number of records that can be exported through the UI (e.g. 10000). If the number of records we want to export are greater than this upper bound, we can filter the list according to date in order to reduce the number of records. We can then export them in chunks that stay under the record limit. For example, first export is incident records from current year, second export is incidents from previous year, etc.

Click the cog wheel to personalize the list columns. Include the following columns, if not already included.

Column
Number
Opened
Short Description
Priority
State
Updated
Business duration
Closed
Resolved
Created
Description
Parent Incident
Problem
Resolution notes
Severity
Impact
Configuration Item
Comments and Work notes

To export the records, right click on any column heading, go to Export and select CSV. You must select CSV as the export format, as it’s the only format that reserves all the comments. After the export is complete, click Download to download the exported records in CSV format.

Converting to JSON

The next step is to take the downloaded CSV file(s) from ServiceNow and convert them to “JSON Lines” format for uploading to AI Manager.

A Python script is provided here to assist in conversion from CSV to JSONL. Note, that this script is a beta helper script and will most likely need customized to your environment.

The script requires Python be installed on your workstation.

Download incidents.py and rename the source CSV file to incidents.csv placing it in the same directory.

Run the following to generate JSON lines formatted output.

python incident.py > incidents.json

incidents.json is the JSON lines formatted file with properly formatted fields for consumption in AI manager.

Copy JSON file to MinIO

The next step is to copy the file to MinIO.

Compress via gzip the file before copying to the model-train console so that the transfer will be quicker. Call the file incidents.json.gz. Copy the file to the model-train console.

oc cp incidents.json.gz $(oc get pods -l app.kubernetes.io/component=model-train-console -o jsonpath='{ .items[*].metadata.name }'):/home/zeno/

Log in to the model-train console.

oc exec -it $(oc get pods -l app.kubernetes.io/component=model-train-console -o jsonpath='{ .items[*].metadata.name }') -- bash

Once logged in to the model-train console, decompress the JSON file.

gunzip incidents.json.gz

Create a MinIO bucket for $SIMILAR_INCIDENTS if not done previously. If the bucket is already defined, we can ignore the BucketAlreadyOwnedByYou error message.

aws s3 mb s3://$SIMILAR_INCIDENTS

Copy incidents.json file to MinIO.

aws s3 cp incidents.json s3://$SIMILAR_INCIDENTS

Train the similar incidents model

From the model-train console, navigate to the incident directory:

cd incident

Back up the original index_incident.sh file.

mv index_incident.sh index_incidents.orig

For v2.1, there is an updated script for launching the training published in the Knowledge Center documentation. View index_incident.sh, copy the text, and paste it in file /home/zeno/incident/index_incident.sh in the model-train console.

There is a line near the end of the script that deploys the model once the training is complete. On that line, the name of the script needs changed from deploy_model.py to deploy_model.pyc (add a c to the end). If this is not done, we will see an error message at the end of the script output and the model will not be deployed.

Assign executable permission to the script.

chmod +x index_incidents.sh

In order to start the training job, we need the application_group_id and the application_id for the application. These IDs are generated by AI Manager when a new application (and group) are defined in the console. The values of these IDs can be found in the console. If we go to the application page in the console, the values are found in the URL or we can start an insights model wizard and find the values in the instructions.

Run the following command to train the incident similarity model, using the values for app_group_id and app_id appropriate to your environment. We are using nohup to start the process in the background

app_group_id=ghgjy5um; app_id=71cyvyal; nohup bash index_incidents.sh s3://$SIMILAR_INCIDENTS/incidents.json $app_group_id $app_id >nohup.out 2>&1 </dev/null &

This can run for some time, in the sample below there were almost 10,000 incidents and the runtime was nearly 30 minutes.

<user1>$ app_group_id=ghgjy5um; app_id=71cyvyal; bash index_incidents.sh s3://$SIMILAR_INCIDENTS/incidents.json $app_group_id $app_id
Downloading data file from s3://similar-incident-service/incidents.json to /tmp/tmp.rAX5IVwwsP
download: s3://similar-incident-service/incidents.json to ../../../tmp/tmp.rAX5IVwwsP
Successfully downloaded file
Normalizing incients from /tmp/tmp.rAX5IVwwsP to /tmp/tmp.rAX5IVwwsP.out
Indexing incients from /tmp/tmp.rAX5IVwwsP.out
curl -X POST "https://db2-aimanager-aio-similar-incidents-service.aimgr21.svc:8000/v2/upload_incidents/ghgjy5um/71cyvyal?init_es=true&need_normalization=false" -H "accept: application/json" -H "Content-Type: multipart/form-data" -F "file=@/tmp/tmp.rAX5IVwwsP.out;type=application/json" -v --cacert /etc/ssl/certs/ca-root-cert.pem
Note: Unnecessary use of -X or --request, POST is already inferred.

Verify model

Run the following command inside of the model-train console to verify that the model was trained and is inside ElasticSearch database.

curl -s -u $ES_USERNAME:$ES_PASSWORD -XGET $ES_URL/_cat/indices --insecure | grep normalized-incidents

The example below shows expected output.

<user1>$ curl -s -u $ES_USERNAME:$ES_PASSWORD -XGET $ES_URL/_cat/indices --insecure | grep normalized-incidents
green open normalized-incidents-ghgjy5um-71cyvyal PEcA5QFGQkuha6zct8mhBA 1 1 9523 0 62.9mb 30.7mb

We can also verify the model by drilling down the application in the console and viewing the Insight Models tab. There we should see Status Running for the Similar incident model and version 0.

Congratulations, we have configured incident similarity for AI Manager!