Extract Data from a Response

For nearly all requests (with the exception of a couple endpoints which return a CSV list), your Cybereason platform returns a response in JSON form. However, this response may contain quite a bit of data that is not of value for you in your workflow.

For example, when you retrieve a list of Malops, the response contains numerous lines of JSON data. However, you may only be interested in the root cause, Malop type, root cause name, detection time, and status of the Malop. The other fields, which include last update time, hash value of the root cause Element, and so forth, do not hold value for you.

Given this situation, you can use JSON parsing to access only the important parts of the response. Depending on what framework (cURL, REST API clients, or a script written in a programming language) you run your request, there numerous options to perform additional JSON parsing. Every programming language contains the ability to parse JSON information, extract this information, and use it elsewhere.

The examples presented in this topic use Python, but the concept of how to retrieve the information is the same for every programming language.

This example uses the use-case of performing a file search with the API. When you perform a file search in the API, you must run multiple requests. The first initiates the file search operation on your platform, and returns a batch ID number for this file search operation. The next request takes the batch ID and retrieves relevant results for that batch.

In this example, you parse the response for the first request, extract the batch ID field from the first request and use this value to run an additional request.

If you look in this example, the script adds a line - batch_id = your_response_1[‘batchId’] - which utilizes the Python syntax for finding a single value in a JSON response and assigns this value to the variable batch_id.

import json
import requests

username = "<user name>"
password = "<password>"
server = "yourserver.com"
port = "443"

data = {
      "username": username,
      "password": password
    }

headers = {"Content-Type": "application/json"}

base_url = "https://" + server + ":" + port

login_url = base_url + "/login.html"

session = requests.session()

login_response = session.post(login_url, data=data, verify=True)

print (login_response.status_code)
print (session.cookies.items())

# Request URL to start the file search operation

endpoint_url_1 = "rest/sensors/action/fileSearch"

api_url_1 = base_url + endpoint_url_1

# These are the variables that represent different keys in the first request.

file_name = "ShadowCopy.exe"
file_path = "C:\\Temp"

# This is the code that sends the first request to start the file search operation.

query = json.dumps({"filters":[],"fileFilters":[{"fieldName":"fileName","values":[file_name],"operator":"ContainsIgnoreCase"},{"fieldName":"paths","values":[file_path],"operator":"ContainsIgnoreCase"}]})

api_response_1 = session.request("POST", api_url, data=query, headers=headers)

your_response_1 = json.loads(api_response_1.content)

# This line extracts the batch_id value from the response to use in the second response.

batch_id = your_response_1['batchId']

# Request URL to retrieve file search details

endpoint_url_2 = "rest/sensors/action/fileSearch/"

# This is the code that sends the second request to retrieve the results for the selected batch.

api_url_2 = base_url + endpoint_url_2 + str(batch_id)

api_response_2 = session.request("GET", api_url, headers=headers)

your_response_2 = json.loads(api_response_2.content)

# This line prints the results for file search.

print(json.dumps(your_response_2, indent=4, sort_keys=True))

This example retrieves a limited amount of data from a response. When you send a request to retrieve investigation query results, you receive a huge volume of data in return. However, you may only want a very targeted bit of this data.

In this example, you parse the query response to list the suspicions for a specific result instance.

If you look in this example (near the bottom), there is:

A line that create a list of results. Investigation query result instances are identified by the GUID for result instance (process, file, and so forth). This line creates a results list for the results_list variable.
A second line that extracts the process name, and suspicions for that process from the response. This line then prints the results GUID, combined with the process name and suspicions.

import requests
import json

# Login information

username = "[email protected]"
password = "password"
server = "myserver.com"
port = "443"

data = {
    "username": username,
    "password": password
}

headers = {"Content-Type": "application/json"}

base_url = "https://" + server + ":" + port

login_url = base_url + "/login.html"

session = requests.session()

login_response = session.post(login_url, data=data, verify=True)

print (login_response.status_code)
print (session.cookies.items())

# Request URL

endpoint_url = "/rest/visualsearch/query/simple"

api_url = base_url + endpoint_url

# These are the variables that represent different fields in the request.

query_element = "Process"

# This is the code that sends the first request to run the investigation query.

query = json.dumps({"queryPath":[{"requestedType":query_element,"filters":[{"facetName":"hasSuspicions","values":[True]}],"isResult":True}],"totalResultLimit":1000,"perGroupLimit":100,"perFeatureLimit":100,"templateContext":"SPECIFIC","queryTimeout":120000,"customFields":["elementDisplayName","creationTime","endTime"]})

api_response = session.request("POST", api_url, data=query, headers=headers)

your_response = json.loads(api_response.content)

# This code extracts the information from the response and prints selected fields in a more accessible way.

results_list = your_response['data']['resultIdToElementDataMap'].keys()

print(your_response['data']['resultIdToElementDataMap'][0] + "- Process name: " + your_response['data']['resultIdToElementDataMap'][0]['simpleValues']['elementDisplayName']['values'][0:] + "- Suspicions: " + your_response['data']['resultIdToElementDataMap'][0]['suspicions'].keys())

This example retrieves specific data from a response. Sometimes in an investigation query, you want to retrieve the same information for multiple results. However, the challenge with the investigation query response is to parse over each results instance, find the necessary fields, and save this information.

In this example, you parse the entire response and list specific fields for all results instances.

If you look in this example (near the bottom), there is:

A line that create a list of results. Investigation query result instances are identified by the GUID for result instance (process, file, and so forth). This line creates a results list for the results_list variable.
A function that extracts the process name, and suspicions for that process from the response. This line then prints the results GUID, combined with the process name and suspicions. This function also runs repeatedly on each item in the results list, and extracts the relevant fields from each results instance.

import requests
import json

# Login information

username = "[email protected]"
password = "password"
server = "myserver.com"
port = "443"

data = {
    "username": username,
    "password": password
}

headers = {"Content-Type": "application/json"}

base_url = "https://" + server + ":" + port

login_url = base_url + "/login.html"

session = requests.session()

login_response = session.post(login_url, data=data, verify=True)

print (login_response.status_code)
print (session.cookies.items())

# Request URL

endpoint_url = "/rest/visualsearch/query/simple"

api_url = base_url + endpoint_url

# These are the variables that represent different fields in the request.

query_element = "Process"

# This is the code that sends the first request to run the investigation query.

query = json.dumps({"queryPath":[{"requestedType":query_element,"filters":[{"facetName":"hasSuspicions","values":[True]}],"isResult":True}],"totalResultLimit":1000,"perGroupLimit":100,"perFeatureLimit":100,"templateContext":"SPECIFIC","queryTimeout":120000,"customFields":["elementDisplayName","creationTime","endTime"]})

api_response = session.request("POST", api_url, data=query, headers=headers)

your_response = json.loads(api_response.content)

# This code extracts the information from the response and prints selected fields in a more accessible way.

results_list = your_response['data']['resultIdToElementDataMap'].keys()

# This is the function that iterates (repeats) for each item in the results and extracts the relevant data from the response.

def getSuspicionsList(guid_list):

  for x in guid_list:
      print(x + "- Process name: " + your_response['data']['resultIdToElementDataMap'][x]['simpleValues']['elementDisplayName']['values'][0:] + "- Suspicions: " + your_response['data']['resultIdToElementDataMap'][x]['suspicions'].keys())

getSuspicionsList(results_list)