Example script for automated alert actions via External Alerting


Recommended Posts

  • LogicMonitor Staff

Below is a PowerShell script that's a handy starting point if you want to trigger actions based on specific alert types. In a nutshell, it takes a number of parameters from each alert and has a section of if/else statements where you can specify what to do based on the alert. It leverages LogicMonitor's External Alerting feature so the script runs local to whatever Collector(s) you configure it on.

I included a couple of example actions for pinging a device and for restarting a service. It also includes some handy (optional) functions for logging as well as attaching a note to the alert in LogicMonitor.

NOTE: this script is provided as-is and you will need to customize it to suit your needs. Automated actions are something that must be approached with careful planning and caution!! LogicMonitor cannot be responsible for inadvertent consequences of using this script.

If you want try it out, here's how to get started:

  1. 1. Update the variables in the appropriate section near the top of the script with optional API credentials and/or log settings. Also change any of the if/elseif statements (starting around line #95) to suit your needs.
  2. 2. Save the script onto your Collector server under the Collector's agent/lib directory (typically "C:\Program Files (x86)\LogicMonitor\Agent\lib"). I named the file "alert_central.ps1" but feel free to call it something else.
  3. 3. In your LogicMonitor portal go to Settings, then External Alerting.
  4. 4. Click the Add button.
  5. 5. Set the 'Groups' field as needed to limit the actions to alerts from any appropriate group of resources. (Be sure the group's devices would be reachable from the Collector running the script!)
  6. 6. Choose the appropriate Collector in the 'Collector' field.
  7. 7. Set 'Delivery Mechanism' to "Script"
  8. 8. Enter the name you saved the script as (in step #2) in the 'Script' field (ex. "alert_central.ps1").
  9. 9. Paste the following into the 'Script Command Line' field (NOTE: if you add other parameters here then be sure to also add them to the 'Param' line at the top of the script):
  10.     "##ALERTID##" "##ALERTSTATUS##" "##LEVEL##" "##HOSTNAME##" "##SYSTEM.SYSNAME##" "##DSNAME##" "##INSTANCE##" "##DATAPOINT##" "##VALUE##" "##ALERTDETAILURL##" "##DPDESCRIPTION##"

    image.thumb.png.67f5e84d7a0c007c65e2c269e6ccfbe1.png
     
  11. 10. Click Save.

This uses LogicMonitor's External Alerting feature so there are some things to be aware of:

  • The Collector(s) oversee the running of the script, so be conscience to any additional overhead the script actions may cause.
  • It could take up to 60 seconds for the script to trigger from the time the alert comes in.
  • This example is a PowerShell script so best suited for Windows-based collectors, but could certainly be re-written as a shell script for Linux-based collectors.

Here's a screenshot of a cleared alert where the script auto-restarted a Windows service and attached a note based on its actions.

image.thumb.png.52f137a00c3ad60d7e026dcc4767efb7.png

Below is the PowerShell script:

# ----
# This PowerShell script can be used as a starting template for enabling
# automated remediation for alerts coming from LogicMonitor.
# In LogicMonitor, you can use the External Alerting feature to pass all alerts
# (or for a specific group of resources) to this script.
# ----
# To use this script:
#    1. Update the variables in the appropriate section below with optional API and log settings.
#    2. Save this script onto your Collector server under the Collector's agent/lib directory. I recommend naming it "alert_central.ps1" but feel free to call it something else.
#    3. In your LogicMonitor portal go to Settings, then click External Alerting.
#    4. Click the Add button.
#    5. Set the 'Groups' field as needed to limit the actions to alerts from any appropriate group of resources.
#    6. Choose the appropriate Collector in the 'Collector' field.
#    7. Set 'Delivery Mechanism' to "Script"
#    8. Enter the name you saved the PowerShell script as in the 'Script' field (ex. "alert_central.ps1").
#    9. Paste the following into the 'Script Command Line' field (including the quotes but not the "#" at the very start of the line):
#       "##ALERTID##" "##ALERTSTATUS##" "##LEVEL##" "##HOSTNAME##" "##SYSTEM.SYSNAME##" "##DSNAME##" "##INSTANCE##" "##DATAPOINT##" "##VALUE##" "##ALERTDETAILURL##" "##DPDESCRIPTION##"
#    10. Click Save.

# The following line captures alert information passed from LogicMonitor (defined in step #9 above)...
Param ($alertID = "", $alertStatus = "", $severity = "", $hostName = "", $sysName = "", $dsName = "", $instance = "", $datapoint = "", $metricValue = "", $alertURL = "", $dpDescription = "")

###--- SET THE FOLLOWING VARIABLES AS APPROPRIATE ---###
# OPTIONAL: LogicMonitor API info for updating alert notes (the API user will need "Acknowledge" permissions)...
$accessId = ''
$accessKey = ''
$company = ''
# OPTIONAL: Set a filename in the following variable if you want specific alerts logged. (example: "C:\lm_alert_central.log")...
$logFile = ''

###############################################################
## HELPER FUNCTIONS (you likely won't need to change these)  ##

# Function for logging the alert to a local text file if one was specified in the $logFile variable above...
Function LogWrite ($logstring = "")
{
    if ($logFile -ne "") {
        $tmpDate = Get-Date -Format "dddd MM/dd/yyyy HH:mm:ss"

        # Using a mutex to handle file locking if multiple instances of this script trigger at once...
        $LogMutex = New-Object System.Threading.Mutex($false, "LogMutex")
        $LogMutex.WaitOne()|out-null
        "$tmpDate, $logstring" | out-file -FilePath $logFile -Append
        $LogMutex.ReleaseMutex()|out-null
    }
}

# Function for attaching a note to the alert...
function AddNoteToAlert ($alertID = "", $note = "")
{
    # Only execute this if the appropriate API information has been set above...
    if ($accessId -ne '' -and $accessKey -ne '' -and $company -ne '') {
        # Encode the note...
        $encodedNote = $note | ConvertTo-Json

        # API and URL request details...
        $httpVerb = 'POST'
        $resourcePath = '/alert/alerts/' + $alertID + '/note'
        $url = 'https://' + $company + '.logicmonitor.com/santaba/rest' + $resourcePath
        $data = '{"ackComment":' + $encodedNote + '}'

        # Get current time in milliseconds...
        $epoch = [Math]::Round((New-TimeSpan -start (Get-Date -Date "1/1/1970") -end (Get-Date).ToUniversalTime()).TotalMilliseconds)

        # Concatenate general request details...
        $requestVars_00 = $httpVerb + $epoch + $data + $resourcePath

        # Construct signature...
        $hmac = New-Object System.Security.Cryptography.HMACSHA256
        $hmac.Key = [Text.Encoding]::UTF8.GetBytes($accessKey)
        $signatureBytes = $hmac.ComputeHash([Text.Encoding]::UTF8.GetBytes($requestVars_00))
        $signatureHex = [System.BitConverter]::ToString($signatureBytes) -replace '-'
        $signature = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($signatureHex.ToLower()))

        # Construct headers...
        $auth = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch
        $headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
        $headers.Add("Authorization",$auth)
        $headers.Add("Content-Type",'application/json')

        # Make request to add note.. 
        $response = Invoke-RestMethod -Uri $url -Method $httpVerb -Body $data -Header $headers

        # Change the following if you want to capture API errors somewhere...
        # LogWrite "API call response: $response"
    }
}

# Empty placeholder for capturing any note we might want to attach back to the alert...
$alertNote = ""


###############################################################
## CUSTOMIZE THE FOLLOWING AS NEEDED TO HANDLE SPECIFIC ALERTS FROM LOGICMONITOR...

# Actions to take if the alert is new or re-opened (note: status will be "active" or "clear")...
if ($alertStatus -eq 'active') {
	
	# Perform actions based on the type of alert...

	# Ping alerts...
	if ($dsName -eq 'Ping' -and $datapoint -eq 'PingLossPercent') {
		# Insert action to take if a device becomes unpingable. In this example we'll do a verification ping & capture the output...
		$job = ping -n 4 $sysName

		# Restore line feeds to the output...
		$job = [string]::join("`n", $job)

		# Add ping results as a note on the alert...
		$alertNote = "Automation script output: $job"


	# Restart a specific Windows service. I'm using the Print Spooler service as an example...
	} elseif ($dsName -eq 'WinService-' -and $datapoint -eq 'State' -and $instance -eq 'Print Spooler') {
		# Get an object reference to the Windows service...
		$tmpService = Get-Service -DisplayName "$instance" -ComputerName $sysName

		# Only trigger if the service is still stopped...
		if ($tmpService.Status -eq "Stopped") {
			# Start the service...
			$tmpService | Set-Service -Status Running

			# Capture the current state of the service as a note on the alert...
			$alertNote = "Attempted to auto-restart the service. Its new status is " + $tmpService.Status
		}


	} elseif ($dsName -eq 'HTTPS-' -and $datapoint -eq 'CantConnect') {
		# Insert action here to take if there's a website error.

		# Attach a note to the LogicMonitor alert...
		$alertNote = "Action X performed on this alert"
	}

}


###############################################################
###############################################################
## Final functions for backfilling notes and/or logging as needed
## (you likely won't need to change these)

# Section that updates the LogicMonitor alert if 'alertNote' is not empty...
if ($alertNote -ne "") {
	AddNoteToAlert $alertID $alertNote
}

# Log the alert (only triggers if a filename is given in the $logFile variable near the top of this script)...
LogWrite "$alertID,$alertStatus,$severity,$hostName,$sysName,$dsName,$instance,$datapoint,$metricValue,$alertURL,$dpDescription"

 

Edited by Kevin Ford
  • Like 3
  • Upvote 1
Link to post
Share on other sites
  • 3 months later...

I like this and got it working in a test environment.  Then I kicked off a collector upgrade to see what would happen to the file in the \lib directory and poof, it's gone.  :(

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.