Gary Dewrell

Another crazy idea: Add ability to pass alert message from custom script

Recommended Posts

I have many custom data sources. I know I can only pass numeric results back to Lomo but I would love to be able to pass back to Lomo a text string that could be automatically added to a custom alert message.  This would give me the ability to include more details in an alert message.  

  • Upvote 1

Share this post


Link to post
Share on other sites

You can reference properties (including automatic properties) as tokens, but there is no facility to send arbitrary text (e.g., enum to text lookup).  Unfortunately, since the alert system is simple token substitution and has no programmatic capability, you cannot do any sort of conditional display or substitution.  I have railed on this one virtually since we started with the product, but no joy, not even a hint of a plan to fix alerting.  Our only option has been to use custom integrations with actual templating (like ERB, Template::Toolkit, etc.), but we have found various bugs (like custom email integrations do not pass ACK or SDT notices), which make it more annoying than it needs to be.

Share this post


Link to post
Share on other sites
4 hours ago, mnagel said:

You can reference properties (including automatic properties) as tokens, but there is no facility to send arbitrary text (e.g., enum to text lookup).  Unfortunately, since the alert system is simple token substitution and has no programmatic capability, you cannot do any sort of conditional display or substitution.  I have railed on this one virtually since we started with the product, but no joy, not even a hint of a plan to fix alerting.  Our only option has been to use custom integrations with actual templating (like ERB, Template::Toolkit, etc.), but we have found various bugs (like custom email integrations do not pass ACK or SDT notices), which make it more annoying than it needs to be.

I just thought of a hack that might work.  In the AD section, generate a bunch of auto properties that look like auto.enum.DSName.DPName.enumValue = enumName (substituting your DS name, DP name and enum value/name pairs, of course).  Then in your alert message you may be able to reference ##AUTO.ENUM.DSNAME.DPNAME.##VALUE####.  I have not tested whether nested token references work, but if they do, this could take away some of the pain.  It might work as well with or without AD to put the definitions into a propertysource.  Complete conjecture at this point, but I will probably whip something up to test as soon as I have some time....

Share this post


Link to post
Share on other sites

On a test run with some enum properties defined, no joy.  Maybe it can be done via an undocumented method, like braces or something, or more likely there is just no support for constructing tokens this way.

Help, I see a value of ##AUTO.ENUM.FAKEDS.FAKEDP.##VALUE####!  Waaaah!

I did create the enum mappings as a property source (could also just be manually defined at the root, but using a PS is a lot faster for many items).  This will pollute the property list pretty badly when used extensively unless there is an option to keep them hidden in the UI.  Enum support really needs to be added to LogicModules directly.

Share this post


Link to post
Share on other sites

I haven't thought of a way to do this either. Best I've come up with is for script-based DataSources is to output any text info in script output that DataPoints will ignore but will show up if you use Raw Data > Poll Now and scroll down to scriptOutput section. No where as as good as having the details in the alert message/ticket itself, but better than having to log into the problem system and work it out.

P.S. Calling LM"Lomo", ha! :)

Share this post


Link to post
Share on other sites

In addition to enum management within modules, the right fix is to replace the simple token single-pass token substitution functionality with an actual templating engine.  See https://docs.groovy-lang.org/docs/next/html/documentation/template-engines.html for a list -- LM needs to pick one and provide some sort of library function to enable management of template fragments.

Share this post


Link to post
Share on other sites

Just the ability to pass the alert message through a scripted function that returns a string back the alert handling process would meet many people's requirements in this space.  The alert message script could be part of the data source same as discovery and collection scripts. At datapoint level would be nice, but even at module level, it would be fine as we could if()...else or switch() case for data points.

Share this post


Link to post
Share on other sites

Another reason why I'd like to pass the alert message through a script first would be for SSL cert alerts. Rather than "SSL cert will expire in n days" I would prefer to have it say "SSL cert will expire on <date>".  With a alert message script we could perform that calculation.

Share this post


Link to post
Share on other sites

There are gobs of reasons to be able to transform the raw tokens into meaningful rather than vanilla output.  Most times, the output is just shy of "Something is wrong, good luck."  We created a real template system for Nagios that provided very rich conditional information with callbacks to be able to collect additional data from impacted systems and this is one thing I really miss.  LM has no apparent interest in doing anything like that, or at least has been silent in every F/R thread it has been mentioned within.  My best option so far has been to send all tokens to our ticket system (which is a maintenance headache as the integration must list every token explicitly), but then I hit a wall with email integration bugs.  Need to figure out how to put up a middle web API instead...

Share this post


Link to post
Share on other sites

LM Employee here, from the Sales Engineering team. Hello!

What if, for scripted datasources, we had a token that contained the stdout and another token that contained the stderr of the most recent poll?

You could add to the groovy telling it to call the LM API and put that kind of data as a property on the device/instance. You'd be able to get the entire raw output into a token, and then into an alarm notification.

Agreed that real templating is needed. Personally I'd love to see Jinja2.

Edited by Stuart Weenig
  • Upvote 1

Share this post


Link to post
Share on other sites
1 hour ago, Stuart Weenig said:

What if, for scripted datasources, we had a token that contained the stdout and another token that contained the stderr of the most recent poll?

You could add to the groovy telling it to call the LM API and put that kind of data as a property on the device/instance. You'd be able to get the entire raw output into a token, and then into an alarm notification.

Agreed that real templating is needed. Personally I'd love to see Jinja2.

 

For this use it seems a bit messy to me and puts the work onto the integrated system to do the filtering, as stdout might be hundred of lines. Still sounds useful to have though as a workaround, although hopefully not as an excuse to build something more proper. :)

Not sure what you mean about the property thing. Right now the only way I know of to create auto properties is from Active Discovery scripts, which only run every 15 minutes at it's quickest, and only for multi-instance, or with PropertySources which only runs once a day. I think that would add too much delay and you may not know which message went with which alert (if it's flapping).

Share this post


Link to post
Share on other sites

Yes, it would be messy, but looking for minimum viable product to get moving in the right direction. If we had stdout/stderr as tokens, it wouldn't take much to build processing to loop over the lines, apply regex, push through a template, etc.

What i mean with the properties thing is to have the groovy script actually call the LM API as part of its execution. It would do an HTTP patch to the /device/devices/{deviceId}/devicedatasources/{hdsId}/instances or /device/devices/{deviceId}/properties/{name} endpoint (to add as instance or device property, respectively). The information pushed to the LM API in that HTTP patch would be the value of some variable in the groovy script. 

For example:

Say you have a groovy based datasource: "IsItOkay", that gathers some data from the device. One of the pieces of data it gathers is the message you would want included in the alarm. Let's say the groovy script stores this message in a variable called `message`. After the part of the script that does the data collection finishes, you would have a section of the code that would do an HTTP Patch to the LM API updating/creating a property called `IsItOkay.detailedMessage` with the value of the `message` variable. The script would finish.

Once the script is finished, the data is compared to the threshold and since something is bad, it would open an alarm. The alarm body would reference all the tokens it references today with the addition of the `IsItOkay.detailedMessage` property. 

Share this post


Link to post
Share on other sites

ok, I gotcha.

I always worry when the answer becomes having LM call it's own API. It's a powerful option but I feel is a hacky workaround and becomes an excuse to not add functionality to the product directly. Need to know how many alerts a customer has? write a datasource that queries the API. Dashboard widget not provide the data you need? Write a datasource that directly modifies widgets...etc. Also as an MSP, I don't own the collector systems and concerned that the customer would be able to extract the API token (which must be write-enabled) and able to modify LM themselves (or their part of it).

But I'm all for having options and moving in the right direction though.

Share this post


Link to post
Share on other sites

Yeah, it's not great that we have to resort to it as often as we do. I'm working internally to get a new type of "collector" (i.e. the dropdown when building a datasource where you normally pick SNMP, WMI, BATCHSCRIPT, JDBC, etc.) called LMAPI. The idea would be to make it much easier to just call an LM API resource (without having to setup all the things you have to now) and parse through the data with a simple groovy script. 

One thing I do with my LM API calls (well for all API calls that aren't associated with a particular device) is that I create a resource in the portal called "[portalname].logicmonitor.com". Among several advantages is the ability to monitor that resource with an auto balanced collector group, meaning that collectors can go offline without suddenly losing the data. Ownership of the keys is also addressed there.

  • Upvote 1

Share this post


Link to post
Share on other sites

Would be good to have STDOUT and STDERR as standard tokens. Though as they could contain a lot of content, would need a special view, opens in a new dialog to scroll through the content ideally in a console font, rather than the current method of viewing properties using the Info tab.

Share this post


Link to post
Share on other sites

Having them as tokens is different than storing them as properties. It's pretty easy to have active discovery scripts set properties for stdout and stderr, but as you mention, the info tab would get even more cluttered.

But if they were available as tokens, they could be used in alarm messages. 

Share this post


Link to post
Share on other sites

Yes, that would make what I described above much easier. I mentioned it before, but I'm lobbying internally for there to be baked in methods for doing lm_api gets. That would pave the way for lm_api RW methods like lm_api.setProp("propname","propvalue"). Obviously, this only need to be added for use within the collection script as the discovery script already has this available for instance level properties and property sources has it available for device level properties.

That FR gets a +1 from me.

Edited by Stuart Weenig

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.