• 0

Making a Backup for Cisco ISE in LogicMonitor


Question

I am trying to setup a ConfigSource in LogicMonitor for Cisco ISE backups. We want the backup to only pull when a change is made in ISE. The out-of-the-box Cisco_IOS and Cisco_NXOS ConfigSources don't seem to work for ISE. So I tried to make a ConfigSource from the LM page https://www.logicmonitor.com/support/logicmodules/articles/creating-a-configsource but I haven't been able to get it to work properly. I made a copy of Cisco_IOS so I could have the Config Checks and removed the scripts. I have the following:

AppliesTo:

(
    ( 
      startsWith( system.sysinfo, "Cisco Identity Services Engine" )
    ) 
  ) &&
  ( 
    (ssh.user && ssh.pass ) || 
    (config.user && config.pass) 
)

 

Parameters:

import com.santaba.agent.groovyapi.expect.Expect

host = hostProps.get("system.hostname");
user = hostProps.get("config.user");
pass = hostProps.get("config.pass");

cli=Expect.open(host, user, pass);
cli.expect("#");

cli.send("terminal length 0\n");
cli.expect("#");

cli.send("show running-config all\n");
cli.expect(/Current configuration.*\n/);

cli.send("exit\n");
cli.expect("#exit");

config=cli.before();
config = config.replaceAll(/ntp clock-period \d+/,"ntp clock-period ");

cli.expectClose();
println config;

 

Test comes back with the error: 

The script failed, elapsed time: 60 seconds - End of stream reached, no match found
java.io.IOException: End of stream reached, no match found

 

Does anyone have any ideas? 

Link to post
Share on other sites

Recommended Posts

  • 1
  • Administrators
25 minutes ago, grantae said:

End of stream reached, no match found

This usually means that you've sent some command to the device and are waiting for a particular response that doesn't come. The lines in your code that are waiting are the cli.expect() ones. It's difficult to tell which one timed out. You can add some println statements before/after them to know which ones executed successfully. println(cli.before()) is especially handy in determining what has come back from the device.

  • Like 1
Link to post
Share on other sites
  • 1
  • Administrators

I'd suggest using before the first cli.expect statement, in case that first expect statement is the one expecting the "#" and never getting it back from the device. You might want to put a sleep statement (sleep(n) where n is the number of milliseconds the script should wait before executing the next line) between logging in and doing the println(cli.before()) so that the device has a chance to send something back to you after logging in.

A lower level way to work through this is to replicate the activity of the script using your own SSH client (e.g. putty on Windows or terminal in Linux/Mac). Log in using the same credentials, make a note of what get sent to you when you log in. After providing the username and password, are you sent directly to the prompt? How do you recognize what the prompt is? Think about how you use your human brain to understand that the device is waiting for your input. Then, what command do you send and what do you expect back? When you get that back, how do you (as a human) recognize that the device has sent you everything it's going to send? Usually because it returns to the prompt. You need to teach your script not only to send the command but also to recognize the output coming back from the device and know when the device is done sending the response back and is waiting for another command.

In this case, the script (trying to emulate human behavior) is looking for a #, likely because the prompt probably uses something like "my-cisco-ise# " as the prompt. 

 

Welcome to the deep end where you get to learn groovy and expect in one step. 😉

 

  • Like 1
Link to post
Share on other sites
  • 1
  • Administrators

Groovy doesn't require semicolons. Also, you can't printlin(cli.before()) before you define what cli actually is.

If you do:

cli=Expect.open(host, user, pass);
println(cli.before())

You need to give your device some time to respond before printing out what has been received so far. I also just remembered that you can do println(cli.stdout()) as well. So, maybe something like this:

cli=Expect.open(host, user, pass);
sleep(10000)
println(cli.stdout())

Which would login, wait for 10 seconds for the device to log you in and present the command prompt. Then println(cli.stdout()) prints out everything that has been sent back and forth to the moment.

  • Like 1
Link to post
Share on other sites
  • 1
  • Administrators

Perhaps I should explain what these lines do:

/*This logs into the device. Once logon happens, the script can send text to the 
device and the device can send text back to the script. Any text sent back and
forth will be stored in cli.stdout().*/
cli=Expect.open(host, user, pass);

/*This sleeps for 10 seconds. The purpose of this is to give your device time to send your script 
something now that it's logged on. This can include a message of the day, logon banner, etc., but also 
includes (hopefully eventually) a prompt: some text indicating that the device is ready to receive a 
command from the script.*/
sleep(10000)
  
/*When expect runs, it is constantly listening to the device to see if anything comes in. This line prints 
out the contents of cli.before(), which is the holding place for any text received before the most recent 
expect match...*/
println(cli.before())

/* This is an expect match. This tells the script to look for a # sign in the text streaming from the 
device. Usually, this is used to signal to the script that the device is ready to receive commands. It's 
ready to receive commands because the device sent back the prompt which is usually like "my-cisco-ice#"
When the script sees the # sign in the text coming from the device, it takes everything before that pound 
sign and puts it into cli.before()*/
cli.expect("#");

In each of your attempts, you had cli.expect("#"). It would seem that the prompt for this device doesn't contain the pound sign so the script isn't not finding it. The script is waiting for a prompt with a # in it, but the device doesn't use the # sign (or it doesn't in the current mode) in its prompt.  Because the script never finds it, the script exits with a non-zero exit code. When testing a script from the UI, a non-zero exit code doesn't actually display the output. In order to get that, you'll need to go into the collector console and issue a !groovy command. This will give you a dialog box where you can pick a device and run a groovy script. Then it'll give you the whole output.

I'm doing this against a linux box. The prompt is like this. I found this out simply by manually logging into the device and looking at what the prompt was. Notice that mine does not contain a #. If my script is expecting the device to send a #, the script will timeout waiting for something that will never get sent.

[sweenig@staticcollector ~]$ 

So, i've setup my expect statements to be slightly different. I had to create a regex expression to match on the prompt. I used https://www.regexpal.com/ to make sure my expression caught the whole thing and nothing else from my manual session.

I ran this script against one of my linux devices (centos). 

import com.santaba.agent.groovyapi.expect.Expect

host = hostProps.get("system.hostname")
user = hostProps.get("ssh.user")
pass = hostProps.get("ssh.pass")

prompt = /\[.+\]\$\s/ //this is the regex that identifies my prompt "[sweenig@staticcollector ~]$ "

cli=Expect.open(host, user, pass) //connect
cli.expect(prompt) //Waiting for prompt...
cli.send("ls -l ~ \n") //Issuing command
cli.expect(prompt) //Waiting for prompt...

println("Full Output:\n" + "="*60)
println(cli.stdout())
println("="*60)

println("Just the output between the prompts:\n" + "="*60)
println(cli.before())
println("="*60)

return 0

 

This is what the output looks like:

returns 0
output:
Full Output:
============================================================
Last login: Mon Aug 31 17:07:19 2020 from localhost
[sweenig@staticcollector ~]$ ls -l ~ 
total 278764
-rwxrwxr-x. 1 sweenig sweenig     23284 Mar 25 04:15 LogicmonitorBootstrap64_6.bin
-rwxr-xr-x. 1 root    root    285427235 Mar 25 04:17 LogicmonitorSetup.bin
[sweenig@staticcollector ~]$ 
============================================================
Just the output between the prompts:
============================================================
ls -l ~ 
total 278764
-rwxrwxr-x. 1 sweenig sweenig     23284 Mar 25 04:15 LogicmonitorBootstrap64_6.bin
-rwxr-xr-x. 1 root    root    285427235 Mar 25 04:17 LogicmonitorSetup.bin

============================================================

 

The key is that my cli.expect statements were looking for a prompt that actually exists in the text returned by the device.

  • Like 1
Link to post
Share on other sites
  • 1
  • Administrators
22 minutes ago, grantae said:

Not sure what cli.expect() I should use for the line following cli.send("exit\n")

Once you've done what you need to do to log off the device a simple "cli.expectClose()" is all that's needed. At that point, you don't need to wait for anything else before closing the SSH connection. I added some comments to your code. 

Something to understand: cli.before() grabs everything between the most recent cli.expect() and the previous cli.expect(). Since you had "cli.expect("[screen is terminating]");" before the cli.before(), it was trying to grab the data after it had already exited the buffer. Does that make sense? Your cli.before() needs to be right after you know the config has been completely received. 

import com.santaba.agent.groovyapi.expect.Expect
host = hostProps.get("system.hostname");
user = hostProps.get("config.user");
pass = hostProps.get("config.pass");
prompt = /.+\#\s/ //<---this may be problematic as it might match on the # and everything before it. You might need to be more specific.

cli=Expect.open(host, user, pass);
cli.expect(prompt);

cli.send("terminal length 0\n");
cli.expect(prompt);

cli.send("show running-config\n");
//cli.expect(/Current configuration.*\n/); //<--If the previous command ends at the prompt, then just expect the prompt.
cli.expect(prompt)

cli.send("exit\n");
//cli.expect("[screen is terminating]"); //You've already exited, no need to wait for anything before closing the connection. You probably wouldn't get it anyway.

config=cli.before(); //println(cli.before()) should be just fine. However, it might include "show running-config" on the first line, which is technically not part of the config.
config = config.replaceAll(/ntp clock-period \d+/,"ntp clock-period "); //<--- Not sure ntp clock period is different every time you show the config. This removes the difference so that LM doesn't alert you that the config has been changed. There are better ways to do this. Just create an exception in the configcheck.

cli.expectClose();
println config;

 

  • Like 1
Link to post
Share on other sites
  • 1
  • Administrators

It seems that you've setup the ConfigSource as multi-instance (didn't you clone from something else?). You probably don't want this since there is only one config file on the device you're interested in.

You have a couple options, you can:

  • Create a new ConfigSource and copy the parameters from your clone into it without selecting multi-instance
  • Hard code the active discovery to just discover the one instance. You'd do this by changing your discovery script to:
println("0##Running Configuration")

 

Then put the Expect script in the collection script field (not the Active Discovery field).

Also, you should put a "return 0" at the end of your script to let LM know that the script finished successfully.

  • Like 1
Link to post
Share on other sites
  • 1
  • Administrators

It could still be an issue. Your prompt can use string substitution just like a double quoted string:

prompt = /${host}\/${user}\#\s/

host and user were defined earlier in the script, so if they don't contain the right contents, you can pull other properties that do.

hostProps.get("system.hostname") pulls the system.hostname property, which is the DNS/IP used to communicate with the device, not necessarily the actual device's hostname. For that, you might want to use system.displayname. Just depends on what the values are of the properties on the device in question.

I happened to be looking right at it when you posted it. 😉 

  • Like 1
Link to post
Share on other sites
  • 1
  • Administrators

Look in the resource tree under the device and you should see a C inside a circle with the display name of the ConfigSource. You should be able to click on it and go to the "Configs" tab (if not already displayed when you click on it. That page has the details showing when it was last checked. We show only the deltas and don't actually store the whole thing, just the original and the deltas.

  • Like 1
Link to post
Share on other sites
  • 1
  • Administrators

LM doesn't currently have any way of using the gathered config to push back to the device. So, you'd be manually copying and pasting into the device. The expectation is that you would know not to include that.

Alternatively, you can modify your script to exclude the "show running-config" from the output. (recommended for your benefit)

/* Instead of this
println(cli.before());

Do this: */
cli.before().eachLine{
  if(!(it=~/show running-config/)){
    println(it)
  }
}
//Or something close to that. YMMV

 

  • Like 1
Link to post
Share on other sites
  • 1
  • Administrators

"it" is the loop variable. So, with cli.before().eachLine{}, you're looping through each line of the cli.before() text. So, "it" is the current line.  It's checking to see if the current line doesn't (that's the !) match "show running-config". 

  • Like 1
Link to post
Share on other sites
  • 1
  • Administrators

If you don't specify a loop variable, Groovy decides on one for you and calls it "it". You could specify it differently like this:

cli.before().eachLine{ loopvariable ->
  if(!(loopvariable=~/show running-config/)){
    println(loopvariable)
  }
}

 

Or

cli.before().eachLine{ line -> 
  if(!(line=~/show running-config/)){
    println(line)
  }
}

 

28 minutes ago, grantae said:

Also did we have to use ~/show running-config/ or was there a non-regex way to identify the string? Curious since regex is new to me so I had to Google the ~ thing. It said it forced a regex string? Just want to know more about it.

That's just habit on my side. You could have done it=="show running-config".

  • Like 1
Link to post
Share on other sites
  • 0

Thank you for the suggestion. This is the first time I am trying to script with Groovy and my overall scripting experience is pretty low, so forget me if I didn't use println(cli.before()) correctly.

------------

import com.santaba.agent.groovyapi.expect.Expect

host = hostProps.get("system.hostname");
user = hostProps.get("config.user");
pass = hostProps.get("config.pass");

cli=Expect.open(host, user, pass);
cli.expect("#");
println(cli.before())     <-------------------------

cli.send("terminal length 0\n");
cli.expect("#");

cli.send("show running-config all\n");
cli.expect(/Current configuration.*\n/);

cli.send("exit\n");
cli.expect("#exit");

config=cli.before();
config = config.replaceAll(/ntp clock-period \d+/,"ntp clock-period ");

cli.expectClose();
println config;

----------------------

With it added after the first section, I still got the same error. No change. I also tried the line after each section with no change.

Did I just use it wrong or is the script failing immediately? 

Link to post
Share on other sites
  • 0

Thank you for helping me learn more about groovy and expect. 

----------------------------


cli=Expect.open(host, user, pass);
println(cli.before())
cli.expect("#");

Results in the same error.

-----------------------------

println(cli.before())
cli=Expect.open(host, user, pass);
cli.expect("#");

Results:

The script failed, elapsed time: 0 seconds - No such property: cli for class: Script129
Possible solutions: class
groovy.lang.MissingPropertyException: No such property: cli for class: Script129
Possible solutions: class
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:66)
	at org.codehaus.groovy.runtime.callsite.PogoGetPropertySite.getProperty(PogoGetPropertySite.java:51)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGroovyObjectGetProperty(AbstractCallSite.java:310)
	at Script129.run(Script129.groovy:7)

So I don't think it works there.

--------------------------------

Noticed I didn't put a ; after the println(cli.before()) so tried 

cli=Expect.open(host, user, pass);
cli.expect("#");
println(cli.before());

and got:

The script failed, elapsed time: 60 seconds - Timeout
com.santaba.agent.groovyapi.expect.expectj.TimeoutException: Timeout

--------------------------------- (Surprised it ran without the semicolon)

I did something similar to your PuTTY suggestion when I was checking if the password worked. I logged in with the user I made for LogicMonitor to verify the password and it goes to: "<ISE-hostname>/<username>#". So I believe cli.expect("#") would be correct. I also noted that "show logins cli" shows the LogicMonitor account logging in from the LogicMonitor IP for 00:01 a few times (Probably from running the test script).

Link to post
Share on other sites
  • 0

cli=Expect.open(host, user, pass);
sleep(10000)
println(cli.stdout())
cli.expect("#");

Results: 

The script failed, elapsed time: 60 seconds - End of stream reached, no match found
java.io.IOException: End of stream reached, no match found

-----------------------------

cli=Expect.open(host, user, pass);
cli.expect("#");
sleep(10000)
println(cli.stdout())

Results:

The script failed, elapsed time: 60 seconds - Timeout
com.santaba.agent.groovyapi.expect.expectj.TimeoutException: Timeout

--------------------------

cli=Expect.open(host, user, pass);
println(cli.before())
cli.expect("#");

Results: 

The script failed, elapsed time: 60 seconds - Timeout
com.santaba.agent.groovyapi.expect.expectj.TimeoutException: Timeout

------------------------

cli=Expect.open(host, user, pass);
sleep(10000)
println(cli.before())
cli.expect("#");

Results: 

The script failed, elapsed time: 60 seconds - End of stream reached, no match found
java.io.IOException: End of stream reached, no match found

------------------

They all come back with the same error. Am I missing an import or something?

Link to post
Share on other sites
  • 0

OK, that helps explain the prompt thing. I was thinking it just needed to end # not that it was looking for the whole prompt. So I used https://www.regexpal.com/ to verify a correct prompt. I also typed through my cli.send while logged in as the LogicMonitor account. 

*login*
terminal length 0
show running-config
exit

----------------------

When I exit the device the follow displays:

<host/user># exit

[screen is terminating]
Connection to <host> closed.
bash-4.3$

Not sure what cli.expect() I should use for the line following cli.send("exit\n")

---------------------My Current Script---------------------------

import com.santaba.agent.groovyapi.expect.Expect

host = hostProps.get("system.hostname");
user = hostProps.get("config.user");
pass = hostProps.get("config.pass");

prompt = /.+\#\s/

cli=Expect.open(host, user, pass);   <----login
cli.expect(prompt); <----wait for prompt

cli.send("terminal length 0\n"); <----Set so show run shows all at one time
cli.expect(prompt); <----wait for prompt

cli.send("show running-config\n");       <------------ Changed from cli.send("show running-config all\n"); since show running-config all isn't a command
cli.expect(/Current configuration.*\n/); <--Guess this one is expecting the config (Currently with the terminal length 0 show run ends on the prompt)

cli.send("exit\n");  <---- Exit the device
cli.expect("[screen is terminating]"); <----- Not sure what to expect here

config=cli.before(); <------ Seems unnecessary (just change println config to println(cli.before()) )
config = config.replaceAll(/ntp clock-period \d+/,"ntp clock-period "); <--- Not sure 

cli.expectClose();
println config;

------------------------------------------------------------------------

I've played with variations of the above script and tried /* commenting */ parts of it out with println(cli.before()) and/or println(cli.stdout()) added in different sections. Also tried ending it with return 0 and switching the config.user and config.pass to ssh.user and ssh.pass. 

Still getting: 

The script failed, elapsed time: 60 seconds - End of stream reached, no match found
java.io.IOException: End of stream reached, no match found

Earlier in the day I also sometimes got a different error, but now I cannot get the error to come up again. It Was something about timing out after 47 seconds. 

Link to post
Share on other sites
  • 0

import com.santaba.agent.groovyapi.expect.Expect

host = hostProps.get("system.hostname");
user = hostProps.get("config.user");
pass = hostProps.get("config.pass");

prompt = /.+\#\s/

cli=Expect.open(host, user, pass);
cli.expect(prompt);

cli.send("terminal length 0\n");
cli.expect(prompt);

cli.send("show running-config\n");
cli.expect(prompt);
println(cli.before())

cli.send("exit\n");
cli.expectClose();

Result:

Success: There would be no instances discovered for the selected device.

Not sure what is happening, was expecting a print out of show run to return. Maybe my Config Checks are stopping it?

---------------------------------------------------------------------------------------------------------------

I have the following Config Checks (took from Cisco_IOS):

image.png.f626f75580ccb725d16464a8279d20ad.pngimage.png.17e9b1c0f527b3f7a09dda442cc2fc42.png

 

 

 

Link to post
Share on other sites
  • 0

Update: I removed the Config Checks and ran it, but I still get the same results. Maybe it is an issue with the prompt like you mentioned earlier? The prompt is "HOSTNAME/USERname# " so would I be able to make prompt /<host/user>\#\s/? (Not sure how to call host / user if I even can.)

Wow you replied fast. lol Thanks 

Edited by grantae
Link to post
Share on other sites
  • 0

Prompt:

HOSTNAME/USERname# 

Breakdown of /${host}\/${user}\#\s/:

/        <-Start
${host}  <- host defined earlier
\/        <- matches /
${user}  <- user defined earlier
\#        <- matches #
\s        <- space
/        <-End

----------------------

host = hostProps.get("system.hostname");
user = hostProps.get("config.user");
pass = hostProps.get("config.pass");
display = hostProps.get("system.displayname");

//prompt = /.+\#\s/
prompt = /${display}\/${user}\#\s/

Let me try this.

  • Like 1
Link to post
Share on other sites
  • 0

That seemed to work. Aside from some weirdness like ! and removing show running-config. I think I can play with the Config Checks to fix that. (They look like the address that issue, mostly.)

When this script actually runs instead of "Test Script" does it use the Config Checks, but not in the test? 

Link to post
Share on other sites
  • 0

How do I check if the ConfigSource is actually running every hour?

Where is the backup saved when it runs?

I want to make sure the final product looks correct.

Also thank you so much for all your help. I wouldn't have been able to do this without your help. It was a great learning experience. 

Link to post
Share on other sites
  • 0

Final Product:

import com.santaba.agent.groovyapi.expect.Expect

host = hostProps.get("system.hostname");
user = hostProps.get("config.user");
pass = hostProps.get("config.pass");
display = hostProps.get("system.displayname");

/* Define prompt to work for HOSTNAME/USERname#
/                 <-Start
${display}  <- display defined earlier
\/               <- matches /
${user}      <- user defined earlier
\#              <- matches #
\s              <- space
/               <-End
*/
prompt = /${display}\/${user}\#\s/

//Login and expect prompt
cli=Expect.open(host, user, pass);
cli.expect(prompt);

//Set terminal length to 0 so show run shows all at once then expect prompt
cli.send("terminal length 0\n");
cli.expect(prompt);

//show running-config then expect prompt then print everything before the above expect to the next above expect
//NOTE: The line show running-config will also be printed and can be ignored in a Config Check
cli.send("show running-config\n");
cli.expect(prompt);
println(cli.before());

//Exit device
cli.send("exit\n");
cli.expectClose();

Link to post
Share on other sites
  • 0

OK, I found it. I now understand that the Config Check doesn't change the file it literally just does a diff ignoring the parts we told it to ignore to determine if it should save it or not.

Since my file has "show running-config" in it would that mess anything up? Is the backup rollback process, just copy and paste the file into the device config or is there a fancy LM way to roll back where the file needs to be exact?

Link to post
Share on other sites
  • 0

Can confirm this removes the show running-config line:

cli.before().eachLine{
  if(!(it=~/show running-config/)){
    println(it)
  }
}

Can you explain the it=~/show running-config/ part, so I understand what the code is doing?

Is the code defining "it" as the regex string "show running-config", but we are println(it) so it isn't defined as "show running-config"? I know it is printing when "it" is not "show running-config" but not sure how it is doing that.

  • Like 1
Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.