mnagel

netflow data access via API

Recommended Posts

Among other reasons, it would be very desirable to have access to Netflow data via the API for at least these reasons:

* detect missing netflow data due to misconfiguration or similar; this includes both persistent lack of data and data gaps
* detect and alert on unusual traffic in a customized manner

Missing data without awareness is probably the single biggest weakness in LM, and with Netflow there is literally nothing that can be done short of visual inspection of every device in the UI.

Thanks,
Mark

 

Share this post


Link to post
Share on other sites

@mnagel Do you mean the flow data as it appears in the Traffic view?

Edited by Mosh

Share this post


Link to post
Share on other sites
15 minutes ago, Mosh said:

@mnagel Do you mean the flow data as it appears in the Traffic view?

@mosh Correct -- there is no access to this information via the API and no alerting capability, so if data is not populating for any reason, it is an unpleasant surprise when the data is needed.  I run into this all the time, but have been able to create alert rules for key datapoints that let me know when stuff is broken (e.g., I found recently SNMP breaks when you have intermediate firewalls since the code re-uses a single session and sometimes passes data too infrequently to keep the "connection" active).

 

Share this post


Link to post
Share on other sites
42 minutes ago, mnagel said:

@mosh Correct -- there is no access to this information via the API and no alerting capability, so if data is not populating for any reason, it is an unpleasant surprise when the data is needed.  I run into this all the time, but have been able to create alert rules for key datapoints that let me know when stuff is broken (e.g., I found recently SNMP breaks when you have intermediate firewalls since the code re-uses a single session and sometimes passes data too infrequently to keep the "connection" active).

 

There's no documentation, however, I can see these REST endpoints are invoked by the Traffic view:

Top Talkers
/santaba/rest/device/devices/{deviceID}/topTalkersGraph?netflowFilter={}&end=1532968403&start=1532961173&time=2hour&_=1532968368909

Endpoints
/santaba/rest/device/devices/{deviceID}/endpoints?filter=type%3A"destination"&sort=-usage&time=2hour&start=1532961173&end=1532968403&netflowFilter={}&size=10&offset=0&_=1532968368905

Ports
/
santaba/rest/device/devices/3942/ports?sort=-usage&time=2hour&start=1532961306&end=1532968536&netflowFilter={}&size=10&offset=0&_=1532968503441

Flows
/santaba/rest/device/devices/{deviceID}/flows?sort=-usage&time=2hour&start=1532957720&end=1532964950&netflowFilter={}&size=10&offset=0&_=1532964896784
 

Edited by Mosh

Share this post


Link to post
Share on other sites

(BTW, re SNMP, I ended up implementing a uptime via SNMP poll just to alert if there is no SNMP response.)

  • Upvote 1

Share this post


Link to post
Share on other sites
On 7/30/2018 at 8:36 AM, Mosh said:

@mnagel Do you mean the flow data as it appears in the Traffic view?

@mosh Correct -- there is no access to this information via the API and no alerting capability, so if data is not populating for any reason, it is an unpleasant surprise when the data is needed.  I run into this all the time, but have been able to create alert rules for key datapoints that let me know when stuff is broken (e.g., I found recently SNMP breaks when you have intermediate firewalls since the code re-uses a single session and sometimes passes data too infrequently to keep the "connection" active).

Very nice!  I have an initial script now that pulls top flows for netflow-enabled devices, so now I must see what I will do with this.  Minimally, detect lack of data and either opsnotes additions for exceptional usage or email alerts (or both).  Definitely important to include the size parameter, though.  Core of current in-progress script:

my $devices = $lmapi->get_all(path => "/device/devices", fields => "id,displayName,enableNetflow");
for my $device (@$devices) {
    if ($device->{enableNetflow} == JSON::true) {
        for my $flow ($lmapi->get_all(path => "/device/devices/$device->{id}/flows", size => 10, sort => '-usage', time => '2hour')) {
            print Dumper($flow);
        }
    }
}

Share this post


Link to post
Share on other sites
On 8/13/2018 at 9:52 PM, mnagel said:

@mosh Correct -- there is no access to this information via the API and no alerting capability, so if data is not populating for any reason, it is an unpleasant surprise when the data is needed.  I run into this all the time, but have been able to create alert rules for key datapoints that let me know when stuff is broken (e.g., I found recently SNMP breaks when you have intermediate firewalls since the code re-uses a single session and sometimes passes data too infrequently to keep the "connection" active).

Very nice!  I have an initial script now that pulls top flows for netflow-enabled devices, so now I must see what I will do with this.  Minimally, detect lack of data and either opsnotes additions for exceptional usage or email alerts (or both).  Definitely important to include the size parameter, though.  Core of current in-progress script:

my $devices = $lmapi->get_all(path => "/device/devices", fields => "id,displayName,enableNetflow");
for my $device (@$devices) {
    if ($device->{enableNetflow} == JSON::true) {
        for my $flow ($lmapi->get_all(path => "/device/devices/$device->{id}/flows", size => 10, sort => '-usage', time => '2hour')) {
            print Dumper($flow);
        }
    }
}

 

I believe the two parameters "start" and "end" are the epochs to use for the "time" parameter. Could be that omitting these means current time is automatically used.  If you wanted to narrow the time period to check for data, you should be able to play with these two params.

Edited by Mosh

Share this post


Link to post
Share on other sites
2 hours ago, Mosh said:

 

I believe the two parameters "start" and "end" are the epochs to use for the "time" parameter. Could be that omitting these means current time is automatically used.  If you wanted to narrow the time period to check for data, you should be able to play with these two params.

Yep, I concluded they were autoset based on current time, which is typically what I want (verify if we have recent data, generate alerts based on usage, etc.).  But sure, if I want to do a sliding analysis over previous windows, those params will be needed.

Share this post


Link to post
Share on other sites
On 8/14/2018 at 4:15 PM, mnagel said:

Yep, I concluded they were autoset based on current time, which is typically what I want (verify if we have recent data, generate alerts based on usage, etc.).  But sure, if I want to do a sliding analysis over previous windows, those params will be needed.

So it turns out in v109 this and a bunch of other endpoints broke (e.g., netscans).  It is not like we were not warned this might happen when using undocumented endpoints.  From what I have heard, it seems that now that the v2 API is in development, all the endpoints we were using to obtain data otherwise impossible to access stopped working.

[mnagel@colby lmapi-scripts]$ ./get-top-talkers --company xxx
Problem with request:
    Request: https://xxx.logicmonitor.com/santaba/rest/device/devices/7/flows?size=10&sort=-usage
    Response Status: 406 Not Acceptable
 at ./get-top-talkers line 53

Sigh.

Share this post


Link to post
Share on other sites

Forgot to include the headers I was referring to:

Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: en-US,en;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Charset: utf-8, iso-8859-1;q=0.5 

Share this post


Link to post
Share on other sites
2 hours ago, mnagel said:

I have been given a tentative fix, will post if I get it working...

OK -- under control for now.  DISCLAIMER: everything is subject to change, as it was before when accessing undocumented REST resources.

The short answer is you must now include an 'X-version: 2' header in your request headers, or add 'v=2' as a query param.  When you do this, it is supposed to work for any endpoint except one (can't recall which), but I updated my module to use v2 only as needed for specific endpoint patterns. 

The other important finding is that the data structure returned for the v2 API is different.  Instead of items being a subkey of data, it is (for now anyway) a top-level key.  I also had been checking the status key (copy of the HTTP response code), but that was removed. 

I am still working on updates to my scripts, but the netflow script is working again.  I will post my revised LMAPI.pm to github when it is stable.

  • Upvote 1

Share this post


Link to post
Share on other sites
2 hours ago, mnagel said:

OK -- under control for now.  DISCLAIMER: everything is subject to change, as it was before when accessing undocumented REST resources.

The short answer is you must now include an 'X-version: 2' header in your request headers, or add 'v=2' as a query param.  When you do this, it is supposed to work for any endpoint except one (can't recall which), but I updated my module to use v2 only as needed for specific endpoint patterns. 

The other important finding is that the data structure returned for the v2 API is different.  Instead of items being a subkey of data, it is (for now anyway) a top-level key.  I also had been checking the status key (copy of the HTTP response code), but that was removed. 

I am still working on updates to my scripts, but the netflow script is working again.  I will post my revised LMAPI.pm to github when it is stable.

Updated WM::LMAPI.pm at https://github.com/willingminds/lmapi-scripts.git -- still working on more issues, but this handles most of my pain.

Share this post


Link to post
Share on other sites

@Mosh @mnagel We will publish the ability to get netflow data with v2 of the API, which is coming soon. v2 is not quite ready to be published yet, so use it with caution until it's officially published (it is subject to change), but we should be ready in the next couple of releases - stay tuned!

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.