• 0

Regarding CPU and Memory and Disk Performance Data Collection for multiple servers


usnishguha
 Share

Question

Hi Everyone,

 Can anyone please let me know if they are aware of any method/script that can be used to get the CPU, Memory and Disk related performance data in csv for a list of servers between a specific start time and end time.

Thanks in advance.

Link to comment
Share on other sites

18 answers to this question

Recommended Posts

  • 1
  • Administrators

The API call to get at the data that's shown on the raw data tab is documented here: https://www.logicmonitor.com/swagger-ui-master/dist/#/Data/getDeviceDatasourceInstanceData.

1) You'll need to get the device id. You can get that by calling "/device/devices/?fields=displayName,id,name"

2) You'll also need to get the devicedatasourceid (known as "hdsId" in the documentation). Do this call to get the list of datasources by name and id: "/device/devices/{deviceId}/devicedatasources?fields=dataSourceName,id"

3) You'll also need to get the instance id that you want to fetch data for. Do this call to get the list of instances: "/device/devices/{deviceId}/devicedatasources/{hdsId}/instances/?fields=id,name"

4) Use the deviceId, hdsId, and instance_id to make this call to get the data: "/device/devices/{deviceId}/devicedatasources/{hdsId}/instances/{instance_id}/data"

Once you have the data, you can combine it however you want.

You can do this in the python sdk by using the following methods:

1) Get device id list:

for device in lm.get_device_list().items:
  print(f"{device.id} {device.name} ({device.display_name})")

2) Get device datasource id (in this case for device with ID 43):

for x in lm.get_device_datasource_list(43).items:
    print(f"{x.id} {x.data_source_name}")

3) Get the instance id:

for x in lm.get_device_datasource_instance_list(43,5128).items:
  print(f"{x.id} {x.name}")

4) Use the deviceId, hdsId, and instance id to get the data:

data = lm.get_device_datasource_instance_data(43,5128,112712652)
#at this point, data has what you want in it. 
for x in range(30):
  print(f"{data.time[x]} {data.values[x]}")

#OUTPUT
1620261167000 [0.0, 0.0, 0.0, 0.0, 0.8083, 23122.2917, 93.9583, 0.0, 0.0, 0.0, 0.0, 0.0, 30046.425, 97.5833, 1.0, 314.0, 0.0, 0.185, 0.2404, 0.185, 0.2404, 12744918.0, 1.0, 'No Data', 'No Data', 1.0]
1620261047000 [0.0, 0.0, 0.0, 0.0, 0.5833, 27667.775, 140.4917, 0.0, 0.0, 0.0, 0.0, 0.0, 33068.3417, 128.05, 1.0, 314.0, 0.0, 0.2213, 0.2645, 0.2213, 0.2645, 12732918.0, 1.0, 'No Data', 'No Data', 1.0]
1620260927000 [0.0, 0.0, 0.0, 0.0, 0.575, 21000.0083, 78.25, 0.0, 0.0, 0.0, 0.0, 0.0, 29561.4417, 83.825, 1.0, 314.0, 0.0, 0.168, 0.2365, 0.168, 0.2365, 12720918.0, 1.0, 'No Data', 'No Data', 1.0]
1620260807000 [0.0, 0.0, 0.0, 0.0, 1.0667, 24940.7833, 90.7833, 0.0, 0.0, 0.0, 0.0, 0.0, 29460.3333, 92.15, 1.0, 314.0, 0.0, 0.1995, 0.2357, 0.1995, 0.2357, 12708918.0, 1.0, 'No Data', 'No Data', 1.0]
1620260687000 [0.0, 0.0, 0.0, 0.0, 0.8333, 21588.0083, 69.5917, 0.0, 0.0, 0.0, 0.0, 0.0, 28425.675, 76.2917, 1.0, 314.0, 0.0, 0.1727, 0.2274, 0.1727, 0.2274, 12696918.0, 1.0, 'No Data', 'No Data', 1.0]
1620260567000 [0.0, 0.0, 0.0, 0.0, 0.675, 19949.3, 69.7, 0.0, 0.0, 0.0, 0.0, 0.0, 28177.125, 76.1583, 1.0, 314.0, 0.0, 0.1596, 0.2254, 0.1596, 0.2254, 12684918.0, 1.0, 'No Data', 'No Data', 1.0]
1620260447000 [0.0, 0.0, 0.0, 0.0, 0.5083, 21323.925, 69.5083, 0.0, 0.0, 0.0, 0.0, 0.0, 28169.5333, 76.4167, 1.0, 314.0, 0.0, 0.1706, 0.2254, 0.1706, 0.2254, 12672918.0, 1.0, 'No Data', 'No Data', 1.0]
1620260327000 [0.0, 0.0, 0.0, 0.0, 0.7167, 21547.7667, 69.5583, 0.0, 0.0, 0.0, 0.0, 0.0, 27970.5333, 76.1, 1.0, 314.0, 0.0, 0.1724, 0.2238, 0.1724, 0.2238, 12660917.0, 1.0, 'No Data', 'No Data', 1.0]
1620260207000 [0.0, 0.0, 0.0, 0.0, 0.65, 19852.1333, 69.55, 0.0, 0.0, 0.0, 0.0, 0.0, 28590.5917, 76.1583, 1.0, 314.0, 0.0, 0.1588, 0.2287, 0.1588, 0.2287, 12648917.0, 1.0, 'No Data', 'No Data', 1.0]
1620260087000 [0.0, 0.0, 0.0, 0.0, 0.65, 21166.5167, 69.4667, 0.0, 0.0, 0.0, 0.0, 0.0, 28446.6083, 76.0917, 1.0, 314.0, 0.0, 0.1693, 0.2276, 0.1693, 0.2276, 12636918.0, 1.0, 'No Data', 'No Data', 1.0]
1620259967000 [0.0, 0.0, 0.0, 0.0, 1.15, 21610.7833, 70.4167, 0.0, 0.0, 0.0, 0.0, 0.0, 27947.575, 76.3667, 1.0, 314.0, 0.0, 0.1729, 0.2236, 0.1729, 0.2236, 12624918.0, 1.0, 'No Data', 'No Data', 1.0]
1620259847000 [0.0, 0.0, 0.0, 0.0, 0.7083, 20190.3667, 69.1167, 0.0, 0.0, 0.0, 0.0, 0.0, 28349.1667, 76.0083, 1.0, 314.0, 0.0, 0.1615, 0.2268, 0.1615, 0.2268, 12612918.0, 1.0, 'No Data', 'No Data', 1.0]
1620259727000 [0.0, 0.0, 0.0, 0.0, 0.6167, 21531.1167, 72.75, 0.0, 0.0, 0.0, 0.0, 0.0, 32798.825, 77.9667, 1.0, 314.0, 0.0, 0.1722, 0.2624, 0.1722, 0.2624, 12600918.0, 1.0, 'No Data', 'No Data', 1.0]
1620259607000 [0.0, 0.0, 0.0, 0.0, 0.775, 22277.75, 71.325, 0.0, 0.0, 0.0, 0.0, 0.0, 29394.2083, 77.8167, 1.0, 314.0, 0.0, 0.1782, 0.2352, 0.1782, 0.2352, 12588918.0, 1.0, 'No Data', 'No Data', 1.0]
1620259487000 [0.0, 0.0, 0.0, 0.0, 0.6167, 20045.0667, 69.9667, 0.0, 0.0, 0.0, 0.0, 0.0, 28847.25, 76.8083, 1.0, 314.0, 0.0, 0.1604, 0.2308, 0.1604, 0.2308, 12576918.0, 1.0, 'No Data', 'No Data', 1.0]
1620259367000 [0.0, 0.0, 0.0, 0.0, 0.5417, 21151.7917, 69.1167, 0.0, 0.0, 0.0, 0.0, 0.0, 27920.675, 75.3333, 1.0, 314.0, 0.0, 0.1692, 0.2234, 0.1692, 0.2234, 12564918.0, 1.0, 'No Data', 'No Data', 1.0]
1620259247000 [0.0, 0.0, 0.0, 0.0, 0.5417, 21280.6833, 70.25, 0.0, 0.0, 0.0, 0.0, 0.0, 29305.6333, 76.5083, 1.0, 314.0, 0.0, 0.1702, 0.2344, 0.1702, 0.2344, 12552918.0, 1.0, 'No Data', 'No Data', 1.0]
1620259127000 [0.0, 0.0, 0.0, 0.0, 0.5417, 19288.0333, 68.9, 0.0, 0.0, 0.0, 0.0, 0.0, 28707.925, 75.7583, 1.0, 314.0, 0.0, 0.1543, 0.2297, 0.1543, 0.2297, 12540918.0, 1.0, 'No Data', 'No Data', 1.0]
1620259007000 [0.0, 0.0, 0.0, 0.0, 0.5417, 22780.9833, 70.8, 0.0, 0.0, 0.0, 0.0, 0.0, 28817.0583, 76.2333, 1.0, 314.0, 0.0, 0.1822, 0.2305, 0.1822, 0.2305, 12528918.0, 1.0, 'No Data', 'No Data', 1.0]
1620258887000 [0.0, 0.0, 0.0, 0.0, 0.75, 20836.5333, 68.2083, 0.0, 0.0, 0.0, 0.0, 0.0, 28034.7667, 74.55, 1.0, 314.0, 0.0, 0.1667, 0.2243, 0.1667, 0.2243, 12516918.0, 1.0, 'No Data', 'No Data', 1.0]
1620258767000 [0.0, 0.0, 0.0, 0.0, 0.5333, 19077.65, 68.425, 0.0, 0.0, 0.0, 0.0, 0.0, 27929.9667, 74.3167, 1.0, 314.0, 0.0, 0.1526, 0.2234, 0.1526, 0.2234, 12504918.0, 1.0, 'No Data', 'No Data', 1.0]
1620258647000 [0.0, 0.0, 0.0, 0.0, 0.6, 20671.7667, 67.9167, 0.0, 0.0, 0.0, 0.0, 0.0, 27612.3833, 74.5417, 1.0, 314.0, 0.0, 0.1654, 0.2209, 0.1654, 0.2209, 12492918.0, 1.0, 'No Data', 'No Data', 1.0]
1620258527000 [0.0, 0.0, 0.0, 0.0, 0.775, 21883.3167, 71.175, 0.0, 0.0, 0.0, 0.0, 0.0, 28669.65, 77.5917, 1.0, 314.0, 0.0, 0.1751, 0.2294, 0.1751, 0.2294, 12480918.0, 1.0, 'No Data', 'No Data', 1.0]
1620258407000 [0.0, 0.0, 0.0, 0.0, 0.6417, 137304.1917, 157.925, 0.0, 0.0, 0.0, 0.0, 0.0, 38044.15, 108.525, 1.0, 314.0, 0.0, 1.0984, 0.3044, 1.0984, 0.3044, 12468918.0, 1.0, 'No Data', 'No Data', 1.0]
1620258287000 [0.0, 0.0, 0.0, 0.0, 0.5417, 21183.7833, 69.55, 0.0, 0.0, 0.0, 0.0, 0.0, 28358.05, 76.5333, 1.0, 314.0, 0.0, 0.1695, 0.2269, 0.1695, 0.2269, 12456918.0, 1.0, 'No Data', 'No Data', 1.0]
1620258167000 [0.0, 0.0, 0.0, 0.0, 0.5, 21261.9167, 69.125, 0.0, 0.0, 0.0, 0.0, 0.0, 27748.5083, 75.6917, 1.0, 314.0, 0.0, 0.1701, 0.222, 0.1701, 0.222, 12444918.0, 1.0, 'No Data', 'No Data', 1.0]
1620258047000 [0.0, 0.0, 0.0, 0.0, 0.5833, 20101.2833, 69.5833, 0.0, 0.0, 0.0, 0.0, 0.0, 28711.5583, 75.9417, 1.0, 314.0, 0.0, 0.1608, 0.2297, 0.1608, 0.2297, 12432918.0, 1.0, 'No Data', 'No Data', 1.0]
1620257927000 [0.0, 0.0, 0.0, 0.0, 0.525, 21318.0917, 70.3417, 0.0, 0.0, 0.0, 0.0, 0.0, 28355.2583, 76.6417, 1.0, 314.0, 0.0, 0.1705, 0.2268, 0.1705, 0.2268, 12420918.0, 1.0, 'No Data', 'No Data', 1.0]
1620257807000 [0.0, 0.0, 0.0, 0.0, 0.525, 21842.075, 70.55, 0.0, 0.0, 0.0, 0.0, 0.0, 28007.65, 76.2917, 1.0, 314.0, 0.0, 0.1747, 0.2241, 0.1747, 0.2241, 12408918.0, 1.0, 'No Data', 'No Data', 1.0]
1620257687000 [0.0, 0.0, 0.0, 0.0, 0.6167, 19956.9833, 70.325, 0.0, 0.0, 0.0, 0.0, 0.0, 28823.6583, 77.1833, 1.0, 314.0, 0.0, 0.1597, 0.2306, 0.1597, 0.2306, 12396918.0, 1.0, 'No Data', 'No Data', 1.0]

The only oddity i found was that when doing the raw get to this same endpoint, you get back an additional array with the names of the datapoints, presumably so you know which value in each values array contains which datapoint's values. That's not returned through the sdk, as far as i can tell. I'd have to do some playing around with it to know if they are always returned in the same order across instances. If they are, a simple subscript would get you the value you want:

data = lm.get_device_datasource_instance_data(43,5128,112712652)
#at this point, data has what you want in it. 
for x in range(30):
  print(f"{data.time[x]} {data.values[x][12]}") #fetch the 13th element in the values array, which i think is OutOctets
  
#OUTPUT
1620261767000 32891.8333
1620261647000 30996.375
1620261527000 34751.2583
1620261407000 32650.15
1620261287000 30647.1917
1620261167000 30046.425
1620261047000 33068.3417
1620260927000 29561.4417
1620260807000 29460.3333
1620260687000 28425.675
1620260567000 28177.125
1620260447000 28169.5333
1620260327000 27970.5333
1620260207000 28590.5917
1620260087000 28446.6083
1620259967000 27947.575
1620259847000 28349.1667
1620259727000 32798.825
1620259607000 29394.2083
1620259487000 28847.25
1620259367000 27920.675
1620259247000 29305.6333
1620259127000 28707.925
1620259007000 28817.0583
1620258887000 28034.7667
1620258767000 27929.9667
1620258647000 27612.3833
1620258527000 28669.65
1620258407000 38044.15
1620258287000 28358.05

From there you could combine the data however you want, using whatever aggregation method you'd like. There are python libraries for handling this kind of thing.

Link to comment
Share on other sites

  • 0

@Stuart Weenig, thanks a lot for your detailed answer. I am struggling to some extent with Python SDK ( I am new to this). I have tried to download the tar.gz and extracted it. Then I tried running a sample from here: https://www.logicmonitor.com/support/rest-api-developers-guide/logicmonitor-sdks. But it's giving me this error,

ModuleNotFoundError: No module named 'certifi'

I am probably doing it wrong. Is there any url which helps in starting with the SDKs?

Thanks in advance.

Link to comment
Share on other sites

  • 0

Hi @Stuart Weenig, I was able to get past the "certifi" module error after installing it. But can you please let me know about this problem,

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'lm' is not defined
>>>
 

*seems it doesn't find lm module. 

Link to comment
Share on other sites

  • 0

i'm trying to get to this data using Powershell... no errors, it successfully grabs the device, DS, and instances using the same functions... but the data returns an empty array for data that is present in the LM resources interface... thoughts?

  • $resourcePath  = "/device/devices/$($device.id)/devicedatasources/$($DS_CPU.id)/instances/$($inst_CPU.id)/data"

my output shows me this:

  • Gathering /device/devices/6966/devicedatasources/503122/instances/9311756/data

No errors are thrown during the run.  It's confounding me a bit.

Link to comment
Share on other sites

  • 0
  • Administrators

Would suggest you get postman up and running and vet out all the calls you are trying to make before you try to code them in. That will eliminate the question of whether it's something about the call or Posh. Just a suggestion. 

That said, when i did your call, the $($DS_CPU.id) variable is the id (9512), not the dataSourceId (24131557):

image.thumb.png.bbe1ca9eef748493d36e02c7dd537dad.png

 

And my $($inst_CPU.id) was the id (146237749), not any other id. That one stumped me for a while until i learned that lesson.

image.thumb.png.577c7b8533485af799b5f15a5d9ce59d.png

 

So, my final call was {{url}}/device/devices/9/devicedatasources/9512/instances/146237749/data

Where my $($device.id) is 9, $$DS_CPU.id) is 9512 (id) and my $($inst_CPU.id) was 146237749. 

Link to comment
Share on other sites

  • 0

I'll be getting postman running tomorrow... my time was up and had to move to another effort.  Thank you for pointing the tool out to me.  It looks really useful.  and thanks for poking at my code.  I'll verify I have the right IDs for each of those pieces as well. (Funny Note... I typed this up yesterday and didn't hit submit).  I've verified that I have the right pieces and parts.  You're examining the JSON in your examples... I use Powershell ISE and explore the return data using the variables.  I have verified that I'm using the deviceDataSource rather than the dataSource:

PS C:\Users\cole.mcdonald> $inst_CPU

id                       : 8976881
dataSourceId             : 5725412
deviceDataSourceId       : 427972
groupId                  : 538636
groupName                : @default
name                     : Perfmon_CPU-_Total
displayName              : _Total
description              : 
lockDescription          : False
deviceId                 : 5378

 

and my call was:

/device/devices/5378/devicedatasources/427972/instances/8976881/data

 

(I switched devices to verify that it wasn't just empty)  I've tried grabbing data from both the instance directly as well as the graph data.  I'm going to dig into postman... but looking at it, I don't know that it will necessarily show me anything different than I'm getting using straight powershell... and I generally dislike GUI'd applications for this sort of thing as I find them horribly inefficient (I'm old and grew up on CLI).

Here's my Postman return:

{
    "dataSourceName": "Perfmon_CPU",
    "dataPoints": [
        "PercentProcessorTimeCounter",
        "PercentProcessorTime"
    ],
    "values": [],
    "time": [],
    "nextPageParams": ""
}

 

So it looks like it's succeeding in the call, just no data returned.  Odd, because the graph shows that there should be historical data there to grab.

Link to comment
Share on other sites

  • 0
  • Administrators

I think it's the 427972 that's not right. Hard to say without navigating your IDs directly. 

Here's what I'd do (in code or in postman). If this is what you're already doing, ignore:

/device/devices -> find the "id" of the device of interest (here, i'll call it $device_id)

/device/devices/$($device_id)/devicedatasources -> find the "id" (not the dataSourceId) of the datasource of interest (here, i'll call it $DS_id)

/device/devices/$($device_id)/devicedatasources/$($DS_id)/instances -> find the "id" of the instance of interest (here, i'll call it $instance_id)

/device/devices/$($device_id)/devicedatasources/$($DS_id)/instances/$($instance_id)/data -> should have data here from the last hour

 

For what it's worth, i was skeptical about postman too. I'm old school CLI as well. But i quickly got addicted. The other thing i got addicted to was the Python SDK. Combining it with my custom LM wrapper, the code becomes:

from lm import lm
data = lm.get_device_datasource_instance_data(9,9512,146237749)

Where 9 is the $device_id, 9512 is the $DS_id, and 146237749 is the $instance_id. 

Link to comment
Share on other sites

  • 0

Yeah, I'm using the ID of each bit exclusively.  I figured we need the instance IDs as they apply to the specific resource rather than the id of the DS itself... which would be the same for every instance of that DS as applied to any resource.  I've built quite a few Relational DB driven apps ;)  I've spent my time chasing IDs... and then banging my head against my desk once the "obvious" problem was unconvered.

The instance shows the deviceID, the deviceDataSourceID, and the instanceIDs as their own properties in the returned JSON (here parsed into a PSObject).  that dds ID should be the specific ID of the DS as applied to that device.  Everything seems to be correct on the walk up to the "/data"  I'm a little surprised we can't just get "/instance/#####/data" from it as a direct access to the endpoint of the walk since it's a unique ID for the instance itself.  Overly complicated data access method that is prone to error and misinterpretation.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share