Import Datasource via API and PowerShell

[nrichards 4]

Hello All,

I'm attempting to import datasources into our LogicMonitor instance, using the API and PowerShell.  The following documentation only provides a CURL example, which isn't really sufficient for us.

https://www.logicmonitor.com/support/rest-api-developers-guide/datasources/import-datasources-from-xml/

Usage: 

Import-LMDatasource -Credential $credentials -FilePath 'c:\repositories\LogicModules\DataSources\CustomDataSource.xml'

I am assuming that you have an array containing your accessId, accessKey and company name, prior to calling the function.  The parameter FilePath is the full path to the XML file to be uploaded.

function Import-LMDatasource
{
    [CmdletBinding()]
    param (
        [Parameter(Mandatory = $false)]
        [array]$Credential,
        [Parameter(Mandatory = $false)]
        [ValidateNotNullOrEmpty()]
        [string]$FilePath
    )
    Begin {
        if(-not($Credential))
        {
            $accessId = Read-Host -Prompt "Please supply accessId:"
            $accessKey = Read-Host -Prompt "Please supply accessKey:"
            $company = Read-Host -Prompt "Please supply company:"
        }
        else {
            $accessId = $Credential.accessId
            $accessKey = $Credential.accessKey
            $company = $Credential.company
        }

        $httpVerb = 'POST'

        $resourcePath = '/setting/datasources'        
        $queryParams = '/importxml'
        $data = ''

        $url = 'https://' + $company + '.logicmonitor.com/santaba/rest' + $resourcePath + $queryParams
        $epoch = [Math]::Round((New-TimeSpan -Start (Get-Date -Date '1/1/1970') -End (Get-Date).ToUniversalTime()).TotalMilliseconds)
        
        $contentType = [System.Web.MimeMapping]::GetMimeMapping($FilePath)
    }
    Process 
    {
        $requestVars = $httpVerb + $epoch + $data + $resourcePath
        $hmac = New-Object System.Security.Cryptography.HMACSHA256
        $hmac.Key = [Text.Encoding]::UTF8.GetBytes($accessKey)
        $signatureBytes = $hmac.ComputeHash([Text.Encoding]::UTF8.GetBytes($requestVars))
        $signatureHex = [System.BitConverter]::ToString($signatureBytes) -replace '-'
        $signature = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($signatureHex.ToLower()))
        $auth = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch
  
        Add-Type -AssemblyName System.Net.Http

        $httpClientHandler = New-Object System.Net.Http.HttpClientHandler
        $httpClient = New-Object System.Net.Http.HttpClient $httpClientHandler
        $httpClient.DefaultRequestHeaders.Authorization = $auth
        $packageFileStream = New-Object System.IO.FileStream @($filePath, [System.IO.FileMode]::Open)

        $contentDispositionHeaderValue = New-Object System.Net.Http.Headers.ContentDispositionHeaderValue 'form-data'
        $contentDispositionHeaderValue.Name = 'file'
        $contentDispositionHeaderValue.FileName = (Split-Path -Path $FilePath -Leaf)

        $streamContent = New-Object System.Net.Http.StreamContent $packageFileStream
        $streamContent.Headers.ContentDisposition = $contentDispositionHeaderValue
        $streamContent.Headers.ContentType = New-Object System.Net.Http.Headers.MediaTypeHeaderValue $contentType

        $content = New-Object System.Net.Http.MultipartFormDataContent
        $content.Add($streamContent)

        $response = $httpClient.PostAsync($url, $content).Result

        if(!$response.IsSuccessStatusCode)
        {
            $responseBody = $response.Content.ReadAsStringAsync().Reult
            $errorMessage = "Status code {0}.  Reason {1}.  Server reported the following message: {2}." -f $response.StatusCode, $response.ReasonPhrase, $responseBody
            throw [System.Net.Http.HttpRequestException] $errorMessage
        }

        return $response.Content.ReadAsStringAsync().Result

        # $httpClient.Dispose()
        # $response.Dispose()
    }
    End 
    {

    }
}

Result:

{"errmsg":"Request content is too large, max allowed size is 10240","status":1007}

Dot Sourcing the function at runtime allows me to inspect the variables set during execution:

$response

Version             : 1.1
Content             : System.Net.Http.StreamContent                                                
StatusCode          : OK
ReasonPhrase        : OK                         
Headers             : {[Date, System.String[]], [Server, System.String[]]}                         
RequestMessage      : Method: POST, RequestUri: 'https://<instance>.logicmonitor.com/santaba/rest/setting/datasources/importxml', Version: 1.1, Content:          
                      System.Net.Http.MultipartFormDataContent, Headers:
                      {
                        Authorization: LMv1 <ACCESSTOKEN-OBFUSCATED> 
                        Content-Type: multipart/form-data; boundary="17ba48f6-b5e9-48c6-9002-7544d99025d8"
                        Content-Length: 11858
                      }   
IsSuccessStatusCode : True

Uploading the exact same datasource XML via the GUI works.  I think I'm most of the way there, but obviously something I'm doing is bloating the request size and tripping this limit.

Has anyone had any success with uploading datasources via the API?

Many Thanks,

~Nick

[Sarah Terry 20]

Hi Nick,

This one is a bit tricky - we didn't publish much around this resource because there wasn't a lot of interest and also it starts to overlap with where the LogicModule Exchange is useful.  That being said, we'll update the docs to provide more useful examples.  We previously only supported Basic Auth for this resource, but have since added support for authenticating via API Tokens.  In order to get the token authentication to work, you need the following:

1. (1) the payload needs to include the xml file content enclosed in boundaries (can be random)
2. (2) the content type needs to be set to multipart/form-data and it needs to call out the boundaries

And in your code above, you'll want the /importxml to be tacked on to the resource path (I recognize that doesn't align with the proper REST resource naming, and we'll improve it in a future release).  So a successful example in PowerShell looks like this:

  
<# account info #>
$accessId = '48v2wRzfK94y53sq5EuF'
$accessKey = 'aB=k9opY87U~^_cvHiq~98G-j6Q_ja4sr6J8-4+)'
$company = 'api'

<#File & Request info#>
$uploadFile = 'test.xml'
$FilePath = "C:\Users\Sarah\Desktop\test.xml"
$httpVerb = 'POST'
$resourcePath = '/setting/datasources/importxml'        
$queryParams = ''

$file = Get-Content $FilePath

$boundary = [System.Guid]::NewGuid().ToString()

$data = '------$boundary
Content-Disposition: form-data; name="file"; filename="test.xml"
Content-Type: text/xml

'+$file+'

------$boundary--'

<# Construct URL #>
$url = 'https://' + $company + '.logicmonitor.com/santaba/rest' + $resourcePath + $queryParams

<# Get current time in milliseconds #>
$epoch = [Math]::Round((New-TimeSpan -start (Get-Date -Date "1/1/1970") -end (Get-Date).ToUniversalTime()).TotalMilliseconds)

<# Concatenate Request Details #>
$requestVars = $httpVerb + $epoch + $data + $resourcePath

<# Construct Signature #>
$hmac = New-Object System.Security.Cryptography.HMACSHA256
$hmac.Key = [Text.Encoding]::UTF8.GetBytes($accessKey)
$signatureBytes = $hmac.ComputeHash([Text.Encoding]::UTF8.GetBytes($requestVars))
$signatureHex = [System.BitConverter]::ToString($signatureBytes) -replace '-'
$signature = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($signatureHex.ToLower()))

<# Construct Headers #>
$auth = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("Authorization",$auth)
$headers.Add("Content-Type",'multipart/form-data; boundary=----$boundary')

<# Make Request #>
$response = Invoke-RestMethod -Uri $url -Method $httpVerb -Body $data -Header $headers 

<# Print status and body of response #>
$status = $response.status
$body = $response.data

Write-Host "Status:$status"
Write-Host "Response:$body"

Let me know if this doesn't work for you, or if you have additional issues/feedback.

Thanks!

Sarah

[nrichards 4]

Hi Sarah, 

Thanks for this.  My import is now working following the example you provided.  Though I am getting some formatting issues for the Collector Attributes > Groovy Script section.  I don't know if its actually an issue yet, but imagine that it might be.

I understand your comment completely;

Quote

"This one is a bit tricky - we didn't publish much around this resource because there wasn't a lot of interest and also it starts to overlap with where the LogicModule Exchange is useful. 

We're looking to commit our internally amended / created data sources into version control, and import them into our production instance under change, hence the desire for this.

Many Thanks,

~Nick 

[Preston Gallwas 0]

Apologies to drag this old thread up, but it's relevant to Sarah Terry's reply

The above example/script doesn't seem to be working for me - Only getting HTTP code 200.  When I switch to v2 api (by header or query param) its authentication failed.  I noticed the content-type header was a literal and $boundary wasn't being evaluated so I changed that but to no avail.  Is it possible to get some assistance with this?  We've got quite a few we want to import and are generating from a third party api.

[Preston Gallwas 0]

Well, the Datasources "import" from file option pointed me in the right direction.  Turns out my XML file wasn't valid (it had a wrong encoding).  I fixed that and will give this another go the next time I have a datasource to import.