Add an HTTP query helper

[misje]

After writing a lot of artifacts querying REST APIs (to be released), I have found the need for a helper method that does the following:

• Handle login to APIs using a secret and an additional request, retrieving a short-lived token
• Handle pagination with either known or unknown total size
• Log debug messages (like progress) and errors if the query or login fails
• Return results as individual rows

I was originally planning to add this to the exchange, but I am hoping it is considered useful enough to be part of the main project. It does not do that much, but personally I have enjoyed this utility a lot. Hopefully others will find great use for it too.

[scudette]

This is a cool artifact but I am not sure that all REST APIs are actually the same - do you have an example of the APIs that this works on?

I am also trying to find a good way to test this artifact

[misje]

APIs are not all the same, certainly not. This why I added some assumptions/limitations to the artifact description, and a number of parameters can be changed to adapt do different APIs. There are of course cases where this helper fall short, for instance if the query parameters are nested or complex, where the offset and size parameters are in a nested object, array etc.

As for testing, I suppose a simple "echo server" would do. It would be super simple to implement in Python using the integrated http.server module and a simple BaseHTTPRequestHandler class. I can certainly look at building some tests, if you would like.

Here are some examples:

Defender for Endpoint machine list

The API syntax is the same for all of Microsoft's security APIs. Login required, and pagination required (limit depends on API).

SELECT *
FROM Artifact.Server.Utils.QueryAPI(
  APIName='Machines',
  LoginSecret='mdatp_login',
  LoginURL='https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/token',
  TokenField='access_token',
  URL='https://api.security.microsoft.com/api/machines',
  QueryParams=dict(
    `$filter`="onboardingStatus eq 'Onboarded'"),
  ErrorField='error_description',
  ResultsField='value', PageSize=100, MaxPages=20, OffsetField='$skip', LimitField='$top'
  )

Action1 endpoints (free)

Login and pagination required.

SELECT *
FROM Artifact.Server.Utils.QueryAPI(
  LoginSecret='action1_login',
  URL='https://app.eu.action1.com/api/3.0/endpoints/managed/<org_id>',
  PageSize=100,
  MaxPages=20,
  TokenField='access_token',
  ErrorField='user_message',
  ResultsField='items',
  OffsetField='from',
  LimitField='limit',
  TotalField='total_items')

Snipe-IT hardware list

No login needed, but pagination required (500 row limit).

SELECT *
FROM Artifact.Server.Utils.QueryAPI(
  APIName='Assets',
  Secret='snipeit_hw_list',
  PageSize=500,
  OffsetField='offset',
  LimitField='limit',
  MaxPages=20)

AlienVault OTX (public/free)

SELECT
    *
FROM Artifact.Server.Utils.QueryAPI(
  URL='https://otx.alienvault.com/api/v1/indicator/hostname/google.com/malware',
  ResultsField='data',
  ErrorField='details',
  OffsetField='page',
  TotalField='count'
  OffsetStart=1,
  PageSize=10,
  MaxPages=3,
  )

AbuseIPDB (free)

SELECT *
FROM Artifact.Server.Utils.QueryAPI(
  Secret='abuseipdb_reports',
  URL='https://api.abuseipdb.com/api/v2/reports',
  QueryParams=dict(
    ipAddress='35.216.188.88'),
  ResultsField='data.results',
  ErrorField='errors.detail',
  OffsetField='data.page',
  LimitField='data.perPage',
  TotalField='data.total',
  OffsetStart=1,
  PageSize=10,
  MaxPages=3)