OCI provider refresh fails consistently with HTTP 429 TooManyRequests

[ferr7s]

Summary

We are testing ManageIQ with Oracle Cloud Infrastructure (OCI) to validate provider integration. Provider authentication validates successfully, but inventory refresh fails consistently with HTTP 429 TooManyRequests. After retrying later, we also observe 404 errors. This prevents stable inventory collection.

Environment

ManageIQ version: Kasparov

Deployment: Docker (container-based)

Database: Internal

Provider: Oracle Cloud Infrastructure (OCI)

Region configured: <e.g. sa-saopaulo-1>

What we successfully validated

AWS provider works in the same ManageIQ deployment (credentials validate and refresh succeeds).

OCI provider can be added in the UI.

OCI credentials validate as Valid.

During refresh, we see multiple successful OCI API calls (HTTP 200) while iterating compartments/services before the 429 occurs.

Steps to reproduce

Add an Oracle Cloud provider:

Region:

Tenancy OCID:

User OCID:

API private key:

Save provider and confirm Default Credentials = Valid

Trigger refresh (or wait for scheduled refresh):

UI: Compute → Clouds → Providers → → Refresh (or Refresh relationships)

Expected behavior

Inventory refresh should complete successfully (or handle throttling via retries/backoff) and populate resources.

Actual behavior

We observe successful OCI API responses (HTTP 200) first, and then the refresh fails with HTTP 429 TooManyRequests. After retrying later, we also observe 404 / authorization-not-found messages.

Examples of successful OCI API responses (HTTP 200) during refresh

{"@timestamp":"2026-03-10T14:18:12.868046","hostname":"","pid":287,"tid":"a94c","service":"oracle","level":"debug","message":"API Response Received:\nData: [#<OCI::Identity::Models::AvailabilityDomain ... @name="BAZQ:SA-SAOPAULO-1-AD-1" ... @compartment_id="">]\nStatus code: 200\nHeaders: #Net::HTTPOK:..."}
{"@timestamp":"2026-03-10T14:18:12.963601","hostname":"","pid":287,"tid":"a94c","service":"oracle","level":"debug","message":"API Response Received:\nData: [#<OCI::Identity::Models::AvailabilityDomain ... @name="BAZQ:SA-SAOPAULO-1-AD-1" ... @compartment_id="">]\nStatus code: 200\nHeaders: #Net::HTTPOK:..."}
{"@timestamp":"2026-03-10T14:18:25.072453","hostname":"","pid":297,"tid":"a94c","service":"oracle","level":"debug","message":"API Response Received:\nData: #<OCI::Identity::Models::User ... @name="" ...>\nStatus code: 200\nHeaders: #Net::HTTPOK:..."}
{"@timestamp":"2026-03-10T14:18:25.176987","hostname":"","pid":297,"tid":"cd00","service":"oracle","level":"debug","message":"API Response Received:\nData: [#<OCI::Streaming::Models::StreamSummary ... @name="" ...>]\nStatus code: 200\nHeaders: #Net::HTTPOK:..."}

Error sequence (429 followed by authorization/resource-not-found message)

{"@timestamp":"2026-03-10T13:39:16.768612","hostname":"","pid":287,"tid":"a94c","service":"evm","level":"err","message":"MIQ(MiqQueue#deliver) Message id: [33206], Error: [{ 'message': 'Too many requests for the tenant', 'status': 429, 'code': 'TooManyRequests', 'opc-request-id': '' }]"}
{"@timestamp":"2026-03-10T13:39:16.768811","hostname":"","pid":287,"tid":"a94c","service":"evm","level":"err","message":"[ManageIQ::Providers::BaseManager::Refresher::PartialRefreshError]: { 'message': 'Too many requests for the tenant', 'status': 429, 'code': 'TooManyRequests', 'opc-request-id': '' } Method:[block in method_missing]"}
{"@timestamp":"2026-03-10T13:40:32.384294","hostname":"","pid":297,"tid":"c404","service":"evm","level":"err","message":"[OCI::Errors::ServiceError]: Authorization failed or requested resource not found. Method:[block in method_missing]"}

The last line indicates Authorization failed or requested resource not found. The OCI user belongs to a group with broad tenancy-level policies (listed below), so we do not expect authorization to be the root cause.

OCI policies applied (group: ManageIQ)

Allow group ManageIQ to inspect all-resources in tenancy

Allow group ManageIQ to read all-resources in tenancy

Allow group ManageIQ to use all-resources in tenancy

Allow group ManageIQ to manage streams in tenancy

Allow group ManageIQ to use stream-pull in tenancy

Allow group ManageIQ to use stream-push in tenancy

Frequency

429: always (every refresh attempt after initial successful calls)

404: after 429 (observed on subsequent attempts)

Question

Is there a recommended way to work around or mitigate the HTTP 429 throttling during OCI refresh so we can reliably Discover and visualize all OCI resources in ManageIQ?

[Fryguy]

@agrare Can you take a look at this one? Either we've got an N+1 in there, or we need to add some throttling - @ferr7s Do you happen do know what OCI's API limits are?

[ferr7s]

@Fryguy I'm not sure if the OCI let you change or know the API Limit they use, I tried to find it on the OCI GUI and could not find it.

On the ManageIQ I tried to reduce the pool on the event_catcher config and reduce the refresh_interval on the oracle provider.

At the same time I tried to limit the permission of the user to only see some compartments but it did not take effect and got all the compartments ( I think that's happen because the request is made to the tenant )

[Fryguy]

Can you give us an idea of the scale of your environment? Number of vms, number of compartments, etc? I think we're going to bed that in order to replicate.

[ferr7s]

Our OCI environment is relatively small. Currently, we have 16 compartments configured and 5 virtual machines running in total.

[agrare]

The issue is the number of compartments, because we have to issue API calls for each resource for each compartment which gives us our NxM (N resource types x M compartments) and grows pretty quickly.

I created 9 compartments (plus 1 root compartment) and also see the [OCI::Errors::ServiceError]: Too many requests for the tenant error.

I'll see if they have added a way to request all resources without having to specify the compartment, if not we'll need a HTTP 429 error handler in the inventory collector.