Connector
PowerBI
Feature area
Metadata ingestion
Describe the bug
Summary
When ingesting data from Power BI, the agent starts by fetching all groups from Power BI
via the GetGroupsAsAdmin endpoint. Here it uses a default page size limit of 100 which
cannot be overridden to a higher value. When a tenant
has more than 100 groups, the endpoint is paged through and called multiple times
by design. This is only reliable if the number of groups in the tenant is small. The larger the
number, the more likely it is that other services using the same endpoint will impact the
ingestion process or the ingestion process will impact other services using the same endpoint.
Problem
GetGroupsAsAdmin enforces a strict per-tenant rate limit of 50 requests
per hour or 15 requests per minute
(docs).
While ingestion still completes, the page size of 100 generates an unnecessary
number of calls against an endpoint with a strict rate limit. This has two
consequences:
- Shared quota impact — the extra calls consume a rate limit that is shared
across the whole tenant, potentially affecting other services that rely on the
same endpoint, especially as the number of groups approaches 5000.
- Lockout at scale — when the number of groups exceeds 5000, OpenMetadata can
effectively lock other services out for one hour and eventually time out after
approximately 50 minutes.
Increasing the page size in the agent configuration does not help, because it is
overridden by a hardcoded limit in the code.
To Reproduce
To see that page size in the agent configuration is ignored, set the page size to any number between 101 and 5000 in a tenant with at least 101 groups.
Expected behavior
When changing the page size to 150, the endpoint should be called once if the tenant has between 101 and 150 groups.
Connection / ingestion config
Logs
OS
No response
Python version
No response
OpenMetadata version
No response
OpenMetadata Ingestion package version
No response
Additional context
Creating a PR (shortly), to solve the problem
Pre-submission checklist
Connector
PowerBI
Feature area
Metadata ingestion
Describe the bug
Summary
When ingesting data from Power BI, the agent starts by fetching all groups from Power BI
via the
GetGroupsAsAdminendpoint. Here it uses a default page size limit of 100 whichcannot be overridden to a higher value. When a tenant
has more than 100 groups, the endpoint is paged through and called multiple times
by design. This is only reliable if the number of groups in the tenant is small. The larger the
number, the more likely it is that other services using the same endpoint will impact the
ingestion process or the ingestion process will impact other services using the same endpoint.
Problem
GetGroupsAsAdminenforces a strict per-tenant rate limit of 50 requestsper hour or 15 requests per minute
(docs).
While ingestion still completes, the page size of 100 generates an unnecessary
number of calls against an endpoint with a strict rate limit. This has two
consequences:
across the whole tenant, potentially affecting other services that rely on the
same endpoint, especially as the number of groups approaches 5000.
effectively lock other services out for one hour and eventually time out after
approximately 50 minutes.
Increasing the page size in the agent configuration does not help, because it is
overridden by a hardcoded limit in the code.
To Reproduce
To see that page size in the agent configuration is ignored, set the page size to any number between 101 and 5000 in a tenant with at least 101 groups.
Expected behavior
When changing the page size to 150, the endpoint should be called once if the tenant has between 101 and 150 groups.
Connection / ingestion config
Logs
OS
No response
Python version
No response
OpenMetadata version
No response
OpenMetadata Ingestion package version
No response
Additional context
Creating a PR (shortly), to solve the problem
Pre-submission checklist