Lookup error (403) on public endpoints for customManager/customDatasources

[chouetz]

How are you running Renovate?

A Mend.io-hosted app

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

No response

Please tell us more about your question or problem

Hello 👋🏽
I would like some help to understand why Renovate seems to work alternatively with my configuration, using a public endpoint as customDatasource

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "enabledManagers": ["custom.regex"],
  "labels": ["dependencies"],
  "customManagers" : [
    {
      "customType": "regex",
      "fileMatch": [".protoc-version"],
      "matchStrings": [
        "(?<currentValue>[0-9]+.[0-9]+(-rc[0-9]+)?)"
      ],
      "depNameTemplate": "protoc",
      "versioningTemplate": "loose",
      "datasourceTemplate": "custom.protoc"
    }
  ],
  "customDatasources": {
    "protoc": {
      "defaultRegistryUrlTemplate": "https://api.github.com/repos/protocolbuffers/protobuf/releases",
      "transformTemplates": [
        "{\"releases\": $map($.[tag_name,published_at], function($v) { {\"version\": $v[0], \"releaseTimestamp\": $v[1] } }) }"
      ]
    }
  }
}

When I activate this configuration a pull-request is created, but the following runs of renovate close it.
Looking at the job runs on my dashboard, I see the following:

DEBUG: GET https://api.github.com/repos/protocolbuffers/protobuf/releases = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=403 retryCount=0, duration=4)
DEBUG: Failed to look up custom.protoc package protoc
{
  "dependency": "protoc"
  "packageFile": ".protoc-version"
}

The endpoint I use requires no authentication (a simple curl https://api.github.com/repos/protocolbuffers/protobuf/releases works fine) so I don't understand why the PR is created when I enable the rule, and is then auto-closed following these 403 errors.
Am I missing something in the configuration?

Thanks in advance for your help.

Logs (if relevant)

Logs

Replace this text with your logs, between the starting and ending triple backticks

[github-actions[bot]]

Hi there,

Get your discussion fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this. Discussions without reproductions are less likely to be converted to Issues.

Please follow these steps:

1. Read our guide on creating a minimal reproduction.
2. Go to our minimal reproduction template repository.
3. Select the Use this template button to create a new repository based on the template.
4. Work on the minimal reproduction in your own repository.
5. Fill out the information in your repository's README.md.
6. Add the link to your reproduction to the first post of your Discussion. If you are not the original author, you can post a new comment with the link.

If you need help with running renovate on your minimal reproduction repository, please refer to our Running Renovate guide.
Good luck,

The Renovate team

[chouetz]

Here is the minimal reproduction for this issue https://github.com/chouetz/renovate-reproduction-34204
The error on the dashboard is the same

Thanks a lot in advance for your support

[rarkins]

This doesn't reproduce for me when I run locally.

My best guess is:

• This custom datasource isn't using the app's github token for auth
• Although this is a public repo, github limits requests to api.github.com to 60 requests per hour
• The Renovate app's IP's already have exceeded that rate for unauthenticated

[chouetz]

Thanks for your test.
Do you know if I can update the configuration to force using the app's github token?
On the log I can also see DEBUG: hostRules: no authentication for api.github.com
Using an app would indeed reduce the risk of rate limiting (even if we don't receive a rate limitation error message)

[rarkins]

One way or another there needs to be a hostRules matching the hostType of your custom manager. Adding 'custom' to

renovate/lib/constants/platforms.ts

Lines 22 to 31 in 858f26b

	export const GITHUB_API_USING_HOST_TYPES = [
	'github',
	'github-releases',
	'github-release-attachments',
	'github-tags',
	'pod',
	'hermit',
	'github-changelog',
	'conan',
	];

might also solve it, but I'm not sure if there's any unintended consequences of that

[chouetz]

Hello!
I forced the hostRules on the configuration to use the token, and the logs now display this

DEBUG: hostRules: applying Bearer authentication for api.github.com
DEBUG: Using queue: host=api.github.com, concurrency=16
DEBUG: GET https://api.github.com/repos/protocolbuffers/protobuf/releases = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=401 retryCount=0, duration=28)

So maybe I misunderstand the logs but it seems it now says it uses an authentication for the call and I still get the 401 error.

As it seems it works from time to time, disabling the pruneStaleBranches could also be a workaround, even if I would like to understand the error.

[rarkins]

You're getting a 401 response, which indicates something wrong with the token

[MPV]

Any ideas on how a usable setup of this could look like?
I described in a thread below how I'm also experiencing almost the same (but without the hostRules fix just yet).

[MPV]

For the hosted Renovate, what might be needed to get the custom datasource to use a proper/built-in token for api.github.com?

In the logs I see for example:

DEBUG: hostRules: no authentication for api.github.com
DEBUG: GET https://api.github.com/repos/ibmruntimes/semeru11-binaries/releases = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=403 retryCount=0, duration=5)
DEBUG: Failed to look up custom.semeru package java
{
  "dependency": "java"
  "packageFile": ".tool-versions"
}

It worked before (just 2 days ago), but just started failing for me like this / over here:

• Update dependency java to v11.0.26+4_openj9-0.49.0 (from 11.0.20.1+1_openj9-0.40.0) - autoclosed MPV/renovate-asdf-semeru#3
• https://developer.mend.io/github/MPV/renovate-asdf-semeru/-/job/01953a7e-8f83-78fe-96db-812e5a2379c3
• Dependency Dashboard MPV/renovate-asdf-semeru#2

[chouetz]

On my side I tested with

 "hostRules": [
    {
      "hostType": "custom.protoc",
      "token": "process.env.GITHUB_TOKEN"
    }
  ]

But it seems it is now only working because I set a PAT in server side. This is unsatisfactory for me I think I'll try to do the update suggested above in renovate/lib/constants/platforms.ts as a contribution and test if it has no negative consequences

[rarkins]

The string "process.env.GITHUB_TOKEN" will never work in a JSON file. It's a literal string and won't do any env substitution

[chouetz]

Ho you are right indeed. Another workaround which could be accepted (at least in my situation it's ok) is to set pruneStaleBranches to `false.

[chouetz]

Ok there is an alternative solution to my initial issue: as a I rely from "standard" Github content, I can also use github-release or git-refs datasources (I missed them when reading the docs).
Sorry for the inconvenience, using these datasources solved the issues on my end.
Many thanks again for your prompt and valuable support.

[MPV]

Sounds promising (and fingers crossed it solves my use case too). 🤞

Would you be up for sharing/showing how you did that? 🙏🏻