[Bug] Improper regex used in removeTemplates when parsing docker-compose files

[Sculas]

How are you running Renovate?

A Mend.io-hosted app

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

No response

Please tell us more about your question or problem

A couple of months ago, I noticed that suddenly, almost all of my docker-compose dependencies weren't detected anymore.
After some research, I've finally figured it out today. Recently, this PR was merged: #31206

That PR added an option to remove Jinja2 templates from a YAML file, so the YAML parser can properly parse it.
However, the regex used to remove certain templates seems to be incorrect. See the following docker-compose file using Jinja2:

# {{ ansible_managed }}
---
version: "3"

networks:
  {{ docker_network }}:
    external: true

services:
  dozzle:
    container_name: dozzle
    image: amir20/dozzle:v8.8.2
    restart: unless-stopped
    networks:
      - "{{ docker_network }}"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

Now let's look at what parseSingleYaml does. If removeTemplates is set, it runs: content.replace(regEx(/\s+{{.+?}}:.+/gs), '')

renovate/lib/util/yaml.ts

Lines 143 to 151 in f40c035

	function massageContent(content: string, options?: YamlOptions): string {
	if (options?.removeTemplates) {
	return content
	.replace(regEx(/\s+{{.+?}}:.+/gs), '')
	.replace(regEx(/{{`.+?`}}/gs), '')
	.replace(regEx(/{{.+?}}/gs), '')
	.replace(regEx(/{%`.+?`%}/gs), '')
	.replace(regEx(/{%.+?%}/g), '');
	}

If we put that regex and the contents into something like RegExr (link contains reproduction), we can see almost the whole file is matched, which then explains why Renovate is unable to parse any of my docker-compose files:

DEBUG: docker-compose.extractPackageFile(templates/dozzle/docker-compose.yml.j2)
DEBUG: Parsing Docker Compose config YAML failed
{
  "err": {
    "message": "Schema error",
    "stack": "ZodError: Schema error\n    at Object.get error [as error] (/usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/zod/lib/types.js:55:31)\n    at ZodUnion.parse (/usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/zod/lib/types.js:160:22)\n    at parseSingleYaml (/usr/local/renovate/lib/util/yaml.ts:136:17)\n    at Object.extractPackageFile (/usr/local/renovate/lib/modules/manager/docker-compose/extract.ts:39:29)\n    at extractPackageFile (/usr/local/renovate/lib/modules/manager/index.ts:75:9)\n    at getManagerPackageFiles (/usr/local/renovate/lib/workers/repository/extract/manager-files.ts:58:43)\n    at /usr/local/renovate/lib/workers/repository/extract/index.ts:57:28\n    at async Promise.all (index 2)\n    at extractAllDependencies (/usr/local/renovate/lib/workers/repository/extract/index.ts:54:26)\n    at extract (/usr/local/renovate/lib/workers/repository/process/extract-update.ts:140:28)\n    at extractDependencies (/usr/local/renovate/lib/workers/repository/process/index.ts:158:26)\n    at Object.renovateRepository (/usr/local/renovate/lib/workers/repository/index.ts:71:9)\n    at attributes.repository (/usr/local/renovate/lib/workers/global/index.ts:202:11)\n    at start (/usr/local/renovate/lib/workers/global/index.ts:187:7)\n    at /usr/local/renovate/lib/renovate.ts:19:22",
    "issues": [
      "Expected object, received null",
      "Expected object, received null"
    ]
  }
  "packageFile": "templates/dozzle/docker-compose.yml.j2"
}

So then, what's the fix? It seems that the regex can't find the end of the template.
So my first attempt was wrapping it in quotation marks, which unfortunately didn't work as well yet.
I also had to remove the Ansible header for it to work:

---
version: "3"

networks:
  "{{ docker_network }}":
    external: true

# ... same as before

I'm not entirely sure what the proper fix for this is, but I at least thought I'd report it. If I have to live with this, then that's perfectly fine as well for me, since in the end, YAML will parse it properly anyway (it just ignores the quotation marks).

Logs (if relevant)

No response

[rarkins]

I was sure I had responded to this yesterday, but can't find any comment here, or find a duplicate discussion.

This is related though: #33105

@viceice I think the jinja2 template removal seems too aggressive. It's not just removing templates, it's removing whitespace and a colon around it in an attempt to "repair" the result too, but that ends up breaking things like above.

[viceice]

🤔 yes, somebody impproved my original regex, so it's now too greedy. not sure if we can implement a fully working solution without a proper parser.

[rarkins]

@viceice were you the somebody? #29061

[viceice]

maybe 🤷‍♂️

[viceice]

maybe the reflex need to be /^\s+{{.+?}}:.+$/gs

[rarkins]

What about \s*?{{.+?}}(:.+?)??

[viceice]

🤷‍♂️

[rarkins]

Should we remove /\s+{{.+?}}:.+/gs altogether, or narrow it? For example to /\n\s+{{.+?}}:\s*\n/gs

[rarkins]

#33190

[MGSousa]

Since we have now a more proper way of stripping variables with stripTemplates function.
What about completely removing the regex code?

I tested without it on the docker-compose manager locally and it works, all images are correctly discovered and updated without any AggregateError issues.

[rarkins]

Thanks for the clarification. What we need is this:

• A new test which currently fails
• new test works after code changes
• old tests continue working