Puppet upgrade breaks APT

[TobiPeterG]

Describe the Bug

So far, we ran puppet 7 on our machines. Since it's going EOL soon, we are in the process of migrating to puppet8.
We use this module to automatically update the agents to puppet 8. While this works, we get this error:

Error: /Stage[main]/Apt::Update/Exec[apt_update]: Failed to call refresh: '/usr/bin/apt-get update' returned 100 instead of one of [0]
Error: /Stage[main]/Apt::Update/Exec[apt_update]: '/usr/bin/apt-get update' returned 100 instead of one of [0]

And sometimes:

Error: Could not prefetch package provider 'apt': Execution of '/usr/bin/apt-mark showmanual' returned 100: E: Conflicting values set for option Signed-By regarding source https://apt.puppet.com/ focal: /etc/apt/keyrings/GPG-KEY-puppet-20250406.asc != 
E: Conflicting values set for option Signed-By regarding source https://apt.puppet.com/ focal: /etc/apt/keyrings/GPG-KEY-puppet-20250406.asc != 
E: The list of sources could not be read.

When running APT, we see this:

apt update
E: Conflicting values set for option Signed-By regarding source https://apt.puppet.com/ focal: /etc/apt/keyrings/GPG-KEY-puppet-20250406.asc != 
E: Conflicting values set for option Signed-By regarding source https://apt.puppet.com/ focal: /etc/apt/keyrings/GPG-KEY-puppet-20250406.asc != 
E: The list of sources could not be read.
E: Conflicting values set for option Signed-By regarding source https://apt.puppet.com/ focal: /etc/apt/keyrings/GPG-KEY-puppet-20250406.asc != 
E: Conflicting values set for option Signed-By regarding source https://apt.puppet.com/ focal: /etc/apt/keyrings/GPG-KEY-puppet-20250406.asc != 
E: The list of sources could not be read.
root@playground1:/tmp# apt update
E: Conflicting values set for option Signed-By regarding source https://apt.puppet.com/ focal: /etc/apt/keyrings/GPG-KEY-puppet-20250406.asc != 
E: Conflicting values set for option Signed-By regarding source https://apt.puppet.com/ focal: /etc/apt/keyrings/GPG-KEY-puppet-20250406.asc != 
E: The list of sources could not be read.
root@playground1:/tmp# apt install hiera-eyaml
E: Conflicting values set for option Signed-By regarding source https://apt.puppet.com/ focal: /etc/apt/keyrings/GPG-KEY-puppet-20250406.asc != 
E: Conflicting values set for option Signed-By regarding source https://apt.puppet.com/ focal: /etc/apt/keyrings/GPG-KEY-puppet-20250406.asc != 
E: The list of sources could not be read.
E: Conflicting values set for option Signed-By regarding source https://apt.puppet.com/ focal: /etc/apt/keyrings/GPG-KEY-puppet-20250406.asc != 
E: Conflicting values set for option Signed-By regarding source https://apt.puppet.com/ focal: /etc/apt/keyrings/GPG-KEY-puppet-20250406.asc != 
E: The list of sources could not be read.

We had a look at the lists in /etc/apt/sources.list.d/ and found a puppet7-release.list and the pc_repo.list. On systems where we manually upgraded puppet to version 8 we also had puppet8-release.list. The release lists come from the release packages we use to initially install puppet on our machines (as recommended in the official docu https://www.puppet.com/docs/puppet/8/install_puppet.html). The release lists however do not contain a link to the asc file to sign it, the new pc_repo list does. Removing the release lists "solves" the issue, but is not elegant. We have them on every machine and they are created using the official way to install puppet.

Our config looks like this:

class {'puppet_agent':
  package_version => 'latest',
  collection      => 'puppet8',
  service_names   => [],
  config          => [{section => main, setting => runinterval, value => '5m'}],
  before          => Service['puppet'],
}

service { 'puppet':
  ensure => running,
  enable => true,
}

The puppet8-release.list looks like this:

# Puppet 8 focal Repository
deb http://apt.puppet.com focal puppet8

pc_repo.list:
cat /etc/apt/sources.list.d/pc_repo.list

# This file is managed by Puppet. DO NOT EDIT.
# pc_repo
deb [signed-by=/etc/apt/keyrings/GPG-KEY-puppet-20250406.asc] https://apt.puppet.com focal puppet8

Curiously, on systems running puppet7, the pc_repo.list looks like this:

# This file is managed by Puppet. DO NOT EDIT.
# pc_repo
deb https://apt.puppet.com focal puppet7

and does not include the signing key.

Am I missing something?
Should we install puppet differently in the future?
Shouldn't the puppet release deb create the file with link to the signing key?

Expected Behavior

The puppet upgrade should not break APT

Steps to Reproduce

Steps to reproduce the behavior:

1. Install puppet initially using the release package from their website https://apt.puppetlabs.com/
2. Run puppet with this module with the config shown above (setting puppet8)
3. try to use APT

Environment

• Version 4.22.0
• Platform Ubuntu 20.04

Additional Context

---

[cliff-wakefield-acurus]

Also running into this issue.

Previously apt would just warn that two sources existed for the puppet repositories, now it errors as one is signed, and the other is not.

Unclear why this module and the puppet agent official package cannot be made to co-exist with each other. Not sure how this was not tested for either.

[joshcooper]

I can reproduce as well (installed 7.34.0 and attempted to upgrade to latest puppet8 on apt.puppet.com):

# dpkg -l | grep puppet
ii  puppet-agent               7.34.0-1focal                     amd64        The Puppet Agent package contains all of the elements needed to run puppet, including ruby, facter, and hiera.
ii  puppet7-release            7.0.0-20focal                     all          Release packages for the Puppet 7 repository
# cat /etc/apt/sources.list.d/puppet7-release.list 
# Puppet 7 focal Repository
deb http://apt.puppet.com focal puppet7
# cat /etc/apt/sources.list.d/pc_repo.list         
# This file is managed by Puppet. DO NOT EDIT.
# pc_repo
deb [signed-by=/etc/apt/keyrings/GPG-KEY-puppet-20250406.asc] https://apt.puppet.com focal puppet8
# apt update
E: Conflicting values set for option Signed-By regarding source https://apt.puppet.com/ focal: /etc/apt/keyrings/GPG-KEY-puppet-20250406.asc != 
E: The list of sources could not be read.
# cat manifest.pp 
class {'puppet_agent':
  package_version => 'latest',
  collection      => 'puppet8',
}
# puppet apply manifest.pp        
Notice: Compiled catalog for f9d832890253 in environment production in 0.15 seconds
Error: Could not prefetch package provider 'apt': Execution of '/usr/bin/apt-mark showmanual' returned 100: E: Conflicting values set for option Signed-By regarding source https://apt.puppet.com/ focal: /etc/apt/keyrings/GPG-KEY-puppet-20250406.asc != 
E: The list of sources could not be read.

Our options seem to be:

1. Republish the puppet7-release packages to include signed-by
2. Update this module to remove signed-by from the pc.repo
3. Have the puppet_agent module delete the old repo, so they don't conflict
4. Other options?

Also the old apt.puppet.com distribution server won't be receiving updates, due to https://www.puppet.com/products/puppet-core.

Release packages on the new distribution do correctly contain signed-by as expected

# curl -sLO https://apt-puppetcore.puppet.com/public/puppet7-release-focal.deb
# dpkg -i puppet7-release-focal.deb 
Selecting previously unselected package puppet7-release.
(Reading database ... 4675 files and directories currently installed.)
Preparing to unpack puppet7-release-focal.deb ...
Unpacking puppet7-release (10.5.0-1focal) ...
Setting up puppet7-release (10.5.0-1focal) ...
# cat /etc/apt/sources.list.d/puppet7-release.list 
# Puppet 7 focal Repository
deb [signed-by=/etc/apt/keyrings/puppet-keyring.gpg] https://apt-puppetcore.puppet.com focal puppet7

And the puppet_agent module will do the same when this is merged 8783558#diff-be83ab7a0bf1eb873740284913898396895efcd5db84723fdd565fb7a698838bR100-R101

[bastelfreak]

Hi,

Republish the puppet7-release packages to include signed-by

IMO yes

I also think it doesn't make sense to use the puppet*-release packages and the puppetlabs/puppet_agent module. My profiles has this:

  # remove the old repo setup
  package { ['puppet7-release', 'puppet8-release', 'puppet-release']:
    ensure => 'absent',
  }
  # purging the old package won't properly remove the old repo setup
  -> file { ['/etc/apt/sources.list.d/puppet7-release.list', '/etc/apt/sources.list.d/puppet8-release.list', '/etc/apt/sources.list.d/puppet-release.list']:
    ensure => 'absent',
  }
  # configure the puppet repo
  -> class { 'puppet_agent':
    package_version => $package_version, # ensures that all agents run the same version as the puppetserver
    collection      => $collection,
    service_names   => [],
  }
  contain puppet_agent

It would be great if puppetlabs/puppet_agent could take care of purging the puppet*-release packages.