JenkinsApi::Exceptions::ForbiddenWithCrumb: Access denied. Please ensure that Jenkins is set up to allow access to this operation. A crumb was used in attempt to access operation.

[thebravoman]

I have migrated to jenkins 2.249 and this started happening.
(at the bottom I confirm that it is an issue with the library using jenkins_api_client 1.5.3)

E, [2020-10-14T17:46:29.384574 #4] ERROR -- : JenkinsApi::Exceptions::ForbiddenWithCrumb: Access denied. Please ensure that Jenkins is set up to allow access to this operation. A crumb was used in attempt to access operation. Access denied. Please ensure that Jenkins is set up to allow access to this operation. 
Traceback (most recent call last):
       16: from bin/rails:9:in `require'
       15: from railties (6.0.3.3) lib/rails/commands.rb:18:in `<top (required)>'
       14: from railties (6.0.3.3) lib/rails/command.rb:46:in `invoke'
       13: from railties (6.0.3.3) lib/rails/command/base.rb:69:in `perform'
       12: from thor (1.0.1) lib/thor.rb:392:in `dispatch'
       11: from thor (1.0.1) lib/thor/invocation.rb:127:in `invoke_command'
       10: from thor (1.0.1) lib/thor/command.rb:27:in `run'
        9: from railties (6.0.3.3) lib/rails/commands/console/console_command.rb:102:in `perform'
        8: from railties (6.0.3.3) lib/rails/commands/console/console_command.rb:19:in `start'
        7: from railties (6.0.3.3) lib/rails/commands/console/console_command.rb:70:in `start'
        6: from (irb):3
        5: from (irb):3:in `rescue in irb_binding'
        4: from lib/jenkins_client_factory.rb:11:in `force_job_workoff'
        3: from jenkins_api_client (26f0de9cab9b) lib/jenkins_api_client/job.rb:858:in `build'
        2: from jenkins_api_client (26f0de9cab9b) lib/jenkins_api_client/client.rb:422:in `api_post_request'
        1: from jenkins_api_client (26f0de9cab9b) lib/jenkins_api_client/client.rb:450:in `rescue in api_post_request'
JenkinsApi::Exceptions::ForbiddenWithCrumb (Access denied. Please ensure that Jenkins is set up to allow access to this operation. A crumb was used in attempt to access operation. Access denied. Please ensure that Jenkins is set up to allow access to this operation. )

The client is created with

  def self.new_client
    ENV["JENKINS_USERNAME"] || raise("no JENKINS_USERNAME provided")
    ENV["JENKINS_PASSWORD"] || raise("no JENKINS_PASSWORD provided")

    crt = (ENV["JENKINS_CRT"] || File.read("#{Rails.root}/keys/platform@jenkins.crt.pem"))
    crt || raise("no JENKINS_CRT provided")
    key = (ENV["JENKINS_KEY"] || File.read("#{Rails.root}/keys/platform@jenkins.key.pem"))
    key || raise("no JENKINS_KEY provided")

    # This crt and key files could be generated with
    # openssl pkcs12 -in path.p12 -out newfile.crt.pem -clcerts -nokeys
    # openssl pkcs12 -in path.p12 -out newfile.key.pem -nocerts -nodes
    # This of course is only if you have p12 file.
    JenkinsApi::Client.new(:server_url=>CAST_CONFIG["jenkins"]["server_url"],
                           crt: crt,
                           key: key,
                           ssl: true,
                           :username=>ENV["JENKINS_USERNAME"],
                           :password=>ENV["JENKINS_PASSWORD"])
  end

and the call is

new_client.job.build(build_name, {"param1"=>value1})

Here is the output in the console

opening connection to examplehost.com:443...
opened
starting SSL for examplehost.com:443...
SSL established, protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384
<- "GET /jenkins/api/json?tree=useCrumbs HTTP/1.1\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nAccept: */*\r\nUser-Agent: Ruby\r\nAuthorization: Basic xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+\r\nConnection: close\r\nHost: examplehost.com\r\n\r\n"
-> "HTTP/1.1 200 OK\r\n"
-> "Server: nginx/1.18.0 (Ubuntu)\r\n"
-> "Date: Wed, 14 Oct 2020 17:46:23 GMT\r\n"
-> "Content-Type: application/json;charset=utf-8\r\n"
-> "Content-Length: 64\r\n"
-> "Connection: close\r\n"
-> "X-Content-Type-Options: nosniff\r\n"
-> "X-Jenkins: 2.249.2\r\n"
-> "X-Jenkins-Session: XXXXXXXX\r\n"
-> "X-Frame-Options: deny\r\n"
-> "Content-Encoding: gzip\r\n"
-> "\r\n"
reading 64 bytes...
-> "\x1F\x8B\b\x00\x00\x00\x00\x00\x00..."
read 64 bytes
Conn close
I, [2020-10-14T17:46:23.167153 #4]  INFO -- : Crumbs turned on.  Fetching from the server.
opening connection to examplehost.com:443...
opened
starting SSL for examplehost.com:443...
SSL established, protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384
<- "GET /jenkins/crumbIssuer/api/json HTTP/1.1\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nAccept: */*\r\nUser-Agent: Ruby\r\nAuthorization: Basic xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+\r\nConnection: close\r\nHost: examplehost.com\r\n\r\n"
-> "HTTP/1.1 200 OK\r\n"
-> "Server: nginx/1.18.0 (Ubuntu)\r\n"
-> "Date: Wed, 14 Oct 2020 17:46:23 GMT\r\n"
-> "Content-Type: application/json;charset=utf-8\r\n"
-> "Content-Length: 155\r\n"
-> "Connection: close\r\n"
-> "X-Content-Type-Options: nosniff\r\n"
-> "X-Jenkins: 2.249.2\r\n"
-> "X-Jenkins-Session: XXXXXXXX\r\n"
-> "X-Frame-Options: deny\r\n"
-> "Content-Encoding: gzip\r\n"
-> "Set-Cookie: JSESSIONID.67fe54fc=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.node0; Path=/jenkins; HttpOnly\r\n"
-> "Expires: Thu, 01 Jan 1970 00:00:00 GMT\r\n"
-> "\r\n"
reading 155 bytes...
-> "\x1F\x8B\b\x00\x00\x00\x00\x00\x00..."
read 155 bytes
Conn close
opening connection to examplehost.com:443...
opened
starting SSL for examplehost.com:443...
SSL established, protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384

<- "POST /jenkins/job/build_name/buildWithParameters HTTP/1.1\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nAccept: */*\r\nUser-Agent: Ruby\r\nJenkins-Crumb: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \r\nContent-Type: application/x-www-form-urlencoded\r\nAuthorization: Basic 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+\r\nConnection: close\r\nHost: examplehost.com\r\nContent-Length: 25\r\n\r\n"
<- "params1=value1"
-> "HTTP/1.1 403 Forbidden\r\n"
-> "Server: nginx/1.18.0 (Ubuntu)\r\n"
-> "Date: Wed, 14 Oct 2020 17:46:24 GMT\r\n"
-> "Content-Type: text/html;charset=iso-8859-1\r\n"
-> "Transfer-Encoding: chunked\r\n"
-> "Connection: close\r\n"
-> "X-Content-Type-Options: nosniff\r\n"
-> "Set-Cookie: JSESSIONID.67fe54fc=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.node0; Path=/jenkins; HttpOnly\r\n"
-> "Cache-Control: must-revalidate,no-cache,no-store\r\n"
-> "Content-Encoding: gzip\r\n"
-> "\r\n"
-> "164\r\n"

Update 1:
I have seen https://stackoverflow.com/a/62957734/1266681
which mentions

According to Jenkins Directive First you have to check your Jenkins version if the version is < 2.176.2 then per Jenkins guideline CSRF tokens (crumbs) are now only valid for the web session they were created in to limit the impact of attackers obtaining them. Scripts that obtain a crumb using the /crumbIssuer/api URL will now fail to perform actions protected from CSRF unless the scripts retain the web session ID in subsequent requests.

I am not sure if Jenkins_api_client does this because we have just migrated to 2.249.2 and probably that is the cause.

Update 2
It might not be an issue with the library

I just did

$ curl -v http://localhost:8080/jenkins/crumbIssuer/api/json --user theuser
# Than took the crumb
curl -X POST http://localhost:8080/jenkins/job/jobName/buildWithParameters --user theuser -H 'Jenkins-Crumb: cccccccccc'

and jenkins returned

<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 403 No valid crumb was included in the request</title>
</head>
<body><h2>HTTP ERROR 403 No valid crumb was included in the request</h2>
<table>
<tr><th>URI:</th><td>/jenkins/job/jobName/buildWithParameters</td></tr>
<tr><th>STATUS:</th><td>403</td></tr>
<tr><th>MESSAGE:</th><td>No valid crumb was included in the request</td></tr>
<tr><th>SERVLET:</th><td>Stapler</td></tr>
</table>
<hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.30.v20200611</a><hr/>

</body>
</html>

Update 3:
The trick i think is that after "2.176 a session is required" but the two values send for Set-Cookie are different
The first one is
-> "Set-Cookie: JSESSIONID.67fe54fc=node01itvfkmabhuf21dsikujbn1il449.node0; Path=/jenkins; HttpOnly\r\n"
and the second is
-> "Set-Cookie: JSESSIONID.67fe54fc=node02geew3fiibpk1j4w6eciwzipn50.node0; Path=/jenkins; HttpOnly\r\n"

Update 4:
I can confirm it is an issue with the library jenkins_api_client

The reason is the following
I go to https://support.cloudbees.com/hc/en-us/articles/219257077-CSRF-Protection-Explained
and I follow the explanation of how to correctly call jenkins after 2.176 with curl
The code is

# Replace with your Jenkins URL and admin credentials
SERVER="http://localhost:8080"
# File where web session cookie is saved
COOKIEJAR="$(mktemp)"
CRUMB=$(curl -u "admin:admin" --cookie-jar "$COOKIEJAR" "$SERVER/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,%22:%22,//crumb)")
curl -X POST -u "admin:admin" --cookie "$COOKIEJAR" -H "$CRUMB" "$SERVER"/job/someJob/build

By using this code I have successfully triggered a build.
This call preserves the session between the crumbIssuer and the post request.

I cant see a way to do this with jenkins_api_client

[thebravoman]

I managed to fix this by adding

    end

      http.open_timeout = @http_open_timeout
      http.read_timeout = @http_read_timeout

      response = http.request(request)
+      @cookies = response["set-cookie"]
      case response
        when Net::HTTPRedirection then
          # If we got a redirect request, follow it (if flag set), but don't
          # go any deeper (only one redirect supported - don't want to follow
          # our tail)
          if follow_redirect
            redir_uri = URI.parse(response['location'])
            response = make_http_request(
              Net::HTTP::Get.new(redir_uri.path, false)
            )
          end
      end

I will prepare a pull request for this.

[jesseadams]

This didn't fix my issue, but #292 did.

[NathanZook]

This is a wholly inadequate way to handle cookies. There are multiple places in the file where cookies need to be read--including another place in the above snippet. But beyond that, look at https://stackoverflow.com/questions/1486703/how-to-implement-cookie-support-in-ruby-net-http . Not the selected response, of course. Look further in. Cookies are actually complex beasts. Dropping their attributes is an error. As I understand the standards, cookies with certain attribute settings MUST not be accepted. I could see a server deliberately sending a cookie with invalid attributes & reading it back to trap bots.

[thebravoman]

@NathanZook by "this is a wholly inadequate" do you mean #292 or you mean

      response = http.request(request)
+      @cookies = response["set-cookie"]
      case response
        when Net::HTTPRedirection then

If it is the second one, then yes, this is a workaround for my case that gives more information about the issue, but it is not a solution to the issue.