Does solid queue have issues working with PGBouncer ? #537
Comments
|
Hey @jwoodrow! Solid Queue shouldn't have any problems with PGBouncer. The error you're getting seems to be related to Solid Queue's database not being correctly created. It's about a table missing: Have you run |
|
Hi @rosa I saw the other issue mentioning this (using In case I had borked my database (since it wasn't a production environment) I even replaced my database with a completely different one and redeployed and regenerated my database on it. When using the connection pool url (PGBouncer) I get this error almost immediately and enter a crash loop. I've also connected directly to the database to observe the DDL and the tables and indices are all there. |
|
Thanks for the extra information 🙏 That's very strange! Does your main application's database work fine with PGBouncer? Solid Queue doesn't access the DB in a different way from the main app; it uses Active Record and Rails's connection handling under the hood 🤔 |
|
Yes, it's currently the way it is setup actually. I'm using direct connection only for solid_queue in that env and all other dynos are configured to use the connection pool 🤔 |
|
Very strange indeed 🤔 And what happens when you try to enqueue a job from your app? Does it also fail with a |
|
I just tried it out, both from controller actions and the rails console, no errors pop up as long as I have direct connection only on solid_queue workers |
|
Ah, I'm sorry it seems there is indeed an issue with PGBouncer (always relating to SolidQueue) also with other dynos, just much fewer because of less activity, I'm also using schema caching could this be related somehow ? |
|
I'm beginning to think this is really more of a Rails issue than anything. I've noticed that the queries being generated by SolidQueue did not explicitly provide the My monkey patch is two-fold just as an FYI
Monkey Patch (loaded in the initializers)module SolidQueueSupervisorPatch
private
def supervise
loop do
break if stopped?
set_procline
process_signal_queue
next if stopped?
recover_supervisor
reap_and_replace_terminated_forks
interruptible_sleep(1.second)
end
ensure
shutdown
end
def handle_claimed_jobs_by(terminated_fork, status)
return unless (registered_process = process&.supervisees&.find_by(name: terminated_fork.name))
error = SolidQueue::Processes::ProcessExitError.new(status)
registered_process.fail_all_claimed_executions_with(error)
end
def recover_supervisor
return if process.present?
@pid = nil
register
end
end
ActiveSupport.on_load(:solid_queue) do
# We're opting for a different schema approack so we need schema prefixing instead of table prefixing
SolidQueue.singleton_class.define_method(:table_name_prefix) { 'solid_queue.' }
SolidQueue::Supervisor.prepend SolidQueueSupervisorPatch
endI don't believe I'll let it run a bit more and keep you posted on if this was actually a Rails issue all along Update: so far have not seen any issues even loading solid_queue with 3k jobs in a short burst with pg_bouncer with my changes |
|
Oh huh! Great digging and very interesting find! However, I haven't fully understood it. Do you mean you've set a different |
|
That's exactly it, unless mod.respond_to?(:table_name_prefix)
define_method(:table_name_prefix) { "#{name}_" }
ActiveSupport.on_load(:active_record) do
mod.singleton_class.redefine_method(:table_name_prefix) do
"#{ActiveRecord::Base.table_name_prefix}#{name}_"
end
end
end(There's also something about prefixing the Usually this isn't an issue, but there appears to be some moments when Rails or PGBouncer or both, lose the notion of the As the code from Either way I do think the base issue is in how Rails and PGBouncer interact within isolated engines, and schemas disappearing out of nowhere 😅 |
Hi !
I was trying to migrate from Resque to SolidQueue on our staging environment before attempting production, but I'm running into issues.
Our database.yml looks something like this
with these applied to all environments equally and where the
_database_urlvariables prioritize a connection pool URL from PGBouncer (we are using heroku) since we use up a lot more than 500 open connections otherwise.The issue seems to be that when we are using the connection pool when running either
bin/jobsandbundle exec rails solid_queue:startthen we get random errors like these:but if I open a bash terminal on that application (disabling the workers from the procfile manually first) then proceed to unset the connection pool URL (so it falls back to direct connection to the database) and run
bin/jobsorrails solid_queue:startthen the error goes awayI'm thinking maybe solid queue has issues not being directly connected to the database for some queries, maybe from the supervisor ?
I don't mind supervisors and the such having direct access, they don't need many connections. But I want my jobs'
performto run using the connection pool to avoid saturating them, if there was a way to configure that so that different configs indatabase.ymlcan be used forSolidQueueactions and others for the execution of the worker itself, maybe pooling would no longer be an issueThe text was updated successfully, but these errors were encountered: