Process Jobs In Order By Key

[ProgrammerZ]

I'm trying to find a way to preserve the execution order of jobs exactly as they were dispatched by arbitrary key.

For example when we sync data with external API we need to process all the jobs for a particular resource in sequence to avoid errors. If the Job is delayed – we need to halt any other jobs to that resource until the first one is processed. Then second in order and so on until all will be resolved by some key.

I can’t use Chained Jobs or Batched (at least to my knowledge) since these jobs are dispatched in no particular order.

I'm looking for ways to accomplish this using Laravel's existing job queue system without adding unnecessary complexity.

[ghost]

You must use FIFO queue. SQS FIFO queue for example. No code change needed.

[ProgrammerZ]

@bulletproof-coding Thanks. I see SQS queues still have max delay of 15min, not sure how that work if we need to delay queue for hours. Checking docs.

Still looking for ideas for FIFO queue to be built within Laravel itself.

[ghost]

FIFO queues and delay in the same sentence is nonsense:). If you delay a job it will block all other jobs. Do not use mysql queue in production because it has issues acquiring the row lock in an atomic way.

[ProgrammerZ]

It seems I’m misunderstanding the term FIFO. Everywhere I read its definition, the only requirement I see is: "FIFO ensures that the first element added to the queue is the first to be removed," in different variations. There is no mention that the queue cannot be delayed or halted. However, if the term FIFO does not allow delays, then I don’t need FIFO. :)

I’m not trying to block the whole queue—actually, the opposite. For example, I could send all these jobs that need to be processed this way to a separate queue and add a separate worker to process them one by one. That would partially solve the issue, but only until a job is released back into the queue for any reason. So, it doesn’t fully solve the problem.

I don’t think this is the right approach, but currently, the only solution I can think of is to add a column to the jobs table to store a key with a sequence number. Each job that needs to be processed in order would first check during dispatch whether any other jobs exist with the same key. It would then assign itself an appropriate sequence number. Before processing, the job would ensure that no other jobs with the same key are ahead of it; otherwise, it would be released back into the queue for later processing. However, this approach doesn’t feel quite right to me.

[ghost]

You can use a normal queue that supports delay and dispatch on the FIFO queue only when you want them to be processed.

[ProgrammerZ]

Thanks, I will look deeper into SQS FIFO queue, but based on my current understanding it would not solve the issue without hacks around SQS FIFO queue. I don't need to delay the Job itself when dispatched, only then it can not be completed successfully, for example API unavailable for a day. Technically it's called releasing job back to the queue.