Pitch: Queuing System

[tanner0101 (Tanner)]

I think this is a great proof of concept. Really nice work.

@jdmcd

I want to add a config structure similar to MigrationsConfig or DatabasesConfig instead of doing everything statically, but this works for now.

I agree. I think this could help cleanup the Job type reference lookup code and make the package easier to configure overall.

@jdmcd

I believe the current lpush Redis system eliminates the need for a database mechanism to track which jobs are done since whichever queue process gets to that item in the queue first will pop it off and make it unavailable for any other horizontal instances.

That's wonderful if Redis does indeed handle the horizontal scaling for us. Do you have any references to documentation or otherwise that states LPUSH + RPOP works this way?

[jdmcd (Jimmy McDermott)]

@tanner0101

Do you have any references to documentation or otherwise that states LPUSH + RPOP works this way?

I don't have any explicit documentation that states that, but judging by this from the documentation it looks like we're safe (unless I'm not thinking of something):

Removes and returns the last element of the list stored at key .
RPOP | Redis

So I'm thinking that the first worker to pop the job will get the metadata and since it will be immediately removed (with O(1) complexity) the other worker will never have a chance to grab it. Thoughts?

[jdmcd (Jimmy McDermott)]

To follow up, maybe it makes more sense to use BRPOP: https://redis.io/commands/brpop

[John-Connolly]

Your assumption about how redis serves clients is correct. Redis will only give that Job to one client. Having implemented this library GitHub - John-Connolly/SwiftQ: Distributed Task Queue I think the best pattern is to use the reliable queue method. This means having 2 queues and using the brpoplpush command. This will allow for jobs not to be lost if one of the consumers crash. There is documentation on this method here RPOPLPUSH | Redis

[jdmcd (Jimmy McDermott)]

Thanks for all of that info! I'll definitely look into it. This is the first time I've stumbled across SwiftQ - is it still under active development? It looks fairly similar to what I'm trying to build.

Question for anyone in the thread - how does a system like this handle server restarts? Sure, the persistence makes sure that no metadata is lost, but what about the job(s) currently being handled?

[John-Connolly]

No problem! It is kinda, I have been really busy with school lately and haven't had much time to work on it. I do plan on updating it to use NIO, I just need to find the time. For server restarts I think most systems have a way of shutting down gracefully, so they will stop consuming from the queue and wait until all jobs are finished before exiting. If a server crashes another server that is monitoring the processing queue for stale jobs would re-queue it. Here are some docs on how Sidekiq does it Signals · mperham/sidekiq Wiki · GitHub

[jdmcd (Jimmy McDermott)]

Interesting, I'll have to figure out the best way to accomplish that with NIO/Vapor. Thanks again!

[jdmcd (Jimmy McDermott)]

Hi all,

I've been trying to figure out the best way to accomplish signal catching in Swift. This is what I stumbled upon, courtesy of this blog post: Trapping Signals with Swift on Linux

import Glibc

enum Signal:Int32 {
case HUP    = 1
case INT    = 2
case QUIT   = 3
case ABRT   = 6
case KILL   = 9
case ALRM   = 14
case TERM   = 15
}

typealias SigactionHandler = @convention(c)(Int32) -> Void
               
let hupHandler:SigactionHandler = { signal in
  print("Received HUP signal, reread config file")
}
                  
func trap(signum:Signal, action:SigactionHandler) {
  var sigAction = sigaction()  
                    
  sigAction.__sigaction_handler = unsafeBitCast(action, sigaction.__Unnamed_union___sigaction_handler.self)
                    
  sigaction(signum.rawValue, &sigAction, nil)
}
                  
// This method works
trap(.INT) { signal in
  print("Received INT signal")
  exit(0)
}

// And this works of course
trap(.HUP, action:hupHandler)

while true {
  select(0, nil, nil, nil, nil)
}

I've never worked directly with signal handling like this before so I want to make sure I'm on the right path before shoving it into the project. Can anyone chime in with whether or not this would be an appropriate way to handle the problem?

[Jean-Daniel]

I’m not sure you can safely call print in a signal handler. The list of safe operations in a signal handler are rather short (see man signal-safety).

On thing I consider problematic with high level language and signal handler is that it is very difficult to define what can be done, as many operations will rely on the Swift runtime and you have no guarantee these side effects are safe. For instance, using class instances may produce call to retain/release.

[jdmcd (Jimmy McDermott)]

@Jean-Daniel

as many operations will rely on the Swift runtime and you have no guarantee these side effects are safe

Got it. So I guess this warrants discussion around both if it's worth it to try and gracefully shut down on a SIGTERM (I think it is) and the best way to go about doing that in Swift. Would love ideas from anyone that has them.