Pitch: Queuing System

pitch

(Tanner) #21

I think this is a great proof of concept. Really nice work. :+1:

I agree. I think this could help cleanup the Job type reference lookup code and make the package easier to configure overall.

That's wonderful if Redis does indeed handle the horizontal scaling for us. Do you have any references to documentation or otherwise that states LPUSH + RPOP works this way?


(Jimmy McDermott) #22

I don't have any explicit documentation that states that, but judging by this from the documentation it looks like we're safe (unless I'm not thinking of something):

So I'm thinking that the first worker to pop the job will get the metadata and since it will be immediately removed (with O(1) complexity) the other worker will never have a chance to grab it. Thoughts?


(Jimmy McDermott) #23

To follow up, maybe it makes more sense to use BRPOP: https://redis.io/commands/brpop


(John Connolly) #24

Your assumption about how redis serves clients is correct. Redis will only give that Job to one client. Having implemented this library https://github.com/John-Connolly/SwiftQ I think the best pattern is to use the reliable queue method. This means having 2 queues and using the brpoplpush command. This will allow for jobs not to be lost if one of the consumers crash. There is documentation on this method here https://redis.io/commands/rpoplpush


(Jimmy McDermott) #25

Thanks for all of that info! I'll definitely look into it. This is the first time I've stumbled across SwiftQ - is it still under active development? It looks fairly similar to what I'm trying to build.

Question for anyone in the thread - how does a system like this handle server restarts? Sure, the persistence makes sure that no metadata is lost, but what about the job(s) currently being handled?


(John Connolly) #26

No problem! It is kinda, I have been really busy with school lately and haven't had much time to work on it. I do plan on updating it to use NIO, I just need to find the time. For server restarts I think most systems have a way of shutting down gracefully, so they will stop consuming from the queue and wait until all jobs are finished before exiting. If a server crashes another server that is monitoring the processing queue for stale jobs would re-queue it. Here are some docs on how Sidekiq does it https://github.com/mperham/sidekiq/wiki/Signals


(Jimmy McDermott) #27

Interesting, I'll have to figure out the best way to accomplish that with NIO/Vapor. Thanks again!


(Jimmy McDermott) #28

Hi all,

I've been trying to figure out the best way to accomplish signal catching in Swift. This is what I stumbled upon, courtesy of this blog post: https://dev.iachieved.it/iachievedit/trapping-signals-with-swift-on-linux/

import Glibc

enum Signal:Int32 {
case HUP    = 1
case INT    = 2
case QUIT   = 3
case ABRT   = 6
case KILL   = 9
case ALRM   = 14
case TERM   = 15
}

typealias SigactionHandler = @convention(c)(Int32) -> Void
               
let hupHandler:SigactionHandler = { signal in
  print("Received HUP signal, reread config file")
}
                  
func trap(signum:Signal, action:SigactionHandler) {
  var sigAction = sigaction()  
                    
  sigAction.__sigaction_handler = unsafeBitCast(action, sigaction.__Unnamed_union___sigaction_handler.self)
                    
  sigaction(signum.rawValue, &sigAction, nil)
}
                  
// This method works
trap(.INT) { signal in
  print("Received INT signal")
  exit(0)
}

// And this works of course
trap(.HUP, action:hupHandler)

while true {
  select(0, nil, nil, nil, nil)
}

I've never worked directly with signal handling like this before so I want to make sure I'm on the right path before shoving it into the project. Can anyone chime in with whether or not this would be an appropriate way to handle the problem?


(Jean-Daniel) #29

Iā€™m not sure you can safely call print in a signal handler. The list of safe operations in a signal handler are rather short (see man signal-safety).

On thing I consider problematic with high level language and signal handler is that it is very difficult to define what can be done, as many operations will rely on the Swift runtime and you have no guarantee these side effects are safe. For instance, using class instances may produce call to retain/release.


(Jimmy McDermott) #30

Got it. So I guess this warrants discussion around both if it's worth it to try and gracefully shut down on a SIGTERM (I think it is) and the best way to go about doing that in Swift. Would love ideas from anyone that has them.