Pylons Worker Threads

I was reading a thread over at pylons-discuss about worker threads in pylons. Worker threads are useful if you need to execute a long running task, but want to return to the user immediately. You could run a new thread per request, but if you have many requests, it is probably better to queue things up.

To do this, you start a thread and use python’s Queue for managing the tasks. Here is a very basic implementation:

config/environment.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
# PYLONS IMPORTS

from myapp.lib.myworker import start_myworker

def load_environment(global_conf, app_conf):
    """Configure the Pylons environment via the ``pylons.config``
    object
    """
   
    # PYLONS STUFF GOES HERE
   
    # start worker
    start_myworker()

lib/myworker.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
import Queue
import threading

worker_q = Queue.Queue()

class MyWorkerThread(threading.Thread):
    def run(self):
        print 'Worker thread is running.'
       
        while True:
            msg = worker_q.get()
            try:
                # do a long running task...
                print 'We got %s, do something with it!' % (msg)
            except Exception, e:
                print 'Unable to process in worker thread: ' + str(e)
            worker_q.task_done()                
   
def start_myworker():
    worker = MyWorkerThread()
    worker.start()

And in your controller….

1
2
3
4
5
from myapp.lib.myworker import worker_q

class WorkerController(BaseController):    
    def do_task(self):
      worker_q.put({'id': generate_some_unique_id(), 'msg': 'your data goes here'})
Posted Wednesday, March 4th, 2009 under Technology.

Tags: , ,

Similar Posts

  • Chris Van

    Thanks Chris,

    You’ve done three things for me here that I am thankful for, one is show me how to call helper programs from the lib directory, two is a simple example of threading in python, and three is using the built in python queue.

    Much appreciated, and very clear too.

    Thanks a lot,

    Chris.

  • chrismoos

    Glad you found it helpful. Also, if you want to pass a SQLAlchemy object into the queue, you will experience thread issues. If you wanted to pass a user into your queue, either pass the individual values you need (user.name, user.username) etc,., or pass (user.id) and then in the thread you query and get the object there.

  • http://entic.net Anil

    Thanks for the great code! I am having problems with passing variables to the thread though. I keep getting:

    No object (name: app_globals) has been registered for this thread

    In the start_worker, I have:

    worker = TagMail(app_globals.MAX_TAG_LENGTH)
    worker.start()

    Any help?
    Thanks

  • Chris Moos

    Maybe you need to import app_globals? Pylons right?

  • http://entic.net Anil

    I have imported it. Still no luck. Oh well. Thanks!

    from pylons import app_globals
    x = app_globals.MAX_TAG_LENGTH
    worker = TagMail(x)
    worker.start()

  • http://jaux.net jaux

    @Anil simply change start_myworker() to start_myworker(max_tag_len) and call it in load_environment with appropriate variable. app_globals is a pylons global which is available for pylons threads (the threads pylons creates to handle requests).