• Python/Django enthusiast:
  • factory_boy => Sprint with me!
  • python-ldap, django-ldapdb
• Lead dev @Polyconseil: Electric carsharing solutions (We're hiring!)
• Online:
  • @raphaelbarrois
  • https://github.com/rbarrois
  • Xelnor@Freenode

A big Django project (7 years old):

• 300 models
• Django startup time (./manage.py runserver): 12 seconds

Benchmarking:

[lazy]
startup       = 13   s
memory        = 903  MB
first_query   = 3747 ms

How does it work, exactly?

1. uWSGI start
2. uWSGI prepares worker processes
3. uWSGI starts accepting requests
4. Request arrives => load wsgi.py
5. wsgi.py loads Django
6. Django handles the request
7. time passes
8. Worker is recycled; GOTO 2

Default uWSGI configuration:

lazy = true

help: Set lazy mode (load apps in workers instead of master)

Let's disable it:

lazy = false

[lazy]
startup       = 13   s
memory        = 903  MB
first_query   = 3747 ms

[prefork]
startup       = 132  s
memory        = 769  MB
first_query   = 85   ms

Without lazy, each worker loads the whole codebase post-fork. But it's ready for the first request.

1. uWSGI start
2. uWSGI loads wsgi.py
3. uWSGI forks worker processes
4. Worker loads Django
5. Worker is ready, uWSGI accepts requests
6. Django handles the request
7. time passes
8. Worker is recycled; GOTO 3

import os
DJANGO_WARMUP_URL = os.environ.get('DJANGO_WARMUP_URL')
app = None

def get_wsgi_application():
    from django.core import wsgi
    global app
    app = wsgi.get_wsgi_application()
    if DJANGO_WARMUP_URL:
        warmup_django()
    return app

def warmup_django():

    # Setup the whole wsgi standard headers we do not care about.
    env = make_warmup_wsgi_env()
    wsgiref.util.setup_testing_defaults(env)

    # Boot the app
    def start_response(status, response_headers, exc_info=None):
        assert status == "200 OK"
        return io.BytesIO()  # Fake socket

    global app
    app(env, start_response)

    # Close connections before forking
    from django.db import connections
    for conn in connections.all():
        conn.close()

[lazy]
startup       = 13   s
memory        = 903  MB
first_query   = 3747 ms

[prefork]
startup       = 132  s
memory        = 769  MB
first_query   = 85   ms

[prewarm]
startup       = 9    s
memory        = 316  MB
first_query   = 121  ms

1. uWSGI start
2. uWSGI loads wsgi.py, which calls get_wsgi_application()
3. uWSGI forks worker processes
4. Worker is ready, uWSGI accepts requests
5. Django handles the request
6. time passes
7. Worker is recycled; GOTO 3

NB: only 25 MB per extra worker!

Details online:

https://gist.github.com/rbarrois/044b2d9f7052a6542f7a3d79695d64c6