Django is a brilliant web framework. In fact it is my most favourite one for various reasons. An year and a half ago, I switched to Python and Django for all my web development. I am a big fan of the eco system and the many third party packages. Particularly I use Django REST Framework whenever I need to create APIs. Having said that, Django was more than good enough for basic HTTP requests. But the web has changed. We now have HTTP/2 and web sockets. Django could not support them well in the past. For the web socket part, I usually had to rely on Tornado or NodeJS (with the excellent Socket.IO library). They are good technologies but most of my web apps being in Django, I really wished there were something that could work with Django itself. And then we had Channels. The project is meant to allow Django to support HTTP/2, websockets or other protocols with ease.
Concepts
The underlying concept is really simple - there are channels and there are messages,
there are producers and there are consumers - the whole system is based on passing messages
on to channels and consuming/responding to those messages.
Let’s look at the core components of Django Channels first:
channel- A channel is a FIFO queue like data structure. We can have many channels depending on our need.message- A message contains meaningful data for the consumers. Messages are passed on to the channels.consumer- A consumer is usually a function that consumes a message and take actions.interface server- The interface server knows how to handle different protocols. It works as a translator or a bridge between Django and the outside world.
How does it work?
A http request first comes to the Interface Server which knows how to deal with a specific type of
request. For example, for websockets and http, Daphne is a popular interface server. When a
new http/websocket request comes to the interface server (daphne in our case), it accepts the request
and transforms it into a message. Then it passes the message to the appropriate channel. There are
predefined channels for specific types. For example, all http requests are passed to http.request channel.
For incoming websocket messages, there is websocket.receive. So these channels receive the messages when
the corresponding type of requests come in to the interface server.
Now that we have channels getting filled with messages, we need a way to process these messages and
take actions (if necessary), right? Yes! For that we write some consumer functions and register them to
the channels we want. When messages come to these channels, the consumers are called with the message.
They can read the message and act on them.
So far, we have seen how we can read an incoming request. But like all web applications, we should
write something back too, no? How do we do that? As it happens, the interface server is quite clever.
While transforming the incoming request into a message, it creates a reply channel for that particular
client request and registers itself to that channel. Then it passes the reply channel along with the message.
When our consumer function reads the incoming message, it can pass a response to the reply channel attached
with the message. Our interface server is listenning to that reply channel, remember? So when a response is sent
back to the reply channel, the interface server grabs the message, transforms it into a http response and sends
back to the client. Simple, no?
Writing a Websocket Echo Server
Enough with the theories, let’s get our hands dirty and build a simple echo server. The concept is simple. The server accepts websocket connections, the client writes something to us, we just echo it back. Plain and simple example.
Install Django & Channels
pip install channels
That should do the trick and install Django + Channels. Channels has Django as a depdency, so when you install channels, Django comes with it.
Create An App
Next we create a new django project and app -
django-admin.py startproject djchan
cd djchan
python manage.py startapp realtime
Configure INSTALLED_APPS
We have our Django app ready. We need to add channels and our django app (realtime) to the INSTALLED_APPS list under settings.py.
Let’s do that:
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
"channels",
"realtime"
]
Write our Consumer
After that, we need to start writing a consumer function that will process the incoming websocket messages and send back the response:
# consumers.py
def websocket_receive(message):
text = message.content.get('text')
if text:
message.reply_channel.send({"text": "You said: {}".format(text)})
The code is simple enough. We receieve a message, get it’s text content (we’re expecting that the websocket
connection will send only text data for this exmaple) and then push it back to the reply_channel - just like
we planned.
Channels Routing
We have our consume function ready, now we need to tell Django how to route messages to our consumer. Just like URL routing, we need to define our channel routings.
# routing.py
from channels.routing import route
from .consumers import websocket_receive
channel_routing = [
route("websocket.receive", websocket_receive, path=r"^/chat/"),
]
The code should be self explanatory. We have a list of route objects. Here we select the channel name
(websocket.receive => for receieving websocket messages), pass the consumer function and then configure
the optional path. The path is an interesting bit. If we didn’t pass a value for it, the consumer will
get all the messages in the websocket.receive channel on any URL. So if someone created a websocket connection
to / or /private or /user/1234 - regardless of the url path, we would get all incoming messages. But
that’s not our intention, right? So we restrict the path to /chat so only connections made to that url
are handled by the consumer. Please note the beginning /, unlike url routing, in channels, we have to use it.
Configuring The Channel Layers
We have defined a consumer and added it to a routing table. We’re more or less ready. There’s just a final bit of configuration we need to do. We need to tell channels two things - which backend we want to use and where it can find our channel routing.
Let’s briefly talk about the backend. The messages and the channels - Django needs some sort of data store or message queue to back this system. By default Django can use in memory backend which keeps these things in memory but if you consider a distributed app, for scaling large, you need something else. Redis is a popular and proven piece of technology for these kinds of scenarios. In our case we would use the Redis backend.
So let’s install that:
pip install asgi_redis
And now we put this in our settings.py:
CHANNEL_LAYERS = {
"default": {
"BACKEND": "asgi_redis.RedisChannelLayer",
"CONFIG": {
"hosts": [("localhost", 6379)],
},
"ROUTING": "realtime.routing.channel_routing",
},
}
Running The Servers
Make sure that Redis is running (usually redis-server should run it). Now run the django app:
python manage.py runserver
In local environment, when you do runserver - Django launches both the interface server and necessary
background workers (to run the consumer functions in the background). But in production,
we should run the workers seperately. We will get to that soon.
Trying it Out!
Once our dev server starts up, let’s open up the web app. If you haven’t added any django views, no worries, you should still see the “It Worked!” welcome page of Django and that should be fine for now. We need to test our websocket and we are smart enough to do that from the dev console. Open up your Chrome Devtools (or Firefox | Safari | any other browser’s dev tools) and navigate to the JS console. Paste the following JS code:
socket = new WebSocket("ws://" + window.location.host + "/chat/");
socket.onmessage = function(e) {
alert(e.data);
}
socket.onopen = function() {
socket.send("hello world");
}
If everything worked, you should get an alert with the message we sent. Since we defined a path, the websocket connection works only on /chat/. Try modifying the JS code and send a message to some other url to see how they don’t work. Also remove the path from our route and see how you can catch all websocket messages from all the websocket connections regardless of which url they were connected to. Cool, no?
Our Custom Channels
We have seen that certain protocols have predefined channels for various purposes. But we are not limited to those. We can create our own channels. We don’t need to do anything fancy to initialize a new channel. We just need to mention a name and send some messages to it. Django will create the channel for us.
Channel("thumbnailer").send({
"image_id": image.id
})
Of course we need corresponding workers to be listenning to those channels. Otherwise nothing will happen. Please note that besides working with new protocols, Channels also allow us to create some sort of message based task queues. We create channels for certain tasks and our workers listen to those channels. Then we pass the data to those channels and the workers process them. So for simpler tasks, this could be a nice solution.
Scaling Production Systems
Running Workers Seperately
On a production environment, we would want to run the workers seperately (since we would not run runserver on
production anyway). To run the background workers, we have to run this command:
python manage.py runworker
ASGI & Daphne
In our local environment, the runserver command took care of launching the Interface server and background
workers. But now we have to run the interface server ourselves. We mentioned Daphne already. It works
with the ASGI standard (which is commonly used for HTTP/2 and websockets). Just like wsgi.py, we now need to
create a asgi.py module and configure it.
import os
from channels.asgi import get_channel_layer
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "djchan.settings")
channel_layer = get_channel_layer()
Now we can run the server:
daphne djchan.asgi:channel_layer
If everything goes right, the interface server should start running!
ASGI or WSGI
ASGI is still new and WSGI is a battle tested http server. So you might still want to keep using wsgi for your http only parts and asgi for the parts where you need channels specific features.
The popular recommendation is that you should use nginx or any other reverse proxies in front and route the
urls to asgi or uwsgi depending on the url or Upgrade: WebSocket header.
Retries and Celery
The Channels system does not gurantee delivery. If there are tasks which needs the certainity, it is highly recommended to use a system like Celery for these parts. Or we can also roll our own checks and retry logic if we feel like that.