Openduty is an open source incident escalation tool, which is something like Pagerduty but free and much simpler. It was born during a hackathon at Ustream back in 2014. The project received a lot of attention in the devops community, and was also featured in Devops weekly andPycoders weekly.It is listed at Full Stack Python as an example Django project. This article is going to include some design decisions we made during the hackathon, and detail some of the main components of the Opendutysystem.
Design
When we started the project, we already knew what we wanted to end up with:
- We had to work quickly—it was a hackathon after all
- An API similar to Pagerduty
- Ability to send notifications asynchronously
- A nice calendar to organize on—call schedules can’t hurt anyone, right?
- Tokens for authorizing notifiers
So we chose the corresponding components to reach our goal.
Get the job done quickly
If you have to develop apps rapidly in Python, Django is the framework you choose. It’s a bit heavyweight, but hey, it gives you everything you need and sometimes even more. Don’t get me wrong; I’m a big fan of Flask also, but it can be a bit fiddly to assemble everything by hand at the start. Flask may pay off later, and you may win on a lower amount of dependencies, but we only had 24 hours, so we went with Django.
An API
When it comes to Django and REST APIs, one of the GOTO soluitions is The Django REST Framework. It has all the nuts and bolts you’ll need when you’re assembling an API, like serializers, authentication, and permissions. It can even give you the possibility to make all your API calls self-describing. Let me show you howserializers work in the Rest Framework.
class OnCallSerializer(serializers.Serializer):
person = serializers.CharField()
email = serializers.EmailField()
start = serializers.DateTimeField()
end = serializers.DateTimeField()The code above represents a person who is on-call on the API. As you can see, it is pretty simple; you just have to define the fields. It even does the validation for you, since you have to give a type to every field. But believe me, it’s capable of more good things like generating a serializer from your Django model:
class SchedulePolicySerializer(serializers.HyperlinkedModelSerializer):
rules = serializers.RelatedField(many=True, read_only=True)
class Meta:
model = SchedulePolicy
fields = ('name', 'repeat_times', 'rules')This example shows how you can customize a ModelSerializer, make fields read-only, and only accept given fields from an API call.
Async Task Execution
When you have tasks that are long-running, such as generating huge reports, resizing images, or even transcoding some media, it is a common practice thatyou must move the actual execution of those out of your webapp into a separate layer. This decreases the load on the webservers, helps in avoiding long or even timing out requests, and just makes your app more resilient and scalable. In the Python world, the go-to solution for asynchronous task execution is called Celery. In Openduty, we use Celery heavily to send notifications asynchronously and also to delay the execution of any given notification task by the delay defined in the service settings.
Defining a task is this simple:
@app.task(ignore_result=True)
def send_notifications(notification_id):
try:
notification = ScheduledNotification.objects.get(id = notification_id)
if notification.notifier == UserNotificationMethod.METHOD_XMPP:
notifier = XmppNotifier(settings.XMPP_SETTINGS)
#choosing notifier removed from example code snippet
notifier.notify(notification)
#logging task result removed from example snippet
raise
And calling an already defined task is also almost as simple as calling any regular function:
send_notifications.apply_async((notification.id,) ,eta=notification.send_at)This means exactly what you think: Send the notification with the id: notification.id at notification.send_at.
But how do these things get executed?
Under the hood, Celery wraps your decorated functions so that when you call them, they get enqueued instead of being executed directly.
When the celery worker detects that there is a task to be executed, it simply takes it from the queue and executes it asynchronously.
Calendar
We use django-scheduler for the awesome-looking calendar in Openduty. It is a pretty good project generally, supports recurring events, and provides you with a UI for your calendar, so you won’t even have to fiddle with that.
Tokens and Auth
Service token implementation is a simple thing. You want them to be unique, and what else would you choose if not aUUID? There is a nice plugin for Django models used to handle UUID fields, called django-uuidfield. It just does what it says—addingUUIDField support to your models. User authentication is a bit more interesting, so we currently support plain Django Users, and you can use LDAP as your user provider.
Summary
This was just a short summary about the design decisions made when we coded Openduty. I also demonstrated the power of the components through some snippets that are relevant. If you are on a short deadline, consider using Django and its extensions. There is a good chance that somebody has already done what you need to do, or something similar, which can always be adapted to your needs thanks to the awesome power of the open source community.
About the author
BálintCsergő is a software engineer from Budapest, currently working as an infrastructure engineer at Hortonworks. He lovesUnix systems, PHP, Python, Ruby, the Oracle database, Arduino, Java, C#, music, and beer.
