Elastic Search with Django Rest Framework

Objective: Adding a full-text search in Django Rest Framework using elastic search.

What is Elastic Search?

Elasticsearch is a distributed, open-source search and analytics engine designed for handling large volumes of data. It is built on top of the Apache Lucene library and allows you to perform full-text searches, faceted searches, and geospatial searches, among other features.

One of the main advantages of Elasticsearch is its scalability. It can be easily scaled horizontally by adding more nodes to a cluster, which allows it to handle increasing amounts of data and search queries.

---

Project Setup

First, we will need a running elastic search database. For this, we will use the docker.

# dockerfile
FROM elasticsearch:8.5.3

ENV discovery.type single-node
ENV xpack.security.enabled=false

EXPOSE 9200 9300

CMD ["/usr/share/elasticsearch/bin/elasticsearch"]

Explanation

discovery.type single-node This configuration is useful in development or testing environments, where you may only need a single node for testing or small-scale projects.

xpack.security.enabled=false this disables the usage of X-Pack security features. X-Pack is a set of additional security features provided by Elasticsearch, such as authentication, authorization, and encryption.

Second, we will need a Django project configured with DRF and a simple model with some data in it. the packages we are going to use currently work with Python v3.6 to v3.9, so please make sure you are using the correct version.

mkdir es_with_drf && cd es_with_drf
conda create -n elastic-with-drf
conda activate elastic-with-drf
conda install python=3.9

pip install django djangorestframework
django-admin startproject config .
python manage.py startapp app1

# config/settings.py
INSTALLED_APPS = [
    ...
    "app1",
    "rest_framework",
]

REST_FRAMEWORK = {
    'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.PageNumberPagination',
    'PAGE_SIZE': 50,
    'DEFAULT_RENDERER_CLASSES': ['rest_framework.renderers.JSONRenderer']
}

# app1/models.py
from django.db import models

class City(models.Model):
    name = models.CharField(max_length=100)
    state_code = models.CharField(max_length=100)
    state_name = models.CharField(max_length=100)
    country_code = models.CharField(max_length=100)
    country_name = models.CharField(max_length=100)

    def __str__(self):
        return "%s: %s: %s" % (self.name, self.state_code, self.country_code)

For the data, I will be using https://github.com/dr5hn/countries-states-cities-database. After importing the data. We will work on adding the elastic search to this project.

pip install django-elasticsearch-dsl-drf

then connect our project to the elastic search container that we started in docker.

# config/settings.py

INSTALLED_APPS = [
    ...
    "django_elasticsearch_dsl",
    "django_elasticsearch_dsl_drf",
]

ELASTICSEARCH_DSL = {
    'default': {
        'hosts': 'localhost:9200'
    },
}

create a document file in the app folder to specify the model which we want to index in our elastic search

# app1/documents.py

from django_elasticsearch_dsl import Document
from django_elasticsearch_dsl.registries import registry
from app1.models import City


@registry.register_document
class CityDocument(Document):
    class Index:
        name = 'city'

    settings = {
        'number_of_shards': 1,
        'number_of_replicas': 0
    }

    class Django:
        model = City
        fields = ["name", "state_code", "state_name", "country_code", "country_name"]

then we will need a serializer for the City document we just created.

# app1/serializers.py

from django_elasticsearch_dsl_drf.serializers import DocumentSerializer
from app1.documents import CityDocument


class CityDocumentSerializer(DocumentSerializer):
    class Meta:
        document = CityDocument
        fields = "__all__"

finally we can create a view and add it to the URLs file

# app1/views.py

from django_elasticsearch_dsl_drf.viewsets import BaseDocumentViewSet
from django_elasticsearch_dsl_drf.filter_backends import CompoundSearchFilterBackend
from elasticsearch.exceptions import NotFoundError

from app1.documents import CityDocument
from app1.serializers import CityDocumentSerializer
from rest_framework.exceptions import NotFound


class CityDocumentView(BaseDocumentViewSet):
    document = CityDocument
    serializer_class = CityDocumentSerializer

    filter_backends = [CompoundSearchFilterBackend]

    # Define search fields
    search_fields = {
        'name': {'fuzziness': 'AUTO'},
    }

    def list(self, request, *args, **kwargs):
        try:
            return super().list(request, *args, **kwargs)
        except NotFoundError:
            raise NotFound(f"{self.index} index not found")

You can look into this for more customizations https://django-elasticsearch-dsl-drf.readthedocs.io/

# config/urls.py

from django.contrib import admin
from django.urls import path, include

from app1.views import CityDocumentView

from rest_framework.routers import DefaultRouter

router = DefaultRouter()
books = router.register("city", CityDocumentView, basename="citydocument")

urlpatterns = [
    path("", include(router.urls)),
    path("admin/", admin.site.urls),
]

now if we hit the API, it will generate an exception index_not_found_exception because we haven't indexed the City data in elastic search. To resolve this, we will index the data by entering this command.

python manage.py search_index --populate

Now when we hit http://127.0.0.1:8000/city/ we will get this response (based on the stored data)

{
    "count": 10000,
    "next": "http://localhost:8000/city/?page=2",
    "previous": null,
    "facets": {},
    "results": [
        {
            "name": "Departamento de Sarmiento",
            "country_code": "AR",
            "state_code": "G",
            "state_name": "Santiago del Estero",
            "country_name": "Argentina",
            "id": 1501
        },
        {
            "name": "Farah",
            "country_code": "AF",
            "state_code": "FRA",
            "state_name": "Farah",
            "country_name": "Afghanistan",
            "id": 22
        },
        ....
    ]
}

and to perform a search http://localhost:8000/city/?search=delih, it will return the correct data.

{
    "count": 19,
    "next": null,
    "previous": null,
    "facets": {},
    "results": [
        {
            "name": "Delhi",
            "country_code": "IN",
            "state_code": "DL",
            "state_name": "Delhi",
            "country_name": "India",
            "id": 45445
        },
        {
            "name": "Delia",
            "country_code": "IT",
            "state_code": "82",
            "state_name": "Sicily",
            "country_name": "Italy",
            "id": 60369
        },
        ....
    ]
}

Source Code