Knowpapa.com - a developer's blog

Django Haystack + Elasticsearch + Autocomplete + Faceting Tutorial

In this tutorial, we will implement a ‘Products’ search similar to what you would find on any e-commerce store.

The search results can be refined using multiple facets as shown here. We will also implement search auto complete which will provide real time suggestions as the user types in his search query.

Here’s the end result:

Haystack Search Demo

You can download the source code for the complete working project from github.


Please note that Elasticsearch version 5 and even version 2 are not compatible with Django Haystack as on date of writing (11.1.2017).

The trick to getting Django Haystack to work right out of the box is to choose the correct versions.

Here’s the combination that worked for me and I suggest you stick to it.

  • elasticsearch-1.7.6
  • django-haystack==2.5.1
  • elasticsearch==5.0.1

lets get started.

Installing and running Elasticsearch server

I am assuming that you have a new Django project up and running.

Next, download Elasticsearch version 1.7.6 on your PC (download from here). Once downloaded, extract the zip file into any convenient location.

Navigate to the directory where you have extracted Elasticsearch and open its bin folder in a bash terminal.

cd path-to-the-folder-of-elasticsearch-1.7.6/bin
./elasticsearch

Running the above bash script should start the Elasticsearch server. Visit http://127.0.0.1:9200/. If the server is up and running, it should show a message similar to this:

{
  "status" : 200,
  "name" : "Zero-G",
  "cluster_name" : "elasticsearch",
  "version" : {
  "number" : "1.7.6",
  "build_hash" : "c730b59357f8ebc555286794dcd90b3411f517c9",
  "build_timestamp" : "2016-11-18T15:21:16Z",
  "build_snapshot" : false,
  "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

Installation on Django

Next install django haystack and elasticsearch package for Django.

pip install django-haystack==2.5.1
pip install elasticsearch==5.0.1

Next add haystack to the list of installed apps in your settings.py

INSTALLED_APPS = [
  ...
  'haystack',
]

Also add the following to your settings.py file:

HAYSTACK_CONNECTIONS = {
  'default': {
  'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
  'URL': 'http://127.0.0.1:9200/',
  'INDEX_NAME': 'products_tutorial',
  },
}

Since we are implementing a Product search I have named its index ‘products_tutorial’. You can name it whatever you want to call it.

Defining Models

Lets define the models in `models.py`. Here’s the simplified product model:

class Product(models.Model):
  title = models.CharField(unique=True, max_length=255, db_index=True, default='')
  slug = models.SlugField(null=True, blank=True, unique=True, max_length=255)
  description = models.TextField(db_index=True)
  brand = models.CharField(db_index=True, max_length=255)
  category = models.ForeignKey(Category, related_name='category')
  image = models.ImageField(upload_to = 'product_images/', default = 'product_images/no-img.jpg')
  timestamp = models.DateTimeField(auto_now=True)

  def get_absolute_url(self):
  return reverse('product',
  kwargs={'slug': self.slug})


  def __str__(self):
  return self.title

Let’s also define the Category Model which has been added as a foreign key to the above products model. We will use the category as one of our facets for filtering the results.

class Category(models.Model):
  name = models.CharField(max_length=255, db_index=True, unique=True)

  def __str__(self):
  return self.name

Defining the Search Index

Next let’s define the search index in a separate file named `search_indexes.py`. It is important that this file be named search_indexes.py as this is what is used by Django haystack for indexing the search items.

class ProductIndex(indexes.SearchIndex, indexes.Indexable):
  text = indexes.EdgeNgramField(
  document=True, use_template=True,
  template_name='search/indexes/catalogue/product_text.txt')
  title = indexes.EdgeNgramField(model_attr='title')
  description = indexes.EdgeNgramField(model_attr="description", null=True)
  date_updated = indexes.DateTimeField(model_attr='date_updated')

  subcategory = indexes.CharField(
  model_attr='subcategory',
  faceted=True)

  def get_model(self):
  return Product

  def index_queryset(self, using=None):
  return self.get_model().objects.filter(
  date_updated__lte=datetime.datetime.now())

Note that the above code references a file named ‘product_text.txt’ which must be placed in the templates directory in the location specified therein.

Add the following content to the file product_text.txt.

{{object.title}}
{{ object.description|default:"" }}

Adding Views and Forms

We will be defining 4 views in our file.

1) HomeView – This shows the home page. Not necessary for search but it demonstrates how to integrate search form in a different page.
2) FacetedSearchView – This shows the faceted search result.
3) ProductView – This is not related to search but this is where the user lands when he clicks on any individual search item.
4) autocomplete – This returns result for ajax requests that will be send for suggesting search terms.

Here’s the views.py file:

from django.views.generic import TemplateView
from django.views.generic.detail import DetailView
from django.http import JsonResponse
from haystack.generic_views import FacetedSearchView as BaseFacetedSearchView
from haystack.query import SearchQuerySet

from .models import Product
from .forms import FacetedProductSearchForm

class HomeView(TemplateView):
  template_name = "home.html"

class ProductView(DetailView):
  template_name = "product.html"
  model = Product

def autocomplete(request):
  sqs = SearchQuerySet().autocomplete(
  content_auto=request.GET.get(
  'query',
  ''))[
  :5]
  s = []
  for result in sqs:
  d = {"value": result.title, "data": result.object.slug}
  s.append(d)
  output = {'suggestions': s}
  return JsonResponse(output)


class FacetedSearchView(BaseFacetedSearchView):

  form_class = FacetedProductSearchForm
  facet_fields = ['category', 'brand']
  template_name = 'search_result.html'
  paginate_by = 3
  context_object_name = 'object_list'

Note that FacetedSearchView is a child class of BaseFacetedSearchView defined in django haystack.
Since we want to filter results based on multiple facets, namely the brand and category fields, we add them to the list titled ‘facet_fields’.

We could have done away without defining a form_class but since we ant to show checkboxes next to each facet item, we define a custom form in forms.py.

from haystack.forms import FacetedSearchForm

class FacetedProductSearchForm(FacetedSearchForm):

  def __init__(self, *args, **kwargs):
  data = dict(kwargs.get("data", []))
  self.categories = data.get('category', [])
  self.brands = data.get('brand', [])
  super(FacetedProductSearchForm, self).__init__(*args, **kwargs)

  def search(self):
  sqs = super(FacetedProductSearchForm, self).search()
  if self.categories:
  query = None
  for category in self.categories:
  if query:
  query += u' OR '
  else:
  query = u''
  query += u'"%s"' % sqs.query.clean(category)
  sqs = sqs.narrow(u'category_exact:%s' % query)
  if self.brands:
  query = None
  for brand in self.brands:
  if query:
  query += u' OR '
  else:
  query = u''
  query += u'"%s"' % sqs.query.clean(brand)
  sqs = sqs.narrow(u'brand_exact:%s' % query)
  return sqs

Defining Templates

I will not show the template code for home.html and product.html as they are not directly related to search. However there are two templates that are important for search.

1) The Search Form
2) Search Result Page

I have implemented search form in a separate file named ‘search_form.html’. This way I can include it the header of all my pages.

Here’s the template for the search form.

lt&;div class="col-md-6">
  lt&;form class="search-form-container" action="/find/" method="get" >
  lt&;div class="form-group">
  lt&;div class="icon-addon addon-lg">
  lt&;input type="text" placeholder="what are you looking for ?" class="form-control" name="q" id="q" autocomplete="off">
  lt&;div id="selction-ajax">lt&;/div>
  lt&;/div>
  lt&;/div>
  lt&;/form>
lt&;/div>

Note that the form action attribute refers to a url named ‘/find/’. So we will have to define the url accordingly.

Also note the empty div section with the id “selction-ajax”. This div will be used to fill in the autocomplete suggestions in the real time.

Next, let’s define the search result page, defined in the file search_result.html

{% extends "layout.html" %}
{% block title %}
    "{{ query }}" | {{ block.super }}
{% endblock %}
{% block content %}
lt&;div class="container">
lt&;div class="row">
{% if  page_obj.object_list %}
lt&;div class="col-md-3">
	lt&;h3>Filterslt&;/h3>
        lt&;dl>
        {% if facets.fields.category %}
        lt&;dt>Filter by Categorylt&;/dt>
        {% for category in facets.fields.category %}
        {% if category.1 != 0 %}
        lt&;dd>
        lt&;input class="facet" id="{{category.0|cut:" "}}" type="checkbox" name="category" value="{{ category.0 }}" 
        data-toggle="toggle" /> {{ category.0 }} ({{ category.1 }})
        lt&;/dd>
        {% endif %}
        {% endfor %}
        {% endif %}
        lt&;/dl>
        lt&;div>
        lt&;input class="btn btn-info btn-sm pull-right" type="submit" value="apply filter" onclick="return onFacetChangeApplied();" />
        lt&;/div>
        lt&;dl>

        {% if facets.fields.brand %}
        lt&;dt>Filter by Brandlt&;/dt>
        {% for brand in facets.fields.brand %}
        {% if brand.1 != 0 %}
        lt&;dd>
        lt&;input class="facet" id="{{brand.0|cut:" "}}" type="checkbox" name="brand" value="{{ brand.0 }}" /> {{ brand.0 }} ({{ brand.1 }})
        lt&;/dd>
        {% endif %}
        {% endfor %}
        {% endif %}
        lt&;/dl>
        lt&;div>
        lt&;input class="btn btn-info btn-sm pull-right" type="submit" value="apply filter" onclick="return onFacetChangeApplied();" />
	
        lt&;/div>
    lt&;/div>
{% endif %}
	lt&;div class="col-md-9">
            lt&;div class="row">
                lt&;div class="col-md-6 col-xs-6">
                      Search result for: lt&;label> {{query}} lt&;/label>
               lt&;/div>
               lt&;div class="col-md-6 col-xs-6 align-right">
               
                   Showing {{ page_obj.start_index }} - {{ page_obj.end_index }} of total 
                    {{ page_obj.paginator.count }}
                    results on page {{ page_obj.number }} of {{ page_obj.paginator.num_pages }}
               lt&;/div>
            lt&;/div>    
            lt&;div>
	    {% if  page_obj.object_list %}
                lt&;ol class="row top20">
                    {% for result in page_obj.object_list %}
                        lt&;div class="showcase col-sm-6 col-md-4">
		   lt&;a href="{{ result.object.get_absolute_url }}">
		      lt&;h3>{{result.object.title}}lt&;/h3>
			            lt&;img src="{{ result.object.image.url }}" class="img-responsive">
		   lt&;/a>
		    lt&;h4 class="text-center">lt&;span class="label label-info">{{result.object.brand}}lt&;/span>lt&;/h4>
		    lt&;/div>
                    {% endfor %}
                lt&;/ol>
            lt&;/div>
		    {% if is_paginated %}
		      lt&;ul class="pagination pull-right">
		        {% if page_obj.has_previous %}
		          lt&;li>lt&;a href="?q={{ query }}&page={{ page_obj.previous_page_number }}">«lt&;/a>lt&;/li>
		        {% else %}
		          lt&;li class="disabled">lt&;span>«lt&;/span>lt&;/li>
		        {% endif %}
		        {% for i in paginator.page_range %}
		          {% if page_obj.number == i %}
		            lt&;li class="active">lt&;span>{{ i }} lt&;span class="sr-only">(current)lt&;/span>lt&;/span>lt&;/li>
		          {% else %}
		            lt&;li>lt&;a href="?q={{ query }}&page={{ i }}">{{ i }}lt&;/a>lt&;/li>
		          {% endif %}
		        {% endfor %}
		        {% if page_obj.has_next %}
		          lt&;li>lt&;a href="?q={{ query }}&page={{ page_obj.next_page_number }}">»lt&;/a>lt&;/li>
		        {% else %}
		          lt&;li class="disabled">lt&;span>»lt&;/span>lt&;/li>
		        {% endif %}
		      lt&;/ul>
		    {% endif %}
        {% else %}
	    lt&;p> Sorry, no result found for the search term  lt&;strong>{{query}} lt&;/strong>lt&;/p>
	{% endif %}
	lt&;/div>
lt&;/div>
lt&;/div>
{% endblock %}

Note that the above template takes care of faceting based on multiple fields . In our case it loops through the category and the brand facet.

Search result pagination is taken care by default pagination code in Django. The complete search result in to be found in page_obj.object_list which is to be looped through to get the individial search results.

Further the returned object is wrapped in SearchResult object. So to access the individual objects you have to call result.object.itsfield.

Defining URLs

Having defined our views, forms, and templates, let’s finally connect our views to the URLs in the file urls.py.

from django.conf.urls import url
from django.contrib import admin
from .views import HomeView, ProductView, FacetedSearchView, autocomplete
from .settings import MEDIA_ROOT, MEDIA_URL
from django.conf.urls.static import static

urlpatterns = [
  url(r'^$', HomeView.as_view()),
  url(r'^admin/', admin.site.urls),
  url(r'^product/(?P[\w-]+)/$', ProductView.as_view(), name='product'),
  url(r'^search/autocomplete/$', autocomplete),
  url(r'^find/', FacetedSearchView.as_view(), name='haystack_search'),
  
] + static(MEDIA_URL, document_root=MEDIA_ROOT)

Adding Products to be searched

Having added the models, views, templates and urls, let’s next run the migrations and then start the Django server.

python manage.py makemigrations
python manage.py migrate
python manage.py runserver

If you are following along with the actual code for this tutorial from the github page, you need not add products as the sqlite database and the product images are included in the repository.

Or you can add the models to Django admin interface:

from django.contrib import admin
from .models import Product, Category

admin.site.register(Category)

class ProductAdmin(admin.ModelAdmin):
  prepopulated_fields = {"slug": ("title",)}

admin.site.register(Product, ProductAdmin)

Then visit Django admin (http://127.0.0.1:8000/admin/) and add a few products from the admin interface.

Javascript for Autocomplete and Facet Based Filtering

In order to implement the autocomplete feature, we will use a javascript library called
jQuery-Autocomplete.

Download the jquery.autocomplete.js file and add it to your static/js folder.

You will also need jquery. I have used jquery version 2.1.1 but it should probably work with other versions.

In addition create a file named ‘our_search_code.js’ where we will write some custom jquery.

Include these three files in all your pages where you need the search form to appear.

lt&;script src="{% static 'js/jquery.min.js' %}">lt&;/script>
lt&;script src="{% static 'js/jquery.autocomplete.js' %}">lt&;/script>
lt&;script src="{% static 'js/our_search_code.js' %}">lt&;/script>

To enable auto complete add the following code to our_search_code.js file.

$(function () {
  'use strict';

  $('#q').autocomplete({
  serviceUrl: "http://127.0.0.1:8000/search/autocomplete/",
  minChars: 2,
  dataType: 'json',
  type: 'GET',
  onSelect: function (suggestion) {
  console.log( suggestion.value + ', data :' + suggestion.data);
  }
});

});

The above lines of code will make autocomplete functional but since we have not yet indexed the products. They will not show up now.

We will do that in the next step.

Meanwhile lets add the following lines of javascript code to take care of faceting. The following code simply adds event listenders to click of apply button which when clicked sends a GET request with appropriate parameters for faceting.

function getParameterByName(name, url) {
  if (!url) {
  url = window.location.href;
  }
  name = name.replace(/[\[\]]/g, "\\$&");
  var regex = new RegExp("[?&]" + name + "(=([^&#]*)|&|#|$)"),
  results = regex.exec(url);
  if (!results) return null;
  if (!results[2]) return '';
  return decodeURIComponent(results[2].replace(/\+/g, " "));
}




function onFacetChangeApplied(){
	var url = window.location.href.split("?")[0];
	var search_query = getParameterByName('q');
	var url_with_search_query = url + '?q=' + search_query 
	$('input:checkbox.facet').each(function () {
  	var sThisVal = (this.checked ? $(this).val() : null);
  var sThisName = (this.checked ? $(this).attr('name') : null);
  if(sThisVal !== null){
  	url_with_search_query += '&'+encodeURIComponent(sThisName)+'='+encodeURIComponent(sThisVal);
  }
  });
	location.href = url_with_search_query;
	return true;
} 


function getQueryParams(){
  var vars = {}, hash;
  var hashes = window.location.href.slice(window.location.href.indexOf('?') + 1).split('&');
  for(var i = 0; i < hashes.length; i++)
  {
  hash = hashes[i].split('=');
  vars[hash[1]] = hash[0] ;
  }
  return vars;
}


$( document ).ready(function() {
	var all_params = getQueryParams();
	console.log();
	$.each( all_params, function( key, value ) {
		id = decodeURIComponent(key).replace(/\s/g,'');
		$('#'+id).attr('checked', 'checked');
		});
	
});


Building Index

Run the following command to build the actual index.

  python manage.py rebuild_index

Now you should have a fully functional search with autocomplete and faceting !