Using the elasticsearch Profile API

Some time ago I found an issue with Elasticsearch explaining a Profile API that they were working on. The recently released 2.2 version of Elasticsearch contains the first experimental version of the API. The API is still experimental, meaning that it could change or be removed completely in the future.

Having given you the warning, let us have a look at what we can do with the profile API. In this blog post, I give you an overview of the capabilities with a very basic example. After this blog you should have a good idea what you can do with the profile API.

The sample documents

Before we can start, we need some sample documents. Below you can find the documents that I used together with the mapping. To make the output easier, I configured just one shard with no replicas and a document with the standard analyzed string field, together with two alternatives: keyword analyzer with lowercase filter and raw.

PUT /companies
{
  "settings": {
    "number_of_replicas": 0,
    "number_of_shards": 1,
    "analysis": {
      "analyzer": {
        "askeyword": {
          "type": "custom",
          "tokenizer": "keyword",
          "filter": ["lowercase"]
        }
      }
    }
  },
  "mappings": {
    "company": {
      "properties": {
        "name": {
          "type": "string", 
          "index": "analyzed",
          "fields": {
            "asone": {
              "type": "string", 
              "analyzer": "askeyword"
            },
            "raw": {
              "type": "string",
              "index": "not_analyzed"
            }
          }
        }
      }
    }
  }
}

Below the documents that we insert.

PUT /companies/company/1
{
  "name": "Yuma Zaventem"
}
PUT /companies/company/2
{
  "name": "Elastic"
}
PUT /companies/company/3
{
  "name": "ServiceHouse"
}
PUT /companies/company/4
{
  "name": "Bol.com"
}
PUT /companies/company/5
{
  "name": "ANWB"
}

Introduction

You can enable the Profile API results using the top level property profile, set it to true to enable obtaining profile results. The profile results are calculated for each shard. So, if you are searching over multiple indices, you are also searching over multiple shards. In the results a shard is identified by the unique identifier of the node, the name of the index and the number of the shard. For an example, check the id field in the next code block which shows part of the response. But first the request with a very basic query.

GET /companies/_search
{
  "profile": true,
  "query": {
    "match": {
      "name": "Yuma"
    }
  }
}

"profile": {
  "shards": [
    {
      "id": "[0]",
      "searches": [
        {
          "query": [],
          "rewrite_time": 2914,
          "collector": []
        }
      ]
    }
  ]
}

At the moment the result consists of a shards array with each shard consisting of an id according to the mentioned format and an array of searches. This search array usually consists of one item. The item contains 3 elements: The query results, the rewrite time and the collector giving more information about how Lucene handled the query including the time it took for lucene. In the next sections we take a deeper dive into the different elements of the response

Query Section

Queries can have a pretty complicated structure. Just like with the explain API if you break it down, it is a repetition of the same elements. The basic building blocks are:

• query_type: for instance a TermQuery
• lucene: The query part in Lucene style, for example message:lucene
• time: number of ms it took Lucene to execute the query
• breakdown: Has more detailed information about the query
• children: This is the repetition, more of the same kind of blocks

The query type is the query type as created based on the original query, but often not the same as the original query. For example, a match query with multiple terms will become a bool query consisting of a number of Term queries. Check the following example:

query
  query_type: BooleanQuery
  lucene: name:Yuma name:zaventem
  time: 0.3146980000ms
  breakdown: 
  children:
    query_type: TermQuery
    lucene: name:Yuma
    time: 0.1017570000ms
    breakdown:
 
    query_type: TermQuery
    lucene: name:zaventem
    time: 0.02729200000ms
    breakdown:

The example shows the query_type, being BooleanQuery and TermQuery, it also shows the Lucene query that is executed and the time it took in milliseconds. The final piece is the breakdown part. This piece contains more detailed information about what took so long. Check the docs in the reference section to learn more about this very advanced statistic.

Rewrite section

As described in the query section, the query is rewritten. In the example a match query is rewritten as a bool query containing two term queries. This section just returns the amount of milliseconds it took to rewrite the query.

Collectors section

Collectors are Lucene's mechanism to keep track of all other activities. Each collector has a name and a keyword description of why the specific collector was used. Finally the collector shows the time the collection took. A few example collectors are.

• SimpleTopScoreDocCollector: Used when asking for the documents with the highest score used as sorting
• SimpleFieldCollector: Used when returning documents sorted by their name
• TotalHitCountCollector: Used when nu documents are requested, but only the count (when size:0 is used)
• MultiCollector: Used when a query is used in combination with a aggregation for instance.
• GlobalOrdinalsStringTermsAggregator: Used when a Terms aggregation is requested.

There are more types than mentioned here, they are in the documentation. The next code block shows a sample output extracted from the Profile API result.

"collector": [
  {
    "name": "SimpleTopScoreDocCollector",
    "reason": "search_top_hits",
    "time": "0.02907200000ms"
  }
]

Concluding

In my opinion, the Profile API is a very nice addition to the available API's. It enables you to learn more about what is actually happening when executing searches in Elasticsearch. It is easy to use and not to hard to understand. As a warning though, using the profile API is not cheap and should not be used by default in production. Also, not everything is supported yet by the Profile API; suggestions, for instance, are not supported. At the moment, there is no GUI for the Profile API, but Elastic is working on a Kibana plugin to help you visualize the Profile results. Maybe I'll introduce it in my own GUI project as well; I need to think about that.

Reference

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-profile.html