Kibana is an open source (Apache Licensed), browser based analytics and search dashboard for ElasticSearch. Kibana is a snap to setup and start using. Written entirely in HTML and Javascript it requires only a plain webserver, Kibana requires no fancy server side components. Kibana strives to be easy to get started with, while also being flexible and powerful, just like Elasticsearch.
Kibana is a great tool for real time data analytics and simply being on this page has put you on your way to making the most of it! If you have not downloaded Kibana yet, you can get it here: Download Kibana. We recommend you start this tutorial with a clean Elasticsearch instance.
By the end of this document you will have:
We will assume you have already:
We will be using the entire collected works of Shakespeare as our example data. In order to make the best use of Kibana you will likely want to apply a mapping to your new index. Let’s create the shakespeare index with the following mapping. Our data will have more fields than this, but these are the ones we want to explicitly map. Specifically we do not want to analyze speaker and play_name. You’ll see why later on.
Run in a terminal:
curl -XPUT http://localhost:9200/shakespeare -d '
{
"mappings" : {
"_default_" : {
"properties" : {
"speaker" : {"type": "string", "index" : "not_analyzed" },
"play_name" : {"type": "string", "index" : "not_analyzed" },
"line_id" : { "type" : "integer" },
"speech_number" : { "type" : "integer" }
}
}
}
}
';
Great, we’ve created the index. Now we want to import the data. You can download the entire works of shakespeare in elasticsearch bulk import format here: shakespeare.json
Import it into your local elasticsearch instance with the following command. This may take a few minutes. Shakespeare wrote a lot of stuff!
Now that you have data, let’s do something with it. Point your browser at your local webserver, the one with Kibana already installed.
If you have Kibana extracted to your document root you will be presented with this lovely welcome page. Click Sample Dashboard
And there you have your sample dashboard! Now if you
started with a clean elasticsearch instance you’ll see a very heavily
weighted pie chart. This represents the type of documents in your index.
As you can see, 99% them are lines for characters, with only a few
denoting acts and scenes.
Below that you will see a long list of JSON formatted lines of shakespeare.
Kibana allows you to search Elasticsearch data via the Lucene Query
String syntax. Queries can be run via the query input at the top of the
page
Type this in the query bar. Then check out the first few rows of the table.
For more information on queries see Queries and Filters
Right now Kibana is pointing at the special Elasticsearch index _all. Think of _all as a composite index that points at all of your indices. Right now you only have one, shakespeare, but you might have more someday and we don’t want Kibana searching all of that data if you’re only looking for your favorite line from Macbeth.
To configure the index click the configure icon in the top right:
From here, you can set your index to shakespeare and ensure that Kibana only searches the shakespeare index
Congratulations, you’ve installed and configured kibana and dipped your toes in the water. Next, check out some of our videos and other tutorials for more advanced usage. Now you can try adding your own panels to a blank dashboard. For more information on queries see Rows and Panels
Charts and tables and maps oh my! But how do we constrain the data displayed on them. This is the role of queries and filters. Because Kibana is powered by Elasticsearch it supports the powerful Lucene Query String syntax, as well as making use of some of Elasticsearch’s filter capabilities.
We will assume you have already:
Our dashboard looks like this, we will be searching through lines of shakespeare. If you would like to follow along with the screenshots in this guide you can download the exported dashboard schema
Type this very simple query into the search bar:
You will notice, in the table, your first hit is Hamlet as expected. However, look at the next line by Sir Andrew, it does not contain “to be”, nor does it contain “not to be”. The search we have actually performed is: to OR be OR or OR not OR to OR be.
We can also match the entire phrase:
Or in specifc fields:
We can express complex searches with AND/OR, note these words must be capitalized:
Or parantheses:
Numeric ranges can also be easily searched:
And of course to search everything:
In some cases, you might want to compare the results of two seperate queries. Kibana can handle multiple queries by joining them with a logical OR and using the knowledge that they should be treated as seperate to influence visualizations.
Click the + to the right of the query input to add another query.
You should now have this:
In the lefthand, green input, type "to be" and in the right, yellow input, type "not to be". This will search for every document that has "to be" OR "not to be" in it, and break that down on our hits pie chart. So we go from a big green circle to:
To remove a query, click the x icon that appears when you mouse over the query input field
Kibana will automatically determine a free color to use for your queries, but you can set your color explicitly if you’d like by clicking the colored dot associated with the query to open the query settings drop down. From here you can change the color of the query, or set a new value for the legend
Many Kibana charts are interactive and can be used to filter the view of your data. For example, click on the first bar of the chart. You will end up with something like this. As you can see, the chart has become one big green bar. This is because a filter was added to match that term in the play_name field.
“But where” you say?
The answer lies in that little white star that appeared on the filtering tab. Give the tab a click and we can see that a filter has added to the filtering panel. From the filtering panel filters can be added, edited, toggled and removed. Many panels have the ability to add filters, including the table, histogram, map and others.
Filters can also be added manually by clicking the +
Now that you have a solid handle on filters and queries you might be interested in how they’re described in the Kibana schema. If you’re interested in inserting queries or filters via URL parameters you might also take a look at Templated and Scripted Dashboards
Kibana’s dashboards are organized into a system of rows and panels. These can be added, removed and rearranged to suite your needs.
In this section we will cover:
We will assume you have already:
From the homescreen, select option #3, this will load a Blank Dashboard. By default, the blank dashboard is configured to look at Elasticsearch’s _all index, which points to all of your indices. If you would like to configure Kibana to point at a different index, see Using Kibana for the first time
Your new blank dashboard will appear with the query and filter sections expanded, a time filter input in the navigation bar at the top, and not a whole lot else. Click the Add a row button on the right to add your first row.
Give your row a title and click Create Row. You will see your new row appear in the list of rows to the left. Click Save
Now that you have a row, you will notice a few new elements on your dashboard. Chiefly the 3 brightly colored rectangles to the left. Move the mouse over them
Ah hah! these buttons allow you to accomplish 3 things
For now we’ll focus on the green button in the row controls. Give it a try. You can also click the grey button labeled Add panel to empty row, but it’s grey, and what fun is that?
Let’s add a terms panel. The terms panel allows us to make use of the Elasticsearch terms facet to find the most popular values in a field.
As you can see, the terms panel has a number of optional settings, however we’ll focus on the general settings in the first section for now:
Great! Now you have a panel! You may recognize this breakdown of data from the pie chart in Using Kibana for the first time. The shakespeare data set is comprised largely of lines of speech, with a few markers between acts and scenes thrown in there.
The blue button collapses your rows. Panels in collapsed rows do not refresh data and thus require no Elasticsearch resources to keep around. Collapsed rows are great for data that you don’t need to see often. Click the blue button again to expand the row.
The query and filter sections at the top can also be collapsed. Click the colored label to collapse and expand
Rows can be renamed, resized and edited via the row editor. Click the orange cog icon button to open the row editor.
The same dialog also allows you to change the order and size of panels, as well as remove them.
Panels can be dragged and dropped within their own row, or into another row, by dragging the crosshair shaped move icon in the top right of the panel.
Click the remove icon in the top right of a panel to remove it from the dashboard. Panels can also be removed from the row editor.
Rows can be re-ordered and removed via the dashboard configuration screen. Click the cog in the top left of the screen, and select the Rows tab to make changes to the row layout. You may remember this screen from when we added our first row.
The arrows to the left allow you to change the order of the rows on your dashboard. The X is used to remove rows.
Before you close your browser you might want to save this new dashboard. See Saving and Loading dashboards
You’ve built an amazing dashboard! Now you want to share it with your team, maybe turn on auto-refresh and put it up on a big screen? Kibana can persist dashboard designs to Elasticsearch and allow you to recall them via the load menu or links.
Saving your flashy new interface is as easy as opening the save drop down by clicking the Save icon in the upper righthand corner. Give your new dashboard a name and hit enter. Now your dashboard has been saved to a special Elasticsearch index called kibana-int
To search a list of your saved dashboards click the Load icon in the upper righthand corner. From here you can load, share and remove dashboards.
Saved dashboards can be shared via the URL in your brower’s navigation bar. Dashboards persisted to Elasticsearch will have a URL that points to them, such as:
In this example MYDASHBOARD is the title we have given the dashboard when saving.
You can also share links to adhoc dashboards by generating a temporary URL to them by clicking the Share icon in the upper righthand corner of Kibana.
By default these temporary URLs live for 30 days.
Dashboards can also be saved to disk on your server as .json files. Place them in the app/dashboards directory and access them via
Where MYDASHBOARD.json is the name of the file on disk. Note the path /#dashboard/file/ appears similar to the above path for accessing Elasticsearch stored dashboards, but in this case we are accessing a file instead of elasticsearch. For information on exporting the dashboard schema to a file see The Dashboard Schema Explained
Now that you know how to save dashboards, how to load them and how to access them on disk you might be interested in passing them parameters via the URL so you can programatically make links to data from other applications. See Templated and Scripted Dashboards
Kibana dashboards can be easily created right in the browser, and in most cases, the browser is all you will need to build a powerful, informative, interface. When you need that extra little bit of tweaking, Kibana allows for the editing of the dashboard schema.
NOTE: This section is for advanced users only. JSON syntax is very strict, a stray comma or missing brace will cause your dashboard not to load.
We will be using the above dashboard as an example. You can export the schema of any dashboard by clicking the Save icon in the top right, selecting advanced and choosing Export Schema. You can download this schema here: schema.json
While the dashboard is made up of one large JSON document, we will break it down into pieces for the purpose of instruction in this guide.
Like any JSON document, we start with an opening brace.
{
"services": {
Services are persisted objects that are used by multiple panels. There are currently 2 services that attach to the dashboard object and if omitted will be automatically populated with query and filter services.
"query": {
"list": {
"0": {
"query": "play_name:\"Romeo and Juliet\"",
"alias": "",
"color": "#7EB26D",
"id": 0,
"pin": false,
"type": "lucene",
"enable": true
}
},
"ids": [
0
]
},
The query service is controlled largely by the query dropdown at the top of the dashboard. It has 2 properties:
"filter": {
"list": {
"0": {
"type": "querystring",
"query": "speaker:ROMEO",
"mandate": "must",
"active": true,
"alias": "",
"id": 0
}
},
"ids": [
0
]
}
},
Filters behave much like queries, but are not selectable on a panel level, they filter the entire dashboard view. Much like the query service, there are are two properties: list and ids. Each of these behaves like their query counter part.
"pulldowns": [
Pulldowns are a special type of panel. Rather, they’re a special place you can put panels. The panels in the pulldown section work just like panels in a row except they don’t get a span, they’re all full width. Also panels in pulldowns are not movable or editable by the user. They’re great for panels that take input. the pulldown property is an array that contains panel objects. For documentation for specific panels see Kibana Panels
{
"type": "query",
"collapse": false,
"notice": false,
"enable": true,
"query": "*",
"pinned": true,
"history": [
"play_name:\"Romeo and Juliet\"",
"playname:\"Romeo and Juliet\"",
"romeo"
],
"remember": 10
},
{
"type": "filtering",
"collapse": false,
"notice": true,
"enable": true
}
],
Pulldown panels have 2 options that traditional row based panels do not have:
The loader property describes the behavior of the save and load buttons at the top of the dashboard.
"loader": {
"save_gist": false,
"save_elasticsearch": true,
"save_local": true,
"save_default": true,
"save_temp": true,
"save_temp_ttl_enable": true,
"save_temp_ttl": "30d",
"load_gist": false,
"load_elasticsearch": true,
"load_elasticsearch_size": 20,
"load_local": false,
"hide": false
},
rows are where panels are usually placed, and the only place they can be added via the browser UI.
"rows": [
{
"title": "Charts",
"height": "150px",
"editable": true,
"collapse": false,
"collapsable": true,
Row objects contain a panel array as well as several other properties describing the row:
The panels array property of a row contains a list of panel objects in the order in which they appear in the row. For a list of common panel properties and documentation for specific panels see Kibana Panels
"panels": [
{
"error": false,
"span": 8,
"editable": true,
"type": "terms",
"loadingEditor": false,
"field": "speech_number",
"exclude": [],
"missing": false,
"other": false,
"size": 10,
"order": "count",
"style": {
"font-size": "10pt"
},
"donut": false,
"tilt": false,
"labels": true,
"arrangement": "horizontal",
"chart": "bar",
"counter_pos": "above",
"spyable": true,
"queries": {
"mode": "all",
"ids": [
0
]
},
"tmode": "terms",
"tstat": "total",
"valuefield": "",
"title": "Longest Speeches"
},
{
"error": false,
"span": 4,
"editable": true,
"type": "goal",
"loadingEditor": false,
"donut": true,
"tilt": false,
"legend": "none",
"labels": true,
"spyable": true,
"query": {
"goal": 111397
},
"queries": {
"mode": "all",
"ids": [
0
]
},
"title": "Percentage of Total"
}
]
}
],
The index property contains information on the Elasticsearch index that Kibana communicates with.
"index": {
"interval": "none",
"default": "_all",
"pattern": "[logstash-]YYYY.MM.DD",
"warm_fields": false
},
The following are top level dashboard settings
"failover": false,
"editable": true,
"style": "dark",
"refresh": false
}
By default, importing of schemas is disabled. It can be re-enabled via the Controls tab of the dashboard settings screen by checking the Local file option. Schemas can be imported from the Load icon in the top left of the dashboard by selecting Advanced and choosing a file to import.
Kibana has support for creating dashboards dynamically via templates, and more advanced scripts. This allows you to create a based dashboard, then influence it with parameters, such as inserting a new query or filter via the URL.
Templates and scripts must be stored on disk, storing them to Elasticsearch is not supported. They must also be created by editing or creating a schema. Because of this we highly recommend reading The Kibana Schema Explained
Dashboards are stored in the app/dashboards directory within the Kibana installation directory. You will notice there are 2 types of files: .json files and .js files.
.json files are templated dashboards. An example of templating can be found on the query and filter objects of the logstash.json dashboard. Templates use handlebar syntax that allow you insert javascript clauses into your json. URL parameters are available on the ARGS object. For example, a snippet from logstash.json‘s (on github) query and filter services:
"0": {
"query": "{{ARGS.query || '*'}}",
"alias": "",
"color": "#7EB26D",
"id": 0,
"pin": false
}
[...]
"0": {
"type": "time",
"field": "@timestamp",
"from": "now-{{ARGS.from || '24h'}}",
"to": "now",
"mandate": "must",
"active": true,
"alias": "",
"id": 0
}
This allows us to set 2 parameters, query and from in the url. If they are not set, they will default to the value after the ||. For example, this would search for status:200 in the last 7 days:
Note: Take careful note of the file part of #/dashboard/file/logstash.json
http://yourserver/index.html#/dashboard/file/logstash.json?query=status:200&from=7d
Scripted dashboards are much more powerful than templated dashboards. Of course, with power comes responsibility, and scripted dashboards are more complex to build. The goal in a scripted dashboard is to build and return a javascript object that describes the dashboard schema. A well commented example scripted dashboard can be found in app/dashboards/logstash.js (on github). This file accomplishes the same goals as logstash.json, but with the added power of scripts we can do things like split queries on commas:
Note: Take careful note of the script part of #/dashboard/script/logstash.js. This instructs kibana to treat this file as javascript.
http://yourserver/index.html#/dashboard/script/logstash.js?query=status:403,status:404&from=7d
This will create 2 query objects, status:403 and status:404 and chart them seperately. In fact this dashboard takes another parameter that describes the string by which to split the query on: split
http://yourserver/index.html#/dashboard/script/logstash.js?query=status:403!status:404&from=7d&split=!
We can see this happening in logstash.js (on github) here:
// In this dashboard we let users pass queries as comma separated list to the query parameter.
// Or they can specify a split character using the split parameter
// If query is defined, split it into a list of query objects
// NOTE: ids must be integers, hence the parseInt()s
if(!_.isUndefined(ARGS.query)) {
queries = _.object(_.map(ARGS.query.split(ARGS.split||','), function(v,k) {
return [k,{
query: v,
id: parseInt(k,10),
alias: v
}];
}));
} else {
// No queries passed? Initialize a single query to match everything
queries = {
0: {
query: '*',
id: 0,
}
};
}
This dashboard takes more parameters than this, all of which are described in the comments at the top of logstash.js (on github)
config.js is where you will find the core Kibana configuration. This file contains parameter that must be set before kibana is run for the first time.
The URL to your elasticsearch server. You almost certainly don’t want http://localhost:9200 here. Even if Kibana and Elasticsearch are on the same host. By default this will attempt to reach ES at the same host you have kibana installed on. You probably want to set it to the FQDN of your elasticsearch host
Note: this can also be an object if you want to pass options to the http client. For example:
+elasticsearch: {server: "http://localhost:9200", withCredentials: true}+
This is the default landing page when you don’t specify a dashboard to load. You can specify files, scripts or saved dashboards here. For example, if you had saved a dashboard called ‘WebLogs’ to elasticsearch you might use:
default_route: /dashboard/elasticsearch/WebLogs,
The default ES index to use for storing Kibana specific object such as stored dashboards
An array of panel modules available. Panels will only be loaded when they are defined in the dashboard, but this list is used in the “add panel” interface.
Kibana dashboards are made up of blocks called panels. Panels are organized into rows and can serve many purposes, though most are designed to provide the results of a query or multiple queries as a visualization. Other panels may show collections of documents or allow you to insert instructions for your users.
Panels can be configured easily via the Kibana web interface. For more advanced usage, such as templated or scripted dashboards, documentation of panel properties is available in this section. You may find settings here which are not exposed via the web interface.
Each panel type has its own properties, however there are several that are shared.
Status: Experimental
Bettermap is called bettermap for lack of a better name. Bettermap uses geographic coordinates to create clusters of markers on map and shade them orange, yellow and green depending on the density of the cluster.
To drill down, click on a cluster. The map will be zoomed and the cluster broken into smaller cluster. When it no longer makes visual sense to cluster, individual markers will be displayed. Hover over a marker to see the tooltip value/
Important
bettermap requires an internet connection to download its map panels.
This object describes the queries to use on this panel.
Status: Stable
A pseudo panel that lets you add other panels to be arranged in a column with defined heights. While the column panel is stable, it does have many limitations, including the inability to drag and drop panels within its borders. It may be removed in a future release.
Status: Stable
The goal panel display progress towards a fixed goal on a pie chart
This object describes the queries to use on this panel.
Status: Stable
The histogram panel allow for the display of time charts. It includes several modes and tranformations to display event counts, mean, min, max and total of numeric fields, and derivatives of counter fields.
A query can be specified, the results of which will be displayed as markers on the chart. For example, for noting code deploys.
Min and max y-axis values
This object describes the queries to use on this panel.
Status: Stable
The hits panel displays the number of hits for each of the queries on the dashboard in a configurable format specified by the ‘chart’ property.
This object describes the queries to use on this panel.
Status: Stable
The map panel translates 2 letter country or state codes into shaded regions on a map. Currently available maps are world, usa and europe.
This object describes the queries to use on this panel.
Status: Experimental
The sparklines panel shows tiny time charts. The purpose of these is not to give an exact value, but rather to show the shape of the time series in a compact manner
This object describes the queries to use on this panel.
Status: Stable
The table panel contains a sortable, pagable view of documents that. It can be arranged into defined columns and offers several interactions, such as performing adhoc terms aggregations.
This object describes the queries to use on this panel.
Status: Stable
A table, bar chart or pie chart based on the results of an Elasticsearch terms facet.
This object describes the queries to use on this panel.
Status: Stable
The text panel is used for displaying static text formated as markdown, sanitized html or as plain text.
Status: Beta
A stock-ticker style representation of how queries are moving over time. For example, if the time is 1:10pm, your time picker was set to “Last 10m”, and the “Time Ago” parameter was set to “1h”, the panel would show how much the query results have changed since 12:00-12:10pm
This object describes the queries to use on this panel.