Menu
Looks like a simple usecase but for some reason I just can't figure out how to do this, or google a clear example.
Lets say I have a message stored in logstash message:
"info: 2015-11-28 22:02:19,232:common:INFO:ENV: Production User:None:Username:None:LOG: publishing to bus "
And I want to search in kibana (version 4) for the phrase:"publishing to bus" I'll get a set of results But if I'll search for: "None:LOG: publishing to bus" Then I get "No results found".
While Obviously this phrase does exists and is returned by the previous search.
So my question is basically - What is going on? What is the correct way to search for a possible long phrase and why does the second example fail.
EDIT: The stored JSON.
{
"_index": "logz-ngdxrkmolklnvngumaitximbohqwbocg-151206_v1",
"_type": "django_logger",
"_id": "AVF2DPxZZst_8_8_m-se",
"_score": null,
"_source": {
"log": " publishing to bus {'user_id': 8866, 'event_id': 'aibRBPcLxcAzsEVRtFZVU5', 'timestamp': 1449384441, 'quotes': {}, 'rates': {u'EURUSD': Decimal('1.061025'), u'GBPUSD': Decimal('1.494125'), u'EURGBP': Decimal('0.710150')}, 'event': 'AccountInstrumentsUpdated', 'minute': 1449384420}",
"logger": "common",
"log_level": "INFO",
"message": "2015-12-06 06:47:21,298:common:INFO:ENV: Production User:None:Username:None:LOG: publishing to bus {'user_id': 8866, 'event_id': 'aibRBPcLxcAzsEVRtFZVU5', 'timestamp': 1449384441, 'quotes': {}, 'rates': {u'EURUSD': Decimal('1.061025'), u'GBPUSD': Decimal('1.494125'), u'EURGBP': Decimal('0.710150')}, 'event': 'AccountInstrumentsUpdated', 'minute': 1449384420}",
"type": "django_logger",
"tags": [
"celery"
],
"path": "//path/to/logs/out.log",
"environment": "Staging",
"@timestamp": "2015-12-06T06:47:21.298+00:00",
"user_id": "None",
"host": "path.to.host",
"timestamp": "2015-12-06 06:47:21,298",
"username": "None"
},
"fields": {
"@timestamp": [
1449384441298
]
},
"highlight": {
"message": [
"2015-12-06 06:47:21,298:common:INFO:ENV: Staging User:None:Username:None:LOG: @kibana-highlighted-field@publishing@/kibana-highlighted-field@ @kibana-highlighted-field@to@/kibana-highlighted-field@ @kibana-highlighted-field@bus@/kibana-highlighted-field@ {'user_id': **, 'event_id': 'aibRBPcLxcAzsEVRtFZVU5', 'timestamp': 1449384441, 'quotes': {}, 'rates': {u'EURUSD': Decimal('1.061025'), u'GBPUSD': Decimal('1.494125'), u'EURGBP': Decimal('0.710150')}, 'event': 'AccountInstrumentsUpdated', 'minute': 1449384420}"
]
},
"sort": [
1449384441298
]
}
290
See the current release documentation . The Kibana Query Language (KQL) is a simple syntax for filtering Elasticsearch data using free text search or field-based search. KQL is only used for filtering data, and has no role in sorting or aggregating the data.
Use the search box without any fields or local statements to perform a free text search in all the available data fields. If no data shows up, try expanding the time field next to the search box to capture a broader range. Searching for the word elasticsearch finds all instances in the data in all fields. The query in Kibana is not case-sensitive.
When querying Elasticsearch in Kibana you can either use the traditional Lucene query syntax or the newer Kibana Query Language (KQL). If you are using Kibana 7.0 or later, Kibana Query Language is included as a default. In this article we provide the basics for both approaches and provide example searches.
To match an exact string, use quotation marks. For example, "get elasticsearch" queries the whole string. Kibana allows searching individual fields. Check all available fields on the bottom left menu pane under Available fields: To perform a search in a specific field, use the following syntax: The query syntax depends on the field type.
Accodrding to Elasticsearch, it uses standard analyzer as default. The standard analyzer tokenizes the message field as follows:
"2015-12-06 06:47:21,298:common:INFO:ENV: Production User:None:Username:None:LOG: publishing to bus {'user_id': 8866, 'event_id': 'aibRBPcLxcAzsEVRtFZVU5', 'timestamp': 1449384441, 'quotes': {}, 'rates': {u'EURUSD': Decimal('1.061025'), u'GBPUSD': Decimal('1.494125'), u'EURGBP': Decimal('0.710150')}, 'event': 'AccountInstrumentsUpdated', 'minute': 1449384420}"
{
"tokens": [
{
"token": "2015",
"start_offset": 0,
"end_offset": 4,
"type": "<NUM>",
"position": 0
},
{
"token": "12",
"start_offset": 5,
"end_offset": 7,
"type": "<NUM>",
"position": 1
},
{
"token": "06",
"start_offset": 8,
"end_offset": 10,
"type": "<NUM>",
"position": 2
},
{
"token": "06",
"start_offset": 11,
"end_offset": 13,
"type": "<NUM>",
"position": 3
},
{
"token": "47",
"start_offset": 14,
"end_offset": 16,
"type": "<NUM>",
"position": 4
},
{
"token": "21,298",
"start_offset": 17,
"end_offset": 23,
"type": "<NUM>",
"position": 5
},
{
"token": "common:info:env",
"start_offset": 24,
"end_offset": 39,
"type": "<ALPHANUM>",
"position": 6
},
{
"token": "production",
"start_offset": 41,
"end_offset": 51,
"type": "<ALPHANUM>",
"position": 7
},
{
"token": "user:none:username:none:log",
"start_offset": 52,
"end_offset": 79,
"type": "<ALPHANUM>",
"position": 8
},
{
"token": "publishing",
"start_offset": 81,
"end_offset": 91,
"type": "<ALPHANUM>",
"position": 9
},
{
"token": "to",
"start_offset": 92,
"end_offset": 94,
"type": "<ALPHANUM>",
"position": 10
},
{
"token": "bus",
"start_offset": 95,
"end_offset": 98,
"type": "<ALPHANUM>",
"position": 11
},
{
"token": "user_id",
"start_offset": 100,
"end_offset": 107,
"type": "<ALPHANUM>",
"position": 12
},
{
"token": "8866",
"start_offset": 109,
"end_offset": 113,
"type": "<NUM>",
"position": 13
},
{
"token": "event_id",
"start_offset": 115,
"end_offset": 123,
"type": "<ALPHANUM>",
"position": 14
},
{
"token": "aibrbpclxcazsevrtfzvu5",
"start_offset": 125,
"end_offset": 147,
"type": "<ALPHANUM>",
"position": 15
},
{
"token": "timestamp",
"start_offset": 149,
"end_offset": 158,
"type": "<ALPHANUM>",
"position": 16
},
{
"token": "1449384441",
"start_offset": 160,
"end_offset": 170,
"type": "<NUM>",
"position": 17
},
{
"token": "quotes",
"start_offset": 172,
"end_offset": 178,
"type": "<ALPHANUM>",
"position": 18
},
{
"token": "rates",
"start_offset": 184,
"end_offset": 189,
"type": "<ALPHANUM>",
"position": 19
},
{
"token": "ueurusd",
"start_offset": 192,
"end_offset": 199,
"type": "<ALPHANUM>",
"position": 20
},
{
"token": "decimal",
"start_offset": 201,
"end_offset": 208,
"type": "<ALPHANUM>",
"position": 21
},
{
"token": "1.061025",
"start_offset": 209,
"end_offset": 217,
"type": "<NUM>",
"position": 22
},
{
"token": "ugbpusd",
"start_offset": 220,
"end_offset": 227,
"type": "<ALPHANUM>",
"position": 23
},
{
"token": "decimal",
"start_offset": 229,
"end_offset": 236,
"type": "<ALPHANUM>",
"position": 24
},
{
"token": "1.494125",
"start_offset": 237,
"end_offset": 245,
"type": "<NUM>",
"position": 25
},
{
"token": "ueurgbp",
"start_offset": 248,
"end_offset": 255,
"type": "<ALPHANUM>",
"position": 26
},
{
"token": "decimal",
"start_offset": 257,
"end_offset": 264,
"type": "<ALPHANUM>",
"position": 27
},
{
"token": "0.710150",
"start_offset": 265,
"end_offset": 273,
"type": "<NUM>",
"position": 28
},
{
"token": "event",
"start_offset": 277,
"end_offset": 282,
"type": "<ALPHANUM>",
"position": 29
},
{
"token": "accountinstrumentsupdated",
"start_offset": 284,
"end_offset": 309,
"type": "<ALPHANUM>",
"position": 30
},
{
"token": "minute",
"start_offset": 311,
"end_offset": 317,
"type": "<ALPHANUM>",
"position": 31
},
{
"token": "1449384420",
"start_offset": 319,
"end_offset": 329,
"type": "<NUM>",
"position": 32
}
]
}
The phrase "Production User:None:Username:None:LOG: publishing to bus "
{
"token": "production",
"start_offset": 41,
"end_offset": 51,
"type": "<ALPHANUM>",
"position": 7
},
{
"token": "user:none:username:none:log",
"start_offset": 52,
"end_offset": 79,
"type": "<ALPHANUM>",
"position": 8
},
{
"token": "publishing",
"start_offset": 81,
"end_offset": 91,
"type": "<ALPHANUM>",
"position": 9
},
{
"token": "to",
"start_offset": 92,
"end_offset": 94,
"type": "<ALPHANUM>",
"position": 10
},
{
"token": "bus",
"start_offset": 95,
"end_offset": 98,
"type": "<ALPHANUM>",
"position": 11
}
So if you search "publishing to bus" the elasticsearch matches the above three token and return the document.
if you search "None:LOG: publishing to bus" "None:LOG:" doesn't match fully so it doesn't return the document.
you can try "User:None:Username:None:LOG: publishing to bus" to get the result.
150
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
