Version du 25 octobre 2014 à 11:39

Check results for grammar rules according to payloads values.

Sample

For example, the document :

 { "fullName": "BOULEVARD|1 DE|1 PARIS|1 07|2 L|3 HOPITAL|3" }

with the following mapping :

 "fullName" : {"type": "string", "term_vector" : "with_positions_offsets_payloads", "index_analyzer":"myAnalyzer"}

and settings like :

 {
   "index" : {
      "analysis" : {
          "analyzer": {
              "myAnalyzer" : {
                  "type" : "custom",
                  "tokenizer" : "whitespace",
                  "filter" : ["delimited_payload_filter", "lowercase", "tokencount_payload_filter"]
              }
          },
          "filter" : {
              "delimited_payload_filter" : {
                "type": "delimited_payload_filter",
                "delimiter" : "|",
                "encoding" : "int"
              },
              "tokencount_payload_filter" : {
                "type": "tokencountpayloads",
                "factor": 1000
              }
      }
 } 

will match the query :

 {
   "span_payloadchecker" : {
       "clauses": [
         { "span_multipayloadterm" : { "fullName": "BOULEVARD" } },
         { "span_multipayloadterm" : { "fullName": "PARIS" } },
         { "span_multipayloadterm" : { "fullName": "HOPITAL" } }
       ],
       "checker": { "type":"Grouped" }
   }
 }

but not match the following :

 {
   "span_payloadchecker" : {
       "clauses": [
         { "span_multipayloadterm" : { "fullName": "BOULEVARD" } },
         { "span_multipayloadterm" : { "fullName": "HOPITAL" } }
         { "span_multipayloadterm" : { "fullName": "PARIS" } },
       ],
       "checker": { "type":"Grouped" }
   }
 }

because eventhough they got the same payload BOULEVARD and PARIS are not grouped together in the second request.

Checkers

Notice the following checkers :

How it works

Elasticsearch is a flat inverted index document, it does not natively make a difference between one term or another. As an example, it uses frequencies to seek for terms in a corpus.

However, in a simple sentence, like an adress, terms might be very differents. Frequency might not be usable in such cases.

For example for the request :

 57 BD DE L HOPITAL 75 PARIS

We are not intend to get such results :

 75 BD DE L HOPITAL 75013 PARIS (75 as the number of the adress)
 75 rue de paris 57 L HOPITAL (the town L HOPITAL in Moselle, it do not exists in real life, it's an example)

The exactitude of the number of the adresse is here more important than its frequency in the corpus.

To get the most pertinent results, ElasticSearch uses lucene engine to get a score for each results. PayloadCheckerSpanQuery overload that scorer with a new score for each results. Actually, it descores bad results.

Scoring

Scoring System of PayloadCheckerSpanQuery got two objectives :

1. give the most representative results
2. get an absolute notation , than mean an AI would pick the good choice between the results (see bulk mode)

The scoring concept follows the SpanScorer and the BooleanScorer, and is inspired by scoring in JDONREFv2 and JDONREFv3 (adapted to inversed index). It was necessary to change the DefaultSimilarity class.

The bulk mode is a mode where score get a ceiling. The ceiling for a result is the maximum score it is possible to get with it. So the bulk mode is simply the score divided by the maximum score. Then it's applying a rule of three to the ceiling for commodity (something like 100% or 20/20).

The bulk mode is less performant than traditionnal way, so it is not recommended for autocomplete, and it is made like an option mode. It is not necessary when a human being can choose between results. But an IA can choose automatically a result when the score is for example over 19/20.

Customise Scoring

(soon)

See also

EdgeNGramWithPayloadsFilter

TokenCountPayloadFilter