Version du 2 décembre 2014 à 23:55

Detokenize tokens in order to reduce term query frequencies for performance consideration.

Sample

For example, the document :

 { "fullName": "BOULEVARD PARIS 07 HOPITAL" }

with the following mapping :

 "fullName" : {"type": "string", "index_analyzer":"myAnalyzer", "search_analyzer":"myAnalyzer"}

and settings like :

 {
   "index" : {
      "analysis" : {
          "analyzer": {
              "myAnalyzer" : {
                  "type" : "custom",
                  "tokenizer" : "whitespace",
                  "filter" : ["lowercase", "unsplit_filter"]
              }
          },
          "filter" : {
              "unsplit_filter" : {
                 "type": "unsplit",
                 "min_words_unsplitted" : 3,
                 "keep_originals" : false
              }
          }
     }
   }
 } 

will index these terms :

• 07BOULEVARDHOPITALPARIS
• 07BOULEVARDHOPITAL
• 07BOULEVARDPARIS
• 07HOPITALPARIS
• BOULEVARDHOPITALPARIS

so that the request :

 {
   "query_string" :
   {
       "default_field":"fullName",
       "query":"BOULEVARD PARIS 07",
       "analyzer" : "myAnalyzer"
   }
 }

will match the document.

Note that 07BOULEVARDPARIS got a very lower frequency than 07, BOULEVARD, and PARIS.

How it works

For each token given to the unsplit filter, all ordered combinations are considered. That mean unsplit filter can be used with synonyms and ngrams, and nearly all other token filters (tell me which one does not works !).

The combination tokens got an 'UNSPLITED' token type.

There is a few parameters in order to control theses combinations.

Features