JDONREFv4 Plugin : Différence entre versions

De JDONREF Wiki
(→‎Includes)
(→‎Includes)
 
(48 révisions intermédiaires par le même utilisateur non affichées)
Ligne 1 : Ligne 1 :
−
L'intérêt du plugin JDONREFv3ES pour ElasticSearch est multiple. Il fournit :
  
  +
The JDONREF plugin provides a way to index adresses compatible with french format and search them in a single line format, with 100% accurate results and great order.
−
* les [[JDONREFv3ES_Mappings|mappings]] utilisés par JDONREF ;
  
  +
−
* un [[JDONREFv3ES_synonymes|fichier de synonymes]] adapté à la norme Française ;
  
  +
It works over the provided PayloadCheckerSpanQuery that target to combines whatever work rules you want (not only adresses) to search data in a single line format, with 100% accurate results and great order.
−
* un [[JDONREFv3ES_Type|type adresse]] associé à un [[JDONREFv3ES_AdressTokenFilter|TokenFilter adresse]] permettant d'indexer les adresses pour une recherche plus efficace (gestion des numéros, etc ...) ;
  
  +
Scorers are adaptable for human or computer uses.
−
* une [[JDONREFv3ES_Query|requête]] qui permet un calcul de note adapté à l'homme comme à la machine ;
  
  +
  +
Hercule had better watch out !
  +
  +
===== Plugin installation =====
  +
  +
Simply run at the root of your ''ElasticSearch v1.7'' installation :
  +
  +
bin/plugin --install elasticsearch-jdonrefv4-0.5 --url http://sourceforge.net/p/jdonref/code/HEAD/tree/Dev/JDONREFv4/dist/elasticsearch-jdonrefv4-0.5.zip?format=raw
  +
  +
This will download the last beta version plugin from sourceforge.
  +
  +
Don't forget restart elasticsearch.
  +
  +
  +
You can follow [[JDONREFv3ES_plugin_GettingStarted|the getting started guide]] for jdonref plugin.
  +
  +
There is an embedded site in order to test the query like this :
  +
  +
http://url_elasticsearch:port/_plugin/jdonrefv4-0.5/JDONREFv4Test.html
   
 
===== Releases =====
  
===== Releases =====
   
 
{| border="1"
  
{| border="1"
−
| '''JDONREFv3ES plugin'''
+
| '''JDONREF plugin'''
 
| '''elasticsearch'''
 
| '''elasticsearch'''
 
| '''release date'''
  
| '''release date'''
Ligne 20 : Ligne 38 :
 
| 0.2beta
  
| 0.2beta
 
| 1.1.X
  
| 1.1.X
−
| 2014-08-25 (a bit later than expected : notation problem)
  
  +
| 2014-08-25
 
| adress based score with integration tests
  
| adress based score with integration tests
 
|-
  
|-
 
| 0.2
  
| 0.2
 
| '''1.3.X'''
  
| '''1.3.X'''
−
| 2014-09-30
 +
| 2014-11-15
−
| production version : adress based score with production test.
 +
| production version : adress based score with production tests. human validation only. miss bulk scorer and explain.
 
|-
  
|-
 
| 0.3
  
| 0.3
 
| 1.3.X
  
| 1.3.X
−
| 2015-03-31
 +
| 2015-12-15
−
| score tuning
 +
| reduce high frequency term frequencies (improve performances)
  +
|-
  +
| 0.4beta
  +
| 1.3.X
  +
| 2015-05-15
  +
| minimum_should_match
  +
|-
  +
| 0.4
  +
| 1.3.X
  +
| 2015-12-31
  +
| bulk scorer & minimum_should_match. Performance Problems need to be solved again ! Waiting for ES support.
  +
|-
  +
| 0.5beta
  +
| 1.7.X
  +
| 2016-02-26
  +
| Performances. Need a big tuning by now !
  +
|-
  +
| 0.5
  +
| 1.7.X
  +
| 2016-06-31
  +
| final wildcard, fuzzyness, general performances
 
|}
  
|}
   
Ligne 41 : Ligne 59 :
 
| '''disponibilité'''
  
| '''disponibilité'''
 
|-
  
|-
−
| [[JDONREFv3ES_Query|jdonrefv3es]]
 +
| [[JDONREFv3ES_Query|jdonrefv4]]
 
| query/scorer
  
| query/scorer
−
| 0.1beta<br>0.2beta (weights and term can't be changed).
 +
| since 0.2beta (weights and term can't be changed for now).
 
|-
  
|-
−
| [[PayloadCheckerSpanQuery|PayloadCheckerSpanQuery]]
  
  +
| [[PayloadCheckerSpanFilter|PayloadCheckerSpanFilter]]
−
| query/scorer
  
  +
| filter
−
| 0.2beta (grammar filter based on payloads used by jdonrefv3es).
 +
| since 0.4beta (grammar filter based on payloads, not used by jdonrefv4 any more).
 
|-
  
|-
−
| [[JDONREFv3ES_Mappings#Le_mapping_poizon|POI et Zones]]
  
  +
| [[UnsplitFilter|UnsplitFilter]]
−
| mapping
  
  +
| token filter
−
| 0.2beta
  
  +
| since 0.3beta (reduce tokens frequencies, not used by jdonrefv4 any more).
 
|-
  
|-
−
| [[JDONREFv3ES_Mappings#Le_mapping_adresse|adresse]]
  
  +
| [[JDONREFv4_Mappings#Le_mapping_poizon|POI et Zones]], [[JDONREFv4_Mappings#Le_mapping_adresse|adresse]], [[JDONREFv4_Mappings#Le_mapping_troncon|troncon]], [[JDONREFv4_Mappings#Le_mapping_voie|voie]], [[JDONREFv4_Mappings#Le_mapping_commune|commune]], [[JDONREFv4_Mappings#Le_mapping_departement|departement]], [[JDONREFv4_Mappings#Le_mapping_pays|pays]]
 
| mapping
  
| mapping
−
| 0.1beta<br>0.2beta
 +
| since 0.2beta
 
|-
  
|-
−
| [[JDONREFv3ES_Mappings#Le_mapping_troncon|troncon]]
  
  +
| [[JDONREFv4_synonymes|jdonrefv3es_synonym.fr.txt]]
−
| mapping
  
−
| 0.1beta<br>0.2beta
  
−
|-
  
−
| [[JDONREFv3ES_Mappings#Le_mapping_voie|voie]]
  
−
| mapping
  
−
| 0.1beta<br>0.2beta
  
−
|-
  
−
| [[JDONREFv3ES_Mappings#Le_mapping_commune|commune]]
  
−
| mapping
  
−
| 0.1beta<br>0.2beta
  
−
|-
  
−
| [[JDONREFv3ES_Mappings#Le_mapping_departement|departement]]
  
−
| mapping
  
−
| 0.1beta<br>0.2beta
  
−
|-
  
−
| [[JDONREFv3ES_Mappings#Le_mapping_pays|pays]]
  
−
| mapping
  
−
| 0.1beta<br>0.2beta
  
−
|-
  
−
| [[JDONREFv3ES_synonymes|jdonrefv3es_synonym.fr.txt]]
  
 
| French Synonyms'file
  
| French Synonyms'file
−
| 0.1beta
 +
| since 0.1beta
 
|-
  
|-
−
| jdonrefv3es-settings.json
 +
| [[jdonrefv4-settings.json]]
 
| settings
  
| settings
−
| 0.1beta (jdonrefv3es_index, jdonrefv3es_search)<br>0.2beta (+jdonrefv3es_codes_index)
  
  +
| since 0.2beta jdonrefv4_index, jdonrefv4_search, jdonrefv4_codes_index
 
|-
  
|-
 
| JDONREFv4TermSimilarity
  
| JDONREFv4TermSimilarity
 
| similarity
  
| similarity
−
| 0.2beta
 +
| since 0.2beta
 
|-
  
|-
 
| [[EdgeNGramWithPayloadsFilter]]
  
| [[EdgeNGramWithPayloadsFilter]]
 
| token filter
  
| token filter
−
| 0.2beta (edgengram with payloads support)
 +
| since 0.2beta (edgengram with payloads support, not used by jdonrefv4 any more)
 
|-
  
|-
 
| [[TokenCountPayloadFilter]]
  
| [[TokenCountPayloadFilter]]
 
| token filter
  
| token filter
−
| 0.2beta (insert in payload the count of token with given payload - for use by PayloadVersusTypeSpanQuery).
 +
| since 0.2beta (insert the number of token with given payload in each such payload - used by PayloadCheckerSpanQuery#All).
 
|-
  
|-
−
| [[GroupedPayloadSpanQuery]]
  
  +
| [[SynonymWithPayloadsFilter]]
−
| query
  
  +
| token filter
−
| 0.2beta (remove results where payloads are not grouped together).
 +
| since 0.2beta (synonym with payloads support, not used by jdonrefv4 any more).
 
|-
  
|-
−
| [[PayloadVersusTypeSpanQuery]]
  
  +
| JDONREFv3Test.html
−
| query
  
−
| 0.2beta (remove results where all payloads term matches for given types).
  
−
|-
  
−
| [[JDONREFv3ES_Type|jdonrefv3es_adresse]]
  
−
| type
  
−
|
 
−
|-
  
−
| [[JDONREFv3ES_AdressTokenFilter|jdonrefv3es_adresse]]
  
−
| tokenFilter
  
−
|
 
−
|-
  
−
| [[JDONREFv3ES_filter|combosynonym]]
  
−
| TokenFilter
  
−
|
 
−
|-
  
−
| test_0.html test_1.html
  
 
| site
  
| site
−
| 0.1beta (simple test)
  
  +
| since 0.2beta (site for jdonrefv3es and index jdonref) Set 10000 for maxSizePerType with dedicated index of 5 shards for each type !
 
|-
  
|-
−
| JDONREFv3Test.html
 +
| JDONREFv4Test.html
 
| site
  
| site
−
| 0.2beta (site for jdonrefv3es and index jdonref)
  
  +
| since 0.4. Set 10000 for maxSizePerType with dedicated index of 5 shards for each type !
 
|}
  
|}
   
 
See also [[JDONREFv3ES_TODO|todo list]].
  
See also [[JDONREFv3ES_TODO|todo list]].
−  
−
===== Installation du plugin =====
  
−  
−
Pour installer le plugin, vous pouvez simplement exécuter la commande suivante :
  
−  
−
bin/plugin --install elasticsearch-jdonrefv3es-0.2beta --url http://sourceforge.net/p/jdonref/code/HEAD/tree/Dev/JDONREF_ES_PLUGIN_POC/dist/elasticsearch-jdonrefv3es-0.2beta.zip?format=raw
  
−  
−
N'oubliez pas de redémarrer elasticsearch.
  
−  
−
Le plugin n'offre pour le moment qu'une requête. Le site de test fourni est pour le moment sans rapport avec cette requête.
  
−
La requête est utilisable tel que décrit dans [[jdonrefv3es_plugin_configuration|ici]].
  
−  
−
Le site est accessible avec une url comme http://url_elasticsearch:port/_plugin/jdonrefv3es-0.2beta/JDONREFv3Test.html.html.
  
−  
−
'''Attention, la version 1.3 de elasticsearch est nécessaire à partir de la version 0.2beta du plugin'''
  
−  
−
===== Prérequis avant d'utiliser le plugin =====
  
−
Le site fonctionne uniquement avec un index de nom "jdonref" qui respecte le mapping proposé par le plugin.
  
−
Par contre, la requête peut être utilisée sur n'importe quel index qui respecte le mapping.
  
−  
−
Avant de créer cet index, le fichier jdonrefv3es_synonym.fr.txt doit être déployé dans /etc/elasticsearch ou dans la chaîne du path.
  
−  
−
===== Utilisation du plugin =====
  
−  
−
Voir notamment le [[JDONREFv3ES_plugin_GettingStarted|guide de première utilisation]].
  
−  
−
===== Mise à jour du plugin =====
  
−  
−
Il s'agit de le désinstaller, puis de le réinstaller.
  

Version actuelle en date du 26 février 2016 à 22:05

The JDONREF plugin provides a way to index adresses compatible with french format and search them in a single line format, with 100% accurate results and great order.

It works over the provided PayloadCheckerSpanQuery that target to combines whatever work rules you want (not only adresses) to search data in a single line format, with 100% accurate results and great order. Scorers are adaptable for human or computer uses.

Hercule had better watch out !

Plugin installation

Simply run at the root of your ElasticSearch v1.7 installation :

bin/plugin --install elasticsearch-jdonrefv4-0.5 --url http://sourceforge.net/p/jdonref/code/HEAD/tree/Dev/JDONREFv4/dist/elasticsearch-jdonrefv4-0.5.zip?format=raw

This will download the last beta version plugin from sourceforge.

Don't forget restart elasticsearch.


You can follow the getting started guide for jdonref plugin.

There is an embedded site in order to test the query like this :

http://url_elasticsearch:port/_plugin/jdonrefv4-0.5/JDONREFv4Test.html

Releases
JDONREF plugin elasticsearch release date detail
0.1beta 1.0.0.RC1 2014-04-30 POC
0.2beta 1.1.X 2014-08-25 adress based score with integration tests
0.2 1.3.X 2014-11-15 production version : adress based score with production tests. human validation only. miss bulk scorer and explain.
0.3 1.3.X 2015-12-15 reduce high frequency term frequencies (improve performances)
0.4beta 1.3.X 2015-05-15 minimum_should_match
0.4 1.3.X 2015-12-31 bulk scorer & minimum_should_match. Performance Problems need to be solved again ! Waiting for ES support.
0.5beta 1.7.X 2016-02-26 Performances. Need a big tuning by now !
0.5 1.7.X 2016-06-31 final wildcard, fuzzyness, general performances
Includes
name type disponibilité
jdonrefv4 query/scorer since 0.2beta (weights and term can't be changed for now).
PayloadCheckerSpanFilter filter since 0.4beta (grammar filter based on payloads, not used by jdonrefv4 any more).
UnsplitFilter token filter since 0.3beta (reduce tokens frequencies, not used by jdonrefv4 any more).
POI et Zones, adresse, troncon, voie, commune, departement, pays mapping since 0.2beta
jdonrefv3es_synonym.fr.txt French Synonyms'file since 0.1beta
jdonrefv4-settings.json settings since 0.2beta jdonrefv4_index, jdonrefv4_search, jdonrefv4_codes_index
JDONREFv4TermSimilarity similarity since 0.2beta
EdgeNGramWithPayloadsFilter token filter since 0.2beta (edgengram with payloads support, not used by jdonrefv4 any more)
TokenCountPayloadFilter token filter since 0.2beta (insert the number of token with given payload in each such payload - used by PayloadCheckerSpanQuery#All).
SynonymWithPayloadsFilter token filter since 0.2beta (synonym with payloads support, not used by jdonrefv4 any more).
JDONREFv3Test.html site since 0.2beta (site for jdonrefv3es and index jdonref) Set 10000 for maxSizePerType with dedicated index of 5 shards for each type !
JDONREFv4Test.html site since 0.4. Set 10000 for maxSizePerType with dedicated index of 5 shards for each type !

See also todo list.

Récupérée de « https://adullact.net/plugins/mediawiki/wiki/jdonref-v2/index.php?title=JDONREFv4_Plugin&oldid=484 »