검색/ElasticSearch

elasticsearch-plugin 생성하기

듐듐다다 2021. 1. 13. 16:14
반응형

목적

  • 자동완성 기능을 위해 사용할 자동완성 플러그인 이 정상적으로 install되지 않는 원인 분석을 위함

  • 플러그인 생성 방법 및 테스트 내용을 기록하기 위함

테스트

  1. 기능 없는 플러그인으로 정상 설치 여부를 확인한다.

  2. javacafe-analyzer 소스를 이용하여 플러그인을 만들고 정상 설치 여부를 확인한다.

1. 기능 없는 플러그인 테스트

1-1. 프로젝트 생성

plugin 생성 참고url : http://www.technocratsid.com/how-to-create-an-elasticsearch-6-4-1-plugin/

 

How to create an Elasticsearch 6.4.1 Plugin

A plugin provides a way to extend or enhance the basic functionality of Elasticsearch without having to fork it from GitHub. Elasticsearch supports a plugin framework which provides many custom plu…

www.technocratsid.com

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <name>autocomplete-plugin-test</name>
    <groupId>org.elasticsearch.plugin.autocomplete</groupId>
    <artifactId>autocomplete-test-plugin</artifactId>
    <version>1.0</version>
    <packaging>jar</packaging>
    <description>autocomplete plugin test</description>

    <properties>
        <elasticsearch.version>6.5.4</elasticsearch.version>
        <autocomplete.plugin.classname>org.autocomplete.test.plugin.ExamplePlugin</autocomplete.plugin.classname>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <autocomplete.assembly.descriptor>${basedir}/src/main/assemblies/plugin.xml</autocomplete.assembly.descriptor>
    </properties>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.2</version>
                <configuration>
                    <includes>
                        <include>**/*Tests.java</include>
                    </includes>
                    <skipTests>true</skipTests>
                </configuration>
            </plugin>
            <plugin>
                <artifactId>maven-source-plugin</artifactId>
                <version>3.2.0</version>
                <executions>
                    <execution>
                        <id>attach-sources</id>
                        <goals>
                            <goal>jar</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <appendAssemblyId>false</appendAssemblyId>
                    <outputDirectory>${project.build.directory}/releases/</outputDirectory>
                    <descriptors>
                        <descriptor>${autocomplete.assembly.descriptor}</descriptor>
                    </descriptors>
                </configuration>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>attached</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>${elasticsearch.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.13.1</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

</project>

assemblies

  • plugin을 zip형태로 묶기 위한 설정

  • 위치 :

  • 내용 :

    • plugin생성에 필요한 plugin-descriptor.properties 파일 생성 위치를 명시한다.

    • 생성된 plugin.jar와 plugin정보를 담고있는 plugin-descriptor.properties의 위치는 zip의 root 위치에 있어야 하므로 프로젝트 경로를 생성하지 않도록 includeBaseDirectory = false로 해준다.

<?xml version="1.0"?>
<assembly>
    <id>plugin</id>
    <formats>
        <format>zip</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <files>
        <file>
            <source>${basedir}/src/main/resources/plugin-descriptor.properties</source>
            <outputDirectory>/</outputDirectory>
            <filtered>true</filtered>
        </file>
    </files>
    <dependencySets>
        <dependencySet>
            <outputDirectory>/</outputDirectory>
            <unpack>false</unpack>
        </dependencySet>
    </dependencySets>
</assembly>

plugin-descriptor.properties

  • plugin 기본정보를 담은 파일

  • 위치 :    

 

 

 

 

 

 

 

description=${project.description}
version=${project.version}
name=${project.artifactId}
classname=${autocomplete.plugin.classname}
elasticsearch.version=${elasticsearch.version}
java.version=${maven.compiler.target}

  • elastic

docker run -p 9200:9200 -p 9300:9300 --name elastic -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:6.5.4

  • kibana (추후 index 생성 및 호출 용도)

docker run -d --link elastic:elastic-url -e "ELASTICSEARCH_URL=http://elastic-url:9200" -p 5601:5601 --name kibana docker.elastic.co/kibana/kibana:6.5.4

1-3. 프로젝트 빌드 및 plugin install

프로젝트 빌드 ~ elasticsearch-plugin install

 

elasticsearch가 설치된 서버에 들어가서 다음을 진행

sh-4.2# pwd
/usr/share/elasticsearch
wget https://raw.githubusercontent.com/mondayus/analyzer/main/autocomplete-plugin-2.0.zip
sh-4.2# bin/elasticsearch-plugin install file:./autocomplete-plugin-2.0.zip

  • 다음의 경우에 해당 에러가 발생함

    • zip 파일 내에 plugin-descriptor.properties이 없는 경우

    • plugin 버전과 설치한 elasticsearch 버전이 상이한 경우

    • plugin-descriptor.properties이 zip의 root 경로에 없을 경우

    • zip 파일을 열지 못할 경우

  • zip파일 확인

    • wget을 이용해 zip 파일을 다운로드 후 unzip -t [파일명]을 이용하여 zip의 유효성을 체크한다.

    • 다음과 같은 결과가 보이면 zip 파일을 열지 못할 경우 에 해당한다.

sh-4.2# wget https://github.com/mondayus/analyzer/blob/main/autocomplete-test-plugin-1.0.zip
--2020-12-14 07:11:09--  https://github.com/mondayus/analyzer/blob/main/autocomplete-test-plugin-1.0.zip
Resolving github.com (github.com)... 15.164.81.167
Connecting to github.com (github.com)|15.164.81.167|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'autocomplete-test-plugin-1.0.zip'

    [ <=>                                                      ] 90,970      --.-K/s   in 0.09s   

2020-12-14 07:11:10 (946 KB/s) - 'autocomplete-test-plugin-1.0.zip' saved [90970]

sh-4.2# unzip -t autocomplete-test-plugin-1.0.zip                                               
Archive:  autocomplete-test-plugin-1.0.zip
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of autocomplete-test-plugin-1.0.zip or
        autocomplete-test-plugin-1.0.zip.zip, and cannot find autocomplete-test-plugin-1.0.zip.ZIP, period.

 

  • 해결방법

    • zip을 해제한 채로 git에 업로드 하고 서버내에서 zip으로 묶는다.

sh-4.2# pwd
/usr/share/elasticsearch
sh-4.2# wget https://github.com/mondayus/analyzer/blob/main/autocomplete-test-plugin-1.0.jar
--2020-12-14 07:19:22--  https://github.com/mondayus/analyzer/blob/main/autocomplete-test-plugin-1.0.jar
Resolving github.com (github.com)... 15.164.81.167
Connecting to github.com (github.com)|15.164.81.167|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'autocomplete-test-plugin-1.0.jar'

    [ <=>                                                      ] 95,473      --.-K/s   in 0.09s   

2020-12-14 07:19:22 (994 KB/s) - 'autocomplete-test-plugin-1.0.jar' saved [95473]

sh-4.2# wget https://github.com/mondayus/analyzer/blob/main/plugin-descriptor.properties
--2020-12-14 07:19:31--  https://github.com/mondayus/analyzer/blob/main/plugin-descriptor.properties
Resolving github.com (github.com)... 15.164.81.167
Connecting to github.com (github.com)|15.164.81.167|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'plugin-descriptor.properties'

    [ <=>                                                      ] 94,436      --.-K/s   in 0.09s   

2020-12-14 07:19:32 (1020 KB/s) - 'plugin-descriptor.properties' saved [94436]

sh-4.2# zip -r els-plugin.zip ./autocomplete-test-plugin-1.0.jar ./plugin-descriptor.properties 
  adding: autocomplete-test-plugin-1.0.jar (deflated 78%)
  adding: plugin-descriptor.properties (deflated 78%)
sh-4.2# unzip -t els-plugin.zip
Archive:  els-plugin.zip
    testing: autocomplete-test-plugin-1.0.jar   OK
    testing: plugin-descriptor.properties   OK
No errors detected in compressed data of els-plugin.zip.
sh-4.2#

 

  • 에러를 상세히 보면 plugin-descriptor.properties내의 description이 빠진 것으로 예상되지만.. 그렇지 않다.

  • git에 올라간 파일 내용도 정상임을 확인.

  • 실제 서버에 올린 파일 내용도 위와 같은지 확인한다… 정상이 아님.

  • 파일 수정 후 신규 zip 파일 생성하고 다시 테스트

=======================plugin-descriptor.properties====================
description=${project.description}
version=${project.version}
name=${project.artifactId}
classname=${autocomplete.plugin.classname}
elasticsearch.version=${elasticsearch.version}
java.version=${maven.compiler.target}
==============================after build===============================
description=autocomplete plugin test
version=1.0
name=autocomplete-test-plugin
classname=org.autocomplete.test.plugin.ExamplePlugin
elasticsearch.version=6.5.4
java.version=1.8

파일 수정 후 신규 zip 파일 생성하고 다시 테스트

sh-4.2# cat plugin-descriptor.properties 
description=autocomplete plugin test
version=1.0
name=autocomplete-test-plugin
classname=org.autocomplete.test.plugin.ExamplePlugin
elasticsearch.version=6.5.4
java.version=1.8
sh-4.2# rm -rf els-plugin.zip
sh-4.2# zip -r els-plugin.zip ./autocomplete-test-plugin-1.0.jar ./plugin-descriptor.properties 
  adding: autocomplete-test-plugin-1.0.jar (deflated 78%)
  adding: plugin-descriptor.properties (deflated 36%)
sh-4.2# bin/elasticsearch-plugin install file:els-plugin.zip 
-> Downloading file:els-plugin.zip
[=================================================] 100%?? 
Exception in thread "main" java.lang.IllegalStateException: failed to load plugin autocomplete-test-plugin due to jar hell
	at org.elasticsearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:517)
	at org.elasticsearch.plugins.InstallPluginCommand.jarHellCheck(InstallPluginCommand.java:764)
	at org.elasticsearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:727)
	at org.elasticsearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:792)
	at org.elasticsearch.plugins.InstallPluginCommand.install(InstallPluginCommand.java:775)
	at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:231)
	at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:216)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
	at org.elasticsearch.cli.Command.main(Command.java:90)
	at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)
Caused by: java.util.zip.ZipException: zip END header not found
	at java.base/java.util.zip.ZipFile$Source.zerror(ZipFile.java:1529)
	at java.base/java.util.zip.ZipFile$Source.findEND(ZipFile.java:1430)
	at java.base/java.util.zip.ZipFile$Source.initCEN(ZipFile.java:1437)
	at java.base/java.util.zip.ZipFile$Source.<init>(ZipFile.java:1268)
	at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1231)
	at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:726)
	at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:843)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:246)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:176)
	at java.base/java.util.jar.JarFile.<init>(JarFile.java:346)
	at java.base/java.util.jar.JarFile.<init>(JarFile.java:317)
	at java.base/java.util.jar.JarFile.<init>(JarFile.java:256)
	at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:178)
	at org.elasticsearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:503)
	... 12 more

  • 중복하는 jar가 있을때 발생

  • find lib plugins 명령어를 쳐서 중복 jar를 확인 후 제거 한다.

2. javacafe-analyzer로 플러그인 생성

  • 자동완성 플러그인 에서 javacafe-analyzer.6.5.4.zip 소스를 받는다.

  • pom.xml을 수정한다.(elasticsearch version 6.4.3 등 맞지 않는 설정 수정)

  • 테스트와 동일한 방법으로 zip을 생성한여 설치한다.

  • 테스트와 동일한 에러 발생..

Exception in thread "main" java.lang.IllegalStateException: failed to load plugin javacafe-analyzer due to jar hell
	at org.elasticsearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:517)
	at org.elasticsearch.plugins.InstallPluginCommand.jarHellCheck(InstallPluginCommand.java:764)
	at org.elasticsearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:727)
	at org.elasticsearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:792)
	at org.elasticsearch.plugins.InstallPluginCommand.install(InstallPluginCommand.java:775)
	at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:231)
	at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:216)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
	at org.elasticsearch.cli.Command.main(Command.java:90)
	at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)
Caused by: java.util.zip.ZipException: zip END header not found
	at java.base/java.util.zip.ZipFile$Source.zerror(ZipFile.java:1529)
	at java.base/java.util.zip.ZipFile$Source.findEND(ZipFile.java:1430)
	at java.base/java.util.zip.ZipFile$Source.initCEN(ZipFile.java:1437)
	at java.base/java.util.zip.ZipFile$Source.<init>(ZipFile.java:1268)
	at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1231)
	at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:726)
	at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:843)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:246)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:176)
	at java.base/java.util.jar.JarFile.<init>(JarFile.java:346)
	at java.base/java.util.jar.JarFile.<init>(JarFile.java:317)
	at java.base/java.util.jar.JarFile.<init>(JarFile.java:256)
	at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:178)
	at org.elasticsearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:503)

  • Caused by: java.util.zip.ZipException: zip END header not found 가 발생하는 이유는 zip 파일이 정상이 아니기 때문이라고 함.

  • zip 생성 방식을 zip -r [filename.zip] [file] 에서 elasticsearch가 떠있는 java 버전의 jar 이용해서 zip을 만들어서 테스트 해본다. ( jar cvf [filename.zip] [file]) => 동일 에러 발생

  • !!!!!!!!!

  • 문제 해결 후 정상 설치함. 원인은 git으로 부터 파일을 받을때 raw링크를 이용해서 받지 않아서 . 업로드 파일이 아닌 git 주소에 해당하는 html을 통으로 받았기 때문.
    https://raw.githubusercontent.com/mondayus/analyzer/develop/analyzer.zip 를 이용해 받고 정상 설치를 확인함.

sh-4.2# wget https://raw.githubusercontent.com/mondayus/analyzer/develop/analyzer.zip
--2020-12-16 02:39:21--  https://raw.githubusercontent.com/mondayus/analyzer/develop/analyzer.zip
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.88.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.88.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 26614 (26K) [application/zip]
Saving to: 'analyzer.zip'

100%[===========================================================================>] 26,614      --.-K/s   in 0.02s   

2020-12-16 02:39:22 (1.28 MB/s) - 'analyzer.zip' saved [26614/26614]

sh-4.2# bin/elasticsearch-plugin install file:analyzer.zip 
-> Downloading file:analyzer.zip
[=================================================] 100%?? 
-> Installed javacafe-analyzer

올바른 설치 방향 및 테스트 결과

  1. 자동완성 플러그인 소스 다운로드

  2. elasticsearsch 버전 변경 (pom.xml) 및 maven install

  3. zip 파일 git에 업로드

  4. elasticsearch 에서 wget or curl -O 를 이용한 zip 파일 다운로드 ( git html주소가 아닌 실제 파일 다운로드 주소를 사용해야함)

sh-4.2# wget https://raw.githubusercontent.com/mondayus/analyzer/develop/analyzer.zip
--2020-12-16 02:39:21--  https://raw.githubusercontent.com/mondayus/analyzer/develop/analyzer.zip
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.88.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.88.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 26614 (26K) [application/zip]
Saving to: 'analyzer.zip'

100%[===========================================================================>] 26,614      --.-K/s   in 0.02s   

2020-12-16 02:39:22 (1.28 MB/s) - 'analyzer.zip' saved [26614/26614]

bin/elasticsearch-plugin install file:./~.zip 을 이용하여 plugin 설치

sh-4.2# bin/elasticsearch-plugin install file:analyzer.zip 
-> Downloading file:analyzer.zip
[=================================================] 100%?? 
-> Installed javacafe-analyzer

  1. elasticsearch를 재시작 하여 plugin 적용

  2. 끝

반응형