Tutorial FreeBSD Elasticsearch and Logstash - Installing and setting up

In today's world, a website's ability to search files, analyze and visualize small and large amounts of data is critical. This will of course be very beneficial for the development of organizations and businesses, which require speed in searching for data or files on a website. There are many search engines that web developers use. Elasticsearch is one of many fast, powerful, and popular search and data analysis engines. Elasticsearch has gained immense popularity over the years.

Since its inception until today, Elasticsearch has grown into something much bigger than when it was first introduced. Elasticsearch is not just a simple search engine, but more than that, it has become an entire ecosystem known as the “Elastic Stack.” This thing has become a superstar, capable of handling all sorts of tasks, from basic website searches and log data analysis to complex data processing.




Elasticsearch is an open source application built on Apache Lucene. This application, written in Java, is used as a distributed and RESTful search and analysis engine capable of handling many use cases. Elasticsearch was first introduced by Elasticsearch N.V. in 2010, and is now part of the Elastic Stack.

Elasticsearch was created with extremely high scalability in mind, so it is able to handle the challenges of scaling indexes horizontally and can handle large amounts of data like a pro.


There are many excellent features offered by Elasticsearch, one of which is its very fast search and analysis capabilities. Elasticsearch is capable of searching in real-time, namely within one second. Therefore, it is natural that many people use it to handle large data volumes easily, efficiently and simply.


1. Prerequisites

OS: FreeBSD 13.2
Hostname: ns3
IP address: 192.168.5.2
Logstash version: logstash8-8.11.3
Elasticsearch version: elasticsearch8-8.11.3
Web server: Apache24
Dependencies: jna bash wazuh-server
Java version:
a. openjdk version "17.0.9" 2023-10-17
b. OpenJDK Runtime Environment (build 17.0.9+9-1)
c. OpenJDK 64-Bit Server VM (build 17.0.9+9-1, mixed mode, sharing)


2. Install Elasticsearch with Ports

Before starting the elasticsearch installation, make sure the Java and Apache applications are running on FreeBSD. You can use both Java and Apache versions following the instructions above. OK, let's just start installing the elasticsearch dependencies.

Install dependencies
root@ns3:~ # pkg install jna bash wazuh-server
The next process after dependencies is installing elasticsearch. Use the FreeBSD port system, so that all libraries can be installed.

/usr/ports/textproc/elasticsearch8
root@ns3:~ # cd /usr/ports/textproc/elasticsearch8
root@ns3:/usr/ports/textproc/elasticsearch8 # make install clean
Then, you create a start up script, so that elasticsearch can run automatically on FreeBSD. Open the rc.conf file and type the script below.

/etc/rc.conf
root@ns3:~ # ee /etc/rc.conf
elasticsearch_enable="YES"
elasticsearch_user="elasticsearch"
elasticsearch_group="elasticsearch"
elasticsearch_config="/usr/local/etc/elasticsearch"
elasticsearch_login_class="root"
elasticsearch_java_home="/usr/local/openjdk17"


3. Configuration Elasticsearch

Once you have completed the installation, continue with setting up elasticsearch. The main file you have to configure is elasticsearch.yml. Open the elasticsearch.yml file and change only a few scripts, like the example below.

/usr/local/etc/elasticsearch/elasticsearch.yml
root@ns3:~ # ee /usr/local/etc/elasticsearch/elasticsearch.yml
cluster.name: nextcloud2
node.name: node-1
path.data: /var/db/elasticsearch
path.logs: /var/run/elasticsearch
bootstrap.memory_lock: true
network.host: 192.168.5.2
http.port: 9200
discovery.seed_hosts: ["127.0.0.1", "[::1]"]
discovery.type: single-node
xpack.ml.enabled: false
xpack.security.enabled: false
xpack.security.enrollment.enabled: false
Although Elasticsearch has out-of-the-box features, you can extend its functionality by adding plugins to provide advanced analysis and process different types of data. You can install each plugin on a compatible Elasticsearch cluster, just like any other Elasticsearch plugin.

In this article we will try to install the Elasticsearch plugin and interact with it using the Elasticsearch API. Run the command below to install commonly used plugins.

Install plugin ingest-attachment
root@ns3:~ # /usr/local/lib/elasticsearch/bin/elasticsearch-plugin install ingest-attachment
The command above, installs the Elasticsearch plugin to index and search base64 encoded documents in formats such as RTF, PDF, and PPT. You can also install the "analysis-phonetic" plugin to identify search results performed by elasticsearch.

Install plugin analysis-phonetic
root@ns3:~ # /usr/local/lib/elasticsearch/bin/elasticsearch-plugin install analysis-phonetic
Another plugin you can install is “ingest-user-agent”. This plugin is used to parse the User Agent header of HTTP requests to provide identifying information about the client sending each request.

Install plugin ingest-user-agent
root@ns3:~ # /usr/local/lib/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent
Run the ownership command on the db and log files.

Install plugin ingest-user-agent
root@ns3:~ # chown -R elasticsearch:elasticsearch /var/db/elasticsearch
root@ns3:~ # chown -R elasticsearch:elasticsearch /var/run/elasticsearch
The next step is to run elasticsearch.

Run elasticsearch
root@ns3:~ # service elasticsearch restart
After that, you test whether Elasticsearch responds to queries.

Test elasticsearch
root@ns3:~ # curl -X GET "192.168.5.2:9200"
{
  "name" : "node-1",
  "cluster_name" : "nextcloud2",
  "cluster_uuid" : "6oYpEqMpTPanRtTSe61E_w",
  "version" : {
    "number" : "8.11.3",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "64cf052f3b56b1fd4449f5454cb88aca7e739d9a",
    "build_date" : "2023-12-08T11:33:53.634979452Z",
    "build_snapshot" : false,
    "lucene_version" : "9.8.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}


4. Install Logstash8

Logstash is a real-time event processing engine for managing events and logs. You can use it to collect logs, parse them, and save them for later use. Usually Logstash is paired with elasticsearch. In this section we will explain how to install Logstash on FreeBSD. After installing and configuring Logstash, we will connect it to elasticsearch, so that it can be used simultaneously.

To install Logstash, you can use the PKG package, as in the example below.

Install logstash8
root@ns3:~ # pkg install logstash8
Add the below script to the rc.conf file.

/etc/rc.conf
root@ns3:~ # ee /etc/rc.conf
logstash_enable="YES"
logstash_user="logstash"
logstash_group="logstash"
logstash_home="/usr/local/logstash"
logstash_config="/usr/local/etc/logstash"
logstash_log="YES"
logstash_java_home="/usr/local/openjdk17"
logstash_java_opts=""
logstash_opts=""
Edit the logstash.conf file, so that it can connect to elasticsearch.

/etc/rc.conf
root@ns3:~ # ee /usr/local/etc/logstash/logstash.conf
input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://192.168.5.2:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}
Run Logstash with the service command.

Run Logstash
root@ns3:~ # service logstash restart
Stopping logstash.
Waiting for PIDS: 906, 906.
Starting logstash.
After everything has been configured, you see the results by checking elasticsearch, whether it is running normally or there is a wrong script. Open the Google Chrome web browser, type "http://192.168.5.2:9200/". The result will look like the image below.



Overall, Elasticsearch is an advanced search and data analysis engine, works in real time, is fast and has high performance. Its ability to handle complex queries and provide fast search results makes it a valuable asset for organizations and companies dealing with large amounts of data.
Iwan Setiawan

I Like Adventure: Mahameru Mount, Rinjani Mount I Like Writer FreeBSD

Post a Comment

Previous Post Next Post