Send data to Elasticsearch

Overview

You can send logs to Elasticsearch via HTTP using our HTTP Data sink.

Prerequisites

Connecting to an Elasticsearch cluster requires different configurations depending on whether you are using Elastic Cloud or a self-managed cluster.

To connect to Elastic Cloud, you need the Cloud ID and an API Key. Both of these credentials can be obtained from the Elastic Cloud console.

You will typically pass these values into a configuration object, like this Go example:

cfg := elasticsearch.Config{
    CloudID: "YOUR_CLOUD_ID", // Found in the cloud console
    APIKey:  "YOUR_API_KEY"   // Generated via the cloud console
}
es, err := elasticsearch.NewClient(cfg)

Ensure you securely store your generated API Key immediately after creation, as Elastic Cloud does not allow you to view the key again for security reasons.

Connecting to a Self-Managed Cluster

Security features,specifically Authentication and TLS, are enabled by default. Trust the cluster's certificates to establish a secure HTTPS connection.

When a self-managed Elasticsearch cluster is started for the first time, the terminal output provides the necessary security credentials. You will see a block containing the following critical information:

elastic user password: The initial password for the built-in superuser.
HTTP CA certificate SHA-256 fingerprint: A unique hash used to identify and verify the cluster's self-signed security certificate.

You should record both the elastic user password and the CA fingerprint for use in your client connection configuration. These are commonly stored as environment variables, such as ELASTIC_PASSWORD and CERT_FINGERPRINT.

The root CA Certificate used by the cluster (named http_ca.crt) is located within your Elasticsearch configuration directory (e.g., $ES_CONF_PATH/certs/). If you are running Elasticsearch in a container environment like Docker, consult the relevant documentation for how to retrieve this file.

Once you have access to the http_ca.crt file, you pass its contents via the client configuration's CACert field:

cert, _ := os.ReadFile("/path/to/http_ca.crt") // Load the certificate file content

cfg := elasticsearch.Config{
    Addresses: []string{
        "https://localhost:9200", // Cluster address
    },
    Username: "elastic",
    Password: ELASTIC_PASSWORD, // Use the recorded password
    CACert:   cert               // Pass the certificate content
}
es, err := elasticsearch.NewClient(cfg)

For more detailed information, see this article.

Data sink configuration

To start sending data to Elasticsearch, follow these steps:

Create a new HTTP Data sink. To do it, go to Data sinks > New Data sink and double-click HTTP.

Give your Data sink a Name and, optionally, add a Description and some Tags. Click Finish when you're done.

Now, drag your Data sink to the required Pipeline canvas. Link it to the required Listener/Action and double-click it to configure it.

Fill the following parameters as follows:

HTTP method* - Choose POST
URL* - Enter <elastic_endpoint>
Choose the field containing the Message you wish to send on.
Content-Type - Choose your required data type.
Support special characters - Set to true
Use gzip, HTTP headers Set as required

In the Bulk configuration section, fill in the parameters as follows:

Bulk allow* - Set to true
Choose Manual delimiter* and leave it as new line (\n).
Maximum number of buffers per server URL* - Enter 50.
Event amount*, Event time limit* - Set both to 1.

Set the Authentication type* to API key. Enter your API key name* and choose the required API key value*. Create a Secret containing your API key or select one already created.