:plugin: avro
:type: codec

///////////////////////////////////////////
START - GENERATED VARIABLES, DO NOT EDIT!
///////////////////////////////////////////
:version: %VERSION%
:release_date: %RELEASE_DATE%
:changelog_url: %CHANGELOG_URL%
:include_path: ../../../../logstash/docs/include
///////////////////////////////////////////
END - GENERATED VARIABLES, DO NOT EDIT!
///////////////////////////////////////////

[id="plugins-{type}s-{plugin}"]

=== Avro codec plugin

include::{include_path}/plugin_header.asciidoc[]

==== Description

Read serialized Avro records as Logstash events

This plugin is used to serialize Logstash events as 
Avro datums, as well as deserializing Avro datums into 
Logstash events.

[id="plugins-{type}s-{plugin}-ecs_metadata"]
==== Event Metadata and the Elastic Common Schema (ECS)

The plugin behaves the same regardless of ECS compatibility, except adding the original message to `[event][original]`.

==== Encoding

This codec is for serializing individual Logstash events 
as Avro datums that are Avro binary blobs. It does not encode 
Logstash events into an Avro file.


==== Decoding

This codec is for deserializing individual Avro records. It is not for reading
Avro files. Avro files have a unique format that must be handled upon input.

.Partial deserialization
[NOTE]
================================================================================
Avro format is known to support partial deserialization of arbitrary fields,
providing a schema containing a subset of the schema which was used to serialize
the data.
This codec *doesn't support partial deserialization of arbitrary fields*.
Partial deserialization _might_ work only when providing a schema which contains
the first `N` fields of the schema used to serialize the data (and 
in the same order).
================================================================================

==== Usage
Example usage with Kafka input.

[source,ruby]
----------------------------------
input {
  kafka {
    codec => avro {
        schema_uri => "/tmp/schema.avsc"
    }
  }
}
filter {
  ...
}
output {
  ...
}
----------------------------------

[id="plugins-{type}s-{plugin}-options"]
==== Avro Codec Configuration Options

[cols="<,<,<",options="header",]
|=======================================================================
|Setting |Input type|Required
| <<plugins-{type}s-{plugin}-ecs_compatibility>> | <<string,string>>|No
| <<plugins-{type}s-{plugin}-encoding>> | <<string,string>>, one of `["binary", "base64"]`|No
| <<plugins-{type}s-{plugin}-password>> |<<password,password>>|No
| <<plugins-{type}s-{plugin}-proxy>> |<<uri,uri>>|No
| <<plugins-{type}s-{plugin}-schema_uri>> |<<string,string>>|Yes
| <<plugins-{type}s-{plugin}-ssl_certificate>> |<<path,path>>|No
| <<plugins-{type}s-{plugin}-ssl_certificate_authorities>> |list of <<path,path>>|No
| <<plugins-{type}s-{plugin}-ssl_cipher_suites>> |<<array,array>>|No
| <<plugins-{type}s-{plugin}-ssl_enabled>> |<<boolean,boolean>>|No
| <<plugins-{type}s-{plugin}-ssl_key>> |<<path,path>>|No
| <<plugins-{type}s-{plugin}-ssl_keystore_password>> |<<password,password>>|No
| <<plugins-{type}s-{plugin}-ssl_keystore_path>> |<<path,path>>|No
| <<plugins-{type}s-{plugin}-ssl_keystore_type>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-ssl_supported_protocols>> |<<array,array>>|No
| <<plugins-{type}s-{plugin}-ssl_truststore_password>> |<<password,password>>|No
| <<plugins-{type}s-{plugin}-ssl_truststore_path>> |<<path,path>>|No
| <<plugins-{type}s-{plugin}-ssl_truststore_type>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-ssl_verification_mode>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-tag_on_failure>> |<<boolean,boolean>>|No
| <<plugins-{type}s-{plugin}-target>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-username>> |<<string,string>>|No
|=======================================================================

&nbsp;

[id="plugins-{type}s-{plugin}-ecs_compatibility"]
===== `ecs_compatibility`

* Value type is <<string,string>>
* Supported values are:
** `disabled`: Avro data added at root level
** `v1`,`v8`: Elastic Common Schema compliant behavior (`[event][original]` is also added)

Controls this plugin's compatibility with the {ecs-ref}[Elastic Common Schema (ECS)].

[id="plugins-{type}s-{plugin}-encoding"]
===== `encoding`

* Value can be any of: `binary`, `base64`
* Default value is `base64`

Set encoding for Avro's payload. 
Use `base64` (default) to indicate that this codec sends or expects to receive base64-encoded bytes.

Set this option to `binary` to indicate that this codec sends or expects to receive binary Avro data.


[id="plugins-{type}s-{plugin}-password"]
===== `password`

* Value type is <<password,password>>
* There is no default value for this setting.

Password for HTTP basic authentication when fetching remote schemas.
Used together with `username`.

[id="plugins-{type}s-{plugin}-proxy"]
===== `proxy`

* Value type is <<uri,uri>>
* There is no default value for this setting.

The address of a forward HTTP proxy to use when contacting a remote schema registry.

[id="plugins-{type}s-{plugin}-schema_uri"]
===== `schema_uri` 

  * This is a required setting.
  * Value type is <<string,string>>
  * There is no default value for this setting.

schema path to fetch the schema from.
This can be a 'http' or 'file' scheme URI
example:

* http - `http://example.com/schema.avsc`
* file - `/path/to/schema.avsc`

[id="plugins-{type}s-{plugin}-tag_on_failure"]
===== `tag_on_failure` 

  * Value type is <<boolean,boolean>>
  * Default value is `false`

tag events with `_avroparsefailure` when decode fails

[id="plugins-{type}s-{plugin}-ssl_certificate"]
===== `ssl_certificate`

* Value type is <<path,path>>
* There is no default value for this setting.

Path to PEM encoded certificate file for client authentication (mutual TLS).
You may use this setting or <<plugins-{type}s-{plugin}-ssl_keystore_path>>, but not both simultaneously.

*Example*
[source,ruby]
----------------------------------
ssl_certificate => "/path/to/client.crt"
----------------------------------

[id="plugins-{type}s-{plugin}-ssl_certificate_authorities"]
===== `ssl_certificate_authorities`

* Value type is a list of <<path,path>>
* There is no default value for this setting.

Path to PEM encoded CA certificate file(s) for server verification.
This is an alternative to using <<plugins-{type}s-{plugin}-ssl_truststore_path>>.
You may use this setting or <<plugins-{type}s-{plugin}-ssl_truststore_path>>, but not both simultaneously.

*Example*
[source,ruby]
----------------------------------
ssl_certificate_authorities => ["/path/to/ca.crt"]
----------------------------------

[id="plugins-{type}s-{plugin}-ssl_cipher_suites"]
===== `ssl_cipher_suites`

* Value type is <<array,array>>
* There is no default value for this setting.

The list of cipher suites to use, listed by priorities.
Supported cipher suites vary depending on which version of Java is used.

[id="plugins-{type}s-{plugin}-ssl_key"]
===== `ssl_key`

* Value type is <<path,path>>
* There is no default value for this setting.

Path to PEM encoded private key file for client authentication.
Must be used together with <<plugins-{type}s-{plugin}-ssl_certificate>>.
The private key must be unencrypted (passphrase-protected keys are not supported).

*Example*
[source,ruby]
----------------------------------
ssl_key => "/path/to/client.key"
----------------------------------

[id="plugins-{type}s-{plugin}-ssl_enabled"]
===== `ssl_enabled`

* Value type is <<boolean,boolean>>
* There is no default value for this setting.

Enable SSL/TLS secured communication to remote schema registry.
When using HTTPS schema URIs, SSL is automatically enabled.

[id="plugins-{type}s-{plugin}-ssl_keystore_path"]
===== `ssl_keystore_path`

* Value type is <<path,path>>
* There is no default value for this setting.

The path to the JKS or PKCS12 keystore file for client certificate authentication.
Use this when the schema registry requires mutual TLS (mTLS) authentication.

[id="plugins-{type}s-{plugin}-ssl_keystore_password"]
===== `ssl_keystore_password`

* Value type is <<password,password>>
* There is no default value for this setting.

The password for the keystore file specified in <<plugins-{type}s-{plugin}-ssl_keystore_path>>.


[id="plugins-{type}s-{plugin}-ssl_keystore_type"]
===== `ssl_keystore_type`

* Value type is <<string,string>>
* There is no default value for this setting.

The format of the keystore file. It must be either `jks` or `pkcs12`.

[id="plugins-{type}s-{plugin}-ssl_supported_protocols"]
===== `ssl_supported_protocols`

* Value type is <<array,array>>
* Default value is `[]` (uses Java defaults)
* Valid values are: `TLSv1.1`, `TLSv1.2`, `TLSv1.3`

List of allowed SSL/TLS protocol versions.
When not specified, the JVM defaults are used.

[id="plugins-{type}s-{plugin}-ssl_truststore_path"]
===== `ssl_truststore_path`

* Value type is <<path,path>>
* There is no default value for this setting.

The path to the JKS or PKCS12 truststore file containing certificates to verify
the schema registry server's certificate.

*Example*
[source,ruby]
----------------------------------
input {
  kafka {
    codec => avro {
        schema_uri => "https://schema-registry.example.com:8081/schemas/ids/1"
        ssl_truststore_path => "/path/to/truststore.jks"
        ssl_truststore_password => "${TRUSTSTORE_PASSWORD}"
    }
  }
}
----------------------------------

[id="plugins-{type}s-{plugin}-ssl_truststore_password"]
===== `ssl_truststore_password`

* Value type is <<password,password>>
* There is no default value for this setting.

The password for the truststore file specified in <<plugins-{type}s-{plugin}-ssl_truststore_path>>.

[id="plugins-{type}s-{plugin}-ssl_truststore_type"]
===== `ssl_truststore_type`

* Value type is <<string,string>>
* There is no default value for this setting.

The format of the truststore file. It must be either `jks` or `pkcs12`.

[id="plugins-{type}s-{plugin}-ssl_verification_mode"]
===== `ssl_verification_mode`

* Value type is <<string,string>>
* Default value is `"full"`
* Valid options are: `full`, `none`

Options to verify the server's certificate:

* `full`: Validates that the provided certificate has an issue date that's within the not_before and not_after dates; chains to a trusted Certificate Authority (CA); has a hostname or IP address that matches the names within the certificate. (recommended)
* `none`: Performs no certificate validation. **Warning:** Disabling this severely compromises security (https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf)

*Example*
[source,ruby]
----------------------------------
input {
  kafka {
    codec => avro {
        schema_uri => "https://schema-registry.example.com:8081/schemas/ids/1"
        ssl_certificate_authorities => ["/path/to/ca.crt"]
        ssl_verification_mode => "full"
    }
  }
}
----------------------------------

[id="plugins-{type}s-{plugin}-target"]
===== `target`

* Value type is <<string,string>>
* There is no default value for this setting.
* This is only relevant when decode data into an event

Define the target field for placing the values. If this setting is not
set, the Avro data will be stored at the root (top level) of the event.

*Example*
[source,ruby]
----------------------------------
input {
  kafka {
    codec => avro {
        schema_uri => "/tmp/schema.avsc"
        target => "[document]"
    }
  }
}
----------------------------------

[id="plugins-{type}s-{plugin}-username"]
===== `username`

* Value type is <<string,string>>
* There is no default value for this setting.

Username for HTTP basic authentication when fetching remote schemas.
Used together with `password`.

*Example*
[source,ruby]
----------------------------------
input {
  kafka {
    codec => avro {
        schema_uri => "https://schema-registry.example.com:8081/schemas/ids/1"
        username => "registry_user"
        password => "${REGISTRY_PASSWORD}"
    }
  }
}
----------------------------------
