Logstash avro output cannot be decoded by apache avro-tools -


i beginner both logstash , avro. setting system logstash producer kafka queue. however, running problem avro serialized events produced logstash cannot decoded avro-tools jar (version 1.8.2) apache provides. furthermore, notice serialized output logstash , avro-tools differs.

we have following setup:

  • logstash version 5.5
  • logstash avro codec version 3.2.1
  • kafka version 0.10.1
  • avro-tools jar version 1.8.2

as example, consider following schema:

{ "name" : "avrotestschema", "type" : "record", "fields" : [ {   "name" : "testfield1",   "type" : "string"   },   {   "name" : "testfield2",   "type" : "string"   } ] }

and following json string:

{"testfield1":"somestring","testfield2":"anotherstring"}

when serializing using logstash. logstash config file:

input {   stdin {     codec => json   } }  filter {  mutate {     remove_field => ["@timestamp", "@version"]   } }  output {   kafka {     bootstrap_servers => "localhost:9092"     codec => avro {       schema_uri => "/path/to/testschema.avsc"     }     topic_id => "avrotestout"   }   stdout {     codec => rubydebug   } }

output (using cat):

fhnvbwvzdhjpbmcayw5vdghlcnn0cmluzw==

when serializing using avro-tools. command:

java -jar avro-tools-1.8.2.jar jsontofrag --schema-file testschema.avsc message.json

output

somestringanotherstring

command:

java -jar avro-tools-1.8.2.jar fromjson --schema-file testschema.avsc message.json

output:

objavro.codenullavro.schema▒{"type":"record","name":"avrotestschema","fields":[{"name":"testfield1","type":"string"},{"name":"testfield2","type":"string"}]}▒▒▒▒&70▒▒hs▒u2somestringanotherstring▒▒▒▒&70▒▒hs▒u

so our question is: how configure logstash such output becomes compatible apache avro-tools jar?

update: found out logstash produced avro output base64 encoded. cannot find happens, , how make avro-tools compatible

as mentioned in update, found out standard logstash avro codec adds non optional base64 encoding avro output. found undesirable. forked codec , made encoding configurable. tested , worked out of box on several of our systems.

the fork available on github: https://github.com/rubyan/logstash-codec-avro

to set (or unset) base64 encoding, add logstash config file:

output {      stdout {         codec => avro {             schema_uri => "schema.avsc"             base64_encoding => false         }     } }

Comments

Post a Comment

Popular posts from this blog

PHP and MySQL WP -

i trying accomplish 2 things here code. writing php directly wp page using insert php plugin. 1) query mysql database , return results 1 column 4 columns. 2) make each result own hyperlink directs user new wp page runs new query on page. i able query show results , turn results dynamic hyperlink, cannot formatting down. right returns result 1 column. here have: [insert_php] $servername = "localhost"; $username = "login"; //edited privacy $password = "password"; //edited privacy $database = "database"; //edited privacy // create connection $conn = new mysqli($servername, $username, $password, $database); // check connection if ($conn->connect_error) { die("connection failed: " . $conn->connect_error); } echo "connected successfully"; //get results database $result = mysqli_query($conn,"select mine mines order mine"); $all_property = array(); //declare array saving property //showing proper...
Read more

android - InAppBilling registering BroadcastReceiver in AndroidManifest -

i'm implementing inapp billing in android app. works well, however, i'm trying decouple broadcast receiver activity manifest. suggestion in android's trivialdrive sample: // important: dynamically register broadcast messages updated purchases. // register receiver here instead of <receiver> in manifest // because call getpurchases() @ startup, therefore can ignore // broadcasts sent while app isn't running. // note: registering listener in activity bad idea, done here // because sample. regardless, receiver must registered after // iabhelper setup, before first call getpurchases(). currently there's class extends broadcastreceiver : public class iabbroadcastreceiver extends broadcastreceiver { /** * intent action receiver should filter for. */ public static final string action = "com.android.vending.billing.purchases_updated"; private final iabbroadcastlistener mlistener; public iabbroadcastreceiver(iabbroadcastlistener listener) { m...
Read more

go - golang pprof for c library code -

i have been trying profile golang application pprof, seems details of c code(linked static library) missed in output result. flame graph from frame graph can see c code time consuming reported within "runtime._externalcode", no details @ all. , sure static library compiled debug enable. so there way make pprof details(like stack trace) of 3rd party c code? or there other golang profiling tool this?
Read more