Avro is a row-orientedremote procedure call and data serialization framework developed within Apache's Hadoop project. It uses JSON for defining data types and protocols, and serializes data in a compact binary format. Its primary use is in Apache Hadoop, where it can provide both a serialization format for persistent data, and a wire format for communication between Hadoop nodes, and from client programs to the Hadoop services. Avro uses a schema to structure the data that is being encoded. It has two different types of schema languages; one for human editing and another which is more machine-readable based on JSON. It is similar to Thrift and Protocol Buffers, but does not require running a code-generation program when a schema changes. Apache Spark SQL can access Avro as a data source.
Four bytes, ASCII 'O', 'b', 'j', followed by the Avro version number which is 1 .
File metadata, including the schema definition.
The 16-byte, randomly-generated sync marker for this file.
For data blocks Avro specifies two serialization encodings: binary and JSON. Most applications will use the binary encoding, as it is smaller and faster. For debugging and web-based applications, the JSON encoding may sometimes be appropriate.
Schema definition
Avro schemas are defined using JSON. Schemas are composed ofprimitive types and complex types. Simple schema example:
Serializing and deserializing
Data in Avro might be stored with its corresponding schema, meaning a serialized item can be read without knowing the schema ahead of time.
Example serialization and deserialization code in Python
Serialization: import avro.schema from avro.datafile import DataFileReader, DataFileWriter from avro.io import DatumReader, DatumWriter schema = avro.schema.Parse.read) # need to know the schema to write. According to 1.8.2 of Apache Avro writer = DataFileWriter, DatumWriter, schema) writer.append writer.append writer.close
File "users.avro" will contain the schema in JSON and a compact binary representation of the data: $ od -v -t x1z users.avro 0000000 4f 62 6a 01 04 14 61 76 72 6f 2e 63 6f 64 65 63 >Obj...avro.codec< 0000020 08 6e 75 6c 6c 16 61 76 72 6f 2e 73 63 68 65 6d >.null.avro.schem< 0000040 61 ba 03 7b 22 74 79 70 65 22 3a 20 22 72 65 63 >a..< 0000400 00 05 f9 a3 80 98 47 54 62 bf 68 95 a2 ab 42 ef >......GTb.h...B.< 0000420 24 04 2c 0c 41 6c 79 73 73 61 00 80 04 02 06 42 >$.,.Alyssa.....B< 0000440 65 6e 00 0e 00 06 72 65 64 05 f9 a3 80 98 47 54 >en....red.....GT< 0000460 62 bf 68 95 a2 ab 42 ef 24 >b.h...B.$< 0000471
Deserialization: reader = DataFileReader, DatumReader) # the schema is embedded in the data file for user in reader: print user reader.close
This outputs:
Languages with APIs
Though theoretically any language could use Avro, the following languages have APIs written for them:
C
C++
C#
Go
Haskell
Java
Javascript
Perl
PHP
Python
Ruby
Rust
Scala
Avro IDL
In addition to supporting JSON for type and protocol definitions, Avro includes experimental support for an alternative interface description language syntax known as Avro IDL. Previously known as GenAvro, this format is designed to ease adoption by users familiar with more traditional IDLs and programming languages, with a syntax similar to C/C++, Protocol Buffers and others.
