Death Valley Bloom
Death Valley is a desert – it gets only about 2 inches of rainfall per year. And since life depends on water, the valley name is quite appropriate. Life in Death Valley is dormant – some plant seeds …
9 photographs
Big Data Schema Evolution
Schema migrations in the relational world are now common practice. The best practices for evolving a database schema are well known, where a migration gets applied before the code that needs to use it is rolled out. If …
Type All The Things!
Scala is statically typed. Yet we rarely use it to our advantage. For example let’s say you have a function with the signature: def convert(a: Double, b: Double): String Of course, the author hasn’t written a comment, and …
Spark on AWS EMR – The Missing Manual
Apache Spark recently received top level support on Amazon Elastic MapReduce (EMR) cloud offering, joining applications such as Hadoop, Hive, Pig, HBase, Presto, and Impala. This is exciting for me, because most of my workloads run on EMR, …
Handling Avro records in Scalding
In this post I’ll try to cover how to write and read Avro records in Scalding pipelines. To begin, a reminder that Avro is a serialization format, and Scalding is a scala API on top of Hadoop. If …