In the "Hakk the planet!" blog post series I'm aiming to answer common Akka community questions in a more "guided tour" way, such that they can serve for reference for anyone learning about a particular area of Akka.

This time I'll explain how to implement your own Akka HTTP Marshallers, and when to actually notice that that's the thing you should do.

If you're new to Marshallers in Akka HTTP (or Spray, as Akka's marshalling infrastructure is based on the Spray one – which Akka HTTP superseded), you may want to first read the documentation about Marshalling in Akka HTTP.

Akka HTTP (Un)Marshalling infrastructure

For a detailed overview of the marshalling infra please refer to the documentation, however let's repeat the bare minimum here so that we know what we're working with here. The marshalling infra in Akka HTTP is:

  • Implicits based
    • good: composable, compile-time checked (also – no lookups during runtime which marshaller should be used, it's already composed during compile time)
    • bad: may sometimes be problematic to set up right if you're learning the ropes or have some very complex type structure to (un)marshal
  • Supports marshalling to/from multiple content types for a given type
  • By itself does not do much marshalling, just the basic things
    • typically used in tandem with a JSON (or other data formats) library via support traits such as SprayJsonSupport via akka-http-spray-json
    • or any other JSON library via Heiko Seeberger's Akka HTTP JSON (including Argonout, circe, Json4s, Play JSON, uPickle...)
  • Also available for Java, in which case a marshaller must be passed in to operations which require it (because no implicits)

Spotting a problem

In today's post we'll solve a community question from akka-user, in which Igor has trouble unmarshalling an HttpEntity to an Either instance.

In Akka 2.4.3 we did not provide a default Unmarshaller for Either, so the below code would not (yet) compile. While working on this port (and answering Igor I actually implemented such eitherUnmarshaller, so it may end up in Akka HTTP 2.4.4).

import akka.http.scaladsl.model.HttpResponse  
import akka.http.scaladsl.unmarshalling.Unmarshal  
import akka.stream.Materializer  
import akka.actor.ActorSystem

implicit val system = ActorSystem(getClass.getSimpleName)  
implicit val materializer = ActorMaterializer() // needed for marshalling infra  
import system.dispatcher // needed because marshalling infra returns Future[T]

val response: HttpResponse = ???

// we want to make this compile:
Unmarshal(response.entity).to[Either[String, Long]]  

And this would error out with something similar like:

Error:(14, 34) could not find implicit value for parameter um: akka.http.scaladsl.unmarshalling.Unmarshaller[akka.http.scaladsl.model.ResponseEntity,Either[String,Long]]  
    Unmarshal(response.entity).to[Either[String, Long]]
                              ^                       ^

The error is correct and expected, however it leaves newcomers to this style of API lost. Let's, besides "just make it work" also dive into figuring out what's missing here so future coders have an easier time once if they get such error.

Finding the missing Unmarshaller

I'll be using IntelliJ IDEA in this example (version 2016, Community Edition with Scala plugin is sufficient), there one can highlight implicit parameters by typing meta + shift + P, and after selecting the scope:

... we're able to see all implicit parameters involved in this method call:

So now we know unmarshalling requires a dispatcher, materializer (at least in Akka 2.4.3), and finally - the Unmarshaller[HttpEntity.Strict, Either[String, Int]]. What does this tell us? That there is no such unmarshaller in scope.

The next step is usually (in the general case, when you're debugging such problem) to find if you got all the imports or support traits right. For example, if a Unmarshaller[..., JsValue] or something pointing towards JSON is missing, you perhaps have forgotten to add with SprayJsonSupport to your class. It may abso be helpful to look through the built-in Unmarshallers, which are defined in GenericUnmarshallers, PredefinedFromEntityUnmarshallers etc, all of which are defined in the akka.http.scaladsl.unmarshalling package (and similarily for marshalling).

Now that we've checked that indeed, there is no pre-defined marshaller that we could have used, it's time to implement one ourselves:

Create a custom Unmarshaller for Either

Admittedly, we're starting out with a rather complex Marshaller for our first one, but that's also an interesting exercise and once you "get" this one, the next one will be a piece of cake.

As for the type we need to provide, we're looking at something like: Unmarshaller[HttpEntity, Either[L, R]] (which we can use the FromEntityUnmarshaller[Either[L, R]] alias for – these common aliases are defined in the marshalling package).

We'll do so by providing such an implicit def that combines two marshallers, and attempts to use first the right and then the left one, we do it in this order since Either[L, R] is usually understood as R being the "successful case"–which in our case means "the more likely type we'll want to unmarshal, and if it did not work, we'll try the other one":

implicit def eitherUnmarshaller[L, R](implicit  
    ua: FromEntityUnmarshaller[L], 
    ub: FromEntityUnmarshaller[R]): FromEntityUnmarshaller[Either[L, R]] = ???

In the real-life example I'll also want it to take a ClassTag for both of the classes so I can make a very useful error message (instead of just a "meh" error message). So let's look at the full implementation:

implicit def eitherUnmarshaller[A, B](implicit  
  ua: FromEntityUnmarshaller[A], rightTag: ClassTag[A],
  ub: FromEntityUnmarshaller[B], leftTag: ClassTag[B]
  ): FromEntityUnmarshaller[Either[A, B]] =
Unmarshaller.withMaterializer {  
    implicit ex: ExecutionContext ⇒ 
    implicit mat: Materializer ⇒ 
    value: HttpEntity ⇒    
  // sneaky perf optimisation (if Future is completed already we avoid touching the ExecutionContext at all in FastFuture
  // this improvement is coming to Scala 2.12.x, if I remember correctly, thanks to Viktor Klang (?)
  import akka.http.scaladsl.util.FastFuture._

  // unmarshal as B and if successful pack it as Right
  def right = ub(value).fast.map(Right(_))
  // since A is our "2nd try" we want to keep the first exception here too!
  def fallbackLeft: PartialFunction[Throwable, Future[Either[A, B]]] = {
    case rightFirstEx ⇒ 
      val left = ua(value).fast.map(Left(_))

      // combine EitherUnmarshallingException by carring both exceptions
      left.transform(
        s = identity,
        f = leftSecondEx => new EitherUnmarshallingException(
          rightClass = rightTag.runtimeClass, right = rightFirstEx,
          leftClass = leftTag.runtimeClass, left = leftSecondEx)
        )
    }

  right.recoverWith(fallbackLeft)
}

So starting from the top here:

  • we require FromEntityUnmarshallers for both A and B to be available – makes sense, since we'll try to apply them
  • next we create an Unmarshaller.withMaterializer, as we need the Materializer for to be able to call the ub and ua unmarshallers
    • very often one would get away with just Unmarshaller { ex => value => Future(f(value)) } if for example we'd use some JSON library inside the unmarshaller.
    • notice that the result of unmarshalling is a Future – that's right, we're async here as well.
  • next we apply the right unmarshaller and if it failed (the Future would be failed), we apply the left unmarshaller using recoverWith on the first Future
    • since I want to report a very nice error message to users of this unmarshaller, I want to keep both exceptions – which is the reason for a bit more code to handle this rather than a plain recoverWith(case _ => ua(value).map(Left(_))

Once we have implemented this let's see if we're done already, and if not... why?

Are we done yet?

Okey, so we programmed our first custom Unmarshaller – actually a non-trivial one at that as well. Are we done yet? Compiler says: no. Why?

[error] could not find implicit value for parameter um: akka.http.scaladsl.unmarshalling.Unmarshaller[akka.http.scaladsl.model.HttpEntity.Strict,Either[String,Int]]
[error]       val testRight = Unmarshal(HttpEntity("42")).to[Either[String, Int]]
[error]                                                     ^

So it seems as if we didn't actually proceed – it's the same implicit missing. However we can do better than that, first let's use IntelliJ to debug this issue (and then I'll show how the same can be done manually as well – however it requires more boilerplate typing):

So we investigate the implicit parameters again (shift + meta + P) and expand all the implicit parameters–we notice that while yes the eitherUnmarshaller was indeed picked up, however one of its parameters was not able to be located – we couldn't find a FromEntityUnmarshaller[R]. Here a quick look at the eitherUnmarshaller reveals that R in our case is Int – so what we're missing is an Unmarshaller[HttpEntity, Int] (FromEntityUnmarshaller[T] is a type-alias for Unmarshaller[HttpEntity, T]).

Well, true – Akka does not define a general HttpEntity to Int unmarshaller, as it seems to be a not that common case (if we're convinced otherwise, we may add one). Implementing this one will be much simpler than the previous one actually, here's all that is needed:

implicit val rawIntFromEntityUnmarshaller: FromEntityUnmarshaller[Int] =  
  Unmarshaller.withMaterializer { 
    implicit ex ⇒ implicit mat ⇒ entity: HttpEntity ⇒
      entity.dataBytes
        .runFold(ByteString.empty)(_ ++ _) // concat the entire request body
        .map(_.utf8String.toInt) // map the resulting Future to Int
    }
  }

Or we can re-use the stringUnmarshaller provided by Akka (use it with caution, it'll load the entire payload into memory, into the String) to implement the rawIntFromEntityUnmarshaller using the following one-liner:

implicit val rawIntFromEntityUnmarshaller: FromEntityUnmarshaller[Int] =  
  PredefinedFromEntityUnmarshallers.stringUnmarshaller.map(_.toInt)

Such Unmarshallers would usually be put into some common trait and companion object using the following pattern (so you can easily access them as either mix-in or import):

trait MyUnmarshallers {  
  def rawIntFromEntityUnmarshaller = ???
  // ... 
}
object MyUnmarshallers extends MyUnMarshallers  

Finding the missing Unmarshaller, with just scalac

For this section forget about the rawIntFromEntityUnmarshaller that we just implemented. Let's step back and see how we would find out that we're missing just that, but we didn't want to use IntelliJ to help us with that search.

As promised, I don't want to leave you hanging if you'd like to achieve the same thing, but don't want to rely on an IDE to help you out here (also because they sometimes get confused in other ways than scalac actually sees things), so here's a pattern to debug missing implicit parameters using plain scalac:

First, convert find where the implicits are – in our case the to method needs a number of implicits. Next we'll want to make those passed in explicitly, however we'll obtain the values for them using implicitly– which triggers the same lookup mechanism as the implicit parameters would:

val um = implicitly[Unmarshaller[Strict, Either[String, Int]]]  
val ex = implicitly[ExecutionContext]  
val mat = implicitly[Materializer]  
val testRight = Unmarshal(HttpEntity("42")).to[Either[String, Int]](um, ex, mat)  

Now, that the implicit lookups are separated into specific implicitly calls, scalac will tell us basically the same thing, that the marshaller can't be found:

[error] could not find implicit value for parameter e: akka.http.scaladsl.unmarshalling.Unmarshaller[akka.http.scaladsl.model.HttpEntity.Strict,Either[String,Int]]
[error]       val um = implicitly[Unmarshaller[Strict, Either[String, Int]]]
[error]                          ^
[error] one error found

Now we'll help the compiler by manually providing that unmarshaller – we know which one we want to apply here, since that's the one we just implemented:

val either: FromEntityUnmarshaller[Either[String, Int]] =  
  Unmarshaller.eitherUnmarshaller[String, Int]  
val um =  
  implicitly[Unmarshaller[Strict, Either[String, Int]]](either)

And now scalac will actually point out that it's not able to provide us with an instance of that marshaller, since it's missing one of its implicit parameters:

[error] could not find implicit value for parameter ub: akka.http.scaladsl.unmarshalling.FromEntityUnmarshaller[Int]
[error]       val either: FromEntityUnmarshaller[Either[String, Int]] = Unmarshaller.eitherUnmarshaller[String, Int]
[error]

So now we know that we're missing just the FromEntityUnmarshaller[Int]–once we provide that one we're good to go! This technique can be used for any kind of missing or conflicting implicits, so be sure to remember it if you stumble into such trouble.

Looking at the tests

To round the menu up with a nice desert, let's have a look at the complete tests for the marshaller we were just working on:

import akka.http.scaladsl.unmarshalling.Unmarshaller.EitherUnmarshallingException  
import org.scalatest.{ BeforeAndAfterAll, FreeSpec, Matchers }  
import akka.http.scaladsl.testkit.ScalatestUtils  
import akka.actor.ActorSystem  
import akka.stream.ActorMaterializer  
import akka.http.scaladsl.model._  
import scala.concurrent.duration._

import scala.concurrent.Await

class UnmarshallingSpec extends FreeSpec with Matchers with BeforeAndAfterAll with ScalatestUtils {  
  implicit val system = ActorSystem(getClass.getSimpleName)
  implicit val materializer = ActorMaterializer()
  import system.dispatcher

  "The GenericUnmarshallers" - {
    // format: OFF
    implicit val rawIntFromEntityUnmarshaller: FromEntityUnmarshaller[Int] =
      Unmarshaller { implicit ex ⇒ entity ⇒ entity.toStrict(1.second).map(_.data.utf8String.toInt) }
    implicit val rawlong: FromEntityUnmarshaller[Long] =
      Unmarshaller { implicit ex ⇒ entity ⇒ entity.toStrict(1.second).map(_.data.utf8String.toLong) }
    // format: ON

    "eitherUnmarshaller should unmarshal its Right value" in {
      // we'll find:
      // PredefinedFromEntityUnmarshallers.eitherUnmarshaller[String, Int] will be found
      //
      // which finds:
      //   rawIntFromEntityUnmarshaller: FromEntityUnmarshaller[Int]
      //  +
      //   stringUnmarshaller: FromEntityUnmarshaller[String]

      val testRight = Unmarshal(HttpEntity("42")).to[Either[String, Int]]
      Await.result(testRight, 1.second) should ===(Right(42))
    }

    "eitherUnmarshaller should unmarshal its Left value" in {
      val entity = HttpEntity("I'm not a number, I'm a free man!")
      val testLeft = Unmarshal(entity).to[Either[String, Int]]
      Await.result(testLeft, 1.second) should ===(Left("I'm not a number, I'm a free man!"))
    }

    "eitherUnmarshaller report both error messages if unmarshalling failed" in {
      type ImmenseChoice = Either[Long, Int]
      val testLeft = Unmarshal(HttpEntity("I'm not a number, I'm a free man!")).to[ImmenseChoice]
      val ex = intercept[EitherUnmarshallingException] {
        Await.result(testLeft, 1.second)
      }

      ex.getMessage should include("Either[long, int]")
      ex.getMessage should include("attempted int first")
      ex.getMessage should include("Right failure: For input string")
      ex.getMessage should include("Left failure: For input string")
    }

  }

  override def afterAll() = system.terminate()
}

Tests pass and we've learned how to debug and implement (Un)Marshallers – great success!

Cheering

This post is an answer to an akka-user question, a guide on implementing your own unmarshallers (and debugging them) as well as in-depth description of the PullRequest #20274 in case you'd like to track if this unmarshaller makes it's way into an Akka HTTP release (comments of course welcome on the PR as well).

In the next posts of this series we'll look into some more advanced techniques as well as other areas of Akka. Until then, happy hakking!

Versions used

  • Akka 2.4.3
    • in which akka-http-core is stable, however akka-http (the DSLs) are still experimental and may change a bit
  • Scala 2.11.8