Leverage Mesos for running Spark Streaming jobs in production (Iulian Dragos, Tech Lead @Typesafe)

Mesos is a general purpose cluster manager that can scale to tens of thousands of nodes and that can handle mixed data loads and general applications. Mesos is being used in large deployments such as Twitter and AirBnB. Its versatility makes it particularly appealing to organizations that have a mixed workload and want to maximize their cluster utilization (are there any other?). But how exactly does it work when the workload is a long-running Spark Streaming job? Particularly, how does one deal with failures that are bound to happen at this scale, without data loss and service disruptions?

In this talk we’ll discuss how Spark integrates with Mesos, the differences between client and cluster deployments, and compare and contrast Mesos with Yarn and standalone mode. Then we’ll look at deploying a Spark Streaming application that should run 24/7 and show how to deploy, configure and tune a Mesos cluster such that:

  • the application runs efficiently and uses only the resources it needs
  • if any of the nodes fails (including the driver), the application recovers without data loss

Availability of seats is very limited. Please register only if you really attend the event. Thanks and we are happy to see you soon!

Btw: You find the review of our last Dev.Wednesday on our blog: http://ict.swisscom.ch/2015/12/dev-wednesday-cloud/




Genfergasse 14

Genfergasse 14, 3011 Bern, Schweiz

map Open in Google maps | Apple maps | Citymapper

Sign in to edit

Complete your event page