tag:blogger.com,1999:blog-52905637985857977202024-03-04T00:00:12.361-08:00Scala-bilityJames McCabehttp://www.blogger.com/profile/07411857278648943057noreply@blogger.comBlogger14125tag:blogger.com,1999:blog-5290563798585797720.post-19279997100755333792016-07-22T08:50:00.002-07:002018-10-12T04:03:44.713-07:00Spark, a Worked Example: Speeding Up DNA SequencingProcessing huge amounts of data takes time. To do it efficiently we can write code in a functional style and use the opportunities for parallelism that FP allows us. Still, if we want to spread the code over thousands of processors, there is extra work involved in distributing it and managing failure scenarios.
<br /><br />
Fortunately things have got much easier in recent years with the emergence of helper software like <b>Spark</b> which will take your code and run it reliably in a cluster. This assumes that you follow certain conventions. Specifically, your program should be a pipeline of transformations between collections of elements, especially collections of key-value pairs. Such a collection in the Spark system is called an <b>RDD</b> (Resilient Distributed Dataset).
<br/><br />
Spark is written in Scala. The API supports other languages too like Python and R, but let's stick with Scala, and illustrate how a sample problem can be solved by modelling the solution in this way and feeding the code to Spark. I will take a simplified task from the area of genomics.
<br/><br />
DNA may be sequenced by fragmenting a chromosome into many small subsequences, then reassembling them by matching overlapping parts to get the right order and forming a single correct sequence.In the FASTA format, each sequence is composed of characters: one of T/C/G/A. Here are four sample sequence fragments:
<br /><br />
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; margin-bottom: 16px; overflow: auto; padding: 16px; word-wrap: normal;"><code style="background: transparent; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; line-height: inherit; margin: 0px; overflow: visible; padding: 0px; word-break: normal; word-wrap: normal;">>Frag_56
ATTAGACCTG
>Frag_57
CCTGCCGGAA
>Frag_58
AGACCTGCCG
>Frag_59
GCCGGAATAC
</code></pre>
<br />
Can you see the overlapping parts? If you put the sequences together, the result is be <code style="background-color: rgba(0, 0, 0, 0.0392157); border-radius: 3px; box-sizing: border-box; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; margin: 0px; padding: 0.2em 0px;">ATTAGACCTGCCGGAATAC</code>.
<br /><br />
A naive solution would do it the same way you do it in your head: pick one sequence, look through all the others for a match, combine the match with what you already have, look though all the remaining ones for a match again... and so on, until all sequences have been handled or there are no more matches.
<br /><br />
Let's try a better algorithm, one which divides the solution into a number of tasks, each of which can be parallelized. First, you may want to download <a href="http://spark.apache.org/downloads.html">Spark</a>. You will also need <a href="http://www.scala-lang.org/download/">Scala</a> and <a href="http://www.scala-sbt.org/download.html">sbt</a>.
<br /><br />
We need Spark because the number of sequence fragments in a real-life scenario would be many millions. (However, I will continue to illustrate the algorithm using the toy data above.)
<br /><br />
The skeleton of a Spark program looks like this:
<br /><br />
<pre class="brush: scala">
import org.apache.spark.{SparkContext, SparkConf}
import org.apache.spark.rdd.RDD
object MergeFasta {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setAppName("MergeFasta")
val sc = new SparkContext(conf)
// The algorithm's code will go here.
}
}
</pre>
<code>main</code> is the entry point. (The Scala <code>extends App</code> syntax also works, but the documentation recommends not using it.) The first thing any Spark program needs to do is get a reference to a <code>SparkContext</code> so it can send computations to the Spark execution environment.
<br />
<br />
Now, back to the algorithm. We start with a set of small sequences (see above). These would typically be read from a file and converted to an RDD. Throughout the program we try to keep all data wrapped in RDD data structures so Spark knows how to deal with and parallelize the processing. To get an RDD from a file, we can use Spark's own function, <code>sc</code>.<code>textFile</code>(<code>"data.txt"</code>), or use regular Scala code to read the file, parse it into a collection, and call <code>sc</code>.<code>parallelize</code>(<code>anyScalaCollection</code>).
<br /><br />
By the way if you want to play about with Spark functions like this, the Spark software includes a REPL which you can fire up as follows:
<br /><br />
<pre>
$ spark-shell
scala> val myRDD = sc.parallelize(List(1,2))
myRDD: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[1] at parallelize at <console>:27
scala> myRDD.collect()
res2: Array[Int] = Array(1, 2)
</pre>
<br />
Notice how the <code>collect()</code> function is used to convert an RDD back into a Scala collection so we can view it.
<br /><br />
Anyway, now that we have a starting RDD in the program, the first step of the algorithm is to compare each sequence with every other sequence to form a matrix. For each pair, calculate the number of characters at the end of the first sequence that match the characters at the beginning of the second (0 if there is no match). The matrix looks like this:
<br /><br /><br />
<table style="border-collapse: collapse; border-spacing: 0px; box-sizing: border-box; color: #333333; display: block; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 24px; margin-bottom: 16px; margin-top: 0px; overflow: auto; width: 888px; word-break: keep-all;"><thead style="box-sizing: border-box;">
<tr style="background-color: white; border-top-color: rgb(204, 204, 204); border-top-style: solid; border-top-width: 1px; box-sizing: border-box;"><th style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;">Key sequence</th><th align="left" style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;">Other sequences</th></tr>
</thead><tbody style="box-sizing: border-box;">
<tr style="background-color: white; border-top-color: rgb(204, 204, 204); border-top-style: solid; border-top-width: 1px; box-sizing: border-box;"><td style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;">ATTAGACCTG</td><td align="left" style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;">(CCTGCCGGAA, 4), (AGACCTGCCG, 7), (GCCGGAATAC, 1)</td></tr>
<tr style="background-color: #f8f8f8; border-top-color: rgb(204, 204, 204); border-top-style: solid; border-top-width: 1px; box-sizing: border-box;"><td style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;">CCTGCCGGAA</td><td align="left" style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;">(ATTAGACCTG, 0), (AGACCTGCCG, 0), (GCCGGAATAC, 7)</td></tr>
<tr style="background-color: white; border-top-color: rgb(204, 204, 204); border-top-style: solid; border-top-width: 1px; box-sizing: border-box;"><td style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;">AGACCTGCCG</td><td align="left" style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;">(ATTAGACCTG, 0), (CCTGCCGGAA, 7), (GCCGGAATAC, 4)</td></tr>
<tr style="background-color: #f8f8f8; border-top-color: rgb(204, 204, 204); border-top-style: solid; border-top-width: 1px; box-sizing: border-box;"><td style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;">GCCGGAATAC</td><td align="left" style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;">(ATTAGACCTG, 0), (CCTGCCGGAA, 1), (AGACCTGCCG, 0)</td></tr>
</tbody></table>
<br />
There are different ways you can build this matrix. Let's assume we have a function <code>matchingLength</code> that calculates the number of characters that overlap between two sequences. Then we can use this code:<br />
<br />
<pre class="brush: scala">
val seqList = seqSet.collect().toList
val matrix = seqSet.map(seq => (seq, seqList.filterNot(_ == seq)))
matrix.map {
case (keySeq, seqs) => (keySeq, seqs.map(seq => (seq, matchingLength(keySeq, seq))))
}
</pre>
<br />
The nice thing is that if you are used to dealing with Scala collections, this will look very familiar. We are actually operating not on Scala collections but RDDs. More good news: RDDs are based on the FP technique of <b>lazy evaluation</b>, so no time is wasted on building intermediate collections. The work is postponed until you call <code>collect()</code> or something similar.
<br /><br />
The next step in the algorithm is this: for each row in the matrix, eliminate all values except the one with the longest match. Also eliminate inadequate matches, i.e. where the common part is not more than half the length of a sequence. The reduced matrix looks like:
<br /><br /><br />
<table style="border-collapse: collapse; border-spacing: 0px; box-sizing: border-box; color: #333333; display: block; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px; line-height: 24px; margin-bottom: 16px; margin-top: 0px; overflow: auto; width: 888px; word-break: keep-all;"><thead style="box-sizing: border-box;">
<tr style="background-color: white; border-top-color: rgb(204, 204, 204); border-top-style: solid; border-top-width: 1px; box-sizing: border-box;"><th style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;">Key sequence</th><th align="left" style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;">Sequence with overlap length</th></tr>
</thead><tbody style="box-sizing: border-box;">
<tr style="background-color: white; border-top-color: rgb(204, 204, 204); border-top-style: solid; border-top-width: 1px; box-sizing: border-box;"><td style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;">ATTAGACCTG</td><td align="left" style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;">(AGACCTGCCG, 7)</td></tr>
<tr style="background-color: #f8f8f8; border-top-color: rgb(204, 204, 204); border-top-style: solid; border-top-width: 1px; box-sizing: border-box;"><td style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;">CCTGCCGGAA</td><td align="left" style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;">(GCCGGAATAC, 7)</td></tr>
<tr style="background-color: white; border-top-color: rgb(204, 204, 204); border-top-style: solid; border-top-width: 1px; box-sizing: border-box;"><td style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;">AGACCTGCCG</td><td align="left" style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;">(CCTGCCGGAA, 7)</td></tr>
<tr style="background-color: white; border-top-color: rgb(204, 204, 204); border-top-style: solid; border-top-width: 1px; box-sizing: border-box;"><td style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;"><span style="background-color: #f8f8f8;">GCCGGAATAC</span></td><td align="left" style="border: 1px solid rgb(221, 221, 221); box-sizing: border-box; padding: 6px 13px;"><span style="background-color: #f8f8f8;">(</span><span style="background-color: #f8f8f8;">CCTGCCGGAA</span><span style="background-color: #f8f8f8;">, 1)</span></td></tr>
</tbody></table>
<br />
This <b>transformation</b> can be achieved using the <code>mapValues</code> function:
<br /><br />
<pre class="brush: scala">
val matrixMaximums = matrix.mapValues(seq => seq.maxBy(_._2))
</pre>
<br />
The last line of the matrix is not a good match, so we can lop it off by applying a <code>filter</code>:
<br /><br />
<pre class="brush: scala">
matrixMaximums.filter { case (key, (seq, length)) => length > seq.length / 2 }
</pre>
<br />
Now we have the sequences that we can put together to make the final sequence. We still need to get them in the right order though. Which is the first sequence? It will be any sequence that does not appear as a value anywhere.
<br /><br />
<pre class="brush: scala">
val values: RDD[String] = matrix.values.map(_._1)
val startingSeqs: RDD[String] = matrix.keys.subtract(values)
</pre>
<br />
<code>subtract</code> is one of the many useful transformations available on an RDD, and does what you would expect. If you want to see all the available RDD transformations check the <a href="http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.RDD">Spark API</a>. There are <a href="http://homepage.cs.latrobe.edu.au/zhe/ZhenHeSparkRDDAPIExamples.html">examples of using each one here.</a><br />
<br />
We extract the starting sequence (signalling an error if there wasn't one and only one), and then "follow" that through the matrix: from <code>ATTAGACCTG</code> to <code>AGACCTGCCG</code> to <code>CTGCCGGAA</code> to <code>GCCGGAATAC</code> where the trail ends. To implement this, we may call <code>lookup</code> on the RDD repeatedly inside of a recursive function that builds the String. Each sequence found is appended using the given length (7 in all cases here) to get a final sequence of <code>ATTAGACCTGCCGGAATAC</code>.
<br /><br />
To see the final result we can just use <code>println</code> . But how to run it? First compile it to a jar file. sbt is a good choice for building. The full code <a href="https://github.com/oranda/fastamerge">is available as an sbt project here</a>. When you've cloned or downloaded that, call <code>sbt package</code> in the normal way and then issue the command:
<br /><br />
<pre>
spark-submit --class "MergeFasta" --master local[4] target/scala-2.11/mergefasta_2.11-0.1.jar
</pre>
<br />
The <code>local[4]</code> means that for test purposes we will just be running the program locally rather than on a real cluster, but trying to distribute the work to all 4 processors of the local machine.
<br /><br />
When you run <code>spark-submit</code> it will produce a lot of output about how it is parallelizing tasks. You might prefer to see this in the Web UI later. If you want to turn off excessive stdout logging in the meantime, go to Spark's <code>conf</code> directory, copy <code>log4j.properties.template</code> to <code>log4j.properties</code> and set <code>log4j.rootCategory</code> to <code>WARN</code>. Then you will just see the result at the end.
<br /><br />
The next question is: how fast was it really? The Spark history server can look at the application logs stored on the filesystem and construct a UI. There is more <a href="http://spark.apache.org/docs/latest/monitoring.html">about configuration here</a>. But the main thing is to run this from the Spark root:
<br /><br />
<pre>
./sbin/start-history-server.sh
</pre>
<br />
Then navigate in your browser to <code>http://localhost:18080/</code>. You will see a list representing all the times you called <code>spark-submit</code>. Click on the latest run, the top one. Now there is a list of "jobs," the subtasks Spark parallelized and how long they took. The jobs run on RDDs will just be a few milliseconds each because of the strategy of deferring the bulk of the work to the end (lazy evaluation), where something like <code>collect() </code>is called.
<br /><br />
When I first ran the code, by far the slowest thing in the list of jobs was the <code>count()</code> on line 78, taking 2 seconds. <code>count()</code> is not a transformation: it is an <b>action</b> and it is not lazy but eager, forcing all the previously queued operations to be performed. So it is necessary to dig a bit deeper to find out where the true bottleneck is. Clicking on the link reveals a graph of what Spark has been doing:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZ0dEisGH779X1K5KpFaLLdA_5UEBnbNaZglU9w3vOTz9wwna7E6yb79zvZRPfULu9iMP1dfOpckmFG4IIDo-sVPYmiFojPQWGg35xv457yH0xy5GiMdwNlI5Db2fv-fJHtWdWAO48O2U/s1600/SparkUI.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZ0dEisGH779X1K5KpFaLLdA_5UEBnbNaZglU9w3vOTz9wwna7E6yb79zvZRPfULu9iMP1dfOpckmFG4IIDo-sVPYmiFojPQWGg35xv457yH0xy5GiMdwNlI5Db2fv-fJHtWdWAO48O2U/s1600/SparkUI.png" /></a></div>
<span style="color: #333333; line-height: 24px;"><span style="font-family: "times" , "times new roman" , serif;"><br /></span></span>
<span style="color: #333333; line-height: 24px;"><span style="font-family: "times" , "times new roman" , serif;"><br /></span></span>
This suggests that the really slow thing is the <code>subtract</code> operation on line 76. Let's look at that code again:<br />
<br />
<pre class="brush: scala">
val values: RDD[String] = matrix.values.map(_._1)
val startingSeqs: RDD[String] = matrix.keys.subtract(values)
</pre>
<br>
You can see that the <code>matrix</code> RDD is used twice. The problem is that by default Spark is not keeping the <code>matrix</code> in memory, and it has to recompute it. To solve this, the <code>cache</code> function needs to be called just before the above code:<br />
<br />
<pre class="brush: scala">
matrix.cache()
</pre>
<br />
After running everything again, the statistics breakdown looks like this:
<span style="color: #333333; font-family: "times" , "times new roman" , serif;"><span style="line-height: 24px;"><br /></span></span>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdq06j84O1Pudvby-EmLj9db94Pu1t9pl8F3Mdane-z8KkRsoNmsMELI9kFIDLxp8sIxvlH9bpF5KrBbeYm9GQsw-UNZAJZyoQlFzAcX1kdCCgwHHpgGYFJY7F1CHxgDAHQNR5L_n7U-o/s1600/SparkUI_after.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdq06j84O1Pudvby-EmLj9db94Pu1t9pl8F3Mdane-z8KkRsoNmsMELI9kFIDLxp8sIxvlH9bpF5KrBbeYm9GQsw-UNZAJZyoQlFzAcX1kdCCgwHHpgGYFJY7F1CHxgDAHQNR5L_n7U-o/s1600/SparkUI_after.png" /></a>
<br /><br />
So, thanks to caching, the time has been nearly halved.<br /><br />
There are still many optimizations possible and perhaps you can think of a better algorithm entirely. However, hopefully this post has shown how Spark can be useful for parallelizing the solution to a simple problem. Again, the full code can be found <a href="https://github.com/oranda/fastamerge">here</a>.
James McCabehttp://www.blogger.com/profile/07411857278648943057noreply@blogger.com1tag:blogger.com,1999:blog-5290563798585797720.post-36610075264851572672015-05-13T16:10:00.001-07:002015-06-08T09:11:30.020-07:00Scala.js and React: Building an Application for the Web<a href="http://scala.js/">Scala.js</a> compiles Scala code to JavaScript. I noticed on the
<link course="" https:="" reactive="" www.coursera.org=""></link><a href="https://www.coursera.org/course/reactive">Reactive Programming course at Coursera</a> that Scala.js has been integrated into it to implement a basic spreadsheet on a Web page,
suggesting good support from the Scala establishment. The principal developer of Scala.js is a collaborator
of Martin Odersky at EPFL, Sébastien Doeraene, and <a href="http://event.scaladays.org/scaladays-amsterdam-2015#!#schedulePopupExtras-6936">he will be speaking about it next month (June 2015) at Scala Days</a>
in Amsterdam.<br />
<br />
Before getting into the sample application, let's talk about the motivation for Scala.js. Basically, the Web continues to be a powerful platform for application development. Despite its many problems, it has features that the desktop and mobile phone cannot match. For example, the install process for a web application is negligible from a user's perspective: just load the web page for the first time and it is already there. Also, web applications can connect with each other via hyperlinks and REST calls.<br />
<br />
Unfortunately on the Web, JVM languages have traditionally been limited to the server-side. Functionality
on the client-side is dominated by JavaScript. The trouble is, from a developer's perspective, two languages
mean a lot of extra complexity. You can't share code between them, and to pass objects at runtime
between client and server, you end up writing serialization and validation logic in both languages.
It would be preferable to implement a web application in a single language.<br />
<br />
How is this possible, given that browsers only understand JavaScript? One possibility is to run
JavaScript on the server-side, which is the approach that Node.js takes. However, JavaScript meets
with many complaints: it is essentially untyped, and you can't take advantage of the solidity and
scalability that the JVM has been providing on the server-side for so long. It would be better to
use a language on both sides that can take advantage of the performance of JVM, the safety
of typing, and also the FP principles crossing over into the mainstream during the past ten years. This
is made possible though the use of <b>transpilers</b> which convert one source language (Scala in the case
of Scala.js) to JavaScript.<br />
<br />
One of the big challenges in Web programming is coordinating events so that elements in the view (client-side)
are updated when the model (server-side) changes. For desktop apps, the
<link design_patterns="" https:="" observer="" sourcemaking.com=""></link><a href="https://sourcemaking.com/design_patterns/observer">Observer design pattern</a> is often used, but on the Web, it takes a bit more work, and we often employ the help of some
MVC (Model-View-Controller) Web framework. The most general term for getting changes to propagate automatically
like this (as opposed to manually making calls from the view all the time) is "Reactive Programming". A particular
form of Reactive Programming is <b>Functional Reactive Programming (FRP)</b> which is about capturing relationships
in a composable way between values that change over time ("signals"). A related approach is to use a message passing system like Akka that keeps components loosely coupled. In both cases the key goals are to avoid the inefficiency of blocking operations and the hazards of mutable
data, so making the overall system <a href="http://www.reactivemanifesto.org/">scalable and resilient</a>.<br />
<br />
I would propose that the term <b>FWP</b> (Functional Web Programming) be used to cover systems that bring FP and
Reactive Programming to the Web: including <a href="http://elm-lang.org/">Elm</a>, <link: github.com="" https:="" om="" omcljs=""><a href="https://github.com/omcljs/om">ClojureScript/Om</a>, and Play Iteratees. The FWP implementation I have chosen is a combination of Scala.js and React: this article describes setting up a simple application without going into depth about FRP or being a full
tutorial.<br />
<br />While Scala.js was being developed, back in the JavaScript world frameworks were being developed that tackled the
issues of Reactive Programming. One of the most popular has been
<b><a href="https://facebook.github.io/react/">React</a></b>, developed by Facebook and Instagram. When it was introduced in May 2013, it surprised seasoned developers as it seemed to violate establish best practices. In JavaScript updating
the browser DOM is slow, so it was common to only update the necessary parts when the backing model changed. However,
in React when any component's state is changed, a complete re-render is done from the application developer's perspective. It's almost like serving a whole new page, guaranteeing that every place data is displayed it will be up-to-date. This
avoids the dangers of mutating state but it seems like it would be very slow: still, it actually outperforms other frameworks like AngularJS thanks to some a clever diffing algorithm involving a "virtual DOM" that is maintained independently of the browser's actual DOM.<br />
<br />
<br />
</link:><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFmpmp5SxXXySGIVwmd2FJtsiOEaK-lic6EhwfXbOx35NDrbaZ4dTcx4Nn_lQTm7MA5HSz2CzjkvMHagM1QErfcYvqa20cViN_9jftNe9-J4RGQzm3XDGrhgGQce6FJyyN4xrtrIjcYvI/s1600/libanius-scalajs-react-v0.2-screenshot.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFmpmp5SxXXySGIVwmd2FJtsiOEaK-lic6EhwfXbOx35NDrbaZ4dTcx4Nn_lQTm7MA5HSz2CzjkvMHagM1QErfcYvqa20cViN_9jftNe9-J4RGQzm3XDGrhgGQce6FJyyN4xrtrIjcYvI/s400/libanius-scalajs-react-v0.2-screenshot.png" width="346" /></a>Another advantage of React is that developers concentrate on re-using components rather than templates where messy logic tends to accumulate. Components are typically written in JSX, a JavaScript extension language, and then translated
to actual JavaScript using a preprocessor. For instance, consider the score text at the top left
in the Libanius app (see screenshot). In JSX this would be written:<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<pre class="brush: javascript">var ScoreText = React.createClass({
render: function() {
return (
<span className="score-text">
Score: {this.props.scoreText}
</span>
);
}
});
React.render(
<ScoreText />,
document.getElementById('content')
);<b>
</b></pre>
<br />
This is JavaScript plus some syntactic sugar. Notice how the score variable is passed in using the <code>props</code> for the component.<br />
<link: elm-lang.org="" http:=""><br />The code above is converted to normal JavaScript by running the preprocessor. On the command-line you would typically run something like this to watch your source directory and translate code to raw JavaScript in the build directory whenever anything changes:</link:><br />
<div>
<br />
<pre class="brush: plain">> jsx --watch src/ build/</pre>
<br />
This is using standard React so far. However this is not the way we do it with Scala.js.<br />
<br />
Firstly, there exists a Scala library from Haoyi Li called
<a href="http://lihaoyi.github.io/scalatags/">Scalatags</a>, that includes Scala
equivalents for HTML tags and attributes. Let's assume we have a file <code>QuizScreen.scala</code> in which we are
writing the view. The core code may start off looking a bit like this:<br />
<br />
<pre class="brush: scala">@JSExport
object QuizScreen {
@JSExport
def main(target: html.Div) = {
val quizData = // … Ajax call
target.appendChild(
span(`class` := "alignleft", "Score: " + quizData.score)
// ... more view code here
)
}
} </pre>
<span style="font-weight: normal;">
<br />Notice that <code>span</code> is a Scala method (the Scalatags library has to be imported). Assuming
you've configured SBT to use Scala.js (see below), this is converted to JavaScript in SBT by calling:</span><br />
<link: elm-lang.org="" http:=""><link: github.com="" https:="" om="" omcljs=""><code><br /></code></link:></link:>
<link: elm-lang.org="" http:=""><link: github.com="" https:="" om="" omcljs=""><code>> fastOptJS </code></link:></link:><br />
<link: elm-lang.org="" http:=""><link: github.com="" https:="" om="" omcljs=""><br /></link:></link:>
<link: elm-lang.org="" http:=""><link: github.com="" https:="" om="" omcljs=""><code></code>A good thing about the Scala.js compiler is that it keeps the target JavaScript small by eliminating any code
from included libraries that is not used. To stop this from happening on the entry point itself
in <code>QuizScreen</code>, it is necessary to use the <code>@JSExport</code> annotation both on the object and the
<code>main</code> method. This guarantees that <code>main()</code> will be callable from JavaScript. </link:></link:><br />
<br />
<link: elm-lang.org="" http:=""><link: github.com="" https:="" om="" omcljs="">So now we have seen the React way and the Scala.js way. How do we combine them? A good option is to use
the <link github.com="" https:="" japgolly="" scalajs-react=""></link><a href="https://github.com/japgolly/scalajs-react%3Escalajs-react">scalajs-react </a>library from David Barri. Now the
ScoreText component looks like this:<br />
</link:></link:><br />
<pre class="brush: scala">
val ScoreText = ReactComponentB[String]("ScoreText")
.render(scoreText => <.span(^.className := "alignleft", "Score: " + scoreText))
.build
</pre>
<br />
Compare this with the JSX version. The Scala code is more concise. Notice that the <code>render()</code> method is present. It's also possible to use other React lifecycle methods if necessary, like <code>componentDidMount()</code> for initializing the component.<br />
<link: elm-lang.org="" http:=""><link: github.com="" https:="" om="" omcljs=""><br /></link:></link:>
<link: elm-lang.org="" http:=""><link: github.com="" https:="" om="" omcljs="">Notice also that it uses a Scala <code>span</code>. A specialized version of Scalatags is used here. At first the
extra symbols look intimidating, but just remember that <code><</code> is used for tags and <code>^</code> is used for attributes, and they are imported like this: <br />
</link:></link:><br />
<pre class="brush: scala">import japgolly.scalajs.react.vdom.prefix_<^._
</pre>
<br />
In classic JavaScript React, a component can hold state, as seen in references to <code>this.state</code> and
the <code>getInitialState()</code> method, which might look like this.<br />
<br />
<pre class="brush: scala"> getInitialState: function() {
return {data: []};
}
</pre>
<span style="font-weight: normal;"><br />
The Scala version lets us define the state more clearly because it is a strongly typed language. For example,
the state for the <code>QuizScreen</code> looks like this:</span>
<br />
<pre class="brush: scala">case class State(userToken: String, currentQuizItem: Option[QuizItemReact] = None,
prevQuizItem: Option[QuizItemReact] = None, scoreText: String = "",
chosen: Option[String] = None, status: String = "")
</pre>
<br />
<code></code>It is best to centralize the state for the screen like this and pass it down to components,
rather than having each sub-component have its own separate state object. That could get out of hand!<br />
<br />
By the way, you can compose components just as you can in classic React. The central component of
the <code>QuizScreen</code> object is the QuizScreen component, and it contains the
<code>ScoreText</code> component along with the various other bits and pieces. The code below shows
how this is all put together.<br />
<br />
<pre class="brush: scala">val QuizScreen = ReactComponentB[Unit]("QuizScreen")
.initialState(State(generateUserToken))
.backend(new Backend(_))
.render((_, state, backend) => state.currentQuizItem match {
// Only show the page if there is a quiz item
case Some(currentQuizItem: QuizItemReact) =>
<.div(
<.span(^.id := "header-wrapper", ScoreText(state.scoreText),
<.span(^.className := "alignright",
<.button(^.id := "delete-button",
^.onClick --> backend.removeCurrentWordAndShowNextItem(currentQuizItem),
"DELETE WORD"))
),
QuestionArea(Question(currentQuizItem.prompt,
currentQuizItem.responseType,
currentQuizItem.numCorrectResponsesInARow)),
<.span(currentQuizItem.allChoices.map { choice =>
<.div(
<.p(<.button(
^.className := "response-choice-button",
^.className := cssClassForChosen(choice, state.chosen,
currentQuizItem.correctResponse),
^.onClick --> backend.submitResponse(choice, currentQuizItem), choice))
)}),
PreviousQuizItemArea(state.prevQuizItem),
StatusText(state.status))
case None =>
if (!state.quizEnded)
<.div("Loading...")
else
<.div("Congratulations! Quiz complete. Score: " + state.scoreText)
})
.buildU</pre>
<br />
The central component (<code>QuizScreen</code>) contains the other components (implementations not shown) and also has
access to a <code>State</code> and a <code>Backend</code>. The backend contains logic that is a bit
more extended. For example, in the code above, observe that <code>submitResponse</code> is
called above on the <code>backend</code> when a button is clicked by the user. The code invoked is:<br />
<br />
<pre class="brush: scala">class Backend(scope: BackendScope[Unit, State]) {
def submitResponse(choice: String, curQuizItem: QuizItemReact) {
scope.modState(_.copy(chosen = Some(choice)))
val url = "/processUserResponse"
val response = QuizItemAnswer.construct(scope.state.userToken, curQuizItem, choice)
val data = upickle.write(response)
val sleepMillis: Double = if (response.isCorrect) 200 else 1000
Ajax.post(url, data).foreach { xhr =>
setTimeout(sleepMillis) { updateStateFromAjaxCall(xhr.responseText, scope) }
}
}
def updateStateFromAjaxCall(responseText: String, scope: BackendScope[Unit, State]): Unit = {
val curQuizItem = scope.state.currentQuizItem
upickle.read[DataToClient](responseText) match {
case quizItemData: DataToClient =>
val newQuizItem = quizItemData.quizItemReact
// Set new quiz item and switch curQuizItem into the prevQuizItem position
scope.setState(State(scope.state.userToken, newQuizItem, curQuizItem,
quizItemData.scoreText))
}
}
// more backend methods...
}
</pre>
<br />
<code>submitResponse</code> makes an Ajax POST call to the server, collects the results and
updates the <code>State</code> object. The React framework will take care of the rest, i.e.
updating the DOM to reflect the changes to <code>State</code>.<br />
<br />
In making the Ajax call, the upickle library (again from Haoyi Li) is used for
serialization/deserialization. This is also used on the server side of our Scala.js application.
The core of the server side is a Spray server. A simple router is defined which recognizes the call to <code>processUserResponse</code> made above:<br />
<br />
<pre class="brush: scala">object Server extends SimpleRoutingApp {
def main(args: Array[String]): Unit = {
implicit val system = ActorSystem()
lazy val config = ConfigFactory.load()
val port = config.getInt("libanius.port")
startServer("0.0.0.0", port = port) {
// .. get route not shown here
post {
path("processUserResponse") {
extract(_.request.entity.asString) { e =>
complete {
val quizItemAnswer = upickle.read[QuizItemAnswer](e)
upickle.write(QuizService.processUserResponse(quizItemAnswer))
}
}
}
}
}
}
}
</pre>
<br />
The "processUserResponse" path extracts the post data using upickle then passes the call on to
the <code>QuizService</code> singleton which contains the mid-tier business logic, and relies on
the core Libanius library to run the main back-end functionality on the Quiz. I won't go into
detail about this logic, but note that both for historical reasons and future portability it relies
on simple files to hold quiz data rather than a database system.<br />
<br />
Back to the Spray server: when the <code>QuizScreen</code> page is initially loaded, this route is used:<br />
<br />
<pre class="brush: scala"> get {
pathSingleSlash {
complete{
HttpEntity(
MediaTypes.`text/html`,
QuizScreen.skeleton.render
)
}
}
</pre>
<br />
The <code>QuizScreen</code> mentioned here is <i>not</i> the <code>QuizScreen</code> on the
client-side that is described above. In fact, it is a <i>server-side</i> <code>QuizScreen</code>
that makes a call to the client-side <code>QuizScreen</code>. Like this:<br />
<br />
<pre class="brush: scala">object QuizScreen {
val skeleton =
html(
head(
link(
rel:="stylesheet",
href:="quiz.css"
),
script(src:="/app-jsdeps.js")
),
body(
script(src:="/app-fastopt.js"),
div(cls:="center", id:="container"),
script("com.oranda.libanius.scalajs.QuizScreen().main()")
)
)
}
</pre>
<br />
Again the tags are from Scalatags. The main call is in the last <code>script</code> tag. Recall that on
the client-side we use <code>@JSExport</code> to make the <code>QuizScreen().main()</code> available:<br />
<br />
<pre class="brush: scala">
@JSExport
def main(): Unit = {
QuizScreen() render document.getElementById("container")
}
</pre>
<br />
Also notice in the skeleton above, there are two included JavaScript libraries:<br />
<ul>
<li><code>app-fastopt.js</code>: In a Scala.js application, the <code>*-fastopt.js</code> file is the final output of the <code>fastOptJS</code> task, containing the JavaScript code that has been generated from your Scala code.</li>
<li><code>app-jsdeps.js</code>: In a Scala.js application, the <code>*-jsdeps.js</code>, contains all additional JavaScript libraries: in our case, the only thing it incorporates is <code>react-with-addons.min.js</code>. </li>
</ul>
Here are the essentials of the SBT configuration, which can be used as a starting point for other Scala.js projects, as it just uses the most basic dependencies, including Scalatags, upickle for serialization, and utest for testing.<br />
<br />
<pre class="brush: scala">import sbt.Keys._
name := "Libanius Scala.js front-end"
// Set the JavaScript environment to Node.js, assuming that it is installed, rather than the default Rhino
scalaJSStage in Global := FastOptStage
// Causes a *-jsdeps.js file to be generated, including (here) React
skip in packageJSDependencies := false
val app = crossProject.settings(
unmanagedSourceDirectories in Compile +=
baseDirectory.value / "shared" / "main" / "scala",
libraryDependencies ++= Seq(
"com.lihaoyi" %%% "scalatags" % "0.5.1",
"com.lihaoyi" %%% "utest" % "0.3.0",
"com.lihaoyi" %%% "upickle" % "0.2.8"
),
scalaVersion := "2.11.6",
testFrameworks += new TestFramework("utest.runner.Framework")
).jsSettings(
libraryDependencies ++= Seq(
"org.scala-js" %%% "scalajs-dom" % "0.8.0",
"com.github.japgolly.scalajs-react" %%% "core" % "0.8.3",
"com.github.japgolly.scalajs-react" %%% "extra" % "0.8.3",
"com.lihaoyi" %%% "scalarx" % "0.2.8"
),
// React itself (react-with-addons.js can be react.js, react.min.js, react-with-addons.min.js)
jsDependencies += "org.webjars" % "react" % "0.13.1" / "react-with-addons.js" commonJSName "React"
).jvmSettings(
libraryDependencies ++= Seq(
"io.spray" %% "spray-can" % "1.3.2",
"io.spray" %% "spray-routing" % "1.3.2",
"com.typesafe.akka" %% "akka-actor" % "2.3.6",
"org.scalaz" %% "scalaz-core" % "7.1.2"
)
)
lazy val appJS = app.js.settings(
// make the libanius core JAR available
// ...
unmanagedBase <<= baseDirectory(_ / "../shared/lib")
)
lazy val appJVM = app.jvm.settings(
// make sure app-fastopt.js, app-jsdeps.js, quiz.css, the libanius core JAR, application.conf
// and shared source code is copied to the server
// ...
)
</pre>
<br />
As you can see, the special thing about a Scala.js client-server SBT configuration is that it is divided into
three parts: <code>js</code>, <code>jvm</code>, and <code>shared</code>. The <code>js</code> folder contains
code to be compiled by ScalaJS, the <code>jvm</code> folder contains regular Scala code used on the server-side,
and the <code>shared</code> folder contains code and configuration that should be accessible to both <code>js</code> and <code>jvm</code>. This is achieved by using the <code>crossProject</code> builder from Scala.js, which constructs two separate projects, the <code>js</code> one and the <code>jvm</code> one.<br />
<br />
So far we've been assuming that any generated JavaScript will run in a browser. However, Scala.js also works with
"headless runtimes" like Node.js or PhantomJS to ensure you can run it from the command-line on the server-side too:
this is important in testing. Notice the <code>scalaJSStage in Global := FastOptStage</code> line above.<br />
<br />
Now for a grand overview of the web application, let's look at the directory structure. You can see how slim the
application really is: there are only a few key source files.<br />
<br />
<pre brush="plain">libanius-scalajs-react/
<b> build.sbt</b>
app/
js/
src/
main/
scala/
com.oranda.libanius.scalajs/
<b>QuizScreen.scala</b>
target/
jvm/
src/
main/
resources/
<b>application.conf</b>
scala/
com.oranda.libanius.sprayserver/
<b>QuizScreen.scala</b>
<b>QuizService.scala</b>
<b>Server.scala</b>
target/
shared/
lib/
<b>libanius-0.982.jar</b>
src/
main/
resources/
<b>quiz.css</b>
scala/
com.oranda.libanius.scalajs/
<b>ClientServerObjects.scala</b>
QuizItemReact
</pre>
<br />
Again notice there is a <code>QuizScreen</code> on both the server-side and client-side: the former calls the latter.<br />
<br />
One thing that I didn't mention yet is the <code>quiz.css file</code> that is used in the client-side <code>QuizScreen</code>. This is just an old-fashioned CSS file, but of course it also possible to use LESS files. Furthermore, if you don't anticipate having a graphic designer want to change your styles, you could even go the whole way in making your application <i>type safe</i>, and write your styles in Scala with <a href="https://github.com/japgolly/scalacss">ScalaCSS</a>.<br />
<br />
The full code for this front-end to Libanius is <a href="https://github.com/oranda/libanius-scalajs-react">on Github</a>. As of writing there is a <a href="https://thawing-stream-3905.herokuapp.com/">deployment on Heroku</a> (may not be supported indefinitely). For a full tutorial on Scala.js, see <a href="http://lihaoyi.github.io/hands-on-scala-js/">Hands-on Scala.js</a> from Haoyi Li. There is also a <a href="http://www.scala-js.org/doc/tutorial.html">small official tutorial</a>.</div>
James McCabehttp://www.blogger.com/profile/07411857278648943057noreply@blogger.com46tag:blogger.com,1999:blog-5290563798585797720.post-60517392061570515482015-04-06T06:58:00.000-07:002015-04-06T07:23:42.945-07:00Speaking Actors Living in the Phone<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjr9DXoHh3c6F4yeJow84DlaAJ_SHNbnzS7twMh6du0dyjef6VVpP66Pe9CPgSHcgh_mySVcgUGt3k_6lb5Nz6odjt1WQMT4ejPCX9ZVDz3AfqKI9G9BuEZTbSELb6-p6PlV_-Kql6nb6k/s1600/photoQuizScreen.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjr9DXoHh3c6F4yeJow84DlaAJ_SHNbnzS7twMh6du0dyjef6VVpP66Pe9CPgSHcgh_mySVcgUGt3k_6lb5Nz6odjt1WQMT4ejPCX9ZVDz3AfqKI9G9BuEZTbSELb6-p6PlV_-Kql6nb6k/s1600/photoQuizScreen.jpg" height="200" width="179" /></a>
A Hollywood actor who couldn't speak wouldn't get far. How soon before we say the same of mobile apps?<br />
<br />
This article is a bit off-topic for this blog, but it's a bit of fun and the subject deserves more attention. Over the past 18 months the <a href="http://en.wikipedia.org/wiki/Google_Text-to-Speech">Google Text-to-Speech API</a> has come a long way on the Android platform. I decided to take advantage of it in the <a href="https://play.google.com/store/apps/details?id=com.oranda.libanius&hl=en">Libanius quiz app</a>. For example, when the app presents a quiz item, it should speak the prompt ("Clairvoyance" in the picture).<br />
<br />
Libanius uses the actor model from Akka to coordinate its subsystems. To see how Android can be set up to use Akka actors, see <a href="http://typesafe.com/activator/template/macroid-akka-pingpong">this</a> from Typesafe, or <a href="http://scala-bility.blogspot.de/2013/08/akka-in-your-pocket.html">this</a> from me. Assuming the Akka infrastructure is in place, a speaking actor can be created like this:<br />
<br />
<br />
<pre class="brush: scala">class Voice(implicit ctx: Context) extends Actor with TextToSpeech.OnInitListener {
val tts = new TextToSpeech(ctx, this)
override def receive = {
case Speak(text: String, quizGroupKeyType: String) => // see next code snippet
case _ => logError("Voice received an unknown command")
}
}
</pre>
<br />
The <code>Context</code> of the Android activity is passed to the actor because the <code>TextToSpeech</code> class needs it for initialization. <code>Speak</code> is just a case class for the message that is sent to the actor. We have to implement what happens when that message is received.<br />
<br />
The prompt could be in various different languages. The Google API supplies a different voice for each language, so we need to take advantage of this.<br />
<br />
<pre class="brush: scala"> case Speak(text: String, quizGroupKeyType: String) =>
KnownQuizGroups.getLocale(quizGroupKeyType).foreach(<b>speak</b>(text, _))
</pre>
<br />
<code>KnownQuizGroups</code> is a simple map of Libanius quiz group types (e.g. "Spanish") to locales<br />
recognized by <code>TextToSpeech</code>.<br />
<br />
The <code>speak</code> method finally uses the <code>TextToSpeech</code> API to set the voice according to the given locale, and actually speak the text. It works fine whether the text is a word or a full sentence.<br />
<br />
<pre class="brush: scala">def speak(text: String, locale: Locale) {
setSpeechLanguage(locale)
tts.speak(text, TextToSpeech.QUEUE_FLUSH, null)
}
private[this] def setSpeechLanguage(locale: Locale): Unit = {
val result: Int = tts.setLanguage(locale)
if (result == TextToSpeech.LANG_MISSING_DATA || result == TextToSpeech.LANG_NOT_SUPPORTED)
logError("Language is not available.")
}
</pre>
<br />
Also we must remember in the <code>Speak</code> code to reset the language after the message is spoken.<br />
<br />
<pre class="brush: scala"> setSpeechLanguage(DEFAULT_LOCALE)
</pre>
<br />
While we're adding sound to the Libanius quiz, how about a ping sound for a correct answer, and a buzz sound for a wrong answer? This is a job for another actor: <code>SoundPlayer</code>. The guts of it look like this:<br />
<br />
<pre class="brush: scala">class SoundPlayer(implicit ctx: Context) extends Actor {
val audioManager: AudioManager =
ctx.getApplicationContext.getSystemService(Context.AUDIO_SERVICE).asInstanceOf[AudioManager]
val soundPool: SoundPool = new SoundPool(10, AudioManager.STREAM_MUSIC, 0)
var soundPoolMap = Map[SoundSample, Int]()
override def receive = {
case Load() =>
loadSounds()
case Play(soundSample: SoundSample) =>
val curVolume = audioManager.getStreamVolume(AudioManager.STREAM_MUSIC)
soundPool.play(soundPoolMap(soundSample), curVolume, curVolume, 1, 0, 1f)
case _ =>
logError("SoundPlayer received an unknown command")
}
}
</pre>
<br />
There is <a href="https://github.com/oranda/libanius-android/tree/master/src/main/scala/com/oranda/libanius/mobile/actors">more complete code</a> on Github. This is all for the Android platform, but of course you can also use Akka actors for sound on other platforms like the Mac: see Alvin Alexander's <a href="http://alvinalexander.com/scala/wikipedia-page-reader-scala-akka">Wikipedia Reader </a>page for some sample code.
Have fun!James McCabehttp://www.blogger.com/profile/07411857278648943057noreply@blogger.com0tag:blogger.com,1999:blog-5290563798585797720.post-30907597798288258802014-09-23T14:30:00.001-07:002014-10-04T19:38:35.758-07:00The QuizItemSource TypeclassWhen talking about scaling an application, people are usually referring to performance: how to design the software so that
limited hardware can support millions of users? But it is also important to be able to scale the <i>feature set</i> of an
application. If users say they want a new button, you don't want to have to make extensive refactorings of your core model classes as is all too often the case in industry. The key to both of these related demands is <b>decoupling</b>.
<br />
<br />
While writing the Libanius quiz application, I found <b>typeclasses</b> to be very useful in decoupling behaviour from
the model classes. Typeclasses (formally spelt "type classes") originated in Haskell, but can be used
in Scala too, being a language that supports FP well. I'll describe one use of typeclasses in Libanius, an
example that should be less "theoretical" and more extended than the ones that are usually given.
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJ-f2kvSAywFCzn1h3zKBvj_s9AaO3x9wwdh9zqil8HByeHsW7bL8NJXwK_g1Txb1Ssyml7PPljG2-FEumH6eI2VOV-xnmznJkNSfMs_LGaC55iyPwFMfHmXijF3ZFifDUk6-sj62W2_Q/s1600/FindQuizItem.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJ-f2kvSAywFCzn1h3zKBvj_s9AaO3x9wwdh9zqil8HByeHsW7bL8NJXwK_g1Txb1Ssyml7PPljG2-FEumH6eI2VOV-xnmznJkNSfMs_LGaC55iyPwFMfHmXijF3ZFifDUk6-sj62W2_Q/s1600/FindQuizItem.jpg" height="332" width="400" /></a>To start with, the core model in Libanius is a simple containment hierarchy as shown in the diagram. The
<code>Quiz</code> data structure
contains all the information necessary to provide a stream of quiz items to a single user. It is split by topic into <code>QuizGroup</code>s, which, for performance reasons, are further partitioned into "memory levels" according to how
well the user has learned the items. The flow of a typical request then is <code>Quiz.findQuizItem()</code>
followed by <code>QuizGroup.findQuizItem()</code> calls, followed by <code>QuizGroupMemoryLevel.findQuizItem()</code> calls.
<br />
<br />
In practice the <code>findQuizItem()</code> signatures and functionality were similar but not identical. Rather than have
them scattered throughout the model classes, it was desirable to put them together in a single place where the
similarites could be observed and moulded, avoiding any unnecessary repetition. The same was true of other operations in
the model classes. Pulling them out would also solve the problem of too much bloat in the model.
<br />
<br />
Furthermore, although the <code>Quiz</code> data structures are optimized for finding quiz items, I wanted to be able
to extend the same action to other entities. A wide range of scenarios can be imagined:
<br />
<br />
<ol>
<li>A Twitter feed of quotes might be seen as a mapping from speakers to quotations, which could be turned into a Who-Said-What quiz.</li>
<li>A set of web pages with recipes could be read as a "What ingredient goes with what dish?" quiz.</li>
<li>In a chat forum context, a person might be thinking of quiz questions and feeding them through.</li>
<li>A clever algorithm might not simply be "finding" quiz items in a store, but dynamically generating them based on previous answers. </li>
<li>The Quiz might not be a Scala data structure, but a database. This could be a NoSQL in-memory database like Redis or leveldb. It could also be a traditional RDBMS like Postgres accessed via Slick.</li>
</ol>
<br />
What links all these cases together is that a <code>findQuizItem()</code> function is necessary but with a new implementation. So the immediate priority is to <b>decouple</b> the <code>findQuizItem()</code> action from the simple <code>Quiz</code> data structure. Notice that
this action might be applied to structures that didn't even know they were supposed to be quizzes. We don't want to
have to retroactively modify classes like <code>TwitterFeed</code> and <code>ChatParticipant</code> to make them
"quiz aware".
<br />
<br />
If you know a lot of Java, some design patterns might occur to you here, like Adapter, Composite, or Visitor. In the Visitor pattern, in particular, you can separate out arbitary operations from the data structures and form a parallel hierarchy
to run those operations. But the original data structures still have to have an <code>accept()</code> method defined. In
Scala a more convenient alternative, graced with all the polymorphism you could ever need, is the typeclass construct.
Typeclasses excel at separating out orthogonal concerns.
<br />
<br />
Firstly a trait is defined for the typeclass with the behaviour we wish to separate out. Let's begin by calling the main function <code>produceQuizItem()</code> rather than <code>findQuizItem()</code>, to abstract it more from the implementation:<br />
<br />
<pre class="brush: scala">trait QuizItemSource[A] {
def produceQuizItem(component: A): Option[QuizItem]}
}
</pre>
<br />
The component could be a <code>Quiz</code>, a <code>QuizGroup</code>, a <code>QuizGroupMemoryLevel</code> or anything
else. Instead of calling <code>produceQuizItem()</code> <i>on</i> the component, the component is passed as a parameter
to the operation.
<br />
<br />
Next, define some sample implementations (aka typeclass instances) for the entities (aka model components) that we already have:
<br />
<br />
<pre class="brush: scala">object modelComponentsAsQuizItemSources {
implicit object quizAsSource extends QuizItemSource[Quiz] {
def produceQuizItem(quiz: Quiz): Option[QuizItem]} =
… // implementation calls methods on quiz
}
implicit object quizGroupAsSource extends QuizItemSource[QuizGroup] {
def produceQuizItem(quizGroup: QuizGroup): Option[QuizItem]} =
… // implementation calls methods on quizGroup
}
implicit object quizGroupMemoryLevelAsSource extends QuizItemSource[QuizGroupMemoryLevel] {
def produceQuizItem(quizGroupMemoryLevel: QuizGroupMemoryLevel): Option[QuizItem]} =
… // implementation calls methods on quizGroupMemoryLevel
}
}
</pre>
<br />
These sample implementations can be accessed in client code once it is imported in this way:
<br />
<br />
<pre class="brush: scala">import modelComponentsAsQuizItemSources._
</pre>
<br />
However, we don't want to call implementations directly, like <code>quizGroupAsSource.produceQuizItem(myQuizGroup)</code>.
Instead we want to just call <code>produceQuizItem(myQuizGroup)</code> and let the correct implementation be
selected automatically: this is what polymorphism is about. To facilitate this, let's define a function
in the middle for accessing the implementations (and this is really the final piece of the puzzle for typeclasses!):
<br />
<br />
<pre class="brush: scala">object QuizItemSource {
def produceQuizItem[A](component: A)(implicit qis: QuizItemSource[A]): Option[QuizItem] =
qis.produceQuizItem(component)
}
</pre>
<br />
If you're coming from other languages, it's the use of the implicit that is at first difficult to grasp, but this is
really the Scala feature that makes typeclasses convenient to use, where they are not in, say, Java. Let me explain what is happening here:
<br />
<br />
Suppose in client code, the <code>modelComponentsAsQuizItemSources</code> has been imported as above. Now the <code>quizGroupAsSource</code> is <i>implicitly</i> in scope, among other implementations. If there is now a call to
<code>QuizItemSource.produceQuizItem(quizGroup)</code>, the compiler, seeing that <code>quizGroup</code> is of type
<code>QuizGroup</code> will match <code>qis</code> to an implicit of type <code>QuizItemSource[QuizGroup]</code>: this is <code>quizGroupAsSource</code>, the correct implementation.
<br />
<br />
The great thing is that the correct <code>QuizItemSource</code> implementation will always be selected based on the type of entity passed to <span style="font-family: monospace;">produceQuizItem</span>, and also the <i>context</i> of the client, specifically the imports it has chosen. Also, we have achieved our original goal
that whenever we realize we want to turn an entity into a quiz (<code>TwitterFeed</code>, <code>ChatParticipant</code>,
etc.) we don't have to modify that class: instead, we add an instance of the typeclass to <code>modelComponentsAsQuizItemSources</code> and import it.<br />
<br />
This approach is a break with OO, moving the emphasis from objects to actions. Instead of adding actions (operations) to objects, we are adding objects to an action generalization (typeclass). We can make that action
as powerful as we like by continuing to add objects to it. Again, unlike in OO, those objects do not need
to be custom-built to be aware of our action.
<br />
<br />
That is the main work done. There are two more advanced points to cover.
<br />
<br />
<u>Firstly</u>, it is possible to add some "syntactical sugar" to the definition of <code>produceQuizItem()</code> above. In this code
<br />
<br />
<pre class="brush: scala">def produceQuizItem[A](component: A)(implicit qis: QuizItemSource[A]): Option[QuizItem] =
qis.produceQuizItem(component)
</pre>
<br />
… the <code>qis</code> variable may be viewed as clutter. We can push it into the background using <b>context bound syntax</b>:
<br />
<br />
<pre class="brush: scala">def produceQuizItem(A : QuizItemSource](component: A): QuizItem =
implicitly[QuizItemSource].produceQuizItem(component)
</pre>
<br />
What happens is that the context bound <code>A : QuizItemSource</code> establishes an implicit parameter, which is then pulled out by <b>implicitly</b>. This syntax is a bit shorter, and looks shorter still if you do not need to call implicitly but are simply passing the implicit parameter down to another function.
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibklKngJkFFHDFxBEa-KL1_g9qL5jaG9CIOG5YKJ1-Xhsf8kRNrjLgLxrVZD-g6BgMHz9YKnLc2RvwviQ-sXnMUitCEJMh_TKqKmmNIPFBHnc75Zrm9rkhPamql1Y1w6WjqHLXF8vhgWc/s1600/hardHatArea.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibklKngJkFFHDFxBEa-KL1_g9qL5jaG9CIOG5YKJ1-Xhsf8kRNrjLgLxrVZD-g6BgMHz9YKnLc2RvwviQ-sXnMUitCEJMh_TKqKmmNIPFBHnc75Zrm9rkhPamql1Y1w6WjqHLXF8vhgWc/s1600/hardHatArea.png" /></a><br />
<br />
<image: area="" hard="" hat="">
<u>Secondly</u>, it is possible to have a typeclass with <b>multiple type parameters</b>. Context bound syntax is not
convenient for this case, so, first, let's go back to the original syntax: </image:><br />
<br />
<br />
<br />
<pre class="brush: scala">def produceQuizItem[A](component: A)(implicit qis: QuizItemSource[A]): Option[QuizItem] =
qis.produceQuizItem(component)
</pre>
<br />
Suppose a <code>Quiz</code> were passed in, the <code>quizAsSource</code> instance will be selected. You will
remember this looked like this:
<br />
<br />
<pre class="brush: scala">implicit object quizAsSource extends QuizItemSource[Quiz] {
def produceQuizItem(quiz: Quiz): Option[QuizItem]} =
… // implementation calls methods on quiz
}
</pre>
<br />
However, I simplified a bit here. The original implementation did not in fact return simple quiz items in the case of a <code>Quiz</code>. Whenever the <code>Quiz</code> retrieved a quiz item from a <code>QuizGroup</code>, it would add some information. Specifically it would generate some "wrong choices" for the question and return that to the user to allow a multiple-choice presentation of the quiz item. So the <code>QuizItem</code> would be wrapped in a <code>QuizItemViewWithChoices</code> object. The signature had to be:
<br />
<br />
<pre class="brush: scala"> def produceQuizItem(quiz: Quiz): Option[QuizItemViewWithChoices]} = …
</pre>
<br />
Now the problem is this does not conform to the typeclass definition. So let's massage the typeclass definition a bit by turning the return type into a new parameter, <code>C</code>:
<br />
<br />
<pre class="brush: scala">trait QuizItemSource[A, C] {
def produceQuizItem(component: A): Option[C]
}
</pre>
<br />
Then the typeclass instance for <code>Quiz</code> will look as desired:
<br />
<br />
<pre class="brush: scala">implicit object quizAsSource extends QuizItemSource[Quiz, QuizItemViewWithChoices] {
def produceQuizItem(quiz: Quiz): Option[QuizItemViewWithChoices]} =
… // implementation calls methods on quiz
}
</pre>
<br />
The intermediate function for accessing the typeclass becomes a bit more complex. Recall this
was previously:
<br />
<br />
<pre class="brush: scala"> def produceQuizItem[A](component: A)
(implicit qis: QuizItemSource[A]): Option[QuizItem] =
qis.produceQuizItem(component)
</pre>
<br />
The second type parameter for the return type is added as <code>C</code>:
<br />
<br />
<pre class="brush: scala"> def produceQuizItem[A, C](component: A)
(implicit qis: QuizItemSourceBase[A, C], c: C => QuizItem): Option[C] =
qis.produceQuizItem(component)
</pre>
<br />
<code>C</code> is constrained so that the return value must be compatible with <code>QuizItem</code>. To make
<code>QuizItemViewWithChoices</code> compatible with <code>QuizItem</code>, it is not necessary
to make it a subclass with lots of boilerplate as in Java. Instead an implicit conversion is defined next to the
<code>QuizItemViewWithChoices</code> class like this:
<br />
<br />
<pre class="brush: scala">object QuizItemViewWithChoices {
// Allow QuizItemViewWithChoices to stand in for QuizItem whenever necessary
implicit def qiView2qi(quizItemView: QuizItemViewWithChoices): QuizItem =
quizItemView.quizItem
}
</pre>
<br />
Now it all works!
<br />
<br />
That ends the description of the <code>QuizItemSource</code> typeclass. For more examples, see the other typeclasses in the <a href="https://github.com/oranda/libanius/">Libanius source code</a> such as <code>CustomFormat</code> and <code>ConstructWrongChoices</code>, which cover other concerns orthogonal to the model, or <a href="https://www.youtube.com/watch?v=sVMES4RZF-8">view this excellent video tutorial by Dan Rosen</a>.James McCabehttp://www.blogger.com/profile/07411857278648943057noreply@blogger.com1tag:blogger.com,1999:blog-5290563798585797720.post-75071055130625816232014-03-03T15:19:00.000-08:002014-03-04T08:45:51.042-08:00The Akka Event Bus on Android<b>Event buses</b> are useful for keeping components in a system loosely coupled, and we generally see them in two areas: distributed systems, and UIs. This post is about the use of an event bus in a UI, specifically Android.
<br />
<br />
There exist event bus libraries for Android like
<a href="https://github.com/square/otto">Otto</a> and
<a href="https://github.com/greenrobot/EventBus">greenrobot</a>.
There also exists an
<a href="http://doc.akka.io/docs/akka/snapshot/java/event-bus.html">Event Bus API for Akka</a>. What I haven't seen yet is the Akka Event Bus set up on Android. So, here it is!
<br />
<br />
First a word on the motivation. In the <a href="https://github.com/oranda/libanius-android">Libanius Android</a> application, I recently separated part of a screen (in Android's parlance, an Activity) into its own separate class. A problem was that, in response to certain events, the new class needed to add data back to the data structure in the main class for the Activity. My initial approach was to define a closure that could do the work of adding the data, and pass the closure as an argument to the new class (the sub-Activity). However, this felt a bit ad hoc, so I decided it was time to introduce a general event-handling mechanism into Libanius Android. Having helped create an event bus for the GWT UI framework before, I thought it would be interesting to see if the same idea could be applied in Android.
<br />
<br />
The Libanius Event Bus just extends Akka's ActorEventBus. It is nearly as simple as this:
<br />
<br />
<pre class="brush: scala">class LibaniusActorEventBus extends ActorEventBus with LookupClassification
with AppDependencyAccess {
type Event = EventBusMessageEvent
type Classifier = String
protected def classify(event: Event): Classifier = event.channel
protected def publish(event: Event, subscriber: Subscriber) {
subscriber ! event
}
}
</pre>
<br />
The diagram below shows what use we are actually making of the event bus:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiutva2W3vrVb44RxRGxMRj1xcKn9d2ib9iLbYkOQF_UEXZqfQwPx8aqfB6q1HQCWcd14oTOzfAUgmIS1KneJvFITxjB0S_ByW3Yy7XmqY0-eyDZMmORipEzy7QZhyphenhyphenQfd8z_3VfChAkMy4/s1600/eventbus.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiutva2W3vrVb44RxRGxMRj1xcKn9d2ib9iLbYkOQF_UEXZqfQwPx8aqfB6q1HQCWcd14oTOzfAUgmIS1KneJvFITxjB0S_ByW3Yy7XmqY0-eyDZMmORipEzy7QZhyphenhyphenQfd8z_3VfChAkMy4/s1600/eventbus.jpg" height="264" width="640" /></a></div>
<br />
<br />
The Android parts are OptionsScreen and DictionarySearch. In the UI, DictionarySearch is actually part of the OptionsScreen activity, but they communicate through the event bus. When the user clicks on a button from DictionarySearch to add a new word to the quiz, a NewQuizItemMessage is constructed. This event is just a wrapper for the quiz item and an identifying header:<br />
<br />
<pre class="brush: scala">case class NewQuizItemMessage(header: QuizGroupHeader, quizItem: QuizItem)
extends EventBusMessage
</pre>
<br />
The NewQuizItemMessage is published to the QUIZ_ITEM_CHANNEL by calling sendQuizItem() in LibaniusActorSystem. LibaniusActorSystem is a facade to LibaniusActorEventBus and other actors in the system not discussed here.
<br />
<br />
<pre class="brush: scala">object LibaniusActorSystem extends AppDependencyAccess {
val system: ActorSystem = ActorSystem("LibaniusActorSystem")
val appActorEventBus = new LibaniusActorEventBus
val QUIZ_ITEM_CHANNEL = "/data_objects/quiz_item"
def sendQuizItem(qgh: QuizGroupHeader, quizItem: QuizItem) {
val newQuizItemMessage = NewQuizItemMessage(qgh, quizItem)
appActorEventBus.publish(EventBusMessageEvent(QUIZ_ITEM_CHANNEL, newQuizItemMessage))
}
}
</pre>
<br />
That's the guts of it. There is some more code in OptionsScreen to create an actor to subscribe to the QUIZ_ITEM_CHANNEL, and actually add the QuizItem when it sees the event. For more detail, see the code in the <a href="https://github.com/oranda/libanius-android">Libanius Android</a> project.James McCabehttp://www.blogger.com/profile/07411857278648943057noreply@blogger.com0tag:blogger.com,1999:blog-5290563798585797720.post-64988778722521433922014-01-03T23:39:00.000-08:002014-01-03T23:39:30.404-08:00Extending Play with AngularJS: Nested MVCs in a WebappI finally got a bit of spare time to tighten up the JavaScript in <a href="https://github.com/oranda/libanius-play">Libanius-Play</a>, the web interface to my quiz application. For a long time, I was dissatisfied with the way data was being passed around from server to client and then back from client to server: there was too much repetition. The JavaScript library jQuery was of use to create AJAX callbacks, but still, any time there was a new parameter, it had to be added in too many places. <br />
<br />
<b>AngularJS</b> is another JavaScript library. Like jQuery it is geared towards asynchronous communication, but it provides an MVC framework. The Play Framework is also MVC, so the result is an MVC-within-MVC architecture (see diagram). It seems to work quite well.
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrqzLR9cHl_i9SDJJBfe00Q3tTYHFDFq1tACpKTsmNvy4Fe6IPs-sXGy2mbdwLmJdnapf5Xgmu_ZaX7EuYfEApp8FuI29ptKLROk_lTjrbEtUu-IE5MuoLREEFpyAJectVY_i25N83_Ps/s1600/libanius-nestedMVC-small.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="533" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrqzLR9cHl_i9SDJJBfe00Q3tTYHFDFq1tACpKTsmNvy4Fe6IPs-sXGy2mbdwLmJdnapf5Xgmu_ZaX7EuYfEApp8FuI29ptKLROk_lTjrbEtUu-IE5MuoLREEFpyAJectVY_i25N83_Ps/s1600/libanius-nestedMVC-small.jpg" width="720" /></a></div>
<br />
<br />
The AngularJS controller has a services layer to fetch data from the Scala back-end. Then it updates the page. The nice thing about it is that I don't have to update variables in the web page one by one: I can just <b>bind</b> variables in the web page to attributes in the model. This is achieved in the HTML using AngularJS tags like ng-model and the use of curly braces like this:
<br />
<br />
<pre class="brush: html">
<span id="prompt-word">{{ quizData.prompt }}</span>
</pre>
<br />
There are a number of variables like this scattered through the HTML, but they can all be updated at once in a single line of JavaScript if the Play backend returns a JSON structure. (The data structure can be defined as a case class in Scala on the server, and will be dynamically mirrored as a JavaScript structure on the client-side without the need to map attributes manually.) Here is a simplified version of what happens in the JavaScript controller after the server has processed a user response: basically it returns a new quiz item.
<br />
<br />
<pre class="brush: js">
services.processUserResponse($scope.quizData, response).then(function(freshQuizData) {
$scope.quizData = freshQuizData.data // updates values in the web page automagically
labels.setColors($scope.quizData) // extra UI work using a custom JavaScript module called labels
}
</pre>
<br />
In the first line, services.processUserResponse is a function in the services.js file which
sends user input to the server, and immediately returns a <b>promise</b> of new data. The promise has a handy <b>then</b> function which takes a callback argument, describing what happens when the data actually comes through, i.e. updating the view.<br />
<br />
All in all, a good experience with AngularJS.James McCabehttp://www.blogger.com/profile/07411857278648943057noreply@blogger.com0tag:blogger.com,1999:blog-5290563798585797720.post-2778292764956208692013-12-18T08:33:00.000-08:002013-12-18T08:33:14.786-08:00Private ConstructorsPrivate constructors: mostly you won't need them. Commonly in Scala we just define a class and let the constructor and parameters be exposed to the world:
<br />
<pre class="brush: scala">
case class SocialFeedCache(tweets: List[Tweet]) { ... }
</pre>
But suppose the tweets need to be organized for efficiency, and we don't want the cache to be constructed in an invalid state? Then we might have a factory method that performs the necessary initialization. In Java this would be a static method. In Scala we put it in a companion object:
<br />
<pre class="brush: scala">
object SocialFeedCache {
def apply(tweets: List[Tweet]): SocialFeedCache = {
// ... organize the tweets ...
SocialFeedCache(organizedTweets) // call the primary constructor
}
}
</pre>
So now we must always call the factory method rather than the primary constructor. How can we enforce this? How can we block anyone from calling the constructor directly? A first attempt might look like this:
<br />
<pre class="brush: scala">
private case class SocialFeedCache(tweets: List[Tweet]) { ... }
</pre>
But this doesn't just hide the constructor; it hides the whole class! In fact there will be a compile-time error, because the factory method in object SocialFeedCache is trying to expose a class that is now hidden.
<br /><br />
The correct syntax is actually:
<br />
<pre class="brush: scala">
case class SocialFeedCache private(tweets: List[Tweet]) { ... }
</pre>
<br />
And that's all there is to it. This is a useful technique whenever you have special initialization taking place in factory methods.James McCabehttp://www.blogger.com/profile/07411857278648943057noreply@blogger.com0tag:blogger.com,1999:blog-5290563798585797720.post-91597804573691614312013-08-17T08:05:00.000-07:002013-09-17T15:15:38.288-07:00Akka in Your PocketAlthough Akka is usually thought of in connection with massively scalable applications on the server side, it is a generally useful model for concurrency that can be deployed in an ordinary PC application or even a mobile app.<br />
<br />
My Nexus 4 phone has four cores. If I have some CPU-intensive processing in an Android app, I observe that I can speed it up nearly 300% by spreading load across four actors. This can mean the difference between one second response time and quarter-second response time, quite noticeable to the user. Therefore, along with regular Scala futures, Akka is a useful tool that Scala provides to Android developers to make the most of their users' computing resources.<br />
<br />
One tricky bit is getting the initial configuration correct with SBT. After some experimentation, I found that these Proguard settings, adapted from an early version of <a href="https://github.com/scottaj/android-scala-example">this project</a>, work. (ProGuard shrinks and optimizes an application, but when you include a library like Akka, it needs to be reminded to keep certain classes using "keep options.")<br />
<pre><span class="inner-pre" style="font-size: 8px;">
proguardOption in Android :=
"""
|-keep class com.typesafe.**
|-keep class akka.**
|-keep class scala.collection.immutable.StringLike {*;}
|-keepclasseswithmembers class * {public <init>(java.lang.String, akka.actor.ActorSystem$Settings, akka.event.EventStream, akka.actor.Scheduler, akka.actor.DynamicAccess);}
|-keepclasseswithmembers class * {public <init>(akka.actor.ExtendedActorSystem);}
|-keep class scala.collection.SeqLike {public protected *;}
|
|-keep public class * extends android.app.Application
|-keep public class * extends android.app.Service
|-keep public class * extends android.content.BroadcastReceiver
|-keep public class * extends android.content.ContentProvider
|-keep public class * extends android.view.View {public <init>(android.content.Context);
| public <init>(android.content.Context, android.util.AttributeSet); public <init>
| (android.content.Context, android.util.AttributeSet, int); public void set*(...);}
|-keepclasseswithmembers class * {public <init>(android.content.Context, android.util.AttributeSet);}
|-keepclasseswithmembers class * {public <init>(android.content.Context, android.util.AttributeSet, int);}
|-keepclassmembers class * extends android.content.Context {public void *(android.view.View); public void *(android.view.MenuItem);}
|-keepclassmembers class * implements android.os.Parcelable {static android.os.Parcelable$Creator CREATOR;}
|-keepclassmembers class **.R$* {public static <fields>;}
""".stripMargin
</span></pre>
<br />
Note: this has only been tested to work with Akka 2.1.0 and Scala 2.10.1. If you use other versions, you may have do some more tinkering/googling.James McCabehttp://www.blogger.com/profile/07411857278648943057noreply@blogger.com0tag:blogger.com,1999:blog-5290563798585797720.post-54693936688342484612013-08-03T07:04:00.001-07:002013-08-03T10:29:27.390-07:00Using 2.10 Futures in AndroidIn an Android app, it is usually necessary to run some tasks in the background so they don't block<br />
the UI. The standard way to do this in Java code is with an <b>AsyncTask</b>. Basically you fill out the body of an AsyncTask with two parts: what needs to happen in the background, and what needs to happen in the foreground once that is complete. The return value of the background part is passed as a parameter to the foreground part:<br />
<br />
Here is an example expressed in Scala, as always more concise than Java:<br />
<br />
<pre class="brush: scala"> new AsyncTask[Object, Object, String] {
override def doInBackground(args: Object*): String = calculateScore
override def onPostExecute(score: Int) { showScore(score) }
}.execute()
</pre>
<br />
From a Scala perspective, it might occur to you that this is a bit similar to what a <b>Future</b> does.
Futures are more general and concise, so in my opinion they are to be preferred. Indeed, <a href="http://engineering.linkedin.com/incubator/technology-behind-eatin-android-apps-scala-ios-apps-and-play-framework-web-services">following the example of Terry Lin</a>, I tried to get the identical functionality as the above working using the Future trait in Scala 2.10, and it worked fine. The new code has this structure:<br />
<br />
<pre class="brush: scala"> future {
calculateScore
} map {
runOnUiThread(new Runnable { override def run() { showScore(score) } }))
}</pre>
<br />
This code is a bit shorter. The one thing to remember is that after the score is returned, we have<br />
to remind Android to return control to the main thread -- that's what the runOnUiThread() is for.James McCabehttp://www.blogger.com/profile/07411857278648943057noreply@blogger.com0tag:blogger.com,1999:blog-5290563798585797720.post-76555687420019404602013-07-04T06:39:00.002-07:002013-08-03T06:24:12.040-07:00A Benefit of Implicit Conversions: Avoiding Long Boilerplate in WrappersIn OO practice we use wrapper classes a lot. So much in fact, that the Gang of Four identified several types of wrappers (differentiated more by purpose than structure): Adapters, Decorators, Facades, and Proxies. In this blog post I am going to concentrate on the <b>Proxy</b> design pattern, but the same technique of using <b>implicit def</b> to cut down code repetition can be applied for many different types of wrappers.<br />
<br />
As an example, I'm going to take a data structure from my open-source quiz app <a href="http://github.com/oranda/libanius">Libanius</a>. It's not necessary to go into the full data model of the app, but one of the central domain objects is called a <b>WordMappingValueSet</b>, which holds pairs of words and associated user data. This data is serialized and deserialized from a custom format in a simple file.<br />
<br />
The problem is that the deserialization, which involves parsing, is rather slow for tens of thousands of WordMappingValueSet's, especially on Android, the target platform. The app performs this slow initialization on startup, but we don't want to keep the user waiting before starting the quiz. How can the delay be avoided? The solution is to wrap the WordMappingValueSet in a <b>Lazy Load Proxy</b>, which gives the app the illusion that it immediately possesses all these WordMappingValueSet objects when in fact all it has is a collection of proxies that each contain the serialized String for a WordMappingValueSet. The expensive work of deserializing the WordMappingValueSet is done by the proxy only when each object is really needed.<br />
<br />
The question now is how to implement the Lazy Load Proxy. The classic object-oriented way<br />
to do it, as documented by the Gang of Four book, is to create an interface to the WordMappingValueSet that contains all the same methods, and then a WordMappingValueSetLazyProxy implementation of that interface. The client code instantiates the proxy object but it makes calls on the interface as if it were calling the real object. In this way, the illusion is preserved so not much has to change in the client code, but "under the hood" what is happening is that the proxy is deserializing the String and then forwarding each call to the real object.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXTnK5hKLQMdowUnzBMGEmwlRwe2EP2_JU0lD6G4nooKeZHxI3p6JQVfPwUmN98354bRAhzWCV_Hs2xfcDraMkj6Vgp4SBSHz_7H08hSqlA55UZ12s2nffWVV-_WKCgDkGicTMz3oGnZE/s926/WordMappingValueSetLazyProxy.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="204" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXTnK5hKLQMdowUnzBMGEmwlRwe2EP2_JU0lD6G4nooKeZHxI3p6JQVfPwUmN98354bRAhzWCV_Hs2xfcDraMkj6Vgp4SBSHz_7H08hSqlA55UZ12s2nffWVV-_WKCgDkGicTMz3oGnZE/s640/WordMappingValueSetLazyProxy.jpeg" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
In the above diagram, the function fromCustomFormat() does the work of deserialization. In Java this would be a static method; in Scala it is placed within the singleton object WordMappingValueSet. Inside WordMappingValueSetLazyProxy, the serialized String is stored, and fromCustomFormat() is called the first time one of the business methods is called, before the call is delegated to the method of the same name in WordMappingValueSet.<br />
<br />
This kind of wrapper is very common in the Java world. But there is a problem. There are a lot of methods in WordMappingValueSet. These all need to be duplicated in IWordMappingValueSet and WordMappingValueSetLazyProxy. It violates the principle of DRY (Don't Repeat Yourself). Implementing this in Scala, I would go from about a hundred lines of code to two hundred lines. (In Java I would be going from two hundred lines to four hundred lines.) Furthermore, every time I need to add a method to WordMappingValueSet, I now need to do the same to the other two types, so the lightweight flexibility of the data structure has been lost.<br />
<br />
In Scala, there is a better solution: use an implicit conversion. Firstly, as before, the WordMappingValueSet is wrapped inside a WordMappingValueSetLazyProxy:<br />
<br />
<pre class="brush: scala">case class WordMappingValueSetLazyProxy(valuesString: String) {
lazy val wmvs: WordMappingValueSet = WordMappingValueSet.fromCustomFormat(valuesString)
}</pre>
<br />
But that's all that's necessary for that class. We won't be defining the same methods all over again. Notice also that the <b>lazy</b> keyword, provided by Scala, means that fromCustomFormat() won't actually be called until some method is called on wmvs, i.e. until it is really needed.<br />
<br />
But the real trick to make this work is to define the implicit conversion:<br />
<br />
<pre class="brush: scala">object WordMappingValueSetLazyProxy {
implicit def proxy2wmvs(proxy: WordMappingValueSetLazyProxy): WordMappingValueSet = proxy.wmvs
}</pre>
<br />
This "converts" a WordMappingValueSetLazyProxy to a WordMappingValueSet. More specifically, it says to the compiler, "Whenever a method is called on WordMappingValueSetLazyProxy such that it doesn't seem to compile because no such method exists, try calling that method on WordMappingValueSet instead and see if that works." So we get the benefits of adding all the methods of WordMappingValueSet to WordMappingValueSetLazyProxy without having to actually do it by hand as in Java.<br />
<br />
There are other types of proxies, objects that give you the illusion that a resource has been acquired before it actually has. Perhaps the most common type is the network proxy. This is how EJBs are implemented, and perhaps the most frequent criticism of EJBs, since the early days of 2.x, is the large amount of code redundancy: for each domain object, you need to write a plethora of interfaces surrounding it. A lot of tools and techniques have emerged over the years to deal with the redundancy: code generation, AOP byte code weaving, annotations, etc. Could implicit conversions have avoided this? Some believe that implicits are too powerful and scary, but considering how much time has been spent over the years on frameworks involving reams of XML to work around the limitations of Java, it seems there might be a case for allowing this kind of power to be built into the language. Scala has this.James McCabehttp://www.blogger.com/profile/07411857278648943057noreply@blogger.com0tag:blogger.com,1999:blog-5290563798585797720.post-33668184908069527412013-06-01T14:39:00.000-07:002013-06-01T14:42:06.236-07:00Scala on the Rise in Germany<a href="http://de.linkedin.com/pub/kevin-shale/60/a00/652">Kevin Shale</a> was nice enough to invite me to a Scala meetup in Cologne on May 27. I had previously been to the Scala meeting in Berlin, but this was my first time with the <a href="https://www.xing.com/net/pri852682x/sugcb">Scala User Group in Cologne</a>. The talk covered <a href="http://typelevel.org/projects/scalaz/">scalaz</a>, the functional Scala library from Tony Morris. The discussion afterwards was broader and also interesting.<br />
<br />
I understand previous meetings were pretty small, but twice the number of people came this time. After taking London and Silicon Valley by storm, it's amazing how fast Scala is catching on in Germany too. I have already had recruiters from Munich and Berlin asking me about it.James McCabehttp://www.blogger.com/profile/07411857278648943057noreply@blogger.com0tag:blogger.com,1999:blog-5290563798585797720.post-22756511306612092302013-04-21T08:37:00.000-07:002013-04-21T08:37:13.218-07:00Functional Programming at CourseraMany professions are based on the idea that you do a big amount of learning upfront (university) and then only small updates will be necessary throughout the rest of your career. However, in software development technology advances so quickly that your whole career is really one of continuous learning. To further this
end, MOOCs (Massive Open Online Courses) have been become popular, especially over the past year or two. I don't see them as replacing traditional universities. However, they are a very useful resource, both for reminding yourself what you learnt in university and introducing you to new paradigms as they become popular
in your field of expertise.
<br />
<br />
A great example is the <a href="https://www.coursera.org/course/progfun">Functional Programming Principles in Scala</a> course at Coursera. Coursera is based at Stanford University, and this particular course is led by the Swiss professor Martin Odersky, inventor of the Scala language. I am now on the fourth week of the
seven-week course. Each week requires the student to watch a couple of hours of videos interspersed with quizzes, and then take some more serious exercises, which take several hours more and submitted online using the Scala tool sbt. A student can do this work at any time during the week, meaning there doesn't need to be any interruption to one's regular work schedule. At the end of the course, a certificate is available, as well as the total score on the exercises.<br />
<br />
In practise the majority of students drop out of online courses soon after starting them. However, I fully intend to pursue this one to the end, as I think the material is both excellently presented and highly relevant to the direction the software world is moving in. Furthermore, completing the exercises feels like solving puzzles in a way that I found quite addictive, and I shall be sorry when the course is complete and there are no more.<br />
<br />
For instance this week's lectures, on "Types and Pattern Matching," set the stage for exercises on creating Huffman trees, which was quite interesting. In this course, there is an emphasis on data structures and algorithms that reminds of my days studying Computer Science at university twenty years ago. In those days we learnt Scheme, a LISP variant and a classic functional programming language. These days there are more popular FP languages like Scala and Clojure, which are entering the mainstream for many good reasons including allowing access to the JVM and Java libraries.<br />
<br />
Anyway I highly recommend the course. This is the second run of it and I expect there will be another chance for people to take it in a few months.James McCabehttp://www.blogger.com/profile/07411857278648943057noreply@blogger.com0tag:blogger.com,1999:blog-5290563798585797720.post-50206311158462648162012-10-16T08:37:00.000-07:002013-08-03T06:40:03.591-07:00Load Testing with GatlingRecently, I've been working with an organization that had a huge increase in traffic due to mobile phones, and needed to determine how its legacy application would handle the millions of users flooding into its servers. They were accustomed to using JMeter, as it's a full-featured mature Java application for load testing and also useful for functional testing.<br />
<br />
However, it turns out that when the numbers get really big, you have to figure out how to deal with the load of JMeter itself! You end up <a href="http://jmeter.apache.org/usermanual/remote-test.html"> distributing the JMeter application across several servers</a>. Surely this shouldn't be necessary? Also, because it's so UI-oriented, the "scripts" you use for testing are big clunky XML files. You might prefer writing them in a modern programming language, giving you the option to integrate them into a general network management system.<br />
<br />
Enter <a href="http://gatling-tool.org/">Gatling</a>. This tool is based on the modern Akka framework, so it has much less overhead than JMeter. In fact, I was able to run it on a single machine all the time. Furthermore, since the scripts you write are in Scala, you can make it as easy or complex as the needs of the organization warrant. A simple script (or "simulation") looks like this:
<br />
<br />
<br />
<pre class="brush: scala">val myScenario = scenario("My Sequence Of Actions")
.feed(csv("feed_of_users.csv"))
.exec(http("Confirm Newspaper Subscription")
.get("/myapp/hasSubscription?userId=" + "${userId}")
.check(status.is(200))
.check(regex("""<response status='ok' """).exists))
</pre>
<br />
Run gatling.sh, and choose the script you want to run. After it finishes Gatling gives you a HTML page filled with graphs. Here is an example. This graph is actually showing a problem with the application under the test. The bumps in the red line mean that connections are being dropped from time to time.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWYOu892YOtSymbhyhAjiXsMCs-m1SvCyybREk1Cv025UZea86AuElF8PtUX6YT_vfYoRj6ETiYnq_deg6Fd_7sa1dS20Zr6Myd3yqhgofSnCP1vDA66Ud1bfUVma88mdeH2cweLMMPSU/s1600/gatlingGraph.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="197" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWYOu892YOtSymbhyhAjiXsMCs-m1SvCyybREk1Cv025UZea86AuElF8PtUX6YT_vfYoRj6ETiYnq_deg6Fd_7sa1dS20Zr6Myd3yqhgofSnCP1vDA66Ud1bfUVma88mdeH2cweLMMPSU/s320/gatlingGraph.png" width="500" /></a></div>
<br />
In summary, Gatling is not as fancy as JMeter yet, but I think this is the future of load testing.
<div>
</div>
James McCabehttp://www.blogger.com/profile/07411857278648943057noreply@blogger.com0tag:blogger.com,1999:blog-5290563798585797720.post-51398023051387189552012-09-30T09:55:00.003-07:002013-08-03T06:39:09.265-07:00Keywords as Clutter: Doing More with Less<body>
When I first started writing Scala, I found it very easy to code in a Java-like
way. Perhaps a little too easy in fact. The problem is that if you persist in writing
code in an imperative style, you miss a lot of the FP (functional) advantages that
Scala provides.<br />
<br />
For example, consider a simple <b>for loop</b> in Java, the unenhanced version where
you need the index:<br />
<br />
<pre class="brush:java">
for (int i = 1; i <= 10; i++) {
System.out.println("I like the number " + i);
}
</pre>
<br />
It is easy to transliterate this into Scala like this:<br />
<br />
<pre class="brush: scala">
for (i <- 1 to 10)
println("I like the number " + i)
</pre>
<br />
This is OK. However, a more idiomatic way is to use a <b>foreach</b> function:<br />
<br />
<pre class="brush: scala">
Range(1, 10).foreach(println("I like the number " + i))
</pre>
<br />
This is functional, rather than procedural. People will have various
reasons for preferring this style. But the interesting thing to me is that
foreach is not a Scala keyword like for. It is
just another function. Code is easier to compose when you get rid of the
magic of keywords. You can do more with less.
Another magic keyword is <b>if</b>. Again, Scala allows you to use it, and
sometimes you need it. However, if you think about it a little, very often
you find that using a Scala idiom is better. For example, it is tempting to
translate a Java null check to something like this:
<br />
<br />
<pre class="brush: scala">
if (functionResult.isDefined) // dubious style
doMyProcessing(functionResult.get)
</pre>
<br />
However this check on the functionResult Option can be a bit tedious.
The same thing can be written like this:<br />
<br />
<pre class="brush: scala">
functionResult.foreach { doMyProcessing(_) }
</pre>
<br />
This may look a bit odd if functionResult is only a single value. However, by
writing it in this way you have succeeded in moving the boring check down into
the functionResult Option implementation. This is especially useful when you have a lot of such "null checks" and wish to compose them. You don't want the code to become spaghetti-like with if statements.
<a href="http://blog.tmorris.net/scalaoption-cheat-sheet/">Here is a classic cheat sheet</a> about dealing with Option's in a functional way.
A different way of getting rid of if is to use <b>pattern matching</b>. Here is some Scala written in a Java-like way:
<br />
<br />
<pre class="brush: scala">
if (actorMessage == "toast_warm")
pop()
else if (actorMessage == "toast_cold")
toastSomeMore()
else
println("Who moved my toast?")
</pre>
<br />
Better is the Scala idiom:
<br />
<br />
<pre class="brush: scala">
actorMessage match {
case "toast_warm" => pop()
case "toast_cold" => toastSomeMore()
case _ => println("Who moved my toast?")
}
</pre>
<br />
Admittedly, match and case are still keywords. But in this case, it's probably worth it. Unlike the Java case label, Scala's case clause can take, not just an int or a String, but anything, including nested structures. That is the power of pattern matching.
I am collecting Scala idioms, so if you can think of other good examples, especially ones that do away with keywords, add a comment.
</body>James McCabehttp://www.blogger.com/profile/07411857278648943057noreply@blogger.com0