The Software Simpleton

Nomadic cattle rustler and inventor of the electric lasso

ember.js - Data Down, Actions up…better but Still Not Far Enough

The Ghost of Christmas Present

It is ironic that the thing that first drew me to ember is the thing that ended up causing the most excruciating pain. I am of course talking about two way data binding. An example of this is in this jsbin and in the code below which renders a list of todos. Two way databinding is employed to synchronise the isFinished boolean property of the todo and with the checked property of the input component:

1
2
3
4
5
6
{{#each todo in active}}
  <tr>
    <td>{{input type="checkbox" name="isFinished" checked=todo.isFinished}}</td>
    <td>{{todo.description}}</td>
  </tr>
{{/eachn}}

After years of DOM fiddling with jquery, this was instantly appealing as changes to the DOM are synchronised with the bound model without any extra code on from the developer. While this was great for breath taking ember demos, it led to a pandora’s box full of problems in produciton with earlier version of ember-data. In the first versions of ember-data, binding a model in this way led to the model’s isDirty flag getting set and if the model was not saved or the changes were not rolled back, the model remained in this dirty state and multiple errors would occurr as undefined methods were called on the current dirty state. I hacked around this pain by using the buffered proxy and other outlandish hacks but dealing with this is not something I ever want to experience again and still causes me to scream out in my sleep which is distressing for my partner.

While a lot of this is down to the fragility, rigidity and unforgiving nature of the early ember-data statemanager, the problem remains that it should not be the responsibility of the downstream input component to directly mutate the upstream model field and unexpected side effects occurred because of this. The problem is not particularly bad in this example but it becomes a big problem when you have a tree of nested components and shared data amongst these components, then it becomes difficult or sometimes impossible to reason about or track down what actually mutated the state. A component has its data changed in some other component that shares the same state has no idea where the mutation occurred. Ember needs a data flow story that is currently missing and computed properties and observables make the flow of data in an ember application very difficult to reason about and cascading partial updates from partially resolved asynchronous requests can get very stressful, I’ll elaborate more on this later but the ember core team seems to have reacted (pun intended) to some of the good things that react are doing and one way databinding will be the default as of ember 2.0.

The Ghost of Christams Future

Thankfully with the announcement of the ember 2.0 roadmap, the core team appear to recognise the dangers of this rampant mutation and they want to move a way from two way databinding, at least by default anyway. Databinding will be one way by default in ember 2.0 which greatly simplifies the interaction. The new paradigm that is being promoted goes by the meme data down and actions up. Child components will get their data from parent components, so when the parent component changes, the child component’s UI is updated. Data will flow down to the child components and the child components will notify their parent components of data changes by calling actions or pushing data up. The parent component should have the responsibility of persisting the changes and pushing those changes to the children. Now I fully support the data down approach, this makes absolute sense to me but I think that there are still problems with the data up approach, more on this later. But first let me now flesh out the data down, actions up approach with a refactor of the original example in this jsbin and below is the updated index.hbs template:

1
2
<h1>Todos</h1>
{{todo-list todos=model}}

It seems that the proxying controllers ArrayController and ObjectController are going to be depreceated and components will be the main currency so my todo-list becomes the top level component and below is its template:

1
2
3
4
5
6
7
8
9
<table>
  <tbody>
    {{#each todo in active}}
    <tr>
      {{todo-item todo=todo isFinishedChanged="isFinishedChangedHandler" tagName="td" }}
    </tr>
    {{/each}}
  </tbody>
</table>

The main thing to take away from the above code sample is that I am passing the parent component’s isFinishedChangedhandler to the child todo-item component. There is new syntax planned for actions but at the time of writing, I am using ember 1.10 and this remains the only way of passing handlers down. Below is the component’s code:

todo-list.js
1
2
3
4
5
6
7
8
9
App.TodoListComponent = Ember.Component.extend({
  actions: {
    isFinishedChangedHandler: function(todo) {
      todo.toggleProperty('isFinished');
      todo.save();
    }
  },
  active: Ember.computed.filterBy('todos', 'isFinished', false)
});

The todo-item child component that is rendered from the parent todo-list component has the following template:

1
<input type="checkbox" name="todo"/> {{todo.description}}

Gone is the input component and the bound isFinished property with the input's checked property.

Instead the isFinishedChangedHandler is called by the downstream component in the change event:

todo-item.js
1
2
3
4
5
6
App.TodoItemComponent = Ember.Component.extend({
  change: function(e) {
    this.sendAction('isFinishedChanged', this.get('todo'));
    return false;
  }
});

I’ve used sendAction but I expect this to change in ember 2.0. This is nice and neat in this simple of simplest examples. Once a checkbox is clicked the isFinishedChangedHandler of the parent controller is called and any state changes are synchronised down to the child components via the one way binding.

Ok great, I buy into this with this example but what if we have a more involved form with a number of text input fields, are we saying that we should write handlers for each input? This would very quickly get verbose, in react land, they have the ReactLink and I think in ember they are proposing something along the lines of this:

1
2
<input value={{mut firstName}}>
<input value={{mut lastName}}>

Data Flow in a Real world application

I see a lot of hype about html syntax for components, ok my templates look more like html but what I want from ember is a real dataflow story. React has covered this with flux and even better again with reflux.js where the data flow is unidirectional. In the real world application that I work on, I had a reasonably complex data table where all fields in the table needed to be editable in various ways and I was dealing with a large dataset, the table is infinitely scrolled and every sort operation etc. goes to the server. The sort macro etc. are useless to me in this scenario because all the objects are not in memory. There is a textbox on the top right which will filter the records further by whatever search term is entered. Below is how the table looks: Now where ember really shone was allowing me to add components to each table cell with a little bit of hacking that I blogged about here. This was great, every table cell is editable with a little help from html5’s contenteditable. The backing dataset of this table can have 1000s of records and there are multiple filters and sorting operations that all cause a requery of the external asynchronous data source. How does data come in and out of the the ember application? Currently ember lacks a data flow story which is disappointing in my opinion. How do I orchestrate the data coming from the server with its many filters and sort properties are selected from user input?

So the way I got round this was to use a combination a combination of query params, computed properties and observables. When the user selects a filter from the left menu in the screenshot above, a transitionTo action will occurr to a url with the filter added to the query params, something like addressbook/people/tagged?tag=191&company=856, this will cause execution to return to the route and select the new data. What do we do if a column is sorted or what do we do if some search text is entered to further filter the data? My solution is to maintain through a computed property the current parameters that will be sent to the server:

filterparams.js
1
2
3
4
5
6
7
8
9
10
11
12
13
filterParams: Ember.computed('filter', 'user', 'tag', 'company', function() {
    var company, contactimportjob, params, tag, user;
    params = {
      filter: this.get('filter'),
      page_size: this.get('pageSize')
    };

    params.company = this.get('company');
    params.user = this.get('user');
    params.tag = this.get('tag');

    return params;
  })

I do something similar with the sort which triggers a query param change that goes back to the route to fetch the new data. For the search I use an observer that listens for a bound property change:

search.js
1
2
3
4
5
6
7
8
9
10
11
12
  searchDidChange: Ember.observer("searchText", function() {
    var filterParams, params, searchText;
    if (this.get('filter') === null) {
      return;
    }
    searchText = this.get('searchText');
    filterParams = this.get('filterParams') || {};
    params = Ember.merge(filterParams, {
      like: searchText
    });
    return Ember.run.debounce(this, 'contentChanged', Ember.copy(params), 150);
  })

I could have used query params for this but I did not, I am also having to debounce as the input changes which is less than ideal. The solution I have come up with feels disjointed and I had to remind myself of how it all fits together when I write this. I am also dealing with partial updates happening from data coming in that is not fully resolved from the asynchronous source. You end up putting conditional logic or using inline observers that can really get quite annoying.

The problem with the above is that changes happen with responses to computed properties and observables that cause state mutation in a very disjointed fashion. With something like flux every user initiated change in the UI is communicated to the dispatcher via an action. The same goes for any data coming from a REST endpoint, where an action can be fired to the dispatcher. This is a top down approach that is consistent across the application and I’m not rolling out new ways of tackling problems for each scenario. This is a top down orchestrated workflow and not side effect driven like we have with ember and computed properties. I’ve really lost faith in the computed property and observable paradigm which causes no end of problems with asynchronous data. Ember seems to put more focus on quirky little macros that work on small datasets or nice html syntax for htmlbars which really does not make my life much easier. I can live with bind-attr for now, but I ended up forking and using a bastardised version of ember-data because I was sick of replacing one set of problems for another on every new release. Why is ember-data not the focus, I would love to know. Please feel free to shout me down for this last sentence as this really is just my opinion.

So to sum up, I feel that ember is at least making a move in the right direction with data down, action up but more needs to be done. One way databinding by default is also the right move. But we need a dataflow story and we need to forget about things like nicer html syntax or at least forget about the flashy demo stuff and concentrate on the problems that user’s have with real data on real applications. The fact that ember-data is still not 1.0 is just insane. This gives weight to my argument that ember often lets the more mundane important stuff take a back seat to the flashy nicer stuff, e.g. html syntax for components. Ember has some of the best developers out there working on it so they really can and should address these problems.

With all this said, the application I work on is in production and we have paying users but a lot of time was spent on issues with ember and ember-data rather than creating features. Ember is free and it has helped us in a lot of ways and I find it really useful for a lot of things like creating widgets. I can’t remember the last time I downloaded a jquery plugin. The router is also a thing of great beauty and something that really shines. Ember-cli is also a great tool also. I hope my criticism is constructive and not bitchy as I have a lot of respect for the developers.

React seems to be moving to fully embrace immutable data structures and I find this very exciting. Immutability opens up a new world of possiblities and makes things like change tracking, memoization and undo/redo functionality trivial.

Fast Prime Number Generation in Clojure

I’ve tried to stay away from building anything meaningful while I have been learning clojure and instead I have been solving problems on the various problem sites such as 4Clojure. I want to concentrate on learning the language as much as possible before getting bogged down with the practicalities that accompany building something. I came across the Hackerrank site which has a project euler contest. In this contest, I came across a problem which was one of the first problems I solved on 4clojure, the problem states:

By listing the first six prime numbers: 2,3,5,7,11 and 13, we can see that the 6th prime is 13.
What is the N’th prime number?

Input Format
First line contains T that denotes the number of test cases. This is followed by T lines, each containing an integer, N.

Output Format
Print the required answer for each test case.

Constraints
1≤T≤103
1≤N≤104

I had a quick look at my original 4clojure code that solved this problem that I wrote months ago and I was disgusted by what I saw, I was totally convinced that I could do much better and this problem would be toast in no time. I quickly rustled up the following solution:

second.clj
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
(ns scratch.core
  (require [clojure.string :as str :only (split-lines join)]))

(defn is-prime? [n]
  (empty? (filter #(= 0 (mod n  %)) (range 2 n))))

(defn nth-prime [n]
  (last (take n (filter #(is-prime? %) (iterate inc 2)))))

(defn print-primes [[x & xs]]
  (prn x)
  (if (seq xs)
    (recur xs)))

(let [input "1\n10001"
      ranks (rest (map read-string (str/split-lines input)))
      primes (map nth-prime ranks)]
  (time (print-primes primes))) ; Elapsed time: 103534.576 msecs

The input variable on line 15 simulates how the hackerrank checker gets input into the solution checking program which is in reality via STDIN but I have stubbed it here with one test case of 100001. The checking program checks the results by reading input from STDOUT which is the whole point of the print-primes function on line 10. Hacker rank supplies a number of unknown inputs or tests into your program which you actually never find out what they are but I am simply supplying 100001 as a significantly big number to test against.

I was feeling pretty chuffed with myself after writing this as the code is seriously concise and I was using some techniques that I perceived to be cool techniques such as an infinite lazy sequence in the form of (iterate inc 2) on line 8 which will iterate infinitely unless a constraint is put on the ever expanding sequence. The constraint in this instance is the take n expression on the same line which is further constrained to only prime numbers returned from the lazy sequence via the (filter #(is-prime? %)) expression.

Imagine my horror to discover that this was infinitely slower than my original 4clojure code, coming in at a diet bursting 103534.576 msecs. At this point, I started to have suspicions about the costs of lazy sequences.

I did some research to see if I could improve the algorithm and made a couple of common sense refactorings that are listed below:

third.clj
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
(ns scratch.core
  (require [clojure.string :as str :only (split-lines join)]))

(defn is-prime? [n]
  (or (= 2 n)
   (not-any? #(= 0 (mod n %)) (range 3 (inc (Math/sqrt n)) 2))))

(defn nth-prime [n]
  (last (take n (filter #(is-prime? %) (cons 2 (iterate (partial + 2) 3))))))

(defn print-primes [[x & xs]]
  (prn x)
  (if (seq xs)
    (recur xs)))

(let [input "1\n10001"
      ranks (rest (map read-string (str/split-lines input)))
      primes (map nth-prime ranks)]
(time (print-primes primes))) ; Elapsed time: 500.085 msecs

This executes in a significantly faster 500.085 msecs. I will quickly outline the refactorings:

The orignal is-prime? function looked like this:

isprime.clj
1
2
(defn is-prime? [n]
    (empty? (filter #(= 0 (mod n  %)) (range 2 n))))

The first problem with the above is that it will expand out the whole sequence and then check the count to verify that it does not contain any non-prime numbers. What I needed was a way to short circuit this function the first time a non-prime is encountered, as it turns out there is not-any? which will return false the first time it encounters a logical true.

The next problem with is-prime? is that the sequence that I am feeding into the filter function above, which is (range 2 n). The point of this range function is to ineffeciently pull out all the divisors of the number that we are checking is-prime? against. If one of the divisors is divisable by anything other than itself and one then it is a prime number. It turns out that there is a common algorithm to make this more efficient which states that you only need to test the numbers that are equal to or less than the square root of that number. This is because we can always find the divisors of a number in pairs. One member of the pair will be less than the square root and the other will be more.

For example, here are the pairs of divisors of 100:

1 and 100 (because 1 X 100)
2 and 50 (because 2 X 50)
4 and 25 (because 4 X 25)
5 and 20 (because 10 X 20)
10 and 10 (because 10 X 10)

Each number on the left is less than the square root, and each number on the right is more than the square root. Therefore when we look for the divisors of a number, it necessary to look only up to the square root.

The point is:

When we look for divisors of a number,
it is necessary to look only up to its square root.

So if we call (range 2 100) it expands out to (2 3 4 5 6 7 8 9 10 11 12 13 14.....etc...99), but we only need to do is expand out to the square root of 100.

We can refactor the range call to (range 2 (inc (Math/sqrt 100))) which will expand out to a much more efficient (2 3 4 5 6 7 8 9 10).

We can also take it further than that because we can also exclude any number that is greater than 2 and is even or divisable by 2 because it is quite obvioiusly a non prime number. We can refactor the range function to (range 3 (inc (Math/sqrt 100)) 2) which yields (3 5 7 9).

Below is the refactored is-prime? function:

refactoredisprime.clj
1
2
3
(defn is-prime? [n]
  (or (= 2 n)
   (not-any? #(= 0 (mod n %)) (cons 2 (range 3 (inc (Math/sqrt n)) 2)))))

My original nth-prime function also dealt with even numbers:

orig.clj
1
2
(defn nth-prime [n]
  (last (take n (filter #(is-prime? %) (iterate inc 2)))))

I refactored it nth-prime to only include 2 and the subsequent odd numbers:

refactored.clj
1
2
(defn nth-prime [n]
  (last (take n (filter #(is-prime? %) (cons 2 (iterate (partial + 2) 3))))))

I entered my code into hackerank and……the same thing happened, the first 2 tests passed and the remaining tests timed out. I was depressed by these results as I really thought that I could get on with my life afer this.

My next hack was to remove everything lazy and see what the time difference was. The algorithms stayed the same but everything lazy went:

nonlazy.clj
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
(ns scratch.core
  (require [clojure.string :as str :only (split-lines join)]))

(defn is-prime? [n]
  (if (= n 2)
    true
    (let [root (Math/floor (int (Math/sqrt n)))]
      (loop [i 3]
        (if (> i root) true
            (if (= 0 (mod n i)) false
                (recur (+ i 2))))))))

(defn n-primes [n]
  (loop [curr 3 acc [2]]
    (if (= (count acc) n)
      acc
      (recur (+ 2 curr) (if (is-prime? curr)
                          (conj acc curr)
                          acc)))))

(defn nth-prime [n]
  (last (n-primes n)))

(defn print-primes [[x & xs]]
  (prn x)
  (if (seq xs)
    (recur xs)))

(let [input "1\n10001"
      ranks (rest (map read-string (str/split-lines input)))
      primes (map nth-prime ranks)]
  (time (print-primes  primes)))  ; Elapsed time: 178.132

This came in at a much quicker 178.132 msecs and reafirrmed my suspicions that there is indeed a cost to lazyness. The above passed 3 of the 5 hackerrank tests so I guess this was progress but still no cigar.

Back to the drawing board, this was getting personal and I was not going to give up lightly.

My final throw of the dice involved me getting to grips with something that I was avoiding up unitil now, namely the Sieve of Eratosthenes. This is a reknowned alogrithm for marking multiples off any given limit starting with multiples of 2.

My first problem is that all the examples that I could find were using the sieve with numbers up to a certain number or a limit and not the nth number. I hit wikipidea again and found that there is an algorithm for estimating the limit that the nth number would equate to:

1
a(n) = n*(log n + log (log n))

Below is my code that uses the estimation technique (calc-limit) and the now infamous sieve of Eratosthenes:

solongandthanksforallthefish.clj
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
(ns scratch.core
  (require [clojure.string :as str :only (split-lines join)]))

(defn calc-limit [n]
  (let [log (Math/log n)
        loglog (Math/log log)
        logsum (+ log loglog)]
    (-> n (* logsum) int (+ 3))))

(defn primes [n]
  (let [root (-> n (Math/sqrt) inc int)
        sieve (boolean-array n true)]
    (loop [i 2]
      (when (< i (Math/sqrt n))
        (when (aget sieve i)
          (loop [j (* i 2)]
            (when (< j n)
              (aset sieve j false)
              (recur (+ j i)))))
        (recur (inc i))))
    (filter #(aget sieve %) (range 2 n))))

(defn nth-prime [n]
  (cond
    (= n 1) 2
    (= n 2) 3
    :else (last (take n (primes (calc-limit n))))))

(defn print-primes [[x & xs]]
  (prn x)
  (if (seq xs)
    (recur xs)))

(let [input "1\n10001"
      ranks (rest (map read-string (str/split-lines input)))
      primes (map nth-prime ranks)]
  (time (print-primes  primes)))  ; Elapsed time: 52.271 msecs

The sieve eleimates the non primes by:

  1. Crossing off multiples of x, only at and not 2 * x.
  2. Eliminate even numbers from the sieve.
  3. Elmintate further small numbers from the sieve.

This comes in at a very impressive52.271 msecs, I think this is fast but alas it was not fast enough for hackerrank and only 4 of the 5 tests passed.

At this point, I am giving up and moving on with my life. I now know things about prime numbers that I really have no business knowing.

Please feel free to offer a solution to this problem in the comments below. People have solved this hackerrank problem in the time allowed so it is possible but just not by me.

I am defeated on this and it does not feel good.

Connected Graph With Clojure

I’ve fallen in love with clojure and as part of my learning, I’ve set myself the goal of solving all the problems on the excellent 4clojure site. I was a painter and decorater by profession up until the age of 30 so I have no background in computer science or in depth mathematics but part of my fascination with 4clojure is that some of the harder problems introduce concepts that are completely foreign to me like graph theory. I get a kick out of digging into the theory behind the problem before trying to code my solution.

Problem 91 states:

Given a graph, determine whether the graph is connected. A connected graph is such that a path exists between any two given nodes.

-Your function must return true if the graph is connected and false otherwise.

-You will be given a set of tuples representing the edges of a graph. Each member of a tuple being a vertex/node in the graph.

-Each edge is undirected (can be traversed either direction).

You are then presented with a list of input and outputs that require a common function to balance both sides of the argument, e.g.

1
2
(= true (__ #{[:a :b] [:b :c] [:c :d] [:x :y]
              [:d :a] [:b :e] [:x :a]}))

The job of the user is to provide a function that will replace the __ characters to complete the expression.

The elements of the above set represents the edges of a graph. A graph edge can be thought of as a line or path that connects two graph nodes or vertices as they are known in graph speak. Each element of the set contains a vector with two vertices that have a path between them. We could visualise the above graph in the image below:

The problem stated that the function must return true if a path exists between any two nodes. You can see from the above diagram that each node is acccessible from each other.

Another of the expressions in the problem provides a set of edges that are not connected:

1
2
(= false (__ #{[:a :b] [:b :c] [:c :d]
               [:x :y] [:d :a] [:b :e]}))

You can visualise the unconnected graph below:

Another way to view the above graph is to say that it is made up of two components. We are dealing with undirected graphs which means that the edges can be travelled in either direction or more specifically, the edges do not point in any direction. A component of an undirected graph is a subgraph in which any two vertices are connected. In the above diagram, there are two components, the one on the right with the x and y vertices and the other component which is made up of the remaining vertices. The connected example can be thought of as one component.

Show Me the Code

If we are to use this concept of components as is outlined above, then we could say that if we can get the number of components in a graph and if that number is equal to one then the graph is connected. If we have more than one component then quite obviously the graph is disconnected.

After consulting the book of knowledge, a.k.a. wikipedia, I found this entry that gave me an outline of an algorithm that I could use to tell whether I had one connected component or not:

A search that begins at some particular vertex “`v“` will find the entire connected component containing “`v“` (and no more) before returning. To find all the connected components of a graph loop through its vertices , starting a new breadth first search whenever the loop reaches a vertex that has not already been included in a previously found connected component.

Armed with the above algorithm, I was able to construct the code to solve the problem:

connected.clj
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
(= true (
    (fn [g]
      (letfn [(connected? [g]
          (loop [q (conj [] (ffirst g)) visited #{}]
              (if (empty? q)
                  (let [rem (filter #(not (contains? visited %)) (flatten (for [e g] e)))]
                  (= empty? rem))
              (let [v1 (peek q)
                  edges (filter (partial some #{v1}) g)
                  vertices (filter (partial not= v1) (flatten edges))
                  unvisited (filter #(not (contains? visited %)) vertices)]
                (recur (into (rest q) unvisited) (into (conj visited v1) unvisited))))))]
     (connected? g)))
#{[:a :b] [:b :c] [:c :d]
              [:x :y] [:d :a] [:b :e] [:x :a]}))
  • On line 3, is a function connected? that does the actual work.
  • Line 4 starts a loop recur sequence with some intial bindings. I am selecting the first vertex of the graph which is the first element of the set and the first element from the first vector of that set. I am also initialising a map that will keep track of which vertices of the graph have been visited. In the above algorithm, this the particular vertex where the search begins. This vertex is added to a vector that we will use to process vertices that we have not yet visited.
  • line 5 checks that we have any unvisited vertices left to process.
  • If not, line 8 selects the first element from the unvisited vector and binds it to the v1 symbol.
  • line 9 selects any edges or connected vertices that contain v1 vertice that we are searching on.
  • line 10 selects the vertices from the edges that are not equal to v1.
  • line 11 filters out any vertices that we have already processessed before calling recur on line 12.
  • The recur on line 12 will add the unvisited nodes to the processing queue and also record what we have visited.
  • The code will continue until there are no vertices left to process at which point the (if (empty? q) statement on line 5 will be true.
  • If the graph is connected or only contains 1 component then the visited map that we have been building up will contain all the vertices of the graph or the flattened elements of the vectors that make up the set.

I’m still new to clojure and there is definitely better ways of doing this. One problem is that this function is in no way lazy.

I found this a hugely enjoyable problem to solve.

Clojurescript - Handling Mouse Events With core.async

I came across a situation recently where I needed to know the x and y co-ordinates of the mouse while the user was dragging over a canvas element. The mousemove event will give me the x and y co-ordinates via the offfsetX and offsetY properties of the event object but I only want to record these values when the user is dragging, i.e. when the mouse is down and the mousedown event has been triggered and we are receiving mousemove events. When the user stops dragging and the mouseup event is fired, I want to stop recording the co-ordinates.

The solution was to use the illusory synchronous appearance of core.async. In my opinion, the key to having user friendly asynchronous code is to have it read as synchronous as possible and I think that core.async outshines promises, generators and any other abstraction that I have encountered to read synchronously.

I am using the excellent om library to render the html and below I am declaring 3 channels in the IInitState lifecycle event that will be stored in the local state and can be retrieved in subsequent lifecycle events.

init.cljs
1
2
3
4
5
om/IInitState
(init-state [_]
  {:mouse-down (chan)
   :mouse-up (chan)
   :mouse-move (chan 1 (map (fn [e] (clj->js {:x (.-offsetX e) :y (.-offsetY e)}))))})

I am creating 3 channels to handle the mousedown, mouseup and mousemove events and storing them in the local state.

On line 4 I am creating a channel and also supplying a transducer to the channel that will transform each event object that is placed on the channel.

In the IDidMount lifecycle event that is triggered when the component is mounted onto the dom and is displayed below, I am retrieving the channels from the local state and creating event listeners for the mouse events that I am interested in, MOUSEDOWN, MOUSEUP and MOUSEMOVE. All these event handlers do is basically put! the event object onto the relevant channel:

didmount.cljs
1
2
3
4
5
6
7
8
9
10
11
12
13
14
om/IDidMount
(did-mount [_]
  (let [canvas (q ".tutorial")
        ctx (.getContext canvas "2d")
        mouse-down (om/get-state owner :mouse-down)
        mouse-up (om/get-state owner :mouse-up)
        mouse-move (om/get-state owner :mouse-move)]

    ; we use put! because we are not in a go block
    (listen canvas EventType.MOUSEDOWN #(put! mouse-down %))
    (listen canvas EventType.MOUSEMOVE #(put! mouse-move %))
    (listen canvas EventType.MOUSEUP #(put! mouse-up %))

    (set! (.-lineWidth ctx) 3)))

Now we come to the meat and two potatoes of the piece. Below is the IWillMount handler that is called before the component is mounted onto the dom:

willmount.cljs
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
om/IWillMount
(will-mount [_]
  (let [mouse-down (om/get-state owner :mouse-down)
        mouse-up (om/get-state owner :mouse-up)
        mouse-move (om/get-state owner :mouse-move)]

    (go-loop []
      (loop []
        (alt! [mouse-down] ([down]
                              (do (log "we hit mouse down and onto next loop.") nil))))
      (log "we won't reach here until mouse down.")
      (loop []
        (alt! [mouse-up] ([up] (do (log "we hit mouse up, final recur and back to previous loop.")  nil))
              [mouse-move] ([coords]
                              (do
                                (log coords)
                                (recur)))))
      (recur))))

As before, in lines 3-5 I retrieve the channels from local state.

On line 7 a go-loop is created. A go-loop is a convenience or shorthand for (go (loop ...)). The go-loop appears to create an infinite loop but actually it creates a statemachine that turns synchronous/blocking looking code into asynchronous non-blocking code. If there are no events then the go block is suspended. The go macro will expand out calls to <!, >!, alts! or in our case alt! into an asynchronous non-blocking statemachine.

On line 8 of the above, we use a combination of loop and alt! to suspend execution until a mousedown event is triggered. Execution will not reach the second inner loop on line 12 until the alt! expression on line 9 receives the mousedown event and returns nil on line 10.

alt! works like a sort of poor man’s pattern matching by dispatching execution to the right hand side s-expression of the left hand side channel that has received the event.

When the mousedown event is triggered, execution then proceeds to line 12 in a nice synchronous manner where a second loop will listen for mouseup or mousemove events via the alt! expression on line 13.

If mousemove events are received then we are simply logging the transformed event object that is transformed via the transducer that was passed to the channel initialisation in IInitState:

trans.cljs
1
:mouse-move (chan 1 (map (fn [e] (clj->js {:x (.-offsetX e) :y (.-offsetY e)}))))

While the stream of mousemove events are received on the mouse-move channel on line 14, the recur statement on line 17 will keep execution on this inner loop.

When the user stops dragging and the mouseup event is fired and received on the mouse-up channel on line 13. nil is returned which breaks execution out of this inner loop and execution continues onto the recur on line 18 which belongs to the go-loop and execution goes back to the first inner loop on line 8 where the code appears to block as it listens for the next mousedown event.

I found this a very interesting approach and the code appears beautifully synchronous which of course it is not. I also like the ability to supply a transducer to the core.async channel to transform the incoming input. This decouples things nicely.

Clojurescript - Using Transducers to Transform Native Javascript Arrays

I’ve been digging into clojurescript more and more but as I am still quite new to cljs, I find myself over zealously calling the clj->js and js->clj interop functions that transform javascript arrays to clojurescript vectors, maps, lists etc. and vice versa. I found this frustrating as I want to use the power of the clojurescript language and not have to drill down to javascript unless it absolutely necessary.

I’ve been writing a react wrapper component which has pretty much dictated that I need to be dealing with native javascript objects at all times and as such I am having to call these interop functions. An example of this is in the code below:

keys.cljs
1
2
3
4
5
6
7
(let [prevKeys (.keys js/Object (or prevChildMapping (js-obj)))
      nextKeys (.keys js/Object (or nextChildMapping (js-obj)))
      keysToEnter (clj->js (filter #(not (.hasOwnProperty prevChildMapping %)) nextKeys))
      keysToLeave (clj->js (filter #(not (.hasOwnProperty nextChildMapping %)) prevKeys))]

      (set! (.-keysToEnter this) keysToEnter)
      (set! (.-keysToLeave this) keysToLeave)))))

On lines 3 and 4, I am calling clj->js to transform a clojurescript PersistentVector into the javascript native array equivalent. What I really wanted was to call the clojurescript sequence functions map, reduce, filter etc. on native javascript objects. I asked if this was possible in the clojurescript irc and transducrs were put forward as a means of achieving the goal.

Transducers

I had heard of transducers in the clojure world without taking the trouble to see what all the fuss was about but I had no idea that they were available in clojurescript. I’m now going to give a brief introduction as to what transducers are but there is lots of good material out there that probably do a better job and Rich Hickey’s strangeloop introduction to them is a great start.

I always address a new concept by first of all determining what problem does the new concept solve and with tranducers the problem is one of decoupling. You are probably familiar with filter which returns all items in a collection that are true in terms of a predicate function:

filter.cljs
1
(filter odd? (range 0 10)) ;=> (1 3 5 7 9)

It should be noted that filter could be constructed using reduce.

filter-odd.cljs
1
2
3
4
5
6
7
(defn filter-odd
  [result input]
  (if (odd? input)
    (conj result input)
    result))

(reduce filter-odd [] (range 0 10))

The problem with the above is that we cannot replace conj on line 4 with another builder function like+. This problem holds true for all the pre-transducer sequence functions like map, filter etc. Transducers set out to abstract away operations like conj so that the creation of the resultant datastructure is decoupled from the map/filter logic.

conj and + are reducing functions in that they take a result and an input and return a new result. We could refactor our filter-odd function to a more generic filtering function that allows us to supply different predicates and reducing funtions by using higher order functions:

filtering.cljs
1
2
3
4
5
6
7
8
9
10
(defn filtering
  [predicate]
  (fn [reducing]
    (fn [result input]
      (if (predicate input)
        (reducing result input)
        result))))

(reduce ((filtering odd?) conj) [] (range 0 10)) ;=>[1 3 5 7 9]
(reduce ((filtering even?) +) 0 (range 0 10)) ; => 20

The above is not as scary as it looks and you can see on lines 9 and 10 that we are able to supply different reducing functions (conj and +). This is the problem that transducers set out to solve, the reducing function is now abstracted away so that the creation of the datastructure is decoupled from the sequence function (filter, map etc.) logic.

As of clojure 1.7.0 most of the core sequence functions (map, filter etc.) are gaining a new 1 argument arity that will return a transducer that, for example this call will return a transducer from filter:

odd.cljs
1
(filter odd?)

One of the new ways (but not the only way) to apply transducers is with the transduce function. The transduce function takes the following form:

transduce.cljs
1
transduce(xform, f, init, coll)

The above states that transduce will reduce a collection coll with the inital value init, applying a transformation xform to each value and applying the reducing function f.

We can now apply this to our previous example

xform.cljs
1
2
3
4
5
(def xform
  (filter odd?))

(transduce xform + 0 (range 0 10)) ;=> 25
(transduce xform conj [] (range 0 10)) ;=>  ;=>[1 3 5 7 9]

I hope it is obvious that (range 0 10) is coll and [] is the init, xform is the transducer function and + or conj are the reducing functions.

Meanwhile Back in Javascript land……

If we now shift back to our specific example, we can use a transducer to transform a native javascript array because a transducer is fully decoupled from input and output sources.

This is the current code that we want to refactor:

cljs.cljs
1
(clj->js (filter #(not (.hasOwnProperty prevChildMapping %)) nextKeys))

So the first question is what would the reducing function be when dealing with native arrays? The answer is the native array push push method which adds a new item to an array. My first ill thought out attempt at the above looked something like this:

transduce.cljs
1
(transduce (filter #(not (.hasOwnProperty prevChildMapping %))) (.-push #js[]) #js [] nextKeys)

This is completely wrong because I had not grasped what is required of the reducing funcion. A reducing function takes a result and an input and returns a new result e.g.

conj.cljs
1
2
(conj [1 2 3] 4) ;=> [1 2 3 4]
(+ 10 1) ;=> 11

The push function does not satisfy what is required as the push function actually returns the length of the array which is not what is expected. What was needed was someway of turning the push function into a function that behaved in a way that the transducer expected. The push function would need to return the result:

arr.cljs
1
(fn [arr x] (.push arr x) arr)

But as it turns out, this also does not work because a reducing function to transduce has to have 0, 1 and 2 arities and our reducing function only has 1.

As it turns out, both clojure and clojurescript provide a function called completing that takes a function and returns a function that is suitable for transducing by wrapping the reducing funtion and adding an extra arity that simply calls the identity function behind the scenes. Below is the completing function from the clojure.core source.

completing.cljs
1
2
3
4
5
6
7
(defn completing
  ([f] (completing f identity))
  ([f cf]
     (fn
       ([] (f))
       ([x] (cf x))
       ([x y] (f x y)))))

My final code ended up looking like this:

keys.cljs
1
(keysToEnter (transduce (filter #(not (.hasOwnProperty prevChildMapping %))) (completing (fn [arr x] (.push arr x) arr)) #js [] nextKeys)

The reducing function that uses the native javascript push function is wrapped in completing that makes it suitable for transducing.

I think I’ve ended up with more code than I started with and I also think that this is a poor example of transducers but I wanted to outline the mechanics involved in using transducers with native javascript arrays as I could find absolutely nothing on the google etc. so hopefully this will point somebody else in the right direction.

If I have got anyting wrong in this explanation then please leave a comment below.

Clojure - Installing a Local Jar With Leiningen

I ran into a situation earlier where I wanted to use the 0.8.0-beta2 version of om but it had not yet been pushed to clojars.

The answer was to install the jar in a directory local to the project I was working on.

Here are the steps I took to install the om jar locally:

Ember.js - Programmatically Bind From a Handlebars Helper

Warning: The examples use Ember 1.7.1

This is a quick update to my last post where I created a helper that bound data from an external json hash.

One problem I ran into was that if I was not creating links via the link-to helper then the properties were not bound. In line 11 of the gist below, I am returning a simple string that will render the unbound property wrapped in a span tag.

bad.js
1
2
3
4
5
6
7
8
9
10
11
12
Ember.Handlebars.registerBoundHelper('getProperty', function(context, property, options) {
  var defaults, prop;
  defaults = {
    className: "",
    context: "this",
    avatar: false
  };
  property = Ember.merge(defaults, property);
  prop = context.get(property.binding);
  if (!property.hasOwnProperty("route")) {
    return new Handlebars.SafeString("<span>" + prop + "</span>");
  }

The solution was to call out to the handlebars bind helper after updating the options hash on lines 2-5 below:

good.js
1
2
3
4
5
  if (!property.hasOwnProperty("route")) {
    options.contexts = [context];
    options.types = ["ID"];
    return Ember.Handlebars.helpers.bind.call(context, property.binding, options);
  }

Here is an updated jsbin with a full working example.

I am not sure if this is relevant for life after ember 1.7.1 but these techniques have worked well for me thus far.

Ember.js - Rendering Dynamic Content

Warning: The examples use Ember 1.7.1

I’m not going to go into great detail in this post as I think the code examples will be out of date post ember 1.7.1.

I recently had the problem of how to make a complex table reusable accross different datasets. As each dataset would contain objects with different fields, I would not be able to use the usual handlebars syntax of binding:

1
{{property}}

My solution was to create a json hash that is similar to the one in the gist below that specified which fields I was binding to and whether or not, I was going to render a simple text field, a link or for complex scenarios a component:

hash.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
App.IndexController = Ember.ArrayController.extend({
  columns: Ember.A([
   {
      heading: "Heading",
      binding: "name",
      route: "company"
    },
    {
      heading: "Address",
      binding: "address"
    },
    {
      heading: 'full',
      component: 'full-contact',
      bindings: ['name', 'address']
    }
  ])
});
  • The address structure on line 9 is the most basic as it just specifies a property to bind to.
  • The structure on line 4 contains a route property to signify that I want a link generated.
  • The structure on line 13 contains a component property that unsurprisingly will render a component. A component can also take an array of bindings on line 15 that will have the effect of calling the component like below:
1
{{full-contact name=name address=address}}

Now in my template, I just iterate over this columns collection and either call a handlebars helper that renders the text or link (line 9) or I call out to a different helper that will render the component (line 7)

If I am rendering a simple text or link, then I call the helper that is outlined below:

getproperty.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
Ember.Handlebars.registerBoundHelper('getProperty', function(context, property, options) {
  var args, defaults, id, prop;

  defaults = {
    className: "",
    context: "this",
    avatar: false
  };

  property = Ember.merge(defaults, property);
  prop = context.get(property.binding);

  if (Ember.isEmpty(prop)) {
    return "";
  }

  if (!property.hasOwnProperty("route")) {
    return new Handlebars.SafeString("<span>" + prop + "</span>");
  }

  args = Array.prototype.slice.call(arguments, 2);

  options.types = ["ID", "STRING", "ID"];
  options.contexts = [context, context, context];

  id = property.context ? "" + property.context + ".id" : "id";

  args.unshift(id);
  args.unshift(property.route);
  args.unshift(property.binding);

  return Ember.Handlebars.helpers["link-to"].apply(context, args);
});

If I am rendering a component then this helper is called:

component.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Ember.Handlebars.registerHelper('renderComponent', function(contextPath, propertyPath, options) {
  var context, helper, property;

  context = Ember.Handlebars.get(this, contextPath, options);
  property = Ember.Handlebars.get(this, propertyPath, options);

  helper = Ember.Handlebars.resolveHelper(options.data.view.container, property.component);

  options.contexts = [];
  options.types = [];

  property.bindings.forEach(function(binding) {
    options.hash[binding] = binding;
    options.hashTypes[binding] = "ID";
    options.hashContexts[binding] = context;
  });

  return helper.call(context, options);
});

Here is a working jsbin.

That is all I have to say on the matter but I would love to hear an alternative or better approach to the above.

Clojurescript - Om - Dynamic Components and Unique Keys

As I get older my tolerance for javascript seems to be getting worse so I’ve been employing clojurescript as a shield from the true horrors of javascript. I’ve been playing around wih om which acts as an interface to facebook’s react.

While developing with om, I kept getting the same javascript error after rendering a dynamic list like below:

If you have done any developing with om, then I am confident in saying that you will have come across this warning after rendering such a list:

Each child in an array should have a unique “key” prop. Check the renderComponent call using <tbody>. See http://fb.me/react-warning-keys for more information.

Here is the code I was using to render such a list:

old.cljs
1
2
3
4
5
6
7
8
9
10
11
12
(defn clips-view [{:keys [clips]} owner]
  (reify
    om/IRender
    (render [this]
      (html/html
        [:div.well
         [:table.table.table-bordered.table-hover.table-striped
          [:tbody
           (if (empty? clips)
             [:tr
              [:td.text-center {:colSpan "2"} "No Clips!"]]
             (om/build-all clip-view clips))]]]))))

Line 12 in the above is the villain of this piece as there is no way that I could find of passing a func that will set the key property that react uses to identify each dynamic child.

I have been trying to ignore this warning as it did not cause any code to stop executing but I kept seeing this question pop up on the irc channel and I could not find a good answer on the google.

It also turns out that not supplying a react key for each dynamic item could lead to some very unexpected behaviour.

After consulting the om docs I discovered that om/build can take a third :opts argument and one of the allowed keys is a :react-key which is one of the solutions to the problem.

Armed with this information, I refactored the above code to the following:

refactor.cljs
1
2
3
4
5
6
7
8
9
10
11
12
(defn clips-view [{:keys [clips]} owner]
  (reify
    om/IRender
    (render [this]
      (html/html
        [:div.well
         [:table.table.table-bordered.table-hover.table-striped
          [:tbody
           (if (empty? clips)
             [:tr
              [:td.text-center {:colSpan "2"} "No Clips!"]]
             (map-indexed #(om/build clip-view %2 {:react-key %1}) clips))]]]))))

The only change is on line 12:

1
(map-indexed #(om/build clip-view %2 {:react-key %1}) clips))

I have used the simplest case of an index for the :react-key but if I was rendering from a list where each item had a unique identifier then I would use the :key option to specify an element property that would be bound as the react key and would look something like this:

b.clj
1
(map #(om/build clip-view % {:key :id}) clips))

(thanks to Anna Pawlicka) for mentioning this in the comments below.

Or probably the best option of is to pass an :opts map containing a :key option to om/build-all:

a.clj
1
(om/build-all clip-view clips {:key :id})

Thanks to Daniel Szmulewicz for mentioning this on twitter.

I can now paste a link to this post when somebody asks how to get rid of this warning on the clojurescript irc channel.

Clojure - Lazy Sequences for the Masses

I have recently fallen head over heals in love with clojure and I’ve decided to run through the problems in the excellent 4Clojure site to ensure that I have a good understanding of the language before I even think about dealing with IO or building something meaningful. I’m currently on question 60 which has introduced the concept of lazy sequences and is displayed below:

r.clj
1
2
3
4
5
Write a function which behaves like reduce, but returns each intermediate value of the
reduction.  Your function must accept either two or three arguments, and the return
sequence must be lazy.

(= (take 5 (__ + (range))) [0 1 3 6 10])

What struck me as totally strange when I first read this problem is that an infinite list is being passed to the function in the form of (range). I had only previously seen range used with at least an end argument like below:

range.clj
1
2
user=> (range 10)
(0 1 2 3 4 5 6 7 8 9)

But if no start or end arguments are supplied then the default for end is infinity.

d.clj
1
2
3
;; default value of 'end' is infinity
user=> (range)
(0 1 2 3 4 5 6 7 8 9 10 ... 12770 12771 12772 12773 ... n

So why on earth would you want to supply an ever expanding range? The answer to this puzzle within a puzzle was to clarify what range actually returns which according to the docs is:

Returns a lazy seq of nums from start (inclusive) to end (exclusive), by step, where start defaults to 0, step to 1, and end to infinity.

The 4clojure problem at the start of this post also asks the user to return a lazy sequence for the solution. So what is a lazy sequence? As a newbie to clojure, I struggled to find good information of what an actual lazy sequence was and the docs left me with an unclear understanding of how I should use lazy-seq to construct a lazy sequence.

Lazy Sequences

Laziness in this context means that you can specify a computation that could theoretically take forever to complete, but you can evalutate as much or as little of it as you need. A very simple example of this is below:

take.clj
1
2
user=> (take 10 (range))
(0 1 2 3 4 5 6 7 8 9)

In the above example, range is being called with no end argument which according to the docs will mean that end defaults to infinity but if we type the expression into the repl, a sequence with the first 10 elements is immediately evaluated so it would appear that take 10 is not waiting for range to reach infinity before grabbing the first 10 elements.

A closer look at what type of sequence is returned starts to give the game away:

class.clj
1
2
user=> (class (take 10 (range)))
clojure.lang.LazySeq

As range with no end argument defaults to infinity, the only way that this could possibly work is if range returns a lazy sequence or one that realizes its elements as requested. The original 4Clojure problem stated that the sequence returned from the solution must be lazy in order to work with an infinite range:

t.clj
1
(= (take 5 (__ + (range))) [0 1 3 6 10])

After a quick google to determine how I can return a lazy sequence, I came across the clojure docs for the lazy-seq function but I did not fully grasp what the explanation meant:

Takes a body of expressions that returns an ISeq or nil, and yields
a Seqable object that will invoke the body only the first time seq
is called, and will cache the result and return it on all subsequent
seq calls. See also - realized?

In my haste to complete the problem, I simply thought that all I had to do was wrap an existing function in a lazy-seq call and everything would just magically work. Here is my original attempt at solving the question:

first.cljs
1
2
3
4
5
(= (take 5 ((fn [func se]
    (lazy-seq
        (reduce (fn [acc item]
                      (conj acc (func (last acc) item)))
                        [(first se)] (rest se)))) +  (range))) [0 1 3 6 10])

All I have done is wrap a call to reduce on line 3 in a call to lazy-seq on line 2 which of course resulted in a stackoverflow because reduce does not return a lazy sequence and instead will execute until the computation is complete which in this case is never because reduce is being called on an infinite list. I needed to go back to the drawing board. What I needed to do was create a sequence that is realized as needed.

After a bit more digging it became apparent that lazy sequences are not really sequences that simply return a list of elements but are more comparable to iterators in other languages such as java that conform to an a sequence api. With a lazy sequence it is up to the client or the calling function to decide how many elements to consume. A lazy function is just a data structure that waits until you ask it for a value. So how does one return a lazy sequence?

The Simplest Example Imaginable

Below is the simplest example that I could think of:

simple.clj
1
2
3
4
(defn add-one [se]
  (lazy-seq (cons (inc (first se)) (add_one (rest se)))))

(take 5 (add-one (range))) ;;;;;;=>(1 2 3 4 5)
  • Line 1 defines a function named add-one that accepts a sequence that could be infinite.
  • Line 2 is where the action happens, the lazy-seq macro is employed to create a lazy sequence whose body is an expression. The body of this lazy sequence uses the cons function which takes an element and a sequence and returns a new sequence by prepending the element to the sequence. The element in this case is the result of incrementing the first element of the list with the expression:
inc.clj
1
(inc (first se))

The sequence that the element is prepended to is the result of recursively calling the add-one function again. If a lazy sequence was not returned from add-one then add-one would be called over and over again until a stackoverflow exception was thrown. The whole point of the lazy-seq macro is to circumvent this and ensure that the function will only be accessed as the elements are requested. A sequence that is returned from lazy-seq contains the requested element and a pointer or a function to get the next element.

Another mistake I made in my original misunderstanding was that cons was returning a lazy sequence or was somehow involved in returning a lazy sequence. cons does not return a lazy sequence (the thing that does that is lazy-seq), but it is a good tool to use as part of buidling a lazy sequence.

The 4clojure Solution

Before I show my solution, I will remind you of the original 4Clojure problem.

red.clj
1
2
3
4
Write a function which behaves like reduce, but returns each intermediate value of the reduction.
Your function must accept either two or three arguments, and the return sequence must be lazy.

(= (take 5 (__ + (range))) [0 1 3 6 10])

The question is really asking the user to recreate clojure’s core reductions function. Below is my solution that took me quite a while to get to but I am very pleased with.

res.clj
1
2
3
4
5
6
7
8
9
10
11
12
(= (take 5 (
  (fn my-reduct
    ([func coll]
       (my-reduct func (first coll) (rest coll)))

    ([func firstArg coll]
      (letfn [(reduct [f init se]
                (lazy-seq (when-not (empty? se)
                            (let [res (f init (first se))]
                              (cons res (reduct f res (rest se)))))))]
        (lazy-seq (cons firstArg (reduct func firstArg coll))))))
                    + (range))) [0 1 3 6 10])

The important things to note are that an infinite range is passed into the named anonymous function my-reduct on line 12 but becase a lazy sequence is returned on line 8 and line 11, the resulting sequence will only be returned as requested. Because the take function is used on line line 1 to request the lazy sequence with an arguement of 5, the function will only be called 5 times.

I found lazy sequences very difficult to wrap my head around and part of that process was to write this post.

If I have inaccuately described anything or I can improve this explanation then please leave a comment below.