Tech behind Tech

In this part we will start implementing Add,  View and View all functionalities.

The services we are going to build are

View All Addresses – GET – http://localhost:8080/addresses
View single address – GET – http://localhost:8080/addresses/:id
Add Address – POST – http://localhost:8080/addresses

Interactive Development using slime

I mentioned in last part that after every change to our code we need to restart our server for our changes to be reflected. This is a pain and against clojure (lisp) philosophy . It would be great if we could eval our modified buffers in emacs and those changes reflected immediately in our jetty server. Thankfully there is very easy way to do this.

In core.clj we are starting jetty adapter. We can start this jetty server in background using future and reload the namespace that we changed. This way we can do interactive development without restarting our server.

PS: Make sure you have emacs setup with slime. I feel emacs is the best IDE for clojure. Again this is my opinion. There are lot of information on how to do this.

I talked about jetty adapter. It is time for us to look at some what little deeper in Compojure.

Compojure

Compojure is based on a library  called Ring. In fact most of clojure web frameworks are based on Ring. So to understand Compojure, it is important to understand Ring.

Ring has three components.

1) Handlers:

Handlers are main functions that  process a request. We define handlers using defroutes macro.

2) Middleware:

Middleware are functions that could be chained together to process a request. Middleware functions can take any number of arguments, but the spec stats that first argument should be an handler and function should return an handler. An example for middleware is logging all requests that comes to your webserver.  Ring and compojure comes with some standard middleware. We will see in next part how to create our own middleware.

3) Adapters:

Adapters are functions could adapt our handler to a web server. We are using jetty adapter to tie our handler to jetty server.

Refactor code to make it easy for interactive development

Edit core.clj

(ns address_book.core
  (:use [compojure.core]
        [ring.adapter.jetty])
  (:require [compojure.route :as route]))

(defroutes example
  (GET "/" [] "<h1>My Address Book!</h1>")
  (route/files "/" {:root "public"})
  (route/not-found "Page not found"))

(future (run-jetty (var address-book) {:port 8080}))

1) Add swank-clojure to our project

swank-clojure comes with lein plugin which will allow us to start a swank server and from emacs you can connect using slime. There are lot of documentation about swank-clojure and slime. Let me know if you need more information. If I see enough interest, I could blog about it or at least point to some good resources.

Edit project.clj

(defproject address_book "1.0.0-SNAPSHOT"
  :description "Address Book"
  :dependencies [[org.clojure/clojure "1.2.0"]
                 [org.clojure/clojure-contrib "1.2.0"]
                 [compojure "0.4.1"]
                 [ring/ring-jetty-adapter "0.2.3"]]
  :dev-dependencies [[swank-clojure "1.2.1"]])

and run

lein deps

Now you should be able to start swank server using

lein swank

2) Break core.clj into different namespaces

Current core.clj is doing two things. Setting up routes and starting jetty server. Lets break it into web_server.clj and routes.clj

rm src/address_book/core.clj

create src/address_book/routes.clj

(ns address_book.routes
  (:use [compojure.core])
  (:require [compojure.route :as route]))

(defroutes address-book
  (GET "/" [] "<h1>My Address Book!</h1>")
  (route/files "/" {:root "public"})
  (route/not-found "Page not found"))

create src/address_book/webserver.clj

(ns address_book.webserver
  (:use [compojure.core]
        [ring.adapter.jetty]
        [address_book.routes :as routes]))

(future (run-jetty (var address-book) {:port 8080}))

3) At last lets create our Address namespace

In this code, I am going to use atoms for persistence. In future I will probably show how we can persist in mysql db using clj-records.

create src/address_book/address.clj

(ns address-book.address
  (:use [address-book.utils number])
  (:refer-clojure :exclude (find create)))

(def STORE (atom {:1 {:id :1 :name "Siva Jagadeesan" :street1 "88 7th" :street2 "#203" :city "Cupertino" :country "USA" :zipcode 98802}}))

(defn create [attrs]
  (let [id (random-number)
        new-attrs (merge {:id id} attrs)]
    (swap! STORE merge {id new-attrs})
    new-attrs))

(defn find-all []
  (vals @STORE))

(defn find [id]
  ((to-keyword id) @STORE))

(defn update [id attrs]
  (let [updated-attrs (merge (find id) attrs)]
    (swap! STORE assoc id updated-attrs)
    updated-attrs))

(defn delete [id]
  (let [old-attrs (find id)]
    (swap! STORE dissoc id)
    old-attrs))

Create utils/number.clj

(ns address-book.utils.number
  (:import [java.util Date]))

(defn to-keyword [num]
  (if-not (keyword? num)
    (keyword (str num))
    num))

(defn random-number []
  (to-keyword (.getTime (Date.))))

PS: I am not writing tests to keep this blog short. But I would advice everyone to write tests.

4)   Lets update routes for add, view and view all functionalities

</span>
<pre>(ns address_book.routes
  (:use [compojure.core])
  (:require [address-book.address :as address]
            [compojure.route :as route]
            [clj-json.core :as json]))

(defn json-response [data & [status]]
  {:status (or status 200)
   :headers {"Content-Type" "application/json"}
   :body (json/generate-string data)})

(defroutes handler
  (GET "/addresses" [] (json-response (address/find-all)))
  (GET "/addresses/:id" [id] (json-response (address/find id)))
  (POST "/addresses" {params :params}  (json-response (address/create params)))

  (route/files "/" {:root "public"})
  (route/not-found "Page not found"))

(def address-book
     handler)

Compojure comes with GET, POST, PUT, DELETE, HEAD and ANY macro to define routes. These macros are self explanatory. Currently we have used GET and POST to define our routes.

Parsing Parameters

Compojure binds request parameters to params.

To get all parameters from request you can destructure map like we did in

POST "/addresses" {params :params}  (json-response (address/create params)))

To get particular parameters from request you can use Compojure sugar syntax like we did in
[sourceode](GET “/addresses/:id” [id] (json-response (address/find id)))[/sourcecode]

5)  Lets build our front end
PS: front end code is a hack. Please don’t follow these codes as a good practice.

Edit public/index.html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
        "http://www.w3.org/TR/html4/strict.dtd">
<html lang="en">
<head>
	<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
	<title>My Address Book</title>
        <link href="/css/address.css" media="screen" rel="stylesheet" type="text/css" />
        <script src="/js/jquery-1.4.2.min.js" type="text/javascript"></script>
        <script src="/js/jquery-ui-1.8.1.custom.min.js" type="text/javascript"></script>
        <script src="/js/jquery.form.js" type="text/javascript"></script>
        <script src="/js/address_book.js" type="text/javascript"></script>
        <script src="/js/address_list.js" type="text/javascript"></script>
</head>
<body>
<div id="wrap">
	<div id="header"><h1>My Address Book</h1></div>

	<div id="main">
          <table id="address">
            <tr><td>Name</td><td id="name"/></tr>
            <tr><td>Street1</td><td id="street1"/></tr>
            <tr><td>Street2</td><td id="street2"/></tr>
            <tr><td>City</td><td id="city"/></tr>
            <tr><td>Country</td><td id="country"/></tr>
            <tr><td>ZipCode</td><td id="zipcode"/></tr>
          </table>
	  <table id="address-list" border="1" width="100%" rules="rows" align="center">
            <tr>
              <th>Name</th>
              <th>Action</th>
            </tr>
            <tr>
              <td >row 1, cell 1</td>
              <td align="center">row 1, cell 2</td>
            </tr>
          </table>
	</div>
	<div id="sidebar">
          <form id="address-form" class="formular" method="post" action="/addresses">
	    <fieldset class="login">
	      <legend>New Address</legend>
	      <div>
		<label for="name">Name</label> <input type="text" id="name" name="name">
	      </div>
              <div>
		<label for="street1">Street1</label> <input type="text" id="street1" name="street1">
	      </div>
               <div>
		<label for="street2">Street2</label> <input type="text" id="street2" name="street2">
	      </div>
              <div>
		<label for="city">City</label> <input type="text" id="city" name="city">
	      </div>
              <div>
		<label for="country">Country</label> <input type="text" id="country" name="country">
	      </div>
               <div>
		<label for="zipcode">ZipCode</label> <input type="text" id="zipcode" name="zipcode">
	      </div>

              <input class="submit" type="submit" value="Create"/>
	    </fieldset>
          </form>
	</div>
	<div id="footer">
	  <p><a href="TechbehindTech.com">TechBehindTech</a> -- Siva Jagadeesan</p>
	</div>
</div>
</body>
</html>

Create public/css/address.css

body,html {
    margin:0;
    padding:0;
    color:#000;
    background:#a7a09a;
}
#wrap {
    width:970px;
    margin:0 auto;
    background: #fffeff;
}
#header {
    padding:5px 10px;
    background: #054477;
	text-align: right;
	color: #fffeff;
}
h1 {
    margin:0;
}
#main {
    float:left;
    width:580px;
    padding:10px;
	background-color: #fffeff;
}
h2 {
    margin:0 0 1em;
	background-color: #fffeff;
}
#sidebar {
    float:right;
    width:350px;
    padding:10px;
    background-color: #fffeff;
}
#footer {
    clear:both;
    padding:5px 10px;
    background: #fed47f;
}
#footer p {
    margin:0;
}
* html #footer {
    height:1px;
}

form * {margin:0;padding:0;} /* Standard margin and padding reset, normally done on the body */

legend {
	color:#000; /* IE styles legends with blue text by default */
	*margin-left:-7px;
	font-weight: bold;
	font-size: 20px;
}
fieldset {
	border:1px solid #dedede; /* Default fieldset borders vary cross browser, so make them the same */
}
fieldset div {
	overflow:hidden; /* Contain the floating elements */
	display:inline-block;
	padding: 10px;
}
fieldset div {display:block;} /* Reset element back to block leaving layout in ie */
label {
	float:left; /* Take out of flow so the input starts at the same height */
	width:8em; /* Set a width so the inputs line up */
}

Create public/js/address_book.js

$(document).ready(function() {
    $("#address").hide();
    $("#address-list").showAddressList();

    $(".address-link").live("click",function(e){
        $.getJSON($(this).attr("href"), function(json) {
            $("#address").showAddress(json);
        });
        e.preventDefault();
    });

    $('#address-form').submit(function(event){
        event.preventDefault();
        var $this = $(this);
        var url = $this.attr('action');
        var dataToSend = $this.serialize();
        var callback = function(data){
            $("#address-list").addAddress(data);
        };
        var options = {
            success:   callback,
            url: url,
            type:     "POST",
            dataType: "json",
            clearForm: true
        };
        $(this).ajaxSubmit(options);
    });

});

Create public/js/address_list.js

function action_links(data){
    var link = "<a href=\"\">edit</a> ";
    link += " | <a href=\"\">delete</a>";
    return "edit | delete";
}

$.fn.showAddressList = function(){
    return this.each(function(){
        var that = this;
        $.getJSON("addresses", function(json) {
            $(that).html(" <tr> <th>Name</th> <th>Action</th> </tr>");
            $.each(json,function(i,data) {
                $(that).append("<tr id=\"address-"+ data.id +" \"><td><a class=\"address-link\" href=\"addresses/" + data.id +"\">" + data.name + "</a></td><td align='center'>" + action_links(data) + "</td></tr>");
            });
        });
    });
};

$.fn.addAddress = function(json){
    var data = json;
    var that = this;
    return this.each(function(){
        var that = this;
        $(that).append("<tr id=\"address-"+ data.id +" \"><td><a class=\"address-link\" href=\"addresses/" + data.id +"\">" + data.name + "</a></td><td align='center'>" + action_links(data) + "</td></tr>");
    });
};

$.fn.showAddress = function(json){
    var data = json;
    var that = this;
    return this.each(function(){
        $(that).slideDown('slow');
        $(that).find("#name").html(data.name);
        $(that).find("#street1").html(data.street1);
        $(that).find("#street2").html(data.street2);
        $(that).find("#city").html(data.city);
        $(that).find("#country").html(data.country);
        $(that).find("#zipcode").html(data.zipcode);
    });
};

Download jquery-1.4.2.min.js, jquery-ui-1.8.1.custom.min.js and jquery.form.js to public/js folder.

6)  Start the web server

[/sourcecode]lein repl src/address_book/webserver.clj [/sourceode]

Wow that was lot of stuff. We will look more into writing our own middlewares in next part.

Source code is now available at github. Created branches for each part.

Part 5 is posted.

In part2 we saw how to setup a skeleton project with Compojure. In this part we will see how to add static files to a Compojure project.

For our application, we will use JQuery as front end and Clojure as a backend. I will concentrate more on Compojure.

1) Create folders for static files.

mkdir public
mkdir public/css
mkdir public/js

It is better to have static files under a seperate directory that way you could use your webserver to serve these files. For now we will use Compojure to serve these files.

2) Create index.html under public folder

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
	<head>
		<meta http-equiv="content-type" content="text/html; charset=utf-8" />
		<title>My Address book</title>
		<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
	</head>

  <body id="main">
    <p>
      My Address Book
    </p>
  <body>
</html>

3) Start your server

lein repl src/address_book/core.clj

4) Access index.html

Go to http://localhost:8080/index.html . You will get “Page not found” instead of “My Address Book”. This is happening because we have not told compojure about our static files folder.

Route helper functions in Compojure

In Compojure we have two route helper functions. We have seen one already in core.clj file.

1) not-found :

This function is used to capture all other routes that are not defined in Compojure. As we still have not defined index.html we got “Page not found”.

2) files :

This function is used to inform Compojure where static files are present.

Lets change our core.clj to handle static files

(ns address_book.core
  (:use [compojure.core]
        [ring.adapter.jetty])
  (:require [compojure.route :as route]))

(defroutes example
  (GET "/" [] "<h1>My Address Book!</h1>")
  (route/files "/" {:root "public"})
  (route/not-found "Page not found"))

(run-jetty example {:port 8080})

Restart your server (there is a way to avoid restarting using futures. I will show that in a later blog) and access http://localhost:8080/index.html . Now you should see “My Address Book”.

In this part we saw how we can handle static files. In next part we will implement Add , View and View All functionalities.

PS: Using Hiccup you can create HTML and css in clojure. I will try to cover this in future blogs.

Source code is now available at github . Created branches for each part.

Checkout PART4 …

In part 1 I introduced the address book application that we are going to build.

In this part we will setup our skeleton project with compojure.

1) Install Leiningen

http://github.com/technomancy/leiningen/blob/master/README.md

2) Create a new project using Leiningen

lein new address_book

3) Add Compojure to our project:

a) Edit project.clj

(defproject address_book "1.0.0-SNAPSHOT"
  :description "Address Book"
  :dependencies [[org.clojure/clojure "1.1.0"]
                 [org.clojure/clojure-contrib "1.1.0"]
                 [compojure "0.4.1"]
                 [ring/ring-jetty-adapter "0.2.3"]])

b) Install dependencies

lein deps

This should install all dependencies of compojure

c) Test whether our setup is working

Edit src/address_book/core.clj

(ns address_book.core
  (:use [compojure.core]
           [ring.adapter.jetty])
  (:require [compojure.route :as route]))

(defroutes example
  (GET "/" [] "My Address Book!")
  (route/not-found "Page not found"))

(run-jetty example {:port 8080})

Run the server

lein repl src/address_book/core.clj

Goto http://localhost:8080 and you should see “My Address Book!”

In next part we will start implementing our functionalities.

In this part, I am going to talk about what application we are going to implement.

Lets build a simple address book web application using compojure. This will be a RESTFUL application. As we go through this example we will see some interesting aspects of compojure.

Functionalities

• Add / Edit / Delete Address book
• View All Addresses
• View a single Address

Domain

Address

• Name
• Street1
• Street2
• City
• Country
• ZipCode

REST interfaces we will implement

• View All Addresses – GET – http://localhost:8080/addresses
• View single address – GET – http://localhost:8080/addresses/:id
• Add Address - POST – http://localhost:8080/addresses
• Edit Address - PUT – http://localhost:8080/addresses/:id
• Delete Address – DELETE – http://localhost:8080/addresses/:id

Next part 2 we will setup our base project with compojure.

Map

Map applies a function to each element in a collection and returns a new collection with the results of each function.

user> (map #(+ 10 %1) [ 1 3 5 7 ])
(11 13 15 17)

Reduce

Reduce applies a function on all elements of a collection and returns a value. This value could also be another collection.

user> (reduce * [2 3 4])
24

Filter

Filter applies a predicate function to each element in a collection and returns a new filtered collection.

user> (filter even? [1 2 3 4 5 6])
(2 4 6)

Problem :

We need to parse a xml string and be able to query using xpath style tag list.

Ex :

<friends>
  <person>
   <name>Siva</name>
  </person>
</friends>

I need a function that can do this,

(get-value xml :person :name)

Solution :

To parse and query xml we need to do these following three things in clojure.

1) Convert xml string (file) to Struct Map

Clojure core comes with a build in xml library (clojure.xml http://richhickey.github.com/clojure/clojure.xml-api.html) that has a parse function which takes in InputStream and returns a struct map that represents xml.

(defn get-struct-map [xml]
  (let [stream (ByteArrayInputStream. (.getBytes (.trim xml)))]
    (xml/parse stream)))

user> (get-struct-map xml)
{:tag :friends, :attrs nil, :content [{:tag :person, :attrs nil, :content [{:tag :name, :attrs nil, :content ["Siva"]}]}]}

This struct map is cumbersome to query.

2) Convert Struct Map to Zipper Data Structure

“A zipper is a technique of representing an aggregate data structure so that it is convenient for writing programs that traverse the structure arbitrarily and update its contents, especially in purely functional programming languages.” http://en.wikipedia.org/wiki/Zipper_%28data_structure%29

To make it easy for us to traverse we will change struct map to zipper data structure. Clojure comes with zip library ( it is short form for zipper ) http://richhickey.github.com/clojure/clojure.zip-api.html

We will this zip library to convert xml struct map to zipper data structure.

user> (clojure.zip/xml-zip xml-struct)
[{:tag :friends, :attrs nil, :content [{:tag :person, :attrs nil, :content [{:tag :name, :attrs nil, :content ["Siva"]}]}]} nil]

3) Use Zip-filter library and query zipper data structure.

Now that we have our xml in zipper data structure we could use zip-filter library that is present in clojure.contrib. http://richhickey.github.com/clojure-contrib/zip-filter-api.html

user> (clojure.contrib.zip-filter.xml/xml-> zipper-struct :person :name)
("Siva")

(ns com.sivajag.utils.xml
  (:import (java.io ByteArrayInputStream))
  (:require [clojure.xml :as xml])
  (:require [clojure.zip :as zip])
  (:require [clojure.contrib.zip-filter.xml :as zf]))

(defn get-struct-map [xml]
  (if-not (empty? xml)
    (let [stream (ByteArrayInputStream. (.getBytes (.trim xml)))]
      (xml/parse stream))))

(defn get-value [xml & tags]
  (apply zf/xml1-> (zip/xml-zip (get-struct-map xml)) (conj (vec tags) zf/text)))

user> (get-value xml :person :name)
"Siva"

Happy Coding!!!

What is Future?

It is a simple mechanism to execute a snippet of code on a background thread.

How to use Future?

We can use Future in two ways.

1) Using macro “future”

	
  (def f
	(future
 		(prn "going to sleep...")
		(Thread/sleep 10000)
		(prn "slept good")))

2) Using function “future-call”

(defn long-process[]
  (Thread/sleep 10000))

(future-call long-process)

Under the shell both these ways are equivalent. The macro is syntax sugar and it gets converted to future-call.

Even though “long-process” function is put to sleep for 10 secs, it will return immediately. It returns a “future” object. We can use this object to inspect whether the function has been executed in background thread.

(future-done? f)
(future-cancelled? f)
(future-cancel f)

Limitations of Future?

As you known clojure comes with a bounded thread pool and unbounded thread pool. Mostly CPU bound operations should be using bounded thread pool and IO operations should be using unbound thread pool. Clojure Future uses unbounded thread pool, so it is good for IO operations but not for CPU bound operations. So for CPU bound operations Future might not be a good choice. But there is a open source project that helps to overcome this limitations. Check out http://github.com/amitrathore/medusa

I was looking for a simple web crawler in clojure and came across clj-web-crawler (http://github.com/heyZeus/clj-web-crawler) . It took me less than 2 minutes to get started. (thanks to Leiningen)

1) In project.clj add these two lines for :dependencies

                 [com.jmeeks/clj-web-crawler "0.1.0-SNAPSHOT"]
                 [org.clojars.kjw/commons-httpclient "3.1"]	

2) Run lein deps and you ready to play with clj-web-crawler.

Checkout http://github.com/heyZeus/clj-web-crawler for more information

Tip 1 : Adding License info

To add your license info, add these following lines in project.clj

:license {:name "Eclipse Public License - v 1.0"
          :url "http://www.eclipse.org/legal/epl-v10.html"
          :distribution :repo
          :comments "same as Clojure"}

Tip 2: Turning off auto delete of lib folder

:disable-implicit-clean true

Tip 3: Turning on reflection warning calls

:warn-on-reflection true

Tip 4: Changing JVM level options

:jvm-opts "-Xmx1g"

Problem :

We need to parse a xml string and be able to query using xpath style tag list.

Ex :

<friends>
  <person>
   <name>Siva</name>
  </person>
</friends>

I need a function that can do this,

(get-value xml :person :name)

returns “Siva”

Solution :

To parse and query xml we need to do these following three things in clojure.

1) Convert xml string (file) to Struct Map

Clojure core comes with a build in xml library (clojure.xml http://clojure.github.com/clojure/clojure.xml-api.html) that has a parse function which takes in InputStream and returns a struct map that represents xml.

(defn get-struct-map [xml]
  (let [stream (ByteArrayInputStream. (.getBytes (.trim xml)))]
    (xml/parse stream)))

user> (get-struct-map xml)
{:tag :friends, :attrs nil, :content [{:tag :person, :attrs nil, :content [{:tag :name, :attrs nil, :content ["Siva"]}]}]}

This struct map is cumbersome to query.

2) Convert Struct Map to Zipper Data Structure

“A zipper is a technique of representing an aggregate data structure so that it is convenient for writing programs that traverse the structure arbitrarily and update its contents, especially in purely functional programming languages.” http://en.wikipedia.org/wiki/Zipper_%28data_structure%29

To make it easy for us to traverse we will change struct map to zipper data structure. Clojure comes with zip library ( it is short form for zipper ) http://clojure.github.com/clojure/clojure.zip-api.html

We will this zip library to convert xml struct map to zipper data structure.

user> (clojure.zip/xml-zip xml-struct)
[{:tag :friends, :attrs nil, :content [{:tag :person, :attrs nil, :content [{:tag :name, :attrs nil, :content ["Siva"]}]}]} nil]

3) Use Zip-filter library and query zipper data structure.

Now that we have our xml in zipper data structure we could use zip-filter library that is present in clojure.contrib. http://clojure.github.com/clojure-contrib/zip-filter-api.html

user> (clojure.contrib.zip-filter.xml/xml-> zipper-struct :person :name)
("Siva")

Putting this altogether

(ns com.sivajag.utils.xml
  (:import (java.io ByteArrayInputStream))
  (:require [clojure.xml :as xml])
  (:require [clojure.zip :as zip])
  (:require [clojure.contrib.zip-filter.xml :as zf]))

(defn get-struct-map [xml]
  (if-not (empty? xml)
    (let [stream (ByteArrayInputStream. (.getBytes (.trim xml)))]
      (xml/parse stream))))

(defn get-value [xml & tags]
  (apply zf/xml1-> (zip/xml-zip (get-struct-map xml)) (conj (vec tags) zf/text)))

user> (get-value xml :person :name)
"Siva"

Happy Coding!!!