Being a data scientist (yup, that what we are called nowadays, apparently we are all the rage) and looking for my first bike to buy made me think (a lot) about what ride wanted.

Hundreds of pages on various forums talking about which bike is good, which one is better, reliability, position, torque, sound, etc etc. Yet in the end, once you have set your mind on one or two models, you still have dozens (if not hundreds) of classifieds for these models, and it is sometime hard to know whether the price is fair, a good deal, or an overpriced deal.

So, having lots of time on my hands at the moment, I started taking a quantitative approach.

• Step 1 : Get all the data (or as much as I can).
• Step 2 : Analysis, model
• Step 3 : Profit (now that I have a model, I can know whether any ad is a good deal or not).

Now that we have a server running our database, we want to make it also able to run our rails app. As I said in the previous post, I could use the full application cookbook, but I’d have to create my own cookbook and recipe to use that properly. For now, I’ll just rely on rackbox (and its dependency appbox).

Managing with appbox

The appbox cookbook does pretty much the stuff that we configured earlier (setting up users etc). Since it’s a dependency of rackbox, we are pretty much forced to use it (rackbox does call appbox default recipe).

Having added appbox to our Berksfile, I had to modify the roles/base.rb :

• Let app box know that my admin user name is wam.
• Generate a ssh key for to login as the deploy user.
• Make sure the app box recipe is run before I run my sudo and users::sysadmins recipe.

We’ll start by creating the deploy key :

1

$ ssh-keygen -t rsa -C "wam@scube" -f ~/.ssh/scube_deploy_rsa

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

name "base"
description "Base role applied to all nodes."
run_list(
  "recipe[apt]",
  "recipe[build-essential]",
  "recipe[git]",
  "appbox",
  "recipe[users::sysadmins]", # Necessary to run after appbox to add our stuff
  "recipe[sudo]" # Same
)
override_attributes(
  "appbox"  => {
    "deploy_keys" => [
      "ssh-rsa [...]" # content of ~/.ssh/scube_deploy_rsa.pub
    ],
    "admin_user" => "wam",
  }
)

Adding a rack server

I’ve chosen to use passenger + nginx, which is a popular choice among the rails community. I was tempted for a moment by puma on jruby, but I want my app online faster and will bother changing this kind of thing later (chef makes it easy to test new nodes with new recipes..)

Let’s create a roles/app_server.rb :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

name "app_server"
description "Serving http requests for the app. Main app server"
run_list(
  "rackbox"
)
override_attributes(
  "rackbox" => {
    "ruby" => {
      "versions"=> ["1.9.3-p385"],
      "global"=> "1.9.3-p385"
    },
    "apps"=> {
      "passenger"=> [
        {"appname"=> "scube", "hostname"=> "my.hostname.com"}
      ]
    }
  }
)

Now to make it work with vagrant, two changes are necessary in our Vagrantfile :

• We need to make sure that the chef version we are using is 11 or more. By default my vagrant was using chef 10, and the rackbox (more specifically the runit it uses) was throwing an error (NameError: Cannot find a resource for load_new_resource_state on ubuntu version 12.04).
• We add a port mapping to access the http port of our server on localhost:8888

1
2
3
4

# Stuff here
config.vm.network :forwarded_port, guest: 80, host: 8888
# Put this line just before your config.vm.provision :chef_solo line
config.vm.provision :shell, :inline => "gem install chef --version 11.4.2 --no-rdoc --no-ri --conservative"

The :shell provision makes sure that vagrant updates the chef gem before actually running our chef-solo provision. Now it’s on to vagrant provision, stuff should appear in green and http://localhost:8888 should show a 404 error. Yes of course, we haven’t deployed our app yet. That’ll be next.

Now that we have our vagrant box working with chef, let’s use chef to configure our services and our app.

Chef roles

Assuming that you have read some basics about chef, you’ll know that the cookbooks we have downloaded provide recipes for installing various software. We could ask vagrant to install a few recipes, but it’s probably better to assemble them in roles. We’ll then assign the roles to one or several nodes, or use all of them on our box for testing.

For now, we probably want to have one base role (to install common software on all our nodes) and two roles to serve our application :

• database\_master : a simple install of postgres should be fine here.
• app\_server : this one will serve our RoR app.

One could think of other roles (workers, redis etc), but for my purpose and for now these two (and the base role) should be fine.

Base role

We want our base role to include the following :

• apt, git, sudo and build-essential should be installed. We’ll use the default cookbooks/recipes for each of these.
• users setup : Should create the users (with their ssh key), give them sudo rights. We’ll use the users cookbook.

We start by editing our Berksfile to make sure all the cookbooks are included (sudo, apt, git, build-essential, users). Then let’s create a role file.

1
2
3
4
5
6
7
8
9

name "base"
description "Base role applied to all nodes."
run_list(
  "recipe[apt]",
  "recipe[build-essential]",
  "recipe[git]",
  "recipe[users::sysadmins]",
  "recipe[sudo]"
)

The order matters here ! apt should appear first (it’s used to handle packages), build-essential is used by pretty much everything, and especially by ruby-shadow which is a gem dependency of users.

Reading the documentation of the users cookbook, we see that we should define the users in a data bag (a way of telling chef about some data, list, including potentially encrypted password and ssh keys).

Chef solo doesn’t work very well with data bags (or the CLI doesn’t work very well), so we’ll just create the file manually. Also, we see in the users cookbook that it requires chef-solo-search to run with chef-solo.

Adding cookbook 'chef-solo-search', git: "https://github.com/edelight/chef-solo-search.git" to our Berksfile should be good enough.

1
2

$ mkdir data_bags/users
$ vim data_bags/users/wam.json

1
2
3
4
5
6
7
8
9
10
11
12
13
14

{
  "id": "wam", //or your user name
  // The following should be the result of openssl passwd -1 plainpasswd
  // but that's md5 on a mac. Alternatively run mkpasswd -m sha-512 -S mySalt on a linux machine
  "password": "$6$[...]098/",
  "ssh_keys": [
    // Copy paste from your ssh public key
    "ssh-rsa AAA123...xyz== foo"
    ],
  "groups": [ "sysadmin" ],
  "uid": 2001,
  "shell": "\/bin\/bash",
  "comment": ""
}

Now we need to modify our Vagrantfile to use this role (and not the dummy git recipe we were using). An extra bit of precaution is needed here : the sudo cookbook/recipe will install sudo qnd configure it by default for the sysadmin group (lucky us, our user is a member). It will override vagrant’s sudo config, breaking vagrant provision using chef-solo. To avoid that, we use vagrant’s chef.json config to override the sudo configuration attributes for vagrant :

1
2
3
4
5
6
7
8
9

  chef.add_role "base"
  chef.json = {
    :authorization => {
      :sudo => {
        :users => ["vagrant"],
        :passwordless => true
      }
    }
  }

Then it’s on to vagrant provision, and ssh to whatever port was forwarded to 22 (for me it was ssh localhost -p 2222) to see that you log in using your ssh key.

If you hit a json parsing exception when chef reads your user json file, make sure you don’t have trailing commas. You can check your JSON easily in irb using require 'json'; JSON.parse(File.read('data_bags/users/wam.json')).

Creating a custom cookbook …

There’s a big choice to do here. We could either create a whole separate cookbook just for our app, configured with many default recipes, or for now just use an already created one.

It is very likely that I’ll have to create a cookbook at some point, because it’s the only way to have your own recipes and reach a high enough level of customization.

.. or use and existing one

I originally had a look at the database cookbook but finally decided to go the fast way by using two very neat cookbooks, rackbox and databox. It will probably make sense to use database and application cookbooks, but they seem to be easier to work with when you are using a proper chef server and your own cookbook/recipes.

rackbox includes appbox by default, which creates its own users for deployment/app running. I have found that these cookbooks are a bit limited for my taste (for example, they don’t use data_bags, which are a proper way of encrypting password instead of storing them in your chef repository… Well, next time.

Setting up our roles

Let’s start by adding the cookbooks to our Berksfile and run berks install

1
2
3

cookbook "runit", ">= 1.1.2"  # HACK: force-use this version
cookbook "databox"
cookbook "rackbox"

and create our roles/database_master.rb. We are using non encrypted passwords here, which isn’t very secure. We should actually use encrypted data bags, but they don’t play very nicely with roles (they are supposed to be used with recipes, which would mean custom cookbook), nor do they play nicely with knife solo (although a plugin exist, but it didn’t work very well in my tests). Let’s start this way, we’ll see later to move to a more robust non solo chef.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

name "database_master"
description "Master postgresql node"

run_list(
  "databox::postgresql" # Or "databox" to include mysql as well
)
default_attributes(
  :databox => {
    :db_root_password => "PASSWORD_HERE",
    :postgresql => [
      {
        "database_name" => "myapp_production",
        "username" => "myapp",
        "password" => "ANOTHER_PASSWORD_HERE"
      }
     ]
  }
)

Now running vagrant provision (or vagrant up or vagrant reload depending on whether your current vagrant box is up or not) should run this recipe, adding the myapp database. We can test that in vagrant ssh

1
2
3
4

$ psql -h localhost -d myapp_production -U myapp -W
Password for user myapp:
#psql (9.1.9)
#[...] Yeepee

What’s next

Next post will be about configuring a proper rails box using rackbox, setting up capistrano to deploy … then deploy to a vps and get closer to production. I’m still not entirely happy with this deployment today. I should move to a proper cookbook, as I said, to get more customization options. For now, I want my app out, and will probably work a bit more later depending on how successful it is. The beauty of chef, after all, is that it makes it easy to set up new nodes and new deployments.

References

Chef cookbooks for busy ruby developers

I have updated a rails app I have been working on recently to a more recent version of rails 3.2, and all my tests where failing. Finally managed to have that working, figured I’d show how.

Mocking inherited_resources helpers in views specs.

I know I shouldn’t be using inherited_resources anymore (see here and here) but I want to release my app before I change everything to use responders.

So, my tests where failing because I was using the resource, collection and resource_class helpers from some views I was using. So first my tests are failing because resource_class isn’t available in my views. I would have thought that the controller helpers were available in the views, but they aren’t. The solution is easy. Let’s add the following to our spec/support directory :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

module ViewResourceMacros
  def has_resource(name, &block)
    before do
      # Creates the resource
      @resource ||= yield
      # Assign to the symbol we wanted, so it's available in the view
      assign(name, @resource)
      # Assigns to @name so that we can use that in our assertions
      instance_variable_set("@#{name}", @resource)

      # If we pass an array, it's for stubing a collection, if not it's for stubbing a single object
      if @resource.is_a?(Array)
        view.stub(:collection) { @resource }
        view.stub(:resource_class) {@resource.first.class}
      else
        view.stub(:resource) {@resource}
        view.stub(:resource_class) {@resource.class}
      end
    end
  end
end

RSpec.configure do |config|
  config.extend ViewResourceMacros, :type => :view
end

And see how to transform our old (failing) test :

1
2
3
4
5
6
7
8
9
10
11
12

require 'spec_helper'

describe "cars/edit.html.haml" do
  before(:each) do
    assign(:car, @car = Factory.create(:car))
  end

  it "renders the edit view" do
    render
    rendered.should contain(@car.name)
  end
end

becomes :

1
2
3
4
5
6
7
8
9
10

require 'spec_helper'

describe "cars/edit.html.haml" do
  has_resource(:car) { Factory.create(:car) }

  it "renders the edit view" do
    render
    rendered.should contain(@car.name)
  end
end

Using shared inherited partial in our views specs

Rails 3.1+ offers views inheritance, so I changed my code to have the following :

1
2
3
4
5
6
7
8
9
10
11

class BaseController

end

class CarsController < BaseController
  inherit_resources
end

class PlanesController < BaseController
  inherit_resources
end

Then I just created a base/new.html.haml and a base/edit.html.haml views, to use the views inheritance.

1
2

%h1 Create #{resource_class.model_name.human}
= render :partial => "form"

1
2

%h1 Edit #{resource}
= render :partial => "form"

And I have two _form.html.haml partials, one for each controller. Now the next issue is that our edit and new views are shared, but we still want to test the _form.html.haml partial.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

require 'spec_helper

describe "cars/_form.html.haml" do
  {
    new: -> { Car.new }
    edit: -> { Factory.create(:car) }
  }.each do |name, block|
    context "when called in ##{name}" do
      has_resource(name, block)

      it "renders the form" do
        render
        rendered.should have_selector("form")
      end
    end
  end
end

Shared partial

Finally, when testing for example cars/index.html.haml which uses a partial toolbar.html.haml that actually exists in base views, the following lines are necessary :

1
2
3

before do
  views.lookup_context.prefixes << "base"
end

This was raised as an issue to the rails team, but they commented (rightly I think) that the inheritance is related to the controller, not the views, so the test case shouldn’t know about it and you’ll have to declare it manually using the lines above.

Now let’s go back and make these tests green again.

So, let’s be honest, I’ve been quite lazy here. My latest rails application was deployed with Heroku. Lots of fun, a very pleasant experience. But now I have this new shinny application that I don’t want to deploy using Heroku :

• Because I am worried that if I start with Heroku, I’ll be too lazy to switch later.
• Because I know that at some point, my application is going to need plugins and binaries that I can’t get on Heroku.
• Because I’d rather have a portable application that I can deploy easily on any type of server.

Just a quick post, having finally figured out how to install dep_selector on mac os x. The issue is that having xcode installed which configures clang as the default compiler, some native gems break.

It took me some time, but at long least I can now build a native gem using gcc instead of clang on mac os x. And you’ll see it’s not very easy to have ruby change its compiler for gems native extensions compilation.

I haven’t posted anything for a while now, and after hours of trying to find a solution to my problem, I thought I should share. So here we go.

The problem

Your nice and shiny iOS app is supposed to have two data components : User data and Seed data. For example, you want to have some (seeded) list of postcodes. The size of data is too big to be shipped with your app, and we assume that the model is too complex to be just filled by your application at runtime from a downloaded csv/txt file.

So, you start thinking that hey, you’ll generate a sqlite database (persistent data store as they say), put it on a server and have your app download and use it. You can either duplicate the whole stack (NSManagedObjectContext, NSPersistentStoreCoordinator and NSManagedObjectModel) or, according to apple :

You typically use configurations if you want to store different entities in different stores. A persistent store coordinator can only have one managed object model, so by default each store associated with a given coordinator must contain the same entities. To work around this restriction, you can create a model that contains the union of all the entities you want to use. You then create configurations in the model for each of the subsets of entities that you want to use. You can then use this model when you create a coordinator. When you add stores, you specify the different store attributes by configuration. When you are creating your configurations, though, remember that you cannot create cross-store relationships.

Well, that’s pretty much all the doc you’ll get from apple. There are a few mentions of this problem there or there but not in a clear enough form for me. So, here’s how it works…

So it all started when I asked a friend to test this new app I’m building : “On your main page, garantee. You should really check your spelling, people will know you are French !”.

So here I went, naively googling for “website spell checker”, with either pricey reports or simple websites where I’d have to copy-paste my text from all my pages. Then, I found AfterTheDeadline. It looks great, open-source, recently bought by wordpress.com, and they even provide an API.

So, spent my evening doing stuff, and here we are with a new gem : Merimee (github).