Ruby through rails part 5: Bundler Dsl

Note: All path are relative to bundler gem path. For these blog, i am currently using ruby 2.1.2 and bundler 1.6.3.

While going through Bundler source code earlier, we have seen how bundler evaluates the Gemfile and creates function for each of keyword like gem, source and etc. Now we are going to see the implementation details of a few of these functions. Here is the `gem` function.

#lib/bundler/dsl.rb
def gem(name, *args)
   if name.is_a?(Symbol)
     raise GemfileError, %{You need to specify gem names as Strings. Use 'gem "#{name.to_s}"' instead.}
   end

   options = args.last.is_a?(Hash) ? args.pop.dup : {}
   version = args

   normalize_options(name, version, options)
   dep = Dependency.new(name, version, options)

   # ...
end

The `gem` function accepts one mandatory parameter – the name of the gem and other parameters is an array (called splat parameters). These can be any of :version, :git, :github, :platforms, :source or :group. The * operator (pronounced “star,” “unarray,” or, among the whimsically inclined, “splat”) does a kind of unwrapping of its operand into its components, those components being the elements of its array representation. This function first checks if gem name is provided as symbol and if it is, it throws an exception. The next line check if the splat parameter is a hash. If it is, then it pops and clones the last arguments. Otherwise, the argument is assumed to be the version or if nothing has been mentioned, an empty array. For Example:

 gem 'rails', '4.1.1'
 gem 'mongoid-grid_fs', github: 'ahoward/mongoid-grid_fs'
 gem 'simple_form', :git => 'git://github.com/plataformatec/simple_form.git'

Now, each time the gem function is  invoked, it receives ‘rails’, ‘mongoid-grid_fs’ and ‘simple_form’ as the name. In the case of ‘rails’, the ‘version’ is set as 4.1.1 and variable options as empty hash. In the other two invocations  to the `gem` function, options will have a populated hash and ‘version’ will be an empty array since args have been popped!

Tip 1: Using pop is an interesting programming idea when you want to populate 2 variables differently depending on the input type!

Now, the variables ‘name’, ‘version’ and ‘options’ are passed to `normalize_options`.

def normalize_options(name, version, opts)
  normalize_hash(opts)

  git_names = @git_sources.keys.map(&:to_s)

  # ...
end

Function ‘normalize_options’ change the keys in options variable from symbol to string. Notice the @git_sources in the code. This contains two keys :github and :gist. While digging deeper, I found that where this instance variable gets set. – during the initialization of class Bundler::Dsl. The function ‘add_github_sources’ is shown as below:

# lib/bundler/dsl.rb
def initialize
  @source          = nil
  @sources         = []
  @git_sources     = {}
  @dependencies    = []
  @groups          = []
  @platforms       = []
  @env             = nil
  @ruby_version    = nil
  add_github_sources
end
def add_github_sources
  git_source(:github) do |repo_name|
    repo_name = "#{repo_name}/#{repo_name}" unless repo_name.include?("/")
    "git://github.com/#{repo_name}.git"
  end

  git_source(:gist){ |repo_name| "https://gist.github.com/#{repo_name}.git" }
end

def git_source(name, &block)
  unless block_given?
    raise InvalidOption, "You need to pass a block to #git_source"
  end

  if valid_keys.include?(name.to_s)
    raise InvalidOption, "You cannot use #{name} as a git source. It " \
      "is a reserved key. Reserved keys are: #{valid_keys.join(", ")}"
  end

  @git_sources[name.to_s] = block
end

‘add_git_sources’ function calls git_source which pass two parameters: name of source (gist & github) and a block. If name passed to git_source function is not from valid keys, then it will raise InvalidOption error. ‘valid_keys’ used in above is function returns an array of keywords (show in the code snippet below). ‘git_source’ function create the @git_sources hash that contains the name of source as key and the block as value.  now, the instance variable ‘@git_sources’ has keys :github and :gist. Value of these keys is a block that gets evaluated in the ‘normalize’ function.

Tip 2: You can use blocks as values in a hash!

def valid_keys
  @valid_keys ||= %w(group groups git path name branch ref tag require submodules platform platforms type)
end

Let’s get back to `normalise_options`. Lets see what happens when an invalid key is passed. For Example:

#In Gemfile add rmagick gem with some arbitrary options.

gem 'rmagick', :aaa => 'surely you are joking'

# Error message it shows when doing 'bundle install':
# You passed :aaa as an option for gem 'rmagick', but it is invalid. Valid options are: group, groups, git, path, name, branch, ref, tag, require,
# submodules, platform, platforms, type

Again in function ‘normalize_options’, instance variable @groups is initiated (as empty array []) in class(Bundler::Dsl) initialization and get set when group function is called but during normal call to gem, it is just nil. Option group (opts[:group]) is set if the group option is passed gem (gem ‘better_erros’, :group => :development). If group or groups options not passed to gem, then gem is put under :default group.

def normalize_options(name, version, opts)
      # ...
      groups = @groups.dup
      opts["group"] = opts.delete("groups") || opts["group"]
      groups.concat Array(opts.delete("group"))
      groups = [:default] if groups.empty?

      platforms = @platforms.dup
      opts["platforms"] = opts["platform"] || opts["platforms"]
      platforms.concat Array(opts.delete("platforms"))
      platforms.map! { |p| p.to_sym }
      platforms.each do |p|
        next if VALID_PLATFORMS.include?(p)
        raise GemfileError, "`#{p}` is not a valid platform. The available options are: #{VALID_PLATFORMS.inspect}"
      end
      # ...
end

Here is an example showing how the instance variable @groups gets used

# Here the instance variable @groups is [] and does not get set. 
# Empty @groups variable get dup #to another local variable
# Local variable opts[:group] is set and its just concat to 
# another local variable 'groups'
gem 'better_errors', :group => :development

# Here 'groups' variable is assigned empty array using dup 
# from @group variable, so its group is default.
gem 'rails'

#in below case, instance variable @groups is set
group :development, :test do
  gem 'better_errors'
  gem 'binding_of_caller'
  gem 'brakeman', :require => false
end

Here is the code for the method group

# lib/bundler/dsl.rb
def group(*args, &blk)
  @groups.concat args
  yield
ensure
  args.each { @groups.pop }
end

The group method adds args(:development, :test) to the @groups instance variable. The yield statement calls the block code which in turn calls ‘normalize_options’ function. Similar to the instance variable @groups, we have also @platforms variable which is initiliazed in Bundler::Dsl. Instance variable @platforms is checked for invalid platforms.

Back again to normalize_options method and.

def normalize_options(name, version, opts)
  # ...
  git_name = (git_names & opts.keys).last
  if @git_sources[git_name]
    opts["git"] = @git_sources[git_name].call(opts[git_name])
  end

  ["git", "path"].each do |type|
    if param = opts[type]
      if version.first && version.first =~ /^\s*=?\s*(\d[^\s]*)\s*$/
        options = opts.merge("name" => name, "version" => $1)
      else
        options = opts.dup
      end
      source = send(type, param, options, :prepend => true) {}
      opts["source"] = source
    end
  end

  opts["source"]  ||= @source
  opts["env"]     ||= @env
  opts["platforms"] = platforms.dup
end

In above code, variable git_names contain :gist and :github (as we have seen in above code snippets.). Options passed (‘opts’) and ‘git_names’ are used  to find if options contain :gist and :github as options. If :gist or :github passed, then the block i.e. the key for the git_name is invoked and returns the github or gist url and stored in ‘opts[:git]’. For example:

gem 'mongoid-grid_fs', github: 'ahoward/mongoid-grid_fs
# Call to @git_sources[git_name].call(opts[git_name]) will return git://github.com/ahoward/mongoid grid_fs.git

After this, the “git” and “path” options are checked in normalize_options function. If variable opts contains ‘git’ or ‘path’, then the version format is checked and ‘name’ and ‘version’ are merged into opts. The ‘send’ function is called with parameter as type (either git or path), params (i.e. opts[:git] or opts[:path]), options and the prepend value passed as true.

Tip 3:  ‘send’ is defined as a method on the Object class and it invokes the function whose name is first parameter. This is the essence of meta-programming.

In our case, it will try to find the function named git or path. Of  course, these function are defined as show below ‘git’ and ‘path’ is shown below:

# lib/bundler/dsl.rb
def path(path, options = {}, source_options = {}, &blk)
  source Source::Path.new(normalize_hash(options).merge("path" => Pathname.new(path))), source_options, &blk
end

def git(uri, options = {}, source_options = {}, &blk)
  unless block_given?
    msg = "You can no longer specify a git source by itself. Instead, \n" \
              "either use the :git option on a gem, or specify the gems that \n" \
              "bundler should find in the git source by passing a block to \n" \
              "the git method, like: \n\n" \
              "  git 'git://github.com/rails/rails.git' do\n" \
              "    gem 'rails'\n" \
              "  end"
    raise DeprecatedError, msg
  end
  source Source::Git.new(normalize_hash(options).merge("uri" => uri)), source_options, &blk
end

The classes Source::Git and Source::Path are defined in ‘lib/bundler/source/git.rb’ and ‘lib/bundler/source/path.rb’ respectively. Here is an example of Bundler::Source::Git object.

 #<Bundler::Source::Git:0x000000024a31f0 @options={\"github\"=>\"ahoward/mongoid-grid_fs\", \"git\"=>\"git://github.com/ahoward/mongoid-grid_fs.git\", \"uri\"=>\"git://github.com/ahoward/mongoid-grid_fs.git\"}, @glob=\"{,*,*/*}.gemspec\", @allow_cached=false, @allow_remote=false, @uri=\"git://github.com/ahoward/mongoid-grid_fs.git\", @branch=nil, @ref=\"master\", @submodules=nil, @name=nil, @version=nil, @copied=false, @local=false>

and here is an example of Bundler::Source::Path

#<Bundler::Source::Path:0x00000001e6e8e8 @options={\"path\"=>#<Pathname:../docsplit>}, @glob=\"{,*,*/*}.gemspec\", @allow_cached=false, @allow_remote=false, @path=#<Pathname:../docsplit>, @name=nil, @version=nil, @original_path=#<Pathname:../docsplit>> and block true"
"sources #<Bundler::Source::Git:0x00000001dfa768 @options={\"github\"=>\"marcosbeirigo/mina_extensions\", \"git\"=>\"git://github.com/marcosbeirigo/mina_extensions.git\", \"uri\"=>\"git://github.com/marcosbeirigo/mina_extensions.git\"}, @glob=\"{,*,*/*}.gemspec\", @allow_cached=false, @allow_remote=false, @uri=\"git://github.com/marcosbeirigo/mina_extensions.git\", @branch=nil, @ref=\"master\", @submodules=nil, @name=nil, @version=nil, @copied=false, @local=false>

Function ‘path’ contain ruby library Pathname. Pathname represents a path on the file system, providing convenient access to functionality that is otherwise scattered across a handful of other classes like File, FileTest, and Dir. Both ‘git’ and ‘path’ invoke function source. (We have discussed source function in previous blog) but I had intentionally left out the else part of source function. Let see this source function again.

def source(source, options = {})
  case source
  when :gemcutter, :rubygems, :rubyforge then
    Bundler.ui.warn "The source :#{source} is deprecated because HTTP " \
          "requests are insecure.\nPlease change your source to 'https://" \
          "rubygems.org' if possible, or 'http://rubygems.org' if not."
        rubygems_source.add_remote "http://rubygems.org"
    return
  when String
    rubygems_source.add_remote source
    return
  else
    @source = source
    if options[:prepend]
      @sources = [@source] | @sources
    else
      @sources = @sources | [@source]
    end

    yield if block_given?
    return @source
  end
ensure
  @source = nil
end

The functions ‘path’ and ‘git’ pass instance of class as source, hence else part is called. It prepends the source, in this case, the instance of the class, to already existing sources stored in @sources with pipe operator (|) and returns the source. The function returns @source which get stored in opts[:source] which would getting used later during installation process.

Now, Going to back start of our blog and back into the ‘gem’ function.

def gem(name, *args)
  if name.is_a?(Symbol)
    raise GemfileError, %{You need to specify gem names as Strings. Use 'gem "#{name.to_s}"' instead.}
  end

  options = args.last.is_a?(Hash) ? args.pop.dup : {}
  version = args

  normalize_options(name, version, options)
  dep = Dependency.new(name, version, options)

  # if there's already a dependency with this name we try to prefer one
  if current = @dependencies.find { |d| d.name == dep.name }
    if current.requirement != dep.requirement
      if current.type == :development
        @dependencies.delete current
      elsif dep.type == :development
        return
      else
        raise GemfileError, "You cannot specify the same gem twice with different version requirements.\n" \
                            "You specified: #{current.name} (#{current.requirement}) and #{dep.name} (#{dep.requirement})"
      end

    else
      Bundler.ui.warn "Your Gemfile lists the gem #{current.name} (#{current.requirement}) more than once.\n" \
                          "You should probably keep only one of them.\n" \
                          "While it's not a problem now, it could cause errors if you change the version of just one of them later."
    end

    if current.source != dep.source
      if current.type == :development
        @dependencies.delete current
      elsif dep.type == :development
        return
      else
        raise GemfileError, "You cannot specify the same gem twice coming from different sources.\n" \
                            "You specified that #{dep.name} (#{dep.requirement}) should come from " \
                            "#{current.source || 'an unspecified source'} and #{dep.source}\n"
      end
    end
  end

  @dependencies << dep
end

All the gems mentioned in the Gemfile are passed to the Dependency class which is defined in  “lib/bundler/dependency.rb”. The Dependency class contains the name, version, type, source, group, platforms & env as attributes. The instance variable @dependencies is initiated during Bundler::Dsl.new call as empty array.

In the code above, the if.. else block checks if the gem with that name already exists in our dependencies, and tries to find suitable solution:

  1. If current gem requirement is different and it is in development environment, then it will deleted from @dependencies.
  2. If the existing gem in @dependencies is different and is in development environment, then it simply returns.
  3. If the gem exists but does not satisfy above two condition, then process will be aborted and errors messages will prompted on console.
  4. If current gem is not in @dependencies, then it is added to it  (@dependencies << dep).

Note, that the first 2 steps are repeated for all gems, even if source of both gems is different.

We have now learned about who the gem function works. In upcoming blog posts, we are going to discuss the gemspec function and how the Dependancies work.


One thought on “Ruby through rails part 5: Bundler Dsl

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s