Ruby

The Fedora Ruby SIG is in the process of updating the main Ruby package in Fedora to Ruby 1.9. We hope to get it in by Fedora 17, while shipping a Ruby 1.8 compatability package for those who wish to continue using the older version (though more and more upstream projects are phasing this out). Alternatively, since JRuby is now in Fedora, end-users may opt to utilize that interpreter for both Ruby 1.8 and 1.9 support.

The Ruby 1.9 effort is being led by Vit Ondruch who has done alot of the packaging work and submitted the initial guidelines draft, which we are currently discussing/revising on the mailing lists. They are largely an extension of the existing guidelines to better nail things down and cover various issues that have come up over the years, but all are welcome to join the discussion and share their comments / opinions on the best way to move forward.

A Ruby 1.9.3 testing repository has been setup for those who wish to try the new packages out. Feel free to share any feedback or issues on list.

As part of our dev and release cycle for aeolus we build alot of rpms and yum repositories.

To help with this process, especially for the aeolus puppet configuration recipes, we whipped up a couple of rake tasks to create rpms and yum repositories.

The latest versions of the tasks can be found here and can simply be included in your Rakefile as such:

require 'rubygems'
require 'rake/rpmtask'
require 'rake/yumtask'
 
CURRENT_DIR  = File.dirname(__FILE__)
YUM_REPO     = "#{CURRENT_DIR}/repo"
RPM_SPEC     = "my-package.spec"
 
# Build the rpm
rpm_task =
Rake::RpmTask.new(RPM_SPEC) do |rpm|
  rpm.need_tar = true
  rpm.package_files.include("bin/*", "recipes/**/*")
  rpm.topdir = "#{RPMBUILD_DIR}"
end
 
# Construct yum repo
Rake::YumTask.new(YUM_REPO) do |repo|
  repo.rpms << rpm_task.rpm_file
end

And without further ado, here is the rpm task

# Define a package task library to aid in the definition of RPM
# packages.
 
require 'rubygems'
require 'rake'
require 'rake/packagetask'
 
require 'rbconfig' # used to get system arch
 
module Rake
 
  # Create a package based upon a RPM spec.
  # RPM packages, can be produced by this task.
  class RpmTask < PackageTask
    # RPM spec containing the metadata for this package
    attr_accessor :rpm_spec
 
    # RPM build dir
    attr_accessor :topdir
 
    # Include extra_release information in the rpm
    attr_accessor :include_extra_release
 
    def initialize(rpm_spec)
      init(rpm_spec)
      yield self if block_given?
      define if block_given?
    end
 
    def init(rpm_spec)
      @include_extra_release = true
      @rpm_spec = rpm_spec
 
      # parse this out of the rpmbuild macros,
      # not ideal but better than hardcoding this
      File.open('/etc/rpm/macros.dist', "r") { |f|
        f.read.scan(/%dist\s*\.(.*)\n/)
        @distro = $1
      }
 
      # Parse rpm name / version out of spec
      # hacky way to do this for now
      #   (would be nice to implement a full blown rpm spec parser for ruby)
      File.open(rpm_spec, "r") { |f|
        contents = f.read
        @name    = contents.scan(/\nName: .*\n/).first.split.last
        @version = contents.scan(/\nVersion: .*\n/).first.split.last
        @release = contents.scan(/\nRelease: .*\n/).first.split.last
        @release.gsub!("%{?dist}", ".#{@distro}")
        @arch    =  contents.scan(/\nBuildArch: .*\n/) # TODO grab local arch if not defined
        if @arch.nil?
          @arch = Config::CONFIG["target_cpu"] # hoping this will work for all cases,
                                               # can just run the 'arch' cmd if we want
        else
          @arch = @arch.first.split.last
        end
      }
      super(@name, @version)
 
      @rpmbuild_cmd = 'rpmbuild'
    end
 
    def define
      super
 
      directory "#{@topdir}/SOURCES"
      directory "#{@topdir}/SPECS"
 
      desc "Build the rpms"
      task :rpms, [:include_extra_release] => [rpm_file]
 
      # FIXME properly determine :package build artifact(s) to copy to sources dir
      file rpm_file, [:include_extra_release] => [:package, "#{@topdir}/SOURCES", "#{@topdir}/SPECS"] do |t,args|
        @include_extra_release = args.include_extra_release != "false"
        git_head = `git log -1 --pretty=format:%h`
        extra_release = "." + Time.now.strftime("%Y%m%d%k%M%S").gsub(/\s/, '') + "git" + "#{git_head}"
        cp "#{package_dir}/#{@name}-#{@version}.tgz", "#{@topdir}/SOURCES/"
        cp @rpm_spec, "#{@topdir}/SPECS"
        sh "#{@rpmbuild_cmd} " +
           "--define '_topdir #{@topdir}' " +
           "--define 'extra_release #{@include_extra_release ? extra_release : ''}' " +
           "-ba #{@rpm_spec}"
      end
    end
 
    def rpm_file
      # at some point we should support all a spec's subpackages as well
      "#{@topdir}/RPMS/#{@arch}/#{@name}-#{@version}-#{@release}.#{@arch}.rpm"
    end
  end
end

And here is the yum task

# Define a package task library to aid in the definition of YUM
# repos.
 
require 'rubygems'
require 'rake'
 
module Rake
 
  # Create a yum repo with specified rpms
  class YumTask < TaskLib
    # Yum repository location which to build
    attr_accessor :yum_repo
 
    # RPMs to pull into yum repo
    attr_accessor :rpms
 
    def initialize(yum_repo)
      init(yum_repo)
      yield self if block_given?
      define if block_given?
    end
 
    def init(yum_repo)
      @yum_repo = yum_repo
      @createrepo_cmd = 'createrepo'
      @rpms = []
    end
 
    def define
      desc "Build the yum repo"
      task :create_repo => @rpms do
        @rpms.each { |rpm|
          rpmc = rpm.split('.')
          arch = rpmc[rpmc.size-2]
          arch_dir = @yum_repo + "/" + arch
          FileUtils.mkdir_p arch_dir unless File.directory? arch_dir
          cp_r rpm, "#{@yum_repo}/#{arch}"
        }
        sh "#{@createrepo_cmd} -v #{@yum_repo}"
      end
    end
  end
end

Have at it!

Just a quick update, both Ruby 1.8.7 and Rails 2.3.8 are now official in and will be shipping with Fedora 14. Many thanks to those who helped for all their hard work, and for those relying on these rpms please testing your software and systems to ensure compatibility and prevent any surprises if/when you upgrade. Feel free to report any issues you find (and please do if you find them).

I'm going to be putting development on these on hold for a little while while I work on other things, namely my main project deltacloud (make sure to check it out, the team has done alot of great work on it and it's coming along nicely) and getting jruby and passenger into Fedora. I intend to get back to this eventually, to help Jeroen with the parallel installable Ruby 1.9 work and figuring out a strategy for Rails 3.0.0 support. Until then, have fun!

Recently I implemented a simple thread pool in Ruby for a side project I'm working on and figured I'd share.

AFAIK its pretty solid, it does what it needs to do without any deadlocks or race conditions (feel free to correct me if you find one), though it probably could use a little simplification and 'Rubyification'. As an added bonus I've included an optional timeout feature, if a job being run by a thread in the pool takes longer than the specified time, it is killed and that thread is then free to pick up another task. Enjoy!

  require 'thread'
 
  # Work item to be executed in thread pool
  class ThreadPoolJob
    attr_accessor :handler
    attr_accessor :params
 
    def initialize(*params, &block)
      @params = params
      @handler = block
    end
  end
 
 
  # Launches a specified number of threads on instantiation,
  # assigning work to them as it arrives
  class ThreadPool
 
    # Encapsulate each thread pool thread in object
    class ThreadPoolJobRunner
      attr_accessor :time_started
 
      def initialize(thread_pool)
        @thread_pool = thread_pool
        @timeout_lock = Mutex.new
        @thread_lock  = Mutex.new
      end
 
      def run
        @thread_lock.synchronize {
          @thread = Thread.new {
            until @thread_pool.terminate
              @timeout_lock.synchronize { @time_started = nil }
              work = @thread_pool.next_job
              @timeout_lock.synchronize { @time_started = Time.now }
              work.handler.call *work.params unless work.nil?
            end
          }
        }
      end
 
      def check_timeout(timeout)
        @timeout_lock.synchronize {
          if !@time_started.nil? && Time.now - @time_started > timeout
            stop
            run
          end
        }
      end
 
      def stop
        @thread_lock.synchronize {
          if @thread.alive?
            @thread.kill
            @thread.join
          end
        }
      end
    end
 
    # Create a thread pool with a specified number of threads
    def initialize(num_threads, args = {})
      @num_threads = num_threads
      @timeout     = args[:timeout]
      @job_runners = []
      @job_runners_lock = Mutex.new
      @terminate = false
      @terminate_lock = Mutex.new
 
      @work_queue  = Queue.new
 
      0.upto(@num_threads) { |i|
        runner = ThreadPoolJobRunner.new(self)
        @job_runners << runner
        runner.run
      }
 
      # optional timeout thread
      unless @timeout.nil?
        @timeout_thread = Thread.new {
          until terminate
            sleep @timeout
            @job_runners_lock.synchronize {
              @job_runners.each { |jr|
                jr.check_timeout(@timeout)
              }
            }
          end
        }
      end
    end
 
    # terminate reader
    def terminate
      @terminate_lock.synchronize { @terminate }
    end
 
    # terminate setter
    def terminate=(val)
      @terminate_lock.synchronize { @terminate = val }
    end
 
    # Add work to the pool
    def <<(work)
      @work_queue.push work
    end
 
    # Return the next job queued up
    def next_job
      @work_queue.pop
    end
 
    # Terminate the thread pool
    def stop
      terminate = true
      @timeout_thread.join unless @timout_thread.nil?
      @work_queue.clear
      @job_runners_lock.synchronize { @job_runners.each { |jr| jr.stop } }
    end
  end

Attached is the presentation I have prepared in both openoffice and text formats.

Before I get to rails, note that an updated Ruby 1.8.7 srpm is now available. The latest 1.8.7 release tarball is now being used (p299), there are many fixes and improvements, and its pretty close to the final version which will be pushed into Fedora (of course any and all feedback and additional improvements are more than welcome). Many thanks go out to Jim Meyering and Mamoru Tasaka for all their hard work and help to get to this point.

I'm also pleased to announce new Ruby on Rails rpms (built against Ruby 1.8.7) corresponding to the 2.3.8 and 3.0.0.beta4 Rails releases. Admittedly these will most likely require additional tweaking to resolve any issues as they are discovered, but what's there should be the complete Rails suite packaged and ready to deploy. Feedback for any / all of these packages would also be much appreciated.

Without further ado, here is the Rails 2.3.8 suite:
activesupport: srpm rpm
activerecord: srpm rpm
actionpack: srpm rpm
actionmailer: srpm rpm
activeresource: srpm rpm
rails: srpm rpm

And here is the the Rails 3.0.0.beta4 suite:
activesupport: srpm rpm
activerecord: srpm rpm
actionpack: srpm rpm
actionmailer: srpm rpm
activeresource: srpm rpm
rails: srpm rpm

Also note you will need an updated sqlite3-ruby rpm to build either set, which I provide here (rpm here).

Enjoy!

For those that are interested, I managed to cobble the Ruby 1.8.6 and 1.9.1 spec files together to form a Ruby 1.8.7 rpm (srpm here) built against Fedora 11 (yes I really need to update to F13, just haven't had the cycles yet). It's not 100% perfect, I tried to simplify the package as much as possible, removing any unneeded cruft, but in the process may have remove a couple things we wanted to keep (namely pulling the upstream ruby-tk branch for the ruby-tcltk package). But what is there is working as evident by the attached screenshot, and anything missing can easily be readded by using the 1.8.6 specfile as a reference.

Next I'm going to be working towards Rails 3.0.0 rpms and a Ruby 1.8.7 / Fedora appliance that pulls all of this in. After this we should have a base platform to try some widespread integration testing for multiple versions of Ruby on Fedora.

Enjoy!

I am pleased to release a Fedora 12 based appliance complete w/ a stack of rpms built against and related to Ruby 1.9.1.

This was built using Polisher, which I improved extensively and configured to generate various RPMs and yum repos. There are now many more rpms being generated/hosted since my last announcement, and after alot of work I finally got ruby-shadow to compile against Ruby 1.9.1. The packages themselves were built in mock, and made use of Polisher's project dependency support, so that projects built against the 'devel' repo were being built against the Ruby 1.9 package that resided there.

To make use of the appliance, download and extract the tarball, and run 'virt-image' on the polisher-devel.xml file (you will need the appliance tools installed). This will take care of all the work for you, you can then just open virt-manager, virt-viewer, or any other virt client and use a Ruby 1.9.1 build for Fedora!

Many thanks go out to those on ruby-sig for all the work to support Ruby on Fedora, especially Jeroen for the Ruby 1.9.1 spec file (and all the other work/research) which I initially based this of off. There were several other changes that went into it as issues were discovered, particularily some patches from Ruby 1.8.6 had to be readded to fix issues, and care had to be taken due to the fact that rubygems and rake were both merged into the official Ruby project stating in Ruby 1.9 (right now I figured the easiest way to handle this was to change those packages to be empty and just depend on / pull in the Ruby rpm).

There is still alot more work to do, eg nits w/ a few of the specific scripts, support for more projects, and a working Ruby 1.8.7 appliance (the maintenance repo and 1.8.7 specfile are still a work in progress). But what is there is a very easy way to run Ruby 1.9 natively on Fedora, and to start testing various software stacks. Until the next update, I leave you with this, a screenshot of Fedora 12 running a ruby-gnome app built/running against Ruby 1.9 (source code is attached to this post). Enjoy!

Alot of changes have gone into Polisher since my last update (eg the initial release). Most notably, the focus of Polisher has been changed and expanded, it now serves as a generic upstream project post-release processing tool, which can be used to configure any number of upstream projects w/ many various sources, and associate events with them to be triggered on releases. Polisher still works w/ the gemcutter API (namely the subscriptions / webhooks) as well as with gem2rpm, but now they are special cases in the generic upstream project management framework.

In the backend, Polisher is now based on Sinatra, which is far more lightweight and flexible than the previous solution, Rails. Rails is a powerful framework in itself, but I was looking for something small to help me establish the primary REST interface to the webapp.

Polisher also provides a nice DSL interface which can be used to define and run mass project / source registrations and trigger events in order to automate many various releases at the same time. I went through and picked several popular ruby-related packages from the current Fedora Repos, and wrote Polisher scripts for all of them, establishing several yum repositories for various Ruby related stacks.

Currently I have setup:

* stable - same packages that are currently in Fedora updates
* maintenance - later releases in the same major version branch of the packages currently in stable
* devel - packages that work w/ Ruby 1.9 (great resource)
* rawhide - all latest project versions

By tweaking the Polisher scripts and rpm spec file templates, the contents of these repos can be changed and new ones can be setup for any target software stack.

Any Fedora / yum user can simply point a yum repo entry towards any of those before running a yum update to get all the latest Ruby packages provided by that repo. Warning no functionality is guaranteed, there may currently be major breakage as Polisher and the scripts themselves are a work in progress. Furthermore, I'm in the process of setting up various Fedora based appliances, each pointing at a different Polisher generated repo by default. Any user can easily startup a VM on their local machine based off any of those appliances and instantly have Fedora w/ a complete Ruby stack for whichever version of Ruby they want to use. It's a work in progress, and I will have a link up soon to the downloadable images.

It would be awesome to get some help from anyone in the community that's interested in Ruby on Fedora (or any upstream project post-release processing on any flavor of Linux in general, as it's trivial to expand Polisher to handle any input/output package or repository format). It is a bit of effort to get Polisher scripts written for every Ruby related package, but if this thing starts rolling, I can see this as being a useful tool in the "upstream project to Fedora repo" process as mass parallel source releases can be triggered and handled easily in a committable / reproducible / reliable fashion.

Well that's it for now, be sure to note the many updates to deltacloud and many to my own projects, more on all of those in the near future.

Until next time, take it easy.

Just finished unorphaning, packaging, and submitting JRuby for Fedora. There were 13 Java packages (JRuby and 12 dependencies) that had to be updated and submitted, and getting them all in might take some time, but what's there is working, most packages build on Koji fine and the rest depend on other packages in the set that have to make it into the repos first. I've done quite a bit of Java work in the past (RIP Mr. Logistics!) but wouldn't say I'm a guru so if something is amiss feel free to comment in Bugzilla.

During this process I also looked into packaging Maglev, a cool new ruby interpreter that runs as a persistent server, which an end user's Ruby processes / vms running on any machine can connect to in order to manage objects and run code. No database backend is required, all persistence is done in the server and thus it scales very nicely and is highly optimized/fast (see this good article for more info). Unfortunately Maglev is still alpha and the build / install system is somewhat unconventional (neither of which is a problem for packaging for Fedora) but additionally the licensing situation is currently fairly messy and I'm not fully sure how compliant it is w/ the Fedora Licensing Guidelines. I'm going to keep track of this and potentially see what can be done if things change.

So for today, only one Ruby interpreter :-D All in all the following are the packages which need to be approved (those w/ a * next to them depend on other packages in the set). Fedora/Jython should also be able to benefit from these packages, as the newer versions of jython depend on some of these as well.

jffi nailgun yydebug yecht jnr-x86asm jgrapht jaffl* jcodings jnr-constants (formerly constantine) bytelist* jnr-posix (formerly jna-posix)* joni* jruby*

If anyone is interested in helping out w/ any aspect of the Fedora/Ruby integration feel free to join the discussion on the ruby-sig mailing list (particularly regarding supporting the various versions of Ruby and gems). Enjoy!