Yesterday I was experimenting with Google App Engine for a little project that I was working on. I literally started from ground zero and could do my thing after a long night. I’m blogging about it to share you the idea and save you some time googling solutions. So, here is the thing in brief: I wanted to parse some feed on periodical basis and send an email with new entries.
Looks like a 10 minutes with Nokogiri and cron jobs. Actually that’s true except of the fact that I need to pay for a VPS so that I can run the script with various gems (since I needed to do some work on the feeds before sending, but that’s for another topic) and for using cron jobs. Thus, I decided to try something new this time and I went with GAE since it has memecached that I can use as an LRU cache, also it has cronjobs and finally it has a mail system. I’m using JRuby and Sinatra for this project.
Here are the steps:
Install JRuby, I’m using RVM on my machine:
rvm install jruby-1.5.5
rvm use jrubyInstall needed gems, those versions are the ones which worked for me:
gem install jbundle gem install appengine-sdk -v "1.4.0" gem install google-appengine -v "0.0.19"
Create a simple app:
appcfg.rb generate_app notifier
cd notifierNow let’s create the following files (Please note that the following code can be further enhanced):
Gemfile
# Critical default settings: disable_system_gems disable_rubygems bundle_path ".gems/bundler_gems" # List gems to bundle here: gem "appengine-apis" gem "appengine-rack" gem "sinatra" gem "sinatra-reloader" gem "jruby-rack", "1.0.4" gem "nokogiri","1.5.0.beta.3"
config.ru
(fill the sender/receiver emails and the feed URL)
require 'sinatra' set :sender_email, 'SENDER_EMAIL' set :receiver_email, 'RCEIVER_EMAIL' set :feed_url, 'FEED_URL' require 'app' run Sinatra::Application
app.rb
require 'digest/sha1' require 'appengine-apis/memcache' require 'appengine-apis/logger' require 'appengine-apis/mail' require 'appengine-apis/urlfetch' require 'sinatra' require 'sinatra/reloader' if development? require 'nokogiri' class Fetcher def initialize(sender_email,receiver_email,feed_url) @sender_email = sender_email @receiver_email = receiver_email @feed_url = feed_url end def act send_email(create_message(get_results)) "done" end private def memcache @memcached ||= AppEngine::Memcache.new(:namespace=>"cache") end # get the new results only def filter_results(results) res = [] results.each do |hashed_guid,info| if memcache.get(hashed_guid).nil? memcache.set(hashed_guid, 1) res << info end end res end # parse the feed def get_results results = {} doc = Nokogiri::XML(AppEngine::URLFetch.fetch(@feed_url).body) doc.css('item').each do |item| title = item.at_css('title').content link = item.at_css('link').content description = item.at_css('description').content guid = item.at_css('guid').content results[Digest::SHA1.hexdigest(guid)] = {:title => title,:description => description, :link => link} end filter_results(results) end # create email message def create_message(results) template = "" results.each do |info| template += <<-DOC Title: #{info[:title]} Description: #{info[:description]} Link: #{info[:link]} -------------------------------------------------------------------------------------------- DOC end template end def send_email(message) AppEngine::Mail::send(@sender_email,@receiver_email,"New updates for the site",message) if message.length > 0 end end get '/' do "Hello World!" end get '/notify' do Fetcher.new(settings.sender_email,settings.receiver_email,settings.feed_url).act end
cron.xml
(You can set the period here, I’m using 15 minutes intervals)
<?xml version="1.0" encoding="UTF-8"?> <cronentries> <cron> <url>/notify</url> <description>fetch new data</description> <schedule>every 15 minutes</schedule> </cron> </cronentries>
Now, create an application-id at appspot.com. Then go to Administration >> Permissions and make sure the sender/receiver emails play some role in the app. Personally, I granted them the developer role.
Now, go back to the source and modify the application-id in WEB-INF/app.yaml
application your-app-id
We should be ready now, start development server, watch the console and make sure no exceptions are there:
dev_appserver.rb .
If everything is OK, go to http://localhost:8080/notfiy and you should see “Done”.
Let’s go live now:
(the first time you execute this, you will prompted to submit the GAE email/pass combination)
appcfg.rb update .
It should now be running at http://your-app-id.appspot.com, and in few minutes you should be receiving the first email. If not, go to app engine page then to Main >> Logs and check what’s the problem.
I like this stack cause it’s Zero cent cost, Zero deployment time, flexible, and it gives me exactly the freedom I want in terms of needed gems. Give it a shot, you won’t regret it!
What is our development OS ?
I suppose it is not Windows. Few months ago, I failed to make a dummy application work on Windows.
I’m using Ubuntu, but I’m pretty sure it should work on windows since JRuby is supporting Windows very well.
Pingback: How You Can Use the Internet to Learn a Foreign Language
Awesome write!
Awesome read man.
Thanks a lot for this wonderful post. Its the reason that get me started with JRuby which I have been itching try out. I think you should have mentioned the version of JRuby, that you are using. Yesterday, I installed JRuby using rvm and it installed the Nailgun which is JRuby 1.5.6. I am following this post as guideline, haven’t completed yet but found that there are couple of notable difference in between this tutorials and the actual process, for instance with the version of appengine gem, bundler and Gemfile itself.
Error message : “‘disable_system_gems’ has been removed from the Gemfile DSL, and is no longer supported. See the README for more information on upgrading from Bundler 0.8.”
I have modified the Gemfile as following and did ‘bundle install’.
#Gemfile
source :rubygems
gem “appengine-apis”
gem “appengine-rack”
gem “sinatra”
gem “sinatra-reloader”
gem “jruby-rack”, “1.0.4″
gem “nokogiri”
@ranendra, if you use the rvm use jruby, then the console will use JRuby and the error you had above won’t show as JRuby right now uses different bundler.
@khell : thank you again for the quick reply. I did perform ‘rvm use jruby’ and I have created a gist[1] from the logs till now .
1. https://gist.github.com/861469
@ranendra, 2 things:
1- You don’t need to keep prefixing the commands with “jruby” like doing: “jruby whatever_statement”. Executing the statements as mentioned above is enough, since when doing “rvm use jruby”, JRuby will be your default Ruby version. You can check that by doing “ruby -v”. Also, please notice that in RVM, you have to execute the statement “rvm use whatever_ruby_version” for each console you open, unless you use the “rvm default …” statement.
2- Why are you doing “bundle install” ? That’s not required so that you run the app since the gems will get bundled for you upon running the development server or going live. So, don’t run that statement at all.
Try again and tell me how it goes
wonderful tutorial, thanks so much for taking the time to do such a good job on this
Great write up, just a minor (I hope) issue.
When I ran ‘dev_appserver.rb .’ from the notifier sub-directory I received the following output:
I am using rvm and issued the ‘rvm use jruby’.
Thanks again.
Never mind. It turned out to be that I needed to alter the Gemfile from:
To:
Thanks!
Thanks for the catch! Updated the post.
Thank you for tutorial, but i experienced one problem, after installing rvm and required gems i run ‘appcfg.rb generate_app test’ but get error:
$ appcfg.rb generate_app test
/home/wolf/.rvm/gems/jruby-1.6.0/gems/appengine-tools-0.0.17/lib/appengine-tools/gem_bundler.rb:63 warning: already initialized constant RUBY_ENGINE
=> Bundling gems
ERROR: While executing gem … (RuntimeError)
Unknown command bundle
What i do wrong? My system is ubuntu, i tried do this withour using rvm with same error.
@Sergiy, when it worked for me I was using jruby 1.5.6, but I think the recent rvm 1.2.9 update provides access only to jruby 1.6.0, and I do not see 1.5.6 in the list. Maybe that is your problem.
@Lyndon, this is problem with bundler08 gem http://groups.google.com/group/ruby-bundler/browse_thread/thread/bbdd4f6be5661290
@Sergiy
I’m having the same error here. Did you figure out a solution?
I suspect that it’s related to the gem command version. Since this error happens when the following is executed from appengine-tools/lib/appengine-tools/gem_bundler.rb:
Gem::CommandManager.instance.run(Gem.configuration.args)@khell, @hoshposh
Which version is your Rubygems?
Thanks.
OK guys, do this:
rvm install jruby-1.5.5
I have updated the article.
I’m using rubygems version 1.3.7
@andrecardoso, yes i had a solution, i use latest jruby with old rubygems
$ rvm use jruby
Using /home/wolf/.rvm/gems/jruby-1.6.0
$ gem list
*** LOCAL GEMS ***
addressable (2.2.4)
appengine-apis (0.0.23, 0.0.22)
appengine-rack (0.0.12)
appengine-sdk (1.4.0)
appengine-tools (0.0.17)
bouncy-castle-java (1.5.0145.2)
bundler (1.0.10)
bundler08 (0.8.5)
closure-compiler (1.0.0)
dm-appengine (0.1.3)
dm-core (1.1.0)
google-appengine (0.0.19)
jbundle (0.0.8)
jruby-enginize (0.9)
jruby-jars (1.6.0)
jruby-launcher (1.0.7)
jruby-openssl (0.7.3)
jruby-rack (1.0.7)
lexidecimal (0.0.1)
rack (1.2.2)
rake (0.8.7)
rubyzip (0.9.4)
sinatra (1.2.0)
tilt (1.2.2)
$ gem -v
1.3.5
Then to avoid error ‘ERROR: While executing gem … (RuntimeError) can’t add a new key into hash during iteration’, i made some monkey-patching in
nly, :except
gems/jruby/gems/bundler08-0.8.5/lib/bundler08/dependency.rb
Changed first lines in initialize method:
class Dependency < Gem::Dependency
attr_reader :name, :version, :require_as,
attr_accessor :source
def initialize(name, opts = {}, &block)
options = {}
opts.each do |k, v|
options[k.to_s] = v
end
And after this it works.
Gr8 right up. Got me into trying things with jruby and appengine for the first time and these seem like a cool thing together.
I had a question that might not be 100% relevant to the post but would appreciate if anyone here knows, I want to use the just released jruby 1.6.0 but in the 1.9.2 compatibility mode. Does anyone have any idea how to do so? Or even a close guess at where I should look at?
I got the app to run locally on the new jruby with the –1.9 switch, but what abt when deploying?
This may change, but the current appengine gems should be installed into a ruby 1.8.6 rvm, or simply into the system ruby. The development server runs whatever version of jruby is specified in the Gemfile (or the most current version).
Sadly, the most recent version of JRuby (1.6.0) contains a jar that is too big, and jruby-rack greater than 1.0.4 seem to have issues. For now please use these gems:
gem ‘jruby-jars’ ’1.5.6′
gem ‘jruby-rack’, ’1.0.4′
but the thing is, it’s not just bundling those gems, it’s also passing the –1.9 switch.
I think my sinatra app now bundles the jruby-jars 1.6.0 (at least that’s what i guess when i have jruby-core-1.6.0.jar in WEB-INF/lib).
But again, I don’t fully understand this and I don’t have the big picture of how this whole appengine jruby integration works
Great post, has saved us boodles of time. I don’t suppose you’ve integrated RSpec into the setup? Am really struggling to get it to run.