Jaspersoft on VMware Micro Cloud Foundry
Cloud Foundry is VMware’s Platform as a Server (PaaS) offering. It’s good. It’s potentially a very big deal. Infrastructure as a Service (IaaS) systems like Amazon EC2 and Eucalyptus are better know. But PaaS has the potential to be bigger is several ways. (That’s a topic for another article.)
But as we were updating JasperReports Server Professional 4.5 for Cloud Foundry we ran into some problems. Here is a summary of the problems and solutions.
Problems on cloudfoundry.com
- Not enough memory
- Not enough files
It’s free to get started on cloudfoundry.com. It’s hard to complain about free… but it’s not impossible. Here I go. Two gigabytes is not enough. I would rather pay for four or eight than be stuck with two free gigabytes. I’m sure the Cloud Foundry team will be happy enough to accommodate this request soon. Someday they’ll offer a paid production version. But for now it’s only available for free and only with 2 Gb of memory. This is not enough. Specifically, it’s not enough for JasperReports Server. It says so right in the documentation. [warning: that link only works if you have access to the premium documentation, but for $100/year it's a fantastic deal.]
Actually, 2 Gb really is enough to run JasperReports Server. It works fine for me. But the whole reason you probably want JasperReports Server on Cloud Foundry is because you already created an application there, and now you want to do some analytics or reporting. That means you have a maximum of 1 Gb remaining for JasperReports Server if you allocate any memory to the other application(s). A single gigabyte is really not enough.
While everyone understands “not enough memory”, the complaint “not enough files” is a little weird. This stems from the fact that Cloud Foundry only allows a process to have 256 open files. 256 files is not enough.¹ This is a bug, and it will be solved someday, I’m sure. But for now it’s a problem.
Solutions on Micro Cloud Foundry
- Unlimited memory
- Plenty of files
Brief non-whiny version
You should probably read the full article once to understand what’s going on. But for future reference, here is the very brief version of the required changes.
- Edit micro.vmx. Set virtualhw.version = “6″
- Set memory for the Micro Cloud Foundry instance to 4.5 Gb or more.
- Install vmc v0.3.16 or later: gem install vmc –pre
- Edit /var/vcap/packages/dea/dea/lib/dea/agent.rb.
num_fds = 1024
- vmc env-add ‘my_application’ “JAVA_OPTS=$JAVA_OPTS -XX:MaxPermSize=256m”
Those steps are too brief to be meaningful the first time through… so you’re stuck reading the full article at least once. Tough luck.
Micro Cloud Foundry puts all of the power of PaaS into your own hands. In theory this means I should be able to fix anything that I don’t like about cloudfoundry.com. In practice… it’s true! (Well done, guys.) But it’s not always simple. It requires a mix of different skills. Here are the details needed to get JasperReports Server and other applications running well on Cloud Foundry.
When you download Micro Cloud Foundry you find that the default amount of memory allotted to the VM is just 1 Gb. Pshaw. I laugh at your 1 Gb of memory. We need at least 2 Gb. But this means we need at least 4 Gb. That’s because Cloud Foundry will only make half of the total memory available to a standard user. Increasing this limit is not quite intuitive to new users of VMware. Here’s what is needed.
First edit the file micro.vmx in any text editor. You can read all of the details about why this is true in the VMware knowledge base. In short, you need to make this change:
original: virtualhw.version = "4" modified: virtualhw.version = "6"
You need to make that change before launching the instance. If you have already launched it, then you can first confirm the piddly little 1 Gb.
C:\>vmc info VMware's Cloud Application Platform For support visit http://support.cloudfoundry.com Target: http://api.jaspersoft-webinar.cloudfoundry.me (v0.999) Client: v0.3.16.beta.6 User: email@example.com Usage: Memory (0B of 1.0G total) <-- Doh! This is not enough for anything fun. Services (0 of 16 total) Apps (0 of 16 total)
Then stop the instance and make the change.
After editing micro.vmx, then you need to use VMware Player or VMware Workstation to allocate more memory to the instance. You need a bit more than 4 Gb because not quite the entire 4 Gb is made available to the instance when it’s running.
Then on launching (or re-launching) the instance you’ll see option 3 “Reconfigure Memory” clearly highlighted in orange. Choose 3. Let it use all of the memory you have just allocated, and restart. Now all is good:
C:\>vmc info VMware's Cloud Application Platform For support visit http://support.cloudfoundry.com Target: http://api.jaspersoft-webinar.cloudfoundry.me (v0.999) Client: v0.3.16.beta.6 User: firstname.lastname@example.org Usage: Memory (0B of 5.5G total)
Well… things are not quite good enough yet. There is a bug in the current (as of 24 April 2012) stable version of vmc which causes deployment to fail when you have extra memory allocated. Never fear, this has already been fixed. But you need to upgrade to a beta release of vmc to get the fix. You need v0.3.16.beta.6 or later. Installing vmc is covered in detail in Cloud Foundry’s documentation. Updating to the pre-release version (beta version) is simple:
gem install vmc --pre
If you don’t upgrade to the latest and greatest vmc, then you’re destined for an error like this:
vmc push Would you like to deploy from the current directory? [Yn]: y Application Name: jrs-sample Application Deployed URL [jrs-sample.jaspersoft.cloudfoundry.me]: y can't dup NilClass C:/Ruby192/lib/ruby/gems/1.9.1/gems/rubyzip2-2.0.1/lib/zip/zip.rb:1131:in `dup' C:/Ruby192/lib/ruby/gems/1.9.1/gems/rubyzip2-2.0.1/lib/zip/zip.rb:1131:in `block in dup' C:/Ruby192/lib/ruby/gems/1.9.1/gems/rubyzip2-2.0.1/lib/zip/zip.rb:1131:in `map' C:/Ruby192/lib/ruby/gems/1.9.1/gems/rubyzip2-2.0.1/lib/zip/zip.rb:1131:in `dup' C:/Ruby192/lib/ruby/gems/1.9.1/gems/rubyzip2-2.0.1/lib/zip/zip.rb:1367:in `initialize' C:/Ruby192/lib/ruby/gems/1.9.1/gems/rubyzip2-2.0.1/lib/zip/zip.rb:1378:in `new' C:/Ruby192/lib/ruby/gems/1.9.1/gems/rubyzip2-2.0.1/lib/zip/zip.rb:1378:in `open' C:/Ruby192/lib/ruby/gems/1.9.1/gems/rubyzip2-2.0.1/lib/zip/zip.rb:1400:in `foreach' C:/Ruby192/lib/ruby/gems/1.9.1/gems/vmc-0.3.15/lib/cli/zip_util.rb:29:in `entry_lines' C:/Ruby192/lib/ruby/gems/1.9.1/gems/vmc-0.3.15/lib/cli/frameworks.rb:44:in `block in detect' C:/Ruby192/lib/ruby/gems/1.9.1/gems/vmc-0.3.15/lib/cli/frameworks.rb:33:in `chdir' C:/Ruby192/lib/ruby/gems/1.9.1/gems/vmc-0.3.15/lib/cli/frameworks.rb:33:in `detect' C:/Ruby192/lib/ruby/gems/1.9.1/gems/vmc-0.3.15/lib/cli/commands/apps.rb:434:in `push' C:/Ruby192/lib/ruby/gems/1.9.1/gems/vmc-0.3.15/lib/cli/runner.rb:440:in `run' C:/Ruby192/lib/ruby/gems/1.9.1/gems/vmc-0.3.15/lib/cli/runner.rb:14:in `run' C:/Ruby192/lib/ruby/gems/1.9.1/gems/vmc-0.3.15/bin/vmc:5:in `' C:/Ruby192/bin/vmc:19:in `load' C:/Ruby192/bin/vmc:19:in `'
Plenty of files
A Java application based on Spring needs lots of files. You can SSH to your Micro Cloud Foundry instance to see how many files may be opened at once:
root@micro:~# ulimit -n 1024
Somewhat surprisingly, the above result is totally irrelevant. Your tomcat instance can still only open 256 files. This is built-in to the Cloud Foundry Ruby code. How can you prove this to yourself? Like this:
root@micro:~# ps aux | grep tomcat 22001 2993 0.8 7.9 1425380 454164 ? SNl 18:00 0:20 /var/vcap/packages/dea_jvm/bin/java [... lots of stuff ...] org.apache.catalina.startup.Bootstrap start root@micro:~# cat /proc/2993/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds ... Max open files 256 256 files ...
Fortunately, Cloud Foundry has a way to modify this value. You need to set the parameter DEFAULT_APP_NUM_FDS.
Unfortunately, Cloud Foundry has a bug and it ignores the value you set.
Fortunately, you can ignore this ignored parameter and just hard-code the value you want.
Edit this file:
if limits = message_json['limits'] mem = limits['mem'] if limits['mem'] num_fds = limits['fds'] if limits['fds'] disk = limits['disk'] if limits['disk'] end # Jaspersoft decides to ignore the above code # Instead we overwrite the value: num_fds = 1024
With this hard-coded hack in place, the file limit is increased:
root@micro:~# ps aux | grep tomcat 22001 2993 0.8 7.9 1425380 454164 ? SNl 18:00 0:20 /var/vcap/packages/dea_jvm/bin/java [... lots of stuff ...] org.apache.catalina.startup.Bootstrap start root@micro:~# cat /proc/2993/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds ... Max open files 1024 1024 files ...
Surely the need for this hack will go away someday soon. I’ll update this article when it does.
But you aren’t quite done yet. The reason you need more files is because Spring-base applications need lots of files. And the reason they need lots of files closely related to the fact that they instantiate lots of different classes. The use of many classes implies the use of a lot of PermGen space. So you need to increase this from its tiny little default value. Cloud Foundry makes this easy. (This part isn’t a hack like the last part was.)
vmc stop jrs-pro-451 vmc env-add 'jrs-pro-451' "JAVA_OPTS=$JAVA_OPTS -XX:MaxPermSize=256m" vmc start jrs-pro-451
Without this change, JasperReports Server will boot up successfully. But it runs into problems quickly. It’s an important change. It may not be obvious why the application is having troubles. When you look at the log it’s clear:
vmc logs jrs-pro-451 INFO: Server startup in 17742 ms Apr 19, 2012 6:32:45 PM org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process SEVERE: Error reading request, ignored java.lang.OutOfMemoryError: PermGen space
Once you have all the memory you could want and all the files you can shake a stick at, you’re on your way to taking full advantage of Cloud Foundry PaaS.
Although cloudfoundry.com does not today (24 April 2012) accommodate full-sized applications the way it should, it’s entirely to the Cloud Foundry team’s credit that Micro Cloud Foundry gives as much power as required to do what we need.
I’m sure cloudfoundry.com will catchup someday soon. I’ll update this article when it does.
¹ “256 files is not enough.” For the Strunk and White fans out there, I would like to be clear that I did not accidentally start a sentence with number which I failed to spell out completely. Rather, it’s an ill-considered rule that I gleefully violated.