Seriously?

Malware stuffed into a USB battery charger?

This is just as bad as a trojan on a key fob and… smart phone? . Actually, this is worse because people don’t expect utility to run.

Is nothing sacred?

I’d just like to point out, being a big Mac-fan, that it would have been trivial to write an exploit Macs with this as well. Though it can be frustrated by enabling the Firewall. But face it, nobody (translate to not many) does but me.

Lessons lessons

What’s the lesson here? Don’t trust a battery company software? Don’t use autorun? I’m sure Microsoft thought that it sounded like a great idea way back in the day.

You auto-be-running

Apparently, autorun has been renamed to autoplay (<sarcasm>Where the heck have I been, what a change!</sarcasm>).

According to KB967715 they had it disabled in XP prior to SP2. Now it’s enabled for XP SP2+, Vista, and 7. However, for Vista and 7 users, it asks you if you want to auto-run it first.

Trust

Again, it’s about trust. Do you trust Energizer to make you software? Apparently not any more. Even if you have auto-run disabled, it appears as though you can still be compromised because you trusted the application to run. So, a Mac is just as vulnerable in this respect.

So the next time a peripheral decides to offer you friendly software: just say no!

Left unsaid

I’m sure there’s tons of room for comments such as: “It’s the manufacturers overseas!” and, while that MIGHT be true, it would be much better if we didn’t trust all these random devices and gadgets we have.

The error message

/public/../config/../vendor/rails/railties/lib/initializer.rb:271:in `require_frameworks': RubyGem version error: rack(1.0.0 not ~> 1.0.1) (RuntimeError)
	from /public/../config/../vendor/rails/railties/lib/initializer.rb:134:in `process'
	from /public/../config/../vendor/rails/railties/lib/initializer.rb:113:in `send'
	from /public/../config/../vendor/rails/railties/lib/initializer.rb:113:in `run'
	from /public/../config/environment.rb:9
	from /public/dispatch.fcgi:21:in `require'
	from /public/dispatch.fcgi:21

Old news?

If you have this problem, odds are you’ve been pouring over Google and you’re here because you have not resolved (comment #7) your problem. I’ve seen the “solutions” out there and they all claim: “Remove rack 1.1.0. Install rack 1.0.1”. I tried it and it didn’t solve my problem. The trick is, my error message was different. Also, most others are having trouble with WEBrick. I was having issues with lighttpd. I tried everything: commenting out line 23 in action_controller.rb (bad idea, by the way) in my vendor/rails/actionpack/action_controller/lib/action_controller.rb. Added gem ‘rack’, ‘= 1.0.1’ in dispatch.fcgi (moved the error message, but didn’t fix it).

The solution

Manually remove the residual rack-1.0.0 installation.

I will assume you’ve already tried:

sudo gem uninstall rack

Do be careful!

sudo rm -rf /opt/local/lib/ruby/gems/1.8/gems/rack-1.0.0
sudo gem install rack --version "= 1.0.1"

What happened?

The problem was that rack-1.0.0 wasn’t uninstalled completely. RubyGems thought it was, but it was still being included in the code. Why? Because it existed here:

/opt/local/lib/ruby/gems/1.8/gems/rack-1.0.0

wojno% gem list | rack
rack (1.0.1)

I don’t know how it was stranded there. I only use gem to add/remove gems, yet somehow it was left there to rot my brain while I’m under deadline.

Fun!

Keep in mind, I use MacPorts and keep it fairly updated.

A Pavlov doggy gets his treat!

NOTE: This is a disproven thought experiment: unable to prove guessed password succeeded and pass the information along.

Prompt

I’ve just finished reading (yes, the whole thing) an article about securing Rails applications. While it talks about security with Rails specifically in mind, it isn’t limited to just Rails applications. It is a good read for anyone who creates, maintains, or audits web applications of any flavor. You can read the Ruby on Rails Security Guide now or later if you like. I’ll explain as much as necessary to get my point across for this article.

One section in particular concerns Cross-Site Request Forgery attacks. According to the article, such attacks are carried out by placing HTML (yes, HTML) pointing to a specific action in a Rails application. The article used the example of using an image to delete critical information from a target site. The article continues to explain that one can use javascript to create a form that a user mistakenly clicks to use POST attacks instead of simple GET requests. (I have figured out a way to modify this so clicking is not required below)

This method of attack got me thinking. I had never specifically thought of doing this to attack my own sites. It makes sense though and naturally follows from the application of the semantics of the HTTP protocol. It also got me thinking of how one might use it to further exploit a site.

Is it worth it?

I believe it is. If using no more evidence that it exists as chapter 3 in this article of 8 regarding security measures, the practical result is that, unmitigated, your application’s security is based upon the security of other, well-traveled applications. The article continues to explain a simple fix as provided by Rails. The ONLY fix is the protect from forgery secret. I haven’t evaluated in depth exactly how effective it is, but I’m currently cooking up a new attack based on the CSRF vector.

What’s your Vector Victor?

Security is no joke. I believe I’ve come up with an attack using the CSRF vector that could potentially be devastating. There’s really no one to report this to as it applies to every site that is vulnerable to this type of attack. I believe it should be called the CSRF-MPDBF: The Cross-Site Request Forgery Massively Parallel Distributed Brute Force attack.

How does it work?

It uses the POST exploit as discussed in the article using javascript to commandeer the computers of any visitor to try a random password or two and report success to another site using the same CSRF exploit.

The Set-up (Huh? No really, how would that work?)

Let’s presume there are 2 sites: A and B. Site A is some website for which the attacker wants login credentials. The attacker may, or may not know the login name of a user. Having it certainly helps the attacker, but it’s not strictly necessary for this exercise. Site B is a forum with lose security policies that allows javascript code to be inserted into pages by users. Examples aren’t limited to forums, but a multitude of other “public service” sites.

Presume that Site A has a simple log-in form, with or without SSL; encryption is moot. Also presume that this site doesn’t use request forgery protection. It’s just a simple login for such as:

<form action="/security/login" method="POST" />
<p><label for="username">Username:</label><input name="username" type="text" /></p>
<p><label for="password">Password:</label><input name="password" type="password" /></p>
<p><input type="submit" value="Login" />
</form>

Site A will look at the username and password in the POST body and do a look up of the username, then see if the password matches. I’m not talking about SQL injection here. Presume it’s a Rails application and it’s using the sanitize_sql functions correctly. If the login is successful, the URL will change (a redirection will occur), otherwise, presume the application displays the same message (same URL) without redirecting. Most applications redirect after login. But it is possible to confirm login another way. This is just one example. The key point is that an attacker is able to reliably confirm a successful login (and conversely, an unsuccessful one).

Are we there yet?

Sorry, let’s get to some steps of the attack:

The attack code

The first step is for the attacker to create special javascript code as in the above article with one key difference: target the login script and include a username and GUESSED password.

Say what? A GUESSED password?

Yes, guessed. I told you that this was a brute force attack. Javascript includes a random number generator:

Math.random()

But that will never work!

Oh really? Why not? Every javascript engine is different. No browser and no visit to the page will likely use the same password. If you post some comment to a popular site and get a lot of people to trigger this, the target site A could be compromised in a matter of days, if not hours.

Dude, where’s my code?

FIIINE! This javascript is inserted on site B to attack site A (modified from the article: “On Rails Security”

<a href="http://www.b.com/" onmouseover="
  var f = document.createElement('form');
  function passwordGuess() { /* ... details here ... */ }
  f.style.display = 'none';
  this.parentNode.appendChild(f);
  f.method = 'POST';
  f.action = 'http://www.a.com/account/destroy';
  var x = document.createElement('input'));
  f.appendChild(x);
  x.setAttribute('name','username');
  x.setAttribute('value','TARGETUSER');
  x = document.createElement('input'));
  f.appendChild(x);
  x.setAttribute('name','password');
  x.setAttribute('value',passwordGuess());
  f.submit();
  return false;">To the harmless survey</a>

Here’s how it works. I added a stub function for the password guessing routine as that is not what is at issue here. This code is to be injected into a forum post or SQL injection or whatever vector such that it appears to public users of site B. When the javascript is called, on click, it creates a form in HTML. This is a bit strange. Javascript modifies the DOM to add a form. The form is then submitted as though the user had submitted a form from site B. But server A doesn’t know the form exists on site B and processes the username and password as though the request came from site A. This, my friends, is effectively a login attempt. The user is completely unaware that anything occurred. It’s executed when the hover over the link and when they click, the link works as desired. This is different from the original article as it worked on click. The unwitting user at site B cannot detect that this attack has occurred unless he or she inspects the traffic. What normal user does that?

2010/02/01 – Editor’s Note: I’m trying to get this to work, I cannot seem to get the status of a password guess using Javascript. This might be a hair-brained idea after all.

OK, but the attacker can’t get that random password!

Not true. Use the same attack vector. Presume site B or some other site C is vulnerable to the same type of attack. To deliver the credentials back to the attacker simply modify the javascript to detect if the page was redirected. Once the credentials are verified, post the brute forced credentials to a public location. An attacker could even use the same type of attack to post those results. The attacker simply checks the destination site for updates and snarfs the credentials. Then the real fun begins and I don’t have to explain what can happen at this point.

Impact?

Well, think of it in simple terms. Every browser visiting site B’s compromised page becomes a weapon. Each person that goes to this page on site B with the code contributes to the attack. Assuming site B is very popular, but lax on code insertion filtering, site A will be attacked x times, where x is the number of users that trigger the attack on site B. If the attack runs m times (multiple attempts), then site A will be attacked x*m times. If B is a popular site, then x can easily approach millions of triggers in a period of a few weeks. Assuming a small dictionary can be loaded, a clever attacker could author the script to try word combinations, or insert symbols between, before, and after words, try numbers, etc. If the username is known in advance, it’s only a matter of time before that account on site A is compromised by visitors on site B. If multiple sites with B’s lax security exist, then the attack is: m*(x₁ + x₂ + …x_n) where x is set sites like B to which the malicious code is posted.

It’s sort of like creating a bot network that only exists when users go to a specific page or pages.

But it’s a random password, won’t the same password be tried again and again?

Yes. There will, undoubtedly, be collisions in the the attack space. The key is that, depending on the javascript random engine, there will be significantly greater quantities of untried attacks. Even pseudo-random number generators try to be difficult to predict. In this case, this randomness works in favor of the attack by created more unique generations. A super-sophistocated attack could post failed results to a forum. These results could then be compiled to a random number black list that will skip over previously failed trials. That would significantly reduce the collisions (not eliminate) but is more sophisticated and again, if the PRNG has a decent algorithm, won’t be necessary.

Proof of concept?

Ha, no way. I would not write one in the wild.

Has anyone else thought of this?

I suppose I should have looked this up before I started writing. However, writing down an idea helps solidify the thought. I didn’t find anything with a quick Google Search but that isn’t thorough research. Most of the hits involve the tokens getting brute forced. My example assumes there are no tokens. But you can see how even brute forcing the tokens leads to slowing the progress of an attack.

Why is this so bad?

Because there’s no one IP source to block. It’s distributed and not easily detectable. Couple this with a password dictionary somewhere and you have a serious attack. Couple it further without the need to click something to trigger the attack (below) and it’s even more wide-spread.

No Clicking Required

The article gives the example of javascript code that creates a form, then replaces a link a user would normally click with the form such that it performs the CSRF attack. It is possible to create an exploit using the same vector but not require a click to activate it. Simply create a form as indicated in the article with my modifications, then write a script to periodically execute it asynchronously and without redirecting the user to the login page. This hides the attack and permits multiple guesses; as many as can be squeezed into the time the user is on the B site.

How bad is it?

I would imagine most of the big sites already mitigate this attack using forgery tokens. While no security expert, I believe that Facebook is using them based on their login form. I’ve removed some of the worthless table formatting, added line breaks, and only used the HTML form fields. For some reason, the field is listed as having the id=“lsd”. I have no idea why.

<form method="POST" action="https://login.facebook.com/login.php?login_attempt=1" id="login_form">
<input type="hidden" name="charset_test" value="&euro;,&acute;,€,´,水,Д,Є" />
<input type="hidden" id="locale" name="locale" value="en_US" autocomplete="off" />
<input type="hidden" id="non_com_login" name="non_com_login" autocomplete="off" />
[...]
<input type="checkbox" class="inputcheckbox " id="persistent" name="persistent" value="1" />
[...]
<input type="text" class="inputtext" title="Email" placeholder="Email" id="email" name="email" value="" />
<input type="password" class="inputpassword" id="pass" name="pass" value="" />
<input type="text" class="inputtext hidden_elem DOMControl_placeholder" id="pass_placeholder" name="pass_placeholder" value="" />
<input value="Login" type="submit" class="UIButton_Text" />
<input type="hidden" name="charset_test" value="&euro;,&acute;,€,´,水,Д,Є" />
<input type="hidden" id="lsd" name="lsd" value="KboH0" autocomplete="off" />
</form>

The 5 characters seem to change from browser to browser. It appears to be generated, then persisted using cookies (cookie name: lsd).

The real problem will be with smaller sites that are written with custom code that do not use the forgery prevention tokens. I cannot even begin to guess how many sites are affected.

“And what was the real lesson? Don’t leave things in the fridge.”

Spike may have been onto something there (quote is from Cowboy Bebop, “Toys in the Attic”). Don’t get soft on your security. As you can see, we’re all in this together. Your weakness is someone else’s exploit vector. You won’t just become a statistic, you’ll be actively, albeit unwittingly, contributing to an attack on other sites at the discretion of less scrupled persons.

Read the article! Don’t leave your applications in security cold-storage. It’s important.

Disclaimer

Again, this is an informational article to help site programmers build more secure sites. I am not responsible for people that misuse these ideas or examples.

I’ve finally updated the Typo software for my blog. The site started running funky; as in not letting me edit articles. It was a big version jump: 4.1 to 5.4.1. I followed the directions and only had one snag: Rubygems installs EDGE Typo (which was broken). But I had a backup, so I just rolled back and all was well.

But it would be nice if the rubygems did NOT install the edge version, but only included stable releases.

BE WARNED

The rubygem: “typo” will update to edge (unstable). You should always be sure to check the typo site: http://typosphere.org to be sure you specify the version when updating your ruby gems:

sudo gem install typo --version 5.4.1

This is just an example, replace with your desired version.

Background

Lately, I’ve been updating my Mac and Linux systems (AS I SHOULD!) and I’ve been getting errors for the passed few months from my Rails-based applications. They’ll be 500 errors (server error).

The Problem

It seems that a Ruby1.9 (from Ruby1.8) update has added the function “chars” to the core String class. Unfortunately, Rails prior to version 2 (I have not had this issue with newer, Rails 2 projects, I can only assume it’s not an issue: consider it unconfirmed) uses this method/attribute in a Rails ActiveSupport extension that allows you to slice up strings in an easier way.

Specifically, the ActiveSupport library assumes that the String class does NOT define the “chars” function. However, with Ruby1.9, it does. This leads to a conflict of types. Rails wants to use an attribute (as defined with ActiveSupport), Ruby1.9 is offering a method. The method wins and Ruby thinks the result of “chars” is an enumerator when it’s really expecting a String. Under Ruby 1.8, String included Enumerable as a mixin module. In 1.9, String no longer includes this.

The Rails team saw a need for the method chars and created one in activesupport/lib/active_support/core_ext/string/unicode.rb, then the Ruby core team added their own to the string class. Thus the conflict and thus the error.

The Solution

I picked this up from various websites reporting the same issue. They’re just not easy to find. You need to insert this code into the config/boot.rb file in the affected application. I put it at the top.

# Fix for ./script/../config/../config/../vendor/
#rails/activerecord/lib/../../activesupport/lib/
#active_support/core_ext/string/access.rb:43:in
# `first':NoMethodError: undefined method `[]' for
##<Enumerable::Enumerator:0x103767ec0>
unless '1.9'.respond_to?(:force_encoding) 
String.class_eval do 
    begin 
      remove_method :chars 
    rescue NameError 
      # OK 
    end 
  end 
end 
# /Fix

It effectively removes the function “chars” from the String class that is added in from Ruby1.9. Hopefully, the Rails equivalent is the same and as or more secure.. But only does this if the String class has the method “force_encoding”. Why force_encoding? because it appeared in 1.9 along with “chars.” That’s the only link. So, should they remove “force_encoding” later, this fix won’t work any more.

Suggestions

Upgrade Rails to 2.0+ That should fix you up. Or avoid Ruby 1.9… I don’t recommend that though. They’ve isolated some of the core functionality into libraries. Ruby should have less of a memory footprint. I’d imagine there are security fixes .

It is also unfortunate that this situation occurred. I’m not sure how it can be avoided in the future. Things get updated. A little warning about them would be nice, however I’m actually amazed it hasn’t happened already. Kudos to the Rails/Ruby team.

Background

I have a couple of Mac workstations at the office. One is used extensively for photo editing (CS4). At any rate, I’m trying to set up a new e-mail account.

The Symptoms

Even though I entered the new password correctly SEVERAL times, it never worked when “Remember password in keychain” was selected. I thought it was a fluke.

I tried setting it up again today. However, I spent almost an hour tracking this one down. I noticed that, after editing the account settings that a dialog window would appear and claim that it was not possible to save the settings. It claimed that the permissions on ~/Library/Preferences were not correct. I heeded it’s warning well.

The Investigation

I popped open my swiss army knife: AppleKey + Space “Terminal” [return]. There, I looked at the file permissions for

ls lah

The seemingly coincidental success with not using the “Remember my password in keychain” got me thinking; I tried to add a new item to the keychain. The login keychain. I was dumbfounded when greeted with the following message: “An error has occured. Unable to add an item to the current keychain… [Unix Permissions]”. What the heck was wrong? The owner is right, group is irrelevant, and the permissions mask was 600.

The Discovery

On a hunch, I did: “ls -leh” on a suspect directory. And there it was:

drwx------+

The + at the end of the permissions mask isn’t just for show. It’s not a sticky bit either. It’s the extended attribute for ACLs. Yes. Macs use ACLs. There’s almost no information about this online. You can get more information by using the man pages:

man chmod

And I saw it:

  0 group:everyone deny delete

There was one other ACL for domain administrators. But this one was it. Macs, Unix, and Linux machines in particular typically use the write to temporary and copy-back method of writing files. It helps reduce corruption and in this case, probably enabled the system to run as long as it did without major incident. Sadly, with this technique, the final copy, which is actually a move, never completes successfully. The result is that preferences for “well behaved” applications fail to save. Keychain wouldn’t update either due to the same principle.

The Solution

You have to eliminate that one particular ACL in the ~/Library folder. Doing so enables you to write again. Denies always over-rule the allows.

chmod -R -a "everyone deny delete" ~/Library

Remember, I’m not responsible for what happens if you use that. Do your research and be sure yourself.

The Aftermath

Keychain now accepts new items. Mail now accepts new passwords. Preferences save without incident.

This entire episode was the result of ACL’s being improperly applied. I’m not sure how it happened but I do have a guess.

The Possible Reason

About a month back, we tried a tablet. It was a WaCom Bamboo tablet. The software was designed for CS3. We tried it with CS4. That was a mistake. To be fair, it never said it worked with CS4. It said CS3 and then stopped at that platform. I goofed.

CS4 became extremely unstable. Photoshop crashed many more times each day than normal. InDesign refused to open at all. Uninstalling the suite using the uninstaller and Adobe’s secret clean-up script (which told me that it took no action) did not fix it. I had to remove all the preferences that bore the mark of Adobe (anything with “adobe” in the name of the .plist file or folder). I reinstalled it and it worked as though it were brand new again (after updates).

However, this was not before I performed a “Disk Permissions Repair” as suggested by forums. This utility is located in your Applications > Utilities > Disk Utility under your Mac drive. I believe that doing this action resulted in the mis-placed ACL’s.

I hope this helps someone who is or has suffered this problem before.

Overview of Xen

If you’re here, odds are you’ve heard of virtualization and, specifically, xen. Xen is an amazing piece of technology. So amazing that a company named Citrix has created a product sourced from it: XenServer.

As an additional bit of background, I love the Linux distro: Gentoo. Gentoo is distinguished because of it’s package system: portage. Wouldn’t it be nice if one could combine the amazing Xen with the amazing Gentoo? Turns out, it’s really hard.

One of the primary draws to Xen is the ability to do what’s called a migration. And to boot, the ability to do a live migration, that is, moving a virtual machine from one physical host, to another without powering the virtual machine down. This has enormous consequences for maintenance of physical hardware. Why schedule downtime when you can avoid it all together?

So Xen, the hypervisor, provides drivers to the guest instance (also known as the domU (for unprivileged domain, or domain… unprivileged)). While Xen can host just about any operating system, in order to take advantage of the special features of XenServer, such as memory usage statistics and, of course, migrations (including live migrations), you need to have a paravirtualized operating system (PV).

With a paravirtualized operating system, the hypervisor (that’s Xen remember), boots the guest operating system and installs a special device and a place to put hypervisor information as follows:

• /proc/xen
• /sys/hypervisor

Goal

So the goal should be clear by now: make Gentoo Linux a paravirtualized guest domU in Citrix XenServer5.5.

How to do it

Warning

Remember, it’s not easy. You should know how to handle Gentoo and XenServer.

Also: this is based on my notes of my experience. I am not responsible for any damages frustration or any other liabilities incurred by using the steps in this article. I am not responsible for how this information is used (or not used).

This “guide” is extremely raw. Only the major steps are covered and some details have been left out. This is not a step-by-step, testedly proven guide. It’s assumed you know how to edit files and navigate your way through a kernel compilation and installation, most of which you can get from the Gentoo Handbook, though.

The good news, I have created working paravirtualized instances of Gentoo. If that’s your goal, you might pick something up.

Requirements

You’ll need to have working before you start with the remainder of this guide:

• XenServer5.5 installed on 2 physical servers
• XenServer5.5 configured with shared storage, such as CIFS (yuck), NFS, or, even more special, iSCSI
• XenCenter with the above XenServer’s in a server pool (Yes, sadly, this one requires a Windows operating system)

The Overview

1. Setup your shared storage
2. Install XenServers
3. Install XenCenter (but you’ve done this already)
4. Install all networking as per your requirements ;-)
5. Get a Gentoo install CD (I guess any install CD will do)
6. Create your VM
7. Install Gentoo
8. Get the Xen linux kernel
9. Compile your kernel
10. Install the bootloader
11. Configure /etc/fstab
12. Install Citrix Xen Tools
13. Configure the VM in Xen

Setup our shared storage, XenServers and XenCenter

I’m not covering this. Citrix has plenty of documentation and, frankly, I didn’t read most of it. Just install it and read the docs when you have questions about options. This is by far, the easiest part.

Get a Gentoo install CD

Now, you don’t NEED a Gentoo-branded install CD. I used it as I followed the directions at the Gentoo site. You can find a Gentoo install CD here:

http://www.gentoo.org/main/en/where.xml

Keep your arch in mind. Mine was EM64T (aka: Intel64, AKA: amd64) ¹, which means I had to download the amd64 architecture version. If you don’t pick this, you’ll have to cross compile and that’s just ugly.

So. Go ahead and download it from a mirror near you. No need to burn it to a CD/DVD. XenCenter can mount ISO’s directly via the ISO Library shared storage. If you don’t want to set that up (but WHY? It’s so useful!), then go ahead and burn your precious CD and then pop it into the server.

Provision the Virtual Machine

From within XenCenter, provision a new virtual machine from the “Other” template. Create a 10GB virtual disk. Believe me, 10GB is pretty much the minimum.

Mount the Gentoo install CD as the CD drive in the new virtual machine.

Fire up your virtual machine

Now, follow the directions to install Gentoo until you reach the Kernel part. Remember, pick the install guide for your architecture. The guides are mostly the same (with a few differences) anyway. Remember, install everything as described in the manual until you get to the “Configuring the Kernel” part.

Get the Xen kernel

Now, when I said I wanted Gentoo, I really want Gentoo’s package manager. In order to create a PV (paravirtualized, remember) kernel, you need the Xen drivers. OOOOOK. So you need to go get those. But the only way to do that is to use a Xen-enabled Kernel. Bottom line, you can’t use the gentoo-sources kernel. I tried, but I couldn’t get it to recognize the paravirtualized devices.

So, let’s download the Xen-kernel from xen.org. Download it into your chrooted Gentoo install. You can find the kernel at http://xen.org/download/ Snag the “Linux 2.6.18 with Xen 3.4.x support source tarball” or later version if available. I highly recommend wget.

cd /usr/src/
wget "http://bits.xensource.com/oss-xen/release/3.4.0/linux-2.6.18-xen-3.4.0.tar.gz"
gzip -d linux-2.6.18-xen-3.4.0.tar.gz
tar -xf linux-2.6.18-xen-3.4.0.tar
rm linux-2.6.18-xen-3.4.0.tar
ln -s linux-2.6.18-xen-3.4.0 linux
cd linux

Now it’s time to configure your kernel. I’m not sure why, but there are TONS of extraneous options. You really only need the Xen options. Just do a search for Xen and make sure they all say “y.” However, since we’re building a domU instance, you don’t need any that say: “backend.” You must configure your processor options, however. Pick your architecture type, configure your SMP options. You really don’t need the Kernel Hacks. I said no to them all. Be sure to configure your file-system drivers. If you used the Gentoo instructions (as you should have thus far), you’ll need to include ext3 support. You do not need ext2 support, but I recommend it (at least as a module) as you’ll need to edit your boot partition if anything is wrong. Once you’re done with that, go ahead and compile your kernel.

make && make modules_install

You may now resume the Gentoo install instruction until you reach the part about bootloaders.

Install the Bootloader

Now that you’re here for bootloaders, I’ll tell you right off that it’s pretty much the same as the Gentoo installation instructions. There’s a few caveats with the kernel command. Follow the instructions from the Gentoo installation, but use the following for the kernel line in /boot/grub/grub.conf (aka: menu.lst):

kernel /boot/YOURKERNELFILE root=/dev/xvda3 xencons=tty console=tty0

So that says that you installed your kernel at /boot/YOURKERNELFILE (from the Gentoo kernel installation instructions, you should have copied your compiled kernel from arch/x86_64/boot/vmlinuz to /boot/YOURKERNELFILE. Of course, you probably didn’t name it YOURKERNELFILE and named it after what it is, like: xen3.4.0-linux2.6.18.

The root= should point to the file system where your root “/” directory is stored. You have to use the device path: “/dev/xvda3” and not /dev/hda or /dev/sda (first, they won’t exist and if they do, will cause your file systems to corrupt). Consequently, “/etc” needs to be readable. Xencons and console is required to tell XenServer how to talk with the PV guest².

Go ahead and finish the Gentoo install. Before you reboot, see the next step in this article

Configure /etc/fstab

Your disks will appear as /dev/xvdAN where A is the letter of the “disk” which is really your shared storage as provided by XenServer. N is the partition number. You should have 3 partitions if you followed the Gentoo instructions. Make sure you edit your /etc/fstab to use those devices rather than /dev/hdAN, and if you use /dev/sdAN, you’ll cause your disks to encounter errors when doing lots of writing. Believe me. You’ll notice that your disks will re-mount as read-only. If you see this, STOP using /dev/sda! /dev/hda works, but again, to fully paravirtualize, use /dev/xvda.

Go ahead and finish the Gentoo installation. Go all the way until you have to reboot. Don’t reboot yet.

Install Citrix Xen Tools

First, download the xen source from xen.org: http://bits.xensource.com/oss-xen/release/3.4.1/xen-3.4.1.tar.gz

Download that file by:

cd /usr/local/src
wget "http://bits.xensource.com/oss-xen/release/3.4.1/xen-3.4.1.tar.gz"
gzip -d xen-3.4.1.tar.gz
tar -xf xen-3.4.1.tar
rm xen-3.4.1.tar
cd xen-3.4.1
make tools

You’ll see errors about firmware. Ignore them. You only care about tools/xenstore. And by the time you see the firmware errors, it will be built (or should be at the time of this writing).

Copy the xenstore executable to /usr/sbin
Copy all the libxenstore.so.* files to /usr/lib

cd tools/xenstore
cp xenstore /usr/sbin
cp libenstore.so* /usr/lib

Now, mount the XenTools ISO provided by XenServer (it’s a “CD” provided by XenServer. Access it via the VM configuration the XenCenter console).

Before we begin, you must emerge the RPM utility. Yes, this Red Hat’s package management, but Citrix does not yet provide an ebuild. There is a Gentoo port for RPM though, so we’ll use that:

emerge rpm

Once that finishes, navigate the XenTools CD to the Linux directory. There you’ll see the utilities. Unpack the one in the Linux folder:

rpm -i --nodeps xe-guest-utilities-5.5.0-458.x86_64.rpm

—nodeps is required as we have all the dependencies, but we don’t want the RPM system to manage our software.

When the package is unpacked, you’ll see a new script in /etc/init.d/xe-linux-distribution

rc-update add xe-linux-distribution default

to run this at boot, but first, we need to fix it.

You’ll have 3 more script files in /usr/sbin:

• xe-daemon
• xe-linux-distribution
• xe-update-guest-attrs

Edit xe-linux-distribution. Add the following function below the identify_lsb() function (don’t put it IN the function, put it BELOW the function).

identify_gentoo()
{
	gentoo_release="$1"
	if [ ! -e "${gentoo_release}" ] ; then
		return 1
	fi
	distro="gentoo"
	eval $(cat ${gentoo_release} | awk '{ print "release=" $5 }' )
	if [ -z "${release}" ] ; then
		return 1
	fi
	eval $(echo $release | awk -F. -- '{ print "major=" $1 ; print "minor=" $2 }' )
	if [ -z "${major}" -o -z "$minor" ] ; then
		return 1
	fi
	write_to_output "${distro}" "${major}" "${minor}" "${distro}"
}

(Editor’s note: Line 12 edited, added a space after “—”)

Next, you should add the following line

identify_gentoo "/etc/gentoo-release" && exit 0

put it just below the following line:

identify_debian /etc/debian_version && exit 0

That’s it. It should work. The /etc/gentoo-release file has the details about the Gentoo kernel you’re using. All this script does is read that file and print out the details in a format other Citrix tools use. Shutdown the Gentoo VM and move the next step in this article.

/etc/init.d/shutdown

Configure the VM

Make sure the Gentoo VM is powered down. Go to a XenServer console (accessible from XenCenter). This is different from a VM console. You access this by selecting a server and then the console tab. Any server will do. The XenVM database is shared in XenCenter (if you have shared storage that is, which you need for this article).

You’ll have to get the uuid of the Gentoo VM first. You can see a list by doing:

xe vm-list

Record the uuid for future reference. Next, you need to set the VM boot-up options. This let’s Xen boot your kernel. Exciting right!?

xe vm-param-set uuid=UUID PV-kernel=/boot/kernel PV-bootloader=pygrub PV-args=xencons=tty HVM-boot-policy=""

(Editor’s Note: uuid is not a flag, but an argument)

The above sets the kernel for the bootloader, sets the xen bootloader for the domU kernel (gentoo) and clears the HVM-boot-policy (allows for ordering the disks before boot, we don’t want this as this prevents Xen from booting your machine paravirtually).

Now, make the disk created for this VM bootable.

xe vm-disk-list vm=NAMEOFVIRTUALMACHINE | less

You’ll see a list of disks. The disk type will be a VDI. Record the uuid for future reference. Hit “q” to exit less Now, make the disk bootable:

xe vbd-param-set bootable=true uuid=UUIDOFPREVIOUSSTEP

You may now boot the machine as a PV and it should start without issue. If you need to troubleshoot, create a second VM (using the “other” template) and boot from the Gentoo CD. Detach the storage from the server you’re configuring (that doesn’t work) and attach it to the new “Rescue” VM. You can mount the drives and chroot to configure the operating system environment, if necessary. YOu can even re-build the kernel. However, you cannot yet migrate the VM, that is until you install Citrix’s tools.

You should now be able to see memory usage, network interfaces, migrate the machine, and suspend the machine, once you start it that is.

Conclusion

That’s about it. Your Gentoo domU guest should not only be restartable, but migratable! I reconstructed these instructions based on what I did to get one guest instance built. I may have missed a few things.

Disclaimer

Citrix is the Trade Mark of Citrix Systems, Inc. I am not affiliated with Citrix Systems, Inc. in any way.

Remember, I’m not responsible for your use or misuse of the information in this article.

Lack of documentation

When trying to do this myself, I was and still am stymied by the lack of documentation about paravirtualization kernels. Even Xen.org is pretty useless when it comes to documentation. They cover how to make a dom0 (which is important, I admit), but fail to cover how to construct that kernels for the ultimate goal of Xen: running virtual machines.

There is nothing from Citrix about xenbus or xenstore. This entire guide was created from trial and error. Believe me, the bootloader errors are not fun to deal.

¹ Yeah, confusing. Ain’t it?

² If you see a on Gentoo domU boot about the Kernel Freeing Unused Memory and then seemingly stalling, odds are, you forgot to tell Xen about the console. Your OS is running, you just can’t speak with it, which makes it useless.

This article is part 2 of Going SSL with Your Own Root CA

Putting your Root CA to work

Now that you have your own Root CA, it doesn’t do anyone any good unless you use it. You use it by creating certificates for services. The rule is one certificate per-IP address. If you need additional certificates, you should have an IP address for each certificate. Any domain (or sub-domain, etc.) listed in the Certificate’s CN (Common Name) should resolve to its own unique IP address. Now, nothing is set in stone. But that’s the way it should be done. I’ve used multiple certificates for different domains on the same server using Virtual Hosts (same IP) before, but Apache warned me every time I started the server that this is bad practice. If you’re using Apache, it will still work, but you’ve been warned (again).

Disclaimers

This guide is for amateur, non-production use only. I make no warranties or guarantees as to the correctness of this document. This guide involves escalating privileges and may cause irreparable damage if used improperly. You use this document at your own risk and I hereby disclaim any responsibility or liability for your actions or non-actions.

Assumptions

I’m assuming you’re using Apache 2.X with mod_ssl already installed. I’m also assuming you’ve already set-up a vhost file with the site you want to SSL already created and configured minus the specification of the server certificate, and trust chain. I realize this is a lot to assume. This can help you setup an Apache server with SSL.

Creating a Server Key

Just as with the Root CA key, you need a server key. Now, if you already have one, relax, skip this step (though I still urge you to create and migrate to the “secure” environment created in the next step). You can re-use your server key as many times as you like until you want to upgrade the private key quality (or the key is compromised). If you have not created a server key, you’ll need to do so now.

We’ll use OpenSSL again, but first, let’s create a safe place for our key. The key needs to be readable by the server upon start-up (meaning, the key cannot be encrypted) or Apache (or any other server) will be unable to use it without first asking you for a password. If you don’t mind typing it in each time, that’s your prerogative. However, for live or even development environments, it’s simply impractical.

Just as with the Root CA key, you must keep this server key confidential as well. If it is breached, anyone can masquerade as your web server; thus defeating the purpose of the certificate. If you need to move the key, you are advised to use OpenSSL to encrypt it first, however, that is a tutorial for another day.

Create the “secure” environment

Create a comfy place for your key. I like to put it in /etc/apache2/ssl with Gentoo, but the better place is likely /usr/local/etc/apache2/ssl and if you use Windows: there are no rules for file placement. With that: create your directory:

sudo mkdir /usr/local/etc/apache2
sudo mkdir /usr/local/etc/apache2/ssl
sudo chmod 0770 /usr/local/etc/apache2/ssl
sudo groupadd apache_ssl_admins
sudo chown :apache_ssl_admins /usr/local/etc/apache2/ssl
sudo usermod -a -G apache_ssl_admins YOURUSERNAME # varies by OS

Here’s what I’ve done:

1. Created an apache2 directory (if it didn’t exist, if this call fails, just make sure the directory is there)
2. Created the SSL directory where we’ll store the server key
3. Changed the permissions for the directory such that only root can access it
4. Created a group called “apache_ssl_admins”
5. Changed the ownership of the new directory to allow members of the apache_ssl_admins access
6. Added you to the new group called “apache_ssl_admins”, note that this step may vary among Operating Systems so modify this command as required.

You’ll need to log out and log back into your server for the group membership to take effect.

Now, I’ve put the word “secure” in quotes as if anyone were to compromise the server and gain access to the apache_ssl_admins group, your key is then accessible. Your key is only as secure as your server.

Once you’ve logged back in, cd to the newly created “secure” environment.

cd /usr/local/etc/apache2/ssl

Creating the server key

You will now generate a server key. I highly recommend that you study the documentation for OpenSSL. It’s very very rough, but it will help you understand the commands you’re entering. It may be necessary to update these commands as machines become faster and the encryption level becomes insufficient.

openssl genrsa -out www.pem 2048

This generates a new file called www.pem in your “secure” environment. This is an RSA private key with a bit length of 2048. This bit-length is considered to be good by today’s standards. You may use any power of two; though, be warned, larger numbers require more time to initiate SSL connections. Smaller numbers will result in a less secure SSL handshake (and vicariously a less secure SSL session).

Again, it is a good idea to encrypt this key with 3des. I’ve not done so to facilitate this demonstration. Please see the previous tutorial for instructions about encrypting your server’s private key. The instructions for encrypting your Root CA private key are applicable to this purpose.

Generate a Certificate Signing Request

The Certificate Signing Request (CSR) is used by Root CA’s to create certificates for people who need them. A CSR is a uniform way to telling Root CA’s exactly what the certificate should say. The Root CA merely signs the CSR and viola: a certificate. Well, it’s not all that simple. The Root CA also adds an expiration date to the signature or other fields deemed necessary or desired. But before it can be signed, the CSR must be generated by using the server’s key:

openssl req -new -sha1 -out www.csr -key www.pem

This command will create a file called www.csr in your “secure” environment. This file is a Certificate Signing Request. You may expose this file publicly, though that is not required if you’re signing it yourself. This new request requires that it be identified using the SHA1 hashing algorithm. This algorithm is safer than MD5 as it has been proven that MD5 collisions can be generated in a reasonable time.

You will be asked a series of questions after running this command. Answer them honestly. When you see the CN (Common Name), enter the domain name and subdomain for your certificate. I.E. for my domain: christopher.wojno.com, use as the CN: “christopher.wojno.com”

Sign your CSR and create a real server Certificate

Now that you have a CSR, we’ll use our Root CA certificate and key to sign the CSR. This creates a certificate you can use in your Apache server (or other SSL-capable web server). If you created your Root CA using my previous article, then you may use the paths I have specified here. Otherwise, you’ll need to adjust the -CA, -CAkey and -CAserial parameters to match what you have used.

openssl x509 -req -days 365 -in www.csr -CA /usr/local/etc/certificate_authority/certs/cacert.crt -CAkey /usr/local/etc/certificate_authority/private/cakey.pem -CAserial /usr/local/etc/certificate_authority/serial.srl -out www.crt

This is a lot to swallow in one step. This used to be where I became completely lost with regard to SSL Certificate generation. The command is very complex so I’ll break it down part by part:

• We’re asking OpenSSL to do something
• Specifically, we’re asking it to invoke it’s x509 library (SSL certificate chains)
• We’re then specifying that our input will be a CSR (-req)
• We want the resulting certificate to be valid for 365 days (-days 365) from date of signing
• The CSR is the input (-in www.csr)
• We’re using the CA public certificate created in the previous tutorial: (-CA /usr/local/etc/certificate_authority/certs/cacert.crt)
• We’re using the CA’s private key created in the previous tutorial to sign this CSR: (-CAkey /usr/local/etc/certificate_authority/private/cakey.pem)
• We’re using the CA’s serial file to mark the resulting certificate: (-CAserial /usr/local/etc/certificate_authority/serial.srl)
• Finally, we’re outputting a new certificate for the web server: (-out ../www.crt)

That little command does quite a bit. After running it, you will be prompted for the CA Root private key password (if you created one as I recommended that you do). It should automatically increment the serial file (serial.srl) so that future certificates do not have the same serial number used to identify certificates.

Now we have an SSL certificate for your server located in /usr/local/etc/apache2. Time to tell Apache where it is.

Telling Apache about our certificate

This next step takes place in the Virtual Host configuration file (or in your httpd.conf if your OS distro has not broken that file up yet). The true location of this next step can only be described in a universal fashion by describing the purpose: website configuration. Locate that file. In Gentoo, it’s located at: /etc/apache2/vhosts.d/00_default_ssl_vhost.conf I am unsure as to its location on other operating systems.

Once you’ve located this file, you’ll need to edit it (this may require privilege escalation). Edit this file in your favorite editor (or least favorite, there is no favorite requirement).

Add or amend the following lines in the VirtualHost section of the website you wish to secure using the new SSL Certificate. The hostname of this virtual host must match the CN name you specified when you created the CSR for this site.

SSLCertificateFile /usr/local/etc/apache2/ssl/www.crt
SSLCertificateKeyFile /usr/local/etc/apache2/ssl/www.pem
SSLCertificateChainFile /usr/local/certificate_authority/certs/cacert.crt

This tells Apache:

1. Where our certificate can be accessed so that the server may present it to requesters
2. Where the server’s private key file is so that the certificate may be used to decrypt encrypted requests from requesters (these requests are encrypted using the certificate in step 1)
3. Where the certificate authority’s certificate is located so that Apache is able to append it to the certificate presented in step 1 (to avoid having the user’s browser look it up first)

Again, I’m assuming that you’ve already set up Apache to use SSL:

1. You’ve installed mod_ssl
2. You have enabled it in the /etc/conf.d/apache2 file
3. You’ve allowed access to port 443 through all firewalls (or other port you wish to use for SSL)
4. You’ve configured your Apache instance to listen on the SSL port

If you’ve not done so, Apache will not understand the SSL commands and will not start.

If all goes well, when you access the website at the CN name on your certificate and use the https protocol (i.e. https://CN where CN is the name used when you were asked for the Common Name when generating the CSR), you will see your website and will not be prompted to accept the authenticity of the certificate. Again, I assume you’ve completed the previous tutorial and have installed your own Root CA (used to sign this certificate) into your machine’s trusted Root certificates.

Should something go wrong

Debugging OpenSSL certificate problems is tricky and complicated. OpenSSL provides a tool for doing so over the network. I used it extensively during my first attempts at creating my certificates. Use:

openssl s_client -connect CN:PORT -debug

See the documentation# for more information. Be warned, this is a developer tool and comes with few instructions and brief explanations.

The Prompt

Recently, my slew of SSL certificates expired for my site and e-mail. Like most people, I didn’t write anything down when I set it up. After all, I was just trying to explore what I could do with my server at that point. So, I decided to make my own Root CA for my site’s certificates. That way, my family and I can use my root certificate to verify the authenticity of my servers and services.

What are you talking about?

SSL Certificates. SSL/TLS is the secure socket layer. It uses the PKI (public key infrastructure) to provide both data encryption and confidentiality. You can review the Wikipedia article about PKI and SSL

Anywho, with a fully working SSL/TLS system in place, you can be relatively sure that your client-server (and server-server) communications are not only encrypted, but being held between the party with which you believe you are communicating. This means you can use your e-mail and web server on a public network and not worry about your password being sent in the clear or digested format. It also means you’re sure you’re talking with your servers and not someone pretending to be a gateway. It’s nice, but it’s much more complicated to set up.

One of the problems with self-signed certificates is that you are not sure if the certificate is authentic the first time you connect to the server. As I have SSL for my mail and https server, I’d have to independently verify each certificate via the SHA1 hash and identifying information to be absolutely sure my communications are secure.

Where does the Root CA Thingy come in?

A CA is short for “Certificate Authority.” A Certificate authority can sign certificates that you make. Because the CA is trusted, when the CA signs your certificate, it transfers that trust to your certificate. When you get your operating system (or your web browser or e-mail client), there are several Root CA’s pre-installed. They pay lots of cashey money to be there so that your browser, operating system, and e-mail client will recognize theirs and certificates they sign as trusted.

OK, so How do you do it?

Disclaimers

This guide is for amateur, non-production use only. I make no warranties or guarantees as to the correctness of this document. This guide involves escalating privileges and may cause irreparable damage if used improperly. You use this document at your own risk and I hereby disclaim any responsibility or liability for your actions or non-actions.

Overview

1. Create a “proper” and “secure” environment for your CA private key and certificates
2. Create a CA private key
3. Create a CA Certificate Signing Request (CSR)
4. Self-sign the CA CSR using the CA’s private key thereby creating a CA Certificate
5. Install the CA Certificate locally

Create a “proper” and “secure” environment

Before we begin, we need a “secure” place to put our CA Root’s private key. We cannot let this fall into the wrong hands (or any hands other than ours).

When you see YOURUSERNAME, replace it with the username you use to access the system onto which you are creating the Root CA’s certificate.

sudo mkdir /usr/local/etc/certificate_authority
sudo groupadd ca_admins
sudo usermod -a -G ca_admins YOURUSERNAME
cd /usr/local/etc/certificate_authority
sudo mkdir certs
sudo mkdir private
sudo chmod 0770 private
sudo chmod 0775 certs
sudo chown :ca_admins private
sudo chown :ca_admins certs

You’ll need to log out and then log back in to properly join the ca_admins group. Be sure to cd to the /usr/local/etc/certificate_authority directory when you do so. The following steps assume that you are in that directory.

We’ll be generating the certificate’s private key. It’s important that it never be exposed. I highly recommend that you encrypt it now, when it’s being created, but in this example, I will not do so to facilitate this demonstration. I’ll indicate where you’ll need to put in the encryption.

I’ve created an environment that is relatively “secure.” That is, it is only as secure as your server. We will generate the private key in the private folder. I have designated this as readable only to root and members of the ca_admins group. I’ve also put commands in to create that group and have you join it.

When you create the CA private key, it will be placed under “private.” When the certificate is generated and signed, the resulting certificate will be placed in the certs directory. The only file that needs to be kept secret is the CA private key. We generate it in the next step.

The proper location for local configuration changes should be the /usr/local/etc folder. It’s a toss up as to whether or not a CA’s private key is a local configuration resource. It may be better to place all of this in the /var/local/certificate_authority. The serial file most assuredly belongs here (created later) as would a Root CA CSR configuration file (not used in this document). However, for the purposes of this document, we’ll keep it as is.

In this step, I’ve also created a certs directory and a private directory. Certificates that we create for the CA Root should be placed in the certs folder. The private key for the CA Root should be placed in the private directory, as it will not be world-readable.

The CA Private Key

I like RSA certificates, so I generate RSA server private keys.

openssl genrsa -out private/cakey.pem 2048

If you want to create an encrypted key, specify this:

openssl genrsa -out private/cakey.pem -des3 2048

Create a CA CSR and Sign it (single step)

openssl req -new -x509 -key private/cakey.pem -out certs/cacert.crt -days 3600 -extensions v3_ca

This creates a 10-year (abouts) certificate signing request that is immediately signed by the CA’s Root private key to produce the certificate (certs/cacert.crt). We’ll be using it as our Root CA certificate. If you put a password on the key, you’ll be asked for that password again. The extensions section is for version 3 certificates. V3 certificates have a flag in the certificate explaining what its role is. You’ll also be asked a series of questions to identify your certificate. Answer them honestly. When you get to the CN field, do NOT enter your domain name or similar. Name it something that’s impossible to be a domain and does not conflict with another Root CA. I called my “Wojno CA Root” modeled after Apple’s certificate name: “Apple CA Root”.

You now have a real certificate. Now you need to install it locally onto every machine that will use a certificate signed by this Root CA Certificate. Yes, this is bothersome, but it sure beats having to specify an override for every certificate you generate. It also lowers the risk of accepting a bogus certificate by your users, in the event you miss one or more.

Installing the CA Certificate locally

This process varies between operating systems and/or browsers. Apple uses the Keychain. Windows has a root certificate listing in the Internet Options panel (accessible via the control panel). I have no clue where it would be in Vista. You’re on your own there. Once you have it installed, you’ll be able to create as many certificates and sign them and all will be automatically trusted because you now trust the Root CA Certificate that will sign them all. Special note: do NOT install stranger root certificates! This will pose a security risk if you go about installing root certificates willy-nilly.

Create the serial file

Back in your ca directory where you created the private and certs folders:

echo "01" > serial.srl
sudo chmod 0660 serial.srl
sudo chown :ca_admins serial.srl

Certificates issued by this CA will be imprinted with a serial number. Every time you sign a certificate, the serial file will be incremented and updated. Only ca_admins will be able to modify or read this file. While hiding the contents of this file is not absolutely paramount, the less that is known, the harder it is to break something (this is a generic statement and not necessarily true, but is usually true).

For use with Apache2’s mod_ssl, please see the next article in the series.