A fundamental problem in Cloud management is “how do I get the remote instance to do what I want it to?”. Taking this task on for a few systems is doable with a number of techniques, making it scale for many thousands is not quite as simple. At RightScale, we have been on the “bleeding edge” of this issue since the early days of cloud computing, and we have learned a lot along the way. One of those lessons has led to our implementation of the RightLink agent and the RightNet protocol.

Introduction to RightLink

RightLink (link) is the instance side agent that supports RightScale’s RightNet protocol. The agent provides an improved and secure ability to leverage RightScale to manage large numbers of instances in the cloud. In the RightScale architecture, we leverage a light-weight RightLink agent on every instance to support our latest automation features. Prior to RightLink, which was released a bit over year ago, RightScale leveraged the command execution features of SSH to perform tasks on remote instances. With the introduction of RightNet and the RightLink agent, we are no longer reliant on SSH access for instance management.

The RightLink agent communicates with the core RightScale systems using the Advanced Message Queuing Protocol (AMQP). RightNet leverages AMQP’s Simple Authentication and Security Layer (SASL) support to perform a basic session authentication to ensure that the RightLink agent is talking to a legitimate RightScale core component (a broker in our lingo). This session authentication uses a shared key to authenticate the ends. After the session is authenticated RightNet uses payload encryption (openssl with X509 certificates, PKCS7 envelopes and AES 256 CBC cipher for encryption) to protect that data while in transit, and to provide a much stronger authentication mechanism (public-private key versus only the shared key of the session). Both of these security features are to ensure that packets are properly segmented and protected in the highly multi-tenant aspect of the cloud.

All our version 5 (v5) RightImages (and Multi Cloud Images, MCIs) include the RightLink agent by default. We started releasing v5 images over a year ago, and have seen a large, but not complete, adoption. For those of you still on v4 images, I am going to try to give you a couple more security motivations that may encourage you down the upgrade path.

1. Ability to restrict SSH access on the instance: Because RightLink does not use SSH you can restrict access to the ssh service on Linux systems. With non RightLink enabled images (i.e., v4 and earlier by default), the RightScale platform ran scripts on the instance by ssh-ing into that instance directly, thus the need for ssh port to be accessible on the instance from the RightScale platform usually meant that it was accessible from any IP address. This created some exposure with potential brute force attacks. I will say that by default, RightImages configured SSHD to support public-key authentication only, so the risk of brute force password guessing was not an issue. What was an issue was that any vulnerability found in the SSHD server would then be potentially exploitable by anyone on the Internet.  With RightLink, this exposure can be mitigated.
2. Managed SSH: In addition, v5 RightImages introduce a “Managed SSH Login” feature. This allows you to use a different SSH key for each user logging into a server. It can either use an SSH key uploaded by each user or the dashboard can generate a key for each user.  When using EC2 you may still select an EC2 SSH Key when launching the instance, however, it’s only really necessary if you need to log-in before RightLink starts to troubleshoot something in the bootstrap process. Note that the SSH connection is from your desktop system (wherever you are running the dashboard UI from, not RightScale) to your instance, thus working seamlessly with any SSH access restrictions you put in place.

SPOILER-ALERT: one of the items we are working on for RightLink v5.8 (next version coming out) is a Managed SSH Login that will bind each RightScale authentication principal to a distinct, non-root Unix user whenever they login via the dashboard. This is intended to improve the login auditing as well a enable each user to load a customized shell profile. We’d be very interested in your feedback as to the usefulness and desire of this specific feature.

Upgrade options

The cleanest and best way to move to v5 images is to find a v5 ServerTemplate, clone it and make the modifications needed to effectively duplicate the functionality you currently have. This will work like a charm if you if you did your scripts right and took a modular approach to deployment.

Next option is to change the RightImage (i.e. Multi Cloud Image, MCI) you’re using to a v5 one and relaunch. The V5 execution of RightScripts is almost fully compatible with v4 so, in theory, that’s all you need to do. The catch typically is that this brings updated versions of the OS and packages with it and may cause some incompatibilities. You will probably spend a bit more time troubleshooting this avenue.

Lastly, you can get RightNet support by RightLink enabling your v4 instance (see http://support.rightscale.com/12-Guides/RightLink/04-Creating_RightScale-enabled_Images_with_RightLink), and many might be motivated to go that route. I would encourage you to move to v5. While you’ll get the “not using ssh for command and control” benefit, you will miss many other benefits of the v5 image update.

Why Again?

Because there are some really cool features in v5:

• Managed SSH
• Bug fixes
• Faster Execution of Operational Scripts
• Added Chef Support in addition to RightScritps

More details can be found http://support.rightscale.com/06-FAQs/FAQ_0180_-_What_are_the_differences_between_v4_and_v5_RightImages%3F

It will take a bit of effort, but I guarantee the improvements you gain will be worth it! My one-liner of advice to those RightScale customers with older versions ”if you’re one of those hanging onto v4 or earlier you really should upgrade.”

Filed under: Chef, EC2, Security Tagged: Cloud Management, RightImage, RightLink, RightScale, SSH

I recently called up my buddy who used to be vice president of marketing at SugarCRM. I asked him if he ever encountered companies that were building their own CRM solutions internally. “No, that’s dumb,” he said. “That’s why they came to Sugar, so they could use ours. It’s too much work to do it yourself.”

Building your own Salesforce.com? Yup, sounds like a lot of work. Yet here at RightScale, I see many companies trying to build their own cloud management solutions. Perhaps it is the DevOps mindset that has made cloud computing so popular: “If I can’t get approval, I’ll just do it myself on the side.” Or perhaps it is because we are still in the early stages of cloud, and people are experimenting and discovering what is possible internally versus what is available in the market.

I did an informal poll of our sales team, and here’s what they said were the top reasons companies try to make their own solutions rather than use a cloud management product:

1. They want control, or the ability to highly customize their environment.
2. PaaS and IaaS, as concepts, seem simple, easy to jump on. A “cloud computing management platform” seems like a complex paradigm to adopt.
3. Because they can, and they want the challenge of exploring a new frontier.
4. The cost of a cloud management solution is too high.

OK, so these appear to be valid reasons at first glance. But these statements are typically founded in misconceptions about cloud management solutions in general or RightScale in particular, which I’ll address here:

Control: RightScale is not a PaaS service. We let you get into everything – perhaps more so than we should. Change the images if you must, run custom scripts against our API, and export usage data to include in your own data warehouse. Fifty-two percent of the servers running on RightScale are controlled by completely custom ServerTemplates, not ones we provide. Our product philosophy is to let you “get under the hood” if you need to – so please do.

Complexity: Cloud management is complex, and I don’t argue that. What RightScale aims to do is provide a layer of abstraction that makes the difficult and mundane tasks, like auto-scaling, much easier. It is unfortunate that the term seems complex, because if anything, a cloud management solution can make managing your entire cloud infrastructure and applications so much easier.

Conquering the new frontier: You’re being told by your boss to “Learn cloud now – just figure it out.” You want to truly understand what’s possible, how to build it, and deliver on expectations. As you start down this path, you cobble together some tools to accomplish your first foray into the cloud. Unfortunately, technologists have a tendency to “reinvent the wheel” as they continue along their path to the cloud. We’re many steps ahead, and we’re happy to share what we’ve already learned.

Cost: Netflix is a poster-child for DIY cloud, and has been forthcoming about its experience, which has helped grow this new paradigm. Netflix “designed its cloud architecture so that it has the option to move to an Amazon Web Services competitor” if needed, according to this NetworkWorld article. At a recent conference, Adrian Cockcroft, Cloud Architect for Netflix, mentioned that Netflix has 50+ engineers working on this cloud-independent solution. Doing some quick math, that’s about $8.3 MM per year Netflix spends building and maintaining this platform. That could buy a lot of RightScale Enterprise Editions!

At the end of the day, we see many customers who come to us after they outgrow their own internal solutions. They eventually discover that there are just too many things to stitch together: configuration management, systems automation, monitoring, application automation, provisioning, user permissions, reporting…it goes on.

We have hundreds of employees and have spent many millions creating the most comprehensive cloud management platform in the world. And we designed our product to drive the same way no matter which cloud you choose. So while cloud management may seem like a fun weekend project to tackle, it’s not – please don’t try it at home.

Yes, Amazon is still the dominant cloud, but a tornado of new clouds is swirling. The next thing your boss will likely ask is, “So what if we wanted to use this other cloud instead?”

Update Feb 3, 2012: Since I published this post, I’ve received a lot of feedback regarding DIY in the cloud computing space.

A few of our customer developers pointed out that they actually appreciated learning the cloud through RightScale – it gave them both an understanding of the underlying IaaS cloud as well insight into ideal cloud management frameworks. Forbes ran an article on how this extensive cloud computing knowledge is in high demand in IT and beyond, and we’re starting to see RightScale listed as a required skill on some of these job postings.

Next, I’ve heard from a few more larger companies who have built their own internal cloud management solution. They also cited approximately 50 engineers in their cloud computing groups, so it seems this is the sweet spot for development and maintenance of a robust internal solution. Let’s not forget about the PaaS-like solutions we offer with our ServerTemplates in this regard – it is not just automated provisioning that these larger companies ultimately need to build.

I’m not saying you can’t “do it yourself” in cloud computing (or in anything for that matter), I just want to encourage developers to avoid the trappings of #3 above – namely ignoring off-the-shelf solutions in the interest of personal discovery. It may work in the short term… until you hit one of the many walls that we’ve already had to plow through. At that point, you’ll either have to scale the solution and team, or re-architect for a product that offers the necessary solutions already.

Filed under: Cloud Computing Tagged: Cloud Computing, Cloud Management, DIY cloud, Netflix cloud, RightScale Cloud Management