Cloud Governance Concepts
Of all the exciting topics to blog about, today, I choose to focus on the least exciting, but arguably one of the most impactful: Cloud governance is a subset of data governance and while definitions from the major Cloud providers vary, the following 2 themes appear consistently:
- Cost Governance
- Data Security
I was fortunate enough to attend the Gartner Catalyst Conference in 2019 when attending conferences was still a thing, and one of the themes that resonated with me was sidebar commentary from the analysts about the cost enigma associated with modern Cloud offerings. If memory serves, one of the keynote addresses used a shock tactic statement to grab the audience attention along the lines of:
“I just received a $10M bill in the mail from AWS, and I have no idea what this is for!“
So, I have not received a $10M bill from anyone, but I did have to deal with an annoying $50 monthly deduction on my Walmart credit card (which I forgot I owned) from one of the AWS tenancies that I use for testing that I also forgot about turning off. Yes, it’s my fault. I need to keep track. It could be worse.
The cloud vendor does not care to remind me. Why would they? I agreed to use a resource and pay for it, by firing up an EC2 instance with some block storage and a load balancer in my non-default region and it is my responsibility to ensure that if it’s not being used, then it is turned off. Well, it would be nice if I could be reminded with a monthly email with some note like: “Hey, we noticed you have several resources that have been idling and burning cash, maybe you should consider turning them off or removing them?”, but that’s not going to happen. Lesson learnt. Scale those idle resources by a factor of 100 or a 1000 or 10000, and it stops being annoying…it gets damn serious.
Enter Cost governance or a systematic framework for controlling the costs of your Cloud infrastructure. Cloud computing resources are rentals. You do not own it. You decide when you are done and stop using it and generally speaking, at that point, you stop getting charged for it. One of the major enablers of this problem is the ease of procurement. Anyone with a credit card can sign-up and voila a shadow IT department is created. Over the last 25 years, I’ve consulted and observed in many organizations that not all computing infrastructure is managed by a central IT department. For various reasons, even some legitimate, there often exists one or more departments within organizations that have their own infrastructure. It could be in the organization’s official data centers or it could be in the broom closet next to the network patch panel and now it could be in any number of clouds. This last statement should send you screaming for the hills, because your corporate data could be anywhere in the world, out of the purview of your IT security compliance team, possibly jeopardizing SLAs that ensure data residency, data sovereignty, or even worse, not protecting your data assets from the big bad internet because someone decided they knew better or they could not wait for central IT to dedicate cycles to their project or “the business needed the data yesterday!“
So, what can you do about it?
Governance policies to the rescue.
First, and this is serious, it must come from the TOP of the organization that ALL Cloud computing requirements must go through a central place. Non-conformance penalties should be articulated to discourage shadow IT departments, with a clear prohibition message. Even Test/Demo/POC tenancies should be visible and preferably controlled with oversight by central IT.
Data security breaches are inevitable when private tenancies exist with organizational data outside of the purview of your organization. Would you let an employee/consultant/contractor leave your premises with a data server? An often-overlooked security vector relates to backups. While database-as-a-service instances from Oracle implement in-transit and at-rest encryption, you still need policies to protect backups. If you can encrypt backups, that is awesome, but remember to keep encryption keys separate from backups.
Cloud governance policy frameworks sound super boring, but they go a long way in avoiding major cost and security risks, especially if these are adopted upfront. Cloud governance is applicable regardless of the vendor. The mainstream vendors offer similar tools and mechanisms to aid with governance but it is still pretty cryptic to figure out how much you owe at any point in time. However, they are getting better at this.
Oracle Cloud Infrastructure (OCI) has a pretty neat infrastructure resource management framework that uses a Tenancy (A tenancy is associated with one organization) that comprises logical Compartments to group resources. These Compartments are governed by classical Identity Management principles (Users in Groups with English-like policies that govern how users or groups interact with resources or compartments).
I find it useful to map existing departmental or line-of-business groups that own on-prem infrastructure to compartments as well as to map existing IT infrastructure groups to Cloud infrastructure groups. DBAs or Network Admins working on-prem are doing similar work on Cloud resources. In fact, this may be the opportunity you’ve been waiting for to finally align these teams with the resources they are meant to manage and optimize your infrastructure support teams while securing your infrastructure.
Tags may be applied to each individual OCI resource. These are optional and the most rudimentary tags are simple key-value pairs. OCI upped their game by offering Tag Namespaces, that allow you to use a predefined set of tags like (CreatedBy and CreatedOn) or any set of predefined tags that make sense in your organization. In many of the tenancies that I manage, I have a periodic report of untagged resources that are designated for follow-up and recycling if they do not have identification tags. The tagging framework allows for cost-tracking tags, so I can report by line-of-business or department to cross-charge or track resource costs within the tenancy.
In your Cloud governance policy doc, it is essential to prescribe that all Cloud resources conform to a standardized tagging framework, with a process for handling untagged resources. This approach fosters accountability and efficiency while improving security by preventing rogue resources from being inserted into compartments by bad actors. Group departmental resources into Compartments. Design the IT department of your dreams conceptually and fit human resources into an optimal cloud identity framework that leverages: Users, Groups and Policies to govern resources within Compartments.
OCI lets you download CSV files with details of your usage, so this can be ingested and analyzed to identify usage and cost patterns.
Some descriptive analysis is also provided on the console, but as you can see, the aggregated data is not real-time…actually it’s a couple of days behind. And these are estimates, not actual costs so there may be some deviations.
There is an API framework available that exposes most services including the Oracle CloudAccount Metering REST API. It has some limitations but could be integrated with the cloud usage CSV report data to analyze this data.
OCI also allows you to create budgets on a compartment basis or on a cost-tracking tag basis. You can then set alerts for these budgets which notify you when exceeded and allows you to minimize runaway costs.
Armed with a proactive cloud governance policy, and leveraging tooling provided by the cloud vendors, you can ensure that your costs are tracked, you only use the resources you need, and your data is better protected.