How to Visualize Your Cloud deployments - CloudMapper

September 6, 2018 Gabe Maentz

As you are aware, I am a big fan of visualizations. In fact one of my most popular set of posts centers on using RVTools to collect and visualize a VMware environment. As much of my focus is now centered on cloud deployments I wanted to highlight some of the tools I have found particularly useful for visualizing AWS and Azure. These are:

CloudMapper

CloudMapper is a tool from Duo Security for visualizing Amazon Web Services (AWS) cloud environments. It was build out of a need to help people perform their jobs easier by providing simple and interactive visualizations of their AWS account. CloudMapper runs a collection process against your AWS account to prepare and build an interactive visualization for each component along with their connections. Some have called it Google Maps for your AWS account, and to put it simply CloudMapper shows how your AWS environment actually looks.

To see the level of interaction check out their online demo of a deployed application in the us-east-1 region. Below is the CloudMapper visualization of the web applicatoin deployment highlighted in several of my Terraform posts.

CloudMapper was built by Scott Piper in conjunction with Duo Security and luckily for us, they have open sourced their work and continue to maintain it. Of the three tools mentioned, it definetly provides the most robust view in terms of connectivity and security for visualinzing an AWS environment. To get started using CloudMapper check out the product page as well as the installation and setup details on GitHub.

Benefits:

Especially good for seeing how resources are connected, and visualizing your AWS environment.
Interactive web diagram is extremely handy for understanding and validating your deployment.
I have found CloudMapper to be the most thorough tool of the three highlighted
Free / Open Source

Nice to Haves:

Setup is several steps and more involved compared to the other tools. I did run into a number of compatibility issues with some of the backend Python packages - which reminded me that yes, it is open source.
Collection phase can take some time, as it is truly pulling all the metadata information for your entire AWS account across all components and regions.
Would be nice to have this exported in different formats - currently supports PNG and JSON only. Visio and PDF are some of other formats similar tools support.
Support only for AWS, it would nice to be able to see support for other clouds (Azure, GCP, etc.)

VMworld 2018 - Where's Gabe?

August 21, 2018 Gabe Maentz

Next week I will be attending my fifth VMworld, and have had the fortune of seeing the conference through the eyes of a customer, partner, reseller and blogger. Last week I received news that VMware would be sponsoring me with a blogger pass to attend this year's show for which I am most grateful. I find that VMworld is the most valuable for me when I enter with a plan. I would encourage others - especially first timers - to build out a plan for what the want to get out of the conference.

My blogging and professional development for VMworld 2018 will focus on the areas of automation and orchestration within the VMware ecosystem and across heterogeneous clouds. I am most interested in further developing deeper product and tooling expertise, while hearing first hand from businesses and community members how they are refining their skill sets to keep up with the demands placed upon them in a multi-cloud world.

vmware {code} Hackathon 2018

One of the events I am most looking forward to this year is the Hackathon. As a participant two years ago, it is exciting to see the level of attention and interest for this year's event. This event highlights the creativity, diversity, vulnerability and genuine 'can do' attitude that I have come to love in the VMware community. Want to get out of your comfort zone and hang with some tremendous people - the Hackathon is for you. I am not yet formally on a team (remember, I just found out I would be attending the conference last week), but I still have time. Some of the teams I plan to keep an close eye on are:

vMafia 2.0 - Building a vSphere Client Plugin to run PowerShell / PowerCLI within the new HTML5 web console.
comdivisin - automation around vRO, PowerCLI
VDM-Hackers - creators of VirtualDesignMaster.io, this team is going to set big goals and do great (fun) things
Pythonic - build a bolt-on user interface to Vault to make it easy for homelab users to get started with secrets management.
View API & PS - improve the vmware.hv.helper module by adding new functions or improving them and if time allows to extend the vCheck for Horizon View
vmug-cloud-labs - nested lab builds within VMware Cloud on AWS.
PS API Gateway - Building a PowerShell API gateway to interact with all VMware services in a simple standard format.

Tech Field Day - Extra

At VMworld 2015, I had my first in person Tech Field Day experience as a presenter. The Tech Field Day events bring together innovative IT product vendors and independent thought leaders to share information and opinions in a presentation and discussion format. Since then I have been fortunate to be a delegate to several Tech Field Day events. Tuesday afternoon of VMworld 2018, I will again be joining the delegate ranks to engage with what look to be like three interesting companies. I know very little about each of them and excited to learn more.

JetStream Software - The JetStream Software platform empowers infrastructure spanning multiple data centers and cloud services. Applications enjoy cloud-scale elasticity, and can migrate seamlessly, all with enterprise grade performance, availability and resilience.
Quali - Blueprint, model, and deploy on-demand, self-service, sandbox environments for labs, data centers, and hybrid cloud.
Runecast - Proactively use VMware KBs, best practices and security hardening guidelines to protect your environment

These sessions will be live streamed, with the recordings and coverage made available on the Tech Field Day website.

Break out Sessions

Signing up for the conference late does come with some disadvantages, and one of those is getting into popular sessions. None the less there are some great sessions which I plan to take in first hand, and the others I a plan to view online when they become available.

Culture Shift: The Heavy Lifting Behind Digital Transformation [LDT1895BU]
Monday, August 27, 11:30 AM - 12:30 PM | Lagoon D, Level 2

Introduction to Cloud Native App Development on K8s (Maker Space Workshop) [CODE5627U]
Monday, August 27, 01:00 PM - 02:50 PM

Deep Dive: How Fannie Mae Releases vRealize Content Using Lifecycle Manager [MGT2174BU]
Wednesday, August 29, 09:30 AM - 10:30

Instrumenting all things Kubernetes (Maker Space Workshop) [CODE5633U]
Wednesday, August 29, 01:00 PM - 02:50

Top 5 Processes to Accelerate DevOps with Josh Miller [MTE5074U]
Wednesday, August 29, 03:15 PM - 04:00

Destination Digital: Plot a Course for Transformation with Cloud Migration [LDT2207BU]
Thursday, August 30, 10:30 AM - 11:30 AM

Infrastructure as Code for the Enterprise [CODE5570U]
Thursday, August 30, 12:15 PM - 12:45 PM

Infrastructure as Code(IaC) [VMTN5615U]
Thursday, August 30, 01:45 PM - 02:00 PM

vExpert Session - Mystery CLOUD Session 1 - Tuesday August 28th, 4:15pm-5:15pm Mystery AUTOMATION Session 2 - Wednesday August 29th, 5:15pm-6:15pm

vBrownBag

Schedule and time permitting, I am prepared to present a vBrownBag TechTalk on my recent work with Terraform on AWS and VMware. I have found that the vBrownBag stage and surrounding area in the Hang Space is a great place to meet and engage with other attendees and community members. If you would like to connect this is usually the place to find me.

Community, Parties and Events

Of course one of the best things about VMWorld is to interact with the vCommunity, and this community likes to have a good time. Unfortunately I will not be able to attend vBreakfast, Open Acts or VMunderground this year because I don't arrive until Monday of the show. You should definitely check out these great community events, as well as the full list of other parties. Rumor is that some of my favorites including Snoop Dogg, Run DMC & the Roots will be in attendance. I am traveling with some first time Las Vegas goers, so hope to spend some time enjoying the sites and sounds with them. At this point I am confirmed to be at both the vExpert party and vSoccer events on Tuesday night, but if history is an indicator I will find my way to several others.

Contact Info

Of course, if you are attending the show I would love to connect with you schedules permitting. Hit me up on twitter @gmaentz or via email at gabe@maentz.net . I will be heading back home on Thursday evening.

Using Terraform to Up Your Automation Game: Multi-Environment/Multi-Region

August 16, 2018 Gabe Maentz

In the last two posts we have been working with Terraform to automate the buildout of our Virtual Private Cloud (VPC) in AWS and deploy a fleet of scalable web infrastructure. Our next step is to complete the infrastructure build so that it is identical across our development/test, staging & production environments. Each of these environments reside in different regions: development (us-west-2), staging (us-east-1) and production (us-east-2).

File Layout

In the case of deploying our development environment, our work is really close to being complete. We already have a working set of code that deploys into the us-west-2 region, which is contained in the main.tf and outputs.tf files. We will add one step, and that is to create a folder structure that allows our code to easily be managed and referenced. At the top level we create a separate folder for each of our environments, and within each of these folders a sub-folder specifying the type of fleet we are deploying . In the case of our development web server deployment, the folder structure is simple and looks like this:

. ├── dev │

   └── webservers │

   ├── main.tf │

   ├── outputs.tf

There are a number of benefits with laying out our files in this manner - including isolation, re-use and managing state - which Yevgeniy Brikman does an excellent job of describing. As Brikman indicates - "This file layout makes it easy to browse the code and understand exactly what components are deployed in each environment. It also provides a good amount of isolation between environments and between components within an environment, ensuring that if something goes wrong, the damage is contained as much as possible to just one small part of your entire infrastructure."

Deploy Development

Now that our file layout is the way we want it, let's deploy development. Very simple, and something we have done a few times now. Since we moved our two files into the a new folder location, we will need to initialize the deployment (terraform init), plan it (terraform plan) and finally deploy (terraform apply).

Once complete we can browse over to see the development deployment.

Deploy Staging

One of the most powerful benefits of deploying our infrastructure as code in a modular way is reusability. In fact, to build out our staging environment is only a matter of a couple of steps. First we will create a staging folder in which to store our files, and then we will copy over our main.tf and output.tf files. We will then make a few edits to the main.tf including the following updates: region, IP address space, tags, ami, cluster name, cluster size and key_names. Looking at the differences between development and staging is as simple as running a compare between the main.tf files in the dev and staging folders. The differences are highlighted below:

Once we are happy with the updates, the sequence to deploy is exactly what we are use to. This time we will run our initialize, plan and deployment from within the staging folder. Once complete we can browse over to see the staging deployment.

Production Deployment

Our production environment will mimic our staging environment with only a few edits including deployment to the us-east-2 region, and will start with 8 webservers in the fleet and scale as needed. Once again leveraging infrastructure as code we will simply copy the main.tf and output.tf files out of staging and make our edits. The differences between staging and production are highlighted below:

Now we use the power of Terraform to deploy, and Viola...production is LIVE.

Visualizing Our Deployment

Now that our 3 deployments are complete, we can see the folder structure that has been build out maintaining a separate state for each environment.

To document and visualize our build out, I like to use hava.io which builds architecture diagrams for AWS and Azure environments. As you can see all three environments are active and we can drill into any of them to see the details, including pricing estimates - production ($115/month), staging ($91/month), dev ($72/month).

Mission Complete

Our mission was to create and deploy a set of auto-scaling web servers, front ended by a load balancer for our development, staging and production environments across 3 different AWS regions. Through the power of infrastructure as code we utilized Terraform to define, plan and automate a consistent set of deployments. Mission Complete.

Terraform Series

This is part of a Terraform series in which we cover:

Building a Virtual Private Cloud

Using Terraform to Up Your Automation Game - Building the Fleet

August 13, 2018 Gabe Maentz

Populating our Virtual Private Cloud

In the previous post we successfully created our Virtual Private Cloud (VPC) in AWS via infrastructure as code utilizing Terraform, which provided us the ability to stand up and tear down our infrastructure landing pad on demand. Now that our landing pad is complete and can be deployed at any time, let's build our fleet of load balanced web servers.

Building the Fleet Using Terraform Modules

Taking a similar approach to our VPC build out we will once again utilize Terraform modules, this time to create and build out our web server fleet. In addition to the Terraform Module Registry there are a number of different sources from which to select ready built modules - including GitHub. For our web server cluster we will utilize a short and simple webserver_cluster module that I have made available in my GitHub terraform repository.

This module creates a basic web server cluster which leverages an AWS launch configuration and auto scaling group to spin up the EC2 instances that will be perfuming as web servers. It also places a load balancer in front of these servers which balances traffic amongst them and performs health checks to be sure the fleet is bullet proof. The module also configures the necessary security groups to allow http traffic inbound. All we need to do is to specify the size and number of the web servers and where to land them.

To call this module we simply need to append to the main.tf file to call the webserver_cluster module and specify how our web server fleet should be built.

In the code statement above we simply call out the source of our webserver_cluster module which resides in GitHub, specify a name for our cluster, the image and size server to use, a key name should we need to connect to an instance, the minimum and maximum number of servers to deploy, along with the VPC and subnets to place them in (referenced from our VPC build out).

In this case we are going to deploy two web servers to the public subnets we built in our VPC.

Deploying the Fleet

After updating our main.tf file with the code segment above, let's now initialize and test the deployment of our web servers. Since we are adding a new module plan we must rerun our terraform init command to load the module. We can then execute a terraform plan for validation and finally terraform apply to deploy our fleet of web servers to the public subnets of or VPC residing in AWS us-west-2.

Validate the Plan and Deploy using terraform plan and terraform apply.

Accessing the Fleet

So our deployment is complete, but how can we access it? When building infrastructure, Terraform stores hundreds of attribute values for all of our resources. We are often only interested in just a few of these resource, like the DNS name of our load balancer to access the website. Outputs are used to identify and tell Terraform what data is important to show back to the user.

Outputs are stored as variables and it is considered best practice to organize them in a separate file within our repository. We will create a new file called outputs.tf in the same directory as our main.tf file and specify the key pieces of information about our fleet, including: DNS name of the load balancer, private subnets, public subnets, NAT IPs, etc.

After creating and saving the outputs.tf file, we can issue a terraform refresh against our deployed environment to refresh its state and see the outputs. We could have also issued a terraform output to see these values, and they will be displayed the next time terraform apply is executed.

Browsing to the value contained in our elb_dns_name output, we see our website. Success.

Scaling the Fleet

So now that our fleet is deployed, let's scale it. This is a very simple operation requiring just a small adjustment to the min and max size setting within the webserver_cluster module. We will adjust two lines in main.tf and rerun our plan/deployment.

....  min_size            = 8 max_size            = 10  ....

Viola. Our web server fleet now has been scaled up with an in place update that has no service disruption. This showcases the power of infrastructure as code and AWS auto-scaling groups.