Managing VMware Proactively with Runecast

As technologists, how do you troubleshoot problems? Does this cycle sound familiar:

  1. bad thing happens

  2. Google the error message

  3. land on a knowledge base/blog article

  4. fix the issue

  5. if not…. rinse and repeat.

I am sure we can all relate.

As admins/operators we are constantly pressed for getting things working in short order. This results in much of our time reactively troubleshooting issues, digging through logs and using Google to try to find a fix. While most of us have the desire to stay on top of issues and become more proactive - the reality is that it is an uphill battle.

I am sure we can all relate.

Runecast - For VMWare admins, built by VMWARE ADMINS

At VMworld this year I got to learn a little bit about Runecast during their Tech Field Day presentation. Runecast is a company focused on helping VMware admins with the task of keeping their environments healthy, secure and in line with published best practices. The company was founded by a set of practitioners who ran the VMware Center of Excellence for IBM, where they were in the trenches living and breathing VMware for many years. From these trenches arose the idea for product which would help tackle many of the common operational problems VMware operators face: combing through VMware knowledge base articles and logs, awareness and adherence to VMware best practices, security compliance checks, and spending too much time troubleshooting. A product was born to address these problems and it is called Runecast Analyzer.

Runcast Analyzer deploys as a single VM that aggregates information from a variety of sources including the VMware Knowledge Base, social media, and security hardening guides and synthesizes them into a central repository. Using this information, the analyzer runs a discovery against the environment to identify potential issues before they have the ability to cause outages. The technical details of how this is done is nicely explained in the white board/chalk talk presented by Runecast co-founder, Stanimir Markov.

RUNECAST Analyzer

Once deployed, Runecast Analyzer can be accessed via it’s web interface and presents several different views highlighting the health of your VMware environment. Runecast is not a performance and capacity alerting tool (as there are many of those available), but rather places its focus on configuration, manageability, security and VMware best practice conformance. The dashboard below shows the overall health based on those standards and allows you to drill down into items that may be of most importance in your organization.

RunecastDashboard.png

Looking at the inventory view for critical items across this vSphere environment, it is easy to see a series of patches that should be installed on the vCenter and ESXi hosts. This details of this critical alert provides the relative risk rating, KB article reference, and resolution details for how to address the problem. Runecast does not currently provide the ability to take remediation action from within the web interface, but that is something that may be provided in the future.

RunecastPatch.png

Best Practices, Security Hardening, Compliance & VMware KBs

In it’s first iterations Runecast Analyzer was focused on analyzing configuration items contained within VMware Knowledge Base articles, best practices and security hardening guides. Recently the Analyzer has been expanded to include log analysis and specific security/compliance standards (DISA STIG and PCI DSS). This means that it can cross check against VMware logs for known issues, as well as call out items that don’t comply with specific security standards.

Below is a shot of the inventory view in which all items can be categorized, sorted and filtered based on what is most important including a categorization by product and impact. It is encouraging to see these new items added into the product, and I can envision additional sources and levels of analysis being included moving forward.

RunecastCategories.png


Try it Out

During the Tech Field Day presentation there was a cool demo of Runecast Analyzer which you should check out, but why not try it for yourself? Runecast provides both an online/interactive demo as well as a free trial of Runecast Analyzer for you to run in your environment. Also, if you happen to be a vExpert you can take advantage of their NFR offering. This was my first exposure to Runecast, and overall I would have to say I am highly impressed. This is a product for VMware admins, built by VMware admins and aimed at helping VMware admins move ever close to proactive management of their environments.

Disclaimer:  I was personally invited to attend Tech Field Day Xtra at VMworld 2018. I was not compensated for my time or travel.  I am not required to blog on any content; blog posts are not edited or reviewed by the presenters or Tech Field Day team before publication.

Creating A CloudMapper Virtual Appliance using Packer

One of my favorite visualization tools for diagraming Amazon Web Services (AWS) environments is Duo CloudMapper. CloudMapper helps you understand visually what exists in your AWS accounts by running a collection against the environment and providing an interactive web page. This is extremely handy for identifying possible network misconfigurations, along with a slew of other benefits. For a full listing why I like this tool check out my post on How to Visualize Your Cloud Deployments with CloudMapper.

Despite it’s power, one of the challenges I have found is to simply get it started and working. CloudMapper is open source built upon other open source products and I have found that there are inevitably build and dependency issues that suck up my time before I can simply use the tool. For these reasons and to make things easier in general, I chose to create and deploy CloudMapper as virtual appliance.

Building the Virtual Appliance

I utilized Packer to provision my CloudMapper virtual appliance. Packer is excellent for creating machine images for multiple platforms from a single source configuration. In this case we will build out an Amazon Machine Image (AMI) with Packer, which will take care of all package installation and dependencies for the build out. You can learn more about all the Packer goodness on the HashiCorp website and Paul Kirby provides a nice overview in his Packer PluralSight course.

  1. Install Packer

  2. Download the cloudmapper.packer template from my GitHub account. (Packer templates are simply JSON files that specify the various components used to create the machine image, and where the build of the image will be saved. In our case we will be creating and deploying our virtual appliance into AWS, but Packer comes with support to build images for Amazon EC2, CloudStack, DigitalOcean, Docker, Google Compute Engine, Microsoft Azure, QEMU, VirtualBox, VMware, and more.)

  3. Specify AWS Credentials for creating our virtual appliance. There are a number of ways to accomplish this but we will use environment variables.

       $ export AWS_ACCESS_KEY_ID="awsaccesskey" 
       $ export AWS_SECRET_ACCESS_KEY="awsecretkey"
  4. Build the image.

    $ packer build -var aws_region="us-west-2" -var ami_id="ami-6cd6f714" -var python_version="3.5.6" cloudmapper.packer
        # aws_region is where the image will be stored.
        # ami_id is the base Amazon Linux image in the region.
        # python version of your choice.

    There are currently some issues with CloudMapper and Python 3.7, so I am using the recommend version of 3.5.6

  5. The build process will take ~10-15 minutes as it needs to compile and pull down all of the components. Once it is complete, Packer will notify of your unique AMI that can now be used for deployment.

packami.png

Deploying the Virtual Appliance

Now that the image for our virtual appliance is available in AWS, let deploy it and run CloudMapper. My preferred way to deploy would be using Terraform but for purposes of this post we will step through the manual steps.

  • Launch an instance using the newly created CloudMapper image. You can accept the defaults providing your instance a public IP with SSH access.

myami.png

Configure CloudMapper by logging in via SSH and performing the final initialization steps. (While these could be automated and built into the image, I get sensitive about saving AWS credentials anywhere even if my image is private. I prefer to specify them when needed.)

  • $ aws configure

    You can specify a full access account to run CloudMapper but I like least privilege so have setup a “Visualization” IAM user with the privileges specified in the CloudMapper readme.

cloudmapperiam.png
myIAMAccount.png
  • Configure CloudMapper’s account information in the config.json file to match aws credentials:

    $ cd ~/cloudmapper
    $ pipenv run python3 cloudmapper.py configure add-account --config-file config.json --name AWS_USERNAME --id AWS_ACCESS_KEY_ID
       #AWS_USERNAME is “friendly name” tied to IAM account   
       #AWS_ACCESS_KEY_ID is the AWS Access Key ID specified in aws configure.

  • Run CloudMapper’s collection against the environment. The collection phase can take some time, as it is truly pulling all the metadata information for your entire AWS account across all components and regions.

    $ pipenv run python3 cloudmapper.py collect --account AWS_USERNAME
  • Prepare the results and launch the webserver to display them.

    $ pipenv run python3 cloudmapper.py prepare --config config.json --account AWS_USERNAME
    $ pipenv run python3 cloudmapper.py webserver --public

  • Create and attach a security group to the instance to make the site publicly available.

securitygroupcloudmapperweb.png
securitygroupcloudmapperwebassign.png
  • Browse to public DNS address of your virtual appliance on port 8000

Please note that these steps show running this instance with a publicly available website. You can certainly deploy this to a private subnet and access through a bastion server, etc which is recommended. It would also make sense to put this site behind a login which I have noted as an opportunity for further improvement. Be sure to stop this instance when you are done using it.

devstagingprod_cloudmapper.png

Further Improvements

Having a readily available virtual appliance that just works is perfect, but there are some further improvements that I think would be handy:

  • Create a docker image of CloudMapper that can be run as a container. (There are some folks who have built this)

  • Save the collection data to an external volume so that it doesn’t live in the running appliance.

  • Create the virtual appliance that can be deployed within other Packer supported platforms, namely vSphere and Azure.

  • Lock down the website behind a username and password.

How to Visualize Your Cloud deployments - CloudMapper

As you are aware, I am a big fan of visualizations.  In fact one of my most popular set of posts centers on using RVTools to collect and visualize a VMware environment.  As much of my focus is now centered on cloud deployments I wanted to highlight some of the tools I have found particularly useful for visualizing AWS and Azure.  These are:

  1. CloudMapper

  2. CloudCraft

  3. Hava.io

CloudMapper

CloudMapper is a tool from Duo Security for visualizing Amazon Web Services (AWS) cloud environments.  It was build out of a need to help people perform their jobs easier by providing simple and interactive visualizations of their AWS account.  CloudMapper runs a collection process against your AWS account to prepare and build an interactive visualization for each component along with their connections. Some have called it Google Maps for your AWS account, and to put it simply CloudMapper shows how your AWS environment actually looks.

To see the level of interaction check out their online demo of a deployed application in the us-east-1 region. Below is the CloudMapper visualization of the web applicatoin deployment highlighted in several of my Terraform posts.

cloudmapper.png

CloudMapper was built by Scott Piper in conjunction with Duo Security and luckily for us, they have open sourced their work and continue to  maintain it.  Of the three tools mentioned, it definetly provides the most robust view in terms of connectivity and security for visualinzing an AWS environment.  To get started using CloudMapper check out the product page as well as the installation and setup details on GitHub.

Benefits:

  • Especially good for seeing how resources are connected, and visualizing your AWS environment.

  • Interactive web diagram is extremely handy for understanding and validating your deployment.

  • I have found CloudMapper to be the most thorough tool of the three highlighted

  • Free / Open Source

Nice to Haves:

  • Setup is several steps and more involved compared to the other tools. I did run into a number of compatibility issues with some of the backend Python packages - which reminded me that yes, it is open source.

  • Collection phase can take some time, as it is truly pulling all the metadata information for your entire AWS account across all components and regions.

  • Would be nice to have this exported in different formats - currently supports PNG and JSON only. Visio and PDF are some of other formats similar tools support.

  • Support only for AWS, it would nice to be able to see support for other clouds (Azure, GCP, etc.)

VMworld 2018 - Where's Gabe?

Next week I will be attending my fifth VMworld, and have had the fortune of seeing the conference through the eyes of a customer, partner, reseller and blogger.  Last week I received news that VMware would be sponsoring me with a blogger pass to attend this year's show for which I am most grateful.  I find that VMworld is the most valuable for me when I enter with a plan.  I would encourage others - especially first timers - to build out a plan for what the want to get out of the conference.

My blogging and professional development for VMworld 2018 will focus on the areas of automation and orchestration within the VMware ecosystem and across heterogeneous clouds.  I am most interested in further developing deeper product and tooling expertise, while hearing first hand from businesses and community members how they are refining their skill sets to keep up with the demands placed upon them in a multi-cloud world.

vmware {code} Hackathon 2018

One of the events I am most looking forward to this year is the Hackathon.  As a participant two years ago, it is exciting to see the level of attention and interest for this year's event.  This event highlights the creativity, diversity, vulnerability and genuine 'can do' attitude that I have come to love in the VMware community.  Want to get out of your comfort zone and hang with some tremendous people - the Hackathon is for you.  I am not yet formally on a team (remember, I just found out I would be attending the conference last week), but I still have time.  Some of the teams I plan to keep an close eye on are:

  • vMafia 2.0 - Building a vSphere Client Plugin to run PowerShell / PowerCLI within the new HTML5 web console.
  • comdivisin - automation around vRO, PowerCLI
  • VDM-Hackers - creators of VirtualDesignMaster.io, this team is going to set big goals and do great (fun) things
  • Pythonic - build a bolt-on user interface to Vault to make it easy for homelab users to get started with secrets management.
  • View API & PS - improve the vmware.hv.helper module by adding new functions or improving them and if time allows to extend the vCheck for Horizon View
  • vmug-cloud-labs - nested lab builds within VMware Cloud on AWS.
  • PS API Gateway - Building a PowerShell API gateway to interact with all VMware services in a simple standard format.

Tech Field Day - Extra

At VMworld 2015, I had my first in person Tech Field Day experience as a presenter.  The Tech Field Day events bring together innovative IT product vendors and independent thought leaders to share information and opinions in a presentation and discussion format.  Since then I have been fortunate to be a delegate to several Tech Field Day events.  Tuesday afternoon of VMworld 2018, I will again be joining the delegate ranks to engage with what look to be like three interesting companies.  I know very little about each of them and excited to learn more. 

  • JetStream Software - The JetStream Software platform empowers infrastructure spanning multiple data centers and cloud services. Applications enjoy cloud-scale elasticity, and can migrate seamlessly, all with enterprise grade performance, availability and resilience.
  • Quali - Blueprint, model, and deploy on-demand, self-service, sandbox environments for labs, data centers, and hybrid cloud.
  • Runecast - Proactively use VMware KBs, best practices and security hardening guidelines to protect your environment

These sessions will be live streamed, with the recordings and coverage made available on the Tech Field Day website.

Break out Sessions

Signing up for the conference late does come with some disadvantages, and one of those is getting into popular sessions.  None the less there are some great sessions which I plan to take in first hand, and the others I a plan to view online when they become available.

Culture Shift: The Heavy Lifting Behind Digital Transformation [LDT1895BU]
Monday, August 27, 11:30 AM - 12:30 PM | Lagoon D, Level 2

Introduction to Cloud Native App Development on K8s (Maker Space Workshop) [CODE5627U]
Monday, August 27, 01:00 PM - 02:50 PM

Deep Dive: How Fannie Mae Releases vRealize Content Using Lifecycle Manager [MGT2174BU]
Wednesday, August 29, 09:30 AM - 10:30

Instrumenting all things Kubernetes (Maker Space Workshop) [CODE5633U]
Wednesday, August 29, 01:00 PM - 02:50 

Top 5 Processes to Accelerate DevOps with Josh Miller [MTE5074U]
Wednesday, August 29, 03:15 PM - 04:00

Destination Digital: Plot a Course for Transformation with Cloud Migration [LDT2207BU]
Thursday, August 30, 10:30 AM - 11:30 AM

Infrastructure as Code for the Enterprise [CODE5570U]
Thursday, August 30, 12:15 PM - 12:45 PM

Infrastructure as Code(IaC) [VMTN5615U]
Thursday, August 30, 01:45 PM - 02:00 PM

vExpert Session - Mystery CLOUD Session 1 - Tuesday August 28th, 4:15pm-5:15pm Mystery AUTOMATION Session 2 - Wednesday August 29th, 5:15pm-6:15pm

vBrownBag

Schedule and time permitting, I am prepared to present a vBrownBag TechTalk on my recent work with Terraform on AWS and VMware.  I have found that the vBrownBag stage and surrounding area in the Hang Space is a great place to meet and engage with other attendees and community members.  If you would like to connect this is usually the place to find me.

Community, Parties and Events

Of course one of the best things about VMWorld is to interact with the vCommunity, and this community likes to have a good time.  Unfortunately I will not be able to attend vBreakfast, Open Acts or VMunderground this year because I don't arrive until Monday of the show.  You should definitely check out these great community events, as well as the full list of other parties.  Rumor is that some of my favorites including Snoop Dogg, Run DMC & the Roots will be in attendance. I am traveling with some first time Las Vegas goers, so hope to spend some time enjoying the sites and sounds with them.  At this point I am confirmed to be at both the vExpert party and vSoccer events on Tuesday night, but if history is an indicator I will find my way to several others.

Contact Info

Of course, if you are attending the show I would love to connect with you schedules permitting.  Hit me up on twitter @gmaentz or via email at gabe@maentz.net .  I will be heading back home on Thursday evening.