Discovering Private Keys & Certificates in Unsecured File Shares and VMS

Attending a InfraGard event recently, I was made aware of problem that I never gave much thought to before but probably should have - securing the private keys for SSL and SSH certificates.  Much like usernames and passwords, public and private keys for certificates to encrypt and authenticate you to various internet services are critically important to manage and protect.  I also learned that many people don't rotate their certificates nearly as frequently as recommended (how many of us have GoDaddy certificates that are set to expire for 3 years) and often private private keys are saved in simple file and network shares.  Very similar to saving usernames and passwords in an Excel spreadsheet or text file.

Just like a password, to ensure the security of your private key it is best practice to limit access to members of your organization who absolutely need to have control over it. It is also best practice to change your private key (and re-key any associated certificates) if a member of your team who had access to the private key leaves your organization.  The challenge is finding these keys and identifying who is possibly using them.

Identifying Private & Public Keys

There are many samples of private keys and certificates that you can download to see the makeup of a particular crt or key file for user and machine authentication.  Opening these in a common editor will show the how they are crafted:

Building Intelligence to Indentify Keys & Certificates in File Shares

One of the foundational tenets of DataGravity is to utilize intelligence on unstructured file shares and VMs to determine where sensitive data is being saved and accessed.  In this case we will identify and discover private keys/certificates in file shares and VMs with DataGravity's automated detection and intelligence.  This can be done using the Intelligence Management interface, allowing us to create a custom tag, attach it to a Discovery Policy, and then find and be alerted on this information.

We can simply give our new tag a name 'Certificates Key' and then a color indicator for importance (Red is the universal sign for 'Very Important') and a Description.

The pattern we will be looking inside the files to identify if they are a certificates or keys is the 'Begin' and 'End' lines for private keys and certificates.  The Regex expression that I found useful for this is listed below.

(-----(\bBEGIN\b|\bEND\b) ((\bRSA PRIVATE KEY\b)|(\bCERTIFICATE\b))-----)

As seen when testing for the beginning and ending of these certificates my Match Pattern is working as expected.

Now that I have the tools in place to identify Private Keys and Certificates, I simply need to update my Intelligence Profile to automatically discovery when new certificates and keys are found.

Idenfity, Discover, & Notify

With the new intelligence tag for Certificate Key created and applied to my Intelligence Profile, I can very easily search and discover with DataGravity all instances of those files.

1. Search for instances of the newly created tag - Certificates Key
2. Indentify the number of Results
3. Preview any of the Files to confirm that it is a key or certificate
4. You will notice DataGravity also can identify beyond the file extension to find this information.  In this case the private key was saves as a Text file, but we can still see by previewing the file that it contains the private key information.
5. Export or Subscribe to the Search to be notified when Private Keys or Certificates are saved.

Parity throughout the System

The newly created tags now are accessible through search as well as in all of the key visuals provided by DataGravity including: File Analytics, File Details, Activity Reporting and Trending - across file shares and VMs.  Extremely powerful for understanding where this sensitive data lives, who is accessing it, and then notifying other systems for full discovery such as a PKI Key Management system.

Visualizing RVTools

One of the my most popular blog posts stems from a presentation I provided at a local VMUG over 3 years ago. During my presentation, inspired by Kendrick Coleman, I talked about everyday low cost & free tools that help in administering a VMware environment, and one of my favorites is RVTools.  In fact when paired with Microsoft Excel Pivot Tables, RVTools is extremely powerful.

Fast forward a little bit, and as visualization tools have become more accessible and easy to use, I would like to share with you another way that you can utilize RVTools to help visualize your VMware environment.  One such visualization product that I was recently introduced to is Tibco Spotfire Desktop.  Spotfire is not a free tool, although they do offer a 30-day trial which I am using for this post.  Spotfire at it's most basic level, allows you to import an Excel spreadsheet, CSV, and a variety of other data inputs and visualize them in a number of different ways.

I definitely subscribe to a 'picture is worth 1000 words' so naturally I wanted to see how I could use an export of RVTools to create meaningful Spotfire visuals.  I started small, building on my previous RVTools & Pivot Tables post where we categorized our VMs by O/S, Avg. CPU & Avg. Memory Size.  This is a quick recap of what that looked like in tabular form:

The first thing to do is to install Spotfire, which is straight forward, although support is currently limited to only Microsoft operating systems - no love for my Mac.  Once installed it is very simple to open a source file that will provide the data for the visualizations.  In my case I am opening the Excel file that was exported by RVTools and used to create my pivot tables.

I am then prompted to select the Worksheet that we will build from and I selected the tabvinfo Worksheet as it is an excellent source of VM data.

What I really like about Tibco Spotfire is that once the data file is imported, the Recommended Visualization screen appears to help assist with different ways to visualize the data.  Scatter Plots, Graphs, Pie Charts - take your pick.  I love how you can select different data categories and rows and the recommended visuals change dynamically.

Below I picked OS as my building block, and you can see the various visuals instantly provided. These can then be added to your working dashboard/project using the Add this button.

After adding as many visuals as you wold like, it is very straight forward to duplicate, copy or edit any visual.  I changed the Tree Size view to show Avg. CPU and then duplicated it to show Avg. Memory.

I outlined my dashboard in a quadrant view to easily showcase the breakdown of VMs by OS, Avg. CPU, Avg. Memory, and power state - On/Off.  Very powerful and easy way to consume the data - which as you hover over any visual (box, pie slice or bar) will also highlight the equivalent piece of relevant data in the other quadrants.  

So I can see below that Windows Server 2008 R2 64-bit accounts for 33.9% of my VMs, the majority of which are powered on, accounting for fourth largest average number of CPUs, and fifth largest average memory footprint.

I really like the simplicity of the product and plan on building some more visuals, perhaps by examining some further RVTools output.

Utilizing Amazon Web Services (AWS) Storage GAteway to archive backups

Recently I have been exploring some practical, relatively low cost methods for utilizing cloud services in my lab environment.  In this exploration I found a great video tutorial produced by Luke Miller (a Veeam SE Manager) who walks through how to use the AWS Storage Gateway for placing backup copy jobs onto AWS S3 storage.  I liked the video so much that I thought I would implement it in my lab to augment my existing Veeam setup and Vice Versa backup copies.  Needless to say I have been very impressed with the ease of deployment and the effectiveness of using S3 as an archive storage tier for my lab, that I wanted to share my setup for those who may be interested in doing the same.

The Goal

Just a quick overview of the goal:  Utilize a 15TB Amazon S3 volume as a remote backup archive presented to a local on-premises Windows 2012 R2 application server.

Per Amazon: The AWS Storage Gateway is an on-premises virtual appliance that provides seamless and secure integration between your on-premises applications and AWS's storage infrastructure. The service enables you to securely upload data to the AWS cloud for scalable and cost-effective storage. The AWS Storage Gateway allows you to create iSCSI storage volumes, storing all or just your recently accessed data on-premises for low-latency access, while asynchronously uploading this data to Amazon S3, minimizing the need to scale your local storage infrastructure.  The gateway fits into an existing infrastructure as shown in the diagram provided below - in my case the application server will be running Windows 2012 R2, the host is ESXi 5.5 and I am running both direct attached storage and a DataGravity NFS datastore.

Deploying the gateway

This Gateway VM can be accessed and downloaded from the your AWS Management Console, and provides a nice step-by-step walk through to deploy the gateway.  Utilizing the Gateway-Cached volume so that I can store most of the cold, archive data up to Amazon S3.  I plan on testing out the Virtual Tape and Stored Volume configurations as well but that is for another post.

I am utilizing ESXi in my lab, so we will deploying the gateway as an OVA file.  Supported hardware, hypervisor versions, network, etc. are well documented, so I can begin the deployment.

I will assume that if you are reading this you most likely have no issue deploying an OVA file in your environment, but Amazon does have the process very well documented in case you need a reference.  Once deployed on my ESXi host, AWS recommends that you validate NTP for the host and sync the gateway guest VM with the host's time.

Provision Local Disk

The gateway VM does need disk allocated for both it's cache storage and an upload buffer.  The guideline and recommendation is to allocate at least 20 percent of your existing file store for your cache storage, and at least 150 GB as an upload buffer.  For this deployment I will be using (2) 150GB virtual hard disks - one for the cache storage and one for the upload buffer.  For further detail on AWS recommendations for sizing these, there is some great guidelines and recommendations.

It is important to modify the SCSI controller type for these disks to VMware Paravirtual

Now we are ready to power on the Gateway VM, and the last step is to activate it within AWS.  This is done by entering in the IP Address of the Gateway VM once it powers on.  Yes, it is possible to specify a static IP Address for your Gateway by launching it's console and working your way through the network configuration menu, but in my case I utilized the DHCP address that it picks up. 

AWS does charge a $125 per month fee for each activated gateway.  There is a 60 day free trial, which I am currently utilizing to determine if I find value in the service.  I do like this option, and yes I am seeing value.

Configuring the local storage of your VM gateway is really just a matter of specifing which disk will be used for the Cache Storage and which will be utilized for the Upload Buffer.  Both of my disks are equivalent in size, so pretty much a no brainer, but it is something important to take note of.

Lastly we will provision the capacity of the S3 volume that we want to present.  In my case I will be testing with a number of different backup and copy jobs, so I provisioned a 15 TB volume.

Presenting S3 Volume to Windows application server

Now that I have 15 TB of cloud storage to work with, let's complete the deployment by connecting the dots by presenting this storage to my application server.  This works by using using the AWS Gateway VM as an ISCSI target, and in my case utilizing the default Windows 2012 R2 ISCSI initiator.

Once the disk can be seen by the server, then it can be enabled, formated, and assigned a drive letter for use.  I will initially be using this as a Veeam Backup Copy repository, so I assign it the drive letter V:\

Now the disk is ready to use and you can start consuming AWS S3 storage natively via your application server. In my initial use case I am using this space as a Backup Copy repository for my lab backups with Veeam, so that I have an offsite copy of my backups.

Summary

Overall I have been very pleased with the ease of presenting cloud storage to my lab environment, and the AWS storage gateway setup, configuration and execution is extremely straight forward.  After running this for a little while we will see what our monthly bill looks like.  Thanks Luke for the nice video overview.

Finding the right Part

Living in the proverbial "Rust Belt" of the United States, I have the opportunity to work with many manufacturing companies.  These companies are very rich with unstructured data (files, drawings, spreadsheets, orders, bill of materials, etc.) and there is a key piece of information that is often contained in this data: The Part Number.  While finding products by part number is relatively easy when stored in structured table formats like inventory databases, finding the supporting reference documentation through a part number lookup is not trivial.  Let's look how DataGravity can make that easier.

I say tomato, You say tomato

The first task is to identify the right part number to look for, which can be complicated because every manufacturer typically utilizes a different nomenclature for identifying a part. For example, when referring to a "Hardware, screw, machine, 4-40, 3/4" long, panhead, Phillips" some manufacturers may articulate that product as follows:

Manufacturer A uses part number "4-40-3/4"-pan-phil",
Manufacturer B uses part number "100-440-0.750-3434-A".
Manufacturer C uses part number "TSR-1002".

Many companies will use their own part numbering system to help standardize, but can also certainly complicate things because many times the same product by be referenced by different part numbers.  This makes finding parts quickly amongst a vast amount of product documentation near impossible. DataGravity's intelligence management eases this process by providing the ability to define and identify multiple part numbers by their various nomenclatures.

This can be readily done by leveraging the custom tags available in the system (not to mention all of the amazing other meta tags that ship standard with the system) and defining the part number(s) of interest. 

The custom tags allow us to define what the Tag name will be, in this case 'Part Number' and then what items or patterns to look for to qualify as a tag.  This way we can look for 4 different part numbers which all may refer to the same product without issue or complexity.  We can even test our patterns when defining the tag to be sure it is going to pull back what we would expect.

Now that we have defined what we are looking for, let's specify that we should look for this information on our files shares or VMs.  This is done by adding our the Part Number tag to the file share profile.

the results

Once the tag has been created and applied to the shares and/or VMs of interest - we have full access to utilize that tag within the search functions of the system.  This allows us to simply search by Part Number and find all of the supporting documentation and files of reference that contain that data - complete with a preview of the files themselves.

In the second half of this post, we will explore how we can use the power of this newly created tag to not only identify the data that holds these part numbers, but how this can assist us when part numbers need to be updated.