Tuesday, April 28, 2015

VNS3 API #3: Launch and configure VNS3 with a single script

This is my third post in a series looking at using the VNS3 API, having previously covered clientpacks and snapshots.

Dependencies


This script needs bash, curl and the AWS CLI. You can install these yourself, or use a Docker container that already has the required dependencies e.g.

 sudo docker run -it cohesivenet/awscli  

Before starting make sure that your AWS API key, secret key and region are configured:

 aws configure  

The script


View or download the full script from GitHub Gist. This post will go through each section and explain what's going on.

Variables


 #!/bin/bash  
 VNS3_AMI=ami-ea084f82  
 VNS3_TYPE=t2.small  
 VNS3_SUBNET=subnet-b123b456  
 VNS3_GROUP=sg-ea123456  
 VNS3_NAME=myVNS3  
 VNS3_PW=pa55Word  
 VNS3_LIC=license.gpg  
 VNS3_TK=MyS3cret  

After the hashbang for bash the script begins by setting variables:
  • VNS3_AMI is the Amazon Machine Image that will be launched. ami-ea084f82 is the latest version at the time of writing in the US-East-1 region (3.5.0.7). Ask support@cohesive.net to enable the appropriate AMI for your account in the region you're using.
  • VNS3_TYPE is the EC2 instance type. t2.small is fine for test installations, but m3.medium is generally the smallest recommended for production.
  • VNS3_SUBNET defines the VPC subnet that the VNS3 will be deployed in. So that it can be reached later it should be a public subnet (e.g. one that has an Internet gateway configured for its default route).
  • VNS3_GROUP is the security group that the VNS3 will run with. This should allow access on TCP:8000 from the machine running the script (otherwise the VNS3 API will be unreachable). The group should also allow UDP:1194 from the group that VNS3 clients will be started in.
  • VNS3_NAME is the name that will be used to tag the VNS3 instance on EC2 and within the VNS3 admin console.
  • VNS3_PW is the password used by the API and the admin console
  • VNS3_LIC is the name of the license file provided by support@cohesive.net
  • VNS3_TK is the security token for the topology (used to peer additional managers in a multi manager system)

Allocating an Elastic IP

Although the VNS3 instance could be started with a public IP it's best practice to use an EIP as that can then be moved onto another instance if there ends up being an issue with the initial instance.

 # Allocate an EIP  
 EIP=$(aws ec2 allocate-address --domain vpc --output text)  
 EIP_IP=$(echo $EIP | cut -d ' ' -f3)  
 EIP_ID=$(echo $EIP | cut -d ' ' -f1)  
 echo EIP allocated $EIP_ID $EIP_IP  

The script calls 'allocate-address' and passes the output of that command as text to the variable EIP. The EIP variable is then cut into an IP and an ID for use later on, and those variables are echoed to show progress.

Launching the VNS3 instance

The main command here is mostly taking the parameters from the variables at the start and using them to launch an instance of VNS3.

 # Launch instance  
 VNS3_ID=$(aws ec2 run-instances --image-id $VNS3_AMI --count 1 --instance-type \  
 $VNS3_TYPE --subnet-id $VNS3_SUBNET --security-group-ids $VNS3_GROUP \  
 --no-associate-public-ip-address --block-device-mappings \  
 "[{\"DeviceName\": \"/dev/sda1\",\"Ebs\":{\"VolumeType\":\"gp2\"}}]" \  
 --query Instances[].InstanceId --output text)  
 echo Instance launched $VNS3_ID  

The two noteworthy things are:
  1. Use of --block-device-mappings to specify a general purpose SSD for the primary volume.
  2. The combination of --output text and --query to retrieve the instance ID, which is used elsewhere in the script.
The instance ID is echoed to show progress.

Naming the instance

When launching an instance from the AWS web interface it's possible to add tags including the name, but this needs a separate command when using the CLI. The run-instances command doesn't have a documented tag parameter (though the example output does show name tags). This can be made up for using the create-tags command:

 # Name the instance  
 aws ec2 create-tags --resources $VNS3_ID --tags Key=Name,Value=$VNS3_NAME  
 echo Instance named $VNS3_NAME and waiting for it to be running  

Assigning the Elastic IP

The instance can be named straight away, but the EIP can't be assigned until later in its life cycle:

 # Wait for instance to be running  
 aws ec2 wait instance-running --instance-ids $VNS3_ID  
 # Assign EIP  
 EIP_ASSOC=$(aws ec2 associate-address --instance-id $VNS3_ID \  
 --allocation-id $EIP_ID)  
 echo Assigned IP $EIP_IP  

It's worth noting at this stage that the script isn't doing any error checking, so even if the assignment fails there will still be a console message saying that the IP has been assigned.

Waiting for the VNS3 API

The startup of VNS3 services isn't directly related to the life cycle of the instance as seen by AWS. The first wait is for the instance to pass status checks, and then the VNS3 API is polled until it's available:

 # Wait for instance to pass status checks and for API to be available  
 aws ec2 wait instance-status-ok --instance-ids $VNS3_ID  
 VNS3_UP=`curl --silent -k -X GET -u api:$VNS3_ID \  
 https://$EIP_IP:8000/api/status/system | grep licensed`  
 while [ "$VNS3_UP" = "" ]  
 do  
   sleep 15  
   echo -n "."  
   VNS3_UP=`curl --silent -k -X GET -u api:$VNS3_ID \  
   https://$EIP_IP:8000/api/status/system | grep licensed`  
 done  

The system status call isn't actually available for use until after licensing, but that doesn't matter since the response that it's not available is sufficient to show that the API is ready for action.

Reseting the API password

On AWS the default password for VNS3 is the instance ID, which is a 'secret' that should only be known to the person who launched it. This section changes that password to one defined in the script:

 # Reset API password  
 APIPW_ST=$(curl --silent -k -X PUT -u api:$VNS3_ID -d \  
 '{"password":"'$VNS3_PW'"}' -H 'Content-Type:application/json' \  
 https://$EIP_IP:8000/api/api_password)  
 echo API password set to $VNS3_PW  

Installing the license

First the license file is uploaded, and then it's set using defaults, which causes the VNS3 to restart:

 # Upload license file  
 LICFL_ST=$(curl --silent -k -X PUT -u api:$VNS3_PW --data-binary @$VNS3_LIC \  
 -H 'Content-Type:text/plain' https://$EIP_IP:8000/api/license)  
 echo License file uploaded  
 # Set license (using defaults)  
 LICEN_ST=$(curl --silent -k -X PUT -u api:$VNS3_PW -d '{"default":true}' \  
 -H 'Content-Type: application/json' \  
 https://$EIP_IP:8000/api/license/parameters)  
 echo License set with defaults  

At this stage the import snapshot API call could be used if creating a clone of an existing VNS3 manager rather than making a new one.

Waiting for VNS3 to restart

Now that VNS3 is licensed the system status API will return uptime once it's available:

 # Wait for instance to restart  
 echo Waiting for VNS3 to restart  
 VNS3_UP=`curl --silent -k -X GET -u api:$VNS3_PW \  
 https://$EIP_IP:8000/api/status/system | grep uptime`  
 while [ "$VNS3_UP" = "" ]  
 do  
   sleep 15  
   echo -n "."  
   VNS3_UP=`curl --silent -k -X GET -u api:$VNS3_PW \  
   https://$EIP_IP:8000/api/status/system | grep uptime`  
 done  
 echo VNS3 restarted  

Resetting the user password

The user password is set to the same value as the API password (though an extra variable could be used to make them different):

 # Reset user password  
 USRPW_ST=$(curl --silent -k -X PUT -u api:$VNS3_PW -d \  
 '{"enabled":"true","admin_username":"vnscubed","admin_password":"'$VNS3_PW'"}' \  
 -H 'Content-Type:application/json' https://$EIP_IP:8000/api/admin_ui)  
 echo User password set to $VNS3_PW  

Generating the keyset

Each VNS3 overlay network needs a unique set of cryptographic keys that are generated by the manager. The process is CPU intensive, and takes a little time, so the script has to wait for it to be completed:

 # Generate keyset  
 KEYGN_ST=$(curl --silent -k -X PUT -u api:$VNS3_PW -d \  
 '{"token":"'$VNS3_TK'", "topology_name":"'$VNS3_NAME'"}' \  
 -H 'Content-Type:application/json' https://$EIP_IP:8000/api/keyset)  
 echo Generating keyset  
 export KEYSET_ST=$(curl --silent -k -X GET -u api:$VNS3_PW \  
 https://$EIP_IP:8000/api/keyset | grep false)  
 while [ "$KEYSET_ST" != "" ]  
 do  
   sleep 15  
   echo -n "."  
   export KEYSET_ST=$(curl --silent -k -X GET -u api:$VNS3_PW \  
   https://$EIP_IP:8000/api/keyset | grep false)  
 done  
 echo Keyset generated  

Manager peering

The final step is to make the VNS3 manager a peer (with itself) so that the overlay network becomes active:

 # Make self peer  
 export MGRPR_ST=$(curl --silent -k -X PUT -u api:$VNS3_PW -d '{"id":"1"}' \  
 -H 'Content-Type:application/json' https://$EIP_IP:8000/api/peering/self)  
 echo Self peered  

Fin

When the script completes it echoes the URL to access the manager UI (though at this stage the manager is ready to be used by other scripts that might be part of automated orchestrations such as configuring clientpacks for servers joining an overlay):

 echo 'All ready to go at https://'$EIP_IP':8000'  

Acknowledgement


I'd like to thank Amazon's Martin Elwin for his excellent 'Deep Dive - Advanced Usage of the AWS CLI' at the recent AWS Summit in London, as the guidance he provided was very helpful in crafting the AWS pieces of this script.

Friday, April 24, 2015

Security, Cloud and Networking Weekly News Roundup: April 20 - 23

This week's news in cloud, networking, and security - week of April 20th: 

Until this week, Amazon's Web Services figures were buried in overall North American Sales “Other” category. We now know that in Q1 2015, AWS logged $1.57 billion in revenue, up 49% from the year-ago period. More impressively, AWS logged operating income of $265M for the quarter, up from $245 million in 2014. Bezos: Amazon Web Services is a $5 billion business—and growing fast by Barb Darrow via Fortune

A study from CloudLock found that organizations have an average of 1.2M files stored in the cloud, 10x that of last year. But security is still an overlooked aspect of cloud usage, with an average of 4,000 instances of exposed credentials. "Cloud Cybersecurity Report: The Extended Perimeter," via SC Magazine

Microsoft's cloud, which consists of both their Azure and Office 365 products, reported a 106% growth in revenue and now is on an annualized revenue run rate of $6.3 billion. Microsoft Sales Rise, Earnings Slip for FQ3 on Strong Cloud Business Sales via the VAR Guy


HP announces new cyber security approach at RSA, with “new school of cyber defense” with partners FireEye (FEYE), Securonix and Adallom. The new approach involves protecting the interactions between users, applications, and data exchanges. Where perimeter defense once was the dominant thinking, now analytics, data-centric cloud access protection, mobile application reputation analysis, contextual threat intelligence sharing, and incident response are primary tools. HP, FireEye, Others in “New School” Cybersecurity Collaborations via The VAR Guy

"Network virtualization is becoming one of the key elements of any modern cloud deployment. Cohesive's VNS3 technology is ideally suited to many cloud network designs and provides fast access to further enhance network technologies. With added security and connectivity from VNS3, our CenturyLink Cloud customers can easily connect flexible, secure networks to customers and partners around the globe," said David Shacochis, vice president of Cloud Platform at CenturyLink.  Learn more about VNS3 in the CenturyLink Cloud Marketplace



Catch up with Cohesive Networks: 

  • 20 - 24 April - attending RSA Conference in San Francisco
  • 24 April - Chris Swan speaking at Commonwealth Cybersecurity Forum at 12:10 'Where is my big data: security, privacy and jurisdictions in the cloud'
  • 30 April - CloudCamp London "Containers Everywhere, what's all the fuss about?"
  • 30 April - sponsoring & hosting the Secret Service Chicago Electronic Crimes Task Force April meeting
  • 11 May - CloudCamp Chicago "Big Data"
  • 2 - 4 June - attending InfoSec London
  • 24 - 25 June - exhibiting, sponsoring & speaking at Cloud World Forum
  • 1 July AWS Summit Chicago 

Friday, April 17, 2015

Security, Cloud and Networking Weekly News Roundup: April 13 - 17



This week's news in cloud, networking, and security - week of April 13th: 

The PCI Security Standards Council (PCI SSC) version 1.1. of the PCI Card Production Security Requirements is now available. The updated standard includes physical and logical security measures for any organization handling credit card payments, and version 1.1 includes new requirements for firewalls, access controls, and cryptographic keys. New security requirements for payment card vendors via Net Security 

66% of manufacturers surveyed by IDC report using more than 2 applications in the public cloud. Most report moving IT operations to the cloud first, but 30 – 35% of respondents indicate operations, supply chain and logistics, sales, or engineering expect to benefit from cloud. Majority of Manufacturers Worldwide Using Public or Private Cloud via Finchannel

The Verizon 2015 Data Breach Investigations Report found 10 of the top attack patterns accounted for 96% of data breaches in 2014. The biggest attach vector (28.5%) involved point of sale (POS) systems, 19% used malware, 18% were espionage, and 10% took advantage of insider misuse.
Image via Verizon Data Breach Investigations Report 2015


Equinix announces sixth London datacentre via Business Cloud News

Nokia (NOK) plans to acquire French networking equipment-maker Alcatel-Lucent for $16.6B. The move is intended to put Nokia into the cloud services and Internet of Things markets. Nokia Announces Alcatel-Lucent Acquisition - News Roundup via the VAR Guy
  • 20 - 24 April - attending RSA Conference in San Francisco
  • 24 April - Chris Swan speaking at Commonwealth Cybersecurity Forum at 12:10 'Where is my big data: security, privacy and jurisdictions in the cloud'
  • 30 April - CloudCamp London "Containers Everywhere, what's all the fuss about?"
  • 30 April - sponsoring & hosting the Secret Service Chicago Electronic Crimes Task Force April meeting
  • 11 May - CloudCamp Chicago "Big Data"
  • 2 - 4 June - attending InfoSec London
  • 24 - 25 June - exhibiting, sponsoring & speaking at Cloud World Forum
  • 1 July AWS Summit Chicago 

Friday, April 10, 2015

Security, Cloud and Networking Weekly News Roundup: April 6 - 10

This week's news in cloud, networking, and security - week of April 6th: 

Why 2015 will be the year that the cloud comes of age via The Next Web
Image Credit: the Next Web



At this week's AWS Summit in San Francisco AWS chief Andrew Jassy announced the Elastic File System, a "file system that grows and shrinks, automatically." AWS also announced a new Machine Learning system along with 2 marketplaces for desktop applications and WorkSpaces applications. Amazon Opens ‘Marketplace’ For Apps, Talks of Stealing Enterprise Workloads via Barron's

HP announces their withdraw from public cloud this week in the New York Times article HP Comes to Terms with the Cloud. HP, which recently split into 2 companies to focus on business technology and one on consumer-facing personal computers and printers, will continue to selling servers for large enterprises and cloud companies.

Research from Accenture and Ponemon Institute examines the difference between proactive (or "leapfrog" in the report) companies and static organizations react and value security. "[Proactive] companies exceed Static companies in viewing the following features of security technologies as very important: pinpointing anomalies in network traffic; prioritizing threats, vulnerabilities and attacks; curtailing unauthorized sharing of sensitive or confidential data; and enabling adaptive perimeter controls."

The U.S. National Institutes of Health will now allow researchers to use cloud services to store and analyze data in genetics research. Cloud services must meet NIH data-use and security standards, and major cloud providers such as Amazon AWS, Microsoft, and Google already comply. The cloud scores NIH approval for gene research via Info World

Upcoming events Cohesive is hosting and attending:
  • 15 April attending the AWS Summit London at ExCel 
  • 20 - 24 April - attending RSA Conference in San Francisco
  • 30 April CloudCamp London "Containers Everywhere, what's all the fuss about?"
  • 30 April - Cohesive Sponsoring & hosting the Secret Service Chicago Electronic Crimes Task Force April meeting
  • 12 May CloudCamp Chicago "Big Data"
  • 2 - 4 June InfoSec London
  • 1 July AWS Summit Chicago 

Wednesday, April 8, 2015

VNS3 API examples #2 Snapshots

VNS3 sits at the confluence of network function virtualisation (NFV) - networks made out of software, and software defined networking (SDN) - networks configured by software. The key to that configuration is our API, which is thoroughly documented in the VNS3 3.5 API Instructions.

The aim of this blog series is to provide some practical examples of using the API to perform typical administration tasks with VNS3. Each example will be illustrated by a simple shell script.
Let your API take notes for you. Image credit: Flickr user marcoarment

Firstly some conventions


All of these scripts need to connect to the manager and make use of the API password. Each example will use a manager IP (MGRIP) of '10.0.0.10' and an API password (APIPW) set to 'pa55Word'. Please substitute the correct values in your own scripts, and note that the manager IP will generally need to be its internal IP address.

Dependencies


This script needs bash and curl to be present.

Creating and fetching a snapshot


 #!/bin/bash  
 command -v curl >/dev/null 2>&1 || { echo "This script requires curl, but it's not installed. Aborting." >&2; exit 1; }  
 MGRIP=10.0.0.10   
 APIPW=pa55Word   
 SNAPSHOTJSON=`curl -k -X POST -u api:$APIPW https://$MGRIP:8000/api/snapshots`  
 SNAPSHOTNAME=$(echo $SNAPSHOTJSON | grep -Po '"response":{.*?[^\\]"' | awk '{split($0,a,"{"); print a[2]}' )  
 curl -k -X GET -H 'Content-Type: application/json' https://api:$APIPW@$MGRIP:8000/api/snapshots/${SNAPSHOTNAME//\"} -o ${SNAPSHOTNAME//\"}  
 echo "Created and retrieved $SNAPSHOTNAME"  



View or download this script from GitHub Gist.

The script doesn't take any parameters, and saves the newly created snapshot using its date/time based name:
 $ ./snapshot.sh  
  % Total  % Received % Xferd Average Speed  Time  Time   Time Current  
                  Dload Upload  Total  Spent  Left Speed  
 100  201 100  201  0   0  160   0 0:00:01 0:00:01 --:--:--  160  
  % Total  % Received % Xferd Average Speed  Time  Time   Time Current  
                  Dload Upload  Total  Spent  Left Speed  
 100 935k 100 935k  0   0 3538k   0 --:--:-- --:--:-- --:--:-- 3556k  
 Created and retrieved "snapshot_20150318_1426685707_10.0.0.10"  

Share this Post

Related Posts Plugin for WordPress, Blogger...