The purpose of this document is to provide information necessary to install and run the wintermute.ai server software in the current release.

wintermute.ai server is a Cloud Access Security Broker (CASB) which acts as an IaaS encryption gateway, secure IaaS gateway and cloud broker. It ensures regulatory compliance with company guidelines for IaaS cloud access for corporate users.

1.1  Structure of this document

1.1.1  Chapters:

This Early Access Release means that users will have free, time-limited access to the beta version s of the wintermute.ai server software.

More information about the conditions for using of this software can be found in the EARLY ACCESS AND EVALUATION END USER LICENSE AGREEMENT in the Appendix of this document.

This RC2 release contains only a limited number of the features which will be included in the final release.

RC2 release provides the following unique features:

3.1 Supported Clouds

wintermute.ai server supports following cloud technologies and providers:

• Amazon Web Services (AWS)
• OpenStack based IaaS providers

3.2  In-House

wintermute.ai server runs inside a company on the company's own hardware.

• Cloud providers do not have any access to it.
• All security-related data, such as encryption or SSH keys, are stored only on the company's hardware.
• Users have full control over the server and all the data which it handles.

3.3  Storage Encryption

wintermute.ai server automatically encrypts all used cloud block storage (EBS) transparently in the background.

• All encryption is done transparently in the background.
• Automatic encryption key management also runs in the background.

3.4  Encryption Keys

wintermute.ai server automatically manages all storage and SSH encryption keys.

• Keys are stored and managed automatically.
• Cloud providers have no access to the encryption keys.
• Storage encryption keys are never stored in the cloud.
• Only public SSH keys are injected into the cloud instance; private keys are always saved inside the wintermute.ai server .

• Key export may be prevented to ensure there can be no bypassing of the wintermute.ai server for cloud and data access.

• As all keys are saved inside the company, if a particular user is not available, recovery can be done by another administrator.

3.5  Easy Access

3.6  Compliant

wintermute.ai server ensures that IaaS cloud computing complies with company rules, leaving a company's cloud users free to get on with tasks within specific boundaries.

• Cloud providers and locations, which certain users are allowed to use, can be defined.
• Server enables specification of allowed instance images and instance types.
• Administrators are allowed to define, restrict and control which data is used in the cloud.

3.7  Hardening

wintermute.ai server hardens and automatically secures a company's cloud infrastructure in line with tailored corporate guidelines.

3.9  Cloud Automation

3.10  Web SSH Client

3.11 Customization

This Quickstart Reference Guide provides step-by-step instructions on how to install a working wintermute.ai server without diving too deep into the details. As example Amazon Web Services are used as IaaS service provider.

4.1  Set up wintermute.ai server host machine

• Ensure the machine meets the Hardware Requirements , Operating System Requirements and the General Requirements in the Installation Guide chapter of this document.
• Ensure host's packaging system can install additional software (also from the Internet) on demand.
• Ensure sudo rights are available on the host machine.
• Ensure Amazon Web Services (AWS) Access Key ID and Secret Access Key are available.

4.2  Install wintermute.ai server software

• Install wintermute.ai server software according to the Software Installation instructions in the Installation Guide chapter of this document.

4.3  Login in to the wintermute.ai serve r

• Extract from the installation log file the login URL, login name and initial login password as described in First Login in the Admin Guide chapter of this document.
• Log in to the wintermute.ai server and change the default password as described in First L ogin in the Admin Guide chapter of this document.

4.4  Install FREE trial license

• Create a license request according to First Login in the Admin Guide chapter of this document.
• Send this request to license@wintermute.ai ; a FREE trial license will be sent back within 24 hours.
• Install the license as described in First Login in the Admin Guide chapter of this document.

4.5  Configure Cloud Provider

• Configure and enable an AWS account on the wintermute.ai server as described in Cloud Provider in the Admin Guide chapter of this document.

4.6  Configure Instance Image

• Select a Debian 8.6 image in the most suitable region.

• Import the selected image by using the Import Image String and following the instructions in Instance Images in the Admin Guide chapter of this document.
• Edit the imported Image as described in Instance Images in the Admin Guide chapter of this document:
  • Set type of image to "Debian".
  • Set image to "active".

4.7  Start Test Instance

Create a new instance as described in Instances in the User Guide chapter of this document and using the following settings (region and image name may vary so adapt them as required):

4.7.1  m3.medium instance type with Debian 8.6 image

4.7.2  No additional storage for this test run

4.7.3  Activate only updates and reboot

4.7.4  Launch the instance

4.7.5  Connect to the machine

When the machine status is 'running', open an ssh connection to it as follows:

4.7.6  Terminate machine after tests

4.8  Next Steps

After this quick setup and test, the user should finalize the proper configuration of the wintermute.ai server as described in the Admin Guide chapter of this document.

5.1.1  General Requirements

The wintermute.ai server contains sensitive information like encryption keys and cloud credentials. Therefore the following safety measures should be considered when setting it up:

• If no external access is required, the wintermute.ai server should be installed inside the company behind a firewall.
• If external access (e.g. by external workers) is required, the wintermute.ai server should be placed inside a DMZ and external access to it should be strictly regulated.

• If data transfer to and from the cloud will be performed, only the required data storage segments (e.g. Network File System shares) should be mounted on the wintermute.ai server .

• Console/SSH login to the wintermute.ai server should be allowed only for admin users.
• As the SSH and storage encryption keys are generated on demand and saved only inside the wintermute.ai server , the wintermute.ai server and it main directories (see Appendix ) should be included in the corporate backup. If the wintermute.ai server fails (e.g. hard disk error) there will be no means beside backup to recover the generated encryption and SSH keys.

5.1.2  Hardware Requirements

Virtual or real hardware with the following minimum requirements can be used:

5.1.3  Operating System Requirements

wintermute.ai server runs on Linux and requires Python 2.7.X to be available on the system. The following distributions and version are supported out-of-the box:

Vendor	Version
Debian	8.x, 9.x
Ubuntu	14.04, 16.04, 16.10, 17.04, 17.10
RedHat Enterprise Linux (RHEL)	6.x , 7.x
CentOS	6.x , 7.x
Fedora	24, 25, 26

5.1.4  Required Software Packages

The wintermute.ai server installation script setup.sh requires that the underlying software management system is properly configured and allows additional installation of required software packages, e.g. on a RedHat-based system, a " yum -y install python-virtualenv " call should either report that this package is already installed, or install it and all the dependent packages on the fly.

The system where the wintermute.ai server should be installed must meet all the requirements described in the previous sections ( Hardware Requirements , Operating System Requirements , Required Software Packages ) before the installation process can be started.

1. Extract the downloaded wintermute.ai server software package to a temporary location.

   > tar -C /tmp -xvzf wmais-1.0.0rc2-debian8-x86_64.tgz
   ...
   >

2. Change to the newly created product directory, which has the same name as the software package but without the .tgz extension.

   > cd /tmp/wmais-1.0.0rc2-debian8-x86_64
   > 

   
   

3. Start the installation process by calling the setup.sh script.
   If installing on a system with manual Python 2.7.x installation (e.g. RHEL6, CentOS6) use two additional call parameters when executing setup.sh . For more details see Appendix: Manual installation: Python 2.7.x

   > sudo -H ./setup.sh
   ...

   
   

   
   

4. If the installation finishes successfully , output similar to that in the screen shot below will be displayed. If not, the installation log file in /var/log/wmais/install.log should be analyzed ( use less -r to display the color codes in the log file properly).

   

5. Confirmation that the wintermute.ai server components are running correctly can be done by executing the following shell command:

   > service wmais status
   ...
   >

Following a successful wintermute.ai server installation some post-install configuration changes can be made. This should be done before starting any production work.

All changes should be applied to the /opt/wmai/ VERSION /wmais/settings.py file (replace VERSION with the current version of the wintermute.ai server ). There are more variables in the settings.py file which can be fine tuned later; they are described in the Appendix: Fine Tuning section.

6.1.1  SSH access to instances

As most cloud instances will run with public Internet addresses they will be vulnerable to different kind of attacks. So, for security reasons it's a good idea to restrict access to the running instances and only open to the public those services which are necessary. Secure Shell (SSH) access, which is used by the wintermute.ai server for instance managing, should be open for this purpose. An own security group can be managed inside the cloud provider; the following settings may be changed in line with the type of internet access used:

WMAIS_SG_IP_ADDRESS = "static"

Variable WMAIS_SG_IP_ADDRESS can have the following settings:

• static - means the wintermute.ai server will access the cloud instances from one static address or address pool from one IP range. The wintermute.ai server will use a web service, which is defined in the WMAIS_SG_AUTO_IP_DISCOVER variable, to discover its own static address.
  
  
• 1 . 2.3.4 - a static IP address, from which the wintermute.ai server will access the cloud instances, can be manually defined.
  
  
• dynamic - this keyword indicates that the wintermute.ai server will access instances from different IPs so the access to the SSH service won't be restricted by firewall rules and can be accessed from every IP address.

WMAIS_SG_IP_CIDR = "/32"

Where WMAIS_SG_IP_ADDRESS is set to a static or manual IP definition and there is a range of proxies or gateways from where wintermute.ai server access will originate, the range can be defined by modifying the CIDR setting.

WMAIS_SG_AUTO_IP_DISCOVER = "http://checkip.amazonaws.com/"

This variable defines the accessible web service which returns the IP address of the current request. It's used to auto-discover the IP address from which the wintermute.ai server will access the cloud instances.

6.1.2  Data Settings

This setting defines the file system location from where data can be transfered to and from the cloud.

WMAIS_DATA_BASE_PATH = "/home/%USER%/cloud"

The macro %USER% will be replaced with the login name of the current user.

With the above setting a user with the login name "bob123" could only synchronize data with the cloud from the directory /home/bob123/cloud . In order to avoid a breakout from the defined data base path, the wintermute.ai server will skip transfer of any symbolic links.

The data base path can be any path which is accessible in the wintermute.ai server file system so it could also point to a network mounted storage area, e.g.: /nfs/storage1/cloud/%USER%

6.1.3  Time Zone

Define the local time zone by adjusting the following variable:

WMAIS_DEFAULT_TIMEZONE = 'Europe/Berlin'

After a successful wintermute.ai server installation the login information and a temporary password which was randomly chosen during the installation is displayed in a similar format to that shown in the following screen shot:

If the output wasn't saved after the installation it can be found in the installation log file: /var/log/wmais/install.log

6.2.1  Logging in

In a browser, open the URL contained in the final installation report and log in as an admin user with the temporary password.

6.2.2 Change Admin Password

It's very important that right after the installation, at the first login, the admin password is changed. This can be done by clicking the person icon in the upper right corner of the page and then selecting "Profile: admin" from the menu.

In the "Edit User" dialog, the admin password and other details can be changed.

6.2.3  Request License

A freshly installed wintermute.ai server requires a license to become fullly functional. An Early Access user can get this license for FREE. Execute the following steps:

Click the person icon in the upper right corner of the page and then select "License" from the menu.

In the License dialog, click "Request New License" to get a License Request for the current wintermute.ai server machine.

The License Request (text between +request and -request marker) should be send as email to license@wintermute.ai

Within the next 24 hours a FREE evaluation license will be issued. It can be installed by clicking on the "Install License" button in the License dialog.

After the license is installed the wintermute.ai server will be fully functional.

Before cloud services can be used at least one cloud provider account must be configured.

6.3.1 Amazon Web Services

To configure Amazon Web Services account: select "Settings" from the wintermute.ai server menu and click "New Provider" -> "AWS" to start the configuration.

The following settings can be configured in the "AWS Settings" dialog:

Tab	Configuration Item	Description
Credentials	Name	Provides a meaningful name of the current AWS access credentials. Multiple AWS credential can be registered with different names.
	Description	Provides a short description about the current cloud provider configuration.
	Access Key Id	This key is a unique identifier similar to a username. It can be requested from AWS IAM console.
	Secret Access Key	This is a secret key similar to a password. It can be requested from the AWS IAM console.
Regions	Regions	Defines in which regions wintermute.ai server users are allowed to use cloud resources.
Instance Types	Instance Types	Specifies which instance types (or virtual machines) wintermute.ai server users are allowed to use. The bigger the amount of CPU cores and storage selected, the more expensive it is. Consult the https://aws.amazon.com/ec2/instance-types/ page for different instance types and their costs.
Summary	n/a	A short overview over the current settings is given in this tab. If everything is properly configured, click "Save" to activate the configuration.

Fill all the required login data as described in the above table and press "Synchronize configuration" button to verify your login details and to load dynamic Regions and Instance Types from the cloud vendor.

If the correct login credentials were provided the "Configurations synchronization" task should finish successful and three additional tabs should become visible. Click on the "Regions" tab and select the region to which access with the current provider account should be granted. Click on "Instance Types" tab and select the instance types which should be available in the current account.

After saving the new configuration in the "Summary" tab, it will appear in the "Settings" menu and can be changed at anytime by selecting "Action"→"Edit".

6.3.2 OpenStack based IaaS Providers

To configure an OpenStack based IaaS account: select "Settings" from the wintermute.ai server menu and click "New Provider" -> "OpenStack" to start the configuration.

The following settings can be configured in the "OpenStack Settings" dialog:

If the correct login credentials were provided the "Configurations synchronization" task should finish successful and three additional tabs should become visible.  Click on "Instance Types" tab and select the instance types which should be available in the current account. After saving the new configuration in the "Summary" tab, it will appear in the "Settings" menu and can be changed at anytime by selecting "Action"→"Edit".

6.3.3 Provider Permissions

following permissions settings can be changed by clicking "Action"→"Edit security". And allows the corresponding users group to:

Access Setting	Allows
Create instances	start new instances at this provider
Create instance from template	start new instances based on saved templates
Create instance template	create templates, which is a saved start up scenario of an instance
Create volume	create and use block storage volumes at this provider
Import images	import instance images from this provider
Import volumes	import already existing
View provider configuration	this is the most important setting which allows the users group to use the provider at all. Without this the provider won't be displayed at users menus.

Instance images are templates for the creation of virtual servers. Each image includes a template for the root volume which generally contains an operating system and additional applications. Users can select an image provided by the cloud provider or the user community, or create and upload their own. Images have unique id's in each region they are available. That means the same Debian 8.0 image can have different id's in different regions.

wintermute.ai server allows for the selection of available images from the cloud provider registry. Image creation and the uploading of images are currently not supported by wintermute.ai server . If a user would like to use their own instance image they would need to upload it to the provider registry using the tools supplied by the cloud provider. After doing this, the image will be available for use on the wintermute.ai server .

For security reasons it's highly recommended that a user uses only their own or official images from trustworthy sources, like Linux Distribution Providers.

6.4.1 AWS

The following table provides a short overview about cloud image lists provided by the major Linux Distributions for Amazon Web Services:

6.4.2 OpenStack

OpenStack manages a list of official OpenStack images from the major Linux Distributions, this list can be found here: http://docs.openstack.org/image-guide/obtain-images.html

6.4.3  Supported Client Distributions

following official vendor images were tested as wintermute.ai clients and are known to be working fine with this release of wintermute.ai server :

6.4.4  Import Image

To make images available for wintermute.ai server , click on the "Images" menu and select "Import Image" -> "AWS". The AWS cloud provider must be properly configured before this can be done.

wintermute.ai server will now import a full list from the cloud provider which can take a while. This list can be very long, and, without sufficient information, it may be difficult to ascertan whether the image is trustworthy or not.

It is therefore recommended that the id of the desired cloud image is identified and used to search for the image  in the register. The search can be performed by defining the id in the form image_id:XXXXX in the search field of the images dialog, where XXXXX is the id of the image. Click "Refresh" to search for and display the results.

After selecting the desired image from the import or search results list, click "Import Image" to import the image into the wintermute.ai server .

Image attributes can be changed using the following settings:

Attribute	Description
Name	Name of the corresponding image. Usally this name can be retained as it was imported from the provider's registry.
Type	Select the type of Operating System which this image contains. Not defining this setting properly can lead to a non-functional image inside the wintermute.ai server !
Description	Provide a short description of this image. This description will be displayed later in the selection menu and will be the only way to identify the image. e.g. "Ubuntu 15.10".
Enabled	Only after enabling this check box the image will be activated inside the wintermute.ai server .
User	This setting is only required if a user's images or non-standard images are used. Enter the user name of the image admin user where the SSH key is injected. This field can be left blank if official images of the supported Linux Distributions (Debian, Ubuntu, Fedora, CentOS, RHEL, SLES) are used or if the image uses "ec2-user" as the system admin user.

After saving the changes, the image will become available for use inside the wintermute.ai server .

Image Permissions can be changed by clicking "Edit security" menu and and allow the corresponding users group to:

Attribute	Description
Change image	Modify the image settings
View image	this is the most important setting which allows the users group to use the image at all. Without this the image won't be displayed at users image section

In the current version of wintermute.ai server non-administrator users must be created inside the server and corresponding passwords need to be defined in the server too. Future versions will allow connecting wintermute.ai server to corporate directory services and importing users and passwords from there.

6.5.1 User Creation

Administrators can create new users or edit and adapt permissions for existing users.To manage users, click on the "Security" menu.

The following attributes can be defined in a user record when creating a new user or editing an existing user's details:

6.5.2  User Permissions

After a user is created, different permissions in the wintermute.ai server can be granted. Fine tuning these permissions allows administrators to create boundaries inside of which users can execute their cloud tasks. To change user permissions, click on the "Actions" menu for the corresponding user and then select "Edit security". The following permissions can be granted to a user by selecting the corresponding check box:

Scenarios allow wintermute.ai server users to automatize their cloud tasks. At the moment User Defined Scripts (UDS) is the first supported automation technology. These are bash-ba sed scripts which can be managed directly inside the wintermute.ai server , can use special wintermute.ai server variables and can be applied to cloud instances to customize them. Admin and regular users can create their own private scripts or share them with other users of the current wintermute.ai server .

6.6.1  User Defined Scripts

To manage UDS scripts, select "Scenarios" from the main wintermute.ai server menu. A list of existing UDS scripts will be displayed. On a fresh installed wintermute.ai server only an example generic.sh script will be available.

6.6.2  UDS Creation

Admin and regular users can create a new UDS by clicking the "Create UDS" button or edit existing scripts by selecting "Edit" from the script "Actions" menu.

The following parameters can be set by defining a new UDS script:

Security for exisitng UDS can be change by clicking on edit existing scripts by selecting "Edit Security" from the script "Actions" menu. It allows the corresponding users group to:

6.6.3  UDS Handling

There are some specifics which should be considered when using UDS scripts.

6.6.3.1  Serial Execution

All UDS scripts which were selected for running during cloud instance creation will be by default executed in serial according to their sequence order.

6.6.3.2 Parallel Execution

If there is a need to parallelize UDS script execution, then the tasks inside the script should be sent to the background with some of following shell scripting techniques:

Using WMAIC Shell Library Function

by importing the WMAIC Shell Library in to the UDS name space the wmaic_run_in_background() shell function become available, by placing this function call in the begin of a UDS script will ensure that this script will be executed in the background and handle all TTY and connection losts

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

#!/bin/bash

if [ -f /etc/profile.d/wmaic.sh ];then
    source /etc/profile.d/wmaic.sh
else
    echo "WMAIC shell library not found, aborting!" 1>&2
    exit 1
fi

wmaic_run_in_background

# continue here with your own UDS code

Manual

if you are, for some reasons, not able to use the WMAIC Shell Library consider to integrate following techniques in to your UDS script:

• Using a nohup shell call to start executing parts of the shell script in the background and ignoring the HUP signal.
• Using a trap "" HUP command inside the script to catch the HUP signal and continue execution.
• Using double background execution to detach from the TTY and avoid becoming a zombie process in case of SSH break-up: (my_cmd.sh &) & .

• Use a at job to start executing parts of the shell script in the background and detaching from the current TTY.

Please consult the BASH/SHELL programming documentation for more details on how to implement these techniques.

Beside the simple start and securing of a number of instances wintermute.ai server provides also the ability to automate cloud tasks. This chapter provides a few examples, ranging from simple UDS scripts to more complex swarm and docker based scenarios.

6.7.1 UDS Scripting

UDS Scripts were introduced in the previous chapter. These scripts are created and managed on the wintermute.ai server, but are executed on the managed cloud instances. wintermute.ai server copies these scripts to the corresponding clients, prepares the environment and execute them in the order defined by the user.

Every cloud instance managed by the wintermute.ai server contains a WMAIC shell library which can be used in the UDS scripts. This shell library contains:

• WMAIC Shell Variables which can be directly used in the UDS scripts and are described in the Appendix 8.1
• WMAIC Shell Functions, which provide the UDS scripts additional functionality like templates and are described in the Appendix 8.2

WMAIC Shell Variables are automatically set by the wintermute.ai server.

To be able to use the WMAIC shell library inside of a UDS script, it should be loaded in the very first lines of the script:

1
2
3
4
5
6

#!/bin/bash

if [ -f /etc/profile.d/wmaic.sh ];then
    source /etc/profile.d/wmaic.sh
fi
...

6.7.1.1 Example: direct use of WMAIC Variables

In this simple example should demonstrate the direct use of WMAIC Variables which are listed and described in the Appendix 8.1.

Lets assume we would like to create SSL-Certificate for an application which, the common name of this certificate should always point to the current cloud instance host name. For that we are going to use the $WMAIC_PUBLIC_HOSTNAME variable.

shell eXit() function is provided by the WMAIC shell library, it prints out an message to the STDERR and aborts the script execution with a non-zero exit value.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

#!/bin/bash

if [ -f /etc/profile.d/wmaic.sh ];then
    source /etc/profile.d/wmaic.sh
else
    echo "WMAIC shell library not found, aborting!" 1>&2
    exit 1
fi

openssl req -x509 -newkey rsa:2048 -keyout /tmp/wmaic_key.pem -out /tmp/wmaic_cert.pem -days 3000 -nodes -subj "/CN=${WMAIC_PUBLIC_HOSTNAME}" || eXit "SSL cert creation failed"
... 

6.7.1.2 Example: simple PaaS

During the instance startup tags can be applied to a managed cloud instance. The following example script applies system changes according to the defined tags and provides ready to use services on the corresponding instances.

Three different implementation approaches are shown as an example:

• lines 13-22 : Direct system configuration by using simple shell tools and WMAIC shell library functions like wmaic_parse_template which can be used to convert a configuration file template with WMAI* variables inside into a customized configuration file. Shell can be used for smaller system changes.
• lines 25-31 : Using a system configuration management system like puppet, chef, ansible or others. This can be used if such a tool with ready to use system configuration modules is already in place. This can be used for the most complex situations.
• lines 33-46 : Using docker to start a ready to use Wordpress-Engine, which consists of two linked containers. The first one contains a MySQL-DB the second one Wordpress.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

#!/bin/bash

if [ -f /etc/profile.d/wmaic.sh ];then
    source /etc/profile.d/wmaic.sh
else
    echo "WMAIC shell library not found, aborting!" 1>&2
    exit 1
fi

for tag in ${WMAIC_INSTANCE_TAGS[@]};do 
    case $tag in
        dbserver)
            # as we are on a RedHat-Family system with 'yum' package manager
            # and are installing MariaDB (community developed branch of MySQL)
            yum -y install mariadb-server
            
            # generating a configuration file based on our own template which was
            # copied to the instance in to the encrypted data partition
            wmaic_parse_template -i /mnt/data1/templates/mariadb-server.tmpl -o /etc/my.cnf.d/mariadb-server.cnf

            # making sure the mariadb service is enabled and running with our new config file
            chkconfig mariadb on; service mariadb stop ; service mariadb start
            ;;
        webserver)
            # here we are using puppet configuration management tool to setup a full webserver
            yum -y install puppet

            # we have copied a special created puppet cloud environment directory and a pre-compiled 
            # puppet catalog file for a webserver role. 
            # This can be used by a puppet stand-alone client (no puppet server access required)
            puppet apply --environment cloud1 --catalog /mnt/data1/puppet/webserver.json
            ;;
        wordpress)
            # here we are using docker container engine to deploy a wordpress installation
            yum -y install docker

            # make sure docker is running and enabled
            chkconfig docker on ; service docker start

            # launching MySQL docker container and using the WMAIC_SECRET variable as the root password
            docker run --name wmais-mysql -e MYSQL_ROOT_PASSWORD=${WMAIC_SECRET} -d mysql

            # starting wordpress container and linking it to the MySQL-Container
            docker run --name wmais-wordpress --link wmais-mysql:mysql -p 80:80 -d wordpress

            echo "Wordpress is ready for use, please connect to: http://${WMAIC_PUBLIC_HOSTNAME}"
            ;;
        # add here action parts for other tags
    esac
done

exit 0

6.7.2 Swarm

A swarm is a group of instances. All members of this group are aware of each other. wintermute.ai server achieves this by keeping up to date WMAIC_SWAR_* variables on each swarm instance. These variables (described in the Appendix 8.1) provide information whether the current instance is a member of a swarm or not, the IP addresses of other swarm members, tags assigned to the other swarm members and so on. Functions described in the Appendix 8.2 can be used to interact with swarm instances, e.g. check ports, if services are already up, wait for instances, create firewall cloak around the swarm and allow public access only to defined machines in the swarm, etc.

6.7.2.1 Swarm Creation

Swarm creation is realized the same way as the creation of a single instance. The only difference is that the field "Swarm Name" is filled in with the corresponding swarm name. Also usually the amount of swarm instances is more than one, but the swarm could be also started with a single instance and extended later with additional nodes (see next chapter). Please note following specifics when creating the swarm

Creation Dialog

6.7.2.2 Modify running Swarm

As soon as a swarm is created, instances can be added or removed from the swarm. As soon as an instance is terminated, it will leave swarm automatically and all remaining swarm members will be notified with a swarm-leave action message. To add a new swarm member, just open "Create Instance" dialog, configure all fields as required for start of a single or multiple hosts and select the swarm to join from the "Swarm Name" selection menu.

6.7.3 Swarm Example Use Case

As an swarm example use case we will create a Docker based Wordpress setup composed of three instances with three different roles:

• Nginx: SSL termination and reverse-proxy. Optional later use as a load balancer and static-content delivery.
• Wordpress: wordpress application
• MySQL: MySQL DB server for wordpress

as basis OS we use Ubuntu 16.04 LTS

Additional Requirements are:

• all UDS scripts should be generic as much as possible, so that they can be reused in other scenarios
• all swarm instances should be protected with firewall rules
• all communication inside the swarm should be allowed by the firewall
• only the nginx instance should be accessible from the 'outside' on the HTTP(S) ports 80 and 443

6.7.3.1 NGINX Reverse Proxy

For the nginx setup two components are required:

• configuration file which can be dynamically adapted to new or leaving wordpress hosts
• UDS script which installs, configures and adapts nginx configuration during the system lifetime

Configuration Template

In this example a configuration template is used to generate a suitable nginx configuration file during the installations and update it on related swarm changes.

upstream upservers {
        ${WMAIU_UPSTREAM_SERVERS}
}

server {
    listen 80;
    return 301 https://$host$request_uri;
}

server {
        listen       443 ssl;
        server_name  ${WMAIC_PUBLIC_HOSTNAME};

        ssl on;
        ssl_certificate_key     /etc/nginx/ssl/wmaic_key.pem;
        ssl_certificate         /etc/nginx/ssl/wmaic_cert.pem;

        location / {
                proxy_set_header        Host $host;
                proxy_set_header        X-Real-IP $remote_addr;
                proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header        X-Forwarded-Proto $scheme;
                proxy_pass              http://upservers;
                proxy_read_timeout      90;
        }
}

The above template looks like a regular nginx configuration file but with some special WMAIS template tags:

• line 2 : usually this line should contains references to upstream servers to which nginx will forward the connections. Syntax should be like " server ip-address:port; ". Instead of this a variable name is placed here, this variable will be generated dynamically, on the corresponding swarm instances and will contain the actual "server ..." statements for the current swarm. wmaic_parse_template function will replace this variable in the template with the actual values and generate the final nginx configuration file.
• line 12 : to allow nginx to route properly the request the current configuration file will respond to the current cloud instance hostname, this variable will be defined automatically by the wintermute.ai server .

NGINX UDS

following nginx UDS script should be generic and not only limited to this wordpress scenario. As such it requires three parameters which defines:

• -t : tag of the swarm instance to which requests will be forwarded to. Since the cloud instance hostnames and ips are highly dynamical, the use of tags is more appropriate for identification.
• -p : port to which requests will be forwarded to
• -c : location of the configuration template file (see above) on the file system of an instance
  
  

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81

#!/bin/bash
#
# Ubuntu based nginx reverse proxy UDS, accepts three call arguments:
#
#   -p <port>   defines the port to which connection should be forwarded
#   -t <tag>    defines host tags to which the forward should be send
#   -c <conf>   defines where the nginx reverse proxy config template is 
#               located on the WMAIC client
#
# e.g.: nginx_reverse_proxy.sh -c /mnt/crypted01/data1/nginx.tmpl -p 3000 -t nodejs
#

if [ -f /etc/profile.d/wmaic.sh ];then
    source /etc/profile.d/wmaic.sh
else
    echo "WMAIC shell library not found, aborting!" 1>&2
    exit 1
fi

wmaic_run_in_background

export DEBIAN_FRONTEND=noninteractive 
export DEBIAN_PRIORITY=critical

FORWARD_TO_PORT="3000"
FORWARD_TO_NODES="appserver"
NGINX_CONF_TEMPLATE="/mnt/crypted01/templates/nginx.tmpl"

while getopts "p:t:c:" option; do
    case "${option}" in
        p) FORWARD_TO_PORT="${OPTARG}"
           ;;
        t) FORWARD_TO_NODES="${OPTARG}"
           ;;
        c) NGINX_CONF_TEMPLATE="${OPTARG}"
           ;;
        *) eXit "unknown call argument: ${option}"
           ;;
    esac
done
shift $((OPTIND-1))

if ! wmaic_exclusive_run;then
    eXit "another copy of this script seems to run"
fi 

if wmaic_is_tag_assigned "nginx";then
    if wmaic_is_swarm_member;then
        wmaic_wait_for_instances
        wmaic_refresh_env
        cloak_swarm_arg="-s"
        export WMAIU_UPSTREAM_SERVERS=$(wmaic_get_instances -i -t "${FORWARD_TO_NODES}" -s '\n' -p 'server\s' -x ":${FORWARD_TO_PORT};")
    else
        cloak_swarm_arg=""
        export WMAIU_UPSTREAM_SERVERS="server 127.0.0.1:${FORWARD_TO_PORT};"
    fi

    wmaic_cloak_instance ${cloak_swarm_arg}
    wmaic_open_port -p 80,443

    if ! wmaic_is_swarm_update_action;then
        apt-get -q -y install docker.io openssl || eXit "failed to install 'docker openssl'"
        systemctl enable docker.service
        if ! systemctl --no-pager status docker.service >/dev/null;then
            systemctl --no-pager start docker.service || eXit "failed to start the docker daemon"
        fi
        mkdir -p /usr/local/wmaic/nginx/{ssl,conf.d} || eXit "failed to create /usr/local/wmaic/nginx/{ssl,conf.d} directories"
        openssl req -x509 -newkey rsa:2048 -keyout /usr/local/wmaic/nginx/ssl/wmaic_key.pem -out /usr/local/wmaic/nginx/ssl/wmaic_cert.pem -days 3000 -nodes -subj "/CN=${WMAIC_PUBLIC_HOSTNAME}" || eXit "SSL cert creation failed"
        wmaic_parse_template -i ${NGINX_CONF_TEMPLATE} -o /usr/local/wmaic/nginx/conf.d/wmaic.conf || eXit "failed to create nginx.conf file from template ${NGINX_CONF_TEMPLATE}"
        docker run --name wmaic_nginx --restart=always -d -p 80:80 -p 443:443 -v /usr/local/wmaic/nginx/conf.d/wmaic.conf:/etc/nginx/conf.d/wmaic.conf:ro -v /usr/local/wmaic/nginx/ssl:/etc/nginx/ssl -v /var/run/docker.sock:/tmp/docker.sock:ro nginx || eXit "failed to start nginx docker container"
        wmaic_register_for_swarm_updates
    else
        docker exec wmaic_nginx service nginx status || eXit "nginx docker container is not running properly"
        wmaic_parse_template -f -i ${NGINX_CONF_TEMPLATE} -o /usr/local/wmaic/nginx/conf.d/wmaic.conf || eXit "failed to re-create nginx.conf file from template"
        docker exec wmaic_nginx service nginx reload || eXit "nginx configuration reload failed"
    fi
fi

wmaic_release_exclusive_run

exit 0

• lines 13-18 : load the WMAIC shell library with special variables and function
• line 20 : the wmaic_run_in_background tag makes sure that the script execution (which is default set to serial ) is executed in background. This is very important when using UDS in swarms, as there can be a lot of asynchronous communication and to prevent dead-lock situations, UDS scripts which interact with other swarm components should run in background.
• lines 22-27 : define a few default variables
• lines 29-41 : extract script call parameters
• lines 43-45 : in a swarm environment a lot of changes can happen simultaneously (e.g. 3 instances die). wmaic_exclusive_run () function can used to ensure that the script is executed only once. In this example UDS script exits if another instance is running, other use of wmaic_exclusive_run() can keep the script in a wait-loop until the other script instances exit and then proceed with the execution of the current script.
• line 47 : here is another example how to check with wmaic_is_tag_assigned() function if a tag is assigned to the current cloud instance. So the script will continue only if the tag "nginx" is assigned to the swarm instance
• line 48 : as we are creating a generic nginx reverse proxy UDS which can be applied to a swarm instance or to a stand-alone machine, here we can define settings for each of this scenarios
• line 49 : as different swarm instances may need some time to become available, we use wmaic_wait_for_instances to pause the script until the details (IP and tags) of all other swarm instances are available
• line 50 : after the swarm details are complete, we refresh our WMAIC shell library with all special variables.
• lines 51 and 54 : in the line 58 we activate a iptables based firewall protection around our swarm. As this is a generic nginx UDS script we allow all swarm members to communicate with each other without borders (-s option). If this is a stand-alone nginx installation, we just protect the stand-alone instance with firewall without any unrestricted communication from the outside.
• lines 52 and 55 : here we dynamically generate the WMAIU_UPSTREAM_SERVERS variable which we use in the above nginx configuration file. In this example we have two different cases how this variable can be defined:
  • line 55 : if this is a stand-alone instance we assume that the upstream server is located on the same machine and we only need to define the FORWARD_TO_PORT, which is provided as a call argument of the script.
  • line 52 : on a swarm instances we use wmaic_get_instance() function to get the information about other swarm instances with the requested tag. In our example the tag will be "wordpress". As we would need one upstream server definition per line, we define "\n" (newline) as separator (-s). Also to get nginx compatible upstream server definition we define "server " as prefix (-p) and postfix ":port;" (-x) on each upstream server record.
• line 58 : wmaic_cloak_instance() is used to build a iptables based firewall protection around the instance. In a swarm the communication with other swarm instances should be allowed without limitations (-s parameter)
• line 59 : as it can be seen from the swarm architecture picture above, the nginx-instance should be the only one which is accessible from the outside internet, with the wmaic_open_port() function the HTTP and HTTPS ports are opened for public access.
• line 61 : in the line 61 we register this script to receive swarm updates (leave or join of new nodes), this means our script is running in two modes, the first time during the initial setup of the instance and all following times as soon as a swarm update happens. With the wmaic_is_swarm_update_action() function we check if this is the initial run or swarm update. Lines 61-71 will be executed only during the initial install and lines 73-75 only during swarm updates. All other lines of the script will be executed on both runs.
• lines 62-66 : install docker and other required software and enable the docker services in the system
• line 67 : as all data in a docker container gets lost after container termination, we create data directories on the host system which we will mount-bind in to the docker container and which will ensure that the data is safe even after re-building of a docker container.
• line 68 : creation of a self-signed SSL certificate for the current hostname of the instance.
• line 69 : parsing the nginx template configuration file and replacing the WMAIU_UPSTREAM_SERVERS variable and others with the values which have generated before in the script.
• line 70 : starting the docker with the official nginx container, ensure HTTP and HTTPS ports are forwarded to the container and mount our directories with persistent data.
• line 71 : with the wmaic_register_for_swarm_updates() function this script is registered to be re-executed on all future swarm updates (new instances join or leave). For more details see description of the line 60.
• lines 73-75 : these lines are only executed on swarm update actions:
  • line 73 : ensure that the docker service is running fine
  • line 74 : re-generate the nginx configuration file to make sure any new upstream nodes are properly placed in the config file, or nodes which have left are removed from the config.
  • line 75 : force nginx to reload the new generated configuration file
• line 79 : releases the exclusive script execution, means other nginx UDS script instances can now continue the execution.

6.7.3.2 Wordpress

Wordpress UDS accepts three optional parameters:

• -d : defines the IP and the PORT of the MySQL DB server which should be used
• -u : defines an optional DB user which should be used, if not defined then user 'root' will be used
• -p : password for the optional DB user
  
  

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85

#!/bin/bash
#
# Ubuntu based wordpress UDS, accepts following _optional_ args:
#
#   -d <dbms>      define the IP:PORT of the MySQL server
#   -u <user>      define optional DB user, if not 'root' will be used
#   -p <pass>      define password for the optional DB user, if not 
#                  for SWARM instances the SWARM-ID is used, for standalone
#                  WMAIC clients the local-secret variable is used
#

if [ -f /etc/profile.d/wmaic.sh ];then
    source /etc/profile.d/wmaic.sh
else
    echo "WMAIC shell library not found, aborting!" 1>&2
    exit 1
fi

wmaic_run_in_background

export DEBIAN_FRONTEND=noninteractive 
export DEBIAN_PRIORITY=critical

MYSQL_DB=""
MYSQL_USER=""
MYSQL_PASS=""


while getopts "d:u:p:" option; do
    case "${option}" in
        d) MYSQL_DB="-e WORDPRESS_DB_HOST=${OPTARG}"
           ;;
        u) MYSQL_USER="-e WORDPRESS_DB_USER=${OPTARG}"
           ;;
        p) MYSQL_PASS="-e WORDPRESS_DB_PASSWORD=${OPTARG}"
           ;;
        *) eXit "unknown call argument: ${option}"
           ;;
    esac
done
shift $((OPTIND-1))

if ! wmaic_exclusive_run;then
    eXit "another copy of this script seems to run"
fi 

if wmaic_is_tag_assigned "wordpress";then
    if wmaic_is_swarm_member;then
        wmaic_wait_for_instances
        wmaic_refresh_env
        cloak_swarm_arg="-s"
        if [ -z "${MYSQL_DB}" ];then
            DB_HOSTS=( $(wmaic_get_instances -t "mysql" -x ':3306') )
            MYSQL_DB="-e WORDPRESS_DB_HOST=${DB_HOSTS[0]}"
        fi
        if [ -z "${MYSQL_PASS}" ];then
            MYSQL_PASS="-e WORDPRESS_DB_PASSWORD=${WMAIC_SWARM_ID}"
        fi
    else
        cloak_swarm_arg=""
        MYSQL_DB="-e WORDPRESS_DB_HOST=127.0.0.1:3306"
        if [ -z "${MYSQL_PASS}" ];then
            MYSQL_PASS="-e WORDPRESS_DB_PASSWORD=${WMAIC_SECRET}"
        fi
    fi

    wmaic_cloak_instance ${cloak_swarm_arg}
    wmaic_wait_for_services -w 5 -x 60 -X 10 -t mysql -p 3306     

    if ! wmaic_is_swarm_update_action;then
        apt-get -q -y install docker.io || eXit "failed to install 'docker'"
        systemctl enable docker.service
        if ! systemctl --no-pager status docker.service >/dev/null;then
            systemctl --no-pager start docker.service || eXit "failed to start the docker daemon"
        fi
        docker run --name wmaic_wordpress --restart=always -d -p 3000:80 ${MYSQL_DB} ${MYSQL_USER} ${MYSQL_PASS} -v /var/run/docker.sock:/tmp/docker.sock:ro wordpress || eXit "failed to start wordpress docker container"
        wmaic_register_for_swarm_updates
    else
        :
    fi
fi

wmaic_release_exclusive_run

exit 0

the UDS script is created with a structure which is similar to the nginx UDS, so only different lines are described here:

• line 53 : if a mysql db host wasn't provided via call parameters then get (all) hosts with the tag "mysql" and append the port number to the hostname (':3306)
• line 54 : prepare the MySQL Environment variable as it is required by the wordpress docker image, in this example we are using only one MySQL server, so if in line 53 more than one MySQL tagged hosts were found, we take only the first ([0])
• line 57 : if the instance is part of a swarm then we take the WMAIC_SWARM_ID variable (which is a random uuid string) as common known secret and use it as MySQL password
• line 61 : on a stand-alone instance we assume that the MySQL DB is running on the same machine
• line 63 : on a stand-alone instance we use the WMAIC_SECRET variable for the MySQL password. This variable is only visible inside of the system
• line 68 : as wordpress initialization can only be done after a database service is available, we wait here for TCP port 3306 to become available on all machines with the tag 'mysql'. We try to connect to this port 60 times, with 10 seconds pause between the attempts and 5 seconds connection timeout. As soon as database becomes available, the script execution continues. In case it is not available after 60 attempts, the scripts exits with an error.

6.7.3.3 MySQL-DB

MySQL UDS accepts four optional parameters:

• -d : defines an optional database inside of MySQL to be created during the initial setup
• -u : defines an optional DB user to be created
• -p : password for the optional DB user
• -c : define a customized MySQL DB config template file, this template file, if provided, will be parsed with wmaic_parse_template() function
  
  

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85

#!/bin/bash
#
# Ubuntu based MySQL server UDS, accepts following _optional_ args:
#
#   -d <db>        define a DB which should be created inside of MySQL
#   -u <user>      define optional user
#   -p <pass>      define password for the optional user, if not 
#                  for SWARM instances the SWARM-ID is used, for standalone
#                  WMAIC clients the local-secret variable is used
#   -c <config>    define a customized MySQL config file on the WMAIC client
#                  use full path to the file
#

if [ -f /etc/profile.d/wmaic.sh ];then
    source /etc/profile.d/wmaic.sh
else
    echo "WMAIC shell library not found, aborting!" 1>&2
    exit 1
fi

wmaic_run_in_background

export DEBIAN_FRONTEND=noninteractive 
export DEBIAN_PRIORITY=critical

MYSQL_DB=""
MYSQL_USER=""
MYSQL_PASS=""
MYSQL_CFG=""
MYSQL_MOUNT_CFG=""

while getopts "d:u:p:c:" option; do
    case "${option}" in
        d) MYSQL_DB="-e MYSQL_DATABASE=${OPTARG}"
           ;;
        u) MYSQL_USER="-e MYSQL_USER=${OPTARG}"
           ;;
        p) MYSQL_PASS="-e MYSQL_PASSWORD=${OPTARG}"
           ;;
        p) MYSQL_CFG="${OPTARG}"
           ;;
        *) eXit "unknown call argument: ${option}"
           ;;
    esac
done
shift $((OPTIND-1))

if ! wmaic_exclusive_run;then
    eXit "another copy of this script seems to run"
fi 

if wmaic_is_tag_assigned "mysql";then
    if wmaic_is_swarm_member;then
        wmaic_wait_for_instances
        wmaic_refresh_env
        cloak_swarm_arg="-s"
        MYSQL_ROOT_PASS="-e MYSQL_ROOT_PASSWORD=${WMAIC_SWARM_ID}"
    else
        cloak_swarm_arg=""
        MYSQL_ROOT_PASS="-e MYSQL_ROOT_PASSWORD=${WMAIC_SECRET}"
    fi

    wmaic_cloak_instance ${cloak_swarm_arg}

    if ! wmaic_is_swarm_update_action;then
        apt-get -q -y install docker.io || eXit "failed to install 'docker'"
        systemctl enable docker.service
        if ! systemctl --no-pager status docker.service >/dev/null;then
            systemctl --no-pager start docker.service || eXit "failed to start the docker daemon"
        fi
        mkdir -p /usr/local/wmaic/mysql/{db,conf.d} || eXit "failed to create /usr/local/wmaic/mysql/{db,conf.d} directies"
        if [ -n "${MYSQL_CFG}" ];then
            wmaic_parse_template -i ${MYSQL_CFG} -o /usr/local/wmaic/mysql/conf.d/wmaic.cnf || eXit "failed to create mysql.conf file from template ${MYSQL_CFG}"
            MYSQL_MOUNT_CFG="-v /usr/local/wmaic/mysql/conf.d/wmaic.cnf:/etc/mysql/conf.d/wmaic.cnf:ro"
        fi
        docker run --name wmaic_mysql --restart=always -d -p 3306:3306 ${MYSQL_MOUNT_CFG} ${MYSQL_DB} ${MYSQL_USER} ${MYSQL_PASS} ${MYSQL_ROOT_PASS} -v /usr/local/wmaic/mysql/db:/var/lib/mysql -v /var/run/docker.sock:/tmp/docker.sock:ro mysql || eXit "failed to start mysql docker container"
        wmaic_register_for_swarm_updates
    else
        :
    fi
fi

wmaic_release_exclusive_run

exit 0

the UDS script is created with a structure similar to the nginx and wordpress UDS above, no new statement types are used.

6.7.3.3 Starting the Swarm

Following steps need to be executed to start the above nginx/wordpress/mysql swarm:

1. Create the above UDS scripts inside of wintermute.ai server
2. Place the nginx.tmpl file in to a data directory on your local wintermute.ai server from where it is allowed to be uploaded to the clients.
3. Ensure to have a Ubuntu 16.04 image available inside of wintermute.ai server
4. Start a instance (swarm) with following settings:

Instance Tab:

fill out the Instance Tab similar to the screen shot below, vli-ubuntu-1604 is our Ubuntu 16.04 image and fullOpen is our security group which allows unlimited access to the instances on the provider level.

Storage Tab:

The directory /tmp/data1 is configured in our example as the one allowed for data uploads, in this directory we have placed the nginx.tmpl file.

Setup Tab

select the UDS script to be executed on the instances, the sequence order doesn't matter as they will be only executed on the instances with matching tags. Provide also parameters to the nginx.tmpl file according to your definitions above.

Summary Tab

review the setup and start the swarm

Connect to the Swarm

as soon as swarm is up and running, connect to the public IP address of the machine with the tag nginx:

after successful HTTPS connection a Wordpress setup screen should be shown:

and after clicking to 2-3 wordpress setup dialogs a full configured wordpress instance is ready for use:

The wintermute.ai server has a browser-based, responsive user interface and can be accessed from any computer or mobile devices. Ask your IT department for the access URL of your local wintermute.ai server , your login user name and password.

On a freshly installed wintermute.ai server the access URL, login and temporary password are displayed in the last lines of the installation output and in the installation log file: /var/log/wmais/installation. log

Open the wintermute.ai server access URL and log in:

After the first login the user should immediately change their initial password by clicking on the user icon in the upper right corner and selecting "Profile: loginname " from the menu. In the "Edit User" dialog the user can update their settings and change their password.

Log out by clicking on the person icon in the upper right corner and selecting "Logout" from the menu.

7.2.1  Overview

After a successful login the first page is the dashboard which is the main page for a user's interaction with the wintermute.ai server .

On  the dashboard a summary of all important information about a user's activities is displayed.

In the main pane the following items are displayed:

• Instances Til e - displays how many instances (virtual machines) belonging to the current user are running successfully in the cloud.
• EBS Drives Tile - displays how many additional (and optional) encrypted EBS drives are attached to the running instances.
• EBS Size Tile - displays how big (GB) these additional EBS drives are.
• Tasks Tile - displays how many tasks are currently running. Tasks are explained in more detail later in this chapter.
• Logs - displays tasks' activities in the form of a log journal.
• Created Instances Graph - displays how many instances were deployed by the current user in the past.
• Active Instances Graph - displays how many instances belonging to the current user are active and currently running.
• EBS Volumes Graph - displays how many EBS drives were used by the user's cloud instances in the past. Please note that every cloud instance hast at least one EBS volume which contains its operating system. If selected the instance can have additional (encrypted) EBS volumes attached, which are managed by wintermute.ai server .

The user can click on tiles and log entries to retrieve additional information.

7.2.2  Tasks

A single user can have multiple tasks running on the wintermute.ai server , e.g.: starting 5 cloud instances; downloading data from an instance; patching 3 instances; destroying obsolete instances; etc. To ensure that all these tasks can be executed in parallel, a task manager is provided by the wintermute.ai server . To get an overview of past and current running tasks, the user can click on the tasks icon in the upper right corner of the wintermute.ai server interface :

In the screen shot above there are no running tasks, 4 of 5 tasks were completed successfully and one "create_instance" task failed.

There could be different reasons for an instance startup failure, e.g.:

• the user selected a non-existing software package to be installed; or
• the user tried to upload/ download data from/ to a directory without the correct permissions.

Clicking on the task log details of the execution or the failure will display a reason for the failure. This should be the first step to analyze the failure.

7.3.1  List Instances

By clicking on the "Instances" tile on the dashboard or the "Instances" wintermute.ai server menu in the navigation pane, a detailed view of all active instances is displayed.

Every active instance is displayed as a row record in a table and has the following attributes:

Column Name	Description	Extra Info
ID	Instance name which was selected during the instance start and instance ID which was assigned by the cloud provider.
Provider	Name of the cloud provider where this instance is running.
Type	Cloud type, displayed as an icon. The current version of wintermute.ai server supports only Amazon Web Services. Future versions will support OpenStack and other cloud technologies.
OS	Operating System, displayed as an icon. Supported are: Debian, Ubuntu, Fedora, CentOS, RHEL and SLES.
Storage	Indicates whether additional (encrypted) EBS storage is attached and how big it is.
Status	Current status of the instance. Status can be: creating, initializing, running, rebooting, mounting, shutting-down, terminating, stopped or failed.
Created/Updated	Time stamp of instance creation and last modification.
Controls	Action tasks which can be applied to the instance.

Based on the current status the corresponding row changes its color.

• green - ready to go; instance is running properly
• yellow - not ready: initializing, rebooting, terminating or failed
• white - creating,stopped or shutting-down phase
  
  

Fields which are marked with a in the "Extra Info" column in the above table display additional useful information when clicked. The following screen shot shows additional information about the "ID" field in a pop-up:

7.3.2  Start New Instaces

To start a new instance, from the "Instances" page, click on the "Launch New Instance" button.

The following table explains the different options the user can define in the "Create Instance" dialog:

instance#:tag

firstInstance#-lastInstance#:tag 

name[:install|:uninstall][:package|:group]

service:

7.3.3  Create Cloud Templates

Cloud Templates can be used to save all settings which were made during a cloud session setup and re-load them any time later. To do this, click the "Save as template" button in the "Summary" tab of the "Create Instance" dialog.

The current cloud session setup will be saved under the "Name" identifier in the templates list.

7.3.4  Use Cloud Templates

A previous saved cloud session setup can be loaded for reuse by clicking on the "From Template" selection in the "Create Instance" dialog and selecting the required session by name from the list.

After the session is loaded from the template, the user can adjust all parameters before starting the new session.

7.3.5  Delete Cloud Templates

A previously saved cloud session template can be deleted by loading it as described in "Use Cloud Templates" chapter above and by clicking the "Delete template" button in the bottom right corner of the "Create Instance" dialog.

7.3.6  Instance Controls

Active instances can be controlled by clicking the "Actions" menu for the corresponding machine in the "Instances" list. The action tasks which can be applied are:

7.3.7  Batch Controls

Similar to instance controls, some actions can by applied to a group of instances by selecting the check box in the first column of the "Instances" page and then selecting an action from the "Batch Actions" menu. The actions result is the same as for single instances.

Please note that the wintermute.ai server administrator defines in the configuration a file system location from which data upload is allowed. So, a user can provide either a full file system path to this directory (or subdirectories of it) or a relative path from inside this directory. Providing any other file system path will result in a error. This ensures that only data which is explicitly allowed to be used in the cloud can be uploaded.

After clicking on the "Storage" wintermute.ai server menu a detailed view of all active used additional (encrypted) storage EBS volumes is displayed.

Every active instance is displayed as a row record in a table and has the following attributes:

Column Name	Description	Extra Info
ID	Volume id which was assigned by the cloud provider to this storage volume.
Type	Displays storage type as an icon.
Size	Displays storage size in GB.
Attached	Displays instance id which was assigned by the cloud provider to the instance to which the storage is attached.
Device	Shows the device name inside the Linux OS to which this storage volume is connected.
Files System	Shows the device mount point inside the Linux file system.
Encryption	A lock icon indicates that the corresponding storage volume is encrypted.
Created/Updated	Time stamp of storage volume creation and last modification.
Controls	Action tasks which can be applied to the storage volume.

Fields which are marked with a in the "Extra Info" column in the above table are able to display additional useful information when clicked.

7.4.1 Storage Control

Active storage volumes can be controlled by clicking on the "Actions" menu of the corresponding storage.

The following dynamic bourne shell variables are available in the /etc/profile.d/wmaic.sh file and can be sourced from within UDS scripts:

Following shell (bash) functions become available inside of UDS scripts as soon as the WMAIC Shell Library is loaded, this should be done in the first lines of a UDS script:

#!/bin/bash

if [ -f /etc/profile.d/wmaic.sh ];then
    source /etc/profile.d/wmaic.sh
else
    echo "WMAIC shell library not found, aborting!" 1>&2
    exit 1
fi
#
# now all UDS function are available
#
# continue with the regular UDS shell code
#
...

8.2.1 eXit()

USAGE: eXit [reason text]

this function prints out the optional "reason text" to STDERR and 
exit then with a non-zero value, which means ERROR in shell

EXAMPLE: 
  mkdir /var/lib/test || eXit "failed to create directory: /var/lib/test"

8.2.2 wmaic_parse_template()

USAGE: wmaic_parse_template -i  -o  [-f] [-b] 
       [-B ]

this function parses the input-file and replaces all ${VARIABLES} which
starts with the string "WMAI" by their content in the current script.

This could be the default "WMAIC_" variables which are available by default
on all WMAIC clients, or this could be also user generated variables
which starts with "WMAIU_" prefix. 

Multi-line variable content is also supported. Variable syntax should
always include curly braces, e.g.: ${WMAIU_VARNAME} instead of $WMAIU_VARNAME

ARGUMENTS:

-i     define the input template file which should be parsed

-o    define where the final, parsed file should be saved

-f                 'force' flag for overwriting already existing output-files

-b                 backup the output-file before overwriting it

-B        define the backup postfix which is appended to the 
                   backup file name. 
                   DEFAULT: '_BKP'

EXAMPLE: 
  wmaic_parse_template -i /mnt/data1/nginx.tmpl -o /etc/nginx/conf.d/my.conf

8.2.3 wmaic_exclusive_run()

USAGE: wmaic_exclusive_run [-x] [-w]

this function ensures that no other instance of the script is running
on the current host. If another instance is running this functions
returns a non-zero value.

Other behavior can be achieved by using following optional arguments.

ARGUMENTS:

-x   exit with error if exclusive run can't be achieved

-w   wait until exclusive run can be achieved
     re-check LOCK every 5 seconds

EXAMPLE:
  if ! wmaic_exclusive_run;then
      eXit "another copy of this script seems to run"
  fi

8.2.4 wmaic_release_exclusive_run()

USAGE: wmaic_release_exclusive_run

this function releases the exclusive run which was enabled by the
wmaic_exclusive_run() function. This should be done in the last lines
of a UDS script.

EXAMPLE:
  wmaic_release_exclusive_run

8.2.5 wmaic_get_instances()

USAGE: wmaic_get_instances [-i] [-t ] [-s ] [-p ] 
                           [-x   report only instances which have this tags assigned
                    multiple tags are to be provided as comma separated list

-s       defines separator which will be used for join
                    instances together in to a string. DEFAULT: space

                    Extra format characters ca be used in separator 
                    definition:
                    '/s' - is replaced by a space in the output record
                    '/n' - is replaced by a newline in the output record
                    '/t' - is replaced by a tab in the output record

-p          define optional prefix which should be placed in the 
                    begin of every hostname before join in to a string

-x         define optional postfix which should be appended to 
                    the end of every hostname before join in to a string

EXAMPLE1: get all hosts with zookeeper tag in a string which is ready to be
          used in a client call.

          wmaic_get_instances -i -t zookeeper -s ',' -p 'zk://' -x ':2181/api'
        
          could RETURN a string like: 
            zk://10.0.0.1:2181/api,zk://10.0.0.2:2181/api,zk://10.0.0.3:2181/api

EXAMPLE2: get all nodejs hosts and produce a multi-line string output which 
          can be used in the nginx reverse proxy configuration as upstream 
          servers which are listening on port 3000.

          wmaic_get_instances -i -t nodejs -s '\n' -p 'server\s' -x ":3000;"
        
          could RETURN a multi-line string like: 
            server 10.0.0.20:3000;
            server 10.0.0.21:3000;
            server 10.0.0.22:3000;

8.2.6 wmaic_wait_for_instances()

USAGE: wmaic_wait_for_instances

this function is used in a swarm environment and it pauses a UDS script 
execution until the current execution host receives full information 
about other swarm members. 

As in bigger swarms it can take some time until all instances are up
it's a good idea to use this function to make sure all required
information is available.

Please refresh the environment variables in the current UDS script by 
using 'wmaic_refresh_env()' to ensure that all instances information is
updated inside of the UDS script.

EXAMPLE: if wmaic_is_swarm_member;then
             wmaic_wait_for_instances
             wmaic_refresh_env
             # continue here with your UDS code
             ...
         fi

8.2.7 wmaic_wait_for_services()

USAGE: wmaic_wait_for_services <-t ] | -n  
                                <-p 
                                [-i] [-w ] [-x ] 
                                [-X ] [-v]

this function checks if a service is listening on the defined TCP ports 
on a group of swarm members. Swarm members can be addressed either by
tags and/or IPs or by '-a' flag which will check all swarm members.

This function requires at least one port definition and one of the following
arguments: '-t' or '-n' or '-a', for details see ARGUMENTS below.

This function is very useful in a swarm environment with service dependencies
e.g. application setup should only be done _after_ a DB service is up.

Please note: if the check on at least one machine (after all retries) fails 
the whole UDS script execution will be terminated.

ARGUMENTS:

-i                  include localhost in service check. DEFAULT: skip it

-w         time out in seconds on connect attempt

-x         define the number of connection retries. DEFAULT: forever

-X         pause in seconds before connection retry. DEFAULT: 10

-t   test services only on instances with the defined tags

-n   test services only on the defined IPs

-a                  test services on all swarm hosts. Please note that this
                    wouldn't include the localhost, for this purpose '-i'
                    flag must also be provided

-p           define one, multiple or TCP port-ranges to be checked
                    for services. Syntax example:

                    single port      : -p 443
                    multiple ports   : -p 80,443
                    port ranges      : -p 20-25
                    mixed definition : -p 80,443,6666-6669

-v                  turn on the verbose mode

EXAMPLE: check for database service on TCP port 3306 on all swarm members 
         with the tag 'dbserver'. Do 60 retries with 10 second pause between
         check attempts and 5 seconds timeout on TCP connection.

         wmaic_wait_for_services -i -t dbserver -p 3306 -w 5 -x 60 -X 10

8.2.8 wmaic_cloak_instance()

USAGE: wmaic_cloak_instance [-s] [-i 

this function builds a native firewall around the current cloud instance.
By default the firewall rules allow all OUTPUT connections and deny all
INPUT connection to the host. The only port which is open for INPUT by
default is the SSH port '22', this is required for WMAIS->WMAIC 
communication.

For opening additional ports for public use the WMAIC function
'wmaic_open_port()' should be used, for details see documentation below.

For opening access for other hosts (e.g. other swarm members) see arguments
below.

ARGUMENTS:

-s                  if a member of a swarm allow full INPUT access to all 
                    ports from all other swarm members

-i   allow full INPUT access to all ports from this IPs, 
                    IP ranges can be defined in CIDR notation

-t   allow full INPUT access to all ports from instances 
                    with the defined tags

EXAMPLE: setup the firewall cloak around the current host but allow all
         income communication on all ports from the swarm members

         wmaic_cloak_instance -s

8.2.9 wmaic_open_port()

USAGE: wmaic_open_port [-u] -p  [-a ]

this function opens one or multiple UDP or TCP ports for INPUT access
to all (public) or limited number of hosts (define by their IPs).

Please note that this function should only be called after the 
WMAIC firewall cloak was initialized with the 'wmaic_cloak_instance()'
function, otherwise the whole UDS script execution may fail.

ARGUMENTS:

-u          open UDP type port. DEFAULT: TCP

-p   define port(s) to be opened in the WMAIC firewall cloak
            for external (public) INPUT access

-a     limit access to the defined ports only for this IPS instead
            of a general public INPUT access

EXAMPLE: on a web-server cloud instance enable WMAIC firewall cloak
         which allows limitless communication inside of a swarm and 
         opens the HTTP (80) and HTTPS (443) ports for public access

         wmaic_cloak_instance -s
         wmaic_open_port -p 80,443

8.2.10 wmaic_register_for_swarm_updates()

USAGE: wmaic_register_for_swarm_updates

this function register the current UDS script to be called on 
swarm update events. A swarm event can happen if new instances
join the swarm or instances are terminated and leave the swarm.

The WMAIC function 'wmaic_is_swarm_update_action()' returns a 
true (shell return value 0) if this is a swarm update, otherwise
it is the initial call of the UDS script during the instance
setup phase.

EXAMPLE:
  wmaic_register_for_swarm_updates

8.2.11 wmaic_is_tag_assigned()

USAGE: wmaic_is_tag_assigned 

this function returns true (shell return value 0) if the "tag" which
is provided as argument is assigned to the current cloud instance.
Otherwise false (shell return value non-zero) is returned.

EXAMPLE: install and configure nginx web server on the current cloud 
         instance only if the "nginx" tag is assigned

  if wmaic_is_tag_assigned 'nginx';then
      # insert here the shell code for nginx setup and configuration
      ...
  fi

8.2.12 wmaic_is_swarm_member()

USAGE: wmaic_is_swarm_member

this function returns true (shell return value 0) if the current
cloud instance is a member of a swarm, false (shell return value 
non-zero) otherwise.

EXAMPLE:
  if wmaic_is_swarm_member;then
      # enter here all shell code for swarm behavior 
      ...
  else
      # enter here shell code for a stand-alone cloud instance
      ...
  fi

8.2.13 wmaic_is_swarm_update_action()

USAGE: wmaic_is_swarm_update_action

all UDS scripts are first executed during the cloud instance 
initialization phase, at this first execution by using the WMAIC
function 'wmaic_register_for_swarm_updates' UDS can also be registered
for the execution during the swarm update action.

This function returns true (shell return value 0) if the current
UDS script execution is done during a swarm update action stage. 
Otherwise false (shell return value non-zero) is returned.

EXAMPLE: install and configure docker based nginx web server on a 
         Ubuntu based system during the initial run of the UDS script,
         register the script for swarm updates and on swarm updates 
         re-configure and reload the new new nginx config file.

  if ! wmaic_is_swarm_update_action;then
      # this is the initial run, install and configure everything from scratch
      apt-get -q -y install docker.io || eXit "failed to install 'docker'"
      systemctl enable docker.service
      mkdir -p /usr/local/wmaic/nginx/conf.d || eXit "failed to create /usr/local/wmaic/nginx/conf.d directory"
      wmaic_parse_template -i /mnt/data1/nginx.tmpl -o /usr/local/wmaic/nginx/conf.d/wmaic.conf || eXit "failed to create nginx.conf file"
      docker run --name wmaic_nginx --restart=always -d -p 80:80 -p 443:443 -v /usr/local/wmaic/nginx/conf.d/wmaic.conf:/etc/nginx/conf.d/wmaic.conf:ro -v /var/run/docker.sock:/tmp/docker.sock:ro nginx || eXit "failed to start nginx docker container"
      wmaic_register_for_swarm_updates
  else
      # this is a update run, re-create the config file and reload the service only
      wmaic_parse_template -f -i /mnt/data1/nginx.tmpl -o /usr/local/wmaic/nginx/conf.d/wmaic.conf || eXit "failed to re-create nginx.conf file"
      docker exec wmaic_nginx service nginx reload || eXit "nginx configuration reload failed"
  fi

8.2.14 wmaic_get_current_stage()

USAGE: wmaic_get_current_stage

this function returns the ${WMAIC_STAGE} variable if defined, this variable
contains the name of the cloud instance setup stage during which the UDS 
script was called.

Currently following setup stages are supported:

* post-data-transfer  - this is the stage after the cloud instance is completely
                        setup, full patched, rebooted (if required) and all data
                        is transfered to the instance
* swarm-update-action - this is a stage which is entered on every swarm update,
                        update could be if a new instance is joining a swarm or
                        is leaving

8.2.15 wmaic_run_in_background()

USAGE: wmaic_run_in_background

by default UDS scripts are executed in serial order as defined by their sequence
numbers during the instance start dialog. By placing the 'wmaic_run_in_background'
shell function call in a UDS script the script will be executed in background,
parallel to other UDS scripts, if also defined. WMAIS SSH connection can break-up
the UDS script will be still executed in the background.

8.2.16 wmaic_refresh_env()

USAGE: wmaic_refresh_env

this function refresh the shell environment in the current UDS script, this is 
useful after the script has paused by waiting for other swarm nodes or services.

EXAMPLE: wait for getting all information from swarm nodes and refresh the
         environment after it happen

  wmaic_wait_for_instances
  wmaic_refresh_env

8.3.1  settings.py

The following fine tunings should be applied to the /opt/wmai/ VERSION /wmais/settings.py file. Replace VERSION with the current version of the wintermute.ai server .

8.3.1.1  Dashboard History

Define in days how long the activity list should be kept on the dashboard.

WMAIS_DASHBOARD_HISTORY_DAYS = 90

8.3.1.2  Data Transfer Retries

Define with these two settings how often on a failed or aborted data transfer a retry should be made and how long the pause (in seconds) between these retries should be. The number of WMAIS_DATA_TRANSFER_RETRIES can be increased if there is a weak and unstable Internet connection or a big amount of data is being transferred.

WMAIS_DATA_TRANSFER_RETRIES = 4
WMAIS_DATA_TRANSFER_PAUSE_BETWEEN_RETRY = 10

8.3.1.3  Instance Connection Attempts

wintermute.ai server uses native cloud provider API calls to create an instance and SSH connection to configure instances from within. Usually it takes some time after an instance is created and booted up before it becomes available f or SSH access. Using the following two parameters a user can fine tune how many SSH connection attempts are made before setup will be marked as failed and can define the length of the pause between the connection attempts:

WMAIS_CONNECTION_ATTEMPTS = 50
WMAIS_CONNECTION_RETRY_PAUSE = 10

8.3.1.4   Keep Running Failed Instances

If an instance setup fails the wintermute.ai server has two options as to how to continue.

1. Cancel instance setup and stop the instance execution so the instance will be suspended to cloud storage and won't consume anymore CPU time. The user can boot-up this instance later and examine it if required, or terminate it any time.
2. Cancel instance setup but keep the instance running in a failed state which will consume CPU time but allow the user (or admin) quickly to examine the failure.

The following variable can be set to "False" for option #1 or "True" for option #2:

WMAIS_KEEP_INSTANCE_RUNNING_ON_ERROR = True

8.3.2  Cloud Sessions Directory

Every active cloud session saves information and log files in a unique session id directory under /var/lib/wmai/sessions .Once an hour session directories of terminated cloud sessions are cleaned up and removed. However, if it's useful to keep these session tracks for auditing purposes, the session cleaning script /etc/cron.hourly/wmaisCleaner can be removed.

Following important Configuration Files and Directories are used on a wintermute.ai server system and should be included in to corporate backup:

/opt

/opt/wmai

/opt/wmai/VERSION

/opt/wmai/current

/opt/wmai/current/wmais/settings.py

/opt/wmai/ssl

/opt/extlib

/opt/extlib/VERSION

/opt/extlib/current

/var/log/wmais

/var/lib/wmai

/lib/systemd/system/wmais.service

/etc/init.d/wmais

/etc/default/wmais

/etc/cron.hourly/wmaisCleaner

If web access in a company is realized through a proxy server the following steps must be executed before starting the wintermute.ai server installation. All commands are executed in the BASH shell. They may be rewritten for a preferrerd alternative shell. ">" represents the shell prompt and shouldn't be executed !

Defining a proxy with the fqdn host name proxy1.company.com running on port 3128 without username and password:

> export http_proxy="http://proxy1.company.com:3128"
> export https_proxy="http://proxy1.company.com:3128"
>

Defining a proxy with the fqdn host name proxy1.company.com running on port 3128 with user username and no password:

> export http_proxy="http://user@proxy1.company.com:3128"
> export https_proxy="http://user@proxy1.company.com:3128"
>

Defining proxy with the fqdn host name proxy1.company.com running on port 3128 , with user username and mypass password:

> export http_proxy="http://user:mypass@proxy1.company.com:3128"
> export https_proxy="http://user:mypass@proxy1.company.com:3128"
>

wintermute.ai server is built on top of Python 2.7.x, manual installation of it are required on systems which are by default shipped with older or newer versions of Python.

As an example for manual Python 2.7.x installation this chapter will cover install on a RedHat Enterprise Linux 6.x (RHEL6) system.

8.6.1 Install Required System Tools

> sudo yum -y groupinstall "Development tools"
...
> sudo yum -y install wget curl zlib-devel bzip2-devel openssl-devel ncurses-devel postgresql-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel git shadow-utils gmp gmp-devel rsync
... 

8.6.2 Download Python Sources

> cd /tmp 
tmp> wget https://www.python.org/ftp/python/2.7.14/Python-2.7.14.tgz 
...
tmp> tar xzf Python-2.7.14.tgz
... 
tmp> cd Python-2.7.14
Python-2.7.14>

8.6.3 Compile Python

Python-2.7.14> ./configure --prefix=/usr/local --with-ensurepip --enable-unicode=ucs4 --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"

8.6.4 Install Python

Python-2.7.14> sudo make altinstall

8.6.5 Check Python 2.7.x Installation

Python-2.7.14> /usr/local/bin/python2.7 --version
...
Python-2.7.14> /usr/local/bin/python2.7-config --prefix 
...
Python-2.7.14> /usr/local/bin/pip2.7 --version
...

8.6.6 Install virtualenv

Python-2.7.14> sudo /usr/local/bin/pip2.7 install virtualenv 
...
Python-2.7.14> /usr/local/bin/virtualenv --version
...

 sudo -H ./setup.sh -d rhel6 -V /usr/local/bin/virtualenv

sudo -H ./setup.sh -d centos6 -V /usr/local/bin/virtualenv