documentation - wintermute.ai server 1.0.0rc2

1. About

The purpose of this document is to provide information necessary to install and run the wintermute.ai server software in the current release.

wintermute.ai server is a Cloud Access Security Broker (CASB) which acts as an IaaS encryption gateway, secure IaaS gateway and cloud broker. It ensures regulatory compliance with company guidelines for IaaS cloud access for corporate users.


1.1  Structure of this document

1.1.1  Chapters:


2. Early Access Release

This Early Access Release means that users will have free, time-limited access to the beta version s of the wintermute.ai server software.

Beta software may contain defects. The primary purpose of this type of software is to obtain feedback on performance and identification of defects. Users are advised to safeguard important data, use caution and not to rely in any way on the correct functioning or performance of the software and accompanying materials.

More information about the conditions for using of this software can be found in the EARLY ACCESS AND EVALUATION END USER LICENSE AGREEMENT in the Appendix of this document.

This RC2 release contains only a limited number of the features which will be included in the final release.


3. Features

RC2 release provides the following unique features:

3.1 Supported Clouds

wintermute.ai server supports following cloud technologies and providers:

  • Amazon Web Services (AWS)
  • OpenStack based IaaS providers

3.2  In-House

wintermute.ai server runs inside a company on the company's own hardware.

  • Cloud providers do not have any access to it.
  • All security-related data, such as encryption or SSH keys, are stored only on the company's hardware.
  • Users have full control over the server and all the data which it handles.

3.3  Storage Encryption

wintermute.ai server automatically encrypts all used cloud block storage (EBS) transparently in the background.

  • All encryption is done transparently in the background.
  • Automatic encryption key management also runs in the background.

3.4  Encryption Keys

wintermute.ai server automatically manages all storage and SSH encryption keys.

  • Keys are stored and managed automatically.
  • Cloud providers have no access to the encryption keys.
  • Storage encryption keys are never stored in the cloud.
  • Only public SSH keys are injected into the cloud instance; private keys are always saved inside the wintermute.ai server .
  • Key export may be prevented to ensure there can be no bypassing of the wintermute.ai server for cloud and data access.

  • As all keys are saved inside the company, if a particular user is not available, recovery can be done by another administrator.

3.5  Easy Access

3.6  Compliant

wintermute.ai server ensures that IaaS cloud computing complies with company rules, leaving a company's cloud users free to get on with tasks within specific boundaries.

  • Cloud providers and locations, which certain users are allowed to use, can be defined.
  • Server enables specification of allowed instance images and instance types.
  • Administrators are allowed to define, restrict and control which data is used in the cloud.

3.7  Hardening

wintermute.ai server hardens and automatically secures a company's cloud infrastructure in line with tailored corporate guidelines.

3.9  Cloud Automation

3.10  Web SSH Client

3.11 Customization


4. Quickstart

This Quickstart Reference Guide provides step-by-step instructions on how to install a working wintermute.ai server without diving too deep into the details. As example Amazon Web Services are used as IaaS service provider.

4.1  Set up wintermute.ai server host machine

  • Ensure the machine meets the Hardware Requirements , Operating System Requirements and the General Requirements in the Installation Guide chapter of this document.
  • Ensure host's packaging system can install additional software (also from the Internet) on demand.
  • Ensure sudo rights are available on the host machine.
  • Ensure Amazon Web Services (AWS) Access Key ID and Secret Access Key are available.

4.2  Install wintermute.ai server software

  • Install wintermute.ai server software according to the Software Installation instructions in the Installation Guide chapter of this document.

4.3  Login in to the wintermute.ai serve r

  • Extract from the installation log file the login URL, login name and initial login password as described in First Login in the Admin Guide chapter of this document.
  • Log in to the wintermute.ai server and change the default password as described in First L ogin in the Admin Guide chapter of this document.

4.4  Install FREE trial license

  • Create a license request according to First Login in the Admin Guide chapter of this document.
  • Send this request to license@wintermute.ai ; a FREE trial license will be sent back within 24 hours.
  • Install the license as described in First Login in the Admin Guide chapter of this document.

4.5  Configure Cloud Provider

  • Configure and enable an AWS account on the wintermute.ai server as described in Cloud Provider in the Admin Guide chapter of this document.

4.6  Configure Instance Image

  • Select a Debian 8.6 image in the most suitable region.
Region AMI ID Instance Type Import Image String
ap-northeast-1 (Tokyo) ami-1f4a9a7e m3.medium image_id:ami-1f4a9a7e
ap-northeast-2 (Seoul) ami-a85481c6 m3.medium image_id:ami-a85481c6
ap-south-1 (Mumbai) ami-6aed9805 m3.medium image_id:ami-6aed9805
ap-southeast-1 (Singapore) ami-83cb6ee0 m3.medium image_id:ami-83cb6ee0
ap-southeast-2 (Sydney) ami-74300017 m3.medium image_id:ami-74300017
eu-central-1 (Frankfurt) ami-30e01d5f m3.medium image_id:ami-30e01d5f
eu-west-1 (Ireland) ami-ff0e758c m3.medium image_id:ami-ff0e758c
sa-east-1 (Sao Paulo) ami-bbc350d7 m3.medium image_id:ami-bbc350d7
us-east-1 (N. Virginia) ami-9d6c128a m3.medium image_id:ami-9d6c128a
us-west-1 (N. California) ami-84cc83e4 m3.medium image_id:ami-84cc83e4
us-west-2 (Oregon) ami-2a34e94a m3.medium image_id:ami-2a34e94a
  • Import the selected image by using the Import Image String and following the instructions in Instance Images in the Admin Guide chapter of this document.
  • Edit the imported Image as described in Instance Images in the Admin Guide chapter of this document:
    • Set type of image to "Debian".
    • Set image to "active".

4.7  Start Test Instance

Create a new instance as described in Instances in the User Guide chapter of this document and using the following settings (region and image name may vary so adapt them as required):

4.7.1  m3.medium instance type with Debian 8.6 image


4.7.2  No additional storage for this test run

4.7.3  Activate only updates and reboot



Activating and applying update and reboot on this Debian 8.x,  t2.micro machine takes about 10 minutes. Without active update and reboot the machine startup will be around 1 minute.

For a short startup test this might be OK but for security reasons updates should always be applied to a machine before using it in production. (warning)

4.7.4  Launch the instance


4.7.5  Connect to the machine

When the machine status is 'running', open an ssh connection to it as follows:




4.7.6  Terminate machine after tests

4.8  Next Steps

After this quick setup and test, the user should finalize the proper configuration of the wintermute.ai server as described in the Admin Guide chapter of this document.


5. Installation Guide

5.1 Requirements

5.1.1  General Requirements

The wintermute.ai server contains sensitive information like encryption keys and cloud credentials. Therefore the following safety measures should be considered when setting it up:

  • If no external access is required, the wintermute.ai server should be installed inside the company behind a firewall.
  • If external access (e.g. by external workers) is required, the wintermute.ai server should be placed inside a DMZ and external access to it should be strictly regulated.
  • If data transfer to and from the cloud will be performed, only the required data storage segments (e.g. Network File System shares) should be mounted on the wintermute.ai server .

  • Console/SSH login to the wintermute.ai server should be allowed only for admin users.
  • As the SSH and storage encryption keys are generated on demand and saved only inside the wintermute.ai server , the wintermute.ai server and it main directories (see Appendix ) should be included in the corporate backup. If the wintermute.ai server fails (e.g. hard disk error) there will be no means beside backup to recover the generated encryption and SSH keys.

5.1.2  Hardware Requirements

Virtual or real hardware with the following minimum requirements can be used:

Item Value Comment
RAM 2 GB
CPU 2 cores
Disk Space 10 GB default installation path: /opt
Network Internet Access

These are minimum requirements which are enough for testing wintermute.ai server with a few users. If a user base is large, more RAM, CPUs and faster network access should be added !

5.1.3  Operating System Requirements

wintermute.ai server runs on Linux and requires Python 2.7.X to be available on the system. The following distributions and version are supported out-of-the box:

Vendor Version
Debian 8.x, 9.x
Ubuntu 14.04, 16.04, 16.10, 17.04, 17.10
RedHat Enterprise Linux (RHEL) 6.x (star) , 7.x
CentOS 6.x (star) , 7.x
Fedora 24, 25, 26

(star) For some older versions of these distributions, Python 2.7 can be manually installed. Instructions can be found in Appendix: Manual installation: Python 2.7.x .

sudo rights are required on the corresponding machine for successful wintermute.ai server installation !


5.1.4  Required Software Packages

The wintermute.ai server installation script setup.sh requires that the underlying software management system is properly configured and allows additional installation of required software packages, e.g. on a RedHat-based system, a " yum -y install python-virtualenv " call should either report that this package is already installed, or install it and all the dependent packages on the fly.

OS Package Manager Software Packages to be installed
Ubuntu apt-get python, python-virtualenv, augeas-tools, gettext, curl, python-dev, git, build-essential, g++, libgmp10, libgmp10-dev, openssl, rsync, libssl-dev
Debian apt-get python, python-virtualenv, augeas-tools, gettext, curl, python-dev, git, build-essential, g++, libgmp10, libgmp10-dev, openssl, rsync, libssl-dev
Fedora yum python, python-virtualenv, augeas, gettext, curl, python-devel, git, shadow-utils, gcc, gcc-c++, mpir, mpir-devel, openssl, rsync, openssl-devel
CentOS yum python, python-virtualenv, augeas, gettext, curl, python-devel, git, shadow-utils, gcc, gcc-c++, gmp, gmp-devel, openssl, rsync, openssl-devel
RHEL yum python, python-virtualenv, augeas, gettext, curl, python-devel, git, shadow-utils, gcc, gcc-c++, gmp, gmp-devel, openssl, rsync, openssl-devel
SLES zypper python, python-pip, augeas, gettext-tools, curl, python-devel, git, gcc, gcc-c++, libgmp10, gmp-devel, openssl, rsync, libopenssl-devel


The system where the wintermute.ai server should be installed must meet all the requirements described in the previous sections ( Hardware Requirements , Operating System Requirements , Required Software Packages ) before the installation process can be started.


5.2 Software Installation

The wintermute.ai server host machine must have a proper DNS name configured that is accessible from within the company. Use hostname --long in the command shell to test the settings are correct.

If a proxy service is used to connect to the internet in the company, proxy settings must be defined, as described in Appendix: Proxy Configuration, in the shell in which the installation starts.

  1. Extract the downloaded wintermute.ai server software package to a temporary location.

    > tar -C /tmp -xvzf wmais-1.0.0rc2-debian8-x86_64.tgz
    ...
    >
  2. Change to the newly created product directory, which has the same name as the software package but without the .tgz extension.

    > cd /tmp/wmais-1.0.0rc2-debian8-x86_64
    > 

  3. Start the installation process by calling the setup.sh script.
    (lightbulb) If installing on a system with manual Python 2.7.x installation (e.g. RHEL6, CentOS6) use two additional call parameters when executing setup.sh . For more details see Appendix: Manual installation: Python 2.7.x

    > sudo -H ./setup.sh
    ...



  4. If the installation finishes successfully , output similar to that in the screen shot below will be displayed. If not, the installation log file in /var/log/wmais/install.log should be analyzed ( (lightbulb) use less -r to display the color codes in the log file properly).

  5. Confirmation that the wintermute.ai server components are running correctly can be done by executing the following shell command:

    > service wmais status
    ...
    >

6. Admin Guide

6.1 Post-Install Configuration

Following a successful wintermute.ai server installation some post-install configuration changes can be made. This should be done before starting any production work.

All changes should be applied to the /opt/wmai/ VERSION /wmais/settings.py file (replace VERSION with the current version of the wintermute.ai server ). There are more variables in the settings.py file which can be fine tuned later; they are described in the Appendix: Fine Tuning section.

6.1.1  SSH access to instances

As most cloud instances will run with public Internet addresses they will be vulnerable to different kind of attacks. So, for security reasons it's a good idea to restrict access to the running instances and only open to the public those services which are necessary. Secure Shell (SSH) access, which is used by the wintermute.ai server for instance managing, should be open for this purpose. An own security group can be managed inside the cloud provider; the following settings may be changed in line with the type of internet access used:

WMAIS_SG_IP_ADDRESS = "static"

Variable WMAIS_SG_IP_ADDRESS can have the following settings:

  • static - means the wintermute.ai server will access the cloud instances from one static address or address pool from one IP range. The wintermute.ai server will use a web service, which is defined in the WMAIS_SG_AUTO_IP_DISCOVER variable, to discover its own static address.

  • 1 . 2.3.4 - a static IP address, from which the wintermute.ai server will access the cloud instances, can be manually defined.

  • dynamic - this keyword indicates that the wintermute.ai server will access instances from different IPs so the access to the SSH service won't be restricted by firewall rules and can be accessed from every IP address.

WMAIS_SG_IP_CIDR = "/32"

Where WMAIS_SG_IP_ADDRESS is set to a static or manual IP definition and there is a range of proxies or gateways from where wintermute.ai server access will originate, the range can be defined by modifying the CIDR setting.

WMAIS_SG_AUTO_IP_DISCOVER = "http://checkip.amazonaws.com/"

This variable defines the accessible web service which returns the IP address of the current request. It's used to auto-discover the IP address from which the wintermute.ai server will access the cloud instances.

6.1.2  Data Settings

This setting defines the file system location from where data can be transfered to and from the cloud.

WMAIS_DATA_BASE_PATH = "/home/%USER%/cloud"

The macro %USER% will be replaced with the login name of the current user.

With the above setting a user with the login name "bob123" could only synchronize data with the cloud from the directory /home/bob123/cloud . In order to avoid a breakout from the defined data base path, the wintermute.ai server will skip transfer of any symbolic links.

The data base path can be any path which is accessible in the wintermute.ai server file system so it could also point to a network mounted storage area, e.g.: /nfs/storage1/cloud/%USER%

6.1.3  Time Zone

Define the local time zone by adjusting the following variable:

WMAIS_DEFAULT_TIMEZONE = 'Europe/Berlin'


6.2 First Login

After a successful wintermute.ai server installation the login information and a temporary password which was randomly chosen during the installation is displayed in a similar format to that shown in the following screen shot:

If the output wasn't saved after the installation it can be found in the installation log file: /var/log/wmais/install.log

6.2.1  Logging in

In a browser, open the URL contained in the final installation report and log in as an admin user with the temporary password.


6.2.2 Change Admin Password

It's very important that right after the installation, at the first login, the admin password is changed. This can be done by clicking the person icon in the upper right corner of the page and then selecting "Profile: admin" from the menu.

In the "Edit User" dialog, the admin password and other details can be changed.

6.2.3  Request License

A freshly installed wintermute.ai server requires a license to become fullly functional. An Early Access user can get this license for FREE. Execute the following steps:

Click the person icon in the upper right corner of the page and then select "License" from the menu.


In the License dialog, click "Request New License" to get a License Request for the current wintermute.ai server machine.


The License Request (text between +request and -request marker) should be send as email to license@wintermute.ai


Within the next 24 hours a FREE evaluation license will be issued. It can be installed by clicking on the "Install License" button in the License dialog.

After the license is installed the wintermute.ai server will be fully functional.


6.3 Cloud Provider

Before cloud services can be used at least one cloud provider account must be configured.

6.3.1 Amazon Web Services

To configure Amazon Web Services account: select "Settings" from the wintermute.ai server menu and click "New Provider" -> "AWS" to start the configuration.



The following settings can be configured in the "AWS Settings" dialog:

Tab Configuration Item Description



Credentials

Name Provides a meaningful name of the current AWS access credentials. Multiple AWS credential can be registered with different names.
Description Provides a short description about the current cloud provider configuration.
Access Key Id This key is a unique identifier similar to a username. It can be requested from AWS IAM console.
Secret Access Key This is a secret key similar to a password. It can be requested from the AWS IAM console.
Regions Regions Defines in which regions wintermute.ai server users are allowed to use cloud resources. (star)
Instance Types Instance Types Specifies which instance types (or virtual machines) wintermute.ai server users are allowed to use. The bigger the amount of CPU cores and storage selected, the more expensive it is. Consult the https://aws.amazon.com/ec2/instance-types/ page for different instance types and their costs.
Summary n/a A short overview over the current settings is given in this tab. If everything is properly configured, click "Save" to activate the configuration.

(star) To use services in the China (Beijing) Region, you must have an account and credentials that are specific to the China (Beijing) Region. Accounts and credentials for other AWS regions won't work for the China (Beijing) Region. Likewise, accounts and credentials for the China (Beijing) Region won't work for other AWS regions. For information about the region, see http://www.amazonaws.cn/en/ .


Fill all the required login data as described in the above table and press "Synchronize configuration" button to verify your login details and to load dynamic Regions and Instance Types from the cloud vendor.


If the correct login credentials were provided the "Configurations synchronization" task should finish successful and three additional tabs should become visible. Click on the "Regions" tab and select the region to which access with the current provider account should be granted. Click on "Instance Types" tab and select the instance types which should be available in the current account.


After saving the new configuration in the "Summary" tab, it will appear in the "Settings" menu and can be changed at anytime by selecting "Action"→"Edit".



6.3.2 OpenStack based IaaS Providers

To configure an OpenStack based IaaS account: select "Settings" from the wintermute.ai server menu and click "New Provider" -> "OpenStack" to start the configuration.



The following settings can be configured in the "OpenStack Settings" dialog:

Tab Configuration Item Description



Credentials

Name Provides a meaningful name of the current OpenStack IaaS access credentials. Multiple OpenStack providers can be registered with different names.
Description Provides a short description about the current cloud provider configuration.
Auth URL OpenStack Auth URL which should be provided by the corresponding OpenStack provider. This URL points to the OpenStack Keystone Service and should be provided without any API-Version postfix in the URL. For an example see the screenshot below.
Project Project or tenant name. This ID will be created by the OpenStack provider and groups together all users of a customer.
Name Login Name of the OpenStack user.
Password Password for the OpenStack user login.
Instance Types Instance Types Specifies which instance types (or virtual machines) wintermute.ai server users are allowed to use. The bigger the amount of CPU cores and storage selected, the more expensive it is.
Summary n/a A short overview over the current settings is given in this tab. If everything is properly configured, click "Save" to activate the configuration.


If the correct login credentials were provided the "Configurations synchronization" task should finish successful and three additional tabs should become visible.  Click on "Instance Types" tab and select the instance types which should be available in the current account. After saving the new configuration in the "Summary" tab, it will appear in the "Settings" menu and can be changed at anytime by selecting "Action"→"Edit".



6.3.3 Provider Permissions

following permissions settings can be changed by clicking "Action"→"Edit security". And allows the corresponding users group to:

Access Setting

Allows

Create instances start new instances at this provider
Create instance from template start new instances based on saved templates
Create instance template create templates, which is a saved start up scenario of an instance
Create volume create and use block storage volumes at this provider
Import images import instance images from this provider
Import volumes import already existing
View provider configuration (warning) this is the most important setting which allows the users group to use the provider at all. Without this the provider won't be displayed at users menus.


6.4 Instance Images

Instance images are templates for the creation of virtual servers. Each image includes a template for the root volume which generally contains an operating system and additional applications. Users can select an image provided by the cloud provider or the user community, or create and upload their own. Images have unique id's in each region they are available. That means the same Debian 8.0 image can have different id's in different regions.

wintermute.ai server allows for the selection of available images from the cloud provider registry. Image creation and the uploading of images are currently not supported by wintermute.ai server . If a user would like to use their own instance image they would need to upload it to the provider registry using the tools supplied by the cloud provider. After doing this, the image will be available for use on the wintermute.ai server .

For security reasons it's highly recommended that a user uses only their own or official images from trustworthy sources, like Linux Distribution Providers.

6.4.1 AWS

The following table provides a short overview about cloud image lists provided by the major Linux Distributions for Amazon Web Services:

Distribution URL Comment
Debian https://wiki.debian.org/Cloud/AmazonEC2Image
Ubuntu https://cloud-images.ubuntu.com/locator/ec2/
Fedora https://getfedora.org/de/cloud/download/index.html
CentOS https://wiki.centos.org/Cloud/AWS Before using CentOS images for the first time, the user must accept the terms and subscribe to CentOS inside of AWS Marketplace . After this has been done, CentOS images can be used inside the wintermute.ai server .
RHEL https://aws.amazon.com/partners/redhat/

Before using RedHat Enterprise Linux images for the first time, the user must accept the terms and subscribe to RHEL inside of AWS Marketplace . After this has been done, RHEL images can be used inside the wintermute.ai server .

(warning) Please note, as this is a commercial product, beside the AWS infrastructure costs, the user would need to pay License fees for RHEL usage. Exact price information can be found on RedHat's page inside AWS Marketplace (see link).

SLES https://aws.amazon.com/de/partners/suse/

Before using SuSE Enterprise Server images for the first time, the user must accept the terms and subscribe to SLES inside of AWS Marketplace . After this has been done, SLES images can be used inside the wintermute.ai server .

(warning) Please note, as this is a commercial product, beside the AWS infrastructure costs, the user would need to pay License fees for SLES usage. Exact price information can be found on SuSE's page inside AWS Marketplace (see link).

6.4.2 OpenStack

OpenStack manages a list of official OpenStack images from the major Linux Distributions, this list can be found here: http://docs.openstack.org/image-guide/obtain-images.html

Please note, before importing an OpenStack image into wintermute.ai serve r it must be made available inside of the Image-Managing-Service (glance) of the corresponding OpenStack provider. The most convenient way for this is to use the OpenStack Horizon Web-Interface as described here: http://docs.openstack.org/user-guide/dashboard-manage-images.html


6.4.3  Supported Client Distributions

following official vendor images were tested as wintermute.ai clients and are known to be working fine with this release of wintermute.ai server :

Vendor Version
Debian 7.x, 8.x, 9.x
Ubuntu 14.x, 15.x, 16.x, 17.x
RedHat Enterprise Linux (RHEL) 6.x, 7.x
CentOS 6.x, 7.x
Fedora 21 - 26
SuSE Linux Enterprise Server (SLES) 11, 12

6.4.4  Import Image

To make images available for wintermute.ai server , click on the "Images" menu and select "Import Image" -> "AWS". The AWS cloud provider must be properly configured before this can be done.


wintermute.ai server will now import a full list from the cloud provider which can take a while. This list can be very long, and, without sufficient information, it may be difficult to ascertan whether the image is trustworthy or not.

It is therefore recommended that the id of the desired cloud image is identified and used to search for the image  in the register. The search can be performed by defining the id in the form image_id:XXXXX in the search field of the images dialog, where XXXXX is the id of the image. Click "Refresh" to search for and display the results.


After selecting the desired image from the import or search results list, click "Import Image" to import the image into the wintermute.ai server .


After the first import the image will be available in a deactivated state inside the wintermute.ai serve r. To be able to use it, it must be activated. To do this, select "Actions"->"Edit" for the corresponding image in the Image List dialog.




Image attributes can be changed using the following settings:

Attribute Description
Name Name of the corresponding image. Usally this name can be retained as it was imported from the provider's registry.
Type

Select the type of Operating System which this image contains.

(warning) Not defining this setting properly can lead to a non-functional image inside the wintermute.ai server !


Description Provide a short description of this image. This description will be displayed later in the selection menu and will be the only way to identify the image. e.g. "Ubuntu 15.10".
Enabled (warning) Only after enabling this check box the image will be activated inside the wintermute.ai server .
User This setting is only required if a user's images or non-standard images are used. Enter the user name of the image admin user where the SSH key is injected. This field can be left blank if official images of the supported Linux Distributions (Debian, Ubuntu, Fedora, CentOS, RHEL, SLES) are used or if the image uses "ec2-user" as the system admin user.

After saving the changes, the image will become available for use inside the wintermute.ai server .


Image Permissions can be changed by clicking "Edit security" menu and and allow the corresponding users group to:

Attribute Description

Change image

Modify the image settings
View image (warning) this is the most important setting which allows the users group to use the image at all. Without this the image won't be displayed at users image section



6.5 User Management

In the current version of wintermute.ai server non-administrator users must be created inside the server and corresponding passwords need to be defined in the server too. Future versions will allow connecting wintermute.ai server to corporate directory services and importing users and passwords from there.

For security reasons all data transfers are executed on the underlying Linux system with users' id's (non-root). This ensures that a user can only transfer data where they have access to it on the underlying Linux File System. It also prevents problems with "root-squash" exported Network File Systems. For this reason the user login in the wintermute.ai server must be exactly the same as on the underlying Linux system otherwise data transfer won't work.

6.5.1 User Creation

Administrators can create new users or edit and adapt permissions for existing users.To manage users, click on the "Security" menu.


The following attributes can be defined in a user record when creating a new user or editing an existing user's details:

Attribute Description
User Actual user login name. This should be exactly the same as the user's login name on the underlying system.
First Name User's first name.
Last Name User's last name.
Email User's email address
Time zone User's time zone
Enabled Activate this check box to allow the user to use the wintermute.ai server .
Is Administrator Gives a user administrator rights for the system. Be careful with this permission !
Password User's password in the wintermute.ai server . After a temporary password is defined by the administrator, a user is able to change it (recommended).

6.5.2  User Permissions

After a user is created, different permissions in the wintermute.ai server can be granted. Fine tuning these permissions allows administrators to create boundaries inside of which users can execute their cloud tasks. To change user permissions, click on the "Actions" menu for the corresponding user and then select "Edit security". The following permissions can be granted to a user by selecting the corresponding check box:

Permission Description Note
Create user User is allowed to create other user accounts It is recommended to grant this permission to full or part admin users only.
Change user Modify user accounts With this permission a user can modify properties and security of another user.
Create UDS User is able to create and share own UDS files.
Download SSH Keys User is able to export and download his own SSH keys

Users' ssh keys are automatically managed by the wintermute.ai serve r. A public key is automatically injected in a new cloud instances and allows a user a secure SSH-based login to the instance.

With this permission a user can, in theory, download the private SSH key, import it in to a SSH client program (e.g. PuTTY), and, by bypassing the wintermute.ai server , interact with cloud instances. Without this permission a user is forced to use the controlled cloud access via the wintermute.ai server .

It is recommended to grant this permission to full or part admin users only.

Control Tasks User is able to cancel running tasks

Users can always cancel their own tasks. Users with this permission can cancel all tasks.

View Logs User is able to view log entries View logs of the system.
Manage Licenses User is able to manage licenses It is recommended to grant this permission to full or part admin users only.
Get Server Logs User is able to view wintermute.ai server logs It is recommended to grant this permission to full or part admin users only.

6.6 Scenarios

Scenarios allow wintermute.ai server users to automatize their cloud tasks. At the moment User Defined Scripts (UDS) is the first supported automation technology. These are bash-ba sed scripts which can be managed directly inside the wintermute.ai server , can use special wintermute.ai server variables and can be applied to cloud instances to customize them. Admin and regular users can create their own private scripts or share them with other users of the current wintermute.ai server .

6.6.1  User Defined Scripts

To manage UDS scripts, select "Scenarios" from the main wintermute.ai server menu. A list of existing UDS scripts will be displayed. On a fresh installed wintermute.ai server only an example generic.sh script will be available.

6.6.2  UDS Creation

Admin and regular users can create a new UDS by clicking the "Create UDS" button or edit existing scripts by selecting "Edit" from the script "Actions" menu.


The following parameters can be set by defining a new UDS script:

Parameter Description Default Value
Name Shell script name. Use the same syntax as per any local scripts (e.g. myTestScript.sh). The script will be saved remotely on the instance machine with this name. It is recommended that if scripts will be called from other scripts, meaningful names should be chosen. n/a
Description This should be a short description of the purpose of this script that will help to identify it in the future. n/a
Type The type of this script. Currently only bash scripts are officially supported but in theory other scripts with suitable hash-bang (#!...) lines can be used. bash
Platform Defines on which platform this script can be executed. This could be a generic script which is platform independent or a script which can be run only on special platforms, like Debian Linux, but not on other Linux distributions. Linux generic
Stage During the setup of a cloud instance several setup stages are passed: software installation and updates, user management, data encryption and others. UDS script execution can be hooked in before, during or after a stage. However, in the current version of wintermute.ai server only the execution after the last setup stage post-data-transfer can be selected. That means that the UDS script will only be executed after the instance is completely configured, all settings are applied and all data transfers are complete. post-data-transfer
Access

Defines whether this script is private or can be shared with other people using the wintermute.ai server .

Publishing UDS script inside the wintermute.ai server is a great opportunity to share work and re-use scripts created by others. This will help members of a team not to have to re-invent the wheel over and over again.

private
Parameters Default script parameters may be defined here. However, when activating a UDS script for execution on an instance, a user would need to redefine them. The UDS script will be executed with these parameters on the cloud instance. The script author is responsible for ensuring that the corresponding script can handle call options and parameters. It is recommended bash's getopts function is used for this purpose.
Content

This setting contains the content of the UDS script itself. It should contain valid bash hash-bang ( #!/bin/bash ) as the first line and follow the rules of a proper bash script. wintermute.ai server creates a shell environment file in the location /etc/profile.d/wmaic.sh . After sourcing this file a lot of special variables (and functions in future versions) will be available inside the UDS script. By using them inside the UDS script the behavior based on the current instance environment can be adapted, e.g. the variable $WMAIC_PROCESSOR_COUNT to start the number of parallel computer processes for each of the available cpu cores is one . In this way the script can use all the CPU power available without any changes on 4,8,16,32 CPU systems.

A full list of available variables can be found in Appendix: UDS Variables .



Security for exisitng UDS can be change by clicking on edit existing scripts by selecting "Edit Security" from the script "Actions" menu. It allows the corresponding users group to:

Access Setting Allows
Change UDS change the current UDS, even if the aren't the UDS owners
View UDS use (execute) the current UDS. So by enabling this setting the UDS is shared and can be executed by user from the users group


6.6.3  UDS Handling

There are some specifics which should be considered when using UDS scripts.

6.6.3.1  Serial Execution

All UDS scripts which were selected for running during cloud instance creation will be by default executed in serial according to their sequence order.

UDS scripts, like all other instance customization tasks, are executed remotely via SSH by the wintermute.ai server . If a UDS script starts a long term (for hours or days) execution run, it is possible that the SSH connection will break up, and the UDS script receive a HUP signal and lose the current TTY. UDS script developers should consider this situation and handle it with one of the techniques described below.

6.6.3.2 Parallel Execution

If there is a need to parallelize UDS script execution, then the tasks inside the script should be sent to the background with some of following shell scripting techniques:

Using WMAIC Shell Library Function

by importing the WMAIC Shell Library in to the UDS name space the wmaic_run_in_background() shell function become available, by placing this function call in the begin of a UDS script will ensure that this script will be executed in the background and handle all TTY and connection losts

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
#!/bin/bash

if [ -f /etc/profile.d/wmaic.sh ];then
    source /etc/profile.d/wmaic.sh
else
    echo "WMAIC shell library not found, aborting!" 1>&2
    exit 1
fi

wmaic_run_in_background

# continue here with your own UDS code

Manual

if you are, for some reasons, not able to use the WMAIC Shell Library consider to integrate following techniques in to your UDS script:

  • Using a nohup shell call to start executing parts of the shell script in the background and ignoring the HUP signal.
  • Using a trap "" HUP command inside the script to catch the HUP signal and continue execution.
  • Using double background execution to detach from the TTY and avoid becoming a zombie process in case of SSH break-up: (my_cmd.sh &) & .
  • Use a at job to start executing parts of the shell script in the background and detaching from the current TTY.

Please consult the BASH/SHELL programming documentation for more details on how to implement these techniques.



6.7 Cloud Automation

Beside the simple start and securing of a number of instances wintermute.ai server provides also the ability to automate cloud tasks. This chapter provides a few examples, ranging from simple UDS scripts to more complex swarm and docker based scenarios.

6.7.1 UDS Scripting

UDS Scripts were introduced in the previous chapter. These scripts are created and managed on the wintermute.ai server, but are executed on the managed cloud instances. wintermute.ai server copies these scripts to the corresponding clients, prepares the environment and execute them in the order defined by the user.

Every cloud instance managed by the wintermute.ai server contains a WMAIC shell library which can be used in the UDS scripts. This shell library contains:

  • WMAIC Shell Variables which can be directly used in the UDS scripts and are described in the Appendix 8.1
  • WMAIC Shell Functions, which provide the UDS scripts additional functionality like templates and are described in the Appendix 8.2

WMAIC Shell Variables are automatically set by the wintermute.ai server.

To be able to use the WMAIC shell library inside of a UDS script, it should be loaded in the very first lines of the script:

1
2
3
4
5
6
#!/bin/bash

if [ -f /etc/profile.d/wmaic.sh ];then
    source /etc/profile.d/wmaic.sh
fi
...

6.7.1.1 Example: direct use of WMAIC Variables

In this simple example should demonstrate the direct use of WMAIC Variables which are listed and described in the Appendix 8.1.

Lets assume we would like to create SSL-Certificate for an application which, the common name of this certificate should always point to the current cloud instance host name. For that we are going to use the $WMAIC_PUBLIC_HOSTNAME variable.

(info) shell eXit() function is provided by the WMAIC shell library, it prints out an message to the STDERR and aborts the script execution with a non-zero exit value.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
#!/bin/bash

if [ -f /etc/profile.d/wmaic.sh ];then
    source /etc/profile.d/wmaic.sh
else
    echo "WMAIC shell library not found, aborting!" 1>&2
    exit 1
fi

openssl req -x509 -newkey rsa:2048 -keyout /tmp/wmaic_key.pem -out /tmp/wmaic_cert.pem -days 3000 -nodes -subj "/CN=${WMAIC_PUBLIC_HOSTNAME}" || eXit "SSL cert creation failed"
... 


6.7.1.2 Example: simple PaaS

During the instance startup tags can be applied to a managed cloud instance. The following example script applies system changes according to the defined tags and provides ready to use services on the corresponding instances.

Three different implementation approaches are shown as an example:

  • lines 13-22 : Direct system configuration by using simple shell tools and WMAIC shell library functions like wmaic_parse_template which can be used to convert a configuration file template with WMAI* variables inside into a customized configuration file. Shell can be used for smaller system changes.
  • lines 25-31 : Using a system configuration management system like puppet, chef, ansible or others. This can be used if such a tool with ready to use system configuration modules is already in place. This can be used for the most complex situations.
  • lines 33-46 : Using docker to start a ready to use Wordpress-Engine, which consists of two linked containers. The first one contains a MySQL-DB the second one Wordpress.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
#!/bin/bash

if [ -f /etc/profile.d/wmaic.sh ];then
    source /etc/profile.d/wmaic.sh
else
    echo "WMAIC shell library not found, aborting!" 1>&2
    exit 1
fi

for tag in ${WMAIC_INSTANCE_TAGS[@]};do 
    case $tag in
        dbserver)
            # as we are on a RedHat-Family system with 'yum' package manager
            # and are installing MariaDB (community developed branch of MySQL)
            yum -y install mariadb-server
            
            # generating a configuration file based on our own template which was
            # copied to the instance in to the encrypted data partition
            wmaic_parse_template -i /mnt/data1/templates/mariadb-server.tmpl -o /etc/my.cnf.d/mariadb-server.cnf

            # making sure the mariadb service is enabled and running with our new config file
            chkconfig mariadb on; service mariadb stop ; service mariadb start
            ;;
        webserver)
            # here we are using puppet configuration management tool to setup a full webserver
            yum -y install puppet

            # we have copied a special created puppet cloud environment directory and a pre-compiled 
            # puppet catalog file for a webserver role. 
            # This can be used by a puppet stand-alone client (no puppet server access required)
            puppet apply --environment cloud1 --catalog /mnt/data1/puppet/webserver.json
            ;;
        wordpress)
            # here we are using docker container engine to deploy a wordpress installation
            yum -y install docker

            # make sure docker is running and enabled
            chkconfig docker on ; service docker start

            # launching MySQL docker container and using the WMAIC_SECRET variable as the root password
            docker run --name wmais-mysql -e MYSQL_ROOT_PASSWORD=${WMAIC_SECRET} -d mysql

            # starting wordpress container and linking it to the MySQL-Container
            docker run --name wmais-wordpress --link wmais-mysql:mysql -p 80:80 -d wordpress

            echo "Wordpress is ready for use, please connect to: http://${WMAIC_PUBLIC_HOSTNAME}"
            ;;
        # add here action parts for other tags
    esac
done

exit 0

6.7.2 Swarm

A swarm is a group of instances. All members of this group are aware of each other. wintermute.ai server achieves this by keeping up to date WMAIC_SWAR_* variables on each swarm instance. These variables (described in the Appendix 8.1) provide information whether the current instance is a member of a swarm or not, the IP addresses of other swarm members, tags assigned to the other swarm members and so on. Functions described in the Appendix 8.2 can be used to interact with swarm instances, e.g. check ports, if services are already up, wait for instances, create firewall cloak around the swarm and allow public access only to defined machines in the swarm, etc.

6.7.2.1 Swarm Creation

Swarm creation is realized the same way as the creation of a single instance. The only difference is that the field "Swarm Name" is filled in with the corresponding swarm name. Also usually the amount of swarm instances is more than one, but the swarm could be also started with a single instance and extended later with additional nodes (see next chapter). Please note following specifics when creating the swarm


Creation Dialog

Tab Field Description Example





Instance

Name the same meaning as on single instance start
Description the same meaning as on single instance start
Tags can have the same meaning as on a single instance, but usually different machines in the swarm can have different roles, so the assigned tags to a machine should reflect this different roles. 1-3:keepalived, 1-3:haproxy, 4-6:appserver, 7-9:dbserver
Operating System the same meaning as on single instance start
Instance Type the same meaning as on single instance start
Security Group

can have the same meaning as on single instance, but usually most swarm instances need to communicate with each others, either usable Security Group is prepared on the provider site or it's recommended to fully open all communication of the swarm nodes and use the UDS function wmaic_cloak_instance () to create a native, Linux iptables firewall rules around the swarm instances which protect them to the outer world and allow the communication inside the swarm.

(warning) Please note, you must ensure that the wmaic_cloak_instance() was executed properly on all machines before apply a full open Security Group to the instances


Instances define how many instances should be started on the initial swarm creation
Swarm Name with the swarm name definition in this field a swarm will be created, even if it contain only one machine in the begin.
Storage all

the same meaning as on a single instance.

(info) Please note, that the storage definitions and uploads will be applied to all machines. If you would like to make uploads only to few machines or different uploads to different machines then you would need to create a swarm with a few machines of the same type and then add other groups of machines with different settings to this swarm. This is in details described in the next chapter.


Setup all

the same meaning as on a single instance.

(info) Please note, that the UDS script will be applied to all swarm member instances, the Examples in the Chapter 6.7.3 below will show how to make sure that even when activated UDS scripts will be only applied to the right instances. e.g. applying a db-install-UDS only to instances with the DB tag.


Summary all the same meaning as on single instance start




6.7.2.2 Modify running Swarm

As soon as a swarm is created, instances can be added or removed from the swarm. As soon as an instance is terminated, it will leave swarm automatically and all remaining swarm members will be notified with a swarm-leave action message. To add a new swarm member, just open "Create Instance" dialog, configure all fields as required for start of a single or multiple hosts and select the swarm to join from the "Swarm Name" selection menu.

As during the "Create Instance" dialog only instances of the same type can be created, adding new instances with different attributes to a existing swarm is the best way for the setup of a heterogeneous swarm.

e.g. adding a special instance with more RAM and a bigger HardDisk for the DB role.



6.7.3 Swarm Example Use Case

As an swarm example use case we will create a Docker based Wordpress setup composed of three instances with three different roles:

  • Nginx: SSL termination and reverse-proxy. Optional later use as a load balancer and static-content delivery.
  • Wordpress: wordpress application
  • MySQL: MySQL DB server for wordpress

as basis OS we use Ubuntu 16.04 LTS




Additional Requirements are:

  • all UDS scripts should be generic as much as possible, so that they can be reused in other scenarios
  • all swarm instances should be protected with firewall rules
  • all communication inside the swarm should be allowed by the firewall
  • only the nginx instance should be accessible from the 'outside' on the HTTP(S) ports 80 and 443

6.7.3.1 NGINX Reverse Proxy

For the nginx setup two components are required:

  • configuration file which can be dynamically adapted to new or leaving wordpress hosts
  • UDS script which installs, configures and adapts nginx configuration during the system lifetime

Configuration Template

In this example a configuration template is used to generate a suitable nginx configuration file during the installations and update it on related swarm changes.

upstream upservers {
        ${WMAIU_UPSTREAM_SERVERS}
}

server {
    listen 80;
    return 301 https://$host$request_uri;
}

server {
        listen       443 ssl;
        server_name  ${WMAIC_PUBLIC_HOSTNAME};

        ssl on;
        ssl_certificate_key     /etc/nginx/ssl/wmaic_key.pem;
        ssl_certificate         /etc/nginx/ssl/wmaic_cert.pem;

        location / {
                proxy_set_header        Host $host;
                proxy_set_header        X-Real-IP $remote_addr;
                proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header        X-Forwarded-Proto $scheme;
                proxy_pass              http://upservers;
                proxy_read_timeout      90;
        }
}

The above template looks like a regular nginx configuration file but with some special WMAIS template tags:

  • line 2 : usually this line should contains references to upstream servers to which nginx will forward the connections. Syntax should be like " server ip-address:port; ". Instead of this a variable name is placed here, this variable will be generated dynamically, on the corresponding swarm instances and will contain the actual "server ..." statements for the current swarm. wmaic_parse_template function will replace this variable in the template with the actual values and generate the final nginx configuration file.
  • line 12 : to allow nginx to route properly the request the current configuration file will respond to the current cloud instance hostname, this variable will be defined automatically by the wintermute.ai server .

NGINX UDS

following nginx UDS script should be generic and not only limited to this wordpress scenario. As such it requires three parameters which defines:

  • -t : tag of the swarm instance to which requests will be forwarded to. Since the cloud instance hostnames and ips are highly dynamical, the use of tags is more appropriate for identification.
  • -p : port to which requests will be forwarded to
  • -c : location of the configuration template file (see above) on the file system of an instance

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
#!/bin/bash
#
# Ubuntu based nginx reverse proxy UDS, accepts three call arguments:
#
#   -p <port>   defines the port to which connection should be forwarded
#   -t <tag>    defines host tags to which the forward should be send
#   -c <conf>   defines where the nginx reverse proxy config template is 
#               located on the WMAIC client
#
# e.g.: nginx_reverse_proxy.sh -c /mnt/crypted01/data1/nginx.tmpl -p 3000 -t nodejs
#

if [ -f /etc/profile.d/wmaic.sh ];then
    source /etc/profile.d/wmaic.sh
else
    echo "WMAIC shell library not found, aborting!" 1>&2
    exit 1
fi

wmaic_run_in_background

export DEBIAN_FRONTEND=noninteractive 
export DEBIAN_PRIORITY=critical

FORWARD_TO_PORT="3000"
FORWARD_TO_NODES="appserver"
NGINX_CONF_TEMPLATE="/mnt/crypted01/templates/nginx.tmpl"

while getopts "p:t:c:" option; do
    case "${option}" in
        p) FORWARD_TO_PORT="${OPTARG}"
           ;;
        t) FORWARD_TO_NODES="${OPTARG}"
           ;;
        c) NGINX_CONF_TEMPLATE="${OPTARG}"
           ;;
        *) eXit "unknown call argument: ${option}"
           ;;
    esac
done
shift $((OPTIND-1))

if ! wmaic_exclusive_run;then
    eXit "another copy of this script seems to run"
fi 

if wmaic_is_tag_assigned "nginx";then
    if wmaic_is_swarm_member;then
        wmaic_wait_for_instances
        wmaic_refresh_env
        cloak_swarm_arg="-s"
        export WMAIU_UPSTREAM_SERVERS=$(wmaic_get_instances -i -t "${FORWARD_TO_NODES}" -s '\n' -p 'server\s' -x ":${FORWARD_TO_PORT};")
    else
        cloak_swarm_arg=""
        export WMAIU_UPSTREAM_SERVERS="server 127.0.0.1:${FORWARD_TO_PORT};"
    fi

    wmaic_cloak_instance ${cloak_swarm_arg}
    wmaic_open_port -p 80,443

    if ! wmaic_is_swarm_update_action;then
        apt-get -q -y install docker.io openssl || eXit "failed to install 'docker openssl'"
        systemctl enable docker.service
        if ! systemctl --no-pager status docker.service >/dev/null;then
            systemctl --no-pager start docker.service || eXit "failed to start the docker daemon"
        fi
        mkdir -p /usr/local/wmaic/nginx/{ssl,conf.d} || eXit "failed to create /usr/local/wmaic/nginx/{ssl,conf.d} directories"
        openssl req -x509 -newkey rsa:2048 -keyout /usr/local/wmaic/nginx/ssl/wmaic_key.pem -out /usr/local/wmaic/nginx/ssl/wmaic_cert.pem -days 3000 -nodes -subj "/CN=${WMAIC_PUBLIC_HOSTNAME}" || eXit "SSL cert creation failed"
        wmaic_parse_template -i ${NGINX_CONF_TEMPLATE} -o /usr/local/wmaic/nginx/conf.d/wmaic.conf || eXit "failed to create nginx.conf file from template ${NGINX_CONF_TEMPLATE}"
        docker run --name wmaic_nginx --restart=always -d -p 80:80 -p 443:443 -v /usr/local/wmaic/nginx/conf.d/wmaic.conf:/etc/nginx/conf.d/wmaic.conf:ro -v /usr/local/wmaic/nginx/ssl:/etc/nginx/ssl -v /var/run/docker.sock:/tmp/docker.sock:ro nginx || eXit "failed to start nginx docker container"
        wmaic_register_for_swarm_updates
    else
        docker exec wmaic_nginx service nginx status || eXit "nginx docker container is not running properly"
        wmaic_parse_template -f -i ${NGINX_CONF_TEMPLATE} -o /usr/local/wmaic/nginx/conf.d/wmaic.conf || eXit "failed to re-create nginx.conf file from template"
        docker exec wmaic_nginx service nginx reload || eXit "nginx configuration reload failed"
    fi
fi

wmaic_release_exclusive_run

exit 0
  • lines 13-18 : load the WMAIC shell library with special variables and function
  • line 20 : the wmaic_run_in_background tag makes sure that the script execution (which is default set to serial ) is executed in background. This is very important when using UDS in swarms, as there can be a lot of asynchronous communication and to prevent dead-lock situations, UDS scripts which interact with other swarm components should run in background.
  • lines 22-27 : define a few default variables
  • lines 29-41 : extract script call parameters
  • lines 43-45 : in a swarm environment a lot of changes can happen simultaneously (e.g. 3 instances die). wmaic_exclusive_run () function can used to ensure that the script is executed only once. In this example UDS script exits if another instance is running, other use of wmaic_exclusive_run() can keep the script in a wait-loop until the other script instances exit and then proceed with the execution of the current script.
  • line 47 : here is another example how to check with wmaic_is_tag_assigned() function if a tag is assigned to the current cloud instance. So the script will continue only if the tag "nginx" is assigned to the swarm instance
  • line 48 : as we are creating a generic nginx reverse proxy UDS which can be applied to a swarm instance or to a stand-alone machine, here we can define settings for each of this scenarios
  • line 49 : as different swarm instances may need some time to become available, we use wmaic_wait_for_instances to pause the script until the details (IP and tags) of all other swarm instances are available
  • line 50 : after the swarm details are complete, we refresh our WMAIC shell library with all special variables.
  • lines 51 and 54 : in the line 58 we activate a iptables based firewall protection around our swarm. As this is a generic nginx UDS script we allow all swarm members to communicate with each other without borders (-s option). If this is a stand-alone nginx installation, we just protect the stand-alone instance with firewall without any unrestricted communication from the outside.
  • lines 52 and 55 : here we dynamically generate the WMAIU_UPSTREAM_SERVERS variable which we use in the above nginx configuration file. In this example we have two different cases how this variable can be defined:
    • line 55 : if this is a stand-alone instance we assume that the upstream server is located on the same machine and we only need to define the FORWARD_TO_PORT, which is provided as a call argument of the script.
    • line 52 : on a swarm instances we use wmaic_get_instance() function to get the information about other swarm instances with the requested tag. In our example the tag will be "wordpress". As we would need one upstream server definition per line, we define "\n" (newline) as separator (-s). Also to get nginx compatible upstream server definition we define "server " as prefix (-p) and postfix ":port;" (-x) on each upstream server record.
  • line 58 : wmaic_cloak_instance() is used to build a iptables based firewall protection around the instance. In a swarm the communication with other swarm instances should be allowed without limitations (-s parameter)
  • line 59 : as it can be seen from the swarm architecture picture above, the nginx-instance should be the only one which is accessible from the outside internet, with the wmaic_open_port() function the HTTP and HTTPS ports are opened for public access.
  • line 61 : in the line 61 we register this script to receive swarm updates (leave or join of new nodes), this means our script is running in two modes, the first time during the initial setup of the instance and all following times as soon as a swarm update happens. With the wmaic_is_swarm_update_action() function we check if this is the initial run or swarm update. Lines 61-71 will be executed only during the initial install and lines 73-75 only during swarm updates. All other lines of the script will be executed on both runs.
  • lines 62-66 : install docker and other required software and enable the docker services in the system
  • line 67 : as all data in a docker container gets lost after container termination, we create data directories on the host system which we will mount-bind in to the docker container and which will ensure that the data is safe even after re-building of a docker container.
  • line 68 : creation of a self-signed SSL certificate for the current hostname of the instance.
  • line 69 : parsing the nginx template configuration file and replacing the WMAIU_UPSTREAM_SERVERS variable and others with the values which have generated before in the script.
  • line 70 : starting the docker with the official nginx container, ensure HTTP and HTTPS ports are forwarded to the container and mount our directories with persistent data.
  • line 71 : with the wmaic_register_for_swarm_updates() function this script is registered to be re-executed on all future swarm updates (new instances join or leave). For more details see description of the line 60.
  • lines 73-75 : these lines are only executed on swarm update actions:
    • line 73 : ensure that the docker service is running fine
    • line 74 : re-generate the nginx configuration file to make sure any new upstream nodes are properly placed in the config file, or nodes which have left are removed from the config.
    • line 75 : force nginx to reload the new generated configuration file
  • line 79 : releases the exclusive script execution, means other nginx UDS script instances can now continue the execution.
div

6.7.3.2 Wordpress

Wordpress UDS accepts three optional parameters:

  • -d : defines the IP and the PORT of the MySQL DB server which should be used
  • -u : defines an optional DB user which should be used, if not defined then user 'root' will be used
  • -p : password for the optional DB user

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
#!/bin/bash
#
# Ubuntu based wordpress UDS, accepts following _optional_ args:
#
#   -d <dbms>      define the IP:PORT of the MySQL server
#   -u <user>      define optional DB user, if not 'root' will be used
#   -p <pass>      define password for the optional DB user, if not 
#                  for SWARM instances the SWARM-ID is used, for standalone
#                  WMAIC clients the local-secret variable is used
#

if [ -f /etc/profile.d/wmaic.sh ];then
    source /etc/profile.d/wmaic.sh
else
    echo "WMAIC shell library not found, aborting!" 1>&2
    exit 1
fi

wmaic_run_in_background

export DEBIAN_FRONTEND=noninteractive 
export DEBIAN_PRIORITY=critical

MYSQL_DB=""
MYSQL_USER=""
MYSQL_PASS=""


while getopts "d:u:p:" option; do
    case "${option}" in
        d) MYSQL_DB="-e WORDPRESS_DB_HOST=${OPTARG}"
           ;;
        u) MYSQL_USER="-e WORDPRESS_DB_USER=${OPTARG}"
           ;;
        p) MYSQL_PASS="-e WORDPRESS_DB_PASSWORD=${OPTARG}"
           ;;
        *) eXit "unknown call argument: ${option}"
           ;;
    esac
done
shift $((OPTIND-1))

if ! wmaic_exclusive_run;then
    eXit "another copy of this script seems to run"
fi 

if wmaic_is_tag_assigned "wordpress";then
    if wmaic_is_swarm_member;then
        wmaic_wait_for_instances
        wmaic_refresh_env
        cloak_swarm_arg="-s"
        if [ -z "${MYSQL_DB}" ];then
            DB_HOSTS=( $(wmaic_get_instances -t "mysql" -x ':3306') )
            MYSQL_DB="-e WORDPRESS_DB_HOST=${DB_HOSTS[0]}"
        fi
        if [ -z "${MYSQL_PASS}" ];then
            MYSQL_PASS="-e WORDPRESS_DB_PASSWORD=${WMAIC_SWARM_ID}"
        fi
    else
        cloak_swarm_arg=""
        MYSQL_DB="-e WORDPRESS_DB_HOST=127.0.0.1:3306"
        if [ -z "${MYSQL_PASS}" ];then
            MYSQL_PASS="-e WORDPRESS_DB_PASSWORD=${WMAIC_SECRET}"
        fi
    fi

    wmaic_cloak_instance ${cloak_swarm_arg}
    wmaic_wait_for_services -w 5 -x 60 -X 10 -t mysql -p 3306     

    if ! wmaic_is_swarm_update_action;then
        apt-get -q -y install docker.io || eXit "failed to install 'docker'"
        systemctl enable docker.service
        if ! systemctl --no-pager status docker.service >/dev/null;then
            systemctl --no-pager start docker.service || eXit "failed to start the docker daemon"
        fi
        docker run --name wmaic_wordpress --restart=always -d -p 3000:80 ${MYSQL_DB} ${MYSQL_USER} ${MYSQL_PASS} -v /var/run/docker.sock:/tmp/docker.sock:ro wordpress || eXit "failed to start wordpress docker container"
        wmaic_register_for_swarm_updates
    else
        :
    fi
fi

wmaic_release_exclusive_run

exit 0

the UDS script is created with a structure which is similar to the nginx UDS, so only different lines are described here:

  • line 53 : if a mysql db host wasn't provided via call parameters then get (all) hosts with the tag "mysql" and append the port number to the hostname (':3306)
  • line 54 : prepare the MySQL Environment variable as it is required by the wordpress docker image, in this example we are using only one MySQL server, so if in line 53 more than one MySQL tagged hosts were found, we take only the first ([0])
  • line 57 : if the instance is part of a swarm then we take the WMAIC_SWARM_ID variable (which is a random uuid string) as common known secret and use it as MySQL password
  • line 61 : on a stand-alone instance we assume that the MySQL DB is running on the same machine
  • line 63 : on a stand-alone instance we use the WMAIC_SECRET variable for the MySQL password. This variable is only visible inside of the system
  • line 68 : as wordpress initialization can only be done after a database service is available, we wait here for TCP port 3306 to become available on all machines with the tag 'mysql'. We try to connect to this port 60 times, with 10 seconds pause between the attempts and 5 seconds connection timeout. As soon as database becomes available, the script execution continues. In case it is not available after 60 attempts, the scripts exits with an error.

6.7.3.3 MySQL-DB

MySQL UDS accepts four optional parameters:

  • -d : defines an optional database inside of MySQL to be created during the initial setup
  • -u : defines an optional DB user to be created
  • -p : password for the optional DB user
  • -c : define a customized MySQL DB config template file, this template file, if provided, will be parsed with wmaic_parse_template() function

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
#!/bin/bash
#
# Ubuntu based MySQL server UDS, accepts following _optional_ args:
#
#   -d <db>        define a DB which should be created inside of MySQL
#   -u <user>      define optional user
#   -p <pass>      define password for the optional user, if not 
#                  for SWARM instances the SWARM-ID is used, for standalone
#                  WMAIC clients the local-secret variable is used
#   -c <config>    define a customized MySQL config file on the WMAIC client
#                  use full path to the file
#

if [ -f /etc/profile.d/wmaic.sh ];then
    source /etc/profile.d/wmaic.sh
else
    echo "WMAIC shell library not found, aborting!" 1>&2
    exit 1
fi

wmaic_run_in_background

export DEBIAN_FRONTEND=noninteractive 
export DEBIAN_PRIORITY=critical

MYSQL_DB=""
MYSQL_USER=""
MYSQL_PASS=""
MYSQL_CFG=""
MYSQL_MOUNT_CFG=""

while getopts "d:u:p:c:" option; do
    case "${option}" in
        d) MYSQL_DB="-e MYSQL_DATABASE=${OPTARG}"
           ;;
        u) MYSQL_USER="-e MYSQL_USER=${OPTARG}"
           ;;
        p) MYSQL_PASS="-e MYSQL_PASSWORD=${OPTARG}"
           ;;
        p) MYSQL_CFG="${OPTARG}"
           ;;
        *) eXit "unknown call argument: ${option}"
           ;;
    esac
done
shift $((OPTIND-1))

if ! wmaic_exclusive_run;then
    eXit "another copy of this script seems to run"
fi 

if wmaic_is_tag_assigned "mysql";then
    if wmaic_is_swarm_member;then
        wmaic_wait_for_instances
        wmaic_refresh_env
        cloak_swarm_arg="-s"
        MYSQL_ROOT_PASS="-e MYSQL_ROOT_PASSWORD=${WMAIC_SWARM_ID}"
    else
        cloak_swarm_arg=""
        MYSQL_ROOT_PASS="-e MYSQL_ROOT_PASSWORD=${WMAIC_SECRET}"
    fi

    wmaic_cloak_instance ${cloak_swarm_arg}

    if ! wmaic_is_swarm_update_action;then
        apt-get -q -y install docker.io || eXit "failed to install 'docker'"
        systemctl enable docker.service
        if ! systemctl --no-pager status docker.service >/dev/null;then
            systemctl --no-pager start docker.service || eXit "failed to start the docker daemon"
        fi
        mkdir -p /usr/local/wmaic/mysql/{db,conf.d} || eXit "failed to create /usr/local/wmaic/mysql/{db,conf.d} directies"
        if [ -n "${MYSQL_CFG}" ];then
            wmaic_parse_template -i ${MYSQL_CFG} -o /usr/local/wmaic/mysql/conf.d/wmaic.cnf || eXit "failed to create mysql.conf file from template ${MYSQL_CFG}"
            MYSQL_MOUNT_CFG="-v /usr/local/wmaic/mysql/conf.d/wmaic.cnf:/etc/mysql/conf.d/wmaic.cnf:ro"
        fi
        docker run --name wmaic_mysql --restart=always -d -p 3306:3306 ${MYSQL_MOUNT_CFG} ${MYSQL_DB} ${MYSQL_USER} ${MYSQL_PASS} ${MYSQL_ROOT_PASS} -v /usr/local/wmaic/mysql/db:/var/lib/mysql -v /var/run/docker.sock:/tmp/docker.sock:ro mysql || eXit "failed to start mysql docker container"
        wmaic_register_for_swarm_updates
    else
        :
    fi
fi

wmaic_release_exclusive_run

exit 0

the UDS script is created with a structure similar to the nginx and wordpress UDS above, no new statement types are used.

6.7.3.3 Starting the Swarm

Following steps need to be executed to start the above nginx/wordpress/mysql swarm:

  1. Create the above UDS scripts inside of wintermute.ai server
  2. Place the nginx.tmpl file in to a data directory on your local wintermute.ai server from where it is allowed to be uploaded to the clients.
  3. Ensure to have a Ubuntu 16.04 image available inside of wintermute.ai server
  4. Start a instance (swarm) with following settings:

Instance Tab:

fill out the Instance Tab similar to the screen shot below, vli-ubuntu-1604 is our Ubuntu 16.04 image and fullOpen is our security group which allows unlimited access to the instances on the provider level.


Storage Tab:

The directory /tmp/data1 is configured in our example as the one allowed for data uploads, in this directory we have placed the nginx.tmpl file.


Setup Tab

select the UDS script to be executed on the instances, the sequence order doesn't matter as they will be only executed on the instances with matching tags. Provide also parameters to the nginx.tmpl file according to your definitions above.

Summary Tab

review the setup and start the swarm

Connect to the Swarm

as soon as swarm is up and running, connect to the public IP address of the machine with the tag nginx:


after successful HTTPS connection a Wordpress setup screen should be shown:


and after clicking to 2-3 wordpress setup dialogs a full configured wordpress instance is ready for use:


7. User Guide

7.1 Login and Password

The wintermute.ai server has a browser-based, responsive user interface and can be accessed from any computer or mobile devices. Ask your IT department for the access URL of your local wintermute.ai server , your login user name and password.

On a freshly installed wintermute.ai server the access URL, login and temporary password are displayed in the last lines of the installation output and in the installation log file: /var/log/wmais/installation. log

Open the wintermute.ai server access URL and log in:

After the first login the user should immediately change their initial password by clicking on the user icon in the upper right corner and selecting "Profile: loginname " from the menu. In the "Edit User" dialog the user can update their settings and change their password.

Log out by clicking on the person icon in the upper right corner and selecting "Logout" from the menu.

7.2 Dashboard

7.2.1  Overview

After a successful login the first page is the dashboard which is the main page for a user's interaction with the wintermute.ai server .

On  the dashboard a summary of all important information about a user's activities is displayed.

In the main pane the following items are displayed:

  • Instances Til e - displays how many instances (virtual machines) belonging to the current user are running successfully in the cloud.
  • EBS Drives Tile - displays how many additional (and optional) encrypted EBS drives are attached to the running instances.
  • EBS Size Tile - displays how big (GB) these additional EBS drives are.
  • Tasks Tile - displays how many tasks are currently running. Tasks are explained in more detail later in this chapter.
  • Logs - displays tasks' activities in the form of a log journal.
  • Created Instances Graph - displays how many instances were deployed by the current user in the past.
  • Active Instances Graph - displays how many instances belonging to the current user are active and currently running.
  • EBS Volumes Graph - displays how many EBS drives were used by the user's cloud instances in the past. Please note that every cloud instance hast at least one EBS volume which contains its operating system. If selected the instance can have additional (encrypted) EBS volumes attached, which are managed by wintermute.ai server .

The user can click on tiles and log entries to retrieve additional information.

7.2.2  Tasks

A single user can have multiple tasks running on the wintermute.ai server , e.g.: starting 5 cloud instances; downloading data from an instance; patching 3 instances; destroying obsolete instances; etc. To ensure that all these tasks can be executed in parallel, a task manager is provided by the wintermute.ai server . To get an overview of past and current running tasks, the user can click on the tasks icon in the upper right corner of the wintermute.ai server interface :

In the screen shot above there are no running tasks, 4 of 5 tasks were completed successfully and one "create_instance" task failed.

There could be different reasons for an instance startup failure, e.g.:

  • the user selected a non-existing software package to be installed; or
  • the user tried to upload/ download data from/ to a directory without the correct permissions.

Clicking on the task log details of the execution or the failure will display a reason for the failure. This should be the first step to analyze the failure.

7.3 Instances

7.3.1  List Instances

By clicking on the "Instances" tile on the dashboard or the "Instances" wintermute.ai server menu in the navigation pane, a detailed view of all active instances is displayed.


Every active instance is displayed as a row record in a table and has the following attributes:

Column Name Description

Extra

Info

ID

Instance name which was selected during the instance start and instance ID which was assigned by the cloud provider.

(tick)
Provider Name of the cloud provider where this instance is running. (tick)
Type Cloud type, displayed as an icon. The current version of wintermute.ai server supports only Amazon Web Services. Future versions will support OpenStack and other cloud technologies. (tick)
OS Operating System, displayed as an icon. Supported are: Debian, Ubuntu, Fedora, CentOS, RHEL and SLES. (tick)
Storage Indicates whether additional (encrypted) EBS storage is attached and how big it is. (tick)
Status Current status of the instance. Status can be: creating, initializing, running, rebooting, mounting, shutting-down, terminating, stopped or failed. (tick)
Created/Updated Time stamp of instance creation and last modification.
Controls Action tasks which can be applied to the instance.


Based on the current status the corresponding row changes its color.

  • green - ready to go; instance is running properly
  • yellow - not ready: initializing, rebooting, terminating or failed
  • white - creating,stopped or shutting-down phase

Fields which are marked with a (tick) in the "Extra Info" column in the above table display additional useful information when clicked. The following screen shot shows additional information about the "ID" field in a pop-up:


7.3.2  Start New Instaces

To start a new instance, from the "Instances" page, click on the "Launch New Instance" button.


The following table explains the different options the user can define in the "Create Instance" dialog:

Tab Section Description Comment
Instance Name Instance identifier which is displayed in different instance lists. The name should be short but concise. e.g. : RHEL7.1@EU-Center
Description Use this field to provide a more detailed description of the instance'(s) use and purpose. e.g.: 5 machines for LAMP tests at Frankfurt DC
Tags

Tags are single words which can be applied to a single or multiple instances and are passed through to the UDS scripts inside a variable. Depending on these tags UDS script can customize the machine for the given purpose. Instances are addressed in numerical order. Syntax is as follows:

instance#:tag

or

firstInstance#-lastInstance#:tag 

Multiple tag definitions can be provided in a comma "," separated list.

See following example for tagging a swarm of 5 instances which are used for a LAMP test cluster:

1-5:php, 2-3:apache, 4-5:mysql, 1:haproxy


Instance 1 to 5 got the tag "php"; Instances 2 and 3 got an additional "apache" tag; Instances 4 and 5 got an additional "mysql" tag; Instance 1 got an additional "haproxy" tag

Operating System From this list a user can select all instance (OS) images which were imported and activated by the wintermute.ai server administrator. Every image is bound to a (geographical) location.
Instance Type From this list a user can select the instance machine types which were activated by the wintermute.ai server administrator. The more CPU cores and RAM, the more expensive the usage time is.
Security Group Security groups are a kind of firewall with rules which regulates the public TCP/IP access to the instance. It's highly recommended to open only the really necessary access ports for public (Internet) access. Different access rules can be bound to a "Security Group" and applied to a machine. The local wintermute.ai server administrator should notify its users if any additional Security Groups need to be applied to new instances.
Instances Number of instances which should be started during the current launch. e.g.: 5
Swarm Name Define if the current instance(s) should be part of a swarm and define the swarm name. Swarm member machines share common informations and are aware of each other (e.g. tags of other swarm members, their IP addresses, etc. Swarms are great for managing group or cluster of machines and are described in detail in the chapter 6.7
Storage


Instance Storage

Select which storage strategy should be realized during the current cloud session. The following options are available:

  • New - create a new EBS storage volume and attach it to the instance after launch.
  • Reuse - reuse already available EBS storage volume and attach it to the instance after launch.
  • None - no extra storage is required.

Please note, wintermute.ai server encryption can only be applied to the additional storage volumes; by selecting "None", encryption won't be possible.


Instance Volumes

This selection appears only if "New" additional EBS storage volume was selected previously. The following options are available:

  • Encrypted - by enabling this check box the newly created EBS volume will be automatically encrypted with a newly generated encryption key. The wintermute.ai server will manage all the encrypting and key management automatically in the background.
    (warning) For security it's highly recommended that this setting is enabled.
  • GB - storage volume size in GB.
  • Storage Type - type of the storage backend. The following values are available:
    • General Purpose (SSD)
    • Provisioned IOPS (SSD)
    • Magnetic
  • Mount Point - file system directory where the attached EBS storage volume should be mounted.
  • Device - define the device name to which the attached EBS system should be bound. If unsure what to select, keep the default setting which is known to work on all supported Linux distributions.

e.g.:

Mount Point: /mnt/data

Device: /dev/xvdf

Data Synchronization

This section defines whether local data from the wintermute.ai server should be automatically uploaded through an encrypted channel to the instance.

This selection appears only if "New" additional EBS storage volume was selected previously. The following options are available:

  • Upload - this check box activates data upload.
  • Data Source - this defines a file system directory, mostly a network file system mount, into which content should be uploaded in the cloud instance.
  • Data Destination - this defines a files system directory in the instance.
    (warning) For security reasons this should be either the mount point of the encrypted EBS volume or a subdirectory of it.

Please note that the wintermute.ai server administrator defines in the configuration a file system location from which data upload is allowed. So, user can provide either a full file system path to this directory (or subdirectories of it) or a relative path from inside this directory. Providing any other file system path will result in a error. This ensures that only data which is explicitly allowed to be used in the cloud can be uploaded.

e.g.: user bob123 would like to upload data in to a new cloud instance.

  • Encrypted EBS volume is mounted on the instance to the mount point: /mnt/data
  • Cloud Data Directory defined by the admin is set to /home/%USER%/cloud
  • User bob123 places data he would like to use in the cloud in the directory: /home/bob123/cloud/datadeck1
  • Data Source is set to: /home/bob123/cloud/datadeck1
  • Data Destination is set to: /mnt/data

So, after the instance is up and running, the content of bob123 's directory /home/bob123/cloud/datadeck1 will be available in the cloud instance in the directory /mnt/data .

Reuse Volume

This selection appears only if "Reuse" additional EBS storage volume was previously selected. The following options are available:

  • Volume List - select from the list of volumes the one which should be reused.
  • Device - define the device name to which the attached EBS system should be bound. If unsure what to select, keep the default setting which is known to work on all supported Linux distributions.

Setup




Update

By enabling this option after the instance is booted up for the first time, system updates will be installed.


This will only work if the OS package manager (yum, apt-get or zypper) inside the cloud image is properly configured and can fetch update packages from the Internet.

It is possible if using an older instance image that hundreds of software packages need to be downloaded and upgraded; this can take some time.
Reboot By enabling this in conjuction with the "Update" option, a reboot of a system will be initiated if during the update a linux kernel update was installed. This can delay the system readying by about 1-2 minutes.
Software Packages Management

With this field additional software packages or software groups (if supported by the package manager) can be triggered for un/install. Syntax for the package manager is as follows:

name[:install|:uninstall][:package|:group]

install and package are preset as defaults, so a definition of a package name is equal to name :package:install.

More then one package/group or mixed form of them can be managed by putting them in a comma  "," separated list.

This will only work if the OS package manager (yum, apt-get or zypper) inside the cloud image is properly configured and can fetch software packages from the Internet.

e.g.:

screen, mc, cups:uninstall, Development Tools:group, Development Libraries:group

Service Management

With this field system services (daemons) can be managed. Syntax for service management is as follows:

service:

Multiple service definitions can be provided as a comma "," separated list. By using "start", the service will be started on the instance and also enabled to start after a reboot. By using "stop", the service will be stopped (if running) and disabled to prevent it from startup after a reboot.

e.g.:

sshd:start, cupsd:stop

UDS

User Defined Script can be used to customize cloud instances. The following options can be selected:

  • Sequence - define in which order of execution this script should run. It's a good idea to define sequences with gaps between so that the on-demand insertion of an additional script could be performed.
  • Script - select available script from menu. This could either be a user's own script or a shared script from other users on the wintermute.ai server.
  • Parameters - if the script requires or allows additional call parameters, they can be provided here.

By clicking on the "Add" button the script will be inserted at the defined execution position.

UDS scripts are executed in serial. To make them run in parallel please consult the Scenarios chapter in the Admin Guide of this book.

e.g.:

Seq UDS Params
5 configEnv.sh -f /mnt/data/myEnv.cfg
10 runMPI.sh -i /mnt/data/input.deck
15 postProcess.sh -d /mnt/data/results/



Password

By enabling this option it is possible to change the password of an existing user or create a new user in the cloud instance.

The following fields need to be filled in:

  • User - username of the user whose password should be changed. If this username does not exist yet in the system, it will be automatically created.
  • Password - new password for the user.
  • Repeat Password - repeat the password for the user in order to avoid mistakes.

Summary Review This tab provides a quick review of all selected instance startup options. The user can either go back and mak e changes if something is missing, or click "Create" to create a new cloud session.

7.3.3  Create Cloud Templates

Cloud Templates can be used to save all settings which were made during a cloud session setup and re-load them any time later. To do this, click the "Save as template" button in the "Summary" tab of the "Create Instance" dialog.

The current cloud session setup will be saved under the "Name" identifier in the templates list.


7.3.4  Use Cloud Templates

A previous saved cloud session setup can be loaded for reuse by clicking on the "From Template" selection in the "Create Instance" dialog and selecting the required session by name from the list.

After the session is loaded from the template, the user can adjust all parameters before starting the new session.

7.3.5  Delete Cloud Templates

A previously saved cloud session template can be deleted by loading it as described in "Use Cloud Templates" chapter above and by clicking the "Delete template" button in the bottom right corner of the "Create Instance" dialog.

7.3.6  Instance Controls

Active instances can be controlled by clicking the "Actions" menu for the corresponding machine in the "Instances" list. The action tasks which can be applied are:

Action Description
Edit Security

This opens a sub-menu with permissions settings, which allows the corresponding users group to:

Access Setting Allows
Change Instance
Edit instance properties
Control Instance
Start / stop an instance
View Instance
See an instance in various views
SSH Console Start a Secure SHell connection to the corresponding host. The SSH connection will be displayed directly in a new browser image, SSH key handling and authentication will be done by the wintermute.ai server automatically in the background.
Web https Open a HTTPS Web access in a new browser tab to the current cloud instance. This will of course only work if a web-server is serving on the HTTPS port (443) and this port access is allowed by the security group and (if enabled) local firewall on the host.
Web http Open a HTTP Web access in a new browser tab to the current cloud instance. This will of course only work if a web-server is serving on the HTTP port (80) and this port access is allowed by the security group and (if enabled) local firewall on the host.
Upload Data

Users can use this action to start a data upload to the cloud instance. The following parameters can be defined in the "Upload Data" dialog:

  • Source Path - define on the wintermute.ai server the local path which contains data for upload.
  • Target Path - define the remote path on the cloud instance with which the content of the "Source Path" directory should be synchronized.

(lightbulb) Please always add a trailing "/" to both paths' definitions.

(lightbulb) Data synchronization is realized in a way that only new or changed data is uploaded (where the data base already exists , e.g. from a data synchronization task performed during the instance setup).

Please note that the wintermute.ai server administrator defines in the configuration a file system location from which data upload is allowed. So, a user can provide either a full file system path to this directory (or subdirectories of it) or a relative path from inside this directory. Providing any other file system path will result in a error. This ensures that only data which is explicitly allowed to be used in the cloud can be uploaded.

Download Data

User can use this action to start a data download from the cloud instance. The following parameters can be defined in the "Download Data" dialog:

  • Source Path - define on the wintermute.ai server the local path which contains data for download.
  • Target Path - define the remote path on the cloud instance with which the content of the "Source Path" directory should be synchronized.

(lightbulb) Please add always a trailing "/" to both paths definitions.

(lightbulb) Data synchronization is realized in a way that only new or changed data is downloaded (where the data base already exists, e.g. from a data synchronization task during the instance setup).

Please note that the wintermute.ai server administrator defines in the configuration a file system location from which data download is allowed. So, a user can provide either a full file system path to this directory (or subdirectories of it) or a relative path from inside this directory. Providing any other file system path will result in a error. This ensures that only data which is explicitly allowed to be used in the cloud can be downloaded.

Mount Storage This action can be used if additional, encrypted storage volumes are attached to the instance. After an instance reboot these volumes can't be mounted automatically as the encryption passphrase is never stored in the cloud but only on the wintermute.ai server . By using this action wintermute.ai server will ensure that the remote encrypted EBS storage volume will be mounted on the corresponding cloud instance.
Power Off With this action the instance is shut down and powered off  so that it's not consuming any CPU time but is still available and saved in the cloud storage. After switching to this "stopped" status, the instance can be re-started again with the "Power On" action which will have become available in the "Actions" menu.
Terminate Terminate the corresponding instance. Unlike "Power Off", this action will completely destroy the instance and no re-starting will be possible. After this action is completed, the instance won't consume any CPU time or use any storage space.

7.3.7  Batch Controls

Similar to instance controls, some actions can by applied to a group of instances by selecting the check box in the first column of the "Instances" page and then selecting an action from the "Batch Actions" menu. The actions result is the same as for single instances.


Please note that the wintermute.ai server administrator defines in the configuration a file system location from which data upload is allowed. So, a user can provide either a full file system path to this directory (or subdirectories of it) or a relative path from inside this directory. Providing any other file system path will result in a error. This ensures that only data which is explicitly allowed to be used in the cloud can be uploaded.


7.4 Storage

After clicking on the "Storage" wintermute.ai server menu a detailed view of all active used additional (encrypted) storage EBS volumes is displayed.

Every active instance is displayed as a row record in a table and has the following attributes:

Column Name Description

Extra

Info

ID Volume id which was assigned by the cloud provider to this storage volume. (tick)
Type Displays storage type as an icon. (tick)
Size Displays storage size in GB.
Attached Displays instance id which was assigned by the cloud provider to the instance to which the storage is attached. (tick)
Device Shows the device name inside the Linux OS to which this storage volume is connected.
Files System Shows the device mount point inside the Linux file system.
Encryption A lock icon indicates that the corresponding storage volume is encrypted. (tick)
Created/Updated Time stamp of storage volume creation and last modification.
Controls Action tasks which can be applied to the storage volume.

Fields which are marked with a (tick) in the "Extra Info" column in the above table are able to display additional useful information when clicked.

7.4.1 Storage Control

Active storage volumes can be controlled by clicking on the "Actions" menu of the corresponding storage.

Action Description
Edit security

This opens a sub-menu for permissions settings, it allows the corresponding users group to:

Access Setting Allows
Change Volume To change volume settings
Control Volume Control volume, like deletion
View Volume Give access to the volume, means it shares the volume with the group members
Upload Data see upload data description in the Instance Controls chapter
Download Data see download data description in the Instance Controls chapter

8. Appendices

8.1 UDS Variables

The following dynamic bourne shell variables are available in the /etc/profile.d/wmaic.sh file and can be sourced from within UDS scripts:

Variable Name Variable Type Description Example
WMAIC_ARCHITECTURE string system architecture of the current instance WMAIC_ARCHITECTURE="x86_64"
WMAIC_AVAILABILITY_ZONE string cloud provider availability/geographic zone WMAIC_AVAILABILITY_ZONE="eu-central-1b"
WMAIC_CREATED string instance creation time stamp with syntax YYYYMMDDHHmm WMAIC_CREATED="201503150822"
WMAIC_DEBUG string internal variable which indicates if the wintermute.ai client is running in debug mode WMAIC_DEBUG="False"
WMAIC_DOMAIN string DNS domain of the current instance WMAIC_DOMAIN="eu-central-1.compute.internal"
WMAIC_FQDN string full (long) host name of the current instance WMAIC_FQDN="ip-172-31-2-57.eu-central-1.compute.internal"
WMAIC_HOSTNAME string short host name of the current instance (without domain) WMAIC_HOSTNAME="ip-172-31-2-57"
WMAIC_IMAGE_ID string instance image ID WMAIC_IMAGE_ID="ami-02724d1f"
WMAIC_INET_IF array of strings array of available instance network interfaces WMAIC_INET_IF=("lo" "eth0")
WMAIC_INSTANCE_ID string cloud provider internal ID of the current instance WMAIC_INSTANCE_ID="i-a754e31b"
WMAIC_INSTANCE_NAME string instance name as it was defined during the start inside the wintermute.ai serve r WMAIC_INSTANCE_NAME="deb8@FFM"
WMAIC_INSTANCE_TAGS array of strings this array contains all tags assigned to the current cloud instance WMAIC_INSTANCE_TAGS='([0]="nginx")'
WMAIC_INSTANCE_TYPE string instance type, as declared by the cloud provider WMAIC_INSTANCE_TYPE="t2.small"
WMAIC_KERNEL string kernel type of the current instance WMAIC_KERNEL="Linux"
WMAIC_KERNEL_MAJOR string major version of the kernel WMAIC_KERNEL_MAJOR="3"
WMAIC_KERNEL_MINOR string minor version of the kernel WMAIC_KERNEL_MINOR="16"
WMAIC_KERNEL_RELEASE string full kernel release string WMAIC_KERNEL_RELEASE="3.16.0-4-amd64"
WMAIC_KERNEL_VERSION string full kernel version WMAIC_KERNEL_VERSION="3.16.0"
WMAIC_MEM_AVAILABLE string total available system memory in bytes WMAIC_MEM_AVAILABLE="1970036736"
WMAIC_NET_CIDR array of strings instance network interfaces CIDR values; the array index corresponds to the one from WMAIC_INET_IF array WMAIC_NET_CIDR=("8" "20")
WMAIC_NET_IPV4 array of strings instance network interfaces IPv4 values; the array index corresponds to the one from WMAIC_INET_IF array WMAIC_NET_IPV4=("127.0.0.1" "172.31.2.57")
WMAIC_NET_MASK array of strings instance network interfaces IPv4 netmask values; the array index corresponds to the one from WMAIC_INET_IF array WMAIC_NET_MASK=("255.0.0.0" "255.255.240.0")
WMAIC_NET_ROUTE_IF string instance default route network interface WMAIC_NET_ROUTE_IF="eth0"
WMAIC_NET_ROUTE_IP string instance default router IP WMAIC_NET_ROUTE_IP="172.31.0.1"
WMAIC_OS string instance Operating System WMAIC_OS="Linux"
WMAIC_OS_DISTRO string instance distribution WMAIC_OS_DISTRO="debian"
WMAIC_OS_FAMILY string distribution family WMAIC_OS_FAMILY="Debian"
WMAIC_OS_MAJOR_REL string Operating System major release WMAIC_OS_MAJOR_REL="8"
WMAIC_OS_MINOR_REL string Operating System minor release WMAIC_OS_MINOR_REL="2"
WMAIC_PATCHED string time stamp of when last system patches were applied to the current system with syntax YYYYMMDDHHmm WMAIC_PATCHED="201512071606"
WMAIC_PROCESSOR_COUNT string number of processor cores WMAIC_PROCESSOR_COUNT="1"
WMAIC_PUBLIC_HOSTNAME string public (accessible from the internet) instance host name WMAIC_PUBLIC_HOSTNAME="ec2-52-29-219-165.eu-central-1.compute.amazonaws.com"
WMAIC_PUBLIC_IP4 string public (accessible from the internet) instance IPv4 address WMAIC_PUBLIC_IP4="52.29.219.165"
WMAIC_REBOOTED string time stamp of when the instance was properly rebooted by the wintermute.ai server with syntax YYYYMMDDHHmm WMAIC_REBOOTED="201503150840"
WMAIC_REBOOT_REQUIRED string this variable contains the value if a reboot is required but wasn't executed for some reason(s). This could be the case if kernel patches were applied to a system but a long job run prevented the system from being rebooted. WMAIC_REBOOT_REQUIRED="False"
WMAIC_SECRET string this variable contain a random 32 character long string (hex convert) which can be used ans common secret between processes on the current cloud instance. WMAIC_SECRET="d9c1c6c0c437adca2c4b94f5db423b39"
WMAIC_SWAP_AVAILABLE string total available system SWAP space in bytes WMAIC_SWAP_AVAILABLE="0"
WMAIC_SWARM_ID string a random UUID string which is only shared between swarm members, this string can also be used as common secret between all swarm members WMAIC_SWARM_ID="df36bd41-9ddf-4e0b-99b9-7f52d39214bd"
WMAIC_SWARM_MEMBER string this variable indicates if the current cloud instance is a member of a swarm (value "True") or is not (value "False") WMAIC_SWARM_MEMBER="True"
WMAIC_SWARM_MEMBERS_EXT_IPS array of strings

this variable contains IP addresses of all other swarm members.

(info) the IP address of the current cloud node itself isn't included in this list. To get this information use the variable $WMAIC_PUBLIC_IP4

WMAIC_SWARM_MEMBERS_EXT_IPS='([0]="35.156.138.191" [1]="35.156.127.45")'
WMAIC_SWARM_MEMBER_TAGS array of strings

this variable contains tags of all other swarm members, the field index corresponds to the index of the WMAIC_SWARM_MEMBER_EXT_IPS variable. e.g. on index position 7 in both variables is stored information to the same swarm member.

(info) the tags of the current cloud node itself aren't included in this list. To get them use the variable $WMAIC_INSTANCE_TAGS

WMAIC_SWARM_MEMBER_TAGS='([0]="mysql" [1]="wordpress")'
WMAIC_SWARM_SIZE string this variable contain the size (amount of instances) of the whole swarm, including the current cloud instance WMAIC_SWARM_SIZE="3"
WMAIC_SYSTEM_ID string pseudo unique system identifier WMAIC_SYSTEM_ID="1fac3902"
WMAIC_TIMEZONE string contains the current active time zone on the instance WMAIC_TIMEZONE="UTC"
WMAIC_INSTANCE_TAGS array of strings this variable contains tags of the instance WMAIC_INSTANCE_TAGS=("webserver" "nginx" "mariadb")

8.2 UDS Functions

Following shell (bash) functions become available inside of UDS scripts as soon as the WMAIC Shell Library is loaded, this should be done in the first lines of a UDS script:

#!/bin/bash

if [ -f /etc/profile.d/wmaic.sh ];then
    source /etc/profile.d/wmaic.sh
else
    echo "WMAIC shell library not found, aborting!" 1>&2
    exit 1
fi
#
# now all UDS function are available
#
# continue with the regular UDS shell code
#
...

8.2.1 eXit()

USAGE: eXit [reason text]

this function prints out the optional "reason text" to STDERR and 
exit then with a non-zero value, which means ERROR in shell

EXAMPLE: 
  mkdir /var/lib/test || eXit "failed to create directory: /var/lib/test"

8.2.2 wmaic_parse_template()

USAGE: wmaic_parse_template -i  -o  [-f] [-b] 
       [-B ]

this function parses the input-file and replaces all ${VARIABLES} which
starts with the string "WMAI" by their content in the current script.

This could be the default "WMAIC_" variables which are available by default
on all WMAIC clients, or this could be also user generated variables
which starts with "WMAIU_" prefix. 

Multi-line variable content is also supported. Variable syntax should
always include curly braces, e.g.: ${WMAIU_VARNAME} instead of $WMAIU_VARNAME

ARGUMENTS:

-i     define the input template file which should be parsed

-o    define where the final, parsed file should be saved

-f                 'force' flag for overwriting already existing output-files

-b                 backup the output-file before overwriting it

-B        define the backup postfix which is appended to the 
                   backup file name. 
                   DEFAULT: '_BKP'

EXAMPLE: 
  wmaic_parse_template -i /mnt/data1/nginx.tmpl -o /etc/nginx/conf.d/my.conf

8.2.3 wmaic_exclusive_run()

USAGE: wmaic_exclusive_run [-x] [-w]

this function ensures that no other instance of the script is running
on the current host. If another instance is running this functions
returns a non-zero value.

Other behavior can be achieved by using following optional arguments.

ARGUMENTS:

-x   exit with error if exclusive run can't be achieved

-w   wait until exclusive run can be achieved
     re-check LOCK every 5 seconds

EXAMPLE:
  if ! wmaic_exclusive_run;then
      eXit "another copy of this script seems to run"
  fi

8.2.4 wmaic_release_exclusive_run()

USAGE: wmaic_release_exclusive_run

this function releases the exclusive run which was enabled by the
wmaic_exclusive_run() function. This should be done in the last lines
of a UDS script.

EXAMPLE:
  wmaic_release_exclusive_run

8.2.5 wmaic_get_instances()

USAGE: wmaic_get_instances [-i] [-t ] [-s ] [-p ] 
                           [-x   report only instances which have this tags assigned
                    multiple tags are to be provided as comma separated list

-s       defines separator which will be used for join
                    instances together in to a string. DEFAULT: space

                    Extra format characters ca be used in separator 
                    definition:
                    '/s' - is replaced by a space in the output record
                    '/n' - is replaced by a newline in the output record
                    '/t' - is replaced by a tab in the output record

-p          define optional prefix which should be placed in the 
                    begin of every hostname before join in to a string

-x         define optional postfix which should be appended to 
                    the end of every hostname before join in to a string

EXAMPLE1: get all hosts with zookeeper tag in a string which is ready to be
          used in a client call.

          wmaic_get_instances -i -t zookeeper -s ',' -p 'zk://' -x ':2181/api'
        
          could RETURN a string like: 
            zk://10.0.0.1:2181/api,zk://10.0.0.2:2181/api,zk://10.0.0.3:2181/api

EXAMPLE2: get all nodejs hosts and produce a multi-line string output which 
          can be used in the nginx reverse proxy configuration as upstream 
          servers which are listening on port 3000.

          wmaic_get_instances -i -t nodejs -s '\n' -p 'server\s' -x ":3000;"
        
          could RETURN a multi-line string like: 
            server 10.0.0.20:3000;
            server 10.0.0.21:3000;
            server 10.0.0.22:3000;

8.2.6 wmaic_wait_for_instances()

USAGE: wmaic_wait_for_instances

this function is used in a swarm environment and it pauses a UDS script 
execution until the current execution host receives full information 
about other swarm members. 

As in bigger swarms it can take some time until all instances are up
it's a good idea to use this function to make sure all required
information is available.

Please refresh the environment variables in the current UDS script by 
using 'wmaic_refresh_env()' to ensure that all instances information is
updated inside of the UDS script.

EXAMPLE: if wmaic_is_swarm_member;then
             wmaic_wait_for_instances
             wmaic_refresh_env
             # continue here with your UDS code
             ...
         fi

8.2.7 wmaic_wait_for_services()

USAGE: wmaic_wait_for_services <-t ] | -n  
                                <-p 
                                [-i] [-w ] [-x ] 
                                [-X ] [-v]

this function checks if a service is listening on the defined TCP ports 
on a group of swarm members. Swarm members can be addressed either by
tags and/or IPs or by '-a' flag which will check all swarm members.

This function requires at least one port definition and one of the following
arguments: '-t' or '-n' or '-a', for details see ARGUMENTS below.

This function is very useful in a swarm environment with service dependencies
e.g. application setup should only be done _after_ a DB service is up.

Please note: if the check on at least one machine (after all retries) fails 
the whole UDS script execution will be terminated.

ARGUMENTS:

-i                  include localhost in service check. DEFAULT: skip it

-w         time out in seconds on connect attempt

-x         define the number of connection retries. DEFAULT: forever

-X         pause in seconds before connection retry. DEFAULT: 10

-t   test services only on instances with the defined tags

-n   test services only on the defined IPs

-a                  test services on all swarm hosts. Please note that this
                    wouldn't include the localhost, for this purpose '-i'
                    flag must also be provided

-p           define one, multiple or TCP port-ranges to be checked
                    for services. Syntax example:

                    single port      : -p 443
                    multiple ports   : -p 80,443
                    port ranges      : -p 20-25
                    mixed definition : -p 80,443,6666-6669

-v                  turn on the verbose mode

EXAMPLE: check for database service on TCP port 3306 on all swarm members 
         with the tag 'dbserver'. Do 60 retries with 10 second pause between
         check attempts and 5 seconds timeout on TCP connection.

         wmaic_wait_for_services -i -t dbserver -p 3306 -w 5 -x 60 -X 10

8.2.8 wmaic_cloak_instance()

USAGE: wmaic_cloak_instance [-s] [-i 

this function builds a native firewall around the current cloud instance.
By default the firewall rules allow all OUTPUT connections and deny all
INPUT connection to the host. The only port which is open for INPUT by
default is the SSH port '22', this is required for WMAIS->WMAIC 
communication.

For opening additional ports for public use the WMAIC function
'wmaic_open_port()' should be used, for details see documentation below.

For opening access for other hosts (e.g. other swarm members) see arguments
below.

ARGUMENTS:

-s                  if a member of a swarm allow full INPUT access to all 
                    ports from all other swarm members

-i   allow full INPUT access to all ports from this IPs, 
                    IP ranges can be defined in CIDR notation

-t   allow full INPUT access to all ports from instances 
                    with the defined tags

EXAMPLE: setup the firewall cloak around the current host but allow all
         income communication on all ports from the swarm members

         wmaic_cloak_instance -s

8.2.9 wmaic_open_port()

USAGE: wmaic_open_port [-u] -p  [-a ]

this function opens one or multiple UDP or TCP ports for INPUT access
to all (public) or limited number of hosts (define by their IPs).

Please note that this function should only be called after the 
WMAIC firewall cloak was initialized with the 'wmaic_cloak_instance()'
function, otherwise the whole UDS script execution may fail.

ARGUMENTS:

-u          open UDP type port. DEFAULT: TCP

-p   define port(s) to be opened in the WMAIC firewall cloak
            for external (public) INPUT access

-a     limit access to the defined ports only for this IPS instead
            of a general public INPUT access

EXAMPLE: on a web-server cloud instance enable WMAIC firewall cloak
         which allows limitless communication inside of a swarm and 
         opens the HTTP (80) and HTTPS (443) ports for public access

         wmaic_cloak_instance -s
         wmaic_open_port -p 80,443

8.2.10 wmaic_register_for_swarm_updates()

USAGE: wmaic_register_for_swarm_updates

this function register the current UDS script to be called on 
swarm update events. A swarm event can happen if new instances
join the swarm or instances are terminated and leave the swarm.

The WMAIC function 'wmaic_is_swarm_update_action()' returns a 
true (shell return value 0) if this is a swarm update, otherwise
it is the initial call of the UDS script during the instance
setup phase.

EXAMPLE:
  wmaic_register_for_swarm_updates


8.2.11 wmaic_is_tag_assigned()

USAGE: wmaic_is_tag_assigned 

this function returns true (shell return value 0) if the "tag" which
is provided as argument is assigned to the current cloud instance.
Otherwise false (shell return value non-zero) is returned.

EXAMPLE: install and configure nginx web server on the current cloud 
         instance only if the "nginx" tag is assigned

  if wmaic_is_tag_assigned 'nginx';then
      # insert here the shell code for nginx setup and configuration
      ...
  fi


8.2.12 wmaic_is_swarm_member()

USAGE: wmaic_is_swarm_member

this function returns true (shell return value 0) if the current
cloud instance is a member of a swarm, false (shell return value 
non-zero) otherwise.

EXAMPLE:
  if wmaic_is_swarm_member;then
      # enter here all shell code for swarm behavior 
      ...
  else
      # enter here shell code for a stand-alone cloud instance
      ...
  fi


8.2.13 wmaic_is_swarm_update_action()

USAGE: wmaic_is_swarm_update_action

all UDS scripts are first executed during the cloud instance 
initialization phase, at this first execution by using the WMAIC
function 'wmaic_register_for_swarm_updates' UDS can also be registered
for the execution during the swarm update action.

This function returns true (shell return value 0) if the current
UDS script execution is done during a swarm update action stage. 
Otherwise false (shell return value non-zero) is returned.

EXAMPLE: install and configure docker based nginx web server on a 
         Ubuntu based system during the initial run of the UDS script,
         register the script for swarm updates and on swarm updates 
         re-configure and reload the new new nginx config file.

  if ! wmaic_is_swarm_update_action;then
      # this is the initial run, install and configure everything from scratch
      apt-get -q -y install docker.io || eXit "failed to install 'docker'"
      systemctl enable docker.service
      mkdir -p /usr/local/wmaic/nginx/conf.d || eXit "failed to create /usr/local/wmaic/nginx/conf.d directory"
      wmaic_parse_template -i /mnt/data1/nginx.tmpl -o /usr/local/wmaic/nginx/conf.d/wmaic.conf || eXit "failed to create nginx.conf file"
      docker run --name wmaic_nginx --restart=always -d -p 80:80 -p 443:443 -v /usr/local/wmaic/nginx/conf.d/wmaic.conf:/etc/nginx/conf.d/wmaic.conf:ro -v /var/run/docker.sock:/tmp/docker.sock:ro nginx || eXit "failed to start nginx docker container"
      wmaic_register_for_swarm_updates
  else
      # this is a update run, re-create the config file and reload the service only
      wmaic_parse_template -f -i /mnt/data1/nginx.tmpl -o /usr/local/wmaic/nginx/conf.d/wmaic.conf || eXit "failed to re-create nginx.conf file"
      docker exec wmaic_nginx service nginx reload || eXit "nginx configuration reload failed"
  fi

8.2.14 wmaic_get_current_stage()

USAGE: wmaic_get_current_stage

this function returns the ${WMAIC_STAGE} variable if defined, this variable
contains the name of the cloud instance setup stage during which the UDS 
script was called.

Currently following setup stages are supported:

* post-data-transfer  - this is the stage after the cloud instance is completely
                        setup, full patched, rebooted (if required) and all data
                        is transfered to the instance
* swarm-update-action - this is a stage which is entered on every swarm update,
                        update could be if a new instance is joining a swarm or
                        is leaving

8.2.15 wmaic_run_in_background()

USAGE: wmaic_run_in_background

by default UDS scripts are executed in serial order as defined by their sequence
numbers during the instance start dialog. By placing the 'wmaic_run_in_background'
shell function call in a UDS script the script will be executed in background,
parallel to other UDS scripts, if also defined. WMAIS SSH connection can break-up
the UDS script will be still executed in the background.

8.2.16 wmaic_refresh_env()

USAGE: wmaic_refresh_env

this function refresh the shell environment in the current UDS script, this is 
useful after the script has paused by waiting for other swarm nodes or services.

EXAMPLE: wait for getting all information from swarm nodes and refresh the
         environment after it happen

  wmaic_wait_for_instances
  wmaic_refresh_env


8.3 Configuration Fine Tuning

8.3.1  settings.py

The following fine tunings should be applied to the /opt/wmai/ VERSION /wmais/settings.py file. Replace VERSION with the current version of the wintermute.ai server .

Please note: usually the wintermute.ai server is delivered with good working default settings so execute fine tunning only if there is a need for it.

8.3.1.1  Dashboard History

Define in days how long the activity list should be kept on the dashboard.

WMAIS_DASHBOARD_HISTORY_DAYS = 90

8.3.1.2  Data Transfer Retries

Define with these two settings how often on a failed or aborted data transfer a retry should be made and how long the pause (in seconds) between these retries should be. The number of WMAIS_DATA_TRANSFER_RETRIES can be increased if there is a weak and unstable Internet connection or a big amount of data is being transferred.

WMAIS_DATA_TRANSFER_RETRIES = 4
WMAIS_DATA_TRANSFER_PAUSE_BETWEEN_RETRY = 10

Please note the wintermute.ai server executes an encrypted data transfer to and from the cloud instances. In the case of a breakup, data transfer will be resumed and continued from the last transfer state. This means that some breakups during the transfer shouldn't cause a problem and a successful data transfer can still be effected.

8.3.1.3  Instance Connection Attempts

wintermute.ai server uses native cloud provider API calls to create an instance and SSH connection to configure instances from within. Usually it takes some time after an instance is created and booted up before it becomes available f or SSH access. Using the following two parameters a user can fine tune how many SSH connection attempts are made before setup will be marked as failed and can define the length of the pause between the connection attempts:

WMAIS_CONNECTION_ATTEMPTS = 50
WMAIS_CONNECTION_RETRY_PAUSE = 10

8.3.1.4   Keep Running Failed Instances

If an instance setup fails the wintermute.ai server has two options as to how to continue.

  1. Cancel instance setup and stop the instance execution so the instance will be suspended to cloud storage and won't consume anymore CPU time. The user can boot-up this instance later and examine it if required, or terminate it any time.
  2. Cancel instance setup but keep the instance running in a failed state which will consume CPU time but allow the user (or admin) quickly to examine the failure.

The following variable can be set to "False" for option #1 or "True" for option #2:

WMAIS_KEEP_INSTANCE_RUNNING_ON_ERROR = True

8.3.2  Cloud Sessions Directory

Every active cloud session saves information and log files in a unique session id directory under /var/lib/wmai/sessions .Once an hour session directories of terminated cloud sessions are cleaned up and removed. However, if it's useful to keep these session tracks for auditing purposes, the session cleaning script /etc/cron.hourly/wmaisCleaner can be removed.


8.4 Configuration Files

Following important Configuration Files and Directories are used on a wintermute.ai server system and should be included in to corporate backup:

Path Type Description
/opt
directory base installation directory
/opt/wmai
directory wmai directory contains wintermute.ai server software
/opt/wmai/VERSION
directory contains the VERSION release of the wintermute.ai server software (e.g. /opt/wmai/1.0.0rc1/)
/opt/wmai/current
sym link points to the current active release of wintermute.ai server
/opt/wmai/current/wmais/settings.py
file configuration and fine tuning file of the current wintermute.ai server release
/opt/wmai/ssl
directory directory with wintermute.ai server SSL keys which are used across different release versions
/opt/extlib
directory contains 3rd party support tools and libraries which are installed during the wintermute.ai server setup
/opt/extlib/VERSION
directory contains 3rd party support tools and libraries for the VERSION release of the wintermute.ai server
/opt/extlib/current
sym link points to the current active extlib VERSION
/var/log/wmais
directory wintermute.ai server logs directory
/var/lib/wmai
directory wintermute.ai server directory with variable run-time data
/lib/systemd/system/wmais.service
file wintermute.ai server startup unit for systemd systems
/etc/init.d/wmais
file wintermute.ai server startup script for Sys-V-Init systems
/etc/default/wmais
file default settings for the current relase of the wintermute.ai server
/etc/cron.hourly/wmaisCleaner
file clean up script for outdated /var/lib/wmai/ variable data

8.5 Proxy Configuration

If web access in a company is realized through a proxy server the following steps must be executed before starting the wintermute.ai server installation. All commands are executed in the BASH shell. They may be rewritten for a preferrerd alternative shell. ">" represents the shell prompt and shouldn't be executed !

It's important to execute the setup.sh script call in the same shell in which the following proxy definitions are written !


Defining a proxy with the fqdn host name proxy1.company.com running on port 3128 without username and password:

> export http_proxy="http://proxy1.company.com:3128"
> export https_proxy="http://proxy1.company.com:3128"
>


Defining a proxy with the fqdn host name proxy1.company.com running on port 3128 with user username and no password:

> export http_proxy="http://user@proxy1.company.com:3128"
> export https_proxy="http://user@proxy1.company.com:3128"
>


Defining proxy with the fqdn host name proxy1.company.com running on port 3128 , with user username and mypass password:

> export http_proxy="http://user:mypass@proxy1.company.com:3128"
> export https_proxy="http://user:mypass@proxy1.company.com:3128"
>

If web access in the company is only possible via proxy, the appropriate export commands should also be appended to the /etc/default/wmais file, but only after the wintermute.ai server installation is finished as this file is overwritten during the installation.

HTTP/HTTPS proxy can be used for the API connection to the cloud provider (e.g. AWS) and used for instance and storage creation and management. However, in this version of wintermute.ai server for the final customization, encryption, strengthening and access, a direct SSH connection (TCP port 22) from wintermute.ai server to the instances must be possible .


8.6 Manual installation: Python 2.7.x

wintermute.ai server is built on top of Python 2.7.x, manual installation of it are required on systems which are by default shipped with older or newer versions of Python.

As an example for manual Python 2.7.x installation this chapter will cover install on a RedHat Enterprise Linux 6.x (RHEL6) system.

If Internet access in the company is realized via a proxy server, make sure you set the proper shell variables as described in the Appendix: Proxy Configuration

8.6.1 Install Required System Tools

> sudo yum -y groupinstall "Development tools"
...
> sudo yum -y install wget curl zlib-devel bzip2-devel openssl-devel ncurses-devel postgresql-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel git shadow-utils gmp gmp-devel rsync
... 

8.6.2 Download Python Sources

> cd /tmp 
tmp> wget https://www.python.org/ftp/python/2.7.14/Python-2.7.14.tgz 
...
tmp> tar xzf Python-2.7.14.tgz
... 
tmp> cd Python-2.7.14
Python-2.7.14>

8.6.3 Compile Python

Python-2.7.14> ./configure --prefix=/usr/local --with-ensurepip --enable-unicode=ucs4 --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"

8.6.4 Install Python

Python-2.7.14> sudo make altinstall

8.6.5 Check Python 2.7.x Installation

Python-2.7.14> /usr/local/bin/python2.7 --version
...
Python-2.7.14> /usr/local/bin/python2.7-config --prefix 
...
Python-2.7.14> /usr/local/bin/pip2.7 --version
...

8.6.6 Install virtualenv

Python-2.7.14> sudo /usr/local/bin/pip2.7 install virtualenv 
...
Python-2.7.14> /usr/local/bin/virtualenv --version
...

After a manual installation of a Python 2.7.x version to an alternative path the wintermute.ai server setup.sh script should be called with two additional parameters.

RHEL 6.x:

 sudo -H ./setup.sh -d rhel6 -V /usr/local/bin/virtualenv

CentOS 6.x:

sudo -H ./setup.sh -d centos6 -V /usr/local/bin/virtualenv

Please ensure that the Optional Repository which are provided by RedHat are enabled on the wintermute.ai server host system before continue with the wintermute.ai server installation.


8.7 License

EARLY ACCESS AND EVALUATION END USER LICENSE AGREEMENT



IMPORTANT - READ THIS CAREFULLY BEFORE INSTALLING, USING OR ELECTRONICALLY ACCESSING THIS PROPRIETARY PRODUCT.

This End User License Agreement ("EULA") is a legal agreement between YOU as Licensee and " wintermute.ai UG , Germany" on an evaluation and early access basis for the " wintermute.ai server " software product, which includes any versions and may include associated media, printed materials and documentation. This also include any updates or upgrades to or new versions of the original software products, if and when made available to you by " wintermute.ai UG ". By installing, copying, or otherwise using the " wintermute.ai server " product, YOU agree to be bound by the terms of this EULA. If you do not agree to be bound by these terms and conditions, do not download, install, or use any " wintermute.ai UG " software products.


(1) DEFINITIONS

In this End User License Agreement, unless the contrary intention appears,

* " EULA " means this Early Access and Evaluation End User License Agreement
* " wintermute.ai UG " means the company "wintermute.ai UG (haftungsbeschränkt), Germany"
* " Licensee " means YOU the end user, or the organization (if any) on whose behalf YOU are taking the EULA.
* " the Software " means "wintermute.ai server" product, which includes any versions and may include associated media, printed materials and documentation. This also include any updates or upgrades to or new versions of the original Software product, if and when made available to you by "wintermute.ai UG".
* " Evaluation " means using of the Software which is available free of charge for evaluation purposes for a limited period of time.
* " Evaluation period " means limited evaluation of the Software during 90 days upon issue of the Evaluation license.
* " Evaluation license " means a temporary, free of charge, license file which is valid for the Evaluation period. It activates the Software functionality and allows a limited usage of the Software.
* " Early Access " means access to the Software beta versions


(2)  LICENSE GRANT

Subject to the terms of this Agreement, " wintermute.ai UG " grants to Licensee, during the Term, a limited, revocable, non-exclusive, non-transferable, non-sublicensable license to install and use the Software on a single computer for a Evaluation period after downloading for the sole purpose of evaluating the Software. Licensee shall not use the Software in a production environment. Licensee may only use the Software for evaluating the Software solely for Licensee's own internal business purpose in accordance with the Software documentation and other terms and conditions of this Agreement. Licensee's right to use the Software shall lapse at the end of the Evaluation Period and, at that time, the Software will then become inoperable.


(3)  RESTRICTIONS ON USE

Licensee acknowledges that the Software and the structure, organization, and source code thereof constitute
valuable trade secrets of " wintermute.ai UG ".  Accordingly, except as expressly permitted in Section 2 or as otherwise authorized by " wintermute.ai UG " in writing, Licensee will not, and will not permit any third party to:
a) modify, adapt, alter, translate, or create derivative works from the Software;
b) sublicense, lease, rent, loan, sell, distribute, make available or otherwise transfer the Software to any third party;
c) reverse engineer, decompile, disassemble, or otherwise attempt to derive the source code for the Software; or
d) otherwise use or copy the Software except as expressly allowed under Section 2 above.


(4) SOFTWARE EVALUATION

Evaluation versions of the Software is not free software. Subject to the terms of this agreement, Licensee is hereby licensed to use this software products for evaluation purposes without charge for a Evaluation period. Evaluation license which is issued by "wintermute.ai UG" may limit any of Softwares functionality and parameters. Upon expiration of the Evaluation license, the Software must be removed from the computer. " wintermute.ai UG " may issue new Evaluation license on request and at their discretion.


(5) EARLY ACCESS TO SOFTWARE

Early access is access to the beta versions of the Software. Beta software licensed hereunder is believed to contain defects and a primary purpose of this beta software is to obtain feedback on software performance and identification of defects. Licensee is advised to safeguard important data, to use caution and not to rely in any way on the correct functioning or performance of the software and accompanying materials.


(6)  OWNERSHIP

As between the parties, the Software and all modifications and improvements to the Software, and all worldwide intellectual property rights and proprietary rights relating thereto or embodied therein, are the exclusive property of " wintermute.ai UG " and its suppliers.
" wintermute.ai UG " and its suppliers reserve all rights in and to the Software not expressly granted to Customer in Section 2, and no other licenses or rights are granted by implication, estoppel or otherwise.


(7) DELIVERY AND ACCEPTANCE

The Software will be delivered electronically pursuant to " wintermute.ai UG " standards download procedures.
The Evaluation license must be requested by the Licensee after installing the Software and will be delivered electronically to the Licensee. The Software is deemed accepted upon delivery.


(8) IMPLEMENTATION AND SUPPORT

" wintermute.ai UG " is under no obligation to provide technical support under the terms of this license, and provides no assurance that any specific errors or discrepancies in the Software will be corrected; provided, however, " wintermute.ai UG " may, in its sole discretion, provide Licensee with certain support and consultation free of charge to assist in the evaluation activities of Customer under this Agreement. The furnishing of such support or consultation will not subject " wintermute.ai UG " to any liability, whether in contract, tort or otherwise. Licensee is responsible for providing all applicable hardware and any third party software or required installation and configuration services required for the operation of the Software. Any third party software license agreements will be agreed to by Licensee and the applicable third party software vendor.


(9) CONFIDENTIALITY

Licensee agree to hold the Software and the documentation in confidence, disclosing the Software only to authorized users having a need to use the Software as permitted by this EULA and to take all reasonable precautions to prevent disclosure to other parties.


(10) DISCLAIMER OF WARRANTIES

Early access and evaluation Software is provided to the end user without charge for evaluation purpose only, and are therefore being provided "AS IS". " wintermute.ai UG " and its suppliers expressly disclaim any and all other representations and warranties, either express, implied, statutory, or otherwise, with respect thereto, including any implied warranty of merchantability, fitness for a particular purpose, title, non-infringement, or the continuous, uninterrupted, error-free, virus-free, or secure access to or operation of the Software. " wintermute.ai UG " expressly disclaims any warranty as to the accuracy or completeness of any information or data accessed or used in connection with the Software or documentation.
" wintermute.ai UG " is not responsible for any delays, delivery failures, or any other loss or damage resulting from the transfer of data over communications networks and facilities, including the Internet, and Licensee acknowledges that the Software and Documentation may be subject to limitations, delays and other problems inherent in the use of such communications facilities. The Software is not fault-tolerant and not designed or intended for use in hazardous environments, including without limitation, in the operation of aircraft or other modes of human mass transportation, nuclear or chemical facilities, life support systems, implantable medical equipment, motor vehicles or weaponry systems, or any other application in which failure of the software could lead to death or serious bodily injury of a person, or to severe physical or environmental damage (each, a “High Risk Use”). " wintermute.ai UG " expressly disclaims any express or implied warranty or representation of fitness for High Risk Use.


(11) LIMITATION OF LIABILITY

In no event shall " wintermute.ai UG " or its suppliers be liable for:
a) any indirect, incidental, punitive, special or consequential damages, including damages for loss of revenues or profits, loss of use, business interruption, or loss of data. Whether in an action in contract or tort, even if it has been advised of the possibility of such damages or
b) any other damages in excess of one hundered dollars (US $100).

Licensee acknowledges that this provision reflects the agreed upon allocation of risk for this agreement and that " wintermute.ai UG " would not enter into this agreement without these limitations on its liability.


(12) COMPLIANCE WITH LAWS.

Customer shall comply with all laws, regulations, rules, ordinances and orders applicable to its use of the Software.


(13) TERM AND TERMINATION

This Agreement is effective upon acceptance by Licensee and shall continue until expiration of the evaluation period unless sooner terminated by " wintermute.ai UG " for any reason whatsoever. Licensee may terminate this Agreement at any time by destroying the Software and notifying " wintermute.ai UG " at: license@wintermute.ai
If Licensee breaches any provision of this Agreement, this Agreement will automatically terminate.
Upon the expiration of the Evaluation period or any termination of this Agreement, the license granted to Customer will terminate. The provisions of Sections 3 through 14 shall survive termination or expiration of this Agreement for any reason.


(14) GENERAL

YOU ACKNOWLEDGE THAT YOU HAVE READ THIS LICENSE AGREEMENT, UNDERSTAND IT AND AGREE TO BE BOUND BY ITS TERMS AND CONDITIONS. YOU FURTHER AGREE THAT THIS AGREEMENT, TOGETHER WITH ONE OR MORE EVALUATION LICENSE ORDER CONFIRMATIONS FOR THE SOFTWARE, IS THE COMPLETE AND EXCLUSIVE STATEMENT OF THE AGREEMENT BETWEEN YOU AND " wintermute.ai UG " WHICH SUPERSEDES ANY PROPOSAL, OR PRIOR AGREEMENT, ORAL OR WRITTEN, AND ANY OTHER COMMUNICATIONS BETWEEN YOU AND " wintermute.ai UG " RELATING TO THE SUBJECT MATTER HEREOF.