Overview

This demo sets up 2 VMs in Google Compute Engine and installs an ELK stack spread over both machines. From one host logstash reaches towards twitter API, collects tweets with a certain buzzword and feeds this into Elasticsearch running on the other node. Kibana is used as front end to analyse the stored tweets.

The demo is shown out of an Ansible Tower environment. All playbooks reside in a GitHub repository (called elk-demo), all credentials (to connect to twitter or GCE, …) are stored as credentials within tower. The different playbooks are put into a workflow, which would also delete the GCE VMs in case of an error. This workflow also offers a survey asking for the buzzword right at the beginning.

While the main goal is to showcase the use of Ansible Tower to set up some IT Service Stack starting at Compute level right up to the Application, we will also connect to kibana (manually) and assure to see tweets and be able to analyse the same. This is mainly to proof the setup is working.

As i want this demo to be easily reproduced, i put some effort in an automated setup and mentioning all prerequisites which will be needed. You could start from scratch by following the tips within the prerequisite paragraphs. E.g. To obtain Ansible Tower as the key tool of this demo, you could just setup a Vagrant Box up with everything needed as explained here [1].

Set up the Demo

Prerequisites

It is essential to have a functional Tower installation and credentials to reach out to the different environments and services needed for the setup aimed for.

Tower installation and first step Tower preparation

We need a Tower installation and two git repositories synced into it. A third, private git-repository needs to be created to hold your personal credentials.

  • To set up a Tower installation on your personal Notebook, it’s easiest to use vagrant-tower, as detailed here [1]. This creates a VM with full blown Ansible Tower installation.
  • You will need to install quite some packages and modules into the vagrant tower file. This is defined within a Jupyter Notebook, found here [2]
    Hint: This is a Jupyter notebook making execution of the commands much easier, but you can copy & paste the commands into command line as well if you do not have a Jupyter App running.
  • With vagrant ssh you can ssh into the before created Tower VM.
  • within vagrant-tower i clone the git repositories:

[vagrant@ansible-tower ~]$ mkdir projects
[vagrant@ansible-tower ~]$ cd projects
[vagrant@ansible-tower projects]$ git clone https://github.com/mschreie/elk-demo
[vagrant@ansible-tower projects]$ git clone https://github.com/mschreie/elk-demo-setup

Explanation: The elk-demo-setup project is run from CMD-Line and configures everything within Ansible Tower so that the Demo can be executed. This project is NEEDED to be synced to some place to be executed manual.
The elk-demo repository will be used inside Ansible Tower. Syncing this at the cmd-line as well is just for reference. You are able to view the files via cmd-line tools more easily.

Credentials Repository

All credentials are stored in vaulted files within a third repository, called elk-demo-credentials. As we store personal credentials to access certain services, this repo needs to be a separate one for each user of this demo. Therefore the repo is to be created within your personal git account and synced next to the elk-demo and elk-demo-setup repositories.
Symlinks reach out of elk-demo and elk-demo-setup. For the sym-links to work it is essential, that the repository, the directory structure and the file names are set up correctly. We will create the repository first: 

cd ~/projects
mkdir elk-demo-credentials
cd elk-demo-credentials
cat <<EOF > README.md
This repository is a private repository consitsting credential-files for elk-demo or elk-demo-setup projects. Out of these projects i've created symlinks, so please 
do not
- change repository name (as this would also chane the directory name
- alter directory names
- file names
Each individual user would need the exact same repositry underneath his github account and sync it next to the elk-demo projects. Symlinks should then point to your credential files witouht any further alterations.
EOF

mkdir elk-demo
git init
git add README.md
git commit -m "Initializing Credentials Repo - yet without credentials"
git remote add origin git@github.com:<gituser>/elk-demo-credentials.git
curl -u '<gituser>' https://api.github.com/user/repos -d '{"name":"elk-demo-credentials", "private": true}'
git push -u origin master

Please assure within the GitHub webfrontend, that this repository is setup correctly and marked private.

We will create the files within this repository during the “preparation” section of the blog.

To create a vaulted file i’d recommend the following process:

  1. create a file with the correct name and the correct content
    like: vi <filename>
    You will find example files alongside each vaulted file for reference of exact file layout. The values to add should be gathered during the setup of all prerequisites.
  2. run ansible vault encrypt <filename>
    You will be asked for a vault key twice. This vault key is needed each time the file should be opened.

After any change you can commit and upload to your central repository:

git add --all .
git commit -m "some meaningful comment explaining the change"
git push

At the end of the preparation you should have the following files within the projects/elk-demo-credentials/elk-demo directory:

credentials.yml
elk-demo-gce_service_account.json
ssh_github_key
ssh_github_key.pub
sshkey_elk-demo
sshkey_elk-demo.pub

credentials.yml and elk-demo-gce_service_account.json files are vaulted. The private ssh-keys are protected with a pass phrase. The credential files are used during the setup and get injected as credentials into Ansible Tower. Example files are at each tech place in clear text.
These files need to be created / adopted with the credentials you will gather within the further prerequisites sections. 

Hint: Within the elk-demo files you will also find credentials for mail-sending and ford dns adjustment. Both is not part of this setup as of today. So don’t bother to adjust these credentials.

Start preparation of credentials.yml

We put some infos into the credential.yml to start with. The credentials to connect to the Tower Server:

vi elk-demo-credentials/elk-demo/credentials.yml
tower_user: admin
tower_password: xxxxx
tower_host: 1.2.3.4
tower_base_url: https://{{ tower_host }}/api/v2/

We also need to define the elk-demo repository:

vi elk-demo-credentials/elk-demo/credentials.yml
scm_url: https://github.com/mschreie/elk-demo
scm_branch: master

And in case you need a vaulted file within the demo, we also defined the vault password:

cat elk-demo-credentials/elk-demo/credentials.yml
vault_key: xxxxxxx

ssh key pair to acces VMs

i’ve created a (local) ssh-key-pair for this demo. The public key will be configured within the GCE project to be injected into each VM created. The “comment” added via -C option of ssh-keygen is essential. This comment “cloud-user” defines the username on the VM.

mschreie-mac:elk-demo mschreie$ ssh-keygen -C "cloud-user" -f sshkey_elk-demo

I’ve also added a pass-phrase to this ssh-key. Within a vaulted file yet again you will describe the location of the private key file and the passphrase. You also need to know both things, if you want to ssh to a VM manually.

Both key files go to elk-demo-credentials/elk-demo/sshkey_elk-demo
and elk-demo-credentials/elk-demo/sshkey_elk-demo.pub 

mschreie-mac:elk-demo mschreie$ mv sshkey_elk-demo* elk-demo-credentials/elk-demo/

The values are put into the file: credentials.yml

vi elk-demo-credentials/elk-demo/credentials.yml
vms_user: username set within the ssh_key_file
vms_ssh_keyfile: ssh_keyfile_with_passphrase
vms_ssh_keyfile_passphrase: xxxxxx

Twitter Developer Account

The Twitter Developer Account is (as far as i see) cost free, but needs to be applied for. Please assure to read and understand and accept the terms of use. Compliance with any regulations is in your own responsibility.

Follow these steps to achieve a twitter developer account:

  • Navigate to the developer Portal [3] and click “Apply for an account” and answer the form. Assure email-address and phone number are correct.
  • Within the confirmation email you find a link to further webpage where you need to create an app key.
  • On https://dev.twitter.com/apps you will also need to create an access token.

The outcome of this exercise should give you the following information, again put into the credentials file:

vi elk-demo-credentials/elk-demo/credentials.yml
app_key:     XxxXXxXXxXXxXXxXXxXXxXXxX
app_secret:  xxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxx
token_key:   xxXXxxXXxx-XXxxXXxxXXxXxXxXXXXxXXXXXXxxXxxXxXxXxXx
token_secret: XxXXXXxXXXXxxXXXxXXXxxxXXXxxXXXxxXXXxXXXxxxxX

# API key: XxxXXxXXxXXxXXxXXxXXxXXxX
# API key secret: xxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxx
# Bearer Token: XXXXXXXXXXXXXXXXXXXXXxxXXxXXXXXXXXXXXXXXXXxXXxxxXxXXxxxXXxxXXXXXXXXxXxxxXxXxxXXXXXxXXXXXxxXXXXXxXXXXxxXXXXXXxXXX
# Access token: xxXXxxXXxx-XXxxXXxxXXxXxXxXXXXxXXXXXXxxXxxXxXxXxXx
# Access token secret: XxXXXXxXXXXxxXXXxXXXxxxXXXxxXXXxxXXXxXXXxxxxX

GCE account

I’ve create a private GCE account at [4]. Google offers a free trial for the first 90 days. Please assure to read and understand and accept the terms of use. Compliance with any regulations is in your own responsibility.

  • Navigate to [4] and click “Try it free”. This brings you to a 2 paged registration from. You need to enter email-address, name and also credit card or bank account information. From what i gathered, this will not be charges during the free trial. A switch to charged services will not happen without explicit action from users side. BUT: Please assure to read and understand and accept the terms. Compliance with any regulations or ordering free or chargeable services is at your own risk and in your own responsibility.
  • if i recall correctly within the console you still needed to activate the free trial.

A dedicated project within the GCE account

Within this GCE account i created a project called “elk-demo”.

  • Navigate to googles cloud console [5] and assure you are logged in with the correct user (upper right corner).
  • On the right of the headline “Google Cloud Platform” you find a drop down field with “My Fist Project” or “Select a project”. Selecting the drop down opens a new window with a button “New Project”.
  • Name the project “elk.-demo”. Each project has a Project name, a project id and a project number, similar to this:

Project name:  elk-demo 
Project ID:  elk-demo-123456
Project number: 999999999999

You will only need the Project ID. Be aware the project ID has a 6 digit number attached like “elk-demo-123456”

For this project i created a service account, which enables automated access for the Ansible Playbooks.

  • Within googles cloud console [5] assure you are logged in with the correct user and have “elk-demo” selected as project.
  • Follow the instructions from “Creating a service account” documentation [6]. You don’t need to specify any “Service account permissions” nor need to “Grant users access to this service account”.
  • Stepping through this procedure downloads a json file with the credentials (including the private key file), similar to this:

{
"type": "service_account",
  "project_id": "elk-demo-123456",
"private_key_id": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
  "private_key": "-----BEGIN PRIVATE KEY-----\nAAAAAAAAAAAAAAAAAA....",
  "client_email": "999999999999-compute@developer.gserviceaccount.com",
  "client_id": "111111111111111111111",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/999999999999-compute%40developer.gserviceaccount.com"
}

I’ve renamed the json-file and moved it into the credentials repository

mv elk-demo*.json ~/projects/elk-demo-credentials/elk-demo/elk-demo-gce_service_account.json 

and i’ve pointed the according parameter within the credential.yml to the file just downloaded:

vi elk-demo-credentials/elk-demo/credentials.yml
gce_credential_file: files/elk-demo-gce_service_account.json   ## the downloaded json-file vaulted via ansible-vault

I’ve also uploaded the public ssh-key file, which enables ssh-access to any VM i great within that project.

  • Within googles cloud console [5] assure you are logged in with the correct user and have “elk-demo” selected as project.
  • Navigate to Compute Engine -> Metadata
  • select the “SSH keys” tab and click “Edit”, then “+ Add item”
  • Paste the public key as fount in sshkey_elk-demo.pub into the field and press “Save”.
  • The key should show up in the list of ssh keys and should name “cloud-user” as username.
    Every new VM created in this GCE project will now have a user “cloud-user” with the ssh key added to the auhtorized_keys file. You should be able to connect to a created VM e.g. like this

ssh -i sshkey_elk-demo cloud-user@1.2.3.4

credentials.yml as a whole

We have all the external accounts set up correctly and all credentials stored where they should be. For all “vaulted” files i use the same vault password so that Ansible is able to open all files by asking for just one password.

For your reference please find the complete credentials.yml file next: 

cat ansible/group_vars/all/credentials.yml_example
tower_user: admin
tower_password: xxxxx
tower_host: 1.2.3.4
tower_base_url: https://{{ tower_host }}/api/v2/
gce_credential_file: downloaded_gce_json_file # hopefully this can be vaulted as well...

git_user: yourgithubaccount
git_ssh_keyfile: ssh_keyfile_with_passphrase
git_ssh_keyfile_passphrase: xxxxxxx

vms_user: username set within the ssh_key_file
vms_ssh_keyfile: ssh_keyfile_with_passphrase
vms_ssh_keyfile_passphrase: xxxxxx
scm_url: https://github.com/mschreie/elk-demo
scm_branch: master

# in the elk-demo you will put a vaulted file vars/nginx_usercred.yml
# the vault secret is put here. It will be stored in Ansible Tower and used to read the vaulted file on
# playbook execution
vault_key: xxxxxxx

app_key: XxxXXxXXxXXxXXxXXxXXxXXxX
app_secret: xxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxx
token_key: xxXXxxXXxx-XXxxXXxxXXxXxXxXXXXxXXXXXXxxXxxXxXxXxXx
token_secret: XxXXXXxXXXXxxXXXxXXXxxxXXXxxXXXxxXXXxXXXxxxxX

### twitter naming:
 # API key: XxxXXxXXxXXxXXxXXxXXxXXxX
 # API key secret: xxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxxXXxx
 # Bearer Token:XXXXXXXXXXXXXXXXXXXXXxxXXxXXXXXXXXXXXXXXXXxXXxxxXxXXxxxXXxxXXXXXXXXxXxxxXxXxxXXXXXxXXXXXxxXXXXXxXXXXxxXXXXXXxXXX
 # Access token: xxXXxxXXxx-XXxxXXxxXXxXxXxXXXXxXXXXXXxxXxxXxXxXxXx
 # Access token secret: XxXXXXxXXXXxxXXXxXXXxxxXXXxxXXXxxXXXxXXXxxxxX

Set up of Ansible Tower to actually set up the demo

As we set up the demo automagically we only need to run some ansible playbooks:

  • With vagrant ssh you can ssh into the before created Tower VM.
  • cd into the ansible directory of the setup project:
    cd project/elk-demo-setup/ansible
  • And start running one playbook afer the other in the given order. Asure you get asked for the vault-key:

ansible-playbook --ask-vault-pass 020_setup-credentials.yml
ansible-playbook --ask-vault-pass 030_setup-project.yml
ansible-playbook --ask-vault-pass 040_setup-jobs.yml

For each cmd you will be asked to enter the vault password.

Hint: If the very first playbook comes up with an error similar to..

ERROR! couldn't resolve module/action 'awx.awx.tower_credential_type'. This often indicates a misspelling, missing collection, or incorrect module path.

…you most likely miss the installation of libraries and modules, as explained here [2].
Hint: This is a Jupyter notebook making execution of the commands much easier, but you can copy & paste the commands into command line as well if you do not have a Jupyter App running.

Checking the setup

With your favourite web browser navigated to the tower https://10.42.0.42/ (as given in the output of vagrant ssh, further above) after logging in with the credentials provided through “vagrant ssh“.
Check to have following credentials:

  • “Twitter API Credential”
  • “GCE Service Account for elk-demo”
  • GitHub Credential
  • VMs SSH Credential
  • Vault Key

Check to have

  • a project “elk-demo” and assure sync of that project works well.
  • an inventory “elk-demo Inventory” and assure sync of the corresponding inventory source works. It is fine to see no hosts within that inventory (as we did not create any yet)

Check to have following Job Templates (within the menu item):

  • delete GCE VMs
  • deploy ELK stack
  • wait VMs
  • create GCE VMs

In the same menu there should also be the “elk-demo Workflow Template” workflow template.

Running the Demo

Execution

To run the demo you just execute the “elk-demo Workflow Template” workflow template. This should ask you for a key word. “Trump” or “COVID” works very well at the moment and for username and password.
While the Workflow takes a time you can investigate on the IP-Adress of a host called “elastic”.

  • Wait for the job “create GCE VMs” to finish and the next one to start. The start of the next jobs should trigger an update of the inventory.
  • Navigate to the “elk-demo Inventory” and click on “Hosts” locate the “elastic” line and find an non-private IP-Adress (most likely one staring with 34 or 35). If needed press “VIEW MORE” in the related groups column.
  • point a new browser window to that IP-Address (as http://<IP-Address&gt;)

You will need to wait a little before the connection works till all jobs of the workflow have run through.
HINT: The workflow has some “exception handling” in place. A job, which finished errornouse triggers job which deletes the VMs again. This delete job will (hopefully) finish successful rendering the status of the workflow to be successful as well. So check wether each job within the workflow finished successful.

  • Login with the credentials you defined within the survey file.
  • This should give you access to elastic search Web-UI.

You will need to do some rudimentary adjustments:

A “Let’S get started” screen appears.
Click “Explore on my own”.
Click on “Discover” within the “Visualize and Explore Data”. This shows up the “index patterns” screen.
Click “+ Create index pattern”

You need to create index pattern within kibana:

  • name it “twitter_elastic__example*” (you should see twitter_elastic__example as one line item in a list of indexes further down.
  • In the “Time field” drop down box choose “@timestamp” – which should be available, if everything went correctly.
  • click “create index pattern”
  • Move over to “Discover” by selecting the menu (three parallel horizontal bars”) in the upper left corner, then “Discover” from that menu
  • this should display some tweets already
  • On the left hand side choose the fields “user.name” and “text” to get the two relevant fields presented.
  • change refresh rate to “every 30 seconds” in the upper left box.

Now you should be able to do some analyses.

Explanations

While the run-through is great you need to point out some stuff to make this relevant for the audience. Possible topics / facts to point out

  • Credentials are put into Tower. They can be used to e.g. reach out to GCE, to connect to the VM and gain root access. AT the same time the person executing the workflow can be locked away from these credentials not giving him the chance to gain root-privilege himself or create his own GCE VMs.
  • The Twitter API Credential is based on a self defined “custom credential”, where we defined the attributes needed to be stored.
  • The playbooks are stored in GitHub, which adds the state of the art version control to your code.
  • The elk-demo Inventory source is a dynamic inventory. Interesting here is the “update on launch” option, which assures that the inventory is updated before a job, which uses this inventory uses it. As the update is again an Ansible playbook, we can define the verbosity level here as well.
  • The elk-demo project, defines the git- repository where the playbooks are found. It uses the credentials to connect to git.
  • The “create GCE VMs” execute the loop.yml playbook, found in the elk-project. It uses the gce credentials as it needs to create VMs there.
  • We could look at the loop.yml at the cmd-line:
    Within vagrant box look cd to project/elk-demo/ansible
    We find the group_vars/all/all.yml file which is a file pulled in for all hosts belonging to the “all” group, which is all. This file defines a list “gce_machines” with two names “elastic” and “logstash”.
    loop.yml playbook defines one play running on localhost (so on the ansible server itself). It includes to run gce_create in a loop looping through the two names of the gce_machines list.
    include is one way to call task defined elsewhere. Roles
    gce_create.yml utilised gce_* module. The huge and ever growing number of dedicated modules reaching to all sorts of devices is one success factor of Ansible. The gce credentials are taken from the Tower credential as seen in the defenition of the job.
  • The Workflows within Tower allows to combine different Playbooks into one flow. Simple Error-Handling can be introduced. A Survey can be added to each job. You can also run jobs parallel and continue whit the job there after when one / all jobs before finished (successful).

Conclusion

Demoing the power of Ansible Tower can easily be done through this demo. The playbook scripts allow loads of discussions on playbook creation, possible setups and best practices. The Setup with credentials, a git repository XXX the playbooks and a workflow with three different steps and some error handling again allow a lot of explanation.

If this is not enough you could also point out the 100% Ansible automated setup of this whole demo. While the initial creation of these automation scripts did take longer than clicking in the web-us, the reproducibility is now tremendous.

Links / Ressources

[1] https://mdschreier.com/2020/07/22/automated-setup-of-ansible-tower-for-personal-use/

[2] https://github.com/mschreie/jupyter/blob/master/Setup%20vagrant%20tower%20for%20ELK%20Demo.ipynb

[3] https://developer.twitter.com/content/developer-twitter/en/docs/basics/developer-portal/overview

[4] https://cloud.google.com/compute

[5] https://console.cloud.google.com/

[6] https://developers.google.com/identity/protocols/oauth2/service-account#creatinganaccount