Overview

I like the idea to use Jupyter Notebooks to combine documentation and code, while the code can be directly executed from where it is (without cut&paste). Jupyter notebooks are primarily invented to run Julia, Python, and R, which also lead to this name. They are widely used by data scientists but i’m exploring how Jupiter Notebooks can be beneficial for bash and Ansible related automation.
For an introduction to Jupyter Notebooks i’d recommend [1]

A use case for the use of Jupyter Notebooks could be to set up a local Ansible Tower installation and configure it from scratch. So you need to start at a point, where Ansible Playbooks are not working yet. A different idea could be to write a blog and offer easy-to-consume coding. Hiding the cmd-line steps into Ansible code might be inappropriate or might alter the message a blogger wants to get across.

Solution

Before looking into how to install, we need to know what to install. And for that i dug into, how the installed setup enables execution of shell commands or scripts. So let’s first look on how shell code can be executed within Jupyter notebooks.

Execute shell code in a jupyter notebook

Jupiter Notebooks default to execute python scripts. For ease of use there are several ways you can run bash or bash-similar commands within this environment.

a) you can use use “magics” which is nicely explained here [2]. Adding %%bash in front of each code cell is somehow helpful. Unfortunately this starts a sub-process, so any change (e.g variable definition or change of current working directory) done gets lost at the end of the cell.

b) you have python substitutes for some common shell commands (like %cd or %cp, ..) these work without sub-process and therefor the change is persistent throughout the whole notebook and across the different cells of that Jupyter notebook. Unfortunately these substitutes are limited. Some explanation can be found here [3].

c) for shell based workflow you need to have a bash-kernel based Jupyter notebook. For this to be available install the bash kernel (as pointed out in the next paragraph) and start a bash based Jupyter notebook.

Install jupyter notebook with bash kernel on a mac

As of today i’m using a Mac and focus to get things done from there. I default to use brew, if at all possible and yes you can brew install jupyter to install jupyter notebook environment. This is nicely demonstrated here [4] . This works for python based notebooks but i did not get any further. Therefor i brew uninstall jupyter again and went for:

Markuss-MacBook-Pro:jupyter mschreie$ pip install jupyter
Markuss-MacBook-Pro:jupyter mschreie$ pip install bash_kernel
Markuss-MacBook-Pro:jupyter mschreie$ python -m bash_kernel.install

Start and Use jupyter

The jupyter notebook kernel is started on the command line and finds it’s notebooks in the current working directory. I’ve created a “./projects/jupyter/” directory, where the notebooks can reside. I sync and version this directory via git.

To start the environment, i cd into the directory mentioned and execute jupyter notebook from there. Every new notebook gets created in cwd and every notebook in that directory gets found through the web-UI.

mschreie-mac:~ mschreie$ cd ~/projects/jupyter
mschreie-mac:jupyter mschreie$  mschreie$ jupyter notebook

This opens a URL with your default web browser, which does not work in my case (Safari). It also prints out the url in cmd line window. Opening the given URL works great in chrome, where i open it via cut & past. You can then create a new notebook, bash or python based.

Create NEW Bash-based Jupiter Notebook

Within the notebook you create cells, which are “code” cells or “markdown” cells. Code cells can be executed markdown is for explanatory text.

being critical

entering parameters

I have found no good way to make Jupyter-Notebooks interactive. So any parameter which needs to be adjusted, i put into the first cmd-cell of the notebook as a varable defenition. Every user can adjust this to her needs and execute the notebook. This has limitations on passwords, as the risk of exposing passwords like this is high. I have no solution as of today.

diff of new version

As the Notebook includes the output of the commands directly into the Notebook, version control has issues to generate a useful diff. My approach is to clear all output before uploading the notebook into a git repository. There are other ideas around this as well [5,6], but nothing i looked into yet.

Conclusion

The Jupyter notebooks are a great way to combine code and documentation in one place. It is handy to avoid cut&paste, but might have some limitations to work arround as well. CMD-Line heros should give it a chance. Syncing with GitHub in are more sophisticated way might be a separate topic.

Footnotes

[1] Usage introduction into Jupyter Notebooks:
https://realpython.com/jupyter-notebook-introduction/

[2] Magic keys to execute code of different languages within python:
https://realpython.com/jupyter-notebook-introduction/

[3] Shell related Magic commands:
https://jakevdp.github.io/PythonDataScienceHandbook/01.05-ipython-and-shell-commands.html

[4] Installation via brew:
https://medium.com/@blessedmarcel1/how-to-install-jupyter-notebook-on-mac-using-homebrew-528c39fd530f

[5] Integration of GitHub and Jupyter Notebooks: https://towardsdatascience.com/version-control-with-jupyter-notebooks-f096f4d7035a

[6] broader overview on github and Jupyter integration:
https://nextjournal.com/schmudde/how-to-version-control-jupyter