opsschool-curriculum/sysadmin_tools.rst

Tools for productivity
**********************

The work of operations is often helped by using a combination of tools. The
tools you choose to use will depend heavily on your environment and work style.
Over time you may even change the tools you use.

This is a list of commonly used tools, we recommend trying as many of them as
you have time for. Finding the right tool is often like finding a good pair of
gloves - they have to fit just right!


Terminal emulators
==================

OSX: iTerm2
-----------

Rather than stick with the default terminal, many sysadmins on OS X install
iTerm2_. iTerm2's greater flexibility and larger feature set have made it
a de facto standard among technical Mac users. Like most modern
terminals, it supports vertical and horizontal splits, tabs, profiles,
etc. In addition, it stores the rendered buffer for a period of time
which serves as a way to view previous things that might have been lost
due to editor crashes or the like.

.. _iTerm2: http://iterm2.com/

Linux: Gnome Terminator
-----------------------

Many advanced Linux sysadmins prefer tiling window managers such as i3_,
Awesome_, xmonad_, herbstluftwm_, and others. However, these window managers
can be daunting to a new user when compared to a more Windows-like desktop
environment like Unity_, KDE_, or XFCE_. A popular compromise is to use a
tiling terminal emulator, such as Terminator_, within a regular window
manager.

Terminator_ has some features which are convenient for performing similar
commands in several places at once, such as grouping terminals together and
then broadcasting your keystrokes to all terminals in the group.

.. _i3: http://i3wm.org/
.. _Awesome: http://awesome.naquadah.org/
.. _xmonad: http://xmonad.org/
.. _herbstluftwm: http://herbstluftwm.org/
.. _Unity: https://unity.ubuntu.com/
.. _KDE: http://www.kde.org/
.. _XFCE: http://www.xfce.org/
.. _Terminator: http://gnometerminator.blogspot.com/p/introduction.html

SSH
===

Common: ``ssh``
---------------
If your operating system has a good terminal emulator available (e.g., Linux, BSD,
OSX), then the ``ssh`` command line utility is by far the best tool to use for
SSH interactions.

Windows: PuTTY
--------------
If you use Windows as your primary work space, PuTTY is possibly one of the best
SSH clients available.

Android: Connectbot
-------------------
Connectbot is a free app available in the Google Play store, and offers most of
the functionality you would need from an SSH client.

iOS: iSSH
---------
A universal app available on the App Store, iSSH supports most--if not all--of
the features a demanding power user needs.

SSH Use Cases
=============
Far more than just a secure way to access a shell on a remote machine, ssh has
evolved to support a large collection of ways to encapsulate traffic over the
encrypted channel. Each of these is useful in different situations. This
section will present some common problems, and will present how to solve the
problems with ssh. It is important to note that all port forwarding and
proxying occurs over the secure ssh channel. Besides working around firewall
rules, the tunnels also provided a layer of security where there may not
otherwise be one.

Single port on foreign machine firewalled
-----------------------------------------
For this usecase, consider a loadbalancer--``lb-foo-1``--with a Web management
interface listening on port 9090. This interface is only routable to a LAN
private to the datacenter--10.10.10.0/24. Its IP address is 10.10.10.20. There
is a machine on the 10.10.10.0/24 network with ssh access--``jumphost-foo-1``,
accessible via DNS with the LAN IP 10.10.10.19. Therefore, one way to
access ``lb-foo-1`` is via bouncing through ``jumphost-foo-1``. OpenSSH
supplies the ``-L`` option to bind a local port to a port opened on the
other side of the tunnel. On the remote end, ssh opens a connection to
the host and port specified to the ``-L`` option. An example for this
usecase:

``ssh -L 9090:lb-foo-1:9090 jumphost-foo-1``

This will open a connection from ``jumphost-foo-1`` to ``lb-foo-1`` on
port 9090, and will bind the local port 9090 to a tunnel through the
connection to that port. All traffic sent to the local port 9090 will
reach ``lb-foo-1:9090``. It is important to note that the host given to
``-L`` uses the perspective of the machine ssh connects to directly.
This is important to remember for environments using hosts files or
private / split-horizon DNS. In this usecase, an invocation like the
following would work as well:

``ssh -L 9090:10.10.10.20:9090 jumphost-foo-1``

In this case, ``jumphost-foo-1`` would try to connect to ``10.10.10.20``
directly, skipping the DNS lookup.

Tunnel all traffic through remote machine
-----------------------------------------
For this usecase, consider the rest of the machines in the 10.10.10.0/24
network. There are many ways to access them, including VPNs of various
sorts. In addition to creating tunnels to specific ports, OpenSSH can
also create a SOCKS proxy. OpenSSH provides the ``-D`` option for this.
As an example, to bounce from ``jumphost-1`` to the rest of the network:

``ssh -D 9999 jumphost-foo-1``

Once the SOCKS proxy is in place, the operating system or applications
needing access to the proxy need to be configured to use it. This will
vary per-application. Some operating systems (OS X) provide system-wide
proxy configuration.

Reverse Tunneling
------------------

In some situations it may be necessary to forward ports from the remote
machine to the local one. While not as common as other port forwarding
techniques, it is often the only option available in the situations
where it sees use. For this feature, OpenSSH listens to a port on the
remote machine and sends the traffic to the local host and port. The
syntax for the ``-R`` option is the same as for the ``-L`` option
described above. As an example, to have OpenSSH tunnel the remote port
9090 to the local port 8000 on host ``workstation``:

``ssh -R 9090:workstation:8000 jumphost-foo-1``

Then, programs on ``jumphost-foo-1``--or that can access its port
9090--will be able to access the local port 8000 on ``workstation``.

Tunneling stdin and stdout
--------------------------

This method makes using a jumphost transparent. The idea is to tunnel
not a single port, but the entire connection over the ssh channel. This
usage is typically a matter of configuration in the ``ssh_config`` file,
taking advantage of its wildcard-capable ``Host`` directive::

  Host *.lan
  ProxyCommand ssh -W %h:22 jumphost-foo-1

This configuration uses the ProxyCommand feature to cooperate with the
``-W`` flag. Hostnames are resolved from the perspective of the
jumphost. In this example, the machines use an internal pseudo-top level
domain of ``.lan``. To reach, e.g., ``www-1.lan``:

``ssh www-1.lan``

Before doing DNS resolution, OpenSSH will look in its ``ssh_config``
file for Host entries. Therefore, internal DNS on the foreign end is
sufficient.


Multiplexers
============
Operations work regularly involves connecting to remote servers (there will be
times when you do nothing but work on remote servers - parts of this curriculum
were even typed on remote servers rather than contributors desktops and
laptops!).

There are however two limitations to working this way:

#. You'll often need to be connected to more than one remote system at a time.
   Opening a whole new terminal each time can result in a lot of windows cluttering
   up precious screen space.
#. What happens if your internet connection stops working? All of your
   connections are reset. Any work you might have been doing on the remote servers
   can be lost.

Multiplexers are a good solution to this.
They allow you to run multiple "virtual" windows inside a single windows.
For example:

.. epigraph::
   Bob works on 10 remote servers, all of which run Linux.
   Bob's internet connection at work is questionable.
   To work around this, Bob connects to ``server1`` which is at his data centre.
   It is a reliable server which is close to the other servers Bob works on.
   On ``server1``, Bob starts a multiplexer. The multiplexer gives Bob a regular
   looking command prompt, and Bob continues his work.

   If Bob's internet connection drops, he can reconnect to ``server1``, and then
   re-attach to the multiplexer he started previously. His session is in the
   same state he left it before being disconnected, and he can continue his
   work.

   The multiplexer also lets Bob open more than one command prompt and switch
   between them as he needs to. Bob can now connect to many servers and see them
   all in one window.

.. _gnu-screen:

GNU Screen
----------
``screen`` is one of the longest lived multiplexers. Almost everyone who has
used a multiplexer has used screen, and you can't go far wrong with it.

``screen`` is a full-screen window manager that multiplexes a physical terminal
between several processes (typically interactive shells).  It is useful for
creating sessions that can be disconnected from and reconnected to later.  This
is useful for running tasks that can take a long time that you do not want to
have an ssh session timeout on, such as a large database import.  In these cases
cron is also a very good way to run one off long running tasks.

``screen`` is also **very useful** for creating sessions that users can share.

Installation
~~~~~~~~~~~~
Debian and descendants (Ubuntu, Mint, Suse, etc):

.. code-block:: console

  apt-get install screen

On RedHat-style distributions install with the command:

.. code-block:: console

  yum install screen

Basic usage
~~~~~~~~~~~
Create a session:

.. code-block:: console

  screen -S session1

To detach from a session - in the session type Ctrl+a+d

List available screen sessions:

.. code-block:: console

  screen -ls

.. code-block:: console

  [gary@mc9 ~]# screen -ls
  There are screens on:
          21707.session2  (Detached)
          21692.session1  (Detached)
          21936.session3  (Attached)
  3 Sockets in /var/run/screen/S-gary.
  [gary@mc9 ~]#

Here we can see 3 screen sessions are running, 2 detached and 1 attached.

Reattach to a session:

.. code-block:: console

  screen -r session1

Share a session:

User Alice starts session:

.. code-block:: console

  screen -S session1

User Bob can then attach to the same session (both Alice and Bob can send commands to the session):

.. code-block:: console

  sudo screen -x alice/session1

Non root users, must use sudo to attach to another user's session.

Create a session with a log:

.. code-block:: console

  screen -L -S session1

``screen`` will output the session log to the user's home directory with the
file ``~/screenlog.0`` (0 being the session id).  PuTTY is also as a very useful
and featureful ssh client that can be used for logging ssh sessions locally
(Windows and Linux).  ``screen`` can be used within a PuTTY session.

Create a session with a log and 20000 lines of scrollback in the terminal:

.. code-block:: console

  screen -h 20000 -L -S session1


Configuration
~~~~~~~~~~~~~
``screen`` has a fairly extensive set of configuration options, when screen is invoked, it executes initialization commands from the files ``/etc/screenrc`` and ``.screenrc`` in the user's home directory.

Further info
~~~~~~~~~~~~

.. code-block:: console

  man screen

There is a nifty cheat sheet for the most important ``screen`` and ``tmux`` keybindings (see below in tmux references [3]_).

.. _tmux:

Tmux
----
``tmux`` [#]_  is relatively new compared to
``screen``. It covers the same basic feature set and has added a few
more advanced features. It is recommended you get comfortable with
``screen`` first before attempting to use ``tmux``.

In this chapter you will learn to start a tmux session, get to know a
few first keyboard shortcuts and detach from and re-attach to the
session.

Installation
~~~~~~~~~~~~
tmux is available on Debian and its descendants like Ubuntu or Mint
with the command:

.. code-block:: console

  apt-get install tmux

On RedHat-style distributions you will have to use the :term:`EPEL` repository to
get a pre-built package, and install with the command:

.. code-block:: console

  yum install tmux

On MacOS you can use Homebrew to install via:

.. code-block:: console

  brew install tmux

tmux basics
~~~~~~~~~~~
``tmux`` is usually started with the command ``tmux`` in a
terminal window. Depending of your version of tmux you will see either
a line at the bottom of the screen or nothing at all. ``tmux`` is
controlled with keyboard shortcuts, the default shortcut usually is
``ctrl-b``. If you press ``ctrl-b`` and then a ``t`` in the newly
started tmux window you should see the local time displayed as a large
digital clock. If you hit ``ctrl-b`` and ``c`` you should see a new
empty window with an empty input prompt.

If you want to detach from the session you have to hit ``ctrl-b`` and
``d``. The ``tmux`` window will disappear and you will see a message
``[detached]`` in your terminal window. All the shells and processes
you started onside the ``tmux`` session continue to run, you can see
this with a simple

.. code-block:: console

  ps -ef | grep tmux

You should see something like the following:

.. code-block:: console

  cdrexler 13751     1  0 Nov30 ?        00:00:41 tmux

You will notice that the ``tmux`` process has a parent process id of 1
which means that it is not a child process of the shell you started it
in anymore. Accordingly you can leave your working shell, start a new
one and attach to the running tmux process again which is very handy
if your connectivity is flaky or you have to work from different
locations. If you check the process table for the process id of the
tmux process

.. code-block:: console

  ps -ef | grep 13751

you will find that is the parent process of the two shells you created
in the beginning of the chapter:

.. code-block:: console

   cdrexler  4525 13751  0 17:54 pts/2    00:00:00 -zsh
   cdrexler  4533 13751  0 17:54 pts/5    00:00:00 -zsh

If you want to get an overview of the running tmux processes on your
system you can use the command

.. code-block:: console

  tmux ls

It will list all available ``tmux`` sessions on your system [#]_. If there
is only one you can attach to it with the command:

.. code-block:: console

  tmux att

If there is more than one session the output of ``tmux ls`` will look like this:

.. code-block:: console

   0: 3 windows (created Fri Nov 30 18:32:37 2012) [80x38]
   4: 1 windows (created Sun Dec  2 17:44:15 2012) [150x39] (attached)

You will then have to select the right session with the ``-t`` command line switch:

.. code-block:: console

  tmux att -t 4

``tmux`` runs as a server process that can handle several sessions so
you should only see one tmux process per user per system.

You should see the original session with the two shells again after
running this command.

tmux configuration
~~~~~~~~~~~~~~~~~~~
``tmux`` is configured via a
config file which is usually called :file:`.tmux.conf` that should live in
your ``$HOME`` directory.

A typical :file:`.tmux.conf` looks like this:

.. code-block:: ini

   #set keyboard shortcut to ctrl-g
   unbind C-b
   set -g prefix C-g
   bind C-g send-prefix
   bind g send-prefix
   #end of keybord shortcut setting
   # Highlight active window
   set-window-option -g window-status-current-bg red
   # Set window notifications
   setw -g monitor-activity on
   set -g visual-activity on
   #automatically rename windows according to the running program
   setw -g automatic-rename
   #set scroll back buffer
   set -g history-limit 10000
   set -g default-terminal "xterm-256color"
   set -g base-index 1
   set -g status-left '#[fg=green]#H

This illustrates a method to change the default keybinding and some
useful settings.

Please note that you can force ``tmux`` to use another configfile with
the ``-f`` command line switch like so:

.. code-block:: console

  tmux -f mytmuxconf.conf

There is a nifty cheat sheet [#]_ for the most important
``screen`` and ``tmux`` keybindings or even a whole book about tmux [#]_.



byobu
-----
[Byobu]_ is a wrapper around one of screen or tmux. It
provides profile support, F-keybindings, configuration utilities and a
system status notification bar for most Linux, BSD or Mac operating systems.

Byobu is available in major distros as a packaged binary. Launching byobu will
run whichever the package maintainer included as a dependency, if you have both
installed, you can select explicitly with byobu-{screen,tmux}.  Basic
configuration is launched with F9 or byobu-config.

Scrollback starts with F7, mark the start of your buffer by hitting the
spacebar, then use the spacebar again to mark the end of your selection (which
will be copied into byobu's clipboard automatically). Press Byobu-Escape
(typically Ctrl-A) + [ to paste.

References
----------
.. [#] http://tmux.sourceforge.net/
.. [#] Please note that ``tmux ls`` will *only* list tmux sessions that belong to your userid!
.. [#] http://www.dayid.org/os/notes/tm.html
.. [#] http://pragprog.com/book/bhtmux/tmux
.. [#] https://launchpad.net/byobu
.. [Byobu] http://byobu.co/

Shell customisations
====================

As you read in :doc:`shells_101`, your shell is your primary tool during the
work day. It's also incredibly customisable to suit your needs. Let's look at
some changes you can make.

How to customise your shell
---------------------------

Your shell's configuration is stored in its ``rc`` file. For bash, this file is
``~/.bashrc``. Each time you edit this, you can reload the configuration by
typing:

.. epigraph::
   ``source ~/.bashrc``

Changing your prompt
--------------------

Your default prompt probably looks something like this:

.. epigraph::
   ``bash-3.2$``

That's pretty plain and doesn't tell you much. In fact, all it does tell you is
that you're using Bash version 3.2, and that you are not the root user (the
``$`` at the end signifies a regular user, whereas if you were root, you would
see a ``#`` instead).

Let's change this up a little. Edit your ``~/.bashrc`` file, and add this line
to the end:

.. epigraph::
   ``PS1="\u@\h \w> "``

Save, quit, and then reload your ``.bashrc`` file. Your prompt should change to
something like this:

.. epigraph::
   ``avleen@laptop ~>``

Much better! Now your know your username, the name of the machine you're on (in
this case "``laptop``"), and the directory you're in ("``~``" is your home
directory).

The ``PS1`` variable has a lot of different options you can use to customise it
further.


Mosh
====

Mosh (MObile SHell) is an alternative to remote shell commands, such as ``ssh``
or ``rsh``. The beauty of the Mosh protocol is that it supports intermittent
connectivity without losing the remote session.

You can start a ``mosh`` session just like you would with ``ssh`` on one side
of town with one IP address, shut your laptop and go home, then open your
laptop and connect back to your Mosh session like it was never interrupted.
Also, if your wifi is spotty or internet connection is intermittent, ``mosh``
doesn't break the session when the connection drops out, unlike ``ssh``.
Mosh does not wait for the remote server to confirm each keystroke before
displaying it to a terminal. Instead, it displays the typed characters locally
and confirms entry on the remote end. There are packages available for GNU/Linux,
FreeBSD, Solaris, Mac OS X, Chrome and even Android apps.

Mosh must be installed on both the client and remote server. When a session
is started it spins up a mosh-server and a local mosh-client process. It can
be installed in a home directory without privileged access. Mosh respects
your current ``~/.ssh/config`` so migrating from ``ssh`` to ``mosh`` is
relatively seamless.

SSH connections work by sending a stream of data back and forth between the client
and server. This stream can be broken in various ways, such as the connection
timing out due to inactivity, or the client machine suspending state and shutting
down network devices. The Mosh protocol is based on UDP packets compared to the SSH
protocol that uses TCP packets. Mosh is the first application to use the Stateless
Syncronization Protocol. Instead of a stream of data between the server and client
that makes up a session over SSH, Mosh works by keeping a mirrored copy of the
session on the client and server and syncronizing the changes between them.

While ``ssh`` has been around long enough to have time tested security,
``mosh`` is relatively new and has not been through the same extensive testing.
It is the first application to use the SSP protocol. Mosh does tunnel traffic
encrypted with AES-128 in OES mode, however Mosh hasn't been under the
security spotlight as long as SSH has.

Examples
--------

Mosh works just like ssh:

.. code-block:: console

  mosh username@remoteserver.org

You can also have mosh utilize ssh style commands such as: This specifies the
private key to use to authenticate with the remove server.

.. code-block:: console

  mosh username@remoteserver.org --ssh="ssh -i ~/.ssh/identity_file"

This tells mosh to connect via the ssh port 1234 on the remote server, where
ssh normally runs on port 22.

.. code-block:: console

  mosh username@remoteserver.org --ssh="ssh -p 1234"``

References
----------

.. http://mosh.mit.edu


Ticketing systems
=================


Note-taking
===========

Wiki
----

EverNote
--------

OneNote
-------