Why must 'self' be used explicitly in method definitions and calls?

Sequence Types --- str, unicode, list, tuple, bytearray, buffer, xrange

Put an example image here and finish this tutorial to be like hello_embedded

Gate One includes an authentication API that can be used when embedded into other applications.  It allows the application embedding Gate One to pre-authenticate users so they won't have to re-authenticate when their browser connects to the Gate One server.  Here's how it works:

The secret is not output to the terminal to avoid it being captured in session logs.

You'll need to have a look at your server.conf to see what the 'secret' is:

In the above example our API key would be, ``"NDEzMWEwYTdlZTAzNDkxMWIwMDI4YzJmZTk4YzI4OWJjM"`` and our API secret would be, ``"M2U5YTMxMGQ3OWNlNDJlMTg5NmY0NmUyOTk5MWYwYWFiN"``.

You can set the API Key and secret to whatever you like by editing your server.conf.  By default they're random, 45-character strings but they can be any combination of characters other than colons and commas--even `Unicode <http://en.wikipedia.org/wiki/Unicode>`_!.  The following is a perfectly valid API key and secret:

``ʕ•ᴥ•ʔ ／人 ◕ ‿‿ ◕ 人＼:↑ ↑ ↓ ↓ ← → ← → Ⓑ Ⓐ ♥‿♥``

You can set the API Key and secret to whatever you like by editing your server.conf.  By default they're random, 45-character strings but they can be any combination of characters other than colons and commas--even 

To use Unicode strings in your server.conf you must have an encoding marked at the top of the file like so::

# -*- coding: utf-8 -*-

To use Unicode strings in your server.conf you must have an encoding marked at the top of the file like so:

The next step is to generate a `JSON <http://en.wikipedia.org/wiki/JSON>`_ object (auth) from your application and pass it to :js:func:`GateOne.init`.  The 'auth' object must contain the following information:

api_key
The key that was generated when you ran ``./gateone.py --new_api_key``

upn
The username or userPrincipalName (aka UPN) of the user you wish to preauthenticate.

timestamp
A JavaScript-style timestamp:  13 characters representing the amount of seconds since the epoch (January 1, 1970)

signature
A valid `HMAC <http://en.wikipedia.org/wiki/Hash-based_message_authentication_code>`_ signature that is generated from the api_key, upn, and timestamp (in that order).

 signature that is generated from the api_key, upn, and timestamp (in that order).

signature_method
The HMAC signature method that was used to sign the authentication object.  Currently, only HMAC-SHA1 is supported.

api_version
The version of Gate One's API authentication to use.  Currently, only '1.0' is valid.

authobj = {
    'api_key': 'MjkwYzc3MDI2MjhhNGZkNDg1MjJkODgyYjBmN2MyMTM4M',
    'upn': 'joe@company.com',
    'timestamp': '1323391717238',
    'signature': "f6c6c82281f8d56797599aeee01a5e3efab05a63",
    'signature_method': 'HMAC-SHA1',
    'api_version': '1.0'
}

This object would then be passed to :js:func:`GateOne.init` like so:

Assuming the signature is valid Gate One would then inherently trust that the user connecting over the WebSocket is joe@company.com.

Assuming the signature is valid Gate One would then inherently trust that the user connecting over the WebSocket is 

Authentication objects (aka "authentication tokens") are only valid within the time frame specified in the :option:`--api_timestamp_window` setting.  They also can't be used more than once (to negate replay attacks).

Authentication objects (aka "authentication tokens") are only valid within the time frame specified in the 

 setting.  They also can't be used more than once (to negate replay attacks).

The following are examples demonstrating how to generate valid 'auth' objects in various programming languages.

import time, hmac, hashlib, json
secret = "secret"
authobj = {
    'api_key': "MjkwYzc3MDI2MjhhNGZkNDg1MjJkODgyYjBmN2MyMTM4M",
    'upn': "joe@company.com",
    'timestamp': str(int(time.time() * 1000)),
}
hash = hmac.new(secret, digestmod=hashlib.sha1)
hash.update(authobj['api_key'] + authobj['upn'] + authobj['timestamp'])
authobj['signature'] = hash.hexdigest()
valid_json_auth_object = json.dumps(authobj)

Here's a create_signature() function that can be used as a shortcut to those hash calls above::

Here's a create_signature() function that can be used as a shortcut to those hash calls above:

def create_signature(secret, *parts):
    import hmac, hashlib
    hash = hmac.new(secret, digestmod=hashlib.sha1)
    for part in parts:
        hash.update(str(part))
    return hash.hexdigest()

>>> create_signature(secret, api_key, upn, timestamp)
'f6c6c82281f8d56797599aeee01a5e3efab05a63'

require 'cgi'
require 'openssl'
require 'json'
secret = 'secret'
authobj = {
    'api_key' => 'MjkwYzc3MDI2MjhhNGZkNDg1MjJkODgyYjBmN2MyMTM4M',
    'upn' => 'joe@company.com',
    'timestamp' => (Time.now.getutc.to_i * 1000).inspect,
    'signature_method' => 'HMAC-SHA1',
    'api_version' => '1.0'
}
authobj['signature' = OpenSSL::HMAC.hexdigest('sha1', secret, authobj['api_key'] + authobj['upn'] + authobj['timestamp'])
valid_json_auth_object = JSON.generate(authobj)

$secret = 'secret'
$authobj = array(
    'upn' => $_SERVER['REMOTE_USER'],
    'timestamp' => time() . '0000',
);
$authobj['signature'] = hash_hmac('sha1', $authobj['api_key'] . $authobj['upn'] . $authobj['timestamp'], $secret);
$valid_json_auth_object = json_encode($authobj)

You must decide where you want Gate One to appear in your application.  The simplest way to do this is to create a DIV element like so:

By default `#gateone` will fill itself out to the full size of its parent element.  So it is usually a good idea to wrap it inside of a container which has some explicit dimensions set:

 will fill itself out to the full size of its parent element.  So it is usually a good idea to wrap it inside of a container which has some explicit dimensions set:

<div style="width: 60em; height: 30em;">
    <div id="gateone"></div>
</div>

Before you can initialize Gate One you need to include gateone.js in the web page.  You can, of course, just copy it out of `/opt/gateone/static` and include it in a `<script>` tag somewhere in your web page.  However, to ensure that you're always running the latest version of gateone.js it is recommended that you source the script from the Gate One server itself:

Before you can initialize Gate One you need to include gateone.js in the web page.  You can, of course, just copy it out of 

 tag somewhere in your web page.  However, to ensure that you're always running the latest version of gateone.js it is recommended that you source the script from the Gate One server itself:

You can also load the script dynamically via JS (if you know how).

A *much better* interactive tutorial is available in Gate One's 'tests' directory: <gateone dir>/tests/hello_embedded/.  The future of the documentation you're reading is uncertain.

 interactive tutorial is available in Gate One's 'tests' directory: <gateone dir>/tests/hello_embedded/.  The future of the documentation you're reading is uncertain.

This tutorial will walk you through embedding Gate One into a completely different web application.  It is divided into two parts:

Advanced: API-based authentication, "embedded mode", and customizing *everything*.

Advanced: API-based authentication, "embedded mode", and customizing 

We'll assume you have Gate One installed and running with the following settings (in your server.conf)::

We'll assume you have Gate One installed and running with the following settings (in your server.conf):

auth = None # Anonymous authentication
port = 443
disable_ssl = False
origins = "*" # Disable origin checking for now (this will be covered in Part 2)
url_prefix = "/" # Keep it simple for the tutorial

'origins' is the only setting above that is differs from defaults.

Before we continue please test your Gate One server by loading it in your browser.  This will also ensure that you've accepted the server's SSL certificate (if necessary).

Gate One's SSL certificate must be trusted by clients in order to embed Gate One.  This usually means purchasing an SSL certificate when you move to production.

The :js:func:`GateOne.init` function takes a number of optional arguments but for this example all we need is `goURL`.

 function takes a number of optional arguments but for this example all we need is 

Put that somewhere in your `window.onload` function and you should see something like this:

hooks = {
    'Web': [
        (r"/bookmarks/fetchicon", FaviconHandler),
        (r"/bookmarks/export", ExportHandler),
        (r"/bookmarks/import", ImportHandler),
    ],
    'WebSocket': {
        'bookmarks_sync': save_bookmarks,
        'bookmarks_get': get_bookmarks,
        'bookmarks_deleted': delete_bookmarks,
        'bookmarks_rename_tags': rename_tags,
    }
}

Unescape HTML code refs; c.f. http://wiki.python.org/moin/EscapingHtml

Reads the Netscape-style bookmarks.html in string, *html* and returns a
list of Bookmark objects.

Recursively looks inside *json_dict* trying to find tags associated with the
given *url*.  Returns the tags found as a list.

Given *json_str*, returns a list of bookmark objects representing the data
contained therein.

, returns a list of bookmark objects representing the data

Used to read and write bookmarks to a file on disk.  Can also synchronize
a given list of bookmarks with what's on disk.  Uses a given bookmark's
``updateSequenceNum`` to track what wins the "who is newer?" comparison.

Opens the bookmarks stored in self.user_dir.  If not present, an
empty file will be created.

Saves self.bookmarks to self.bookmarks_path as a JSON-encoded list.

Given *bookmarks*, synchronize with self.bookmarks doing conflict
resolution and whatnot.

, synchronize with self.bookmarks doing conflict

Returns a list of bookmarks newer than *updateSequenceNum*.
If *updateSequenceNum* is 0 or undefined, all bookmarks will be
returned.

Goes through all bookmarks and renames all tags named *old_tag* to be
*new_tag*.

Retrives the biggest favicon-like icon at the given URL.  It will try to
fetch apple-touch-icons (which can be nice and big) before it falls back
to grabbing the favicon.

Works with GET and POST requests but POST is preferred since it keeps the URL from winding up in the server logs.

Fetches the icon at the given URLs, stopping when it finds the biggest.
If an icon is not found, calls itself again with the next icon URL.
If the icon is found, writes it to the client and finishes the request.

Takes a bookmarks.html in a POST and returns a list of bookmarks in JSON
format

Takes a JSON-encoded list of bookmarks and returns a Netscape-style HTML
file.

Returns a JSON-encoded list of bookmarks updated since the last
*updateSequenceNum*.

If *updateSequenceNum* resolves to False, all bookmarks will be sent to
the client.

Given a *json_dict*, updates *urls_list* with each URL as it is found
within.

bookmarks.py - A plugin for Gate One that adds fancy bookmarking capabilities.

hooks = {
        'playback_save_recording': save_recording,
    }
}

Handles uploads of session recordings and returns them to the client in a
self-contained HTML file that will auto-start playback.

NOTE: The real crux of the code that handles this is in the template.

playback.py - A plugin for Gate One that adds support for saving and playing
back session recordings.

Yes this only contains one function and it is exposed to clients through a WebSocket hook.

This is how you add a URL handler to Gate One...  This example attaches
itslef to https://<your Gate One server>/example in the 'Web' hook at the
bottom of this file.  It works just like any Tornado
:class:`~tornado.web.RequestHandler`.  See:

itslef to 

/<your Gate One server>/example in the 'Web' hook at the

...for documentation on how to write a :class:`tornado.web.RequestHandler` for
the Tornado framework.  Fairly boilerplate stuff.

The only reason we use :class:`gateone.BaseHandler` instead of a vanilla :class:`tornado.web.RequestHandler` is so we have access to Gate One's :func:`gateone.BaseHandler.get_current_user` function.

Handle an HTTP GET request to this :class:`~tornado.web.RequestHandler`.
Connect to:

Example Handler for an `HTTP POST <http://en.wikipedia.org/wiki/POST_(HTTP)>`_
request.  Doesn't actually do anything.

Gate One includes a mechanism for plugins to send messages from terminal
programs directly to plugins written in Python.  It's called the "Special
Optional Escape Sequence Handler" or SOESH for short.  Here's how it works:
Whenever a terminal program emits, "\x1b]_;" it gets detected by Gate One's
:class:`~terminal.Terminal` class (which lives in `terminal.py`) and it will
execute whatever callback is registered for SOESH.  Inside of Gate One this
callback will always be :func:`gateone.TerminalWebSocket.esc_opt_handler`.

Whenever a terminal program emits, "x1b]_;" it gets detected by Gate One's

) and it will
callback will always be 

This `WebSocket <https://developer.mozilla.org/en/WebSockets/WebSockets_reference/WebSocket>`_
action gets exposed to the client automatically by way of the 'WebSocket'
hook at the bottom of this file.  The way it works is like this:

Whenever a message is received via the `WebSocket <https://developer.mozilla.org/en/WebSockets/WebSockets_reference/WebSocket>`_
Gate One will automatically
decode it into a Python :class:`dict` (only JSON-encoded messages are accepted).
Any and all keys in that :class:`dict` will be assumed to be 'actions' (just
like :js:attr:`GateOne.Net.actions` but on the server) such as this one.  If
the incoming key matches a registered action that action will be called
like so::

 (only JSON-encoded messages are accepted).
Any and all keys in that 

 but on the server) such as this one.  If
like so:

key(value)
# ...or just:
key() # If the value is None ('null' in JavaScript)

...where *key* is the action and *value* is what will be passed to said
action as an argument.  Since Gate One will automatically decode the message
as JSON the *value* will typically be passed to actions as a single :class:`dict`.
You can provide different kinds of arguments of course but be aware that
their ordering is unpredictable so always be sure to either pass *one*
argument to your function (assuming it is a :class:`dict`) or 100% keyword
arguments.

 is what will be passed to said
as JSON the 

.
their ordering is unpredictable so always be sure to either pass 

The *tws* keyword argument must always be present in WebSocket actions.  It
represents the user's current instance of Gate One's :class:`~gateone.TerminalWebSocket`
class. Think of it as the equivalent of, "self" inside of any given
:class:`~gateone.TerminalWebSocket` function.  For example, the following
function inside of :class:`~gateone.TerminalWebSocket` (inside of gateone.py)::

 keyword argument must always be present in WebSocket actions.  It
represents the user's current instance of Gate One's 

def pong(self, timestamp): # Docstring removed to save space
    message = {'pong': timestamp}
    self.write_message(json_encode(message))

def pong(timestamp, tws=None):
    tws.write_message(json_encode(message))

The typical naming convention for `WebSocket <https://developer.mozilla.org/en/WebSockets/WebSockets_reference/WebSocket>`_
actions is: `<plugin name>_<action>`.  Whether or not your action names
match your function names is up to you.  All that matters is that you line
up an *action* (string) with a *function* in `hooks['WebSocket']` (see below).

.  Whether or not your action names
up an 

*action* duplicates the functionality of Gate One's built-in
:func:`gateone.TerminalWebSocket.pong` function.  You can see how it is
called by the client (browser) inside of example.js (which is in this
plugin's 'static' dir).

 function.  You can see how it is

hooks = {
    'Web': [(r"/example", ExampleHandler)],
        'example_action': example_websocket_action
    },
    'Escape': example_opt_esc_handler,
    #'Auth': create_user_ssh_dir
}

example.py - A plugin to demonstrate how to write a Python plugin for Gate One.
Specifically, how to write your own web handlers, WebSocket actions, and take
advantage of all the available hooks and built-in functions.

This plugin is heavily commented with useful information.  Click on the `[source]` links to the right of any given class or function to see the actual code.

This plugin is heavily commented with useful information.  Click on the 

 links to the right of any given class or function to see the actual code.

Returns the frames of *golog_path* as a list that can be used with the
playback_log.html template.

 as a list that can be used with the

If *limit* is given, only return that number of frames (e.g. for preview)

Calls _enumerate_logs() via a :py:class:`multiprocessing.Process` so it
doesn't cause the :py:class:`~tornado.ioloop.IOLoop` to block.

Log objects will be returned to the client one at a time by sending
'logging_log' actions to the client over the WebSocket (*tws*).

'logging_log' actions to the client over the WebSocket (

Enumerates all of the user's logs and sends the client a "logging_logs"
message with the result.

If *limit* is given, only return the specified logs.  Works just like
`MySQL <http://en.wikipedia.org/wiki/MySQL>`_: limit="5,10" will retrieve
logs 5-10.

Calls :func:`_retrieve_log_flat` via a :py:class:`multiprocessing.Process`
so it doesn't cause the :py:class:`~tornado.ioloop.IOLoop` to block.

Whether or not the result should go into a new window or an iframe.

Writes the given *log_filename* to *queue* in a flat format equivalent to::

*settings* - A dict containing the *log_filename*, *colors*, and *theme* to
use when generating the HTML output.

Calls :func:`_retrieve_log_playback` via a
:py:class:`multiprocessing.Process` so it doesn't cause the
:py:class:`~tornado.ioloop.IOLoop` to block.

Writes a JSON-encoded message to the client containing the log in a
self-contained HTML format similar to::

self-contained HTML format similar to:

{
    'result': "Success",
    'log': <HTML rendered output>,
    'metadata': {<metadata of the log>}
}

It is expected that the client will create a new window with the result of
this method.

Calls :func:`_save_log_playback` via a :py:class:`multiprocessing.Process`

The difference between this function and :py:meth:`_retrieve_log_playback`
is that this one instructs the client to save the file to disk instead of
opening it in a new window.

{
    'data': <HTML rendered output>,
    'mimetype': 'text/html'
    'filename': <filename of the log recording>
}

hooks = {
        'logging_get_logs': enumerate_logs,
        'logging_get_log_flat': retrieve_log_flat,
        'logging_get_log_playback': retrieve_log_playback,
        'logging_get_log_file': save_log_playback,
    }
}

logging.py - A plugin for Gate One that provides logging-related functionality.

/home/riskable/workspace/GateOne/gateone/plugins/ssh/ssh.py:docstring of ssh

Called when there's a failure trying to open a sub-shell via OpenSSH's
`Master mode <http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing>`_
multiplexing capability.

Called when there's an error trying to execute a command in the slave.

Called when there's an error trying to generate a public/private keypair.

Called when there's an error trying to save public/private keypair or
certificate.

Called when we try to generate/decode something that requires a passphrase
but no passphrase was given.

Given a :class:`gateone.TerminalWebSocket` (*tws*) instance, return the
current user's ssh directory

wait_for_prompt(term, cmd, errorback, callback, m_instance, matched)

Called by :func:`termio.Multiplex.expect` inside of :func:`execute_command`,
clears the screen and executes *cmd*.  Also, sets an
:func:`~termio.Multiplex.expect` to call :func:`get_cmd_output` when the
end of the command output is detected.

Captures the output of the command executed inside of
:func:`wait_for_prompt` and calls *callback* if it isn't `None`.

Calls `m_instance.terminate()` and deletes it from the OPEN_SUBCHANNELS dict.

Called when the sub-channel times out by way of an
:class:`termio.Multiplex.expect` pattern that should never match anything.

got_error(m_instance, match=None, term=None, cmd=None, tws=None)

*match* is here in case we want to use it for a positive match of an error.

 is here in case we want to use it for a positive match of an error.

Execute the given command (*cmd*) on the given *term* using the existing
SSH tunnel (taking advantage of `Master mode <http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing>`_)
and call *callback* with the output of said command and the current
:class:`termio.Multiplex` instance as arguments like so::

If *callback* is not provided then the command will be executed and any
output will be ignored.

 is not provided then the command will be executed and any

This will not result in a new terminal being opened on the client--it simply executes a command and returns the result using the existing SSH tunnel.

Called by :func:`ws_exec_command` when the output of the executed command
has been captured successfully.  Writes a message to the client with the
command's output and some relevant metadata.

 when the output of the executed command

Takes the necessary variables from *settings* and calls :func:`execute_command`.

*settings* should be a dict that contains a 'term' and a 'cmd' to execute.

 should be a dict that contains a 'term' and a 'cmd' to execute.

This function can be used to quickly execute a command and return its result from the client over an existing SSH connection without requiring the user to enter their password!  See execRemoteCmd() in ssh.js.

This handler allows the client to view, edit, and upload the known_hosts
file associated with their user account.

Determine what the user is asking for and call the appropriate method.

Determine what the user is updating by checking the given arguments and
proceed with the update.

Writes the connection string associated with *term* to the `WebSocket <https://developer.mozilla.org/en/WebSockets/WebSockets_reference/WebSocket>`_
like so::

In ssh.js we attach an action (aka handler) to :js:attr:`GateOne.Net.actions`
for 'sshjs_reconnect' messages that attaches the connection string to
`GateOne.terminals[*term*]['sshConnectString']`

Returns the private SSH key associated with *name* to the client.  If
*public* is `True`, returns the public key to the client.

Returns a the hash of the given host's public key by making a remote
connection to the server (not just by looking at known_hosts).

Calls :func:`openssh_generate_new_keypair` or
:func:`dropbear_generate_new_keypair` depending on what's available on the
system.

openssh_generate_new_keypair(name, path, keytype=None, passphrase='', bits=None, comment='', tws=None)

Generates a new private and public key pair--stored in the user's directory
using the given *name* and other optional parameters (using OpenSSH).

using the given 

If *keytype* is given, it must be one of "ecdsa", "rsa" or "dsa" (case
insensitive).  If *keytype* is "rsa" or "ecdsa", *bits* may be specified to
specify the size of the key.

 is given, it must be one of "ecdsa", "rsa" or "dsa" (case
insensitive).  If 

Defaults to generating a 521-byte ecdsa key if OpenSSH is version 5.7+. Otherwise a 2048-bit rsa key will be used.

dropbear_generate_new_keypair(name, path, keytype=None, passphrase='', bits=None, comment='', tws=None)

openssh_generate_public_key(path, passphrase=None, settings=None, tws=None)

Generates a public key from the given private key at *path*.  If a
*passphrase* is provided, it will be used to generate the public key (if
necessary).

 is provided, it will be used to generate the public key (if

Stores the given *settings['private']* and/or *settings['public']* keypair
in the user's ssh directory as *settings['name']* and/or
*settings['name']*.pub, respectively.  Either file can be saved independent
of each other (in case this function needs to be called multiple times to
save each respective file).

.pub, respectively.  Either file can be saved independent

Also, a *settings['certificate']* may be provided to be saved along
with the private and public keys.  It will be saved as
*settings['name']*-cert.pub.

 may be provided to be saved along

I've found the following website helpful in understanding how to use OpenSSH with SSL certificates: http://blog.habets.pp.se/2011/07/OpenSSH-certificates

I've found the following website helpful in understanding how to use OpenSSH with SSL certificates: 

Using signed-by-a-CA certificates is very handy because allows you to revoke the user's SSH key(s).  e.g. If they left the company.

Removes the identity associated with *name*.  For example if *name* is
'testkey', 'testkey' and 'testkey.pub' would be removed from the user's
ssh directory (and 'testkey-cert.pub' if present).

 is

Sends a message to the client with a list of the identities stored on the
server for the current user.

*anything* is just there because the client needs to send *something* along
with the 'action'.

Given a list of *identities*, mark them as defaults to use in all outbound
SSH connections by writing them to `<user's ssh dir>/.default_ids`.  If
*identities* is empty, no identities will be used in outbound connections.

, mark them as defaults to use in all outbound
SSH connections by writing them to 

Whenever this function is called it will overwrite whatever is in `.default_ids`.

Whenever this function is called it will overwrite whatever is in 

Handles text passed from the special optional escape sequance handler.  We
use it to tell ssh.js what the SSH connection string is so it can use that
information to duplicate sessions (if the user so desires).  For reference,
the specific string which will call this function from a terminal app is::

the specific string which will call this function from a terminal app is:

To be called by the 'Auth' hook that gets called after the user is done
authenticating, ensures that the `<user's dir>/ssh` directory exists.

authenticating, ensures that the 

Opens a sub-channel of communication by executing a new shell on the SSH
server using OpenSSH's `Master mode <http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing>`_
capability (it spawns a new slave) and returns the resulting
:class:`termio.Multiplex` instance.  If a slave has already been opened for
this purpose it will re-use the existing channel.

server using OpenSSH's 

 instance.  If a slave has already been opened for

hooks = {
    'Web': [(r"/ssh", KnownHostsHandler)],
        'ssh_get_connect_string': get_connect_string,
        'ssh_execute_command': ws_exec_command,
        'ssh_get_identities': get_identities,
        'ssh_get_public_key': get_public_key,
        'ssh_get_private_key': get_private_key,
        'ssh_get_host_fingerprint': get_host_fingerprint,
        'ssh_gen_new_keypair': generate_new_keypair,
        'ssh_store_id_file': store_id_file,
        'ssh_delete_identity': delete_identity,
        'ssh_set_default_identities': set_default_identities
    },
    'Escape': opt_esc_handler,
    'Auth': create_user_ssh_dir
}

ssh.py - A plugin for Gate One that adds additional SSH-specific features.

#!/usr/bin/env python
#
#       Copyright 2011 Liftoff Software Corporation
#

# Meta
__version__ = '1.0'
__license__ = "AGPLv3 or Proprietary (see LICENSE.txt)"
__version_info__ = (1, 0)
__author__ = 'Dan McDougall <daniel.mcdougall@liftoffsoftware.com>'

# Import stdlib stuff
import os, sys, re, gzip
from time import sleep
from datetime import datetime
from optparse import OptionParser

# Import our own stuff
from utils import raw
from gateone import PLUGINS

# 3rd party imports
from tornado.escape import json_encode, json_decode

__doc__ = """\
.. _log_viewer:

Log Viewer
==========
Allows the user to play back a given log file like a video (default) or display
it in a syslog-like format.  To view usage information, run it with the --help
switch:

.. ansi-block::
    \x1b[1;31mroot\x1b[0m@host\x1b[1;34m:/opt/gateone $\x1b[0m ./logviewer.py --help
    Usage:  logviewer.py [options] <log file>

    Options:
      --version       show program's version number and exit
      -h, --help      show this help message and exit
      -f, --flat      Display the log line-by-line in a syslog-like format.
      -p, --playback  Play back the log in a video-like fashion. This is the
                    default view.
      --pretty        Preserve font and character renditions when displaying the
                    log in flat view (default).
      --raw           Display control characters and escape sequences when
                    viewing.

Here's an example of how to display a Gate One log (.golog) in a flat, greppable
format:

    \x1b[1;31mroot\x1b[0m@host\x1b[1;34m:/opt/gateone $\x1b[0m ./logviewer.py --flat
    Sep 09 21:07:14 Host/IP or SSH URL [localhost]: modern-host
    Sep 09 21:07:16 Port [22]:
    Sep 09 21:07:16 User: bsmith
    Sep 09 21:07:17 Connecting to: ssh://bsmith@modern-host:22
    Sep 09 21:07:17
    Sep 09 21:07:17 bsmith@modern-host's password:
    Sep 09 21:07:20 Welcome to Ubuntu 11.04 (GNU/Linux 2.6.38-11-generic x86_64)
    Sep 09 21:07:20
    Sep 09 21:07:20  * Documentation:  https://help.ubuntu.com/
    Sep 09 21:07:20 Last login: Thu Sep 29 08:51:27 2011 from portarisk
    Sep 09 21:07:20 \x1b[1;34mbsmith\x1b[0m@modern-host\x1b[1;34m:~ $\x1b[0m ls
    Sep 09 21:07:21 why_I_love_gate_one.txt  to_dont_list.txt
    Sep 09 21:07:21 \x1b[1;34mbsmith\x1b[0m@modern-host\x1b[1;34m:~ $\x1b[0m

About Gate One's Log Format
===========================
Gate One's log format (.golog) is a gzip-compressed unicode (UTF-8) text file
consisting of time-based frames separated by the unicode character, U+F0F0F0.
Each frame consists of JavaScript-style timestamp (because it is compact)
followed by a colon and then the text characters of the frame.  A frame ends
when a U+F0F0F0 character is encountered.

Here are two example .golog frames demonstrating the format::

    1317344834868:\\x1b[H\\x1b[2JHost/IP or SSH URL [localhost]: <U+F0F0F>1317344836086:\\r\\nPort [22]: <U+F0F0F>

Gate One logs can be opened, decoded, and parsed in Python fairly easily::

    import gzip
    golog = gzip.open(path_to_golog).read()
    for frame in golog.split(u"\U000f0f0f".encode('UTF-8')):
        frame_time = float(frame[:13]) # First 13 chars is the timestamp
        # Timestames can be converted into datetime objects very simply:
        datetime_obj = datetime.fromtimestamp(frame_time/1000)
        frame_text = frame[14:] # This gets you the actual text minus the colon
        # Do something with the datetime_obj and the frame_text

.. note:: U+F0F0F0 is from Private Use Area (PUA) 15 in the Unicode Character Set (UCS). It was chosen at random (mostly =) from PUA-15 because it is highly unlikely to be used in an actual terminal program where it could corrupt a session log.

Class Docstrings
================
"""

# Globals
SEPARATOR = u"\U000f0f0f" # The character used to separate frames in the log
RE_OPT_SEQ = re.compile(r'\x1b\]_\;(.+?)(\x07|\x1b\\)', re.MULTILINE)
RE_TITLE_SEQ = re.compile(
    r'.*\x1b\][0-2]\;(.+?)(\x07|\x1b\\)', re.DOTALL|re.MULTILINE)

# TODO: Support Fast forward/rewind/pause like Gate One itself.

def playback_log(log_path, file_like, show_esc=False):
    """
    Plays back the log file at *log_path* by way of timely output to *file_like*
    which is expected to be any file-like object with write() and flush()
    methods.

    If *show_esc* is True, escape sequences and control characters will be
    escaped so they can be seen in the output.
    """
    log = gzip.open(log_path).read()
    prev_frame_time = None
    # Skip first frame
    for i, frame in enumerate(log.split(SEPARATOR.encode('UTF-8'))[1:]):
        try:
            frame_time = float(frame[:13]) # First 13 chars is the timestamp
            frame = frame[14:] # Skips the colon
            if i == 0:
                # Write it out immediately
                file_like.write(frame.decode('UTF-8'))
                prev_frame_time = frame_time
            else:
            # Wait until the time between the previous frame and now has passed
                wait_time = (frame_time - prev_frame_time)/1000.0
                sleep(wait_time) # frame times are in milliseconds
                if show_esc:
                    frame = raw(frame)
                file_like.flush()
        except ValueError:
            # End of file.  No biggie.
            return

def escape_escape_seq(text, preserve_renditions=True, rstrip=True):
    """
    Escapes escape sequences so they don't muck with the terminal viewing *text*
    Also replaces special characters with unicode symbol equivalents (e.g. so
    you can see what they are without having them do anything to your running
    shell)

    If *preserve_renditions* is True, CSI escape sequences for renditions will
    be preserved as-is (e.g. font color, background, etc).

    If *rstrip* is true, trailing escape sequences and whitespace will be
    removed.
    """
    esc_sequence = re.compile(
        r'\x1b(.*\x1b\\|[ABCDEFGHIJKLMNOQRSTUVWXYZa-z0-9=]|[()# %*+].)')
    csi_sequence = re.compile(r'\x1B\[([?A-Za-z0-9;@:\!]*)([A-Za-z@_])')
    #esc_rstrip = re.compile('[ \t]+\x1b.+$')
    out = u""
    esc_buffer = u""
    # If this seems confusing it is because text parsing is a black art! ARRR!
    for char in text:
        if not esc_buffer:
            if char == u'\x1b': # Start of an ESC sequence
                esc_buffer = char
            # TODO: Determine if we should bring this back:
            #elif ord(char) in replacement_map:
                #out += replacement_map[ord(char)]
            else: # Vanilla char.  Booooring.
                out += raw(char)
        else: # Something interesting is going on
            esc_buffer += char
            if char == u'\x07' or esc_buffer.endswith(u'\x1b\\'): # Likely title
                esc_buffer = u'' # Nobody wants to see your naked ESC sequence
                continue
            elif esc_buffer.endswith('\x1b\\'):
                esc_buffer = u'' # Ignore
                continue
            # Nobody wants to see plain ESC sequences in the buf...
            match_obj = esc_sequence.match(esc_buffer)
            if match_obj:
                seq_type = match_obj.group(1)
                esc_buffer = u'' # Just when you thought you've ESC'd...
                continue
            # CSI ESC sequences...  These are worth a second look
            match_obj = csi_sequence.match(esc_buffer)
                csi_type = match_obj.group(2)
                if csi_type == 'm' and preserve_renditions: # mmmmmm!
                    out += esc_buffer # Ooh, naked viewing of pretty things!
                esc_buffer = u'' # Make room for more!
                continue
    if rstrip:
        # Remove trailing whitespace + trailing ESC sequences
        #return esc_rstrip.sub('', out).rstrip()
        return out.rstrip()
    else: # All these trailers better make for a good movie
        return out

def flatten_log(log_path, preserve_renditions=True, show_esc=False):
    """
    Given a log file at *log_path*, return a str of log lines contained within.

    be preserved as-is (e.g. font color, background, etc).  This is to make the
    output appear as close to how it was originally displayed as possible.
    Besides that, it looks really nice =)

    visible in the output.  Trailing whitespace and escape sequences will not be
    removed.

    NOTE: Converts our standard recording-based log format into something that
    can be used with grep and similar search/filter tools.
    """
    lines = gzip.open(log_path).read()
    out = ""
    out_line = ""
    cr = False
    # We skip the first frame, [1:] because it holds the recording metadata
    for frame in lines.split(SEPARATOR.encode('UTF-8'))[1:]:
        frame = frame.decode('UTF-8', 'ignore')
        try:
            # Convert to datetime object
            frame_time = datetime.fromtimestamp(frame_time/1000)
            if show_esc:
                frame_time = frame_time.strftime(u'\x1b[0m%b %m %H:%M:%S')
            else: # Renditions preserved == I want pretty.  Make the date bold:
                frame_time = frame_time.strftime(
                    u'\x1b[0;1m%b %m %H:%M:%S\x1b[m')
            for char in frame[14:]:
                if '\x1b[H\x1b[2J' in out_line: # Clear screen sequence
                    # Handle the clear screen (usually ctrl-l) by outputting
                    # a new log entry line to avoid confusion regarding what
                    # happened at this time.
                    out_line += "^L" # Clear screen is a ctrl-l or equivalent
                    if show_esc:
                        adjusted = raw(out_line)
                    else:
                        adjusted = escape_escape_seq(out_line, rstrip=True)
                    out += frame_time + ' %s\n' % adjusted
                    out_line = ""
                    continue
                if char == u'\n':
                    else:
                    cr = False
                elif char in u'\r':
                    # Carriage returns need special handling.  Make a note of it
                    cr = True
                else:
                    # \r without \n means that characters were (likely)
                    # overwritten.  This usually happens when the user gets to
                    # the end of the line (which would create a newline in the
                    # terminal but not necessarily the log), erases their
                    # current line (e.g. ctrl-u), or an escape sequence modified
                    # the line in-place.  To clearly indicate what happened we
                    # insert a '^M' and start a new line so as to avoid
                    # confusion over these events.
                    if cr:
                        out_line += "^M"
                        out += frame_time + ' '
                        if show_esc:
                            adjusted = raw(out_line)
                        else:
                            adjusted = escape_escape_seq(out_line, rstrip=True)
                        out += adjusted
                        out += '\n'
                        out_line = ""
                    out_line += char
                    cr = False
        except ValueError as e:
            pass
    return out

if __name__ == "__main__":
    """Parse command line arguments and view the log in the specified format."""
    usage = ('\t%prog [options] <log file>')
    parser = OptionParser(usage=usage, version=__version__)
    parser.disable_interspersed_args()
    parser.add_option("-f", "--flat",
        dest="flat",
        default=False,
        action="store_true",
        help="Display the log line-by-line in a syslog-like format."
    )
    parser.add_option("-p", "--playback",
        dest="playback",
        default=True,
        action="store_false",
        help=("Play back the log in a video-like fashion. This is the default "
              "view.")
    )
    parser.add_option("--pretty",
        dest="pretty",
        help=("Preserve font and character renditions when displaying the log "
              "in flat view (default).")
    )
    parser.add_option("--raw",
        dest="raw",
        help="Display control characters and escape sequences when viewing."
    )
    (options, args) = parser.parse_args()
    if len(args) < 1:
        print("ERROR: You must specify a log file to view.")
        parser.print_help()
        sys.exit(1)
    log_path = args[0]
    if options.flat:
        result = flatten_log(
            log_path, preserve_renditions=options.pretty, show_esc=options.raw)
        print(result)
    else:
        playback_log(log_path, sys.stdout)

#
#

# TODO: Complete this docstring...
notice.py - A plugin for Gate One that adds an escape sequence handler that will
tell the client browser to display a message whenever said escape sequence is
encountered.  Any terminal program can emit the following escape sequence to
display a message in the browser::

    \\x1b]_;notice|<the message>\\x07

.. note that the above example has double slashes...  Don't do that in your actual code.  They're just there to make sure it shows up properly in the HTML documentation.

Very straightforward and also very powerful.

Hooks
-----
This Python plugin file implements the following hooks::

    hooks = {
        'Escape': notice_esc_seq_handler,
    }

Docstrings
----------
"""

# Meta
__license__ = "GNU AGPLv3 or Proprietary (see LICENSE.txt)"

# Special optional escape sequence handler (see docs on how it works)
def notice_esc_seq_handler(message, tws):
    """
    Handles text passed from the special optional escape sequance handler to
    display a *message* to the connected client (browser).

    .. seealso:: :class:`gateone.TerminalWebSocket.esc_opt_handler` and :func:`terminal.Terminal._opt_handler`
    """
    message = {'notice': message}
    tws.write_message(message)

hooks = {
    'Escape': notice_esc_seq_handler,
}

#!/usr/bin/python
# -*- encoding: iso-8859-1 -*-

"""
Python syslog client.

This code is placed in the public domain by the author.
Written by Christian Stigen Larsen.

This is especially neat for Windows users, who (I think) don't
get any syslog module in the default python installation.

See RFC3164 for more info -- http://tools.ietf.org/html/rfc3164

Note that if you intend to send messages to remote servers, their
syslogd must be started with -r to allow to receive UDP from
the network.
"""

import socket

FACILITY = {
    'kern': 0, 'user': 1, 'mail': 2, 'daemon': 3,
    'auth': 4, 'syslog': 5, 'lpr': 6, 'news': 7,
    'uucp': 8, 'cron': 9, 'authpriv': 10, 'ftp': 11,
    'local0': 16, 'local1': 17, 'local2': 18, 'local3': 19,
    'local4': 20, 'local5': 21, 'local6': 22, 'local7': 23,
}

LEVEL = {
    'emerg': 0, 'alert':1, 'crit': 2, 'err': 3,
    'warning': 4, 'notice': 5, 'info': 6, 'debug': 7
}

def syslog(message, level=LEVEL['notice'], facility=FACILITY['daemon'],
    host='localhost', port=514):
    """
    Send syslog UDP packet to given host and port.
    """
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    data = '<%d>%s' % (level + facility*8, message)
    sock.sendto(data, (host, port))
    sock.close()

#
#
# NOTE:  Named logging_plugin.py instead of "logging.py" to avoid conflics with the existing logging module

# TODO: Fix the flat log viewing format.  Doesn't look quite right.
# TODO: Write search functions.
# TODO: Add some search indexing capabilities so that search will be fast.
# TODO: Add a background process that cleans up old logs.
# TODO: Write a handler that displays a page where users can drag & drop .golog files to have them played back in their browser.

Hooks
-----

    hooks = {
        'WebSocket': {
            'logging_get_logs': enumerate_logs,
            'logging_get_log_flat': retrieve_log_flat,
            'logging_get_log_playback': retrieve_log_playback,
            'logging_get_log_file': save_log_playback,
        }
    }

"""

# Meta

# Python stdlib
import os
import logging
import gzip
import time
import re
from multiprocessing import Process, Queue

# Our stuff
from gateone import BaseHandler, PLUGINS
from logviewer import flatten_log
from termio import retrieve_first_frame, retrieve_last_frame
from termio import get_or_update_metadata
from utils import get_translation, json_encode

_ = get_translation()

# Tornado stuff
import tornado.template
import tornado.ioloop
from tornado.escape import json_decode

# Globals
PROCS = {} # For tracking/cancelling background processes
# Matches Gate One's special optional escape sequence (ssh plugin only)
RE_OPT_SSH_SEQ = re.compile(
    r'.*\x1b\]_\;(ssh\|.+?)(\x07|\x1b\\)', re.MULTILINE|re.DOTALL)
# Matches an xterm title sequence

# Helper functions
def retrieve_log_frames(golog_path, rows, cols, limit=None):
    """
    Returns the frames of *golog_path* as a list that can be used with the
    playback_log.html template.

    If *limit* is given, only return that number of frames (e.g. for preview)
    """
    out_frames = []
    from terminal import Terminal
    terminal_emulator = Terminal
    term = terminal_emulator(
        # 14/7 for the em_height should be OK for most browsers to ensure that
        # images don't always wind up at the bottom of the screen.
        rows=rows, cols=cols, em_dimensions={'height':14, 'width':7})
    frames = gzip.open(golog_path).read().split(SEPARATOR.encode('UTF-8'))[1:]
    if not limit:
        limit = len(frames)
    for frame in frames[:limit]:
        if len(frame) > 14:
            frame_time = int(float(frame[:13]))
            frame_screen = frame[14:] # Skips the colon
            term.write(frame_screen)
            scrollback, screen = term.dump_html()
            out_frames.append({'screen': screen, 'time': frame_time})
    return out_frames # Skip the first frame which is the metadata

# Handlers

# WebSocket commands (not the same as handlers)
def enumerate_logs(limit=None, tws=None):
    """
    Calls _enumerate_logs() via a :py:class:`multiprocessing.Process` so it
    doesn't cause the :py:class:`~tornado.ioloop.IOLoop` to block.

    Log objects will be returned to the client one at a time by sending
    'logging_log' actions to the client over the WebSocket (*tws*).
    """
    # Sometimes IOLoop detects multiple events on the fd before we've finished
    # doing a get() from the queue.  This variable is used to ensure we don't
    # send the client duplicates:
    results = []
    if tws.settings['session_logging'] == False:
        message = {'notice': _(
            "NOTE: User session logging is disabled.  To enable it, set "
            "'session_logging = True' in your server.conf.")}
        tws.write_message(message)
        return # Nothing left to do
    user = tws.get_current_user()['upn']
    users_dir = os.path.join(tws.settings['user_dir'], user) # "User's dir"
    io_loop = tornado.ioloop.IOLoop.instance()
    global PROCS
    if user not in PROCS:
        PROCS[user] = {}
    else: # Cancel anything that's already running
        fd = PROCS[user]['queue']._reader.fileno()
        if fd in io_loop._handlers:
            io_loop.remove_handler(fd)
        if PROCS[user]['process']:
            try:
                PROCS[user]['process'].terminate()
            except OSError:
                # process was already terminated...  Nothing to do
                pass
    PROCS[user]['queue'] = q = Queue()
    PROCS[user]['process'] = Process(
        target=_enumerate_logs, args=(q, user, users_dir, limit))
    def send_message(fd, event):
        """
        Sends the log enumeration result to the client.  Necessary because
        IOLoop doesn't pass anything other than *fd* and *event* when it handles
        file descriptor events.
        """
        message = q.get()
        #logging.debug('message: %s' % message)
        if message == 'complete':
            total_bytes = 0
            logs_dir = os.path.join(users_dir, "logs")
            log_files = os.listdir(logs_dir)
            for log in log_files:
                log_path = os.path.join(logs_dir, log)
                total_bytes += os.stat(log_path).st_size
            out_dict = {
                'total_logs': len(log_files),
                'total_bytes': total_bytes
            }
            # This signals to the client that we're done
            message = {'logging_logs_complete': out_dict}
            tws.write_message(message)
            return
        message = json_encode(message)
        if message not in results:
            # Keep track of how many/how much
            if results:
                results.pop() # No need to keep old stuff hanging around
            results.append(message)
    # This is kind of neat:  multiprocessing.Queue() instances have an
    # underlying fd that you can access via the _reader:
    io_loop.add_handler(q._reader.fileno(), send_message, io_loop.READ)
    # We tell the IOLoop to watch this fd to see if data is ready in the queue.
    PROCS[user]['process'].start()

def _enumerate_logs(queue, user, users_dir, limit=None):
    """
    Enumerates all of the user's logs and sends the client a "logging_logs"
    message with the result.

    If *limit* is given, only return the specified logs.  Works just like
    `MySQL <http://en.wikipedia.org/wiki/MySQL>`_: limit="5,10" will retrieve
    logs 5-10.
    """
    logs_dir = os.path.join(users_dir, "logs")
    log_files = os.listdir(logs_dir)
    log_files = [a for a in log_files if a.endswith('.golog')] # Only gologs
    log_files.sort() # Should put them in order by date
    log_files.reverse() # Make the newest ones first
    out_dict = {}
    for log in log_files:
        log_path = os.path.join(logs_dir, log)
        logfile = gzip.open(log_path)
        logging.debug("Getting metadata from: %s" % log_path)
        metadata = get_or_update_metadata(log_path, user)
        if not metadata:
            # Broken log file -- may be being written to
            continue # Just skip it
        metadata['size'] = os.stat(log_path).st_size
        out_dict['log'] = metadata
        message = {'logging_log': out_dict}
        queue.put(message)
        # If we go too quick sometimes the IOLoop will miss a message
        time.sleep(0.01)
    queue.put('complete')

def retrieve_log_flat(settings, tws=None):
    """
    Calls :func:`_retrieve_log_flat` via a :py:class:`multiprocessing.Process`
    so it doesn't cause the :py:class:`~tornado.ioloop.IOLoop` to block.

    :arg dict settings: A dict containing the *log_filename*, *colors*, and *theme* to use when generating the HTML output.
    :arg instance tws: The current :class:`gateone.TerminalWebSocket` instance (connected).

    Here's the details on *settings*:

    :arg settings['log_filename']: The name of the log to display.
    :arg settings['colors']: The CSS color scheme to use when generating output.
    :arg settings['theme']: The CSS theme to use when generating output.
    :arg settings['where']: Whether or not the result should go into a new window or an iframe.
    """
    settings['container'] = tws.container
    settings['prefix'] = tws.prefix
    settings['user'] = user = tws.get_current_user()['upn']
    settings['users_dir'] = os.path.join(tws.settings['user_dir'], user)
    settings['gateone_dir'] = tws.settings['gateone_dir']
            try:
                pass
        target=_retrieve_log_flat, args=(q, settings))
        """
        """
        io_loop.remove_handler(fd)

def _retrieve_log_flat(queue, settings):
    """
    Writes the given *log_filename* to *queue* in a flat format equivalent to::

        ./logviewer.py --flat log_filename

    *settings* - A dict containing the *log_filename*, *colors*, and *theme* to
    use when generating the HTML output.
    """
    out_dict = {
        'result': "",
        'log': "",
        'metadata': {},
    }
    # Local variables
    out = []
    spanstrip = re.compile(r'\s+\<\/span\>$')
    gateone_dir = settings['gateone_dir']
    user = settings['user']
    users_dir = settings['users_dir']
    container = settings['container']
    prefix = settings['prefix']
    log_filename = settings['log_filename']
    theme = "%s.css" % settings['theme']
    colors = "%s.css" % settings['colors']
    log_path = os.path.join(logs_dir, log_filename)
    if os.path.exists(log_path):
        out_dict['metadata'] = get_or_update_metadata(log_path, user)
        out_dict['metadata']['filename'] = log_filename
        out_dict['result'] = "Success"
        flattened_log = flatten_log(log_path)
        flattened_log = flattened_log.replace('\n', '\r\n') # Needed to emulate an actual term
        # Use the terminal emulator to create nice HTML-formatted output
        from terminal import Terminal
        term = Terminal(rows=100, cols=300)
        term.write(flattened_log)
        scrollback, screen = term.dump_html()
        # Join them together
        log_lines = scrollback + screen
        # rstrip the lines
        log_lines = [a.rstrip() for a in log_lines]
        # Fix things like "<span>whatever [lots of whitespace]    </span>"
        for i, line in enumerate(log_lines):
            out.append(spanstrip.sub("</span>", line))
        out_dict['log'] = out
    else:
        out_dict['result'] = _("ERROR: Log not found")
    message = {'logging_log_flat': out_dict}
    queue.put(message)

def retrieve_log_playback(settings, tws=None):
    """
    Calls :func:`_retrieve_log_playback` via a
    :py:class:`multiprocessing.Process` so it doesn't cause the
    :py:class:`~tornado.ioloop.IOLoop` to block.
    """
    settings['url_prefix'] = tws.settings['url_prefix']
            try:
                pass
        target=_retrieve_log_playback, args=(q, settings))
        """
        """

def _retrieve_log_playback(queue, settings):
    """
    Writes a JSON-encoded message to the client containing the log in a
    self-contained HTML format similar to::

        ./logviewer.py log_filename

    The output will look like this::

        {
            'result': "Success",
            'log': <HTML rendered output>,
            'metadata': {<metadata of the log>}
        }

    It is expected that the client will create a new window with the result of
    this method.
    """
    #print("Running retrieve_log_playback(%s)" % settings);
    if 'where' not in settings: # Avoids a KeyError if it is missing
        settings['where'] = None
        'html': "", # Will be replace with the rendered template
        'where': settings['where'] # Just gets passed as-is back to the client
    }
    url_prefix = settings['url_prefix']
    # Important paths
    # NOTE: Using os.path.join() in case Gate One can actually run on Windows
    # some day.
    templates_path = os.path.join(gateone_dir, 'templates')
    colors_path = os.path.join(templates_path, 'term_colors')
    themes_path = os.path.join(templates_path, 'themes')
    plugins_path = os.path.join(gateone_dir, 'plugins')
    logging_plugin_path = os.path.join(plugins_path, 'logging')
    template_path = os.path.join(logging_plugin_path, 'templates')
    # recording format:
    # {"screen": [log lines], "time":"2011-12-20T18:00:01.033Z"}
    # Actual method logic
        # First we setup the basics
        try:
            rows = out_dict['metadata']['rows']
            cols = out_dict['metadata']['cols']
        except KeyError:
        # Log was created before rows/cols metadata was included via termio.py
        # Use some large values to ensure nothing wraps and hope for the best:
            rows = 40
            cols = 500
        out_dict['result'] = "Success" # TODO: Add more error checking
        # Next we render the theme and color templates so we can pass them to
        # our final template
        with open(os.path.join(colors_path, colors)) as f:
            colors_file = f.read()
        colors_template = tornado.template.Template(colors_file)
        rendered_colors = colors_template.generate(
            container=container,
            prefix=prefix,
            url_prefix=url_prefix
        )
        with open(os.path.join(themes_path, theme)) as f:
            theme_file = f.read()
        theme_template = tornado.template.Template(theme_file)
        # Setup our 256-color support CSS:
        colors_256 = ""
        from gateone import COLORS_256
        for i in xrange(256):
            fg = "#%s span.fx%s {color: #%s;}" % (
                container, i, COLORS_256[i])
            bg = "#%s span.bx%s {background-color: #%s;} " % (
            fg_rev = "#%s span.reverse.fx%s {background-color: #%s; color: inherit;}" % (
            bg_rev = "#%s span.reverse.bx%s {color: #%s; background-color: inherit;} " % (
            colors_256 += "%s %s %s %s\n" % (fg, bg, fg_rev, bg_rev)
        colors_256 += "\n"
        rendered_theme = theme_template.generate(
            colors_256=colors_256,
        )
        # NOTE: 'colors' are customizable but colors_256 is universal.  That's
        # why they're separate.
        # Lastly we render the actual HTML template file
        # NOTE: Using Loader() directly here because I was getting strange EOF
        # errors trying to do it the other way :)
        loader = tornado.template.Loader(template_path)
        playback_template = loader.load('playback_log.html')
        preview = 'false'
        if settings['where']:
            preview = 'true'
            recording = retrieve_log_frames(log_path, rows, cols, limit=50)
        else:
            recording = retrieve_log_frames(log_path, rows, cols)
        playback_html = playback_template.generate(
            theme=rendered_theme,
            colors=rendered_colors,
            preview=preview,
            recording=json_encode(recording),
        )
        out_dict['html'] = playback_html
    else:
    message = {'logging_log_playback': out_dict}

def save_log_playback(settings, tws=None):
    """
    Calls :func:`_save_log_playback` via a :py:class:`multiprocessing.Process`
    """
    q = Queue()
    global PROC
    PROC = Process(target=_save_log_playback, args=(q, settings))
    PROC.daemon = True # We don't care if this gets terminated mid-process.
        """
        """
    PROC.start()
    return

def _save_log_playback(queue, settings):
    """

    The difference between this function and :py:meth:`_retrieve_log_playback`
    is that this one instructs the client to save the file to disk instead of
    opening it in a new window.

        {
            'data': <HTML rendered output>,
            'mimetype': 'text/html'
            'filename': <filename of the log recording>
        }

    """
        'result': "Success",
        'mimetype': 'text/html',
        'data': "", # Will be replace with the rendered template
    }
    short_logname = log_filename.split('.golog')[0]
    out_dict['filename'] = "%s.html" % short_logname
    # some day.
        try:
            rows = 40
            cols = 500
        )
            colors_256 += "%s %s" % (fg, bg)
        )
        recording = retrieve_log_frames(log_path, rows, cols)
        )
        out_dict['data'] = playback_html
    else:
    message = {'save_file': out_dict}

# Temporarily disabled while I work around the problem of gzip files not being
# downloadable over the websocket.
#def get_log_file(log_filename, tws):
    #"""
    #Returns the given *log_filename* (as a regular file) so the user can save it
    #to disk.
    #"""
    #user = tws.get_current_user()['upn']
    #logging.debug("%s: get_log_file(%s)" % (user, log_filename))
    #users_dir = os.path.join(tws.settings['user_dir'], user) # "User's dir"
    #users_log_dir = os.path.join(users_dir, 'logs')
    #log_path = os.path.join(users_log_dir, log_filename)
    #out_dict = {'result': 'Success'}
    #if os.path.exists(log_path):
        #with open(log_path) as f:
            #out_dict['data'] = f.read()
    #else:
        #out_dict['result'] = _(
            #'SSH Plugin Error: Log not found at %s' % log_path)
    #message = {'save_file': out_dict}
    #tws.write_message(message)

hooks = {
    }
}

#
#
# For license information see LICENSE.txt

Gate One utility functions and classes.
"""

# Meta
__version__ = '1.1'
__version_info__ = (1, 1)

import os
import signal
import sys
import random
import re
import errno
import uuid
import mimetypes
import fcntl
from datetime import timedelta

# Import 3rd party stuff
from tornado import locale
try:
    from tornado.escape import json_encode as _json_encode
    from tornado.escape import json_decode
except ImportError: # Tornado isn't available
    from json import dumps as _json_encode
    from json import loads as json_encode
from tornado.escape import to_unicode, utf8

# Globals
MACOS = os.uname()[0] == 'Darwin'
# This matches JUST the PIDs from the output of the pstree command
#RE_PSTREE = re.compile(r'\(([0-9]*)\)')
#RE_OPT_SSH_SEQ = re.compile(
    #r'.*\x1b\]_\;(ssh\|.+?)(\x07|\x1b\\)', re.MULTILINE|re.DOTALL)
## Matches an xterm title sequence
#RE_TITLE_SEQ = re.compile(
    #r'.*\x1b\][0-2]\;(.+?)(\x07|\x1b\\)', re.DOTALL|re.MULTILINE)
# This is used by the raw() function to show control characters
REPLACEMENT_DICT = {
    0: u'^@',
    1: u'^A',
    2: u'^B',
    3: u'^C',
    4: u'^D',
    5: u'^E',
    6: u'^F',
    7: u'^G',
    8: u'^H',
    9: u'^I',
    #10: u'^J', # Newline (\n)
    11: u'^K',
    12: u'^L',
    #13: u'^M', # Carriage return (\r)
    14: u'^N',
    15: u'^O',
    16: u'^P',
    17: u'^Q',
    18: u'^R',
    19: u'^S',
    20: u'^T',
    21: u'^U',
    22: u'^V',
    23: u'^W',
    24: u'^X',
    25: u'^Y',
    26: u'^Z',
    27: u'^[',
    28: u'^\\',
    29: u'^]',
    30: u'^^',
    31: u'^_',
    127: u'^?',
}
# These should match what's in the syslog module (hopefully not platform-dependent)
FACILITIES = {
    'auth': 32,
    'cron': 72,
    'daemon': 24,
    'kern': 0,
    'local0': 128,
    'local1': 136,
    'local2': 144,
    'local3': 152,
    'local4': 160,
    'local5': 168,
    'local6': 176,
    'local7': 184,
    'lpr': 48,
    'mail': 16,
    'news': 56,
    'syslog': 40,
    'user': 8,
    'uucp': 64
}

# Exceptions
class UnknownFacility(Exception):
    """
    Raised if `string_to_syslog_facility` is given a string that doesn't match
    a known syslog facility.
    """
    pass

class MimeTypeFail(Exception):
    """
    Raised by `create_data_uri` if the mimetype of a file could not be guessed.
    """
    pass

class SSLGenerationError(Exception):
    """
    Raised by `gen_self_signed_ssl` if an error is encountered generating a
    self-signed SSL certificate.
    """
    pass

class ChownError(Exception):
    """
    Raised by `recursive_chown` if an OSError is encountered while trying to
    recursively chown a directory.
    """
    pass

# Functions
def noop(*args, **kwargs):
    """Do nothing (i.e. "No Operation")"""
    pass

def write_pid(path):
    """Writes our PID to *path*."""
    try:
        pid = os.getpid()
        with open(path, 'w') as pidfile:
            # Get a non-blocking exclusive lock
            fcntl.flock(pidfile.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
            pidfile.seek(0)
            pidfile.truncate(0)
            pidfile.write(str(pid))
    except:
        raise
    finally:
        try:
            pidfile.close()
        except:
            pass

def read_pid(path):
    """Reads our current PID from *path*."""
    return str(open(path).read())

def remove_pid(path):
    """Removes the PID file at *path*."""
    try:
        os.remove(path)
    except:
        pass

def shell_command(cmd, timeout_duration=5):
    """
    Resets the SIGCHLD signal handler (if necessary), executes *cmd* via
    :func:`commands.getstatusoutput`, then re-enables the SIGCHLD handler (if it
    was set to something other than SIG_DFL).  Returns the result of
    `getstatusoutput` which is a tuple in the form of::

        (exitstatus, output)

    If the command takes longer than *timeout_duration* seconds, it will be
    auto-killed and the following will be returned::

        (255, _("ERROR: Timeout running shell command"))
    """
    from commands import getstatusoutput
    existing_handler = signal.getsignal(signal.SIGCHLD)
    default = (255, _("ERROR: Timeout running shell command"))
    if existing_handler != 0: # Something other than default
        # Reset it to default so getstatusoutput will work properly
        try:
            signal.signal(signal.SIGCHLD, signal.SIG_DFL)
            # "Signal only works in the main thread" - no big deal.  This just
            # means we never needed to call signal in the first place.
            pass
    result = timeout_func(
        getstatusoutput,
        args=(cmd,),
        default=default,
        timeout_duration=timeout_duration
    )
    try:
        signal.signal(signal.SIGCHLD, existing_handler)
    except ValueError:
        # Like above, signal only works from within the main thread but our use
        # of it here would only matter if we were in the main thread.
        pass
    return result

def json_encode(obj):
    """
    On some platforms (CentOS 6.2, specifically) `tornado.escape.json_decode`
    doesn't seem to work just right when it comes to returning unicode strings.
    This is just a wrapper that ensures that the returned string is unicode.
    """
    return to_unicode(_json_encode(obj))

def get_translation():
    """
    Looks inside GATEONE_DIR/server.conf to determine the configured locale and
    returns a matching locale.get_translation function.  Meant to be used like
    this:

        >>> from utils import get_translation
        >>> _ = get_translation()
    """
    gateone_dir = os.path.dirname(os.path.abspath(__file__))
    server_conf = os.path.join(gateone_dir, 'server.conf')
    try:
        locale_str = os.environ.get('LANG', 'en_US').split('.')[0]
        with open(server_conf) as f:
            for line in f:
                if line.startswith('locale'):
                    locale_str = line.split('=')[1].strip()
                    locale_str = locale_str.strip('"').strip("'")
                    break
    except IOError: # server.conf doesn't exist (yet).
        # Fall back to os.environ['LANG']
        # Already set above
        pass
    user_locale = locale.get(locale_str)
    return user_locale.translate

def gen_self_signed_ssl(path=None):
    """
    Generates a self-signed SSL certificate using pyOpenSSL or the openssl
    command depending on what's available,  The resulting key/certificate will
    use the RSA algorithm at 4096 bits.
    """
    try:
        import OpenSSL
        # Direct OpenSSL library calls are better than executing commands...
        gen_self_signed_func = gen_self_signed_pyopenssl
    except ImportError:
        gen_self_signed_func = gen_self_signed_openssl
    try:
        gen_self_signed_func(path=path)
    except SSLGenerationError as e:
        logging.error(_(
            "Error generating self-signed SSL key/certificate: %s" % e))

def gen_self_signed_openssl(path=None):
    """
    This method will generate a secure self-signed SSL key/certificate pair
    (using the `openssl <http://www.openssl.org/docs/apps/openssl.html>`_
    command) saving the result as 'certificate.pem' and 'keyfile.pem' to *path*.
    If *path* is not given the result will be saved in the current working
    directory.  The certificate will be valid for 10 years.
    """
    if not path:
        path = os.path.abspath(os.curdir)
    keyfile_path = "%s/keyfile.pem" % path
    certfile_path = "%s/certificate.pem" % path
    subject = (
        '-subj "/OU=%s (Self-Signed)/CN=Gate One/O=Liftoff Software"' %
        os.uname()[1] # Hostname
    )
    gen_command = (
        "openssl genrsa -aes256 -out %s.tmp -passout pass:password 4096" %
        keyfile_path
    )
    decrypt_key_command = (
        "openssl rsa -in %s.tmp -passin pass:password -out keyfile.pem" %
    )
    csr_command = (
        "openssl req -new -key %s -out temp.csr %s" % (keyfile_path, subject)
    )
    cert_command = (
        "openssl x509 -req "    # Create a new x509 certificate
        "-days 3650 "           # That lasts 10 years
        "-in temp.csr "         # Using the CSR we just generated
        "-signkey %s "          # Sign it with keyfile.pem that we just created
        "-out %s"               # Save it as certificate.pem
    )
    cert_command = cert_command % (keyfile_path, certfile_path)
    exitstatus, output = shell_command(gen_command, 30)
    if exitstatus != 0:
        error_msg = _(
            "An error occurred trying to create private SSL key:\n%s" % output)
        if os.path.exists('%s.tmp' % keyfile_path):
            os.remove('%s.tmp' % keyfile_path)
        raise SSLGenerationError(error_msg)
    exitstatus, output = shell_command(decrypt_key_command, 30)
            "An error occurred trying to decrypt private SSL key:\n%s" % output)
    exitstatus, output = shell_command(csr_command, 30)
            "An error occurred trying to create CSR:\n%s" % output)
        if os.path.exists('temp.csr'):
            os.remove('temp.csr')
    exitstatus, output = shell_command(cert_command, 30)
            "An error occurred trying to create certificate:\n%s" % output)
        if os.path.exists(certfile_path):
            os.remove(certfile_path)
    # Clean up unnecessary leftovers
    os.remove('%s.tmp' % keyfile_path)
    os.remove('temp.csr')

def gen_self_signed_pyopenssl(notAfter=None, path=None):
    """
    (using pyOpenSSL) saving the result as 'certificate.pem' and 'keyfile.pem'
    in *path*.  If *path* is not given the result will be saved in the current
    working directory.  By default the certificate will be valid for 10 years
    but this can be overridden by passing a valid timestamp via the
    *notAfter* argument.

    Examples::

        >>> gen_self_signed_ssl(60 * 60 * 24 * 365) # 1-year certificate
        >>> gen_self_signed_ssl() # 10-year certificate
    """
    try:
            "Error: You do not have pyOpenSSL installed.  Please install "
            "it (sudo pip install pyopenssl.")
    pkey = OpenSSL.crypto.PKey()
    pkey.generate_key(OpenSSL.crypto.TYPE_RSA, 4096)
    # Save the key as 'keyfile.pem':
    with open(keyfile_path, 'w') as f:
        f.write(OpenSSL.crypto.dump_privatekey(
            OpenSSL.crypto.FILETYPE_PEM, pkey))
    cert = OpenSSL.crypto.X509()
    cert.set_serial_number(random.randint(0, sys.maxint))
    cert.gmtime_adj_notBefore(0)
    if notAfter:
        cert.gmtime_adj_notAfter(notAfter)
    else:
        cert.gmtime_adj_notAfter(60 * 60 * 24 * 3650)
    cert.get_subject().CN = '*'
    cert.get_subject().O = 'Gate One Certificate'
    cert.get_issuer().CN = 'Untrusted Authority'
    cert.get_issuer().O = 'Self-Signed'
    cert.set_pubkey(pkey)
    cert.sign(pkey, 'md5')
    with open(certfile_path, 'w') as f:
        f.write(OpenSSL.crypto.dump_certificate(
            OpenSSL.crypto.FILETYPE_PEM, cert))

def none_fix(val):
    """
    If *val* is a string that utlimately means 'none', return None.  Otherwise
    return *val* as-is.  Examples::

        >>> none_fix('none')
        None
        >>> none_fix('0')
        None
        >>> none_fix('whatever')
        'whatever'
    """
    if isinstance(val, basestring) and val.lower() in ['none', '0', 'no']:
        return None
    else:
        return val

def str2bool(val):
    """
    Converts strings like, 'false', 'true', '0', and '1' into their boolean
    equivalents.  If no logical match is found, return False.  Examples::

        >>> str2bool('false')
        False
        >>> str2bool('1')
        True
        >>> st2bool('whatever')
        False
    """
    if isinstance(val, basestring) and val.lower() in ['1', 'true', 'yes']:
        return True
    else:
        return False

def generate_session_id():
    """
    Returns a random, 45-character session ID.  Example:

    .. code-block:: python

        >>> generate_session_id()
        "NzY4YzFmNDdhMTM1NDg3Y2FkZmZkMWJmYjYzNjBjM2Y5O"
        >>>
    """
    import base64
    session_id = base64.b64encode(
        utf8(uuid.uuid4().hex + uuid.uuid4().hex))[:45]
    if bytes != str: # Python 3
        return str(session_id, 'UTF-8')
    return session_id

def mkdir_p(path):
    """
    Pythonic version of "mkdir -p".  Example equivalents::

        >>> import commands
        >>> mkdir_p('/tmp/test/testing') # Does the same thing as below:
        >>> commands.getstatusoutput('mkdir -p /tmp/test/testing')

    .. note:: This doesn't actually call any external commands.
    """
    try:
        os.makedirs(path)
    except OSError as exc: # Python >2.5
        if exc.errno == errno.EEXIST:
            pass
        else: raise

def cmd_var_swap(cmd,
        session=None, session_hash=None, user_dir=None, user=None, time=None):
    """
    Returns *cmd* with special inline variables swapped out for their respective
    argument values.  The special variables are as follows:

        ==============  ==============
        %SESSION%       *session*
        %SESSION_HASH%  *session_hash*
        %USERDIR%       *user_dir*
        %USER%          *user*
        %TIME%          *time*

    This allows for unique or user-specific values to be swapped into command
    line arguments like so::

        ssh_connect.py -M -S '/tmp/gateone/%SESSION%/%r@%L:%p'

    The values passed into this function can be whatever you like.  They don't
    necessarily have to match their intended values.
    """
    if session:
        cmd = cmd.replace(r'%SESSION%', session)
    if session_hash:
        cmd = cmd.replace(r'%SESSION_HASH%', session_hash)
    if user_dir:
        cmd = cmd.replace(r'%USERDIR%', user_dir)
    if user:
        cmd = cmd.replace(r'%USER%', user)
    if time:
        cmd = cmd.replace(r'%TIME%', str(time))
    return cmd

def short_hash(to_shorten):
    """
    Converts *to_shorten* into a really short hash depenendent on the length of
    *to_shorten*.  The result will be safe for use as a file name.
    """
    import struct, binascii, base64
    packed = struct.pack('q', binascii.crc32(utf8(to_shorten)))
    return str(base64.urlsafe_b64encode(packed)).replace('=', '')

def get_process_tree(parent_pid):
    """
    Returns a list of child pids that were spawned from *parent_pid*.

    .. note:: Will include parent_pid in the output list.
    """
    parent_pid = str(parent_pid) # Has to be a string
    ps = which('ps')
    retcode, output = shell_command('%s -ef' % ps)
    out = [parent_pid]
    pidmap = []
    # Construct the pidmap:
    for line in output.splitlines():
        split_line = line.split()
        pid = split_line[1]
        ppid = split_line[2]
        pidmap.append((pid, ppid))
    def walk_pids(pidmap, checkpid):
        """
        Recursively walks the given *pidmap* and updates the *out* variable with
        the child pids of *checkpid*.
        """
        for pid, ppid in pidmap:
            if ppid == checkpid:
                out.append(pid)
                walk_pids(pidmap, pid)
    walk_pids(pidmap, parent_pid)
    return out

def kill_dtached_proc(session, term):
    """
    Kills the dtach processes associated with the given *term* and all its
    sub-processes.  Requires *session* so it can figure out the right
    processess to kill.
    """
    logging.debug('kill_dtached_proc(%s, %s)' % (session, term))
    dtach_socket_name = 'dtach_%s' % term
    to_kill = []
    for f in os.listdir('/proc'):
        pid_dir = os.path.join('/proc', f)
        if os.path.isdir(pid_dir):
            try:
                pid = int(f)
            except ValueError:
                continue # Not a PID
            try:
                with open(os.path.join(pid_dir, 'cmdline')) as f:
                    cmdline = f.read()
                if cmdline and session in cmdline:
                    if dtach_socket_name in cmdline:
                        to_kill.append(pid)
            except Exception as e:
                #logging.debug("Couldn't read the cmdline of PID %s" % pid)
                #logging.debug(e)
                pass # Already dead, no big deal.
                # Uncomment above if you're having problems or think otherwise.
    for pid in to_kill:
        kill_pids = get_process_tree(pid)
        for _pid in kill_pids:
            _pid = int(_pid)
            try:
                os.kill(_pid, signal.SIGTERM)
                pass # Process already died.  Not a problem.

def kill_dtached_proc_macos(session, term):
    """
    A Mac OS-specific implementation of `kill_dtached_proc` since Macs don't
    have /proc.  Seems simpler than :func:`kill_dtached_proc` but actually
    having to call a subprocess is less efficient (due to the sophisticated
    signal handling required by :func:`shell_command`).
    """
    logging.debug('kill_dtached_proc_macos(%s, %s)' % (session, term))
    cmd = (
        "%s -ef | "
        "grep %s/dtach_%s | " # Limit to those matching our session/term combo
        "grep -v grep | " # Get rid of grep from the results (if present)
        "awk '{print $2}' " % (ps, session, term) # Just the PID please
    )
    exitstatus, output = shell_command(cmd)
        pid_to_kill = line.strip() # Get rid of trailing newline
        for pid in get_process_tree(pid_to_kill):
            try:
                os.kill(int(pid), signal.SIGTERM)

def killall(session_dir):
    """
    Kills all running Gate One terminal processes including any detached dtach
    sessions.

    :session_dir: The path to Gate One's session directory.
    """
    sessions = os.listdir(session_dir)
            try:
                if pid == os.getpid():
                    continue # It would be suicide!
            cmdline_path = os.path.join(pid_dir, 'cmdline')
            if os.path.exists(cmdline_path):
                try:
                    with open(cmdline_path) as f:
                        cmdline = f.read()
                except IOError:
                    # Can happen if a process ended as we were looking at it
                    continue
            for session in sessions:
                if session in cmdline:
                    try:
                        os.kill(pid, signal.SIGTERM)
                    except OSError:
                        pass # PID is already dead--great
                elif 'python' in cmdline:
                    if 'gateone.py' in cmdline:
                        try:
                            os.kill(pid, signal.SIGTERM)
                        except OSError:
                            pass # PID is already dead--great

def killall_macos(session_dir):
    """
    A Mac OS X-specific version of `killall` since Macs don't have /proc.
    """
    # TODO: See if there's a better way to keep track of subprocesses so we
    # don't have to enumerate the process table at all.
    for session in sessions:
        cmd = (
            "ps -ef | "
            "grep %s | " # Limit to those matching the session
            "grep -v grep | " # Get rid of grep from the results (if present)
            "awk '{print $2}' | " # Just the PID please
            "xargs kill" % session # Kill em'
        )
        exitstatus, output = shell_command(cmd)

def create_plugin_links(static_dir, templates_dir, plugin_dir):
    """
    Creates symbolic links for all plugins in the ./static/ and ./templates/
    directories.  The equivalent of:

    .. ansi-block::

        \x1b[1;31mroot\x1b[0m@host\x1b[1;34m:~ $\x1b[0m ln -s *plugin_dir*/<plugin>/static *static_dir*/<plugin>
        \x1b[1;31mroot\x1b[0m@host\x1b[1;34m:~ $\x1b[0m ln -s *plugin_dir*/<plugin>/templates *templates_dir*/<plugin>

    This is so plugins can reference files in these directories using the
    following straightforward paths::

        https://<gate one>/static/<plugin name>/<some file>
        https://<gate one>/render/<plugin name>/<some file>

    This function will also remove any dead links if a plugin is removed.
    """
    # Clean up dead links before we do anything else
    for f in os.listdir(static_dir):
        if os.path.islink(f):
            if not os.path.exists(os.readlink(f)):
                os.unlink(f)
    for f in os.listdir(templates_dir):
    # Create symbolic links for each plugin's respective static directory
    for directory in os.listdir(plugin_dir):
        plugin_name = directory
        directory = os.path.join(plugin_dir, directory) # Make absolute
        for f in os.listdir(directory):
            if f == 'static':
                abs_src_path = os.path.join(directory, f)
                abs_dest_path = os.path.join(static_dir, plugin_name)
                try:
                    os.symlink(abs_src_path, abs_dest_path)
                except OSError:
                    pass # Already exists
            if f == 'templates':
                abs_dest_path = os.path.join(templates_dir, plugin_name)
                try:

def get_plugins(plugin_dir):
    """
    Adds plugins' Python files to `sys.path` and returns a dictionary of
    JavaScript, CSS, and Python files contained in *plugin_dir* like so::

        {
            'js': [ // NOTE: These would be be inside *plugin_dir*/static
                '/static/happy_plugin/whatever.js',
                '/static/ssh/ssh.js',
            ],
            'css': ['/cssrender?plugin=bookmarks&template=bookmarks.css'],
            // NOTE: CSS URLs will require '&container=<container>' and '&prefix=<prefix>' to load.
            'py': [ // NOTE: These will get added to sys.path
                'happy_plugin',
                'ssh'
            ],
        }

    \*.js files inside of *plugin_dir*/<the plugin>/static will get automatically
    added to Gate One's index.html like so:

    .. code-block:: html

        {% for jsplugin in jsplugins %}
            <script type="text/javascript" src="{{jsplugin}}"></script>
        {% end %}

    \*.css files will get imported automatically by GateOne.init()
    """
    out_dict = {'js': [], 'css': [], 'py': []}
    plugins_conf_path = plugin_dir + '.conf'
    try:
        enabled_plugins = open(plugins_conf_path).readlines()
        if not enabled_plugins or "*" in enabled_plugins:
            logging.debug(_('Loading all plugins'))
            enabled_plugins = None
    except IOError:
        logging.debug(_('Plugins conf file not found, loading all plugins'))
        enabled_plugins = None

        if enabled_plugins and directory not in enabled_plugins:
            continue

        plugin = directory
        http_static_path = '/static/%s' % plugin
        plugin_files = os.listdir(directory)
        if "__init__.py" in plugin_files:
            out_dict['py'].append(plugin) # Just need the base
            sys.path.insert(0, directory)
        else: # Look for .py files
            for plugin_file in plugin_files:
                if plugin_file.endswith('.py'):
                    plugin_path = os.path.join(directory, plugin_file)
                    sys.path.insert(0, directory)
                    (basename, ext) = os.path.splitext(plugin_path)
                    basename = basename.split('/')[-1]
                    out_dict['py'].append(basename)
        for plugin_file in plugin_files:
            if plugin_file == 'static':
                static_dir = os.path.join(directory, plugin_file)
                for static_file in os.listdir(static_dir):
                    if static_file.endswith('.js'):
                        http_path = os.path.join(http_static_path, static_file)
                        out_dict['js'].append(http_path)
                    elif static_file.endswith('.css'):
                        out_dict['css'].append(http_path)
            if plugin_file == 'templates':
                templates_dir = os.path.join(directory, plugin_file)
                for template_file in os.listdir(templates_dir):
                    if template_file.endswith('.css'):
                        http_path = "/cssrender?plugin=%s&template=%s" % (
                            plugin, template_file)
    # Sort all plugins alphabetically so the order in which they're applied can
    # be controlled somewhat predictably
    out_dict['py'].sort()
    out_dict['js'].sort()
    out_dict['css'].sort()
    return out_dict

def load_plugins(plugins):
    """
    Given a list of *plugins*, imports them.

    .. note::  Assumes they're all in `sys.path`.
    """
    out_list = []
    for plugin in plugins:
        imported = __import__(plugin, None, None, [''])
        out_list.append(imported)
    return out_list

def merge_handlers(handlers):
    """
    Takes a list of Tornado *handlers* like this::

        [
            (r"/", MainHandler),
            (r"/ws", TerminalWebSocket),
            (r"/auth", AuthHandler),
            (r"/style", StyleHandler),
                ...
            (r"/style", SomePluginHandler),
        ]

    ...and returns a list with duplicate handlers removed; giving precedence to
    handlers with higher indexes.  This allows plugins to override Gate One's
    default handlers.  Given the above, this is what would be returned::

        [
                ...
        ]

    This example would replace the default "/style" handler with
    SomePluginHandler; overriding Gate One's default StyleHandler.
    """
    regexes = []
    handlers.reverse()
    for handler in handlers:
        if handler[0] not in regexes:
            regexes.append(handler[0])
            out_list.append(handler)
    out_list.reverse()

# NOTE: This function has been released under the Apache 2.0 license.
# See: http://code.activestate.com/recipes/577894-convert-strings-like-5d-and-60s-to-timedelta-objec/
def convert_to_timedelta(time_val):
    """
    Given a *time_val* (string) such as '5d', returns a `datetime.timedelta` object
    representing the given value (e.g. timedelta(days=5)).  Accepts the
    following '<num><char>' formats:

    =========   ======= ===================
    Character   Meaning Example
    s           Seconds '60s' -> 60 Seconds
    m           Minutes '5m'  -> 5 Minutes
    h           Hours   '24h' -> 24 Hours
    d           Days    '7d'  -> 7 Days

        >>> convert_to_timedelta('7d')
        datetime.timedelta(7)
        >>> convert_to_timedelta('24h')
        datetime.timedelta(1)
        >>> convert_to_timedelta('60m')
        datetime.timedelta(0, 3600)
        >>> convert_to_timedelta('120s')
        datetime.timedelta(0, 120)
    """
    num = int(time_val[:-1])
    if time_val.endswith('s'):
        return timedelta(seconds=num)
    elif time_val.endswith('m'):
        return timedelta(minutes=num)
    elif time_val.endswith('h'):
        return timedelta(hours=num)
    elif time_val.endswith('d'):
        return timedelta(days=num)

def convert_to_bytes(size_val):
    """
    Given a *size_val* (string) such as '100M', returns an integer representing
    an equivalent amount of bytes.  Accepts the following '<num><char>' formats:

    =========== ==========  ===================
    Character   Meaning     Example
    B (or none) Bytes       '100' or '100b' -> 100
    K           Kilobytes   '1k' -> 1024
    M           Megabytes   '1m' -> 1048576
    G           Gigabytes   '1g' -> 1073741824
    T           Terabytes   '1t' -> 1099511627776
    P           Petabytes   '1p' -> 1125899906842624
    E           Exabytes    '1e' -> 1152921504606846976
    Z           Zettabytes  '1z' -> 1180591620717411303424L
    Y           Yottabytes  '7y' -> 1208925819614629174706176L

    .. note:: If no character is given the *size_val* will be assumed to be in bytes.

    .. tip:: All characters will be converted to upper case before conversion (case-insensitive).

        >>> convert_to_bytes('2M')
        2097152
        >>> convert_to_bytes('2g')
        2147483648
    """
    symbols = "BKMGTPEZY"
    letter = size_val[-1:].strip().upper()
    if letter.isdigit(): # Assume bytes
        letter = 'B'
        num = size_val
    else:
        num = size_val[:-1]
    assert num.isdigit() and letter in symbols
    num = float(num)
    prefix = {symbols[0]:1}
    for i, size_val in enumerate(symbols[1:]):
        prefix[size_val] = 1 << (i+1)*10
    return int(num * prefix[letter])

def process_opt_esc_sequence(chars):
    """
    Parse the *chars* passed from :class:`terminal.Terminal` by way of the special,
    optional escape sequence handler (e.g. '<plugin>|<text>') into a tuple of
    (<plugin name>, <text>).  Here's an example::

        >>> process_opt_esc_sequence('ssh|user@host:22')
        ('ssh', 'user@host:22')
    """
    plugin = None
    text = ""
    try:
        plugin, text = chars.split('|')
    except Exception as e:
        pass # Something went horribly wrong!
    return (plugin, text)

def raw(text, replacement_dict=None):
    """
    Returns *text* as a string with special characters replaced by visible
    equivalents using *replacement_dict*.  If *replacement_dict* is None or
    False the global REPLACEMENT_DICT will be used.  Example::

        >>> test = '\\x1b]0;Some xterm title\x07'
        >>> print(raw(test))
        '^[]0;Some title^G'
    """
    if not replacement_dict:
        replacement_dict = REPLACEMENT_DICT
    out = u''
        charnum = ord(char)
        if charnum in replacement_dict.keys():
            out += replacement_dict[charnum]
        else:
            out += char
    return out

def string_to_syslog_facility(facility):
    """
    Given a string (*facility*) such as, "daemon" returns the numeric
    syslog.LOG_* equivalent.
    """
    if facility.lower() in FACILITIES:
        return FACILITIES[facility.lower()]
    else:
        raise UnknownFacility(_(
            "%s does not match a known syslog facility" % repr(facility)))

def create_data_uri(filepath):
    """
    Given a file at *filepath*, return that file as a data URI.

    Raises a `MimeTypeFail` exception if the mimetype could not be guessed.
    """
    mimetype = mimetypes.guess_type(filepath)[0]
    if not mimetype:
        raise MimeTypeFail("Could not guess mime type of: %s" % filepath)
    with open(filepath, 'rb') as f:
        data = f.read()
    encoded = str(base64.b64encode(data)).replace('\n', '')
    if len(encoded) > 65000:
        logging.warn(
            "WARNING: Data URI > 65,000 characters.  You're pushing it buddy!")
    data_uri = "data:%s;base64,%s" % (mimetype, encoded)
    return data_uri

def human_readable_bytes(nbytes):
    """
    Returns *nbytes* as a human-readable string in a similar fashion to how it
    would be displayed by 'ls -lh' or 'df -h'.
    """
    K, M, G, T = 1 << 10, 1 << 20, 1 << 30, 1 << 40
    if nbytes >= T:
        return '%.1fT' % (float(nbytes)/T)
    elif nbytes >= G:
        return '%.1fG' % (float(nbytes)/G)
    elif nbytes >= M:
        return '%.1fM' % (float(nbytes)/M)
    elif nbytes >= K:
        return '%.1fK' % (float(nbytes)/K)
    else:
        return '%d' % nbytes

def which(binary, path=None):
    """
    Returns the full path of *binary* (string) just like the 'which' command.
    Optionally, a *path* (colon-delimited string) may be given to use instead of
    `os.environ` ['PATH'].
    """
    if path:
        paths = path.split(':')
    else:
        paths = os.environ['PATH'].split(':')
    for path in paths:
        if not os.path.exists(path):
            continue
        files = os.listdir(path)
        if binary in files:
            return os.path.join(path, binary)
    return None

def timeout_func(func, args=(), kwargs={}, timeout_duration=10, default=None):
    """
    Sets a timeout on the given function, passing it the given args, kwargs,
    and a *default* value to return in the event of a timeout.  If *default* is
    a function that function will be called in the event of a timeout.
    """
    import threading
    class InterruptableThread(threading.Thread):
        def __init__(self):
            threading.Thread.__init__(self)
            self.result = None

        def run(self):
            try:
                self.result = func(*args, **kwargs)
            except:
                self.result = default

    it = InterruptableThread()
    it.start()
    it.join(timeout_duration)
    if it.isAlive():
        if hasattr(default, '__call__'):
            return default()
        else:
            return default
    else:
        return it.result

def valid_hostname(hostname, allow_underscore=False):
    """
    Returns True if the given *hostname* is valid according to RFC rules.  Works
    with Internationalized Domain Names (IDN) and optionally, hostnames with an
    underscore (if *allow_underscore* is True).

    The rules for hostnames:

        * Must be less than 255 characters.
        * Individual labels (separated by dots) must be <= 63 characters.
        * Only the ASCII alphabet (A-Z) is allowed along with dashes (-) and dots (.).
        * May not start with a dash or a dot.
        * May not end with a dash.
        * If an IDN, when converted to Punycode it must comply with the above.

    IP addresses will be validated according to their well-known specifications.

        >>> valid_hostname('foo.bar.com.') # Standard FQDN
        True
        >>> valid_hostname('2foo') # Short hostname
        True
        >>> valid_hostname('-2foo') # No good:  Starts with a dash
        False
        >>> valid_hostname('host_a') # No good: Can't have underscore
        False
        >>> valid_hostname('host_a', allow_underscore=True) # Now it'll validate
        True
        >>> valid_hostname(u'ジェーピーニック.jp') # Example valid IDN
        True
    """
    # Convert to Punycode if an IDN
    try:
        hostname = hostname.encode('idna')
    except UnicodeError: # Can't convert to Punycode: Bad hostname
    if len(hostname) > 255:
    if hostname[-1:] == ".": # Strip the tailing dot if present
        hostname = hostname[:-1]
    allowed = re.compile("(?!-)[A-Z\d-]{1,63}(?<!-)$", re.IGNORECASE)
    if allow_underscore:
        allowed = re.compile("(?!-)[_A-Z\d-]{1,63}(?<!-)$", re.IGNORECASE)
    return all(allowed.match(x) for x in hostname.split("."))

def recursive_chown(path, uid, gid):
    """Emulates 'chown -R *uid*:*gid* *path*' in pure Python"""
    error_msg = _(
        "Error: Gate One does not have the ability to recursively chown %s to "
        "uid %s/gid %s.  Please ensure that user, %s has write permission to "
        "the directory.")
    try:
        os.chown(path, uid, gid)
    except OSError as e:
        if e.errno in [errno.EACCES, errno.EPERM]:
            raise ChownError(error_msg % (path, uid, gid, repr(os.getlogin())))
        else:
            raise
    for root, dirs, files in os.walk(path):
        for momo in dirs:
            _path = os.path.join(root, momo)
            try:
                os.chown(_path, uid, gid)
            except OSError as e:
                if e.errno in [errno.EACCES, errno.EPERM]:
                    raise ChownError(error_msg % (
                        _path, uid, gid, repr(os.getlogin())))
                else:
                    raise
        for momo in files:
            try:
                else:
                    raise

def drop_privileges(uid='nobody', gid='nogroup', supl_groups=None):
    """
    Drop privileges by changing the current process owner/group to
    *uid*/*gid* (both may be an integer or a string).  If *supl_groups* (list)
    is given the process will be assigned those values as its effective
    supplemental groups.  If *supl_groups* is None it will default to using
    'tty' as the only supplemental group.  Example::

        drop_privileges('gateone', 'gateone', ['tty'])

    This would change the current process owner to gateone/gateone with 'tty' as
    its only supplemental group.

    .. note:: On most Unix systems users must belong to the 'tty' group to create new controlling TTYs which is necessary for 'pty.fork()' to work.

    .. tip:: If you get errors like, "OSError: out of pty devices" it likely means that your OS uses something other than 'tty' as the group owner of the devpts filesystem.  'mount | grep pts' will tell you the owner.
    """
    import pwd, grp
    running_gid = gid
    if not isinstance(uid, int):
        # Get the uid/gid from the name
        running_uid = pwd.getpwnam(uid).pw_uid
    running_uid = uid
    if not isinstance(gid, int):
        running_gid = grp.getgrnam(gid).gr_gid
    if supl_groups:
        for i, group in enumerate(supl_groups):
            # Just update in-place
            if not isinstance(group, int):
                supl_groups[i] = grp.getgrnam(group).gr_gid
        try:
            os.setgroups(supl_groups)
        except OSError as e:
            logging.error(_('Could not set supplemental groups: %s' % e))
            exit()
    # Try setting the new uid/gid
    try:
        os.setgid(running_gid)
        logging.error(_('Could not set effective group id: %s' % e))
        exit()
    try:
        os.setuid(running_uid)
        logging.error(_('Could not set effective user id: %s' % e))
        exit()
    # Ensure a very convervative umask
    new_umask = 0o77
    old_umask = os.umask(new_umask)
    final_uid = os.getuid()
    final_gid = os.getgid()
    human_supl_groups = []
    for group in supl_groups:
        human_supl_groups.append(grp.getgrgid(group).gr_name)
    logging.info(_(
        'Running as user/group, "%s/%s" with the following supplemental groups:'
        ' %s' % (pwd.getpwuid(final_uid)[0], grp.getgrgid(final_gid)[0],
                 ",".join(human_supl_groups))
    ))

# Misc
if MACOS: # Apply mac-specific stuff
    kill_dtached_proc = kill_dtached_proc_macos
    killall = killall_macos

# Used in case bell.ogg can't be found or can't be converted into a data URI
fallback_bell = "data:audio/ogg;base64,T2dnUwACAAAAAAAAAABCw2VcAAAAAEKIowgBHgF2b3JiaXMAAAAAAUSsAAAAAAAAgDgBAAAAAAC4AU9nZ1MAAAAAAAAAAAAAQsNlXAEAAACMEDEUDq3///////////////+BA3ZvcmJpcy0AAABYaXBoLk9yZyBsaWJWb3JiaXMgSSAyMDEwMTEwMSAoU2NoYXVmZW51Z2dldCkEAAAAFAAAAEFSVElTVD1EYW4gTWNEb3VnYWxsCQAAAERBVEU9MjAxMRQAAABUSVRMRT1HYXRlIE9uZSBCZWVwMS8AAABDT01NRU5UUz1Db3B5cmlnaHQgTGlmdG9mZiBTb2Z0d2FyZSBDb3Jwb3JhdGlvbgEFdm9yYmlzIkJDVgEAQAAAJHMYKkalcxaEEBpCUBnjHELOa+wZQkwRghwyTFvLJXOQIaSgQohbKIHQkFUAAEAAAIdBeBSEikEIIYQlPViSgyc9CCGEiDl4FIRpQQghhBBCCCGEEEIIIYRFOWiSgydBCB2E4zA4DIPlOPgchEU5WBCDJ0HoIIQPQriag6w5CCGEJDVIUIMGOegchMIsKIqCxDC4FoQENSiMguQwyNSDC0KImoNJNfgahGdBeBaEaUEIIYQkQUiQgwZByBiERkFYkoMGObgUhMtBqBqEKjkIH4QgNGQVAJAAAKCiKIqiKAoQGrIKAMgAABBAURTHcRzJkRzJsRwLCA1ZBQAAAQAIAACgSIqkSI7kSJIkWZIlWZIlWZLmiaosy7Isy7IsyzIQGrIKAEgAAFBRDEVxFAcIDVkFAGQAAAigOIqlWIqlaIrniI4IhIasAgCAAAAEAAAQNENTPEeURM9UVde2bdu2bdu2bdu2bdu2bVuWZRkIDVkFAEAAABDSaWapBogwAxkGQkNWAQAIAACAEYowxIDQkFUAAEAAAIAYSg6iCa0535zjoFkOmkqxOR2cSLV5kpuKuTnnnHPOyeacMc4555yinFkMmgmtOeecxKBZCpoJrTnnnCexedCaKq0555xxzulgnBHGOeecJq15kJqNtTnnnAWtaY6aS7E555xIuXlSm0u1Oeecc84555xzzjnnnOrF6RycE84555yovbmWm9DFOeecT8bp3pwQzjnnnHPOOeecc84555wgNGQVAAAEAEAQho1h3CkI0udoIEYRYhoy6UH36DAJGoOcQurR6GiklDoIJZVxUkonCA1ZBQAAAgBACCGFFFJIIYUUUkghhRRiiCGGGHLKKaeggkoqqaiijDLLLLPMMssss8w67KyzDjsMMcQQQyutxFJTbTXWWGvuOeeag7RWWmuttVJKKaWUUgpCQ1YBACAAAARCBhlkkFFIIYUUYogpp5xyCiqogNCQVQAAIACAAAAAAE/yHNERHdERHdERHdERHdHxHM8RJVESJVESLdMyNdNTRVV1ZdeWdVm3fVvYhV33fd33fd34dWFYlmVZlmVZlmVZlmVZlmVZliA0ZBUAAAIAACCEEEJIIYUUUkgpxhhzzDnoJJQQCA1ZBQAAAgAIAAAAcBRHcRzJkRxJsiRL0iTN0ixP8zRPEz1RFEXTNFXRFV1RN21RNmXTNV1TNl1VVm1Xlm1btnXbl2Xb933f933f933f933f931dB0JDVgEAEgAAOpIjKZIiKZLjOI4kSUBoyCoAQAYAQAAAiuIojuM4kiRJkiVpkmd5lqiZmumZniqqQGjIKgAAEABAAAAAAAAAiqZ4iql4iqh4juiIkmiZlqipmivKpuy6ruu6ruu6ruu6ruu6ruu6ruu6ruu6ruu6ruu6ruu6ruu6rguEhqwCACQAAHQkR3IkR1IkRVIkR3KA0JBVAIAMAIAAABzDMSRFcizL0jRP8zRPEz3REz3TU0VXdIHQkFUAACAAgAAAAAAAAAzJsBTL0RxNEiXVUi1VUy3VUkXVU1VVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVU3TNE0TCA1ZCQAAAQDQWnPMrZeOQeisl8gopKDXTjnmpNfMKIKc5xAxY5jHUjFDDMaWQYSUBUJDVgQAUQAAgDHIMcQccs5J6iRFzjkqHaXGOUepo9RRSrGmWjtKpbZUa+Oco9RRyiilWkurHaVUa6qxAACAAAcAgAALodCQFQFAFAAAgQxSCimFlGLOKeeQUso55hxiijmnnGPOOSidlMo5J52TEimlnGPOKeeclM5J5pyT0kkoAAAgwAEAIMBCKDRkRQAQJwDgcBxNkzRNFCVNE0VPFF3XE0XVlTTNNDVRVFVNFE3VVFVZFk1VliVNM01NFFVTE0VVFVVTlk1VtWXPNG3ZVFXdFlXVtmVb9n1XlnXdM03ZFlXVtk1VtXVXlnVdtm3dlzTNNDVRVFVNFFXXVFXbNlXVtjVRdF1RVWVZVFVZdl1Z11VX1n1NFFXVU03ZFVVVllXZ1WVVlnVfdFXdVl3Z11VZ1n3b1oVf1n3CqKq6bsqurquyrPuyLvu67euUSdNMUxNFVdVEUVVNV7VtU3VtWxNF1xVV1ZZFU3VlVZZ9X3Vl2ddE0XVFVZVlUVVlWZVlXXdlV7dFVdVtVXZ933RdXZd1XVhmW/eF03V1XZVl31dlWfdlXcfWdd/3TNO2TdfVddNVdd/WdeWZbdv4RVXVdVWWhV+VZd/XheF5bt0XnlFVdd2UXV9XZVkXbl832r5uPK9tY9s+sq8jDEe+sCxd2za6vk2Ydd3oG0PhN4Y007Rt01V13XRdX5d13WjrulBUVV1XZdn3VVf2fVv3heH2fd8YVdf3VVkWhtWWnWH3faXuC5VVtoXf1nXnmG1dWH7j6Py+MnR1W2jrurHMvq48u3F0hj4CAAAGHAAAAkwoA4WGrAgA4gQAGIScQ0xBiBSDEEJIKYSQUsQYhMw5KRlzUkIpqYVSUosYg5A5JiVzTkoooaVQSkuhhNZCKbGFUlpsrdWaWos1hNJaKKW1UEqLqaUaW2s1RoxByJyTkjknpZTSWiiltcw5Kp2DlDoIKaWUWiwpxVg5JyWDjkoHIaWSSkwlpRhDKrGVlGIsKcXYWmy5xZhzKKXFkkpsJaVYW0w5thhzjhiDkDknJXNOSiiltVJSa5VzUjoIKWUOSiopxVhKSjFzTkoHIaUOQkolpRhTSrGFUmIrKdVYSmqxxZhzSzHWUFKLJaUYS0oxthhzbrHl1kFoLaQSYyglxhZjrq21GkMpsZWUYiwp1RZjrb3FmHMoJcaSSo0lpVhbjbnGGHNOseWaWqy5xdhrbbn1mnPQqbVaU0y5thhzjrkFWXPuvYPQWiilxVBKjK21WluMOYdSYisp1VhKirXFmHNrsfZQSowlpVhLSjW2GGuONfaaWqu1xZhrarHmmnPvMebYU2s1txhrTrHlWnPuvebWYwEAAAMOAAABJpSBQkNWAgBRAAAEIUoxBqFBiDHnpDQIMeaclIox5yCkUjHmHIRSMucglJJS5hyEUlIKpaSSUmuhlFJSaq0AAIACBwCAABs0JRYHKDRkJQCQCgBgcBzL8jxRNFXZdizJ80TRNFXVth3L8jxRNE1VtW3L80TRNFXVdXXd8jxRNFVVdV1d90RRNVXVdWVZ9z1RNFVVdV1Z9n3TVFXVdWVZtoVfNFVXdV1ZlmXfWF3VdWVZtnVbGFbVdV1Zlm1bN4Zb13Xd94VhOTq3buu67/vC8TvHAADwBAcAoAIbVkc4KRoLLDRkJQCQAQBAGIOQQUghgxBSSCGlEFJKCQAAGHAAAAgwoQwUGrISAIgCAAAIkVJKKY2UUkoppZFSSimllBJCCCGEEEIIIYQQQgghhBBCCCGEEEIIIYQQQgghhBBCCAUA+E84APg/2KApsThAoSErAYBwAADAGKWYcgw6CSk1jDkGoZSUUmqtYYwxCKWk1FpLlXMQSkmptdhirJyDUFJKrcUaYwchpdZarLHWmjsIKaUWa6w52BxKaS3GWHPOvfeQUmsx1lpz772X1mKsNefcgxDCtBRjrrn24HvvKbZaa809+CCEULHVWnPwQQghhIsx99yD8D0IIVyMOecehPDBB2EAAHeDAwBEgo0zrCSdFY4GFxqyEgAICQAgEGKKMeecgxBCCJFSjDnnHIQQQiglUoox55yDDkIIJWSMOecchBBCKKWUjDHnnIMQQgmllJI55xyEEEIopZRSMueggxBCCaWUUkrnHIQQQgillFJK6aCDEEIJpZRSSikhhBBCCaWUUkopJYQQQgmllFJKKaWEEEoopZRSSimllBBCKaWUUkoppZQSQiillFJKKaWUkkIppZRSSimllFJSKKWUUkoppZRSSgmllFJKKaWUlFJJBQAAHDgAAAQYQScZVRZhowkXHoBCQ1YCAEAAABTEVlOJnUHMMWepIQgxqKlCSimGMUPKIKYpUwohhSFziiECocVWS8UAAAAQBAAICAkAMEBQMAMADA4QPgdBJ0BwtAEACEJkhkg0LASHB5UAETEVACQmKOQCQIXFRdrFBXQZ4IIu7joQQhCCEMTiAApIwMEJNzzxhifc4ASdolIHAQAAAABwAAAPAADHBRAR0RxGhsYGR4fHB0hIAAAAAADIAMAHAMAhAkRENIeRobHB0eHxARISAAAAAAAAAAAABAQEAAAAAAACAAAABARPZ2dTAARdbAAAAAAAAELDZVwCAAAA/HXUPh4pH418bl5YT1NwRjEyMjZSXlFIREdJPi4BAQEBAQFk5Xux+dfV3OoNzQgA5Pbn43/P3d3VXes5f8r9t9LczlNqTTddVZqmKXzn09+b2/PdN0EAAAAgJtP0fs+LVJBTK/YjFq33FABaeh4zH5cpkUqwf47oZrACACTgA2C0IAsAMwDYG3ZDAAAAAIA+AHthPY6sWm3sGGOMEfD4+mitFRQFFDQB2vL8gXff/v/WG28+4ogj3vxrzHkTF9VK/tKotbpmdaMeaq21JunketaKDQz9lA0Mu7moOVcxDxa8MhENvtapACACMGLgGwDKqwcFjXQAsAB+it655982MVPieeJ+JlRYAkAAALDhCgBAgw0AUwDAPR9BgAEAAAAAiKYA4AXQxX27dA04AYWCNnAKKOjgQIFAckdH0qmtZXRJmN6vAazMBFTdYRYEgdg0cu0CziTNbWVWkZ+es8lU+5rFZWAPUP5vAQAACVsBWzAGACYAHoqerKcUuUQbjfedflhYAgAAAByyAEAAAHZNEQAAAAAAAFy/C4AOQHdmy9Uc76o0CTheVaBeIHJLN2oLNQL7yMZRAOAEU/FQaYBAzKrvOjlzCRAbvQIIAQDR0Nfx1nIEV/L7GB6WGrOFXoXZQAceil7sh1S6uEvhe6cOFZYAAAAAG74CANBAArAQDAAAAAAAABzuKgGoA0CoaipdsYgIAO6AAxDMzEwTAEarZQIA4PZP8wkAtg8hswCgAXTKLLgA4FJXKB7EecBxAvAA/nk+wpsU46LMwvukjV7kyhIAAABgCmYAYBUBAAAAAACwgPuzBMARQEmdilDEHlopBF9004mAcMnHpQCCKaH8VPNRZAUsSfzO9oAOtgYAAADwSZieUUAHAP5pvgRXye5iT7zvTI8rSwAAAAAyIgIAAAAAAAAB1+8BwG2AliD0P2swCThs5toAAHxamUnAuvzVLMBA8dLSAgDgDI9N2AKAc5TGapAwUANeKT6yNynziFmwD7ACAACeugrQEC0EAAAAAAAIANo9BEAJqzfBo2hGYZWOFHZmyGZ4WxlQT8YhAAAslu0hHpQ78A4BMN8EQMVSDoAlBIWJB5QCAB6I3XP3t+7ivYt7Hr3tZ6KSaW8CS+d//R+VAQBgY48FwA9tMAAAAAAADmnCnC3/9jiNBeAXgB3PdQVAIEZskEhoPQIAnn8CwFnPvtSYrLH9OyczJgt/TxuABKh0Ru9wOD0AfhkSj1e0TrLneccNAEz+6D2zN2n7Ef8L9l0PrAAAgP3F1EBDAAMBAAAAAAAAgOdMAKDw5SkAKySQH5WPWLxownEAoCgAAMwdoB8FPoHfJQDwcWQCfik+w5v0dcQ/mAdYAQAANgEEAQAAAAAAAAAAeN4KAGBKBwC9EpAoYFA/uzuwCROAAn4p3lo3qSwJwd5VwAoAAHABwEAAAAAAAAAAAABa320AAOAOAJ4lgM0HQsrnEwA9AQUAHvk9w5vUdSQI89SAFQAA4HUBgAoAAAAAAAAAAADcvAkAgPcCgA4IQuIS2FvPS5FADQB+uN1aV8ks8YO6T6YBKwAAwAwAAQIAAAAAAAAAAoDnz00AgCoBgDIUCCfKBic4jefbFe4B6AC+R/0YWj/zFt/JCSftsAFYAQAAvpXA8EEFAAAAAAAABAB7NwAATYcCsHayxKypNjB04bnrWyBbNQoAAMsGyMnWySsMBOTPHUBi8TkHAAAAWTgmHpe8ZJ577KLt5pwjrllHxvxNSGT46e8mAHh7CQzfIoABAAAAAKBh/wbOcWxL7gKgIEsAKE9O1kwADDCnItg5VJggQGfouzPWAvD+MQBk2XZegdHBO0uofAUNAADYBT7nfOf2s92CiBP+iWEFAAC4EzB8MAAAAAAAAAAAcw8AYPcEwGUtNWDKbav5hb2zvE5QDY4AAL3yftfrudUB4Ev0aI8+u5kMkP1ZnaMKSAYYBx4XfWT3vR6REreT0SywAgAA1AEQfTAAAAAAAAAgAdAMAEArQBQ8SSp6bx4dAXfM1i+ho5frBWBgWIsb6xBUH5QiDCUmAACABR4Xfeb3b3tFw3anPxJiBQAA+B6ASAADAAAAAAAAEoB9OwAAmKEAlCl26CWlA/D6ik5ggokA0BMgAg66s0B4wwCggMEA3vZ8Du7f+Yojejrpr2EFAAConoDtgwEAAAAAAAAAmE8BACA4ABRXDetkqFMFNHp4LICqAPZEvmOXeZBXm5UBEuBlAAAA1ANetvweNkqouwgtOE7qjAhWAACAKQEMBgAAAAAAAAAATOcAAGKSAcDBbnhJRQXsscOKA8CvAIB1zadQQZ0x3Eh9H/IsZHin+yoA/pX8GehSYontx3SnemEgOLCBz3ckgP0ugIEAAAAAAAAAAFsJyw+aehMAgGX9AYARTRsVAEiuDQAAIAEe3AA+lvyXq5JPF/9/Dk7haOKwAdj5RAIDAAAAAAAAAAAAAA9fuppWB4CXP2wCAAYDDg4ODg4O"

#
#
#   Thanks to Alan Schmitz for contributing this module!

# Meta
__doc__ = """
This authentication module is built on top of python-pam (or PAM).  The latest
version of which can be found here:  ftp://ftp.pangalactic.org/pub/tummy/ or if
that doesn't work try:  http://packages.debian.org/lenny/python-pam

It was originally written by Alan Schmitz.

The only non-obvious aspect of this module is that the pam_realm setting is only
used when the user is asked to authenticate and when the user's information is
stored in the 'users' directory.  It isn't actually used in any part of the
authentication (PAM doesn't take a "realm" setting).
"""

# Standard library modules
import base64

# 3rd party modules
import PAM
import tornado.httpserver
import tornado.web

class PAMAuthMixin(tornado.web.RequestHandler):
    """
    This is used by PAMAuthHandler in auth.py to authenticate users via PAM.
    """
    def initialize(self):
        """
        Print out helpful error messages if the requisite settings aren't
        configured.
        """
        self.require_setting("pam_realm", "PAM Realm")
        self.require_setting("pam_service", "PAM Service")

    def get_authenticated_user(self, callback):
        """
        Processes the client's Authorization header and call self.auth_basic()
        """
        auth_header = self.request.headers.get('Authorization')
        if auth_header and auth_header.startswith('Basic '):
            self.auth_basic(auth_header, callback)

    def auth_basic(self, auth_header, callback):
        """
        Perform Basic authentication using self.settings['pam_realm'].
        """
        auth_decoded = base64.decodestring(auth_header[6:])
        username, password = auth_decoded.split(':', 2)

        def _pam_conv(auth, query_list, user_data=None):
            resp = []
            for i in range(len(query_list)):
                query, qtype = query_list[i]
                if qtype == PAM.PAM_PROMPT_ECHO_ON:
                    resp.append((username, 0))
                elif qtype == PAM.PAM_PROMPT_ECHO_OFF:
                    resp.append((password, 0))
                else:
                    return None
            return resp

        pam_auth = PAM.pam()
        pam_auth.start(self.settings['pam_service'])
        pam_auth.set_item(PAM.PAM_USER, username)
        pam_auth.set_item(PAM.PAM_TTY, 'console')
        pam_auth.set_item(PAM.PAM_CONV, _pam_conv)
        try:
            pam_auth.authenticate()
            pam_auth.acct_mgmt()
        except Exception as e: # Basic auth failed
            if self.settings['debug']:
                print(e) # Very useful for debugging Kerberos errors
            return self.authenticate_redirect()
        # NOTE: Basic auth just gives us the username without the @REALM part
        #       so we have to add it:
        user = "%s@%s" % (username, self.settings['pam_realm'])
        callback(user)

    def authenticate_redirect(self):
        """
        Informs the browser that this resource requires authentication (status
        code 401) which should prompt the browser to reply with credentials.

        The browser will be informed that we support Basic auth.
        """
        if self._headers_written:
            raise Exception('Headers have already been written')
        self.set_status(401)
        self.add_header(
            "WWW-Authenticate",
            'Basic realm="%s"' % self.settings['pam_realm']
        )
        self.finish()

#
#

# Meta

About This Module
=================
This crux of this module is the Terminal class which is a pure-Python
implementation of the quintessential Unix terminal emulator.  It does its best
to emulate an xterm and along with that comes support for the majority of the
relevant portions of ECMA-48.  This includes support for emulating varous VT-*
terminal types as well as the "linux" terminal type.

The Terminal class's VT-* emulation support is not complete but it should
suffice for most terminal emulation needs (e.g. all your typical command line
programs should work wonderfully).  If something doesn't look quite right or you
need support for certain modes added please feel free to open a ticket on Gate
One's issue tracker:  https://github.com/liftoff/GateOne/issues

Note that Terminal was written from scratch in order to be as fast as possible.
It is extensively commented and implements some interesting patterns in order to
maximize execution speed (most notably for things that loop).  Some bits of code
may seem "un-Pythonic" and/or difficult to grok but understand that this is
probably due to optimizations.  If you know "a better way" please feel free to
submit a patch, open a ticket, or send us an email.  There's a reason why open
source software is a superior development model!

Supported Emulation Types
-------------------------
Without any special mode settings or parameters Terminal should effectively
emulate the following terminal types:

 * xterm (the most important one)
 * ECMA-48/ANSI X3.64
 * Nearly all the VT-* types:  VT-52, VT-100, VT-220, VT-320, VT-420, and VT-520
 * Linux console ("linux")

If you want Terminal to support something else or it's missing a feature from
any given terminal type please `let us know <https://github.com/liftoff/GateOne/issues/new>`_.
We'll implement it!

What Terminal Doesn't Do
------------------------
The Terminal class is meant to emulate the display portion of a given terminal.
It does not translate keystrokes into escape sequences or special control
codes--you'll have to take care of that in your application (or at the
client-side like Gate One).  It does, however, keep track of many
keystroke-specific modes of operation such as Application Cursor Keys and the G0
and G1 charset modes *with* callbacks that can be used to notify your
application when such things change.

Special Considerations
----------------------
Many methods inside Terminal start with an underscore.  This was done to
indicate that such methods shouldn't be called directly (from a program that
imported the module).  If it was thought that a situation might arise where a
method could be used externally by a controlling program, the underscore was
omitted.

Asynchronous Use
----------------
To support asynchronous usage (and make everything faster), Terminal was written
to support extensive callbacks that are called when certain events are
encountered.  Here are the events and their callbacks:

.. _callback_constants:

====================================    ========================================================================
Callback Constant (ID)                  Called when...
:attr:`terminal.CALLBACK_SCROLL_UP`     The terminal is scrolled up (back).
:attr:`terminal.CALLBACK_CHANGED`       The screen is changed/updated.
:attr:`terminal.CALLBACK_CURSOR_POS`    The cursor position changes.
:attr:`terminal.CALLBACK_DSR`           A Device Status Report (DSR) is requested (via the DSR escape sequence).
:attr:`terminal.CALLBACK_TITLE`         The terminal title changes (xterm-style)
:attr:`terminal.CALLBACK_BELL`          The bell character (^G) is encountered.
:attr:`terminal.CALLBACK_OPT`           The special optional escape sequence is encountered.
:attr:`terminal.CALLBACK_MODE`          The terminal mode setting changes (e.g. use alternate screen buffer).

Note that CALLBACK_DSR is special in that it in most cases it will be called with arguments.  See the code for examples of how and when this happens.

Also, in most cases it is unwise to override CALLBACK_MODE since this method is primarily meant for internal use within the Terminal class.

Using Terminal
--------------
Gate One makes extensive use of the Terminal class and its callbacks.  So that's
a great place to look for specific examples (gateone.py and termio.py,
specifically).  Having said that, implementing Terminal is pretty
straightforward::

    >>> import terminal
    >>> term = terminal.Terminal(24, 80)
    >>> term.write("This text will be written to the terminal screen.")
    >>> term.dump()
    [u'This text will be written to the terminal screen.                               ',
    <snip>
    u'                                                                                ']

Here's an example with some basic callbacks:

    >>> def mycallback():
    ...     "This will be called whenever the screen changes."
    ...     print("Screen update! Perfect time to dump the terminal screen.")
    ...     print(term.dump()[0]) # Only need to see the top line for this demo =)
    ...     print("Just dumped the screen.")
    >>> term.callbacks[term.CALLBACK_CHANGED] = mycallback
    >>> term.write("This should result in mycallback() being called")
    Screen update! Perfect time to dump the terminal screen.
    This should result in mycallback() being called
    Just dumped the screen.

.. note:: In testing Gate One it was determined that it is faster to perform the conversion of a terminal screen to HTML on the server side than it is on the client side (via JavaScript anyway).

About The Scrollback Bufffer
----------------------------
The Terminal class implements a scrollback buffer.  Here's how it works:
Whenever a :meth:`Terminal.scroll_up` event occurs, the line (or lines) that
will be removed from the top of the screen will be placed into
:attr:`Terminal.scrollback_buf`. Then whenever :meth:`Terminal.dump_html` is
called the scrollback buffer will be returned along with the screen output and
reset to an empty state.

Why do this?  In the event that a very large :meth:`Terminal.write` occurs (e.g.
'ps aux'), it gives the controlling program the ability to capture what went
past the screen without some fancy tracking logic surrounding
:meth:`Terminal.write`.

More information about how this works can be had by looking at the
:meth:`Terminal.dump_html` function itself.

.. note:: There's more than one function that empties :attr:`Terminal.scrollback_buf` when called.  You'll just have to have a look around =)

"""

import re, logging, base64, StringIO, codecs, unicodedata, tempfile
from array import array
from datetime import datetime, timedelta
from collections import defaultdict
from itertools import imap, izip

# Inernationalization support
import gettext
gettext.install('terminal')

try:
    # We need PIL to detect image types and get their dimensions.  Without the
    # dimenions, the browser will render the terminal screen much slower than
    # normal.  Without PIL images will be displayed simply as:
    #   <i>Image file</i>
    from PIL import Image
except ImportError:
    Image = None
    logging.warning(_(
        "Could not import the Python Imaging Library (PIL) "
        "so images will not be displayed in the terminal"))

# Globals
CALLBACK_SCROLL_UP = 1    # Called after a scroll up event (new line)
CALLBACK_CHANGED = 2      # Called after the screen is updated
CALLBACK_CURSOR_POS = 3   # Called after the cursor position is updated
# <waives hand in air> You are not concerned with the number 4
CALLBACK_DSR = 5          # Called when a DSR requires a response
# NOTE: CALLBACK_DSR must accept 'response' as either the first argument or
#       as a keyword argument.
CALLBACK_TITLE = 6        # Called when the terminal sets the window title
CALLBACK_BELL = 7         # Called after ASCII_BEL is encountered.
CALLBACK_OPT = 8 # Called when we encounter the optional ESC sequence
# NOTE: CALLBACK_OPT must accept 'chars' as either the first argument or as
#       a keyword argument.
CALLBACK_MODE = 9 # Called when the terminal mode changes (e.g. DECCKM)
CALLBACK_RESET = 10 # Called when a terminal reset (^[[!p) is encountered
CALLBACK_LEDS = 11 # Called when the state of the LEDs changes

# These are for HTML output:
RENDITION_CLASSES = defaultdict(lambda: None, {
    0: 'reset', # Special: Return everything to defaults
    1: 'bold',
    2: 'dim',
    3: 'italic',
    4: 'underline',
    5: 'blink',
    6: 'fastblink',
    7: 'reverse',
    8: 'hidden',
    9: 'strike',
    10: 'resetfont', # NOTE: The font renditions don't do anything right now
    11: 'font11', # Mostly because I have no idea what they are supposed to look
    12: 'font12', # like.
    13: 'font13',
    14: 'font14',
    15: 'font15',
    16: 'font16',
    17: 'font17',
    18: 'font18',
    19: 'font19',
    20: 'fraktur',
    21: 'boldreset',
    22: 'dimreset',
    23: 'italicreset',
    24: 'underlinereset',
    27: 'reversereset',
    28: 'hiddenreset',
    29: 'strikereset',
    # Foregrounds
    30: 'f0', # Black
    31: 'f1', # Red
    32: 'f2', # Green
    33: 'f3', # Yellow
    34: 'f4', # Blue
    35: 'f5', # Magenta
    36: 'f6', # Cyan
    37: 'f7', # White
    38: '', # 256-color support uses this like so: \x1b[38;5;<color num>sm
    # Backgrounds
    40: 'b0', # Black
    41: 'b1', # Red
    42: 'b2', # Green
    43: 'b3', # Yellow
    44: 'b4', # Blue
    45: 'b5', # Magenta
    46: 'b6', # Cyan
    47: 'b7', # White
    48: '', # 256-color support uses this like so: \x1b[48;5;<color num>sm
    49: 'backgroundreset', # Special: Set BG to default
    51: 'frame',
    52: 'encircle',
    53: 'overline',
    60: 'rightline',
    61: 'rightdoubleline',
    62: 'leftline',
    63: 'leftdoubleline',
    # aixterm colors (aka '16 color support').  They're supposed to be 'bright'
    # versions of the first 8 colors (hence the 'b').
    # 'Bright' Foregrounds
    90: 'bf0', # Bright black (whatever that is =)
    91: 'bf1', # Bright red
    92: 'bf2', # Bright green
    93: 'bf3', # Bright yellow
    94: 'bf4', # Bright blue
    95: 'bf5', # Bright magenta
    96: 'bf6', # Bright cyan
    97: 'bf7', # Bright white
    # 'Bright' Backgrounds
    100: 'bb0', # Bright black
    101: 'bb1', # Bright red
    102: 'bb2', # Bright green
    103: 'bb3', # Bright yellow
    104: 'bb4', # Bright blue
    105: 'bb5', # Bright magenta
    106: 'bb6', # Bright cyan
    107: 'bb7' # Bright white
})
# Generate the dict of 256-color (xterm) foregrounds and backgrounds
for i in xrange(256):
    RENDITION_CLASSES[(i+1000)] = "fx%s" % i
    RENDITION_CLASSES[(i+10000)] = "bx%s" % i
del i # Cleanup

try:
    unichr(0x10000) # Will throw a ValueError on narrow Python builds
    SPECIAL = 1048576 # U+100000 or unichr(SPECIAL) (start of Plane 16)
except:
    SPECIAL = 63561

def handle_special(e):
    """
    Used in conjunction with :py:func:`codecs.register_error`, will replace
    special ascii characters such as 0xDA and 0xc4 (which are used by ncurses)
    with their Unicode equivalents.
    """
    # TODO: Get this using curses special characters when appropriate
    curses_specials = {
        # NOTE: When $TERM is set to "Linux" these end up getting used by things
        #       like ncurses-based apps.  In other words, it makes a whole lot
        #       of ugly look pretty again.
        0xda: u'┌', # ACS_ULCORNER
        0xc0: u'└', # ACS_LLCORNER
        0xbf: u'┐', # ACS_URCORNER
        0xd9: u'┘', # ACS_LRCORNER
        0xb4: u'├', # ACS_RTEE
        0xc3: u'┤', # ACS_LTEE
        0xc1: u'┴', # ACS_BTEE
        0xc2: u'┬', # ACS_TTEE
        0xc4: u'─', # ACS_HLINE
        0xb3: u'│', # ACS_VLINE
        0xc5: u'┼', # ACS_PLUS
        0x2d: u'', # ACS_S1
        0x5f: u'', # ACS_S9
        0x60: u'◆', # ACS_DIAMOND
        0xb2: u'▒', # ACS_CKBOARD
        0xf8: u'°', # ACS_DEGREE
        0xf1: u'±', # ACS_PLMINUS
        0xf9: u'•', # ACS_BULLET
        0x3c: u'←', # ACS_LARROW
        0x3e: u'→', # ACS_RARROW
        0x76: u'↓', # ACS_DARROW
        0x5e: u'↑', # ACS_UARROW
        0xb0: u'⊞', # ACS_BOARD
        0x0f: u'⨂', # ACS_LANTERN
        0xdb: u'█', # ACS_BLOCK
    }
    specials = {
# Note to self:  Why did I bother with these overly descriptive comments?  Ugh
# I've been staring at obscure symbols far too much lately ⨀_⨀
        128: u'€', # Euro sign
        129: u' ', # Unknown (Using non-breaking spaces for all unknowns)
        130: u'‚', # Single low-9 quotation mark
        131: u'ƒ', # Latin small letter f with hook
        132: u'„', # Double low-9 quotation mark
        133: u'…', # Horizontal ellipsis
        134: u'†', # Dagger
        135: u'‡', # Double dagger
        136: u'ˆ', # Modifier letter circumflex accent
        137: u'‰', # Per mille sign
        138: u'Š', # Latin capital letter S with caron
        139: u'‹', # Single left-pointing angle quotation
        140: u'Œ', # Latin capital ligature OE
        141: u' ', # Unknown
        142: u'Ž', #  Latin captial letter Z with caron
        143: u' ', # Unknown
        144: u' ', # Unknown
        145: u'‘', # Left single quotation mark
        146: u'’', # Right single quotation mark
        147: u'“', # Left double quotation mark
        148: u'”', # Right double quotation mark
        149: u'•', # Bullet
        150: u'–', # En dash
        151: u'—', # Em dash
        152: u'˜', # Small tilde
        153: u'™', # Trade mark sign
        154: u'š', # Latin small letter S with caron
        155: u'›', #  Single right-pointing angle quotation mark
        156: u'œ', # Latin small ligature oe
        157: u'Ø', # Upper-case slashed zero--using same as empty set (216)
        158: u'ž', # Latin small letter z with caron
        159: u'Ÿ', # Latin capital letter Y with diaeresis
        160: u' ', # Non-breaking space
        161: u'¡', # Inverted exclamation mark
        162: u'¢', # Cent sign
        163: u'£', # Pound sign
        164: u'¤', # Currency sign
        165: u'¥', # Yen sign
        166: u'¦', # Pipe, Broken vertical bar
        167: u'§', # Section sign
        168: u'¨', # Spacing diaeresis - umlaut
        169: u'©', # Copyright sign
        170: u'ª', # Feminine ordinal indicator
        171: u'«', # Left double angle quotes
        172: u'¬', # Not sign
        173: u"\u00AD", # Soft hyphen
        174: u'®', # Registered trade mark sign
        175: u'¯', # Spacing macron - overline
        176: u'°', # Degree sign
        177: u'±', # Plus-or-minus sign
        178: u'²', # Superscript two - squared
        179: u'³', # Superscript three - cubed
        180: u'´', # Acute accent - spacing acute
        181: u'µ', # Micro sign
        182: u'¶', # Pilcrow sign - paragraph sign
        183: u'·', # Middle dot - Georgian comma
        184: u'¸', # Spacing cedilla
        185: u'¹', # Superscript one
        186: u'º', # Masculine ordinal indicator
        187: u'»', # Right double angle quotes
        188: u'¼', # Fraction one quarter
        189: u'½', # Fraction one half
        190: u'¾', # Fraction three quarters
        191: u'¿', # Inverted question mark
        192: u'À', # Latin capital letter A with grave
        193: u'Á', # Latin capital letter A with acute
        194: u'Â', # Latin capital letter A with circumflex
        195: u'Ã', # Latin capital letter A with tilde
        196: u'Ä', # Latin capital letter A with diaeresis
        197: u'Å', # Latin capital letter A with ring above
        198: u'Æ', # Latin capital letter AE
        199: u'Ç', # Latin capital letter C with cedilla
        200: u'È', # Latin capital letter E with grave
        201: u'É', # Latin capital letter E with acute
        202: u'Ê', # Latin capital letter E with circumflex
        203: u'Ë', # Latin capital letter E with diaeresis
        204: u'Ì', # Latin capital letter I with grave
        205: u'Í', # Latin capital letter I with acute
        206: u'Î', # Latin capital letter I with circumflex
        207: u'Ï', # Latin capital letter I with diaeresis
        208: u'Ð', # Latin capital letter ETH
        209: u'Ñ', # Latin capital letter N with tilde
        210: u'Ò', # Latin capital letter O with grave
        211: u'Ó', # Latin capital letter O with acute
        212: u'Ô', # Latin capital letter O with circumflex
        213: u'Õ', # Latin capital letter O with tilde
        214: u'Ö', # Latin capital letter O with diaeresis
        215: u'×', # Multiplication sign
        216: u'Ø', # Latin capital letter O with slash (aka "empty set")
        217: u'Ù', # Latin capital letter U with grave
        218: u'Ú', # Latin capital letter U with acute
        219: u'Û', # Latin capital letter U with circumflex
        220: u'Ü', # Latin capital letter U with diaeresis
        221: u'Ý', # Latin capital letter Y with acute
        222: u'Þ', # Latin capital letter THORN
        223: u'ß', # Latin small letter sharp s - ess-zed
        224: u'à', # Latin small letter a with grave
        225: u'á', # Latin small letter a with acute
        226: u'â', # Latin small letter a with circumflex
        227: u'ã', # Latin small letter a with tilde
        228: u'ä', # Latin small letter a with diaeresis
        229: u'å', # Latin small letter a with ring above
        230: u'æ', # Latin small letter ae
        231: u'ç', # Latin small letter c with cedilla
        232: u'è', # Latin small letter e with grave
        233: u'é', # Latin small letter e with acute
        234: u'ê', # Latin small letter e with circumflex
        235: u'ë', # Latin small letter e with diaeresis
        236: u'ì', # Latin small letter i with grave
        237: u'í', # Latin small letter i with acute
        238: u'î', # Latin small letter i with circumflex
        239: u'ï', # Latin small letter i with diaeresis
        240: u'ð', # Latin small letter eth
        241: u'ñ', # Latin small letter n with tilde
        242: u'ò', # Latin small letter o with grave
        243: u'ó', # Latin small letter o with acute
        244: u'ô', # Latin small letter o with circumflex
        245: u'õ', # Latin small letter o with tilde
        246: u'ö', # Latin small letter o with diaeresis
        247: u'÷', # Division sign
        248: u'ø', # Latin small letter o with slash
        249: u'ù', # Latin small letter u with grave
        250: u'ú', # Latin small letter u with acute
        251: u'û', # Latin small letter u with circumflex
        252: u'ü', # Latin small letter u with diaeresis
        253: u'ý', # Latin small letter y with acute
        254: u'þ', # Latin small letter thorn
        255: u'ÿ', # Latin small letter y with diaeresis
    }
    # I left this in its odd state so I could differentiate between the two
    # in the future.
    chars = e.object
    if bytes == str: # Python 2
        # Convert e.object to a bytearray for an easy switch to integers.
        # It is quicker than calling ord(char) on each char in e.object
        chars = bytearray(e.object)
        # NOTE: In Python 3 when you iterate over bytes they appear as integers.
        #       So we don't need to convert to a bytearray in Python 3.
    if isinstance(e, (UnicodeEncodeError, UnicodeTranslateError)):
        s = [u'%s' % specials[c] for c in chars[e.start:e.end]]
        return ''.join(s), e.end
    else:
codecs.register_error('handle_special', handle_special)

# TODO List:
#
#   * We need unit tests!
#   * Add a function that can dump the screen with text renditions represented as their usual escape sequences so applications that try to perform screen-scraping can match things like '\x1b[41mAuthentication configuration' without having to find specific character positions and then examining the renditions on that line.

def _reduce_renditions(renditions):
    """
    Takes a list, *renditions*, and reduces it to its logical equivalent (as
    far as renditions go).  Example::

        [0, 32, 0, 34, 0, 32]

    Would become::

        [0, 32]

    Other Examples::

        [0, 1, 36, 36]      ->  [0, 1, 36]
        [0, 30, 42, 30, 42] ->  [0, 30, 42]
        [36, 32, 44, 42]    ->  [32, 42]
        [36, 35]            ->  [35]
    """
    out_renditions = []
    foreground = None
    background = None
    for i, rend in enumerate(renditions):
        if rend < 29:
            if rend not in out_renditions:
                out_renditions.append(rend)
        elif rend > 29 and rend < 40:
            # Regular 8-color foregrounds
            foreground = rend
        elif rend > 39 and rend < 50:
            # Regular 8-color backgrounds
            background = rend
        elif rend > 91 and rend < 98:
            # 'Bright' (16-color) foregrounds
        elif rend > 99 and rend < 108:
            # 'Bright' (16-color) backgrounds
        elif rend > 1000 and rend < 10000:
            # 256-color foregrounds
        elif rend > 10000 and rend < 20000:
            # 256-color backgrounds
        else:
            out_renditions.append(rend)
    if foreground:
        out_renditions.append(foreground)
    if background:
        out_renditions.append(background)
    return out_renditions

def unicode_counter():
    """
    A generator that returns incrementing Unicode characters that can be used as
    references inside a Unicode array.  For example::

        >>> counter = unicode_counter()
        >>> mapping_dict = {}
        >>> some_array = array('u')
        >>> # Pretend 'marked ...' below is a reference to something impotant :)
        >>> for i, c in enumerate(u'some string'):
                if c == u' ': # Mark the location of spaces
                    # Perform some operation where we need to save a value
                    result = some_evaluation(i, c)
                    # Save some memory by storing a reference to result instead
                    # of the same result over and over again
                    if result not in mapping_dict.values():
                        marker = counter.next()
                        some_array.append(marker)
                        mapping_dict[marker] = result
                    else: # Find the existing reference so we can use it again
                        for k, v in mapping_dict.items():
                            if v == result: # Use the existing value
                                some_array.append(k)
                else:
                    some_array.append(\x00) # \x00 == "not interesting" placeholder
        >>>

    Now we could iterate over 'some string' and some_array simultaneously using
    zip(u'some string', some_array) to access those reference markers when we
    encountered the correct position.  This can save a lot of memory if you need
    to store objects in memory that have a tendancy to repeat (e.g. text
    rendition lists in a terminal).

    .. note:: Meant to be used inside the renditions array to reference text rendition lists such as `[0, 1, 34]`.
    """
    n = 1000 # Start at 1000 so we can use lower characters for other things
    while True:
        yield unichr(n)
        if n == 65535: # The end of unicode in narrow builds of Python
            n = 0 # Reset
        else:
            n += 1

# NOTE:  Why use a unicode array() to store references instead of just a regular array()?  Two reasons:  1) Large namespace.  2) Only need to use one kind of array for everything (convenience).  It is also a large memory savings over "just using a list with references to items in a dict."
def pua_counter():
    """
    A generator that returns a Unicode Private Use Area (PUA) character starting
    at the beginning of Plane 16 (U+100000); counting up by one with each
    successive call.  If this is a narrow Python build the tail end of Plane 15
    will be used as a fallback (with a lot less characters).

    .. note:: Meant to be used as references to non-text objects in the screen array() (since it can only contain unicode characters)
    """
    if SPECIAL == 1048576: # Not a narrow build of Python
        n = SPECIAL # U+100000 or unichr(SPECIAL) (start of Plane 16)
        while True:
            yield unichr(n)
            if n == 1114111:
                n = SPECIAL # Reset--would be impressive to make it this far!
            else:
                n += 1
    else:
        # This Python build is 'narrow' so we have to settle for less
        # Hopefully no real-world terminal will actually want to use one of
        # these characters.  In my research I couldn't find a font that used
        # them.  Please correct me if I'm wrong!
        n = SPECIAL # u'\uf849'
            if n == 63717: # The end of nothing-but-block-chars in Plane 15
                n = SPECIAL # Reset
            else:
                n += 1

class Terminal(object):
    """
    Terminal controller class.
    """
    ASCII_NUL = 0     # Null
    ASCII_BEL = 7     # Bell (BEL)
    ASCII_BS = 8      # Backspace
    ASCII_HT = 9      # Horizontal Tab
    ASCII_LF = 10     # Line Feed
    ASCII_VT = 11     # Vertical Tab
    ASCII_FF = 12     # Form Feed
    ASCII_CR = 13     # Carriage Return
    ASCII_SO = 14     # Ctrl-N; Shift out (switches to the G0 charset)
    ASCII_SI = 15     # Ctrl-O; Shift in (switches to the G1 charset)
    ASCII_XON = 17    # Resume Transmission
    ASCII_XOFF = 19   # Stop Transmission or Ignore Characters
    ASCII_CAN = 24    # Cancel Escape Sequence
    ASCII_SUB = 26    # Substitute: Cancel Escape Sequence and replace with ?
    ASCII_ESC = 27    # Escape
    ASCII_CSI = 155   # Control Sequence Introducer (that nothing uses)
    ASCII_HTS = 210   # Horizontal Tab Stop (HTS)

    charsets = {
        'B': {}, # Default: USA
        '0': { # Line drawing mode
            95: u' ',
            96: u'◆',
            97: u'▒',
            98: u'\t',
            99: u'\x0c',
            100: u'\r',
            101: u'\n',
            102: u'°',
            103: u'±',
            104: u'\n',
            105: u'\x0b',
            106: u'┘',
            107: u'┐',
            108: u'┌',
            109: u'└',
            110: u'┼',
            111: u'⎺', # All these bars and not a drink!
            112: u'⎻',
            113: u'─',
            114: u'⎼',
            115: u'⎽',
            116: u'├',
            117: u'┤',
            118: u'┴',
            119: u'┬',
            120: u'│',
            121: u'≤',
            122: u'≥',
            123: u'π',
            124: u'≠',
            125: u'£',
            126: u'·' # Centered dot--who comes up with this stuff?!?
        }
    }

    RE_CSI_ESC_SEQ = re.compile(r'\x1B\[([?A-Za-z0-9;@:\!]*)([A-Za-z@_])')
    RE_ESC_SEQ = re.compile(r'\x1b(.*\x1b\\|[ABCDEFGHIJKLMNOQRSTUVWXYZa-z0-9=<>]|[()# %*+].)')
    RE_TITLE_SEQ = re.compile(r'\x1b\][0-2]\;(.*?)(\x07|\x1b\\)')
    # The below regex is used to match our optional (non-standard) handler
    RE_OPT_SEQ = re.compile(r'\x1b\]_\;(.+?)(\x07|\x1b\\)')
    RE_NUMBERS = re.compile('\d*') # Matches any number

    def __init__(self, rows=24, cols=80, em_dimensions=None):
        """
        Initializes the terminal by calling *self.initialize(rows, cols)*.  This
        is so we can have an equivalent function in situations where __init__()
        gets overridden.

        If *em_dimensions* are provided they will be used to determine how many
        lines images will take when they're drawn in the terminal.  This is to
        prevent images that are written to the top of the screen from having
        their tops cut off.  *em_dimensions* should be a dict in the form of::

            {'height': <px>, 'width': <px>}
        """
        self.initialize(rows, cols, em_dimensions)

    def initialize(self, rows=24, cols=80, em_dimensions=None):
        """
        Initializes the terminal (the actual equivalent to :meth:`__init__`).
        """
        self.cols = cols
        self.rows = rows
        self.em_dimensions = em_dimensions
        self.scrollback_buf = []
        self.scrollback_renditions = []
        self.title = "Gate One"
        # This variable can be referenced by programs implementing Terminal() to
        # determine if anything has changed since the last dump*()
        self.modified = False
        self.local_echo = True
        self.esc_buffer = '' # For holding escape sequences as they're typed.
        self.show_cursor = True
        self.cursor_home = 0
        self.cur_rendition = unichr(1000) # Should always be reset ([0])
        self.init_screen()
        self.init_renditions()
        self.G0_charset = self.charsets['B']
        self.G1_charset = self.charsets['B']
        self.current_charset = 0
        self.charset = self.G0_charset
        self.set_G0_charset('B')
        self.set_G1_charset('B')
        self.use_g0_charset()
        # Set the default window margins
        self.top_margin = 0
        self.bottom_margin = self.rows - 1
        self.timeout_image = None
        self.specials = {
            self.ASCII_NUL: self.__ignore,
            self.ASCII_BEL: self.bell,
            self.ASCII_BS: self.backspace,
            self.ASCII_HT: self.horizontal_tab,
            self.ASCII_LF: self.linefeed,
            self.ASCII_VT: self.linefeed,
            self.ASCII_FF: self.linefeed,
            self.ASCII_CR: self._carriage_return,
            self.ASCII_SO: self.use_g1_charset,
            self.ASCII_SI: self.use_g0_charset,
            self.ASCII_XON: self._xon,
            self.ASCII_CAN: self._cancel_esc_sequence,
            self.ASCII_XOFF: self._xoff,
            #self.ASCII_ESC: self._sub_esc_sequence,
            self.ASCII_ESC: self._escape,
            self.ASCII_CSI: self._csi,
        }
        self.esc_handlers = {
            # TODO: Make a different set of these for each respective emulation mode (VT-52, VT-100, VT-200, etc etc)
            '#': self._set_line_params, # Varies
            '\\': self._string_terminator, # ST
            'c': self.clear_screen, # Reset terminal
            'D': self.__ignore, # Move/scroll window up one line    IND
            'M': self.reverse_linefeed, # Move/scroll window down one line RI
            'E': self.next_line, # Move to next line NEL
            'F': self.__ignore, # Enter Graphics Mode
            'G': self.next_line, # Exit Graphics Mode
            '6': self._dsr_get_cursor_position, # Get cursor position   DSR
            '7': self.save_cursor_position, # Save cursor position and attributes   DECSC
            '8': self.restore_cursor_position, # Restore cursor position and attributes   DECSC
            'H': self._set_tabstop, # Set a tab at the current column   HTS
            'I': self.reverse_linefeed,
            '(': self.set_G0_charset, # Designate G0 Character Set
            ')': self.set_G1_charset, # Designate G1 Character Set
            'N': self.__ignore, # Set single shift 2    SS2
            'O': self.__ignore, # Set single shift 3    SS3
            '5': self._device_status_report, # Request: Device status report DSR
            '0': self.__ignore, # Response: terminal is OK  DSR
            'P': self._dcs_handler, # Device Control String  DCS
            # NOTE: = and > are ignored because the user can override/control
            # them via the numlock key on their keyboard.  To do otherwise would
            # just confuse people.
            '=': self.__ignore, # Application Keypad  DECPAM
            '>': self.__ignore, # Exit alternate keypad mode
            '<': self.__ignore, # Exit VT-52 mode
        }
        self.csi_handlers = {
            'A': self.cursor_up,
            'B': self.cursor_down,
            'C': self.cursor_right,
            'D': self.cursor_left,
            'E': self.cursor_next_line, # NOTE: Not the same as next_line()
            'F': self.cursor_previous_line,
            'G': self.cursor_horizontal_absolute,
            'H': self.cursor_position,
            'L': self.insert_line,
            'M': self.delete_line,
            #'b': self.repeat_last_char, # TODO
            'c': self._csi_device_status_report, # Device status report (DSR)
            'g': self.__ignore, # TODO: Tab clear
            'h': self.set_expanded_mode,
            'l': self.reset_expanded_mode,
            'f': self.cursor_position,
            'd': self.cursor_position_vertical, # Vertical Line Position Absolute (VPA)
            #'e': self.cursor_position_vertical_relative, # VPR TODO
            'J': self.clear_screen_from_cursor,
            'K': self.clear_line_from_cursor,
            'S': self.scroll_up,
            'T': self.scroll_down,
            's': self.save_cursor_position,
            'u': self.restore_cursor_position,
            'm': self._set_rendition,
            'n': self.__ignore, # <ESC>[6n is the only one I know of (request cursor position)
            #'m': self.__ignore, # For testing how much CPU we save when not processing CSI
            'p': self.reset, # TODO: "!p" is "Soft terminal reset".  Also, "Set conformance level" (VT100, VT200, or VT300)
            'r': self._set_top_bottom, # DECSTBM (used by many apps)
            'q': self.set_led_state, # Seems a bit silly but you never know
            'P': self.delete_characters, # DCH Deletes the specified number of chars
            'X': self._erase_characters, # ECH Same as DCH but also deletes renditions
            'Z': self.insert_characters, # Inserts the specified number of chars
            '@': self.insert_characters, # Inserts the specified number of chars
            #'`': self._char_position_row, # Position cursor (row only)
            #'t': self.window_manipulation, # TODO
            #'z': self.locator, # TODO: DECELR "Enable locator reporting"
        }
        self.expanded_modes = {
            # Expanded modes take a True/False argument for set/reset
            '1': self.set_application_mode,
            '2': self.__ignore, # DECANM and set VT100 mode
            '3': self.__ignore, # 132 Column Mode (DECCOLM)
            '4': self.__ignore, # Smooth (Slow) Scroll (DECSCLM)
            '5': self.__ignore, # Reverse video (might support in future)
            '6': self.__ignore, # Origin Mode (DECOM)
            '7': self.__ignore, # Wraparound Mode (DECAWM)
            '8': self.__ignore, # Auto-repeat Keys (DECARM)
            '9': self.__ignore, # Send Mouse X & Y on button press (maybe)
            '12': self.send_receive_mode, # SRM
            '18': self.__ignore, # Print form feed (DECPFF)
            '19': self.__ignore, # Set print extent to full screen (DECPEX)
            '25': self.show_hide_cursor,
            '38': self.__ignore, # Enter Tektronix Mode (DECTEK)
            '41': self.__ignore, # more(1) fix (whatever that is)
            '42': self.__ignore, # Enable Nation Replacement Character sets (DECNRCM)
            '44': self.__ignore, # Turn On Margin Bell
            '45': self.__ignore, # Reverse-wraparound Mode
            '46': self.__ignore, # Start Logging (Hmmm)
            '47': self.toggle_alternate_screen_buffer, # Use Alternate Screen Buffer
            '66': self.__ignore, # Application keypad (DECNKM)
            '67': self.__ignore, # Backarrow key sends delete (DECBKM)
            '1000': self.__ignore, # Send Mouse X/Y on button press and release
            '1001': self.__ignore, # Use Hilite Mouse Tracking
            '1002': self.__ignore, # Use Cell Motion Mouse Tracking
            '1003': self.__ignore, # Use All Motion Mouse Tracking
            '1010': self.__ignore, # Scroll to bottom on tty output
            '1011': self.__ignore, # Scroll to bottom on key press
            '1035': self.__ignore, # Enable special modifiers for Alt and NumLock keys
            '1036': self.__ignore, # Send ESC when Meta modifies a key
            '1037': self.__ignore, # Send DEL from the editing-keypad Delete key
            '1047': self.__ignore, # Use Alternate Screen Buffer
            '1048': self.__ignore, # Save cursor as in DECSC
            '1049': self.toggle_alternate_screen_buffer_cursor, # Save cursor as in DECSC and use Alternate Screen Buffer, clearing it first
            '1051': self.__ignore, # Set Sun function-key mode
            '1052': self.__ignore, # Set HP function-key mode
            '1060': self.__ignore, # Set legacy keyboard emulation (X11R6)
            '1061': self.__ignore, # Set Sun/PC keyboard emulation of VT220 keyboard
        }
        self.callbacks = {
            CALLBACK_SCROLL_UP: {},
            CALLBACK_CHANGED: {},
            CALLBACK_CURSOR_POS: {},
            CALLBACK_DSR: {},
            CALLBACK_TITLE: {},
            CALLBACK_BELL: {},
            CALLBACK_OPT: {},
            CALLBACK_MODE: {},
            CALLBACK_RESET: {},
            CALLBACK_LEDS: {},
        }
        self.leds = {
            1: False,
            2: False,
            3: False,
            4: False
        }
        png_header = re.compile('.*\x89PNG\r', re.DOTALL)
        png_whole = re.compile('\x89PNG\r.+IEND\xaeB`\x82', re.DOTALL)
        # NOTE: Only matching JFIF and Exif JPEGs because "\xff\xd8" is too
        # ambiguous.
        jpeg_header = re.compile('.*\xff\xd8\xff.+JFIF\x00|.*\xff\xd8\xff.+Exif\x00', re.DOTALL)
        jpeg_whole = re.compile(
            '\xff\xd8\xff.+JFIF\x00.+\xff\xd9(?!\xff)|\xff\xd8\xff.+Exif\x00.+\xff\xd9(?!\xff)', re.DOTALL)
        self.magic = {
            # Dict for magic "numbers" so we can tell when a particular type of
            # file begins and ends (so we can capture it in binary form and
            # later dump it out via dump_html())
            # The format is 'beginning': 'whole'
            png_header: png_whole,
            jpeg_header: jpeg_whole,
        }
        self.matched_header = None
        # These are for saving self.screen and self.renditions so we can support
        # an "alternate buffer"
        self.alt_screen = None
        self.alt_renditions = None
        self.alt_cursorX = 0
        self.alt_cursorY = 0
        self.saved_cursorX = 0
        self.saved_cursorY = 0
        self.saved_rendition = [None]
        self.application_keys = False
        self.image = ""
        self.images = {}
        self.image_counter = pua_counter()
        # This is for creating a new point of reference every time there's a new
        # unique rendition at a given coordinate
        self.rend_counter = unicode_counter()
        # Used for mapping unicode chars to acutal renditions (to save memory):
        self.renditions_store = {
            u' ': [0], # Nada, nothing, no rendition.  Not the same as below
            self.rend_counter.next(): [0] # Default is actually reset
        }
        self.prev_dump = [] # A cache to speed things up
        self.prev_dump_rend = [] # Ditto
        self.html_cache = [] # Ditto
        self.watcher = None # Placeholder for the image watcher thread (if used)

    def init_screen(self):
        """
        Fills :attr:`screen` with empty lines of (unicode) spaces using
        :attr:`self.cols` and :attr:`self.rows` for the dimensions.

        .. note:: Just because each line starts out with a uniform length does not mean it will stay that way.  Processing of escape sequences is handled when an output function is called.
        """
        logging.debug('init_screen()')
        self.screen = [array('u', u' ' * self.cols) for a in xrange(self.rows)]
        # Tabstops
        tabs, remainder = divmod(self.cols, 8) # Default is every 8 chars
        self.tabstops = [(a*8)-1 for a in xrange(tabs)]
        self.tabstops[0] = 0 # Fix the first tabstop (which will be -1)
        # Base cursor position
        self.cursorX = 0
        self.cursorY = 0
        self.rendition_set = False
        self.prev_dump = [] # Force a full dump with an init
        self.prev_dump_rend = []
        self.html_cache = [] # Force this to be reset as well

    def init_renditions(self, rendition=unichr(1000)): # Match unicode_counter
        """
        Replaces :attr:`self.renditions` with arrays of *rendition* (characters)
        using :attr:`self.cols` and :attr:`self.rows` for the dimenions.
        """
        # The actual renditions at various coordinates:
        self.renditions = [
            array('u', rendition * self.cols) for a in xrange(self.rows)]

    def init_scrollback(self):
        """
        Empties the scrollback buffers (:attr:`self.scrollback_buf` and
        :attr:`self.scrollback_renditions`).
        """
        # Close any image files that might be associated with characters

    def add_callback(self, event, callback, identifier=None):
        """
        Attaches the given *callback* to the given *event*.  If given,
        *identifier* can be used to reference this callback leter (e.g. when you
        want to remove it).  Otherwise an identifier will be generated
        automatically.  If the given *identifier* is already attached to a
        callback at the given event that callback will be replaced with
        *callback*.

            :event: The numeric ID of the event you're attaching *callback* to. The :ref:`callback constants <callback_constants>` should be used as the numerical IDs.
            :callback: The function you're attaching to the *event*.
            :identifier: A string or number to be used as a reference point should you wish to remove or update this callback later.

        Returns the identifier of the callback.  to Example::

            >>> term = Terminal()
            >>> def somefunc(): pass
            >>> id = "myref"
            >>> ref = term.add_callback(term.CALLBACK_BELL, somefunc, id)

        .. note:: This allows the controlling program to have multiple callbacks for the same event.
        """
        if not identifier:
            identifier = callback.__hash__()
        self.callbacks[event][identifier] = callback
        return identifier

    def remove_callback(self, event, identifier):
        """
        Removes the callback referenced by *identifier* that is attached to the
        given *event*.  Example::

            >>> term.remove_callback(CALLBACK_BELL, "myref")
        """
        del self.callbacks[event][identifier]

    def remove_all_callbacks(self, identifier):
        """
        Removes all callbacks associated with *identifier*.
        """
        for event, identifiers in self.callbacks.items():
            try:
                del self.callbacks[event][identifier]
            except KeyError:
                pass # No match, no biggie

    def reset(self, *args, **kwargs):
        """
        Resets the terminal back to an empty screen with all defaults.  Calls
        :meth:`Terminal.callbacks[CALLBACK_RESET]` when finished.

        .. note:: If terminal output has been suspended (e.g. via ctrl-s) this will not un-suspend it (you need to issue ctrl-q to the underlying program to do that).
        """
            1: False,
            2: False,
            3: False,
            4: False
        }
        self.esc_buffer = ''
        self.G0_charset = 'B'
        self.current_charset = self.charsets['B']
        self.init_scrollback()
        self.prev_dump = []
        self.html_cache = []
        try:
            self.callbacks[CALLBACK_RESET]()
        except TypeError:
            pass

    def __ignore(self, *args, **kwargs):
        """
        Does nothing (on purpose!).  Used as a placeholder for unimplemented
        functions.
        """
        pass

    def resize(self, rows, cols, em_dimensions=None):
        """
        Resizes the terminal window, adding or removing *rows* or *cols* as
        needed.  If *em_dimensions* are provided they will be stored in
        *self.em_dimensions* (which is currently only used by image output).
        """
        logging.debug("resize(%s, %s)" % (rows, cols))
        if em_dimensions:
            self.em_dimensions = em_dimensions
        if rows == self.rows and cols == self.cols:
            return # Nothing to do--don't mess with the margins or the cursor
        if rows < self.rows: # Remove rows from the top
            for i in xrange(self.rows - rows):
                self.screen.pop(0)
                self.renditions.pop(0)
        elif rows > self.rows: # Add rows at the bottom
            for i in xrange(rows - self.rows):
                line = array('u', u' ' * self.cols)
                renditions = array('u', unichr(1000) * self.cols)
                self.screen.append(line)
                self.renditions.append(renditions)

        # Fix the cursor location:
        if self.cursorY >= self.rows:
            self.cursorY = self.rows - 1

        if cols < self.cols: # Remove cols to the right
            for i in xrange(self.rows):
                self.screen[i] = self.screen[i][:cols - self.cols]
                self.renditions[i] = self.renditions[i][:cols - self.cols]
        elif cols > self.cols: # Add cols to the right
                for j in xrange(cols - self.cols):
                    self.screen[i].append(u' ')
                    self.renditions[i].append(unichr(1000))

        if self.cursorX >= self.cols:
            self.cursorX = self.cols - 1

    def _set_top_bottom(self, settings):
        """
        DECSTBM - Sets :attr:`self.top_margin` and :attr:`self.bottom_margin`
        using the provided settings in the form of '<top_margin>;<bottom_margin>'.

        .. note:: This also handles restore/set "DEC Private Mode Values".
        """
        #logging.debug("_set_top_bottom(%s)" % settings)
        # NOTE: Used by screen and vi so this needs to work and work well!
        if len(settings):
            if settings.startswith('?'):
                # This is a set/restore DEC PMV sequence
                return # Ignore (until I figure out what this should do)
            top, bottom = settings.split(';')
            self.top_margin = max(0, int(top) - 1) # These are 0-based
            if bottom:
                self.bottom_margin = min(self.rows - 1, int(bottom) - 1)
        else:
            # Reset to defaults (full screen margins)
            self.top_margin, self.bottom_margin = 0, self.rows - 1

    def get_cursor_position(self):
        """
        Returns the current cursor positition as a tuple::

            (row, col)
        """
        return (self.cursorY, self.cursorX)

    def set_title(self, title):
        """
        Sets :attr:`self.title` to *title* and executes
        :meth:`Terminal.callbacks[CALLBACK_TITLE]`
        """
        self.title = title
        try:
            for callback in self.callbacks[CALLBACK_TITLE].values():
                callback()
        except TypeError as e:
            logging.error(_("Got TypeError on CALLBACK_TITLE..."))
            logging.error(repr(self.callbacks[CALLBACK_TITLE]))
            logging.error(e)

    def get_title(self):
        """Returns :attr:`self.title`"""
        return self.title

# TODO: put some logic in these save/restore functions to walk the current
# rendition line to come up with a logical rendition for that exact spot.
    def save_cursor_position(self, mode=None):
        """
        Saves the cursor position and current rendition settings to
        :attr:`self.saved_cursorX`, :attr:`self.saved_cursorY`, and
        :attr:`self.saved_rendition`

        .. note:: Also handles the set/restore "Private Mode Settings" sequence.
        """
        if mode: # Set DEC private mode
            # TODO: Need some logic here to save the current expanded mode
            #       so we can restore it in _set_top_bottom().
            self.set_expanded_mode(mode)
        # NOTE: args and kwargs are here to make sure we don't get an exception
        #       when we're called via escape sequences.
        self.saved_cursorX = self.cursorX
        self.saved_cursorY = self.cursorY
        self.saved_rendition = self.renditions[self.cursorY][self.cursorX]

    def restore_cursor_position(self, *args, **kwargs):
        """
        Restores the cursor position and rendition settings from
        :attr:`self.saved_rendition` (if they're set).
        """
        if self.saved_cursorX and self.saved_cursorY:
            self.cursorX = self.saved_cursorX
            self.cursorY = self.saved_cursorY
            self.renditions[self.cursorY][self.cursorX] = self.saved_rendition

    def _dsr_get_cursor_position(self):
        """
        Returns the current cursor positition as a DSR response in the form of::

            '\x1b<self.cursorY>;<self.cursorX>R'

        Also executes CALLBACK_DSR with the same output as the first argument.
        Example::

            self.callbacks[CALLBACK_DSR]('\x1b20;123R')
        """
        esc_cursor_pos = '\x1b%s;%sR' % (self.cursorY, self.cursorX)
        try:
            for callback in self.callbacks[CALLBACK_DSR].values():
                callback(esc_cursor_pos)
            pass
        return esc_cursor_pos

    def _dcs_handler(self, string=None):
        """
        Handles Device Control String sequences.  Unimplemented.  Probablye not
        appropriate for Gate One.  If you believe this to be false please open
        a ticket in the issue tracker.
        """
        pass
        #print("TODO: Handle this DCS: %s" % string)

    def _set_line_params(self, param):
        """
        This function handles the control sequences that set double and single
        line heights and widths.  It also handles the "screen alignment test" (
        fill the screen with Es).

        .. note:: Double-line height text is currently unimplemented (does anything actually use it?).
        """
        try:
            param = int(param)
            logging.warning("Couldn't handle escape sequence #%s" % repr(param))
        if param == 8:
            # Screen alignment test
            self.init_renditions()
            self.screen = [
                array('u', u'E' * self.cols) for a in xrange(self.rows)]
        # TODO: Get this handling double line height stuff...  For kicks

    def set_G0_charset(self, char):
        """
        Sets the terminal's G0 (default) charset to the type specified by
        *char*.

        Here's the possibilities::

            0    DEC Special Character and Line Drawing Set
            A    United Kingdom (UK)
            B    United States (USASCII)
            4    Dutch
            C    Finnish
            5    Finnish
            R    French
            Q    French Canadian
            K    German
            Y    Italian
            E    Norwegian/Danish
            6    Norwegian/Danish
            Z    Spanish
            H    Swedish
            7    Swedish
            =    Swiss
        """
        #logging.debug("Setting G0 charset to %s" % repr(char))
        try:
            self.G0_charset = self.charsets[char]
            self.G0_charset = self.charsets['B']
        if self.current_charset == 0:
            self.charset = self.G0_charset

    def set_G1_charset(self, char):
        """
        Sets the terminal's G1 (alt) charset to the type specified by *char*.

            4    Dutch
            C    Finnish
            5    Finnish
            R    French
            K    German
            Y    Italian
            Z    Spanish
            H    Swedish
            7    Swedish
            =    Swiss
        """
        #logging.debug("Setting G1 charset to %s" % repr(char))
        try:
            self.G1_charset = self.charsets[char]
            self.G1_charset = self.charsets['B']
        if self.current_charset == 1:
            self.charset = self.G1_charset

    def use_g0_charset(self):
        """
        Sets the current charset to G0.  This should get called when ASCII_SO
        is encountered.
        """
        #logging.debug(
            #"Switching to G0 charset (which is %s)" % repr(self.G0_charset))

    def use_g1_charset(self):
        """
        Sets the current charset to G1.  This should get called when ASCII_SI
        """
            #"Switching to G1 charset (which is %s)" % repr(self.G1_charset))
        self.current_charset = 1

    def write(self, chars, special_checks=True):
        """
        Write *chars* to the terminal at the current cursor position advancing
        the cursor as it does so.  If *chars* is not unicode, it will be
        converted to unicode before being stored in self.screen.

        if *special_checks* is True (default), Gate One will perform checks for
        special things like image files coming in via *chars*.
        """
        # NOTE: This is the slowest function in all of Gate One.  All
        # suggestions on how to speed it up are welcome!

        # Speedups (don't want dots in loops if they can be avoided)
        specials = self.specials
        esc_handlers = self.esc_handlers
        csi_handlers = self.csi_handlers
        RE_ESC_SEQ = self.RE_ESC_SEQ
        RE_CSI_ESC_SEQ = self.RE_CSI_ESC_SEQ
        cursor_right = self.cursor_right
        magic = self.magic
        changed = False
        #logging.debug('handling chars: %s' % repr(chars))
        if special_checks:
            # NOTE: Special checks are limited to PNGs and JPEGs right now
            before_chars = ""
            after_chars = ""
            for magic_header in magic.keys():
                if magic_header.match(str(chars)):
                    self.matched_header = magic_header
                    self.timeout_image = datetime.now()
            if self.image or self.matched_header:
                self.image += chars
                match = magic[self.matched_header].search(self.image)
                if match:
                    #logging.debug("Matched image format.  Capturing...")
                    before_chars, after_chars = magic[
                            self.matched_header].split(self.image)
                    # Eliminate anything before the match
                    self.image = match.group()
                    self._capture_image()
                    self.image = "" # Empty it now that is is captured
                    self.matched_header = None # Ditto
                if before_chars:
                    self.write(before_chars, special_checks=False)
                if after_chars:
                    self.write(after_chars, special_checks=False)
                # If we haven't got a complete image after one second something
                # went wrong.  Discard what we've got and restart.
                one_second = timedelta(seconds=1)
                if datetime.now() - self.timeout_image > one_second:
                    self.image = "" # Empty it
                    self.matched_header = None
                    chars = _("Failed to decode image.  Buffer discarded.")
                else:
                    return
        # Have to convert to unicode
        try:
            chars = chars.decode('utf-8', "handle_special")
        except UnicodeDecodeError:
            # Just in case
            try:
                chars = chars.decode('utf-8', "ignore")
            except UnicodeDecodeError:
                logging.error(
                    _("Double UnicodeDecodeError in terminal.Terminal."))
                return
        except AttributeError:
            # In Python 3 strings don't have .decode()
            pass # Already Unicode
        for char in chars:
            charnum = ord(char)
            if charnum in specials:
                specials[charnum]()
            else:
                # Now handle the regular characters and escape sequences
                if self.esc_buffer: # We've got an escape sequence going on...
                    try:
                        self.esc_buffer += char
                        # First try to handle non-CSI ESC sequences (the basics)
                        match_obj = RE_ESC_SEQ.match(self.esc_buffer)
                        if match_obj:
                            seq_type = match_obj.group(1) # '\x1bA' -> 'A'
                            # Call the matching ESC handler
                            #logging.debug('ESC seq: %s' % seq_type)
                            if len(seq_type) == 1: # Single-character sequnces
                                esc_handlers[seq_type]()
                            else: # Multi-character stuff like '\x1b)B'
                                esc_handlers[seq_type[0]](seq_type[1:])
                            self.esc_buffer = '' # All done with this one
                            continue
                        # Next try to handle CSI ESC sequences
                        match_obj = RE_CSI_ESC_SEQ.match(self.esc_buffer)
                            csi_values = match_obj.group(1) # e.g. '0;1;37'
                            csi_type = match_obj.group(2) # e.g. 'm'
                            #logging.debug(
                                #'CSI: %s, %s' % (csi_type, csi_values))
                            # Call the matching CSI handler
                            try:
                                csi_handlers[csi_type](csi_values)
                            except ValueError:
                            # Commented this out because it can be super noisy
                                #logging.error(_(
                                    #"CSI Handler Error: Type: %s, Values: %s" %
                                    #(csi_type, csi_values)
                                #))
                                pass
                            self.esc_buffer = ''
                            continue
                    except KeyError:
                        # No handler for this, try some alternatives
                        if self.esc_buffer.endswith('\x1b\\'):
                            self._osc_handler()
                        else:
                            logging.warning(_(
                                "Warning: No ESC sequence handler for %s"
                                % `self.esc_buffer`
                            ))
                    continue # We're done here
# TODO: Figure out a way to write characters past the edge of the screen so that users can copy & paste without having newlines in the middle of everything.
                changed = True
                if self.cursorX >= self.cols:
                    # Start a newline but NOTE: Not really the best way to
                    # handle this because it means copying and pasting lines
                    # will end up broken into pieces of size=self.cols
                    self.newline()
                    self.cursorX = 0
                    # This actually works but until I figure out a way to
                    # get the browser to properly wrap the line without
                    # freaking out whenever someone clicks on the page it
                    # will have to stay commented.  NOTE: This might be a
                    # browser bug.
                    #self.screen[self.cursorY].append(unicode(char))
                    #self.renditions[self.cursorY].append([])
                    # To try it just uncomment the above two lines and
                    # comment out the self.newline() and self.cusorX lines
                try:
                    self.renditions[self.cursorY][
                        self.cursorX] = self.cur_rendition
                    if charnum in self.charset:
                        char = self.charset[charnum]
                        self.screen[self.cursorY][self.cursorX] = char
                    else:
                        # Double check this isn't a unicode diacritic (accent)
                        # which simply modifies the character before it
                        if unicodedata.combining(char):
                            # This is a diacritic.  Combine it with existing:
                            current = self.screen[self.cursorY][self.cursorX]
                            combined = unicodedata.normalize(
                                'NFC', u'%s%s' % (current, char))
                            # Sometimes a joined combining char can still result
                            # a string of length > 1.  So we need to handle that
                            if len(combined) > 1:
                                for i, c in enumerate(combined):
                                    self.screen[self.cursorY][
                                        self.cursorX] = c
                                    if i < len(combined) - 1:
                                        cursor_right()
                            else:
                                self.screen[self.cursorY][
                                    self.cursorX] = combined
                        else:
                            # Normal character
                            self.screen[self.cursorY][self.cursorX] = char
                except IndexError as e:
                    # This can happen when escape sequences go haywire
                    logging.error(_(
                        "IndexError in write() (rate limiter?): %s" % e))
                    pass
                cursor_right()
        if changed:
            self.modified = True
            # Execute our callbacks
            try:
                for callback in self.callbacks[CALLBACK_CHANGED].values():
                    callback()
            except TypeError:
                pass
            try:
                for callback in self.callbacks[CALLBACK_CURSOR_POS].values():
                pass

    def flush(self):
        """
        Only here to make Terminal compatible with programs that want to use
        file-like methods.
        """
        pass

    def scroll_up(self, n=1):
        """
        Scrolls up the terminal screen by *n* lines (default: 1). The callbacks
        CALLBACK_CHANGED and CALLBACK_SCROLL_UP are called after scrolling the
        screen.

        .. note:: This will only scroll up the region within self.top_margin and self.bottom_margin (if set).
        """
        #logging.debug("scroll_up(%s)" % n)
        for x in xrange(int(n)):
            line = self.screen.pop(self.top_margin) # Remove the top line
            self.scrollback_buf.append(line) # Add it to the scrollback buffer
            if len(self.scrollback_buf) > 1000:
                # 1000 lines ought to be enough for anybody
                self.init_scrollback()
                # NOTE:  This would only be if 1000 lines piled up before the
                # next dump_html() or dump().
            empty_line = array('u', u' ' * self.cols) # Line full of spaces
            # Add it to the bottom of the window:
            self.screen.insert(self.bottom_margin, empty_line)
            # Remove top line's style information
            style = self.renditions.pop(self.top_margin)
            self.scrollback_renditions.append(style)
            # Insert a new empty rendition as well:
            empty_line = array('u', unichr(1000) * self.cols)
            self.renditions.insert(self.bottom_margin, empty_line)
        # Execute our callback indicating lines have been updated
        try:
            for callback in self.callbacks[CALLBACK_CHANGED].values():
            pass
        # Execute our callback to scroll up the screen
        try:
            for callback in self.callbacks[CALLBACK_SCROLL_UP].values():
            pass

    def scroll_down(self, n=1):
        """
        Scrolls down the terminal screen by *n* lines (default: 1). The
        callbacks CALLBACK_CHANGED and CALLBACK_SCROLL_DOWN are called after
        scrolling the screen.
        """
        #logging.debug("scroll_down(%s)" % n)
            self.screen.pop(self.bottom_margin) # Remove the bottom line
            self.screen.insert(self.top_margin, empty_line) # Add it to the top
            # Remove bottom line's style information:
            self.renditions.pop(self.bottom_margin)
            # Insert a new empty one:
            self.renditions.insert(self.top_margin, empty_line)
        try:
            pass

        try:
            pass

    def insert_line(self, n=1):
        """
        Inserts *n* lines at the current cursor position.
        """
        #logging.debug("insert_line(%s)" % n)
        if not n: # Takes care of an empty string
            n = 1
        n = int(n)
        for i in xrange(n):
            # Remove bottom line's style information as well:
            self.screen.insert(self.cursorY, empty_line) # Insert at cursor
            empty_rend = array('u', unichr(1000) * self.cols)
            self.renditions.insert(self.cursorY, empty_rend) # Insert at cursor

    def delete_line(self, n=1):
        """
        Deletes *n* lines at the current cursor position.
        """
        #logging.debug("delete_line(%s)" % n)
            n = 1
        n = int(n)
            self.screen.pop(self.cursorY) # Remove the line at the cursor
            # Remove the line's style information as well:
            self.renditions.pop(self.cursorY)
            # Now add an empty line and empty set of renditions to the bottom of
            # the view
            # Add it to the bottom of the view:
            self.screen.insert(self.bottom_margin, empty_line) # Insert at bottom
            self.renditions.insert(self.bottom_margin, empty_rend)

    def backspace(self):
        """Execute a backspace (\\x08)"""
        try:
            self.renditions[self.cursorY][self.cursorX] = u' '
        except IndexError:
            pass # At the edge, no biggie
        self.cursor_left(1)

    def horizontal_tab(self):
        """Execute horizontal tab (\\x09)"""
        next_tabstop = self.cols -1
        for tabstop in self.tabstops:
            if tabstop > self.cursorX:
                next_tabstop = tabstop
                break
        self.cursorX = next_tabstop

    def _set_tabstop(self):
        """Sets a tabstop at the current position of :attr:`self.cursorX`."""
        if self.cursorX not in self.tabstops:
            for tabstop in self.tabstops:
                if self.cursorX > tabstop:
                    self.tabstops.append(self.cursorX)
                    self.tabstops.sort() # Put them in order :)
                    break

    def linefeed(self):
        """
        LF - Executes a line feed.

        .. note:: This actually just calls :meth:`Terminal.newline`.
        """
        self.newline()

    def next_line(self):
        """
        CNL - Moves the cursor down one line to the home position.  Will not
        result in a scrolling event like newline() does.

        .. note:: This is not the same thing as :meth:`Terminal.cursor_next_line` which preserves the cursor's column position.
        """
        self.cursorX = self.cursor_home
        if self.cursorY < self.rows -1:
            self.cursorY += 1

    def reverse_linefeed(self):
        """
        RI - Executes a reverse line feed: Move the cursor up one line to the
        home position.  If the cursor move would result in going past the top
        margin of the screen (upwards) this will execute a scroll_down() event.
        """
        self.cursorY -= 1
        if self.cursorY < self.top_margin:
            self.scroll_down()
            self.cursorY = self.top_margin

    def newline(self):
        """
        Increases :attr:`self.cursorY` by 1 and calls :meth:`Terminal.scroll_up`
        if that action will move the curor past :attr:`self.bottom_margin`
        (usually the bottom of the screen).
        """
        self.cursorY += 1
        if self.cursorY > self.bottom_margin:
            self.scroll_up()
            self.cursorY = self.bottom_margin
            self.clear_line()

    def _carriage_return(self):
        """
        Executes a carriage return (sets :attr:`self.cursorX` to 0).  In other
        words it moves the cursor back to position 0 on the line.
        """

    def _xon(self):
        """
        Handles the XON character (stop ignoring).

        .. note:: Doesn't actually do anything (this feature was probably meant for the underlying terminal program).
        """
        logging.debug('_xon()')

    def _xoff(self):
        """
        Handles the XOFF character (start ignoring)

        """
        logging.debug('_xoff()')
        self.local_echo = False

    def _cancel_esc_sequence(self):
        """
        Cancels any escape sequence currently being processed.  In other words
        it empties :attr:`self.esc_buffer`.
        """

    def _sub_esc_sequence(self):
        """
        Cancels any escape sequence currently in progress and replaces
        :attr:`self.esc_buffer` with single question mark (?).

        .. note:: Nothing presently uses this function and I can't remember what it was supposed to be part of (LOL!).  Obviously it isn't very important.
        """
        self.write('?')

    def _escape(self):
        """
        Handles the escape character as well as escape sequences that may end
        with an escape character.
        """
        buf = self.esc_buffer
        if buf.startswith('\x1bP') or buf.startswith('\x1b]'):
            # CSRs and OSCs are special
            self.esc_buffer += '\x1b'
        else:
            # Get rid of whatever's there since we obviously didn't know what to
            # do with it
            self.esc_buffer = '\x1b'

    def _csi(self):
        """
        Marks the start of a CSI escape sequence (which is itself a character)
        by setting :attr:`self.esc_buffer` to '\\\\x1b[' (which is the CSI escape
        sequence).
        """
        self.esc_buffer = '\x1b['

    def _capture_image(self):
        """
        This gets called when an image (PNG or JPEG) was detected by
        :meth:`Terminal.write` and captured in :attr:`self.image`.  It cleans up
        the data inside :attr:`self.image` (getting rid of carriage returns) and
        stores a reference to self.image as a single character (using
        :attr:`self.image_counter`) in :attr:`self.screen` at the current cursor
        position.  The actual image will be written to disk and read on-demand
        by :meth:`self__spanify_screen` when it needs to be displayed.  The
        image on disk will be automatically removed  when it is no longer
        visible.

        It also moves the cursor to the beginning of the last line before doing
        this in order to ensure that the captured image stays within the
        browser's current window.

        .. note:: The :meth:`Terminal._spanify_screen` function is aware of this logic and knows that a 'character' in Plane 16 of the Unicode PUA indicates the presence of something like an image that needs special processing.
        """
        # Remove the extra \r's that the terminal adds:
        self.image = str(self.image).replace('\r\n', '\n')
        logging.debug("_capture_image() len(self.image): %s" % len(self.image))
        if Image: # PIL is loaded--try to guess how many lines the image takes
            i = StringIO.StringIO(self.image)
            try:
                im = Image.open(i)
            except IOError:
                # i.e. PIL couldn't identify the file
                return # Don't do anything--bad image
        else: # No PIL means no images.  Don't bother wasting memory.
            return
        ref = self.image_counter.next()
        if self.em_dimensions:
            # Make sure the image will fit properly in the screen
            width = im.size[0]
            height = im.size[1]
            if height <= self.em_dimensions['height']:
                # Fits within a line.  No need for a newline
                num_chars = int(width/self.em_dimensions['width'])
                # Put the image at the current cursor location
                self.screen[self.cursorY][self.cursorX] = ref
                # Move the cursor an equivalent number of characters
                self.cursor_right(num_chars)
            else:
                newlines = int(height/self.em_dimensions['height'])
                self.cursorX = 0
                newlines = abs(self.cursorY - newlines)
                self.newline() # Start with a newline for good measure... For
                # Some reason it seems to look better that way.
                if newlines > self.cursorY:
                    for line in xrange(newlines):
                        self.newline()
                self.newline()
        else:
            # No way to calculate the number of lines the image will take
            self.cursorY = self.rows - 1 # Move to the end of the screen
            # ... so it doesn't get cut off at the top
            self.screen[self.cursorY][self.cursorX] = ref
            self.newline() # Make some space at the bottom too just in case
            self.newline()
        # Write the image to disk in a temporary location
        self.images[ref] = tempfile.TemporaryFile()
        im.save(self.images[ref], im.format)
        self.images[ref].flush()
        # Start up the image watcher thread so leftover FDs get closed when
        # they're no longer being used
        if not self.watcher or not self.watcher.isAlive():
            import threading
            self.watcher = threading.Thread(
                name='watcher', target=self._image_fd_watcher)
            self.watcher.setDaemon(True)
            self.watcher.start()

    def _image_fd_watcher(self):
        """
        Meant to be run inside of a thread, calls close_image_fds() until there
        are no more open image file descriptors.
        """
        logging.debug("starting image_fd_watcher()")
        import time
        quitting = False
        while not quitting:
            if self.images:
                self.close_image_fds()
                time.sleep(5)
            else:
                quitting = True
        logging.debug('image_fd_watcher() quitting: No more images.')

    def close_image_fds(self):
        """
        Closes the file descriptors of any images that are no longer on the
        screen.
        """
        #logging.debug('close_image_fds()') # Commented because it's kinda noisy
        if self.images:
            for ref in self.images.keys():
                found = False
                for line in self.screen:
                    if ref in line:
                        found = True
                        break
                if self.alt_screen:
                    for line in self.alt_screen:
                        if ref in line:
                            found = True
                            break
                if not found:
                    self.images[ref].close()
                    del self.images[ref]

    def _string_terminator(self):
        """
        Handle the string terminator (ST).

        .. note:: Doesn't actually do anything at the moment.  Probably not needed since :meth:`Terminal._escape` and/or :meth:`Terminal.bell` will end up handling any sort of sequence that would end in an ST anyway.
        """
        # NOTE: Might this just call _cancel_esc_sequence?  I need to double-check.
        pass

    def _osc_handler(self):
        """
        Handles Operating System Command (OSC) escape sequences which need
        special care since they are of indeterminiate length and end with
        either a bell (\\\\x07) or a sequence terminator (\\\\x9c aka ST).  This
        will usually be called from :meth:`Terminal.bell` to set the title of
        the terminal (just like an xterm) but it is also possible to be called
        directly whenever an ST is encountered.
        """
        # Try the title sequence first
        match_obj = self.RE_TITLE_SEQ.match(self.esc_buffer)
        if match_obj:
            self.esc_buffer = ''
            title = match_obj.group(1)
            self.set_title(title) # Sets self.title
            return
        # Next try our special optional handler sequence
        match_obj = self.RE_OPT_SEQ.match(self.esc_buffer)
            text = match_obj.group(1)
            self._opt_handler(text)
            return
        # At this point we've encountered something unusual
        #logging.warning(_("Warning: No ESC sequence handler for %s" %
            #`self.esc_buffer`))

    def bell(self):
        """
        Handles the bell character and executes
        :meth:`Terminal.callbacks[CALLBACK_BELL]` (if we are not in the middle
        of an escape sequence that ends with a bell character =).  If we *are*
        in the middle of an escape sequence, calls :meth:`self._osc_handler`
        since we can be nearly certain that we're simply terminating an OSC
        sequence. Isn't terminal emulation grand? ⨀_⨀
        """
        # NOTE: A little explanation is in order: The bell character (\x07) by
        #       itself should play a bell (pretty straighforward).  However, if
        #       the bell character is at the tail end of a particular escape
        #       sequence (string starting with \x1b]0;) this indicates an xterm
        #       title (everything between \x1b]0;...\x07).
        if not self.esc_buffer: # We're not in the middle of an esc sequence
            logging.debug('Regular bell')
            try:
                for callback in self.callbacks[CALLBACK_BELL].values():
                pass
        else: # We're (likely) setting a title
            self.esc_buffer += '\x07' # Add the bell char so we don't lose it
            self._osc_handler()

    def _device_status_report(self):
        """
        Returns '\x1b[0n' (terminal OK) and executes:

        .. code-block:: python

            self.callbacks[CALLBACK_DSR]("\\x1b[0n")
        """
        logging.debug("_device_status_report()")
        response = u"\x1b[0n"
        try:
                callback(response)
            pass
        return response

    def _csi_device_status_report(self, request):
        """
        Returns '\\\\x1b[1;2c' (Meaning: I'm a vt220 terminal, version 1.0) and
        executes:

            self.callbacks[self.CALLBACK_DSR]("\\x1b[1;2c")
        """
        logging.debug("_csi_device_status_report()")
        response = u"\x1b[1;2c"
        try:
            pass

    def set_expanded_mode(self, setting):
        """
        Accepts "standard mode" settings.  Typically '\\\\x1b[?25h' to hide cursor.

        Notes on modes::

            '?1h' - Application Cursor Keys
            '?5h' - DECSCNM (default off): Set reverse-video mode.
            '?12h' - Local echo (SRM or Send Receive Mode)
            '?25h' - Hide cursor
            '?1049h' - Save cursor and screen
        """
        # TODO: Add support for the following:
        # * 3: 132 column mode (might be "or greater")
        # * 4: Smooth scroll (for animations and also makes things less choppy)
        # * 5: Reverse video (should be easy: just need some extra CSS)
        # * 6: Origin mode
        # * 7: Wraparound mode
        logging.debug("set_expanded_mode(%s)" % setting)
        setting = setting[1:] # Don't need the ?
        settings = setting.split(';')
        for setting in settings:
            try:
                self.expanded_modes[setting](True)
            except (KeyError, TypeError):
                pass # Unsupported expanded mode
        try:
            for callback in self.callbacks[CALLBACK_MODE].values():
                callback(setting, True)
            pass

    def reset_expanded_mode(self, setting):
        """
        Accepts "standard mode" settings.  Typically '\\\\x1b[?25l' to show
        cursor.
        """
        logging.debug("reset_expanded_mode(%s)" % setting)
            try:
                self.expanded_modes[setting](False)
        try:
                callback(setting, False)
            pass

    def set_application_mode(self, boolean):
        """
        Sets :attr:`self.application_keys` equal to *boolean*.  Literally:

            self.application_keys = boolean
        """
        self.application_keys = boolean

    def toggle_alternate_screen_buffer(self, alt):
        """
        If *alt* is True, copy the current screen and renditions to
        :attr:`self.alt_screen` and :attr:`self.alt_renditions` then re-init
        :attr:`self.screen` and :attr:`self.renditions`.

        If *alt* is False, restore the saved screen buffer and renditions then
        nullify :attr:`self.alt_screen` and :attr:`self.alt_renditions`.
        """
        #logging.debug('toggle_alternate_screen_buffer(%s)' % alt)
        if alt:
            # Save the existing screen and renditions
            self.alt_screen = self.screen[:]
            self.alt_renditions = self.renditions[:]
            # Make a fresh one
            self.clear_screen()
        else:
            # Restore the screen
            if self.alt_screen and self.alt_renditions:
                self.screen = self.alt_screen[:]
                self.renditions = self.alt_renditions[:]
            # Empty out the alternate buffer (to save memory)
            self.alt_screen = None
            self.alt_renditions = None
        # These all need to be reset no matter what
        self.cur_rendition = unichr(1000)

    def toggle_alternate_screen_buffer_cursor(self, alt):
        """
        Same as :meth:`Terminal.toggle_alternate_screen_buffer` but also
        saves/restores the cursor location.
        """
        #logging.debug('toggle_alternate_screen_buffer_cursor(%s)' % alt)
        if alt:
            self.alt_cursorX = self.cursorX
            self.alt_cursorY = self.cursorY
        else:
            self.cursorX = self.alt_cursorX
            self.cursorY = self.alt_cursorY
        self.toggle_alternate_screen_buffer(alt)

    def show_hide_cursor(self, boolean):
        """
        Literally:

            self.show_cursor = boolean
        """
        self.show_cursor = boolean

    def send_receive_mode(self, onoff):
        """
        Turns on or off local echo dependong on the value of *onoff*:

            self.local_echo = onoff
        """
        #logging.debug("send_receive_mode(%s)" % repr(onoff))
        # This has been disabled because it might only be meant for the
        # underlying program and not the terminal emulator.  Needs research.
        #if onoff:
            #self.local_echo = False
        #else:
            #self.local_echo = True

    def insert_characters(self, n=1):
        """
        Inserts the specified number of characters at the cursor position.
        Overwriting whatever is already present.
        """
        #logging.debug("insert_characters(%s)" % n)
        n = int(n)
            self.screen[self.cursorY].pop() # Take one down, pass it around
            self.screen[self.cursorY].insert(self.cursorX, u' ')

    def delete_characters(self, n=1):
        """
        DCH - Deletes (to the left) the specified number of characters at the
        cursor position.  As characters are deleted, the remaining characters
        between the cursor and right margin move to the left. Character
        attributes (renditions) move with the characters.  The terminal adds
        blank spaces with no visual character attributes at the right margin.
        DCH has no effect outside the scrolling margins.

        .. note:: Deletes renditions too.  You'd *think* that would be in one of the VT-* manuals...  Nope!
        """
        #logging.debug("delete_characters(%s)" % n)
        if not n: # e.g. n == ''
            n = 1
        else:
            n = int(n)
            try:
                self.screen[self.cursorY].pop(self.cursorX)
                self.screen[self.cursorY].append(u' ')
                self.renditions[self.cursorY].pop(self.cursorX)
                self.renditions[self.cursorY].append(unichr(1000))
            except IndexError:
                # At edge of screen, ignore
                #print('IndexError in delete_characters(): %s' % e)
                pass

    def _erase_characters(self, n=1):
        """
        Erases (to the right) the specified number of characters at the cursor
        position.

        .. note:: Deletes renditions too.
        """
        #logging.debug("_erase_characters(%s)" % n)
            n = 1
        else:
            n = int(n)
        distance = self.cols - self.cursorX
        n = min(n, distance)
            self.screen[self.cursorY][self.cursorX+i] = u' '
            self.renditions[self.cursorY][self.cursorX+i] = unichr(1000)

    def cursor_left(self, n=1):
        """ESCnD CUB (Cursor Back)"""
        # Commented out to save CPU (and the others below too)
        #logging.debug('cursor_left(%s)' % n)
        n = int(n)
        self.cursorX = max(0, self.cursorX - n)
        try:
            for callback in self.callbacks[CALLBACK_CURSOR_POS].values():
            pass

    def cursor_right(self, n=1):
        """ESCnC CUF (Cursor Forward)"""
        #logging.debug('cursor_right(%s)' % n)
        if not n:
            n = 1
        n = int(n)
        self.cursorX += n
        try:
            pass

    def cursor_up(self, n=1):
        """ESCnA CUU (Cursor Up)"""
        #logging.debug('cursor_up(%s)' % n)
        if not n:
            n = 1
        n = int(n)
        self.cursorY = max(0, self.cursorY - n)
        try:
            pass

    def cursor_down(self, n=1):
        """ESCnB CUD (Cursor Down)"""
        #logging.debug('cursor_down(%s)' % n)
        if not n:
            n = 1
        n = int(n)
        self.cursorY = min(self.rows, self.cursorY + n)
        try:
            pass

    def cursor_next_line(self, n):
        """ESCnE CNL (Cursor Next Line)"""
        if not n:
            n = 1
        n = int(n)
        try:
            pass

    def cursor_previous_line(self, n):
        """ESCnF CPL (Cursor Previous Line)"""
        if not n:
            n = 1
        n = int(n)
        try:
            pass

    def cursor_horizontal_absolute(self, n):
        """ESCnG CHA (Cursor Horizontal Absolute)"""
        if not n:
            n = 1
        n = int(n)
        self.cursorX = n - 1 # -1 because cols is 0-based
        try:
            pass

    def cursor_position(self, coordinates):
        """
        ESCnH CUP (Cursor Position).  Move the cursor to the given coordinates.

            :coordinates: Should be something like, 'row;col' (1-based) but, 'row', 'row;', and ';col' are also valid (assumes 1 on missing value).

        .. note:: If coordinates is '' (an empty string), the cursor will be moved to the top left (1;1).
        """
        # NOTE: Since this is 1-based we have to subtract 1 from everything to
        #       match how we store these values internally.
        if not coordinates:
            row, col = 0, 0
        elif ';' in coordinates:
            row, col = coordinates.split(';')
        else:
            row = coordinates
            col = 0
        try:
            row = int(row)
            row = 0
        try:
            col = int(col)
            col = 0
        # These ensure a positive integer while reducing row and col by 1:
        row = max(0, row - 1)
        col = max(0, col - 1)
        self.cursorY = row
        self.cursorX = col
        try:
            pass

    def cursor_position_vertical(self, n):
        """
        Vertical Line Position Absolute (VPA) - Moves the cursor to given line.
        """
        n = int(n)
        self.cursorY = n - 1

    def clear_screen(self):
        """
        Clears the screen.  Also used to emulate a terminal reset.

        .. note:: The current rendition (self.cur_rendition) will be applied to all characters on the screen when this function is called.
        """
        #logging.debug('clear_screen()')
        self.init_renditions(self.cur_rendition)

    def clear_screen_from_cursor_down(self):
        """
        Clears the screen from the cursor down (ESC[J or ESC[0J).
        """
        #logging.debug('clear_screen_from_cursor_down()')
        self.screen[self.cursorY:] = [
            array('u', u' ' * self.cols) for a in self.screen[self.cursorY:]
        ]
        c = self.cur_rendition # Just to save space below
        self.renditions[self.cursorY:] = [
            array('u', c * self.cols) for a in self.renditions[self.cursorY:]
        ]

    def clear_screen_from_cursor_up(self):
        """
        Clears the screen from the cursor up (ESC[1J).
        """
        #logging.debug('clear_screen_from_cursor_up()')
        self.screen[:self.cursorY+1] = [
            array('u', u' ' * self.cols) for a in self.screen[:self.cursorY]
        ]
        c = self.cur_rendition
        self.renditions[:self.cursorY+1] = [
            array('u', c * self.cols) for a in self.renditions[:self.cursorY]
        ]

    def clear_screen_from_cursor(self, n):
        """
        CSI *n* J ED (Erase Data).  This escape sequence uses the following rules:

        ======  =============================   ===
        Esc[J   Clear screen from cursor down   ED0
        Esc[0J  Clear screen from cursor down   ED0
        Esc[1J  Clear screen from cursor up     ED1
        Esc[2J  Clear entire screen             ED2
        """
        #logging.debug('clear_screen_from_cursor(%s)' % n)
        try:
            n = int(n)
        except ValueError: # Esc[J
            n = 0
        clear_types = {
            0: self.clear_screen_from_cursor_down,
            1: self.clear_screen_from_cursor_up,
            2: self.clear_screen
        }
        try:
            clear_types[n]()
            logging.error(_("Error: Unsupported number for escape sequence J"))
        # Execute our callbacks
        try:
            pass
        try:
            pass

    def clear_line_from_cursor_right(self):
        """
        Clears the screen from the cursor right (ESC[K or ESC[0K).
        """
        #logging.debug("clear_line_from_cursor_right()")
        saved = self.screen[self.cursorY][:self.cursorX]
        saved_renditions = self.renditions[self.cursorY][:self.cursorX]
        spaces = array('u', u' '*len(self.screen[self.cursorY][self.cursorX:]))
        renditions = array('u',
            self.cur_rendition * len(self.screen[self.cursorY][self.cursorX:]))
        self.screen[self.cursorY] = saved + spaces
        # Reset the cursor position's rendition to the end of the line
        self.renditions[self.cursorY] = saved_renditions + renditions

    def clear_line_from_cursor_left(self):
        """
        Clears the screen from the cursor left (ESC[1K).
        """
        #logging.debug("clear_line_from_cursor_left()")
        saved = self.screen[self.cursorY][self.cursorX:]
        saved_renditions = self.renditions[self.cursorY][self.cursorX:]
        spaces = array('u', u' '*len(self.screen[self.cursorY][:self.cursorX]))
            self.cur_rendition * len(self.screen[self.cursorY][:self.cursorX]))
        self.screen[self.cursorY] = spaces + saved
        self.renditions[self.cursorY] = renditions + saved_renditions

    def clear_line(self):
        """
        Clears the entire line (ESC[2K).
        """
        #logging.debug("clear_line()")
        self.screen[self.cursorY] = array('u', u' ' * self.cols)
        self.renditions[self.cursorY] = array('u', c * self.cols)

    def clear_line_from_cursor(self, n):
        """
        CSI*n*K EL (Erase in Line).  This escape sequence uses the following
        rules:

        ======  ==============================  ===
        Esc[K   Clear screen from cursor right  EL0
        Esc[0K  Clear screen from cursor right  EL0
        Esc[1K  Clear screen from cursor left   EL1
        Esc[2K  Clear entire line               ED2
        """
        #logging.debug('clear_line_from_cursor(%s)' % n)
        try:
            n = int(n)
            n = 0
            0: self.clear_line_from_cursor_right,
            1: self.clear_line_from_cursor_left,
            2: self.clear_line
        }
        try:
            logging.error(_(
                "Error: Unsupported number for CSI escape sequence K"))
        try:
            pass
        try:
            pass

    def set_led_state(self, n):
        """
        Sets the values the dict, self.leds depending on *n* using the following
        rules:

        ======  ======================  ======
        Esc[0q  Turn off all four leds  DECLL0
        Esc[1q  Turn on LED #1          DECLL1
        Esc[2q  Turn on LED #2          DECLL2
        Esc[3q  Turn on LED #3          DECLL3
        Esc[4q  Turn on LED #4          DECLL4

        .. note:: These aren't implemented in Gate One's GUI (yet) but they certainly kept track of!
        """
        leds = n.split(';')
        for led in leds:
            led = int(led)
            if led == 0:
                self.leds[1] = False
                self.leds[2] = False
                self.leds[3] = False
                self.leds[4] = False
            else:
                self.leds[led] = True
        try:
            for callback in self.callbacks[CALLBACK_LEDS].values():
                callback(led)
            pass

    def _set_rendition(self, n):
        """
        Sets :attr:`self.renditions[self.cursorY][self.cursorX]` equal to
        *n.split(';')*.

        *n* is expected to be a string of ECMA-48 rendition numbers separated by
        semicolons.  Example::

            '0;1;31'

        ...will result in::

            [0, 1, 31]

        Note that the numbers were converted to integers and the order was
        preserved.
        """
        #logging.debug("_set_rendition(%s)" % n)
        cursorY = self.cursorY
        cursorX = self.cursorX
        if cursorX >= self.cols: # We're at the end of the row
            try:
                if len(self.renditions[cursorY]) <= cursorX:
                    # Make it all longer
                    self.renditions[cursorY].append(u' ') # Make it longer
                    self.screen[cursorY].append(u'\x00') # This needs to match
                # This can happen if the rate limiter kicks in and starts
                # cutting off escape sequences at random.
                return # Don't bother attempting to process anything else
        if cursorY >= self.rows:
                "cursorY >= self.rows!  This is either a bug or just a symptom "
                "of the rate limiter kicking in."))
            return # Don't bother setting renditions past the bottom
        if not n: # or \x1b[m (reset)
            # First char in PUA Plane 16 is always the default:
            self.cur_rendition = unichr(1000) # Should be reset (e.g. [0])
            return # No need for further processing; save some CPU
        # Convert the string (e.g. '0;1;32') to a list (e.g. [0,1,32]
        new_renditions = [int(a) for a in n.split(';') if a != '']
        # Handle 256-color renditions by getting rid of the (38|48);5 part and
        # incrementing foregrounds by 1000 and backgrounds by 10000 so we can
        # tell them apart in _spanify_screen().
        try:
            if 38 in new_renditions:
                foreground_index = new_renditions.index(38)
                if len(new_renditions[foreground_index:]) >= 2:
                    if new_renditions[foreground_index+1] == 5:
                        # This is a valid 256-color rendition (38;5;<num>)
                        new_renditions.pop(foreground_index) # Goodbye 38
                        new_renditions.pop(foreground_index) # Goodbye 5
                        new_renditions[foreground_index] += 1000
            if 48 in new_renditions:
                background_index = new_renditions.index(48)
                if len(new_renditions[background_index:]) >= 2:
                    if new_renditions[background_index+1] == 5:
                        # This is a valid 256-color rendition (48;5;<num>)
                        new_renditions.pop(background_index) # Goodbye 48
                        new_renditions.pop(background_index) # Goodbye 5
                        new_renditions[background_index] += 10000
            # Likely that the rate limiter has caused all sorts of havoc with
            # escape sequences.  Just ignore it and halt further processing
            return
        out_renditions = []
        for rend in new_renditions:
            if rend == 0:
                out_renditions = [0]
            else:
        if out_renditions[0] == 0:
            # If it starts with 0 there's no need to combine it with the
            # previous rendition...
            reduced = _reduce_renditions(out_renditions)
            if reduced not in self.renditions_store.values():
                new_ref_point = self.rend_counter.next()
                self.renditions_store.update({new_ref_point: reduced})
                self.cur_rendition = new_ref_point
            else: # Find the right reference point to use
                for k, v in self.renditions_store.items():
                    if reduced == v:
                        self.cur_rendition = k
            return
        new_renditions = out_renditions
        cur_rendition_list = self.renditions_store[self.cur_rendition]
        reduced = _reduce_renditions(cur_rendition_list + new_renditions)
        if reduced not in self.renditions_store.values():
            new_ref_point = self.rend_counter.next()
            self.renditions_store.update({new_ref_point: reduced})
            self.cur_rendition = new_ref_point
        else: # Find the right reference point to use
            for k, v in self.renditions_store.items():
                if reduced == v:
                    self.cur_rendition = k

    def _opt_handler(self, chars):
        """
        Optional special escape sequence handler for sequences matching
        RE_OPT_SEQ.  If CALLBACK_OPT is defined it will be called like so::

            self.callbacks[CALLBACK_OPT](chars)

        Applications can use this escape sequence to define whatever special
        handlers they like.  It works like this: If an escape sequence is
        encountered matching RE_OPT_SEQ this method will be called with the
        inbetween *chars* (e.g. \x1b]_;<chars>\x07) as the argument.

        Applications can then do what they wish with *chars*.

        .. note:: I added this functionality so that plugin authors would have a mechanism to communicate with terminal applications.  See the SSH plugin for an example of how this can be done (there's channels of communication amongst ssh_connect.py, ssh.js, and ssh.py).
        """
        try:
            for callback in self.callbacks[CALLBACK_OPT].values():
                callback(chars)
            # High likelyhood that nothing is defined.  No biggie.
            pass

    def _spanify_screen(self):
        """
        Iterates over the lines in *screen* and *renditions*, applying HTML
        markup (span tags) where appropriate and returns the result as a list of
        lines. It also marks the cursor position via a <span> tag at the
        appropriate location.
        """
        #logging.debug("_spanify_screen()")
        results = []
        # NOTE: Why these duplicates of self.* and globals?  Local variable
        # lookups are faster--especially in loops.
        special = SPECIAL
        rendition_classes = RENDITION_CLASSES
        screen = self.screen
        renditions = self.renditions
        renditions_store = self.renditions_store
        if len(self.prev_dump) != len(screen):
            # Fix it to be equal--assume first time/screen reset/resize/etc
            # Just fill it with empty strings (only the length matters here)
            self.prev_dump = [[] for a in screen]
            self.prev_dump_rend = [[] for a in screen]
        # The html_cache may need to be fixed as well
        if len(self.html_cache) != len(screen):
            self.html_cache = [u'' for a in screen] # Essentially a reset
        spancount = 0
        current_classes = set()
        prev_rendition = None
        foregrounds = ('f0','f1','f2','f3','f4','f5','f6','f7')
        backgrounds = ('b0','b1','b2','b3','b4','b5','b6','b7')
        html_entities = {"&": "&amp;", '<': '&lt;', '>': '&gt;'}
        for linecount, line_rendition in enumerate(izip(screen, renditions)):
            line = line_rendition[0]
            rendition = line_rendition[1]
            if linecount != cursorY and self.prev_dump[linecount] == line:
                if '<span class="cursor">' not in self.html_cache[linecount]:
                    if self.prev_dump_rend[linecount] == rendition:
                        # No change since the last dump.  Use the cache...
                        results.append(self.html_cache[linecount])
                        continue # Nothing changed so move on to the next line
            outline = ""
            if current_classes:
                outline += '<span class="%s">' % " ".join(current_classes)
            charcount = 0
            # TODO: Figure out if there's a faster way to process each character
            for char, rend in izip(line, rendition):
                rend = renditions_store[rend] # Get actual rendition
                if ord(char) >= special: # Special stuff =)
                    # Obviously, not really a single character
                    if not Image: # Can't use images in the terminal
                        outline += "<i>Image file</i>"
                        continue # Can't do anything else
                    image_file = self.images[char]
                    image_file.seek(0) # Back to the start
                    image_data = image_file.read()
                    # PIL likes StringIO objects for some reason
                    i = StringIO.StringIO(image_data)
                    try:
                        im = Image.open(i)
                    except IOError:
                        # i.e. PIL couldn't identify the file
                    # TODO: Make these sizes adjustable:
                    if im.size[0] > 640 or im.size[1] > 480:
                        # Probably too big to send to browser as a data URI.
                        if im: # Resize it...
                            # 640x480 should come in <32k for most stuff
                            try:
                                im.thumbnail((640, 480), Image.ANTIALIAS)
                                im.save(image_file, im.format)
                                # Re-read it in
                                image_file.seek(0)
                                image_data = image_file.read()
                            except IOError:
                                # Sometimes PIL will throw this if it can't read
                                # the image.
                                outline += "<i>Problem displaying this image</i>"
                                continue
                        else: # Generic error
                            outline += "<i>Problem displaying this image</i>"
                            continue
                    # Need to encode base64 to create a data URI
                    encoded = base64.b64encode(image_data).replace('\n', '')
                    data_uri = "data:image/%s;base64,%s" % (
                        im.format.lower(), encoded)
                    outline += '<img src="%s" width="%s" height="%s">' % (
                        data_uri, im.size[0], im.size[1])
                    continue
                if char in "&<>":
                    # Have to convert ampersands and lt/gt to HTML entities
                    char = html_entities[char]
                if rend == prev_rendition:
                    # Shortcut...  So we can skip all the logic below
                    changed = False
                else:
                    prev_rendition = rend
                if changed and rend:
                    classes = imap(rendition_classes.get, rend)
                    for _class in classes:
                        if _class and _class not in current_classes:
                            # Something changed...  Start a new span
                            if spancount:
                                outline += "</span>"
                                spancount -= 1
                            if 'reset' in _class:
                                if _class == 'reset':
                                    current_classes = set()
                                    if spancount:
                                        for i in xrange(spancount):
                                            outline += "</span>"
                                        spancount = 0
                                else:
                                    reset_class = _class.split('reset')[0]
                                    if reset_class == 'foreground':
                                        # Remove any foreground classes
                                        [current_classes.pop(i) for i, a in
                                        enumerate(current_classes) if a in
                                        foregrounds
                                        ]
                                    elif reset_class == 'background':
                                        backgrounds
                                        ]
                                    else:
                                        try:
                                            current_classes.remove(reset_class)
                                        except ValueError:
                                            # Trying to reset something that was
                                            # never set.  Ignore
                                            pass
                            else:
                                if _class in foregrounds:
                                    [current_classes.pop(i) for i, a in
                                    enumerate(current_classes) if a in
                                    foregrounds
                                    ]
                                elif _class in backgrounds:
                                    backgrounds
                                    ]
                                current_classes.add(_class)
                    if current_classes:
                        outline += '<span class="%s">' % " ".join(current_classes)
                        spancount += 1
                if linecount == cursorY and charcount == cursorX: # Cursor position
                    if self.show_cursor:
                        outline += '<span class="cursor">%s</span>' % char
                    else:
                        outline += char
                else:
                    outline += char
                charcount += 1
            self.prev_dump[linecount] = line[:]
            self.prev_dump_rend[linecount] = rendition[:]
            if outline:
                # Make sure all renditions terminate at the end of the line
                for whatever in xrange(spancount):
                    outline += "</span>"
                results.append(outline)
                self.html_cache[linecount] = outline
            else:
                results.append(None) # 'null' is shorter than 4 spaces
                self.html_cache[linecount] = None
            # NOTE: The client has been programmed to treat None (aka null in
            #       JavaScript) as blank lines.
        for whatever in xrange(spancount): # Bit of cleanup to be safe
            results[-1] += "</span>"
        return results

    def _spanify_scrollback(self):
        """
        Spanifies everything inside *screen* using *renditions*.  This differs
        from _spanify_screen() in that it doesn't apply any logic to detect the
        location of the cursor (to make it just a tiny bit faster).
        """
        # NOTE: See the comments in _spanify_screen() for details on this logic
        screen = self.scrollback_buf
        renditions = self.scrollback_renditions
        for line, rendition in izip(screen, renditions):
                    try:
                            try:
                                continue
                            continue
                    continue
                else:
                if changed and rend != None:
                                else:
                                        ]
                                        ]
                                    else:
                                        try:
                                            pass
                            else:
                                    ]
                                    ]
                outline += char
            else:
                results.append(None)

    def dump_html(self, renditions=True):
        """
        Dumps the terminal screen as a list of HTML-formatted lines.  If
        *renditions* is True (default) then terminal renditions will be
        converted into HTML <span> elements so they will be displayed properly
        in a browser.  Otherwise only the cursor <span> will be added to mark
        its location.

        .. note:: This places <span class="cursor">(current character)</span> around the cursor location.
        """
        if renditions: # i.e. Use stylized text
            screen = self._spanify_screen()
            scrollback = []
            if self.scrollback_buf:
                scrollback = self._spanify_scrollback()
        else:
            cursorX = self.cursorX
            cursorY = self.cursorY
            screen = []
            for y, row in enumerate(self.screen):
                if y == cursorY:
                    cursor_row = ""
                    for x, c in enumerate(row):
                        if x == cursorX:
                            cursor_row += '<span class="cursor">%s</span>' % c
                        else:
                            cursor_row += char
                    screen.append(cursor_row)
                else:
                    screen.append("".join(row))
        # Empty the scrollback buffer:
        return (scrollback, screen)

    def dump_plain(self):
        """
        Dumps the screen and the scrollback buffer as-is then empties the
        scrollback buffer.
        """
        scrollback = self.scrollback_buf

    def dump_components(self):
        """
        Dumps the screen and renditions as-is, the scrollback buffer as HTML,
        and the current cursor coordinates.  Also, empties the scrollback buffer

        .. note:: This was used in some performance-related experiments but might be useful for other patterns in the future so I've left it here.
        """
        screen = [a.tounicode() for a in self.screen]
        scrollback = []
        if self.scrollback_buf:
            # Process the scrollback buffer into HTML
            scrollback = self._spanify_scrollback(
                self.scrollback_buf, self.scrollback_renditions)
        return (scrollback, screen, self.renditions, self.cursorY, self.cursorX)

    def dump(self):
        """
        Returns self.screen as a list of strings with no formatting.
        No scrollback buffer.  No renditions.  It is meant to be used to get a
        quick glance of what is being displayed (when debugging).

        .. note:: This method does not empty the scrollback buffer.
        """
        out = []
        for line in self.screen:
            line_out = ""
            for char in line:
                if len(char) > 1: # This is an image (or similar)
                    line_out += u'⬚' # Use a dotted square as a placeholder
                else:
                    line_out += char
            out.append(line_out)
        return out

# This is here to make it easier for someone to produce an HTML app that uses
# terminal.py
def css_colors(selector=None):
    """
    Returns a (long) string containing all the CSS styles in order to support
    color in an HTML terminal using the dump_html() function.  If *selector* is
    provided, all styles will be prefixed with said selector like so::

        ${selector} span.f0 { color: #5C5C5C; }

    Example::

        >>> css_colors("#gateone").splitlines()[1]
        '#gateone span.f0 { color: #5C5C5C; } /* Black */'
    """
    from string import Template
    colors_template = Template(CSS_COLORS)
    return colors_template.substitute(selector=selector)

CSS_COLORS = """\
/* 8-color Foregrounds */
${selector} span.f0 { color: #5C5C5C; } /* Black */
${selector} span.f1 { color: #D6292C; } /* Red */
${selector} span.f2 { color: #13A110; } /* Green */
${selector} span.f3 { color: #E9E900; } /* Yellow */
${selector} span.f4 { color: #1B4CFF; } /* Blue */
${selector} span.f5 { color: #E01DC6; } /* Magenta */
${selector} span.f6 { color: #28AAA8; } /* Cyan */
${selector} span.f7 { color: #eee; } /* White */
/* 8-color Foregrounds on 8-color Backgrounds */
${selector} span.f0.b0 { color: #444; background-color: #000; } /* Black on Black */
${selector} span.f0.b1 { color: #000; background-color: #B21818; } /* Black on Red */
${selector} span.f0.b2 { color: #000; background-color: green; } /* Black on Green */
${selector} span.f0.b3 { color: #000; background-color: #FFFF00; } /* Black on Yellow */
${selector} span.f0.b4 { color: #000; background-color: #1818B2; } /* Black on Blue */
${selector} span.f0.b5 { color: #000; background-color: #B218B2; } /* Black on Magenta */
${selector} span.f0.b6 { color: #000; background-color: #18B2B2; } /* Black on Cyan */
${selector} span.f0.b7 { color: #000; background-color: #B2B2B2; } /* Black on White */
/* 8-color Foregrounds with 'bright' (16-color) Backgrounds */
${selector} span.f0.bb0 { color: #444; background-color: #000; } /* Black on Black */
${selector} span.f0.bb1 { color: #000; background-color: #B21818; } /* Black on Bright Red */
${selector} span.f0.bb2 { color: #000; background-color: #00E40C; } /* Black on Bright Green */
${selector} span.f0.bb3 { color: #000; background-color: #FFFF00; } /* Black on Bright Yellow */
${selector} span.f0.bb4 { color: #000; background-color: #1818B2; } /* Black on Bright Blue */
${selector} span.f0.bb5 { color: #000; background-color: #B218B2; } /* Black on Bright Magenta */
${selector} span.f0.bb6 { color: #000; background-color: #18B2B2; } /* Black on Bright Cyan */
${selector} span.f0.bb7 { color: #000; background-color: #B2B2B2; } /* Black on Bright White */
/* 8-color Backgrounds */
${selector} span.b0  { background-color: #000; } /* Black (used when the rendition is set to 49) */
${selector} span.b1  { background-color: #B21818; } /* Red */
${selector} span.b2  { background-color: green; } /* Green */
${selector} span.b3  { background-color: #FFFF00; } /* Yellow */
${selector} span.b4  { background-color: #1818B2; } /* Blue */
${selector} span.b5  { background-color: #B218B2; } /* Magenta */
${selector} span.b6  { background-color: #18B2B2; } /* Cyan */
${selector} span.b7  { background-color: #B2B2B2; } /* White */
/* Reversed Foregrounds */
${selector} span.reverse { color: #000; background-color: #ccc; } /* Black on White (since this is the "Black" CSS) */
${selector} span.reverse.f0 { color: #444; background-color: #000; } /* Black on Black (#444 so you can still read it)*/
${selector} span.reverse.f1 { color: #000; background-color: #D6292C; } /* Red on Black */
${selector} span.reverse.f2 { color: #000; background-color: green; } /* Green on Black */
${selector} span.reverse.f3 { color: #000; background-color: #BDC000; } /* Yellow on Black */
${selector} span.reverse.f4 { color: #000; background-color: #1B4CFF; } /* Blue on Black */
${selector} span.reverse.f5 { color: #000; background-color: #E01DC6; } /* Magenta on Black */
${selector} span.reverse.f6 { color: #000; background-color: #28AAA8; } /* Cyan on Black */
${selector} span.reverse.f7 { color: #000; background-color: #eee; } /* White on Black */
/* 16-color ('Bright') Foregrounds */
${selector} span.bf0 { color: #5C5C5C; } /* Bright Black!  I never knew thee existed! */
${selector} span.bf1 { color: #D6292C; } /* Bright Red */
${selector} span.bf2 { color: #13A110; } /* Bright Green */
${selector} span.bf3 { color: #E9E900; } /* Bright Yellow */
${selector} span.bf4 { color: #1B4CFF; } /* Bright Blue */
${selector} span.bf5 { color: #E01DC6; } /* Bright Magenta */
${selector} span.bf6 { color: #28AAA8; } /* Bright Cyan */
${selector} span.bf7 { color: #eee; } /* Bright White (use this on dentures) */
${selector} span.bf0.bb1 { color: #000; background-color: #B21818; } /* Bright Black on Bright Red */
${selector} span.bf0.bb2 { color: #000; background-color: #00E40C; } /* Bright Black on Bright Green */
${selector} span.bf0.bb3 { color: #000; background-color: #FFFF00; } /* Bright Black on Bright Yellow */
${selector} span.bf0.bb4 { color: #000; background-color: #1818B2; } /* Bright Black on Bright Blue */
${selector} span.bf0.bb5 { color: #000; background-color: #B218B2; } /* Bright Black on Bright Magenta */
${selector} span.bf0.bb6 { color: #000; background-color: #18B2B2; } /* Bright Black on Bright Cyan */
${selector} span.bf0.bb7 { color: #000; background-color: #B2B2B2; } /* Bright Black on Bright White */
/* 16-color ('Bright') Backgrounds */
${selector} span.bb0  { background-color: #000; } /* Bright Black */
${selector} span.bb1  { background-color: #B21818; } /* Bright Red */
${selector} span.bb2  { background-color: #00E40C; } /* Bright Green */
${selector} span.bb3  { background-color: #FFFF00; } /* Bright Yellow */
${selector} span.bb4  { background-color: #1818B2; } /* Bright Blue */
${selector} span.bb5  { background-color: #B218B2; } /* Bright Magenta */
${selector} span.bb6  { background-color: #18B2B2; } /* Bright Cyan */
${selector} span.bb7  { background-color: #B2B2B2; } /* Bright White */
${selector} span.reverse { color: #000; background-color: #ccc; } /* Black on White */
${selector} span.reverse.bf0 { color: #000; background-color: #ccc; } /* Black */
${selector} span.reverse.bf1 { color: #000; background-color: #D6292C; } /* Red */
${selector} span.reverse.bf2 { color: #000; background-color: #13A110; } /* Green */
${selector} span.reverse.bf3 { color: #000; background-color: #BDC000; } /* Yellow */
${selector} span.reverse.bf4 { color: #000; background-color: #1B4CFF; } /* Blue */
${selector} span.reverse.bf5 { color: #000; background-color: #E01DC6; } /* Magenta */
${selector} span.reverse.bf6 { color: #000; background-color: #28AAA8; } /* Cyan */
${selector} span.reverse.bf7 { color: #000; background-color: #eee; } /* White */
${selector} span.encircle {
    border: .1em;
    border-radius: .7em;
    -moz-border-radius: 1em;
    display: inline-block;
    margin-right: -0.15em; /* Because we scale it down to make the box normal-sized we have to move it to the left slightly to have it line up properly */
    border-color: #BBB;
    border-style: solid;
    /* WebKit */
    -webkit-transition: -webkit-transform 1s ease-in-out;
    -webkit-transform: scale(.9);
    /* Firefox */
    -moz-transition: -moz-transform 1s ease-in-out;
    -moz-transform: scale(.9);
    /* IE9+ */
    -ms-transition: -ms-transform 1s ease-in-out;
    -ms-transform: scale(.9);
    /* Opera */
    -o-transition: -o-transform 1s ease-in-out;
    -o-transform: scale(.9);
    /* Future CSS3 standard */
    transition: transform 1s ease-in-out;
    transform: scale(.9);
}
${selector} span.rightline {
    border-right: .2em;
    border-top: 0;
    border-bottom: 0;
    border-left: 0;
    margin-right: -0.2em;
    /* IE9+ */
    /* Opera */
}
${selector} span.rightdoubleline {
    border-right: .3em;
    border-style: double;
    margin-right: -0.3em;
    /* IE9+ */
    /* Opera */
}
${selector} span.leftline {
    border-left: .2em;
    border-right: 0;
    /* IE9+ */
    /* Opera */
}
${selector} span.leftdoubleline {
    border-left: .3em;
    /* IE9+ */
    /* Opera */
}
${selector} span.bold { font-weight: bold; }
${selector} span.dim { opacity: 0.7; }
${selector} span.hidden { color: transparent !important;}
${selector} span.underline { text-decoration: underline; }
${selector} span.overline { text-decoration: overline; }
${selector} span.italic { font-style: italic; }
${selector} span.strike { text-decoration: line-through; }
${selector} span.frame {
    /* An inset shadow emulates a border without making the box bigger */
    -moz-box-shadow: inset 0.1em 0.1em 0.3em #000;
    -webkit-box-shadow: inset 0.1em 0.1em 0.3em #000;
    box-shadow: inset 0.1em 0.1em 0.3em #000;
}
@-webkit-keyframes blinker {
  from { opacity: 1.0; }
  to { opacity: 0.0; }
}
@-moz-keyframes blinker {
}
@keyframes blinker {
}
${selector} span.blink {
    -webkit-animation-name: blinker;
    -webkit-animation-iteration-count: infinite;
    -webkit-animation-timing-function: cubic-bezier(1.0,0,0,1.0);
    -webkit-animation-duration: 1s;
    -moz-animation-iteration-count: infinite;
    -moz-animation-name: blinker;
    -moz-animation-duration: 1s;
    -moz-animation-timing-function: cubic-bezier(1.0,0,0,1.0);
    animation-name: blinker;
    animation-iteration-count: infinite;
    animation-timing-function: cubic-bezier(1.0,0,0,1.0);
    animation-duration: 1s;
}
${selector} span.fastblink {
    -webkit-animation-duration: 0.5s;
    -moz-animation-duration: 0.5s;
    animation-duration: 0.5s;
}
${selector} span.fx0 {color: #000000;} ${selector} span.bx0 {background-color: #000000;}  ${selector} span.reverse.fx0 {background-color: #000000; color: inherit;} ${selector} span.reverse.bx0 {color: #000000; background-color: inherit;}
${selector} span.fx1 {color: #800000;} ${selector} span.bx1 {background-color: #800000;}  ${selector} span.reverse.fx1 {background-color: #800000; color: inherit;} ${selector} span.reverse.bx1 {color: #800000; background-color: inherit;}
${selector} span.fx2 {color: #008000;} ${selector} span.bx2 {background-color: #008000;}  ${selector} span.reverse.fx2 {background-color: #008000; color: inherit;} ${selector} span.reverse.bx2 {color: #008000; background-color: inherit;}
${selector} span.fx3 {color: #808000;} ${selector} span.bx3 {background-color: #808000;}  ${selector} span.reverse.fx3 {background-color: #808000; color: inherit;} ${selector} span.reverse.bx3 {color: #808000; background-color: inherit;}
${selector} span.fx4 {color: #000080;} ${selector} span.bx4 {background-color: #000080;}  ${selector} span.reverse.fx4 {background-color: #000080; color: inherit;} ${selector} span.reverse.bx4 {color: #000080; background-color: inherit;}
${selector} span.fx5 {color: #800080;} ${selector} span.bx5 {background-color: #800080;}  ${selector} span.reverse.fx5 {background-color: #800080; color: inherit;} ${selector} span.reverse.bx5 {color: #800080; background-color: inherit;}
${selector} span.fx6 {color: #008080;} ${selector} span.bx6 {background-color: #008080;}  ${selector} span.reverse.fx6 {background-color: #008080; color: inherit;} ${selector} span.reverse.bx6 {color: #008080; background-color: inherit;}
${selector} span.fx7 {color: #c0c0c0;} ${selector} span.bx7 {background-color: #c0c0c0;}  ${selector} span.reverse.fx7 {background-color: #c0c0c0; color: inherit;} ${selector} span.reverse.bx7 {color: #c0c0c0; background-color: inherit;}
${selector} span.fx8 {color: #808080;} ${selector} span.bx8 {background-color: #808080;}  ${selector} span.reverse.fx8 {background-color: #808080; color: inherit;} ${selector} span.reverse.bx8 {color: #808080; background-color: inherit;}
${selector} span.fx9 {color: #ff0000;} ${selector} span.bx9 {background-color: #ff0000;}  ${selector} span.reverse.fx9 {background-color: #ff0000; color: inherit;} ${selector} span.reverse.bx9 {color: #ff0000; background-color: inherit;}
${selector} span.fx10 {color: #00ff00;} ${selector} span.bx10 {background-color: #00ff00;}  ${selector} span.reverse.fx10 {background-color: #00ff00; color: inherit;} ${selector} span.reverse.bx10 {color: #00ff00; background-color: inherit;}
${selector} span.fx11 {color: #ffff00;} ${selector} span.bx11 {background-color: #ffff00;}  ${selector} span.reverse.fx11 {background-color: #ffff00; color: inherit;} ${selector} span.reverse.bx11 {color: #ffff00; background-color: inherit;}
${selector} span.fx12 {color: #0000ff;} ${selector} span.bx12 {background-color: #0000ff;}  ${selector} span.reverse.fx12 {background-color: #0000ff; color: inherit;} ${selector} span.reverse.bx12 {color: #0000ff; background-color: inherit;}
${selector} span.fx13 {color: #ff00ff;} ${selector} span.bx13 {background-color: #ff00ff;}  ${selector} span.reverse.fx13 {background-color: #ff00ff; color: inherit;} ${selector} span.reverse.bx13 {color: #ff00ff; background-color: inherit;}
${selector} span.fx14 {color: #00ffff;} ${selector} span.bx14 {background-color: #00ffff;}  ${selector} span.reverse.fx14 {background-color: #00ffff; color: inherit;} ${selector} span.reverse.bx14 {color: #00ffff; background-color: inherit;}
${selector} span.fx15 {color: #ffffff;} ${selector} span.bx15 {background-color: #ffffff;}  ${selector} span.reverse.fx15 {background-color: #ffffff; color: inherit;} ${selector} span.reverse.bx15 {color: #ffffff; background-color: inherit;}
${selector} span.fx16 {color: #000000;} ${selector} span.bx16 {background-color: #000000;}  ${selector} span.reverse.fx16 {background-color: #000000; color: inherit;} ${selector} span.reverse.bx16 {color: #000000; background-color: inherit;}
${selector} span.fx17 {color: #00005f;} ${selector} span.bx17 {background-color: #00005f;}  ${selector} span.reverse.fx17 {background-color: #00005f; color: inherit;} ${selector} span.reverse.bx17 {color: #00005f; background-color: inherit;}
${selector} span.fx18 {color: #000087;} ${selector} span.bx18 {background-color: #000087;}  ${selector} span.reverse.fx18 {background-color: #000087; color: inherit;} ${selector} span.reverse.bx18 {color: #000087; background-color: inherit;}
${selector} span.fx19 {color: #0000af;} ${selector} span.bx19 {background-color: #0000af;}  ${selector} span.reverse.fx19 {background-color: #0000af; color: inherit;} ${selector} span.reverse.bx19 {color: #0000af; background-color: inherit;}
${selector} span.fx20 {color: #0000d7;} ${selector} span.bx20 {background-color: #0000d7;}  ${selector} span.reverse.fx20 {background-color: #0000d7; color: inherit;} ${selector} span.reverse.bx20 {color: #0000d7; background-color: inherit;}
${selector} span.fx21 {color: #0000ff;} ${selector} span.bx21 {background-color: #0000ff;}  ${selector} span.reverse.fx21 {background-color: #0000ff; color: inherit;} ${selector} span.reverse.bx21 {color: #0000ff; background-color: inherit;}
${selector} span.fx22 {color: #005f00;} ${selector} span.bx22 {background-color: #005f00;}  ${selector} span.reverse.fx22 {background-color: #005f00; color: inherit;} ${selector} span.reverse.bx22 {color: #005f00; background-color: inherit;}
${selector} span.fx23 {color: #005f5f;} ${selector} span.bx23 {background-color: #005f5f;}  ${selector} span.reverse.fx23 {background-color: #005f5f; color: inherit;} ${selector} span.reverse.bx23 {color: #005f5f; background-color: inherit;}
${selector} span.fx24 {color: #005f87;} ${selector} span.bx24 {background-color: #005f87;}  ${selector} span.reverse.fx24 {background-color: #005f87; color: inherit;} ${selector} span.reverse.bx24 {color: #005f87; background-color: inherit;}
${selector} span.fx25 {color: #005faf;} ${selector} span.bx25 {background-color: #005faf;}  ${selector} span.reverse.fx25 {background-color: #005faf; color: inherit;} ${selector} span.reverse.bx25 {color: #005faf; background-color: inherit;}
${selector} span.fx26 {color: #005fd7;} ${selector} span.bx26 {background-color: #005fd7;}  ${selector} span.reverse.fx26 {background-color: #005fd7; color: inherit;} ${selector} span.reverse.bx26 {color: #005fd7; background-color: inherit;}
${selector} span.fx27 {color: #005fff;} ${selector} span.bx27 {background-color: #005fff;}  ${selector} span.reverse.fx27 {background-color: #005fff; color: inherit;} ${selector} span.reverse.bx27 {color: #005fff; background-color: inherit;}
${selector} span.fx28 {color: #008700;} ${selector} span.bx28 {background-color: #008700;}  ${selector} span.reverse.fx28 {background-color: #008700; color: inherit;} ${selector} span.reverse.bx28 {color: #008700; background-color: inherit;}
${selector} span.fx29 {color: #00875f;} ${selector} span.bx29 {background-color: #00875f;}  ${selector} span.reverse.fx29 {background-color: #00875f; color: inherit;} ${selector} span.reverse.bx29 {color: #00875f; background-color: inherit;}
${selector} span.fx30 {color: #008787;} ${selector} span.bx30 {background-color: #008787;}  ${selector} span.reverse.fx30 {background-color: #008787; color: inherit;} ${selector} span.reverse.bx30 {color: #008787; background-color: inherit;}
${selector} span.fx31 {color: #0087af;} ${selector} span.bx31 {background-color: #0087af;}  ${selector} span.reverse.fx31 {background-color: #0087af; color: inherit;} ${selector} span.reverse.bx31 {color: #0087af; background-color: inherit;}
${selector} span.fx32 {color: #0087d7;} ${selector} span.bx32 {background-color: #0087d7;}  ${selector} span.reverse.fx32 {background-color: #0087d7; color: inherit;} ${selector} span.reverse.bx32 {color: #0087d7; background-color: inherit;}
${selector} span.fx33 {color: #0087ff;} ${selector} span.bx33 {background-color: #0087ff;}  ${selector} span.reverse.fx33 {background-color: #0087ff; color: inherit;} ${selector} span.reverse.bx33 {color: #0087ff; background-color: inherit;}
${selector} span.fx34 {color: #00af00;} ${selector} span.bx34 {background-color: #00af00;}  ${selector} span.reverse.fx34 {background-color: #00af00; color: inherit;} ${selector} span.reverse.bx34 {color: #00af00; background-color: inherit;}
${selector} span.fx35 {color: #00af5f;} ${selector} span.bx35 {background-color: #00af5f;}  ${selector} span.reverse.fx35 {background-color: #00af5f; color: inherit;} ${selector} span.reverse.bx35 {color: #00af5f; background-color: inherit;}
${selector} span.fx36 {color: #00af87;} ${selector} span.bx36 {background-color: #00af87;}  ${selector} span.reverse.fx36 {background-color: #00af87; color: inherit;} ${selector} span.reverse.bx36 {color: #00af87; background-color: inherit;}
${selector} span.fx37 {color: #00afaf;} ${selector} span.bx37 {background-color: #00afaf;}  ${selector} span.reverse.fx37 {background-color: #00afaf; color: inherit;} ${selector} span.reverse.bx37 {color: #00afaf; background-color: inherit;}
${selector} span.fx38 {color: #00afd7;} ${selector} span.bx38 {background-color: #00afd7;}  ${selector} span.reverse.fx38 {background-color: #00afd7; color: inherit;} ${selector} span.reverse.bx38 {color: #00afd7; background-color: inherit;}
${selector} span.fx39 {color: #00afff;} ${selector} span.bx39 {background-color: #00afff;}  ${selector} span.reverse.fx39 {background-color: #00afff; color: inherit;} ${selector} span.reverse.bx39 {color: #00afff; background-color: inherit;}
${selector} span.fx40 {color: #00d700;} ${selector} span.bx40 {background-color: #00d700;}  ${selector} span.reverse.fx40 {background-color: #00d700; color: inherit;} ${selector} span.reverse.bx40 {color: #00d700; background-color: inherit;}
${selector} span.fx41 {color: #00d75f;} ${selector} span.bx41 {background-color: #00d75f;}  ${selector} span.reverse.fx41 {background-color: #00d75f; color: inherit;} ${selector} span.reverse.bx41 {color: #00d75f; background-color: inherit;}
${selector} span.fx42 {color: #00d787;} ${selector} span.bx42 {background-color: #00d787;}  ${selector} span.reverse.fx42 {background-color: #00d787; color: inherit;} ${selector} span.reverse.bx42 {color: #00d787; background-color: inherit;}
${selector} span.fx43 {color: #00d7af;} ${selector} span.bx43 {background-color: #00d7af;}  ${selector} span.reverse.fx43 {background-color: #00d7af; color: inherit;} ${selector} span.reverse.bx43 {color: #00d7af; background-color: inherit;}
${selector} span.fx44 {color: #00d7d7;} ${selector} span.bx44 {background-color: #00d7d7;}  ${selector} span.reverse.fx44 {background-color: #00d7d7; color: inherit;} ${selector} span.reverse.bx44 {color: #00d7d7; background-color: inherit;}
${selector} span.fx45 {color: #00d7ff;} ${selector} span.bx45 {background-color: #00d7ff;}  ${selector} span.reverse.fx45 {background-color: #00d7ff; color: inherit;} ${selector} span.reverse.bx45 {color: #00d7ff; background-color: inherit;}
${selector} span.fx46 {color: #00ff00;} ${selector} span.bx46 {background-color: #00ff00;}  ${selector} span.reverse.fx46 {background-color: #00ff00; color: inherit;} ${selector} span.reverse.bx46 {color: #00ff00; background-color: inherit;}
${selector} span.fx47 {color: #00ff5f;} ${selector} span.bx47 {background-color: #00ff5f;}  ${selector} span.reverse.fx47 {background-color: #00ff5f; color: inherit;} ${selector} span.reverse.bx47 {color: #00ff5f; background-color: inherit;}
${selector} span.fx48 {color: #00ff87;} ${selector} span.bx48 {background-color: #00ff87;}  ${selector} span.reverse.fx48 {background-color: #00ff87; color: inherit;} ${selector} span.reverse.bx48 {color: #00ff87; background-color: inherit;}
${selector} span.fx49 {color: #00ffaf;} ${selector} span.bx49 {background-color: #00ffaf;}  ${selector} span.reverse.fx49 {background-color: #00ffaf; color: inherit;} ${selector} span.reverse.bx49 {color: #00ffaf; background-color: inherit;}
${selector} span.fx50 {color: #00ffd7;} ${selector} span.bx50 {background-color: #00ffd7;}  ${selector} span.reverse.fx50 {background-color: #00ffd7; color: inherit;} ${selector} span.reverse.bx50 {color: #00ffd7; background-color: inherit;}
${selector} span.fx51 {color: #00ffff;} ${selector} span.bx51 {background-color: #00ffff;}  ${selector} span.reverse.fx51 {background-color: #00ffff; color: inherit;} ${selector} span.reverse.bx51 {color: #00ffff; background-color: inherit;}
${selector} span.fx52 {color: #5f0000;} ${selector} span.bx52 {background-color: #5f0000;}  ${selector} span.reverse.fx52 {background-color: #5f0000; color: inherit;} ${selector} span.reverse.bx52 {color: #5f0000; background-color: inherit;}
${selector} span.fx53 {color: #5f005f;} ${selector} span.bx53 {background-color: #5f005f;}  ${selector} span.reverse.fx53 {background-color: #5f005f; color: inherit;} ${selector} span.reverse.bx53 {color: #5f005f; background-color: inherit;}
${selector} span.fx54 {color: #5f0087;} ${selector} span.bx54 {background-color: #5f0087;}  ${selector} span.reverse.fx54 {background-color: #5f0087; color: inherit;} ${selector} span.reverse.bx54 {color: #5f0087; background-color: inherit;}
${selector} span.fx55 {color: #5f00af;} ${selector} span.bx55 {background-color: #5f00af;}  ${selector} span.reverse.fx55 {background-color: #5f00af; color: inherit;} ${selector} span.reverse.bx55 {color: #5f00af; background-color: inherit;}
${selector} span.fx56 {color: #5f00d7;} ${selector} span.bx56 {background-color: #5f00d7;}  ${selector} span.reverse.fx56 {background-color: #5f00d7; color: inherit;} ${selector} span.reverse.bx56 {color: #5f00d7; background-color: inherit;}
${selector} span.fx57 {color: #5f00ff;} ${selector} span.bx57 {background-color: #5f00ff;}  ${selector} span.reverse.fx57 {background-color: #5f00ff; color: inherit;} ${selector} span.reverse.bx57 {color: #5f00ff; background-color: inherit;}
${selector} span.fx58 {color: #5f5f00;} ${selector} span.bx58 {background-color: #5f5f00;}  ${selector} span.reverse.fx58 {background-color: #5f5f00; color: inherit;} ${selector} span.reverse.bx58 {color: #5f5f00; background-color: inherit;}
${selector} span.fx59 {color: #5f5f5f;} ${selector} span.bx59 {background-color: #5f5f5f;}  ${selector} span.reverse.fx59 {background-color: #5f5f5f; color: inherit;} ${selector} span.reverse.bx59 {color: #5f5f5f; background-color: inherit;}
${selector} span.fx60 {color: #5f5f87;} ${selector} span.bx60 {background-color: #5f5f87;}  ${selector} span.reverse.fx60 {background-color: #5f5f87; color: inherit;} ${selector} span.reverse.bx60 {color: #5f5f87; background-color: inherit;}
${selector} span.fx61 {color: #5f5faf;} ${selector} span.bx61 {background-color: #5f5faf;}  ${selector} span.reverse.fx61 {background-color: #5f5faf; color: inherit;} ${selector} span.reverse.bx61 {color: #5f5faf; background-color: inherit;}
${selector} span.fx62 {color: #5f5fd7;} ${selector} span.bx62 {background-color: #5f5fd7;}  ${selector} span.reverse.fx62 {background-color: #5f5fd7; color: inherit;} ${selector} span.reverse.bx62 {color: #5f5fd7; background-color: inherit;}
${selector} span.fx63 {color: #5f5fff;} ${selector} span.bx63 {background-color: #5f5fff;}  ${selector} span.reverse.fx63 {background-color: #5f5fff; color: inherit;} ${selector} span.reverse.bx63 {color: #5f5fff; background-color: inherit;}
${selector} span.fx64 {color: #5f8700;} ${selector} span.bx64 {background-color: #5f8700;}  ${selector} span.reverse.fx64 {background-color: #5f8700; color: inherit;} ${selector} span.reverse.bx64 {color: #5f8700; background-color: inherit;}
${selector} span.fx65 {color: #5f875f;} ${selector} span.bx65 {background-color: #5f875f;}  ${selector} span.reverse.fx65 {background-color: #5f875f; color: inherit;} ${selector} span.reverse.bx65 {color: #5f875f; background-color: inherit;}
${selector} span.fx66 {color: #5f8787;} ${selector} span.bx66 {background-color: #5f8787;}  ${selector} span.reverse.fx66 {background-color: #5f8787; color: inherit;} ${selector} span.reverse.bx66 {color: #5f8787; background-color: inherit;}
${selector} span.fx67 {color: #5f87af;} ${selector} span.bx67 {background-color: #5f87af;}  ${selector} span.reverse.fx67 {background-color: #5f87af; color: inherit;} ${selector} span.reverse.bx67 {color: #5f87af; background-color: inherit;}
${selector} span.fx68 {color: #5f87d7;} ${selector} span.bx68 {background-color: #5f87d7;}  ${selector} span.reverse.fx68 {background-color: #5f87d7; color: inherit;} ${selector} span.reverse.bx68 {color: #5f87d7; background-color: inherit;}
${selector} span.fx69 {color: #5f87ff;} ${selector} span.bx69 {background-color: #5f87ff;}  ${selector} span.reverse.fx69 {background-color: #5f87ff; color: inherit;} ${selector} span.reverse.bx69 {color: #5f87ff; background-color: inherit;}
${selector} span.fx70 {color: #5faf00;} ${selector} span.bx70 {background-color: #5faf00;}  ${selector} span.reverse.fx70 {background-color: #5faf00; color: inherit;} ${selector} span.reverse.bx70 {color: #5faf00; background-color: inherit;}
${selector} span.fx71 {color: #5faf5f;} ${selector} span.bx71 {background-color: #5faf5f;}  ${selector} span.reverse.fx71 {background-color: #5faf5f; color: inherit;} ${selector} span.reverse.bx71 {color: #5faf5f; background-color: inherit;}
${selector} span.fx72 {color: #5faf87;} ${selector} span.bx72 {background-color: #5faf87;}  ${selector} span.reverse.fx72 {background-color: #5faf87; color: inherit;} ${selector} span.reverse.bx72 {color: #5faf87; background-color: inherit;}
${selector} span.fx73 {color: #5fafaf;} ${selector} span.bx73 {background-color: #5fafaf;}  ${selector} span.reverse.fx73 {background-color: #5fafaf; color: inherit;} ${selector} span.reverse.bx73 {color: #5fafaf; background-color: inherit;}
${selector} span.fx74 {color: #5fafd7;} ${selector} span.bx74 {background-color: #5fafd7;}  ${selector} span.reverse.fx74 {background-color: #5fafd7; color: inherit;} ${selector} span.reverse.bx74 {color: #5fafd7; background-color: inherit;}
${selector} span.fx75 {color: #5fafff;} ${selector} span.bx75 {background-color: #5fafff;}  ${selector} span.reverse.fx75 {background-color: #5fafff; color: inherit;} ${selector} span.reverse.bx75 {color: #5fafff; background-color: inherit;}
${selector} span.fx76 {color: #5fd700;} ${selector} span.bx76 {background-color: #5fd700;}  ${selector} span.reverse.fx76 {background-color: #5fd700; color: inherit;} ${selector} span.reverse.bx76 {color: #5fd700; background-color: inherit;}
${selector} span.fx77 {color: #5fd75f;} ${selector} span.bx77 {background-color: #5fd75f;}  ${selector} span.reverse.fx77 {background-color: #5fd75f; color: inherit;} ${selector} span.reverse.bx77 {color: #5fd75f; background-color: inherit;}
${selector} span.fx78 {color: #5fd787;} ${selector} span.bx78 {background-color: #5fd787;}  ${selector} span.reverse.fx78 {background-color: #5fd787; color: inherit;} ${selector} span.reverse.bx78 {color: #5fd787; background-color: inherit;}
${selector} span.fx79 {color: #5fd7af;} ${selector} span.bx79 {background-color: #5fd7af;}  ${selector} span.reverse.fx79 {background-color: #5fd7af; color: inherit;} ${selector} span.reverse.bx79 {color: #5fd7af; background-color: inherit;}
${selector} span.fx80 {color: #5fd7d7;} ${selector} span.bx80 {background-color: #5fd7d7;}  ${selector} span.reverse.fx80 {background-color: #5fd7d7; color: inherit;} ${selector} span.reverse.bx80 {color: #5fd7d7; background-color: inherit;}
${selector} span.fx81 {color: #5fd7ff;} ${selector} span.bx81 {background-color: #5fd7ff;}  ${selector} span.reverse.fx81 {background-color: #5fd7ff; color: inherit;} ${selector} span.reverse.bx81 {color: #5fd7ff; background-color: inherit;}
${selector} span.fx82 {color: #5fff00;} ${selector} span.bx82 {background-color: #5fff00;}  ${selector} span.reverse.fx82 {background-color: #5fff00; color: inherit;} ${selector} span.reverse.bx82 {color: #5fff00; background-color: inherit;}
${selector} span.fx83 {color: #5fff5f;} ${selector} span.bx83 {background-color: #5fff5f;}  ${selector} span.reverse.fx83 {background-color: #5fff5f; color: inherit;} ${selector} span.reverse.bx83 {color: #5fff5f; background-color: inherit;}
${selector} span.fx84 {color: #5fff87;} ${selector} span.bx84 {background-color: #5fff87;}  ${selector} span.reverse.fx84 {background-color: #5fff87; color: inherit;} ${selector} span.reverse.bx84 {color: #5fff87; background-color: inherit;}
${selector} span.fx85 {color: #5fffaf;} ${selector} span.bx85 {background-color: #5fffaf;}  ${selector} span.reverse.fx85 {background-color: #5fffaf; color: inherit;} ${selector} span.reverse.bx85 {color: #5fffaf; background-color: inherit;}
${selector} span.fx86 {color: #5fffd7;} ${selector} span.bx86 {background-color: #5fffd7;}  ${selector} span.reverse.fx86 {background-color: #5fffd7; color: inherit;} ${selector} span.reverse.bx86 {color: #5fffd7; background-color: inherit;}
${selector} span.fx87 {color: #5fffff;} ${selector} span.bx87 {background-color: #5fffff;}  ${selector} span.reverse.fx87 {background-color: #5fffff; color: inherit;} ${selector} span.reverse.bx87 {color: #5fffff; background-color: inherit;}
${selector} span.fx88 {color: #870000;} ${selector} span.bx88 {background-color: #870000;}  ${selector} span.reverse.fx88 {background-color: #870000; color: inherit;} ${selector} span.reverse.bx88 {color: #870000; background-color: inherit;}
${selector} span.fx89 {color: #87005f;} ${selector} span.bx89 {background-color: #87005f;}  ${selector} span.reverse.fx89 {background-color: #87005f; color: inherit;} ${selector} span.reverse.bx89 {color: #87005f; background-color: inherit;}
${selector} span.fx90 {color: #870087;} ${selector} span.bx90 {background-color: #870087;}  ${selector} span.reverse.fx90 {background-color: #870087; color: inherit;} ${selector} span.reverse.bx90 {color: #870087; background-color: inherit;}
${selector} span.fx91 {color: #8700af;} ${selector} span.bx91 {background-color: #8700af;}  ${selector} span.reverse.fx91 {background-color: #8700af; color: inherit;} ${selector} span.reverse.bx91 {color: #8700af; background-color: inherit;}
${selector} span.fx92 {color: #8700d7;} ${selector} span.bx92 {background-color: #8700d7;}  ${selector} span.reverse.fx92 {background-color: #8700d7; color: inherit;} ${selector} span.reverse.bx92 {color: #8700d7; background-color: inherit;}
${selector} span.fx93 {color: #8700ff;} ${selector} span.bx93 {background-color: #8700ff;}  ${selector} span.reverse.fx93 {background-color: #8700ff; color: inherit;} ${selector} span.reverse.bx93 {color: #8700ff; background-color: inherit;}
${selector} span.fx94 {color: #875f00;} ${selector} span.bx94 {background-color: #875f00;}  ${selector} span.reverse.fx94 {background-color: #875f00; color: inherit;} ${selector} span.reverse.bx94 {color: #875f00; background-color: inherit;}
${selector} span.fx95 {color: #875f5f;} ${selector} span.bx95 {background-color: #875f5f;}  ${selector} span.reverse.fx95 {background-color: #875f5f; color: inherit;} ${selector} span.reverse.bx95 {color: #875f5f; background-color: inherit;}
${selector} span.fx96 {color: #875f87;} ${selector} span.bx96 {background-color: #875f87;}  ${selector} span.reverse.fx96 {background-color: #875f87; color: inherit;} ${selector} span.reverse.bx96 {color: #875f87; background-color: inherit;}
${selector} span.fx97 {color: #875faf;} ${selector} span.bx97 {background-color: #875faf;}  ${selector} span.reverse.fx97 {background-color: #875faf; color: inherit;} ${selector} span.reverse.bx97 {color: #875faf; background-color: inherit;}
${selector} span.fx98 {color: #875fd7;} ${selector} span.bx98 {background-color: #875fd7;}  ${selector} span.reverse.fx98 {background-color: #875fd7; color: inherit;} ${selector} span.reverse.bx98 {color: #875fd7; background-color: inherit;}
${selector} span.fx99 {color: #875fff;} ${selector} span.bx99 {background-color: #875fff;}  ${selector} span.reverse.fx99 {background-color: #875fff; color: inherit;} ${selector} span.reverse.bx99 {color: #875fff; background-color: inherit;}
${selector} span.fx100 {color: #878700;} ${selector} span.bx100 {background-color: #878700;}  ${selector} span.reverse.fx100 {background-color: #878700; color: inherit;} ${selector} span.reverse.bx100 {color: #878700; background-color: inherit;}
${selector} span.fx101 {color: #87875f;} ${selector} span.bx101 {background-color: #87875f;}  ${selector} span.reverse.fx101 {background-color: #87875f; color: inherit;} ${selector} span.reverse.bx101 {color: #87875f; background-color: inherit;}
${selector} span.fx102 {color: #878787;} ${selector} span.bx102 {background-color: #878787;}  ${selector} span.reverse.fx102 {background-color: #878787; color: inherit;} ${selector} span.reverse.bx102 {color: #878787; background-color: inherit;}
${selector} span.fx103 {color: #8787af;} ${selector} span.bx103 {background-color: #8787af;}  ${selector} span.reverse.fx103 {background-color: #8787af; color: inherit;} ${selector} span.reverse.bx103 {color: #8787af; background-color: inherit;}
${selector} span.fx104 {color: #8787d7;} ${selector} span.bx104 {background-color: #8787d7;}  ${selector} span.reverse.fx104 {background-color: #8787d7; color: inherit;} ${selector} span.reverse.bx104 {color: #8787d7; background-color: inherit;}
${selector} span.fx105 {color: #8787ff;} ${selector} span.bx105 {background-color: #8787ff;}  ${selector} span.reverse.fx105 {background-color: #8787ff; color: inherit;} ${selector} span.reverse.bx105 {color: #8787ff; background-color: inherit;}
${selector} span.fx106 {color: #87af00;} ${selector} span.bx106 {background-color: #87af00;}  ${selector} span.reverse.fx106 {background-color: #87af00; color: inherit;} ${selector} span.reverse.bx106 {color: #87af00; background-color: inherit;}
${selector} span.fx107 {color: #87af5f;} ${selector} span.bx107 {background-color: #87af5f;}  ${selector} span.reverse.fx107 {background-color: #87af5f; color: inherit;} ${selector} span.reverse.bx107 {color: #87af5f; background-color: inherit;}
${selector} span.fx108 {color: #87af87;} ${selector} span.bx108 {background-color: #87af87;}  ${selector} span.reverse.fx108 {background-color: #87af87; color: inherit;} ${selector} span.reverse.bx108 {color: #87af87; background-color: inherit;}
${selector} span.fx109 {color: #87afaf;} ${selector} span.bx109 {background-color: #87afaf;}  ${selector} span.reverse.fx109 {background-color: #87afaf; color: inherit;} ${selector} span.reverse.bx109 {color: #87afaf; background-color: inherit;}
${selector} span.fx110 {color: #87afd7;} ${selector} span.bx110 {background-color: #87afd7;}  ${selector} span.reverse.fx110 {background-color: #87afd7; color: inherit;} ${selector} span.reverse.bx110 {color: #87afd7; background-color: inherit;}
${selector} span.fx111 {color: #87afff;} ${selector} span.bx111 {background-color: #87afff;}  ${selector} span.reverse.fx111 {background-color: #87afff; color: inherit;} ${selector} span.reverse.bx111 {color: #87afff; background-color: inherit;}
${selector} span.fx112 {color: #87d700;} ${selector} span.bx112 {background-color: #87d700;}  ${selector} span.reverse.fx112 {background-color: #87d700; color: inherit;} ${selector} span.reverse.bx112 {color: #87d700; background-color: inherit;}
${selector} span.fx113 {color: #87d75f;} ${selector} span.bx113 {background-color: #87d75f;}  ${selector} span.reverse.fx113 {background-color: #87d75f; color: inherit;} ${selector} span.reverse.bx113 {color: #87d75f; background-color: inherit;}
${selector} span.fx114 {color: #87d787;} ${selector} span.bx114 {background-color: #87d787;}  ${selector} span.reverse.fx114 {background-color: #87d787; color: inherit;} ${selector} span.reverse.bx114 {color: #87d787; background-color: inherit;}
${selector} span.fx115 {color: #87d7af;} ${selector} span.bx115 {background-color: #87d7af;}  ${selector} span.reverse.fx115 {background-color: #87d7af; color: inherit;} ${selector} span.reverse.bx115 {color: #87d7af; background-color: inherit;}
${selector} span.fx116 {color: #87d7d7;} ${selector} span.bx116 {background-color: #87d7d7;}  ${selector} span.reverse.fx116 {background-color: #87d7d7; color: inherit;} ${selector} span.reverse.bx116 {color: #87d7d7; background-color: inherit;}
${selector} span.fx117 {color: #87d7ff;} ${selector} span.bx117 {background-color: #87d7ff;}  ${selector} span.reverse.fx117 {background-color: #87d7ff; color: inherit;} ${selector} span.reverse.bx117 {color: #87d7ff; background-color: inherit;}
${selector} span.fx118 {color: #87ff00;} ${selector} span.bx118 {background-color: #87ff00;}  ${selector} span.reverse.fx118 {background-color: #87ff00; color: inherit;} ${selector} span.reverse.bx118 {color: #87ff00; background-color: inherit;}
${selector} span.fx119 {color: #87ff5f;} ${selector} span.bx119 {background-color: #87ff5f;}  ${selector} span.reverse.fx119 {background-color: #87ff5f; color: inherit;} ${selector} span.reverse.bx119 {color: #87ff5f; background-color: inherit;}
${selector} span.fx120 {color: #87ff87;} ${selector} span.bx120 {background-color: #87ff87;}  ${selector} span.reverse.fx120 {background-color: #87ff87; color: inherit;} ${selector} span.reverse.bx120 {color: #87ff87; background-color: inherit;}
${selector} span.fx121 {color: #87ffaf;} ${selector} span.bx121 {background-color: #87ffaf;}  ${selector} span.reverse.fx121 {background-color: #87ffaf; color: inherit;} ${selector} span.reverse.bx121 {color: #87ffaf; background-color: inherit;}
${selector} span.fx122 {color: #87ffd7;} ${selector} span.bx122 {background-color: #87ffd7;}  ${selector} span.reverse.fx122 {background-color: #87ffd7; color: inherit;} ${selector} span.reverse.bx122 {color: #87ffd7; background-color: inherit;}
${selector} span.fx123 {color: #87ffff;} ${selector} span.bx123 {background-color: #87ffff;}  ${selector} span.reverse.fx123 {background-color: #87ffff; color: inherit;} ${selector} span.reverse.bx123 {color: #87ffff; background-color: inherit;}
${selector} span.fx124 {color: #af0000;} ${selector} span.bx124 {background-color: #af0000;}  ${selector} span.reverse.fx124 {background-color: #af0000; color: inherit;} ${selector} span.reverse.bx124 {color: #af0000; background-color: inherit;}
${selector} span.fx125 {color: #af005f;} ${selector} span.bx125 {background-color: #af005f;}  ${selector} span.reverse.fx125 {background-color: #af005f; color: inherit;} ${selector} span.reverse.bx125 {color: #af005f; background-color: inherit;}
${selector} span.fx126 {color: #af0087;} ${selector} span.bx126 {background-color: #af0087;}  ${selector} span.reverse.fx126 {background-color: #af0087; color: inherit;} ${selector} span.reverse.bx126 {color: #af0087; background-color: inherit;}
${selector} span.fx127 {color: #af00af;} ${selector} span.bx127 {background-color: #af00af;}  ${selector} span.reverse.fx127 {background-color: #af00af; color: inherit;} ${selector} span.reverse.bx127 {color: #af00af; background-color: inherit;}
${selector} span.fx128 {color: #af00d7;} ${selector} span.bx128 {background-color: #af00d7;}  ${selector} span.reverse.fx128 {background-color: #af00d7; color: inherit;} ${selector} span.reverse.bx128 {color: #af00d7; background-color: inherit;}
${selector} span.fx129 {color: #af00ff;} ${selector} span.bx129 {background-color: #af00ff;}  ${selector} span.reverse.fx129 {background-color: #af00ff; color: inherit;} ${selector} span.reverse.bx129 {color: #af00ff; background-color: inherit;}
${selector} span.fx130 {color: #af5f00;} ${selector} span.bx130 {background-color: #af5f00;}  ${selector} span.reverse.fx130 {background-color: #af5f00; color: inherit;} ${selector} span.reverse.bx130 {color: #af5f00; background-color: inherit;}
${selector} span.fx131 {color: #af5f5f;} ${selector} span.bx131 {background-color: #af5f5f;}  ${selector} span.reverse.fx131 {background-color: #af5f5f; color: inherit;} ${selector} span.reverse.bx131 {color: #af5f5f; background-color: inherit;}
${selector} span.fx132 {color: #af5f87;} ${selector} span.bx132 {background-color: #af5f87;}  ${selector} span.reverse.fx132 {background-color: #af5f87; color: inherit;} ${selector} span.reverse.bx132 {color: #af5f87; background-color: inherit;}
${selector} span.fx133 {color: #af5faf;} ${selector} span.bx133 {background-color: #af5faf;}  ${selector} span.reverse.fx133 {background-color: #af5faf; color: inherit;} ${selector} span.reverse.bx133 {color: #af5faf; background-color: inherit;}
${selector} span.fx134 {color: #af5fd7;} ${selector} span.bx134 {background-color: #af5fd7;}  ${selector} span.reverse.fx134 {background-color: #af5fd7; color: inherit;} ${selector} span.reverse.bx134 {color: #af5fd7; background-color: inherit;}
${selector} span.fx135 {color: #af5fff;} ${selector} span.bx135 {background-color: #af5fff;}  ${selector} span.reverse.fx135 {background-color: #af5fff; color: inherit;} ${selector} span.reverse.bx135 {color: #af5fff; background-color: inherit;}
${selector} span.fx136 {color: #af8700;} ${selector} span.bx136 {background-color: #af8700;}  ${selector} span.reverse.fx136 {background-color: #af8700; color: inherit;} ${selector} span.reverse.bx136 {color: #af8700; background-color: inherit;}
${selector} span.fx137 {color: #af875f;} ${selector} span.bx137 {background-color: #af875f;}  ${selector} span.reverse.fx137 {background-color: #af875f; color: inherit;} ${selector} span.reverse.bx137 {color: #af875f; background-color: inherit;}
${selector} span.fx138 {color: #af8787;} ${selector} span.bx138 {background-color: #af8787;}  ${selector} span.reverse.fx138 {background-color: #af8787; color: inherit;} ${selector} span.reverse.bx138 {color: #af8787; background-color: inherit;}
${selector} span.fx139 {color: #af87af;} ${selector} span.bx139 {background-color: #af87af;}  ${selector} span.reverse.fx139 {background-color: #af87af; color: inherit;} ${selector} span.reverse.bx139 {color: #af87af; background-color: inherit;}
${selector} span.fx140 {color: #af87d7;} ${selector} span.bx140 {background-color: #af87d7;}  ${selector} span.reverse.fx140 {background-color: #af87d7; color: inherit;} ${selector} span.reverse.bx140 {color: #af87d7; background-color: inherit;}
${selector} span.fx141 {color: #af87ff;} ${selector} span.bx141 {background-color: #af87ff;}  ${selector} span.reverse.fx141 {background-color: #af87ff; color: inherit;} ${selector} span.reverse.bx141 {color: #af87ff; background-color: inherit;}
${selector} span.fx142 {color: #afaf00;} ${selector} span.bx142 {background-color: #afaf00;}  ${selector} span.reverse.fx142 {background-color: #afaf00; color: inherit;} ${selector} span.reverse.bx142 {color: #afaf00; background-color: inherit;}
${selector} span.fx143 {color: #afaf5f;} ${selector} span.bx143 {background-color: #afaf5f;}  ${selector} span.reverse.fx143 {background-color: #afaf5f; color: inherit;} ${selector} span.reverse.bx143 {color: #afaf5f; background-color: inherit;}
${selector} span.fx144 {color: #afaf87;} ${selector} span.bx144 {background-color: #afaf87;}  ${selector} span.reverse.fx144 {background-color: #afaf87; color: inherit;} ${selector} span.reverse.bx144 {color: #afaf87; background-color: inherit;}
${selector} span.fx145 {color: #afafaf;} ${selector} span.bx145 {background-color: #afafaf;}  ${selector} span.reverse.fx145 {background-color: #afafaf; color: inherit;} ${selector} span.reverse.bx145 {color: #afafaf; background-color: inherit;}
${selector} span.fx146 {color: #afafd7;} ${selector} span.bx146 {background-color: #afafd7;}  ${selector} span.reverse.fx146 {background-color: #afafd7; color: inherit;} ${selector} span.reverse.bx146 {color: #afafd7; background-color: inherit;}
${selector} span.fx147 {color: #afafff;} ${selector} span.bx147 {background-color: #afafff;}  ${selector} span.reverse.fx147 {background-color: #afafff; color: inherit;} ${selector} span.reverse.bx147 {color: #afafff; background-color: inherit;}
${selector} span.fx148 {color: #afd700;} ${selector} span.bx148 {background-color: #afd700;}  ${selector} span.reverse.fx148 {background-color: #afd700; color: inherit;} ${selector} span.reverse.bx148 {color: #afd700; background-color: inherit;}
${selector} span.fx149 {color: #afd75f;} ${selector} span.bx149 {background-color: #afd75f;}  ${selector} span.reverse.fx149 {background-color: #afd75f; color: inherit;} ${selector} span.reverse.bx149 {color: #afd75f; background-color: inherit;}
${selector} span.fx150 {color: #afd787;} ${selector} span.bx150 {background-color: #afd787;}  ${selector} span.reverse.fx150 {background-color: #afd787; color: inherit;} ${selector} span.reverse.bx150 {color: #afd787; background-color: inherit;}
${selector} span.fx151 {color: #afd7af;} ${selector} span.bx151 {background-color: #afd7af;}  ${selector} span.reverse.fx151 {background-color: #afd7af; color: inherit;} ${selector} span.reverse.bx151 {color: #afd7af; background-color: inherit;}
${selector} span.fx152 {color: #afd7d7;} ${selector} span.bx152 {background-color: #afd7d7;}  ${selector} span.reverse.fx152 {background-color: #afd7d7; color: inherit;} ${selector} span.reverse.bx152 {color: #afd7d7; background-color: inherit;}
${selector} span.fx153 {color: #afd7ff;} ${selector} span.bx153 {background-color: #afd7ff;}  ${selector} span.reverse.fx153 {background-color: #afd7ff; color: inherit;} ${selector} span.reverse.bx153 {color: #afd7ff; background-color: inherit;}
${selector} span.fx154 {color: #afff00;} ${selector} span.bx154 {background-color: #afff00;}  ${selector} span.reverse.fx154 {background-color: #afff00; color: inherit;} ${selector} span.reverse.bx154 {color: #afff00; background-color: inherit;}
${selector} span.fx155 {color: #afff5f;} ${selector} span.bx155 {background-color: #afff5f;}  ${selector} span.reverse.fx155 {background-color: #afff5f; color: inherit;} ${selector} span.reverse.bx155 {color: #afff5f; background-color: inherit;}
${selector} span.fx156 {color: #afff87;} ${selector} span.bx156 {background-color: #afff87;}  ${selector} span.reverse.fx156 {background-color: #afff87; color: inherit;} ${selector} span.reverse.bx156 {color: #afff87; background-color: inherit;}
${selector} span.fx157 {color: #afffaf;} ${selector} span.bx157 {background-color: #afffaf;}  ${selector} span.reverse.fx157 {background-color: #afffaf; color: inherit;} ${selector} span.reverse.bx157 {color: #afffaf; background-color: inherit;}
${selector} span.fx158 {color: #afffd7;} ${selector} span.bx158 {background-color: #afffd7;}  ${selector} span.reverse.fx158 {background-color: #afffd7; color: inherit;} ${selector} span.reverse.bx158 {color: #afffd7; background-color: inherit;}
${selector} span.fx159 {color: #afffff;} ${selector} span.bx159 {background-color: #afffff;}  ${selector} span.reverse.fx159 {background-color: #afffff; color: inherit;} ${selector} span.reverse.bx159 {color: #afffff; background-color: inherit;}
${selector} span.fx160 {color: #d70000;} ${selector} span.bx160 {background-color: #d70000;}  ${selector} span.reverse.fx160 {background-color: #d70000; color: inherit;} ${selector} span.reverse.bx160 {color: #d70000; background-color: inherit;}
${selector} span.fx161 {color: #d7005f;} ${selector} span.bx161 {background-color: #d7005f;}  ${selector} span.reverse.fx161 {background-color: #d7005f; color: inherit;} ${selector} span.reverse.bx161 {color: #d7005f; background-color: inherit;}
${selector} span.fx162 {color: #d70087;} ${selector} span.bx162 {background-color: #d70087;}  ${selector} span.reverse.fx162 {background-color: #d70087; color: inherit;} ${selector} span.reverse.bx162 {color: #d70087; background-color: inherit;}
${selector} span.fx163 {color: #d700af;} ${selector} span.bx163 {background-color: #d700af;}  ${selector} span.reverse.fx163 {background-color: #d700af; color: inherit;} ${selector} span.reverse.bx163 {color: #d700af; background-color: inherit;}
${selector} span.fx164 {color: #d700d7;} ${selector} span.bx164 {background-color: #d700d7;}  ${selector} span.reverse.fx164 {background-color: #d700d7; color: inherit;} ${selector} span.reverse.bx164 {color: #d700d7; background-color: inherit;}
${selector} span.fx165 {color: #d700ff;} ${selector} span.bx165 {background-color: #d700ff;}  ${selector} span.reverse.fx165 {background-color: #d700ff; color: inherit;} ${selector} span.reverse.bx165 {color: #d700ff; background-color: inherit;}
${selector} span.fx166 {color: #d75f00;} ${selector} span.bx166 {background-color: #d75f00;}  ${selector} span.reverse.fx166 {background-color: #d75f00; color: inherit;} ${selector} span.reverse.bx166 {color: #d75f00; background-color: inherit;}
${selector} span.fx167 {color: #d75f5f;} ${selector} span.bx167 {background-color: #d75f5f;}  ${selector} span.reverse.fx167 {background-color: #d75f5f; color: inherit;} ${selector} span.reverse.bx167 {color: #d75f5f; background-color: inherit;}
${selector} span.fx168 {color: #d75f87;} ${selector} span.bx168 {background-color: #d75f87;}  ${selector} span.reverse.fx168 {background-color: #d75f87; color: inherit;} ${selector} span.reverse.bx168 {color: #d75f87; background-color: inherit;}
${selector} span.fx169 {color: #d75faf;} ${selector} span.bx169 {background-color: #d75faf;}  ${selector} span.reverse.fx169 {background-color: #d75faf; color: inherit;} ${selector} span.reverse.bx169 {color: #d75faf; background-color: inherit;}
${selector} span.fx170 {color: #d75fd7;} ${selector} span.bx170 {background-color: #d75fd7;}  ${selector} span.reverse.fx170 {background-color: #d75fd7; color: inherit;} ${selector} span.reverse.bx170 {color: #d75fd7; background-color: inherit;}
${selector} span.fx171 {color: #d75fff;} ${selector} span.bx171 {background-color: #d75fff;}  ${selector} span.reverse.fx171 {background-color: #d75fff; color: inherit;} ${selector} span.reverse.bx171 {color: #d75fff; background-color: inherit;}
${selector} span.fx172 {color: #d78700;} ${selector} span.bx172 {background-color: #d78700;}  ${selector} span.reverse.fx172 {background-color: #d78700; color: inherit;} ${selector} span.reverse.bx172 {color: #d78700; background-color: inherit;}
${selector} span.fx173 {color: #d7875f;} ${selector} span.bx173 {background-color: #d7875f;}  ${selector} span.reverse.fx173 {background-color: #d7875f; color: inherit;} ${selector} span.reverse.bx173 {color: #d7875f; background-color: inherit;}
${selector} span.fx174 {color: #d78787;} ${selector} span.bx174 {background-color: #d78787;}  ${selector} span.reverse.fx174 {background-color: #d78787; color: inherit;} ${selector} span.reverse.bx174 {color: #d78787; background-color: inherit;}
${selector} span.fx175 {color: #d787af;} ${selector} span.bx175 {background-color: #d787af;}  ${selector} span.reverse.fx175 {background-color: #d787af; color: inherit;} ${selector} span.reverse.bx175 {color: #d787af; background-color: inherit;}
${selector} span.fx176 {color: #d787d7;} ${selector} span.bx176 {background-color: #d787d7;}  ${selector} span.reverse.fx176 {background-color: #d787d7; color: inherit;} ${selector} span.reverse.bx176 {color: #d787d7; background-color: inherit;}
${selector} span.fx177 {color: #d787ff;} ${selector} span.bx177 {background-color: #d787ff;}  ${selector} span.reverse.fx177 {background-color: #d787ff; color: inherit;} ${selector} span.reverse.bx177 {color: #d787ff; background-color: inherit;}
${selector} span.fx178 {color: #d7af00;} ${selector} span.bx178 {background-color: #d7af00;}  ${selector} span.reverse.fx178 {background-color: #d7af00; color: inherit;} ${selector} span.reverse.bx178 {color: #d7af00; background-color: inherit;}
${selector} span.fx179 {color: #d7af5f;} ${selector} span.bx179 {background-color: #d7af5f;}  ${selector} span.reverse.fx179 {background-color: #d7af5f; color: inherit;} ${selector} span.reverse.bx179 {color: #d7af5f; background-color: inherit;}
${selector} span.fx180 {color: #d7af87;} ${selector} span.bx180 {background-color: #d7af87;}  ${selector} span.reverse.fx180 {background-color: #d7af87; color: inherit;} ${selector} span.reverse.bx180 {color: #d7af87; background-color: inherit;}
${selector} span.fx181 {color: #d7afaf;} ${selector} span.bx181 {background-color: #d7afaf;}  ${selector} span.reverse.fx181 {background-color: #d7afaf; color: inherit;} ${selector} span.reverse.bx181 {color: #d7afaf; background-color: inherit;}
${selector} span.fx182 {color: #d7afd7;} ${selector} span.bx182 {background-color: #d7afd7;}  ${selector} span.reverse.fx182 {background-color: #d7afd7; color: inherit;} ${selector} span.reverse.bx182 {color: #d7afd7; background-color: inherit;}
${selector} span.fx183 {color: #d7afff;} ${selector} span.bx183 {background-color: #d7afff;}  ${selector} span.reverse.fx183 {background-color: #d7afff; color: inherit;} ${selector} span.reverse.bx183 {color: #d7afff; background-color: inherit;}
${selector} span.fx184 {color: #d7d700;} ${selector} span.bx184 {background-color: #d7d700;}  ${selector} span.reverse.fx184 {background-color: #d7d700; color: inherit;} ${selector} span.reverse.bx184 {color: #d7d700; background-color: inherit;}
${selector} span.fx185 {color: #d7d75f;} ${selector} span.bx185 {background-color: #d7d75f;}  ${selector} span.reverse.fx185 {background-color: #d7d75f; color: inherit;} ${selector} span.reverse.bx185 {color: #d7d75f; background-color: inherit;}
${selector} span.fx186 {color: #d7d787;} ${selector} span.bx186 {background-color: #d7d787;}  ${selector} span.reverse.fx186 {background-color: #d7d787; color: inherit;} ${selector} span.reverse.bx186 {color: #d7d787; background-color: inherit;}
${selector} span.fx187 {color: #d7d7af;} ${selector} span.bx187 {background-color: #d7d7af;}  ${selector} span.reverse.fx187 {background-color: #d7d7af; color: inherit;} ${selector} span.reverse.bx187 {color: #d7d7af; background-color: inherit;}
${selector} span.fx188 {color: #d7d7d7;} ${selector} span.bx188 {background-color: #d7d7d7;}  ${selector} span.reverse.fx188 {background-color: #d7d7d7; color: inherit;} ${selector} span.reverse.bx188 {color: #d7d7d7; background-color: inherit;}
${selector} span.fx189 {color: #d7d7ff;} ${selector} span.bx189 {background-color: #d7d7ff;}  ${selector} span.reverse.fx189 {background-color: #d7d7ff; color: inherit;} ${selector} span.reverse.bx189 {color: #d7d7ff; background-color: inherit;}
${selector} span.fx190 {color: #d7ff00;} ${selector} span.bx190 {background-color: #d7ff00;}  ${selector} span.reverse.fx190 {background-color: #d7ff00; color: inherit;} ${selector} span.reverse.bx190 {color: #d7ff00; background-color: inherit;}
${selector} span.fx191 {color: #d7ff5f;} ${selector} span.bx191 {background-color: #d7ff5f;}  ${selector} span.reverse.fx191 {background-color: #d7ff5f; color: inherit;} ${selector} span.reverse.bx191 {color: #d7ff5f; background-color: inherit;}
${selector} span.fx192 {color: #d7ff87;} ${selector} span.bx192 {background-color: #d7ff87;}  ${selector} span.reverse.fx192 {background-color: #d7ff87; color: inherit;} ${selector} span.reverse.bx192 {color: #d7ff87; background-color: inherit;}
${selector} span.fx193 {color: #d7ffaf;} ${selector} span.bx193 {background-color: #d7ffaf;}  ${selector} span.reverse.fx193 {background-color: #d7ffaf; color: inherit;} ${selector} span.reverse.bx193 {color: #d7ffaf; background-color: inherit;}
${selector} span.fx194 {color: #d7ffd7;} ${selector} span.bx194 {background-color: #d7ffd7;}  ${selector} span.reverse.fx194 {background-color: #d7ffd7; color: inherit;} ${selector} span.reverse.bx194 {color: #d7ffd7; background-color: inherit;}
${selector} span.fx195 {color: #d7ffff;} ${selector} span.bx195 {background-color: #d7ffff;}  ${selector} span.reverse.fx195 {background-color: #d7ffff; color: inherit;} ${selector} span.reverse.bx195 {color: #d7ffff; background-color: inherit;}
${selector} span.fx196 {color: #ff0000;} ${selector} span.bx196 {background-color: #ff0000;}  ${selector} span.reverse.fx196 {background-color: #ff0000; color: inherit;} ${selector} span.reverse.bx196 {color: #ff0000; background-color: inherit;}
${selector} span.fx197 {color: #ff005f;} ${selector} span.bx197 {background-color: #ff005f;}  ${selector} span.reverse.fx197 {background-color: #ff005f; color: inherit;} ${selector} span.reverse.bx197 {color: #ff005f; background-color: inherit;}
${selector} span.fx198 {color: #ff0087;} ${selector} span.bx198 {background-color: #ff0087;}  ${selector} span.reverse.fx198 {background-color: #ff0087; color: inherit;} ${selector} span.reverse.bx198 {color: #ff0087; background-color: inherit;}
${selector} span.fx199 {color: #ff00af;} ${selector} span.bx199 {background-color: #ff00af;}  ${selector} span.reverse.fx199 {background-color: #ff00af; color: inherit;} ${selector} span.reverse.bx199 {color: #ff00af; background-color: inherit;}
${selector} span.fx200 {color: #ff00d7;} ${selector} span.bx200 {background-color: #ff00d7;}  ${selector} span.reverse.fx200 {background-color: #ff00d7; color: inherit;} ${selector} span.reverse.bx200 {color: #ff00d7; background-color: inherit;}
${selector} span.fx201 {color: #ff00ff;} ${selector} span.bx201 {background-color: #ff00ff;}  ${selector} span.reverse.fx201 {background-color: #ff00ff; color: inherit;} ${selector} span.reverse.bx201 {color: #ff00ff; background-color: inherit;}
${selector} span.fx202 {color: #ff5f00;} ${selector} span.bx202 {background-color: #ff5f00;}  ${selector} span.reverse.fx202 {background-color: #ff5f00; color: inherit;} ${selector} span.reverse.bx202 {color: #ff5f00; background-color: inherit;}
${selector} span.fx203 {color: #ff5f5f;} ${selector} span.bx203 {background-color: #ff5f5f;}  ${selector} span.reverse.fx203 {background-color: #ff5f5f; color: inherit;} ${selector} span.reverse.bx203 {color: #ff5f5f; background-color: inherit;}
${selector} span.fx204 {color: #ff5f87;} ${selector} span.bx204 {background-color: #ff5f87;}  ${selector} span.reverse.fx204 {background-color: #ff5f87; color: inherit;} ${selector} span.reverse.bx204 {color: #ff5f87; background-color: inherit;}
${selector} span.fx205 {color: #ff5faf;} ${selector} span.bx205 {background-color: #ff5faf;}  ${selector} span.reverse.fx205 {background-color: #ff5faf; color: inherit;} ${selector} span.reverse.bx205 {color: #ff5faf; background-color: inherit;}
${selector} span.fx206 {color: #ff5fd7;} ${selector} span.bx206 {background-color: #ff5fd7;}  ${selector} span.reverse.fx206 {background-color: #ff5fd7; color: inherit;} ${selector} span.reverse.bx206 {color: #ff5fd7; background-color: inherit;}
${selector} span.fx207 {color: #ff5fff;} ${selector} span.bx207 {background-color: #ff5fff;}  ${selector} span.reverse.fx207 {background-color: #ff5fff; color: inherit;} ${selector} span.reverse.bx207 {color: #ff5fff; background-color: inherit;}
${selector} span.fx208 {color: #ff8700;} ${selector} span.bx208 {background-color: #ff8700;}  ${selector} span.reverse.fx208 {background-color: #ff8700; color: inherit;} ${selector} span.reverse.bx208 {color: #ff8700; background-color: inherit;}
${selector} span.fx209 {color: #ff875f;} ${selector} span.bx209 {background-color: #ff875f;}  ${selector} span.reverse.fx209 {background-color: #ff875f; color: inherit;} ${selector} span.reverse.bx209 {color: #ff875f; background-color: inherit;}
${selector} span.fx210 {color: #ff8787;} ${selector} span.bx210 {background-color: #ff8787;}  ${selector} span.reverse.fx210 {background-color: #ff8787; color: inherit;} ${selector} span.reverse.bx210 {color: #ff8787; background-color: inherit;}
${selector} span.fx211 {color: #ff87af;} ${selector} span.bx211 {background-color: #ff87af;}  ${selector} span.reverse.fx211 {background-color: #ff87af; color: inherit;} ${selector} span.reverse.bx211 {color: #ff87af; background-color: inherit;}
${selector} span.fx212 {color: #ff87d7;} ${selector} span.bx212 {background-color: #ff87d7;}  ${selector} span.reverse.fx212 {background-color: #ff87d7; color: inherit;} ${selector} span.reverse.bx212 {color: #ff87d7; background-color: inherit;}
${selector} span.fx213 {color: #ff87ff;} ${selector} span.bx213 {background-color: #ff87ff;}  ${selector} span.reverse.fx213 {background-color: #ff87ff; color: inherit;} ${selector} span.reverse.bx213 {color: #ff87ff; background-color: inherit;}
${selector} span.fx214 {color: #ffaf00;} ${selector} span.bx214 {background-color: #ffaf00;}  ${selector} span.reverse.fx214 {background-color: #ffaf00; color: inherit;} ${selector} span.reverse.bx214 {color: #ffaf00; background-color: inherit;}
${selector} span.fx215 {color: #ffaf5f;} ${selector} span.bx215 {background-color: #ffaf5f;}  ${selector} span.reverse.fx215 {background-color: #ffaf5f; color: inherit;} ${selector} span.reverse.bx215 {color: #ffaf5f; background-color: inherit;}
${selector} span.fx216 {color: #ffaf87;} ${selector} span.bx216 {background-color: #ffaf87;}  ${selector} span.reverse.fx216 {background-color: #ffaf87; color: inherit;} ${selector} span.reverse.bx216 {color: #ffaf87; background-color: inherit;}
${selector} span.fx217 {color: #ffafaf;} ${selector} span.bx217 {background-color: #ffafaf;}  ${selector} span.reverse.fx217 {background-color: #ffafaf; color: inherit;} ${selector} span.reverse.bx217 {color: #ffafaf; background-color: inherit;}
${selector} span.fx218 {color: #ffafd7;} ${selector} span.bx218 {background-color: #ffafd7;}  ${selector} span.reverse.fx218 {background-color: #ffafd7; color: inherit;} ${selector} span.reverse.bx218 {color: #ffafd7; background-color: inherit;}
${selector} span.fx219 {color: #ffafff;} ${selector} span.bx219 {background-color: #ffafff;}  ${selector} span.reverse.fx219 {background-color: #ffafff; color: inherit;} ${selector} span.reverse.bx219 {color: #ffafff; background-color: inherit;}
${selector} span.fx220 {color: #ffd700;} ${selector} span.bx220 {background-color: #ffd700;}  ${selector} span.reverse.fx220 {background-color: #ffd700; color: inherit;} ${selector} span.reverse.bx220 {color: #ffd700; background-color: inherit;}
${selector} span.fx221 {color: #ffd75f;} ${selector} span.bx221 {background-color: #ffd75f;}  ${selector} span.reverse.fx221 {background-color: #ffd75f; color: inherit;} ${selector} span.reverse.bx221 {color: #ffd75f; background-color: inherit;}
${selector} span.fx222 {color: #ffd787;} ${selector} span.bx222 {background-color: #ffd787;}  ${selector} span.reverse.fx222 {background-color: #ffd787; color: inherit;} ${selector} span.reverse.bx222 {color: #ffd787; background-color: inherit;}
${selector} span.fx223 {color: #ffd7af;} ${selector} span.bx223 {background-color: #ffd7af;}  ${selector} span.reverse.fx223 {background-color: #ffd7af; color: inherit;} ${selector} span.reverse.bx223 {color: #ffd7af; background-color: inherit;}
${selector} span.fx224 {color: #ffd7d7;} ${selector} span.bx224 {background-color: #ffd7d7;}  ${selector} span.reverse.fx224 {background-color: #ffd7d7; color: inherit;} ${selector} span.reverse.bx224 {color: #ffd7d7; background-color: inherit;}
${selector} span.fx225 {color: #ffd7ff;} ${selector} span.bx225 {background-color: #ffd7ff;}  ${selector} span.reverse.fx225 {background-color: #ffd7ff; color: inherit;} ${selector} span.reverse.bx225 {color: #ffd7ff; background-color: inherit;}
${selector} span.fx226 {color: #ffff00;} ${selector} span.bx226 {background-color: #ffff00;}  ${selector} span.reverse.fx226 {background-color: #ffff00; color: inherit;} ${selector} span.reverse.bx226 {color: #ffff00; background-color: inherit;}
${selector} span.fx227 {color: #ffff5f;} ${selector} span.bx227 {background-color: #ffff5f;}  ${selector} span.reverse.fx227 {background-color: #ffff5f; color: inherit;} ${selector} span.reverse.bx227 {color: #ffff5f; background-color: inherit;}
${selector} span.fx228 {color: #ffff87;} ${selector} span.bx228 {background-color: #ffff87;}  ${selector} span.reverse.fx228 {background-color: #ffff87; color: inherit;} ${selector} span.reverse.bx228 {color: #ffff87; background-color: inherit;}
${selector} span.fx229 {color: #ffffaf;} ${selector} span.bx229 {background-color: #ffffaf;}  ${selector} span.reverse.fx229 {background-color: #ffffaf; color: inherit;} ${selector} span.reverse.bx229 {color: #ffffaf; background-color: inherit;}
${selector} span.fx230 {color: #ffffd7;} ${selector} span.bx230 {background-color: #ffffd7;}  ${selector} span.reverse.fx230 {background-color: #ffffd7; color: inherit;} ${selector} span.reverse.bx230 {color: #ffffd7; background-color: inherit;}
${selector} span.fx231 {color: #ffffff;} ${selector} span.bx231 {background-color: #ffffff;}  ${selector} span.reverse.fx231 {background-color: #ffffff; color: inherit;} ${selector} span.reverse.bx231 {color: #ffffff; background-color: inherit;}
${selector} span.fx232 {color: #080808;} ${selector} span.bx232 {background-color: #080808;}  ${selector} span.reverse.fx232 {background-color: #080808; color: inherit;} ${selector} span.reverse.bx232 {color: #080808; background-color: inherit;}
${selector} span.fx233 {color: #121212;} ${selector} span.bx233 {background-color: #121212;}  ${selector} span.reverse.fx233 {background-color: #121212; color: inherit;} ${selector} span.reverse.bx233 {color: #121212; background-color: inherit;}
${selector} span.fx234 {color: #1c1c1c;} ${selector} span.bx234 {background-color: #1c1c1c;}  ${selector} span.reverse.fx234 {background-color: #1c1c1c; color: inherit;} ${selector} span.reverse.bx234 {color: #1c1c1c; background-color: inherit;}
${selector} span.fx235 {color: #262626;} ${selector} span.bx235 {background-color: #262626;}  ${selector} span.reverse.fx235 {background-color: #262626; color: inherit;} ${selector} span.reverse.bx235 {color: #262626; background-color: inherit;}
${selector} span.fx236 {color: #303030;} ${selector} span.bx236 {background-color: #303030;}  ${selector} span.reverse.fx236 {background-color: #303030; color: inherit;} ${selector} span.reverse.bx236 {color: #303030; background-color: inherit;}
${selector} span.fx237 {color: #3a3a3a;} ${selector} span.bx237 {background-color: #3a3a3a;}  ${selector} span.reverse.fx237 {background-color: #3a3a3a; color: inherit;} ${selector} span.reverse.bx237 {color: #3a3a3a; background-color: inherit;}
${selector} span.fx238 {color: #444444;} ${selector} span.bx238 {background-color: #444444;}  ${selector} span.reverse.fx238 {background-color: #444444; color: inherit;} ${selector} span.reverse.bx238 {color: #444444; background-color: inherit;}
${selector} span.fx239 {color: #4e4e4e;} ${selector} span.bx239 {background-color: #4e4e4e;}  ${selector} span.reverse.fx239 {background-color: #4e4e4e; color: inherit;} ${selector} span.reverse.bx239 {color: #4e4e4e; background-color: inherit;}
${selector} span.fx240 {color: #585858;} ${selector} span.bx240 {background-color: #585858;}  ${selector} span.reverse.fx240 {background-color: #585858; color: inherit;} ${selector} span.reverse.bx240 {color: #585858; background-color: inherit;}
${selector} span.fx241 {color: #626262;} ${selector} span.bx241 {background-color: #626262;}  ${selector} span.reverse.fx241 {background-color: #626262; color: inherit;} ${selector} span.reverse.bx241 {color: #626262; background-color: inherit;}
${selector} span.fx242 {color: #6c6c6c;} ${selector} span.bx242 {background-color: #6c6c6c;}  ${selector} span.reverse.fx242 {background-color: #6c6c6c; color: inherit;} ${selector} span.reverse.bx242 {color: #6c6c6c; background-color: inherit;}
${selector} span.fx243 {color: #767676;} ${selector} span.bx243 {background-color: #767676;}  ${selector} span.reverse.fx243 {background-color: #767676; color: inherit;} ${selector} span.reverse.bx243 {color: #767676; background-color: inherit;}
${selector} span.fx244 {color: #808080;} ${selector} span.bx244 {background-color: #808080;}  ${selector} span.reverse.fx244 {background-color: #808080; color: inherit;} ${selector} span.reverse.bx244 {color: #808080; background-color: inherit;}
${selector} span.fx245 {color: #8a8a8a;} ${selector} span.bx245 {background-color: #8a8a8a;}  ${selector} span.reverse.fx245 {background-color: #8a8a8a; color: inherit;} ${selector} span.reverse.bx245 {color: #8a8a8a; background-color: inherit;}
${selector} span.fx246 {color: #949494;} ${selector} span.bx246 {background-color: #949494;}  ${selector} span.reverse.fx246 {background-color: #949494; color: inherit;} ${selector} span.reverse.bx246 {color: #949494; background-color: inherit;}
${selector} span.fx247 {color: #9e9e9e;} ${selector} span.bx247 {background-color: #9e9e9e;}  ${selector} span.reverse.fx247 {background-color: #9e9e9e; color: inherit;} ${selector} span.reverse.bx247 {color: #9e9e9e; background-color: inherit;}
${selector} span.fx248 {color: #a8a8a8;} ${selector} span.bx248 {background-color: #a8a8a8;}  ${selector} span.reverse.fx248 {background-color: #a8a8a8; color: inherit;} ${selector} span.reverse.bx248 {color: #a8a8a8; background-color: inherit;}
${selector} span.fx249 {color: #b2b2b2;} ${selector} span.bx249 {background-color: #b2b2b2;}  ${selector} span.reverse.fx249 {background-color: #b2b2b2; color: inherit;} ${selector} span.reverse.bx249 {color: #b2b2b2; background-color: inherit;}
${selector} span.fx250 {color: #bcbcbc;} ${selector} span.bx250 {background-color: #bcbcbc;}  ${selector} span.reverse.fx250 {background-color: #bcbcbc; color: inherit;} ${selector} span.reverse.bx250 {color: #bcbcbc; background-color: inherit;}
${selector} span.fx251 {color: #c6c6c6;} ${selector} span.bx251 {background-color: #c6c6c6;}  ${selector} span.reverse.fx251 {background-color: #c6c6c6; color: inherit;} ${selector} span.reverse.bx251 {color: #c6c6c6; background-color: inherit;}
${selector} span.fx252 {color: #d0d0d0;} ${selector} span.bx252 {background-color: #d0d0d0;}  ${selector} span.reverse.fx252 {background-color: #d0d0d0; color: inherit;} ${selector} span.reverse.bx252 {color: #d0d0d0; background-color: inherit;}
${selector} span.fx253 {color: #dadada;} ${selector} span.bx253 {background-color: #dadada;}  ${selector} span.reverse.fx253 {background-color: #dadada; color: inherit;} ${selector} span.reverse.bx253 {color: #dadada; background-color: inherit;}
${selector} span.fx254 {color: #e4e4e4;} ${selector} span.bx254 {background-color: #e4e4e4;}  ${selector} span.reverse.fx254 {background-color: #e4e4e4; color: inherit;} ${selector} span.reverse.bx254 {color: #e4e4e4; background-color: inherit;}
${selector} span.fx255 {color: #eeeeee;} ${selector} span.bx255 {background-color: #eeeeee;}  ${selector} span.reverse.fx255 {background-color: #eeeeee; color: inherit;} ${selector} span.reverse.bx255 {color: #eeeeee; background-color: inherit;}

${selector} span.cursor { color: #000; background-color: #ccc; }"""

#
#

# Meta
__version_info__ = (1.0)

Authentication
==============
This module contains Gate One's authentication classes.  They map to Gate One's
--auth configuration option like so:

=============== ===================
--auth=none     NullAuthHandler
--auth=kerberos KerberosAuthHandler
--auth=google   GoogleAuthHandler
--auth=pam      PAMAuthHandler

.. note:: API authentication is handled inside of :ref:`gateone.py`

None or Anonymous
-----------------
By default Gate One will not authenticate users.  This means that user sessions
will be tied to their browser cookie and users will not be able to resume their
sessions from another computer/browser.  Most useful for situations where
session persistence and logging aren't important.

*All* users will show up as ANONYMOUS using this authentication type.

Kerberos
--------
Kerberos authentication utilizes GSSAPI for Single Sign-on (SSO) but will fall
back to HTTP Basic authentication if GSSAPI auth fails.  This authentication
type can be integrated into any Kerberos infrastructure including Windows
Active Directory.

It is great for both transparent authentication and being able to tie sessions
and logs to specific users within your organization (compliance).

.. note:: The sso.py module itself has extensive documentation on this authentication type.

Google Authentication
---------------------
If you want persistent user sessions but don't care to run your own
authentication infrastructure this authentication type is for you.  Assuming,
of course, that your Gate One server and clients will have access to the
Internet.

.. note:: This authentication type is perfect if you're using Chromebooks (Chrome OS devices).

"""

import os

from utils import mkdir_p, generate_session_id
from utils import get_translation

import tornado.auth
import tornado.escape

# Localization support

class BaseAuthHandler(tornado.web.RequestHandler):
    """The base class for all Gate One authentication handlers."""
    def get_current_user(self):
        """Tornado standard method--implemented our way."""
        user_json = self.get_secure_cookie("gateone_user")
        if not user_json: return None
        return tornado.escape.json_decode(user_json)

    def user_login(self, user):
        """
        Called immediately after a user authenticates successfully.  Saves
        session information in the user's directory.  Expects *user* to be a
        string containing the username or userPrincipalName. e.g. 'user@REALM'
        or just 'someuser'.
        """
        logging.debug("user_login(%s)" % user)
        # Make a directory to store this user's settings/files/logs/etc
        user_dir = os.path.join(self.settings['user_dir'], user)
        if not os.path.exists(user_dir):
            logging.info(_("Creating user directory: %s" % user_dir))
            mkdir_p(user_dir)
            os.chmod(user_dir, 0o700)
        session_file = os.path.join(user_dir, 'session')
        session_file_exists = os.path.exists(session_file)
        if session_file_exists:
            session_data = open(session_file).read()
            try:
                session_info = tornado.escape.json_decode(session_data)
            except ValueError: # Something wrong with the file
                session_file_exists = False # Overwrite it below
        if not session_file_exists:
            with open(session_file, 'w') as f:
                # Save it so we can keep track across multiple clients
                session_info = {
                    'upn': user, # FYI: UPN == userPrincipalName
                    'session': generate_session_id()
                }
                session_info_json = tornado.escape.json_encode(session_info)
                f.write(session_info_json)
        self.set_secure_cookie(
            "gateone_user", tornado.escape.json_encode(session_info))

    def user_logout(self, user, redirect=None):
        """
        Called immediately after a user logs out, cleans up the user's session
        information and optionally, redirects them to *redirect* (URL).
        """
        logging.debug("user_logout(%s)" % user)
        if not redirect:
            # Try getting it from the query string
            redirect = self.get_argument("redirect", None)
        if redirect:
            self.write(redirect)
            self.finish()
        else:
            self.write(self.settings['url_prefix'])

class NullAuthHandler(BaseAuthHandler):
    """
    A handler for when no authentication method is chosen (i.e. --auth=none).
    With this handler all users will show up as "ANONYMOUS".
    """
    @tornado.web.asynchronous
    def get(self):
        """
        Sets the 'user' cookie with a new random session ID (*go_session*) and
        sets *go_upn* to 'ANONYMOUS'.
        """
        user = 'ANONYMOUS'
        check = self.get_argument("check", None)
        if check:
            # This lets any origin check if the user has been authenticated
            # (necessary to prevent "not allowed ..." XHR errors)
            self.set_header('Access-Control-Allow-Origin', '*')
            self.user_login(user)
            self.write('authenticated')
            return
        logout = self.get_argument("logout", None)
        if logout:
            self.clear_cookie('gateone_user')
            self.user_logout(user)
            return
        # This takes care of the user's settings dir and their session info
        self.user_login(user)
        next_url = self.get_argument("next", None)
        if next_url:
            self.redirect(next_url)
        else:
            self.redirect(self.settings['url_prefix'])

        """
        This is an override of BaseAuthHandler since anonymous auth is special.
        Generates a unique session ID for this user and saves it in a browser
        cookie.  This is to ensure that anonymous users can't access each
        other's sessions.
        """
        logging.debug("NullAuthHandler.user_login(%s)" % user)
        session_info = {
            'upn': user,
            'session': generate_session_id()
        }

class GoogleAuthHandler(BaseAuthHandler, tornado.auth.GoogleMixin):
    """
    Google authentication handler using Tornado's built-in GoogleMixin (fairly
    boilerplate).
    """
        """
        Sets the 'user' cookie with an appropriate *upn* and *session*.
        """
        if check:
            self.set_header ('Access-Control-Allow-Origin', '*')
            user = self.get_current_user()
            if user:
                logging.debug('GoogleAuthHandler: user is authenticated')
                self.write('authenticated')
            else:
                logging.debug('GoogleAuthHandler: user is NOT authenticated')
                self.write('unauthenticated')
            return
        logout_url = "https://accounts.google.com/Logout"
        if logout:
            user = self.get_current_user()['upn']
            self.user_logout(user, logout_url)
            return
        if self.get_argument("openid.mode", None):
            self.get_authenticated_user(self._on_auth)
            return
        self.authenticate_redirect(
            ax_attrs=["name","email","language","username"])

    def _on_auth(self, user):
        """
        Just a continuation of the get() method (the final step where it
        actually sets the cookie).
        """
        if not user:
            raise tornado.web.HTTPError(500, _("Google auth failed"))
        # NOTE: Google auth 'user' will be a dict like so:
        # user: {
        #     'locale': u'en-us',
        #     'first_name': u'Dan',
        #     'last_name': u'McDougall',
        #     'name': u'Dan McDougall',
        #     'email': u'daniel.mcdougall@liftoffsoftware.com'}
        # Named these 'go_<whatever>' since that is less likely to conflict with
        # anything in the future (should some auth mechanism start returning
        # session IDs of some sort).
        self.user_login(user['email'])
        else:

# Add our KerberosAuthHandler if sso is available
KerberosAuthHandler = None
try:
    from sso import KerberosAuthMixin
    class KerberosAuthHandler(BaseAuthHandler, KerberosAuthMixin):
        """
        Handles authenticating users via Kerberos/GSSAPI/SSO.
        """
        @tornado.web.asynchronous
        def get(self):
            """
            Checks the user's request header for the proper Authorization data.
            If it checks out the user will be logged in via _on_auth().  If not,
            the browser will be redirected to login.
            """
            check = self.get_argument("check", None)
            if check:
                user = self.get_current_user()
                if user:
                    logging.debug('KerberosAuthHandler: user is authenticated')
                    self.write('authenticated')
                else:
                    logging.debug('KerberosAuthHandler: user is NOT authenticated')
                    self.write('unauthenticated')
                self.finish()
                return
            logout = self.get_argument("logout", None)
            if logout:
                user = self.get_current_user()['upn']
                self.clear_cookie('gateone_user')
                self.user_logout(user)
                return
            auth_header = self.request.headers.get('Authorization')
            if auth_header:
                self.get_authenticated_user(self._on_auth)
                return
            self.authenticate_redirect()

        def _on_auth(self, user):
            if not user:
                raise tornado.web.HTTPError(500, _("Kerberos auth failed"))
            # This takes care of the user's settings dir and their session info
            # TODO: Add some LDAP or local DB lookups here to add more detail to user objects
            logging.debug(_("KerberosAuthHandler user: %s" % user))
            next_url = self.get_argument("next", None)
            if next_url:
                self.redirect(next_url)
            else:
                self.redirect(self.settings['url_prefix'])
    pass # No SSO available.

# Add our PAMAuthHandler if it's available
PAMAuthHandler = None
try:
    from authpam import PAMAuthMixin
    class PAMAuthHandler(BaseAuthHandler, PAMAuthMixin):
        """
        Handles authenticating users via PAM.
        """
            """
            """
            if check:
                if user:
                    logging.debug('PAMAuthHandler: user is authenticated')
                else:
                    logging.debug('PAMAuthHandler: user is NOT authenticated')
                    self.get_authenticated_user(self._on_auth)
                return
            if logout:
                return
                return

                raise tornado.web.HTTPError(500, _("PAM auth failed"))
            logging.debug(_("PAMAuthHandler user: %s" % user))
            else:
    pass # No PAM auth available.

#
#

# TODO:
# * Write init scripts to stop/start/restart Gate One safely.  Also make sure that .deb and .rpm packages safely restart Gate One without impacting running sessions.  The setup.py should also attempt to minify the .css and .js files.
# * Write a permissions check function so we don't have to repeat ourselves all over the place inside of main()
# * Make it so that the session_dir gets cleaned up.  Will require some logic in regards dtach detection.

# Meta
__version__ = '1.1.0'
__version_info__ = (1, 1, 0)

# NOTE: Docstring includes reStructuredText markup for use with Sphinx.
__doc__ = '''\
.. _gateone.py:

Gate One
========
Gate One is a web-based terminal emulator written in Python using the Tornado
web framework.  This module runs the primary daemon process and acts as a
central controller for all running terminals and terminal programs.  It supports
numerous configuration options and can also be called with the --kill switch
to kill all running terminal programs (if using dtach--otherwise they die on
their own when gateone.py is stopped).

Dependencies
------------
Gate One requires Python 2.6+ but runs best with Python 2.7+.  It also depends
on the following 3rd party Python modules:

 * `Tornado <http://www.tornadoweb.org/>`_ 2.2+ - A non-blocking web server framework that powers FriendFeed.

The following modules are optional and can provide Gate One with additional
functionality:

 * `pyOpenSSL <https://launchpad.net/pyopenssl>`_ 0.10+ - An OpenSSL module/wrapper for Python.  Only used to generate self-signed SSL keys and certificates.  If pyOpenSSL isn't available Gate One will fall back to using the 'openssl' command to generate self-signed certificates.
 * `kerberos <http://pypi.python.org/pypi/kerberos>`_ 1.0+ - A high-level Kerberos interface for Python.  Only necessary if you plan to use the Kerberos authentication module.
 * `python-pam <http://packages.debian.org/lenny/python-pam>`_ 0.4.2+ - A Python module for interacting with PAM (the Pluggable Authentication Module present on nearly every Unix).  Only necessary if you plan to use PAM authentication.

With the exception of python-pam, both the required and optional modules can usually be installed via one of these commands:

        \x1b[1;34muser\x1b[0m@modern-host\x1b[1;34m:~ $\x1b[0m sudo pip install tornado pyopenssl kerberos

...or:

        \x1b[1;34muser\x1b[0m@legacy-host\x1b[1;34m:~ $\x1b[0m sudo easy_install tornado pyopenssl kerberos

.. note:: The use of pip is recommended.  See http://www.pip-installer.org/en/latest/installing.html if you don't have it.

The python-pam module is available in most Linux distribution repositories.  Simply executing one of the following should take care of it:

        \x1b[1;34muser\x1b[0m@debian-or-ubuntu-host\x1b[1;34m:~ $\x1b[0m sudo apt-get install python-pam

        \x1b[1;34muser\x1b[0m@redhat-host\x1b[1;34m:~ $\x1b[0m sudo yum install python-pam

        \x1b[1;34muser\x1b[0m@gentoo-host\x1b[1;34m:~ $\x1b[0m sudo emerge python-pam

        \x1b[1;34muser\x1b[0m@suse-host\x1b[1;34m:~ $\x1b[0m sudo yast -i python-pam

Settings
--------
All of Gate One's configurable options can be controlled either via command line
switches or by settings in the server.conf file (they match up 1-to-1).  If no
server.conf exists one will be created using defaults (i.e. when Gate One is run
for the first time).  Settings in the server.conf file use the following format::

    <setting> = <value>

Here's an example::

    address = "127.0.0.1;::1;10.1.1.4" # Strings are surrounded by quotes
    port = 443 # Numbers don't need quotes

There are a few important differences between the configuration file and
command line switches in regards to boolean values (True/False).  A switch such
as --debug evaluates to "debug = True" and this is exactly how it would be
configured in server.conf::

    debug = True # Booleans don't need quotes either

.. note:: The following values in server.conf are case sensitive: True, False and None (and should not be placed in quotes).

Running gateone.py with the --help switch will print the usage information as
well as descriptions of what each configurable option does:

    \x1b[1;31mroot\x1b[0m@host\x1b[1;34m:~ $\x1b[0m ./gateone.py --help
    Usage: ./gateone.py [OPTIONS]

    Options:
      --help                           show this help information
      --log_file_max_size              max size of log files before rollover
      --log_file_num_backups           number of log files to keep
      --log_file_prefix=PATH           Path prefix for log files. Note that if you are running multiple tornado processes, log_file_prefix must be different for each of them (e.g. include the port number)
      --log_to_stderr                  Send log output to stderr (colorized if possible). By default use stderr if --log_file_prefix is not set and no other logging is configured.
      --logging=debug|info|warning|error|none Set the Python log level. If 'none', tornado won't touch the logging configuration.
      --address                        Run on the given address.  Default is all addresses (IPv6 included).  Multiple address can be specified using a semicolon as a separator (e.g. '127.0.0.1;::1;10.1.1.100').
      --auth                           Authentication method to use.  Valid options are: none, api, google, kerberos, pam
      --certificate                    Path to the SSL certificate.  Will be auto-generated if none is provided.
      --command                        Run the given command when a user connects (e.g. '/bin/login').
      --config                         Path to the config file.  Default: /opt/gateone/server.conf
      --cookie_secret                  Use the given 45-character string for cookie encryption.
      --debug                          Enable debugging features such as auto-restarting when files are modified.
      --disable_ssl                    If enabled, Gate One will run without SSL (generally not a good idea).
      --dtach                          Wrap terminals with dtach. Allows sessions to be resumed even if Gate One is stopped and started (which is a sweet feature).
      --embedded                       Doesn't do anything (yet).
      --enable_unix_socket             Enable Unix socket support use_unix_sockets (if --enable_unix_socket=True).
      --https_redirect                 If enabled, a separate listener will be started on port 80 that redirects users to the configured port using HTTPS.
      --js_init                        A JavaScript object (string) that will be used when running GateOne.init() inside index.html.  Example: --js_init="{scheme: 'white'}" would result in GateOne.init({scheme: 'white'})
      --keyfile                        Path to the SSL keyfile.  Will be auto-generated if none is provided.
      --kill                           Kill any running Gate One terminal processes including dtach'd processes.
      --locale                         The locale (e.g. pt_PT) Gate One should use for translations.  If not provided, will default to $LANG (which is 'en_US' in your current shell), or en_US if not set.
      --new_api_key                    Generate a new API key that an external application can use to embed Gate One.
      --origins                        A semicolon-separated list of origins you wish to allow access to your Gate One server over the WebSocket.  This value must contain the hostnames and FQDNs (e.g. https://foo;https://foo.bar;) users will use to connect to your Gate One server as well as the hostnames/FQDNs of any sites that will be embedding Gate One. Here's the default on your system: 'https://localhost;https://yourhostname'. Alternatively, '*' may be  specified to allow access from anywhere.
      --pam_realm                      Basic auth REALM to display when authenticating clients.  Default: hostname.  Only relevant if PAM authentication is enabled.
      --pam_service                    PAM service to use.  Defaults to 'login'. Only relevant if PAM authentication is enabled.
      --pid_file                       Path of the pid file.   Default: /var/run/gateone.pid
      --port                           Run on the given port.
      --session_dir                    Path to the location where session information will be stored.
      --session_logging                If enabled, logs of user sessions will be saved in <user_dir>/<user>/logs.  Default: Enabled
      --session_timeout                Amount of time that a session should be kept alive after the client has logged out.  Accepts <num>X where X could be one of s, m, h, or d for seconds, minutes, hours, and days.  Default is '5d' (5 days).
      --sso_realm                      Kerberos REALM (aka DOMAIN) to use when authenticating clients. Only relevant if Kerberos authentication is enabled.
      --sso_service                    Kerberos service (aka application) to use. Defaults to HTTP. Only relevant if Kerberos authentication is enabled.
      --syslog_facility                Syslog facility to use when logging to syslog (if syslog_session_logging is enabled).  Must be one of: auth, cron, daemon, kern, local0, local1, local2, local3, local4, local5, local6, local7, lpr, mail, news, syslog, user, uucp.  Default: daemon
      --syslog_host                    Remote host to send syslog messages to if syslog_logging is enabled.  Default: None (log to the local syslog daemon directly).  NOTE:  This setting is required on platforms that don't include Python's syslog module.
      --syslog_session_logging         If enabled, logs of user sessions will be written to syslog.
      --unix_socket_path               Run on the given socket file.  Default: /var/run/gateone.sock
      --url_prefix                     An optional prefix to place before all Gate One URLs. e.g. '/gateone/'.  Use this if Gate One will be running behind a reverse proxy where you want it to be located at some sub-URL path.
      --user_dir                       Path to the location where user files will be stored.

.. note:: Some of these options (e.g. log_file_prefix) are inherent to the Tornado framework.  You won't find them anywhere in gateone.py.

File Paths
Gate One stores its files, temporary session information, and persistent user
data in the following locations (Note: Many of these are configurable):

================= ==================================================================================
File/Directory      Description
authpam.py        Contains the PAM authentication Mixin used by auth.py.
auth.py           Authentication classes.
certificate.pem   The default certificate Gate One will use in SSL communications.
docs/             Gate One's documentation.
gateone.py        Gate One's primary executable/script. Also, the file containing this documentation
i18n/             Gate One's internationalization (i18n) support and locale/translation files.
keyfile.pem       The default private key used with SSL communications.
logviewer.py      A utility to view Gate One session logs.
plugins/          Plugins go here in the form of ./plugins/<plugin name>/<plugin files|directories>
remote_syslog.py  A module that supports sending syslog messages over UDP to a remote syslog host.
server.conf       Gate One's configuration file.
sso.py            A Kerberos Single Sign-on module for Tornado (used by auth.py)
static/           Non-dynamic files that get served to clients (e.g. gateone.js, gateone.css, etc).
templates/        Tornado template files such as index.html.
terminal.py       A Pure Python terminal emulator module.
termio.py         Terminal input/output control module.
tests/            Various scripts and tools to test Gate One's functionality.
utils.py          Various supporting functions.
users/            Persistent user data in the form of ./users/<username>/<user-specific files>
users/<user>/logs This is where session logs get stored if session_logging is set.
/tmp/gateone      Temporary session data in the form of /tmp/gateone/<session ID>/<files>

Running
-------
Executing Gate One is as simple as:

    \x1b[1;31mroot\x1b[0m@host\x1b[1;34m:~ $\x1b[0m ./gateone.py

.. note:: By default Gate One will run on port 443 which requires root on most systems.  Use `--port=(something higher than 1024)` for non-root users.

Plugins
-------
Gate One includes support for any combination of the following types of plugins:

 * Python
 * JavaScript
 * CSS

Python plugins can integrate with Gate One in three ways:

 * Adding or overriding tornado.web.RequestHandlers (with a given regex).
 * Adding or overriding methods (aka "commands") in TerminalWebSocket.
 * Adding special plugin-specific escape sequence handlers (see the plugin development documentation for details on what/how these are/work).

JavaScript plugins will be added to the <body> tag of Gate One's base index.html
template by way of a single file (`{{gateone_js}}` below) that is the
concatenation of all plugins' JS templates:

.. code-block:: html

    <script type="text/javascript" src="{{gateone_js}}"></script>

CSS plugins are similar to JavaScript but instead of being appended to the
<body> they are added to the <head> by way of a WebSocket download and some
fancy JavaScript inside of gateone.js:

.. code-block:: javascript

    CSSPluginAction: function(url) {
        // Loads the CSS for a given plugin by adding a <link> tag to the <head>
        var queries = url.split('?')[1].split('&'), // So we can parse out the plugin name and the template
            plugin = queries[0].split('=')[1],
            file = queries[1].split('=')[1].split('.')[0];
        // The /cssrender method needs the prefix and the container
        url = url + '&container=' + GateOne.prefs.goDiv.substring(1);
        url = url + '&prefix=' + GateOne.prefs.prefix;
        url = GateOne.prefs.url + url.substring(1);
        GateOne.Utils.loadCSS(url, plugin+'_'+file);
    }

There are also hooks throughout Gate One's code for plugins to add or override
Gate One's functionality.  Documentation on how to write plugins can be found in
the Plugin Development docs.  From the perspective of gateone.py, it performs
the following tasks in relation to plugins:

 * Imports Python plugins and connects their hooks.
 * Creates symbolic links inside ./static/ that point to each plugin's respective /static/ directory and serves them to clients.
 * Serves the index.html that includes plugins' respective .js and .css files.

'''

import os
import sys
import pty
import pwd, grp
import atexit
import ssl
from functools import partial, wraps

# Tornado modules (yeah, we use all this stuff)
try:
    import tornado.httpserver
    import tornado.ioloop
    import tornado.options
    import tornado.web
    import tornado.auth
    import tornado.template
    import tornado.netutil
    from tornado.websocket import WebSocketHandler
    from tornado.options import define, options
    from tornado import locale
    from tornado import version as tornado_version
    from tornado import version_info as tornado_version_info
    print("\x1b[31;1mERROR:\x1b[0m Gate One requires the Tornado framework.  "
          "You probably want to run something like, \x1b[1m'pip install "
          "--upgrade tornado'\x1b[0m.")
    sys.exit(1)

if tornado_version_info[0] < 2 and tornado_version_info[1] < 2:
    print("\x1b[31;1mERROR:\x1b[0m Gate One requires version 2.2+ of the "
            "Tornado framework.  The installed version of Tornado is version "
            "%s." % tornado_version)

# We want this turned on right away
tornado.options.enable_pretty_logging()

# Our own modules
import termio, terminal
from auth import NullAuthHandler, KerberosAuthHandler, GoogleAuthHandler
from auth import PAMAuthHandler
from utils import str2bool, generate_session_id, cmd_var_swap, mkdir_p
from utils import gen_self_signed_ssl, killall, get_plugins, load_plugins
from utils import create_plugin_links, merge_handlers, none_fix, short_hash
from utils import convert_to_timedelta, kill_dtached_proc, FACILITIES, which
from utils import process_opt_esc_sequence, create_data_uri, MimeTypeFail
from utils import string_to_syslog_facility, fallback_bell, json_encode
from utils import write_pid, read_pid, remove_pid, drop_privileges
from utils import recursive_chown, ChownError

# Setup the locale functions before anything else
locale.set_default_locale('en_US')
user_locale = None # Replaced with the actual user locale object in __main__
def _(string):
    """
    Wraps user_locale.translate so we can .encode('UTF-8') when writing to
    stdout.  This function will get overridden by the regular translate()
    function in __main__
    """
    if user_locale:
        return user_locale.translate(string).encode('UTF-8')
    else:
        return string.encode('UTF-8')

# Globals
SESSIONS = {} # We store the crux of most session info here
CMD = None # Will be overwritten by options.command
TIMEOUT = timedelta(days=5) # Gets overridden by options.session_timeout
WATCHER = None # Holds the reference to our timeout_sessions periodic callback
CLEANER = None # The reference to our session logs cleanup periodic callback
GATEONE_DIR = os.path.dirname(os.path.abspath(__file__))
PLUGINS = get_plugins(os.path.join(GATEONE_DIR, 'plugins'))
PLUGIN_WS_CMDS = {} # Gives plugins the ability to extend/enhance TerminalWebSocket
PLUGIN_HOOKS = {} # Gives plugins the ability to hook into various things.
PLUGIN_AUTH_HOOKS = [] # For plugins to register functions to be called after a
                       # user successfully authenticates
# Gate One registers a handler for for terminal.py's CALLBACK_OPT special escape
# sequence callback.  Whenever this escape sequence is encountered, Gate One
# will parse the sequence's contained characters looking for the following
# format:
#   <plugin name>|<whatever>
# The <whatever> part will be passed to any plugin matching <plugin name> if the
# plugin has 'Escape': <function> registered in its hooks.
PLUGIN_ESC_HANDLERS = {}
# This is used to store plugin terminal hooks that are called when a new
# terminal is created (so a plugin could override/attach callbacks to the
# multiplex or terminal emulator instances).
PLUGIN_TERM_HOOKS = {}

# Secondary locale setup
locale_dir = os.path.join(GATEONE_DIR, 'i18n')
locale.load_gettext_translations(locale_dir, 'gateone')
# NOTE: The locale gets set in __main__

# A HUGE thank you to Micah Elliott (http://MicahElliott.com) for posting these
# values here: https://gist.github.com/719710
# This gets used by StyleHandler to generate the CSS that supports 256-colors:
COLORS_256 = {
    # 8-color equivalents:
     0: "000000",
     1: "800000",
     2: "008000",
     3: "808000",
     4: "000080",
     5: "800080",
     6: "008080",
     7: "c0c0c0",
    # "Bright" (16-color) equivalents:
     8: "808080",
     9: "ff0000",
    10: "00ff00",
    11: "ffff00",
    12: "0000ff",
    13: "ff00ff",
    14: "00ffff",
    15: "ffffff",
    # The rest of the 256-colors:
    16: "000000",
    17: "00005f",
    18: "000087",
    19: "0000af",
    20: "0000d7",
    21: "0000ff",
    22: "005f00",
    23: "005f5f",
    24: "005f87",
    25: "005faf",
    26: "005fd7",
    27: "005fff",
    28: "008700",
    29: "00875f",
    30: "008787",
    31: "0087af",
    32: "0087d7",
    33: "0087ff",
    34: "00af00",
    35: "00af5f",
    36: "00af87",
    37: "00afaf",
    38: "00afd7",
    39: "00afff",
    40: "00d700",
    41: "00d75f",
    42: "00d787",
    43: "00d7af",
    44: "00d7d7",
    45: "00d7ff",
    46: "00ff00",
    47: "00ff5f",
    48: "00ff87",
    49: "00ffaf",
    50: "00ffd7",
    51: "00ffff",
    52: "5f0000",
    53: "5f005f",
    54: "5f0087",
    55: "5f00af",
    56: "5f00d7",
    57: "5f00ff",
    58: "5f5f00",
    59: "5f5f5f",
    60: "5f5f87",
    61: "5f5faf",
    62: "5f5fd7",
    63: "5f5fff",
    64: "5f8700",
    65: "5f875f",
    66: "5f8787",
    67: "5f87af",
    68: "5f87d7",
    69: "5f87ff",
    70: "5faf00",
    71: "5faf5f",
    72: "5faf87",
    73: "5fafaf",
    74: "5fafd7",
    75: "5fafff",
    76: "5fd700",
    77: "5fd75f",
    78: "5fd787",
    79: "5fd7af",
    80: "5fd7d7",
    81: "5fd7ff",
    82: "5fff00",
    83: "5fff5f",
    84: "5fff87",
    85: "5fffaf",
    86: "5fffd7",
    87: "5fffff",
    88: "870000",
    89: "87005f",
    90: "870087",
    91: "8700af",
    92: "8700d7",
    93: "8700ff",
    94: "875f00",
    95: "875f5f",
    96: "875f87",
    97: "875faf",
    98: "875fd7",
    99: "875fff",
    100: "878700",
    101: "87875f",
    102: "878787",
    103: "8787af",
    104: "8787d7",
    105: "8787ff",
    106: "87af00",
    107: "87af5f",
    108: "87af87",
    109: "87afaf",
    110: "87afd7",
    111: "87afff",
    112: "87d700",
    113: "87d75f",
    114: "87d787",
    115: "87d7af",
    116: "87d7d7",
    117: "87d7ff",
    118: "87ff00",
    119: "87ff5f",
    120: "87ff87",
    121: "87ffaf",
    122: "87ffd7",
    123: "87ffff",
    124: "af0000",
    125: "af005f",
    126: "af0087",
    127: "af00af",
    128: "af00d7",
    129: "af00ff",
    130: "af5f00",
    131: "af5f5f",
    132: "af5f87",
    133: "af5faf",
    134: "af5fd7",
    135: "af5fff",
    136: "af8700",
    137: "af875f",
    138: "af8787",
    139: "af87af",
    140: "af87d7",
    141: "af87ff",
    142: "afaf00",
    143: "afaf5f",
    144: "afaf87",
    145: "afafaf",
    146: "afafd7",
    147: "afafff",
    148: "afd700",
    149: "afd75f",
    150: "afd787",
    151: "afd7af",
    152: "afd7d7",
    153: "afd7ff",
    154: "afff00",
    155: "afff5f",
    156: "afff87",
    157: "afffaf",
    158: "afffd7",
    159: "afffff",
    160: "d70000",
    161: "d7005f",
    162: "d70087",
    163: "d700af",
    164: "d700d7",
    165: "d700ff",
    166: "d75f00",
    167: "d75f5f",
    168: "d75f87",
    169: "d75faf",
    170: "d75fd7",
    171: "d75fff",
    172: "d78700",
    173: "d7875f",
    174: "d78787",
    175: "d787af",
    176: "d787d7",
    177: "d787ff",
    178: "d7af00",
    179: "d7af5f",
    180: "d7af87",
    181: "d7afaf",
    182: "d7afd7",
    183: "d7afff",
    184: "d7d700",
    185: "d7d75f",
    186: "d7d787",
    187: "d7d7af",
    188: "d7d7d7",
    189: "d7d7ff",
    190: "d7ff00",
    191: "d7ff5f",
    192: "d7ff87",
    193: "d7ffaf",
    194: "d7ffd7",
    195: "d7ffff",
    196: "ff0000",
    197: "ff005f",
    198: "ff0087",
    199: "ff00af",
    200: "ff00d7",
    201: "ff00ff",
    202: "ff5f00",
    203: "ff5f5f",
    204: "ff5f87",
    205: "ff5faf",
    206: "ff5fd7",
    207: "ff5fff",
    208: "ff8700",
    209: "ff875f",
    210: "ff8787",
    211: "ff87af",
    212: "ff87d7",
    213: "ff87ff",
    214: "ffaf00",
    215: "ffaf5f",
    216: "ffaf87",
    217: "ffafaf",
    218: "ffafd7",
    219: "ffafff",
    220: "ffd700",
    221: "ffd75f",
    222: "ffd787",
    223: "ffd7af",
    224: "ffd7d7",
    225: "ffd7ff",
    226: "ffff00",
    227: "ffff5f",
    228: "ffff87",
    229: "ffffaf",
    230: "ffffd7",
    231: "ffffff",
    # Grayscale:
    232: "080808",
    233: "121212",
    234: "1c1c1c",
    235: "262626",
    236: "303030",
    237: "3a3a3a",
    238: "444444",
    239: "4e4e4e",
    240: "585858",
    241: "626262",
    242: "6c6c6c",
    243: "767676",
    244: "808080",
    245: "8a8a8a",
    246: "949494",
    247: "9e9e9e",
    248: "a8a8a8",
    249: "b2b2b2",
    250: "bcbcbc",
    251: "c6c6c6",
    252: "d0d0d0",
    253: "dadada",
    254: "e4e4e4",
    255: "eeeeee"
}

def require_auth(method):
    """
    An equivalent to tornado.web.authenticated for WebSockets
    (TerminalWebSocket, specifically).
    """
    @wraps(method)
    def wrapper(self, *args, **kwargs):
        if not self.get_current_user():
            self.write_message(_("Only valid users please.  Thanks!"))
            self.close() # Close the WebSocket
        return method(self, *args, **kwargs)
    return wrapper

def kill_session(session, kill_dtach=False):
    """
    Terminates all the processes associated with *session* and then removes
    *session* from the global SESSIONS dict.  If *kill_dtach* is True, will
    also call kill_dtached_proc() on the terminal processes.
    """
    for term in list(SESSIONS[session].keys()):
        if isinstance(term, int) and term in SESSIONS[session]:
            if SESSIONS[session][term]['multiplex'].isalive():
                SESSIONS[session][term]['multiplex'].terminate()
            if kill_dtach:
                kill_dtached_proc(session, term)
    del SESSIONS[session]

@atexit.register # I love this feature!
def kill_all_sessions(kill_dtach=False):
    """
    Calls kill_session() on all sessions in the SESSIONS dict.
    """
    for session in SESSIONS.keys():
        kill_session(session, kill_dtach)

def timeout_sessions(kill_dtach=False):
    """
    Loops over the SESSIONS dict killing any sessions that haven't been used
    for the length of time specified in *TIMEOUT* (global).  The value of
    *TIMEOUT* can be set in server.conf or specified on the command line via the
    *session_timeout* value.  Arguments:

     * *kill_dtach* - If True, will call kill_dtached_proc() on each terminal to ensure it dies.

    .. note:: This function is meant to be called via Tornado's ioloop.PeriodicCallback().
    """
    #logging.debug("timeout_sessions() TIMEOUT: %s" % TIMEOUT)
    try:
        if not SESSIONS: # Last client has timed out
            logging.info(_("All user sessions have terminated."))
            global WATCHER
            if WATCHER:
                WATCHER.stop() # Stop ourselves
                WATCHER = None # Reset so authenticate() will know to start it
        for session in list(SESSIONS.keys()):
            if "last_seen" not in SESSIONS[session]:
                # Session is in the process of being created.  We'll check it
                # the next time timeout_sessions() is called.
                continue
            if SESSIONS[session]["last_seen"] == 'connected':
                # Connected sessions do not need to be checked for timeouts
                continue
            if datetime.now() > SESSIONS[session]["last_seen"] + TIMEOUT:
                # Kill the session
                logging.info(_("{session} timeout.".format(session=session)))
                kill_session(session, kill_dtach)
        logging.info(_(
            "Exception encountered in timeout_sessions(): {exception}".format(
                exception=e)
        ))
        import traceback
        traceback.print_exc(file=sys.stdout)

def cleanup_session_logs(users_dir, max_age):
    """
    Cleans up all user's session logs older than *max_age* (timedelta) given the
    *users_dir* (string).  The session log directory is assumed to be:

        *users_dir*/<user>/logs
    """
    logging.debug("cleanup_session_logs()")
    for user in os.listdir(users_dir):
        logs_path = os.path.join(users_dir, user, 'logs')
        if not os.path.exists(logs_path):
            # Nothing to do
            continue
        for log_name in os.listdir(logs_path):
            log_path = os.path.join(logs_path, log_name)
            mtime = time.localtime(os.stat(log_path).st_mtime)
            # Convert to a datetime object for easier comparison
            mtime = datetime.fromtimestamp(time.mktime(mtime))
            if datetime.now() - mtime > max_age:
                # The log is older than max_age, remove it
                os.remove(log_path)

# Classes
class HTTPSRedirectHandler(tornado.web.RequestHandler):
    """
    A handler to redirect clients from HTTP to HTTPS.
    """
        """Just redirects the client from HTTP to HTTPS"""
        port = self.settings['port']
        url_prefix = self.settings['url_prefix']
        host = self.request.headers.get('Host', 'localhost')
        self.redirect(
            'https://%s:%s%s' % (host, port, url_prefix))

class StaticHandler(tornado.web.StaticFileHandler):
    """
    An override of :class:`tornado.web.StaticFileHandler` to ensure that the
    Access-Control-Allow-Origin header gets set correctly.
    """
    # This is the only function we need to override thanks to the thoughtfulness
    # of the Tornado devs.
    def set_extra_headers(self, path):
        """
        Adds the Access-Control-Allow-Origin header to allow cross-origin
        access to static content for applications embedding Gate One.
        Specifically, this is necessary in order to support loading fonts
        from different origins.
        """
        self.set_header('Access-Control-Allow-Origin', '*')

class BaseHandler(tornado.web.RequestHandler):
    """
    A base handler that all Gate One RequestHandlers will inherit methods from.
    """
    # Right now it's just the one function...
        if user_json:
            user = json_decode(user_json)
            if user and 'upn' not in user:
                return None
            return user
    # More may be added in the future

class MainHandler(BaseHandler):
    """
    Renders index.html which loads Gate One.

    Will include the minified version of gateone.js if available as
    gateone.min.js.

    Will encode GATEONE_DIR/static/bell.ogg as a data:URI and put it as the
    <source> of the <audio> tag inside the index.html template.  Gate One
    administrators can replace bell.ogg with whatever they like but the file
    size should be less than 32k when encoded to Base64.
    """
    # TODO: Add the ability for users to define their own individual bells.
    @tornado.web.authenticated
    @tornado.web.addslash
        hostname = os.uname()[1]
        gateone_js = "%sstatic/gateone.js" % self.settings['url_prefix']
        minified_js_abspath = os.path.join(GATEONE_DIR, 'static')
        minified_js_abspath = os.path.join(
            minified_js_abspath, 'gateone.min.js')
        js_init = self.settings['js_init']
        # Use the minified version if it exists
        if os.path.exists(minified_js_abspath):
            gateone_js = "%sstatic/gateone.min.js" % self.settings['url_prefix']
        template_path = os.path.join(GATEONE_DIR, 'templates')
        index_path = os.path.join(template_path, 'index.html')
        head_html = ""
        body_html = ""
        for plugin, hooks in PLUGIN_HOOKS.items():
            if 'HTML' in hooks:
                if 'head' in hooks['HTML']:
                    if hooks['HTML']['head']:
                        for item in hooks['HTML']['head']:
                            head_html += "%s\n" % item
                if 'body' in hooks['HTML']:
                    if hooks['HTML']['body']:
                        for item in hooks['HTML']['body']:
                            body_html += "%s\n" % item
        self.render(
            index_path,
            hostname=hostname,
            gateone_js=gateone_js,
            jsplugins=PLUGINS['js'],
            cssplugins=PLUGINS['css'],
            js_init=js_init,
            url_prefix=self.settings['url_prefix'],
            head=head_html,
            body=body_html
        )

class PluginCSSTemplateHandler(BaseHandler):
    """
    Renders plugin CSS template files, passing them the same *prefix* and
    *container* variables used by the StyleHandler.  This is so we don't need a
    CSS template rendering function in every plugin that needs to use {{prefix}}
    or {{container}}.

    gateone.js will automatically load all \*.css files in plugin template
    directories using this method.
    """
    # Had to disable authentication for this for the embedded stuff to work.
    # Not a big deal...  Just some stylesheets.  To an attacker it's like
    # peering into a window and seeing the wallpaper.
        container = self.get_argument("container")
        prefix = self.get_argument("prefix")
        plugin = self.get_argument("plugin")
        template = self.get_argument("template")
        templates_path = os.path.join(GATEONE_DIR, 'templates')
        plugin_templates_path = os.path.join(templates_path, plugin)
        plugin_template = os.path.join(plugin_templates_path, "%s.css" % plugin)
        self.set_header('Content-Type', 'text/css')
        try:
            self.render(
                plugin_template,
                container=container,
                prefix=prefix,
                url_prefix=self.settings['url_prefix']
            )
        except IOError:
            # The provided plugin/template combination was not found
            logging.error(_("%s.css was not found" % plugin_template))

class JSPluginsHandler(BaseHandler):
    """
    Combines all JavaScript plugins into a single file to keep things simple and
    speedy.
    """
    # No auth for this...  Not really necessary (just serves up a static file).
        self.set_header('Content-Type', 'application/javascript')
        # This is so we can load combined_plugins in a <script> tag without
        # having to accept the SSL certificate:
        plugins = get_plugins(os.path.join(GATEONE_DIR, "plugins"))
        static_dir = os.path.join(GATEONE_DIR, "static")
        combined_plugins = os.path.join(static_dir, "combined_plugins.js")
        if os.path.exists(combined_plugins):
            with open(combined_plugins) as f:
                js_data = f.read()
                if len(js_data) < 100: # Needs to be created
                    self.write(self._combine_plugins())
                    return
                else: # It hasn't changed, send it as-is
                    self.write(js_data)
        else: # File doesn't exist, create it and send it to the client
            self.write(self._combine_plugins())

    def _combine_plugins(self):
        """
        Combines all plugin .js files into one (combined_plugins.js)
        """
        out = ""
        for js_plugin in plugins['js']:
            js_path = os.path.join(GATEONE_DIR, js_plugin.lstrip('/'))
            with open(js_path) as f:
                out += f.read()
        with open(combined_plugins, 'w') as f:
            f.write(out)
        return out

class TerminalWebSocket(WebSocketHandler):
    """
    The main WebSocket interface for Gate One, this class is setup to call
    'commands' which are methods registered in self.commands.  Methods that are
    registered this way will be exposed and directly callable over the
    WebSocket.
    """
    def __init__(self, application, request):
        WebSocketHandler.__init__(self, application, request)
        self.commands = {
            'ping': self.pong,
            'authenticate': self.authenticate,
            'new_terminal': self.new_terminal,
            'set_terminal': self.set_terminal,
            'kill_terminal': self.kill_terminal,
            'c': self.char_handler, # Just 'c' to keep the bandwidth down
            'write_chars': self.write_chars,
            'refresh': self.refresh_screen,
            'full_refresh': self.full_refresh,
            'resize': self.resize,
            'get_bell': self.get_bell,
            'get_webworker': self.get_webworker,
            'get_style': self.get_style,
            #'get_js': self.get_js,
            'enumerate_themes': self.enumerate_themes,
            'manual_title': self.manual_title,
            'debug_terminal': self.debug_terminal
        }
        self.terms = {}
        # So we can keep track and avoid sending unnecessary messages:
        self.titles = {}
        self.user = None
        self.em_dimensions = None
        # This is used to keep track of used API authentication signatures so
        # we can prevent replay attacks.
        self.prev_signatures = []
        self.origin_denied = True # Only allow valid origins

    def allow_draft76(self):
        """
        By overriding this function we're allowing the older version of the
        WebSockets protocol.  As long as communications happens over SSL there
        shouldn't be any security concerns with this.  This is mostly to support
        iOS Safari.
        """

        """
        Mostly identical to the function of the same name in MainHandler.  The
        difference being that when API authentication is enabled the WebSocket
        will expect and perform its own auth of the client.
        """
        if self.user:
            return self.user
        if not user_json:
            if not self.settings['auth']:
                # This can happen if the user's browser isn't allowing
                # persistent cookies (e.g. incognito mode)
                return {'upn': 'ANONYMOUS', 'session': generate_session_id()}
            return None
        return json_decode(user_json)

    def open(self):
        """
        Called when a new WebSocket is opened.  Will deny access to any
        origin that is not defined in self.settings['origin'].
        """
        valid_origins = self.settings['origins']
        if 'Origin' in self.request.headers:
            origin_header = self.request.headers['Origin']
        elif 'Sec-Websocket-Origin' in self.request.headers: # Old version
            origin_header = self.request.headers['Sec-Websocket-Origin']
        origin_header = origin_header.lower() # hostnames are case-insensitive
        if '*' not in valid_origins:
            if origin_header not in valid_origins:
                self.origin_denied = True
                origin = origin_header
                short_origin = origin.split('//')[1]
                denied_msg = _("Access denied for origin: %s" % origin)
                logging.error(denied_msg)
                self.write_message(denied_msg)
                self.write_message(_(
                    "If you feel this is incorrect you just have to add '%s' to"
                    " the 'origin' option in your server.conf.  See the docs "
                    "for details." % short_origin
                ))
                self.close()
        self.origin_denied = False
        # TODO: Make it so that idle WebSockets that haven't passed authentication tests get auto-closed within N seconds in order to prevent a DoS scenario where the attacker keeps all possible ports open indefinitely.
        # client_id is unique to the browser/client whereas session_id is unique
        # to the user.  It isn't used much right now but it will be useful in
        # the future once more stuff is running over WebSockets.
        self.client_id = generate_session_id()
        user = self.get_current_user()
        if user and 'upn' in user:
            logging.info(
                _("WebSocket opened (%s).") % user['upn'])
        else:
            logging.info(_("WebSocket opened (unknown user)."))
        if user and 'upn' not in user: # Invalid user info
            logging.error(_("Unauthenticated WebSocket attempt."))
            # In case this is a legitimate client that simply had its auth info
            # expire/go bad, tell it to re-auth by calling the appropriate
            # action on the other side.
            message = {'reauthenticate': True}
            self.write_message(json_encode(message))

    def on_message(self, message):
        """Called when we receive a message from the client."""
        # This is super useful when debugging:
        logging.debug("message: %s" % repr(message))
        if self.origin_denied:
            logging.error(_("Message rejected due to invalid origin."))
        message_obj = None
        try:
            message_obj = json_decode(message) # JSON FTW!
            if not isinstance(message_obj, dict):
                self.write_message(_("'Error: Message bust be a JSON dict.'"))
                return
        except ValueError: # We didn't get JSON
            self.write_message(_("'Error: We only accept JSON here.'"))
            return
        if message_obj:
            for key, value in message_obj.items():
                if key in PLUGIN_WS_CMDS:
                    try: # Plugins first so they can override behavior if they wish
                        PLUGIN_WS_CMDS[key](value, tws=self)# tws==TerminalWebSocket
                    except (KeyError, TypeError, AttributeError) as e:
                        logging.error(_(
                            "Error running plugin WebSocket action: %s" % key))
                else:
                    try:
                        if value:
                            self.commands[key](value)
                        else:
                            # Try, try again
                            self.commands[key]()
                        logging.debug(e)
                        if self.settings['logging'] == "debug":
                            import traceback
                            traceback.print_exc(file=sys.stdout)
                        logging.error(_('Unknown WebSocket action: %s' % key))

    def on_close(self):
        """
        Called when the client terminates the connection.

        .. note:: Normally self.refresh_screen() catches the disconnect first and this method won't end up being called.
        """
        logging.debug("on_close()")
        if user and user['session'] in SESSIONS:
            # Update 'last_seen' with a datetime object for accuracy
            SESSIONS[user['session']]['last_seen'] = datetime.now()
            # Remove all attached callbacks so we're not wasting memory/CPU on
            # disconnected clients
            for term in SESSIONS[user['session']]:
                if isinstance(term, int):
                    try:
                        multiplex = SESSIONS[user['session']][term]['multiplex']
                        multiplex.remove_all_callbacks(self.callback_id)
                        client_dict = SESSIONS[user['session']][term][
                            self.client_id]
                        term_emulator = multiplex.term
                        term_emulator.remove_all_callbacks(self.callback_id)
                        # Remove anything associated with the client_id
                        multiplex.io_loop.remove_timeout(
                            client_dict['refresh_timeout'])
                        del SESSIONS[user['session']][term][self.client_id]
                    except AttributeError:
                        # User never completed opening a terminal so
                        # self.callback_id is missing.  Nothing to worry about
                        if self.client_id in SESSIONS[user['session']][term]:
                            del SESSIONS[user['session']][term][self.client_id]
                _("WebSocket closed (%s).") % user['upn'])
        else:
            logging.info(_("WebSocket closed (unknown user)."))

    def pong(self, timestamp):
        """
        Responds to a client 'ping' request...  Just returns the given
        timestamp back to the client so it can measure round-trip time.
        """
        message = {'pong': timestamp}
        self.write_message(json_encode(message))

    def authenticate(self, settings):
        """
        Authenticates the client by first trying to use the 'gateone_user'
        cookie or if Gate One is configured to use API authentication it will
        use *settings['auth']*.  Additionally, it will accept
        *settings['container']* and *settings['prefix']* to apply those to the
        equivalent properties (self.container and self.prefix).
        """
        logging.debug("authenticate(): %s" % settings)
        # Make sure the client is authenticated if authentication is enabled
        if self.settings['auth'] and self.settings['auth'] != 'api':
            try:
                if not user:
                    logging.error(_("Unauthenticated WebSocket attempt."))
                    # This usually happens when the cookie_secret gets changed
                    # resulting in "Invalid cookie..." errors.  If we tell the
                    # client to re-auth the problem should correct itself.
                    message = {'reauthenticate': True}
                    self.write_message(json_encode(message))
                    return
                elif user and user['upn'] == 'ANONYMOUS':
                    # This can happen when a client logs in with no auth type
                    # configured and then later the server is configured to use
                    # authentication.  The client must be told to re-auth:
                    return
            except KeyError: # 'upn' wasn't in user
                # Force them to authenticate
                message = {'reauthenticate': True}
                self.write_message(json_encode(message))
                self.close() # Close the WebSocket
        elif self.settings['auth'] and self.settings['auth'] == 'api':
            if 'auth' in settings.keys():
                # 'auth' message should look like this:
                # {
                #    'api_key': 'MjkwYzc3MDI2MjhhNGZkNDg1MjJkODgyYjBmN2MyMTM4M',
                #    'upn': 'joe@company.com',
                #    'timestamp': 1323391717238,
                #    'signature': <gibberish>,
                #    'signature_method': 'HMAC-SHA1',
                #    'api_version': '1.0'
                # }
                #
                # *api_key* is the first half of what gets generated when you
                #   run ./gateone --new_api_key.
                # *upn* is the User Principal Name of the user.  This is
                #   typically something like "joe@company.com".
                # *timestamp* is a JavaScript Date() object converted into an
                #   "time since the epoch": var timestamp = new Date().getTime()
                # *signature* is an HMAC signature of the previous three
                #   variables that was created using the given API key's secret.
                # *signature_method* is the HMAC hashing algorithm to use for
                #   the signature.  Only HMAC-SHA1 is supported for now.
                # *api_version* is the auth API version.  Always "1.0" for now.
                #
                # For reference, here's how to make a signature using PHP:
                # $authobj = array('api_key' => 'M2I1MzJmZjk4MTEwNDk2Zjk4MjMwNmMwMTVkODQzMTEyO', 'upn' => $_SERVER['REMOTE_USER'], 'timestamp' => time() . '0000', 'signature_method' => 'HMAC-SHA1', 'api_version' => '1.0');
                # $authobj['signature'] = hash_hmac('sha1', $authobj['api_key'] . $authobj['upn'] . $authobj['timestamp'], '<secret>');
                # Note that the order matters:  api_key -> upn -> timestamp
                auth_obj = settings['auth']
                if 'api_key' in auth_obj:
                    # Assume everything else is present if the api_key is there
                    api_key = auth_obj['api_key']
                    upn = auth_obj['upn']
                    timestamp = auth_obj['timestamp']
                    signature = auth_obj['signature']
                    signature_method = auth_obj['signature_method']
                    api_version = auth_obj['api_version']
                    if signature_method != 'HMAC-SHA1':
                        message = {
                            'notice': _(
                                'AUTHENTICATION ERROR: Unsupported API auth '
                                'signature method: %s' % signature_method)
                        }
                        self.write_message(json_encode(message))
                        return
                    print("api_keys: %s" % self.settings['api_keys'])
                    secret = self.settings['api_keys'][api_key]
                    # Check the signature against existing API keys
                    sig_check = tornado.web._create_signature(
                        secret, api_key, upn, timestamp)
                    if sig_check == signature:
                        # Everything matches (great!) so now we do due diligence
                        # by checking the timestamp against the
                        # api_timestamp_window setting and whether or not we've
                        # already used it (to prevent replay attacks).
                        if signature in self.prev_signatures:
                            logging.error(_(
                            "API authentication replay attack detected!  User: "
                            "%s, Remote IP: %s, Origin: %s" % (
                                upn, self.request.remote_ip, origin_header)))
                            message = {'notice': _(
                                'AUTH FAILED: Replay attack detected!  This '
                                'event has been logged.')}
                            self.write_message(json_encode(message))
                            self.close()
                            return
                        window = self.settings['api_timestamp_window']
                        then = datetime.fromtimestamp(int(timestamp)/1000)
                        time_diff = datetime.now() - then
                        if time_diff > window:
                            "API authentication failed due to an expired auth "
                            "object.  If you just restarted the server this is "
                            "normal (users just need to reload the page).  If "
                            " this problem persists it could be a problem with "
                            "the server's clock (either this server or the "
                            "server(s) embedding Gate One)."
                            ))
                                'AUTH FAILED: Authentication object timed out. '
                                'Try reloading this page (F5).')}
                                'AUTH FAILED: If the problem persists after '
                                'reloading this page please contact your server'
                                ' administrator to notify them of the issue.')}
                            return
                        logging.debug(_("API Authentication Successful"))
                        self.prev_signatures.append(signature) # Prevent replays
                # Make a directory to store this user's settings/files/logs/etc
                        user_dir = os.path.join(self.settings['user_dir'], upn)
                        if not os.path.exists(user_dir):
                            logging.info(
                                _("Creating user directory: %s" % user_dir))
                            mkdir_p(user_dir)
                            os.chmod(user_dir, 0o770)
                        session_file = os.path.join(user_dir, 'session')
                        if os.path.exists(session_file):
                            session_data = open(session_file).read()
                            self.user = json_decode(session_data)
                        else:
                            with open(session_file, 'w') as f:
                        # Save it so we can keep track across multiple clients
                                self.user = {
                                    'upn': upn, # FYI: UPN == userPrincipalName
                                    'session': generate_session_id()
                                }
                                session_info_json = json_encode(self.user)
                                f.write(session_info_json)
                    else:
                            "WebSocket auth failed signature check."))
            else:
                logging.error(_("Missing API Key in authentication object"))
        else: # Anonymous auth
            # Double-check there isn't a user set in the cookie (i.e. we have
            # recently changed Gate One's settings).  If there is, force it
            # back to ANONYMOUS.
            if settings['auth']: # Authentication from localStorage data
                # Authenticate/decode the encoded auth info
                cookie_data = self.get_secure_cookie(
                    'gateone_user', value=settings['auth'])
                if cookie_data:
                    self.user = json_decode(cookie_data)
            if not self.user:
                # Generate a new session/anon user
                self.user = self.get_current_user()
                # Also store/update their session info in localStorage
                encoded_user = self.create_signed_value(
                    'gateone_user', tornado.escape.json_encode(self.user))
                session_message = {'gateone_user': encoded_user}
                self.write_message(json_encode(session_message))
            if self.user['upn'] != 'ANONYMOUS':
                # Gate One server's auth config probably changed
                #self.close() # Close the WebSocket
                return
        try:
            if user and 'session' in user:
                self.session = user['session']
            else:
                logging.error(_("Authentication failed for unknown user"))
                message = {'notice': _('AUTHENTICATION ERROR: User unknown')}
                return
        except Exception as e:
                "Exception encountered trying to authenticate: %s" % e))
            return
        try:
            # Execute any post-authentication hooks that plugins have registered
            if PLUGIN_AUTH_HOOKS:
                for auth_hook in PLUGIN_AUTH_HOOKS:
                    auth_hook(self.get_current_user(), self.settings)
            logging.error(_("Exception in registered Auth hook: %s" % e))
        # Apply the container/prefix settings (if present)
        # NOTE:  Currently these are only used by the logging plugin
        if 'container' in settings:
            self.container = settings['container']
        if 'prefix' in settings:
            self.prefix = settings['prefix']
        # This check is to make sure there's no existing session so we don't
        # accidentally clobber it.
        if self.session not in SESSIONS:
            # Old session is no good, start a new one:
            SESSIONS[self.session] = {
                'last_seen': 'connected'
            }
        else:
            SESSIONS[self.session]['last_seen'] = 'connected'
        terminals = []
        for term in list(SESSIONS[self.session].keys()):
            if isinstance(term, int):
                terminals.append(term) # Only terminals are integers in the dict
        # Check for any dtach'd terminals we might have missed
        if self.settings['dtach']:
            session_dir = self.settings['session_dir']
            session_dir = os.path.join(session_dir, self.session)
            if not os.path.exists(session_dir):
                mkdir_p(session_dir)
                os.chmod(session_dir, 0o770)
            for item in os.listdir(session_dir):
                if item.startswith('dtach_'):
                    term = int(item.split('_')[1])
                    if term not in terminals:
                        terminals.append(term)
        terminals.sort() # Put them in order so folks don't get confused
        message = {
            'terminals': terminals,
        # This is just so the client has a human-readable point of reference:
            'set_username': self.get_current_user()['upn']
        }
        # TODO: Add a hook here for plugins to send their own messages when a
        #       given terminal is reconnected.
        # Startup the watcher if it isn't already running
        global WATCHER
        if not WATCHER:
            interval = 30*1000 # Check every 30 seconds
            watcher = partial(timeout_sessions, options.dtach)
            WATCHER = tornado.ioloop.PeriodicCallback(watcher, interval)
            WATCHER.start()

    def new_multiplex(self, cmd, term_id, logging=True):
        """
        Returns a new instance of :py:class:`termio.Multiplex` with the proper global and
        client-specific settings.

            * *cmd* - The command to execute inside of Multiplex.
            * *term_id* - The terminal to associate with this Multiplex or a descriptive identifier (it's only used for logging purposes).
            * *logging* - If False, logging will be disabled for this instance of Multiplex (even if it would otherwise be enabled).
        """
        user_dir = self.settings['user_dir']
        try:
        except:
            # No auth, use ANONYMOUS (% is there to prevent conflicts)
            user = r'ANONYMOUS' # Don't get on this guy's bad side
        session_dir = self.settings['session_dir']
        session_dir = os.path.join(session_dir, self.session)
        log_path = None
        syslog_logging = False
        if logging:
            syslog_logging = self.settings['syslog_session_logging']
            if self.settings['session_logging']:
                log_dir = os.path.join(user_dir, user)
                log_dir = os.path.join(log_dir, 'logs')
                # Create the log dir if not already present
                if not os.path.exists(log_dir):
                    mkdir_p(log_dir)
                log_name = datetime.now().strftime('%Y%m%d%H%M%S%f.golog')
                log_path = os.path.join(log_dir, log_name)
        facility = string_to_syslog_facility(self.settings['syslog_facility'])
        return termio.Multiplex(
            cmd,
            log_path=log_path,
            user=user,
            term_id=term_id,
            syslog=syslog_logging,
            syslog_facility=facility,
            syslog_host=self.settings['syslog_host']
        )

    def term_ended(self, term):
        """
        Sends the 'term_ended' message to the client letting it know that the
        given *term* is no more.
        """
        message = {'term_ended': term}

    @require_auth
    def new_terminal(self, settings):
        """
        Starts up a new terminal associated with the user's session using
        *settings* as the parameters.  If a terminal already exists with the
        same number as *settings[term]*, self.set_terminal() will be called
        instead of starting a new terminal (so clients can resume their session
        without having to worry about figuring out if a new terminal already
        exists or not).
        """
        logging.debug("%s new_terminal(): %s" % (
            self.get_current_user()['upn'], settings))
            # This happens when timeout_sessions() times out a session
            # Tell the client it timed out:
            message = {'timeout': None}
            return
        self.current_term = term = settings['term']
        self.rows = rows = settings['rows']
        self.cols = cols = settings['cols']
        if 'em_dimensions' in settings:
            self.em_dimensions = {
                'height': settings['em_dimensions']['h'],
                'width': settings['em_dimensions']['w']
            }
        needs_full_refresh = False
        if term not in SESSIONS[self.session]:
            # Setup the requisite dict
            SESSIONS[self.session][term] = {
                'last_activity': datetime.now(),
                'title': 'Gate One',
                'manual_title': False
            }
        if self.client_id not in SESSIONS[self.session][term]:
            SESSIONS[self.session][term][self.client_id] = {
                # Used by refresh_screen()
                'refresh_timeout': None
            }
        if 'multiplex' not in SESSIONS[self.session][term]:
            # Start up a new terminal
            SESSIONS[self.session][term]['created'] = datetime.now()
            # NOTE: Not doing anything with 'created'...  yet!
            now = int(round(time.time() * 1000))
            try:
            except:
                # No auth, use ANONYMOUS (% is there to prevent conflicts)
                user = 'ANONYMOUS' # Don't get on this guy's bad side
            cmd = cmd_var_swap(CMD,   # Swap out variables like %USER% in CMD
                session=self.session, # with their real-world values.
                session_hash=short_hash(self.session),
                user_dir=user_dir,
                user=user,
                time=now
            )
            resumed_dtach = False
            # Create the session dir if not already present
            if self.settings['dtach']: # Wrap in dtach (love this tool!)
                dtach_path = "%s/dtach_%s" % (session_dir, term)
                if os.path.exists(dtach_path):
                    # Using 'none' for the refresh because the EVIL termio
                    # likes to manage things like that on his own...
                    cmd = "dtach -a %s -E -z -r none" % dtach_path
                    resumed_dtach = True
                else: # No existing dtach session...  Make a new one
                    cmd = "dtach -c %s -E -z -r none %s" % (dtach_path, cmd)
            m = SESSIONS[self.session][term]['multiplex'] = self.new_multiplex(
                cmd, term)
            # Set some environment variables so the programs we execute can use
            # them (very handy).  Allows for "tight integration" and "synergy"!
            env = {
                'GO_USER_DIR': user_dir,
                'GO_USER': user,
                'GO_TERM': str(term),
                'GO_SESSION': self.session,
                'GO_SESSION_DIR': session_dir
            }
            m.spawn(rows, cols, env=env, em_dimensions=self.em_dimensions)
            if resumed_dtach:
                # Send an extra Ctrl-L to refresh the screen and fix the sizing
                # after it has been reattached.
                resize = partial(m.resize, rows, cols, ctrl_l=True,
                                    em_dimensions=self.em_dimensions)
                m.io_loop.add_timeout(
                    timedelta(seconds=2), resize)
        else:
            # Terminal already exists
            if SESSIONS[self.session][term]['multiplex'].isalive():
                # It's ALIVE!!!
                SESSIONS[self.session][term]['multiplex'].resize(
                    rows, cols, ctrl_l=False, em_dimensions=self.em_dimensions)
                message = {'term_exists': term}
                # This resets the screen diff
                SESSIONS[self.session][term]['multiplex'].prev_output[
                    self.client_id] = [None for a in xrange(rows-1)]
                # Remind the client about this terminal's title
                self.set_title(term, force=True)
            else:
                # Tell the client this terminal is no more
                self.term_ended(term)
                return
        # Setup callbacks so that everything gets called when it should
        self.callback_id = callback_id = "%s;%s;%s" % (
            self.client_id, self.request.host, self.request.remote_ip)
        # NOTE: request.host is the FQDN or IP the user entered to open Gate One
        # so if you want to have multiple browsers open to the same user session
        # from the same IP just use an alternate hostname/IP for the URL.
        # Setup the termio callbacks
        refresh = partial(self.refresh_screen, term)
        multiplex = SESSIONS[self.session][term]['multiplex']
        multiplex.add_callback(multiplex.CALLBACK_UPDATE, refresh, callback_id)
        ended = partial(self.term_ended, term)
        multiplex.add_callback(multiplex.CALLBACK_EXIT, ended, callback_id)
        # Setup the terminal emulator callbacks
        term_emulator = multiplex.term
        set_title = partial(self.set_title, term)
        term_emulator.add_callback(
            terminal.CALLBACK_TITLE, set_title, callback_id)
        set_title() # Set initial title
        bell = partial(self.bell, term)
            terminal.CALLBACK_BELL, bell, callback_id)
            terminal.CALLBACK_OPT, self.esc_opt_handler, callback_id)
        mode_handler = partial(self.mode_handler, term)
            terminal.CALLBACK_MODE, mode_handler, callback_id)
        reset_term = partial(self.reset_terminal, term)
            terminal.CALLBACK_RESET, reset_term, callback_id)
        dsr = partial(self.dsr, term)
            terminal.CALLBACK_DSR, dsr, callback_id)
        # Call any registered plugin Terminal hooks
        if PLUGIN_TERM_HOOKS:
            for hook, func in PLUGIN_TERM_HOOKS.items():
                term_emulator.add_callback(hook, func(self))
        # NOTE: refresh_screen will also take care of cleaning things up if
        #       SESSIONS[self.session][term]['multiplex'].isalive() is False
        self.refresh_screen(term, True) # Send a fresh screen to the client
        # Restore application cursor keys mode if set
        if 'application_mode' in SESSIONS[self.session][term]:
            current_setting = SESSIONS[self.session][term]['application_mode']
            self.mode_handler(term, '1', current_setting)
        if self.settings['logging'] == 'debug':
            message = {
                'notice': _(
                    "WARNING: Logging is set to DEBUG.  All keystrokes will be "
                    "logged!")
            }
            self.write_message(message)

    def kill_terminal(self, term):
        """
        Kills *term* and any associated processes.
        """
        logging.debug("killing terminal: %s" % term)
        term = int(term)
            return # Nothing to do
        # Remove the EXIT callback so the terminal doesn't restart itself
        multiplex.remove_callback(multiplex.CALLBACK_EXIT, self.callback_id)
        try:
            if self.settings['dtach']: # dtach needs special love
                kill_dtached_proc(self.session, term)
            if multiplex.isalive():
                multiplex.terminate()
        except KeyError as e:
            pass # The EVIL termio has killed my child!  Wait, that's good...
                 # Because now I don't have to worry about it!
        finally:
            del SESSIONS[self.session][term]

    def set_terminal(self, term):
        """
        Sets `self.current_term = *term*` so we can determine where to send
        keystrokes.
        """
        self.current_term = term

    def reset_terminal(self, term):
        """
        Tells the client to reset the terminal (clear the screen and remove
        scrollback).
        """
        message = {'reset_terminal': term}

    def set_title(self, term, force=False):
        """
        Sends a message to the client telling it to set the window title of
        *term* to whatever comes out of::

            SESSIONS[self.session][term]['multiplex'].term.get_title() # Whew! Say that three times fast!

        Example message::

            {'set_title': {'term': 1, 'title': "user@host"}}

        If *force* resolves to True the title will be sent to the cleint even if
        it matches the previously-set title.

        .. note:: Why the complexity on something as simple as setting the title?  Many prompts set the title.  This means we'd be sending a 'title' message to the client with nearly every screen update which is a pointless waste of bandwidth if the title hasn't changed.
        """
        logging.debug("set_title(%s, %s)" % (term, force))
        if SESSIONS[self.session][term]['manual_title']:
            if force:
                title = SESSIONS[self.session][term]['title']
                title_message = {'set_title': {'term': term, 'title': title}}
                self.write_message(json_encode(title_message))
            return
        title = SESSIONS[self.session][term]['multiplex'].term.get_title()
        # Only send a title update if it actually changed
        if title != SESSIONS[self.session][term]['title'] or force:
            SESSIONS[self.session][term]['title'] = title
            title_message = {'set_title': {'term': term, 'title': title}}
            self.write_message(json_encode(title_message))

    def manual_title(self, settings):
        """
        Sets the title of *settings['term']* to *settings['title']*.  Differs
        from :func:`set_title` in that this is an action that gets called by the
        client when the user sets a terminal title manually.
        """
        logging.debug("manual_title: %s" % settings)
        term = int(settings['term'])
        title = settings['title']
        if not title:
            title = SESSIONS[self.session][term]['multiplex'].term.get_title()
            SESSIONS[self.session][term]['manual_title'] = False
        else:
            SESSIONS[self.session][term]['manual_title'] = True
        SESSIONS[self.session][term]['title'] = title
        title_message = {'set_title': {'term': term, 'title': title}}
        self.write_message(json_encode(title_message))

    def bell(self, term):
        """
        Sends a message to the client indicating that a bell was encountered in
        the given terminal (*term*).  Example message::

            {'bell': {'term': 1}}
        """
        bell_message = {'bell': {'term': term}}
        self.write_message(json_encode(bell_message))

    def mode_handler(self, term, setting, boolean):
        """
        Handles mode settings that require an action on the client by pasing it
        a message like::

            {
                'set_mode': {
                    'mode': setting,
                    'bool': True,
                    'term': term
                }
            }
        """
        logging.debug(
            "mode_handler() term: %s, setting: %s, boolean: %s" %
            (term, setting, boolean))
        if setting in ['1']: # Only support this mode right now
            # So we can restore it:
            SESSIONS[self.session][term]['application_mode'] = boolean
            if boolean:
                # Tell client to enable application cursor mode
                mode_message = {'set_mode': {
                }}
                self.write_message(json_encode(mode_message))
            else:
                # Tell client to disable application cursor mode
                    'bool': False,
                }}

    def dsr(self, term, response):
        """
        Handles Device Status Report (DSR) calls from the underlying program
        that get caught by the terminal emulator.  *response* is what the
        terminal emulator returns from the CALLBACK_DSR callback.

        .. note:: This also handles the CSI DSR sequence.
        """
        SESSIONS[self.session][term]['multiplex'].write(response)

    def _send_refresh(self, term, full=False):
        """Sends a screen update to the client."""
        try:
            SESSIONS[self.session][term]['last_activity'] = datetime.now()
            # This can happen if the user disconnected in the middle of a screen
            # update.  Nothing to be concerned about.
            return # Ignore
        scrollback, screen = multiplex.dump_html(
            full=full, client_id=self.client_id)
        if [a for a in screen if a]:
            output_dict = {
                'termupdate': {
                    'term': term,
                    'scrollback': scrollback,
                    'screen' : screen,
                    'ratelimiter': multiplex.ratelimiter_engaged
                }
            }
            try:
                self.write_message(json_encode(output_dict))
            except IOError: # Socket was just closed, no biggie
                logging.info(
                 _("WebSocket closed (%s)") % self.get_current_user()['upn'])
                multiplex = SESSIONS[self.session][term]['multiplex']
                multiplex.remove_callback( # Stop trying to write
                    multiplex.CALLBACK_UPDATE, self.callback_id)

    def refresh_screen(self, term, full=False):
        """
        Writes the state of the given terminal's screen and scrollback buffer to
        the client using `_send_refresh()`.  Also ensures that screen updates
        don't get sent too fast to the client by instituting a rate limiter that
        also forces a refresh every 150ms.  This keeps things smooth on the
        client side and also reduces the bandwidth used by the application (CPU
        too).

        If *full*, send the whole screen (not just the difference).
        """
        # Commented this out because it was getting annoying.
        # Note to self: add more levels of debugging beyond just "debug".
            #"refresh_screen (full=%s) on %s" % (full, self.callback_id))
        if term:
            term = int(term)
        else:
            return # This just prevents an exception when the cookie is invalid
        try:
            msec = timedelta(milliseconds=50) # Keeps things smooth
            # In testing, 150 milliseconds was about as low as I could go and
            # still remain practical.
            force_refresh_threshold = timedelta(milliseconds=150)
            last_activity = SESSIONS[self.session][term]['last_activity']
            timediff = datetime.now() - last_activity
            sess = SESSIONS[self.session][term]
            client_dict = sess[self.client_id]
            multiplex = sess['multiplex']
            refresh = partial(self._send_refresh, term, full)
            # We impose a rate limit of max one screen update every 50ms by
            # wrapping the call to _send_refresh() in an IOLoop timeout that
            # gets cancelled and replaced if screen updates come in faster than
            # once every 50ms.  If screen updates are consistently faster than
            # that (e.g. a key is held down) we also force sending the screen
            # to the client every 150ms.  This ensures that no matter how fast
            # screen updates are happening the user will get at least one
            # update every 150ms.  It works out quite nice, actually.
            if client_dict['refresh_timeout']:
                multiplex.io_loop.remove_timeout(client_dict['refresh_timeout'])
            if timediff > force_refresh_threshold:
                refresh()
            else:
                client_dict['refresh_timeout'] = multiplex.io_loop.add_timeout(
                    msec, refresh)
        except KeyError as e: # Session died (i.e. command ended).
            logging.debug(_("KeyError in refresh_screen: %s" % e))

    def full_refresh(self, term):
        """Calls `self.refresh_screen(*term*, full=True)`"""
        try:
            logging.debug(_(
                "Invalid terminal number given to full_refresh(): %s" % term))
        self.refresh_screen(term, full=True)

    def resize(self, resize_obj):
        """
        Resize the terminal window to the rows/cols specified in *resize_obj*

        Example *resize_obj*::

            {'rows': 24, 'cols': 80}
        """
        logging.debug("resize(%s)" % repr(resize_obj))
        term = None
        if 'term' in resize_obj:
            term = int(resize_obj['term'])
        self.rows = resize_obj['rows']
        self.cols = resize_obj['cols']
        self.em_dimensions = {
            'height': resize_obj['em_dimensions']['h'],
            'width': resize_obj['em_dimensions']['w']
        }
        ctrl_l = False
        if 'ctrl_l' in resize_obj:
            ctrl_l = resize_obj['ctrl_l']
        if self.rows < 2 or self.cols < 2:
            # Fall back to a standard default:
            self.rows = 24
            self.cols = 80
        # If the user already has a running session, set the new terminal size:
        try:
            if term:
                    self.rows,
                    self.cols,
                    self.em_dimensions,
                    ctrl_l=ctrl_l
                )
            else: # Resize them all
                for term in list(SESSIONS[self.session].keys()):
                    if isinstance(term, int): # Skip the TidyThread
                        SESSIONS[self.session][term]['multiplex'].resize(
                            self.rows,
                            self.cols,
                            self.em_dimensions
                        )
        except KeyError: # Session doesn't exist yet, no biggie
            pass

    def char_handler(self, chars, term=None):
        """
        Writes *chars* (string) to *term*.  If *term* is not provided the
        characters will be sent to the currently-selected terminal.
        """
        #logging.debug("char_handler(%s, %s)" % (repr(chars), repr(term)))
        if not term:
            term = self.current_term
        term = int(term) # Just in case it was sent as a string
        session = self.session
        if session in SESSIONS and term in SESSIONS[session]:
                if chars:
                    SESSIONS[ # Force an update
                        session][term]['multiplex'].ratelimit = time.time()
                    SESSIONS[session][term]['multiplex'].write(chars)

    def write_chars(self, message):
        """
        Writes *message['chars']* to *message['term']*.  If *message['term']*
        is not present, *self.current_term* will be used.
        """
        #logging.debug('write_chars(%s)' % message)
        if 'chars' not in message:
            return # Invalid message
        if 'term' not in message:
            message['term'] = self.current_term
        try:
            self.char_handler(message['chars'], message['term'])
            # Term is closed or invalid
                "Got exception trying to write_chars() to terminal %s"
                % message['term']))
            logging.error(str(e))

    def esc_opt_handler(self, chars):
        """
        Executes whatever function is registered matching the tuple returned by
        :func:`utils.process_opt_esc_sequence`.
        """
        logging.debug("esc_opt_handler(%s)" % repr(chars))
        plugin_name, text = process_opt_esc_sequence(chars)
        if plugin_name:
            try:
                PLUGIN_ESC_HANDLERS[plugin_name](text, tws=self)
                logging.error(_(
                    "Got exception trying to execute plugin's optional ESC "
                    "sequence handler..."))
                logging.error(str(e))

    def get_bell(self):
        """
        Sends the bell sound data to the client in in the form of a data::URI.
        """
        bell_path = os.path.join(GATEONE_DIR, 'static')
        bell_path = os.path.join(bell_path, 'bell.ogg')
        if os.path.exists(bell_path):
            try:
                bell_data_uri = create_data_uri(bell_path)
            except MimeTypeFail:
                bell_data_uri = fallback_bell
        else: # There's always the fallback
            bell_data_uri = fallback_bell
        mimetype = bell_data_uri.split(';')[0].split(':')[1]
        message = {
            'load_bell': {
                'data_uri': bell_data_uri, 'mimetype': mimetype
            }
        }

    def get_webworker(self):
        """
        Sends the text of our go_process.js to the client in order to get around
        the limitations of loading remote Web Worker URLs (for embedding Gate
        One into other apps).
        """
        static_url = os.path.join(GATEONE_DIR, "static")
        webworker_path = os.path.join(static_url, 'go_process.js')
        with open(webworker_path) as f:
            go_process = f.read()
        message = {'load_webworker': go_process}

    def get_style(self, settings):
        """
        Sends the CSS stylesheets matching the properties specified in
        *settings* to the client.  *settings* must contain the following:

            * **container** - The element Gate One resides in (e.g. 'gateone')
            * **prefix** - The string being used to prefix all elements (e.g. 'go\_')

        *settings* may also contain any combination of the following:

            * **theme** - The name of the CSS theme to be retrieved.
            * **colors** - The name of the text color CSS scheme to be retrieved.
            * **plugins** - If true, will send all plugin .css files to the client.
            * **print** - If true, will send the print stylesheet.
        """
        logging.debug('get_style(%s)' % settings)
        out_dict = {'result': 'Success'}
        themes_path = os.path.join(templates_path, 'themes')
        colors_path = os.path.join(templates_path, 'term_colors')
        printing_path = os.path.join(templates_path, 'printing')
        go_url = settings['go_url'] # Used to prefix the url_prefix
        if not go_url.endswith('/'):
            go_url += '/'
        container = settings["container"]
        prefix = settings["prefix"]
        theme = None
        if 'theme' in settings:
            theme = settings["theme"]
        colors = None
        if 'colors' in settings:
            colors = settings["colors"]
        plugins = None
        if 'plugins' in settings:
            plugins = settings["plugins"]
        _print = None
        if 'print' in settings:
            _print = settings["print"]
        if theme:
            theme_path = os.path.join(themes_path, "%s.css" % theme)
            theme_css = self.render_string(
                theme_path,
                colors_256=colors_256,
                url_prefix=go_url
            )
            out_dict['theme'] = theme_css
        if colors:
            color_path = os.path.join(colors_path, "%s.css" % colors)
            colors_css = self.render_string(
                color_path,
            )
            out_dict['colors'] = colors_css
        if plugins:
            # Build a dict of plugins
            out_dict['plugins'] = {}
            plugins_dir = os.path.join(GATEONE_DIR, 'plugins')
            for f in os.listdir(plugins_dir):
                if os.path.isdir(os.path.join(plugins_dir, f)):
                    out_dict['plugins'][f] = ''
            # Add each plugin's CSS template(s) to its respective dict
            for plugin in list(out_dict['plugins'].keys()):
                plugin_templates_path = os.path.join(
                    plugins_dir, plugin, 'templates')
                if os.path.exists(plugin_templates_path):
                    for f in os.listdir(plugin_templates_path):
                        if f.endswith('.css'):
                            plugin_css_path = os.path.join(
                                plugin_templates_path, f)
                            plugin_css = self.render_string(
                                plugin_css_path,
                                container=container,
                                prefix=prefix,
                                url_prefix=go_url
                            )
                            if bytes != str: # Python 3
                                plugin_css = str(plugin_css, 'UTF-8')
                            out_dict['plugins'][plugin] += plugin_css
        if _print:
            print_css_path = os.path.join(printing_path, "default.css")
            print_css = self.render_string(
                print_css_path,
            )
            out_dict['print'] = print_css
        self.write_message(json_encode({'load_style': out_dict}))

    # NOTE: This has been disabled for now.  It works OK but the problem is that
    # some plugin JS needs to load *before* the WebSocket is connected.  Might
    # ultimately be something that just won't work.  I'm still considering
    # ways I could take advantage of this though so I don't want to delete it
    # just yet.
    #def get_js(self):
        #"""
        #Sends all JavaScript files inside of plugins' 'static' directory to the
        #client.
        #"""
        #logging.debug('get_js()')
        #out_dict = {'result': 'Success', 'plugins': {}}
        ## Build a dict of plugins
        #plugins_dir = os.path.join(GATEONE_DIR, 'plugins')
        #for f in os.listdir(plugins_dir):
            #if os.path.isdir(os.path.join(plugins_dir, f)):
                #out_dict['plugins'][f] = {}
        ## Add each found JS file to the respective dict
        #for plugin in out_dict['plugins'].keys():
            #plugin_static_path = os.path.join(plugins_dir, plugin, 'static')
            #if os.path.exists(plugin_static_path):
                #for f in os.listdir(plugin_static_path):
                    #if f.endswith('.js'):
                        #print("f: %s" % f)
                        #plugin_js_path = os.path.join(plugin_static_path, f)
                        #with open(plugin_js_path) as js_file:
                            #plugin_js = js_file.read()
                            #out_dict['plugins'][plugin][f] = plugin_js
        #print("out_dict['plugins']: %s" % out_dict['plugins'])
        #self.write_message({'load_js': out_dict})

    def enumerate_themes(self):
        """
        Returns a JSON-encoded object containing the installed themes and text
        color schemes.
        """
        themes = os.listdir(themes_path)
        themes = [a.replace('.css', '') for a in themes]
        colors = os.listdir(colors_path)
        colors = [a.replace('.css', '') for a in colors]
        message = {'themes_list': {'themes': themes, 'colors': colors}}
        self.write_message(message)

    def debug_terminal(self, term):
        """
        Prints the terminal's screen and renditions to stdout so they can be
        examined more closely.

        .. note:: Can only be called from a JavaScript console like so...

        .. code-block:: javascript

            GateOne.ws.send(JSON.stringify({'debug_terminal': *term*}));
        """
        termObj = SESSIONS[self.session][term]['multiplex'].term
        screen = termObj.screen
        renditions = termObj.renditions
        for i, line in enumerate(screen):
            # This gets rid of images:
            line = [a for a in line if len(a) == 1]
            print("%s:%s" % (i, "".join(line)))
            print(renditions[i])
        # Also check if there's anything that's uncollectable
        import gc
        gc.set_debug(gc.DEBUG_UNCOLLECTABLE|gc.DEBUG_OBJECTS)
        from pprint import pprint
        pprint(gc.garbage)
        print("gc.collect(): %s" % gc.collect())
        print("SESSIONS...")
        pprint(SESSIONS)
        try:
            from pympler import asizeof
            print("screen size: %s" % asizeof.asizeof(screen))
            print("renditions size: %s" % asizeof.asizeof(renditions))
            print("Total term object size: %s" % asizeof.asizeof(termObj))
        except ImportError:
            pass # No biggie

class ErrorHandler(tornado.web.RequestHandler):
    """
    Generates an error response with status_code for all requests.
    """
    def __init__(self, application, request, status_code):
        tornado.web.RequestHandler.__init__(self, application, request)
        self.set_status(status_code)

    def get_error_html(self, status_code, **kwargs):
        self.require_setting("static_url")
        if status_code in [404, 500, 503, 403]:
            filename = os.path.join(self.settings['static_url'], '%d.html' % status_code)
            if os.path.exists(filename):
                f = open(filename, 'r')
                data = f.read()
                f.close()
                return data
        import httplib
        return "<html><title>%(code)d: %(message)s</title>" \
                "<body class='bodyErrorPage'>%(code)d: %(message)s</body></html>" % {
            "code": status_code,
            "message": httplib.responses[status_code],
        }

    def prepare(self):
        raise tornado.web.HTTPError(self._status_code)

class Application(tornado.web.Application):
    def __init__(self, settings):
        """
        Setup our Tornado application...  Everything in *settings* will wind up
        in the Tornado settings dict so as to be accessible under self.settings.
        """
        global PLUGIN_WS_CMDS
        global PLUGIN_HOOKS
        global PLUGIN_ESC_HANDLERS
        global PLUGIN_AUTH_HOOKS
        global PLUGIN_TERM_HOOKS
        # Base settings for our Tornado app
        tornado_settings = dict(
            cookie_secret=settings['cookie_secret'],
            static_url=static_url,
            static_url_prefix="%sstatic/" % settings['url_prefix'],
            gzip=True,
            login_url="%sauth" % settings['url_prefix']
        )
        # Make sure all the provided settings wind up in self.settings
        for k, v in settings.items():
            tornado_settings[k] = v
        # Setup the configured authentication type
        AuthHandler = NullAuthHandler # Default
        if 'auth' in settings and settings['auth']:
            if settings['auth'] == 'kerberos' and KerberosAuthHandler:
                AuthHandler = KerberosAuthHandler
            elif settings['auth'] == 'pam' and PAMAuthHandler:
                AuthHandler = PAMAuthHandler
            elif settings['auth'] == 'google':
                AuthHandler = GoogleAuthHandler
            logging.info(_("Using %s authentication" % settings['auth']))
        else:
            logging.info(_("No authentication method configured. All users will "
                         "be ANONYMOUS"))
        docs_path = os.path.join(GATEONE_DIR, 'docs')
        docs_path = os.path.join(docs_path, 'build')
        docs_path = os.path.join(docs_path, 'html')
        url_prefix = settings['url_prefix']
        if not url_prefix.endswith('/'):
            # Make sure there's a trailing slash
            url_prefix = "%s/" % url_prefix
        # Make the / optional in the regex so it works with the @addslash
        # decorator.  e.g. "/whatever/" would become "/whatever/?"
        index_regex = "%s?" % url_prefix
        # Setup our URL handlers
        handlers = [
            (index_regex, MainHandler),
            # Override the default static handler to ensure the headers are set
            # to allow cross-origin requests.
            (r"%sstatic/(.*)" % url_prefix, StaticHandler, {
                "path": static_url,
            }),
            (r"%sws" % url_prefix, TerminalWebSocket),
            (r"%sauth" % url_prefix, AuthHandler),
            (r"%scssrender" % url_prefix, PluginCSSTemplateHandler),
            (r"%scombined_js" % url_prefix, JSPluginsHandler),
            (r"%sdocs/(.*)" % url_prefix, tornado.web.StaticFileHandler, {
                "path": docs_path,
                "default_filename": "index.html"
            })
        ]
        # Hook up the hooks
        for plugin_name, hooks in PLUGIN_HOOKS.items():
            if 'Web' in hooks:
                # Apply the plugin's Web handlers
                fixed_hooks = []
                if isinstance(hooks['Web'], (list, tuple)):
                    for h in hooks['Web']:
                        # h == (regex, Handler)
                        if not h[0].startswith(url_prefix): # Fix it
                            h = (url_prefix + h[0].lstrip('/'), h[1])
                            fixed_hooks.append(h)
                        else:
                else:
                    if not hooks['Web'][0].startswith(url_prefix): # Fix it
                        hooks['Web'] = (
                            url_prefix + hooks['Web'][0].lstrip('/'),
                            hooks['Web'][1]
                        )
                        fixed_hooks.append(hooks['Web'])
                    else:
                handlers.extend(fixed_hooks)
            if 'WebSocket' in hooks:
                # Apply the plugin's WebSocket commands
                PLUGIN_WS_CMDS.update(hooks['WebSocket'])
            if 'Escape' in hooks:
                # Apply the plugin's Escape handler
                PLUGIN_ESC_HANDLERS.update({plugin_name: hooks['Escape']})
            if 'Auth' in hooks:
                # Apply the plugin's post-authentication functions
                if isinstance(hooks['Auth'], list):
                    PLUGIN_AUTH_HOOKS.extend(hooks['Auth'])
                else:
                    PLUGIN_AUTH_HOOKS.append(hooks['Auth'])
            if 'Terminal' in hooks:
                # Apply the plugin's Terminal hooks (called by new_terminal)
                PLUGIN_TERM_HOOKS.update(hooks['Terminal'])
        # This removes duplicate handlers for the same regex, allowing plugins
        # to override defaults:
        handlers = merge_handlers(handlers)
        # Include JS-only and CSS-only plugins (for logging purposes)
        js_plugins = [a.split('/')[2] for a in PLUGINS['js']]
        css_plugins = []
        for i in css_plugins:
            if '?' in i: # CSS Template
                css_plugins.append(i.split('plugin=')[1].split('&')[0])
            else: # Static CSS file
                css_plugins.append(i.split('/')[1])
        #css_plugins = [a.split('?')[1].split('&')[0].split('=')[1] for a in PLUGINS['css']]
        plugin_list = list(set(PLUGINS['py'] + js_plugins + css_plugins))
        plugin_list.sort() # So there's consistent ordering
        logging.info(_("Loaded plugins: %s" % ", ".join(plugin_list)))
        tornado.web.Application.__init__(self, handlers, **tornado_settings)

def main():
    global _
    global user_locale
    # Default to using the shell's LANG variable as the locale
    try:
        default_locale = os.environ['LANG'].split('.')[0]
    except KeyError: # $LANG isn't set
        default_locale = "en_US"
    user_locale = locale.get(default_locale)
    # NOTE: The locale setting above is only for the --help messages.
    # Simplify the auth option help message
    auths = "none, api, google"
    if KerberosAuthHandler:
        auths += ", kerberos"
    if PAMAuthHandler:
        auths += ", pam"
    # Simplify the syslog_facility option help message
    facilities = list(FACILITIES.keys())
    facilities.sort()
    # Figure out the default origins
    default_origins = [
        'http://localhost',
        'https://localhost',
        'http://127.0.0.1',
        'https://127.0.0.1'
    ]
    # Used both http and https above to demonstrate that both are acceptable
    try:
        additional_origins = socket.gethostbyname_ex(socket.gethostname())
    except socket.gaierror:
        # Couldn't get any IPs from the hostname
        additional_origins = []
    for host in additional_origins:
        if isinstance(host, str):
            default_origins.append('https://%s' % host)
        else: # It's a list
            for _host in host:
                default_origins.append('https://%s' % _host)
    default_origins = ";".join(default_origins)
    config_default = os.path.join(GATEONE_DIR, "server.conf")
    # TODO:  These configuration options are getting a bit unwiedly.  Move them into a separate file or something.  Might want to switch over to using optparse and/or ConfigParser as well.
    define("config",
        default=config_default,
        help=_(
            "Path to the config file.  Default: %s" % config_default),
        type=str
    )
    define(
        "debug",
        help=_("Enable debugging features such as auto-restarting when files "
               "are modified.")
    )
    define("cookie_secret", # 45 chars is, "Good enough for me" (cookie joke =)
        default=None,
        help=_("Use the given 45-character string for cookie encryption."),
        type=str
    )
    define("command",
        # The default command assumes the SSH plugin is enabled
        default=GATEONE_DIR + "/plugins/ssh/scripts/ssh_connect.py -S "
                r"'/tmp/gateone/%SESSION%/%SHORT_SOCKET%' --sshfp "
                "-a '-oUserKnownHostsFile=%USERDIR%/%USER%/ssh/known_hosts'",
        help=_("Run the given command when a user connects (e.g. '/bin/login')."
               ),
        type=str
    )
    define("address",
        default="",
        help=_("Run on the given address.  Default is all addresses (IPv6 "
               "included).  Multiple address can be specified using a semicolon"
               " as a separator (e.g. '127.0.0.1;::1;10.1.1.100')."),
        type=str)
    define("port", default=443, help=_("Run on the given port."), type=int)
    define(
        "enable_unix_socket",
        help=_("Enable Unix socket support."),
        type=bool)
    define(
        "unix_socket_path",
        default="/var/run/gateone.sock",
        help=_("Path to the Unix socket (if --enable_unix_socket=True)."),
        type=str)
    # Please only use this if Gate One is running behind something with SSL:
    define(
        "disable_ssl",
        help=_("If enabled, Gate One will run without SSL (generally not a "
               "good idea).")
    )
    define(
        "certificate",
        default="certificate.pem",
        help=_("Path to the SSL certificate.  Will be auto-generated if none is"
               " provided."),
        type=str
    )
    define(
        "keyfile",
        default="keyfile.pem",
        help=_("Path to the SSL keyfile.  Will be auto-generated if none is"
        type=str
    )
    define(
        "ca_certs",
        help=_("Path to a file containing any number of concatenated CA "
               "certificates in PEM format.  They will be used to authenticate "
               "clients if the 'ssl_auth' option is set to 'optional' or "
               "'required'."),
        type=str
    )
    define(
        "ssl_auth",
        default='none',
        help=_("Enable the use of client SSL (X.509) certificates as a "
               "secondary authentication factor (the configured 'auth' type "
               "will come after SSL auth).  May be one of 'none', 'optional', "
               "or 'required'.  NOTE: Only works if the 'ca_certs' option is "
               "configured."),
        type=str
    )
    define(
        "user_dir",
        default=os.path.join(GATEONE_DIR, "users"),
        help=_("Path to the location where user files will be stored."),
        type=str
    )
    define(
        "session_dir",
        default="/tmp/gateone",
        help=_("Path to the location where session information will be stored."),
        type=str
    )
    define(
        "session_logging",
        help=_("If enabled, logs of user sessions will be saved in "
               "<user_dir>/<user>/logs.  Default: Enabled")
    )
    define( # This is an easy way to support cetralized logging
        "syslog_session_logging",
        help=_("If enabled, logs of user sessions will be written to syslog.")
    )
    define(
        "syslog_facility",
        default="daemon",
        help=_("Syslog facility to use when logging to syslog (if "
               "syslog_session_logging is enabled).  Must be one of: %s."
               "  Default: daemon" % ", ".join(facilities)),
        type=str
    )
    define(
        "syslog_host",
        help=_("Remote host to send syslog messages to if syslog_logging is "
               "enabled.  Default: None (log to the local syslog daemon "
               "directly).  NOTE:  This setting is required on platforms that "
               "don't include Python's syslog module."),
        type=str
    )
    define(
        "session_timeout",
        default="5d",
        help=_("Amount of time that a session is allowed to idle before it is "
        "killed.  Accepts <num>X where X could be one of s, m, h, or d for "
        "seconds, minutes, hours, and days.  Default is '5d' (5 days)."),
        type=str
    )
    define(
        "new_api_key",
        help=_("Generate a new API key that an external application can use to "
               "embed Gate One."),
    )
    define(
        "auth",
        default="none",
        help=_("Authentication method to use.  Valid options are: %s" % auths),
        type=str
    )
    # This is to prevent replay attacks.  Gate One only keeps a "working memory"
    # of API auth objects for this amount of time.  So if the Gate One server is
    # restarted we don't have to write them to disk as anything older than this
    # setting will be invalid (no need to check if it has already been used).
    define(
        "api_timestamp_window",
        default="30s", # 30 seconds
        help=_(
            "How long before an API authentication object becomes invalid.  "
            "Default is '30s' (30 seconds)."),
        type=str
    )
    define(
        "sso_realm",
        help=_("Kerberos REALM (aka DOMAIN) to use when authenticating clients."
               " Only relevant if Kerberos authentication is enabled."),
        type=str
    )
    define(
        "sso_service",
        default='HTTP',
        help=_("Kerberos service (aka application) to use. Defaults to HTTP. "
               "Only relevant if Kerberos authentication is enabled."),
        type=str
    )
    define(
        "pam_realm",
        default=os.uname()[1],
        help=_("Basic auth REALM to display when authenticating clients.  "
        "Default: hostname.  "
        "Only relevant if PAM authentication is enabled."),
        # NOTE: This is only used to show the user a REALM at the basic auth
        #       prompt and as the name in the GATEONE_DIR+'/users' directory
        type=str
    )
    define(
        "pam_service",
        default='login',
        help=_("PAM service to use.  Defaults to 'login'. "
               "Only relevant if PAM authentication is enabled."),
        type=str
    )
    define(
        "embedded",
        help=_("Doesn't do anything (yet).")
    )
    define(
        "dtach",
        help=_("Wrap terminals with dtach. Allows sessions to be resumed even "
               "if Gate One is stopped and started (which is a sweet feature).")
    )
    define(
        "kill",
        help=_("Kill any running Gate One terminal processes including dtach'd "
               "processes.")
    )
    define(
        "locale",
        default=default_locale,
        help=_("The locale (e.g. pt_PT) Gate One should use for translations."
             "  If not provided, will default to $LANG (which is '%s' in your "
             "current shell), or en_US if not set."
             % os.environ.get('LANG', 'not set').split('.')[0]),
        type=str
    )
    define("js_init",
        help=_("A JavaScript object (string) that will be used when running "
               "GateOne.init() inside index.html.  "
               "Example: --js_init=\"{scheme: 'white'}\" would result in "
               "GateOne.init({scheme: 'white'})"),
        type=str
    )
    define(
        "https_redirect",
        help=_("If enabled, a separate listener will be started on port 80 that"
               " redirects users to the configured port using HTTPS.")
    )
    define(
        "url_prefix",
        default="/",
        help=_("An optional prefix to place before all Gate One URLs. e.g. "
               "'/gateone/'.  Use this if Gate One will be running behind a "
               "reverse proxy where you want it to be located at some sub-"
               "URL path."),
        type=str
    )
    define(
        "origins",
        default=default_origins,
        help=_("A semicolon-separated list of origins you wish to allow access "
               "to your Gate One server over the WebSocket.  This value must "
               "contain the hostnames and FQDNs (e.g. https://foo;"
               "https://foo.bar;) users will use to connect to your Gate One "
               "server as well as the hostnames/FQDNs of any sites that will be"
               " embedding Gate One. Here's the default on your system: '%s'. "
               "Alternatively, '*' may be  specified to allow access from "
               "anywhere." % default_origins),
        type=str
    )
    define(
        "pid_file",
        default="/var/run/gateone.pid",
        help=_(
            "Define the path to the pid file.  Default: /var/run/gateone.pid"),
        type=str
    )
    define(
        "uid",
        default=str(os.getuid()),
        help=_(
            "Drop privileges and run Gate One as this user/uid."),
        type=str
    )
    define(
        "gid",
        default=str(os.getgid()),
        help=_(
            "Drop privileges and run Gate One as this group/gid."),
        type=str
    )
    define(
        "session_logs_max_age",
        default="30d",
        help=_("Maximum amount of length of time to keep any given session log "
               "before it is removed."),
        type=str
    )
    # Before we do anythong else, load plugins and assign their hooks.  This
    # allows plugins to add their own define() statements/options.
    imported = load_plugins(PLUGINS['py'])
    new_conf = False # Used below to tell the user that a server.conf was
                     # generated in their chosen language.
    for plugin in imported:
        try:
            PLUGIN_HOOKS.update({plugin.__name__: plugin.hooks})
            pass # No hooks--probably just a supporting .py file.
    # We have to parse the command line options so we can check for a config to
    # load.  The config will then be loaded--overriding command line options but
    # this won't be a problem since we'll re-parse the command line options
    # further down which will subsequently override the config.
    tornado.options.parse_command_line()
    if os.path.exists(options.config):
        tornado.options.parse_config_file(options.config)
    # If you want any missing config file entries re-generated just delete the
    # cookie_secret line...
    if not options.cookie_secret:
        # Generate a default server.conf with a random cookie secret
        # NOTE: This will also generate a new server.conf if it is missing.
          "%s not found or missing cookie_secret.  A new one will be generated."
          % options.config))
        config_defaults = {}
        for key, value in options.items():
            if value._value != None:
                config_defaults.update({key: value._value})
            else:
                config_defaults.update({key: value.default})
        # A few config defaults need special handling
        del config_defaults['kill'] # This shouldn't be in server.conf
        del config_defaults['help'] # Neither should this
        del config_defaults['new_api_key'] # Ditto
        del config_defaults['config'] # Re-ditto
        config_defaults.update({'cookie_secret': generate_session_id()})
        # NOTE: The next four options are specific to the Tornado framework
        config_defaults.update({'log_file_max_size': 100 * 1024 * 1024}) # 100MB
        config_defaults.update({'log_file_num_backups': 10})
        config_defaults.update({'log_to_stderr': False})
        if options.log_file_prefix == None:
            web_log_path = os.path.join(GATEONE_DIR, 'logs')
            config_defaults.update({
                'log_file_prefix': os.path.join(web_log_path, 'webserver.log')})
        else:
            web_log_path = os.path.split(options.log_file_prefix)[0]
        if not os.path.exists(web_log_path):
            mkdir_p(web_log_path)
        if not os.path.exists(config_defaults['log_file_prefix']):
            open(config_defaults['log_file_prefix'], 'w').write('')
        config = open(options.config, "w")
        config.write('# -*- coding: utf-8 -*-\n') # Start with the encoding line
        for key, value in config_defaults.items():
            if isinstance(value, basestring):
                config.write('%s = "%s"\n' % (key, value))
            else:
                config.write('%s = %s\n' % (key, value))
        config.close()
    # Change the uid/gid strings into integers
    try:
        uid = int(options.uid)
        # Assume it's a username and grab its uid
        uid = pwd.getpwnam(options.uid).pw_uid
    try:
        gid = int(options.gid)
        # Assume it's a group name and grab its gid
        gid = grp.getgrnam(options.gid).gr_gid
    if not os.path.exists(options.user_dir): # Make our user_dir
        try:
            mkdir_p(options.user_dir)
        except OSError:
                "Error: Gate One could not create %s.  Please ensure that user,"
                " %s has permission to create this directory or create it "
                "yourself and make user, %s its owner." % (options.user_dir,
                repr(os.getlogin()), repr(os.getlogin()))))
            sys.exit(1)
        # If we could create it we should be able to adjust its permissions:
        os.chmod(options.user_dir, 0o770)
    if os.stat(options.user_dir).st_uid != uid:
        # Try correcting this first
        try:
            recursive_chown(options.user_dir, uid, gid)
        except ChownError as e:
    if not os.path.exists(options.session_dir): # Make our session_dir
        try:
            mkdir_p(options.session_dir)
                "yourself and make user, %s its owner." % (options.session_dir,
        os.chmod(options.session_dir, 0o770)
    if os.stat(options.session_dir).st_uid != uid:
        # Try correcting it
        try:
            recursive_chown(options.session_dir, uid, gid)
    # Re-do the locale in case the user supplied something as --locale
    user_locale = locale.get(options.locale)
    _ = user_locale.translate # Also replaces our wrapper so no more .encode()
    # Create the log dir if not already present (NOTE: Assumes we're root)
    log_dir = os.path.split(options.log_file_prefix)[0]
    if not os.path.exists(log_dir):
        try:
            mkdir_p(log_dir)
            logging.error(_("\x1b[1;31mERROR:\x1b[0m Could not create %s for "
                  "log_file_prefix: %s" % (log_dir, options.log_file_prefix)))
            logging.error(_("You probably want to change this option, run Gate "
                  "One as root, or create that directory and give the proper "
                  "user ownership of it."))
    if os.stat(log_dir).st_uid != uid:
        # Try to correct it
        try:
            recursive_chown(log_dir, uid, gid)
    if options.kill:
        # Kill all running dtach sessions (associated with Gate One anyway)
        killall(options.session_dir)
        sys.exit(0)
    if options.new_api_key:
        # Generate a new API key for an application to use
        api_key = generate_session_id()
        # Generate a new secret
        secret = generate_session_id()
        # Save it
        server_conf = ""
        with open(options.config) as f:
            existing = ""
            for line in f.readlines():
                if line.startswith("api_keys"):
                    existing = line.split('=')[1].strip().strip('"').strip("'")
                    existing += ",%s:%s" % (api_key, secret)
                    line = 'api_keys = "%s"\n' % existing
                server_conf += line
            if not existing:
                server_conf += 'api_keys = "%s:%s"\n' % (api_key, secret)
        open(os.path.join(options.config), 'w').write(server_conf)
        logging.info(_("A new API key has been generated: %s" % api_key))
        logging.info(_("This key can now be used to embed Gate One into other "
                "applications."))
    # Set our CMD variable to tell the multiplexer which command to execute
    global CMD
    CMD = options.command
    # Set our global session timeout
    global TIMEOUT
    TIMEOUT = convert_to_timedelta(options.session_timeout)
    # Make sure dtach is available and if not, set dtach=False
    if not which('dtach'):
        logging.warning(
            _("dtach command not found.  dtach support has been disabled."))
        options.dtach = False
    # Turn our API keys into a dict
    api_keys = {}
    with open(options.config) as f:
        for line in f.readlines():
            if line.startswith("api_keys"):
                values = line.split('=')[1].strip().strip('"').strip("'")
                pairs = values.split(',')
                for pair in pairs:
                    api_key, secret = pair.split(':')
                    if bytes == str:
                        api_key = api_key.decode('UTF-8')
                        secret = secret.decode('UTF-8')
                    api_keys.update({api_key: secret})
    # Fix the url_prefix if the user forgot the trailing slash
    if not options.url_prefix.endswith('/'):
        options.url_prefix += '/'
    # Convert the origins into a list of http:// or https:// origins
    origins = options.origins.lower() # Origins are case-insensitive
    real_origins = origins.split(';')
    if options.origins == '*':
        real_origins = ['*']
    logging.info("Connections to this server will be allowed from the following"
                 " origins: '%s'" % " ".join(real_origins))
    # Define our Application settings
    api_timestamp_window = convert_to_timedelta(options.api_timestamp_window)
    app_settings = {
        'gateone_dir': GATEONE_DIR, # Only here so plugins can reference it
        'debug': options.debug,
        'cookie_secret': options.cookie_secret,
        'auth': none_fix(options.auth),
        'api_timestamp_window': api_timestamp_window,
        'embedded': options.embedded,
        'js_init': options.js_init,
        'user_dir': options.user_dir,
        'logging': options.logging, # For reference, really
        'session_dir': options.session_dir,
        'session_logging': options.session_logging,
        'syslog_session_logging': options.syslog_session_logging,
        'syslog_facility': options.syslog_facility,
        'syslog_host': options.syslog_host,
        'dtach': options.dtach,
        'sso_realm': options.sso_realm,
        'sso_service': options.sso_service,
        'pam_realm': options.pam_realm,
        'pam_service': options.pam_service,
        'locale': options.locale,
        'api_keys': api_keys,
        'url_prefix': options.url_prefix,
        'origins': real_origins,
        'pid_file': options.pid_file
    }
    # Check to make sure we have a certificate and keyfile and generate fresh
    # ones if not.
    if options.keyfile == "keyfile.pem":
        # If set to the default we'll assume they want to use the one in the
        # gateone_dir
        options.keyfile = "%s/keyfile.pem" % GATEONE_DIR
    if options.certificate == "certificate.pem":
        # Just like the keyfile, assume they want to use the one in the
        options.certificate = "%s/certificate.pem" % GATEONE_DIR
    if not os.path.exists(options.keyfile):
        logging.info(_("No SSL private key found.  One will be generated."))
        gen_self_signed_ssl(path=GATEONE_DIR)
    if not os.path.exists(options.certificate):
        logging.info(_("No SSL certificate found.  One will be generated."))
    # Setup static file links for plugins (if any)
    static_dir = os.path.join(GATEONE_DIR, "static")
    # Verify static_dir's permissions
    if os.stat(static_dir).st_uid != uid:
        try: # Just os.chown on this one (recursive could be bad)
            os.chown(static_dir, uid, gid)
                "Error: Gate One does not have permission to write to %s.  "
                "Please ensure that user, %s has write permission to the "
                "directory." % (
                static_dir, os.getlogin())))
    plugin_dir = os.path.join(GATEONE_DIR, "plugins")
    templates_dir = os.path.join(GATEONE_DIR, "templates")
    combined_plugins = os.path.join(static_dir, "combined_plugins.js")
    # Remove the combined_plugins.js (it will get auto-recreated)
    if os.path.exists(combined_plugins):
        os.remove(combined_plugins)
    create_plugin_links(static_dir, templates_dir, plugin_dir)
    # When options.logging=="debug" it will display all user's keystrokes so
    # make sure we warn about this.
    if options.logging == "debug":
        logging.warning(_(
            "Logging is set to DEBUG.  Be aware that this will record the "
            "keystrokes of all users.  Don't be evil!"))
    if options.ssl_auth.lower() == 'required':
        # Convert to an integer using the ssl module
        cert_reqs = ssl.CERT_REQUIRED
    elif options.ssl_auth.lower() == 'optional':
        cert_reqs = ssl.CERT_OPTIONAL
    else:
        cert_reqs = ssl.CERT_NONE
    # Instantiate our Tornado web server
    ssl_options = {
        "certfile": options.certificate,
        "keyfile": options.keyfile,
        "ca_certs": options.ca_certs,
        "cert_reqs": cert_reqs
    }
    if options.disable_ssl:
        proto = "http://"
        ssl_options = None
    else:
        proto = "https://"
    https_server = tornado.httpserver.HTTPServer(
        Application(settings=app_settings), ssl_options=ssl_options)
    https_redirect = tornado.web.Application(
        [(r".*", HTTPSRedirectHandler),],
        port=options.port,
        url_prefix=options.url_prefix
    )
    tornado.web.ErrorHandler = ErrorHandler
    try: # Start your engines!
        if options.enable_unix_socket:
            https_server.add_socket(
                tornado.netutil.bind_unix_socket(options.unix_socket_path))
            logging.info(_("Listening on Unix socket '{socketpath}'".format(
                socketpath=options.unix_socket_path)))
        address = none_fix(options.address)
        if address:
            for addr in address.split(';'):
                if addr: # Listen on all given addresses
                    if options.https_redirect:
                        if options.disable_ssl:
                            "You have https_redirect *and* disable_ssl enabled."
                            "  Please pick one or the other."))
                            sys.exit(1)
                        logging.info(_(
                            "http://{addr}:80/ will be redirected to...".format(
                                addr=addr)
                        ))
                        https_redirect.listen(port=80, address=addr)
                    logging.info(_(
                        "Listening on {proto}{address}:{port}/".format(
                            proto=proto, address=addr, port=options.port)
                    ))
                    https_server.listen(port=options.port, address=addr)
        elif address == '':
            # Listen on all addresses (including IPv6)
            if options.https_redirect:
                if options.disable_ssl:
                        "You have https_redirect *and* disable_ssl enabled."
                        "  Please pick one or the other."))
                    sys.exit(1)
                logging.info(_("http://*:80/ will be redirected to..."))
                https_redirect.listen(port=80, address="")
            logging.info(_(
                "Listening on {proto}*:{port}/".format(
                    proto=proto, port=options.port)))
            https_server.listen(port=options.port, address="")
        # NOTE:  To have Gate One *not* listen on a TCP/IP address you may set
        #        address=None
        write_pid(options.pid_file)
        pid = read_pid(options.pid_file)
        logging.info(_("Process running with pid " + pid))
        # Check to see what group owns /dev/pts and use that for supl_groups
        # First we have to make sure there's at least one pty present
        tempfd1, tempfd2 = pty.openpty()
        # Now check the owning group (doesn't matter which one so we use 0)
        tty_gid = os.stat('/dev/ptmx').st_gid
        # Close our temmporary pty/fds so we're not wasting them
        os.close(tempfd1)
        os.close(tempfd2)
        if uid != os.getuid():
            drop_privileges(uid, gid, [tty_gid])
        # Make sure that old logs get cleaned up
        global CLEANER
        if not CLEANER:
            interval = 5*60*1000 # Check every 5 minutes
            max_age = convert_to_timedelta(options.session_logs_max_age)
            cleaner = partial(
                cleanup_session_logs,
                options.user_dir,
                max_age)
            CLEANER = tornado.ioloop.PeriodicCallback(cleaner, interval)
            CLEANER.start()
        tornado.ioloop.IOLoop.instance().start()
    except KeyboardInterrupt: # ctrl-c
        logging.info(_("Caught KeyboardInterrupt.  Killing sessions..."))
    finally:
        tornado.ioloop.IOLoop.instance().stop()
        remove_pid(options.pid_file)
        logging.info(_("pid file removed."))
        if not options.dtach:
            # If we're not using dtach play it safe by cleaning up any leftover
            # processes.  When passwords are used with the ssh_conenct.py script
            # it runs os.setsid() on the child process which means it won't die
            # when Gate One is closed.  This is primarily to handle that
            # specific situation.
            killall(options.session_dir)
            # Cleanup the session_dir (it is supposed to only contain temp stuff)
            import shutil
            shutil.rmtree(options.session_dir, ignore_errors=True)

    main()

#
#

# Meta

About The SSO Module
====================
sso.py is a Tornado Single Sign-On (SSO) authentication module that implements
GSSAPI authentication via python-kerberos (import kerberos).  If "Negotiate"
authentication (GSSAPI SSO) fails it will gracefully fall back to "Basic" auth
(authenticating a given username/password against your Kerberos realm).

For this module to work you must add 'sso_realm' and 'sso_service' to your
Tornado application's settings.  See the docstring of the KerberosAuthMixin for
how to do this.

This module should work with regular MIT Kerberos implementations as well as
Active Directory (Heimdal is untested but should work fine).  If you're
experiencing trouble it is recommended that you set debug=True in your
application settings.  This will enable printing of Kerberos exception messages.

Troubleshooting
---------------

If your browser asks you for a password (i.e. SSO failed) there's probably
something wrong with your Kerberos configuration on either the client or the
server (usually it's a problem with forward/reverse DNS resolution or an
incorrect or missing service principal in your keytab).

If you're using Active Directory, make sure that there's an HTTP
servicePrincipalName (SPN) matching the FQDN of the host running your Tornado
server.  For example:  HTTP/somehost.somedomain.com@CORP.MYCOMPANY.COM
You may also want a short hostname SPN: HTTP/somehost@CORP.MYCOMPANY.COM

Also make sure that the service principal is in upper case as most clients (
web browsers) will auto-capitalize the principal when verifying the server.

Here's some things to test in order to find problems with your Kerberos config:

Try these from both the client and the server (NOTE: Assuming both are Unix):
kinit -p <user@REALM> # To verify you can authenticate via Kerberos (at all)
nslookup <server FQDN> # To verify the IP address reverse maps properly (below)
nslookup <IP address that 'server FQDN' resolves to>
kvno HTTP/somehost.somedomain.com # To verify your service principal

Remember: Kerberos is heavily dependent on DNS to verify the server and client
are who they claim to be.

I find that it is useful to get GSSAPI authentication working with OpenSSH first
before I attempt to get a custom service principal working with other
applications.  This is because SSH uses the HOST/ prinicipal which is often
taken care of automatically via most Kerberos management tools (including AD).
If you can get SSO working with SSH you can get SSO working with anything else.

"""

# Enable localization support

import kerberos

# NOTE: For some reason if I set this as just an 'object' it doesn't work.
class KerberosAuthMixin(tornado.web.RequestHandler):
    """
    Authenticates users via Kerberos-based Single Sign-On.  Requires that you
    define 'sso_realm' and 'sso_service' in your Tornado Application settings.
    For example::

        settings = dict(
            cookie_secret="iYR123qg4UUdsgf4CRung6BFUBhizAciid8oq1YfJR3gN",
            static_path=os.path.join(os.path.dirname(__file__), "static"),
            login_url="/auth",
            debug=True,
            sso_realm="EXAMPLE.COM",
            sso_service="HTTP" # Should pretty much always be HTTP
        )

    NOTE: If you're using 'HTTP' as the service it must be in all caps or it
    might not work with some browsers/clients (which auto-capitalize all
    services).

    To implement this mixin::

        from sso import KerberosAuthMixin
        class KerberosAuthHandler(tornado.web.RequestHandler, KerberosAuthMixin):

            def get(self):
                auth_header = self.request.headers.get('Authorization')
                if auth_header:
                    return
                self.authenticate_redirect()

            def _on_auth(self, user):
                    raise tornado.web.HTTPError(500, "Kerberos auth failed")
                self.set_secure_cookie("user", tornado.escape.json_encode(user))
                print("KerberosAuthHandler user: %s" % user) # To see what you get
                next_url = self.get_argument("next", None) # To redirect properly
                if next_url:
                    self.redirect(next_url)
                else:
                    self.redirect("/")
    """
        """

        NOTE: It won't hurt anything to override this method in your
        RequestHandler.
        """
        self.require_setting("sso_realm", _("Kerberos/GSSAPI Single Sign-On"))
        self.require_setting("sso_service", _("Kerberos/GSSAPI Single Sign-On"))

        """
        Processes the client's Authorization header and calls
        self.auth_negotiate() or self.auth_basic() depending on what headers
        were provided by the client.
        """
        if auth_header and auth_header.startswith('Negotiate'):
            self.auth_negotiate(auth_header, callback)
        elif auth_header and auth_header.startswith('Basic '):

    def auth_negotiate(self, auth_header, callback):
        """
        Perform Negotiate (GSSAPI/SSO) authentication via Kerberos.
        """
        auth_str = auth_header.split()[1]
        # Initialize Kerberos Context
        result, context = kerberos.authGSSServerInit(
            self.settings['sso_service'])
        if result != 1:
            raise tornado.web.HTTPError(500, _("Kerberos Init failed"))
        result = kerberos.authGSSServerStep(context, auth_str)
        if result == 1:
            gssstring = kerberos.authGSSServerResponse(context)
        else: # Fall back to Basic auth
        # NOTE: The user we get from Negotiate is a full UPN (e.g. user@REALM)
        user = kerberos.authGSSServerUserName(context)
        self.set_header('WWW-Authenticate', "Negotiate %s" % gssstring)
        kerberos.authGSSServerClean(context)

        """
        Perform Basic authentication using Kerberos against
        self.settings['sso_realm'].
        """
        try:
            kerberos.checkPassword(
                username,
                password,
                self.settings['sso_service'],
                self.settings['sso_realm'])
        user = "%s@%s" % (username, self.settings['sso_realm'])

        """

        The browser will be informed that we support both Negotiate (GSSAPI/SSO)
        and Basic auth.

        NOTE: I know this isn't technically a redirect but I wanted to make this
        process as close as possible to how things work in tornado.auth.
        """
            raise Exception(_('Headers have already been written'))
        self.add_header("WWW-Authenticate", "Negotiate")
            'Basic realm="%s"' % self.settings['sso_realm']
        )

#
#

.. note:: Yes this only contains one function and it is exposed to clients through a WebSocket hook.

Hooks
-----

    hooks = {
            'playback_save_recording': save_recording,
        }
    }

"""

# Meta

import os

# Globals
plugin_path = os.path.split(__file__)[0]

def save_recording(settings, tws):
    """
    Handles uploads of session recordings and returns them to the client in a
    self-contained HTML file that will auto-start playback.

    NOTE: The real crux of the code that handles this is in the template.
    """
    from gateone import COLORS_256
    from datetime import datetime
    now = datetime.now().strftime('%Y%m%d%H%m%S') # e.g. '20120208200222'
        'result': 'Success',
        'filename': 'GateOne_recording-%s.html' % now,
        'data': None,
        'mimetype': 'text/html'
    }
    recording = settings["recording"]
    container = settings["container"]
    prefix = settings["prefix"]
    theme = settings["theme"]
    colors = settings["colors"]
    gateone_dir = tws.settings['gateone_dir']
    #playback_plugin_path = os.path.join(plugins_path, 'playback')
    template_path = os.path.join(gateone_dir, 'templates')
    colors_templates_path = os.path.join(template_path, 'term_colors')
    colors_css_path = os.path.join(colors_templates_path, '%s.css' % colors)
    with open(colors_css_path) as f:
        colors_file = f.read()
    themes_templates_path = os.path.join(template_path, 'themes')
    theme_css_path = os.path.join(themes_templates_path, '%s.css' % theme)
    with open(theme_css_path) as f:
        theme_file = f.read()
    colors = tornado.template.Template(colors_file)
    rendered_colors = colors.generate(container=container, prefix=prefix)
    theme = tornado.template.Template(theme_file)
    # Setup our 256-color support CSS:
    colors_256 = ""
    for i in xrange(256):
        fg = "#%s span.fx%s {color: #%s;}" % (
            container, i, COLORS_256[i])
        bg = "#%s span.bx%s {background-color: #%s;} " % (
        colors_256 += "%s %s" % (fg, bg)
    colors_256 += "\n"
    rendered_theme = theme.generate(
        container=container,
        prefix=prefix,
        colors_256=colors_256,
        url_prefix=tws.settings['url_prefix']
    )
    templates_path = os.path.join(plugin_path, "templates")
    recording_template_path = os.path.join(
        templates_path, "self_contained_recording.html")
    with open(recording_template_path) as f:
        recording_template_data = f.read()
    recording_template = tornado.template.Template(recording_template_data)
    rendered_recording = recording_template.generate(
        recording=recording,
        theme=rendered_theme,
        colors=rendered_colors
    )
    out_dict['data'] = rendered_recording

hooks = {
    }
}

#
#

Hooks
-----

    hooks = {
        'Web': [(r"/ssh", KnownHostsHandler)],
            'ssh_get_connect_string': get_connect_string,
            'ssh_execute_command': ws_exec_command,
            'ssh_get_identities': get_identities,
            'ssh_get_public_key': get_public_key,
            'ssh_get_private_key': get_private_key,
            'ssh_get_host_fingerprint': get_host_fingerprint,
            'ssh_gen_new_keypair': generate_new_keypair,
            'ssh_store_id_file': store_id_file,
            'ssh_delete_identity': delete_identity,
            'ssh_set_default_identities': set_default_identities
        },
        'Escape': opt_esc_handler,
        'Auth': create_user_ssh_dir
    }

"""

# Meta

import os, sys, logging, re, time
from functools import partial

# Our stuff
from gateone import BaseHandler
from utils import get_translation, mkdir_p, shell_command, which, json_encode
from utils import noop

# Globals
OPENSSH_VERSION = None
DROPBEAR_VERSION = None
OPEN_SUBCHANNELS = {}
SUBCHANNEL_TIMEOUT = timedelta(minutes=5) # How long to wait before auto-closing
READY_STRING = "GATEONE_SSH_EXEC_CMD_CHANNEL_READY"
READY_MATCH = re.compile("^%s$" % READY_STRING, re.MULTILINE)
VALID_PRIVATE_KEY = valid = re.compile(
    r'^-----BEGIN [A-Z]+ PRIVATE KEY-----.*-----END [A-Z]+ PRIVATE KEY-----$',
    re.MULTILINE|re.DOTALL)
TIMER = None # Used to store temporary, cancellable timeouts
# TODO: make execute_command() a user-configurable option...  So it will automatically run whatever command(s) the user likes whenever they connect to a given server.  Differentiate between when they connect and when they start up a master or slave channel.
# TODO: Make it so that equivalent KnownHostsHandler functionality works over the WebSocket.

class SSHMultiplexingException(Exception):
    """
    Called when there's a failure trying to open a sub-shell via OpenSSH's
    `Master mode <http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing>`_
    multiplexing capability.
    """
    pass

class SSHExecutionException(Exception):
    """
    Called when there's an error trying to execute a command in the slave.
    """
    pass

class SSHKeygenException(Exception):
    """
    Called when there's an error trying to generate a public/private keypair.
    """
    pass

class SSHKeypairException(Exception):
    """
    Called when there's an error trying to save public/private keypair or
    certificate.
    """
    pass

class SSHPassphraseException(Exception):
    """
    Called when we try to generate/decode something that requires a passphrase
    but no passphrase was given.
    """
    pass

def get_ssh_dir(tws):
    """
    Given a :class:`gateone.TerminalWebSocket` (*tws*) instance, return the
    current user's ssh directory
    """
    users_ssh_dir = os.path.join(users_dir, 'ssh')
    return users_ssh_dir

def open_sub_channel(term, tws):
    """
    Opens a sub-channel of communication by executing a new shell on the SSH
    server using OpenSSH's `Master mode <http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing>`_
    capability (it spawns a new slave) and returns the resulting
    :class:`termio.Multiplex` instance.  If a slave has already been opened for
    this purpose it will re-use the existing channel.
    """
    logging.debug("open_sub_channel() term: %s" % term)
    global OPEN_SUBCHANNELS
    if term in OPEN_SUBCHANNELS and OPEN_SUBCHANNELS[term].isalive():
        # Use existing sub-channel (much faster this way)
        return OPEN_SUBCHANNELS[term]
    # NOTE: When connecting a slave via ssh you can't tell it to execute a
    # command like you normally can (e.g. 'ssh user@host <some command>').  This
    # is why we're using the termio.Multiplex.expect() functionality below...
    session = tws.session
    session_dir = tws.settings['session_dir']
    session_path = os.path.join(session_dir, session)
    socket_path = None
    # Find the SSH socket path...
    for f in os.listdir(session_path):
        if f.startswith('ssh:%s:' % term):
            # Grab the SSH socket path from the file
            for line in open(os.path.join(session_path, f)):
                if line.startswith('SSH_SOCKET'):
                    # NOTE: This will includes quotes (which is fine):
                    socket_path = line.split('=')[1].strip()
    # Interesting: When using an existing socket you don't need to give it all
    # the same options as you used to open it but you still need to give it
    # *something* in place of the hostname or it will report a syntax error and
    # print out the help.  So that's why I've put 'go_ssh_remote_cmd' below.
    # ...but I could have just used 'foo' :)
    if not socket_path:
        raise SSHMultiplexingException(_(
            "SSH Plugin: Unable to open slave sub-channel."))
    users_ssh_dir = get_ssh_dir(tws)
    ssh_config_path = os.path.join(users_ssh_dir, 'config')
    if not os.path.exists(ssh_config_path):
        # Create it (an empty one so ssh doesn't error out)
        with open(ssh_config_path, 'w') as f:
            f.write('\n')
    # Hopefully 'go_ssh_remote_cmd' will be a clear enough indication of
    # what is going on by anyone that has to review the logs...
    ssh = which('ssh')
    ssh_command = '%s -x -S%s -F%s go_ssh_remote_cmd' % (
        ssh, socket_path, ssh_config_path)
    OPEN_SUBCHANNELS[term] = m = tws.new_multiplex(
        ssh_command, "%s (sub)" % term)
    # Using huge numbers here so we don't miss much (if anything) if the user
    # executes something like "ps -ef".
    fd = m.spawn(rows=100, cols=200) # Hopefully 100/200 lines/cols is enough
    # ...if it isn't, well, that's not really what this is for :)
    # Set the term title so it gets a proper name in the logs
    m.writeline(u'\x1b]0;Term %s sub-channel\007' % term)
    return m

def wait_for_prompt(term, cmd, errorback, callback, m_instance, matched):
    """
    Called by :func:`termio.Multiplex.expect` inside of :func:`execute_command`,
    clears the screen and executes *cmd*.  Also, sets an
    :func:`~termio.Multiplex.expect` to call :func:`get_cmd_output` when the
    end of the command output is detected.
    """
    logging.debug('wait_for_prompt()')
    m_instance.term.clear_screen() # Makes capturing just what we need easier
    getoutput = partial(get_cmd_output, term, errorback, callback)
    m_instance.expect(READY_MATCH, getoutput, errorback=errorback, timeout=10)
    # Run our command immediately followed by our separation/ready string
    m_instance.writeline(u'%s; echo "%s"' % (cmd, READY_STRING))

def get_cmd_output(term, errorback, callback, m_instance, matched):
    """
    Captures the output of the command executed inside of
    :func:`wait_for_prompt` and calls *callback* if it isn't `None`.
    """
    logging.debug('get_cmd_output()')
    cmd_out = [a.rstrip() for a in m_instance.dump() if a.rstrip()]
    capture = False
    out = []
    for line in cmd_out:
        if capture:
            if READY_STRING in line:
                break
            out.append(line)
        elif READY_STRING in line:
            capture = True
    # This is just a silly trick to get the shell timing out/terminating itself
    # after a timout (so we don't keep the sub-channel open forever).  It is
    # easier than starting a timeout thread, timer, IOLoop.add_timeout(), etc
    # (I tried all those and it seemed to result in m_instance never getting
    # cleaned up properly by the garbage collector--it would leak memory)
    m_instance.unexpect() # Clear out any existing patterns (i.e. keepalive ;)
    m_instance.expect( # Add our inactivity timeout
        "^SUB-CHANNEL INACTIVITY TIMEOUT$", # ^ and $ to prevent accidents ;)
        noop, # Don't need to do anything since this should never match
        errorback=timeout_sub_channel,
        timeout=SUBCHANNEL_TIMEOUT)
    m_instance.scheduler.start() # To ensure the timeout occurs
    cmd_out = "\n".join(out)
    if callback:
        callback(cmd_out, None)

def terminate_sub_channel(m_instance):
    """
    Calls `m_instance.terminate()` and deletes it from the OPEN_SUBCHANNELS dict.
    """
    logging.debug("terminate_sub_channel()")
    m_instance.terminate()
    # Find the Multiplex object inside of OPEN_SUBCHANNELS and remove it
    for key, value in OPEN_SUBCHANNELS.items():
        # This will be something like: {1: <Multiplex instance>}
        if hash(value) == hash(m_instance):
            # This is necessary so the interpreter can properly collect garbage:
            del OPEN_SUBCHANNELS[key]

def timeout_sub_channel(m_instance):
    """
    Called when the sub-channel times out by way of an
    :class:`termio.Multiplex.expect` pattern that should never match anything.
    """
    logging.debug(_(
        "Sub-channel on term %s closed due to inactivity."
        % repr(m_instance.term_id)))
    terminate_sub_channel(m_instance)

def got_error(m_instance, match=None, term=None, cmd=None, tws=None):
    """
    Called if :func:`execute_command` encounters a problem/timeout.

    *match* is here in case we want to use it for a positive match of an error.
    """
    logging.error(_(
        "%s: Got an error trying to capture output inside of "
        "execute_command() running: %s" % (m_instance.user, m_instance.cmd)))
    if tws:
        message = {
            'sshjs_cmd_output': {
                'cmd': cmd,
                'term': term,
                'output': None,
                'result': _(
                    'Error: Timeout exceeded or command failed to execute.')
            }
        }

def execute_command(term, cmd, callback=None, tws=None):
    """
    Execute the given command (*cmd*) on the given *term* using the existing
    SSH tunnel (taking advantage of `Master mode <http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing>`_)
    and call *callback* with the output of said command and the current
    :class:`termio.Multiplex` instance as arguments like so::

        callback(output, m_instance)

    If *callback* is not provided then the command will be executed and any
    output will be ignored.

    .. note:: This will not result in a new terminal being opened on the client--it simply executes a command and returns the result using the existing SSH tunnel.
    """
    logging.debug(
        "execute_command(): term: %s, cmd: %s" % (term, cmd))
    try:
        m = open_sub_channel(term, tws)
    except SSHMultiplexingException as e:
            "%s: Got an error trying to open sub-channel on term %s..." %
            (tws.get_current_user()['upn'], term)))
        return
    # NOTE: We can assume the IOLoop is started and automatically calling read()
    m.unexpect() # Clear out any existing patterns (if existing sub-channel)
    m.term.clear_screen() # Clear the screen so nothing mucks up our regexes
    # Check to make sure we've got a proper prompt by executing an echo
    # statement and waiting for it to complete.  This is more reliable than
    # using a regular expression to match a shell prompt (which could be set
    # to anything).  It also gives us a clear indication as to where the command
    # output begins and ends.
    errorback = partial(got_error, term=term, cmd=cmd, tws=tws)
    wait = partial(wait_for_prompt, term, cmd, errorback, callback)
    m.expect(READY_MATCH, callback=wait, errorback=errorback, timeout=10)
    logging.debug("Waiting for READY_MATCH inside execute_command()")
    m.writeline(u'echo "%s"' % READY_STRING)

def send_result(tws, term, cmd, output, m_instance):
    """
    Called by :func:`ws_exec_command` when the output of the executed command
    has been captured successfully.  Writes a message to the client with the
    command's output and some relevant metadata.
    """
    message = {
        'sshjs_cmd_output': {
            'term': term,
            'cmd': cmd,
            'output': output,
            'result': 'Success'
        }
    }

def ws_exec_command(settings, tws):
    """
    Takes the necessary variables from *settings* and calls :func:`execute_command`.

    *settings* should be a dict that contains a 'term' and a 'cmd' to execute.

    .. tip:: This function can be used to quickly execute a command and return its result from the client over an existing SSH connection without requiring the user to enter their password!  See execRemoteCmd() in ssh.js.
    """
    term = settings['term']
    cmd = settings['cmd']
    send = partial(send_result, tws, term, cmd)
    try:
        execute_command(term, cmd, send, tws=tws)
    except SSHExecutionException as e:
        message = {
                'result': 'Error: %s' % e
            }
        }

# Handlers
class KnownHostsHandler(BaseHandler):
    """
    This handler allows the client to view, edit, and upload the known_hosts
    file associated with their user account.
    """
        """
        Determine what the user is asking for and call the appropriate method.
        """ # NOTE: Just dealing with known_hosts for now but keys are next
        get_kh = self.get_argument('known_hosts', None)
        if get_kh:
            self._return_known_hosts()

    def post(self):
        """
        Determine what the user is updating by checking the given arguments and
        proceed with the update.
        """
        known_hosts = self.get_argument('known_hosts', None)
        if known_hosts:
            kh = self.request.body
            self._save_known_hosts(kh)

    def _return_known_hosts(self):
        """Returns the user's known_hosts file in text/plain format."""
        user = self.get_current_user()['upn']
        logging.debug("known_hosts requested by %s" % user)
        users_dir = os.path.join(self.settings['user_dir'], user) # "User's dir"
        users_ssh_dir = os.path.join(users_dir, 'ssh')
        kh_path = os.path.join(users_ssh_dir, 'known_hosts')
        known_hosts = ""
        if os.path.exists(kh_path):
            known_hosts = open(kh_path).read()
        self.set_header ('Content-Type', 'text/plain')
        self.write(known_hosts)

    def _save_known_hosts(self, known_hosts):
        """Save the given *known_hosts* file."""
        # Letting Tornado's exception handler deal with errors here
        f = open(kh_path, 'w')
        f.write(known_hosts)
        f.close()
        self.write("success")

"""
WebSocket Commands
------------------
"""
def get_connect_string(term, tws):
    """
    Writes the connection string associated with *term* to the `WebSocket <https://developer.mozilla.org/en/WebSockets/WebSockets_reference/WebSocket>`_
    like so::

        {'sshjs_reconnect': {*term*: <connection string>}}

    In ssh.js we attach an action (aka handler) to :js:attr:`GateOne.Net.actions`
    for 'sshjs_reconnect' messages that attaches the connection string to
    `GateOne.terminals[*term*]['sshConnectString']`
    """
    logging.debug("get_connect_string() term: %s" % term)
    for f in os.listdir(os.path.join(session_dir, session)):
        if f.startswith('ssh:'):
            terminal, a_colon, connect_string = f[4:].partition(':')
            terminal = int(terminal)
            if terminal == term:
                # TODO: Make it so we don't have to use json_encode below...
                message = {
                    'sshjs_reconnect': json_encode({term: connect_string})
                }
                tws.write_message(message)
                return # All done

def get_key(name, public, tws):
    """
    Returns the private SSH key associated with *name* to the client.  If
    *public* is `True`, returns the public key to the client.
    """
    if not isinstance(name, (str, unicode)):
            'SSH Plugin Error: Invalid name given, %s' % repr(name))
        message = {'save_file': {'result': error_msg}}
        return out_dict
    if public and not name.endswith('.pub'):
        name += '.pub'
        'result': None, # Yet
        'filename': name,
        'mimetype': 'text/plain'
    }
    key_path = os.path.join(users_ssh_dir, name)
    if os.path.exists(key_path):
        with open(key_path) as f:
            out_dict['data'] = f.read()
        out_dict['result'] = 'Success'
    else:
        out_dict['result'] = _(
            'SSH Plugin Error: Public key not found at %s' % key_path)

def get_public_key(name, tws):
    """
    Returns the user's public key file named *name*.
    """
    get_key(name, True, tws)

def get_private_key(name, tws):
    """
    Returns the user's private key file named *name*.
    """
    get_key(name, False, tws)

def get_host_fingerprint(settings, tws):
    """
    Returns a the hash of the given host's public key by making a remote
    connection to the server (not just by looking at known_hosts).
    """
    if 'port' not in settings:
        port = 22
    else:
        port = settings['port']
    if 'host' not in settings:
        out_dict['result'] = _("Error:  You must supply a 'host'.")
        message = {'sshjs_display_fingerprint': out_dict}
    else:
        host = settings['host']
    logging.debug("get_host_fingerprint(%s:%s)" % (host, port))
        'host': host,
        'fingerprint': None
    }
    m = tws.new_multiplex(
        '%s -p %s -oUserKnownHostsFile=none -F. %s' % (ssh, port, host),
        'get_host_key',
        logging=False) # Logging is false so we don't make tons of silly logs
    def grab_fingerprint(m_instance, match):
        out_dict['fingerprint'] = match.split()[-1][:-1]
        m_instance.terminate()
        del m_instance
    def errorback(m_instance):
        leftovers = [a.rstrip() for a in m_instance.dump() if a.strip()]
            "Error: Could not determine the fingerprint of %s:%s... '%s'"
            % (host, port, "\n".join(leftovers)))
        m_instance.terminate() # Don't leave stuff hanging around!
    m.expect('.+fingerprint .+$', grab_fingerprint, errorback=errorback)
    m.spawn()
    # OpenSSH output example:
    # ECDSA key fingerprint is 28:46:86:3a:c6:f9:63:b8:90:e1:09:69:f2:1d:c8:ce.
    # Dropbear output example:
    # (fingerprint md5 fa:a1:5b:4f:e5:ab:fe:e6:1f:1f:74:20:d7:35:67:c2)

def generate_new_keypair(settings, tws):
    """
    Calls :func:`openssh_generate_new_keypair` or
    :func:`dropbear_generate_new_keypair` depending on what's available on the
    system.
    """
    logging.debug('generate_new_keypair()')
    name = 'id_ecdsa'
    keytype = None
    bits = None
    passphrase = ''
    comment = ''
    if 'name' in settings:
        name = settings['name']
    if 'keytype' in settings:
        keytype = settings['keytype']
    if 'bits' in settings:
        bits = settings['bits']
    if 'passphrase' in settings:
        passphrase = settings['passphrase']
    if 'comment' in settings:
        comment = settings['comment']
    #if which('dropbearkey'):
        #DROPBEAR_VERSION = shell_command('dropbear -V')[1].splitlines()[1]
    if which('ssh-keygen'): # Prefer OpenSSH
        openssh_generate_new_keypair(
            name, # Name to use when generating the keypair
            users_ssh_dir, # Path to save it
            keytype=keytype,
            passphrase=passphrase,
            bits=bits,
            comment=comment,
            tws=tws
        )
    elif which('dropbearkey'):
        dropbear_generate_new_keypair(*args, **kwargs)

def errorback(tws, m_instance):
    logging.debug("keygen errorback()")
    print(m_instance.dump())
    message = {
        'sshjs_keygen_complete': {
            'result': _("There was a problem generating SSH keys: %s"
                        % m_instance.dump()),
        }
    }

def overwrite(m_instance, match):
    """
    Called if we get asked to overwrite an existing keypair.
    """
    logging.debug('overwrite()')
    m_instance.writeline('y')

def enter_passphrase(passphrase, m_instance, match):
    logging.debug("entering passphrase...")
    m_instance.writeline('%s' % passphrase)

def finished(tws, m_instance, fingerprint):
    logging.debug("keygen finished.  fingerprint: %s" % fingerprint)
    message = {
            'result': 'Success',
            'fingerprint': fingerprint
        }
    }

def openssh_generate_new_keypair(name, path,
        keytype=None, passphrase="", bits=None, comment="", tws=None):
    """
    Generates a new private and public key pair--stored in the user's directory
    using the given *name* and other optional parameters (using OpenSSH).

    If *keytype* is given, it must be one of "ecdsa", "rsa" or "dsa" (case
    insensitive).  If *keytype* is "rsa" or "ecdsa", *bits* may be specified to
    specify the size of the key.

    .. note:: Defaults to generating a 521-byte ecdsa key if OpenSSH is version 5.7+. Otherwise a 2048-bit rsa key will be used.
    """
    logging.debug('openssh_generate_new_keypair()')
    openssh_version = shell_command('ssh -V')[1]
    ssh_major_version = int(
        openssh_version.split()[0].split('_')[1].split('.')[0])
    key_path = os.path.join(path, name)
    ssh_minor_version = int(
        openssh_version.split()[0].split('_')[1].split('.')[1][0])
    ssh_version = "%s.%s" % (ssh_major_version, ssh_minor_version)
    ssh_version = float(ssh_version)
    if not keytype:
        if ssh_version >= 5.7:
            keytype = "ecdsa"
        else:
            keytype = "rsa"
    else:
        keytype = keytype.lower()
    if not bits and keytype == "ecdsa":
        bits = 521 # Not a typo: five-hundred-twenty-one bits
    elif not bits and keytype == "rsa":
        bits = 2048
    if not passphrase: # This just makes sure False and None end up as ''
        passphrase = ''
    hostname = os.uname()[1]
    if not comment:
        now = datetime.now().isoformat()
        comment = "Generated by Gate One on %s %s" % (hostname, now)
    ssh_keygen_path = which('ssh-keygen')
    command = (
        "%s "       # Path to ssh-keygen
        "-b %s "    # bits
        "-t %s "    # keytype
        "-C '%s' "  # comment
        "-f %s"     # Key path
        % (ssh_keygen_path, bits, keytype, comment, key_path)
    )
    m = tws.new_multiplex(command, "gen_ssh_keypair")
    call_errorback = partial(errorback, tws)
    m.expect('^Overwrite.*', overwrite, optional=True, timeout=10)
    passphrase_handler = partial(enter_passphrase, passphrase)
    m.expect('^Enter passphrase',
        passphrase_handler, errorback=call_errorback, timeout=10)
    m.expect('^Enter same passphrase again',
    finalize = partial(finished, tws)
    # The regex below captures the md5 fingerprint which tells us the
    # operation was successful.
    m.expect(
        '(([0-9a-f][0-9a-f]\:){15}[0-9a-f][0-9a-f])',
        finalize,
        errorback=call_errorback,
        timeout=15 # Key generation can take a little while
    )
    m.spawn()

def dropbear_generate_new_keypair(name, path,
    """
    .. note:: Not implemented yet
    """
    pass

def openssh_generate_public_key(path, passphrase=None, settings=None, tws=None):
    """
    Generates a public key from the given private key at *path*.  If a
    *passphrase* is provided, it will be used to generate the public key (if
    necessary).
    """
    logging.debug('openssh_generate_public_key()')
    pubkey_path = "%s.pub" % path
    command = (
        "-f %s "    # Key path
        "-y "       # Output public key to stdout
        "2>&1 "     # Redirect stderr to stdout so we can catch failures
        "> %s"      # Redirect stdout to the public key path
        % (ssh_keygen_path, path, pubkey_path)
    )
    import termio
    m = termio.Multiplex(command)
    def request_passphrase(*args, **kwargs):
        "Called if this key requires a passphrase.  Ask the client to provide"
        message = {'sshjs_ask_passphrase': settings}
    def bad_passphrase(m_instance, match):
        "Called if the user entered a bad passphrase"
        settings['bad'] = True
        request_passphrase()
    if passphrase:
        m.expect('^Enter passphrase',
            "%s\n" % passphrase, optional=True, timeout=5)
        m.expect('^load failed',
            bad_passphrase, optional=True, timeout=5)
    elif settings:
            request_passphrase, optional=True, timeout=5)
    def atexit(child, exitstatus):
        "Raises an SSHKeygenException if the *exitstatus* isn't 0"
        if exitstatus != 0:
            raise SSHKeygenException(_(
                "Error generating public key from private key at %s" % path))
    fd = m.spawn(exitfunc=atexit)
    #exitstatus, output = shell_command(command)
    #if exitstatus != 0:
        #raise SSHKeygenException(_(
            #"Error generating public key from private key at %s" % path))

# ssh-kegen example for reference:
    #$ ssh-keygen -b 521 -t ecdsa "Testing" -f /tmp/testkey
    #Generating public/private ecdsa key pair.
    #Enter passphrase (empty for no passphrase):
    #Enter same passphrase again:
    #Your identification has been saved in /tmp/testkey.
    #Your public key has been saved in /tmp/testkey.pub.
    #The key fingerprint is:
    #6b:13:4b:5d:80:bd:21:70:33:f5:b9:15:78:75:08:9a Testing
    #The key's randomart image is:
    #+--[ECDSA  521]---+
    #|      ..++o .o.oo|
    #|       .ooo=..o..|
    #|         .Eo+..  |
    #|         ... o   |
    #|        S . .    |
    #|       . +       |
    #|        =        |
    #|       . .       |
    #|                 |
    #+-----------------+

# dropbearkey example for reference:

    #root@router:~# dropbearkey -t dss -f /tmp/testing
    #Will output 1024 bit dss secret key to '/tmp/testing'
    #Generating key, this may take a while...
    #Public key portion is:
    #ssh-dss AAAAB3NzaC1kc3MAAACBAJ5jU4izsZtJKEojw7gIcc6e3U4X6OENN6081YxSAanfTbKjR0V3Ho6aui2z8o039BVH4S5cVD51vEEvDjirKStM2aMvdrVZkjGH1iOMWY4MQrCl4EqMr7rWikeiZJN6BJ+xmPBUyZuicVDFkBwqC+dKgxml0RTpa7TYBWvp403XAAAAFQDg6vb3afaKM9+DvBW7I4xPxF8a8QAAAIEAjcNHYFrqcWK9lSsw2Oy+w1PEWQuxvWydXXk3MQyiZ/PYaeU/138iCB2pW1fgCksx5CHF8dgtQ7AsFv32gBlxuDgX3EYtPYR0wGJqyU7w9+qaq1T02zmDfW4k2WDfMNz+QWFYHuKzC/aeuEC0BRTLyPVQMHLNAd/F5beCqlIPRPcAAACAfUy1+yNgK2svox6aJRqtpxbMSPDRNTRMAjeTkCeLopesZFYbPvms2c19WkIk2qD9aw3gIxsR4wO+kkvI4BtOs8dXQWS+bc+svJbIYOqmPFo89BJHfbP9wvMhfTlp1uH9LxAG6ZiHHz5fseUgTrwYkSw1beUprikxlca8lQm5v7g= root@RouterStationPro
    #Fingerprint: md5 c6:f9:f2:95:b8:40:ac:f3:53:f1:39:e9:57:a0:58:18

# TODO: Get this validating uploaded keys.
def store_id_file(settings, tws=None):
    """
    Stores the given *settings['private']* and/or *settings['public']* keypair
    in the user's ssh directory as *settings['name']* and/or
    *settings['name']*.pub, respectively.  Either file can be saved independent
    of each other (in case this function needs to be called multiple times to
    save each respective file).

    Also, a *settings['certificate']* may be provided to be saved along
    with the private and public keys.  It will be saved as
    *settings['name']*-cert.pub.

    .. note:: I've found the following website helpful in understanding how to use OpenSSH with SSL certificates: http://blog.habets.pp.se/2011/07/OpenSSH-certificates

    .. tip:: Using signed-by-a-CA certificates is very handy because allows you to revoke the user's SSH key(s).  e.g. If they left the company.
    """
    logging.debug('store_id_file()')
    out_dict = {'result': 'Success'}
    name, private, public, certificate = None, None, None, None
    passphrase = None
    global TIMER
    try:
        if 'name' in settings:
            name = settings['name']
        else:
            raise SSHKeypairException(_("You must specify a valid *name*."))
        if 'private' in settings:
            private = settings['private']
        if 'public' in settings:
            public = settings['public']
        if 'certificate' in settings:
            certificate = settings['certificate']
        if 'passphrase' in settings:
            passphrase = settings['passphrase']
        if not private and not public and not certificate:
            raise SSHKeypairException(_("No files were given to save!"))
        users_ssh_dir = get_ssh_dir(tws)
        private_key_path = os.path.join(users_ssh_dir, name)
        public_key_path = os.path.join(users_ssh_dir, name+'.pub')
        certificate_path = os.path.join(users_ssh_dir, name+'-cert.pub')
        if private:
            if VALID_PRIVATE_KEY.match(private):
                with open(private_key_path, 'w') as f:
                    f.write(private)
                # Without this you get a warning:
                os.chmod(private_key_path, 0600)
            else:
                tws.write_message({'notice': _(
                    "ERROR: Private key is not valid.")})
                return
        if public:
            with open(public_key_path, 'w') as f:
                f.write(public)
            # Now remove the timer that will generate the public key from the
            # private key if it is set.
            if TIMER:
                logging.debug(_(
                    "Got public key, cancelling public key generation timer."))
                io_loop = tornado.ioloop.IOLoop.instance()
                io_loop.remove_timeout(TIMER)
                TIMER = None
        elif private: # No biggie, generate one
            # Only generate a new public key if one isn't uploaded within 2
            # seconds (should be plenty of time since they're typically sent
            # simultaneously but inside different WebSocket messages).
                "Only received a private key.  Setting timeout to generate the "
                "public key if not received within 3 seconds."))
            io_loop = tornado.ioloop.IOLoop.instance()
            deadline = timedelta(seconds=2)
            def generate_public_key(): # I love closures
                openssh_generate_public_key(
                    private_key_path, passphrase, settings=settings, tws=tws)
                get_ids = partial(get_identities, None, tws)
                io_loop.add_timeout(timedelta(seconds=2), get_ids)
            # This gets removed if the public key is uploaded
            TIMER = io_loop.add_timeout(deadline, generate_public_key)
        if certificate:
            with open(certificate_path, 'w') as f:
                f.write(certificate)
        out_dict['result'] = _("Error saving keys: %s" % e)
    message = {
        'sshjs_save_id_complete': out_dict
    }

def delete_identity(name, tws):
    """
    Removes the identity associated with *name*.  For example if *name* is
    'testkey', 'testkey' and 'testkey.pub' would be removed from the user's
    ssh directory (and 'testkey-cert.pub' if present).
    """
    logging.debug('delete_identity()')
    private_key_path = os.path.join(users_ssh_dir, name)
    public_key_path = os.path.join(users_ssh_dir, name+'.pub')
    certificate_path = os.path.join(users_ssh_dir, name+'-cert.pub')
    try:
        if os.path.exists(private_key_path):
            os.remove(private_key_path)
        if os.path.exists(public_key_path):
            os.remove(public_key_path)
        if os.path.exists(certificate_path):
            os.remove(certificate_path)
        out_dict['result'] = _("Error deleting keypair: %s" % e)
    message = {
        'sshjs_delete_identity_complete': out_dict
    }

def get_identities(anything, tws):
    """
    Sends a message to the client with a list of the identities stored on the
    server for the current user.

    *anything* is just there because the client needs to send *something* along
    with the 'action'.
    """
    logging.debug('get_identities()')
    out_dict['identities'] = []
    keytype_re = re.compile('.*\(([A-Z]+)\)$', re.MULTILINE)
    # TODO:  Switch this from using ssh-keygen to determine the keytype to using the string inside the public key.
    try:
        if os.path.exists(users_ssh_dir):
            ssh_files = os.listdir(users_ssh_dir)
            for f in ssh_files:
                if f.endswith('.pub'):
                    # Double-check there's also a private key...
                    identity = f[:-4] # Will be the same name minus '.pub'
                    if identity in ssh_files:
                        id_path = os.path.join(users_ssh_dir, identity)
                        pub_key_path = os.path.join(users_ssh_dir, f)
                        public_key_contents = open(pub_key_path).read()
                        comment = ' '.join(public_key_contents.split(' ')[2:])
                        if public_key_contents.startswith('ecdsa'):
                            keytype = 'ECDSA'
                        elif public_key_contents.startswith('ssh-dss'):
                            keytype = 'DSA'
                        elif public_key_contents.startswith('ssh-rsa'):
                            keytype = 'RSA'
                        else:
                            keytype = 'Unknown'
                        keygen_cmd = "%s -vlf %s" % (ssh_keygen_path, id_path)
                        retcode, key_info = shell_command(keygen_cmd)
                        # This will just wind up as an empty string if the
                        # version of ssh doesn't support randomart:
                        randomart = '\n'.join(key_info.splitlines()[1:])
                        bits = key_info.split()[0]
                        fingerprint = key_info.split()[1]
                        retcode, bubblebabble = shell_command(
                            "%s -Bf %s" % (ssh_keygen_path, id_path))
                        bubblebabble = bubblebabble.split()[1]
                        certinfo = ''
                        cert_path = "%s-cert.pub" % id_path
                        if os.path.exists(cert_path):
                            retcode, certinfo = shell_command(
                            "%s -Lf %s" % (ssh_keygen_path, cert_path))
                        certinfo = ' '.join(certinfo.split(' ')[1:])
                        fixed_certinfo = ''
                        for i, line in enumerate(certinfo.splitlines()):
                            if i == 0:
                                line = line.lstrip()
                            fixed_certinfo += line.replace('    ', ' ')
                            fixed_certinfo += '\n'
                        id_obj = {
                            'name': identity,
                            'public': public_key_contents,
                            'keytype': keytype,
                            'bubblebabble': bubblebabble,
                            'fingerprint': fingerprint,
                            'randomart': randomart,
                            'certinfo': fixed_certinfo,
                            'bits': bits,
                            'comment': comment.rstrip(),
                        }
                        out_dict['identities'].append(id_obj)
        # Figure out which identities are defaults
        default_ids = []
        default_ids_exists = False
        default_ids_path = os.path.join(users_ssh_dir, '.default_ids')
        if os.path.exists(default_ids_path):
            default_ids_exists = True
            with open(default_ids_path) as f:
                default_ids = f.read().splitlines() # Why not readlines()? \n
        # Convert any absolute paths inside default_ids to just the short names
        default_ids = [os.path.split(a)[1] for a in default_ids]
        if default_ids_exists:
            for i, id_obj in enumerate(out_dict['identities']):
                if id_obj['name'] in default_ids:
                    out_dict['identities'][i]['default'] = True
                else:
                    out_dict['identities'][i]['default'] = False
        error_msg = _("Error getting identities: %s" % e)
        logging.error(error_msg)
        out_dict['result'] = error_msg
    message = {
        'sshjs_identities_list': out_dict
    }

def set_default_identities(identities, tws):
    """
    Given a list of *identities*, mark them as defaults to use in all outbound
    SSH connections by writing them to `<user's ssh dir>/.default_ids`.  If
    *identities* is empty, no identities will be used in outbound connections.

    .. note:: Whenever this function is called it will overwrite whatever is in `.default_ids`.
    """
    if isinstance(identities, list): # Ignore anything else
        with open(default_ids_path, 'w') as f:
            f.write('\n'.join(identities) + '\n') # Need that trailing newline

def opt_esc_handler(text, tws):
    """
    Handles text passed from the special optional escape sequance handler.  We
    use it to tell ssh.js what the SSH connection string is so it can use that
    information to duplicate sessions (if the user so desires).  For reference,
    the specific string which will call this function from a terminal app is::

        \x1b]_;ssh|<whatever>\x07

    """
    message = {'sshjs_connect': text}

def create_user_ssh_dir(current_user, settings):
    """
    To be called by the 'Auth' hook that gets called after the user is done
    authenticating, ensures that the `<user's dir>/ssh` directory exists.
    """
    user = current_user['upn']
    users_dir = os.path.join(settings['user_dir'], user) # "User's dir"
    ssh_dir = os.path.join(users_dir, 'ssh')
    try:
        mkdir_p(ssh_dir)
        logging.error(_("Error creating user's ssh directory: %s\n" % e))

hooks = {
    },
}

# Certificate information (as output by ssh-keygen) for reference:
#
#$ ssh-keygen -Lf id_rsa-cert.pub
#id_rsa-cert.pub:
        #Type: ssh-rsa-cert-v01@openssh.com user certificate
        #Public key: RSA-CERT 80:57:2c:18:f9:86:ab:8b:64:27:db:6f:5e:03:3f:d9
        #Signing CA: RSA 86:25:b0:73:67:0f:51:2e:a7:96:63:08:fb:d6:69:94
        #Key ID: "user_riskable"
        #Serial: 0
        #Valid: from 2012-01-08T13:38:00 to 2013-01-06T13:39:27
        #Principals:
                #riskable
        #Critical Options: (none)
        #Extensions:
                #permit-agent-forwarding
                #permit-port-forwarding
                #permit-pty
                #permit-user-rc
                #permit-X11-forwarding

# NOTE: *message['identities'][0]* - An associative array conaining the key's metadata.  For example:
#
#{
    #'name': "testid",
    #'public': "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAChqFprVjC0MKe3qpjjc+WdANOHMgcUl46dJxZ+s5soBTkO6thcJDAbFb36lg3YyzZi/PtDJV5CPp8Mv1SUXUYBqgFZJFBqWwkB0O1ohjtEVzC8+ybrY+hP0zLqykglhOi+6W66HgFwjJGn56uGE7s8UpnSRKtqGq2USyme5gopYlytTw== Generated by Gate One on somehost",
    #'keytype': "ecdsa",
    #'bubblebabble': "xevol-budez-difod-zumif-zofos-vezis-rilep-febel-tufok-lugud-dyxex",
    #'fingerprint': "0e:69:0a:9e:2e:26:2b:91:23:3d:95:4b:65:31:a9:6f",
    #'randomart': "+--[ECDSA  521]---+\n|      oo         |\n|      +.         |\n|     =           |\n|    =  .         |\n| o.o o+ S        |\n|=.oo.oEo         |\n|.oo...  .        |\n|+o               |\n|=o.              |\n+-----------------+",
    #'certinfo': "",
    #'bits': 521,
    #'comment': "Generated by Gate One on somehost",
#}

#
#       Copyright 2011 Liftoff Software Corporation (http://liftoffsoftware.com)
#
# NOTE:  Commercial licenses for this software are available!
#

# TODO: See if we can spin off termio.py into its own little program that sits between Gate One and ssh_connect.py.  That way we can take advantage of multiple cores/processors (for terminal-to-HTML processing).  There's no reason why we can't write something that does what dtach does.  Just need to redirect the fd of self.cmd to a unix domain socket and os.setsid() somewhere after forking (twice maybe?).
# TODO: Make the environment variables used before launching self.cmd configurable

# Meta

About termio
============
This module provides a Multiplex class that can perform the following:

 * Fork a child process that opens a given terminal program.
 * Read and write data to and from the child process (synchronously or asynchronously).
 * Examine the output of the child process in real-time and perform actions (also asynchronously!) based on what is "expected" (aka non-blocking, pexpect-like functionality).
 * Log the output of the child process to a file and/or syslog.

The Multiplex class was built for asynchronous use in conjunction with a running
:class:`tornado.ioloop.IOLoop` instance but it can be used in a synchronous
(blocking) manner as well.  Synchronous use of this module is most likely to be
useful in an interactive Python session but if blocking doesn't matter for your
program please see the section titled, "Blocking" for tips & tricks.

Here's an example instantiating a Multiplex class::

    multiplexer = termio.Multiplex(
        'nethack',
        log_path='/var/log/myapp',
        user='bsmith@CORP',
        term_id=1,
        syslog=True
    )

.. note:: Support for event loops other than Tornado is in the works!

Then *multiplexer* can create and launch a new controlling terminal (tty)
running the given command (e.g. 'nethack')::

    env = {
        'PATH': os.environ['PATH'],
        'MYVAR': 'foo'
    }
    fd = multiplexer.spawn(80, 24, env=env)
    # The fd is returned from spawn() in case you want more low-level control.

Asynchronous input and output from the controlled program is handled via IOLoop.
It will automatically write all output from the terminal program to an instance
of self.terminal_emulator (which defaults to Gate One's `terminal.Terminal`).
So if you want to perform an action whenever the running terminal application
has output (like, say, sending a message to a client) you'll need to attach a
callback::

    def screen_update():
        'Called when new output is ready to send to the client'
        output = multiplexer.dump_html()
        socket_or_something.write(output)
    multiplexer.callbacks[multiplexer.CALLBACK_UPDATE] = screen_update

In this example, `screen_update()` will `write()` the output of
`multiplexer.dump_html()` to *socket_or_something* whenever the terminal program
has some sort of output.  You can also make calls directly to the terminal
emulator (if you're using a custom one)::

        output = multiplexer.term.my_custom_func()
        whatever.write(output)

Writing characters to the controlled terminal application is pretty

    multiplexer.write(u'some text')

Typically you'd pass in keystrokes or commands from your application to the
underlying program this way and the screen/terminal emulator would get updated
automatically.  If using Gate One's `terminal.Terminal()` you can also attach
callbacks to perform further actions when more specific situations are
encountered (e.g. when the window title is set via its respective escape
sequence)::

    def set_title():
        'Hypothetical title-setting function'
        print("Window title was just set to: %s" % multiplexer.term.title)
    multiplexer.term.callbacks[multiplexer.CALLBACK_TITLE] = set_title

Module Functions and Classes
============================
"""

# Stdlib imports
import os, sys, time, struct, io, gzip, re, logging
from copy import copy
from datetime import timedelta, datetime
from itertools import izip
from multiprocessing import Process
from json import loads as json_decode
from json import dumps as json_encode

gettext.install('termio')

# Globals
# NOTE: That unicode character was carefully selected from only the finest
# of the PUA.  I hereby dub thee, "U+F0F0F0, The Separator."
CALLBACK_THREAD = None # Used by add_callback()
POSIX = 'posix' in sys.builtin_module_names

def debug_expect(m_instance, match, pattern):
    """
    This method is used by :meth:`BaseMultiplex.expect` if :attr:`self.debug` is
    True.  It facilitates easy debugging of regular expressions.  It will print
    out precisely what was matched and where.

    .. note::  This function only works with post-process patterns.
    """
    print("%s was matched..." % repr(match))
    out = ""
    for line in m_instance.dump():
        match_obj = pattern.search(line)
            out += "--->%s\n" % repr(line)
            break
        else:
            out += "    %s\n" % repr(line)
    print(out)

def retrieve_first_frame(golog_path):
    """
    Retrieves the first frame from the given *golog_path*.
    """
    found_first_frame = None
    frame = b""
    f = gzip.open(golog_path)
    while not found_first_frame:
        frame += f.read(1) # One byte at a time
        if frame.decode('UTF-8', "ignore").endswith(SEPARATOR):
            # That's it; wrap this up
            found_first_frame = True
    distance = f.tell()
    f.close()
    return (frame.decode('UTF-8', "ignore").rstrip(SEPARATOR), distance)

def retrieve_last_frame(golog_path):
    """
    Retrieves the last frame from the given *golog_path*.  It does this by
    iterating over the log in reverse.
    """
    encoded_separator = SEPARATOR.encode('UTF-8')
    golog = gzip.open(golog_path)
    chunk_size = 1024*128
    # Seek to the end of the file (gzip objects don't support negative seeking)
    distance = chunk_size
    prev_tell = None
    while golog.tell() != prev_tell:
        prev_tell = golog.tell()
        try:
            golog.seek(distance)
            return # Something wrong with the file
        distance += distance
    # Now that we're at the end, go back a bit and split from there
    golog.seek(golog.tell() - chunk_size)
    end_frames = golog.read().split(encoded_separator)
    if len(end_frames) > 1:
        # Very last item will be empty
        return end_frames[-2].decode('UTF-8', 'ignore')
    else:
        # Just a single frame here, return it as-is
        return end_frames[0].decode('UTF-8', 'ignore')

def get_or_update_metadata(golog_path, user, force_update=False):
    """
    Retrieves or creates/updates the metadata inside of *golog_path*.

    If *force_update* the metadata inside the golog will be updated even if it
    already exists.

    .. note::  All logs will need "fixing" the first time they're enumerated like this since they won't have an end_date.  Fortunately we only need to do this once per golog.
    """
    logging.debug('get_or_update_metadata(%s, %s, %s)' % (golog_path, user, force_update))
    if not os.path.getsize(golog_path): # 0 bytes
        return # Nothing to do
    try:
        first_frame, distance = retrieve_first_frame(golog_path)
        # Something wrong with the log...  Probably still being written to
        return
    metadata = {}
    if first_frame[14:].startswith('{'):
        # This is JSON, capture existing metadata
        metadata = json_decode(first_frame[14:])
        # end_date gets added by this function
        if not force_update and 'end_date' in metadata:
            return metadata # All done
    # '\xf3\xb0\xbc\x8f' <--UTF-8 encoded SEPARATOR (for reference)
    # Loop over the file in big chunks (which is faster than read() by an order
    # of magnitude)
    chunk_size = 1024*128 # 128k should be enough for a 100x300 terminal full
    # of 4-byte unicode characters. That would be one BIG frame (i.e. unlikely).
    # Sadly, we have to read the whole thing into memory (log_data) in order to
    # perform this important work (creating proper metadata).
    # On the plus side re-compressing the log can save a _lot_ of disk space
    # Why?  Because termio.py writes the frames using gzip.open() in append mode
    # which is a lot less efficient than compressing all the data in one go.
    log_data = b''
    total_frames = 0
        try:
            chunk = golog.read(chunk_size)
        total_frames += chunk.count(encoded_separator)
        log_data += chunk
        if len(chunk) < chunk_size:
            break
    log_data = log_data.decode('UTF-8', 'ignore')
    start_date = first_frame[:13] # Getting the start date is easy
    last_frame = retrieve_last_frame(golog_path) # This takes some work
    end_date = last_frame[:13]
    version = u"1.0"
    connect_string = None
    from gateone import PLUGINS
    if 'ssh' in PLUGINS['py']:
        # Try to find the host that was connected to by looking for the SSH
        # plugin's special optional escape sequence.  It looks like this:
        #   "\x1b]_;ssh|%s@%s:%s\007"
        match_obj = RE_OPT_SSH_SEQ.match(log_data[:(chunk_size*10)])
            connect_string = match_obj.group(1).split('|')[1]
    if not connect_string:
        # Try guessing it by looking for a title escape sequence
        match_obj = RE_TITLE_SEQ.match(log_data[:(chunk_size*10)])
            # The split() here is an attempt to remove the tail end of
            # titles like this:  'someuser@somehost: ~'
            connect_string = match_obj.group(1)
    # TODO: Add some hooks here for plugins to add their own metadata
    metadata.update({
        u'user': user,
        u'start_date': start_date,
        u'end_date': end_date,
        u'frames': total_frames,
        u'version': version,
        u'connect_string': connect_string,
        u'filename': os.path.split(golog_path)[1]
    })
    # Make a *new* first_frame
    first_frame = u"%s:" % start_date
    first_frame += json_encode(metadata)
    # Replace the first frame and re-save the log
    log_data = log_data.encode('UTF-8') # Encode this first to ensure 'distance'
    log_data = (first_frame + SEPARATOR).encode('UTF-8') + log_data[distance:]
    gzip.open(golog_path, 'w').write(log_data)
    return metadata

class Timeout(Exception):
    """
    Used by :meth:`BaseMultiplex.expect` and :meth:`BaseMultiplex.await`;
    called when a timeout is reached.
    """
    pass

class ProgramTerminated(Exception):
    """
    Called when we try to write to a process that's no longer running.
    """
    pass

# Classes
class Pattern(object):
    """
    Used by :meth:`BaseMultiplex.expect`, an object to store patterns
    (regular expressions) and their associated properties.

    .. note:: The variable *m_instance* is used below to mean the current instance of BaseMultiplex (or a subclass thereof).

    :pattern: A regular expression or iterable of regular expressions that will be checked against the output stream.

    :callback: A function that will be called when the pattern is matched.  Callbacks are called like so:

        >>> callback(m_instance, matched_string)

        .. tip:: If you provide a string instead of a function for your *callback* it will automatically be converted into a function that writes the string to the child process.  Example::

            >>> p = Pattern('(?i)password:', 'mypassword\\n')

    :optional: Indicates that this pattern is optional.  Meaning that it isn't required to match before the next pattern in :attr:`BaseMultiplex._patterns` is checked.

    :sticky: Indicates that the pattern will not time out and won't be automatically removed from self._patterns when it is matched.

    :errorback: A function to call in the event of a timeout or if an exception is encountered.  Errorback functions are called like so:

        >>> errorback(m_instance)

    :preprocess: Indicates that this pattern is to be checked against the incoming stream before it is processed by the terminal emulator.  Useful if you need to match non-printable characters like control codes and escape sequences.

    :timeout: A :obj:`datetime.timedelta` object indicating how long we should wait before calling :meth:`errorback`.

    :created: A :obj:`datetime.datetime` object that gets set when the Pattern is instantiated by :meth:`BaseMultiplex.expect`.  It is used to determine if and when a timeout has been reached.
    """
    def __init__(self, pattern, callback,
            optional=False,
            sticky=False,
            errorback=None,
            preprocess=False,
            timeout=30):
        self.pattern = pattern
        if isinstance(callback, (str, unicode)):
            # Convert the string to a write() call
            self.callback = lambda m, match: m.write(unicode(callback))
        else:
            self.callback = callback
        self.errorback = errorback
        self.optional = optional
        self.sticky = sticky
        self.preprocess = preprocess
        self.timeout = timeout
        self.created = datetime.now()

class BaseMultiplex(object):
    """
    A base class that all Multiplex types will inherit from.

    :cmd: *string* - The command to execute when calling :meth:`spawn`.
    :terminal_emulator: *terminal.Terminal or similar* - The terminal emulator to write to when capturing the incoming output stream from *cmd*.
    :log_path: *string* - The absolute path to the log file where the output from *cmd* will be saved.
    :term_id: *string* - The terminal identifier to associated with this instance (only used in the logs to identify terminals).
    :syslog: *boolean* - Whether or not the session should be logged using the local syslog daemon.
    :syslog_host: *string* - An optional syslog host to send session log information to (this is independent of the *syslog* option above--it does not require a syslog daemon be present on the host running Gate One).
    :syslog_facility: *integer* - The syslog facility to use when logging messages.  All possible facilities can be found in `utils.FACILITIES` (if you need a reference other than the syslog module).
    :debug: *boolean* - Used by the `expect` methods...  If set, extra debugging information will be output whenever a regular expression is matched.
    """
    CALLBACK_UPDATE = 1 # Screen update
    CALLBACK_EXIT = 2   # When the underlying program exits

    def __init__(self,
            cmd,
            terminal_emulator=None, # Defaults to Gate One's terminal.Terminal
            log_path=None,
            user=None, # Only used by log output (to differentiate who's who)
            term_id=None, # Also only for syslog output for the same reason
            syslog=False,
            syslog_host=None,
            syslog_facility=None,
            debug=False):
        self.debug = debug
        self.exitfunc = None
        self.cmd = cmd
        if not terminal_emulator:
            # Why do this?  So you could use/write your own specialty emulator.
            # Whatever you use it just has to accept 'rows' and 'cols' as
            # keyword arguments in __init__()
            from terminal import Terminal # Dynamic import to cut down on waste
            self.terminal_emulator = Terminal
        else:
            self.terminal_emulator = terminal_emulator
        self.log_path = log_path # Logs of the terminal output wind up here
        self.log = None # Just a placeholder until it is opened
        self.syslog = syslog # See "if self.syslog:" below
        self._alive = False
        self.ratelimiter_engaged = False
        self.rows = 24
        self.cols = 80
        self.pid = -1 # Means "no pid yet"
        self.started = "Never"
        self._patterns = []
        self._handling_match = False
        # Setup our callbacks
        self.callbacks = { # Defaults do nothing which saves some conditionals
            self.CALLBACK_UPDATE: {},
            self.CALLBACK_EXIT: {},
        }
        # Configure syslog logging
        self.user = user
        self.term_id = term_id
        self.syslog_buffer = ''
        if self.syslog and not self.syslog_host:
            try:
                import syslog
            except ImportError:
                    "The syslog module is required to log terminal sessions to "
                    "syslog if no syslog_host is set.  The syslog module is not"
                    " required if you want to send syslog messages to a remote "
                    "syslog server but for this to work you must set the "
                    "syslog_host variable either via the command-line switch or"
                    " in your server.conf."))
                sys.exit(1)
            if not syslog_facility:
                syslog_facility = syslog.LOG_DAEMON
            syslog_facility = syslog_facility
            # Sets up syslog messages to show up like this:
            #   Sep 28 19:45:02 <hostname> gateone: <log message>
            syslog.openlog('gateone', 0, syslog_facility)

    def __repr__(self):
        """
        Returns self.__str__()
        """
        return "<%s>" % self.__str__()

    def __str__(self):
        """
        Returns a string representation of this Multiplex instance and the
        current state of things.
        """
        started = self.started
        if started != "Never":
            started = self.started.isoformat()
        out = (
            "%s.%s:  "
            "term_id: %s, "
            "alive: %s, "
            "command: %s, "
            "started: %s"
            % (
                self.__module__,
                self.__class__.__name__,
                self.term_id,
                self._alive,
                repr(self.cmd),
                started
            )
        )
        return out

        """
        callback at the given event, that callback will be replaced with

        *event* - The numeric ID of the event you're attaching *callback* to (e.g. Multiplex.CALLBACK_UPDATE).
        *callback* - The function you're attaching to the *event*.
        *identifier* - A string or number to be used as a reference point should you wish to remove or update this callback later.

        Returns the identifier of the callback.  to Example:

            >>> m = Multiplex()
            >>> ref = m.add_callback(m.CALLBACK_UPDATE, somefunc, id)

        """

        """
        given *event*.  Example:

            >>> m.remove_callback(m.CALLBACK_BELL, "myref")

        """
        try:
            del self.callbacks[event][identifier]
            pass # Doesn't exist anymore--nothing to do

        """
        """
            try:
                pass # Doesn't exist--nothing to worry about

    def _call_callback(self, callback):
        """
        This method is here in the event that subclasses of `BaseMultiplex` need
        to call callbacks in an implementation-specific way.  It just calls
        """
        callback()

    def spawn(self, rows=24, cols=80, env=None, em_dimensions=None):
        """
        This method must be overridden by suclasses of `BaseMultiplex`.  It is
        expected to execute a child process in a way that allows non-blocking
        reads to be performed.
        """
        raise NotImplementedError(_(
            "spawn() *must* be overridden by subclasses."))

    def isalive(self):
        """
        expected to return True if the child process is still alive and False
        otherwise.
        """
            "isalive() *must* be overridden by subclasses."))

    def term_write(self, stream):
        """
        Writes :obj:`stream` to `BaseMultiplex.term` and also takes care of
        logging to :attr:`log_path` (if set) and/or syslog (if
        :attr:`syslog` is `True`).  When complete, will call any
        callbacks registered in :obj:`CALLBACK_UPDATE`.

        :stream: A string or bytes containing the incoming output stream from the underlying terminal program.

        .. note:: This kind of logging doesn't capture user keystrokes.  This is intentional as we don't want passwords winding up in the logs.
        """
        #logging.debug('term_write() stream: %s' % repr(stream))
        # Write to the log (if configured)
        separator = b"\xf3\xb0\xbc\x8f"
        if self.log_path:
            # Using .encode() below ensures the result will be bytes
            now = str(int(round(time.time() * 1000))).encode('UTF-8')
            if not os.path.exists(self.log_path):
                # Write the first frame as metadata
                metadata = {
                    'version': '1.0', # Log format version
                    'rows': self.rows,
                    'cols': self.cols,
                    'start_date': now.decode('UTF-8') # JSON needs strings
                    # NOTE: end_date should be added later when the is read for
                    # the first time by either the logviewer or the logging
                    # plugin.
                }
                # The hope is that we can use the first-frame-metadata paradigm
                # to store all sorts of useful information about a log.
                # NOTE: Using .encode() below to ensure it is bytes in Python 3
                metadata_frame = json_encode(metadata).encode('UTF-8')
                # Using concatenation of bytes below to ensure compatibility
                # with both Python 2 and Python 3.
                metadata_frame = now + b":" + metadata_frame + separator
                self.log = gzip.open(self.log_path, mode='a')
                self.log.write(metadata_frame)
            if not self.log: # Only comes into play if the file already exists
            # NOTE: I'm using an obscure unicode symbol in order to avoid
            # conflicts.  We need to do our best to ensure that we can
            # differentiate between terminal output and our log format...
            # This should do the trick because it is highly unlikely that
            # someone would be displaying this obscure unicode symbol on an
            # actual terminal unless they were using Gate One to view a
            # Gate One log file in vim or something =)
            # "\xf3\xb0\xbc\x8f" == \U000f0f0f == U+F0F0F (Private Use Symbol)
            output = now + b":" + stream + separator
            self.log.write(output)
        # NOTE: Gate One's log format is special in that it can be used for both
        # playing back recorded sessions *or* generating syslog-like output.
        if self.syslog:
            # Try and keep it as line-line as possible so we don't end up with
            # a log line per character.
            if '\n' in stream:
                for line in stream.splitlines():
                    if self.syslog_buffer:
                        line = self.syslog_buffer + line
                        self.syslog_buffer = ''
                    # Sylog really doesn't like any fancy encodings
                    line = line.encode('ascii', 'xmlcharrefreplace')
                    syslog.syslog("%s %s: %s" % (
                        self.user, self.term_id, line))
            else:
                self.syslog_buffer += stream
        # Handle preprocess patterns (for expect())
        if self._patterns:
            self.preprocess(stream)
        self.term.write(stream)
        # Handle post-process patterns (for expect())
            self.postprocess()
        if self.CALLBACK_UPDATE in self.callbacks:
            for callback in self.callbacks[self.CALLBACK_UPDATE].values():
                self._call_callback(callback)

    def preprocess(self, stream):
        """
        Handles preprocess patterns registered by :meth:`expect`.  That
        is, those patterns which have been marked with `preprocess = True`.
        Patterns marked in this way get handled *before* the terminal emulator
        processes the :obj:`stream`.

        """
        preprocess_patterns = (a for a in self._patterns if a.preprocess)
        finished_non_sticky = False
        # If there aren't any preprocess patterns this won't do anything:
        for pattern_obj in preprocess_patterns:
            if finished_non_sticky and not pattern_obj.sticky:
                # We only want sticky patterns if we've already matched once
                continue
            if isinstance(pattern_obj.pattern, (list, tuple)):
                for pat in pattern_obj.pattern:
                    match = pat.search(term_lines)
                    if match:
                        callback = partial(
                            pattern_obj.callback, self, match.group())
                        self._call_callback(callback)
                        if not pattern_obj.sticky:
                            self.unexpect(hash(pattern_obj)) # Remove it
                        break
            else:
                match = pattern_obj.pattern.search(stream)
                if match:
                    callback = partial(
                        pattern_obj.callback, self, match.group())
                    self._call_callback(callback)
                    if not pattern_obj.sticky:
                        self.unexpect(hash(pattern_obj)) # Remove it
            if not pattern_obj.optional:
                # We only match the first non-optional pattern
                finished_non_sticky = True

    def postprocess(self):
        """
        Handles post-process patterns registered by :meth:`expect`.
        """
        # Check the terminal emulator screen for any matching patterns.
        post_patterns = (a for a in self._patterns if not a.preprocess)
        for pattern_obj in post_patterns:
            # For post-processing matches we search the terminal emulator's
            # screen as a single string.  This allows for full-screen screen
            # scraping in addition to typical 'expect-like' functionality.
            # The big difference being that with traditional expect (and
            # pexpect) you don't get to examine the program's output as it
            # would be rendered in an actual terminal.
            # By using post-processing of the text after it has been handled
            # by a terminal emulator we don't have to worry about hidden
            # characters and escape sequences that we may not be aware of or
            # could make our regular expressions much more complicated than
            # they should be.
                continue # We only want sticky patterns at this point
            # For convenience, trailing whitespace is removed from the lines
            # output from the terminal emulator.  This is so we don't have to
            # put '\w*' before every '$' to match the end of a line.
            term_lines = "\n".join(
                [a.rstrip() for a in self.term.dump()]).rstrip()
                    if match:
                        self._handle_match(pattern_obj, match)
                        break
            else:
                match = pattern_obj.pattern.search(term_lines)
                if match:
                    self._handle_match(pattern_obj, match)
            if not pattern_obj.optional and not pattern_obj.sticky:

    def _handle_match(self, pattern_obj, match):
        """
        Handles a matched regex detected by :meth:`postprocess`.  It calls
        :obj:`Pattern.callback` and takes care of removing it from
        :attr:`_patterns` (if it isn't sticky).
        """
        if self._handling_match:
            # Don't process anything if we're in the middle of handling a match.
            # NOTE: This can happen when there's more than one thread,
            # processes, or PeriodicCallback going on simultaneously.  It seems
            # to work better than threading.Lock()
            return
        self._handling_match = True
        callback = partial(pattern_obj.callback, self, match.group())
        self._call_callback(callback)
        if self.debug:
            # Turn on the fancy regex debugger/pretty printer
            debug_callback = partial(
                debug_expect, self, match.group(), pattern_obj.pattern)
            self._call_callback(debug_callback)
        if not pattern_obj.sticky:
            self.unexpect(hash(pattern_obj)) # Remove it

    def writeline(self, line=''):
        """
        Just like :meth:`write` but it writes a newline after writing *line*.

        If no *line* is given a newline will be written.
        """
        self.write(line + u'\r\n')

    def writelines(self, lines):
        """
        Writes *lines* (a list of strings) to the underlying program, appending
        a newline after each line.
        """
        if getattr(lines, '__iter__', False):
            for line in lines:
                self.write(line + u'\r\n')
        else:
            raise TypeError(_(
                "%s is not iterable (strings don't count :)" % type(lines)))

    def dump_html(self, full=False, client_id='0'):
        """
        Returns the difference of terminal lines (a list of lines, to be
        specific) and its scrollback buffer (which is also a list of lines) as a
        tuple::

            (scrollback, screen)

        If a line hasn't changed since the last dump said line will be replaced
        with an empty string in the output.

        If *full*, will return the entire screen (not just the diff).
        if *client_id* is given (string), this will be used as a unique client
        identifier for keeping track of screen differences (so you can have
        multiple clients getting their own unique diff output for the same
        Multiplex instance).
        """
        if client_id not in self.prev_output:
            self.prev_output[client_id] = [None for a in xrange(self.rows-1)]
        try:
            scrollback, html = ([], [])
            if self.term:
                try:
                    result = self.term.dump_html()
                    if result:
                        scrollback, html = result
                        # Make a copy so we can save it to prev_output later
                        preserved_html = html[:]
                except IOError as e:
                    logging.debug(_("IOError attempting self.term.dump_html()"))
                    logging.debug("%s" % e)
            if html:
                if not full:
                    count = 0
                    for line1, line2 in izip(self.prev_output[client_id], html):
                        if line1 != line2:
                            html[count] = line2 # I love updates-in-place
                        else:
                            html[count] = ''
                        count += 1
                    # Otherwise a full dump will take place
                self.prev_output.update({client_id: preserved_html})
            return (scrollback, html)
            # This would be special...
            logging.error(_("ValueError in dump_html(): %s" % e))
            return ([], [])
        except (IOError, TypeError) as e:
            logging.error(_("Unhandled exception in dump_html(): %s" % e))
            if self.ratelimiter_engaged:
                # Caused by the program being out of control
                return([], [
                    _("<b>Program output too noisy.  Sending Ctrl-c...</b>")])
            else:
                import traceback
                traceback.print_exc(file=sys.stdout)

        """
        Dumps whatever is currently on the screen of the terminal emulator as
        a list of plain strings (so they'll be escaped and look nice in an
        interactive Python interpreter).
        """
        return self.term.dump()

    def timeout_check(self, timeout_now=False):
        """
        Iterates over :attr:`BaseMultiplex._patterns` checking each to
        determine if it has timed out.  If a timeout has occurred for a
        `Pattern` and said Pattern has an *errorback* function that function
        will be called.

        Returns True if there are still non-sticky patterns remaining.  False

        If *timeout_now* is True, will force the first errorback to be called
        and will empty out self._patterns.
        """
        remaining_patterns = False
        for pattern_obj in self._patterns:
            if timeout_now:
                if pattern_obj.errorback:
                    errorback = partial(pattern_obj.errorback, self)
                    self._call_callback(errorback)
                    self.unexpect()
                    return False
            if not pattern_obj.timeout:
                # Timeouts of 0 or None mean "wait forever"
                remaining_patterns = True
                continue
            elapsed = datetime.now() - pattern_obj.created
            if elapsed > pattern_obj.timeout:
                if not pattern_obj.sticky:
                    self.unexpect(hash(pattern_obj))
            elif not pattern_obj.sticky:
        return remaining_patterns

    def expect(self, patterns, callback,
            timeout=15,
            position=None,
            preprocess=False):
        """
        Watches the stream of output coming from the underlying terminal program
        for *patterns* and if there's a match *callback* will be called like so::

            callback(multiplex_instance, matched_string)

        .. tip:: You can provide a string instead of a *callback* function as a shortcut if you just want said string written to the child process.

        *patterns* can be a string, an :class:`re.RegexObject` (as created by
        :func:`re.compile`), or a iterator of either/or.  Returns a reference
        object that can be used to remove the registered pattern/callback at any
        time using the :meth:`unexpect` method (see below).

        .. note::  This function is non-blocking!

        .. warning::  The *timeout* value gets compared against the time :meth:`expect` was called to create it.  So don't wait too long if you're planning on using :meth:`await`!

        Here's a simple example that changes a user's password::

            >>> def write_password(m_instance, matched):
            ...     print("Sending Password... %s patterns remaining." % len(m_instance._patterns))
            ...     m_instance.writeline('somepassword')
            >>> m = Multiplex('passwd someuser') # Assumes running as root :)
            >>> m.expect('(?i)password:', write_password) # Step 1
            >>> m.expect('(?i)password:', write_password) # Step 2
            >>> print(len(m._patterns)) # To show that there's two in the queue
                2
            >>> m.spawn() # Execute the command
            >>> m.await(10) # This will block for up to 10 seconds waiting for self._patterns to be empty (not counting optional patterns)
            Sending Password... 1 patterns remaining.
            Sending Password... 0 patterns remaining.
            >>> m.isalive()
            False
            >>> # All done!

        .. tip:: The :meth:`await` method will automatically call :meth:`spawn` if not :meth:`isalive`.

        This would result in the password of 'someuser' being changed to 'somepassword'.  How is the order determined?  Every time :meth:`expect` is called it creates a new :class:`Pattern` using the given parameters and appends it to `self._patterns` (which is a list).  As each :class:`Pattern` is matched its *callback* gets called and the :class:`Pattern` is removed from `self._patterns` (unless *sticky* is `True`).  So even though the patterns and callbacks listed above were identical they will get executed and removed in the order they were created as each respective :class:`Pattern` is matched.

        .. note:: Only the first pattern, or patterns marked as *sticky* are checked against the incoming stream.  If the first non-sticky pattern is marked *optional* then the proceeding pattern will be checked (and so on).  All other patterns will sit in `self._patterns` until their predecessors are matched/removed.

        Patterns can be removed from `self._patterns` as needed by calling `unexpect(<reference>)`.  Here's an example::

            >>> def handle_accepting_ssh_key(m_instance, matched):
            ...     m_instance.writeline(u'yes')
            >>> m = Multiplex('ssh someuser@somehost')
            >>> ref1 = m.expect('(?i)Are you sure.*\(yes/no\)\?', handle_accepting_ssh_key, optional=True)
            >>> def send_password(m_instance, matched):
            ...    m_instance.unexpect(ref1)
            ...    self.writeline('somepassword')
            >>> ref2 = m.expect('(?i)password:', send_password)
            >>> # spawn() and/or await() and do stuff...

        The example above would send 'yes' if asked by the SSH program to accept
        the host's public key (which would result in it being automatically
        removed from `self._patterns`).  However, if this condition isn't met
        before send_password() is called, send_password() will use the reference
        object to remove it directly.  This ensures that the pattern won't be
        accidentally matched later on in the program's execution.

        .. note:: Even if we didn't match the "Are you sure..." pattern it would still get auto-removed after its timeout was reached.

        **About pattern ordering:** The position at which the given pattern will
        be inserted in `self._patterns` can be specified via the
        *position* argument.  The default is to simply append which should be
        appropriate in most cases.

        **About Timeouts:** The *timeout* value passed to expect() will be used
        to determine how long to wait before the pattern is removed from
        self._patterns.  When this occurs, *errorback* will be called with
        current Multiplex instance as the only argument.  If *errorback* is None
        (the default) the pattern will simply be discarded with no action taken.

        .. note:: If *sticky* is True the *timeout* value will be ignored.

        **Notes about the length of what will be matched:**  The entire terminal
        'screen' will be searched every time new output is read from the
        incoming stream.  This means that the number of rows and columns of the
        terminal determines the size of the search.  So if your pattern needs to
        look for something inside of 50 lines of text you need to make sure that
        when you call `spawn` you specify at least `rows = 50`.  Example::

            >>> def handle_long_search(m_instance, matched)
            ...     do_stuff(matched)
            >>> m = Multiplex('someCommandWithLotsOfOutput.sh')
            >>> # 'begin', at least one non-newline char, 50 newlines, at least one char, then 'end':
            >>> my_regex = re.compile('begin.+[\\n]{50}.+end', re.MULTILINE)
            >>> ref = m.expect(my_regex, handle_accepting_ssh_key)
            >>> m.spawn(rows=51, cols=150)
            >>> # Call m.read(), m.spawn() or just let an event loop (e.g. Tornado's IOLoop) take care of things...

        **About non-printable characters:** If the *postprocess* argument is
        True (the default), patterns will be checked against the current screen as
        output by the terminal emulator.  This means that things like control
        codes and escape sequences will be handled and discarded by the terminal
        emulator and as such won't be available for patterns to be checked
        against.  To get around this limitation you can set *preprocess* to True
        and the pattern will be checked against the incoming stream before it is
        processed by the terminal emulator.  Example::

            >>> def handle_xterm_title(m_instance, matched)
            ...     print("Caught title: %s" % matched)
            >>> m = Multiplex('echo -e "\\033]0;Some Title\\007"')
            >>> title_seq_regex = re.compile(r'\\x1b\\][0-2]\;(.*?)(\\x07|\\x1b\\\\)')
            >>> m.expect(title_seq_regex, handle_xterm_title, preprocess=True) # <-- 'preprocess=True'
            >>> m.await()
            Caught title: Some Title
            >>>

        **Notes about debugging:** Instead of using `await` to wait for all of your patterns to be matched at once you can make individual calls to `read` to determine if your patterns are being matched in the way that you want.  For example::

            >>> def do_stuff(m_instance, matched):
            ...     print("Debug: do_stuff() got %s" % repr(matched))
            ...     # Do stuff here
            >>> m = Multiplex('someLongComplicatedOutput.sh')
            >>> m.expect('some pattern', do_stuff)
            >>> m.expect('some other pattern', do_stuff)
            >>> m.spawn()
            >>> # Instead of calling await() just call one read() at a time...
            >>> print(repr(m.read()))
            ''
            >>> print(repr(m.read())) # Oops, called read() too soon.  Try again:
            'some other pattern'
            >>> # Doh!  Looks like 'some other pattern' comes first.  Let's start over...
            >>> m.unexpect() # Called with no arguments, it empties m._patterns
            >>> m.terminate() # Tip: This will call unexpect() too so the line above really isn't necessary
            >>> m.expect('some other pattern', do_stuff) # This time this one will be first
            >>> print(repr(m.read())) # This time I waited a moment :)
            'Debug: do_stuff() got "some other pattern"'
            >>> # Huzzah!  Now let's see if 'some pattern' matches...
            'Debug: do_stuff() got "some pattern"'
            'some pattern'
            >>> # As you can see, calling read() at-will in an interactive interpreter can be very handy.

        **About asynchronous use:**  This mechanism is non-blocking (with the exception of `await`) and is meant to be used asynchronously.  This means that if the running program has no output, `read` won't result in any patterns being matched.  So you must be careful about timing *or* you need to ensure that `read` gets called either automatically when there's data to be read (IOLoop, EPoll, select, etc) or at regular intervals via a loop.  Also, if you're not calling `read` at an interval (i.e. you're using a mechanism to detect when there's output to be read before calling it e.g. IOLoop) you need to ensure that `timeout_check` is called regularly anyway or timeouts won't get detected if there's no output from the underlying program.  See the `MultiplexPOSIXIOLoop.read` override for an example of what this means and how to do it.
        """
        # Create the Pattern object before we do anything else
        if isinstance(patterns, (str, unicode)):
            # Convert to a compiled regex (assume MULTILINE for the sanity of
            # the ignorant)
            patterns = re.compile(patterns, re.MULTILINE)
        if isinstance(patterns, (tuple, list)):
            # Ensure that all patterns are RegexObjects
            pattern_list = []
            for pattern in patterns:
                if isinstance(pattern, str):
                    pattern = re.compile(pattern)
                    pattern_list.append(pattern)
                else:
            patterns = tuple(pattern_list) # No reason to keep it as a list
        # Convert timeout to a timedelta if necessary
        if timeout: # 0 or None mean "wait forever"
            if isinstance(timeout, (str, int, float)):
                timeout = timedelta(seconds=float(timeout))
            elif not isinstance(timeout, timedelta):
                raise TypeError(_(
                    "The timeout value must be a string, integer, float, or a "
                    "timedelta object"))
        pattern_obj = Pattern(patterns, callback,
            optional=optional,
            sticky=sticky,
            errorback=errorback,
            preprocess=preprocess,
            timeout=timeout)
        if not position:
            self._patterns.append(pattern_obj)
        else:
            self._patterns.insert(position, pattern_obj)
        return hash(pattern_obj)

    def unexpect(self, ref=None):
        """
        Removes *ref* from self._patterns so it will no longer be checked
        against the incoming stream.  If *ref* is None (the default),
        `self._patterns` will be emptied.
        """
        if not ref:
            self._patterns = [] # Reset
            return
        for i, item in enumerate(self._patterns):
            if hash(item) == ref:
                self._patterns.pop(i)

    def await(self, timeout=15, rows=24, cols=80, env=None, em_dimensions=None):
        """
        Blocks until all non-optional patterns inside self._patterns have been
        removed *or* if the given *timeout* is reached.  *timeout* may be an
        integer (in seconds) or a `datetime.timedelta` object.

        Returns True if all non-optional, non-sticky patterns were handled
        successfully.

        .. warning:: The timeouts attached to Patterns are set when they are created.  Not when when you call :meth:`await`!

        As a convenience, if :meth:`isalive` resolves to False,
        :meth:`spawn` will be called automatically with *rows*, *cols*,
        and *env* given as arguments.

        await
            To wait with expectation.
        """
        if not self.isalive():
            self.spawn(
                rows=rows, cols=cols, env=env, em_dimensions=em_dimensions)
        start = datetime.now()
        if isinstance(timeout, (str, int, float)):
            timeout = timedelta(seconds=float(timeout))
        elif not isinstance(timeout, timedelta):
                "The timeout value must be a string, integer, float, or a "
                "timedelta object"))
        remaining_patterns = True
        # This starts up the scheduler that constantly checks patterns
        self.read() # Remember:  read() is non-blocking
        while remaining_patterns:
            # First we need to discount optional patterns
            remaining_patterns = False
            for pattern in self._patterns:
                if not pattern.optional and not pattern.sticky:
                    remaining_patterns = True
                    break
            # Now check if we've timed out
            if (datetime.now() - start) > timeout:
                raise Timeout("Lingered longer than %s" % timeout.seconds)
            # Lastly we perform a read() to ensure the output is processed
            self.read() # Remember:  read() is non-blocking
            time.sleep(0.01) # So we don't eat up all the CPU

    def terminate(self):
        """
        expected to terminate/kill the child process.
        """
            "terminate() *must* be overridden by subclasses."))

    def _read(self, bytes=-1):
        """
        This method must be overridden by subclasses of `BaseMultiplex`.  It is
        expected that this method read the output from the running terminal
        program in a non-blocking way, pass the result into `term_write`, and
        then return the result.
        """
            "_read() *must* be overridden by subclasses."))

    def read(self, bytes=-1):
        """
        Calls `_read` and checks if any timeouts have been reached in
        `self._patterns`.  Returns the result of `_read`.
        """
        result = self._read(bytes)
        # Perform checks for timeouts in self._patterns (used by self.expect())
        self.timeout_check()
        return result

    def write(self):
            "write() *must* be overridden by subclasses."))

class MultiplexPOSIXIOLoop(BaseMultiplex):
    """
    The MultiplexPOSIXIOLoop class takes care of executing a child process on
    POSIX (aka Unix) systems and keeping track of its state via a terminal
    emulator (`terminal.Terminal` by default).  If there's a started instance
    of :class:`tornado.ioloop.IOLoop`, handlers will be added to it that
    automatically keep the terminal emulator synchronized with the output of the
    child process.

    If there's no IOLoop (or it just isn't started), terminal applications can
    be interacted with by calling `MultiplexPOSIXIOLoop.read` (to write any
    pending output to the terminal emulator) and `MultiplexPOSIXIOLoop.write`
    (which writes directly to stdin of the child).

    .. note:: `MultiplexPOSIXIOLoop.read` is non-blocking.
    """
    def __init__(self, *args, **kwargs):
        super(MultiplexPOSIXIOLoop, self).__init__(*args, **kwargs)
        from tornado import ioloop
        self.terminating = False
        self.sent_sigint = False
        self.env = {}
        self.io_loop = ioloop.IOLoop.instance() # Monitors child for activity
        #self.io_loop.set_blocking_signal_threshold(2, self._blocked_io_handler)
        #signal.signal(signal.SIGALRM, self._blocked_io_handler)
        self.reenable_timeout = None
        interval = 100 # A 0.1 second interval should be fast enough
        self.scheduler = ioloop.PeriodicCallback(self._timeout_checker,interval)
        self.exitstatus = None
        self._checking_patterns = False

    def __del__(self):
        """
        Makes sure that the underlying terminal program is terminated so we
        don't leave things hanging around.
        """
        logging.debug("MultiplexPOSIXIOLoop.__del__()")
        self.terminate()

        """
        If the IOLoop is started, adds the callback via
        :meth:`IOLoop.add_callback` to ensure it gets called at the next IOLoop
        iteration (which is thread safe).  If the IOLoop isn't started
        *callback* will get called immediately and directly.
        """
        try:
            if self.io_loop.running():
                self.io_loop.add_callback(callback)
            else:
        finally:
            del callback

    def _reenable_output(self):
        """
        Restarts capturing output from the underlying terminal program by
        disengaging the rate limiter.
        """

    def __reset_sent_sigint(self):

    def _blocked_io_handler(self, signum=None, frame=None):
        """
        Handles the situation where a terminal is blocking IO (usually because
        of too much output).  This method would typically get called inside of
        `MultiplexPOSIXIOLoop._read` when the output of an fd is too noisy.
        """
            # This can happen if terminate() gets called too fast from another
            # thread...  Strange stuff, mixing threading, signals, and
            # multiprocessing!
            "Noisy process (%s) kicked off rate limiter." % self.pid))
        self.ratelimiter_engaged = True
        # CALLBACK_UPDATE is called here so the client can be made aware of the
        # fact that the rate limiter was engaged.
        for callback in self.callbacks[self.CALLBACK_UPDATE].values():
            self._call_callback(callback)
        self.reenable_timeout = self.io_loop.add_timeout(
            timedelta(seconds=10), self._reenable_output)

    def spawn(self,
            rows=24, cols=80, env=None, em_dimensions=None, exitfunc=None):
        """
        Creates a new virtual terminal (tty) and executes self.cmd within it.
        Also attaches :meth:`self._ioloop_read_handler` to the IOLoop so that
        the terminal emulator will automatically stay in sync with the output of
        the child process.

        :cols: The number of columns to emulate on the virtual terminal (width)
        :rows: The number of rows to emulate (height).
        :env: Optional - A dictionary of environment variables to set when executing self.cmd.
        :em_dimensions: Optional - The dimensions of a single character within the terminal (only used when calculating the number of rows/cols images take up).
        :exitfunc: Optional - A function that will be called with the current Multiplex instance and its exit status when the child process terminates (*exitfunc(m_instance, statuscode)*).
        """
        self.started = datetime.now()
        #signal.signal(signal.SIGCHLD, signal.SIG_IGN) # No zombies allowed
            "spawn(rows=%s, cols=%s, env=%s, em_dimensions=%s)" % (
                rows, cols, repr(env), repr(em_dimensions)))
        rows = min(200, rows) # Max 200 to limit memory utilization
        cols = min(500, cols) # Max 500 for the same reason
        import pty
        pid, fd = pty.fork()
        if pid == 0: # We're inside the child process
    # Close all file descriptors other than stdin, stdout, and stderr (0, 1, 2)
            try:
                # This ensures that the child doesn't get the parent's FDs
                os.closerange(3, 256)
                pass
            if not env:
                env = {}
            env["COLUMNS"] = str(cols)
            env["LINES"] = str(rows)
            env["TERM"] = "xterm-256color" # TODO: Needs to be configurable
            env["PATH"] = os.environ['PATH']
            env["LANG"] = os.environ.get('LANG', 'en_US.UTF-8')
            env["PYTHONIOENCODING"] = "utf_8"
            # Setup stdout to be more Gate One friendly
            import termios
            # Fix missing termios.IUTF8
            if 'IUTF8' not in termios.__dict__:
                termios.IUTF8 = 16384 # Hopefully not platform independent
            stdin = 0
            stdout = 1
            stderr = 2
            attrs = termios.tcgetattr(stdout)
            iflag, oflag, cflag, lflag, ispeed, ospeed, cc = attrs
            # Enable flow control and UTF-8 input (probably not needed)
            iflag |= (termios.IXON | termios.IXOFF | termios.IUTF8)
            # OPOST: Enable post-processing of chars (not sure if this matters)
            # INLCR: We're disabling this so we don't get \r\r\n anywhere
            oflag |= (termios.OPOST | termios.ONLCR | termios.INLCR)
            attrs = [iflag, oflag, cflag, lflag, ispeed, ospeed, cc]
            termios.tcsetattr(stdout, termios.TCSANOW, attrs)
            # Now do the same for stdin
            attrs = termios.tcgetattr(stdin)
            termios.tcsetattr(stdin, termios.TCSANOW, attrs)
            # The sleep statement below ensures we capture all output from the
            # fd before it is closed...  It turns out that IOLoop's response to
            # changes in the fd is so fast that it can result in the fd being
            # closed the very moment the Python interpreter is reading from it.
            cmd = ['/bin/sh', '-c', self.cmd + '; sleep .1']
            os.dup2(2, 1) # Copy stderr to stdout (equivalent to 2>&1)
            os.execvpe(cmd[0], cmd, env)
            os._exit(0)
        else: # We're inside this Python script
            logging.debug("spawn() pid: %s" % pid)
            self._alive = True
            self.fd = fd
            self.env = env
            self.exitfunc = exitfunc
            self.pid = pid
            self.time = time.time()
            try:
                self.term = self.terminal_emulator(
                    rows=rows,
                    cols=cols,
                    em_dimensions=em_dimensions
                )
                # Terminal emulator doesn't support em_dimensions.  That's OK
                    cols=cols
                )
            # Tell our IOLoop instance to start watching the child
            self.io_loop.add_handler(
                fd, self._ioloop_read_handler, self.io_loop.READ)
            self.prev_output = {}
            # Set non-blocking so we don't wait forever for a read()
            import fcntl
            fl = fcntl.fcntl(sys.stdin, fcntl.F_GETFL)
            fcntl.fcntl(self.fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
            # Set the size of the terminal
            resize = partial(self.resize, rows, cols, ctrl_l=False)
            self.io_loop.add_timeout(timedelta(seconds=2), resize)
            return fd

        """
        Checks the underlying process to see if it is alive and sets self._alive
        appropriately.
        """
        if self.exitstatus == None:
            # This doesn't hurt anything if the process is still running
            if self.pid != -1:
                try:
                    pid, status = os.waitpid(self.pid, os.WNOHANG)
                    if pid: # pid is 0 if the process is still running
                        self.exitstatus = os.WEXITSTATUS(status)
                    # This can happen if the program closes itself very quickly
                    # immediately after being executed.
                    if self.debug: # Useful to know when debugging...
                        print(_(
                            "Could not determine exit status for child with "
                            "PID: %s\n" % self.pid
                        ))
                        print(_("Setting self.exitstatus to 99"))
                    self.exitstatus = 99 # Seems like a good number
        if self._alive: # Re-check it
            try:
                os.kill(self.pid, 0)
                return self._alive
                # Process is dead
                self._alive = False
                if self.debug: # Useful to know when debugging...
                    print(_("Child exited with status: %s\n" % self.exitstatus))
                # Call the exitfunc (if set)
                if self.exitfunc:
                    self.exitfunc(self, self.exitstatus)
                    self.exitfunc = None # Just in case
                return False
        else:
            return False

    def resize(self, rows, cols, em_dimensions=None, ctrl_l=True):
        """
        Resizes the child process's terminal window to *rows* and *cols* by
        first sending it a TIOCSWINSZ event and then sending ctrl-l.

        If *em_dimensions* are provided they will be updated along with the
        rows and cols.

        The sending of ctrl-l can be disabled by setting *ctrl_l* to False.
        """
        logging.debug("Resizing term %s to rows: %s, cols: %s" % (
            self.term_id, rows, cols))
        if rows < 2:
            rows = 24
        if cols < 2:
            cols = 80
        self.term.resize(rows, cols, em_dimensions)
        # Sometimes the resize doesn't actually apply (for whatever reason)
        # so to get around this we have to send a different value than the
        # actual value we want then send our actual value.  It's a bug outside
        # of Gate One that I have no idea how to isolate but this has proven to
        # be an effective workaround.
        import fcntl, termios
        s = struct.pack("HHHH", rows, cols, 0, 0)
        try:
            fcntl.ioctl(self.fd, termios.TIOCSWINSZ, s)
            # Process already ended--no big deal
            return
        if ctrl_l:
            self.write(u'\x0c') # ctrl-l
        # SIGWINCH has been disabled since it can screw things up
        #os.kill(self.pid, signal.SIGWINCH) # Send the resize signal

        """
        Kill the child process associated with `self.fd`.

        .. note:: If dtach is being used this only kills the dtach process.
        """
        if not self.terminating:
            self.terminating = True
        else:
            return # Something else already called it
        logging.debug("terminate() self.pid: %s" % self.pid)
        if self.reenable_timeout:
            self.io_loop.remove_timeout(self.reenable_timeout)
        # Unset our blocked IO handler so there's no references to self hanging
        # around preventing us from freeing up memory
        try:
            self.io_loop.set_blocking_signal_threshold(None, None)
            pass # Can happen if this instance winds up in a thread
        for callback in self.callbacks[self.CALLBACK_EXIT].values():
        # This try/except block *must* come before the exitfunc logic.
        # Otherwise, if the registered exitfunc raises an exception the IOLoop
        # will never stop watching self.fd; resulting in an infinite loop of
        # exitfunc.
        try:
            self.io_loop.remove_handler(self.fd)
            os.close(self.fd)
        except (KeyError, IOError, OSError):
            # This can happen when the fd is removed by the underlying process
            # before the next cycle of the IOLoop.  Not really a problem.
            pass
        self.scheduler.stop()
        # NOTE: Without this 'del' we end up with a memory leak every time
        # a new instance of Multiplex is created.  Apparently the references
        # inside of PeriodicCallback pointing to self prevent proper garbage
        # collection.
        del self.scheduler
        try:
            # TODO: Make this walk the series from SIGINT to SIGKILL
            import signal
            #os.kill(self.pid, signal.SIGINT)
            os.kill(self.pid, signal.SIGTERM)
            #os.kill(self.pid, signal.SIGKILL)
            # The process is already dead--great.
            pass
        if not self.exitstatus:
            pid, status = os.waitpid(-1, os.WNOHANG)
            if pid:
                self.exitstatus = os.WEXITSTATUS(status)
            self.timeout_check(timeout_now=True)
            self.unexpect()
        # Call the exitfunc (if set)
        if self.exitfunc:
            self.exitfunc(self, self.exitstatus)
            self.exitfunc = None
        # Reset all callbacks so there's nothing to prevent GC
        }
        # Commented this out so that you can see what was in the terminal
        # emulator after the process terminates.
        #del self.term
        # Kick off a process that finalizes the log (updates metadata and
        # recompresses everything to save disk space)
        if not self.log_path:
            return # No log to finalize so we're done.
        self.log.close() # Write it out
            "Finalizing the log for pid %s (this can take some time)."
            % self.pid
        ))
        PROC = Process(
            target=get_or_update_metadata,
            args=(self.log_path, self.user),
            kwargs={'force_update': True})
        PROC.start()

    def _ioloop_read_handler(self, fd, event):
        """
        Read in the output of the process associated with *fd* and write it to
        `self.term`.

        :fd: The file descriptor of the child process.
        :event: An IOLoop event (e.g. IOLoop.READ).

        .. note:: This method is not meant to be called directly...  The IOLoop should be the one calling it when it detects any given event on the fd.
        """
        if event == self.io_loop.READ:
            self._call_callback(self.read)
        else: # Child died
                "Apparently fd %s just died (event: %s)" % (self.fd, event)))
            self.terminate()

        """
        Reads at most *bytes* from the incoming stream, writes the result to
        the terminal emulator using `term_write`, and returns what was read.
        If *bytes* is -1 (default) it will read `self.fd` until there's no more
        output.

        Returns the result of all that reading.

        .. note:: Non-blocking.
        """
        #logging.debug("MultiplexPOSIXIOLoop._read()")
        result = b""
        try:
            with io.open(self.fd, 'rb', closefd=False, buffering=0) as reader:
                if bytes == -1:
                    # Even though -1 indicates "read everything" we still
                    # want to read in chunks so we can catch when an fd is
                    # feeding us too much data (so we can engage the rate
                    # limiter).
                    bytes = 8192 # Should be plenty
                    # If we need to block/read for longer than five seconds
                    # the fd is outputting too much data.
                    five_seconds = timedelta(seconds=5)
                    loop_start = datetime.now()
                    while True:
                        updated = reader.read(bytes)
                        if not updated:
                            break
                        if self.ratelimiter_engaged:
                            # Truncate the output to the last 1024 chars
                            self.term_write(updated[-1024:])
                            result += updated[-1024:]
                            # NOTE: If we didn't truncate the output we'd
                            # eventually have to process it all which would
                            # take forever.
                            break
                        elif datetime.now() - loop_start > five_seconds:
                            self._blocked_io_handler()
                        result += updated
                        self.term_write(updated)
                elif bytes:
                    result = reader.read(bytes)
                    self.term_write(result)
        except IOError as e:
            # IOErrors can happen when self.fd is closed before we finish
            # reading from it.  Not a big deal.
            pass
            logging.error("Got exception in read: %s" % `e`)
            import traceback
            logging.error(
                "Got unhandled exception in read (???): %s" % `e`)
            traceback.print_exc(file=sys.stdout)
            if self.isalive():
                self.terminate()
        finally:
            return result

    def _timeout_checker(self):
        """
        Runs `timeout_check` and if there are no more non-sticky
        patterns in :attr:`self._patterns`, stops :attr:`scheduler`.
        """
        if not self._checking_patterns:
            self._checking_patterns = True
            remaining_patterns = self.timeout_check()
            if not remaining_patterns:
                # No reason to keep the PeriodicCallback going
                logging.debug("Stopping self.scheduler (no remaining patterns)")
                try:
                    self.scheduler.stop()
                except AttributeError:
                # Now this is a neat trick:  The way IOLoop works with its
                # stack_context thingamabob the scheduler doesn't actualy end up
                # inside the MultiplexPOSIXIOLoop instance inside of this
                # instance of _timeout_checker() *except* inside the main
                # thread.  It is absolutely wacky but it works and works well :)
                    pass
            self._checking_patterns = False

        """
        .. note:: This is an override of `BaseMultiplex.read` in order to take advantage of the IOLoop for ensuring `BaseMultiplex.expect` patterns timeout properly.

        Calls `_read` and checks if any timeouts have been reached
        in :attr:`self._patterns`.  Returns the result of :meth:`_read`.  This
        is an override of `BaseMultiplex.read` that will create a
        :class:`tornado.ioloop.PeriodicCallback` (as `self.scheduler`) that
        executes :attr:`timeout_check` at a regular interval.  The
        `PeriodicCallback` will automatically cancel itself if there are no more
        non-sticky patterns in :attr:`self._patterns`.
        """
        remaining_patterns = self.timeout_check()
        if remaining_patterns and not self.scheduler._running:
            # Start 'er up in case we don't get any more output
            logging.debug("Starting self.scheduler to check for timeouts")
            self.scheduler.start()
        self.isalive() # This just ensures the exitfunc is called (if necessary)

    def _write(self, chars):
        """
        Writes *chars* to `self.fd` (pretty straightforward).  If IOError or
        OSError exceptions are encountered, will run `terminate`.  All other
        exceptions are logged but no action will be taken.
        """
        #logging.debug("MultiplexPOSIXIOLoop._write()")
        try:
                # This empties the fd buffer so the user gets immediate
                # responses to their keystrokes.  Just note that this can
                # truncating a lot of data.
                with io.open(
                    self.fd, 'rb', closefd=False, buffering=0) as reader:
                        reader.read() # Empty it out
                # In testing, emptying out the fd read buffer in this way
                # increased responsiveness to user keystrokes when the rate
                # limiter is engaged by about 3-4 seconds (which is a lot!)
                self.ratelimiter_engaged = False
                # For reference, the time between reader.read() above and
                # writer.write() below is enough for the 'yes' command to
                # output a few lines.
            with io.open(
                self.fd, 'wt', encoding='UTF-8', closefd=False) as writer:
                    writer.write(chars)
        except (IOError, OSError) as e:
            logging.error("write() exception: %s" % e)

    def write(self, chars):
        """
        Calls `_write(*chars*)` via `_call_callback` to ensure thread safety.
        """
            raise ProgramTerminated(_("Child process is not running."))
        write = partial(self._write, chars)
        self._call_callback(write)

# Here's an example of how termio compares to pexpect:
#import pexpect
#child = pexpect.spawn ('ftp ftp.openbsd.org')
#child.expect ('Name .*: ')
#child.sendline ('anonymous')
#child.expect ('Password:')
#child.sendline ('noah@example.com')
#child.expect ('ftp> ')
#child.sendline ('cd pub')
#child.expect('ftp> ')
#child.sendline ('get ls-lR.gz')
#child.sendline ('bye')
# NOTE: Every expect() in the above example is a blocking call.

# This is the same thing, rewritten using termio:
#import termio
#child = termio.Multiplex('ftp ftp.openbsd.org', debug=True)
## Expectations come first
#child.expect('Name .*:', "anonymous\n")
#child.expect('Password:', 'user@company.com\n')
#child.expect('ftp>$', 'cd pub\n')
#child.expect('ftp>$', 'get ls-lR.gz\n')
#child.expect('ftp>$', 'bye\n')
#child.await() # Blocks until all patterns have been matched or a timeout
# NOTE: If this code were called inside of an already-started IOLoop there would
# be no need to call await(). Everything would be asynchronous and non-blocking.

def spawn(cmd, rows=24, cols=80, env=None, em_dimensions=None, *args, **kwargs):
    """
    A shortcut to::

        >>> m = Multiplex(cmd, *args, **kwargs)
        >>> m.spawn(rows, cols, env)
        >>> return m
    """
    m = Multiplex(cmd, *args, **kwargs)
    m.spawn(rows, cols, env, em_dimensions=em_dimensions)
    return m

def getstatusoutput(cmd, **kwargs):
    """
    Emulates Python's commands.getstatusoutput() function using a Multiplex
    instance.

    Optionally, any additional keyword arguments (\*\*kwargs) provided will be
    passed to the spawn() command.
    """
    # NOTE: This function is primarily here to provide an example of how to use
    # termio.Multiplex instances in a traditional, blocking manner.
    output = ""
    m = Multiplex(cmd)
    m.spawn(**kwargs)
    while m.isalive():
        output += m.read()
    return (m.exitstatus, output)

if POSIX:
    Multiplex = MultiplexPOSIXIOLoop
else:
    raise NotImplementedError(_(
        "termio currently only works on Unix platforms."))

#
#

Hooks
-----

    hooks = {
        'Web': [
            (r"/bookmarks/fetchicon", FaviconHandler),
            (r"/bookmarks/export", ExportHandler),
            (r"/bookmarks/import", ImportHandler),
        ],
            'bookmarks_sync': save_bookmarks,
            'bookmarks_get': get_bookmarks,
            'bookmarks_deleted': delete_bookmarks,
            'bookmarks_rename_tags': rename_tags,
        }
    }

"""

# Meta

import os, sys, logging, time

# Our stuff
from utils import noop, json_encode

# 3rd party stuff
# The following two lines let us import modules in the "dependencies" dir
sys.path.append(os.path.join(plugin_path, "dependencies"))

# Globals
boolean_fix = {
    True: True,
    False: False,
    'True': True,
    'False': False,
    'true': True,
    'false': False
}

def unescape(s):
    """
    Unescape HTML code refs; c.f. http://wiki.python.org/moin/EscapingHtml
    """
    import re
    from htmlentitydefs import name2codepoint
    # Fix the missing one:
    name2codepoint['#39'] = 39
    return re.sub('&(%s);' % '|'.join(name2codepoint), lambda m: unichr(name2codepoint[m.group(1)]), s)

def parse_bookmarks_html(html):
    """
    Reads the Netscape-style bookmarks.html in string, *html* and returns a
    list of Bookmark objects.
    """
    # If this looks impossibly complicated it's because parsing HTML streams is
    # dark voodoo.  I had to push my brains back behind my eyes and into my ears
    # a few times while writing this.
    import html5lib
    p = html5lib.HTMLParser(tree=html5lib.treebuilders.getTreeBuilder("dom"))
    dom_tree = p.parse(html)
    walker = html5lib.treewalkers.getTreeWalker("dom")
    stream = walker(dom_tree)
    level = 0
    tags = []
    h3on = False
    aon = False
    ddon = False
    add_date = None
    url = None
    icon = None
    name = ""
    notes = ""
    for token in stream:
        if 'name' in token:
            if token['name'] == 'dl':
                if token['type'] == 'StartTag':
                    level += 1
                elif token['type'] == 'EndTag':
                    if tags:
                        tags.pop()
                    level -= 1
            if token['name'] == 'dd':
                    ddon = True
                    ddon = False
            if token['name'] == 'h3':
                    h3on = True
                    h3on = False
            if token['name'] == 'a':
                    aon = True
                    aon = False
                    if not add_date: # JavaScript-style 13-digit epoch:
                        add_date = int(round(time.time() * 1000))
                    add_date = int(add_date)
                    if add_date > 9999999999999: # Delicious goes out to 16
                        add_date = int(add_date/1000)
                    if add_date < 10000000000: # Chrome only goes to 10 digits
                        add_date = int(add_date*1000)
                    bm = {
                        'url': url,
                        'name': name.strip(),
                        'tags': [a for a in tags if a], # Remove empty tags
                        'notes': "", # notes
                        'visits': 0, # visits
                        'updated': add_date, # updated
                        'created': add_date, # created
                        'updateSequenceNum': 0, # updateSequenceNum
                        'images': {'favicon': icon}
                    }
                    out_list.append(bm)
                    # Reset everything (just in case)
                    add_date = None
                    url = None
                    icon = None
                    name = ""
        if h3on:
            if token['data']:
                if type(token['data']) == str:
                    tags.append(token['data'])
                elif type(token['data']) == unicode:
        if ddon: # Indicates that there's notes here
                if token['type'] == 'Characters':
                    # Notes get attached to the bookmark we just created
                    out_list[-1]['notes'] = unescape(token['data'].strip())
        if aon:
            if token['type'] == 'StartTag':
                # html5lib changed from using lists to using dicts at some point
                # after 0.90.  Hence the two conditionals below
                if isinstance(token['data'], list):
                    for tup in token['data']:
                        if tup[0] == 'add_date':
                            add_date = tup[1]
                        elif tup[0] == 'href':
                            url = tup[1]
                        elif tup[0] == 'icon':
                            icon = tup[1]
                        elif tup[0] == 'tags':
                            tags = tup[1].split(',') # Delicious-style
                elif isinstance(token['data'], dict):
                        if 'add_date' in tup:
                            add_date = token['data'][tup]
                        elif 'href' in tup:
                            url = token['data'][tup]
                        elif 'icon' in tup:
                            icon = token['data'][tup]
                        elif 'tags' in tup:
                            tags = token['data'][tup].split(',') # Delicious
            elif token['type'] == 'Characters':
                name += unescape(token['data'])

def get_json_tags(json_dict, url):
    """
    Recursively looks inside *json_dict* trying to find tags associated with the
    given *url*.  Returns the tags found as a list.
    """
    tags = []
    # This function has been brought to you by your favorite stock symbol
    if json_dict.has_key('root') and json_dict.has_key('children'):
        for item in json_dict['children']:
            if item['title'] == 'Tags':
                for child in item['children']:
                    if child['type'] == 'text/x-moz-place-container':
                        for subchild in child['children']:
                            if subchild['type'] == 'text/x-moz-place':
                                if subchild['uri'] == url:
                                    tags.append(child['title'])
                                        # "Ahhhhhhh"
                                            # "hhhhhhhh"
                                                # "hhhhhhh"
                                                    # "hhhhhh"
                                                        # "hhhhh"
                                                            # "!!!"
                                                                # <splat>
    return tags

def get_ns_json_bookmarks(json_dict, bookmarks):
    """
    Given a *json_dict*, updates *urls_list* with each URL as it is found
    within.

    .. note:: Only works with Netscape-style bookmarks.json files.
    """
    children = []
    if json_dict.has_key('children'):
        for child in json_dict['children']:
            if child['type'] == 'text/x-moz-place':
                if not bookmarks[0].has_key(child['uri']):
                    # Browser won't let you load file: URIs from HTTP pages
                    if child['uri'][0:6] not in ['place:', 'file:/']:
                        # Note the use of json_dict as bookmarks[1] here:
                        tags = get_json_tags(bookmarks[1], child['uri'])
                        if not tags:
                            tags = ['Untagged']
                        if child.has_key("annos"):
                            notes = child["annos"]
                        else:
                            notes = ""
                        if child['lastModified'] > 9999999999999:
                            # Chop off the microseconds to make it 13 digits
                            child['lastModified'] = int(child['lastModified']/1000)
                        elif child['lastModified'] < 10000000000:
                            child['lastModified'] = int(child['lastModified']*1000)
                        if child['dateAdded'] > 9999999999999: # Delicious
                            child['dateAdded'] = int(child['dateAdded']/1000)
                        elif child['dateAdded'] < 10000000000: # Chrome
                            child['dateAdded'] = int(child['dateAdded']*1000)
                        bm = {
                            'url': child['uri'],
                            'name': child['title'].strip(),
                            'tags': tags,
                            'notes': notes,
                            'visits': 0, # visits
                            'updated': child['lastModified'], # updated
                            'created': child['dateAdded'], # created
                            'updateSequenceNum': 0, # updateSequenceNum
                            'images': {} # No icons in JSON :(
                        }
                        bookmarks[0].update({child['uri']: bm})
            elif child['type'] == 'text/x-moz-place-container':
                get_ns_json_bookmarks(child, bookmarks)

def parse_bookmarks_json(json_str):
    """
    Given *json_str*, returns a list of bookmark objects representing the data
    contained therein.
    """
    # TODO: Get this recognizing and parsing our own JSON format.
    json_obj = json.loads(json_str)
    bookmarks = [{}, json_obj] # Inside a list for persistence
    get_ns_json_bookmarks(json_obj, bookmarks) # Updates urls in-place
    for url, bm in bookmarks[0].items():
        out_list.append(bm)

# Data Structures
class BookmarksDB(object):
    """
    Used to read and write bookmarks to a file on disk.  Can also synchronize
    a given list of bookmarks with what's on disk.  Uses a given bookmark's
    ``updateSequenceNum`` to track what wins the "who is newer?" comparison.
    """
    def __init__(self, user_dir, user):
        """
        Sets up our bookmarks database object and reads everything in.
        """
        self.bookmarks = [] # For temp storage of all bookmarks
        self.user_dir = user_dir
        users_dir = os.path.join(user_dir, user) # "User's dir"
        self.bookmarks_path = os.path.join(users_dir, "bookmarks.json")
        # Read existing bookmarks into self.bookmarks
        self.open_bookmarks()

    def open_bookmarks(self):
        """
        Opens the bookmarks stored in self.user_dir.  If not present, an
        empty file will be created.
        """
        if not os.path.exists(self.bookmarks_path):
            with open(self.bookmarks_path, 'w') as f:
                f.write('[]') # That's an empty JSON list
            return # Default of empty list will do
        with open(self.bookmarks_path) as f:
            self.bookmarks = json_decode(f.read())

    def save_bookmarks(self):
        """
        Saves self.bookmarks to self.bookmarks_path as a JSON-encoded list.
        """
        with open(self.bookmarks_path, 'w') as f:
            f.write(json_encode(self.bookmarks))

    def sync_bookmarks(self, bookmarks):
        """
        Given *bookmarks*, synchronize with self.bookmarks doing conflict
        resolution and whatnot.
        """
        highest_USN = self.get_highest_USN()
        changed = False # For if there's changes that need to be written
        updated_bookmarks = [] # For bookmarks that are newer on the server
        for bm in bookmarks:
            if bm['url'] == "web+deleted:bookmarks/":
                # Remove the existing deleted entry if it exists
                for j, deleted_bm in enumerate(bm['notes']):
                    if deleted_bm['url'] == bookmark['url']:
                        # Remove the deleted bookmark entry
                        bm['notes'].pop(j)
            found_existing = False
            for i, db_bookmark in enumerate(self.bookmarks):
                if bm['url'] == db_bookmark['url']:
                    # Bookmark already exists, check which is newer
                    found_existing = True
                    if bm['updateSequenceNum'] > db_bookmark['updateSequenceNum']:
                        # The given bookmark is newer than what's in the DB
                        self.bookmarks[i] = bm # Replace it
                        highest_USN += 1 # Increment the USN
                        self.bookmarks[i]['updateSequenceNum'] = highest_USN
                        changed = True
                    elif bm['updateSequenceNum'] < db_bookmark['updateSequenceNum']:
                        # DB has a newer bookmark.  Add it to the list to send
                        # to the client.
                        updated_bookmarks.append(db_bookmark)
                    # Otherwise the USNs are equal and there's nothing to do
            if not found_existing:
                # This is a new bookmark.  Add it
                highest_USN += 1 # Increment the USN
                bm['updateSequenceNum'] = highest_USN
                self.bookmarks.append(bm)
                changed = True # So it will be saved
            # Write the changes to disk
            self.save_bookmarks()
        # Let the client know what's newer on the server
        return updated_bookmarks

    def delete_bookmark(self, bookmark):
        """Deletes the given *bookmark*."""
        for i, db_bookmark in enumerate(self.bookmarks):
            if bookmark['url'] == db_bookmark['url']:
                # Remove it
                deleted = self.bookmarks.pop(i)
                # Add it to the list of deleted bookmarks
                special_deleted_bm = None
                for bm in self.bookmarks:
                    if bm['url'] == "web+deleted:bookmarks/":
                        special_deleted_bm = bm
                # The deleted bookmarks 'bookmark' is just a list of URLs that
                # have been deleted along with the time it happened.  This lets
                # us keep multiple browsers in sync with what's been deleted
                # so we don't inadvertently end up re-adding bookmarks that were
                # deleted by another client.
                if not special_deleted_bm:
                    # Make our first entry
                    special_deleted_bm = {
                        'url': "web+deleted:bookmarks/",
                        'name': "Deleted Bookmarks",
                        'tags': [],
                        'notes': [bookmark],
                        'visits': highest_USN + 1,
                        'updated': int(round(time.time() * 1000)),
                        'created': int(round(time.time() * 1000)),
                        'updateSequenceNum': 0,
                        'images': {}
                    }
                    self.bookmarks.append(special_deleted_bm)
                else:
                    # Check for pre-existing
                    updated = False
                    for j, deleted_bm in enumerate(special_deleted_bm['notes']):
                        if deleted_bm['url'] == bookmark['url']:
                            # Update it in place
                            special_deleted_bm['notes'][j] = bookmark
                            updated = True
                    if not updated:
                        special_deleted_bm['notes'].append(bookmark)
                    highest_USN += 1
                    special_deleted_bm['updateSequenceNum'] = highest_USN
                break
        # Save the change to disk
        self.save_bookmarks()

    def get_bookmarks(self, updateSequenceNum=0):
        """
        Returns a list of bookmarks newer than *updateSequenceNum*.
        If *updateSequenceNum* is 0 or undefined, all bookmarks will be
        returned.
        """
        out_bookmarks = []
        for bm in self.bookmarks:
            if bm['updateSequenceNum'] > updateSequenceNum:
                out_bookmarks.append(bm)
        return out_bookmarks

    def get_highest_USN(self):
        """Returns the highest updateSequenceNum in self.bookmarks"""
        highest_USN = 0
            if bm['updateSequenceNum'] > highest_USN:
                highest_USN = bm['updateSequenceNum']
        return highest_USN

    def rename_tag(self, old_tag, new_tag):
        """
        Goes through all bookmarks and renames all tags named *old_tag* to be
        *new_tag*.
        """
            if old_tag in bm['tags']:
                highest_USN += 1
                i = bm['tags'].index(old_tag)
                bm['tags'][i] = new_tag
                # Made a change so we need to increment the USN to ensure sync
                bm['updated'] = int(round(time.time() * 1000))

# Handlers
class FaviconHandler(BaseHandler):
    """
    Retrives the biggest favicon-like icon at the given URL.  It will try to
    fetch apple-touch-icons (which can be nice and big) before it falls back
    to grabbing the favicon.

    .. note:: Works with GET and POST requests but POST is preferred since it keeps the URL from winding up in the server logs.
    """
    # Valid favicon mime types
    favicon_mimetypes = [
        'image/vnd.microsoft.icon',
        'image/x-icon',
        'image/png',
        'image/svg+xml',
        'image/gif',
        'image/jpeg'
    ]
        self.process()

    def process(self):
        url = self.get_argument("url")
        http = tornado.httpclient.AsyncHTTPClient()
        callback = partial(self.on_response, url)
        http.fetch(url, callback, connect_timeout=5.0, request_timeout=5.0)

    def get_favicon_url(self, html):
        """
        Parses *html* looking for a favicon URL.  Returns a tuple of::

            (<url>, <mimetime>)

        If no favicon can be found, returns::

            (None, None)
        """
        import html5lib
        p = html5lib.HTMLParser(
            tree=html5lib.treebuilders.getTreeBuilder("dom"))
        dom_tree = p.parse(html)
        walker = html5lib.treewalkers.getTreeWalker("dom")
        stream = walker(dom_tree)
        fetch_url = None
        mimetype = None
        icon = False
        found_token = None
        for token in stream:
            if 'name' in token:
                if token['name'] == 'link':
                    for attr in token['data']:
                        if attr[0] == 'rel':
                            if 'shortcut icon' in attr[1].lower():
                                found_token = token
                                icon = True
                        elif attr[0] == 'href':
                            fetch_url = attr[1]
                        elif attr[0] == 'type':
                            mimetype = attr[1]
                    if fetch_url and icon:
                        if not mimetype:
                            mimetype = "image/x-icon"
                        if mimetype in self.favicon_mimetypes:
                            return (fetch_url, mimetype)
        return (None, None)

    def on_response(self, url, response):
        try:
            from urlparse import urlparse
        except ImportError: # Python 3.X
            from urllib import parse as urlparse
        if response.error:
            self.write('Unable to fetch icon.')
            return
        try:
            content = response.body.decode('utf-8')
            content = response.body
        parsed_url = urlparse(url)
        (fetch_url, mimetype) = self.get_favicon_url(content)
        if fetch_url:
            if not fetch_url.startswith('http'):
                fetch_url = '%s://%s%s' % (
                    parsed_url.scheme, parsed_url.netloc, fetch_url)
        if not mimetype:
            mimetype = "image/x-icon" # Default
        if not fetch_url:
            fetch_url = '%s://%s/favicon.ico' % (
                parsed_url.scheme, parsed_url.netloc)
        if fetch_url.startswith('http://') or fetch_url.startswith('https://'):
            noop()
        else:
            raise tornado.web.HTTPError(404)
        callback = partial(self.icon_fetch, url, mimetype)
        try:
            http.fetch(
                fetch_url,
                callback,
                connect_timeout=5.0,
                request_timeout=5.0
            )
        except gaierror: # No address associated with hostname
            return

    def icon_multifetch(self, urls, response):
        """
        Fetches the icon at the given URLs, stopping when it finds the biggest.
        If an icon is not found, calls itself again with the next icon URL.
        If the icon is found, writes it to the client and finishes the request.
        """
            if urls:
                url = urls.pop()
                http = tornado.httpclient.AsyncHTTPClient()
                callback = partial(self.icon_multifetch, urls)
                try:
                    http.fetch(url, callback)
                except gaierror:
                    raise tornado.web.HTTPError(404)
            else:
                raise tornado.web.HTTPError(404)
        else:
            if 'Content-Type' in response.headers:
                mimetype = response.headers['Content-Type']
                self.set_header("Content-Type", mimetype)
            else:
                mimetype = "image/vnd.microsoft.icon"
            data_uri = "data:%s;base64,%s" % (
                mimetype,
                response.body.encode('base64').replace('\n', '')
            )
            self.write(data_uri)

    def icon_fetch(self, url, mimetype, response):
        """Returns the fetched icon to the client."""
            return
        data_uri = "data:%s;base64,%s" % (
            mimetype,
            response.body.encode('base64').replace('\n', '')
        )
        self.set_header("Content-Type", mimetype)
        self.write(data_uri)

class ImportHandler(tornado.web.RequestHandler):
    """
    Takes a bookmarks.html in a POST and returns a list of bookmarks in JSON
    format
    """
        html = self.request.body
        if html.startswith('{'): # This is a JSON file
            bookmarks = parse_bookmarks_json(html)
        else:
            bookmarks = parse_bookmarks_html(html)
        self.write(tornado.escape.json_encode(bookmarks))
        # NOTE: The client will take care of storing these at the next sync

class ExportHandler(tornado.web.RequestHandler):
    """
    Takes a JSON-encoded list of bookmarks and returns a Netscape-style HTML
    file.
    """
        bookmarks = self.get_argument("bookmarks")
        bookmarks = tornado.escape.json_decode(bookmarks)
        self.set_header("Content-Type", "text/html")
        self.set_header(
            "Content-Disposition", 'attachment; filename="bookmarks.html"')
        templates_path = os.path.join(plugin_path, "templates")
        bookmarks_html =  os.path.join(templates_path, "bookmarks.html")
        self.render(bookmarks_html, bookmarks=bookmarks)

def save_bookmarks(bookmarks, tws):
    """
    Handles saving *bookmarks* for clients.
    """
        'updates': [],
        'count': 0,
        'errors': []
    }
    try:
        user = tws.get_current_user()['upn']
        bookmarks_db = BookmarksDB(tws.settings['user_dir'], user)
        updates = bookmarks_db.sync_bookmarks(bookmarks)
        out_dict.update({
            'updates': updates,
            'count': len(bookmarks),
        })
        out_dict['updateSequenceNum'] = bookmarks_db.get_highest_USN()
        logging.error("Got exception synchronizing bookmarks: %s" % e)
        out_dict['errors'].append(str(e))
    if out_dict['errors']:
        out_dict['result'] = "Upload completed but errors were encountered."
    else:
        out_dict['result'] = "Upload successful"
    message = {'bookmarks_save_result': out_dict}

def get_bookmarks(updateSequenceNum, tws):
    """
    Returns a JSON-encoded list of bookmarks updated since the last
    *updateSequenceNum*.

    If *updateSequenceNum* resolves to False, all bookmarks will be sent to
    the client.
    """
    bookmarks_db = BookmarksDB(tws.settings['user_dir'], user)
    if updateSequenceNum:
        updateSequenceNum = int(updateSequenceNum)
    else: # This will force a full download
        updateSequenceNum = 0
    updated_bookmarks = bookmarks_db.get_bookmarks(updateSequenceNum)
    message = {'bookmarks_updated': updated_bookmarks}

def delete_bookmarks(deleted_bookmarks, tws):
    """
    Handles deleting bookmars given a *deleted_bookmarks* list.
    """
        'errors': [],
    }
    try:
        for bookmark in deleted_bookmarks:
            out_dict['count'] += 1
            bookmarks_db.delete_bookmark(bookmark)
    except Exception as e: # TODO: Make this more specific
        logging.error("delete_bookmarks error: %s" % e)
        out_dict['result'] = "Errors"
    message = {'bookmarks_delete_result': out_dict}

def rename_tags(renamed_tags, tws):
    """
    Handles renaming tags.
    """
        'updates': []
    }
    for pair in renamed_tags:
        old_name, new_name = pair.split(',')
        bookmarks_db.rename_tag(old_name, new_name)
        out_dict['count'] += 1
    message = {'bookmarks_renamed_tags': out_dict}

hooks = {
    'Web': [
    ],
    }
}

#
#

.. tip:: This plugin is heavily commented with useful information.  Click on the `[source]` links to the right of any given class or function to see the actual code.

Hooks
-----

    hooks = {
        'Web': [(r"/example", ExampleHandler)],
            'example_action': example_websocket_action
        },
        'Escape': example_opt_esc_handler,
        #'Auth': create_user_ssh_dir
    }

"""

# Meta information about the plugin.  Your plugin doesn't *have* to have this
# but it is a good idea.
__license__ = "Apache 2.0" # The "just don't sue me" license

# I like to start my files with imports from Python's standard library...
import os

# Then I like to import things from Gate One itself or my own stuff...
#from gateone import GATEONE_DIR # <--if you need that path it's here

# This is where I'd typically put 3rd party imports such as Tornado...

# Globals
# This is in case we have relative imports, templates, or whatever:
plugin_path = os.path.split(__file__)[0] # Path to our plugin's directory

# Traditional web handler
class ExampleHandler(BaseHandler):
    """
    This is how you add a URL handler to Gate One...  This example attaches
    itslef to https://<your Gate One server>/example in the 'Web' hook at the
    bottom of this file.  It works just like any Tornado
    :class:`~tornado.web.RequestHandler`.  See:

    http://www.tornadoweb.org/documentation/web.html

    ...for documentation on how to write a :class:`tornado.web.RequestHandler` for
    the Tornado framework.  Fairly boilerplate stuff.

    .. note:: The only reason we use :class:`gateone.BaseHandler` instead of a vanilla :class:`tornado.web.RequestHandler` is so we have access to Gate One's :func:`gateone.BaseHandler.get_current_user` function.
    """
    @tornado.web.authenticated # Require the user be authenticated
        """
        Handle an HTTP GET request to this :class:`~tornado.web.RequestHandler`.
        Connect to:

        https://<your Gate One server>/example

        ...to try it out.
        """
        # This is all that's in example_template.html (so you don't have to open
        # it separately):
        #   <html><head><title>{{bygolly}}, it works!</title></head>
        #   <body>
        #   <p>You're logged in as: {{user}}.</p>
        #   <p>Your session ID ends with {{session}}.</p>
        #   </body>
        #   </html>
        #
        # NOTE: I highly recommend using os.path.join() instead of just using
        # '/' everywhere...  You never know; Gate One might run on Windows one
        # day!
        example_template =  os.path.join(templates_path, "example_template.html")
        bygolly = "By golly"
        # The get_current_user() function returns a whole dict of information.
        # What's available in there is dependent on which authentication type
        # you're using but you can be assured that 'upn' and 'session' will
        # always be present.
        user_dict = self.get_current_user()
        # Gate One refers to the username as a userPrincipalName (like Kerberos)
        # or 'upn' for short.  Why?  Because it might actually be a username
        # plus a realm or domain name.  e.g. user@REALM or user@company.com
        username = user_dict['upn']
        session = user_dict['session'][:3] # Just the last three (for security)
            example_template, # The path to a template file
            bygolly=bygolly, # Just match up your template's {{whatever}} with
            user=username,   # the keyword arguments passed to self.render()
            session=session)

        """
        Example Handler for an `HTTP POST <http://en.wikipedia.org/wiki/POST_(HTTP)>`_
        request.  Doesn't actually do anything.
        """
        # If data is POSTed to this handler via an XMLHTTPRequest send() it
        # will show up like this:
        posted_as_a_whole = self.request.body # xhr.send()
        # If data was POSTed as arguments (i.e. traditional form) it will show
        # up as individual arguments like this:
        posted_as_argument = self.get_argument("arg") # Form elem 'name="arg"'
        # This is how you can parse JSON:
        parsed = tornado.escape.json_decode(posted_as_an_argument)
        # For writing JSON it is recommended to use the json_encode() function
        # from Gate One's utils.py (since it takes care of Python 3 support):
        from utils import json_encode
        json_output = {'result': 'Success!'}
        self.write(json_encode(json_output))
        # You'd put self.finish() here if post() was wrapped with tornado's
        # asynchronous decorator.

# WebSocket actions (aka commands or "functions that are exposed")
def example_websocket_action(message, tws=None):
    """
    This `WebSocket <https://developer.mozilla.org/en/WebSockets/WebSockets_reference/WebSocket>`_
    action gets exposed to the client automatically by way of the 'WebSocket'
    hook at the bottom of this file.  The way it works is like this:

    .. rubric:: How The WebSocket Hook Works

    Whenever a message is received via the `WebSocket <https://developer.mozilla.org/en/WebSockets/WebSockets_reference/WebSocket>`_
    Gate One will automatically
    decode it into a Python :class:`dict` (only JSON-encoded messages are accepted).
    Any and all keys in that :class:`dict` will be assumed to be 'actions' (just
    like :js:attr:`GateOne.Net.actions` but on the server) such as this one.  If
    the incoming key matches a registered action that action will be called
    like so::

        key(value)
        # ...or just:
        key() # If the value is None ('null' in JavaScript)

    ...where *key* is the action and *value* is what will be passed to said
    action as an argument.  Since Gate One will automatically decode the message
    as JSON the *value* will typically be passed to actions as a single :class:`dict`.
    You can provide different kinds of arguments of course but be aware that
    their ordering is unpredictable so always be sure to either pass *one*
    argument to your function (assuming it is a :class:`dict`) or 100% keyword
    arguments.

    The *tws* keyword argument must always be present in WebSocket actions.  It
    represents the user's current instance of Gate One's :class:`~gateone.TerminalWebSocket`
    class. Think of it as the equivalent of, "self" inside of any given
    :class:`~gateone.TerminalWebSocket` function.  For example, the following
    function inside of :class:`~gateone.TerminalWebSocket` (inside of gateone.py)::

        def pong(self, timestamp): # Docstring removed to save space
            message = {'pong': timestamp}

    ...could be written inside of a plugin like so::

        def pong(timestamp, tws=None):
            tws.write_message(json_encode(message))

    .. note:: Notice that the ussage of *self* has become *tws*.

    The typical naming convention for `WebSocket <https://developer.mozilla.org/en/WebSockets/WebSockets_reference/WebSocket>`_
    actions is: `<plugin name>_<action>`.  Whether or not your action names
    match your function names is up to you.  All that matters is that you line
    up an *action* (string) with a *function* in `hooks['WebSocket']` (see below).

    *action* duplicates the functionality of Gate One's built-in
    :func:`gateone.TerminalWebSocket.pong` function.  You can see how it is
    called by the client (browser) inside of example.js (which is in this
    plugin's 'static' dir).
    """
    message = {'example_pong': timestamp}
    # WebSockets are asynchronous so you can send as many messages as you want
    message2 = {'notice': 'You just executed the "example_action" action.'}
    tws.write_message(json_encode(message2))
    # Alternatively, you can combine multiple messages/actions into one message:
    combined = {
        'notice': 'Hurray!',
        'bell': {'term': tws.current_term}
    }
    tws.write_message(json_encode(combined))

# Now for some special sauce...  The Special Optional Escape Sequence Handler!
def example_opt_esc_handler(message):
    """
    Gate One includes a mechanism for plugins to send messages from terminal
    programs directly to plugins written in Python.  It's called the "Special
    Optional Escape Sequence Handler" or SOESH for short.  Here's how it works:
    Whenever a terminal program emits, "\\x1b]_;" it gets detected by Gate One's
    :class:`~terminal.Terminal` class (which lives in `terminal.py`) and it will execute whatever
    callback is registered for SOESH.  Inside of Gate One this callback will
    always be :func:`gateone.TerminalWebSocket.esc_opt_handler`.
    """
# SOESH allows plugins to attach actions that will be called whenever a terminal
# encounters the

# Without this 'hooks' dict your plugin might as well not exist from Gate One's
# perspective.
hooks = {
    },
}

toggle_alternate_screen_buffer_cursor() (terminal.Terminal method)

GateOne.terminals[num].X11Title (GateOne.terminals[num] attribute)

GateOne.terminals[num].backspace (GateOne.terminals[num] attribute)

GateOne.terminals[num].columns (GateOne.terminals[num] attribute)

GateOne.terminals[num].created (GateOne.terminals[num] attribute)

GateOne.terminals[num].pasteNode (GateOne.terminals[num] attribute)

GateOne.terminals[num].playbackFrames (GateOne.terminals[num] attribute)

GateOne.terminals[num].prevScreen (GateOne.terminals[num] attribute)

GateOne.terminals[num].screen (GateOne.terminals[num] attribute)

GateOne.terminals[num].screenNode (GateOne.terminals[num] attribute)

GateOne.terminals[num].scrollback (GateOne.terminals[num] attribute)

GateOne.terminals[num].scrollbackNode (GateOne.terminals[num] attribute)

GateOne.terminals[num].scrollbackVisible (GateOne.terminals[num] attribute)

GateOne.terminals[num].scrollbackWriteTimer (GateOne.terminals[num] attribute)

GateOne.terminals[num].sshConnectString (GateOne.terminals[num] attribute)

GateOne.Terminal.closeTermCallbacks (GateOne.Terminal attribute)

GateOne.Terminal.updateTermCallbacks (GateOne.Terminal attribute)

GateOne.Terminal.loadWebWorkerAction() (GateOne.Terminal method)

GateOne.Terminal.reattachTerminalsAction() (GateOne.Terminal method)

GateOne.Terminal.reconnectTerminalAction() (GateOne.Terminal method)

GateOne.Terminal.registerTextTransform() (GateOne.Terminal method)

GateOne.Terminal.resetTerminalAction() (GateOne.Terminal method)

GateOne.Terminal.termUpdateFromWorker() (GateOne.Terminal method)

GateOne.Terminal.unregisterTextTransform() (GateOne.Terminal method)

GateOne.Terminal.updateTerminalAction() (GateOne.Terminal method)

command line option; --api_timestamp_window=string (special: [0-9]+[smhd])

command line option; --auth=string (none|pam|google|kerberos|api)

command line option; --origins=string (semicolon-separated origins)

command line option; --session_timeout=string (special: [0-9]+[smhd])

command line option; --sso_realm=string (Kerberos realm or Active Directory domain)

command line option; --sso_service=string (kerberos service name)

command line option; --syslog_facility=string (auth|cron|daemon|kern|local0|local1|local2|local3|local4|local5|local6|local7|lpr|mail|news|syslog|user|uucp)