Execution API
=============

nbexec adds an ``/api/executions`` resource to the Jupyter notebook RESTful web API.

.. _execution-model:

Execution Model
---------------

nbexec creates an 'Execution' object on executing a notebook. 'Execution Model' is a JSON representation of Execution which has the following properties. The properties marked with an asterisk (*) are always included.

.. list-table::
  :widths: 20, 10, 70
  :header-rows: 1

  * - Name
    - Type
    - Description
  * - exec_id *
    - str
    - Execution ID (UUID).
  * - path *
    - str
    - Notebook file path relative to the Jupyter notebook's root directory.
  * - params *
    - object
    - Runtime parameters given from an API request.
      The values are stored as string instead of defined types in a notebook.
  * - output_path
    - str
    - Output file path. It is ``null`` if ``output_path`` parameter was not specified
      and the execution is not finished yet.
      It is also ``null`` if the execution is failed to start
      (e.g. the notebook is broken, the file is not a notebook).
  * - overwrite *
    - bool
    - Whether to overwrite the existing output file.
  * - jupyter_kernel
    - str
    - Jupyter kernel name.
  * - cell_timeout
    - int
    - Cell timeout in seconds.
  * - status *
    - str
    - Execution status. 'initializing', 'executing', 'completed' or 'error: <msg>'
  * - progress
    - str
    - Execution progress which indicates the current executing cell number
      and the total number of cells (e.g. '5/15'). It is ``null`` if the execution is not started yet.
  * - last_cell_source
    - str
    - The source code of the last started cell. It is ``null`` if any cell has not yet been execute.
  * - started_at
    - float
    - Timestamp of the started time. It is ``null`` if the execution is not started yet.
  * - completed_at
    - float
    - Timestamp of the completed time. It is ``null`` if the execution is not completed yet.

Execution Life Cycle
--------------------

Executions are stored in a in-memory DB by Jupyter notebook server same as notebook sessions. An execution object is created just before the notebook execution and remains in the DB until it is manually deleted. If you create many executions, Jupyter notebook server consumes much memory. It is recommended to delete unused executions periodically or just after they finished.

When Jupyter notebook server shuts down, execution data will be lost. They are not persisted beyond Jupyter notebook server's life time.

API Spec
--------

Common Parameters
~~~~~~~~~~~~~~~~~

All API requests must have ``token``. See :ref:`authentication`.

Sets all parameters into a request body using URL-encoded form data (``application/x-www-form-urlencoded``)
if the HTTP method is ``POST``, ``PUT`` or ``PATCH``; otherwise sets all parameters as a query string into the URL.

Error Response
~~~~~~~~~~~~~~
The error response will depend on the following HTTP status codes:

HTTP status codes
^^^^^^^^^^^^^^^^^

.. list-table::
  :widths: 30, 70
  :header-rows: 1

  * - Status code
    - Description
  * - 400 Bad Request
    - The request was not processed, as the server could not understand what the client is asking for.
  * - 401 Unauthorized
    - The request lacks valid credentials and it should re-request with the required credentials.
  * - 404 Not Found
    - The requested resource was not found.
  * - 500 Internal Server Error
    - The request is valid, but the server could not fulfill it due to some unexpected condition.

The error message is nothing more than reference information for the error code.

GET /api/executions
~~~~~~~~~~~~~~~~~~~
Gets the list of execution models.

Response body (JSON)
^^^^^^^^^^^^^^^^^^^^

.. list-table::
  :widths: 20, 10, 70
  :header-rows: 1

  * - Key
    - Type
    - Description
  * - executions
    - list
    - The list of :ref:`execution models<execution-model>`.

HTTP status codes
^^^^^^^^^^^^^^^^^

.. list-table::
  :widths: 30, 75
  :header-rows: 1

  * - Status code
    - Description
  * - 200 OK
    - The request has succeeded. The list of execution models is in the request body.
  * - 401 Unauthorized
    - Unauthorized request.

POST /api/executions
~~~~~~~~~~~~~~~~~~~~
Executes the notebook with given parameters with creating an Execution object.
The working directory of the execution is same as the parent directory of the notebook.

Parameters
^^^^^^^^^^

The parameters except below are passed as notebook parameters. The parameters marked with an asterisk (*) are required.

.. list-table::
  :widths: 20, 80
  :header-rows: 1

  * - name
    - Description
  * - notebook *
    - * Notebook file path to be executed.
      * The path is relative to the Jupyter notebook’s root directory.
  * - output_path
    - * Output notebook file path.
      * The path is relative to the Jupyter notebook’s root directory.
      * If it is not specified, it is going to be ``<path>-Executed1.ipynb``.
  * - overwrite
    - * Whether to overwrite the output notebook file.
      * ``output_path`` must be set if it is ``true``.
      * If it is ``false``, the file name is incremented such as ``<path>-Executed2.ipynb``.
  * - jupyter_kernel
    - * Jupyter kernel name to execute the notebook.
      * Only IPython kernel is available.
      * If it is not specified, the default kernel of the Jupyter notebook is used.
  * - cell_timeout
    - * Timeout for each cell in seconds.
      * When a cell takes more time than it, the whole execution will be terminated immediately.
      * If it is not specified, cells are never timed out.

Response bosy (JSON)
^^^^^^^^^^^^^^^^^^^^

.. list-table::
  :widths: 20, 10, 70
  :header-rows: 1

  * - Key
    - Type
    - Description
  * - event
    - str
    - Name of the event
  * - timestamp
    - float
    - Time when the event happened
  * - execution
    - object
    - An :ref:`execution model<execution-model>`.

HTTP status codes
^^^^^^^^^^^^^^^^^

.. list-table::
  :widths: 30, 75
  :header-rows: 1

  * - Status code
    - Description
  * - 202 Accepted
    - The execution is started. It sending progress payloads using chunked transfer encoding
      while the execution is completed if ``X-Response-Encoding: chunked`` is set;
      otherwise only ``notebook_start`` event is sent and the execution is run asynchronously.
  * - 400 Bad Request
    - Invalid parameters are specified.
  * - 401 Unauthorized
    - Unauthorized request.
  * - 404 Not Found
    - The notebook file is not found.

.. note::

    * It launches a new process to run the notebook on a Jupyter kernel for each call.
    * It uses 5 random ports which are selected from unused ports to launch Jupyter kernel.

DELETE /api/executions
~~~~~~~~~~~~~~~~~~~~~~
Deletes all executions with shutting down the IPython kernels. This API requires no parameter and returns an empty response body.

HTTP status codes
^^^^^^^^^^^^^^^^^

.. list-table::
  :widths: 30, 75
  :header-rows: 1

  * - Status code
    - Description
  * - 202 Accepted
    - The request has been accepted. It waits deletion is completed
      if ``X-Response-Encoding: chunked`` is set;
      otherwise the deletion is run asynchronously.
  * - 401 Unauthorized
    - Unauthorized request.

GET /api/executions/<exec_id>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Gets the execution model of the given exec_id.

Response bosy (JSON)
^^^^^^^^^^^^^^^^^^^^

.. list-table::
  :widths: 20, 10, 70
  :header-rows: 1

  * - Key
    - Type
    - Description
  * - execution
    - object
    - An execution model.

HTTP status codes
^^^^^^^^^^^^^^^^^

.. list-table::
  :widths: 30, 75
  :header-rows: 1

  * - Status code
    - Description
  * - 200 OK
    - The request has succeeded. The execution model is in the request body.
  * - 401 Unauthorized
    - Unauthorized request.

POST /api/executions/<exec_id>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Executes an action to the given execution. The only available action is ``shutdown``. 

Parameters
^^^^^^^^^^

Parameters marked with an asterisk (*) are always included.

.. list-table::
  :widths: 20, 80
  :header-rows: 1

  * - name
    - Description
  * - action *
    - * The action to be executed.
      * The only available action is ``shutdown`` which shuts down the IPython kernel which is used by the execution.

Response bosy (JSON)
^^^^^^^^^^^^^^^^^^^^

This API returns an empty response body. If ``X-Response-Encoding: chunked`` is set in the request header, this API returns the following response body.

.. list-table::
  :widths: 20, 10, 70
  :header-rows: 1

  * - Key
    - Type
    - Description
  * - execution
    - object
    - An execution model.

HTTP status codes
^^^^^^^^^^^^^^^^^

.. list-table::
  :widths: 30, 75
  :header-rows: 1

  * - Status code
    - Description
  * - 202 Accepted
    - The request has been accepted. It waits the operation is completed
      if ``X-Response-Encoding: chunked`` is set;
      otherwise the operation is run asynchronously.
  * - 400 Bad Request
    - Invalid parameters are specified.
  * - 401 Unauthorized
    - Unauthorized request.

DELETE /api/executions/<exec_id>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Deletes the given execution with shutting down the IPython kernel if it is still alive. This API requires no parameter and returns an empty response body.

HTTP status codes
^^^^^^^^^^^^^^^^^

.. list-table::
  :widths: 30, 75
  :header-rows: 1

  * - Status code
    - Description
  * - 202 Accepted
    - The request has been accepted. It waits deletion is completed
      if ``X-Response-Encoding: chunked`` is set;
      otherwise the deletion is run asynchronously.
  * - 401 Unauthorized
    - Unauthorized request.

Progress Payload
----------------

All payloads has this common structure:

.. code-block:: none

   {
     "event": "<event_name>",
     "timestamp": <timestamp>,
     ...
   }

``event`` is a name of the event, and ``timestamp`` is a time when the event happened. The timestamp value is a float that ``datetime.datetime.timestamp()`` returns.
The top-level object can have other key-values, which depend on the event type.

Events
~~~~~~
* ``notebook_start``: the notebook started
* ``notebook_complete``: the notebook completes all cells
* ``start``: A cell started
* ``end``: A cell ended
* ``notebook_error``: the notebook ended with error

Notebook Events
~~~~~~~~~~~~~~~
``notebook_start`` and ``notebook_complete`` events have ``execution`` which has an :ref:`execution-model` as its value.

Cell Events
~~~~~~~~~~~
``start`` and ``end`` events have ``progress`` and ``cell``.
``progress`` indicates the current notebook progress in format ``<current_cell_num>/<total_cell_num>``. ``cell`` is the same object which exists in ``.ipynb`` file.

Examples
~~~~~~~~

``notebook_start``:

.. code-block:: json

    {
      "event": "notebook_start",
      "timestamp": 1510621481.601824,
      "execution": {
        "exec_id": "f5022ad6-ad17-453f-9d0a-7d7f5bc019c9",
        "path": "/home/user/examples/Example1.ipynb",
        "params": {},
        "output_path": null,
        "overwrite": false,
        "jupyter_kernel": null,
        "cell_timeout": null,
        "status": "executing",
        "progress": null,
        "last_cell_source": null,
        "started_at": 1510621481.601824,
        "completed_at": null
      }
    }

``notebook_complete``:

.. code-block:: json

    {
      "event": "notebook_complete",
      "timestamp": 1510621487.121837,
      "execution": {
        "exec_id": "f5022ad6-ad17-453f-9d0a-7d7f5bc019c9",
        "path": "/home/user/examples/Example1.ipynb",
        "params": {},
        "output_path": "/home/user/examples/Example1-Executed1.ipynb",
        "overwrite": false,
        "jupyter_kernel": null,
        "cell_timeout": null,
        "status": "completed",
        "progress": "10/10",
        "last_cell_source": "1 + 10",
        "started_at": 1510621481.601824,
        "completed_at": 1510621487.121837
      }
    }

``start``:

.. code-block:: json

    {
      "event": "start",
      "timestamp": 1510621482.63849,
      "progress": "1/10",
      "cell": {
        "outputs": [],
        "cell_type": "code",
        "metadata": {
          "nbexec": {
            "start_time": "2017-11-14T01:04:42.638490+00:00"
          },
          "collapsed": true
        },
        "execution_count": 1,
        "source": "1 + 2"
      }
    }

``end``:

.. code-block:: json

    {
      "event": "end",
      "timestamp": 1510621482.650049,
      "progress": "1/10",
      "cell": {
        "outputs": [
          {
            "output_type": "execute_result",
            "data": {
              "text/plain": "3"
            },
            "execution_count": 1,
            "metadata": {}
          }
        ],
        "cell_type": "code",
        "metadata": {
          "nbexec": {
            "end_time": "2017-11-14T01:04:42.650049+00:00",
            "start_time": "2017-11-14T01:04:42.638490+00:00",
            "duration": 0.011559
          },
          "collapsed": true
        },
        "execution_count": 1,
        "source": "1 + 2"
      }
    }

``notebook_error``:

.. code-block:: json

    {
      "event": "notebook_error",
      "timestamp": 1518170320.507459,
      "output_path": "/home/user/examples/Error-Executed1.ipynb",
      "error": "An error occurred while executing the following cell:\n------------------\nraise ValueError()\n------------------\n\nValueError: \n"
    }

Output File
-----------

The output file name of 'Example.ipynb' is by default 'Example-Executed1.ipynb' and the last number is incremented if the same file name already exists.

.. _authentication:

Authentication
--------------

The execution API always requires ``token`` parameter (Jupyter token) and does not use Cookies.

Limitations
-----------
* ``POST /api/executions`` synchronously and ``GET /api/executions`` might rarely return the following error:

  .. code-block:: json

      {
        "error": "Kernel died before replying to kernel_info",
        "event": "notebook_error",
        "exec_id": "18e3be0a-6300-4204-af2c-fb9c129c7fa9",
        "output_path": "sample-Executed567.ipynb",
        "timestamp": 1518570812.971526
      }

  It causes the Jupyter kernel failed to open unused ports on startup because they are used by another process before Jupyter kernel open them.
  Please call ``POST /api/executions`` to execute the notebook again if it is caused.
  You may limit the concurrent calls of ``POST /api/executions`` because the too many concurrent
  calls of ``POST /api/executions`` increase the possibility of this error.
  See also `this issue <https://github.com/jupyter/jupyter_client/issues/347>`_ for more details.
