Application Configuration#

The JupyterHub Outpost uses a configuration file outpost_config.py similar to jupyterhub_config.py of JupyterHub. The Spawner configuration for the Outpost is therefore similar to the Spawner configuration in JupyterHub. The easiest way is configure the Outpost’s configuration file is via the outpostConfig key in the helm chart’s values.yaml file.

Persistent database#

To use a persistent database such as postgresql with JupyterHub Outpost, use extraEnvVarsSecrets in your values.yaml file. All possible values related to the database connection can be found in the source code itself.

Ensure that you have a database such postgres installed and that a JupyterHub Outpost user and database exists.

Example SQL commands for postgresql:

CREATE USER jupyterhuboutpost WITH ENCRYPTED PASSWORD '...';
CREATE DATABASE jupyterhuboutpost OWNER jupyterhuboutpost;

Create a secret in your Outpost namespace with the required values before installing JupyterHub Outpost:

kind: Secret
metadata:
  name: my-db-secret
...
stringData:
  SQL_TYPE: "postgresql"
  SQL_USER: "jupyterhuboutpost"
  SQL_PASSWORD: "..."
  SQL_HOST: "postgres.database.svc"
  SQL_PORT: "5432"
  SQL_DATABASE: "jupyterhuboutpost"

And add the database secret to your Outpost values.yaml file:

...
extraEnvVarsSecrets:
  - my-db-secret

Simple KubeSpawner#

In this example, we use the KubeSpawner to spawn all single-user servers with the same image.

values.yaml file:

outpostConfig: |
  from kubespawner import KubeSpawner
  c.JupyterHubOutpost.spawner_class = KubeSpawner
  c.KubeSpawner.image = "jupyter/minimal-notebook:notebook-7.0.3"

Update or install JupyterHub Outpost with values.yaml file:

helm upgrade --install -f values.yaml --namespace outpost outpost jupyterhub-outpost/jupyterhub-outpost

The JupyterHub OutpostSpawner can override each c.<Spawner>.<key> value using the custom_misc feature. For more information, take a look at the OutpostSpawner configuration. In the example above, custom_misc could be used to dynamically override the c.KubeSpawner.image value.

Customize Logging#

For the logging configuration, the Outpost offers these options (corresponding to the logging options of JupyterHub):

c.JupyterHubOutpost.log_level = ...
c.JupyterHubOutpost.log_datafmt = ...
c.JupyterHubOutpost.log_format = ...

If more customization is required, one can do this directly in the outpost_config.py file itself (possible via the outpostConfig key of the helm chart).

class TornadoGeneralLoggingFilter(logging.Filter):
  def filter(self, record):
    # I don't want to see this log line generated by tornado
    if str(record.msg).startswith("Could not open static file"):
      return False
    return True

logging.getLogger("tornado.general").addFilter(TornadoGeneralLoggingFilter())

import os
logged_logger_name = os.environ.get("LOGGER_NAME", "MyOutpostInstance")
c.JupyterHubOutpost.log_format = f"%(color)s[%(levelname)1.1s %(asctime)s.%(msecs).03d {logged_logger_name} %(name)s %(module)s:%(lineno)d]%(end_color)s %(message)s"

Sanitize Spawner.start response#

JupyterHub Outpost will use the return value of the start function of the configured SpawnerClass to tell JupyterHub where the single-user server will be running. For example, in the KubeSpawner, the response of KubeSpawner.start() will be something like http://jupyter-<id>-<user_id>:<port> and the Outpost will forward this response to JupyterHub.

The JupyterHub OutpostSpawner will take this information and create a ssh port-forwarding process with the option -L 0.0.0.0:<local_jhub_port>:jupyter-<id>-<user_id>:<port>. Afterwards, JupyterHub will look for the newly created single-user server at http://localhost:<local_jhub_port>. If the response of the start function of the configured SpawnerClass in the JupyterHub Outpost service is not correct, OutpostSpawner and Outpost cannot work together properly. To ensure nearly all Spawners can be used anyway, you can override the response send to the OutpostSpawner.

# In the `outpostConfig` key of your helm values.yaml file or your outpost_config.py file:

# This may be a coroutine
def sanitize_start_response(spawner, original_response):
  # ... determine the correct location for the new single-user server
  return "<...>"

c.JupyterHubOutpost.sanitize_start_response = sanitize_start_response

Note

If you don’t know where your single-user server will be running at the end of the start function, you have to return an empty string. In that case, JupyterHub OutpostSpawner won’t create a ssh port-forwarding process. Instead, the start process of the single-user server has to send a POST request to the $JUPYTERHUB_SETUPTUNNEL_URL url. Have a look at the API Endpoints of the OutpostSpawner (https://jupyterhub-outpostspawner.readthedocs.io/en/latest/apiendpoints.html) for more information.

Disable JupyterHub configuration overwrite#

By default, JupyterHubs can overwrite the JupyterHub Outpost configuration with the OutpostSpawner’s custom_misc function. As an administrator of the JupyterHub Outpost service, you can prevent this.

# In the `outpostConfig` key of your helm values.yaml file or your outpost_config.py file:

async def allow_override(jupyterhub_name, misc):
    if jupyterhub_name == "trustedhub":
        return True
    if list(misc.keys()) != ["image"]:
        return False
    return misc.get("image", "None") in ["allowed_image1", "allowed_image2"]

c.JupyterHubOutpost.allow_override = allow_override

The above example leads to the following behaviour:

  • JupyterHub with credential username “trustedhub” can overwrite anything.

  • If a JupyterHub (other than trustedhub) tries to overwrite anything except the image key, it will not be allowed.

  • The given image must be allowed_image1 or allowed_image2.

Note

If custom_misc in the POST request is empty, the allow_override function will not be called.
If allow_override returns False, the JupyterLab will not be started. An error message will be returned to the JupyterHub OutpostSpawner and shown to the user.

Recreate ssh tunnels at startup#

If your JupyterHub Outpost is used as a ssh node in the JupyterHub OutpostSpawner, all port-forwarding processes have to be recreated if the JupyterHub Outpost service was restarted. While restarting, existing ssh port-forwarding process will fail after a few seconds and the user’s single-user server would be unreachable.

By default tunnels will be recreated at JupyterHub Outpost restarts. You can disable this behaviour with the ssh_recreate_at_start key.

# In the `outpostConfig` key of your helm values.yaml file or your outpost_config.py file:

async def restart_tunnels(wrapper, jupyterhub_credential):
    if jupyterhub_credential == "local_jupyterhub":
        return False
    return True

c.JupyterHubOutpost.ssh_recreate_at_start = restart_tunnels
# c.JupyterHubOutpost.ssh_recreate_at_start = False

Note

JupyterHub Outpost will use the stored JupyterHub API token to recreate the port-forwarding process. If the API token is no longer valid, this will fail. The single-user server would then be unreachable and must be restarted by the user.

Flavors - manage resource access for multiple JupyterHubs#

By default, all connected JupyterHubs may use all available resources. It’s possible to configure “flavors” for each connected JupyterHub, offering only a part of the available resources.

For this configuration three attributes are crucial:

  • flavors

  • flavors_undefined_max

  • flavors_update_token

Flavors#

Configure different flavors, which can be used in Spawner configuration.

async def flavors(jupyterhub_name):
    if jupyterhub_name == "privileged":
        return {
            "typea": {
                "max": -1,
                "weight": 10,
                "display_name": "2GB RAM, 1VCPU, 5 days",
                "description": "JupyterLab will run for max 5 days with 2GB RAM and 1VCPU.",
                "runtime": {"days": 5},
            },
        }
    else:
        return {
            "typeb": {
                "max": 10,
                "weight": 9,
                "display_name": "4GB RAM, 1VCPUs, 2 hours",
                "description": "JupyterLab will run for max 2 hours with 4GB RAM and 1VCPUs.",
                "runtime": {"hours": 2},
            },
        }
c.JupyterHubOutpost.flavors = flavors

The connected JupyterHub “privileged” can start infinite singleuser notebook server. The servers will be stopped after 5 days by the JupyterHub Outpost.
Any other connected JupyterHub can start up to 10 singleuser notebook server (all users together for each JupyterHub, not combined for all JupyterHubs).
The according RAM / VCPUs restrictions are configured later on in the config file at c.KubeSpawner.profile_list or c.KubeSpawner.[mem_guarantee|mem_limit|cpu_guarantee|cpu_limit].
JupyterHub OutpostSpawner has to send the chosen flavor in user_options.flavor when starting a notebook server.

Undefined Max#

If JupyterHub OutpostSpawner does not send a flavor in user_options c.JupyterHubOutpost.flavors_undefined_max will be used to limit the available resources. This value is also used, if the given flavor is not part of the previously defined flavors dict. Default is -1, which allows infinite notebook servers for all unknown or unconfigured flavored notebook servers.

c.JupyterHubOutpost.flavors_undefined_max = 0

This example does not allow any notebook server with a flavor, that’s not defined in c.JupyterHubOutpost.flavors. Enables full control of the available resources.

Update Tokens#

The JupyterHub OutpostSpawner offers a APIEndpoint, which receives or offers the current Flavors of all connected JupyterHub Outposts. With this function the Outpost will inform the connected JupyterHubs at each start/stop of a notebook server, about the current flavor situation. The corresponding URL will be given by the OutpostSpawner.

import os
async def flavors_update_token(jupyterhub_name):
    token = os.environ.get(f"FLAVOR_{jupyterhub_name.upper()}_AUTH_TOKEN", "")
    if not token:
        raise Exception(f"Flavor auth token for {jupyterhub_name} not configured.")
    return token
c.JupyterHubOutpost.flavors_update_token = flavors_update_token

In case of an exception the update is not send to JupyterHub. This will not interfere with the start of the notebook server.
Each connected JupyterHub must provide a service token with scope custom:outpostflavors:set to the Outpost administrator.