usage_scenario.yml

The usage_scenario.yml consists of these main blocks:

Start of the file with some basic root level keys
services - (optional) Handles the orchestration of containers
flow - Handles the interaction with the containers (or with the host, see Flows on the host →)
compose-file - (optional) A compose file to include
relations - (optional) Additional repositories to check out
networks - (optional) Handles the orchestration of networks
volumes - (optional) Named volumes to inspect the size of
custom_metrics - (optional) Handles Custom Metrics and SCI

Its format is an extended subset of the Docker Compose Specification, which means that we keep the same format, but disallow some options and also add some exclusive options to our tool. However, keys that have the same name are also identical in function - thought potentially with some limitations. See also the note on unsupported features to disable the warning about that.

Inside the usage_scenario.yml you can use variables. See variables for details.

Basic root level keys

Example for the start of a usage_scenario.yml

---
name: My Hugo Test
author: Arne Tarara <arne@green-coding.io>
description: This is just an example usage_scenario ...

name [str]: Name of the scenario
description [str]: Detailed description of the scenario
author [str]: Author of the scenario
architecture [str] (optional): If your usage_scenario runs only on a specific architecture you can instruct the GMT to check if the architecture of the machine matches. You can specify Linux, Windows and macOS. Omit this key if your scenario has no architecture restriction.
- The value is compared case-insensitively. Note that macOS must be given as macOS and not as Darwin, otherwise the check will always fail.
- Note: Windows with WSL2 and Linux containers would be Linux as architecture
ignore-unsupported-compose [bool] (optional): Ignore unsupported Docker Compose features and still run usage_scenario
version [str|int|float] (optional): Part of the Docker Compose specification. Accepted, but ignored by GMT as it is non-functional.

Please note that when running the measurement you can supply an additional name, which can and should be different from the name in the usage_scenario.yml.

The idea is to have a general name for the usage_scenario.yml and another one for the specific measurement run.

When running the runner.py we would then set --name for instance to: Hugo Test run on my Macbook

Services

Example:

services:
  gcb-wordpress-mariadb:
    image: gcb_wordpress_mariadb
    environment:
      MYSQL_ROOT_PASSWORD: somewordpress
      MYSQL_DATABASE: wordpress
      MYSQL_USER: wordpress
      MYSQL_PASSWORD: wordpress
    ports:
      - 3306:3306
    setup-commands:
      - command: sleep 20
    volumes:
      - /LOCAL/PATH:/PATH/IN/CONTAINER:ro
    networks:
      - wordpress-mariadb-data-green-coding-network
    mem_limit: 2G
    cpus: 2
    shm_size: 256M
    healthcheck:
      test: curl -f http://nc
      interval: "30s"
      timeout: "10s"
      retries: 10
      start_period: "10s"
      start_interval: "5s"
      disable: False
  gcb-wordpress-apache:
    # ...
    depends_on:
      gcb-wordpress-mariadb:
        condition: service_healthy
  gcb-wordpress-dummy:
    # ...
    depends_on:
      - gcb-wordpress-mariadb

services [dict] (optional): (Dictionary of container dictionaries for orchestration)
- A usage_scenario.yml without services is valid. This is the case when you want to benchmark containers that are already running on your host.
- [CONTAINER]: [a-zA-Z0-9_] The name of the container/service
- image: [str] Docker image identifier. If build is not provided the image needs to be accessible locally on Docker Hub. If build is provided it is used as identifier for the image.
- build: [str] (optional) Path to build context. See context for restrictions. Default for dockerfile is Dockerfile. Alternatively, you can provide more detailed build information with:
  - context: [str] (optional) Path to the build context. Needs to be in the path or repo that is passed with --uri to runner.py. Default: ..
  - dockerfile: [str] (optional) Path to Dockerfile. Needs to be in context. Default: Dockerfile.
  - args: [list] (optional) Build arguments that are passed to the build as --build-arg. Supply them as a list of single key-value dictionaries, e.g. a list item in the format: - MY_ARG: my_value. The other forms of the Docker Compose Specification (a plain dict, or list items in the format - MY_ARG=my_value) are currently not supported.
  - target: [str] (optional) Name of the stage in a multi-stage Dockerfile that shall be built and used as the image of the service. It is passed to Kaniko as --target. Allowed characters are [A-Za-z0-9_.-] and the name must start with a letter or digit.
  - All images that GMT builds in one run share a Kaniko layer cache. Services that use the same base image or the same stages of a Dockerfile can therefore reuse already built layers. The cache is a Docker volume that GMT removes at the start and at the end of every run, unless runner.py is called with --dev-cache-build.
- container_name: [a-zA-Z0-9_] (optional) With this key you can overwrite the name of the container. If not given, the defined service name above is used as the name of the container.
- environment: [dict|list] (optional)
  - Either Key-Value pairs for ENV variables inside the container
  - Or list items with strings in the format: MYSQL_PASSWORD=123
- labels: [dict|list] (optional)
  - Either Key-Value pairs for Docker labels set on the container
  - Or list items with strings in the format: my.label=123
  - Without --allow-unsafe the key must match ^[A-Za-z_]+[A-Za-z0-9_.]*$ and the value may be at most 1024 characters long
- ports: [int:int] (optional)
  - Docker container portmapping on host OS to be used with --allow-unsafe flag.
- init: [boolean]
  - Will start a PID 1 init process in the container that helps when you are experiencing zombie processes and memory leaks. See Docker Docs for details
- depends_on: [list|dict] (optional)
  - Can either be an list of services names on which the service is dependent. It affects the startup order and forces the dependency to be started (container state = “running”) before the service is started.
  - Or it can be an dict where you can specify a condition. The condition can have two values:
  - service_healthy: Will wait for the container until the docker healthcheck returns healthy.
  - service_started: Similar to the list syntax -> this will enforce a starting order and just wait until the container state is “running”.
- setup-commands: [list] (optional)
  - List of commands to be run before actual load testing. Mostly installs will be done here. Note that your docker container must support these commands and you cannot rely on a standard linux installation to provide access to /bin
  - command: [str]
  - The command to be executed
  - detach: [bool] (optional, default: false)
  - When the command is detached it will get sent to the background and the setup will continue with the next command. The process is terminated at the end of the run.
  - shell: [str] (optional)
  - Will execute the setup-commands in a shell. Use this if you need shell-mechanics like redirection > or chaining &&. ** Please use a string for a shell command here like sh, bash, ash etc. The shell must be available in your container
  - GMT runs the command as SHELL -o errexit -o nounset -o pipefail -c COMMAND. The command thus fails on the first failing sub-command, on unset variables and on failures inside a pipe. See Strict shell options for details.
  - shell-options: [str|list] (optional)
  - Replaces the default options -o errexit -o nounset -o pipefail for this command. Can only be used together with shell. See Strict shell options.
- volumes: [list] (optional)
  - List of volumes to be mapped in the format source:target[:ro].
  - This key is always read. The --allow-unsafe flag does not gate it, but relaxes how it is handled:
  - In safe mode (the default) only read-only mounts are allowed, so the :ro (or :readonly) option is mandatory, and the source is resolved relative to the folder of your usage_scenario.yml. An exception are mounts that are listed in the allowed_volume_mounts capability of your user, which may be read-write and may be absolute paths or named volumes.
  - With --allow-unsafe the volume spec is handed to docker run -v as given, so read-write mounts and absolute host paths are possible.
  - Note that --skip-unsafe has no effect on volumes. It only applies to ports, environment and labels.
- networks: [list|dict] (optional)
  - The networks to put the container into. If no networks are defined throughout the usage_scenario.yml the container will be put into the default network will all others in the file.
  - When given as a dict you can supply network modifiers per network:
  - aliases: [list] (optional) Additional DNS names the container is reachable under inside that network
  - Joining the Docker host network is restricted and needs --allow-unsafe
- healthcheck: [dict] (optional)
  - Please see the definition of these arguments and how healthcheck works in the official docker compose definition. We just copy them over: Docker compose healthcheck specification
  - test: [str|list]
  - interval: [str]
  - timeout: [str]
  - retries: [integer]
  - start_period: [str]
  - start_interval: [str] Time between health checks during the start period
  - disable: [boolean]
- folder-destination: [str] (optional)
  - Specify where the project that is being measured will be mounted inside of the container
  - Defaults to /tmp/repo
- command: [str|list] (optional)
  - Command to be executed when container is started. When container does not have a daemon running typically a shell is started here to have the container running like bash or sh.
  - Can be given as a string, which is split shell-like, or as a list of arguments.
- entrypoint: [str|list] (optional)
  - Declares the default entrypoint for the service container. This overrides the ENTRYPOINT instruction from the service’s Dockerfile.
  - The value of entrypoint can be an empty string (ENTRYPOINT instruction will be ignored), a string, or a list of arguments.
  - When a string is given it is split shell-like. The first element becomes the entrypoint, all remaining elements are prepended to command.
- log-stdout: [boolean] (optional, default: true)
  - Will log the stdout of the container and make it available through the frontend in the Logs tab.
  - Please see the Best Practices → for when to disable the logging.
- log-stderr: [boolean] (optional, default: true)
  - Will log the stderr of the container and make it available through the frontend in the Logs tab and in error messages.
  - Please see the Best Practices → for when to disable the logging.
- read-notes-stdout: [bool] (optional)
  - Read notes from stdout of the container.
  - Most likely you do not need this, as it also requires customization of your application (writing of a log message in a specific format). It may be helpful if your application has asynchronous operations and you want to know when they have finished. In most cases, it is more appropriate to read the notes from the command’s stdout in your flow (see below).
  - Note that log-stdout has to be enabled (it is the default).
  - Format specification is documented below in section Read-notes-stdout format specification →.
- docker-run-args: [list] (optional)
  - A list of string that should be added to the docker run command of that container.
  - The argument needs to be listed in the user.capabilities json under measurement:orchestrators:docker:allowed_run_args. The string in the user.capabilities can be a regex and must match the whole argument. Opening this up could be a potential security issue!
  - Running with --allow-unsafe bypasses this allow list entirely and every argument is accepted.
- mem_limit: [str|int] (optional)
  - The memory limit of the container, e.g. 2G. If not set, GMT auto-assigns a value.
  - Can also be given as deploy.resources.limits.memory. If both are given they must be identical.
- cpus: [str|int|float] (optional)
  - The number of CPUs the container may use. If not set, GMT auto-assigns all assignable cores.
  - Can also be given as deploy.resources.limits.cpus. If both are given they must be identical.
- shm_size: [str|int] (optional)
  - Size of /dev/shm for the container, passed to docker run --shm-size.
- restart: [str] (optional)
  - Part of the Docker Compose specification. Accepted, but ignored, as GMT does its own orchestration.
- expose: [list] (optional)
  - Part of the Docker Compose specification. Accepted, but ignored, as it is non-functional.

Please note that every key below services will serve as the name of the container later on. You can overwrite the container name with the key container_name.

Also see Resource Limits → for how mem_limit and cpus are auto-assigned and enforced.

The cmd key is not supported anymore. If your file still uses it you will get an error asking you to migrate to command.

Relations

relations:
  server: # will be put in /tmp/relations/server
    url: https://github.com/nextcloud/server
    branch: master
    # no commit hash given means HEAD is checked out
  k6:
    url: https://github.com/grafana/k6
    branch: main
    commit_hash: 352991e5acdf50728a7c8ea7350c33d0de65dfc1

Often it is needed to have additional repositories checked out that you do not want to include in your core repository.

Example: Your software is a HTTP server and you include the benchmarks in your repository. However you need k6 as a helper library also somewhere present.

You can of course include it as a service. But sometimes it is needed to have the repository available for building some stuff or for supplementing your services.

Relations are checked out at a specific branch and revision that you specify. If you do not specify a revision HEAD is always checked out.

All relations are mounted into every container at the mountpoint /tmp/relations/X whereas X represesents the key the relations object from your usage_scenario.yml file.

relations: [dict] (Dictionary of relations, aka additional git repositories, to check out)
- KEY: [a-zA-Z0-9_] The name of the relation. Will later be the location in /tmp/relation/KEY of the container
  - url: URL URL of the repository. Technically can also be SSH URI if you have the git client configured to send credentials. No filesystem paths are allowed though as usage_scenario.yml files should be transportable to different systems and shareable.
  - branch: Allowed git branch characters: The branch to check out for the git repository
  - commit_hash: SHA-1 hash: The revision to check out. HEAD if not specified

Networks

Example:

networks:
  wordpress-mariadb-data-green-coding-network:
  gmt-test-network:
    internal: true

networks: [dict|list] (Dictionary of network dictionaries for orchestration, or a plain list of network names)
- [NETWORK]: [a-zA-Z0-9_-] The key is the name of the network. No value is required.
  - internal: [bool] (optional) Creates the network with --internal, so containers in it have no external connectivity.
- The pre-defined Docker networks host, bridge and none cannot be created here. They already exist and can only be joined.

Networks defined here are created before the containers are started and are removed again after the run. Put a container into a network with the networks key of the service.

Volumes

Example:

volumes:
  my-named-volume:

volumes: [dict|str] (optional) (Dictionary of named volumes, or a single volume name)
- [VOLUME]: [a-zA-Z0-9_-] The key is the name of the volume. No value is required.

Contrary to the Docker Compose Specification GMT does not create named volumes from this key. The volume must already exist on the machine. GMT only inspects the listed volumes and reports their size as Container Volume Sizes in the Machine tab of the Dashboard.

The inspection only runs when all of the following are true:

runner.py is executed with --allow-unsafe (volume information reveals host filesystem paths)
--skip-volume-inspect is not set
the run is on Linux (the inspection is skipped on macOS and Windows)

Flow

Example:

flow:
  - name: Check Website
    container: green-coding-puppeteer-container
    commands:
    - type: console
      command: node /var/www/puppeteer-flow.js
      note: Starting Puppeteer Flow
      read-notes-stdout: true
    - type: console
      command: sleep 30
      note: Idling
    - type: console
      command: node /var/www/puppeteer-flow.js
      note: Starting Puppeteer Flow again
      read-notes-stdout: true
  - name: Shutdown DB
    container: database-container
    commands:
      - type: console
        command: killall postgres

flow: [list] (List of flows to interact with containers)
- name: [.\s0-9a-zA-Z_()-]+ An arbitrary name, that helps you distinguish later on where the load happend in the chart
- container: [a-zA-Z0-9][a-zA-Z0-9_.-]+|null The name of the container in which you want to run the flow
  - The value must be the name of a container defined in services. This is the container_name of the service if it is set, otherwise the key of the service. GMT validates this before the run and aborts if the flow references an unknown container.
  - Set it to null (or leave the value empty) to run the flow directly on the host. See Flows on the host →.
- hidden: true Minimizes a flow set in the frontend
- commands: [list]
- type: [console|playwright]
  - console will execute a shell command inside the container (or on the host for flows with container: null)
  - playwright will execute the playwright command in the container. See the documentation for more details.
- command: [str]
  - The command to be executed. If type is console then piping or moving to background is not supported.
- detach: [bool] (optional, default: false)
  - When the command is detached it will get sent to the background. This allows to run commands in parallel if needed, for instance if you want to stress the DB in parallel with a web request
- note: [str] (optional)
  - A string that will appear as note attached to the datapoint of measurement (optional)
- ignore-errors: [bool] (optional)
  - If set to true the run will not fail if the process in command has a different exit code than 0. Useful if you execute a command that you know will always fail like timeout 0.1 stress -c 1
- shell: [str] (optional)
  - Will execute the command in a shell. Use this if you need shell-mechanics like redirection > or chaining &&.
  - Please use a string for a shell command here like sh, bash, ash etc. The shell must be available in your container
  - For console commands GMT runs the command as SHELL -o errexit -o nounset -o pipefail -c COMMAND. See Strict shell options for details.
- shell-options: [str|list] (optional)
  - Replaces the default shell options for this command. Can only be used together with shell and only for commands of type console. See Strict shell options.
- log-stdout: [boolean] (optional, default: true)
  - Will log the stdout of the command and make it available through the frontend in the Logs tab.
  - Please see the Best Practices → for when to disable the logging.
- log-stderr: [boolean] (optional, default: true)
  - Will log the stderr of the command and make it available through the frontend in the Logs tab and in error messages.
  - Please see the Best Practices → for when to disable the logging.
- read-notes-stdout: [bool] (optional)
  - Read notes from the stdout of the command.
  - This is helpful if you have a long running command that does multiple steps and you want to log every step.
  - Note that log-stdout has to be enabled (it is the default).
  - Format specification is documented below in section Read-notes-stdout format specification →.

Strict shell options

Every console command and every setup-command that has a shell is started with these options by default:

SHELL -o errexit -o nounset -o pipefail -c COMMAND

errexit stops the command at the first sub-command that fails
nounset treats the use of an unset variable as an error
pipefail makes a pipe fail if any command in it fails, not only the last one

This way errors in your commands cannot go unnoticed.

Some shells do not support all of these options. A common case is dash, which is /bin/sh in Debian and Ubuntu based images and has no pipefail. GMT then aborts the run and tells you that the used shell does not support the shell options. See Troubleshooting →. Either use a shell that supports them, like bash, or set shell-options for the command.

shell-options replaces the default options for a single command:

A string is split like a command line, for example -o errexit -o pipefail
A list is used as it is, for example ['-o', 'errexit']
An empty list (shell-options: []) disables all options. Errors in your command may then go unnoticed.

flow:
  - name: Stress
    container: test-container
    commands:
      - type: console
        command: stress-ng -c 1 -t 1 -q
        shell: bash
        shell-options: -o errexit -o pipefail
      - type: console
        command: echo 1
        shell: sh
        shell-options: []

shell-options can only be used together with shell. In flows it is only allowed for commands of type console. GMT rejects the usage_scenario.yml otherwise.

For flows on the host the options depend on the shell, because PowerShell and cmd do not support them. See Flows on the host →.

compose-file:

If you specify the compose-file key the referenced compose file will be included into the usage_scenario. This is a feature so you can develop your app using the standard compose functionality and you don’t need to duplicate your configuration in the usage_scenario.yml file.

Example:

compose-file: !include compose.yml

Please see the !include section for more details on how to use this functionality. Through using this syntax you tell the parser to include the compose.yml file before doing any parsing of the usage_scenario. This is done so you can modify values that are defined in the compose file. For example you could have a section in your compose

services: ...
  gcb-wordpress: ...
    volumes:
      - ./wordpress.conf:/etc/apache2/sites-enabled/wordpress.conf:ro

than loads a configuration into the container. Which is fine for your production system but when doing the benchmark you want to load a different configuration file. This can easily be done by just adding

services: ...
  gcb-wordpress: ...
    volumes:
      - ./wordpress_bench.conf:/etc/apache2/sites-enabled/wordpress.conf:ro

into your usage_scenario.yml. Of course you can also add key/values.

!include

It is possible to include files into your usage_scenario.yml. This is useful, for example, if you want to use different flows for various measurements but the services stay the same. Like this you could have a usage_services.yml that has the services section and in your usage_scenario.yml you can

services: !include services.yml

It is also possible to add a key selector so you can select which content you want

services: !include utils.yml services

Read-notes-stdout format specification

If you have the read-notes-stdout set to true your output must have this format:

TIMESTAMP_IN_MICROSECONDS NOTE_AS_STRING

Example: 1656199368750556 This is my note

Every note will then be consumed and can be retrieved through the API.

Please be aware that the timestamps of the note do not have to be identical with any command or action of the container. If the timestamp however does not fall into the time window of your measurement run it will not be displayed in the frontend.

Unsupported Docker Compose features

All features not listed here are not supported by the Green Metrics Tool.

Since we allow the import of Docker Compose files this can lead to importing unsupported features.

GMT will error in this case. If you do not want that add the ignore-unsupported-compose key after you have tested your usage_scenario.yml file.

Variables

A variable must adhere to the format __GMT_VAR_[\w]+__. An example would be __GMT_VAR_NUMBER__.

The value of the variable is a string. It will be replaced as is without adding " or ’ though.

Example in a usage_scenario.yml:

---
name: Test Stress
author: Dan Mateas
description: test

services:
  test-container:
    image: gcb_stress
    build:
      context: ../stress-application

flow:
  - name: Stress
    container: test-container
    commands:
      - type: console
        command: stress-ng -c 1 -t __GMT_VAR_DURATION__ -q
        note: Starting Stress

Here we can leverage the variable functionality to supply different durations without creating new usage scenarios all the time.

Beware that it is often very helpful to put the whole command in quotes like this:

...
command: "stress-ng -c 1 -t __GMT_VAR_DURATION__ -q"
...

The reason being that when your variable contains YAML control characters like a : you might get into parsing errors.

A run where we want the variable to be 1 as example can be started like this:

python3 runner.py --uri PATH_TO_SCENARIO --variable "__GMT_VAR_DURATION__=1"

See more details in Runner switches →

The API accepts these variables also in the usage_scenario_variables field of the /v1/runs/add endpoint. See the API documentation → for details.

Secret variables

Variables whose name matches __GMT_VAR_SECRET_[\w]+__, for example __GMT_VAR_SECRET_API_TOKEN__, hold secrets like passwords or tokens. They are replaced into the usage_scenario.yml like every other variable, but GMT never stores or displays their plaintext:

The value is encrypted before it is stored. The database and the dashboard only contain the encrypted form. The runner decrypts it only in memory to replace the variable.
The plaintext is redacted from everything GMT prints or saves, like the logged commands, the container and process logs, warnings and error messages.
When you run runner.py on a machine without an encryption key, the secret is stored as *****GMT-REDACTED***** instead and GMT prints a warning.
The API refuses secret variables with HTTP status 422 if encryption is not configured on the server.

How to set up the encryption keys is described in Private repositories and registries →.

Custom Metrics

Custom Metrics are a super set of the SCI metrics which however are configured identically.

Please see Software Carbon Intensity (SCI) → for more information.

Interacting with applications

Measuring locally

usage_scenario.yml

Basic root level keys#

Services#

Relations#

Networks#

Volumes#

Flow#

Strict shell options#

compose-file:#

!include#

Read-notes-stdout format specification#

Unsupported Docker Compose features#

Variables#

Secret variables#

Custom Metrics#

Basic root level keys

Services

Relations

Networks

Volumes

Flow

Strict shell options

compose-file:

!include

Read-notes-stdout format specification

Unsupported Docker Compose features

Variables

Secret variables

Custom Metrics