Collection execution

Collections are ordered list of monitors/tests that are executed in the sequence they are defined in. A Collection by itself doesn't have failure states, it's just a container shell for executing your underlying tests.

Defined Collections in a Workspace can be triggered to run manually by any Workspace user or can also be scheduled to run automatically at regular intervals.

When a run is triggered for a Collection, by user or system, the system executes the steps defined the Collection via a scheduler job that runs at regular intervals.

Total Execution Time

Unlike individual monitoring runs, execution of Collection Steps are not immediate. When a run is triggered for a Collection run, it can take upto a minute for each step in the run to be kicked-off. Add a minute for each Step in your Collection to determine the total execution time for a Collection run.

Here is an example to determine the potential total execution time it takes to complete a Collection Run:

Assume a Collection with three Browser Tests.

Each test can take up 2 minutes to complete. So, 3 steps * 2 minutes = 6 minutes for execution of the steps.

Collection scheduler can take upto a minute per step. So, 3 steps * 1 minute = 3 minutes for Collection scheduler.

Total execution time = 6 minutes for execution of steps + 3 minutes for Collection scheduler = 9 minutes.

Execution Interval

Concurrent runs for a Collection can be triggered when a Collection is scheduled to run automatically. For example, if a Collection run takes 10 minutes to complete and if the Collection is scheduled to run every 5 minutes. Concurrent runs will get invoked.

Schedule your Collections considering the Total Execution Time (as explained above) to avoid concurrent runs.

Execution environment

Steps defined in a Collection do not share the environment for execution of the Steps. Each Step is executed separately in a discrete new environment.

If your monitor has multiple locations selected, the step will run from all the selected locations. If multiple locations are selected, ensure that your tests are accounted for records created from all the locations.

For simplicity, if you are defining a test to be executed via a Collection we recommend selecting a single monitoring location for those tests.

Environment variables

If your individual monitoring checks already have an associated Environment, the variables with the same from from Collection's Environment will override the Monitor level Environment variables. Here is an example scenario:

Assume an API monitor is associated with an Environment with the following variables.

{
    "foo":"foo value 1",
    "bar":"bar value"
}

This monitor is part of a Collection, which is associated with an Environment the following variables defined:

{
    "foo":"foo overridden value",
    "baz":"baz value"
}

During the Collection run, the API monitor will have following variables available.

{
    "foo":"foo overridden value", //foo value is overriden from Collection's env variables
    "bar":"bar value",
    "baz":"baz value" 
}

State

Steps in a Collection do not preserve any state during execution. However, we provide a mechanism to temporarily preserve and retrieve values via APIs. Refer this document fore more details.

caution

Any values preserved via these APIs are not secured and can be accessed with the key. Do not persist credentials, login tokens via these APIs. We recommend using random strings that are hard to guess as keys for storing any values.

Alerting Notifications

Any failure notifications for any monitors will leverage the notification settings or integrations defined for the monitor. If one the steps in a Collection reports a failure, notifications will be sent as per the notification configuration defined on the underlying monitor/test.

If Snooze Alerts is enabled on the monitor/test that is used in a Step, notifications will not be sent for any failures or warnings.