Versions and levels#

pystorms 2.0 adds two keywords to every scenario.

version selects the network. "1" is the scenario as originally published and is the default. "2" is a harder variant of the same network, defined for theta, alpha, gamma, delta and epsilon.

level decides how trustworthy the instrumentation is. "1" is the default and reports the true state. "2" adds reading noise, sensor drift and calibration bias, and lets valves stick. "3" intensifies all of that and adds sensors that drop out entirely.

The two are independent, so you can hold the network fixed and vary only the measurement quality, which is usually what you want when testing whether a controller is robust or merely tuned.

import pystorms

env = pystorms.scenarios.theta(version="2", level="3")

Both keywords accept integers as well as strings. A value a scenario does not define raises ValueError before any simulation is opened, so asking beta or zeta for version="2", or any scenario for level="4", fails immediately rather than silently running the default.

Scenario versions#

Version 2 tightens the objective and modifies the network so that the same controller has less room to look good. The changes per scenario:

Scenario	Version 2
theta	Flow threshold halved to 0.25. The first basin is half as deep. The event ends a day and a half earlier.
alpha	The five weirs join the five orifices in the action space. Every orifice is widened to the interceptor diameter.
gamma	Flow threshold lowered from 4.0 to 3.0. Basins 5 and 9 are removed from the state space, the action space and the performance targets.
delta	Outflow threshold lowered from 12.0 to 0.5. The event runs three days longer with 30 percent more rainfall. The downstream conduit restrictions and the uncontrollable subcatchment flooding are removed, and the routing step is fixed at five seconds so every run has the same number of steps.
epsilon	TSS loading threshold lowered to 70 percent of the original. The event runs to the middle of February with 10 percent more rainfall.
beta, zeta	No second version.

The version 2 networks are built when the scenario is constructed, by rewriting the shipped SWMM input file with swmmio. The shipped file is not touched. The rewritten copy, together with the .rpt and .out files SWMM writes for every run, goes to a per user cache directory: $PYSTORMS_CACHE if set, otherwise $XDG_CACHE_HOME/pystorms/networks or ~/.cache/pystorms/networks.

Note

Processes that build the same version 2 scenario at the same time write the same derived file. When running scenarios in parallel, give every worker its own PYSTORMS_CACHE directory.

Difficulty levels#

The fault schedule is drawn once, when the scenario is built. From then on state() and step() apply it without being told again.

env = pystorms.scenarios.theta(level="2")   # draws the fault schedule
done = False
while not done:
    state = env.state()                      # noisy, drifting, biased readings
    done = env.step(controller(state))       # stuck valves ignore the command

What the levels do#

Lengths below are metres and are converted for networks that run in US units.

	Level 2	Level 3
Reading noise	Gaussian, standard deviation 2.5 cm	Gaussian, standard deviation 15 cm
Sensor drift	15 percent of sensors drift, at 0.15 to 0.45 mm per day	50 percent of sensors drift, at 1 to 2 cm per day
Calibration bias	Multiplicative, within 1 percent	Multiplicative, within 10 percent
Stuck valves	Each valve has a 60 percent chance of sticking once, for 10 to 30 percent of the event	Each valve has an 80 percent chance of sticking once, for 20 to 50 percent of the event
Sensor dropouts	None	Each sensor has an 80 percent chance of dropping out once, for 5 to 20 percent of the event

A few consequences worth knowing:

Noise, drift and bias are lengths, and they are applied to every state the same way, including flow and pollutant concentration states.
Readings can be negative.
A stuck valve keeps the position it had when it stuck, whatever the controller commands, until it is fixed.
A dropped out sensor reports exactly zero.
The performance measure is always computed from the true state of the network. Only what the controller sees is degraded.

Reading the true state#

state() and step() also accept a level argument. Pass level="1" to read the ground truth from a degraded scenario, for instance to log it next to what the controller saw:

env = pystorms.scenarios.theta(level="3")
done = False
while not done:
    measured = env.state()
    actual = env.state(level="1")
    done = env.step(controller(measured))

Asking for a higher level than the scenario was built with raises ValueError, since no fault schedule exists for it.

Inspecting the fault schedule#

The schedule is available on the environment object, env.env:

drift_rates: per sensor drift, in the network’s length unit per day
bias: per sensor multiplicative bias
actuator_schedule: {asset: [(stuck_time, fix_time), ...]} for every valve that sticks, or None when none does
sensor_schedule: the same for sensors that drop out, level 3 only

Seeding#

The faults are drawn from numpy’s global random state. Two runs of the same scenario are therefore not the same experiment unless you seed first. There is no seed argument on the scenarios themselves, so seed numpy.random before construction, once at the top of a benchmark script, and record the seed alongside the numbers:

import numpy as np

np.random.seed(42)
env = pystorms.scenarios.theta(level="3")

Level 1 does not touch the random number generator, so seeding is only needed for levels 2 and 3.

Comparing controllers fairly#

The useful question is not how a controller scores at level 1 but how much of that score survives when the instrumentation degrades. Averaging over several seeds keeps a single lucky fault schedule from deciding the answer:

def evaluate(controller, level, seeds=(1, 2, 3)):
    scores = []
    for seed in seeds:
        np.random.seed(seed)
        with pystorms.scenarios.theta(level=level) as env:
            done = False
            while not done:
                done = env.step(controller(env.state()))
            scores.append(env.performance())
    return np.mean(scores), np.std(scores)

The Versions_and_Levels notebook walks through all of this on theta, and the baseline_controllers directory of the repository holds the controller implementations, tuned parameters and analysis scripts behind the accompanying manuscript.

Changes to reported numbers in 2.0#

Two fixes in 2.0 change numbers that a 1.0 script would have produced:

epsilon now reads the TSS concentration in the outlet conduit rather than in the node upstream of it, so the pollutant state and the loading objective differ by about a tenth of a percent.
At level 3 a dropped out sensor reports zero. Earlier development builds reported noise around zero instead.

Level 1 runs of every other scenario reproduce their 1.0 results exactly. See the changelog for the full list.