=================== Versions and levels =================== ``pystorms`` 2.0 adds two keywords to every scenario. ``version`` selects the network. ``"1"`` is the scenario as originally published and is the default. ``"2"`` is a harder variant of the same network, defined for *theta*, *alpha*, *gamma*, *delta* and *epsilon*. ``level`` decides how trustworthy the instrumentation is. ``"1"`` is the default and reports the true state. ``"2"`` adds reading noise, sensor drift and calibration bias, and lets valves stick. ``"3"`` intensifies all of that and adds sensors that drop out entirely. The two are independent, so you can hold the network fixed and vary only the measurement quality, which is usually what you want when testing whether a controller is robust or merely tuned. .. code:: python import pystorms env = pystorms.scenarios.theta(version="2", level="3") Both keywords accept integers as well as strings. A value a scenario does not define raises ``ValueError`` before any simulation is opened, so asking *beta* or *zeta* for ``version="2"``, or any scenario for ``level="4"``, fails immediately rather than silently running the default. Scenario versions ================= Version 2 tightens the objective and modifies the network so that the same controller has less room to look good. The changes per scenario: .. list-table:: :header-rows: 1 :widths: 10 45 * - Scenario - Version 2 * - theta - Flow threshold halved to 0.25. The first basin is half as deep. The event ends a day and a half earlier. * - alpha - The five weirs join the five orifices in the action space. Every orifice is widened to the interceptor diameter. * - gamma - Flow threshold lowered from 4.0 to 3.0. Basins 5 and 9 are removed from the state space, the action space and the performance targets. * - delta - Outflow threshold lowered from 12.0 to 0.5. The event runs three days longer with 30 percent more rainfall. The downstream conduit restrictions and the uncontrollable subcatchment flooding are removed, and the routing step is fixed at five seconds so every run has the same number of steps. * - epsilon - TSS loading threshold lowered to 70 percent of the original. The event runs to the middle of February with 10 percent more rainfall. * - beta, zeta - No second version. The version 2 networks are built when the scenario is constructed, by rewriting the shipped SWMM input file with ``swmmio``. The shipped file is not touched. The rewritten copy, together with the ``.rpt`` and ``.out`` files SWMM writes for every run, goes to a per user cache directory: ``$PYSTORMS_CACHE`` if set, otherwise ``$XDG_CACHE_HOME/pystorms/networks`` or ``~/.cache/pystorms/networks``. .. note:: Processes that build the same version 2 scenario at the same time write the same derived file. When running scenarios in parallel, give every worker its own ``PYSTORMS_CACHE`` directory. Difficulty levels ================= The fault schedule is drawn once, when the scenario is built. From then on :code:`state()` and :code:`step()` apply it without being told again. .. code:: python env = pystorms.scenarios.theta(level="2") # draws the fault schedule done = False while not done: state = env.state() # noisy, drifting, biased readings done = env.step(controller(state)) # stuck valves ignore the command What the levels do ------------------ Lengths below are metres and are converted for networks that run in US units. .. list-table:: :header-rows: 1 :widths: 22 24 24 * - - Level 2 - Level 3 * - Reading noise - Gaussian, standard deviation 2.5 cm - Gaussian, standard deviation 15 cm * - Sensor drift - 15 percent of sensors drift, at 0.15 to 0.45 mm per day - 50 percent of sensors drift, at 1 to 2 cm per day * - Calibration bias - Multiplicative, within 1 percent - Multiplicative, within 10 percent * - Stuck valves - Each valve has a 60 percent chance of sticking once, for 10 to 30 percent of the event - Each valve has an 80 percent chance of sticking once, for 20 to 50 percent of the event * - Sensor dropouts - None - Each sensor has an 80 percent chance of dropping out once, for 5 to 20 percent of the event A few consequences worth knowing: - Noise, drift and bias are lengths, and they are applied to every state the same way, including flow and pollutant concentration states. - Readings can be negative. - A stuck valve keeps the position it had when it stuck, whatever the controller commands, until it is fixed. - A dropped out sensor reports exactly zero. - The performance measure is always computed from the true state of the network. Only what the controller sees is degraded. Reading the true state ---------------------- :code:`state()` and :code:`step()` also accept a ``level`` argument. Pass ``level="1"`` to read the ground truth from a degraded scenario, for instance to log it next to what the controller saw: .. code:: python env = pystorms.scenarios.theta(level="3") done = False while not done: measured = env.state() actual = env.state(level="1") done = env.step(controller(measured)) Asking for a higher level than the scenario was built with raises ``ValueError``, since no fault schedule exists for it. Inspecting the fault schedule ----------------------------- The schedule is available on the environment object, ``env.env``: ``drift_rates`` per sensor drift, in the network's length unit per day ``bias`` per sensor multiplicative bias ``actuator_schedule`` ``{asset: [(stuck_time, fix_time), ...]}`` for every valve that sticks, or ``None`` when none does ``sensor_schedule`` the same for sensors that drop out, level 3 only Seeding ------- The faults are drawn from numpy's global random state. Two runs of the same scenario are therefore not the same experiment unless you seed first. There is no ``seed`` argument on the scenarios themselves, so seed ``numpy.random`` before construction, once at the top of a benchmark script, and record the seed alongside the numbers: .. code:: python import numpy as np np.random.seed(42) env = pystorms.scenarios.theta(level="3") Level 1 does not touch the random number generator, so seeding is only needed for levels 2 and 3. Comparing controllers fairly ---------------------------- The useful question is not how a controller scores at level 1 but how much of that score survives when the instrumentation degrades. Averaging over several seeds keeps a single lucky fault schedule from deciding the answer: .. code:: python def evaluate(controller, level, seeds=(1, 2, 3)): scores = [] for seed in seeds: np.random.seed(seed) with pystorms.scenarios.theta(level=level) as env: done = False while not done: done = env.step(controller(env.state())) scores.append(env.performance()) return np.mean(scores), np.std(scores) The `Versions_and_Levels notebook `_ walks through all of this on *theta*, and the `baseline_controllers `_ directory of the repository holds the controller implementations, tuned parameters and analysis scripts behind the accompanying manuscript. Changes to reported numbers in 2.0 ================================== Two fixes in 2.0 change numbers that a 1.0 script would have produced: - *epsilon* now reads the TSS concentration in the outlet conduit rather than in the node upstream of it, so the pollutant state and the loading objective differ by about a tenth of a percent. - At level 3 a dropped out sensor reports zero. Earlier development builds reported noise around zero instead. Level 1 runs of every other scenario reproduce their 1.0 results exactly. See the `changelog `_ for the full list.