Release Changelog prior to v1.0.0

Note

This page details the development history prior to the first official open-source release of RiskScape, v1.0.0. This may be of interest for people who have used older evaluation versions of RiskScape for their modelling.

v1.0.0

Built 4 February 2022

Breaking

  • The --home option will no longer be supported. Instead, the auto-import feature on projects replaces the home directory functionality. The --home option, although now hidden on the CLI, can still be used in the interim to customize where your RiskScape settings.ini file is located, although this functionality will likely be replaced in a future release. (GL-663)

  • RiskScape will no longer report whether errors are present in your project, unless you use the --show-project-errors CLI option. (GL-663)

  • Pipelines can no longer be specified as [pipeline ID] sections in the project.ini file. Instead pipelines should be added as a model using framework = pipeline. This change will only impact advanced users who used pipelines directly. As part of this change, the following riskscape pipeline commands have been removed:

    • riskscape pipeline run, replaced by riskscape model run

    • riskscape pipeline list, replaced by riskscape model list

    • riskscape pipeline verify, replaced by riskscape model verify

    • riskscape pipeline graph, replaced by riskscape model run MODEL_ID --graph

    Pipelines can be converted to pipeline models easily by changing

    [pipeline ID]
pipeline|location = ...

    to

    [model ID]
framework = pipeline
pipeline|location = ...

  • The legacy wizard model has been removed. It is no longer possible to run the legacy survey or to run wizard models that had been saved from it, i.e. wizard models created prior to RiskScape v0.11.0. If users have any saved legacy wizard models they should run through the wizard to re-create them. (GL558)

  • You can no longer define RiskScape types using a separate Jython script, i.e. types.py. This was a very old feature and any users still using it should migrate to defining types in their project.ini file instead. Defining the Jython function’s RETURN_TYPE and ARGUMENT_TYPES in the Python .py file, alongside the function, is still supported. However, going forward we recommend following the examples here and defining the function argument-types and return-type in the project.ini file instead.

  • The following commands have been renamed (GL-534).

    • riskscape bookmark format list is now riskscape format list

    • riskscape bookmark format options is now riskscape format info

    • riskscape pipeline step parameters is now riskscape pipeline step info (GL-690)

  • Plugins are no longer loaded from the plugins sub-directory within the RiskScape home directory. Plugins should now be loaded via the settings.ini file in the home directory instead. Refer to Settings plugins for more details on how to configure this. If there are plugins found in the home directory then a warning will now be output. (GL681)

  • The --stats-out CLI option has been removed. This option is unnecessary now, as metrics are always displayed to the CLI and always saved to file after the pipeline completes.

  • The directory bookmark format has been removed. This bookmark format is no longer used. (GL689)

Enhancements

  • A new warning is now displayed when no --project file is set, and there is no project.ini file present in the current working directory (GL655).

  • Add pagination to RiskScape list and info commands. This lets you page through large amounts of data similar to using the more command (GL430)

  • CPython functions now accept the anything argument type. (GL600)

  • When a model or pipeline is run, some performance metrics for the pipeline are now saved in the output directory as a stats.txt file. (GL411)

  • The riskscape bookmark info command now always examines the data that the bookmark contains, by default. Previously, this would only happen when the --measure option (now removed) was used. Use the --brief CLI option to skip examining the bookmark data. For example, if the bookmark contains a lot of data, then the command could take a while to run. (GL609)

  • Projects now support an auto-import feature, which loads any INI files in the same directory or below. This replaces the RiskScape home directory functionality, which has been removed. Auto-import is enabled by adding auto-import = true to the project.ini file (GL663).

  • Plugins may now be added to RiskScape via the settings.ini file. (GL681)

  • New tutorials have been added to RiskScape’s documentation for building and running models, as well as for creating RiskScape projects. (GL-650)

Fixes

  • Cycles in pipelines are now reported with a descriptive error message instead of causing RiskScape to exit unexpectedly (GL615)

  • Fixed issue on Windows where RiskScape would not run if it was installed in a directory containing spaces, e.g. C:\Program Files\RiskScape. Note that the Windows shortcut script instructions have also been updated to handle spaces in the installation directory path. (GL644)

  • Fixed a bug where segmenting lat/long geometry could fail. The failures were caused by the geometry becoming invalid during a reprojection. (GL640)

  • Fixed a bug where RiskScape would appear to hang if the project contained an HTTP bookmark and the target server was not responding. (GL667)

v0.11.0

Built 10 December 2021

Breaking

  • Some pipeline steps have been removed (or hidden) from RiskScape (GL526). These steps have been superseded by improvements to the RiskScape expression language that allow the same functionality with simpler, more general purpose, constructs.

    The removed steps are:

    • aggregate - replaced by the simpler and more expressive group step

    • function - can be done more simply with a select step

    • sample - can be done more simply with a select step and using one of RiskScape’s sampling functions

    • joinloop - no longer used and can be replaced with a join and group

    • flipaxis - not used and deprecated in favour of correct use of CRS

  • The unnest pipeline step now expects the to-unnest parameter to be an expression. This used to be a list of strings. To convert existing pipelines simply remove the quotes from around the to-unnest lists. E.g. unnest(to-unnest: ['foo', 'bar']) -> unnest(to-unnest: [foo, bar]) (GL526)

  • The sort pipeline step has changed how the sort-by parameter is specified (GL526). This used to be a list of strings like:

    sort(sort-by: ['foo desc', 'bar asc'])

    Now the sort-by parameter is an expression containing the attributes to sort by. The sort direction is set in a new parameter direction which is a list containing the sort direction for each attribute. E.g

    sort(sort-by: [foo, bar], direction: ['desc', 'asc'])

  • Some wizard model parameters have been renamed to reflect the latest model workflow terminology (GL622). This means that any wizard models built and saved using RiskScape v0.10.0 will no longer run successfully. Note that pipeline models saved on v0.10.0 will continue to work - these models would have been saved by RiskScape in a models_pipeline_<your-model-name>.ini file.

Enhancements

  • The wizard now shows a menu at the completion of the model survey instead of running the model. (GL551)

  • When the wizard is given a vector-based hazard (e.g from a shapefile), it now lets you pick a selected attribute from the hazard layer to use as the hazard intensity measure. This lets you use both vector and raster hazards with the same consequence function. It also lets you aggregate the hazard intensity measure from the vector layer (i.e. take the max value) when matching all geometry intersections between the hazard-layer and your elements-at-risk. (GL546)

  • RiskScape has upgraded the core spatial libraries that are used. Of particular note is JTS. The updated version of JTS contains features that RiskScape can now utilize to fix invalid geometries. The GeoTools library has also been upgraded to v26.0. (GL500)

  • User-defined functions, such as your risk functions, can now be implemented using CPython, instead of the default Jython implementation. If you have used Python previously, you will find importing modules is simpler using CPython, and it lets you use a wider range of Python packages (e.g. numpy, scipy, etc…). See the how-to guides for help on setting up RiskScape’s CPython integration. (#328)

  • Geometry validation modes ERROR and WARN will now try to fix any invalid geometries. The ERROR or WARN action will only occur if the geometry is unable to be fixed automatically.

    The default mode for geometry validation is now ERROR. (GL500)

  • Geometry predicate functions such as intersects will now produce an error if the input geometries do not have the same coordinate reference system. (GL588)

  • RiskScape will now use a datum aware transform when reprojecting geometry, or warn that reprojections do not account for datum shifts. (GL589)

  • If no other project or home directory options are set, then RiskScape will default to using a project.ini file from the current working directory. (GL586)

  • crs-longitude-first now defaults to true on shapefile bookmarks. This is to match the shapefile format which specifies an X/Y (longitude/latitude) axis order. (GL604)

  • There is now simple support to use pixel-based raster formats as exposure layers - you can now use an ASCII raster or GeoTIFF as input for a pipeline, or, as your exposure layer in the wizard (GL#593).

  • RiskScape can now run on Java 17, the latest Java version with long term support. Users are encouraged to upgrade to Java 17 soon, a future (as yet unplanned) release of RiskScape will require Java 17. (GL592)

Fixes

  • When writing to shapefiles RiskScape will reproject geometries to an X-Y axis order if they are not already. The shapefile format requires geometries to be in the X-Y order. But many coordinate reference systems are have a Y-X axis order (e.g EPSG:2193 New Zealand Transverse Mercator 2000).

    This issue mostly affects data from CSV data sources. Until now it has been worked around by using bookmarks with crs-longitude-first with creates the data with a X-Y axis order. (GL556)

  • to_coverage will now return the same coverage each time it is called if the input arguments are constant. There was a bug which caused a new coverage to be created on every call which caused some models to run a lot slower. (GL581)

  • Fixed a bug where a bad function call expression would trigger an unexpected error when inside a struct declaration, e.g. {square_root('nine')} (GL575)

  • Cutting geometries could fail when the projection file is missing datum shift parameters. This would affect input data that is:

    • lat/long or long/lat

    • not using the WGS84 datum

    • datum shift parameters (TOWGS84) not in the *.prj file RiskScape has now updated how the reprojection required for cutting are done. (GL537)

  • Fixed issues with the terminal where:

    • on Windows the status updates could overwrite other output, such as the pipeline DSL in the wizard

    • aborting a model run could leave the bottom part of the terminal unusable (where the status updates were)

    • aborting a model run on Windows could output junk characters (GL565)

  • Fixed some bugs when using WFS data sources:

    • geometry from WFS sources could not be reprojected (GL597)

    • setting crs-name on WFS bookmarks would cause errors (GL598/GL599)

  • Fixed a bug where cutting geometries could fail if the geometry contained longitudes over 180 degrees. This would affect geometries from the Chatham Islands in EPSG:4167 (NZGD2000). (GL595)

  • Setting crs-name with shapefile now correctly assumes longitude first (GL604)

  • Fixed a bug where sampling a GeoTIFF could return not a number (NaN) when the GeoTIFF is using NaN as the no data value. In this case <nothing> should be returned. (GL596)

0.10.0

Built 30 September 2021

Breaking

  • The riskscape model run command has been revamped. The major difference you will notice is with the -o option. The -o option is now optional rather than required, and is short for --output directory (whereas previously is was short for --outfile filename). The command will now always write to file by default, and default to shapefile when geometry is present in your results. To change the output file format to something else, like CSV, please use the -f or --format CLI option. (GL401)

    If you have already been using the riskscape beta model command on previous releases, then you can now run riskscape model and should not notice any difference.

  • You can still run saved models that are based off the table or default templates. However, the default and table template models themselves will no longer appear in riskscape model list. You will only be able to run the models that appear in riskscape model list.

  • The riskscape model info command has been removed. Use riskscape model framework template info instead.

  • The non-point model template has been removed. The wizard should now be used for this type of modelling.

  • The riskscape model batch command has changed. (GL306)

  • The PARAMETERS keyword is no longer supported for Jython functions. Instead, you can now use model pipeline parameters. Refer to Function parameters for an example of how to use these parameters with your function. (GL451)

  • RiskScape no longer reads Z or M coordinate values from shapefiles. RiskScape never made use of these extra coordinate dimensions so this change is unlikely to be noticed by users. The reason for this change is that the extra coordinate dimensions could trigger other problems, most likely after a segmenting operation. (GL484)

  • When a struct declaration uses the * select-all or ‘splat’ operator, it now allows struct members to be replaced, e.g. input(value: {name: 'bob', height: 1.74}) -> select({*, height: round(height * 1000)}) replaces the height attribute. This makes it easy to replace parts of a struct that came from the input without having to resort to a cumbersome function like merge_struct. Note that it is still illegal syntax to redeclare a struct member explicitly, e.g. {loss: 12, loss: 13}

  • Cutting geometries using the segment function (like the wizard geoprocessing options can do) now always aligns the grid to the bottom left corner of the feature’s bounding box. Previously it would align either to the centroid or 1/2 a grid distance off the centroid, depending on which produced a smaller grid envelope. This can result in slightly different modelling results for non-point models.

  • The asset_at_risk_ratio function has been removed. This function was used to get ratio of an asset’s exposed size. This can be done with the following expression measure(exposed_geom) / measure(geom) (GL401)

  • The built-in geocode function has been renamed to is_exposed. The is_exposed function now may also be called with a third resource argument. The is_exposed function will now be displayed in riskscape function list by default, and can be selected when building models in the wizard. Because the exposure and hazard values passed to the is_exposed function can be any type, it makes it a versatile placeholder when building a basic model. (GL531)

  • The --force or -f option on the pipeline run and eval commands have been changed to --replace or -r. This change now makes the CLI options consistent with the model run command. Note that the -f option is now used for --format. (GL533)

  • The NetCDF bookmark resolver has been changed to produce a relation. Previously it would return a coverage, but it was not possible to access all the data from a multi-dimensional NetCDR dataset.

    The resolver can be used to access the data from one or more user defined layers from the NetCDF file. Note that the NetCDF bookmark is part of the optional NetCDF plugin, and so it is only available if you install the plugin. (GL524)

Enhancements

  • The riskscape wizard command has changed considerably. The wizard now contains support for multiple hazards. Several different hazard-layers can now be combined and sampled in a single operation. The geoprocessing features can now be applied to any layer in your model, not just the exposure-layer. Models created and saved using the wizard on previous releases will continue to work. The old wizard interface is still available by using the riskscape wizard --legacy command. (GL523)

  • The wizard supports undo to go back a question via the <ctrl>-c menu. This makes it easy to go back and change your responses if you make a mistake or try something out, without having to start the wizard process from the start. (GL535)

  • The wizard now makes it easier to customize the results output files produced by your model. You can now interactively pick which attributes to include in the results file, and build aggregation and sort expression easily using the wizard. (GL471)

  • When writing your own pipelines, RiskScape will now assign default names to all pipeline steps. Previously, RiskScape would only assign a default name to the first pipeline step of a given type. Now you only need to specify a name if you want to reference a step elsewhere in the pipeline. (GL440)

  • The map function now accepts non-list arguments. Instead of giving an error, the function will apply the given lambda expression to the argument that wasn’t the list. This has been done to make expressions more reusable and succinct when something may or may not end up being a list, depending on prior steps.

  • A general purpose HDF5 bookmark resolver has been added. This resolver allows users to turn an HDF5 dataset into a relation. Bookmark options exist to configure which dataset (within the HDF5 file) is to be used and which attributes of that dataset should be included. Note that the HDF5 bookmark support is part of the optional HDF5 plugin, and so it is only available if you install the plugin. (GL439)

  • A segment_by_grid function has been added. This new function is similar to the existing segment function except that:

    • it lets you align the grid to a specific to-align input

    • it cuts line geometries using a grid (whereas segment cuts them to length)

    The to-align input may be either a point or a coverage (e.g. a hazard raster file you want to sample). This function now lets you segment your exposure-layer geometry consistently using a common grid. (GL478)

  • A reproject function has been added. This function reprojects geometries to the desired Coordinate Reference System (CRS). The desired CRS can be either text (‘EPSG:4326’) or another geometry that is in the desired CRS. (GL497)

  • A new validate-geometry option has been added to the project INI file and also to relation bookmarks (GL283). This can be set to warn or fail when invalid geometry is detected by RiskScape. Invalid geometry can sometimes occur when non-point geometry is modified, such as during reprojection or cutting. Sometimes invalid geometry may simply be present in your input data file. Note that this setting has a slight performance hit and so is off by default. Invalid geometry may result in a TopologyException when the validate-geometry option is disabled.

    In the future the validate-geometry setting will be extended to include the option of automatically fixing the invalid geometry, if possible. This enhancement is pending a new JTS library release, which will contain new features to support fixing invalid geometry. (GL283)

  • The skip-invalid bookmark setting is now enabled by default. If any invalid tuples (i.e. rows or records of input data) are encountered then a warning will be output but the model pipeline will continue running. skip-invalid can still be disabled for any given bookmark, i.e. to stop running the model pipeline immediately as soon as an invalid tuple is encountered. (GL510)

  • Any -p parameters that are given to riskscape model run command that start with ./ will now be expanded with the full path of your current directory. Previously it would try to resolve relative file-paths and URIs relative to the project’s URI, which could be confusing when the project file was not in the current directory. See (GL511) for more details.

  • The details of a Python function can now be defined in your project.ini file. (GL447) This can make it simpler to define the argument-types. For example:

    [function foo]
location = example.py
argument-types = [floating, integer]
return-type = floating

  • A new combine_coverages function has been added for multi-hazard pipeline support. This lets you combine multiple coverages into one and sample multiple hazard or resource-layers with a single operation. (GL472)

  • RiskScape now supports GeoJSON (GL218) and KML (Keyhole Markup Language - GL203) for both input and output files.

  • The to_coverage function has been enhanced to create a coverage from a bookmark relation directly. Previously it could only be used as an aggregation function. The relation_to_coverage function has been deprecated as to_coverage is now equivalent. (GL490)

  • Additional options have been added to the to_coverage function that now let you turn vector data containing point geometries into a nearest neighbour coverage. This allows for better spatial matching when using a point-based hazard layer (e.g. a site mesh), which is typically used in HDF5 and NetCDF datasets. (GL485)

Fixes

  • The wizard would throw an unexpected exception if an analysis function was chosen with an ID that required quoting in the pipeline DSL. Quoting will be required when the function ID contains characters that are used as part of the RiskScape expression language, in this case - would be the most likely. The wizard now quotes function IDs when this is required. (GL419)

  • Shapefiles that contain special characters (such as macrons) would not be read correctly. This would result in unusual characters in the output where the special characters should have been. RiskScape now uses the <shapefile>.cpg file to select the character encoding to use which should fix this issue. RiskScape will now also create a <shapefile>.cpg file when writing shapefiles. This fix uses new a new GeoTools feature that was part of the 25.2 release. (GL481)

  • Relative paths have been resolved incorrectly in model.ini and pipeline.ini files that are in a different directory to the project file. A location = path-to-file should be resolved relative to the file containing it. But model.ini and pipeline.ini files were resolving these relative to the project file location. This has now been fixed. (GL512)

  • Warnings from pipeline step realization were been treated as errors and preventing the pipeline from running. This has been corrected now and any warnings are logged but the pipeline will run as expected. (GL486)

  • Struct and list expressions have been treating the comma separating elements as optional. That is [1 2] would be treated as [1, 2] for lists, and {foo: 1 bar: 2} would be treated as {foo: 1, bar: 2} for structs. This behaviour could be unexpected or confusing for users. Now RiskScape requires the comma separator in these cases. (GL514)

0.9.0

Built 8 June 2021

Breaking

  • Previously, the percentile(s) aggregation functions were using the nearest-rank inclusive method when used for list aggregation or group pipeline steps, and nearest-rank exclusive when used with the aggregate pipeline step. This has changed so that now nearest rank exclusive is used consistently for all percentile aggregation. (GL394)

  • The new count aggregation function (used by the group pipeline step, and for aggregating lists) has been changed so that when aggregating boolean values, only those that are true get counted. This is now consistent with the older count function (used by the aggregate pipeline step). (GL397)

  • The seismic plugin has now been removed. This has been replaced by the hdf5, openquake, and usgs plugins, as well as by general-purpose new ‘core’ pipeline features. (GL416)

Enhancements

  • The sort step may now be used to calculate a delta between the sorted items. This could be used to sort items based on value and to calculate the value difference between items. (GL398)

  • Expressions now have some support for binary operations on structs, e.g. {a: 10, b: 20} + {a: 3, b: 12} works by applying the operation to all matching members. It will not apply if the left hand side of the operation has extra members or if any of the pairs do not have a compatible operator. Support is also present for applying an operation on a struct and a scalar value, e.g. 2 * {a: 3, b: 4} results in {a: 6, b: 8}.

  • RiskScape has upgraded the core spatial libraries that are used. Of particular note is JTS. The updated version of JTS contains features that RiskScape can now utilize to reduce (or hopefully eliminate) the occurrence of Topology Exceptions. The GeoTools library has also been upgraded to v25.0. (GL340)

  • The buffer function now has options that may be used to control how geometries are enlarged. By default buffer will round off external corners. If this is not desired, then the new options can be used to mitre or bevel corners. Lines can also have square or flat ends. Flat ends do not cause the line to increase in length. (GL391)

  • The wizard now allows the exposure-layer geometries to be enlarged in the geoprocessing phase. The primary use for this is to convert road centre lines into road polygons. When enlarging there are further options to control how geometries are enlarged, including whether any resulting overlaps should be removed. (GL383)

  • Multi-threaded execution is now enabled by default. The --pipeline-threads option is now set by default to the number of processors (or CPU cores) that are available. This should provide the optimal performance. (GL405)

  • Models that use the pipeline framework now support custom parameter replacement. For example, $ can now be used in the pipeline definition to declare a named parameter that can be overridden or replaced when the model pipeline is run. For example, if you declare a $foo parameter, then it can be overridden by --parameter foo='bar' on the command line. Note that any such parameters also need to be defined in the project.ini file using param.name = default-value. (GL-414)

  • When a pipeline framework model is run, a copy of the pipeline definition is now saved in the output directory. This means you can always see the exact pipeline code that was used to produce a given result. This is particularly helpful now that custom $ parameters are supported in the pipeline definition, as it records the actual replacement value that was used. (GL-404)

  • Parameters can now be passed to the beta model run command in an INI-format file. When changing many different parameters in the model, this makes it more convenient to save them in a file and reuse that, rather than typing them all out on the command line each time. (GL409)

  • The default output-base-location may now be specified in the project.ini file. The output-base-location is the base directory that output is written to by the pipeline and beta model commands. If not specified then ‘output’ is the default. (GL406)

  • New aggregation functions bucket and bucket_range have been added that allow for very flexible customization of the aggregated results. These are conceptually similar to SUMIF spreadsheet operations, in that they can aggregate based on a given condition, such as whether the underlying data falls into a certain category. bucket_range is useful for aggregating against predefined bands, for example, counting individual losses that fall within a specific dollar value range. (GL-377)

  • The underlying execution of pipelines and models has been changed to use a more task-centric, scheduler-based approach. This new approach means that model execution can be better tailored to efficiently process individual RiskScape pipeline steps, such as better parallelization of aggregation steps. (GL-266, GL-334, GL-407)

  • Interpolation is now supported with loss-modelling functions, using the new create_continuous and apply_continuous built-in RiskScape functions. This allows efficient execution of a pipeline when the loss function is computationally expensive, or when there is a very large number of data-points. In addition, when a value is common across multiple data-points, such as multiple elements-at-risk that map to the same hazard value, the stack_continuous function can also be used for further efficiencies. (GL-412)

  • Real-time progress diagnostics have been added when running a model. This now clearly shows the number of rows (i.e. exposure/hazard data-points) that are processed by each step of the underlying pipeline. (GL-399)

  • RiskScape pipelines now support dynamically changing the details of a bookmark on the fly, including the bookmarked file location. This can be useful for combining many different data sources, such as for probabilistic modelling. For example, a bookmarked data source can now contain a list of other input files that RiskScape should use. (GL-417)

Fixes

  • Previously RiskScape could throw Topology Exceptions when some geometry operations such as intersection or difference were applied. These exceptions where caused by long standing issues related to rounding, in the underlying JTS library that RiskScape uses for geometry operations. The JTS library has been updated and has added new functionality for dealing with these operations that should eliminate these Topology Exceptions. Now RiskScape has been updated to use these new features of the JTS library. (GL369)

  • Previously RiskScape would let you save a partially complete model when you were halfway through the wizard, but would then refuse to run the saved model. This has now been fixed. For example, this allows you to save only the geoprocessing actions as a model, and then re-run the ‘model’ on different input data files. (GL323)

  • On some Windows systems, RiskScape v0.8.0 could log the following message when saving a shapefile: Could not open/create prefs root node Software\JavaSoft\Prefs at root 0x80000002. This message was harmless and can be ignored. It now no longer gets displayed. (GL373)

  • Previously, if you had a multi-geometry exposure-layer and you used the wizard cut-by-layer option to cut it against a layer in a different CRS, it could result in an error message like Could not find a suitable transverse mercator based projection.... This issue has now been resolved (GL370)

  • When the wizard was used to save either geoprocessing or analysis raw results an exception would be caused later on if a report filter option was used. This has been fixed (GL372)

  • When the wizard cut-by-layer option was used against a layer in a different CRS, in combination with buffering when sampling (i.e. the hazard-buffer option), then RiskScape would fail to match anything against the buffered geometry. This issue has now been resolved (GL375)

  • The shapefile_geotools writer would fail when writing a relation that only contained geometries that were created by the createPoint function and did not have a crs-name set as part of a bookmark. Now the shapefile will be created as expected, but the projection file will be empty as that is unknown. (GL314)

  • When the wizard cut-by-layer option was used against a layer in a different CRS, and ALL_PARTS was used, then the difference could appear to duplicate the exposure. This was due to reprojection introducing small changes to the geometry that were large enough to produce a difference. Now in this case the cut-by-layer is reprojected up front to prevent this issue. But this is only possible if the features in cut-by-layer are known to fit inside the projection bounds of the exposure-layer. (GL376)

  • When the wizard cut-by-layer option was used any multi-geometries that were found would be added to the result list individually. If ALL_PARTS was used then this would also be done for the difference if that was a multi-geometry. This would be done whether the input geometry was itself a multi-geometry or not. This could cause the exposures modelled to be far larger than expected if they were multi-geometries.

    Now any resulting multi-geometries are only added to the resulting list by their parts if the input geometry was not a multi-geometry. So for a line exposure, if multi-line intersections were found, then each part would be added to the list separately. But if the input was a multi-line then the multi-line intersection would be added to the list. This applies to both intersections and differences. (GL382)

  • The to_coverage function used in the geoprocessing filter by layer/attribute step could trigger unexpected exceptions. This would be more likely to occur if the filter by layer contains a larger number of features. Processing has been amended to no longer trigger this exception. (GL390)

  • RiskScape could produce CSV (comma separated values) files that could not be opened by Microsoft Excel. The reason for this was that the file contained columns with more than 32,767 characters which is an Excel limitation. Now RiskScape will truncate columns that would otherwise exceed this limit. (GL384)

0.8.0

Built 3 March 2021

Known Issues

  • Wizard ctrl-c support has been disabled in Windows while we fix (GL371)

  • Using the new shapefile format on windows can produce a harmless but annoying log message

Breaking

  • measure, measure_length and measure_area functions have been enhanced to measure in metres. Previously these functions returned a measurement that was in the units of the geometry’s CRS (Coordinate reference system) which in the case of lat/long CRSs would be degrees. (GL26)

  • rasterize-grid-size is now specified in metres. Bookmarks allow a relation to be rasterized in which case rasterize-grid-size is used to specify how big each raster cell should be. Previously rasterize-grid-size was specified in the unit of the relation’s CRS (Coordinate reference system) which in the case of lat/long CRSs would be degrees. (GL325)

  • beyond, buffer and dwithin geometry function now have the distance argument in metres. Previously these functions required the distance argument to be in the units of the geometry’s CRS (Coordinate reference system) which in the case of lat/long CRSs would be degrees. (GL115)

  • Aggregation functions p75, p90, p95 and p99 have been removed. These functions were aliases to percentile(value, percentile: xx) and were created to allow these percentiles to be used from the wizard. These functions are not required now as the wizard will ask for the percentile should that aggregation be selected. (GL335)

  • The --force option for the riskscape beta model run command has been renamed to --replace. (GL343)

  • The capture pipeline step has been removed. The save pipeline step can be used in its place. (GL321)

  • How shapefiles get written for the pipeline run and model run commands has now changed slightly (GL310). This may now save the shapefile attributes with slightly different names compared to the previous RiskScape version. Instead of asset.name, agg.name, etc attributes will now be saved as name, name1, etc. To restore the old behaviour, you can use --format=shapefile_geotools on the CLI.

Enhancements

  • Performance has been significantly improved when writing large shapefiles. Note that the attribute column-naming behaviour has changed slightly as a result. You will benefit from the improved performance whenever you use --format=shapefile with the riskscape model run command. Note that by default, the old (slower) shapefile writing behaviour will still be used when --format is omitted, e.g. if you use riskscape model run --outfile=output.shp. (GL310)

  • Wizard now has a geoprocessing phase for processing the exposure layer. Available options are:

    • filter, to include only desired exposure features

    • cut-distance, to segment exposure features that are larger than desired size

    • adjust-attributes, to scale attributes by the ratio of cut/original feature (GL313)

  • wizard results now include an exposed_ratio attribute when all_intersections hazard sampling is used. The exposed_ratio may be used to scale losses to the part of the exposure that is actually exposed. (GL311)

  • Wizard now has option to save raw results when all_intersections hazard sampling is used. The raw results include:

    • hazard intersections

    • consequence (when aggregating consequences)

    • exposure The raw results may be useful to show the workings of RiskScape which may be required for auditing purposes. (GL119)

  • Wizard now offers to recalculate the exposure layer’s attributes after cutting (GL337). This lets a user decide what to do to each attribute on a case by case basis.

  • Additional geometry types have been added for line, multiline, point, multipoint, polygon and multipolygon. These specific geometry types can be used instead of geometry if only that geometry type should be used. Care must be taken when dealing with shapefiles as they read all geometries as a multi version (due to the shapefile data format). (GL285)

  • A cut-by-layer option has been added to the wizard’s geoprocessing functionality. This lets you cut the lines/polygons in the exposure-layer by another vector layer (e.g. regional boundaries) before running the model. (GL99)

  • When RiskScape detects a duplicate item (type, function, bookmark, model etc) with the same ID as an existing item, then the error now includes the location of both items. This will make if much easier to find and remove the duplicate item. (GL305)

  • The wizard now lets you specify the file format that the model results should be saved in (i.e. CSV or shapefile). (GL321)

  • A host of new sampling questions that allow the modeller to specify a proximity when sampling from the hazard and area layer, as well as buffering non-point exposure geometry. These allow a margin of error when exposures might not lie exactly over your other layers, such as ports on district boundaries, or flood maps that have been produced using the exposure layer. (GL362, GL361 and GL348)

  • The interactive wizard now supports being interrupted with ctrl-c, which brings up a context menu of wizard actions. Currently disabled on Windows due to GL#371.

  • Similar to GL119, the results of the geoprocessing phase can optionally be saved to get a view of the modified exposure layer before other processing phases are applied (GL365)

Fixes

  • Wizard support for area and resource layers has now been improved. The wizard now supports these layers as either vector or raster input data. Previously, the area-layer had to be in the same CRS as the exposure-layer, but now these layers can be in any CRS. (GL309)

  • Previously an unexpected exception could occur when running RiskScape over shapefile input data. There were two possible causes, both of which have now been fixed:

    • The shapefile contained null geometry (GL317). Any records in the shapefile with null geometry are now treated as invalid tuples, and can be skipped over by specifying skip-invalid = true in the bookmark.

    • The shapefile contained Z-values, but no corresponding M-values (GL265). These shapefiles are now successfully read, as the M-value is optional.

  • When RiskScape read integer values from a shapefile they where not being converted to the correct RiskScape INTEGER type. This caused problems when running expressions that used these attributes. Now RiskScape is converting these attribute values to the correct RiskScape type. (GL319)

  • RiskScape can now read CSV files created by Excel. Previously reading these files would mislabel the first column because Excel start the file with an invisible byte order mark. Now RiskScape tolerates the inclusion of these characters and does not include it when assigning the name to first column. (GL230)

  • Wizard now asks additional questions when aggregations are chosen that require extra configuration (such as percentile aggregation). Previously using percentile aggregation would result in an invalid model as the extra configuration was not collected. (GL335)

  • RiskScape was silently ignoring lines in a INI file type definition that contained typos. For example, if you had typ.name instead of type.name. You will now see ‘spurious attribute’ warnings in the riskscape type list output. (GL322)

  • If a bookmark specified crs-name but did not contain a geometry member then an unexpected error would occur. Now should this occur an error is output saying that a geometry member is required. (GL352)

0.7.4

Built 4 November 2020.

Enhancements

  • More functionality has been added to the interactive wizard (GL290), although it’s still a beta feature. This includes support for:

    • saving the wizard model to INI file.

    • better handling of struct-based (i.e. composite value) hazards or consequences, by aggregating where necessary.

    • renaming columns in the output file.

    Note that the area-layer support will be improved in the next release (currently one limitation is that the area-layer needs to be in the same CRS as the exposure-layer).

  • A new set of riskscape beta model commands have been added. These commands support running new models that were saved in the wizard. Anyone with existing (template) models can continue using the same riskscape model commands they currently do and should not notice any difference. Eventually the riskscape model commands will be replaced by the new beta commands. Note that anyone who wants to try using the new beta commands with a template model, will first need to manually add framework = templated to their model’s INI definition.

  • A new set-attribute parameter has been added to bookmarks. This is an alternative to the existing map-attribute parameter. The difference is set-attribute can be used without specifying a type for the bookmark, whereas map-attribute always requires a type. The idea is that you can use set-attribute to manipulate the input data, such as renaming an attribute in a shapefile, without being forced to define unnecessary types in RiskScape. (GL288)

  • More aggregation functions now support lists as arguments. These include: median, mode, stddev, and percentile (including p99, p95, p90, and p75). (GL264)

  • A new beta pipeline group step (GL264) has been added that allows grouping and aggregating in more consistent, flexible and powerful ways, by making use of aggregate expressions. While this feature is working, it is currently missing a few important aggregate functions (compared to the old style functions) and will not scale as well with large datasets like the existing implementation can (likely leading to out of memory errors).

  • The glossary has been updated with new proposed terminology to use with RiskScape going forward.

0.7.3

Built 30 September 2020.

Breaking

  • The select pipeline step now accepts just a single expression parameter. In practice, most users will not notice a difference, but some pipeline expressions may need to be amended to produce identical results as before the change. (Part of GL213)

Enhancements

  • A beta version of an interactive wizard for building model pipelines has been added. To run the wizard, point RiskScape at your home directory or project file containing your bookmarks and types, and enter the riskscape wizard command. (GL274)

  • A new API for aggregation functions that can also work as normal functions (GL#137). These will eventually be used in a new type of expression, an aggregation expression.

  • A new framework has been added for constructing aggregation functions from expressions using a classic map/reduce style, along with several implementations that make use of it (mean, max, count, sum, to_list).

  • A few new language functions for dealing with lists (append, concat) and nulls ( if_null and null_of) have been added.

  • Expressions now allow use of the select-all * operator (also known as ‘splat’) (GL200). This is helpful for various scenarios where input is copied from one pipeline step to another, or for manipulating structs within an expression. For example, select({*, hello: 'world'}) will append hello world to whatever was emitted from the previous pipeline step.

  • Expressions now support lambdas (GL37). Two new functions have been added, map and call, that use lambdas For example, a list of integers can be turned in to a list of strings with the following expression: map([1, 2, 3], el -> str(el)).

  • More complicated log configurations can now be set from the command line to target specific loggers, e.g. --log-level info,.engine.i18n=error,.engine.rl=debug would set default logging to info level, i18n to error level and RiskScape expression language code to log at debug level.

Fixes

  • When the riskscape type-registry list command was run, RiskScape would exit with an exception. This problem only affected RiskScape v0.7.2 and has now been fixed. Additional unit tests and integration tests have now been added to avoid this type of problem occurring again. (GL276)

  • RiskScape error messages have now been improved when running a model or pipeline that uses vector data (i.e. shapefiles) that contains:

    • Shapes with invalid geometry. Or

    • Shapes with geometry that, when reprojected between two different CRS, becomes invalid.

    Previously, RiskScape would just exit with an EvalException, resulting from a TopologyException. Now RiskScape will provide more details on how to resolve this problem. (GL279, GL281)

  • When running a non-point-asset model with vector data, or using the relation_to_coverage function directly, you would see a message like: WARNING There is code leaving shapefile readers open, this might result in file system locks not being cleared. Depending on the size of the input data, this may have resulted in an exception with the message: Error: Maximum lock count exceeded. This issue has now been resolved. (GL279)

  • When using a non-point-asset model or pipeline with vector data, and the crs-name bookmark parameter is used, it was possible that RiskScape could exit with an UnknownSRIDException when manipulating the geometry. This issue has now been resolved. (GL281)

0.7.2

Built 13 August 2020.

Enhancements

  • Improve performance for the seismic plugin’s riskscape seismic convert-relation command. Converting a large HDF5 file to OQRS format should now run approximately five to ten times faster. As part of these changes, two parameters have been renamed slightly:

    • --cursor-size has been renamed to --cursor-size-mb to make it clear what unit is expected.

    • --quiet has been replaced by --verbose. Quiet is now the default, but you can use --verbose to restore the old behaviour.

  • To help ensure your CRS settings for bookmarks are correct, the geographic bounds used by your model are now displayed whenever a model is run. You can also see more details about a bookmark’s CRS, axis-order, and bounding envelope by using the riskscape bookmark info <name> --measure command. Refer to Axis/Ordinate Order for more details. (GL252)

  • RiskScape expressions now allow hyphens in keywords without needing escaping. This fixes GL232 by avoiding having to change any function or pipeline parameters to avoid or workaround hyphens in their names. For example, join(intersects(foo, bar), join-type: 'INNER') would have previously required quotes around join-type.

Breaking

  • The bookmark crs-force-xy option has been renamed to crs-longitude-first to better reflect what it does. Currently the old crs-force-xy option will continue to be accepted, but may be removed in future. (GL252)

Changes to error reporting

  • Failed functions, bookmarks etc are no longer ‘removed’ from your project on start up. Instead, when they are referenced, e.g trying to use a function in your model, the failure is linked to the parameter or thing that tried to use it to show the user a detailed and traceable error message to help them identify and fix the problem.

  • CLI list commands now show failed things in their listings, along with a description of the failure message.

  • Errors building your project are now hidden unless you run RiskScape with --show-project-errors. Instead, the number of errors is shown with instructions on how to see more details on these errors.

  • Reusing IDs for functions, bookmarks, etc will now cause RiskScape to not run until the collision has been removed. Previously RiskScape would somewhat inconsistently replace or ignore duplicates.

Fixes

  • The --stats-out option was not recording any metrics when a model was run. When a pipeline was run, it did not reliably record metrics for the last second of execution. This has now been fixed. (GL252)

v0.7.1

Built 3 June 2020.

Enhancements

  • New built-in RiskScape functions have been added to support randomness. (GL244) These include:

    • random_choice() for randomly selecting an item from a list. Optionally, a weighting can be specified for each list item.

    • random_uniform() for randomly selecting a floating-point number within a given range.

    • random_norm() for randomly selecting a number from a given normal distribution.

  • A new exp() built-in RiskScape function has been added. For example, to get the constant e, you can now use exp(1). (GL247)

  • The riskscape function info <id> command will now display more detailed help for some built-in RiskScape functions, e.g. polynomial or lognorm_cdf.

  • The non-point model now supports a vector-based hazard layer. A new hazard-expression parameter has been added to the model for accessing or modifying the hazard value. (GL98)

Fixes

  • Some converted OpenQuake datasets (OQRS) files were failing to be read because of a buffer underflow bug in the OQRS parsing code. This has now been fixed (GL253)

  • In previous releases, the norm_pdf() and lognorm_pdf() functions were available for use in RiskScape expressions, however these functions were not fully implemented (and so they always returned zero). These functions now correctly return the Probability Density Function result. (GL242)

  • Improve the error message that gets displayed when trying to use a GeoTIFF file that is in an ESRI proprietary format. (GL248)

  • A more informative error message is now displayed when referencing a bookmark that does not exist, e.g. if the bookmark name contained a typo. (GL208)

  • Some minor improvements to type variance for function parameters have been made. For example, previously a function with a parameter type anything would incorrectly accept nothing or null. (GL244)

v0.7.0

Built 5 May 2020.

Enhancements

  • Some improvements have been made to RiskScape’s error reporting, in particular when parsing files and expressions. More error messages now support internationalization (i.e. are capable of translation).

  • Advanced users can now define their own custom Pipelines using a new purpose-built text file format. For an example of building your own pipeline from scratch, see How to write basic pipelines. Note that pipelines are a beta feature that are still evolving.

Breaking changes

  • RiskScape output on Windows now defaults to UTF-8 encoding. If your terminal is not setup to handle UTF-8 characters, RiskScape will now automatically change your terminal encoding (called the code page) the first time RiskScape runs. When RiskScape does this, a warning will be displayed to let you know this is happening, e.g. Changed console to use UTF-8 encoding (was chcp 850). You don’t have to worry about this message, and shouldn’t notice any difference to your terminal. In previous releases, without the UTF-8 encoding, some characters (e.g. ellipsis) were rendered strangely in the RiskScape output. If, for some reason, you need RiskScape output encoded in a different code page, then in your terminal run set RISKSCAPE_OPTS=-Dfile.encoding=cp1252 before running RiskScape to restore the old behaviour. You can also use chcp 850 to set the terminal encoding back to the default. (GL190)

  • The language that pipelines are defined in has now changed. Most users would not have needed to use pipelines directly, so should not be affected. However, if you did have a pipeline defined in your project file previously, then this will need to be rewritten. Many pipeline steps have been renamed. Pipelines that contain steps with old names will no longer work. Refer to riskscape pipeline step list for available pipeline steps. (GL123)

  • If the hazard argument-type for a model’s function is not defined as nullable, then the function will no longer be called for null hazards (i.e. where no hazard overlaps with a given asset). This means you may notice a slight change in the model’s results for exposure functions that produce output information even when the hazard is null. To define a nullable hazard argument-type for a Python function use:

    ARGUMENT_TYPES = ['my_asset_type', Nullable.of(typeset.getLinkedType('my_hazard_type'))]

    Note that for simple coverage-based hazards, you can alternatively define your hazard type itself as nullable. For example:

    [type flood_hazard]
type = nullable(struct(depth: floating))

  • The default column heading produced by group-by expressions has changed slightly. Group-by expressions will infer an output name if one was not provided. For nested tuple accesses like asset.construction the inferred name was the same as the expression, leading to confusion as the output property name looked like a nested property access. Now any dots . in the inferred name are changed to an underscore _. E.g. riskscape model run --group-by=asset.construction now produces a field called asset_construction instead of asset.construction. (GL50)

Fixes

  • Shapefile output produced by RiskScape could contain null values which is not supported by ESRI software. Now RiskScape will map null values to other constant non-null values to ensure shapefiles can be used in ESRI. The mapped values are:

    • integer/floating/decimal: -9999

    • text: “” (an empty string)

    • geometry: empty point geometry

    • boolean: false

    • date: 1st January 1970 (GL170)

  • Expression parsing would fail with an unexpected error with a misleading message when characters not allowed by the parser were encountered. This now returns a meaningful error message. (GL177)

  • The default model would fail if null hazards were passed to a function that does not accept nullable hazard values. Now the function call is skipped if unexpected null values are encountered. Instead a null result is produced. (GL197)

  • When a riskscape model batch command hit certain error conditions (such as no geometry in input dataset) the output error was not very useful. The batch command now outputs the same useful errors as riskscape model run would. (GL209)

  • Previously the riskscape model info command would not display any help information for template models. E.g. riskscape model info default would display help, but riskscape model info kaiju_stomp would not. This is now fixed. (GL128).

  • On Windows, RiskScape output will now adjust to match the width of the terminal, instead of always being 80 characters wide. (GL195)

  • RiskScape now displays a more helpful error message when a .asc file has a corresponding .prj file in an unsupported format. (GL236)

v0.6.0

Built 25 March 2020.

Note that in this release, major refactors have been made to the following:

  • RiskScape expressions.

  • Model execution.

Enhancements

  • RiskScape now runs pipelines with the scheduler based executor which has improved performance. (GL266)

  • How error messages are handled and displayed has been refactored to better support internationalization of error messages in future. The wording of some error messages may have changed slightly as a result of this (GL32).

  • Pipeline steps now use RiskScape expressions instead of GeoTools expressions/filters. This allows users full access to RiskScape functions from the expression language. (GL68)

  • Bookmarks now use RiskScape expressions instead of GeoTools expressions/filters. This applies to filter, map-attributes and rasterize-expression. (GL72)

  • RiskScape expressions now apply standard maths operator precedence (BEDMAS). (GL64)

  • Models are now run using an underlying pipeline. This allows models and pipelines to share improvements made to performance and other features. (GL23)

  • Query and related code has been removed as it is no longer used by models. (GL53)

  • Various improvements have been made to the RiskScape documentation. (GL74)

  • Internationalization support has been further improved. This includes a new riskscape i18n command to generate the i18n resource bundles. Documented instructions on how to add RiskScape translations have been added. (GL79)

  • New functionality has been added to improve pipeline execution performance. However, this new functionality is still experimental and is disabled by default. (GL53)

  • New built-in functions for sampling - sample and sample_at_centroid for sampling non-point and point geometries (GL97).

  • New model non-point-asset for modelling assets represented with polygon or line geometries (GL25).

  • New built-in function, bookmark, for inserting data from your project’s bookmarks in to a RiskScape expression. This is required for most uses of the sample functions. (GL97)

  • Geometry measuring functions: measure, measure_area and measure_length. (Start of GL26)

  • You can now apply particular common percentile aggregate functions with the shortcut functions p99, p95, p90 and p50.

  • New command pipeline eval for testing out the new Domain Specific Language for constructing pipelines. (GL122)

  • Many built-in RiskScape functions have been renamed so that naming style is consistent. As part of this function resolution will attempt to convert old style names to the new format so that users do not have to immediately change how they call these functions. Users are encouraged to use the new function names, e.g. create_point, geom_from_wkt, is_null and is_not_null. (GL186)

Breaking changes

  • --pipeline-threads option moved to immediately after the riskscape command, e.g. riskscape --pipeline-threads. Older commands, like riskscape pipeline run --pipeline-threads will no longer be accepted.

  • The unused stochastic model has been removed. (GL54)

  • The riskscape plugins command has been removed. This information is now available using the new CLI option riskscape --version. The output now includes the core RiskScape engine version and build date, as well as basic Java/OS information (for support purposes). (GL143)

  • The keyword arguments for the lognorm_cdf() and lognorm_pdf() functions have been renamed from mean and stddev to scale and shape. This was to try to avoid confusion over whether these values were the mean/standard-deviation of the log or raw values. For more details about scale and shape, refer to the underlying Java implementation here. This change means that if you used keyword arguments with the lognorm_cdf() or lognorm_pdf() functions, you will need to update your code. For example:

    lognorm_cdf(x: 1.0, mean: 0.0, stddev: 0.25)

    would need to change to:

    lognorm_cdf(x: 1.0, scale: 0.0, shape: 0.25)

    Note that if keyword arguments are not used, then no change is needed, i.e. the following is still accepted without problem:

    lognorm_cdf(1.0, 0.0, 0.25)

Fixes

  • Writing output to a shapefile would fail if null hazards (or resources) were included (GL152).

  • Model batch command was not picking up the correct output file format, from the --outfile argument. Instead it was writing output in JSON format even though the files been written to had some other formats extension (e.g csv, shp). (GL153)

  • Model batch command would fail on Windows if vary-input contains ‘*’ (GL156)

  • Exceptions from bad Python functions would cause unexpected errors when run from models. This has been changed to report an error message with problem source (GL154).

  • Windows file paths are now accepted in the Projects file, as well as by the --project CLI option. (GL175)

v0.5.5

Built 26 November 2019.

Enhancements

  • How error messages are handled and displayed has been refactored to better support internationalization of error messages in future. However, there should be no noticeable difference to functionality (GL32).

Fixes

  • A map-member-name parameter has been added to the core:apply-function pipeline step, which executes a function against each member of a list. This means the apply-function step can be used for the Minerva vulnerability function modelling without using interpolation (RM641).

  • The seismic:apply-function-preset-hazard pipeline step can now support ‘unpacking’ a single value from a tuple before passing it to the function. This is only done if that function’s last argument is not a struct type (RM641).

v0.5.4

Built 14 November 2019.

Enhancements

  • Internationalization support has been added allow translations of command line help output

  • Bucketing_by aggregation added to allow values to be aggregated to to a bucket that is selected by some independent value (RM642)

Fixes

  • Changes around indexing on joins to improve performance (RM646).