RPS Methodology, Step 4: Run a Screening and Review the Results
At the completion of Step 3, your customized Recovery Potential Screening (RPS) Tool contains data for all your indicators and is ready for use. The best news is, most of the work is done already. The greatest level of effort already occurred in the preceding steps while you reached agreement with your workgroup, developed and evaluated candidate indicators, possibly also compiled new indicators and customized your tool. At this point, you now can carry out different screening runs through quick and easy manipulations of the RPS Tool SETUP tab. You can vary these screening runs iteratively by selecting new combinations of indicators or by weighting existing indicators differently. In fact, setting up and running an initial statewide screening usually takes less than an hour. Repeating additional screening runs with a different selection of indicators, or the same indicators but a different subset of watersheds, can produce additional re-analysis results in just minutes.
Below are the basics for carrying out a single screening run with an RPS Tool. You may also consult the RPS Tool User Manual for a more detailed treatment of the full range of capabilities and functions of the Tool.
Rename the file for each individual screening. You have probably already re-named your custom RPS Tool once and saved it in its customized but as-yet unused condition in a place where you can always find a fresh copy to start new screenings (e.g., on a share drive, or as a read-only copy that must be renamed to edit). From this point on, your first step upon opening the Tool should be to re-name it in some way that keeps the existing primary name but adds something about the screening run you are about to perform. For example:
Originally downloaded Statewide RPS Tool: TN RPS-Scoring-Tool-031015.xlsm
Renamed customized Tool: TN RPSTOOL TDEC081215.xlsm
Customized tool renamed for a specific screening: TN RPSTOOL TDEC081215 NUTRIENTS1.xlsm
The best strategy for file management is to rename immediately at the beginning of each screening run, then save the file after running the screening if you wish to retain that screening run’s results. Because iterations of different screenings accumulate quickly, and saving the file from each run is a rapid and efficient way to organize records of all your results, systematically naming and saving file copies in this manner is recommended.
Choose watersheds. You have an option to begin any screening with all watersheds or a subset of them. If your screening run will screen every watershed statewide, simply click the ADD ALL WATERSHEDS button on the SETUP tab of your Tool. If you would like to screen a specific subset of watersheds, and you have followed instructions under Step 3 to compile a few subsets of interest, go to the SUBSETS tab in your Tool and copy just the watershed IDs from one subset (do not include the column header). Then move to the SETUP tab, place your cursor in the first blank cell of the ADD WATERSHEDS column and select PASTE. The subset’s watershed IDs will fill in the column. If you pasted in the wrong watersheds or wish to change the list, select all the IDs that were pasted, use the DELETE key and paste in new IDs.
Select indicators. Steps 1, 2 and 3 guided you through defining a screening purpose and selecting a menu of indicators generally relevant to that purpose. Step 4 now brings you back to that indicator menu for a more specific indicator selection task – choosing a small number of indicators for the first screening (of potentially many, with varying indicator combinations). Although your indicator menu may have 20, 50 or more available indicators per category, you should use between 3 and 12 per category and ensure that all three categories are represented. Using too few indicators tends to make those selected potentially too influential on the RPS index scores, whereas using too many indicators can mask the influence of individually important indicators under the ‘noise’ of too many others. Further, using several indicators that are different but highly correlated can overemphasize one watershed characteristic unintentionally. Making selections from different indicator subcategories for any given screening run can help avoid over-correlated selections, but you may also wish to do a correlation analysis on a selected set of indicators to clarify where high correlations among indicators occur. One way to use correlation analysis results is to substitute highly correlated indicators for one another in different screening runs.
No matter how you have arrived at your indicator preferences, entering them in the RPS Tool is easily done on the SETUP tab. Click on the drop-down window for each category, and a list of indicator names will appear. Click on those you wish to use and they will fill in under each column.
Assign indicator weights. When indicators are selected on the SETUP tab, note that the column to the right shows a weighting factor that initially is set to 1 for all indicators. You may (and actually should) change the default weights based on any expert insights or information you may have, along with the high likelihood that some indicators are clearly more important than others for any given purpose. To change weights, click on the cell to the right of an indicator name and enter a new weight other than 1. If nothing else, assigning weights of 3, 2, or 1 (implying high, medium, low) based on your expert judgement is recommended over leaving indicators equally weighted. Note that some older versions of the RPS Tool are limited to use of whole integers of 1 or greater as weights.
Run the screening. After selecting the indicators and weights on the SETUP tab, the final step is to click on RUN SCREENING. The Tool will process your screening for a brief period, usually not more than a minute. If you are screening tens of thousands of watersheds, the processing may take several minutes. During processing, the Tool may issue pop-up notes to inform you of any data characteristics that may cause you to rethink the screening parameters. These include notifying you of whether any of the indicators are equal value for all watersheds or may have blank values for some watersheds. Answering yes allows the processing to continue. The pop-up message PROJECT CREATED SUCCESSFULLY means that screening calculations are complete. You’re done!
Review screening results. After your screening finishes processing, the results are automatically compiled in several different formats within your custom Tool. These include tabular, graph and mapped results. On the SUMMARY SCORES tab, you will find the calculated values per watershed for the Ecological, Stressor, and Social indices as well as the Recovery Potential Integrated (RPI) score that combines all three. Each index has been calculated based on the indicators you chose. Before calculation, each indicator’s raw values are normalized to a range from 0 to 1.0 and then multiplied by the weight assigned. The indices are based on a maximum of 100. Also in the results table is the rank order of each watershed per each of the four indices.
Directionality of the index scores and their rank orders is crucial to understanding and using project results. For all the indices, the value gradient of rank ordering is such that the #1rank is the ‘best’ watershed condition for that index. Except for the Stressor index, the index scores are directionally aligned so that the higher scores are the better conditions. The higher Stressor index scores imply higher stressors, which is worse for watershed condition.
The tabs BUBBLE PLOT and HUC12 MAP also display your screening results, although in different formats. The bubble plot is a graph format with each watershed plotted as a round symbol, located based on its unique index scores. The plot has the Ecological index as the Y (vertical) axis, the Stressor index as the X (horizontal) axis and the size of each bubble reflects the Social index score. By using the BUBBLE PLOT OPTIONS tab, substantial ways are available to customize a bubble plot with labels, color codes and other changes, then save the image of the plot. At the HUC12 MAP tab, your screening results are displayed as a simple map of the state with HUC12 boundaries superimposed. The default map after a screening run shows the RPI score values for the watersheds screened in ten color increments, but the map tab allows for customizing the map to display any index or indicator, again with the option to save the image for later use.
Perform a QA/QC check. Automation in the RPS Tool reduces or eliminates substantial opportunities for human error in calculations, but quality control checks should still take place on the screening results in step 4. Some places to check for errors are:
- including the intended selection of indicators in the spreadsheet;
- properly assigned weights;
- whether each indicator's value was directionally aligned properly relative to its category;
- whether and how missing data affects any summary index values; and
- whether specific ‘good’ and ‘poor’ reference watersheds scored as expected.