Consensus when experts disagree : A priority list of invasive alien plant species that reduce ecological restoration success

Determining which species should or should not be on priority lists of invasive alien species is far from trivial. It is rare to have sufficient data to obtain empirical estimates of whether an individual species should be included or excluded and so risk assessment protocols depend heavily on expert-opinion. A challenge is deciding what to do if experts have different opinions. Few published studies have attempted to quantify the diversity of opinion among experts and in most circumstances final lists are presented as if they were the outcome of full consensus. Here we used an iterative expert elicitation process and combined this with statistical modelling to develop a list of “High Threat” alien plant species in the context of ecological restoration. Competition from invasive alien plant species can be a major cause of restoration failure, but not all alien species pose a threat to restoration success. The development of a list of persistent and invasive alien species that competitively exclude native plant species and reduce the probability of successful ecological restoration could improve our ability to distinguish between those sites where ecological restoration is more likely to succeed and those sites where it could be a costly failure. Nine experts assessed 263 alien plant species over multiple assessment rounds. Full consensus among experts was achieved for only 20% of species but the iterative process reduced the number of uncertain allocations. Alien plant prioritisations rarely communicate the range of expert-opinion and here we demonstrate; (i) that opposing opinions can be prevalent, (ii) that capturing this variability is important, and (iii) how to accommodate it analytically. Using a precautionary and transparent approach, we generate a list of 201 alien species thought likely to reduce ecological restoration outcomes. This list synthesises current expertopinion and knowledge and will be used to guide conservation and restoration actions in in New South Wales, Australia.


Introduction
Competition from alien plant species is often a cause of restoration failure (e.g.Norton 2009;Kettenring and Adams 2011;Cordell et al. 2016) and can limit the capacity for recovery of native vegetation following disturbance (Suding et al. 2004).Invasion by alien plant species and subsequent competitive effects can modify native plant composition, abundance and structure and alter ecosystem processes (e.g.Levine et al. 2003;Vilà et al. 2011;Bernard-Verdier and Hulme 2015;Kuebbing and Nunez 2015).Even with considerable management resources it can be difficult to prevent or reverse these impacts (Cordell et al. 2016).In other cases, control activities, while reducing the abundance of the target alien species, can have adverse effects on co-occurring native species (Zavaleta et al. 2001;Kettenring and Adams 2011) and favour other invasive species (Reid et al. 2009;Cordell et al. 2016).
However, not all alien plant species are likely to threaten native biodiversity or pose a significant threat to successful restoration (D' Antonio and Meyerson 2002;Grice 2006;Gross et al. 2015).In Australia, there are over 3 100 naturalised alien plant species (non-native plant species with extant self-sustaining populations outside of intentional cultivation) of which only a small fraction is a concern for conservation managers and restoration practitioners (Downey et al. 2010).Some alien species are transient and do not exert large competitive effects (D 'Antonio and Meyerson 2002) while others may facilitate the persistence of native species (Zavaleta et al. 2001;Schlaepfer et al. 2011).Others are favoured by frequent disturbances and decline in abundance after cessation of disturbance (e.g.many annual crop or pasture weeds) or their impacts can be effectively managed (Meiners et al. 2002;Booy et al. 2017).To be useful for the management of restoration activities and the prediction of restoration success it is necessary to differentiate between alien species which are likely to threaten success and those which are expected to be relatively benign (Parker et al. 1999;Leung et al. 2012;Pearson et al. 2016).
Numerous alien plant prioritisation schemes have been developed (McGeoch et al. 2016).Most focus on assessing the ecological impacts of species, such as competitive effects or ability to modify ecological processes and dynamics; invasiveness; current or potential range; trends in abundance; or the difficulty of managing extant populations (e.g.Randall et al. 2008;Downey et al. 2010;OEPP 2012;Booy et al. 2017).While current and projected range and rates of spread are important for estimating regional or national priorities, they are of lesser concern within a specific restoration site.In this case local abundance, ecological impact and ease of management will arguably have greatest influence on restoration success.
Alien plant prioritisations rely heavily on expertopinion (Vanderhoeven et al. 2017), often applying the opinion of the authors (e.g.Downey et al. 2010) or stakeholder groups (e.g.NSW DPI and OEH 2011).The application of expert knowledge to alien plant prioritisations is often justified; developing priority lists requires knowledge of large numbers of species across broad geographic ranges and for which empirical data can be scarce (Hulme et al. 2013;McGeoch et al. 2016).Because experts have differing experience, knowledge and bias, we should expect to observe a diversity of opinion about the impact of an alien species.While many prioritization methods can accommodate uncertainty (Roy et al. 2018), priority lists are often presented as consensus without explicit consideration of differences in expert-opinion (but see McGeoch et al. 2012;Firn et al. 2015).Vanderhoeven et al. (2017) recommended the adoption of iterative elicitations with pooling of expert responses to explicitly and transparently estimate opinion and variability.
This paper outlines an iterative prioritisation process to identify a list of alien plant species, from here referred to as High Threat alien (HT) species, that have a high likelihood of reducing restoration success.This list of HT species will be used to help guide selection of restoration sites for biodiversity offsetting in the State of New South Wales (NSW), Australia.The aim was to apply expert-opinion, over multiple assessment rounds, and use statistical modelling to estimate variability among experts and assessment rounds.Experts undertook independent assessments but in later assessment stages these were informed by prior results.This approach borrows from the Delphi technique where the aim is to facilitate convergence of expert opinion (Mukherjee et al. 2015) and is suitable in this instance where we expect a wide range of expert opinions but ultimately require a single list of HT species.We used a hierarchical model to account for the multiple experts and multiple assessment stages and developed predictions of each candidate species' probability of being categorised as a HT species.We hypothesised that variation around individual species probabilities of being categorised as HT would decline and convergence in opinion would increase with increasing rounds of assessment.We discuss the benefits and limitations of this approach and outline how the results could be applied to improve predictions at biodiversity offset sites.

Context
In NSW, Australia, the Biodiversity Assessment Method (BAM) provides a consistent framework and method for predicting biodiversity gains at offset sites (see www.environment.nsw.gov.au/biodiversity/assessmentmethod.htm,accessed 25 th of March, 2018).An underlying assumption is that the abundance of some alien plant species, i.e.High Threat (HT) species, are expected to reduce the probability of restoration success and reduce the likely biodiversity gains at those sites.The BAM requires an a priori list of HT species to avoid differences in opinion and experience among field assessors and to ensure transparent and consistent assessment of likely restoration gains at offset sites.
At the time this work was undertaken multiple lists of priority alien species did exist in NSW, including the Weeds of National Significance (Thorp and Lynch 2000) and National Environmental Alert List Weeds (http://environment.gov.au/biodiversity/invasive/weeds/weeds/lists/alert.html).However, these lists were not considered adequate because of numerous omissions but also inclusion of species that were thought unlikely to be HT in the context of restoration success.While there have been other prioritisations of NSW alien plants based on general impacts on biodiversity (e.g.Downey et al. 2010;NSW DPI and OEH 2011), they were not developed within a restoration context and despite reliance on expert opinion, did not report the diversity of opinion among experts.

Assessment process
The assessment process had six components: (1) compile a preliminary list of invasive alien plant species; (2) develop an assessment protocol based around a decision tree and a definition of HT species; (3) identify and select a panel of experts; (4) apply an iterative expert-assessment of proposed species against the assessment criteria; (5) analyse the assessments and (6) prepare a final list of HT species.
Figure 1 presents a diagrammatical summary of the stages in the assessment process.

Stage 1: Compile a preliminary alien plant species list
Assessing all the alien flora of NSW (> 1 665 species Downey et al. 2010) using multiple experts and over several assessment rounds would be an onerous, datademanding and time-consuming task (McGeoch et al. 2016;Vanderhoeven et al. 2017).Instead, we sourced a preliminary list of plant taxa derived from a NSW state-wide process that prioritised widespread alien species based on potential biodiversity impacts (NSW DPI and OEH 2011).While this list was not exhaustive, it provided a pragmatic and useful starting point for initial expert-assessments and benefited from the input of > 100 experts and stakeholders.The NSW Biodiversity Assessment Method applies only to terrestrial vegetation and so prior to providing the list to experts, all fully aquatic species were removed.The final list contained 199 taxa.Some of these taxa were listed as genera, though the majority were as species level entities.

Stage 2: Development of assessment protocol
Within the context of likely impact on restoration success we defined HT species as invasive and persistent alien plant species with the potential to outcompete native species, modify key ecosystem processes (e.g.fire seasonality/intensity or nutrient cycling) and are difficult to control without impacting co-occurring native vegetation.The definition excludes those alien species capable of invading and persisting in native vegetation if they are thought not to competitively exclude native species or if their impacts are relatively easy to manage.This definition differs from that of Groves et al. (2003) who prioritised species if they were thought to be amenable to control.It also differs from other prioritisation approaches because it focuses solely on potential ecological impacts and management difficulty and excludes separate consideration of spatial distribution and trends in abundance (e.g.Randall et al. 2008;Downey et al. 2010).Trends in abundance and distribution are useful for predicting the probability of an interaction between native vegetation communities and the alien species but downplays the potential local impact of species with a limited state-wide distribution.We assumed that if a species met the definition in any part of their current geographic range they should be assigned as a HT species.
We used our definition to create a decision tree (Supplementary material Figure S1) to guide the allocation of species to either High Threat (HT), Not High Threat (NHT) or Uncertain based on a set of five assessment questions: 1. Is the species non-native (alien)? 2. Does the species outcompete and threaten the persistence and/or recruitment of native plant species?
3. Is the probability of improving biodiversity values, through active management or restoration, negatively correlated with the species abundance/cover? 4. Is the species difficult to effectively control or manage?i.e. known and accepted management strategies fail to consistently reduce abundance or prevent spread.
5. Do the known control measures lead to significant loss of native plant cover or native plant richness?
We did not attempt to quantify probability distributions from each expert (e.g.Garthwaite et al. 2005), but rather asked experts to provide their educated best guess (e.g.McCoy et al. 1999) as "likely", "unlikely" or "uncertain".Experts could answer with "uncertain" if they lacked knowledge of the species or if they believed "likely" or "unlikely" were equally probable, though they were not asked to differentiate between these two types of "uncertain" response.
A species could only be assigned HT if the answers to either of questions 2 and 3 (competitive effect and impact on restoration success) were likely.However, even if these were satisfied, if current known control measures were effective (Q4), then the species was not assigned to HT, unless the control measures also result in significant reductions in native plant cover and/or richness (Q5).On the other hand, if the control methods were likely to be ineffective, then regardless of response to Q5 the species was assigned to HT (Figure S1).If the answers to questions 2 and 3 were both unlikely, then regardless of answers to other questions the species was assigned as NHT.Most other combinations of responses resulted in an assignment as uncertain.

Stage 3: Identifying experts and overview of elicitation process
Botanists and plant ecologists were regarded as experts if they had broad knowledge of NSW native and alien flora and > 15 years relevant field experience.A pool of 12 experts was identified and ranked by availability and extent of field experience relevant to invasive plant ecology and native plant restoration.An important assumption was that these experts were a representative and random sample of the population of persons with expert knowledge of invasive alien plant species in NSW.Nine individual experts, drawn from the pool of 12, contributed to 14 unique assessments over three assessment rounds (Table S1, Figure 1).For the initial round of assessment, five experts on the list were contacted, one of whom declined to participate.After completion of the first round of assessments, four new experts from the pool were engaged to repeat the assessment with the aim to resolve uncertain allocations and assess additional taxa.In the third and final round of assessments three new and the original eight experts were contacted.Completed responses were received from six experts -five from previous rounds and one new expert.Five experts were unable to complete the assessment within the allotted timeframe.See Table S1 for allocation of experts to assessment rounds.

Stage 4: Assessment of species
Round 1: Decision-tree derived HT species assessment In the first round, four experts provided independent assessment of the initial 199 species by answering the five assessment questions and where appropriate providing comments justifying their answers.Experts were also invited to nominate and assess additional species that they believed would meet the definition of HT species.This included suggesting individual taxa that were only listed as genera in the initial list.
Completed spreadsheets received from each expert were checked for omissions (one or more questions not answered for a taxon).In a small number of cases experts had provided answers to all but one of the five questions and in these cases the cell was populated with the default rating "uncertain" and the species was retained.The decision tree was used to evaluate individual species and assign each to HT, NHT or Uncertain.
Round 2: Independent review of preliminary assessments The outcomes of the allocations from Round 1 indicated that additional assessment was needed to resolve taxa without majority-opinion, and also to review 57 new taxa added to the list by the experts in Round 1. Rather than simply repeat the assessment with the same four experts, in Round 2, a new group of four experts was asked to assess the 256 taxa (and any new taxa they chose to nominate) so as to increase the sample of expert opinion.Prior to making their assessment they were provided with the anonymous individual decision-tree derived allocations for each taxon from Round 1.However, instead of using the decision tree to make their own assessments, experts in Round 2 heuristically classified each species as HT (the species meets the definition), NHT (the species does not meet the definition) or Uncertain (knowledge or evidence insufficient to make a determination) based on the definition of HT species.
Round 3: Review of preliminary allocations and final assessment Prior to Round 3 all species were given an analytically derived draft assignment of either HT or NHT based on Round 1 and Round 2 majority-opinion (see below).
This list of 261 species (199 preliminary species, 57 species nominated in Round 1 and five species nominated in Round 2) was assessed by six experts (see above).Each expert was asked to use the definition of HT species to review the draft assignments and reallocate each species to HT, NHT or Uncertain.They were also asked to nominate and allocate any additional species.
Stage 5: Analysis of expert-assessments 5a.Analysis of Round 1 and 2 results and draft assignment The results of the first two rounds were compiled and a draft list of HT species was prepared based on simple majority-opinion by summing results across both Rounds 1 and 2. If the number of HT assessments were > NHT, then the species was given a draft allocation as a HT species.The same logic was applied to allocating species as NHT.Species not allocated using majority-opinion based on both rounds were allocated using majority-opinion in the second round alone, which was informed by the results of the first round.If species could still not be allocated (e.g.full consensus-opinion was Uncertain or equal HT and NHT opinions), it was allocated as a HT species.This precautionary approach to identifying HT species was justified on the basis that either no expert was prepared to assign the species as NHT, or as many experts were prepared to assign it HT as those assigning it NHT.We argue that a precautionary approach would minimise the risk of selecting a site for restoration where significant alien plant threats existed.

5b. Final analysis
To account for the complexity among species, experts and allocations, we analysed the matrix of all 263 species and expert-assessments among three assessment rounds using hierarchical mixed models.We then used predictions from this to generate a final list of HT species.Additional species suggested by experts in Rounds 1 and 2 were included in the analyses, but of the nominated species first suggested in Round 3, only those independently nominated by two or more experts were retained (2 species).
While a simple approach would have been to pool the data and calculate majority-opinion (e.g.proportion of HT assignments across all three rounds and within each round), this would have provided no information about variability around estimates or the probability that a randomly selected expert would assign a species as HT.Modelling is a useful tool for estimating the point probability associated with each species and the associated confidence around the estimates.
Broadly we treated the results of the expertassessments in a hierarchical manner.Each expert's estimate for each species was treated as a single observation though grouped by assessment round.Species and experts were each treated as random intercepts and the effect of assessment round was treated as a fixed effect (a factor with three levels) as well as being allowed to vary among species (as a random slope).Additional species, nominated by experts in Rounds 1, 2 or 3 were accounted for with a separate fixed effect (as a factor with two levels) in the model.
Expert-opinion could be treated as a single binary response variable where HT allocations take a value of 1 and NHT allocations take a value of 0, after disregarding the Uncertain responses.However, this approach fails to account for the substantial number of Uncertain allocations (476, 14.5%, of all allocations) and would potentially lead to greater confidence in the results than warranted from the data.As an alternative, we treated the data as three different binary responses that were analysed separately: 1. Probability of HT (prHT), where an observation took a value of 1 if it was scored by an expert as HT and 0 if it was scored as either NHT or Uncertain; 2. Probability of NHT (prNHT), where an observation took a value of 1 if it was scored by an expert as NHT and 0 if it was scored as either HT or Uncertain; 3. Probability of Uncertain (prUnc), where an observation took a value of 1 if it was scored by an expert as Uncertain and 0 if it was scored as either HT or NHT.
Three separate models were used to predict the probability of a species being allocated as either HT, NHT or Uncertain.The data were analysed using generalised linear mixed models with a binomial response and logit link with the package lme4 (Bates et al. 2015) in R (v3.2.2; R Core Team 2016).
For each of prHT, prNHT, prUnc we fit the following binomial model: Where for a specific model (i.e.either prHT, prNHT, prUnc), Y ijkl are the observed binomial opinions for the k th species assessed by the l th expert in the j th assessment round.β are the fixed effect coefficients for the intercept (β 1 ), each assessment round (β 2 and β 3 ) and whether the species was an additional expert nominated species (β 4 ).The random component of the model is composed of normally distributed random intercepts for each of 263 species (α k ) and 9 experts (γ l ) and additional random variation among species in each assessment round, denoted by b k and c k .ε ijkl is the residual variation.
For each species, we estimated the probability of HT, NHT and Uncertain, and associated 95% confidence intervals, for each of the three assessment rounds for an average expert.Predictions and associated 95% confidence intervals (CI) were compared among species within each round.

Stage 6: Final list of HT species
The final list of HT species was based on the predicted probabilities from the hierarchical model.Predictions were available for each of the three assessment rounds but we used predictions from the third round because expert opinions benefited from prior knowledge and review while model estimates and associated error drew from all observations.The following pragmatic rules were used to allocate species as HT and to give each allocation a categorical indication of confidence: 1. Species were allocated as a HT species, with high confidence, if their lower 95% CI for the probability of HT in Round 3 was > 0.5 and their upper 95% CI for probability of NHT was < 0.5; 2. Species were allocated as a HT species, with moderate confidence, if their predicted mean probability of HT was > 0.5 and there was no overlap between the 95% CI for HT and NHT 3. Species were allocated as a HT species, with low confidence, if their predicted mean probability of HT was > NHT but 95% CI for HT and NHT overlapped.
All remaining species were allocated as NHT and an equivalent approach to the above was taken to derive their confidence ratings.
The choice of probability thresholds can be arbitrary although 0.5 was thought to be useful for communicating the results within a policy and management context.A probability of HT > 0.5 indicates a greater than even chance that a random expert will assign the species as HT.In the case of those species with lower 95% CI > 0.5, it suggests that if the "experiment" were repeated there would be a 95% probability of the species having an estimated probability of HT > 0.5.Table 1.Summary of the number of alien plant taxa (percentage in brackets) assigned as High threat (HT) or Not High Threat (NHT) species based on majority-opinion in each round and final allocations after modelling of all data.The number of taxa with low, moderate or high confidence in the final allocations are shown.Final allocations and confidence ratings are based on hierarchical model results (see methods for details).The total number of taxa assessed excludes any additional taxa nominated in that round.Majority-opinion was assumed if the number of HT was > NHT or number of NHT was > HT within that round.

Majority-opinion Allocations
Final

Results
Fifty-three species (20%) were consistently assessed by all experts in all assessment rounds (Table S2).
Of these, 52 species were always rated as HT, while only a single species (Argemone ochroleuca, Mexican Poppy) was consistently rated as NHT by all experts in all assessment rounds.Also, the proportion of species with simple majority-opinion (either majority HT or NHT) increased with each successive round (Table 1).On completion of the final round, all but a single species were allocated based on majorityopinion (Table 1).
The number of species with predicted 95% CI overlapping 0.5 for probability of HT declined with each assessment round (Figure 2) and there was a decline in the mean probabilities of individual species being assigned Uncertain from the first to the third round (Figure 3).Likewise, increasing numbers of assessment rounds improved predictions for a species probability of being assigned NHT (Figure 4).The tightening of 95% CI for the probability of HT was in part due to increasing consensus among experts; in Round 1, 45% of species had diametrically opposed assessments (i.e.assigned as HT and NHT by different experts) as opposed to 17% and 21% in rounds 2 and 3 respectively.
For those species on the preliminary list, their results in the first round were generally indicative of final allocations and predictions.The predicted probabilities for HT (Figure 2) or NHT (Figure 4) in the third round were positively correlated with the predictions in the first and second round.Likewise, the number of experts rating a species as HT in round one was indicative of the species predicted probability of HT in Round 3 (Figure 5).
Of the 263 species, 201 were classified as a HT species and of these, 182 were assigned a high confidence based on lower 95% CI > 0.5 (Table 1, Table S2).A further 19 species were allocated as HT but their allocations were of moderate or low confidence.Results suggest that while opinion was generally in favour of these later species being listed as HT (i.e.simple majority-opinion and tendency that more experts favoured the HT allocation), there was conflicting opinion.

Variation among expert-assessments of high threat species
Approaches to prioritizing invasive alien species rely on assessments of threat or impact on native species and ecosystems, but data to support these assessments are often limited (e.g.Hulme et al. 2013).Prioritisation is therefore often reliant on expert knowledge and opinion (McGeoch et al. 2016).However, differences in opinion among experts can be substantial (Regan et al. 2005).We have clearly demonstrated that opposing opinions can be pervasive in an invasive species assessment process and this alone can induce considerable uncertainty.For example, of the 62-species excluded from the list of HT species (those classified as NHT), 52 were assessed as meeting the HT definition by at least one expert.The range of expert responses highlight that expertderived priority lists presented as consensus need to be treated with some caution as they are likely to mask underlying differences in opinion (Regan et al. 2005).In many ecological applications expert-opinions  are pooled and summarised with the mean or median (e.g.Martin et al. 2005).While measures of central tendency are a simple and widespread method of summarising expert knowledge, they too obscure differences of opinion.
Here we have collated a list of HT species via an iterative approach that explicitly considered differences in opinion.The inclusion or exclusion of each species was transparent and relied primarily on lower 95% CI for each species' predicted probability of being either HT or NHT.Using this approach, we found that we could confidently list 182 species (~ 72%) as HT species.A further 19 species were allocated as HT but with low or moderate confidence.Risk assessments and prioritisations that rely on expert knowledge and opinion should endeavour to use similar approaches that explicitly incorporate inter-rater variation in the creation of final priority lists.
Our approach does still rely on an arbitrary threshold, being a lower 95% CI > 0.5.We consider this threshold appropriate from a precautionary perspective because excluding a species from the HT list could be a substantial risk.However, lower or higher thresholds could be appropriate in specific circumstances, depending on the type of errors that were deemed acceptable.If failure to identify an alien species as HT was unacceptable, then even a lower threshold might be justified.
By explicitly describing the variability of expertopinion we can highlight those species for which opinions diverge.For example, two species classified as NHT had point probabilities for HT that were less than NHT but their HT 95% CI overlapped both 0.5 and their NHT 95% CI (Phalaris aquatica and Solanum mauritianum).Hence expert-opinion, while on average was in favour of a NHT classification, was clearly divided and led to a low degree of confidence in the final allocation.Both species have been described as significant invasive species elsewhere (Phalaris aquatica e.g.see Godfree et al. 2017; Solanum mauritianum e.g.see Olckers 1999) and would be candidates for further assessment if the opportunity arose.Explicitly describing the variation among experts therefore points to those species that would benefit from further research.
Multiple assessment rounds and provision of prior knowledge are a cornerstone of Delphi-like assessment methods and are used to increase consensus among experts (Hsu and Sandford 2007).Here, we have adapted these key elements of Delphi but opted to include additional new experts in each round.While new experts may introduce additional variation, we argue that our results will better reflect the range of expert-opinion than if we had attempted to reach consensus with just the original four experts.In this case despite introduction of five new experts, with the provision of prior opinion and iterative assessment confidence around those species rated as HT or NHT improved (Table 1).Furthermore, model predictions from the first round in our study were indicative of results in subsequent rounds (Figure 2).
While increasing consensus is desirable when the aim is to present a single prioritised list of species, iterative elicitations with feedback can contribute to a centring of opinion, even where this is not warranted (Woudenberg 1991).For example, in our study Xanthium strumarium, was listed as HT in the draft allocations based on analysis of Round 1 and Round 2 data and was rated by all six experts in Round 3 as HT.Yet three of the previous eight opinions in Rounds 1 and 2 had been NHT.Was the final round consensus-opinion a true reflection of the generally held opinion or was it an "artifactual byproduct of the pressure to conformity caused by the statistical feedback" (Woudenberg 1991)?Delphi-like approaches are often considered to represent best practice and our study has benefited from iterative assessment and feedback.However, all approaches to strategic group decision making have their inherent strengths and weaknesses and these need to be considered carefully against the aims of each study.
Our study tackles one key source of uncertainty in assessments of invasive species, that is variation in opinion among experts.While we allowed the experts to categorise a species as "uncertain" we do not know if this is because they lack sufficient data or knowledge to confidently assign it as HT or NHT or if available evidence is conflicting.While experts were encouraged to provide comments in support of their assessments, these were insufficient to diagnose the source of uncertain allocations.An important improvement to the approach we have taken would be for experts to provide an estimate of confidence against each assignment and explicitly provide justification for those estimates (Vanderhoeven et al. 2017).
The preliminary list for assessment round 1, derived from NSW DPI and OEH (2011), contained several taxa listed as genera (e.g.Opuntia spp., Pyracantha spp.).The listing of genera is consistent with other invasive plant prioritisations in Australia (e.g.Weeds of National Significance, Thorp and Lynch 2000).Listing of genera might be reasonable if species within the genus are thought to be similarly invasive (Pyšek and Richardson 2008;Cadotte et al. 2009) and when the invading genera are unrelated to species in the native flora (Strauss et al. 2006).However, variation in invasiveness and impact among species within the same genera would increase assessment uncertainty.Because of this, experts were encouraged to separately identify and assess additional species, if they believed they did not share the same level of impact as others in the genus.Despite this, extending predictions to all species within a genus should be done with caution, especially when the genus encompasses numerous alien species known to occur in NSW.

High threat alien species within the context of restoration success
The list of 201 HT species generated by this study (Table S2) is not exhaustive and non-inclusion does not imply a species will not have an impact on biodiversity or restoration success.Our preliminary list focused on widespread alien species and it is likely that it excluded species with localised distributions and potential to impact on conservation management or restoration outcomes.There was scope for experts to nominate additional alien species and new species were nominated in all assessment rounds.Additional rounds may have led to further additions to this list.Our definition of high threat species prioritised persistent, perennial species.Despite numerous annual species being on the list assessed by the experts, few were prioritised as HT species.However, in some circumstances annual species can form stable states, reinforced by positive feed-back loops, that are then resistant to many restoration activities (Suding et al. 2004;Prober and Thiele 2005).At individual sites, planning should focus on identifying and managing all major alien plant threats and this may include species not identified by our study as HT.
Often invasive plants present as a suite of species, rather than as dominance by a single species (Groves et al. 2003).Sites supporting multiple HT species will require co-ordinated management efforts that ensure that control of a specific HT species does not simply result in its replacement by other HT species (e.g.Kettenring and Adams 2011).Also, relative threats and impacts are likely to vary among HT species, requiring differing levels of restoration effort and priority.While HT species are more likely to pose a threat to biodiversity and or restoration success than NHT species, the mean HT probabilities for individual species (Table S2) should not be interpreted as differences in threat or impact among HT species.Where multiple HT species are present, the results we present cannot be used to develop priorities for management actions.Assessment approaches that assign priorities for alien species based on their relative level of threat and impact (e.g.Booy et al. 2017), in conjunction with local knowledge, would be useful to help guide site based management priorities.

Implications for biodiversity offset policies
Increasingly governments and financial institutions are adopting biodiversity offsetting for managing the impacts of human development on biodiversity (Gardner et al. 2013;Maron et al. 2016).In jurisdictions with biodiversity offset policies, residual negative impacts of development activities are meant to be offset elsewhere by improvements (gains) in habitat quality or increases in the population size of target species (Maron et al. 2016).The success of ecological restoration is a key factor in determining whether biodiversity offsets can meet their obligations (Maron et al. 2012;Curran et al. 2014).Invasive plant control can be a large component of ecological restoration effort at offset sites and competition with alien species can reduce restoration success.Offset sites containing alien species that are likely to impose large management costs or reduce the chance of restoration success increase risks associated with biodiversity offsetting.In NSW, Australia, the presence and abundance of HT species contributes to predictions of the amount of improvement in native vegetation structure and composition (vegetation condition) that might result from conservation management actions and restoration at offset sites (see https://www.lmbc.nsw.gov.au/bamcalc,accessed March 19th 2018).The underlying assumption is that areas dominated by HT species will have lower probabilities of delivering successful outcomes relative to those where HT species are absent.This acts as a risk management strategy and provides a mechanism that explicitly favours offset sites with few or no HT species.Our study has provided a defensible list of HT species to ensure transparent and repeatable predictions of improvements in biodiversity values (gain) likely to be achieved through restoration actions at biodiversity offset sites.

Conclusions
Invasive alien plant control is a major conservation and restoration action but can be resource intensive in terms of time and cost.However, not all alien plant species pose a threat to ecological restoration success.Identifying HT species most likely to affect restoration outcomes is valuable to (1) help prioritise sites where risks to meeting restoration objectives from HT species are low and (2) to help prioritise plant control efforts at individual sites that contain a suite of alien species, many of which may not be HT.The list we have developed here was derived from the opinions of nine botanists and invasive plant ecologists and focuses specifically on identifying persistent and invasive alien species likely to competitively exclude native species and reduce the probabilities of successful ecological restoration.Our results show that there was full consensus-opinion for only 20% of species, but 85% of species could be allocated as HT or NHT with high confidence.Our robust analytical approach has provided a defensible and transparent list of HT species based on the synthesis of current regional knowledge of those alien species most likely to influence the success of conservation and restoration actions.

Figure 1 .
Figure 1.Schematic summary of the multiple stages in the assessment of alien plant species as High Threat (HT) or Not High Threat (NHT) in the context of ecological restoration.

Figure 2 .
Figure 2. Predicted probability of an alien species being assigned as High Threat (HT) in each of three assessment rounds.Upper panels show predictions for the 199 species on the initial list.The bottom panels show predictions for additional species proposed by one or more experts.Vertical lines are 95% confidence intervals.Species are sorted in each panel according to their mean predicted probability of High Threat in Round 3.

Figure 3 .
Figure 3. Predicted probability of an alien species being assigned as uncertain in each of three assessment rounds.Upper panels show predictions for the 199 species on the initial list.The bottom panel shows predictions for additional species proposed by one or more experts.Vertical lines are 95% confidence intervals.Species are sorted according to their mean predicted probability of High Threat in Round 3.

Figure 4 .
Figure 4. Predicted probability of a species being assigned as Not High Threat (NHT) in each of three assessment rounds.Upper panels show predictions for the 199 species on the initial list.The bottom panels show predictions for additional species proposed by one or more experts.Vertical lines are 95% confidence intervals.Species are sorted according to their mean predicted probability of High Threat in Round 3.

Figure 5 .
Figure5.The number of experts assigning a species as High Threat (HT) in Round 1 (x-axis) was positively related to the predicted probability of an average expert classifying a species as HT in assessment Round 3 (y-axis).The data are summarised as a box and whisker plot indicating the median value (line), interquartile range (box), upper and lower values within 1.5 x the interquartile range (whiskers) and outliers (points).Data are for 199 species assessed by all experts in Round 1.