Public contributors' preferences for the organization of remote public involvement meetings in health and social care: A discrete choice experiment study

Abstract Introduction Covid‐19 expanded the use of remote working to engage with public contributors in health and social care research. These changes have the potential to limit the ability to participate in patient and public involvement and engagement (PPIE) for some public contributors. It is therefore important to understand public contributors' preferences, so that remote working can be organized in an optimal way to encourage rather than discourage participation. Methods We use an economic preference elicitation tool, a discrete choice experiment (DCE), via an online survey, to estimate public contributors’ preferences for and trade‐offs between different features of remote meetings. The features were informed by previous research to include aspects of remote meetings that were relevant to public contributors and amenable to change by PPIE organizers. Results We found that public contributors are more likely to participate in a PPIE project involving remote meetings if they are given feedback about participation; allowed to switch their camera off during meetings and step away if/when needed; were under 2.5 h long; organized during working hours, and are chaired by a moderator who can ensure that everyone contributes. Different combinations of these features can cause estimated project participation to range from 23% to 94%. When planning PPIE and engaging public contributors, we suggest that resources are focused on training moderators and ensuring public contributors receive meeting feedback. Discussion and Conclusion Project resources should be allocated to maximize project participation. We provide recommendations for those who work in public involvement and organize meetings on how resources, such as time and financial support, should be allocated. These are based on the preferences of existing public contributors who have been involved in health and social care research. Patient or Public Contribution We had a public contributor (Naheed Tahir) as a funded coapplicant on the UKRI ESRC application and involved members of the North West Coast Applied Research Collaboration (NWC ARC) Public Advisor Forum at every stage of the project. The survey design was informed from three focus groups held with NWC ARC public contributors. The survey was further edited and improved based on the results of six one‐to‐one meetings with public contributors.

working will continue to be used alongside face-to-face meetings and as part of 'hybrid' working. Remote working has provided a valuable way of continuing to do PPIE during the pandemic and ameliorating some of the isolation felt by public contributors during the periods of lockdown in 2020. 1 However, remote working with its dependence on the Internet and communication equipment, needs to be carefully considered in light of socioeconomic and health inequalities. There is a digital divide that maps onto existing socioeconomic inequalities, with those in lower socioeconomic groups and older communities having less access to and opportunities to use remote working technologies. 2 Areas of high deprivation and ethnic minority communities bear the burden of poor health and access to health care and these communities have experienced disproportional harmful effects of the pandemic. 3,4 As a consequence, health inequalities are increasing. 5 Therefore, PPIE conducted remotely has the potential to further disenfranchise already disadvantaged groups and attention needs to be paid to ensuring diversity and inclusion in PPIE remote working.
The likelihood that remote working will continue alongside face-to-face meetings means that disenfranchisement due to the digital divide is added to concerns that PPIE was insufficiently diverse before the pandemic. 6 A recent National Institute of Health & Care Research (NIHR) (a UK-based funder of health and social care research) survey of public contributors found a lack of diversity in the public contributor community in terms of age and socioeconomic status and addressing this is an NIHR priority. 7 This paper reports on a discrete choice experiment (DCE) survey that is part of a larger UK-based study that explored remote working in PPIE in health and social care research during the Covid-19 pandemic in 2020-2021. 8 1.1 | PPIE in health and social care research PPIE has become a widespread phenomenon in health and social care research. The NIHR state: 'Public involvement is at the centre of NIHR health and social care research, and the public has a right to have a say in what and how publicly funded research is undertaken'. 9 The terms 'patient and public involvement and engagement' (PPIE) or public and patient involvement (PPI) are commonly used to capture a broad range of activities that aim to develop effective links between researchers and the general public. We will use a broad definition of PPIE for the purposes of this paper, 'research being carried out 'with' or 'by' contributors of the public rather than 'to', 'about' or 'for' them.' PPIE includes notions of active contribution, 10 and 'good' PPIE is more about coproduction than just involvement. 11 'Coproducing a research project is an approach in which researchers, practitioners and the public work together, sharing power and responsibility from the start to the end of the project, including the generation of knowledge'. 11 We use the term 'remote working' to cover meetings and interactions held without face-to-face contact that use communication technologies such as telephones (landlines, mobiles, smartphones), computers, tablets, online conferencing/meetings software, social media, and apps. Hybrid meetings are where a meeting is held with some participants face-to-face and other participants joining remotely, such as via a video conferencing tool such as Zoom. 12 There is limited research on the feasibility and assessment of remote working quality in PPIE. However, since the start of the  15 This paper contributes to this growing literature.
In one of the few published articles, Lampa et al. 16 reported observations of digital PPIE meetings during the pandemic. They found that meeting organizers need to be committed to solving practical issues and it is important to coproduce the meeting LORIA-REBOLLEDO ET AL. | 147 structure and format with public contributors. Adeyemi et al. 17 discussed three case studies of remote PPIE with marginalized groups and concluded that it is possible to do remote work with such groups, but it also presents some challenges, predominately the challenge of digital poverty and lack of access to equipment and data/WIFI. A better understanding of how to organize and support public contributors with remote working can help engage public contributors and allow teams to design remote working practices that are inclusive and encourage, rather than limit, diversity.

| METHODS
DCEs are a survey-based method grounded in economic theory that assumes the value of a service (in this instance PPIE meetings) comprises the value of the different attributes that describe it. 18 DCEs are a widely used method to elicit preferences from the public, patients, and healthcare professionals. 19 Respondents in a DCE are asked to make a series of choices between two or more hypothetical alternatives describing different types of meeting packages. These packages are further described by different features (herein referred to as attributes) and a corresponding value (herein referred to as levels). For example, a meeting attribute could be the 'time of day the meeting takes place' and the levels could be 'between working hours' and 'between working hours and evenings'. When respondents make choices, they are implicitly trading the attributes and levels that describe the alternatives. This tradeoff information can be used to estimate the relative importance of one attribute over another and predict participation in a defined meeting package.
In this study, public contributors were asked to imagine they were invited to take part in a new project. This would mean that they had to join regular project meetings using video calls. In the DCE, public contributors were then presented with a series of choices. In each choice, they were shown two different ways in which project meetings could be organized. These meetings differed in seven attributes. Contributors were asked to choose to take part in one type of meeting or not take part in the project (e.g., opt-out alternative). The DCE was designed using a state-ofpractice sequential, mixed methods approach. 20,21 The attributes and levels describing the remote meetings were identified and refined from the previous phases of this study (see After analyzing the survey data, we conducted qualitative interviews to further probe and explore the themes (results of the previous phases are reported in Frith et al. 22 and Jones et al. 23 ). This ensured that the included attributes of remote meetings are those which are both most important to PPIE contributors and amenable to change or under the control of meeting organizers.
The data from the previous phases (Phase 1 and Phase 2) of our study identified three stages that influenced how PPIE contributors felt about participating in meetings: what happens before, during and after the meetings. Based on these findings, seven attributes, grouped into these three stages, were used to describe the remote meetings (see Figure 2). Four attributes described features of a meeting's organization that happen before: the length of the meeting, the time of day when the meeting is held, the type of connectivity support that is provided and the technical support provided to help participants to join and contribute during video calls. Two attributes described features that occur during meetings: the etiquette during a remote meeting and the role of the moderator. One attribute described whether any feedback on the contributors' contributions was provided after the meeting. The rationale for the selection of these attributes and levels, based on the previous phases' findings is described below ( Table 1).
Length of the meeting was deemed a key feature that public contributors would want to know ahead of any remote meeting, with the data suggesting participants generally preferring shorter meetings, while some enjoyed meetings with an icebreaker and/or social activity that allowed them to interact with and get to know the other participants, albeit that is likely to lengthen the meeting time overall.
Given the potential impersonal nature of remote meetings, we included the possibility of having a longer meeting with a social activity as one level in the DCE. The meeting's time of day was also important. Some contributors preferred the flexibility of meetings outside working hours and others preferred meetings during working hours, especially those with caring responsibilities. We include the provision of connectivity tools as an attribute that contributors would want to know before, as this allowed us to test whether providing web-enabled devices and reimbursing Internet and electricity costs is a way to overcome the digital divide and increase participation.
Similarly, we included the provision of technical support as this was identified to be a potential driver of public contributor disenfranchisement, especially amongst contributors with limited experience and Internet literacy.
Previous phases identified meeting etiquette as a potential key driver of meeting and project uptake. Public contributors in the interviews discussed the difficulty of balancing long video calls from home with their caring responsibilities, with some describing the The meeting etiquette attribute thus described whether contributors had to keep their cameras on and be ready to contribute during the whole meeting, or whether it was possible to turn them off and step away when needed. The role of the meeting's chair or moderator was another important aspect identified in the interviews and surveys in the previous phases (for our purposes we are using them interchangeably to mean 'the person who is running the meeting' or organizing the meeting, as these are not formal decision making meetings where the chair has a formal role). Contributors were able to distinguish between a good and a sub-par moderator, with the majority agreeing on the importance this can have to the success of a meeting. We described this attribute in terms of a standard moderator who only ensures meetings run smoothly and a good moderator who also makes sure participants feel comfortable and confident to contribute.
A recurring theme in most of the interviews and survey data from the previous phases was the uncertainty of what happens after the meetings and, specifically, whether the public contributors had been listened to and their suggestions are taken on board. We, therefore, included the provision of feedback, either as a personalized report that details how each individual's contribution was used or a general report that explained how the group's contributions were taken onboard, as an attribute that can be both influenced by the meeting organizers and speaks to addressing this uncertainty. While not being an issue exclusive to remote meetings, this feature was deemed key given the nature of remote meetings and the way they can limit nonverbal communications between participants and moderators.
Based on the attributes and levels, there are 273,248 possible unique choice tasks (pairs of meeting descriptions). We used experimental design techniques to reduce these to a more manageable number. Specifically, we created a D-efficient experiment design with vague informative priors and allowing for estimation of nonlinear effects of attributes using Ngene software to reduce the number of choice tasks to 24. 24,25 The aim of this design was to create realistic choice tasks with statistical properties that facilitate the estimation of the effect of each feature. 26 To reduce respondent burden, the resulting design was blocked into three sets to each respondent was asked to complete eight choice questions. 27 Based on this design, a minimum of 49 respondents were required for each analysis block to ensure the estimation of all attribute effects. 28 T A B L E 1 Attributes and levels used in the choice experiment (Great moderator) On ensuring meetings run smoothly and makes an effort to make you feel comfortable and confident about contributing to meeting.
7. Meeting feedback (e.g., sense of contribution) No follow up.
General follow up that tells how broad contributions from the meeting were included.
Personalized follow up that tells how individual contributions were included.
Respondents were randomly assigned to one block and the order of the choice tasks within each block was also randomized to minimize ordering effects. 29 The DCE online survey was comprised of three sections (see

| DCE analysis
The DCE response data indicates which one of the three alternatives a respondent selects in each choice task. The data were analysed using a mixed logit (MXL) model. 30 We assume that respondents (n) choose the alternative (j) that provides them with the highest utility in each of the choice tasks (t). Following random utility theory, 31  where β denotes the parameter of attribute k and x jk is the level that the attribute takes in the scenario h.
We estimate the participation of different remote meeting configurators described in Table 2 Table 2 describe different ways a meeting can be organized, and each involves a different allocation of resources available to meeting organizers. All analysis was done using the statistical software R. Confidence intervals were computed using the delta method.

| Affordance theory
We drew on the concept of affordances to further analyse our data.  Table 3. The modal respondent was an experienced PPIE contributor (e.g., involved in three PPIE projects), had taken part in remote meetings as part of their role, had access to devices and an Internet connection to enable joining remote meetings and was able to take part in remote meetings uninterrupted. Only 22% of respondents were completely certain that past contributions to other projects had been taken onboard (64% were at least very certain). Respondents were more likely to be female, over the age of 45, highly educated (at least University or equivalent), living alone or with no more than one person, and having no caring responsibilities.
We found most respondents were willing to take part in a project that involved remote meetings and were willing to make trade-offs

| DISCUSSION
To our knowledge, this is the first DCE to investigate public contributors' preferences for how remote meetings in PPIE are organized. While remote working has the potential to limit the ability to participate for some public contributors, for others it can increase their participation. 23 People in our sample were generally willing to take part in projects even if this involves remote meetings. However, project participation can vary significantly depending on certain features of the meetings. Our findings suggest how project resources, such as time and financial support, can be best allocated to increase meeting participation by public contributors. Giving participants feedback about how their contributions to meetings are taken onboard by the organizers-how their contribution has made a difference was important to our participants. This has been found in other research in this area 34 and providing feedback to public contributors has been described as an important, but often overlooked, part of PPIE leads' work. 35 Respondents seemed indifferent to whether this is general or personalized feedback, so there is little benefit from the additional resource cost of providing individual reports compared to a general one. We are not able to conclude if the importance of providing feedback is exclusive to remote meeting settings. However, our data suggest that this feature is important regardless of whether the meeting is remote or face-to-face.
Furthermore, most respondents in our survey stated that they were not certain that their contributions had been taken on board in past  Table 5). Bringing an affordance lens to our data enabled us to see how different features and elements of remote meetings interacted to understand how these different features afforded specific types of benefits to public contributors. 37

| CONCLUSION
Our results provide important insights for researchers involved in the design and organization of meetings that include public contributors.
The shift to remote meetings with public contributors caused by Covid-19 is likely to become a feature of PPIE. It is key we understand preferences and key drivers of project uptake to ensure remote meetings are designed so that potential public contributors are not disenfranchised. Hybrid meetings are also becoming popular, and further research is needed on these types of meetings, as public contributors' preferences may be different in a hybrid meeting format, than when working solely online. We found that particular features of remote meetings can have a significant impact on project uptake, in our case ranging from 23% to 93% uptake. We identified features such as the provision of feedback, the role of the moderator, whether contributors need to have their cameras off and can step away, and whether the meeting length can have an impact on potential project uptake. We also found that features such as the provision of connectivity tools and support to connect to meetings did not have a significant effect, although this could be due to our sample having significant experience in remote meetings. Resources would be best allocated to moderator training and the provision of postmeeting feedback instead of arranging long meetings with socializing activities and providing ongoing technical support. These findings are useful for researchers, project managers and PPIE leads to inform the allocation of resources when designing remote meetings with public contributors. An allocation of resources that responds to contributors' preferences will likely result in higher uptake of public involvement in projects.