CIVIL ENGINEERING 365 ALL ABOUT CIVIL ENGINEERING



AbstractMethods to collect data in construction engineering and management (CEM) research are evolving, informed by recent technological advancements. One such method is research charrettes that allow effective interactions and knowledge sharing between expert industry practitioners and academic researchers, all colocated in a single venue, enabling rich data collection and live communication. A pivot point in technological evolution occurred with the COVID-19 pandemic, forcing a global shift to remote work. Hence, planned in-person research charrettes had to shift to remote sessions, relying on virtual conferencing platforms and online data collection mechanisms. Technology-enabled charrettes have allowed the authors to collect significantly richer data sets and ensure a more diverse representation of participants, while saving tremendous amounts of time. With the continuing emergence of technological applications, the world might not go back to functioning fully in person. The authors believe remote research charrettes (RRCs) will still be used in a post-COVID-19 world because of their superior performance. This paper builds on a previous publication that described traditional research charrettes as a method to enhance CEM research a decade ago; it offers a significantly updated and improved RRC method based on the knowledge gained from transitioning a dozen in-person charrettes into RRCs. It also presents performance comparisons between RRCs and traditional charrettes by quantifying metrics indicating how RRCs are more time-efficient and cost-saving, harness more participants from more diverse locations, and enable the collection of richer data sets and four times more industry comments and expert feedback. This paper also provides guidance on the integration of technology with traditional research charrettes, hence contributing to the CEM body of knowledge.IntroductionConstruction engineering and management (CEM) research that is focused on industry improvement is a relatively young and dynamic area that requires innovation and hard work to meet the challenging demand of collecting useful and usable data from industry. Accordingly, the quantitative and qualitative research methods employed in the field of CEM are continuously evolving to help it reach its intended performance. Examples of such research methods include conducting experimental research in construction (Bernold and Lee 2010) and conducting discrete-event simulation studies in CEM (Martinez 2010) as quantitative methods and employing observational research methods to study team performance in construction management (Leicht et al. 2010) as well as employing research charrettes for engaging industry in best practices research (Gibson and Whittington 2010).Research charrettes are structured workshops that facilitate respondent data collection and feedback in a short period of time, combining the useful characteristics of surveys, interviews, and focus groups into one setting (Gibson and Whittington 2010). They are widely used, hybrid methods that combine several data collection approaches into one intensive session, resulting in the development of a number of commonly used CEM planning and management tools such as the Project Definition Rating Index (PDRI) and Shutdown Turnaround Alignment Review (STAR) tools. Research charrettes facilitate efficient interaction with engaged industry representatives in best practices research because data are collected in a dynamic manner.The use of the right research methods and technologies at the right time improves the efficiency and effectiveness of the research, enabling faster and more focused results. New technologies are being employed in collecting data for CEM research while still relying on the most common data collection instruments of questionnaires, interviews, observations, literature reviews, case studies, source document review, and archival documents, among others. Examples of technologies used in CEM data collection research include email exchanges, telephone conversations, video analysis, and informal chats as well as online data collection applications and software such as Qualtrics, Google Forms, and SurveyMonkey and advanced technologies such as drones and the use of virtual and augmented realities. Such technologies increase the productivity, reliability, and versatility of data collection approaches. For example, Qualtrics or Google Forms enable creating, distributing, and collecting responses for surveys and questionnaires remotely, which saves time and resources as well as reduces human error in the process of reentering the data manually. The use of drones and virtual reality allow for precise field data collection specifically at high-risk locations, which ensures safety, precision, and speed in collecting sensitive data.Since it began in late 2019 and later its declaration as a worldwide pandemic in early 2020, COVID-19 has impacted the progress of all industries around the world and research work has been affected with its share of consequences. Specifically, between March and mid-October of 2020, the US academic research enterprise in science, technology, engineering, and mathematics (STEM) fields was disrupted, which impacted intellectual activities and the associated revenues (Radecki and Schonfeld 2020). Although traditional research activities were largely suspended, innovations in research collaboration and scholarly communication took place that allowed the continuation of certain research efforts (Radecki and Schonfeld 2020). This was enabled by shifting to remote work, forcing a quick and challenging adaptation to remote research using advanced technologies. Examples of such technologies particularly for the purpose of communication include teleconferencing tools (e.g., Skype) and updated versions of tools that include more features of audiovisuals, screen sharing, whiteboard, and annotations (e.g., Zoom, WebEx, and Microsoft Teams). Such technologies allowed the continuation of many research efforts that do not require lab or field experiments and can still be carried out using basic research resources such as access to computers and the internet. Hence, advanced teleconferencing tools are being used to conduct meetings as well as interviews and host focus groups sessions.Problem Statement and ObjectivesIn CEM research where industry engagement is needed to uncover best practices, it can be challenging to collect data and feedback from practitioners. Research charrettes are a proven method to enhance data collection with a number of benefits. Considering the advantages of research charrettes and driven by the pandemic that transformed much work to online interaction, as well as the availability of emerging technologies, the authors were compelled by circumstances to test a novel, remote mode of research charrettes.Using online data collection techniques (e.g., Qualtrics, Google Forms, SurveyMonkey) with teleconferencing platforms enabled the authors among other researchers to conduct structured workshops remotely. Such tools and resources facilitated research charrettes, allowing the emergence of a new form referred to by the authors as remote research charrettes (RRCs). Over the several months following the start of the pandemic, the authors used RRC to collect data from industry practitioners in the United States and internationally to build a new planning and management tool to be used by government and industry practitioners. Thus, the hypothesis used in this paper is that RRCs are different than in-person research charrettes in terms of the richness of the data set, the time required to collect data, the diversity of input of personnel, the cost required to conduct the session, and the efficacy of the data set.The objectives of this paper are to describe how research charrettes are being carried out in a remote work environment, identify successful RRC implementation techniques, and compare the performance of remote charrettes to that of traditional in-person research charrettes. The paper will update the research charrettes method, as originally described and discussed in detail by Gibson and Whittington (2010), and compare the results of in-person research charrettes and RRCs.Background on Research Charrettes and Technology Use in ResearchResearch Charrettes: Proven Benefits and Noted LimitationsA charrette is a structured focus group meeting used to elicit creative design ideas and practical input from project stakeholders in a workshop setting. Based on the French word for cart, charrette became part of the architectural vocabulary in the early 1900s at the Ecole des Beaux-Arts in Paris (Healey 1991). Nearing the end of a grueling architectural studio project, students would feverishly work to finish their projects and place them on a cart as it was being wheeled down the aisles between drafting tables to collect projects. The term is broadly used by architects and engineers today for structured, intensive early project planning sessions.Research charrettes were inspired by the use of planning charrettes by designers to benefit from industry experience in producing early project planning ideas in a workshop setting (Gibson and Whittington 2010). The research charrettes method was derived from these planning charrettes and focused on a new element called social data collection (Gibson and Whittington 2010). As identified by Gibson and Whittington (2010), research charrettes combine some of the best features of surveys, interviews, and focus groups. For instance, they satisfy the objectives of focus groups, such as helping participants explore and clarify their thoughts around a particular topic, while also including surveys for structured data collection from participants. Research charrettes are typically aimed at collecting data while also capturing different types of discussions that focus on tool development or validation of an idea or method. They have been used in multiple studies between 1996 and 2019 (e.g., Cho and Gibson 2001; Griffith and Gibson 2001; Yussef et al. 2019) and have gained popularity in the field of CEM particularly between the years 2010 and 2021. Table 1 provides a sample of studies using research charrettes with a summary of the topic description of each study and the objective behind employing research charrettes.Table 1. Summary of studies that used research charrettesTable 1. Summary of studies that used research charrettesStudyTopic descriptionObjective of employing research charrettesWhittington and Whittington (2009)Development of the STAR tool for the management of shutdown, turnaround, and outage projectsTo prioritize the developed tool as well as provide input and suggestions for improvementGibson et al. (2010)Understanding the critical issues that must be addressed during front-end planning (FEP) of infrastructure projectsTo develop relative weights to the project definition rating index elementsEsmaeili et al. (2013)Identifying critical success factors and determining performance metrics on construction projectsTo rank the preliminary list of identified variables of metrics and predictorsKang et al. (2014)Determining new benchmarking theories used in the creation of the Construction Industry Institute’s new benchmarking system of capital project performanceTo refine the questionnaires representing five project phases, FEP, engineering, procurement, construction, and startupFranz et al. (2017)Studying the role of integration in the performance of building construction projectsTo develop a questionnaire to identify practices of successful project teamsCollins et al. (2017)Development of a project scope definition and assessment tool for small industrial construction projectsTo evaluate and prioritize the proposed elements within the toolGrau et al. (2017)Quantifying organizational-behavior influence on cost and schedule predictabilityTo assess the findings from survey responses and characterize practical applicationsEl Asmar and Gibson (2018)Assessing the accuracy of front-end engineering design (FEED) for large industrial projects and measuring its impact on cost performanceTo provide and categorize as well as give weight to accuracy factorsSullivan et al. (2019)Uncovering new market entry decision factors for the sheet metal engineering and construction industryTo identify and prioritize market entry decision factors collected from the literature and from experiential knowledge of industry expertsYussef et al. (2019)Quantifying FEED maturity and its impact on project performance in large industrial projectsTo finalize the maturity elements and their descriptions and to collect quantitative data to test FEED maturity’s impact on project performanceFeghaly et al. (2020)To study the implementation practices of alternative project delivery methods for water infrastructure projects by assessing the state of practice during procurement and executionTo review and validate the development of the surveyGibson et al. (2020)State-of-the-art review on research, practice, and education of prevention through design (PtD) initiativeTo review PtD best practicesAlthough this method was used before 2010, Gibson and Whittington (2010) were the first to provide a detailed description of how to plan and conduct such CEM research charrettes. Research charrettes are traditionally carried out by having industry experts and researchers colocated in a single venue. Charrettes support input and feedback from all participants and allow for project data collection in real time from participants that represent certain stakeholder groups (Yussef et al. 2019). Conducting a successful research charrette requires proper preplanning of logistics as well as availability and allocation of resources such as travel funds and venue funds to carry out the workshops. When performed correctly, they enable collection of a large amount of data by actively collaborating with subject matter experts (Grau et al. 2017) and offer an environment for practitioners to engage in a structured atmosphere while using multiple data collection approaches to obtain responses in less time (Gibson and Whittington 2010). Because of their unique feature as a social data collection approach, research charrette sessions allowed Grau et al. (2017) to monitor the behavior of subject matter experts and to capture their thoughts on applications of the research. Engaging in research charrettes is an opportunity for experts to interact supervised by senior researchers to identify best practices from successful or unsuccessful project environments (Franz et al. 2017). Among the research charrette’s planning limitations is finding a good time and location that suits all the invited participants without delaying the research project. Another limitation is the cost of conducting the session considering travel and venue expenses. Considering the advancement in technology to hold virtual meetings, these limitations can be partially overcome without compromising the value of social interaction.The point of departure and foundation for this paper is Gibson and Whittington (2010), who identified a research charrette development process map to help ensure successful results. The steps identified in the process included plan and develop multiple data collection strategies using industry input; develop charrette agenda and data collection instruments; set up meeting logistics and solicit participation; and conduct research charrettes. They identified the prework as the most important component including involving the sponsor or oversight committee in putting in place the topic, objectives, issues for discussion, a developed and tested data collection strategy, and good background documents. Attendees must be selected based on specific background and experience criteria. As for the number of participants needed, that would depend on the purpose of the study and whether statistical analysis is to be performed or not.Logistics must be well defined prior to the workshop such as material and equipment needs, room layout and seating arrangements, breakout rooms, food and drink availability, handouts, nearby hotels, and parking and meeting location access and security arrangements. Also, handout packets and data collection forms must be carefully prepared and provided to participants at the beginning of the workshop. The meeting agenda and flow should be shared with the participants and should accommodate periods of informal networking as well as structured discussion following the presentation of the topic and research background by the researchers. Gibson and Whittington (2010) concluded that research charrettes enable rich data collection and live, person-to-person communication in an accelerated duration while reducing geographic bias of participants. They also allow researchers to field questions during data collection and help gain more consistent results from the participants collectively. These benefits should not be impacted by the sudden shift to remote work if care is taken in structuring the charrette events. Hence, this change compelled the authors to utilize videoconferencing tools and virtual data collection software to continue using charrettes for their ongoing research projects, which originally planned to use in-person charrettes for data collection.Data and Research Methods in CEM ResearchThere are a number of types of data needed in CEM research, and accordingly different methods to help collect said data. Examples of such data and methods vary from collecting qualitative data or experts’ opinions using Delphi or modified Delphi methods (Cheng 2014; Tayeh and Issa 2021), collecting qualitative data through observational research methods (Leicht et al. 2010), and collecting data based on practitioners’ experience using research charrettes (Gibson and Whittington 2010). Other examples include using questionnaires to collect data for national surveys or opinions on relative importance of project management best practices; structured interviews or facilitated discussions to gain insights from small group of practitioners on a small group of structured questions and answers; source document reviews to collect data for validation; and focus groups to collect qualitative data that are difficult to leverage from practitioners in one-to-one interviews (Whittington and Whittington 2009). As discussed in the “Research Charrettes: Proven Benefits and Noted Limitations” section, research charrettes include the features and advantages of many of the aforementioned methods and have proven to be very efficient in collecting practitioners’ feedback on newly developed frameworks and tools (Whittington and Whittington 2009; Collins et al. 2017; Yussef et al. 2019) to ensure the usability and applicability of the tools, supporting practitioners in their work.Use of Technology in CEM ResearchConsidering that research charrettes are a hybrid method that include social data collection, most of the CEM studies that employ them and use surveys or questionnaires as an approach to collect data require having the participants answer survey questions while being colocated in one venue (Kang et al. 2014; Franz et al. 2017; El Asmar and Gibson 2018). However, more recent studies are utilizing Qualtrics weblink to collect responses by distributing the links using emails (Yussef et al. 2017, 2019; Feghaly et al. 2019, 2020). Qualtrics and Google Forms have proven to be reliable and widely used software platforms for data collection (Shrestha et al. 2016; Ezeldin and Abu Helw 2018; Bsisu 2020).Other advanced technologies are also used for data collection purposes in the CEM field. These include the use of mobile robotic systems for autonomous navigation in indoor areas (Asadi and Han 2020). Drones are also considered to help in collecting more accurate and enhanced real-time data from construction sites (Charlesraj and Rakshith 2020). Similarly, recent studies prove the effectiveness of using virtual and augmented realities in construction engineering education and training while providing useful information for researchers on how to integrate such technologies in their programs (Wang et al. 2018).Videoconferencing platforms are now an essential technical part of the CEM research world. Recent research has shown the benefits of using videoconferencing tools in carrying out remote research efforts and in collecting data for qualitative studies (Gray et al. 2020) as well as providing guidance through sharing best practices and field examples for researchers transitioning in-person assessments and efforts to a synchronous videoconferencing platform (Marhefka et al. 2020). Archibald et al. (2019) acknowledged the importance of Zoom among other advanced communication technologies in carrying out qualitative and mixed-methods research, and they investigated the feasibility and acceptability of using Zoom to collect qualitative interview data to better understand its suitability for such research. Gray et al. (2020) examined how the features of Zoom can be used to conduct successful qualitative research interviews and concluded with advantages discerned by the interviewers and interviewees using Zoom. These include ease of accessibility as well as saving time for other priorities where logistical hindrances such as travel funds and location to conduct face-to-face interviews are avoided. Despite the physical separation, researchers still felt a real connection with interviewees, unlike with telephone interviews.Use of Technology for Design CharrettesIn the prepandemic world, one study was identified that tested the use of an online approach to conducting design charrettes in a virtual environment using Web 2.0 (Ryan et al. 2008). They identified a number of online applications that can be used as alternatives for traditional design charrettes for planning purposes such as Wikiplanning, Meetup, Zebralog, PlaceMatters, and Planetizen.In a recent publication, Dirrigl et al. (2021) discussed how they began adopting the charrettes method to collect data to identify regional environmental and sustainability issues in South Texas; however, their final charrette was disrupted by the COVID-19 pandemic. Accordingly, they used Zoom as a meeting platform to conduct a half-day online charrette in 2020, considering its features of allowing qualitative data collection (Archibald et al. 2019), which is critical to a charrette. They identified the practicability of utilizing the breakout rooms feature and assigning participants in facilitating the discussion. Using the Zoom share screen feature allowed the participants to visualize the shared information, which enhanced the overall process. This is an example of design charrettes where progress was hindered by the COVID-19 situation, but it was overcome with technology.Certainly, in today’s environment, because of COVID-19 and because of the proven new technologies enabling remote work in general, remote research charrettes are ubiquitous; however, no studies were found outlining their efficacy.Remote Research CharrettesRRCs are an updated version of research charrettes such that the workshops are conducted remotely via teleconferencing platforms. As mentioned in the background section, the authors had plans for using research charrettes in an ongoing research study funded by the US Department of Energy (DOE) to engage ideas from the industry and leverage practitioners’ experience. Although plans were set to conduct these in-person research charrettes in predetermined locations, the COVID-19 pandemic started and health regulations and safety restrictions disrupted the plans. Nevertheless, the academic world adjusted by using videoconferencing tools to continue meeting and teaching classes. The authors had previous experience with using data collection tools such as Qualtrics to harvest information from practitioners by sharing a developed questionnaire with a weblink. Accordingly, the authors came up with a new plan to carry on the research charrettes by combining the videoconferencing with online data collection software. The former is to accommodate for the presence of all participants and ensure effective interaction between them and the research team by being able to see them, listen to their thoughts, engage them in discussions, and share with them presentations and documents. The latter is to ensure proper distribution of forms to collect the needed information remotely.RRC MethodRRCs are planned in a similar manner to traditional research charrettes by determining the issues to be discussed and the information to be collected, as shown in Fig. 1. Accordingly, the agenda for the charrette is planned and data collection instruments needed to capture the information are prepared. A Qualtrics-based survey (or other electronic platform) is developed to collect data instead of printing the forms to distribute to the participants during the workshop. Also, necessary documents about the research topic are prepared and sent via email to the participants before the workshop instead of distributing packets during the workshop. Choosing the participants is based on a list of certain criteria determined by the researchers and the sponsor. As for logistics, choosing a convenient geographic location, picking a venue, planning travel, and all other venue-related details such as reserving rooms, preparing audiovisual equipment, and ordering food are not applicable. Logistics are now related to picking a convenient time to meet across several time zones and having adequate resources such as electricity, computers, headset or speaker, microphone, and internet connection for all participants (participants are reminded of their requirements ahead of time). The senior researchers host the workshops and share details about the research and workshop objectives when asking practitioners if they are interested in participating. After receiving confirmations or declinations from the invitees, the research team sorts the participants list and sends save-the-date invitations, documents to read before the workshop, workshop link, and agenda to the confirmed invitees as well as directions on what to prepare before attending the workshop. Through the invitation links, the participants confirm or decline their attendance. The sponsor, who is also a practitioner, can help host and coordinate the workshop. How the remote workshop is carried will be discussed in more detail with an example in the following section.The authors have anecdotally heard of other researchers moving toward using RRCs or similar forms of technology to continue their efforts. Whether this continues post-COVID-19 or not remains to be seen, but RRCs can be used in CEM research among other techniques that require human-based interaction between individuals from different areas of the world to exchange thoughts and conduct technical discussions while collecting data and feedback in the process. As shown by Dirrigl et al. (2021), different software and applications can be coupled with videoconferencing platforms to satisfy the objective of the charrette. The advancement of remote research–enabling technologies exacerbated by COVID-19 pandemic restrictions influenced the need to use the RRC method. Twelve remote workshops conducted by the authors on a single topic have proven the feasibility, effectiveness, and practicality of using RRCs as described subsequently.Research MethodologyTo test the hypothesis regarding the difference between RRC and in-person charrettes, the authors followed a systematic research methodology described here. First, they conducted a number of RRCs, which were prepared, developed, and hosted following a consistent process. Second, the authors documented the process and improved it based on lessons learned. Third, the authors demonstrated an RRC example to specifically highlight the integration of the videoconferencing platform and the data collection technology, while maintaining the social, professional, and technical features of a traditional research charrette. Fourth, the authors collected research charrettes characteristics data from the recently conducted RRCs and compared these data to previous traditional research charrettes from projects conducted by two of the authors. With the identified metrics, they conducted comparative analyses to highlight the performance of RRCs compared to the traditional method. Fifth, throughout conducting the RRCs, the authors identified challenges faced and lessons learned underlying their experiences as well as through participants’ feedback. The authors then discussed the findings in reference to recent studies published on data collection and research mechanisms in a remote environment. Finally, a list of recommendations was developed around how to successfully conduct RRCs. The following sections detail each step of the methodology.Remote Research Charrette ExampleFrom June 2020 to August 2021, the authors conducted 12 RRCs focused on the topic of integrated project and program management as related to earned value management systems (EVMSs). These workshops were collectively aimed at developing and testing a decision-support tool, the Integrated Project/Program Management Maturity and Environment Total Risk Rating (referred to as IP2M METRR). This tool is intended for use on large, complex projects and programs that need to comply with EVMS guidelines. More background on this topic is discussed by Cho et al. (2020) and Aramali et al. (2022b).Of the initial eight RRC workshops focused on developing the new decision-support tool, four tackled the first component of the tool. The researchers aimed to adequately describe and weight the relative importance of 56 EVMS maturity attributes (Aramali et al. 2022a). The other four workshops tackled the second component of the tool; they aimed to identify, describe, and weight 33 environment factors that impact the EVMS implementation. The final set consisted of four workshops that allowed practitioners to use the tool to assess their own projects and programs and to collect project or program quantitative and qualitative performance data such as cost and schedule-related information as well as compliance with the appropriate guidelines, customer satisfaction, and meeting the business objectives.Emails were used to communicate workshop information and to send invitations to prospective participants. All the workshops were conducted via Zoom, with Qualtrics used to collect participant demographic information as well as their input to survey questions used to assess the tool. Proper use of the technology as well as management experience from traditional charrettes enabled the successful conduction of the workshops, while preserving as much as possible the interactive nature of charrettes. Based on experience, the authors feel this melding of technologies would work effectively for carrying out structured and unstructured interviews and focus groups meetings.Similar to numerous research studies, a multitude of statistical analyses can be used to test the hypotheses. In this example, descriptive information about the participants, comments received about strengths and areas for improvement for the tool, as well as feedback and suggestions to advance the tool were collected and evaluated. Forced ranking and sensitivity analysis were used to determine the proper weights of the attributes and factors. For the data validation workshops, testing for statistically significant differences among performance metrics and correlation between the assessment results and performance results took place; also, sensitivity analyses were used to determine thresholds to categorize performance outcomes.EVMS Environment RRC as an Illustrative CaseA research team consisting of four researchers and 27 industry practitioners cocreated the tool. The practitioners are experts in earned value management (EVM) as well as project and program control with an average of 19 years of relevant experience. Together with the authors, the research team drafted the initial IP2M METRR tool with 56 maturity attributes grouped within 10 subprocesses, and 33 environment factors grouped within four categories.After drafting the initial tool, a set of eight RRCs were conducted to prioritize both components of the tool as well as provide practitioners’ feedback for refinement. The following description is for the set of workshops dedicated to the environment component of the tool (the maturity component was also assessed in a similar manner in four other workshops). Each of the workshops consisted of four major sections as shown in the agenda in Fig. 2. The first section consisted of introductions and objectives, starting with a welcome word by the sponsor, followed by brief introductions of each of the participants. Next came an overview of the workshop and research objectives, a confidentiality statement, and the research study milestones. Afterward, definitions of the project key terms set the stage for a presentation of the environment component of the tool with its list of environment factors as well as a detailed example of one environment factor.The second section of the workshop consisted of collecting data to review and critique the environment factors and assessment mechanism, as well as to develop credible weights reflecting the relative importance of each environment factor. The workshops were structured in an organized manner to ensure consistent and efficient data collection while allowing for an open discussion with questions and answers for each factor. The Qualtrics link was shared with the participants in the Zoom chat box so they could start filling in background information about themselves, their organization, and the individual project they were using as the anchor for their input. The hosts had shared documents and instructions with the participants before the workshop so they could each come prepared to the workshop with a recent anchor project. Then instructions were clearly articulated to solicit evaluation input from the attendees. An example of the types of instructions used in this workshop included: “First, collectively participants will review each factor (each factor will be projected on Zoom, in order). Second, participants can ask questions and provide comments and suggestions on each of the environment factors, in order, by category. Participants should use the Qualtrics comment box for each category to provide edits/suggestions/input related to any of the factors.” These sentences show how the hosts are making use of the resources available to enable RRC. Basically, the workshop slides were shared all the time on the screen using the Zoom share screen feature to provide instructions and keep the participants well informed and literally on the same page. Once the participants access Qualtrics, a similar preview (screenshot of each Qualtrics screen) was also shown in the presentation slides to make sure participants can follow the session flow easily without the need to shift between multiple screens. The respondents were then asked to give feedback and suggest edits to each of the factors within each category through a comment box, as shown in Fig. 3. After all the factors for a given category were reviewed, the participants were asked to indicate the relative importance of each factor within this environment category. Questions and comments were encouraged during this process with two-way dialog that could be joined by anyone participating in the session. Fig. 4 shows one example where the rating is on a scale of 1 to 5, where 1 means the most impactful and 5 means the least impactful. The details of the method used to develop the prioritizations are described by Aramali et al. (2021).Once each factor was weighted within each category, the third section of the workshop assessed these high-level categories against one another. The participants were asked to weight the four categories of the environment component by giving each a percentage out of 100 such that the four grades sum up to 100. Moreover, the proposed rating scheme and rating levels were reviewed to collect feedback on their functionality and clarity. Finally, the data collection was wrapped up by giving the respondents a last chance to provide general feedback and to review previous answers and input, if needed, before submitting their responses. Before exiting Qualtrics, the respondents were asked if they would like to receive continuing education unit (CEU) credits for their participation.The fourth and final stage of the workshop concluded by sharing how the envisioned environment rating score sheet would look along with the proposed summary visuals in the form of a maturity and environment rating plot or matrix. Then the (virtual) floor was open for further questions and discussions. These four workshop stages, along with a planned break halfway through the allowed time, detail one example of how a successful RRC can be conducted. However, the process does not stop here. Follow-up work takes place to address the given comments and collected feedback, and to follow up on any missing information.Based on the feedback and weighting data from 20 participants in the first two environment workshops, the authors analyzed the data and combined some factors, reducing the number of factors from 35 to 27. These factors were then ranked by 27 additional practitioners in the third and fourth environment workshops. In addition to prioritizing factors, the environment component of the tool received 368 comments from the first two environment workshops. Most of the comments were addressed by the authors and in some cases experts from the research team were called upon to help (requiring about 20 meeting hours over Zoom), resulting in a new and improved version of the factors that were then presented in the third and fourth workshops. During these final two environment workshops, a total of 308 comments were received from the participants and again were resolved collectively by the authors and industry experts over a second set of about 20 meeting hours via Zoom, before finalizing the tool to its most updated version. Table 2 shows an example of a few of the overarching comments received. The RRC setting as well as Zoom features encouraged and enabled receiving such a large and diverse number of rich comments evaluating the tool and associated approach. The participants were able to interact with other practitioners from diverse industries and locations and exchange thoughts about common challenges and best practices. Unlike in-person research charrettes, informal networking during breaks and breakout session reports were not doable; yet, the Zoom feature of individual or group chatting encouraged traditionally more reserved individuals to voice their opinions and contribute to the discussions.Table 2. Sample of overarching comments from RRC workshopsTable 2. Sample of overarching comments from RRC workshopsEnvironment workshopCommentsComment 1I really like the automated approach to questioning. Very efficient.Comment 2This assessment should be beneficial, especially for management and customers if they will adhere to the principles/guidance from the assessment. Appreciate the ASU [Arizona State University] participants and their work on this. It was very fast with regard to providing comments.Comment 3The Environment assessment model is a very well thought out and articulated model. It will be very instructive to get real-time experience with the model and fine tune where criteria or factors overlap, and how observed performance is translated into ratings.Other observed advantages of this type of RRC data collection method are the valuable questions asked by the practitioners on how to use the envisioned tool, their feedback on the tool and how to further enhance it, the discussions about how the data they are providing will be used in determining performance assessment, and what they gained in the session in terms of knowledge around the EVMS strengths and gaps, hopefully enhancing their ongoing projects and future projects. This feedback was used to improve the tool development effort and to inform the authors’ and steering team’s research efforts.As mentioned previously, four additional RRCs were conducted for tool validation in a similar manner. Qualtrics survey questions inquired about project performance data and included the developed tool to assess the maturity and environment of a sample of completed projects. A total of 35 projects and programs totaling more than $20 billion in cost were used to test the tool.Results: Performance Comparison between RRC and Traditional Research CharrettesIn this section, the authors compare the performance of RRCs using the IP2M METRR example, with its two types of workshops for data collection and tool validation, against the performance of traditional research charrettes relying on data from studies that used research charrettes in the years before the COVID-19 pandemic. These studies are listed in Table 3 along with data describing their characteristics. The performance was compared using several metrics that reflect the quality of the workshops, their efficiency, their value in terms of feedback quality and quantity, and their diversity in terms of participants represented.Table 3. Characteristics of studies employing traditional research charrettes and RRCTable 3. Characteristics of studies employing traditional research charrettes and RRCProjectYearNumber of research charrettesNumber of organizationsNumber of participants per charretteAverage number of participants per charretteTotal number of participantsAverage experience of participants (years)PDRI (industrial)1995234Not available275419PDRI (building)199973513, 6, 12, 9, 7, 6, 16106922STAR20094498, 26, 20, 13176720PDRI (infrastructure)201162816, 8, 7, 13, 9, 11116422PDRI (small infrastructure)201652919, 12, 12, 12, 10136520PDRI (small industrial)201674320, 5, 6, 10, 15, 8, 7107117FEED MATRS201843114, 6, 9, 19124823RRC: IP2M METRR2020–2021126012, 13, 16, 23, 10, 11, 16, 19, 14, 6, 15, 211517619Most of the projects presented in Table 3 occurred between years 1995 and 2018 and employed traditional research charrettes; the last project listed occurred in 2020–2021 using RRCs. The RRC project has the highest number of charrettes conducted (arguably in part because of the lower associated in-person logistics and related costs, which will be explored subsequently) as well as the highest number of participating organizations. This resulted in the largest number of total participants, more than double the highest number we found for a traditional research charrette. Getting participants to any single venue is generally challenging, but the authors were able to hold more charrettes because travel time and associated costs are now invested in RRCs as opposed to physical costs. Fig. 5 shows the average duration of the research charrettes for all the projects presented in Table 3. The RRC project had the shortest duration of less than 4 h including breaks, whereas the other workshops had a 1-h lunch break, which prolonged the duration of the charrette. This translates into a direct 20% workshop time saving when conducting an RRC versus a traditional research charrette. However, this does not take into consideration the additional indirect travel time savings, which are typically much higher, especially once multiplied by all participants.Similarly, a cost comparison was performed by first estimating the cost of conducting research charrettes for each of the projects. Table 4 shows these estimates and their totals for each project. For simplicity, the cost estimate is based on three major anticipated components: flight fee, hotel fee, and food expenses. To come up with a reasonable estimate for the flight and hotel fees, it was assumed that only the participants who come from a geographic location other than the workshop location pay flight and hotel fees. For example, if the workshop location is Texas and the participant comes from Nevada, they are assumed to pay flight and hotel fees. These are referred to in Table 4 as the traveling participants. To determine the number of the traveling participants for each project, the percentage of the participating organizations from locations other than the workshop location was calculated using the participant characteristics data collected. The calculated percentage was multiplied by the total number of workshop participants. The result was then multiplied by an estimated average fee for a US domestic flight and one hotel night, resulting in what the table refers to as flight and hotel expenses for traveling participants. Also, the food expenses for the workshop participants were added to the fees. These data also include the research team, which on average includes two professors and two graduate students, as well as the local participants where an average of $20 per participant was used for the day to conservatively estimate the total food expenses. This value was added to the flight and hotel expenses, resulting in the total expenses for each project, which sum up to a total of about $172,000 for the studied projects that utilized traditional research charrettes with the average being $24,500 per project. This value does not account for all expenses such as fuel cost for local participants to commute to the workshop venue, flight and hotel expenses for the research team when the workshop is not conducted in their home location, venue reservation fees, and printing fees; it is still considered a significant cost, all of which is saved when conducting an RRC, which costs $0 per event. But perhaps even more importantly, other than the estimated cost savings, RRCs save a significant amount of travel time and time away from their usual work location for all the participants. RRCs also help reduce the carbon footprint from air and land travel, making it a more environmentally friendly approach.Table 4. Research charrettes expenses across a variety of projectsTable 4. Research charrettes expenses across a variety of projectsProjectParticipating organizations from locations other than the workshop location (%)Number of traveling participantsFlight and hotel expenses for traveling participants ($)Total number of participants including the research teamFood expenses (approx.) ($)Total expenses (approx.) ($)PDRI (industrial)683727,068581,16028,228PDRI (building)543727,756731,46029,216STAR594029,279711,42030,699PDRI (infrastructure)432720,325681,36021,685PDRI (small industrial)664331,556691,38032,936PDRI (small infrastructure)312216,188751,50017,688FEED MATRS301510,753521,04011,793RRC: IP2M METRR1000018000Total—220162,9256469,320172,245To compare RRCs with the traditional research charrettes, only the development and weighting workshops (for maturity and environment) from the IP2M METRR project are included in the following analysis. The number of comments recorded per workshop are shown in the column graph in Fig. 6, while the line graphs show the cumulative sum of comments received for each project. The results show a significant increase in the number of comments captured for the IP2M METRR workshops, reaching 305 comments in one. The comments captured from the eight RRCs sum up to 1,534 comments in one research project, compared to 819 total comments from 16 traditional research charrettes for the three other combined projects. This translates into almost four times more comments per charrette, demonstrating how the use of technology enabled a more effective capture of expert feedback for this sample.Fig. 6 shows the increase in the number of recorded comments for the RRC. Equally as important, however, is the diversity in the sources of these comments. Conducting remote research charrettes enables experts from anywhere in the world to participate. Fig. 7 shows the diversity of the participants between one project (FEED MATRS) that used traditional research charrettes and one project (IP2M METRR) that used RRC. The geographical diversity is represented by the locations of the workshops’ participants in comparison to the workshop location itself. For example, the first workshop of FEED MATRS was conducted in Houston, so most of the participants came from Texas and only a few of the participants traveled from three other states (California, Colorado, and New Jersey) to attend the workshop. The fourth workshop of FEED MATRS was conducted in Calgary, Alberta, Canada, with all its participants coming from Alberta. Conversely, the IP2M METRR workshops were all conducted remotely, enabling more participants from more diverse locations to attend. This is shown by the larger number of needle pins on the map for IP2M METRR versus the smaller number of pins for FEED MATRS. In total, organizations from 22 states and four countries (United States, Australia, Greece, and Canada) were represented in 12 RRCs. Such diversity ensures wider representation and expertise from more organizations across the country and internationally.Nevertheless, conducting research charrettes online has some drawbacks. Researchers conducting RRCs have more difficulty in fully confirming all reservations in advance because it is very easy to confirm attendance to an online meeting and then cancel, whereas for in-person charrettes participants need to give it more thought and planning and may commit financial resources to reserve accommodations; canceling at the last minute could come with a financial burden. In a traditional in-person research charette, the hosts can more easily observe participants and ensure that all attendees participate in the data collection exercise and answer all questions in the survey before leaving the meeting room. However, in RRCs, the hosts do not have the same control of the room to ensure everyone stays engaged all the time, nor can they guarantee that all participants are filling out the survey and not passively observing the session. This final point can be visualized in Fig. 8, where the number of responses represents the number of participants who provided information and filled out the survey questionnaire. For projects using in-person research charrettes, the number of responses is generally equal to the number of participants, while for the project using RRC the number of responses collected lags behind the number of participants. This is a noted weakness for RRCs; however, even without collecting feedback from all the attendees, RRC still had almost four times more responses per charrette when compared to in-person charrettes.Considering the performance comparisons discussed in this paper, as well as the authors’ experiences in facilitation of both traditional and remote research charrettes over three decades, it can be noted that RRCs have proven to be a more effective and sustainable research data collection method for these types of research projects. Some of the associated metrics include the following: •Four times more comments per workshop;•Three times more comments per participant;•Shorter workshop times, thus it is easier to attract participants;•Zero travel time;•Almost zero financial cost per workshop;•No paper copies of the participant’s workshop packages and survey, as well as no airplane use, thus reducing carbon footprint; and•Less data entry errors because the data are directly input in the database by the respondents, instead of having researchers manually input the collected responses and then having someone else check what was entered versus the actual responses to confirm correct data entry.DiscussionBenefitsOverall, using RRCs is a cheaper way to get much more data. One observation is that remote workshops have enabled more individuals to share their thoughts through asking questions directly and confidentially to the hosts via Zoom’s individual chat feature. Also, a Qualtrics feature allows respondents to pause data entry and carry on with it at a later time, which provides flexibility that proved critical for some participants. The pandemic and travel restrictions played a key role in making RRC a possible and usable method. In the last 2 years, the technology became available to support this change, which was practically impossible a decade ago. And even if it were technologically possible several years ago, people were not ready to change. Today most people are trained and set up for using technologies for virtual communication and to work remotely, so asking practitioners to participate in remote charrettes is not a challenge anymore.Recently, impacted by the COVID-19 pandemic and the resultant forced shift to remote work, many researchers, especially those involved with data collection that relies on human interaction, started accommodating to the situation and finding new methods and techniques to continue their work. Lobe et al. (2020) provide guidance on the videoconferencing services suitable for a project’s needs to transform from face-to-face qualitative data collection to socially distant methods. This study exemplifies such efforts from the medical sector and identifies Zoom among others as convenient platforms; however, they point out the issues of ensuring security and confidentiality of the data in the process. Opara et al. (2021) describe the implementation methodology as the benefits and challenges of using Google Docs to conduct synchronous, online, written interviews through presenting two of their projects’ case studies as live examples. Their study is within the social sciences field. The benefits identified include more privacy for the participants, and “shy” individuals can voice their thoughts better. Also, more flexibility and time savings with respect to “transcribing verbatim immediately.” Similarly, Marhefka et al. (2020) discuss best practices and key considerations in transitioning in-person data collection to videoconferencing platforms in the medical field.Considering the performance results of RRCs, it is evident that they share common advantages with the methods and techniques described here and in the background section. These include saving time, voicing the thoughts of more individuals, increasing the diversity of participants, and reducing travel costs, among others. However, unlike RRCs, all these existing studies do not address the CEM field, do not quantify the performance metrics that were addressed in this paper, and mainly rely on one platform or tool to specifically collect a certain type of data rather than integrating a videoconferencing platform that connects the investigators to the attendees, a data collection tool that captures attendees’ interventions, and a research method that solicits practitioners’ experiences and direct feedback.Challenges of RRCsWhile RRCs allow for collecting a richer data set from experts, the sheer amount of data will require more time for researchers to go through, assess the input, and address it. This can be considered a challenge associated with RRCs. Indeed, the authors have identified several challenges associated with RRCs. The major challenges of this method are technical difficulties, associated with the use of an online meeting platform. Similar to any online tool, the successful use of the Zoom software is contingent on functional infrastructure and services of electricity and internet connection. In some cases, the functionality of these services is jeopardized by local weather conditions leading to cutting off an attendee from the call. Also, because of conducting all work remotely and keeping the computer turned on most of the time, computers might crash at any point in time causing disruptions to the workshop. The ability to rejoin the meeting after being disconnected is contingent on the speed of the systems update or the availability of other devices with installed and functional Zoom software. Also, sound settings of speakers, microphone, headsets, and cameras must be adjusted to the proper settings. None of these issues were required in the traditional charrette method.In terms of workshop logistics, in traditional charrettes participants can usually hear the hosts’ instructions the first time, but this might not always be the case on Zoom. Also, participants arriving late to a virtual meeting (which has happened a number of times) can also exhaust the hosts if they need to repeat instructions several times or answer the same questions that were explained at an earlier time. During in-person sessions, hosts could walk around the room to guide the respondents; in virtual sessions this is not possible, so it requires very explicit instructions and continuous feedback on whether the pace of the workshop is too fast and if everyone is tracking together. Traditionally, presenters can interact with the facial expressions they receive from the audience; however, in Zoom many participants decide to turn the video off, which can be a communication barrier. Moreover, the hosts cannot guarantee that all the attendees at the start of the workshop will attend the whole session. For example, in one case a workshop started with 28 participants and finished with 19. Attrition was observed to be higher in RRCs compared to traditional charrettes. Finally, the workshop time must be carefully set during reasonable working hours across multiple time zones because the attendees are geographically dispersed. However, regardless of the issues discussed here, the benefits of RRCs are several and far outweigh the challenges.Recommendations for Conducting a RRCThere are many similarities in the steps needed to conduct traditional research charrettes and RRCs, such as the preparation work, developing questions and data collection instruments, preparing the presentation, and so on. An extra step here is to ensure the digital functionality of the new data collection instruments by testing them and piloting them prior to the event, and also developing a contingency plan in case internet connection problems occur during the workshop. Examples are discussed in the lessons learned section.The researchers need to hold rehearsal meetings using all the exact same software and data collection instruments. They also need to ensure the slides to be used during the charrette are ready and include all the needed information. One primary purpose of the rehearsal meeting is to guarantee a clear and systematic method of communicating the workshop instructions. Timing, responsibilities, and even breaks are all rehearsed, along with all support tasks.Prior to the event, the research team and the sponsor must cooperate to form a list of needed criteria to start identifying suitable candidates (based on the participants’ background, position, years of experience, location, organization, and so forth). The sponsor’s network is leveraged in this scenario considering their tie with the project’s objectives. Then invitations with the charrette’s description are sent out through emails requesting confirmation from invitees. To attract a large number of suitable participants, it is critical to describe in detail the purpose of the charrette and what the invitees will gain from participating. These benefits may include early involvement in developing a best practices tool that will be widely used, their ability to use the tool early to enhance their projects’ performance, interaction with their peers from other companies and industries, gaining benchmarking insight into their projects compared to other similar projects, and receiving continuing education credit for participation. To increase the number of participants, the sponsor and the research team encouraged participants to also suggest names of experienced colleagues who might be interested in the study. The experience of the participants with the tool’s topic is critical to their ability to provide credible and valuable feedback based on their industry perspective; therefore, one criterion often required is a minimum of 10 years of experience. The resulting average experience of participants generally ends up at around 20 years or so.On the workshop day, the hosts start the virtual meeting 15 min ahead of the scheduled time to ensure there are no technical challenges and to prepare all the meeting documents and needed links (for Qualtrics). They start admitting participants to the virtual room and make announcements of when the workshop will formally start. Graduate research assistants are responsible for keeping track of time and reporting to the hosts, keeping the workshop on schedule, answering any technical questions posted in the chat room, sharing the Qualtrics links and monitoring the data collection process, keeping track of attendees, and writing down participants’ questions, comments, and discussion ideas. The hosts and the sponsor facilitate the charrette and they welcome questions from participants at any time, either orally or through the chat feature.After the charrette concludes, all participants will receive thank you emails for their contributions, and those who provided incomplete responses for the surveys will be contacted and reminded to complete their responses. The participants will also receive project reports and publications once these are finalized.Practical Lessons Learned from RRCsAfter conducting eight tool development charrettes and four tool validation RRCs, a number of practical lessons can be shared. These include: •Test the data collection technology beforehand.•Check the computer and videoconferencing software setup ahead of time and have a contingency plan to use another nearby equipment or location.•Request that the hosts and participants have redundancy in the devices available in case their main device crashes during the workshop.•Ensure hosts have enough mobile data on their phones for hotspot connection in case the Wi-Fi disconnects.•Have more than one host ready to facilitate the full workshop on their own.•Organize files in a systematic way in a shared folder between research team members.•Have one point of contact between the research team and all participants to properly track communication.•Send invitations a couple of weeks in advance, send a reminder, and send a calendar save the date, as well as follow-up emails and requests for missing data.•Share workshop documents with the participants ahead of time, urge them to go over the material before the workshop, and encourage questions.•Reinvite participants of the initial tool development charrettes to the later tool validation charrettes to generate more effective feedback and data collection.•Verify participant list against RSVP roster during the charrette to ensure everyone in the meeting is authenticated.•Share analysis of previously collected data early in the charrette because it motivates contribution and encourages participants to provide more data to benchmark their performance.•Offer to facilitate tool implementation sessions, which increases practitioner buy-in.ConclusionsResearch charrettes are proven to be a widely used method for data collection and tool development in use-inspired research studies that incorporate industry best practices. As described in this paper, this popular method is updated through making use of the new technologies and to accommodate postpandemic work styles. This endeavor shows that conducting remote research charrettes is indeed possible and even comes with several advantages. A number of metrics were used to quantify these advantages in comparison to traditional research charrettes, which include shorter duration per research charrette, more than double the number of participants per project, more diversity in the participants’ geographical representation, and almost four times the number of data received per charrette. Conducting remote workshops is also more convenient because it allows for increasing the number of workshops and the possibility for each event to focus on an individual component of the work, which encourages more focused discussions and better feedback. In addition to the time saved, the costs associated with in-person charrettes are on the order of tens of thousands of dollars, most of which can now be saved using RRCs, in addition to the carbon footprint saved from the reduction in travel. Conversely, some limitations are noted with this method and mostly are related to technical issues related to the availability and consistency of internet service, electricity, and electronic device reliability.This paper serves as a guide to a convenient, resource-light, effective, and overall superior data collection and tool development method for collecting data in CEM research. Hence, it contributes to the CEM research body of knowledge by updating and strengthening an existing proven method by adapting to new technologies as a result of the pandemic. The authors feel that CEM researchers and practitioners will perceive RRC as a pioneering method that is likely to remain popular in the post-COVID-19 world because of all the benefits discussed and documented in this paper. A number of possible research projects can benefit from this method, not only those specific to CEM research. Research projects funded by government agencies, or others, that require the intervention or feedback of multiple stakeholders across a wide geographic area, soliciting opinions, advice, or creative input are all candidates for this method. Hence, even though travel may be possible postpandemic, it is more practical, efficient, and time- and cost-effective to host research charrettes online. This method has proven its effectiveness and offers the ability to capture data through mixed research methods. RRCs may be especially advantageous for studies that require international contributions, even with large time zone differences.Data Availability StatementSome or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.AcknowledgmentsThe authors would like to thank the US Department of Energy’s Office of Project Management for their funding and support of this national study, as well as all the government representatives and industry participants who provided their time and expertise to support this work.References Aramali, V., G. E. Gibson Jr., M. El Asmar, and N. Cho. 2021. “Earned value management system state of practice: Identifying critical subprocesses, challenges, and environment factors of a high-performing EVMS.” J. Manage. Eng. 37 (4): 04021031. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000925. Aramali, V., H. Sanboskani, G. E. Gibson Jr., and M. El Asmar. 2022a. “EVMS maturity and its impact on project cost and schedule performance of large and complex projects: A preliminary analysis.” In Proc., Construction Research Congress 2022. Reston, VA: ASCE. Aramali, V., H. Sanboskani, G. E. Gibson Jr., M. El Asmar, and N. Cho. 2022b. “Forward-looking state-of-the-art review on earned value management systems: The disconnect between academia and industry.” J. Manage. Eng. 38 (3): 03122001. https://doi.org/10.1061/(ASCE)ME.1943-5479.0001019. Archibald, M. M., R. C. Ambagtsheer, M. G. Casey, and M. Lawless. 2019. “Using zoom videoconferencing for qualitative data collection: Perceptions and experiences of researchers and participants.” Int. J. Qual. Methods 18 (Sep): 1–8. https://doi.org/10.1177/1609406919874596. Asadi, K., and K. Han. 2020. “An integrated aerial and ground vehicle (UAV-UGV) system for automated data collection for indoor construction sites.” In Proc., Construction Research Congress 2020: Computer Applications, 846–855. Reston, VA: ASCE. Charlesraj, V., and N. Rakshith. 2020. “Stakeholder perspectives on the adoption of drones in construction projects.” In Proc., ISARC. Int. Symp. on Automation and Robotics in Construction, 1227–1234. London: The International Association for Automation and Robotics in Construction Publications. Cho, N., M. El Asmar, G. E. Gibson Jr., and V. Aramali. 2020. “Earned value management system (EVMS) reliability: A review of existing EVMS literature.” In Proc., Construction Research Congress 2020, 631–639. Reston, VA: ASCE. Dirrigl, F. J., Jr., A. M. Chavez, and H. R. DeYoe. 2021. “Identifying regional environmental and sustainability issues: Adoption of an interdisciplinary design charrette approach.” J. Plann. Educ. Res. (Jun). https://doi.org/10.1177/0739456X211020084. El Asmar, M., and G. E. Gibson. 2018. “The cost impact of front end engineering design (FEED) accuracy for large industrial projects.” In Proc., Int. Structural Engineering and Construction 2018: Responsible Design and Delivery of the Constructed Project. Fargo, ND: International Structural Engineering and Construction. Esmaeili, B., B. Franz, K. R. Molenaar, R. M. Leicht, and J. Messner. 2013. “A review of critical success factors and performance metrics on construction projects.” In Proc., 4th Construction Specialty Conf. Point Claire, QC: Canadian Society for Civil Engineering. Feghaly, J., M. El Asmar, S. Ariaratnam, and W. Bearup. 2019. “Selecting project delivery methods for water treatment plants.” Eng. Constr. Archit. Manage. 27 (4): 936–951. https://doi.org/10.1108/ECAM-06-2019-0308. Gibson, G. E., Jr., E. Bingham, and C. R. Stogner. 2010. “Front end planning for infrastructure projects.” In Proc., Construction Research Congress 2010: Innovation for Reshaping Construction Practice, 1125–1135. Reston, VA: ASCE. Gibson, G. E., D. Grau, and H. Chava. 2020. Prevention through design 2020: Current and future state-of-the-art on research, practice and education, report 1. Tempe, AZ: School of Sustainable Engineering and the Built Environment, Arizona State Univ. Gray, L. M., G. Wong-Wylie, G. R. Rempel, and K. Cook. 2020. “Expanding qualitative research interviewing strategies: Zoom video communications.” Qual. Rep. 25 (5): 1292–1301. https://doi.org/10.46743/2160-3715/2020.4212 . Healey, E. H. 1991. “Planning a library in one week.” Am. Lib. 22 (4): 354–356. Kang, Y., J. Dai, S. Mulva, and J. Choi. 2014. “The 10-10 performance assessment campaign: New theories regarding the benchmarking of capital project performance.” In Proc., Construction Research Congress 2014: Construction in a Global Network, 2335–2344. Reston, VA: ASCE. Leicht, R. M., S. T. Hunter, C. Saluja, and J. I. Messner. 2010. “Implementing observational research methods to study team performance in construction management.” J. Constr. Eng. Manage. 136 (1): 76–86. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000080. Marhefka, S., E. Lockhart, and D. Turner. 2020. “Achieve research continuity during social distancing by rapidly implementing individual and group videoconferencing with participants: Key considerations, best practices, and protocols.” AIDS Behav. 24 (7): 1983–1989. https://doi.org/10.1007/s10461-020-02837-x. Opara, V., S. Spangsdorf, and M. K. Ryan. 2021. “Reflecting on the use of Google Docs for online interviews: Innovation in qualitative data collection.” Qual. Res. (Sep): 1–18. https://doi.org/10.1177/14687941211045192. Ryan, D., N. Brown, and J. Middleton. 2008. “Wikiplanning: The virtual design charrette.” WIT Trans. Ecol. Environ. 117 (Sep): 517–526. https://doi.org/10.2495/SC080491. Shrestha, P. P., J. Batista, and R. Maharjan. 2016. “Impediments in using design-build or construction management-at-risk delivery methods for water and wastewater projects.” In Proc., Construction Research Congress 2016, 380–387. Reston, VA: ASCE. Sullivan, J. J., M. El Asmar, and K. T. Sullivan. 2019. “Consensus-building workshops to uncover new market entry decision factors for the sheet metal engineering and construction industry.” J. Manage. Eng. 35 (2): 04019001. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000679. Wang, P., P. Wu, J. Wang, H.-L. Chi, and X. Wang. 2018. “A critical review of the use of virtual reality in construction engineering education and training.” Int. J. Environ. Res. Public Health 15 (6): 1204. https://doi.org/10.3390/ijerph15061204. Whittington, D. A., and G. E. Whittington. 2009. “Development of the STAR tool for the management of shutdown/turnaround/outage projects.” In Proc., Construction Research Congress 2009: Building a Sustainable Future, 685–694. Reston, VA: ASCE. Yussef, A., M. El Asmar, D. Ramsey, and G. E. Gibson Jr. 2017. “Front end engineering design for large industrial projects: Industry perceptions and state of practice.” In Proc., 15th Engineering Project Organization Conf. Boulder, CO: Engineering Project Organization Society.



Source link

Leave a Reply

Your email address will not be published. Required fields are marked *