Introduction
One of the most important tasks of healthcare facilities throughout the country is to ensure the proper quality of rendered medical services. It can be challenging due to the necessity to cover a variety of factors contributing to the overall outcome of organizations’ programs. In order to do this, it is crucial to evaluate the existing initiatives with the help of different assessment tools. In this case, the evaluation process implies the division on specific stages allowing to receive precise results. The purpose of this report is to present these stages on the example of the Trail Program of the North Star Behavioral Health System facilities. It includes the analysis of interviews with the administrator, description of the program, selection of appropriate evaluation approach and corresponding questions, and the definition of methods and participants.
Summary of the Interview
The conducted interview consisted of a variety of questions allowing to assess the benefits of the facilities’ participation in AOTA School Mental Health Community of Practice (CoP). The first questions were intended to reveal the understanding of the CoP’s goals and the reason for the participation in it (see Appendix A). The answers proved the correspondence of the administrator’s perceptions of the CoP to the objectives presented on the website of the American Occupational Therapy Association (AOTA) (“AOTA communities of practice,” n.d.). In this way, the provided benefits matched the expectations of the North Star Hospital.
Other questions were related to the outcome of the facility’s participation in the AOTA CoP and the degree of satisfaction expressed by participants. As can be seen from the answers, its employees managed to expand their general knowledge, thereby ensuring their professional growth and positive results of the hospital’s programs. Moreover, they established better connections between the organization’s departments contributing to the development of its programs. The only problem that emerged while participating in the CoP was poor time management and the partial inability of specialists to take part in meetings while being busy at work. Nevertheless, the overall experience of employees from the North Star Hospital was valuable in terms of further development of the system’s initiatives.
Information about the Program from the Interview
The conducted interview not only demonstrated the apparent benefits of organizations’ participation in various CoPs but also highlighted the preferences of the facility under consideration regarding the development of programs. Hence, the North Star Hospital’s focus on the provision of healthcare services for children was confirmed by the choice of a CoP oriented on this population group (see Appendix A). It implied specific expectations towards the outcome of programs aimed at patients of school age that can benefit from the exchange of medical information. Even though the institutions involved in the work of the CoP were different from the North Star Hospital since they do not focus on children, the experience was still valuable.
The principal results of this exchange were related to establishing connections between different departments and facilities of the North Star Behavioral Health System. They are essential for the coordination of their work regarding the implementation of the program intended to improve the provision of healthcare services for children in the future (“The Trail Program,” n.d.). In this way, the organization’s participation in the CoP contributed to the realization of the initiatives of the group of facilities and promoted their efficiency. It provides the basis for further consideration of the program with the use of evaluation tools.
Evaluation Approach and its Rationale
The assessment of the North Star Hospital’s program requires its analysis with the help of one of the approaches used for this objective. Therefore, the selection of an appropriate method will define the overall success of the outcome in terms of the accuracy and completeness of the received information. In the case of a healthcare program for children of school age, the best option will be one of the approaches oriented to decisions to be made about it (Fitzpatrick et al., 2011). They aim at revealing problems in existing initiatives of medical facilities and, therefore, will be beneficial in the situation of the North Star Hospital (Fitzpatrick et al., 2011). Hence, their implementation will ensure positive results of the program due to the elimination of various issues.
As for a specific approach, utilization-focused evaluation (UFE) will best suit the organization’s goals. It implies the consideration of primary users for the identification of information needs and, therefore, is oriented on people. The use of UFE will facilitate the decision-making process when discovering issues that should be addressed for increasing the initiative’s efficiency (Fitzpatrick et al., 2011). Moreover, it is useful for informing decisions and the involvement of several stakeholders (Fitzpatrick et al., 2011). Therefore, the rationale for evaluating the North Star Hospital program is the necessity to assess its current success or eliminate problems if there are any. In this way, the focus on stakeholders and opportunities for the solution of emerging problems define the efficiency of this evaluation approach for healthcare programs.
Ten Steps to Review Evaluability
The next stage of preparation for the program assessment with the use of the selected approach is the application of the checklist, determining when to conduct it. This instrument allows analyzing the facility’s situation regarding the implementation of its initiatives and justifies the evaluation on a particular stage of their practical realization (Fitzpatrick et al., 2011). The checklist consists of ten steps with ten questions accordingly, and they should be answered in the specified order to make the correct decision (Fitzpatrick et al., 2011). The first question determined that there is no contractual requirement for evaluation (see Appendix B). However, it does not make the assessment less valuable for the program’s outcome.
The second question showed that the object of evaluation has a tremendous impact on the facility’s operations and their quality, and it was necessary to continue research. The third step was to determine if there is sufficient consensus among stakeholders on the program’s model, and this issue needs further consideration due to the lack of information on the subject. The fourth question proved the consistency of actions with the current program, and it allows moving to other questions. The fifth step was to assess the feasibility of the proposed evaluation in terms of available data and human and fiscal resources. It revealed the correspondence of all circumstances to the needs of researchers.
The sixth question demonstrated the agreement of stakeholders on the intended use of the evaluation, and it was essential because of their support, which contributes to precise results. The seventh consideration was related to the productive use of information by stakeholders, and it appeared to be one of the project’s strengths. However, the decisions will not be made based on their reaction to the evaluation results and, therefore, can be viewed as independent and unbiased, as follows from the eighth question. Moreover, the results will include the provision of dependable information, allowing to realign the program in accordance with revealed issues, as it is stated in the ninth question. The final, tenth step showed that the evaluation is likely to meet acceptable standards of propriety, and it indicates its evaluability.
Identification of Essential Factors
The next consideration that should be included in the evaluation process is a set of factors that influence its results. In the case of UFE, the selected approach for this purpose implies the use of personal factors since they take a central place. Hence, the primary focus in this situation is on stakeholders who can readjust the program in accordance with the outcome of the evaluation. According to Patton, they are individuals or groups of people interested in it and care about the initiative realization (Fitzpatrick et al., 2011). In this way, the approach aims to provide them with extensive information on the current state of affairs, thereby enhancing their collaboration.
Other vital factors in UFE are related to political considerations. They comprise external circumstances that should be considered as well (Fitzpatrick et al., 2011). The survey conducted by Patton was intended to define the most important factors in UFE, and the political situation was the second important after personal aspects (Fitzpatrick et al., 2011). Therefore, it is unreasonable to exclude these issues from the evaluation process.
Evaluation Questions
The selection of an approach allowing to assess the efficiency of the program and determination of essential factors that should be included are followed by the preparation of evaluation questions. They cover the aspects that can influence the overall outcome of the assessed program (see Appendix C). In this case, the principal consideration is consumers’ ability to receive access to all the services provided by the program as well as the correspondence of such services to quality standards. It is followed by the assessment of the impact the program has on the overall efficiency of the examined facility and its sustainable development.
The next subject of the evaluation is the provision of healthcare services and their timeliness and completeness. Another critical issue is the program’s impact on the professional growth of specialists and their experience in current positions. It is essential for determining the role of the facility’s employees in the program and revealing possible problems in the sphere. Since the selected evaluation approach includes the assessment of the political situation, the questions also address the influence of external circumstances on the program’s outcome. At last, the methods that can have a positive impact on the initiative implementation are included.
Type of Program Evaluation Design
In order to conduct the evaluation, it is also important to determine the type of design that combines the techniques for this goal. It will serve as a solid base of evidence allowing to measure the program’s efficiency and to increase the credibility of such a study (Fitzpatrick et al., 2011). The methods that will be used should correspond to the prepared evaluation questions (Fitzpatrick et al., 2011). In this way, they define the program evaluation design and contribute to receiving precise data for assessment.
In view of the above, the most suitable type of evaluation design will be the descriptive method since it allows to process qualitative data received with the help of prepared questions. The rationale for the selection of this method is the scope of issues it is intended to address. They include the experience of stakeholders with the program, the description of types of its users, and the details of the initiative delivery. More specifically, the sub-type of this design is a cross-sectional study as it implies conducting a survey to receive quantitative data to reveal the target group’s attitudes, characteristics, behaviors, and beliefs (Fitzpatrick et al., 2011). In this way, it will be possible to receive extensive information on the program’s current state and reveal problems that should be solved in the future.
Method for Collecting Information
The selected evaluation design and prepared questions given above define the choice of methods for collecting information. In the case of the study, the primary method will be a survey conducted among the program’s principal stakeholders. It will be extremely beneficial for the specified evaluation design due to the fact that it covers a variety of issues instead of focusing on a single aspect of the matter (Fitzpatrick et al., 2011). They are also useful in the case when there is a need to receive information from numerous individuals (Fitzpatrick et al., 2011). Therefore, the survey will be the most important method for collecting information in the study.
It will be complemented by other approaches, which are less informative but still necessary for precision. Thus, the first additional methods will be documents and records that will allow receiving data of the facility’s employees to assess their competency and suitability for the program under consideration (Fitzpatrick et al., 2011). Such data will be related to the personnel’s education, professional experience, salaries, and other essential factors. The second additional method for collecting information will be observation, and it will be beneficial in terms of determining the behavior of the program’s participants and assessing the environment (Fitzpatrick et al., 2011). The combination of the specified methods will allow seeing a clear picture of the initiative’s efficiency and revealing the spheres of possible inefficiencies.
Participants and Sample Size
The next step of preparation for the evaluation of the selected program is the definition of its participants and the sample size. The principal person involved in the process will be the evaluator, and its success will depend on their correct actions and the proper use of methods for collecting and analyzing data. Individuals that will take part in the assessment will mostly relate to the hospital staff since they are responsible for the initiative’s outcome. Other participants will include the patients who will be observed in order to evaluate the program.
As for the sample size, it will reflect the number of respondents in the study. In order to make the evaluation efficient, it is vital to ensure the proper amount of people involved in it. Hence, it should be small enough to ensure precision and big enough to provide extensive information on the topic of research (Fitzpatrick et al., 2011). In the case of the North Star Hospital, the optimal number of participants will be about 30% of all healthcare specialists involved in the program.
Sampling Procedure and Data Analysis Strategy
Since the participants and the sample size are defined, it is vital to apply the sampling method to ensure the selection of appropriate individuals for evaluation. This procedure allows to avoid generalization in the case when researchers need concrete results as in the situation with a specific program that should be assessed (Fitzpatrick et al., 2011). Hence, it will be reasonable to select the sample for assessing the North Star Hospital’s initiative based on provided documentation and staff records. The most experienced employees will be able to provide more extensive information and contribute to the precise results of the evaluation.
The received data will be analyzed with the help of a specific strategy, and its selection also affects the study’s success. This strategy is based on the type of collected information, and, in this case, it will be qualitative data analysis. It implies summarizing findings for each group of stakeholders and analyzing differences between them in order to evaluate the whole program and its progress (Fitzpatrick et al., 2011). Such an approach will be not only beneficial in terms of describing the current state of the initiative but will also allow to reveal the trends in its development and anticipate potential problems.
Brief Description of the Program
Healthcare facilities tend to have long-term plans contributing to the development of specific areas of their work. For this purpose, they design programs addressing pre-defined categories of patients depending on their age, conditions, and other factors. The definition of a program includes all the aspects of such plans implementation. It refers to them as “a set of planned activities” performed with the use of managed resources, focused on specific goals, or addressing particular needs (Fitzpatrick et al., 2011, p. 8). This definition corresponds to the initiative of the North Star Hospital that is a part of the North Star Behavioral Health System known as the Trail Program.
This initiative is intended to provide high-quality healthcare services to children with mental health issues. According to the facility’s website, they encourage “healthy and safe thoughts, feelings, and behaviors,” allowing their patients to overcome specific conditions such as depression, anxiety and other similar states (“The Trail Program,” n.d.). The Trail Program is oriented solely on children and teens aged 4-18, and this fact makes it unique since no other clinics in the area offer services to this category of patients with mental health problems (“The Trail Program,” n.d.). Hence, its evaluation is extremely important due to the lack of analogs.
The primary orientation of the Trail Program is the provision of inpatient acute care for their patients, and it implies the availability of specific services. The facility employs evidence-based practice that includes trauma-focused cognitive behavioral therapy, experiential therapy with activity and recreational based groups, cognitive behavioral therapy for anxiety, depression and anger control, and other methods (“The Trail Program,” n.d.). These approaches are complemented by mild therapy represented by yoga classes and stress management groups (“The Trail Program,” n.d.). Such a variety of options requires particular consideration, and the planned evaluation of the program is the first step on the way to assessing all treatment procedures offered by the North Star Hospital.
The suitability of the selected approach for assessment of the program’s efficiency, utilization-focused evaluation (UFE), is defined by its orientation on people and personal factors. It provides numerous opportunities for the determination of stakeholders’ principal needs and implies the consideration of various groups of participants (Fitzpatrick et al., 2011). Moreover, since it is widely used by healthcare institutions, it will also be beneficial for the North Star Hospital (Fitzpatrick et al., 2011). Another benefit of the use of UFE in order to assess the Trail Program is its additional focus on political circumstances and the external environment (Fitzpatrick et al., 2011). It contributes to the creation of a long-term development plan for the initiatives with the attention to possible complications.
Stakeholders of the Program
The definition of the principal stakeholders of any healthcare program is a vital stage allowing to address their problems in an appropriate way. In the case of the Trail Program of the North Star Hospital, they will be the school children aged 4-18 with mental health problems (“The Trail Program,” n.d.). This category of patients benefits most from the successful implementation of the initiative. The second group of stakeholders is family members of the children eligible for the program. Their interest in maintaining a high quality of provided services is defined by the impact of mental health conditions on the overall life of a family. The third group of stakeholders is healthcare specialists working for the North Star Hospital. Their inclusion is conditional upon the numerous opportunities such programs provide for their professional growth.
The Use of the Logic Model in Evaluation
Another instrument that contributes to the evaluation of healthcare programs is logic modeling. Its use allows to add external factors and thereby make the assessment process more realistic and corresponding to the environment (Ebenso et al., 2019). In the case when researchers use logic models while conducting the evaluation of programs, the objectivity of their results increases (Fitzpatrick et al., 2011). They serve as an extension to already developed plans for a program assessment and mostly focus on a specific issue rather than on the initiative as a whole (Fitzpatrick et al., 2011). In this way, logic models contribute to addressing the revealed problems in an efficient and timely manner and improve the overall results of programs under consideration.
A Logic Model Example for Evaluation
One of the examples of the use of logic models for evaluation of programs is the experience of Nigerian researchers. They applied the concept to the community health workers (CHW) program promoting access to maternity services in order to include contextual factors of the initiative into consideration (Ebenso et al., 2019). In this way, the specialists managed to increase their awareness of current issues and cover greater amounts of data (Ebenso et al., 2019). As a result, the use of the logic model contributed to the clear formulation of the program’s goals and the establishment of relationships among its elements (Ebenso et al., 2019). It allows concluding on the efficiency of logic modeling for evaluation of healthcare programs and justifies the necessity to use it for the assessment of the Trail Program.
Conclusion
The preparation of evaluation of the Trail Program started with the interview with the administrator revealing the facility’s particular focus on the provision of healthcare services for children. It was followed by the selection of an evaluation approach, which is utilization-focused evaluation, and assessment of the program’s evaluability. The successful completion of these two stages led to the need to define essential factors, formulate evaluation questions, choose the study design, method for collecting information, and determine participants and sample size. Thus, the procedure will include the consideration of personal circumstances and the political situation that will be demonstrated through surveys, observations, and review of documentation.
The participants in the process will be evaluators, healthcare specialists working for the North Star Hospital, and their patients. They are also included in the list of stakeholders complemented by their family members. The sample size for the study will be 30% of employees that are most experienced and patients eligible for the program. The received data will be summarized and analyzed for revealing differences in participants’ perceptions. At last, the overall objectivity of evaluation will be tested through logic modeling.
References
AOTA communities of practice (CoPs). (n.d.). American Occupational Therapy Association. 2020. Web.
Ebenso, B., Manzano, A., Uzochukwu, B., Etiaba, E., Huss, R., Ensor, T., Newel, J., Onwujekwe, O., Ezumah, N., Hicks, J., & Mirzoev, T. (2019). Dealing with context in logic model development: Reflections from a realist evaluation of a community health worker programme in Nigeria. Evaluation and Program Planning, 73, 97-110. Web.
Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative approaches and practical guidelines (4th ed.). Pearson.
The Trail Program. (n.d.). North Star Behavioral Health System. 2020. Web.