HCA 603 - Final Evaluation Report

Appendix F

Evaluation Plan for Advanced Practice Providers in Queens, New York

{New York State Department of Health Medicaid Services}

{Allied Professional Efficiency In the Health Care Setting}

Prepared by:

{Thomas Saw Aung}

{Queens Medical Professional Association}

{2019}

1. Introduction and Stakeholder Engagement

Introduction

The United States (US) has not always faced healthcare shortages since the founding of the nation. In the earlier years of the nation, healthcare was a diverse field of different philosophies and beliefs each with numerous followers. There were large and active branches of allopathy, homeopathy, osteopathy, naturopathy as well as additional subgroups and ideals. The effectiveness of the different branches of medicine was constantly shifting and difficult to verify owing to the multifactorial nature of health. However with the rise of centralize government during the tidal wave of regulation in the progressive era, medical products and treatments came under increasing scrutiny. In 1910 the Carnegie Foundation for the Advancement of Education issued the Flexner Report that noted the US had “an enormous over-production of uneducated and ill trained medical practitioners” due to the “existence of a very large number of commercial schools.”^[1] The report highlighted the start of a new era of standardized modern medicine that decreased the number of medical schools and limited the pool of physicians.

The decrease in the physician pool did not go by unnoticed. In 1924 the Rockefeller Foundation’s General Education Board responded to the allegations that “there are too few doctors in the country as a whole, in consequence of the fact that too few are being produced by the medical schools under the current régime of high entrance requirements and prolonged course of study” in a report by Lewis Mayers and Leonard Harrison titled, “The Distributions of Physicians in the United States.”^[2] The report stated that the primary reason for the rural physician shortage are the inherent advantages of the city and towns versus the rural areas, not because the higher medical standards. Based on these findings, the report concludes that the neither reduction of medical school requirements nor the creation of a sub-standard class of physicians for these areas is quite impracticable.

Yet decades later we are still faced with the issue of primary care shortages. In 1993 the Health Resources and Services Administration (HRSA) of the Public Health Service published a report titled, “Health Personnel in the United States 1993: Ninth Report to Congress 1993” that detailed the “dearth of adequately trained primary care personnel.” More than a decade later, Bridget M. Kuehn continues the cry in an article in the Journal of the American Medical Association (JAMA). She says that researchers from the University of Missouri estimate a shortage of possibly more than 40,000 primary care physicians by 2025. However, the growth of primary care doctors is only predicted to increase by 7%, not enough to cover the gap.

Why is the lack of primary care health services critical more than ever? The growth of managed care and its accessory policies such as value-based payments rely heavily on preventative and primary care services. These services require repeat visits as well as organization of health information at an unprecedented scale. The data collected can be used to decipher the overall health of the person and maintain a general level of good health. The data can also be used to gauge the health issue at the primary level before sending the patient off to specialists. However, many patients are now opting to skip the primary care doctor all together and go straight to the specialists. This flow of healthcare is rapidly evolving with patients gaining empowerment thanks to the Internet. Unfortunately, going to specialists that focus different aspects of the body often results in the big picture of overall health being lost. What would a ear, nose and throat doctor (ENT) know about the patient’s urinary tract? What would a cardiologist know about the patient’s digestive system? Primary care health services are the core foundation of healthcare as they tie all the health facts together into a cohesive package.

Years of attempting to increase the number of primary care doctors have proven fruitless as the issue continues to grab headlines in the recent years. Continuing issues of doctor concentration in towns and cities as well as poor financial incentive have generated little momentum for primary care physicians. This continued struggle for staffing has some policy makers eyeing the use of advanced practice providers (APPs). Though more limited in scope and sometimes deem “mid level,” these clinicians are more than capable of filling the primary care role. However, the rise of these advanced practice providers appears to be fly in the face of the Flexner report that says the creation of a sub-standard class of physician is impracticable.  Indeed a number of journals in Medical Economics highlight the uncertainty of the role non-physician clinicians and nonclinical. Charolette Huff notes in her article, “Solving the nation's primary care shortage: increasing the number of U.S. physicians means tackling many complicated issues on numerous fronts” that “it’s still unknown how adding care managers, social workers, and other non-physicians will impact how many patients a practice can treat” (Huff, 2016). To tackle the large complicated issues such as primary care it is better to target the smaller chunks of the problem. We intend to evaluate the possible impact of advanced practice providers for private practices in the New York City (NYC) borough of Queens. By seeing how the advanced practice providers can alleviate both cost and resources, a strong report can be made discussing the expansion of APP in total primary care responsibilities.

Queens Medical Professional Association

Queens Medical Professional Association (QMPA) started amidst the housing crisis of 2008 when Doctor Omnes Amigon and his medical friends decided to tackle the issue of patients without insurance. With the economic crisis leaving more and more patients without a form of employer insurance, many of them tried to pay with cash. These were usually worked out on an ad hoc fashion with different doctors, but the flood of uninsured patients that came in post 2008 once again highlighted the issue. Amigon and his friends decided to team up with other health professionals and support staff to provide a standardize model of care for uninsured patients. Thus, the Queens Medical Professional Association was formed. At the current state, QMPA is an established agency with over 5 years of experience. However, most of the association is a loose network of medical professions with a decentralized structure. Often separate doctors from the association will take charge of their own “pockets” of healthcare.

Evaluation of Advanced Practice Providers in the Queens Medical Professional Association

Findings from the Queens evaluation will be submitted to all participating private practices, multiple medical journals, and the state health department. The participating private practices will be given access to the final report so that they can use it for quality improvement measures. Efforts will be made to see if APP utilization can be effectively implemented without incurring too much cost to clinical revenue. Medical journals more interested in the finance aspect of healthcare such as Medical Economics would be keen on figuring out the opportunity costs of APPs. At the same time it would also invite other practices with APPs to write about their situation to provide a clearer image of the roles, responsibilities, and opportunities of APPs. The state health department would also be concerned about possible limitations to physician assistants that would decrease their effectiveness as primary care providers. This report as well as others can be used to foster discussion about APPs in the primary care capacity for health professional shortage areas (HPSAs).

This local evaluation of primary care practices in Queens is only part of the larger general study towards value based service and preventative health. Since the 1990s, quality improvement and accountability measures in healthcare have been on the rise. Expansions of government provided healthcare (Medicaid and Medicare) as well as incentive programs (Obamacare marketplace) has sped up the calls for more transparency into how healthcare dollars are being spent. The US itself has a problem of healthcare costs spiraling out of control with our prices higher than other nations.^[3] How does this rise in focus on healthcare outcomes relate to APPs and preventative health? Since the ultimate desire is to lower the high US cost of healthcare, a particular focus is being placed on preventative care and healthcare access to decrease possibility of more severe health conditions later on down the line. Preventative health is ultimately pushed as a method of decreasing tertiary healthcare. APPs are connected to the rise of primary care as they are seen as the cheaper alternative to medical physicians with shorter training time. Thus, the APPs supply is able to help supplement the physician shortage and expand available preventative health services.

Evaluation Purpose

·      Purpose of this evaluation:
To determine the efficiency or inefficiency of employing advanced practice providers in primary care clinics accepting Medicaid.

·      Findings from the evaluation to be used:
To see the possible improvements to healthcare workflows that can be made to better the healthcare care experience as well as decrease cost.

·      Evaluation of Advance Practice Providers Paves Way for Patient Centered Care:
The employment of secondary or allied health professionals continues to grow, as the primary care shortage remains ever widening. Increases in efficiency of primary care practices will help to offset the growing healthcare deficit as well as allow for a greater management of preventative health services for patients.

Of vital importance to any evaluation are the stakeholders involved in an evaluation & initiative. These stakeholder groups bring with them their inherent biases. Therefore it is necessary to list the possible stakeholders involved, their specific interest or perspective, and their roles in the evaluation to determine if there is possible role mismatch that increases error.

Stakeholders

·      What individuals and groups have an interest in the outcomes of this evaluation? Examples include program participants, staff, and critics.
Program Participants: Primary care health practices, hospitals, group practices.
Staff: Physicians, Advanced Practice Providers, management/administrative staff, allied health staff, 3^rd party consultants, government health programs.
Critics: Healthcare professionals, government health programs.

·      What perspective of the evaluation are they most interested in? For example, are they interested in the evaluation from a cost angle, effectiveness of the program, a critic, etc.?
Interested about cost and effectiveness of allied health professionals. These will impact long-term returns in both health access and insurance costs.

·      What role will they play in developing or implementing this evaluation plan? Examples include serving on planning team or as external reviewer, collecting data, interpreting findings, receiving results.
Physicians and other health professionals will participate in the gathering of measurements. Office and administrative managers will serve to collect data on site, while 3^rd party consultants evaluate collected data offsite. The consulting company will also have some physicians to help interpret health outcomes. Government health programs and the general public will be notified about the first round of evaluation plans and when the entire evaluation is over.

Local physicians and community health centers will be contacted to determine how patients will respond to the introduction of APPs. A survey will also be distributed to patients after their experience to determine their own thoughts. Particular efforts will be made to determine the optimal combination of languages that could be catered to. If the APPs do not have the required language skills, a translator can be brought in if needed. A third party consultation group specializing in cultural sensitivity will also be involved in the process. Different professionals and groups will meet regularly once a month.

Table F.1. Stakeholder Assessment and Engagement Plan (* indicates member of planning team)

Stakeholder Name	Stakeholder Category	Interest or Perspective	Role in the Evaluation
{May be an individual or a group}	{primary, secondary, tertiary}	{program participant, staff, etc.}	{planning team, external reviewer, etc.}
Physicians	Primary	Program participant	Reviewer, Provide input
Allied Health Professionals	Primary	Program participant	Provide data input
Office Managers/Administration	Secondary	Program reviewer, planner	Planning team, reviewer
3rd Part Consultant in Healthcare	Secondary	Program creator	Planning team, external reviewer
Government health programs	Tertiary	Indirect involvement	Awaits results.
General Public	Tertiary	Indirect involvement	Awaits results.

Cultural Competence

·      Engaging with Diverse Stakeholders:
Operate with bilingual programs in mind as well as cultural consulting. Demographics of the areas surrounding the individual practices will be also taken into account.

·      Incorporate various perspectives:
Various perspectives will be gathered through different practices each with their own assortment of staff and health professionals.

·      Accounting for Cultural Context:
Consultants assigned to minority or varied ethnic groups will be screened for cultural sensitivity prior to evaluation. Studies about specific cultures and their approaches towards healthcare will also bed consulted. Office managers and administration will delegate data collection to admin on site so as to avoid difficult cultural transitions.

2. Description of Advanced Practice Providers

Need

·      Program’s Original Issue and Answer:
Dealing with the primary care shortage in America will require support from advanced practice providers (APPs). The ultimate question is whether or not the APPs can provide just as good service as physicians even with lower cost. An evaluation into their cost and work productivity is needed to see if need they can help increase patient load and maintain good health outcomes at an affordable cost.

Context

·      Program’s Context Within the Present:
Program is testing the hierarchy of medicine with continued intrusion of APPs such as physician assistants and nurse practitioners into the field of primary care medicine. This may lead to clashes with traditional ideals of physician-dominated healthcare provision. Patients may also provide some significant push back when hearing that “mid-level providers” or “non-doctors” will be treating them. Another factor to account for is the race and gender of the advance practice providers as patients within a practice may feel more comfortable discussing their health issues with someone of their own culture.

Population Addressed

·      Population Involved:
Activities are intended for the general population of Queens, but especially for underserved health areas. In particular the health professional shortage areas (HPSA) as indicated by the Health Resources & Services Administration.

Stage of Development

·      Program Duration:
Program is planning on being implemented for a period of 3-4 years.

·      Stage of Development:
The program is currently in its planning stage so as to make sure the comparison studies and analysis will be up to par for the task. Finding a diverse enough pool of candidates of advanced practice providers also remains a difficult issue.

Resources/Inputs

·      Program Resources:
Using Queens Medical Professional Association (QMPA) monetary funds to guide the organization into the next level of healthcare provision. The organization will have the staff in place with interconnected network of healthcare professionals, all close friends. Money will come from the association’s group fund for medical experiments, research, and health activities. Allied health staff that is not advanced practice providers will be asked to cooperate with the new healthcare transition. All of the practices involved with have electronic medical records (EMR) with evaluation reporting functions. A third party technology firm appointed and approved by the EMR company will be on hand to deal with any immediate EMR issues. Insurance companies themselves will contribute resources through their periodic remittances as well as healthcare metric notices.

Activities

·      Program Activities:
Review of charts and patient health outcome analysis through weekly onsite visits with office managers. A randomized assortment of 500 patients out of every pool of 1000 patients will be taken for analysis. The health outcomes in particular are tricky and will be mostly focused on readmissions for acute health issues, as chronic health diseases are more difficult to display improvements for. However, common chronic health conditions such as hypertension and diabetes can be evaluated to a certain extent with specific measurements (blood pressure and blood glucose level, respectively) and medication dosage intake.

Besides the chart reviews, there will also be monthly meet ups with all practice staff from the 11 locations at the main headquarters of QMPA. During these large main meetings, the goal is to have a primary and secondary presentation discussing the main issues spotted in the recent weekly onsite visits. The primary presentation will usually focus on workflows and the top critical metric of the month. The secondary presentation will focus on the various other factors affecting the process such as cultural reception and EMR issues. After the presentations comes the main attraction of the night, the cultural potluck dinner where members play games and catch up with friends.

Outputs

·      Production:
The activities will produce multiple reports regarding the performance of advanced practice providers. First will be the report focusing on EMR recorded data: wait times, number of services billed, number of readmissions or complications. This general EMR summary report will focus on general frequencies and demographic socioeconomic structure. The second report will focus on office workflows and APP integration into the practice. Workflows will require a visual representation to make the connections clear. APP integration will require both APP and physician input to determine if the APPs are completely utilized. The third report will focus on accessory healthcare metrics such as referrals, consults, and follow-up. Both the frequency and efficiency of these metrics will be analyzed.

Aside from the reports, the new arrangement in workflow and staff is expected to produce a greater quantity and quality in care. Greater in quantity thanks to the APP’s ability to deal with minor and base level problems. Greater in quality thanks to the better delegation of doctor expertise for unusual or more severe issues. These expectations may fall short though, so it is always important to have a backup plan in the case of patient overload at a practice such as patient transfers to other practices or using local hospitals.

Outcomes

·      Intended Outcomes:
The program’s intended outcome is to improve primary care practices by allowing for more time between patients for the physicians as well as faster services for the patient. The short term expected outcomes are increases in patient intake, health services, availability, and health professional diversity. The intermediate or medium expected outcomes are better patient health outcomes as a result of improved delegation of duties and services. There are linked to streamlined workflows and better follow up on reimbursement and insurance claims.

·      Long Term Outcomes:
The long term goals of the program are the removal of HPSA designations from select neighborhoods, decrease in hospitalizations reported, and improved healthcare transitions between primary, secondary, and tertiary institutions. All together, the work of APPs in addition to excellent primary care coordination are expected to improve health statistics in an entire neighborhood, not overnight, but perhaps through years, maybe decades of work.

·      Transition between Activities and Outcomes:
Many, many reports and projects designed to test workflows and to determine the effectiveness of APPs. Expect a lot of back and forth month to month with new projects being implemented and old projects being evaluated.

Table F.2. Program Description Template

Resources/Inputs	Activities		Outputs	Outcomes
	Initial	Subsequent		Short-Term/Intermediate	Long-Term
Physicians	Chart review	Demographic chart analysis	Reports	Increase employment of PA & NP	Increase health outcomes in the area
Physician assistants	Time review	Time assessment	Summaries	Increase patient intake	Decrease hospitalizations
Medical assistants	Health outcome summaries	Analysis of report differences	Workflow assessments	Increase availability of care	Improve patient documenting consistency
Associated health professionals	Monthly meet ups	Monthly minutes	Health clinics	Increase number of health clinics	Smoother primary care transition to other services
Patient	Patient survey	Survey analysis	Health professional positions	Increase culture variety of health services	Better recovery from accidental trauma
Nurse practitioner	Billing claims report	Claims investigation	Summaries	Decrease patient wait times	Increase health utilization and decrease hospitalizations
Office manager	Report compilation
Computer EHR, software	Records of patients charts	Analysis of changes to metrics	Report summaries

Logic Model

·      Present in the back of this template report.

3. Evaluation Design

Stakeholder Needs

·      Who will use the evaluation findings?
The evaluation findings will be used by the office managers of the 11 clinics, physicians, medical journals, state and city government officials, and associations such as the American Medical Association and the American Academy of Physician Assistants.

·      What do they need to learn from the evaluation?
The organizations should learn about the effectiveness of APPs in providing and supplementing primary care services within an area. Specific details of various reports should care to the variety of individuals interested in the project (eg Medical Economics journal interested in billing services and cost analysis versus government focus on HPSA relief in certain neighborhoods).

·      What do intended users view as credible information?
The reports focused on billing and insurance remittances as those reports will have solid numbers to compare with other utilization reports. Another focus, especially for those interested in medical office management, are organizational workflows involving APPs. The rest of the reports involved with patient outcomes would interest healthcare providers.

·      How will the findings be used?
The findings of the reports will be used by the 11 practices to improve healthcare outcomes and efficiency. While this may sound rather bland owing to the nature of the issue, large-scale projects in the primary care setting are quite rare already. The reports will provide valuable insight into describing how the different practices adapted to APPs and how various workflow changes can improve performance. The reports will also be submitted to others for similar effect.

·      What evaluation capacity will need to be built to engage these stakeholders throughout the evaluation?
The most important evaluation capacity that needs to be built is flexibility to changing office arrangements and developing procedures to recognizing and addressing problems. Often time practices develop a set of procedures and become locked into them as a result of repetitive reliance. However, there remains room for improvement in all practices, whether that is through implementation of a new technology of software, changing billing services, and other such improvements. Another vital feature for practices to make sure that evaluations are being fully put to use, is a clear procedural layout. Without a set outline for changes and implementation, things are bound to go awry. 

Evaluation Questions

·      What five major questions do you intend to answer through this evaluation?

                       1.         Are advance practice providers effective in providing health outcomes equal to physicians?

                       2.         Are advance practice providers as cost effective or better than physicians?

                       3.         Does an increase in advance practice providers reduce the burden on primary care physicians?

                       4.         Do advanced practice providers ease a patient integration into a patient centered healthcare network?

                       5.         Does the expansion of healthcare to lower requirement professionals enable ethnic groups to have greater access to healthcare?

·      Do the questions align with the Good Evaluation Questions Checklist? (http://www.cdc.gov/asthma/program_eval/AssessingEvaluationQuestionChecklist.pdf.)

Table F.3 Good Evaluation Questions Checklist

Does the evaluation question meet this criterion?	YES	NO	Does not meet criterion but merits inclusion because…
1. Stakeholder engagement
A. Diverse stakeholders, including those who can act on evaluation findings and those who will be affected by such actions (e.g., clients, staff), were engaged in developing the question.	1,2,3,4,5
B. The stakeholders are committed to answering the question through an evaluation process and using the results.	1,2,3,4,5
2. Appropriate fit
A. The question is congruent with the program’s theory of change.	1,2,3,4,5
B. The question can be explicitly linked to program goals and objectives.	1,2,3,4,5
C. The program’s values are reflected in the question.	1,2,3,4,5
D. The question is appropriate for the program’s stage of development.	1,2,3,4	5	Long term goals of removing HPSA designation should be considered a valid goal.
3. Relevance
A. The question clearly reflects the stated purpose of the evaluation.	1,2,3,4,5
B. Answering the question will provide information that will be useful to at least one stakeholder.	1,2,3,4,5
C. Evaluation is the best way to answer this question, rather than some other (non-evaluative) process.	1,2,3,4
4. Feasibility
A. It is possible to obtain an answer to the question ethically and respectfully. Unless an acceptable option can be found, eliminate this question.	1,2,3,4,5
B. Information to answer the question can be obtained with a level of accuracy acceptable to the stakeholders.	1,2,3,5	4	At best surveys and frequency of participation would be our metrics. Does not definitively prove the case though.
C. Sufficient resources, including staff, money, expertise, and time can be allocated to answer the question.	1,2,3,4,5
D. The question will provide enough information to be worth the effort required to answer it.	1,2,3,4,5
E. The question can be answered in a timely manner, i.e., before any decisions potentially influenced by the information will be made.	1,2,3	4,5	Both questions may require significant comparison and time to fully prove and process.
5. In sum…
A. This question, in combination with the other questions proposed for this evaluation, provides a complete (enough) picture of the program.	1,2,3,4,5
B. The question, in combination with the other questions proposed for this evaluation, provides enough information for stakeholders.	1,2,3,4,5

Evaluation Design

·      Evaluation Design: Utilization-Focused Evaluation

·      What is the rationale for using this design?
We are primarily focused on how the APPs stack up to Physicians with far-reaching social changes as a more secondary goal. As a result, the measures we are collecting are specific (such as billing utilization by electronic health records). The specific data inputs make the comparisons of utility easer. An alternative was the rainbow approach, but that was deem too board and not specific enough to clearly demonstrate APP capabilities.

4. Gather Credible Evidence

Data Collection Methods

·      Will new data be collected to answer the evaluation questions and/or will secondary data be used? Can you use data from the performance measurement system?
New data will be collected from 11 different private medical practices and compared to past metrics and other secondary sources.

·      What methods will you use to collect or acquire the data?
Data will be compiled and collected via the electron health record software of choice. The requirement is that the EHR program be complaint with Meaningful Use and Patient Center Medical Home standards. These standards require EHR programs to have self-reporting, data compilation, and interoperability to be eligible for government incentive. Thus, the weekly onsite visits will consist of producing reports from the EHRs at each practice. An additional side discussion with office managers will be done to make sure the data checks out.

·      Will you use a sample? If so, how will you select it?
The samples in this case are the 11 private practices in Queens. In each of the practices, two new APPs will be employed to see if there are any improvements to healthcare with APP introduction. The demographic of the population being treated shifts depending on the location of the practice.

·      How will you identify or create your data collection instruments?
Data collection instruments are selected via EHR software. Other software will be used to create visualizations and presentations for easier access. Analysis of the frequency and numerical data will be done with R Studio software.

·      How will you test instruments for readability, reliability, validity, and cultural appropriateness?
The EHR software will be tested for reliability and validity with sample size testing. A portion of the charts from the data collected will be sample to see if they are correctly in line with the patient’s current health status. On the issue of cultural appropriateness, the EHR can categorize patients into gender, race, and ethnicity, but that is not the main focus of the study.

·      How will you determine the quality and utility of existing data?
The quality of the existing data can be determined by specific follow up visits with those specific patients selected from the total population of patients. Documents from the chart will be compared to the current patients health to determine if the charting matches the actual condition of the patient. Once the patient data has been unidentified, its utility in statistical analysis will be vastly improved, as it will be easier to manipulate.

·      From whom or from what will you collect data (source of data)?

The data about the APPs introduced as well as their patients will be collected via EHR and complied into weekly reports.

Table F.4: Evaluation Questions and Associated Data Collection Methods

Evaluation Question	Data Collection Method	Source of Data
1. Are advance practice providers effective in providing health outcomes equal to physicians?	Aggregate collection of EHR reappointments and complications	EHR of Private Practice
2. Are advance practice providers as cost effective or better than physicians?	Insurance Remittance, EHR account inquiry summaries	Private insurance, government insurance (both state and federal)
3. Does an increase in advance practice providers reduce the burden on primary care physicians?	EHR appointment times, EHR chart completion time, Number of accessory health functions completed	EHR of Private Practice
4. Do advanced practice providers ease a patient’s integration into a patient centered healthcare network?	Insurance service utilization report, Patient survey, patient participation rate in patient center medical home	Insurances, patient surveys, patient center medical home reports
5. Does the expansion of healthcare to lower requirement professionals enable ethnic groups to have greater access to healthcare?	Demographic analysis with Department of Health and Mental Hygiene, Bureau of Labor Statistics, healthcare association statistics	NYC DOHMH, Bureau of Labor Statistics, Associations such as AAPA and AMA.

5.     Data Analysis and Interpretation

Indicators and Standards

·      Measurable or Observable Elements of Program Performance:

v Patient Waiting Times

v Billing Codes for PA/NP services – Frequency + Rate

v Billing Costs of Services given

v Number of readmission/complications, electronic referrals, consultation visits

Table F.5: Rubric Measure of Quality – Checklist

Measures	Specific	Measurable	Timely	Achievable	Relevant
Patient Waiting Times	✔	✔	✔	✔	✔
Billing codes for PA services (Frequency, rate)	✔	✔	✔	✔	✔
Billing Cost of services	✔	✔			✔
Number of readmissions for previous complaints	✔	✔			✔
Number of electronic referrals	✔	✔	✔	✔	✔
Number of follow up or consultation visits	✔	✔	✔		✔

Table F.6: Rubric Measure of Quality – Specific, Measurable, Timely, Achievable, Relevant

Measures	Specific	Measurable	Timely	Achievable	Relevant
Patient Waiting Times	Minutes per visit	Time duration calculations by EHR	Easy to acquire through EHR	Reduction of patient waiting times	Decreased waiting times allows for more patient visits
Billing codes for PA services (Frequency, rate)	Frequency of specific CPT codes	Codes tracked by both EHR and Insurances	Easy to acquire from EHR and insurance	Easy to achieve specific	PA services related to specific CPTs
Billing Cost of services	Insurance Price linked to CPT code	Costs linked to insurance – vary depending on specific insurances	Task will be difficult due to maze of pricing via different insurance	The cost comparison will be difficult owing to insurance	Cost is a huge factor of promoting APPs.
Number of readmissions for previous complaints	Patient visits for same complaints	Readmissions for specific complaints tracked	Will be difficult to track readmissions at different healthcare sites	Tracking down admissions for patients at different sites will be difficult	Necessary to answer APP vs MD quality
Number of electronic referrals	EHR logs of referrals	Referrals tracked for specialists visits	Same procedure done as usual	Referrals easy to track and provide	Less referrals for preventative health
Number of follow up or consultation visits	EHR logs of follow ups/consults	Consultations and follow ups tracked	Same procedures done as usual	Easy to track using EHR	Follow ups necessary aspect of preventative health

·      What constitutes “success”? That is, to what standards will you compare your evaluation findings?
Standards of success for Advanced Practice Provider Efficiency and Cost will be the comparison between the original control group and the new experimental group of with APPs. The control group, prior practices without the APPs, and results from secondary sources and literature will be compared against the experimental group. Success would be categorized as a significant difference (greater than 10%) between the control and experimental group.

Table F.7: Indicators and Success

Evaluation Question	Criteria or Indicator	Standards (What Constitutes “Success”?)
1. Are advance practice providers effective in providing health outcomes equal to physicians?	Number of readmissions for previous complaint or complications post operation.	Difference between control and experimental > 10%
2. Are advance practice providers as cost effective or better than physicians?	Monetary value of billed services via CPT codes.	Difference between control and experimental > 10%
3. Does an increase in advance practice providers reduce the burden on primary care physicians?	Number of electronic referrals, follow up visits, and consultation visits	Difference between control and experimental > 10%
4. Do advanced practice providers ease a patient’s integration into a patient centered healthcare network?	Patient waiting times, number of referrals to other specialists	Difference between control and experimental > 10%
5. Does the expansion of healthcare to lower requirement professionals enable ethnic groups to have greater access to healthcare?	Frequency of various billing services, number of new ethnic patients, HPSA designation	Difference between control and experimental > 10%

Analysis

·      Data Analysis: Inferential Statistics

·      Data Security: Data is stored on non-network connected computers that will share files via passcode encrypted hard drive. All devices are equipped with Business Antivirus that prevent auto run and boot log programs from running.

Interpretation

·      Who will you involve in drawing, interpreting, and justifying conclusions?  Does this group include program participants or others affected by the program?
All health professionals at every tier will be involved in drawing, interpreting, and justifying their conclusions. They will be working with a group of 3^rd party consultants who will try to make the study as impartial as possible. For the administrative touch, the office managers from each of the 11 sites will also be asked to participate as they can clearly see the problems on the ground. Unfortunately, patients will not be participating in the interpretation of data even the ones that were involved in the surveys.

·      What are your plans, including evaluation capacity building activities, to involve them in this process?
All the above participants will be involved through the monthly meeting as well as the annual report summary compilation.

6.     Use and Communication of Evaluation Findings

Use

·      How will evaluation findings be used? By whom?
The evaluation will be used by QMPA as a tester for adding more APPs to primary care clinics. APPs have to first be proven before the rest of the primary care practices in the organizations start adapting them wholesale. In addition, other possible workflow issues can also be hashed out during this period.

Other medical journals and practices interested in the addition of APPs to the clinic will use the findings to discuss the range of possibilities. Some may even debate against the use of APPs in the primary care setting for a variety of reasons. Ultimately, the research will at least start a conversation to help push APPs to into the public spotlight.

·      Who is responsible for creating and monitoring an action plan to guide the implementation of evaluation recommendations?  What follow up is needed?
The office managers and administrative support staff with physicians as co-conspirators. Many of the issues surrounding the evaluation recommendations needs to be push through at the administrative level so as to have any real change on the primary care office. Health professional staff can provide some help by supporting changes and projects, but expected to focus on their healthcare side of the issue (one that often pushes them to their limit of endurance). Administration should bear the big stick to advocate changes in policy as well as follow up audits and reports to make sure that everyone is doing as planned or their best.

·      What lessons learned, including those about evaluation and evaluation capacity building, should be shared? How will they be documented?
Lessons learned about the possible ways to streamline workflow should be shared. What should not be shared are the clinical vulnerabilities and blatant weaknesses. Sharing these runs the risk of patient data being compromised. The road of quality improvements is a long one that needs to be documented with prefilled forms so mistakes are not repeated.

Communication

·      Evaluation Stakeholders Vital to Communications:

v Physicians

v Advanced Practice Provider

v Office Manager/Administrator

v 3^rd Party Consultant

§  Allied Health *(While communications are necessary in preparing them for possible changes, major issues such as billing and evaluation theory should be kept out of the allied health sphere so that they can focus on work).

·      Methods of Communication with Stakeholders:
Weekly Newsletters, Email blasts, Monthly Presentations, face-to-face conversation weekly.

·      Who is best suited to deliver the information?
The program manager as they can talk directly to the office administrative staff at the sites while also contacting the health professional from time to time. Their ability to dip between admin and professions allows them the flexibility to not get too enmeshed within the current workflow.

·      Why are these methods appropriate for the specific evaluation stakeholder audience of interest?
The above methods of writings and face-to-face communication are limited to weekly at most so as to not fill the participants inbox. Health professionals are often extremely pressed for time and will ignore electronic methods of communication (especially if they get daily emails). These methods also help to avoid the large dialogue of group chats that often go awry.

7. Evaluation Management  

Evaluation Team

·      Who will manage and implement this evaluation?
Implementation of the evaluation lies in the hands of office managers, administration, and physicians. The main push will come from the office managers and administrators as they will have the burden of making sure data is prepped for weekly onsite visits.

·      What evaluation skills are needed to successfully conduct this evaluation?
A foundation in statistics and organizational management is needed to successfully complete this evaluation.

·      Have you identified an external reviewer to provide feedback on the evaluation plan?
There will be a third party consultant that serves as the external reviewer.

Table F.8: Roles and Responsibilities of the Evaluation Team Members

Individual	Title or Role	Responsibilities
Physician	Facilitator	Helps make sure that other allied health staff are working along guidelines.
Office manager	Manager	Takes standardized guidelines from main administration and implements them at the practice.
Administrators	Coordinator	Networks among the 11 sites to make sure that all 11 are following the same guidelines.

Data Collection Management

·      What data will be collected?

§  Time duration of office visits

§  Billing code frequency and rate

§  Services cost

§  Visits due to Readmissions OR complications

§  Frequency and rate of Electronic referrals, follow up visits, and consultation visits.

·      What activities are needed to carry out the data collection successfully? When should each of these activities be completed?
Weekly Onsite visits and monthly conferences.

·      Who is responsible for conducting each activity?
Weekly Onsite visits and monthly conferences fall under the purview of main administration. However, some minor input is required from the office managers at the practice.

·      Who will oversee the conduct of the evaluation to assure appropriate implementation?
A third party consultant will oversee conduct of the entire evaluation.

Data Analysis Management and Interpretation

·      How will you ensure the security of the data?
Data will be stored in an offline computer desktop not connected to the main operations. An additional back up will be located on a portable hard drive kept in the main administrator’s office. All the data collected will be done via onsite software transfer to avoid use of cloud or online sources.

·      What data will be analyzed, how, and when?
Data will be analyzed at the end of every week for validity, but the most of the important inferential statistical calculations will be done monthly (roughly in line with the monthly conference meetings).

·      Who is responsible for conducting the analyses?
Analysis of the data will occur via third party consultants experienced with working with HIPAA confidential information and statistics.

·      How will you engage stakeholders in confirming analysis results and interpreting them?
Stakeholders will get to see the analysis results and debate about various issues at the monthly conferences.

Table F.10: Data Analysis Plan

Analysis to Be Performed	Data to Be Analyzed	Person(s) Responsible
5 Number summary data groups	Waiting times, billing codes	Main administration
Normal curve distribution estimation from sample	Number of readmissions or complications,	Office manager
Visual Depictions of data	ALL	Main administation

Communicating and Reporting Management

·      Who are the audiences for reporting the progress made on the evaluation and/or evaluation findings?
Private practice managers, physicians, state and city government officials.

·      What is the purpose of the communications with this audience?
Communicating with private practice managers will help spread the use of APPs on the primary care sector. While physicians can see what they can gain by employing APPs under them. State and city government officials can look to the reports to see if there is possible merit in changing the scope of APPs.

·      What is the most appropriate type of communication method to use with this audience, for this purpose? Who is the most suitable “messenger”?
The most appropriate type of communication with private practices and physicians are direct presentations followed up with a rough outline. This type of communication skips straight to the meat of the issue and allows for rapid utilization. On the other hand, city and state government agencies will be receiving the more polish manual and report produced from the data. The suitable messenger for both occasions are third party consultants and main administrators at QMPA.

·      When will the communication take place?
At the monthly meetings for office managers, administration, physician, and physician support staff. At the end of the evaluation for city and state government.

Table F.11: Communication and Reporting Plan for Private Practice Physicians and Managers

		Audience 1: Private Practice Physicians and Managers
Applicable? (√)X√√	Purpose of Communication		Possible Formats	Possible Messenger	Timing/Dates	Notes
X	Include in decision making about evaluation design/activities
√	Inform about specific upcoming evaluation activities		Newsletters	Administrative coordinator	Weekly
√	Keep informed about progress of the evaluation		Conferences	Administrative coordinator	Monthly
√	Present initial/interim findings		Verbal	Visiting Onsite administrator	Bi-weekly
√	Present complete/final findings		Presentation	Consultant	END
√	Document the evaluation and its findings		Form report	Office Manager	Weekly
√	Document implementation of actions taken because of the evaluation		Report Summary	Office manager	Monthly after END

Adapted from Russ-Eft and Preskill, 2001, pp. 354–357.

Timeline

·      When will planning and administrative tasks occur?
Planning and the administrative groundwork will occur during the summer prior.

·      When will training for data collectors occur?
Training for the data collectors will occur during the 4-month transition from Summer through Fall. The goal is to set up a solid framework of support so that the data collectors can learn from on the job experience as much as possible.

·      When will you pilot test data collection instruments?
The data collect from the 11 practice sites will be tested for validity every week.

·      When will formal data collection, analysis, and interpretation tasks occur?
Formal data analysis will occur on a monthly basis, especially during the

·      When will information dissemination tasks occur?
Information dissemination occurs at every monthly conference with the total summary of data being submitted with the report at the end of the evaluation.

·      Upon mapping all of the above on a single timeline, are there any foreseeable bottlenecks or sequencing issues?
Bottlenecks will occur mostly during the monthly events where everything from event planning to data compilation is done for the big conference. The hope is that preparations will be done on a slow and methodical basis, but knowing human behavior procrastination is expected.

Wrapping Up

• At the end of the evaluation, how will you acknowledge the contributions of planning team members and others who contributed to the successful implementation of the plan?
  
  

• How will you document evaluation lessons learned in the course of implementing the evaluation?
  Lessons learned from various mistakes during the evaluation process will be recorded on to a log summary. The log summary is present to note any issues with data validity as well as possible hiccups being faced by administration.
• How/where will you archive relevant documents, instruments, and data?
  All relevant documents will be archived on to an encrypted USB as well as an offline computer. The EHR equipment maintains the data as long as the patient is active.

___ Evaluation was implemented as planned ___ Changes were made to the plan (describe changes as well as the rationale for changes)

 

Program:   _____Answering the Primary Care Shortage____ Logic Model Situation: Primary care continues to suffer from constant shortages causing deficits to health outcomes. A new generation of allied health professionals and other primary healthcare providers have evolved to answer this shortage.

Inputs		Outputs			Outcomes -- Impact
		Activities	Participation		Short	Medium	Long
- Physicians - Physician Assistant - Medical Assistant - Associated Allied health professionals - Physical Therapist - Nurse Practitioner - Office Manager - Servers - Computers - IT professionals - Electronic Medical Record Software - Field Reports - Insurance claims		- Chart review bundle, 500 per 1000 - Time Review via electronic records - Health outcomes summary - Monthly Meet up - Patient survey - Monthly Billing claims reports	- 11 Primary care practices through Queens throughout the borough - All associated allied health professionals involved in clinic operations - Each clinic catering to a specific ethnic group		- Increase patient intake - Increase in number of available primary care professionals - Increase in number of health clinics - Increase in cultural variety of health services - Decrease in patient wait times	- Improved patient health outcomes - Streamlined workflow increases patient document distribution - Increase in patient insurance claims - Increase in health clinic insurance reimbursement		- Increase in health outcomes in surrounding neighborhoods - Decrease in hospitalizations in the neighborhood - Improved patient documenting consistency allows for smoother primary care to social services transition

Assumptions		External Factors
- Lack of primary care health services hurts both the short and long term health of the population. - Increasing staff and professionals improves health access.		- Socioeconomic status of the neighborhoods surrounding the practices. - Insurance and administrative regulation on reimbursement.

References

Bal, B., & Brenner, L. (2013). Employing a mid-level provider differs from entering a collaborative agreement. Orthopedics Today, 33(9), 24-26.

Bendix, J. (2013). Fix the primary-care shortage without more physicians.(policy: THE BRIDGE BETWEEN POLICY AND HEALTHCARE DELIVERY: The last word). Medical Economics, 90(23), 61.

Boyle, D. (2011). Are you a mid-level provider, a physician extender, or a nurse? Oncology Nursing Forum, 38(5), 497. doi:10.1188/11.ONF.497

Evans, M. (2013). U.S. spending at the top. data: Healthcare prices well above other countries. Modern Healthcare, 43(13), 17.

HRSA report shows primary care shortage continues. (health resources and services administration). (1996). Public Health Reports, 111(1), 2. Retrieved from https://search.proquest.com/docview/230146153

Huff, C. (2016). Solving the nation's primary care shortage: Increasing the number of U.S. physicians means tackling many complicated issues on numerous fronts.(IN DEPTH: Policy). Medical Economics, 93(24), 42.

Kuehn, B. (2008). Reports warn of primary care shortages. Jama, 300(16), 1872-1875. doi:10.1001/jama.300.16.1872

Pearl, R. (2017). U.S. healthcare failing in cost, quality, safety; together we can cure what's ailing our system. Modern Healthcare, 47(20), 27.

United States. (1993). Health personnel in the united states: Ninth report to congress, 1993. (). Rockville, MD: Health Resources and Services Administration. Bureau of Health Professions. Retrieved from https://catalog.hathitrust.org/Record/102340172

Warner, J. H., & Tighe, J. A. (2001). Major problems in the history of american medicine and public health. Boston, Mass.: Wadsworth CENGAGE learning.

---

[1] Primary source obtained from References Warner, J. H., & Tighe, J. A. (2001). Major Problems in the History of American Medicine and Public Health. Boston, Mass.: Wadsworth CENGAGE learning.

[2] Also obtained from Warner & Tighe, 2001, pp. 292-297.

[3] Evans, M. (2013). U.S. spending at the top. data: Healthcare prices well above other countries. Modern Healthcare, 43(13), 17.