Skip to main content
Download PDF
Download ePub

Incremental cost and cost-effectiveness of low-dose, high-frequency training in basic emergency obstetric and newborn care as compared to status quo: part of a cluster-randomized training intervention evaluation in Ghana

Globalization and Health volume 13, Article number: 88 (2017) Cite this article

Abstract

Background

Low-dose, high-frequency (LDHF) training is a new approach best practices to improve clinical knowledge, build and retain competency, and transfer skills into practice after training. LDHF training in Ghana is an opportunity to build health workforce capacity in critical areas of maternal and newborn health and translate improved capacity into better health outcomes.

Methods

This study examined the costs of an LDHF training approach for basic emergency obstetric and newborn care and calculates the incremental cost-effectiveness of the LDHF training program for health outcomes of newborn survival, compared to the status quo alternative of no training. The costs of LDHF were compared to costs of traditional workshop-based training per provider trained. Retrospective program cost analysis with activity-based costing was used to measure all resources of the LDHF training program over a 3-year analytic time horizon. Economic costs were estimated from financial records, informant interviews, and regional market prices. Health effects from the program’s impact evaluation were used to model lives saved and disability-adjusted life years (DALYs) averted. Uncertainty analysis included one-way and probabilistic sensitivity analysis to explore incremental cost-effectiveness results when fluctuating key parameters.

Results

For the 40 health facilities included in the evaluation, the total LDHF training cost was $823,134. During the follow-up period after the first LDHF training—1 year at each participating facility—approximately 544 lives were saved. With deterministic calculation, these findings translate to $1497.77 per life saved or $53.07 per DALY averted. Probabilistic sensitivity analysis, with mean incremental cost-effectiveness ratio of $54.79 per DALY averted ($24.42–$107.01), suggests the LDHF training program as compared to no training has 100% probability of being cost-effective above a willingness to pay threshold of $1480, Ghana’s gross national income per capita in 2015.

Conclusion

This study provides insight into the investment of LDHF training and value for money of this approach to training in-service providers on basic emergency obstetric and newborn care. The LDHF training approach should be considered for expansion in Ghana and integrated into existing in-service training programs and health system organizational structures for lower cost and more efficiency at scale.

Background

In the last 2 decades, under-5 mortality rates have declined by nearly 50% and the rate of decline has increased over the last decade, largely due to increases in coverage and access to essential reproductive, maternal, newborn, and child health (RMNCH) services [1]. Despite these successes in under-5 child health, newborn mortality accounts for 45% of under-5 deaths and reductions in neonatal mortality and stillbirth rates have stagnated with significant gaps in relation to targets, especially in sub-Saharan Africa [1,2,3]. Globally, an estimated 2.7 million neonatal deaths and 2.6 million stillbirths occurred in 2015, of which 23% of neonatal deaths and 26% of stillbirths occurred during the critical intrapartum period: onset of labor to within 24 h of birth [2, 4,5,6,7]. To realize Sustainable Development Goal (SDG) 3—good health and well-being—health systems and implementing partners need to address persistent challenges in providing quality care, but also need evidence on how to most effectively improve care, during the critical pregnancy, intrapartum, and postpartum periods [8].

Ghana has sought to increase access to and affordability of key RMNCH services by expanding national health insurance and providing free maternal care [9]. Despite progress toward the Millennium Development Goals and, more recently, SDG targets, Ghana mirrors some regional positive trends in areas of stunting and wasting but still faces significant challenges in improving newborn health outcomes which is reflected in low levels of progress in skilled birth attendance, neonatal mortality, and under-5 mortality [10]. Among under-5 deaths in Ghana, 40% occurred during the neonatal period in 2012 [1]. For every 1000 live births in Ghana, 28 children die within the first 28 days of life, more than twice the SDG target of 12 per 1000 live births [1]. Despite global declines in absolute numbers of stillbirths of 19% from 2000 to 2015, the absolute number of stillbirths within Ghana only reduced 1.8% over the same time period; the country stillbirth rate was 22 per 1000 total births in 2009 [2, 11]. Recognizing these gaps in progress, the Ghana Health Service (GHS) and Ministry of Health are actively pursuing strategies to improve quality of RMNCH care with focus on a better-trained workforce and improved rural access to quality labor, delivery, and newborn health services [12].

Low-dose, high-frequency (LDHF) training

Clinical care quality is strengthened by increased knowledge, acquired skills, and peer consultation; training in many forms has long been utilized to improve clinical performance and, consequently, RMNCH outcomes [13]. The best and most cost-effective methods of training in low-resource settings, however, continue to be tested. Traditionally, training in low-resource settings has assumed a workshop-based approach wherein a subsample of providers are removed from their clinical environment and trained off site for an intensive period of time. Workshop training may improve attendees’ capacity, but knowledge and skills may not be transferred to other co-workers at the facility nor translated into provider practice or performance [14, 15]. Various methods for continuing medical education and in-service training have established strong evidence on best practices for learning: repetition is correlated to higher retention of knowledge and skill, having the same setting for training as clinical practice is correlated with higher gains in skills and performance, simulation-based practice and interactive methods are more effective than didactic lecture [16,17,18,19].

Low-dose, high-frequency (LDHF) is an approach to improve the in-service training model to better enhance the health workforce capacity, sustain health provider gains in knowledge and skills, and help learners translate clinical lessons into practice [20]. The concepts and terminology of on-site short teaching lessons (low-dose) followed by longer practice sessions (high-frequency) have been applied in different ways and settings from maternal and newborn health trainings by Jhpiego in Uganda to pediatric cardiopulmonary resuscitation training in the Children’s Hospital of Philadelphia in the United States [14,15,16, 21]. LDHF educates with small quantities of content, frequent repetition, increased interaction, simulation-based practice, and supervised direct patient care that keeps providers in the facility environment while learning. Jhpiego has combined evidence with extensive training experience to create an LDHF approach that better targets and supports the health workforce with competency-based education rather than solely focusing on knowledge improvement [22]. The LDHF training program in Ghana was designed after a facility quality improvement project for child health, Project Fives Alive!, identified that in aiming to reduce under-5 mortality, an additional focus on improving clinical skills for quality delivery and postpartum services was necessary to reduce the large proportion of under-5 deaths that occur on the day of birth [23]. The LDHF training implemented in Ghana aims to accelerate newborn survival by improving health provider capacity to provide quality institutional delivery and postpartum services at the facility.

Study objectives

This study examines programmatic costs and health effects, calculated by DALYs averted, of LDHF training on institutional stillbirth rates and on newborn survival outcomes of 40 facilities included in the impact evaluation in three regions of Ghana from 2014 to 2017 and compares the incremental cost-effectiveness of LDHF training with the alternative of no training (the status quo).

Methods

Study setting

Of the 24.7 million people that live in Ghana, 21.4% are in the Central, Western, and Upper West regions where the LDHF training program was jointly planned with the Ghana Health Service and Ministry of Health and coordinated so as not to overlap with existing programs [24]. Utilization of key maternal health care services across Ghana has increased, but delivery at a health facility has not seen the same levels of utilization as other services [25]. In 2011, 87% of pregnant women had four or more antenatal care visits and 83% had a postnatal care visit within 2 days of birth, but only 68% of live births had a skilled attendant at delivery, and key health service utilization is lower in rural populations than in urban [1, 9]. The Central and Western regions have larger proportions of their population in urban areas, 47.1% and 42.4%, respectively, than Upper West region, where 16.3% of population is urban [24]. In the study regions, public and mission hospitals were selected as LDHF training sites if they averaged 30 or more births per month and had at least three skilled birth attendants on staff who actively provided labor, birth, and immediate postnatal care at the facility.

Program description

The LDHF training program was adapted from a 2-week classroom-based basic emergency obstetric and newborn care (BEmONC) package to two 4-day low-dose onsite trainings. The LDHF trainings incorporated high-quality onsite instruction with a whole-team perspective, clinical simulation-based practice with models, mentorship and coaching, and a daily lunch. Following the LDHF trainings, content is reinforced through high-frequency mentorship: individual phone calls between regional GHS master mentors (MMs) and trainees, peer practice with medical models in the facility, and text message reminders and quizzes. Nongovernmental organization (NGO) staff contacted the trained MMs, peer practice coordinators, and providers to ensure mentoring and practice were continuing after the LDHF onsite training and also monitored the mobile messaging system to make sure text messages sending appropriately. In contrast to traditional training models, which train a small subpopulation of health workers outside of the clinical setting, in the LDHF model all midwives and nurses providing labor, delivery, and immediate postpartum care at a participating facility have the opportunity to receive the same training.

Program activities are understood through three phases: Development—adaptation of the curriculum from workshop to LDHF training approach; Startup—planning and conducting MM trainings, health information officer trainings, and regional health team meetings; and, finally, Implementation—organizing and conducting LDHF trainings in two main sessions (LDHF1 to cover basic care and LDHF2 for complications) with provider knowledge and skills assessments, and initiating mobile mentoring (mMentoring) and peer practice. Additional details on the program and health effects measurement are found in the Accelerating Newborn Survival project impact evaluation [26]. A full program description with timeframes for each phase is found in Table 1.

Table 1 Program description
Full size table

Cost data

Assuming a 3-year analytic time horizon corresponding to the program’s length, data on economic costs were collected for all program activities based on a review of program financial records, including purchase orders, original receipts kept on record in the finance department, and recurring overhead allocation reports. Cost details were confirmed and supported by interviews with the LDHF implementation team, especially important because many program costs were summarized and reported at the regional rather than the activity level. Personnel salaries were increased per fiscal year based on informant interviews and human resources records, and applied monthly to appropriate phases during which staff were supporting the program. Market prices or regional price averages were used to value the furniture, equipment, and other resources used by LDHF program staff. Operating costs were included for each phase, by time period and value of personnel effort, building space, maintenance and utilities, and furniture and equipment used by the LDHF training program.

Development costs included costs associated with adapting the BEmONC curriculum, including personnel time and editing services to develop the learning resource package of materials: facilitator and learner manuals, slides for instruction, and standardized assessments. Startup costs included all inputs related to regional health team meetings and trainings for health information officers and MMs. The main inputs were transportation, program staff per diem and resource personnel fees, and food and lodging costs associated with the necessary travel. Implementation costs included all inputs related to the LDHF trainings and mMentoring follow-up. These similarly included transportation, personnel, and accommodations costs but also trainee-associated costs of lunch and training materials as well as equipment and expendables costs for simulation-based trainings, including practice models of MamaNatalie™ and NeoNatalie™, medical equipment, examination gloves, face masks, and cleaning supplies.

Key cost model assumptions are as follows: 1) once averaged from multiple data sources or direct from one single and validated source, a cost is stable and representative of further repeated activities at the facility or regional level, 2) key development and startup costs such as curriculum development were allocated across all facilities that received LDHF training, 3) personnel time and salary cost were measured monthly and then averaged based on the corresponding timeline of each LDHF program phase so as to have one personnel cost for each staff member during each phase, 4) expendable training items are costed based on either facilities or region depending on rate of use and reusability as informed by the LDHF clinical training team. The program costs collected represent Jhpiego’s LDHF training program reach to three regions, 40 facilities, and 428 health workers receiving training.

Health effects data

The impact evaluation of the LDHF training intervention on newborn survival measured health effects by tracking institutional newborn mortality within 24 h of birth or until discharge and institutional intrapartum stillbirths; indicators were tracked at the facility with supplemental registers in 1-month increments and pooled as 6-month average rates [26]. Each tracking period—the 6 month baseline prior to LDHF training, 1–6 months after LDHF training, and 7–12 months after LDHF training—included over 30,000 live births from the study sites in the three regions (38,192; 36,160; and 31,498 respectively). Early newborn mortality reduced 48% from baseline during the 1–6 months immediately following the LDHF training, and reduced 56% from baseline in the 7–12 months period [26]. Intrapartum stillbirths reduced 35% from baseline during the 1–6 months immediately following the LDHF training, and reduced 49% from baseline during the 7–12 months period [26]. Data were recorded by the service providers and verified by study staff. Service providers and facility-based health information officers were trained on the use of supplemental registers, new indicators, and data quality. Aside from these trainings, the study team had minimal onsite presence and clinical practice was continued as usual.

Disability-adjusted life years (DALYs) were calculated in this analysis using estimated lives saved, average life expectancy in Ghana, no age weighting, omitting years lived with disability (YLD), and discounting at 3% so as to discount health effects at the same rate as costs [27]. Unlike the impact of chronic illness or cancer which is heavily influenced by years lived with disability (YLD), the majority health effect impact from newborn mortality is derived from years of life lost [28, 29]. The Global Burden of Disease 2016 valuation of DALYs also reflects this difference: neonatal disorders are the leading cause of death under-5 years but only the fourth contributing cause of YLD, the ratio of all age YLD to DALYs remains low throughout sub-Saharan Africa, and only a quarter of the DALY counts for Ghana are estimated to be from YLD [30, 31]. The omission of YLD for this analysis will make the results slightly more conservative as we may be underestimating the full health impact of improving newborn care as related to morbidity, but it will not considerably change the overall findings of cost-effectiveness.

The cost and health effect data were collected from existing records from the implementation of the LDHF training and from the impact evaluation; the cost-effectiveness analysis using this data was determined nonhuman subjects research by the Johns Hopkins Bloomberg School of Public Health Institutional Review Board.

Analysis

The analysis of cost data was conducted in Excel. All costs are presented in 2015 US dollars. All capital costs are discounted at 3% and one-time development and startup activities annualized for the analytic time horizon of 3 years, matching the actual program timeframe [27]. The activities and costs included in this analysis are from the programmatic perspective. Although we assume that personnel and overhead costs would be much lower if the program were adapted, continued, or supported by a local partner or government, we kept the wide range of actual Jhpiego cost so as to get the most accurate result for incremental cost-effectiveness. We did not include societal costs to beneficiaries as the cost of care seeking is separate from the cost of training health care workers to provide quality care; in this manner, costs to beneficiaries would remain the same whether or not LDHF training was provided. Furthermore, the programmatic perspective can be useful to decision-makers that are weighing benefits with costs and striving to build balanced health systems in the midst of ever increasing health expenditure as a share of the economy [32].

To determine the incremental cost-effectiveness of program activities, a simple decision tree was created using TreeAge Pro 2017 software comparing incremental cost and effects of LDHF training to no training [33]. Base case results of costs and health effects were explored with both univariate sensitivity analysis of each cost category and probabilistic sensitivity analysis with sampling from the expected ranges of costs and effects using the Monte Carlo simulation. All programmatic costs were considered for all waves of implementation. High and low values for each activity were established by taking the maximum option for each input and the minimum option for each input. When multiple data points were not available for a particular cost input, the base case value was inflated or deflated by 25%. The health effects high and low were determined from the best case and worst case scenario based on the standard deviation from the base health effects findings of reduced early newborn mortality and intrapartum stillbirth rates. To conduct a probabilistic sensitivity analysis, distributions were applied to base case parameters which significantly affected the results in univariate analysis. Uncertainty in the results was explored through tornado diagrams, cost-effectiveness planes, and cost-effectiveness acceptability curves [27, 34, 35].

Essential LDHF program model and comparison to workshop training

As part of sensitivity analysis, we tested model variations and developed a major set of modifications through discussion with the LDHF program team to explore sustainability and changes in the LDHF program activities if continued or implemented within the Ghana health system. Assumptions were tested together to assess the cost of the essential activities of the LDHF training program implementation in a sustainable scenario that maximizes efficiencies. This Essential LDHF Program scenario was then compared to a traditional BEmONC workshop training hosted by the Ghana College of Nurses and Midwives (GCNM). The GCNM conducted three workshops on BEmONC targeting educational institution tutors and their teaching facility lead clinical preceptors. Two-week-long workshops were conducted in Accra and Kumasi: one week of classroom-based learning and a second week of field site coaching and hands-on experience, both facilitated by GCNM staff and guest trainers. Follow-up was conducted mainly via phone calls to all trained providers; however, one onsite follow-up visit was conducted to create a case study of how the training had influenced the participating providers. Costs were collected for the traditional workshop with the same methods used to collect the LDHF training costs.

Results

The full cost of the LDHF training program for all 40 facilities over the 3-year analytic time horizon in three regions was $823,134. The mean total cost per facility was $20,578 including $114 for development (0.6% of total), $695 for startup (3.4% of total), and $19,769 for implementation (96.1% of total). As an average of 11 health care providers participated at each facility, the total program mean cost per provider trained is $1974.

Personnel (44% of overall cost) and the LDHF trainings held as two separate facility-based visits were the major drivers of the program costs: 20% of overall cost for LDHF1 and 14% for LDHF2 (Table 2). The second LDHF facility-based training was less costly due to shorter duration and use of previously provided medical models for peer practice.

Table 2 Program costs by wave and total program
Full size table

Cost-effectiveness

With deterministic calculation under which all inputs remain constant and non-random, the base case cost-effectiveness of the LDHF training program is $1497.77 per life saved or $53.07 per DALY averted.

Sensitivity analyses

With deterministic calculations for high and low case sensitivity analyses, we established a best case with measures of high effectiveness and lower limit costs. The best case showed a total program cost of $387,808 and a total of 20,313 DALYs averted; this translates to an incremental cost of $19.09 per DALY averted. In the converse, worst-case-scenario measures of low effectiveness and upper limit costs, the total program cost $1,596,116 with a total of 10,707 DALYs averted; this translates to an incremental cost of $149.08 per DALY averted (Table 3). One-way analysis of key parameters shows that the LDHF training program’s effectiveness, personnel cost, and resources needed for facility-based trainings are the parameters which most impact the incremental cost per DALY averted (Fig. 1).

Table 3 Incremental cost effectiveness of LDHF training in 3 regions, with 40 facilities v No Training
Full size table
Fig. 1
figure 1

Tornado diagram: One way sensitivity analysis top five most impactful model components on the overall ICER

Full size image

Probabilistic sensitivity analysis was conducted to sample from within key cost distributions and effects distributions to better evaluate the robustness of the base case findings (Table 4). Under the probabilistic sensitivity analysis, the mean incremental cost-effectiveness ratio of $54.79 per DALY averted (median $50.68, 95% CI $24.42–$107.01) suggests the LDHF training program as compared to no training has 100% probability of being cost-effective above a willingness to pay threshold of $1480, which was Ghana’s gross national income (GNI) per capita in 2015 (Fig. 2). Each output from the Monte Carlo simulation can be seen plotted in the cost-effectiveness quadrant of increased health effect and increased cost and all simulation results fall under the willingness to pay threshold of $1480 (Fig. 3). The GNI or gross domestic product (GDP) per capita— $1442 in Ghana in 2015 — is a standard measure which helps establish a standard threshold by which to assess interventions, though it is not indicative of affordability or funding [36,37,38]. Under the GDP or GNI per capita thresholds, 100% of the probabilistic sensitivity analysis estimates of incremental cost per DALY averted by the LDHF program would be considered highly cost-effective in Ghana.

Table 4 Sensitivity Analysis parameters for one way, multivariate, and probabilistic sensitivity analysis
Full size table
Fig. 2
figure 2

Incremental cost-effectiveness acceptability curve of LDHF training v No training

Full size image
Fig. 3
figure 3

Cost-effectiveness plane of LDHF Training v Status quo in three regions of Ghana valued at the willingness to pay of $1480 GNI per capita in Ghana

Full size image

Essential LDHF program model comparison to workshop training costs

The traditional workshop approach costs $901 per provider compared to $506 per provider trained on average under the Essential LDHF Program Model, which accounts for expected changes in costs and efficiencies for the program implementation (Table 5).

Table 5 Modeled Essential LDHF Program cost per trainee (nurses and midwives) of LDHF rollout in 40 sites over 4 Waves and BEmONC classroom based workshops as implemented by Ghana College of Nurses and Midwives (GCNM)
Full size table

Discussion

LDHF approach components

The initial investments needed to begin a new approach to training—such as adaptation of the standardized BEmONC training content into an LDHF learning package, the MM training, and procurement of teaching equipment—all contribute to each facility and health provider that receives that LDHF training. Though the costs of these activities alone are significant, development and startup costs were, together, less than 4% of total program cost because they are shared across all facilities that use the LDHF approach with MMs to lead the training. The main cost drivers are the resources expended to support program personnel and the inputs for onsite LDHF trainings during the implementation phase. Any changes to personnel structure or salaries have a significant impact on the program’s overall cost. LDHF trainings, both the first training that focuses on basic delivery and the second training that focuses on complications and emergency care, have many constants in terms of cost but are sensitive to changes in number of days and number of support staff.

A common expectation is that moving from hotel-based traditional 1- or 2-week workshops to onsite trainings will be less expensive; however, onsite training is not guaranteed to be less expensive though there are key differences in cost drivers. Cost drivers of LDHF onsite trainings are travel and accommodation of MMs, the clinical training team, and support staff; in contrast, much of the cost for traditional workshops is for travel and accommodation for the training recipients. In Ghana, the LDHF onsite training setting enables capacity-building for the full workforce responsible for maternal and newborn health (MNH). This enhances the base capacity of many providers and the facility as a whole, rather than sole reliance on a single trained colleague who may or may not be present when needed.

The LDHF program and lessons learned directly support Strategy 6, “Improving the capacity of facility-level health workers to address newborn care,” as set forth in the National Newborn Health Strategy and Action Plan 2014–2018 by the Ministry of Health, Ghana [12]. Through this LDHF approach, Jhpiego has identified one approach to optimal delivery of “in-service competency-based training” on essential obstetric and newborn care for midwives in hospitals and Community-Based Health Planning and Services facilities [12]. Project Fives Alive! demonstrated successful scale-up of systems strengthening efforts, engaging over half of Ghana’s districts through key learning and quality improvement collaborations [23]. Future training programs could follow a similar path to scale and strengthen the LDHF approach to training and leverage existing stakeholder partnerships.

Cost-effectiveness of LDHF training as compared to no training

How do we judge the value for money of this new approach? The impact evaluation of the LDHF training showed that this approach led to increased competency, retained knowledge and competency at 1 year, and associated health effects of decreased newborn deaths and stillbirths at the facility [26]. These effects show that this LDHF approach, with onsite training, peer practice, a low-cost mobile messaging system, and mentorship follow-up by phone, was good value for the money expended as measured by the population health impact in Ghana. The LDHF training program saved an estimated 262 lives based on the reductions seen in newborn deaths at the facility in the first 24 h or before discharge; in addition, an estimated 288 lives were saved based on reductions seen in the rate of institutional intrapartum stillbirths. At 28.2 DALYs per death averted, the program in total averted 15,510 DALYs in the base case scenario, at a cost of $53.07 per DALY averted. By using actual NGO cost, matching the analytic time horizon to the project timeframe of 3 years, and not assuming continued health effects for years beyond the project’s end, the base case cost per DALY is conservative. As the impact evaluation established retention of provider competency and continued improvements at 1 year following LDHF training, improvements in lives saved could be assumed beyond the year of data collected, resulting in greater cost-effectiveness in decreased cost per DALY averted.

We tested the model with univariate, multivariate, and probabilistic sensitivity analyses to better understand how our results would change in relation to changes in key parameters. The cost-effectiveness results were mainly affected by changes in effectiveness of the LDHF training program in improving newborn health outcomes, changes to personnel structures or salaries, or changes to LDHF training costs. In the multivariate deterministic calculation of best case and worst case, the cost per DALY averted ranged from $19.09 to $149.08; in the probabilistic sensitivity analysis, the mean cost per DALY was very similar to the base case but the range was slightly smaller, $24.42–$107.01. All scenarios simulated to test incremental cost-effectiveness of LDHF training compared to no training were highly cost-effective when assessed with the threshold of GNI per capita, $1480 in Ghana in 2015 [39].

In a Lancet priority-setting article of 2006, low-cost opportunities for sub-Saharan Africa in MNH care interventions—such as increasing primary care coverage, care quality, and targeted neonatal packages—range from $82 to $409 per DALY averted [40]. A 2016 comprehensive look at returns on investment evaluated the cost per DALY ranges for several interventions designed to decrease maternal and newborn morbidity and mortality—participatory women’s groups, training of midwives or village health workers, and safe motherhood and facility interventions—and found that they ranged from $150 to $1000 per DALY averted [41]. The LDHF training cost per DALY averted at base case falls lower than either of these ranges, indicating it may be preferable to other MNH interventions in achieving population health impact at less cost. However, though per capita measures such as GDP or GNI are widely used to standardize cost-effectiveness evidence, using these measures alone to assess value for money can inhibit decision-makers in understanding trade-offs and relating cost to their local health budget or resource-restricted context [36, 42]. Ghana has health financing challenges relevant to the possible continuation of LDHF training. Funds approved for MNH do not always reach the health sector and are frequently delayed even when released; total government spending on health per capita, $35 in 2014, still falls below the estimated amount necessary to achieve basic health service package delivery [43, 44]. As public spending in Ghana is only 53% of total health expenditure and the total health expenditure per capita, $146 in 2015, is modeled to increase only to $218 by 2030, scaling a new training program as it was implemented by an NGO may be cost prohibitive [45, 46]. We created a cost model for Essential LDHF Program activities that would estimate the cost-effectiveness of an LDHF program that was integrated into existing health system management structures. LDHF program experts defined the major assumptions for this scenario and found that, on average, it would cost $506 per provider for LDHF training (Table 5). Using the modeled program cost of $224,026 for the 40 facilities in three regions and the lower limit of health effects, the incremental cost would be $20.92 per DALY averted (Table 3). Consideration of the modeled scenario of the Essential LDHF Program activities could provide ways in which to affordably integrate and sustain LDHF training which may have more health impact per dollar spent than other MNH interventions, including alternative trainings.

Comparison to other training alternatives

The most common alternative training is an offsite BEmONC classroom-based workshop; we conducted a cost analysis of such workshop trainings led by the GCNM in Accra and in Kumasi. The GCNM workshops targeted midwifery tutors from pre-service educational institutions across Ghana whereas Jhpiego’s LDHF program targeted midwives practicing in hospitals. For this reason, as well as other confounding variables relative to the assessments and aspects of the respective institutions, health effects of the traditional BEmONC training were not evaluated or pursued for inclusion in the cost-effectiveness analysis. Comparing program costs alone still contributes by clarifying differences in key program activities and the associated costs as well as measuring efficiency when considering total cost per trainee between the workshop and LDHF training. One of the major distinctions between LDHF and workshop-based approaches is that many staff present at the facility can be trained in LDHF whereas one or two representatives from a facility are typically trained in a workshop-based training.

Other new training methods for maternal, newborn, and child health care improvement have been studied, though few connect this training with associated health outcomes. One such program implemented in seven states of South Sudan had several critical components in common with Jhpiego’s LDHF training in Ghana: an 8-day training course for local trainers, a shorter course for trainees, use of mannequins for training, and reusable medical equipment left for newborn resuscitation and care. Over 2 years, this program trained 708 frontline health workers with a sustained improvement of 40.4% in maternal objective structured clinical examination (OSCE) scores and 3.1% in newborn OSCE scores at the time of 2-month follow-up and resulting increased referral rates [47]. Though not directly comparable to Jhpiego LDHF given differences in OSCE assessments, setting, and health worker cadre trained, this is a similar example of MNH training that saw positive results. Other studies have explicitly analyzed the viability of low-dose training: a Kenyan program held 1-day newborn resuscitation trainings for public hospital health workers and found that short onsite trainings had an immediate impact on practice and associated health measures [48].

Limitations

The major limitation was a lack of a comparison group of facilities trained with the alternate approach, a BEmONC traditional workshop, in the same time period, for which the same health outcomes could be measured. Although a GCNM BEmONC workshop covered the same training content as the LDHF program during the same time period, the fundamental differences in facility and trainee characteristics rendered any comparison of facility health outcomes invalid. Jhpiego and the Ghana Health Service coordinated to minimize any MNH trainings that could be potential confounding factors to the impact of the LDHF approach; although a few sporadic trainings may have taken place, no trainings with hands-on practice, follow up, and mentoring were recorded during the timeframe [26]. Future research of LDHF in comparison to the traditional workshop-based approaches needs to go beyond cost comparison and consider health effects resulting from this distinct training alternative.

Conclusion

The LDHF approach to training presents a unique opportunity to build lasting capacity for cadres of frontline health workers that currently receive only basic training. These are the same health providers who may not usually be selected for workshop training programs due to lack of experience or seniority in the facility; arguably, these frontline health workers have the most need to engage with trainers and are most likely to provide care to patients regularly. LDHF responds to this need and—quite literally—meets providers where they are in order and helps them achieve outstanding gains in knowledge and competency, retain that capacity, and use their skills and expertise in effectively improving MNH outcomes.

At $53.07 per DALY averted, this intervention is highly cost-effective when considered with several critical cost-effectiveness thresholds and represents good value for money in Ghana. To accelerate progress toward SDG 3, LDHF training for basic emergency newborn and obstetric care should be considered for adoption and scale-up by the GHS and other implementing partners. Additional modifications building on lessons learned from cost modeling have potential to make the LDHF training program more efficient in future implementations.

Abbreviations

BEmONC:

Basic Emergency Obstetric and Newborn Care

CI:

Confidence Interval

DALY:

Disability-adjusted Life Year

GCNM:

Ghana College of Nurses and Midwives

GDP:

Gross Domestic Product

GHS:

Ghana Health Service

GNI:

Gross National Income

LDHF:

Low-Dose, High-Frequency

MM:

Master Mentor

mMentoring:

Mobile Mentoring

MNH:

Maternal and Newborn Health

NGO:

Nongovernmental Organization

OSCE:

Objective Structured Clinical Examination

SDG:

Sustainable Development Goal

References

  1. Countdown to 2015. Fulfilling the Health Agenda for Women and Children, The 2014 Report [Internet]. Geneva; 2014. Available from: http://countdown2030.org/reports-and-articles/previous-reports/2014-report.

  2. Blencowe H, Cousens S, Jassir FB, Say L, Chou D, Mathers C, et al. National, regional, and worldwide estimates of stillbirth rates in 2015, with trends from 2000: A systematic analysis. Lancet Glob. Heal. [Internet]. Blencowe et al. Open Access article distributed under the terms of CC BY-NC-ND; 2016;4:e98–108. Available from: https://0-dx-doi-org.brum.beds.ac.uk/10.1016/S2214-109X(15)00275-2

  3. Victora CG, Requejo JH, Barros AJD, Berman P, Bhutta Z, Boerma T, et al. Countdown to 2015: a decade of tracking progress for maternal, newborn, and child survival [internet]. Lancet. 2016; Available from: http://0-www-ncbi-nlm-nih-gov.brum.beds.ac.uk/pubmed/26477328

  4. Lawn J, Shibuya K, Stein C. No cry at birth: global estimates of intrapartum stillbirths and intrapartum-related neonatal deaths. Bull World Health Organ. 2005;83:409–17.

    PubMed  PubMed Central  Google Scholar 

  5. Lawn JE, Kerber K, Enweronu-Laryea C, Cousens S. 3.6 million neonatal deaths-what is progressing and what is not? Semin Perinatol. 2010;34:371–86.

    Article  PubMed  Google Scholar 

  6. UNICEF. Levels and trends in child mortality [internet]. New York, New York: New York UNICEF; 2015. Available from: http://www.who.int/maternal_child_adolescent/documents/levels_trends_child_mortality_2013/en/

    Google Scholar 

  7. Lawn JE, Blencowe H, Waiswa P, Amouzou A, Mathers C, Hogan D, et al. Stillbirths: Rates, risk factors, and acceleration towards 2030. Lancet [Internet]. 2016 [cited 2017 Jun 13];387:587–603. Available from: http://www.thelancet.com/pdfs/journals/lancet/PIIS0140-6736(15)00837-5.pdf

  8. United Nations (UN). Draft Outcome Document of the United Nations Summit for the Adoption of the Post-2015 Development Agenda. In: Sixty-ninth Sess. Agenda items 13(a) 115 [Internet]; 2015. p. 13–7. Available from: http://www.un.org/ga/search/view_doc.asp?symbol=A/69/L.85&Lang=E.

    Google Scholar 

  9. Opoku EE a. Utilization of Maternal Care services in Ghana by Region after the Implementation of the Free Maternal Care Policy [Internet]. Public Health. University of North Texas Health Science Center at Fort Worth; 2009. Available from: http://digitalcommons.hsc.unt.edu/theses/78/

  10. GBD 2015 SDG Collaborators, Lim SS, Allen K, Bhutta ZA, Dandona L, Forouzanfar MH, et al. Measuring the health-related Sustainable Development Goals in 188 countries: a baseline analysis from the Global Burden of Disease Study 2015. Lancet. [Internet]. 2016 [cited 2017 Oct 30]; Available from: http://www.thelancet.com/pdfs/journals/lancet/PIIS0140-6736(16)31467-2.pdf

  11. Blencowe H, Cousens S, Jassir FB, Say L, Chou D, Mathers C, et al. National, regional, and worldwide estimates of stillbirth rates in 2015, with trends from 2000: a systematic analysis. Lancet Glob Heal [Internet]. 2016;4:e98–108. Available from: https://0-doi-org.brum.beds.ac.uk/10.1016/S2214-109X(15)00275-2

    Article  Google Scholar 

  12. Ministry of Health G. Ghana National Newborn Health Strategy and Action Plan. 2014-2018;2014:45–6.

    Google Scholar 

  13. Oduro-Mensah E, Kwamie A, Antwi E, Amissah Bamfo S, Bainson HM, Marfo B, et al. Care decision making of frontline providers of maternal and newborn health Services in the Greater Accra Region of Ghana. PLoS One. 2013;8:e55610.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Evans CL, Bazant E, Johnson P, Hiner C, Williams E, Conecker G, et al. Saving Lives at Birth Final Report: Building and Sustaining Capacity of Frontline Health Workers in Uganda. Baltimore: A partnership of: Jhpiego, USAID, Norwegian Ministry of Foreign Affairs; Bill & Melinda Gates Foundation; Grand Challenges Canada; UKaid; American Academy of Pediatrics; Laerdal; 2016.

  15. Evans CL, Johnson P, Bazant E, Bhatnagar N, Zgambo J, Khamis AR. Competency-based training Helping Mothers Survive: Bleeding after Birth for providers from central and remote facilities in three countries. Int. J. Gynecol. Obstet. [Internet]. International Federation of Gynecology and Obstetrics; 2014;126:286–290. Available from: https://0-dx-doi-org.brum.beds.ac.uk/10.1016/j.ijgo.2014.02.021

  16. Sutton RM, Niles D, Meaney PA, Aplenc R, French B, Abella BS, et al. Low-dose, high-frequency CPR training improves skill retention of in-hospital pediatric providers. Pediatrics [Internet]. 2011;128:e145–51. Available from: http://pediatrics.aappublications.org/cgi/doi/10.1542/peds.2010-2105

    Article  Google Scholar 

  17. Bluestone J, Johnson P, Fullerton J, Carr C, Alderman J, BonTempo J. Effective in-service training design and delivery: evidence from an integrative literature review. Hum. Resour. Health [Internet]. 2013 [cited 2017 Nov 24];11:51. Available from: http://0-www-ncbi-nlm-nih-gov.brum.beds.ac.uk/pubmed/24083659.

  18. Satterlee WG, Eggers RG, Grimes DA. Effective medical education: insights from the Cochrane Library. Obstet. Gynecol. Surv. [Internet]. 2008;63:329–333. Available from: http://content.wkhealth.com/linkback/openurl?sid=WKPTLP:landingpage&an=00006254-200805000-00022%5Cnhttp://ovidsp.ovid.com/ovidweb.cgi?T=JS&PAGE=reference&D=med5&NEWS=N&AN=18419832

  19. Rowe AK, De Savigny D, Lanata CF, Victora CG. How can we achieve and maintain high-quality performance of health workers in low-resource settings? Lancet. 2005;366:1026–35.

    Article  PubMed  Google Scholar 

  20. Jhpiego Resource Brief. Low Dose, High Frequency: A Learning Approach to Improve Health Workforce Competence, Confidence, and Performance [Internet]. Baltimore; 2016. Available from: www.healthynewbornnetwork.org/hnn-content/uploads/Jhpiego-LDHF_briefer-1.pdf

  21. Sutton RM, Niles D, Meaney PA, Aplenc R, French B, Abella BS, et al. “Booster” training: evaluation of instructor-led bedside cardiopulmonary resuscitation skill training and automated corrective feedback to improve cardiopulmonary resuscitation compliance of Pediatric Basic Life Support providers during simulated cardiac. Pediatr. Crit. care Med. [Internet]. NIH Public Access; 2011 [cited 2017 Oct 31];12:e116–e121. Available from: http://0-www-ncbi-nlm-nih-gov.brum.beds.ac.uk/pubmed/20625336.

  22. Otolorin E, Gomez P, Currie S, Thapa K, Dao B. Essential basic and emergency obstetric and newborn care: From education and training to service delivery and quality of care. Int. J. Gynecol. Obstet. [Internet]. Elsevier B.V.; 2015;130:S46–S53. Available from: http://0-www-sciencedirect-com.brum.beds.ac.uk/science/article/pii/S0020729215001368

  23. Sodzi-Tettey S, Twum N, Mobisson-Etuk L, Macy L, Roessner J, Baker P. Lessons learned from Ghana’s project fives alive! A practical guide for designing and executing large-scale improvement initiatives [internet]. Cambridge: Massachusetts; 2015. Available from: http://www.ihi.org/resources/Pages/Publications/ProjectFivesAliveLessonsLearnedGuide.aspx

    Google Scholar 

  24. Ghana Statistical Service. Population and housing census [internet], vol. 2012: Ghana Stat Serv Accra; 2010. Available from: http://www.statsghana.gov.gh/docfiles/2010phc/Census2010_Summary_report_of_final_results.pdf

  25. Aseweh Abor P, Abekah-Nkrumah G, Sakyi K, Adjasi CKD, Abor J. The socio-economic determinants of maternal health care utilization in Ghana. Int J Soc Econ [Internet]. 2011;38:628–48. Available from: http://0-www-emeraldinsight-com.brum.beds.ac.uk/doi/10.1108/03068291111139258

    Article  Google Scholar 

  26. Gomez P, Nelson A, Asiedu A, Appiagyei M, Darko P, Duodu J. Accelerating newborn survival in Ghana: the impact of a low-dose, high-frequency health worker training model on intrapartum stillbirth and early newborn mortality. Submitt: Publ; 2017.

    Google Scholar 

  27. WHO-CHOICE. Making choices in health: WHO guide to cost-effectiveness analysis [Internet]. In: Tan-Torres Edejer T, Baltussen R, ADAM T, Hutubessy R, Acharya A, Evans DB, et al., editors. Glob. Program. Evid. Heal. Policy ,World Heal. Organ. Geneva. Geneva: World Health Organization; 2003. Available from: http://www.who.int/choice/publications/p_2003_generalised_cea.pdf.

    Google Scholar 

  28. Jakovljevic M, Malmose-Stapelfeldt C, Milovanovic O, Rancic N, Bokonjic D. Disability, Work Absenteeism, Sickness Benefits, and Cancer in Selected European OECD Countries—Forecasts to 2020. Front. Public Heal. [Internet]. 2017 [cited 2017 Oct 30];5:23. Available from: http://journal.frontiersin.org/article/10.3389/fpubh.2017.00023/full

  29. Shillcutt SD, Lefevre AE, Lee ACC, Baqui AH, Black RE, Darmstadt GL. Forecasting burden of long-term disability from neonatal conditions: results from the Projahnmo i trial, Sylhet, Bangladesh. Health Policy Plan. 2013;28:435–52.

    Article  PubMed  Google Scholar 

  30. Institute for Health Metrics and Evaluation (IHME). GBD Compare Data Visualization. Washingt: Seattle, WA IHME, Univ; 2016.

    Google Scholar 

  31. Collaborators G 2016 Daly and H, Hay SI, Abajobir AA, Abate KH, Abbafati C, Abbas KM, et al. Global, regional, and national disability-adjusted life-years (DALYs) for 333 diseases and injuries and healthy life expectancy (HALE) for 195 countries and territories, 1990–2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet (London, England) [Internet]. Elsevier; 2017 [cited 2017 Oct 30];390:1260–1344. Available from: http://0-www-ncbi-nlm-nih-gov.brum.beds.ac.uk/pubmed/28919118.

  32. Jakovljevic M, Getzen TE. Growth of Global Health Spending Share in Low and Middle Income Countries. Front. Pharmacol. [Internet]. Frontiers; 2016 [cited 2017 Oct 30];7:21. Available from: http://journal.frontiersin.org/Article/10.3389/fphar.2016.00021/abstract

  33. TreeAge Pro 2017, R2.1. TreeAge Software, Williamstown, MA. Software available at: http://www.treeage.com

  34. Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, et al. Consolidated health economic evaluation reporting standards (CHEERS)-explanation and elaboration: A report of the ISPOR health economic evaluation publication guidelines good reporting practices task force. Value Heal. [Internet]. 2013 [cited 2016 Sep 26];16:231–250. Available from: https://0-dx-doi-org.brum.beds.ac.uk/10.1016/j.jval.2013.02.002

  35. Walker D. Cost and cost-effectiveness guidelines: which ones to use? Health Policy Plan. [Internet]. 2001 [cited 2017 Jul 26];16:113–121. Available from: http://0-heapol-oxfordjournals-org.brum.beds.ac.uk/content/16/1/113

  36. Bertram MY, Lauer JA, De Joncheere K, Edejer T, Hutubessy R, Kieny M-P, et al. Cost–effectiveness thresholds: pros and cons. Bull. World Health Organ. [Internet]. 2016 [cited 2017 Jun 21];94:925–930. Available from: http://www.who.int/entity/bulletin/volumes/94/12/15-164418.pdf

  37. Hutubessy R, Chisholm D, Edejer TT-T. Generalized cost-effectiveness analysis for national-level priority-setting in the health sector. Cost Eff. Resour. Alloc. [Internet]. 2003 [cited 2017 Jul 26];1:8. Available from: http://0-www-ncbi-nlm-nih-gov.brum.beds.ac.uk/pubmed/14687420%5Cnhttp://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC320499

  38. Chaired by Jeffry D Sachs. Macroeconomics and Health: Investing in Health for Economic Development [Internet]. Geneva; 2001. Available from: http://apps.who.int/iris/bitstream/10665/42435/1/924154550X.pdf

  39. Points KEY. Ghana : Country Profile [Internet]. DataBank. 2015 [cited 2017 Jun 21]. p. 1–7. Available from: http://data.worldbank.org/country/ghana

  40. Brown BJ, Oladokun RE, Osinusi K. Situation analysis of the existing infant feeding pattern at the commencement of the prevention of mother to child transmission (PMTCT) of HIV programme in Ibadan. Niger. J. Clin. Pract. [Internet]. 2009 [cited 2017 Jun 21];12:421–428. Available from: https://0-www-ncbi-nlm-nih-gov.brum.beds.ac.uk/pubmed/20329685

  41. Stenberg K, Sweeny K, Axelson H, Temmerman M, Sheehan P. Disease control priorities, third edition (volume 2): reproductive, maternal, newborn, and child health. In: Black RE, Laxminarayan R, Temmerman M, Walker N, editors. Reprod. Matern. Newborn, child heal. Dis. Control priorities (volume 2) [internet]. Third edit; 2016. p. 321. Available from: http://elibrary.worldbank.org/doi/book/10.1596/978-1-4648-0348-2.

    Google Scholar 

  42. Marseille E, Larson B, Kazi DS, Kahn JG, Rosen S. Thresholds for the cost – effectiveness of interventions: alternative approaches. Bull World Health Organ [Internet]. 2015;93:118–24. Available from: http://www.who.int/bulletin/volumes/93/2/14-138206/en/

    Article  Google Scholar 

  43. World Health Organization. Data Explorer: National Health Account Indicators [Internet]. Glob. Heal. Expend. Database. 2014 [cited 2017 Nov 25]. Available from: http://apps.who.int/nha/database/ViewData/Indicators/en.

  44. Government Spending Watch. 2014 Planned Spending by the Government of Ghana [Internet]. GSW London; 2015. Available from: http://www.governmentspendingwatch.org/spending-data.

  45. Dieleman JL, Campbell M, Chapin A, Eldrenkamp E, Fan VY, Haakenstad A, et al. Evolution and patterns of global health financing 1995-2014: development assistance for health, and government, prepaid private, and out-of-pocket health spending in 184 countries. Lancet [Internet]. 2017 [cited 2017 Jun 21];389:1981–2004. Available from: http://www.thelancet.com/pdfs/journals/lancet/PIIS0140-6736(17)30874-7.pdf.

  46. Dieleman JL, Campbell M, Chapin A, Eldrenkamp E, Fan VY, Haakenstad A, et al. Future and potential spending on health 2015–40: development assistance for health, and government, prepaid private, and out-of-pocket health spending in 184 countries. Lancet [Internet]. 2017 [cited 2017 Jun 21];389:2005–2030. Available from: http://www.thelancet.com/pdfs/journals/lancet/PIIS0140-6736(17)30873-5.pdf

  47. Nelson BD, Ahn R, Fehling M, Eckardt MJ, Conn KL, El-Bashir A, et al. Evaluation of a novel training package among frontline maternal, newborn, and child health workers in South Sudan. Int. J. Gynecol. Obstet. [Internet]. International Federation of Gynecology and Obstetrics; 2012;119:130–135. Available from: http://0-doi-wiley-com.brum.beds.ac.uk/10.1016/j.ijgo.2012.05.019.

  48. Opiyo N, Were F, Govedi F, Fegan G, Wasunna A, English M. Effect of newborn resuscitation training on health worker practices in Pumwani hospital, Kenya. PLoS One. 2008;3

Download references

Acknowledgments

We extend our gratitude and appreciation to the entire Accelerating Newborn Survival program team at the Jhpiego Ghana office, especially Martha Appiagyei, Patience Darko, Julia Duodu and the clinical training team for their expertise and knowledge of historical detail in LDHF implementation and the finance department for their gracious time and record review for activity-based costing; and Chantelle Allen for her leadership, direction, and support. We also have been assisted immensely by the insight of the headquarters Jhpiego staff, especially Hannah Tappis.

Funding

This study was supported as part of the project Accelerating Newborn Survival in the Project Fives Alive! Sites in Ghana, funded by the Bill & Melinda Gates Foundation.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Author information

Authors and Affiliations

  1. Jhpiego, an affiliate of Johns Hopkins University, 1615 Thames St, Baltimore, MD, #200, USA

    Michelle Willcox, Heather Harrison & Patricia Gomez

  2. Jhpiego, 14 Ollenu Rd, Accra, Ghana

    Amos Asiedu

  3. Jhpiego, Mamba Point, Monrovia, Liberia

    Allyson Nelson

  4. Department of International Health, Johns Hopkins Bloomberg School of Public Health, Johns Hopkins University Global mHealth Initiative, 615 N. Wolfe St, Baltimore, MD, USA

    Amnesty LeFevre

Authors
  1. Michelle Willcox
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Heather Harrison
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amos Asiedu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Allyson Nelson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Patricia Gomez
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Amnesty LeFevre
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

This manuscript was written by MW, with inputs from AL, HH, and clinical team members of the Jhpiego Ghana LDHF program. Study activities were conceived by AL and HH. MW collected program implementation detail and costing data. Cost data were compiled by MW and verified and reviewed by AL, HH, AA, and AN. Impact evaluation results analyzed and shared by AA, AN, and PG. All authors reviewed and provided edits. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Michelle Willcox.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Willcox, M., Harrison, H., Asiedu, A. et al. Incremental cost and cost-effectiveness of low-dose, high-frequency training in basic emergency obstetric and newborn care as compared to status quo: part of a cluster-randomized training intervention evaluation in Ghana. Global Health 13, 88 (2017). https://0-doi-org.brum.beds.ac.uk/10.1186/s12992-017-0313-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s12992-017-0313-x

Keywords

Globalization and Health

ISSN: 1744-8603