This project was made possible through Cooperative Agreement Number HRN-A-00-96-9006 between the US Agency for International Development and Tulane University

ACKNOWLEDGMENTS

We would like to acknowledge the following organizations and individuals for their contributions to the research: Bill Lyerly of USAID and Tom Ventimiglia, formerly of World Vision, without whose support this project would not have been possible, the staff of the World Vision Uganda and our interviewers and interviewees.  Also Bill Eaton and Bill Weiss of Johns Hopkins University, and Nancy Mock of Tulane University, Richard Mollica of Harvard University and Richard Neugebauer of Columbia University, all of whom gave invaluable advice on the design and execution of this project.

INTRODUCTION

This report describes the second field trial of a methodology for assessing mental health problems across cultures.  The first field trial was described in a previous report: ‘Cross-Cultural Assessment of Trauma-Related Mental Illness: A Report of Research Conducted by World Vision Rwanda and The Johns Hopkins University, April 2000', by the same authors.  Both trials represent a collaboration between World Vision (WV) and Johns Hopkins University (JHU) to develop a means by which non-governmental organizations (NGOs) can conduct scientifically valid assessments of the populations they serve.  The initiative is supported and funded by the USAID Africa Bureau Complex Emergency Response and Transition Initiative (CERTI). The objectives of CERTI are to develop a consensus on best health practices during and after a complex humanitarian emergency (CHE), and to strengthen the capacity of organizations providing assistance.  The focus is on Sub-Saharan Africa.

The study area for this trial was all of Rakai district and approximately half of the contiguous Masaka district in southwest Uganda.  The districts are largely rural and the major language is Luganda.  Most residents are poor and high death rates from HIV since the 1980s have had continuing adverse economic and social effects.  This region was chosen for this study because it is the local area of operations for World Vision Uganda (WVU).  WVU programs in this area currently focus on economic and education assistance to families and orphans affected by HIV.  World Vision currently does not offer local mental health programs although staff believe the need to be large.  The results of this field trial will serve as a prelude to mental health interventions to be implemented by WVU.

The background to the overall project and methodology; its rationale, theoretical basis, and explanations of the technical concepts, are described in detail in the report of the first field trial.  Therefore much of this information has been omitted from this report.  The contents here are limited to a consideration of the methods and results of the Uganda field trial, with a focus on the differences between this trial and the first trial in Rwanda.  The objectives of this trial are the same as those listed in the Rwanda document:

PROJECT OBJECTIVES

            1.  To create an instrument adaptation and validation process useful for NGOs and others to quantitatively assess the mental health burden of trauma at the population level across cultures and situations.

            2.  To use this instrument and process to assess part of the mental health burden of trauma for a local civilian population.                  

            3.  To use the resulting data to assess the need for interventions, form the baseline for an intervention process, and (at a future date) to help plan and test the impact of such an intervention.

METHODS                       

Overview       

As in Rwanda we focused on mental illness, this being the most severe aspect of mental health problems in terms of suffering and effects on function.  In Uganda we planned to assess depression and posttraumatic stress disorder (PTSD) as these are the major illnesses known to result from mental trauma.[1]

Our assessment methodology has 6 stages[2]:

1. Collecting ethnographic data on local perceptions of mental health.

2. Analyzing these data to determine if western concepts of mental problems (in this case PTSD and depression) are locally appropriate. 

3.  If so, using these data to adapt and translate existing questionnaires which measure these concepts.

4. Testing the validity of these concepts and questionnaires and adjusting them if necessary.

5. Using these questionnaires in a community-based survey of a random sample of the population. 

6. Analyzing the survey data to

(a) assess the prevalence and local characteristics of these mental health problems;

            (b) further test the validity and reliability of the questionnaires.

Stages 1-3 refer to the creation of the study instrument.  This is a composite of demographic questions, the standard questionnaires that assess mental illness, and a series of questions on function. Stage 4 refers to preliminary testing of the composite instrument in the study population.  Stages 5-6 refer to the actual survey, once the preliminary testing is completed and if the instrument is validated in step 4.  The survey provides prevalence data on mental illness and functional disability as well as further tests of the instrument’s accuracy.   

Ethnographic Mini-Study (stages 1 and 2)

The ethnographic study provides preliminary evidence of the local validity of western mental illness concepts.  This is necessary before standard mental health assessment questionnaires based on these concepts are used.  If, for example, the study failed to find evidence of depression, this would suggest either that depression does not occur locally or is not a major problem in the context of the study.  In either case, the use of standard instruments to assess depression would not be indicated.  If, instead, the study finds good evidence that depression occurs then these standard questionnaires can be used.  The usefulness of the ethnographic data does not end there.  The standard questionnaires will require adaptation and translation to make them suitable for local use.  Once this is done they must then be tested to make sure they are accurate.  The ethnographic data is vital to both these needs.

In Uganda the ethnographic study began with a short training in research and ethnographic methods for the ten World Vision Uganda (WVU) staff who were to be the interviewers.  The first ethnographic method was free listing in which knowledgeable local people generate lists in response to a standardized question.[3] In Uganda this question was :

‘What are the main problems that affect the people of this community as a result of HIV?’

‘As a result of HIV’ was included because of WVU’s special interest in problems resulting from this epidemic.  Each problem was recorded using the words of the respondent, who was then asked to provide a short description.  For problems that sounded mental or emotional, respondents were also asked whom people consulted about these problems - their role, name and contact information.  After completing this first free list the same respondents were asked to complete three more – on the major tasks and duties a person must perform regularly to care for a)themselves, b)their families and c)their community.  Each respondents described the tasks specific to their sex.  As before, responses were recorded using the language of the informants and a short description of each task was obtained. 

The data from the first free list were collapsed to provide a composite list of problems in order of the frequency they were mentioned.  This list gave us the local names and short descriptions of the major mental and emotional problems due to HIV (from the community’s perspective) and how highly people prioritized them compared with other problems.  The list also included links to local people who are knowledgeable about these issues.  The data from the free lists on tasks were similarly collapsed into 6 lists of self, family and community –related tasks, separate for each sex. 

Those identified as knowledgeable about mental health issues in the first free listing were then approached for key informant interviewing, the second ethnographic method.  The purpose was to confirm the local terms and descriptions of mental illness that emerged from the first free lists, and to develop more detailed descriptions of them.  Interviewers began by describing a hypothetical person with many of the mental health symptoms given by free list respondents - only free list symptoms that are part of the DSM criteria for depression and PTSD were used.  Interviewers then asked the informant to name and describe all the problems that these symptoms might represent.  At the second interview with the same informants (the next day) interviewers repeated the same question, to gather any additional thoughts the person may have had since the first interview.

The free list and key informant interview data were reviewed for evidence that depression and PTSD occur in this population.  We were primarily interested in how many of the DSM symptoms of these disorders were mentioned as the results of trauma.  Mention of most of the diagnostic symptoms would be independent evidence that these illnesses occur among this population, even if they are not classified into exactly the same syndromes as described in DSM.[4]

In Uganda local informants described all the diagnostic symptoms for depression.  We therefore concluded that there was sufficient evidence supporting the existence and importance of depression in this population.  In contrast, none of the characteristic symptoms of PTSD were mentioned in the ethnographic study.  The conclusion was that PTSD did not occur or was not a significant issue (at least in the context of HIV) and plans to study it were dropped.

Key informants not only described depression-like symptoms, but also organized them into syndromes similar to depression.  These local syndromes included several symptoms that are not part of the DSM depression criteria (see Results).  The same occurred in Rwanda.  There we conducted a pile sort activity to confirm the link between depression and these local symptoms, to confirm that they were part of the local expression of depression and could be included in the depression questionnaire.  In Uganda we decided that the confirmation of these symptoms was not important to the depression diagnosis and so did not conduct pile sorts.  However, we did include one of the local symptoms in the questionnaire by inserting it at the end of the HSCL, using the same response format (see Appendix A, question B16).  Its connection to the depression symptoms was assessed using data from the validity and survey study data (see Results).  With the elimination of the pile sorts, the completion of the key informant interviews and analysis marked the end of the ethnographic study.  

Finalizing the English version of the Instrument (Stage 3a)

Having completed the depression section of the questionnaire, the function section had to be finalized.  To assess functional disability we created a template in which respondents were asked to select the level of difficulty they experienced in completing specific tasks.  For each task the respondent is asked the level of difficulty they experience in completing that task compared with others of their age and sex.  There are 5 categories to choose from.[5]  The actual tasks inserted into the template vary with each community and are based on the most frequent responses to the three function free lists in the ethnographic study - self, family and community tasks.  Appendix A, Part A shows the template with the tasks for the Uganda study filled in.  Tasks that did not affect other people or were not really tasks were not used.  For example, many respondents in the Rwanda field trial listed prayer as an important task in caring for themselves.  However this was removed because it was not clear that inability to do this would affect others.  Many men listed ‘sending children to school’ as an important task to care for the family.  But their descriptions made it clear that their task in this regard was to earn enough money to pay the fees.  It was removed since it was covered under the ‘earning money’ task.  The most frequently mentioned of the remaining tasks were then inserted into the function questionnaire.[6] 

At the end of the questionnaire we included questions on whether the respondent thought they had the local syndrome identified as similar to depression in the ethnographic study, and whether other people had told them that they had it.  We also included questions on the duration of the local syndrome for the validity study (see Validity Study below).  The final part of the questionnaire was demographic questions on sex, age and education.  This completed the English version of the instrument (see Appendix A for the final English version).

Translation (Stage 3b)

Four translators were hired from the local town of Masaka.  Their qualifications were that they lived in the local area and were fluent in English and Luganda.  Three worked together to translate the English version of the instrument.  This was then back-translated by the fourth translator working in isolation.  All four translators then met together with one of the project directors to reconcile the differences between the translation and back translation.  During this meeting the results of the ethnographic study (the language used by the respondents) were continually consulted.  This was to try to ensure that words chosen for specific symptoms and problems were those known and used by the local population (as opposed to those used by the more educated or in other parts of the country).  On this basis we substituted several words chosen by the translators for words with the same meaning used by the ethnographic study respondents. 

The resulting Luganda version of the instrument was then reviewed by the group of WVU staff who had worked on the ethnographic study, and later by the 22 interviewers hired for the survey.  This resulted in some simplification in the language used to explain the instrument to the respondents.  Once this was completed the instrument was ready for pilot testing. We also developed and translated a standard consent form, using the same procedure (see Appendix D).

Quantitative Study Staff and Procedures

Twenty-two interviewers were hired locally.  Qualifications were a high school education, ability to read and write, ability to walk long distances and availability for the entire study period.  The ten WVU staff from the ethnographic study acted as their supervisors.  Their role consisted of:

·        assisting interviewers to find houses and identify the correct respondent

·        regularly observe and provide feedback to the interviewer on some interviews

·        re-interview 10% of respondents as a check of test-retest validity and of ‘arm chair’ interviewing

·        check all completed interviews for completeness and clarity

·        revisit all refusals to ensure that these were genuine.

Both interviewers and supervisors were given training in interviewing methods and survey procedures, and divided into groups of one supervisor and 2-3 interviewers.

Prior to each interview informed consent was obtained.  If, during the interview, it was apparent that the person was severely distressed this was reported to the supervisor who recorded the respondent’s contact information.  These respondents, and those who are diagnosed as depressed according to the interview, will be re-visited by World Vision staff to be assessed for counseling and possible referral. 

Instrument Testing (Stage 4)

Pilot Study

The purpose of the pilot study was to detect any problems with the interview procedure, the consent form and the instrument (including data entry), and to give the supervisors and interviewers field practice before the survey began.  Each was allocated one of the 30 selected study villages[7] in which they interviewed one man and one woman chosen by convenience.  Interviewers, supervisors and the project directors then met and reviewed the experience.  The process went smoothly and few problems emerged.

Validity Testing

Validity Study

Validity refers to how well the instrument measures what it is supposed to measure.  There are various types of validity, but the most important are criterion and construct validity.  These concepts were explained in detail in the Rwanda field trial report.  Briefly, criterion validity refers to the agreement between the questionnaire and an external measure (criterion) of the same construct known to be accurate (referred to as a ‘gold standard’).  This is not strictly possible with this population, since none of the usual ‘gold standards’ are available.

Instead, we conducted an alternative validity study using an alternative local standard - diagnosis by local people of local illnesses similar to depression.[8]  Supervisors revisited the knowledgeable persons identified in the first free list and asked for the names of people who had the local syndromes most similar to depression.  They also asked for the names of persons who do not have this problem.  Supervisors then assigned these people to interviewers without revealing their reported illness status.

At the end of the instrument were questions asking whether the respondent felt if they had either of the local illnesses.[9]  In the analysis we threw out all responses where the key informant and the respondent disagreed.  This was to try to ensure that we had true cases and true non-cases by reducing the number of inaccurate local diagnoses.  The remaining cases and non-cases of local illnesses formed the external criteria against which the depression instrument was compared - if depression really occurs among the population and can be accurately identified by the instrument there should be a strong correlation between depression diagnosis and the diagnosis by local people of these similar local illnesses.  We used the Phi correlation coefficient because it measures correlations between dichotomous data.[10]

Validity Testing in the Main Survey

Validity testing was not restricted to the validity study or to the depression section of the instrument.  Validity testing was also done using the data from the main survey and on both the depression and function sections of the instrument.  In some cases supervisors reinterviewed respondents using the instrument, as a check of reliability (see below).  Where there were other adults living in the house and available, they were asked to assess the respondent’s function using the same function questions.  Neither respondent or cohabiting adult witnessed the other interview.  Comparison of the results provided a validity test of the respondents’ responses regarding function.  More details of the validity (and reliability) testing done in the main survey are given in the Analysis and Results sections. 

Survey Sampling and Procedure (Stage 5)

We used a 30 cluster sampling method.  We first made a list of all the villages in the study area.  This list included estimates of the population of each village based on local government data.  Systematic random sampling was then used to make weighted selections of 30 villages to be included in the survey.  Weighted means that larger villages were more likely to be chosen.  Interviewers and supervisors went to each village and, with local advice, mapped the village.  These maps were not detailed but merely showed sections, based on separation by roads, paths, hills streams, etc, and approximately how many houses were in each section.  These maps were then used to make a random weighted selection (according to number of houses) of a sector in each village.  We needed to have a minimum of 20 houses in each village so, where a chosen sector had less than 20 houses, a contiguous sector was also chosen.

The 30 villages were allocated among the supervisors and their teams of 2-3 interviewers.  They numbered the houses in the chosen sector(s) of their villages, then used a random number table to select 20 houses.  An interviewer then visited each house and asked an inhabitant to list, in any order, all the adults (persons of at least 18 years of age) living there.  The interviewer would then select the respondent from this list, again using the random number table.  If the respondent was not there an appointment was made to return at a more convenient time, or the interviewer went and found the person if they were not far.  If the respondent refused to be interviewed that interview was marked as a refusal and no-one else from that house was interviewed. Only one person was selected from each house, giving a total sample size of 600.  This number was based on a target of 384 for a simple random sample design, a recommended cluster design effect of 1.3 (Aday, 1989), and an estimated 10% refusal rate and 10% failure rate to find the house or an eligible respondent.[11]  The inflation of the sample size by the design effect is due to the partial loss of randomization inherent in the cluster design: The multistage choice of blocks (or clusters) instead of a single choice of respondents by simple random sampling increases the risk of a non-representative sample being chosen.  Increasing the sample size is an attempt to counteract this.  The sampling method also deviates from pure randomization in the choice of a single respondent from each house, regardless of size.  This gives those living in houses with more adults less chance of being chosen.  However, to conduct true simple random sampling (the most accurate measure) would require a complete list of all the adults living in the study area and their addresses.  This is not available or feasible in this study area, as in many other parts of Africa.

Quantitative Analysis (Stage 6)

Data from each interview in the validity study and survey was entered into a computer and analyzed using SPSS[12] statistical software.  Analysis consisted of tests of validity and reliability, and assessment of depression and functional disability and the associations between them.

Testing Reliability

This was done with the main survey data only.  As discussed in the Rwanda field trial report, there are two elements of reliability that should be tested before an instrument can be used: internal consistency reliability and test-retest reliability.  The former is measured by the Cronbach alpha statistic, which is the average correlation between all questions measuring the same thing (such as depression or function) in the same interview.  As a rule of thumb Cronbach’s alphas should be above 0.7 and ideally between 0.8-0.9.[13]  The reliability of each question can be assessed by calculating the alpha with and without it.  Significant increases in alpha without the question would suggest that the question is not measuring the same thing as the other questions, and should be removed.  Studying the effect of each question in this way is called Item Analysis. We measured Cronbach’s alphas separately for the male function, female function and depression questions.

Test-retest reliability refers to testing reliability over time.  Supervisors re-interviewed approximately 10% of survey respondents 1-3 days after the initial interview.  We then measured the Pearson correlations between the depression scores and the function scores on the initial and repeat interviews.[14]  Opinions vary as to what is an acceptable score, although correlations above 0.7 are considered desirable (Aday, 1989).

Testing Validity in the Main Survey

We also did some instrument criterion validity testing in the main survey.  A depression severity scale was created for each respondent by adding the responses of the depression questions.  The mean score for those with no local illness was compared to that of respondents with a single illness and both illnesses.  If the HSCL is valid mean depression scores should increase in the presence of local illnesses.

Construct validity was also tested in the main survey.  This refers to how well items that are supposed to be associated with each other (for example, various depression questions) are actually associated.  Cronbach’s alpha can be used as a measure of construct validity as can factor analysis.  Factor analysis is similar to Cronbach’s alpha, except that it demonstrates which questions are most highly correlated with each other, rather than the questions as a whole.  This information can be used to understand the different elements of depression that underlie the responses.  Similarity between these constructs and hypothesized constructs support construct validity.  Both the Cronbach’s alpha and the factor analysis were done using the main survey data.  The results of the factor analysis are in Appendix F.

Criterion validity testing of the function questions was also done in the main survey.  If a cohabiting adult was available when supervisors reinterviewed respondents a second interview was conducted with that person.  The second respondent was questioned about the function of the initial respondent, using the same function questions.  Each was interviewed separately.  Each set of responses was added to create separate function assessment scores.[15]  On this scale 0 represents no difficulty with any task, compared with others of the same age and sex, while 36 represents the maximum level of difficulty for every task.  The function score according to the initial respondent was then compared with the score according to the other adult.  Comparison was by plotting the scores and calculating the line of best fit and its corresponding coefficient of determination.  If agreement is good between respondents and cohabiting adults the coefficient of determination should be high and the slope of the line should be close to 45 degrees with its origin through 0 (ie, a change of one on one scale corresponds to a change of one on the other scale).

Depression is known to have a significant effect on function.  Therefore testing for an association between depression diagnosis using our depression questionnaire and reduced function using our function questionnaires also provided some criterion validity, both for the depression questions and the function questions.  This is discussed further in the Results section.     

Measuring Depression and Functional Disability

From the survey data we calculated the prevalence of depression diagnosed according to the DSM algorithm.  We also calculated the prevalence of any ‘health-related’ difficulty with each of the function tasks; separately for men and women and for depressed vs not depressed.  We explored the relationship between depression and function by regression analysis with function assessment score as the dependent variable and depression, local illness, sex, age and education as the independent variables.[16]

RESULTS

Study Site

We studied villages selected from across all of Rakai District and from 8 parishes in Masaka District.  This constitutes the local area of operations for World Vision Uganda.  The total population of this area is 1-2 million.  We selected this site so that World Vision can use the results as an indicator of need and baseline for future interventions.  Both areas have been severely affected by HIV in the past, with high rates of infection and mortality (Nunn et al, 1997; Sewankambo et al, 1994).

Results of the Ethnographic Study

Free Listing

For the first activity (free listing) 10 WVU interviewers spoke with 50 knowledgeable persons in 10 of the 30 villages chosen for the survey.  Interviewers conducted 4 free lists with each respondent on a)problems in the community due to HIV, and important tasks that adults must do regularly to care for b)themselves, c)their families, and d)their communities.  The results of the problem free lists are summarized in Table 1.  From the free lists of important tasks those for which difficulty did not appear to affect others were removed.  From the remainder the nine most frequent responses for men and for women were inserted into the function section of the instrument (see Appendix A).

Key Informant Interviews

The original plan called for free list respondents to identify persons who deal with mental or emotional problems (see Methods).  These persons would then be the key informants for this stage of the ethnographic study.  However, most of the persons so identified were from outside the community (such as social workers and NGO staff) and therefore were not suitable informants.  Therefore, interviewers were advised to reinterview those free list respondents who appeared knowledgeable about mental health issues or to identify new knowledgeable respondents by asking local people for the names of those who dealt with these issues and lived in the community.  In selecting key informants, interviewers were also advised not to interview highly educated persons, to limit the possibility that respondents would express ideas from outside the community that differed from the general community viewpoint.

A total of 20 key informants were interviewed – of whom 10 were interviewed twice.  The first interview began with a description of a hypothetical person with many of the DSM depression symptoms that emerged in the free lists.[17]  Key informants were then asked to name and describe what problems this person might have, and to give detailed descriptions of these problems (see Methods).  Respondents described two main syndromes with depression-like features:

·        Yo’kwekyawa - translated as hating oneself

·        Okwekubagiza - translated as pitying oneself

The reported symptoms of these illnesses are shown in Figure 1.

Table 1: Results of Free Lists on major problems resulting from HIV*