Scott Bristol's 'Values@Hand'.

Validity and Reliability

Summary: Life Journey Map® values measurement methodology challenges many of the popular values measurement techniques. During the ranking process the user rank orders a randomly generated list of 130 values, presented in sets of four. Value ranking scores are the sum of two complete rankings. Validity: 71% of the users report 'High Degree of Fit' upon seeing their top 6 priority values at the end of the values ranking process. Reliability: The between test 'Mental Model' -rank score correlation is .91. (n=2500, 46% men, 54% women).

Life Journey Map® Values Measurement Methodology

Defines these conditions as necessary for Values Measurement:

Values Measurement is based on a values ranking process.
The individual is asked to rank order three or more distinct values at a time (forced ranking).
The individual is presented with a full domain of distinct values to rank order.
The full domain of values is presented randomly to avoid any biasing.
The affective (feeling) nature of values allows them to be either means or ends.
The individual can discard values from the ranking process.
The value definitions shown while ranking are the value definitions in their totality.
Value scores relate directly to the individual’s ranking process.
An immediate feedback loop at the end of data collection asks the individual to confirm the validity of his or her values ranking results.

Considers these techniques inappropriate for Values Measurement:

Focusing on small set of values. (Content Validity Error)
Absence of a feedback loop at the end of data collection that presents initial results and asks the user to evaluate relevancy of results. (Criterion-related Validity Error)
Using a Likert scale when measuring values. (Construct Validity Error)
Using Polarity scale when measuring values. (Construct Validity Error)
Using Forced Choice when measuring values. (Construct Validity Error)
Presenting the full domain of values in a fixed sequence (non-random) for ranking. (Construct Validity Error)
Presenting partial value definitions during individual scoring that then map to different value definitions for reporting results. (Construct Validity Error)
Classifying and measuring separately ‘means values’ and ‘goal values’. (Construct Validity Error)

Validity

Are we measuring what we think we are measuring?

There are three types of validity: 1) content, 2) criterion-related, and 3) construct, each attends to a different question (Kerlinger,1973):

1. Content Validity: Do the number of elements (values) being measured adequately represent the full domain of all potential elements (all human values)?

Naturally occurring full linguistic domains appear to have an upper limit of 500 distinct words (Berlin, 1992).
According to Life Journey Map's® definition, there are hundreds of values (Rokeach, 1968).
Using Christian biblical document analysis, Hall and Tonna were the first identify a large domain of values (125). (Hall, 1986; Tonna, 1995, 1997)

Life Journey Map® methodology:

Measures 130 different 'elemental values', each value is defined by 3 different synonyms thus drawing on a full domain of up to 390 value words.
Measurements indicate that people operate within a personal domain of values-a Field of Meaning with an average size=34 different values.
As a self-correcting document analysis methodology asserts that any written phrase with a value orientation should equate to three or less 'elemental values'. If not, a new 'elemental value' needs to be added to the list of measured values.

Common Errors in Values Measurement:

Most values measurement tools are implicitly biased in that they fail to present a full domain of values that can be cross checked via document analysis.
Brainstorming values fails to generate a full domain of values, it will only generate the individual's or team's Field of Meaning.

2. Criterion-related Validity: Does the measurement predict the external variables or criteria? There are two criterion we are interested in:

a) Does the measurement process result in people identifying a set of priority values?

Life Journey Map® methodology:

In response to the question: "According to your own experience and self awareness do these 6 statements reflect the values that you believe and experience as being most meaningful, significant, and important to you at this time?" From a sample of n=2500; 46% men, 54% women:

71% report a "High Degree of Fit"
28% report a "Moderate Degree of Fit"
0% report a "Low Degree of Fit"
0% report "No degree of Fit"
1% report "Cannot Answer"

Common Errors in Values Measurement:

Non-interactive measurement techniques, such as paper and pencil, fail to present immediate results to the participant and fail to collect participant validation of the results.

b) Do the priority ranked values predict 'value behavior': spontaneous speech and problem-solving orientation?

Life Journey Map® methodology:

Consultant observations indicate 'yes'. No systematic study has been completed at this time.

3. Construct Validity: What factors or constructs account for variance in measurement?

Life Journey Map® methodology:

Uses a ranking schema: important to match mathematical schema to the phenomena (Lakoff & Núñez, 2000).
Uses value definitions in the values ranking process in their totality, neither adding or removing words from reported results.
Individual's can choose to discard or remove values from their ranking domain thus insuring the the 130 domain of values measured is culturally relevant over a large sample of users.

Common Errors in Values Measurement:

Using a Likert scale when measuring values. Example: Rate the degree to which you personally identity with each of the following ‘value statements’. Use a Likert Scale: 1 = No Identification, 2 = Low, 3 = Moderate, 4 = High; 5 = Very High Identification. This process fails to rank order different values, consequently it is an inappropriate measurement schema.
Using a Polarity methodology: Example: Pick one of four statements that best describes your team... Results are mapped onto an "Individualism vs. Communitarianism" polarity scale. This process fails to rank order different values, consequently it is an inappropriate measurement schema.
Using Forced Choice measurement by presenting four different values and then selecting only one that is most important fails to create a rank order relationship between the selected value and other possible choices.
The largest area of construct variance happens when people are asked to manipulate or respond to one set of words that result in a conclusion that uses a different set of words. For example, presenting a partial value definition during an individual scoring that then maps to a different (more worded) value definition when reporting results.
Static questionnaires have no self-correcting mechanism for testing the relevancy of the domain of values being used and adjusting the questions accordingly.

Reliability

If we measure again and again will we get the same results.

One of the challenges of values measurement is to use a ranking schema to measure a full domain of values in a manner that reduces structural biasing.

Both Rokeach and Hartman use the ranking process for values measurement and both use philosophical arguments to define and classify values in a manner that results in the size of the domain of measured values being limited to groupings no larger than 18 (Rokeach,1968,1979; Hartman,1967; Edwards & Davis,1991). In different research settings Rokeach uses different sets of 18 values (Rokeach,1979).

Hall and Tonna (1978) expanded Rokeach’s domain to 125 values but in turn gave up using the ranking schema. They use a forced choice methodology. The goal of this approach, also used by the University Michigan, World Values Survey, is to make the value set ‘full’ or robust enough to use cross culturally.

Life Journey Map® methodology resolves this conflict by using interactive technology to randomly present a large domain of values (130) that are rank ordered in sets of 4. Two complete ranking rounds (steps) are completed. This methodology generates both a process measure and a content measure.

Process Measure: 'Mental Model' refers to a analogical (Gentner & eds.,1983,2001, Johnson-Laird, 1983) manner of decision making. In the ranking process this results in different patterns of scoring; i.e. ranking one value over and over again as top priority vs. ranking many different values as priority. Consequently we can correlate the first step and second step scores according to descending magnitude of ranking. Mental Model-rank score correlation = .91. This is a measure of our cognitive unconscious and demonstrates the reliability of Life Journey Map® values measurement methodology.

Content Measure: Item measurement refers to specific value ranking score assigned in each step. The between step Item Correlation = .45. This is not a particularly high correlation but understandable given the random presentation of the values. This random presentation skews the scoring when an important value ends up at the end of the presentation list disallowing the accumulation of ranking points. The assumption is that the more ranking steps completed the less the influence of randomization and the stabilization of the rank order of specific values.

Preliminary measures indicate that what is reliable over a period of time is the person's set of Field of Meaning Values. This is supported by the 71% High Degree of Fit score reported on top 6 priority values, which are a subset of the individual's Field of Meaning.