The purpose of this proposed program evaluation is to appraise the effectiveness of the Unit Leadership Development Program, a service-wide program in the United States Coast Guard, in terms of conflict management. The Unit Leadership Development Program was introduced in January 2005 and became mandatory for use by all Coast Guard units in June 2005 (U.S. Coast Guard, 2005). Outcome data for various indicators of leadership effectiveness have been collected for a number of years as a normal part of the Coast Guard's ongoing performance evaluation initiatives. In addition, qualitative data will be gathered and evaluated as a part of this program evaluation. The retrieval, analysis, and interpretation of this data for the purposes as outlined within this proposed program evaluation will be new for this initiative.
Coast Guard Maintenance and Logistics Command Atlantic, a staff command located in Norfolk, Virginia, has identified interpersonal conflict between staff members as a key problem. This problem was first identified by a mixed methods assessment conducted during the initial implementation of a leadership assessment (Stinson, 2005).
Interpersonal conflict can be defined in a number of different ways. Lulofs and Cahn (2000) note that interpersonal conflict situations include four distinct characteristics (p. 5). They note that people involved in an interpersonal conflict situation are interdependent, that they “perceive that they seek different outcomes or they favor different means to the same ends,” that “the conflict has the potential to negatively affect the relationship if not addressed,” and there is “a sense of urgency about the need to resolve the issue.” Wilmont and Hocker (2001) believe that interpersonal conflict is “an expressed struggle between at least two interdependent parties who perceive incompatible goals, scarce resources, and interference from others in achieving their goals” (p. 41).
The Maintenance and Logistics Command staff is located in downtown Norfolk in a commercial office building. The staff consists of approximately 450 employees; about half the staff is military. The other half of the staff is civilian employees and civilian contract employees. The staff provides program management and direct support to other Coast Guard units in the fields of naval engineering, civil engineering, electronic engineering, personnel services, health & safety services, and legal services. Some divisions – such as the civil engineering staff – are fairly small; the civil engineering staff has a dozen employees. Other divisions, such as the naval and electronic engineering staffs, are fairly large as they provide a variety of direct services to Coast Guard units east of the Rockies. The naval engineering staff, responsible for boat and cutter repairs and maintenance, has approximately 150 staff members. The Maintenance and Logistics Command is headed by a rear admiral – a one-star flag officer – and a deputy commander who is a senior captain.
The Unit Leadership Development Program is a service-wide program and was introduced in January 2005; all Coast Guard units – including operational units, support units, and staff commands – were required to have implemented the program by July 2005. The program mandates three specific actions by all Coast Guard units. Units are required to conduct an assessment to determine the perceived implementation of the Coast Guard defined leadership competencies at the unit; units are required to analyze the data; units are to prepare an action plan to close the gap.
Beyond these mandated activities, the Unit Leadership Development Program consists of a number of supporting components designed to assist the unit in assessing perceived leadership competency maturity. Many of these supporting components are Internet-based and found at the program’s website, http://learning.uscg.mil/uldp. The program includes the ability for the unit to arrange for a web-based assessment delivered to each member of the unit. The assessment is a survey of 36-questions which reflect the respondents' perceptions of leadership implementation and maturity at the unit (U.S. Coast Guard, 2004a). The assessment's results are provided to the unit point-of-contact as an aggregate; the results are presented according to the leadership competencies. At the present time, only the first 21 competencies, those related to leading self, leading others, and leading performance and change, are reflected in the assessment results. The competencies related to leading the Coast Guard have not yet been incorporated in the assessment tool or the program.
Beyond the assessment tool, the Unit Leadership Development Program's website provides suggestions of specific interventions for each leadership competency. For instance, for the leadership competency of conflict management, 12 interventions are listed as recommended interventions for increasing unit personnel’s competency of conflict management. If a unit wanted to close a perceived or actual gap in this competency, they would implement one or more of the recommended interventions. Another component to the Unit Leadership Development Program is the availability of trained coaches to assist units in developing their implementation plan of the program, in analyzing the assessment data, in choosing appropriate interventions, and in implementing the interventions. While the use of a coach is not required, the personal assistance a coach provides can help ensure the unit follows the program as designed.
While the Unit Leadership Development Program was designed to develop overall leadership within the Coast Guard across all defined leadership competencies, this program evaluation will look only at the leadership competency of conflict management:
Coast Guard leaders facilitate open communication of controversial issues while maintaining relationships and teamwork. They effectively use collaboration as a style of managing contention; confront conflict positively and constructively to minimize impact to self, others and the organization; and reduce conflict and build relationships and teams by specifying clear goals, roles and processes. (U.S. Coast Guard, 2004b p. 4)
The Unit Leadership Development Program at the Maintenance & Logistics Command Atlantic staff will be evaluated both formatively and summatively using a mixed methods approach. The formative evaluation will examine if the program has been implemented appropriately at the staff command. The summative evaluation will examine specific outcomes which are related to leadership and the leadership competency of conflict management. In addition, the summative evaluation will examine staff perceptions about conflict management through various survey assessments.
Research Questions to be Examined
This evaluation looks to answer five research questions:
1. What measures or indicators does the literature indicate serve as a barometer of interpersonal conflict?
2. Has the Maintenance & Logistics Command Atlantic staff implemented the Unit Leadership Development Program according to the standards required by the program and in accordance with the mandate from the Commandant?
3. Does the Unit Leadership Development Program impact conflict management at a staff command?
4. What impact, anecdotally, has the Unit Leadership Development Program had on individual members of a staff?
5. What Unit Leadership Development Program interventions were, according to staff members, most helpful in their own development?
The first two research questions look at formative issues; the next three research questions addresses a summative evaluation. The second and third questions are quantitative in nature; the other research questions are qualitative.
The formative evaluation will examine if and how the staff is completing the mandatory activities; the evaluation will also examine to what extent and how the staff used the other, non-mandatory components, of the program. The mandatory components consist of three activities: assessment, analysis, and action plan. Did the unit use the 36-question assessment provided with the program, or did the unit use another assessment tool or survey? How was the assessment delivered? Did the unit use the data from the survey – whether quantitative, qualitative, or mixed – for an analysis of the current state? Was the analysis completed using the leadership competencies as a filter? Was an action plan developed that appears to address the gaps or chosen leadership competencies? Was the action plan completed? For each of these questions, the formative evaluation would show the program has been implemented as mandated if an assessment was made and the analysis of the assessment data was built around the leadership competencies. In addition, the program mandates the action plan be developed; the Commandant's mandate does not actually specify, however, that the action plan needs to be carried out.
The summative evaluation is quasi-experimental in nature and is an interrupted time series design. The summative evaluation will look at outcome or result measures which serve as indicators to the success or failure – or maturity – of conflict management behaviors at the unit. The measures that will be examined are all currently collected by various program managers at the Maintenance and Logistics Command or higher authority as indicators of program activity or success. They have, to this point, not been analyzed as indicators of conflict management or leadership.
In addition, the summative evaluation will have a qualitative component, using focus groups and interviews to determine staff members’ thoughts and perceptions about the leadership development program and interpersonal conflict within the staff.
Quantitative Data to be Examined
Sutterfield, Friday-Stroud, and Shivers-Blackwell (2007) note that “interpersonal conflict deals with relationship tension” between people within an organizational context (p. 219). Indeed, this notion that interpersonal conflict happens within organizations is paramount to the assertions of Knapp, Putnam, and Davis (1988) who suggest that common conflict instruments measure intent or attitude rather than actual conflict; the instruments measure “abstractions of verbal styles – not actual verbalizations” (p. 420). They suggest that more appropriate measures must reflect “behavioral decisions in specific situations, characteristics of organizational conflicts, and models of fit between persons and environments” (p. 417). Knapp, et al suggests that appropriate measures will “examine person-context interaction” and may use “time series measures to examine cross-situational changes and consistencies in preference for conflict strategies” (p. 425).
Conflict, and the inappropriate or unhealthy resolution of conflict, is reflected in various outcomes or results. When the interpersonal conflict is not resolved either quickly or appropriately, the quality of the relationship between the people changes, as does the climate of the organization (Lulofs & Cahn, 2000, p. 14). These indicators include civil rights and equal opportunity complaints, employee churn rates, and disciplinary events. For each of these indicators, an increase in conflict would drive the metrics upward.
Three types of data will be examined in the course of this evaluation: civil rights data, employee retention data, and disciplinary data. Many civil rights and equal employment issues have, at their very root, conflict and conflict management as a cause. Research by Gallop has shown that leadership behaviors, specifically conflicts between employees and managers which are not managed appropriately, are a root cause for people leaving an organization (Buckingham & Coffman, 1999). Likewise, many incidents which become disciplinary in nature begin as conflicts between employees or conflict between an employee and some accepted norm, rule, or policy. The only measures which will be used in this proposed evaluation will be those that have at least three years of data before Fiscal Year (FY) 2005. Data, specific to the Maintenance and Logistics Command staff, must be available from at least FY 2002 and forward in order to be used for this interrupted time series evaluation. Data from FY 2005 and 2006 will also be used; the program intervention occurred during FY 2005. Results would be expected for FY 2006 and following. To determine if the effect of the Unit Leadership Development Program is long-term and does not diminish with several years, result measures would ideally be tracked for several years.
Civil rights data. The civil rights data will come from two sets of metrics. The Coast Guard tracks the number of informal complaints and formal complaints made with the Equal Employment Opportunity (EEO) domain. EEO complaints, whether actually based on valid discrimination or not, nearly always have some sort of conflict at the root, and usually that conflict has been managed poorly or at least the aggrieved party believes the conflict management has been less than satisfactory. The EEO realm becomes the forum of last resort for many conflicts. A second set of metrics for the civil rights data comes from triennial comprehensive civil rights unit assessments. These assessments include a survey along with qualitative data obtained through interviews and focus groups.
Employment retention data. Employee retention data is available for three important populations. The Coast Guard places great importance, because they have been found to be strong indicators of leadership amiss, on: (a) re-enlistment rates for first term enlisted member, (b) retention rates of junior officers after their initial six-year obligation is over, and (c) civil servant retention or “churn.”
Staff disciplinary data. The final set of behavior data concerns conduct and disciplinary actions. The Coast Guard tracks the number of infractions to the Uniform Code of Military Justice, including instances of “Captain's Mast” , known as Non-Judicial Punishment (NJP) and Courts Martials, which are actual, legal trials similar to civil or criminal courts outside of the military. Most instances of NJP are caused by conflict or poor conflict management, often conflict between the subordinate and the supervisor. Captain's Masts and Courts Martials are actions under the Uniform Code of Military Justices and apply only to military members of the Coast Guard. Civilian employees, who are federal civil servants, receive disciplinary action through other means which is, most often, informal hearings and some punishment levied by an appropriate commissioned officer.
Other quantitative data from surveys. In addition to the behavior data measuring outcomes, the summative evaluation will look at staff member's perceptions about conflict management at the unit. The Coast Guard has used a number of survey instruments over the last ten years and continues to use assessments such as the Unit Leadership Development Program 36 question assessment. The data from each entire assessment will not be examined; rather, the questions relating to conflict management, as defined by the Coast Guard leadership competency, will be analyzed.
The Coast Guard has used many assessment tools in the last ten years which may yield data related to conflict management. The Workforce Cultural Audit was a Coast Guard-wide survey examining cultural issues including civil rights, leadership, and discrimination. The Workforce Cultural Audit was completed once nearly a decade ago. The Coast Guard has completed two service-wide assessments using the Organizational Assessment Survey, an assessment developed by the Office of Management and Budget. The Coast Guard intends to conduct this survey biennially. In addition, the Maintenance and Logistics Command staff has conducted several assessments during the last seven years which may yield appropriate data. One assessment tool used was the Q-12, a survey based on the work of Gallup as published by Buckingham and Coffman (1999). Another tool used more recently is the Unit Leadership Development Program is a 36 question assessment tool. Each of these assessment tools has specific questions about conflict management, and each assessment tool's results data can be analyzed at the staff level.
One weakness with this methodology is that the behavior metrics are actions or results which are merely indicators of conflict management; they do not measure, for instance, the number of conflicts reported or noted. They are only indicators, and as such other things will impact these metrics. For instance, some reported civil rights incidents are exactly that: civil rights incidents; they are not indicators or interpersonal conflict. The Coast Guard does still have members and employees who do not treat people equally and fairly because of their race, color, gender, or other protected classification. Another impact on civil rights complaints is complaints which are resolved in the aggrieved party's favor and result in a large monetary award for the complainant. Nothing breeds formal and informal complaints through the civil rights and EEO system like rumors of employee success for complaints lodged earlier and just resolved. There is a sense of “I'll get mine, too” for some complainants. Other factors could also move the civil rights metrics. And, the same is true for the other two sets of measures, also. For instance, the Coast Guard's well documented response to Hurricanes Katrina and Rita will certainly have some impact on various performance and program measures; but, we will be unable to determine what actually had an impact on the various measures. Was it Katrina or the Unit Leadership Development Program or something else altogether?
Threats to Validity within the Program Evaluation
As noted above in discussing the weaknesses of this proposed program evaluation, a key weakness is the threat to internal validity through history or events. As constructed, the program evaluation does not isolate cause and effect. A number of forces or events – including or not including the Unit Leadership Development Program – could impact the various measures. While this threat cannot be removed entirely, perhaps it can be minimized, at least in the analysis of the data, with qualitative information. Staff members’ perceptions of what caused any changes could be polled during focus groups and interviews. While this would not remove the threat, it would – perhaps – provide data which can be analyzed and shown to minimize the threat.
A second threat to the validity of the proposed program evaluation is also discussed in the weaknesses: the measures do not actually measure conflict management. They may not provide a valid assessment or indication of conflict management. Additional review of the literature needs to be conducted to ensure an accurate representation of conflict is created.
The Coast Guard staff that created the Unit Leadership Development Program certainly hoped that the use of the program would have a positive impact on leadership development and the use of leadership competencies throughout the Coast Guard. This program evaluation seeks to identify impact for a single leadership competency at a single organization. As to the formative research question, it is anticipated that the evaluation will show the Maintenance & Logistics Command Atlantic staff has implemented the program according to the required standards of the program. As to whether or not the program impacts conflict management at a staff command, it is anticipated the data will show a reduction of civil rights cases, increased employee retention, and reduced disciplinary actions. From these anticipated data sets, it is hoped to show the implementation of the Unit Leadership Development Program has a positive relationship with regard to conflict management.
With the bulk of the quantitative data already captured by the Coast Guard, and a supportive staff willing to participate in interviews and focus groups, progress in completing this study should be fairly quick. In addition, impending organizational changes throughout the Coast Guard, the Commandant’s strategic transformation initiatives, makes it imperative to complete the study before the Maintenance and Logistics Atlantic staff is thrust into a new organizational matrix.
The following benchmarks are proposed:
Acceptance of the concept paper:
Completion of the IRB process:
Submission of the proposal:
Acceptance of the proposal:
Quantitative data gathered from Coast Guard sources:
Completion of interviews and focus groups:
Submission of dissertation with results and conclusions:
Acceptance of dissertation and completion of program:
Buckingham, M., & Coffman, C. (1999). First, Break All the Rules: What the world's greatest managers do differently. New York: Simon & Schuster.
Knapp, M. L., Putnam, L. L., & Davis, L. L. (1988). Measuring Interpersonal Conflict in Organizations: Where do we go from here? Management Communications Quarterly, 1(3), 414-429.
Lulofs, R. S., & Cahn, D. D. (2000). Conflict: From theory to action. Boston: Allyn and Bacon.
Stinson, P. A. (2005). Addressing the Need for Leadership Development as a Human Performance Issue in the United States Coast Guard: An intervention designed for MLC LANT. Retrieved December 2, 2005, from here.
Sutterfield, J. S., Friday-Stroud, S. S., & Shivers-Blackwell, S. L. (2007). How NOT to Manage a Project: Conflict management lessons learned from a DOD case study. Journal of Behavioral and Applied Management, 8(3), 218-238.
U.S. Coast Guard. (2004a). ULDP 36: A competency-based assessment instrument for leadership development. New London: Coast Guard Leadership Development Center
U.S. Coast Guard. (2004b). Leadership competencies. Retrieved October 12, 2005, from here.
U.S. Coast Guard. (2005). Commandant’s Priorities – People – Unit leadership Development Program Implementation. Retrieved April 9, 2005, from here.
Wilmont, W. W., & Hocker, J. L. (2001). Interpersonal Conflict (6th ed). New York: McGraw-Hill.