Author's Note from JP Sekulich: My return to coaching coincided with a return to education. As a PhD student in the field of Education, I'm interested in Educational Leadership, Organizational Theory, Organizational Behavior, and Industrial Psychology. After taking a few classes, it became even more clear to me that rowing clubs are the perfect lab for studying these concepts discussed in the scientific journals. It was also clear to me that most coaches I spoke with had some form of coxswain evaluation, but it was usually of the ad hoc variety, even if it was a measure that had been used for many seasons. The inception for this paper was a graduate level survey design class, where the co-author and I thought it would be interesting to approach this casual but wide-spread practice with a more scientific method. Here's our contribution to the coaching field in an effort to further professionalize the occupation and raise the standard of rowing everywhere. The conversation is never fully complete; please contact me with any questions or comments. Also, please feel free to use the measure, but please credit the authors with citation where appropriate.

Introduction
The objectives of this survey are to assess the coxswains on UCSB’s rowing team in the following areas; technical skill, teamwork, and leadership skills. Furthermore, it was our goal to establish a measurement tool which could reliably evaluate the ability of the coxswains as such a measure does not currently exist, and having one would allow rowing coaches to identify areas for improvement among student rowers. There is a dearth of research available on assessing the skills of student athletes, but there has been studies on to coaching and leadership (Pratt, 1989 and Arnold 2000).

As both the coxswain and the coach are leadership positions, we thought that certain items and indicators presented by researchers such as Arnold (2000) and Pratt (1989) could be adapted and used for assessing the leadership skills of the coxswain. Arnold provides some indicators for effective and empowering leadership such as leading by example, encouraging, participative decision making, group management, and showing concern for the team (Arnold, 2000: 254-56). Validity for Arnold’s scale was derived from having participants go through an open-ended interview process about their managers leadership behaviors which lasted for twenty to thirty minutes (Arnold, 2000).

Pratt provides some useful ways for thinking about the types of leadership styles among coaches like authoritarian leaders and democratic leaders, and since the coxswain acts as a coach on the water we thought that these could also be adapted for the survey, but later rounds of validity testing, refinements, and discussions led us away from this as assessing the type of leader coxswains were became tangential to assessing whether or not they were effective leaders (Pratt, 1989).

In addition to adapting some of these indicators to be used to assess the construct of leadership among coxswains, the coach for UCSB’s rowing team was the lead on this project and so first-hand experience of what makes an effective coxswain was also utilized in developing the scale to measure the student athletes. Through the focus group and expert review exercises in class we were able to refine the indicators and items. This process helped us immensely, where we began by thinking about psychological stability and emotional stability of coxswains we soon realized through the feedback from the expert review that we were in fact thinking about leadership abilities and had labeled the indicator in a manner that would not resonate with many people.

Although useful, this process was difficult for our survey design as the indicators and items and the way the survey questions are posed are almost entirely specific to rowing. Thus, getting reviewers unfamiliar with the sport and not on the team to think about the indicators and items was a challenge, but we moved forward anticipating that even generalized feedback about word choice and critique of the meaning and accuracy of the indicators would help to refine the protocol.

In order to make up for shortcomings in the feedback on the survey, JP reached out to the network of rowing coaches he is in contact with and sent them early versions of the survey for review. The coaches that responded provided invaluable feedback on how we should refine the measurement. For instance, organizational skills were originally an item that was part of the sportsmanship/teamwork indicator, but after receiving feedback from the expert reviewers we found that many of them were puzzled by how it related to the indicator of sportsmanship/teamwork. Further discussions and additional rounds of feedback led us to refine the indicator of sportsmanship/teamwork to only include teamwork.

This process of drafting indicators, items, soliciting feedback, and refining our measurement based on the feedback allowed for the development of a more parsimonious survey which can measure the behaviors of coxswains and allow coaches to both identify areas that team members need to improve on, and validate what the coaches observe for themselves during practice and during events.

Methods
The intent of this scale is to provide coaches of most rowing teams (junior, collegiate, community, etc.) with a measure of general skills at the coxswain position. This scale is perhaps less applicable at the national team or elite levels, as the scale is also intended to provide basic technical guidance that elite-level coxswains would be expected to already possess, thus rendering some of the evaluative criteria in this scale redundant or repetitive. As the role of the oarsman can be typically evaluated or observed in terms of fitness on the ergometer (rowing machine, similar to a treadmill) or ability to contribute to a crew’s overall speed in relation to another oarsman through the frequent process of direct switches, this evaluative process seems to happen less so with the coxswain position, and/or is more difficult to maintain validity.

This scale is intended to be submitted to the members of a rowing team, with the specific objectives of providing a quantitative score for the coxswains on that team. The scores can be used to rank the coxswains in terms of skill, or at least in terms of preference by the team. That could be an area of future research for this study – does it measure skill, or does it measure social preference? Considering that members of a rowing team invest a considerable amount of time in their activity and therefore have a vested interest in the performance of their coxswains, as that relates to their own performance, it can be assumed that the validity of the results of this scale will likely increase with the competitive level of the club administering the survey.

At lower levels of competitiveness, this scale would be intended to be more instructional or provide feedback for the coxswains being evaluated. At higher levels of competition, this scale could be used to provide quantitative figures used for personnel selection.

Given the desired evaluative and comparative nature of this scale, it’s recommended that this scale be administered multiple times per year. The ideal timeframe would be well before major competition periods, so coxswains have the opportunity to learn from the feedback and improve, and so the motivation for the oarsmen providing the feedback are motivated by improving their performance opportunities before the intensity of the main selection cycles increase.

For the purposes of this specific measure, the intended sample size was roster of the UCSB men’s rowing team, which in the spring of 2019 was roughly 34 oarsmen. Five coxswains were to be evaluated with this scale. The average roster size (and corresponding sample size) of the rowing team is usually 24-36 oarsmen and 3-5 coxswains.

Results
This scale was developed during the course of the Spring 2019 quarter as part of the requirements for Education 218: Questionnaire Design, an elective course offered by the UCSB Graduate School of Education. The original scale was administered before the start of the quarter. The data from that scale were used in part for selection/crew assignment of varsity coxswains for the spring 2019 regatta season. As the coach of the rowing team and one of the researchers/developers of this study, JP admitted that the results of the early scale reflected actual crew placement of each coxswain on the UCSB team for the Spring 2019 Quarter. However, a desire for greater specificity of desired improvements from the rowers, or qualitative feedback on actual coxswain behavior and/or performance influenced the motivation to further refine the original scale.

Generally speaking, the results of this scale should be used as both an evaluative and comparative tool in selecting coxswains for the various crews that comprise a rowing team. There is support that the rowers completing the questionnaire are motivated to be as objective as possible, as their team performances and successes weigh heavily on this process of personnel selection. It is not recommended that coaches place full emphasis on the results of this measure in determining placement of their coxswains. Rather, it is suggested that coaches use the results of this scale to educate their coxswains on ways to improve within the context of their team. Quantitative data from the scale may reveal a trend of hierarchy in the corps of coxswains on a rowing team, but that does not take into consideration the psychological cohesiveness of work group assignment.

Conclusion
The process that we learned and utilized to develop this survey scale allowed for us to incrementally refine and improve the measurement over the course of the quarter. Coming into the course we had good idea of what our construct would be (i.e. assessing the skills of the coxswains) but the literature review section allowed for us to see how other scholars discuss team dynamics and leadership skills. Conducting the literature review added a layer of depth and more specificity to the construct and the indicators we were using to asses it.

Although, there was not much literature available on the topic of assessing student athletes through survey measurements, what was available gave us a sense of the types of items to use when assessing indicators such as leadership or teamwork skills. The literature review also gave us a sense of some of the things the researchers did to ensure that the measures had validity such as conducting open ended interviews with a sample of the target population. Once we had developed a more comprehensive list of items to go with our indicators for our construct we moved into the second phase of survey development by conducting two open-ended interviews with students in order to receive feedback on our proposed items and indicators and see if they were in fact resonating with people the way we had intended. This was one of the most valuable steps in the process as it allowed for us to review the items which respondents indicated where either out of place or lacked relevance to a particular indicator.

During the expert review phase of the survey design JP was able to network with fellow coaches who took the time to look over what we had developed in terms of indicators and items and provide feedback to us on them. For example, one of the coaches indicated that they liked that one of our items for Sportsmanship was level of effort, because the coxswain is typically the team member that sets the pace for the rest of the crew and the less effort they put in, the less they are likely to get from the crew. Another coach indicated that organizational skills would have to be further contextualized to hold relevance as an item under Sportsmanship. This was something that came up in earlier and later phases of the survey design, and ultimately this feedback led us away from this item.

The cognitive pre-testing phase of the survey was a challenge given that the questions were entirely specific to a rowing team. This meant that the participants that were used for the cognitive pre-tests did not have enough knowledge of the sport or the team to simply read each question and rephrase them in their own words. In order to overcome this limitation, we opted to have the participants think out loud about each of the indicators and each of the items. For example, to assess how well we had developed our leadership items we asked each of the respondents individually to think out loud about what a leader is to them and the type of characteristics they might possess, without using the word lead or leader.

In this phase, we also received more feedback on the item organization, namely it did not have enough context to warrant its inclusion under the indicator of Sportsmanship. It also became clear that when we were talking about Sportsmanship/teamwork that most of the items included where assessing teamwork and that sportsmanship was somewhat tangential to an assessment of coxswains.

Finally, during the pilot testing phase of the survey design, we received feedback from respondents to change the number of choices available from five to seven as five indicates unipolar response options, and seven indicates bipolar response options and our response options were in fact bipolar. Overall these survey design techniques outlined by Gehlbach and Brinkworth (2011) proved to be valuable iterative tools which when put to use helped refine this survey to its current form, which we hope will be used by other professionals in the field.

Appendices

References
Arnold, J. A., Arad, S., Rhoades, J. A., & Drasgow, F. (2000). The empowering leadership questionnaire: The construction and validation of a new scale for measuring leader behaviors. Journal of Organizational Behavior, 21(3), 249-269.

Gehlbach, H., & Brinkworth, M. E. (2011). Measure twice, cut down error: A process for enhancing the validity of survey scales. Review of General Psychology, 15(4), 380-387.

Pratt, S. R., & EITZEN, D. S. (1989). Contrasting leadership styles and organizational effectiveness: The case of athletic teams. Social Science Quarterly, 70(2), 311.

Schwarz, N. (1999). Self-reports: how the questions shape the answers. American psychologist, 54(2), 93.