Peering at the peer review process for conference submissions

For many scholars conference papers are a stepping stone to submitting a journal article. However with increasing time pressures for presentation at conferences, peer review may in practice be the only developmental opportunity from conference attendance. Hence it could be argued that the most important opportunity to acquire the standards and norms of the discipline and develop researchers' judgement is the peer review process - but this depends on the quality of the reviews. In this paper we report the findings of an ongoing study into the peer review process of the Australasian Association for Engineering Education (AAEE) annual conference. We began by examining the effectiveness of reviews of papers submitted to the 2010 conference in helping authors to improve and/or address issues in their research. Authors were also given the chance to rate their reviews and we subsequently analysed both the nature of the reviews and authors' responses. Findings suggest that the opportunity to use the peer review process to induct people into the field and improve research methods and practice was being missed with almost half of the reviews being rated as 'ineffectual'. Authors at the 2011 AAEE conference confirmed the findings from the 2010 data. The results demonstrate the lack of a shared understanding in our community of what constitutes quality research. In this paper in addition to the results of the above-mentioned studies we report the framework being adopted by the AAEE community to develop criteria to be applied at future conferences and describe the reviewer activity aimed at increasing understanding of standards and developing judgement to improve research quality within our engineering education community.