Objective To develop and validate a self-evaluation tool (questionnaire) to identify perceived competence and confidence felt by physiotherapists undertaking respiratory on-call duty. Design A questionnaire to rate specific acute care respiratory competencies was developed from criteria in the Emergency Duty Guidelines. A triangulation approach was used to evaluate content validity. The questionnaire was completed by a group of respiratory physiotherapists and a group of non-respiratory physiotherapists. Feedback from respiratory physiotherapists was through open (written) questions, and 15 participated in semi-structured telephone interviews. Participants A total of 263 participants: a purposive sample of 180 respiratory physiotherapists attending ‘On course for on-call’ courses in Autumn 2005, and a convenience sample of 83 non-respiratory physiotherapists attending on-call training courses or the 2005 Congress of the Chartered Society of Physiotherapy. Results Thirty-eight questionnaire items were compiled and tested (12 assessment and 10 treatment competencies, 10 range and six confidence items). There were 222 responses (86% overall response rate). Cronbach's alpha found internal consistency for all items (a > 0.9). There was a significant difference in competence scores between the respiratory group (median 118, interquartile range 111 to 121) and the non-respiratory group (median 76, interquartile range 64 to 87) (P < 0.001). Participants endorsed the content validity of competencies. The importance of current, regular clinical experience in determining competence was highlighted. Self-concept, self-efficacy and attitude to on-call may affect competence rating, and reliance on self-evaluation alone in the assessment of clinical competence was questioned. Conclusions Respiratory physiotherapists involved in on-call training supported the development of a tool to assess self-perceived competence, and endorsed the content validity of the questionnaire items. The study findings support the validity of the tool as a competency measure since it is able to discriminate between those at different levels of seniority and experience. Participants varied in their approach to rating their own competence in unfamiliar situations; this seemed to depend on self-concept and self-efficacy. Since this tool is based on core respiratory competencies, it may also be used as supportive continuous professional development evidence for the Knowledge and Skills Framework and professional development portfolios. It also has the potential adaptation for use with physiotherapy students and those working in acute respiratory care.