Concept inventories have become increasingly common in STEM disciplines as a means of assessing student conceptual understanding on a given topic, and overall they have led to significant reform in the teaching and learning of content in their respective disciplines. In mathematics, the use of the Calculus Concept Inventory seems, anecdotally and based on a review of the literature, to be growing. Yet peer-reviewed literature on its development and psychometric properties is lacking. Using data from approximately 1800 students across four institutions, we analysed its content validity, internal structure validity, and reliability. We conclude that the data is consistent with a unidimensional model and that the instrument lacks sufficiently strong validity and reliability for its intended use. We therefore recommend the development of a new instrument to better measure conceptual understanding of calculus concepts based on a stronger theoretical framework.