Fluency in terms of speed of speech and (lack of) hesitations such as silent pauses and filled pauses (“uhm”) are part of overall oral proficiency: second language (L2) learners with higher oral proficiency will usually speak faster with less hesitation. Language assessment rubrics, such as those for the Aptis speaking test therefore include aspects of fluency as an indicator for proficiency. Research into the precise relation between different aspects of fluency and overall proficiency, however, is still needed. Such research is usually time-consuming because of the manual labour involved in measuring the aspects of fluency. The first aim of this research was, therefore, to measure aspects of L2 fluency in both Dutch and English automatically, without the use of manual transcriptions or measurements. An existing PRAAT-script to measure speed of speech and silent pausing was rewritten and a new script was added to measure filled pauses automatically. The algorithm for filled pauses was subsequently tested by calculating the accuracy of the script: most syllables that are manually annotated to be filled pauses were indeed correctly classified as such. Finally, we gauged the extent to which the outcomes of the script could be used for language assessment purposes by relating these outcomes to human judgements on fluency. We conclude that without further testing of the accuracy of the algorithm, the current script should not yet be used for the purpose of assessing fluency automatically in a language proficiency test.