Contents Introduction
CCPForge1 is a collaborative software management environment provided by the Software Engineering Support Centre (SESC) based at the Science and Technology Facilities Council and is funded by the Engineering and Physical Sciences Research Council (EPSRC). It has a focus on supporting the EPSRC funded Collaborative Computational Projects2 (CCPs). It launched in 2006 when this type of tool was not widely or freely available and thus was needed to share code development and other tools. The projects within the system are all academic. The underpinning technology did not keep pace with developments, usage declined and in April 2018 the decision was taken to close CCPF orge down at the end of 2018.
Following the change in service leadership in 2015 SESC invested time in understanding the usage of the service through user surveys and analysis of usage behaviour. This short paper discusses the outcomes of this analysis work, identifies lessons learnt and suggests reasons for the behaviour identified. It examines whether these lessons learnt could be more widely applicable, especially to institutional software repositories which are becoming more wide spread in the UK.
User Analysis
CCPForge supports a wide range of domains within the CCP communities. These are mostly from fields using computational science techniques and this is reflected in the most widely used languages which are Fortran, C, C++ and Python.
In 2015 SESC conducted a user survey, with a relatively small response rate (n=43). The main reasons for using CCPF orge are shown in figure 1. Over half felt that having a focus for community was important, although not all CCP codes are in CCPForge. A significant minority indicated that they hadn't chosen the CCPForge platform, but it was expected that they use it - which is the alternative, more negative, view from having a community resource.
The revised 2015 user and content management policy put in place a more rigorous annual user management process. In this an active user was defined as a user who had logged into CCPForge within the last twelve months. The user management process was focussed on removing inactive users. This has been run for the last three years. Figure 2 shows an analysis of how long deleted users had been registered and active in CCPForge. Over half of the users deleted had a short interaction elapse time with CCPF orge. Our interpretation of this is that as private projects are accessible only to registered users, these users are registering to CCPF orge to download the software. Whereas it is more likely that those who have been registered with CCPF orge for longer and with a greater gap between creation date and last logged in date are involved with the development process.
Interestingly in this year's deletion process there has been a large increase in the number of users who have been involved with CCPForge for over five years. One potential explanation is that these were the early adopters of the service and are now retiring or becoming less active in software development. We would be interested in alternative explanations from the wider community.
To see if these usage patterns can be identified in users within active projects, two of our private projects with the greatest number of users (n=57 in both cases) were analysed and the outputs are shown in figure 3. This shows that around a third of users regardless of the project, have a short amount of time between creating the user id and the last login.
The act of managing users encourages people to log in to keep their account active and so all CCPF orge users have logged in within the last eighteen months.
The question is therefore: does this pattern of user behaviour in one specific service with a specific community have any lessons for other services? We suggest that for institutional services in this space, understanding user behaviour patterns and managing them over significant periods of time is key. It also explains the large number of users, there are over 750 current registered users, following the 2018 user management exercise which removed 783 users, and an average of 47 new users a month over the last twelve months. Our user management experience has shown that a significant number of inactive users have simply forgotten they ever had user accounts with CCPForge, due to the volume of queries about this to our helpdesk.
Project Analysis
The SESC team analysed the projects held within CCPForge in 2017; at that time there were 161 user projects split into the following categories:
40 with public repositories (25%)
77 with private repositories (48%)
44 with either no repository or no commits in the last 5 years (27%)
So, nearly half of the content on CCPF orge is in private repositories. As the analysis was done from statistics, it is not possible to determine how much this reflects culture when the projects were set up and how much is due to other factors such as the projects wanting to be able to track usage. CCPForge commit statistics are not available before 2010.
Of the 161 projects, 30 committed code between November 2017 and February 2018, as the closure decision was made. It was clear that a proportion of projects were no longer developed, and some stated that the development version was elsewhere.
There are a variety of code versioning systems in use. These typically show the changes in preferred tool over the last decade. The majority of new projects in the last two years use Git. Figure 4 shows the code versioning systems in use in CCPF orze at the end of April 2018.
The major challenge to managing a service such as CCPForge is being able to identify the status of a project and ensuring that there is a valid owner/project administrator. As a service you do not hold the rights to the project and so you need to have someone responsible for the content. Our experience has shown that as projects cease active development then the connection between the software in a project and any developer becomes looser and there can come a point where as a service owner you don't know who owns a specific project and hence your service terms and conditions need to be clear as to what happens in this case. In 2017 a project administration notification process was run. This resulted in a small number of projects being removed as they were no longer needed and a change in project administrators following role changes which were not notified to SESC.
Conclusions and Lessons Learnt
Clear user and content policy is important and should be in place from the beginning. In our experience when services are new, the focus is on getting people to use them and the issues around managing inactive users and content can sometimes take second place. Active management of the service ensures that it complies with legal requirements and remains relevant to the users of the service.
The use of private repositories and the user registration requirement to enable access to the software, means that services such as CCPForge have two types of user: short-term single purpose users and long-term contributors to a project. This increases the management overheads, but ensures that the individual projects are aware of who is using the software. This is a proxy until software citation is more embedded within the community. It could be argued that the registration is a barrier to further use and reuse of the software.
It is difficult to explain why CCPForge continued to be used as other, more functionally-rich services appeared. From our interactions with our users we believe that it is a combination of familiarity with CCPF orge, provision of enough functionality and the perceived effort to move. This may also be related to the community supported.
Over a decade software projects are created, used, mutate, thrive or die. Services providing hosting services for these projects need to be able to identify the owners and status of the projects to be able to establish the value they are adding. CCPForge was a publically funded service designed to support the advancement of knowledge by enabling researchers to share the code development. As can be demonstrated by the decline in usage, CCPForge's purpose is no longer unique and it is time for it to close. Being able to understand how the service is used and when usage patterns change is an important part of service management.