1. Introduction

The reproducibility crisis in Chemistry has been widely reported on and remains a key challenge for the future of the field. [1] Many reasons for irreproducibility have been debated and include flawed experimental methods more often than fraud. Over 80% of chemists report that they have failed to repeat an experiment from the literature and over 60% have failed to repeat one of their own experiments. [2] One of the key skills to ensure reproducibility is sufficient laboratory experience to apply chemical intuition to a reaction procedure and this lead to a successful outcome, [3–5] This experience is often limited to those who have practised at a postgraduate level or beyond and is a key focus for the application of machine learning in chemistry research. [6–8] One area where a lack of reproducibility can be directly associated to a lack of experience is in undergraduate teaching. This is not just a problem for chemistry, with other sciences reporting reproducibility flaws at the undergraduate level such as P-value hacking, [9] HARKing [10] and drawing conclusions from undersized samples. [11,12] Part of the undergraduate laboratory learning experience is to perfect fundamental techniques that are critical to successful science, and consequentially, reproducible results. But despite standardized training, often there is a variety of skill and practical chemical intuition across a similar cohort of students. Going forward to research-based learning this can result in experimental failure as the students face a steeper learning curve.

One way in which reproducibility can be improved is using standard operating procedures and linking these to measurable outcomes in real-time. Allowing students to recognise how minor differences in an approach change the overall outcome, and to optimize their practise to perfect skills. Using this approach, relatively inexperienced scientists should be able to perform more complex experiments successfully with replicable outcomes. The use of real-time data analysis provides one of these measurable outcomes. The aim is that the outcome of a chemical reaction can be predicted based on its real-time reaction profile and how this compares to previous attempts. With the advent of new process analytical technologies to acquire this real-time data such as ReactIR and ReactNMR, [13–16] it is important to provide learning opportunities for undergraduate students in this area to enrich their learning experience and improve their employability.

At the University of Nottingham students in the third-year of their MSci(Hons) degree in Chemistry or Chemistry with Medicinal and Biological Chemistry undertake mini-projects that give students the opportunity to plan and design their own experiments under guidance from an academic member of staff. [17] In this manner students exhibit higher-order cognitive skills, and this leads to a more valuable learning experience. The aim of this model is to bridge the gap between standard expository laboratory instruction and the open-inquiry model of a research environment. This approach has been successful and the outcomes of some of these projects have been published in the research literature. [17–19]

In a pioneering collaboration between the University of Nottingham and DeepMatter® developed an inquiry-led project based on the use of DigitalGlassware^® to optimise a chemical reaction. The aim of this project was to highlight the variation within student groups even when following a ‘recipe-style’ experiment and to challenge the students to design experiments to reduce this variability, and such improve reproducibility. This student-centred inquiry-based approach allows students to take responsibility for the design and direction of the project and develop the new ‘recipes’ to improve reproducibility using the DigitalGlassware^® platform.

DigitalGlassware^® is an innovative cloud-based digital chemistry platform from DeepMatter^® that allows users to capture and analyse a rich array of information about their chemical reactions. The platform records time course sensor data from the reaction flask against XML-structured experimental protocols (recipes) that contextualise the chemist’s actions in the lab. Outcomes such as yields and purity, and characterisation data such as chromatograms and spectra, can also be retained against the run record, improving analysis and interpretation capabilities. Coupling these rich run records with aggregated cloud based data presents opportunities for aggregated data analysis, machine learning and better understanding of the underlying chemical processes.

DigitalGlassware^® consists of several software and hardware components (Figure 1):

Figure 1. A schematic overview of the DigitalGlassware® platform indicating the software (RecipeRunner and WebApp) and hardware (DigitalController, DeviceX and other devices) connected via the cloud-based service

- The DigitalGlassware^® WebApp: a web-based application which acts as the primary interface for creating recipes and viewing aggregated data.

- The RecipeRunner tablet application: an Android app that accompanies the chemist in the laboratory, offering live sensor traces and the opportunity to record notes and observations against the chemistry.

- The DigitalController: the central hardware device that collates sensor streams from the lab and stores them securely and instantly on the cloud-based service (Figure 2).

Cloud-based storage: The DigitalGlassware^® software components and data storage is cloud-based, providing instant dissemination of reaction results, and improving collaboration opportunities.

To facilitate data capture in the lab, DeepMatter^® has designed DeviceX (Figure 2), a unique multi-sensor probe developed that sits right inside the reaction vessel, providing real-time data (temperature, pressure, UV light levels and more). This data is collected and stored in the cloud alongside each process carried out during the reaction, providing a whole new level of insight and understanding.

Figure 2. An image of a typical DigitalGlassware® equipment set up in the laboratory. The DeviceX multi-sensor probe is inserted into the reaction flask, on the right the DigitalGlassware® WebApp is running on a tablet. Right front – DeviceX Environmental Sensor that is mounted within the fumehood. Left – Digital Controller

For this project each student group had access to a shared DigitalGlassware^® platform, a DigitalController and a DeviceX for means of capturing reaction flask data. The shared cloud-based service allowed the students to analyse and consider their data collectively in an iterative manner, using the outputs to facilitate the design and direction of the project.

2. Experimental Model

The reaction chosen for students to explore the synthesis of N-(1-naphthoyl)-4-methylbenzenesulfonohydrazide as reported in Organic Syntheses (Scheme 1). [20] This reaction requires the students to demonstrate several fundamental techniques essential for success in the organic chemistry laboratory but leaves some scope for interpretation of the procedure.

Scheme 1. N-Sulfonylation reaction between 1-napthoyl chloride and p-toluenesulfonyl hydrazide

The students’ initial chemical intuition would be examined by carrying out the reaction following the standard procedure using the DigitalGlassware^® recipe format. This procedure was prepared in advance of the laboratory and represents a typical expository style procedure and is available at the supporting GitHub (https://github.com/deepmatterltd/dm_nortcliffe) and on the open DigitalGlassware^® platform in deepmatter® ‘s interoperable XML format. The schema for interpreting the data can be accessed online at www.schemas.deepmatter.tech. The provided procedure contained some omissions of experimental specifics that students were expected to use their own experience to make an objective judgement, such as the temperature of the ice-bath or the rate of addition of chemicals. The students incorporated the use of the DeviceX into their reaction set up to collect sensor data in real time. Following completion of the experiment, the students compared the overall experimental outcome (yield and purity of product) and the reaction profile to try and understand why the outcomes were varied. Following this they adapted the recipe on the DigitalGlassware^® WebApp software online ahead of their next session. The initial aim of the project was that students would undergo two rounds of optimization following the initial comparison run. Unfortunately, due to the COVID-19 pandemic students were not able to complete the final round of optimization due to the closure of the University. However, the ultimate goal of optimizing the reaction is secondary to the learning experience of using state-of-the-art chemistry hardware, that has been adopted by the chemical industry, and developing an appreciation for the need for reproducible chemical reactions and how data science can accomplish this.

2.1. Pedagogical Values

These laboratory experiments have the following learning goals

1. To introduce students to the use of real-time data collection using DigitalGlassware^®.

2. To allow students to reflect on their reactions and evaluate reaction outcomes and relate these to experimental procedures.

3. To design new experimental procedures in response to collected data in an effort to optimise a reaction.

4. To introduce students to the power of data science and its potential applications within Chemistry (including artificial intelligence and machine learning).

The overall optimization of the reaction was not set as a goal for the project as more value was placed on the process and the decisions made by the students.

2.2. Hazards

All experiments should be performed in well-ventilated fume hoods; suitable eye protection, hand protection (nitrile gloves) and a laboratory coat should be worn. Eye and skin contact, inhalation and ingestion of all chemicals should generally be avoided. 4-Dimethylaminopyridine is highly toxic by skin absorption, causes eye and skin burns, and all eye and skin contact and inhalation should be avoided. p-Toluenesulfonyl hydrazide is toxic if swallowed and flammable, do not use in a laboratory where there are naked flames. there is a risk of explosion if heated in a closed system. Dichloromethane, tetrahydrofuran, and acetonitrile should be used with special caution, since they may cause damage to the eyes, skin, brain, liver, lung, and central nervous. Dichloromethane is a known carcinogen. All solvents should be handled in a well-ventilated fume hood. Triethylamine and N,N-diisopropylethylamine are flammable liquids, cause skin burns and eye damage and are harmful if swallowed. Citric acid causes serious eye irritation. 1-Naphthoyl chloride is toxic by inhalation, causes eye and skin burns, and all eye and skin contact and inhalation should be avoided. Only use in a well-ventilated fume hood.

3. Results

In the first week, the students worked in pairs to each complete the reaction (pairs a-f) using the same procedure and following the recipe on the DigitalGlassware^® platform. They were asked to work independently and to avoid conferring or comparing their experiment to the others; relying solely on the experimental procedure and their experience and experimental intuition. The reactions were monitored using DeviceX and students were able to monitor the real time data through the tablet. Each pair of students was able to prepare the sulfonamide successfully, although as expected there was significant variation in the overall yield (Table 1).

Table 1. Summary of the results using the initial procedure. Reagents and conditions: 1-Napthoyl chloride in CH2Cl2 added dropwise via syringe to an ice-bath solution of p-toluenesulfonyl hydrazide and 4-dimethylaminopyridine in CH2Cl2. See supporting github for full details

Despite following the same procedure, the overall yield of the reaction was highly variable with a range of 14-60% yield of purified material observed. The reported yield for this reaction is 62%. With the cloud-based data storage, the students had access to the aggregated data of their peers for review and analysis. During the post-experiment review sessions the students could use the visualisation tools within the DigitalGlassware® web application to compare run data and notes, and highlight where deviations occurred amongst their experiments to gain a better understanding of how their actions in the lab led to different outcomes. When the subsequent laboratory experiments were performed the students were in a better place to address these issues to gain more reproducible outcomes We challenged the students to determine some of the possible reasons for the variability of the reaction through looking at the data recorded by the DeviceX probe. To give students time to do this, we allocated dry lab-session in a computer room for students to review the data generated for each run, and compare the results achieved. All the results generated by DeviceX are stored on the DigitalGlassware® WebApp, allowing for review and comparison between scientists on a project.

Based on a review of the six independent experiments the students highlighted two key points of variability – these were the temperature the reaction mixture reached in the ice-bath, and the time of addition of triethylamine to the mixture. The use of the DigitalGlassware^®’s. WebApp allowed for the capture of the differences in the set-up of the ice bath (Figure 3). It can be seen that the depth of immersion within the ice is variable, also that not all students added water as a liquid “carrier” to allow for the transfer of heat between the reaction vessel and the bath. The important aspect to note in these images is the submersion of the RBF in the top picture (A, run 1a) is different from the bottom (B, run 1e): specifically 1e appears to be more submerged than 1a. In terms of yield, experiment 1a reported 48% whilst 1e reported the highest at 60%, a small difference that could be related to the more effective solution cooling in 1e. There is some loose evidence to suggest this as experiment 1b reported a yield of 23% and their RBF is of similar submersion depth to 1a. While this explanation is not the most scientifically rigorous explanation, having the images available to spark this discussion is important, particularly for the students to compare set ups and consider how this could impact their end result. By overlaying the reaction profiles of each run we can see how the temperature of the reaction mixture varied along with the time of addition for 1-napthoyl chloride/CH₂Cl₂ (Figure 4). This overlay clearly shows that, despite following the same set of instructions, this was interpreted by students differently, and led to vastly different temperature profiles. The students recognised that the data showed a large degree of variability in the temperature of the reactions and that this could lead to different reaction outcomes. Based on this they understood the importance of the liquid ‘carrier’ in the experimental setup and improved their experimental technique. While it is common for students to reflect on their lab experience to improve their skills, the use of experimentally collected data using DeviceX gives students a clear visualisation on the how the differences in reaction set up actually affect the reaction taking place, reinforcing the need to improve and develop.

Figure 3. Image notes taken by the students with the RecipeRunner tablet app used to capture the ice bath set-ups. Note the difference in immersion depth in A cf. B

Figure 4. An overlay of temperature profiles captured by the 6 groups during the addition of 1-napthoyl chloride/CH2Cl2 mixture (Reactions 1a-f)

Based on their initial work, the students designed a series of follow up experiments (2-9) to probe the outcome of the reaction by changing a particular variable – some of these variables were based on the data that was generated in week one. A summary of changes and the reasons for this can be found in Table 2. While many of these changes seem self-explanatory to those with significant experience of practical experimental chemistry, these simple changes were suggested based on the student’s own analysis and assessment of the data. The outcome of the initial experiments was unknown and the use of the data and discussion between the pairs was the main tool for coming to these suggestions. By designing their own future experiments, students were able to take greater ownership of their own work and were able to appreciate that the process they had undertaken was a real-life scenario undertaken by professional chemists on a daily basis. [21] With new experiments designed students used the RecipeBuilder module to create new experimental procedures (recipes) and to update the various steps in order to remove potential ambiguity within the language used. Following the new procedures, as expected the students observed significant variability in their outcomes (Table 3).

Table 2. Summary of experimental changes made following the initial run

Table 3. Results of second set of reactions with modifications

Following the second set of results two sets of conditions, 5 and 8, stand out. By controlling and slowing the time of addition of triethylamine (20 mins, condition 5) the yield was significantly improved compared to a ten-minute addition. Ensuring the temperature remained below 0°C. (condition 8) was also found to show a yield improvement compared to no control (Table 2, reaction 9). Other changes to the experimental conditions were found to have a detrimental effect on the reaction; highlighting that not all changes made to a procedure will lead to improved outcomes, for example the use of THF as the solvent (Table 2, reaction 2) or pyridine as base (Table 2, reaction 7) had unfavourable outcomes. The third round of optimisation would have seen students combine example procedures determined in the second round to attempt to run an optimised procedure with each pair.

Unfortunately, due to the COVID-19 pandemic, the third round of optimisation combining multiple parameters based on the data was not possible.

4. Discussion

We sought extensive informal feedback from the students through individual conversations throughout the project and after. They particularly appreciated working with technology and data to inform their decisions – recognising the global boom of data-based science and the importance of this for future employability. They also spoke favourably about working on an experiment that paralleled a real-life challenge – reproducibility and the input of scientists working at DeepMatter^®. The following are quotes from the student authors taken from an evaluation survey of the project.

• “I thoroughly enjoyed working with the equipment provided by DeepMatter^®. Mainly because It was the first time I have worked with technology and data in chemistry.”

• “Working with DeepMatter^® on this project was very interesting; it highlighted to me the inconsistencies in the reporting of findings in the field of synthetic chemistry which ultimately lead to issues with reproducibility.”

• “Although our work was cut short, I thought our findings were conclusive and supported our ideas of how we would optimize our reaction, none of which would have been possible without the data that DeviceX collected.”

• “I think after I graduate, having some underlying knowledge and experience in this field will be inherently useful and will be something not many other undergraduates have.”

• “Since using the DeepMatter^® DeviceX I think using technology and software in the lab is almost essential. The amount of data that can be obtained is immeasurable compared to traditional methods.”

• “The project allowed us to understand what was required for a successful optimisation and made us think of potential apparatus and techniques to optimise a reaction. Additionally, the idea of working with DigitalGlassware^® that has never been used in an undergraduate lab before was very exciting.”

• “We get an insight into what people are developing and how chemistry can be used outside of a lecture theatre.”

• “Being able to monitor these endotherms or exotherms on a graph in real time (as well as the other parameters monitored by DeviceX) forces you to think more deeply about the chemistry that is happening as a result of each step of the method.”

• “We’ve come across new techniques such as using a syringe pump or a dropping funnel, trying out different acids and bases, practicing a quench or a reverse quench and varying the temperature of the reaction.”

• “I think DeepMatter^®’s DigitalGlassware^® would be a great tool to have access to in early-stage undergrad synthetic labs; as an undergraduate in a teaching lab there is a lot to think about, particularly as a first or second-year student, where many of the techniques and pieces of equipment you are using are new to you.”

• Due to the timing of this project (February/March 2020) and the impact of COVID-19 restrictions, students were not assessed on this work. Had assessment taken place the ability of them to work well as a team would have been assessed and how the data they generated allowed them to make decisions in the lab. A laboratory report detailing the experiments undertaken and outputs of data from the DigitalGlassware^® platform with a suitable written narrative of the discussions and decisions made would have identified successful completion of the learning goals. An oral presentation to supplement this would have allowed opportunity to explore student thinking and decision-making processes.

• The overall expectation of students to fully optimise a reaction was not a pedagogical goal of this work, mainly due to the time restraints of the lab. However, for future applications of this project then a specific optimisation should be identified, giving students something to aim for in their work. The judicious choice of chemistry to investigate with students is also important, for example, reactions that are known to give multiple products or variations in diastereomeric / enantiomeric ratio are good examples to investigate due to additional important reaction outcomes, other than yield. One limitation of the work is the scale of the reaction suitable to be analysed. Due to the size of the DeviceX sensor, the tip containing the sensors must be submerged in solvent in order to gather accurate data, it is optimized to work in 100 mL or 250 mL round-bottomed flasks; microscale reactions would not be suitable to analyse with this equipment.

Due to the timing of this project (February/March 2020) and the impact of COVID-19 restrictions, students were not assessed on this work. Had assessment taken place the ability of them to work well as a team would have been assessed and how the data they generated allowed them to make decisions in the lab. A laboratory report detailing the experiments undertaken and outputs of data from the DigitalGlassware^® platform with a suitable written narrative of the discussions and decisions made would have identified successful completion of the learning goals. An oral presentation to supplement this would have allowed opportunity to explore student thinking and decision-making processes.

The overall expectation of students to fully optimise a reaction was not a pedagogical goal of this work, mainly due to the time restraints of the lab. However, for future applications of this project then a specific optimisation should be identified, giving students something to aim for in their work. The judicious choice of chemistry to investigate with students is also important, for example, reactions that are known to give multiple products or variations in diastereomeric/enantiomeric ratio are good examples to investigate due to additional important reaction outcomes, other than yield. One limitation of the work is the scale of the reaction suitable to be analysed. Due to the size of the DeviceX sensor, the tip containing the sensors must be submerged in solvent in order to gather accurate data, it is optimized to work in 100 mL or 250 mL round-bottomed flasks; microscale reactions would not be suitable to analyse with this equipment.

5. Conclusions

The DeepMatter^® DigitalGlassware^® platform is a user-friendly apparatus that has been successfully applied to an undergraduate project investigating reaction optimisation of a typical transformation carried out in industry. DeviceX collected real-time information about the reaction that was monitored via a tablet computer. Students were then able to use this data to develop new experimental procedures to investigate the outcome of the reaction. Through this project, students were able to experience first-hand the value of data in chemistry and how data science will continue to impact on practical chemistry. Student feedback was overwhelmingly positive with students highlighting the skills obtained and their value of them in future employability. Since the completion of this project, DeepMatter^® has continued to work with researchers at the University of Nottingham in further developing projects that utilise the DigitalGlassware^® and DeviceX technology to further expand the usability of this technology and some of these results have recently been published. [22]

The data captured by the students and procedures (recipes) is available for download at https://github.com/deepmatterltd/dm_nortcliffe.

ACKNOWLEDGEMENTS

We thank the University of Nottingham for funds and consumables associated with this work. Special thanks to Peter Morgan-Tansley for helping set up the DigitalGlassware^® equipment at Nottingham.