Previously, we identified and set the file.paths to the location where the .fcs (and .csv) files are stored after daily QC. We also identified the file.paths to the respective instrument and archive folders within your InstrumentQC folder. Afterwards, we modified the Instrument.R file to reflect these paths, and successfully copied over and processed the existing data into an Archive.csv file.
We will now move on to discuss how to set up manual and automated options to facilitate this process as part of a daily routine.We will also discuss the steps involved in the export of the processed data to the GitHub repository for use in creating the website, and how to orchestrate this when you have multiple instruments sending QC data.
Setting up Manual Transfer
Within your forked InstrumentQC folder, you can find an “Examples_staff.qmd” file. This is a .qmd file, that contains within the individual chunks code specific to each instrument that get used when setting up a staff.qmd file for use in manually transferring data from individual instruments.
Start by first duplicating the “Examples_staff.qmd” and rename it as “staff.qmd”. This file name is ignored by Git (through its inclusion in the .gitignore file) allowing for slightly different versions of this file on each instrument to exist without the different file.paths causing version control issues.
Now that you have the staff.qmd file, go ahead and open it. You will encounter a large code chunk containing several lines of code. On the upper-right hand side, you will see a green play button that says “Run Current Chunk”.
When this is clicked, the code-block will run, provide the correct file paths and trigger Rscript you have modified to process the newly acquired QC data and sending it to GitHub.
As long as your file.paths were correctly set up, the only thing staff needs to manually do in the morning to generate the processed data is the following steps:
- Open Rstudio.
- Make sure they are in the InstrumentQC project folder
- Open the staff.qmd file
- Hit the Run Code Chunk button
- And that’s it.
This manual sending of data approach is useful if you have fewer instruments and are already present at the instrument at the time of QC, as the extra minute to hit “Run Code Chunk” doesn’t add that much an inconvenience. Drawbacks are if you forget, the website when rendered will not contain the data for that day, and you will need to go back to the instrument and send the data.
Setting Up Automated Transfer
Another option is to set up an automated transfer of the QC data for each instrument to the GitHub repository at a specific time of the day. This is possible by utilizing the Windows Task Scheduler (the system that schedules your computer updates, etc.) via the TaskScheduleR R package. What this looks like in practice is at the designated time, a black terminal/console window will pop up on the computer screen for around a minute, run the code in the Instrument.R file, push the data to GitHub and then close the terminal window.
One of the considerations with using this approach is deciding at what time of the day you want the automated script to run and process newly acquired QC data. For our institution, QC on all instruments is historically wrapped up most days by 10:30 AM, which is when we schedule the automated tasks to begin for the first instrument. This way, we ensure the websites and dashboards have the most recent data rather than the displayed data being delayed a day due to the automated script running after QC for the day was acquired.
Some of the issues with the automated approach is the sudden appearance black terminal window pop up that pops up at the designated time to run the code.This doesn’t affect anything on the cytometer while acquiring, but it can startle an user who is not expecting it to appear. In our experience, when an user is on the instrument at the time the console window appears, they will either panic and immediately close the black terminal window, or after it appears to not be doing anything for 10 seconds get annoyed and close it down. This results in incomplete processing of the data, with the data not being sent to server, requiring staff intervention.
In our context, since our goal is for the website data to be current (and therefore can’t just set the automated processing to occur during the middle of the night), we set automation for the small window between when morning QC is typically acquired and when users usually first book. We additional have trained/emailed/reminded users to not close the terminal but with mixed results.
Currently, we are working on figuring out a way to provide computer privileges necessary to run Task Manager without opening the console window, or when the computer is logged out, but that remains a work in progress at the moment.
To get started, open the “TaskSchedules.qmd” file for example code. Load the respective required packages with the library function, and navigate to an individual instruments block of code. Modify the file.paths to those you previously identified for your own instrument to find where the new QC data is stored, and the location of your forked InstrumentQC repository that contains the individual Instrument.R scripts.
Next, looking at the code block, we can see there is setup code for three automated tassk: “QC_Instrument_Morning”, “FlagRemoval” and RepoPull. We will discuss each of these in turn, starting with setting the task schedule for QC_Instrument_Morning.
We can see it’s using TaskManagerR’s taskscheduler_create function. Set the taskname to indicate the name of your instrument and that the task is QC related. Then for rscript provide the file.path to the Instrument.R script. Leave the start date as is, but adjust to the desired time for the automated processing to occur. Once you have modified these fields, hit enter to schedule the task.
We can check to see if the task is scheduled by entering the taskscheduler_ls() function into our console and searching for the taskname we set. Additionally, you can give the same taskname to taskscheduler_delete() if you decide you don’t want to run that task at that time anymore.
A Hybrid Approach
At UMGCC FCSS, we use both the automated and manual transfer setups. We schedule automated QC to happen between 10:30 and 10:40 for the various instruments. However, if QC happens earlier (or is delayed) the manual option is available to avoid having automated QC trigger when an user is present and the data for the day has already been sent.
To allow automated QC to discern whether QC has already been sent earlier manually, hitting “Run Code Chunk” on staff.qmd will also produce a Flag.CSV file after completion, that is stored within the InstrumentQC folder. If this file is present, automated QC will skip processing data steps. This results in just a momemtary blip of the console window. We then set a FlagRemoval task (the second one in the code chunk) that at 1 PM every afternoon checks the InstrumentQC folder, and if the Flag.csv file is present, removes it. This resets the cycle, allowing automation to proceed the next day as normal.
To replicate this for your system, open the “Examples_FlagRemoval.qmd”. Copy and paste the code into a new .R script called “FlagRemoval” (also present in the .gitignore file to allow for difference in file.path between instruments). Then return to the TaskSchedules.qmd file, modify the second “FlagRemoval” task to match the new FlagRemoval.R scripts filepath and desired run time, and then create the task.
One thing to remember, regardless of using manual or automated approach, is that the Instrument.R script will first retrieve data from GitHub, then process the data, then send the updated data to GitHub. Consequently, we need to stagger the time between instruments to avoid having the Git commit fails because it doesn’t recognize the GitHub repository that has had changes since it pulled the data but hasn’t finished returning the new data for it’s instrument. In our case, we have the Aurora 3L automated processing run at 10:30, and every subsequent instrument run two minutes later which avoids this issue entirely.
We also are experimenting with a third automated task (RepoPull) that brings in the larger changes involving the Website early in the morning to cut down on console window time around 10 AM. However, we are finding if the computer was shut down or restarted by the last user of the night, this task may not execute.
To set this up on your own system, similarly to when you set FlagRemoval, use the “Examples_RepoPull.qmd” file, create a new “RepoPull.R” script, adjusting the file.paths respectively, navigate back and edit the TaskSchedules.qmd file to reflect these changes, and then run taskscheduler_create to create this third task.
Finally, after a task is run successfully, the instrument computer may produce a .Log file. These can be useful to validate that everything is running as expected. If however the names match on different computers, they may mess with the git commits similiar to if automated QC were run across all instruments at the same time, so I typically add them to the .gitignore file and only view them locally when doing manual QC or troubleshooting.
Conclusion
Both in the previous and current article, you have now modified the .R scripts and processed the existing QC data present on the individual computers. You have also now successfully got the data to transfer to the respective Instrument Archive data folders that are accessible with new daily updates from GitHub. Congratulations, you have made tremendous progress.
In the next chapter, you will start learning how to modify the .qmd files to create individual webpages for each instruments QC data, and then how to combine them into an actual website.