Understand Watchfolders and the Project Store
| Abbreviations Key | |
| CBC | complete blood count |
| CU | University of Colorado–Anschutz Medical Campus |
| EMR | electronic medical record |
| FH | Fred Hutchinson |
| HISE | Human Immune System Explorer |
| IDE | integrated development environment |
| PASC | postacute sequelae of COVID |
At a Glance
Watchfolders are used to upload data, such as lab results, survey data, and EMR data, and transfer it to HISE for analysis. The Project Store is used to store files you plan to work with later.
Descriptions
Watchfolder
A watchfolder is like a pickup spot for data that's ready to be ingested. Data does not persist in a watchfolder. If the data meets specified pipeline requirements, it's claimed by an automated pipeline that picks up and processes the data. For example, when sequencing files are dropped into a watchfolder, the automated sequencing pipeline is triggered. Likewise, Olink files trigger the Olink pipeline, and so on for each file type associated with an automated pipeline. The files then move to a storage bucket for analysis. Users can locate them there using an advanced search. Then they can download the files to their IDE for secondary analysis.
For file formatting instructions, see Use Watchfolders to Ingest Data .
Project Store
Every project has a project-specific, read-only storage space called a Project Store. You can use the Project Store UI to browse file contents, associate files with their corresponding sample references (if you didn't do so during ingestion), preview PDF and JPG files, move files directly into an IDE, mark files for deletion, or upload derived insights for storage.
If a watchfolder has been set up for the Project Store, it is either a Designated Watchfolder, in which all files go directly to the Project Store, or an "allowed file types" watchfolder, in which a file moves to the Project Store only if its type is consistent with one of the file types specified by the project administrator.
Access
Watchfolders
Watchfolders are set up by the HISE administrator on request. Not all projects have watchfolders. If your project has no watchfolders or you're not authorized to ingest data, you won't see any watchfolders or be able to create them. Check with your system administrator to see who is authorized to ingest data for your organization.
If a project does have a watchfolder, access is granted to a subset of project users. To see the available watchfolders for your account and projects, click your name in the upper-right corner of the screen. Then click Watch Folders. For watchfolder setup, contact immunology-support@alleninstitute.org.
Project Store
The Project Store is tied to a specified project. Every project user has access to the Project Store. You might also have access to a watchfolder to ingest data into a Project Store.
Data Ingestion
The goal of the ingest process is to move one or more of the following types of data into HISE:
- CBC results
- Survey results
- Demographic data
- Raw, unprocessed data for automated pipelines
- File data to be analyzed in an IDE
A HISE project for a given partner, such as the PASC project for FH, may have a watchfolder for data ingestion. The folder usually follows this naming convention: [ProjectName PartnerName Watchfolder] (for example, PASC FH Watchfolder). The folder can receive only the file types specified in the File Types column.
Data uploaded to a watchfolder is often picked up by an automated pipeline and moved to an analysis bucket. You can see the analysis label in the Storecolumn in the Files view in HISE. Most watchfolders accept only specific data types. (Wildcard watchfolders accept any file type.) This limitation prevents accidental ingestion of unwanted data into HISE and ensures that the right pipelines handle the ingested data. To view all available data pipelines, navigate to Data Processing > Pipeline Dashboard. You can filter your search by Status, Data stream, Batch ID, Panel ID, or Submission file.
When data is uploaded to a watchfolder, an ingest receipt is created. If the ingest is successful, the receipt shows the name of the source and target files, the partner code (for example, "BRI" or "CU"), ingest type, storage location of the source file, and the data and time the file was last modified. If the upload is not successful, the ingest receipt indicates the type of error. For example, if you upload a file type that doesn't match one of the configured file types for that folder, the ingest receipt shows a "Dismiss" error.