Use Watchfolders to Ingest Data (Tutorial)

Last updated 2026-08-31

At a Glance

Watchfolders are project-specific spaces used to transfer data to HISE. If the data is associated with an automated pipeline, it can be moved to the Project Store for analysis. (For ingestion of data that's not associated with an automated pipeline, see Use the Project Store to Ingest Data .) Contact immunology-support@alleninstitute.org for watchfolder setup or modification.

File Format

Label your ingestion files with the correct file type, such as `LabResults`, `TestResults`, or `clinical_labs`. If your files are not labeled or the label doesn't match the file type, a dismiss error appears on the ingest receipt.

You can create multiple file types (for example, `SurveyResults`) that contain different content types. The regex should include a filename. See the following table for examples, and refer to the boxed instructions to test your filename.

	csv filename	Regex
Lab results	`clinical_data` or `clinical_labs` or `labresults`	`(?i)(.((lab\|test)results).)\|(.((clinical)\_(labs\|data)).)\|(.(clinicaldata).).csv`
Survey data	`survey_data`, `survey_results`, or `surveyresults`	`(?i)(.((survey)results).)\|(.((survey)\_(survey\|data)).)\|(.(surveydata).).csv`
EMR data	`emr_data` or `emrdata`	`(?i)(.(emrdata).)\|(.((emr)\_(data)).).csv

Test Your Filename

1. In the upper-right corner of your screen, click your name.

2. Click Watchfolders.

3. Click the file type you plan to upload, such as OctetStream Lab Results (.csv).

4. Paste your proposed filename into the box, and press Enter.

If your Regex is properly constructed for the selected file type, a green X appears to the right of text entry field, and a link to the watchfolder appears below it. If the format is not correct, a red X appears.

NOTE

You can also use gcloud storage cp to copy files or folders from any computer with Google Cloud CLI access directly to a watchfolder.

To find the Google Cloud Storage bucket for a watch folder, click your email address, select Watch Folders from the drop-down list, locate the watchfolder you want to use, and click its name. HISE opens the associated Google Cloud Storage bucket. Copy the bucket name and replace WatchfolderBucket in the commands below.

To copy a local file to a watchfolder:
storage cp /path/to/YourFileName gs://WatchfolderBucket/

To copy from a cloud storage bucket to a watchfolder:
gcloud storage cp gs://SourceBucket/YourFileName gs://WatchfolderBucket/

To copy a directory and its contents:
gcloud storage cp --recursive /path/to/YourFolder gs://WatchfolderBucket/

Instructions

For special preparation of files containing human metadata, see the preceding section and consult your organization's data privacy policy or legal representative.

To upload properly prepared files of any kind, with or without human metadata, follow the process outlined below. If your data upload requires an automated process, contact us at immunology-support@alleninstitute.org to discuss the options.

1. To upload data to a watchfolder, navigate to HISE and log in with your organization's email account.

2. In the upper-right corner below your email address, make sure you're in the account you want to work with. If not, use the account favorites menu to move to a different account. For details, see Switch Your Account or Projects (Tutorial).

6. Click your email address and select Watch Folders from the drop-down list.

7. Choose the watchfolder for your account and project.

8. Click UPLOAD FILES, or drag and drop your files into the watchfolder. Note that files in Google Drive, such as csv files created in Google Sheets, can't be uploaded directly to your Project Store or dragged into it. Download the file first, and then upload it from your Downloads folder.

8. To see the status of your uploaded files, from the top navigation menu, choose Data Processing > Pipeline Dashboard > Ingest Receipts.

Related Resources

Understand Watchfolders and the Project Store

Ingest Data into the Project Store (Tutorial)