Attach Metadata to Make Files Searchable (Tutorial)

At a Glance

	Abbreviations Key
CSV	comma-separated values
cvf	create verbose filename
ELN	electronic lab notebook
GUID	globally unique identifier
gz	gzip
HISE	Human Immune System Explorer
IDE	integrated development environment
NA	not applicable
SLIMS	simplified laboratory information management system
tar	tape archive (Linux command)
UI	user interface

Automated pipelines let HISE generate result files with attached subject and sample metadata, standardizing data processing for each assay. Certain types of files, however, such as those containing experimental or pilot data, are unsuitable for pipeline ingestion and must be manually uploaded to a Project Store and then downloaded to an IDE. Metadata is not attached to these files.

There are two options for attaching metadata. The first, which we recommend for transparency, reproducibility, and searchability, is to associate metadata with your files at ingest. The second option is to attach the metadata later, after some experimentation. This document offers instructions for both options.

Finally, some projects require that specific demographics data be submitted through a watchfolder. For details, click the button.

Map a Demographics Scheme

Option 1: Attach Metadata at Ingest

The first way to add metadata is to attach it at ingest to files destined for a Project Store. Begin by creating a file manifest. Then tar up your files and ingest them into a watchfolder.

File manifest

The manifest lists all files to be ingested, along with their file types and sample references. To ensure that HISE recognizes the file, it must be named manifest.csv. 

Below is a sample manifest.csv file.

• Lines 1 and 2. The first two rows show the account and project to which the file data belongs. This information is not required but helps ensure that the metadata is associated with the correct account and project (for example, if the file is dropped into the wrong watchfolder).
• Line 3. The third line is a static header showing three columns: file, samples, and fileType. If a file is not associated with a sample, the special keyword reference can be used instead of samples.
• Subsequent lines. The remaining lines contain the actual metadata: each file name, the specific sample(s) with which it's associated, and the file type. Multiple samples are separated with a semicolon (;) as the delimiter.

Instructions

 Create a manifest

1. Navigate to SLIMS, and use your organizational email address to sign in.

2. From the top navigation menu, click the Content tab.

3. Use the filters and checkboxes to find and select the project for which you want to create a manifest file.

4. Remain on the Content tab, and click the ELN tab.

5. In the ELN section, follow the instructions below.

A. Select the content you want to work with.

B. An ELN notebook opens. 

C. Select the block with your content.

D. Click the compass icon.

6. In the Generate HISE Manifest box, choose the type of manifest best suited to your project. (To create a manifest.csv in SLIMS, you must be using a Simplified ELN experiment.)

7. In the lower-left corner, click Finish.

 Create a Tar File and Ingest It into a Watchfolder

After you create a manifest file, tar up the manifest with the files themselves. For instructions, see Step 2 of Ingest Data into the Project Store (Tutorial). When you drop the tar file into the watchfolder, HISE untars the files and uploads the data to the Project Store that's linked to the watchfolder. HISE also associates those files with the samples and file types declared in the manifest.

Option 2: Use the Project Store UI to Attach Metadata

The first method of adding metadata is to attach it at ingest (see Option 1). The second method is to attach the metadata later, after some experimentation. To use this latter method, follow the instructions below.

Instructions

 Choose your account and project

1. Navigate to HISE, and use your organizational email address to sign in.

2. To see the account you're working on, hover over your initials in the upper-right corner.

A. If it's the correct account, proceed to the next step.

B. If it's not the right account, use the following instructions to change it.

i. Click your initials.

ii. From the drop-down menu, choose Account.

iii. On the Account screen, click the Environment card.

iv. From the Accounts tab on the Configure HISE Environment screen, scroll to the Available Accounts list, and click the arrow at right.

v. From the drop-down list, choose the correct account. 

3. To select the projects you're working on, remain on the Accounts tab of the Configure HISE Environment screen.
Below the Available Accounts menu, in the Available Projects section, select the projects you want to work with.

Add metadata

1. Return to the Account screen, and click the Project Stores card.

2. On the Project Stores screen, in the second column, click the name of the project you're working with.

3. To include all files in the watchfolder, select the checkbox at the top of the far-left column.

4. In the upper-right corner of your screen, click Add file metadata.

5. In the Add file metadata dialog box, use the arrows to select the File type, Sample kit GUID, and Batch ID, as shown in the following image.

6. Click Submit.

Related Resources

Create, Edit, or Delete Metadata

Understand Automated Pipelines

Use Specified Metadata Models (Tutorial)