Data Standardization for EMBER-DANDI and EMBER-Vault

This guide walks you through a suggested procedure on how to prepare your dataset for upload using the BIDS standard, with support for intracranial recordings, audio/video, and behavioral data, and optional use of NWB files.

This covers:

The relevant standards
The standardization process and examples
BIDS Standard
Human Audio/Video
Validation
Common Pitfalls
Recommended Workflow

1. Relevant Data Standards

Before formatting your dataset, familiarize yourself with the standards supported by the BBQS Program and the EMBER archive (defined by the BBQS DCAIC Data Standards Working Group):

https://docs.google.com/document/d/1vIJ01La9G76FfGywS3IbG4o1GR4qquS31MoJ2B4_4os/edit?usp=sharing

A quick summary relevant to sensitive, multimodal human datasets (such as audio/video of participants) is outlined below.

Core BIDS

BIDS specification: https://bids.neuroimaging.io/
Defines folder structure, file naming, file types and metadata conventions for different data types

BIDS Extensions Supported

Use for:

Depth electrodes
Microelectrode arrays
Spike/LFP recordings

Audio / Video / Behavioral

BEP047 (motion, audio, video)

https://bids.neuroimaging.io/extensions/beps/bep_047.html

Use for:

Video recordings (e.g., behavior, eye tracking)
Audio recordings (speech, environment)
Motion capture

NWB (Neurodata Without Borders)

https://www.nwb.org/

Recommended for:

Spike times
LFP
Rich electrophysiology metadata

Important: If using NWB files, we recommend the files should still be organized within a BIDS structure for sensitive human data.

2. Standardization Workflow

Preparing your dataset involves three main steps:

Step 1: Organize Files into BIDS Structure

Create folders and filenames according to BIDS rules
Convert to allowable filetypes if needed

Step 2: Add Metadata

JSON sidecars
TSV files (events, channels, electrodes, etc.)
Advise to use LinkML to create/describe their metadata. For advanced: point to complete systems like from Juelich
Advise to use established lab notebook systems to formalize

Step 3: Validate Locally

Use the dandi-cli (which used the NWB inspector, BIDS Validator (https://bids.neuroimaging.io/tools/validator.html) etc) before uploading.
Please see the instructions here: https://github.com/dandi/dandi-cli

Step 4: Upload via Globus

Please see the instructions here: Upload to EMBER-Vault via Globus

3. BIDS Standard

1. Folder Structure (Core BIDS Template)

BIDS specifies a particular set of folders and file names to organize data:

https://bids.neuroimaging.io/getting_started/folders_and_files/index.html

More examples will be made available at:

https://github.com/brain-bbqs

Here is a minimal multimodal dataset layout as an example, including iEEG in NWB formats and audio and video files:

my_dataset/
├── dataset_description.json
├── participants.tsv
├── participants.json
├── sub-01/
│   ├── ses-01/
│   │   ├── ieeg/
│   │   │   ├── sub-01_ses-01_task-rest_ieeg.nwb
│   │   │   ├── sub-01_ses-01_task-rest_ieeg.json
│   │   │   ├── sub-01_ses-01_task-rest_channels.tsv
│   │   │   ├── sub-01_ses-01_task-rest_electrodes.tsv
│   │   │   └── sub-01_ses-01_task-rest_events.tsv
│   │   │
│   │   ├── beh/
│   │   │   ├── sub-01_ses-01_task-rest_beh.tsv
│   │   │   └── sub-01_ses-01_task-rest_beh.json
│   │   │
│   │   ├── video/
│   │   │   ├── sub-01_ses-01_task-rest_video.mp4
│   │   │   └── sub-01_ses-01_task-rest_video.json
│   │   │
│   │   └── audio/
│   │       ├── sub-01_ses-01_task-rest_audio.wav
│   │       └── sub-01_ses-01_task-rest_audio.json

2. BIDS File Naming Rules

As you go, please be sure to refer to the BIDS examples: https://github.com/bids-standard/bids-examples

BIDS uses structured filenames:

sub-<participant>[_ses-<session>]_task-<task>[_modality]

Examples:

sub-01_ses-01_task-rest_ieeg.nwb
sub-01_ses-01_task-speaking_audio.wav
sub-01_ses-01_task-walk_video.mp4

3. Required Metadata Files for BIDS

3.1 dataset_description.json

{
  "Name": "Example Multimodal Dataset",
  "BIDSVersion": "1.9.0",
  "DatasetType": "raw",
  "Authors": ["Jane Doe", "John Smith"]
}

3.2 participants.tsv

participant_id  age sex
sub-01  29  M
sub-02  34  F

3.3 iEEG Metadata (BEP032)

channels.tsv

name    type    units   sampling_frequency
chan01  microelectrode  uV  30000

electrodes.tsv

name    x   y   z
elec01  12.3    -45.2   30.1

3.4 events.tsv

onset   duration    trial_type
0.0 1.0 stimulus
2.0 1.5 response

3.5 Sidecar JSON Example

For an iEEG file:

{
  "SamplingFrequency": 30000,
  "PowerLineFrequency": 60,
  "iEEGReference": "common average",
  "TaskName": "rest"
}

4. Using NWB within BIDS

If using NWB files within your project, they can be used to represent key time series data within BIDS, such as iEEG.

Recommended approach

Store electrophysiology data as .nwb files
Keep behavioral, events, and metadata in BIDS sidecars
Organize the dataset with NWB files within the BIDS structure, getting the best of both worlds

Example:

sub-01_ses-01_task-rest_ieeg.nwb
sub-01_ses-01_task-rest_events.tsv
sub-01_ses-01_task-rest_ieeg.json

When combining these standards, our key recommendations are:

NWB = data container
BIDS = dataset organization + metadata standard

4. Human Audio / Video (BEP047)

Supported formats:

Video: mp4, avi
Audio: wav

Example sidecar:

{
  "SamplingFrequency": 48000,
  "Microphone": "Shure SM7B"
}

For video:

{
  "FrameRate": 30,
  "Resolution": "1920x1080"
}

5. Validation (Required Before Upload)

To use EMBERvault, you must validate your dataset locally. The EMBERvault system is intended to store standardized data for reuse but does not currently format and validate data in-platform.

Command Line Validation for BIDS (Recommended)

You can find detailed instructions on use of the BIDS validator here:

https://bids.neuroimaging.io/tools/validator.html

We do not recommend using the online data validator for datasets containing PHI/PII.

https://bids-validator.readthedocs.io/en/latest/user_guide/command-line.html

Install

(Use the Mac Terminal, Linux Command Line, or Windows Subsystem for Linux)

Using Deno:

https://docs.deno.com/runtime/getting_started/installation/

deno install -ERWN -g -n bids-validator jsr:@bids/validator

Run

deno run -ERWN jsr:@bids/validator <dataset>

When using an extension, you must use the flag --schema with the link to the pull request specific to the extension.

BEP032

--schema https://bids-specification--2307.org.readthedocs.build/en/2307/schema.json

BEP047

--schema https://bids-specification--2231.org.readthedocs.build/en/2231/schema.json

Expected Output

No errors → ready to upload
Warnings → usually acceptable but should be reviewed
Errors → must fix before upload to EMBER-Vault

6. Common Pitfalls

Incorrect filenames

Missing task- or sub- labels

Missing JSON sidecars

Especially for iEEG and video/audio

Inconsistent sampling rates

Must match metadata

Time misalignment

Ensure events align across modalities

If your team is struggling, the EMBER archive team can help. Reach out at info@emberarchive.org.

7. Recommended Workflow (Step-by-Step)

Prepare raw data
- Export from acquisition systems
- Convert electrophysiology to NWB if appropriate, convert other data types to appropriate BIDS formats
Create folder structure
```
mkdir -p my_dataset/sub-01/ses-01/ieeg
```
Rename files to BIDS format
Create metadata files
- dataset_description.json
- participants.tsv
- sidecars
Add events + behavioral data
Validate BIDS dataset
Fix errors
Upload to platform