Note
Go to the end to download the full example code
Run pyLossless on a BIDS dataset.¶
In this notebook, we will run the pyLossless pipeline on a publicly available dataset.
Imports¶
from pathlib import Path
import shutil
import pylossless as ll
Get the data¶
👋 Hello! This is openneuro-py 2024.1.0. Great to see you! 🤗
👉 Please report problems 🤯 and bugs 🪲 at
https://github.com/hoechenberger/openneuro-py/issues
🌍 Preparing to download ds002778 …
📁 Traversing directories for ds002778 : 0 entities [00:00, ? entities/s]
📁 Traversing directories for ds002778 : 7 entities [00:05, 1.18 entities/s]
📁 Traversing directories for ds002778 : 8 entities [00:06, 1.31 entities/s]
📁 Traversing directories for ds002778 : 10 entities [00:06, 1.83 entities/s]
📁 Traversing directories for ds002778 : 14 entities [00:06, 3.27 entities/s]
📁 Traversing directories for ds002778 : 16 entities [00:06, 3.97 entities/s]
📁 Traversing directories for ds002778 : 18 entities [00:07, 4.77 entities/s]
📁 Traversing directories for ds002778 : 20 entities [00:07, 2.76 entities/s]
📥 Retrieving up to 19 files (5 concurrent downloads).
participants.tsv: 0%| | 0.00/1.62k [00:00<?, ?B/s]
participants.tsv: 100%|██████████| 1.62k/1.62k [00:00<00:00, 7.36kB/s]
participants.json: 0%| | 0.00/1.24k [00:00<?, ?B/s]
sub-pd6_ses-off_task-rest_eeg.bdf: 0%| | 0.00/11.5M [00:00<?, ?B/s]
README: 0.00B [00:00, ?B/s]
CHANGES: 0.00B [00:00, ?B/s]
dataset_description.json: 0.00B [00:00, ?B/s]
sub-pd6_ses-off_task-rest_eeg.bdf: 0%| | 50.5k/11.5M [00:00<00:34, 344kB/s]
sub-pd6_ses-off_task-rest_eeg.bdf: 4%|▍ | 457k/11.5M [00:00<00:05, 2.21MB/s]
sub-pd6_ses-off_scans.tsv: 0%| | 0.00/75.0 [00:00<?, ?B/s]
sub-pd6_ses-off_task-rest_eeg.bdf: 10%|▉ | 1.11M/11.5M [00:00<00:02, 3.94MB/s]
sub-pd6_ses-off_task-rest_beh.tsv: 0%| | 0.00/10.0 [00:00<?, ?B/s]
sub-pd6_ses-off_task-rest_beh.json: 0.00B [00:00, ?B/s]
sub-pd6_ses-off_task-rest_channels.tsv: 0%| | 0.00/2.22k [00:00<?, ?B/s]
sub-pd6_ses-off_task-rest_eeg.bdf: 13%|█▎ | 1.53M/11.5M [00:00<00:04, 2.57MB/s]
sub-pd6_ses-off_task-rest_events.tsv: 0%| | 0.00/66.0 [00:00<?, ?B/s]
sub-pd6_ses-off_task-rest_eeg.bdf: 18%|█▊ | 2.02M/11.5M [00:00<00:03, 3.19MB/s]
sub-pd6_ses-on_task-rest_beh.tsv: 0%| | 0.00/10.0 [00:00<?, ?B/s]
sub-pd6_ses-off_task-rest_eeg.json: 0%| | 0.00/471 [00:00<?, ?B/s]
sub-pd6_ses-on_scans.tsv: 0%| | 0.00/74.0 [00:00<?, ?B/s]
sub-pd6_ses-off_task-rest_eeg.bdf: 21%|██ | 2.40M/11.5M [00:00<00:03, 2.52MB/s]
sub-pd6_ses-on_task-rest_beh.json: 0%| | 0.00/433 [00:00<?, ?B/s]
sub-pd6_ses-off_task-rest_eeg.bdf: 34%|███▎ | 3.86M/11.5M [00:01<00:01, 4.94MB/s]
sub-pd6_ses-on_task-rest_events.tsv: 0%| | 0.00/51.0 [00:00<?, ?B/s]
sub-pd6_ses-on_task-rest_eeg.json: 0%| | 0.00/471 [00:00<?, ?B/s]
sub-pd6_ses-on_task-rest_eeg.bdf: 0%| | 0.00/17.4M [00:00<?, ?B/s]
sub-pd6_ses-on_task-rest_channels.tsv: 0%| | 0.00/2.22k [00:00<?, ?B/s]
sub-pd6_ses-off_task-rest_eeg.bdf: 54%|█████▎ | 6.17M/11.5M [00:01<00:00, 9.25MB/s]
sub-pd6_ses-on_task-rest_eeg.bdf: 4%|▍ | 713k/17.4M [00:00<00:02, 7.27MB/s]
sub-pd6_ses-off_task-rest_eeg.bdf: 71%|███████ | 8.12M/11.5M [00:01<00:00, 12.0MB/s]
sub-pd6_ses-on_task-rest_eeg.bdf: 12%|█▏ | 2.15M/17.4M [00:00<00:01, 11.9MB/s]
sub-pd6_ses-off_task-rest_eeg.bdf: 88%|████████▊ | 10.1M/11.5M [00:01<00:00, 14.3MB/s]
sub-pd6_ses-on_task-rest_eeg.bdf: 24%|██▍ | 4.19M/17.4M [00:00<00:00, 16.2MB/s]
sub-pd6_ses-on_task-rest_eeg.bdf: 41%|████ | 7.15M/17.4M [00:00<00:00, 22.0MB/s]
sub-pd6_ses-on_task-rest_eeg.bdf: 64%|██████▍ | 11.1M/17.4M [00:00<00:00, 28.9MB/s]
sub-pd6_ses-on_task-rest_eeg.bdf: 87%|████████▋ | 15.1M/17.4M [00:00<00:00, 33.5MB/s]
✅ Finished downloading ds002778.
🧠 Please enjoy your brains.
Prep the Raw object¶
This data has EOG channels that are not labeled as such. We will manually set the channel types to be “eog” for these channels (i.e. “EXG1”). We will also crop the data to 60 seconds for speed, and load the data in memory, which is required for running the pipeline.
raw.set_channel_types({ch: "eog" for ch in raw.ch_names if ch.startswith("EX")})
raw.load_data().crop(0, 60)
Reading 0 ... 97791 = 0.000 ... 190.998 secs...
Initialize the pipeline¶
The LosslessPipeline instance is the main object that will
run the pipeline. It takes a file path to a Config object
as input. load_openneuro_bids() returned a
Config object, so we will save it to disk and pass the
file path to the LosslessPipeline constructor.
config_path = Path("lossless_config.yaml")
config["filtering"]["notch_filter_args"]["freqs"] = [60]
config.save(config_path)
pipeline = ll.LosslessPipeline(config_path)
Run the pipeline¶
The LosslessPipeline object has a
run_with_raw() method that takes a
Raw object as input.
We will use the Raw object that was returned by
load_openneuro_bids() with the pipeline.
⏩ LOSSLESS: Starting Pylossless Pipeline.
LOSSLESS: Skipping Looking for break periods between tasks
LOSSLESS: 👇 Flagging Noisy Channels.
🧹 Epoching..
Not setting metadata
60 matching events found
No baseline correction applied
0 projection items activated
Using data from preloaded Raw for 60 events and 512 original time points ...
0 bad epochs dropped
🔍 Detecting channels to leave out of reference.
/home/docs/checkouts/readthedocs.org/user_builds/pylossless/checkouts/latest/pylossless/pipeline.py:61: FutureWarning: The current default of copy=False will change to copy=True in 1.7. Set the value of copy explicitly to avoid this warning
data = epochs.get_data() # n_epochs, n_channels, n_times
EEG channel type selected for re-referencing
Applying a custom ('EEG',) reference.
/home/docs/checkouts/readthedocs.org/user_builds/pylossless/checkouts/latest/pylossless/pipeline.py:61: FutureWarning: The current default of copy=False will change to copy=True in 1.7. Set the value of copy explicitly to avoid this warning
data = epochs.get_data() # n_epochs, n_channels, n_times
📋 LOSSLESS: Noisy channels: []
LOSSLESS: 🏁 Finished Flagging Noisy Channels after 0.12 seconds.
LOSSLESS: 👇 Flagging Noisy Time periods.
🧹 Epoching..
Not setting metadata
60 matching events found
No baseline correction applied
0 projection items activated
Using data from preloaded Raw for 60 events and 512 original time points ...
0 bad epochs dropped
🔍 Detecting channels to leave out of reference.
/home/docs/checkouts/readthedocs.org/user_builds/pylossless/checkouts/latest/pylossless/pipeline.py:61: FutureWarning: The current default of copy=False will change to copy=True in 1.7. Set the value of copy explicitly to avoid this warning
data = epochs.get_data() # n_epochs, n_channels, n_times
EEG channel type selected for re-referencing
Applying a custom ('EEG',) reference.
/home/docs/checkouts/readthedocs.org/user_builds/pylossless/checkouts/latest/pylossless/pipeline.py:61: FutureWarning: The current default of copy=False will change to copy=True in 1.7. Set the value of copy explicitly to avoid this warning
data = epochs.get_data() # n_epochs, n_channels, n_times
📋 LOSSLESS: 1.0 second(s) flagged as BAD_LL_noisy
LOSSLESS: 🏁 Finished Flagging Noisy Time periods after 0.11 seconds.
LOSSLESS: 👇 Filtering.
Filtering raw data in 1 contiguous segment
Setting up band-pass filter from 1 - 1e+02 Hz
FIR filter parameters
---------------------
Designing a one-pass, zero-phase, non-causal bandpass filter:
- Windowed time-domain design (firwin) method
- Hamming window with 0.0194 passband ripple and 53 dB stopband attenuation
- Lower passband edge: 1.00
- Lower transition bandwidth: 1.00 Hz (-6 dB cutoff frequency: 0.50 Hz)
- Upper passband edge: 100.00 Hz
- Upper transition bandwidth: 25.00 Hz (-6 dB cutoff frequency: 112.50 Hz)
- Filter length: 1691 samples (3.303 s)
[Parallel(n_jobs=1)]: Done 17 tasks | elapsed: 0.0s
Filtering raw data in 1 contiguous segment
Setting up band-stop filter from 59 - 61 Hz
FIR filter parameters
---------------------
Designing a one-pass, zero-phase, non-causal bandstop filter:
- Windowed time-domain design (firwin) method
- Hamming window with 0.0194 passband ripple and 53 dB stopband attenuation
- Lower passband edge: 59.35
- Lower transition bandwidth: 0.50 Hz (-6 dB cutoff frequency: 59.10 Hz)
- Upper passband edge: 60.65 Hz
- Upper transition bandwidth: 0.50 Hz (-6 dB cutoff frequency: 60.90 Hz)
- Filter length: 3381 samples (6.604 s)
[Parallel(n_jobs=1)]: Done 17 tasks | elapsed: 0.0s
Filtering raw data in 1 contiguous segment
Setting up band-stop filter from 59 - 61 Hz
FIR filter parameters
---------------------
Designing a one-pass, zero-phase, non-causal bandstop filter:
- Windowed time-domain design (firwin) method
- Hamming window with 0.0194 passband ripple and 53 dB stopband attenuation
- Lower passband edge: 59.35
- Lower transition bandwidth: 0.50 Hz (-6 dB cutoff frequency: 59.10 Hz)
- Upper passband edge: 60.65 Hz
- Upper transition bandwidth: 0.50 Hz (-6 dB cutoff frequency: 60.90 Hz)
- Filter length: 3381 samples (6.604 s)
[Parallel(n_jobs=1)]: Done 17 tasks | elapsed: 0.0s
LOSSLESS: 🏁 Finished Filtering after 0.22 seconds.
LOSSLESS: 👇 Flagging uncorrelated channels.
🧹 Epoching..
Not setting metadata
60 matching events found
No baseline correction applied
0 projection items activated
Using data from preloaded Raw for 60 events and 512 original time points ...
1 bad epochs dropped
🔍 Detecting channels to leave out of reference.
/home/docs/checkouts/readthedocs.org/user_builds/pylossless/checkouts/latest/pylossless/pipeline.py:61: FutureWarning: The current default of copy=False will change to copy=True in 1.7. Set the value of copy explicitly to avoid this warning
data = epochs.get_data() # n_epochs, n_channels, n_times
EEG channel type selected for re-referencing
Applying a custom ('EEG',) reference.
0%| | 0/32 [00:00<?, ?it/s]
16%|█▌ | 5/32 [00:00<00:00, 49.39it/s]
31%|███▏ | 10/32 [00:00<00:00, 49.62it/s]
47%|████▋ | 15/32 [00:00<00:00, 48.79it/s]
66%|██████▌ | 21/32 [00:00<00:00, 49.59it/s]
84%|████████▍ | 27/32 [00:00<00:00, 50.05it/s]
100%|██████████| 32/32 [00:00<00:00, 49.89it/s]
📋 LOSSLESS: Uncorrelated channels: []
LOSSLESS: 🏁 Finished Flagging uncorrelated channels after 0.81 seconds.
LOSSLESS: 👇 Flagging Bridged channels.
📋 LOSSLESS: Bridged channels: ['PO3' 'O1' 'Oz' 'O2' 'PO4']
LOSSLESS: 🏁 Finished Flagging Bridged channels after 0.01 seconds.
LOSSLESS: 👇 Flagging the rank channel.
📋 LOSSLESS: Rank channel: ['Fp2']
LOSSLESS: 🏁 Finished Flagging the rank channel after 0.01 seconds.
LOSSLESS: 👇 Flagging Uncorrelated epochs.
🧹 Epoching..
Not setting metadata
60 matching events found
No baseline correction applied
0 projection items activated
Using data from preloaded Raw for 60 events and 512 original time points ...
1 bad epochs dropped
🔍 Detecting channels to leave out of reference.
/home/docs/checkouts/readthedocs.org/user_builds/pylossless/checkouts/latest/pylossless/pipeline.py:61: FutureWarning: The current default of copy=False will change to copy=True in 1.7. Set the value of copy explicitly to avoid this warning
data = epochs.get_data() # n_epochs, n_channels, n_times
EEG channel type selected for re-referencing
Applying a custom ('EEG',) reference.
0%| | 0/26 [00:00<?, ?it/s]
23%|██▎ | 6/26 [00:00<00:00, 49.95it/s]
46%|████▌ | 12/26 [00:00<00:00, 50.06it/s]
69%|██████▉ | 18/26 [00:00<00:00, 50.01it/s]
92%|█████████▏| 24/26 [00:00<00:00, 49.33it/s]
100%|██████████| 26/26 [00:00<00:00, 49.64it/s]
LOSSLESS: 🏁 Finished Flagging Uncorrelated epochs after 0.66 seconds.
LOSSLESS: 👇 Running Initial ICA.
🧹 Epoching..
Not setting metadata
60 matching events found
No baseline correction applied
0 projection items activated
Using data from preloaded Raw for 60 events and 512 original time points ...
1 bad epochs dropped
🔍 Detecting channels to leave out of reference.
/home/docs/checkouts/readthedocs.org/user_builds/pylossless/checkouts/latest/pylossless/pipeline.py:61: FutureWarning: The current default of copy=False will change to copy=True in 1.7. Set the value of copy explicitly to avoid this warning
data = epochs.get_data() # n_epochs, n_channels, n_times
EEG channel type selected for re-referencing
Applying a custom ('EEG',) reference.
Fitting ICA to data using 26 channels (please be patient, this may take a while)
Selecting by non-zero PCA components: 25 components
/home/docs/checkouts/readthedocs.org/user_builds/pylossless/envs/latest/lib/python3.10/site-packages/sklearn/decomposition/_fastica.py:128: ConvergenceWarning: FastICA did not converge. Consider increasing tolerance or the maximum number of iterations.
warnings.warn(
Fitting ICA took 19.0s.
LOSSLESS: 🏁 Finished Running Initial ICA after 19.14 seconds.
LOSSLESS: 👇 Flagging time periods with noisy IC's..
🧹 Epoching..
Not setting metadata
60 matching events found
No baseline correction applied
0 projection items activated
Using data from preloaded Raw for 60 events and 512 original time points ...
1 bad epochs dropped
🔍 Detecting channels to leave out of reference.
/home/docs/checkouts/readthedocs.org/user_builds/pylossless/checkouts/latest/pylossless/pipeline.py:61: FutureWarning: The current default of copy=False will change to copy=True in 1.7. Set the value of copy explicitly to avoid this warning
data = epochs.get_data() # n_epochs, n_channels, n_times
EEG channel type selected for re-referencing
Applying a custom ('EEG',) reference.
/home/docs/checkouts/readthedocs.org/user_builds/pylossless/checkouts/latest/pylossless/pipeline.py:64: FutureWarning: The current default of copy=False will change to copy=True in 1.7. Set the value of copy explicitly to avoid this warning
data = ica.get_sources(epochs).get_data()
📋 LOSSLESS: 3.99 second(s) flagged as BAD_LL_noisy_ICs
LOSSLESS: 🏁 Finished Flagging time periods with noisy IC's. after 0.20 seconds.
LOSSLESS: 👇 Running Final ICA and ICLabel..
🧹 Epoching..
Not setting metadata
60 matching events found
No baseline correction applied
0 projection items activated
Using data from preloaded Raw for 60 events and 512 original time points ...
5 bad epochs dropped
🔍 Detecting channels to leave out of reference.
/home/docs/checkouts/readthedocs.org/user_builds/pylossless/checkouts/latest/pylossless/pipeline.py:61: FutureWarning: The current default of copy=False will change to copy=True in 1.7. Set the value of copy explicitly to avoid this warning
data = epochs.get_data() # n_epochs, n_channels, n_times
EEG channel type selected for re-referencing
Applying a custom ('EEG',) reference.
Fitting ICA to data using 26 channels (please be patient, this may take a while)
Selecting by non-zero PCA components: 25 components
Computing Extended Infomax ICA
Fitting ICA took 10.4s.
NOTE: pick_channels() is a legacy function. New code should use inst.pick(...).
LOSSLESS: 🏁 Finished Running Final ICA and ICLabel. after 16.03 seconds.
✅ LOSSLESS: Pipeline completed! took 0.62 minutes.
View the results¶
The LosslessPipeline object stores flagged channels and ICs in
the flags attribute:
pipeline.flags["ic"]
Get the cleaned data¶
The LosslessPipeline by default does not modify the
Raw object that is passed to it, so none of the flagged channels
or ICs are removed from the Raw object yet. To get the cleaned
Raw object, we need to call the
apply() method. This method takes a
LosslessPipeline as input, which specifies how to apply the flags
to generate a new Raw object.
RejectionPolicy: |
config_fname: None
ch_flags_to_reject: ['noisy', 'uncorrelated', 'bridged']
ic_flags_to_reject: ['muscle', 'ecg', 'eog', 'channel_noise', 'line_noise']
ic_rejection_threshold: 0.3
ch_cleaning_mode: None
remove_flagged_ics: True
Note that we are using the default channel cleaning mode, which is None, meaning
that that the flagged channels will simply be added to raw.info["bads"]. We also
could have specified "interpolate", which means that the flagged
channels would be interpolated using interpolate_bads().
cleaned_raw = rejection_policy.apply(pipeline)
cleaned_raw.plot()
Applying ICA to Raw instance
Transforming to ICA space (25 components)
Zeroing out 7 ICA components
Projecting back using 26 PCA components
Using matplotlib as 2D backend.
<MNEBrowseFigure size 800x800 with 4 Axes>
Save the PyLossless Derivative¶
Let’s save our pipeline output to disk. We need to use our
BIDSPath object to set up a derivatives path to save the
pipeline output to:
BIDSPath(
root: ds002778/derivatives/pylossless
datatype: eeg
basename: sub-pd6_ses-off_task-rest_eeg_ll)
Writing 'ds002778/derivatives/pylossless/README'...
Writing 'ds002778/derivatives/pylossless/participants.tsv'...
Writing 'ds002778/derivatives/pylossless/participants.json'...
Writing 'ds002778/derivatives/pylossless/sub-pd6/ses-off/eeg/sub-pd6_ses-off_space-CapTrak_electrodes.tsv'...
Writing 'ds002778/derivatives/pylossless/sub-pd6/ses-off/eeg/sub-pd6_ses-off_space-CapTrak_coordsystem.json'...
The provided raw data contains annotations, but you did not pass an "event_id" mapping from annotation descriptions to event codes. We will generate arbitrary event codes. To specify custom event codes, please pass "event_id".
Used Annotations descriptions: ['1', '65536', 'BAD_LL_noisy', 'BAD_LL_noisy_ICs']
Writing 'ds002778/derivatives/pylossless/sub-pd6/ses-off/eeg/sub-pd6_ses-off_task-rest_events.tsv'...
Writing 'ds002778/derivatives/pylossless/sub-pd6/ses-off/eeg/sub-pd6_ses-off_task-rest_events.json'...
Writing 'ds002778/derivatives/pylossless/dataset_description.json'...
Writing 'ds002778/derivatives/pylossless/sub-pd6/ses-off/eeg/sub-pd6_ses-off_task-rest_eeg.json'...
Writing 'ds002778/derivatives/pylossless/sub-pd6/ses-off/eeg/sub-pd6_ses-off_task-rest_channels.tsv'...
Copying data files to sub-pd6_ses-off_task-rest_eeg.edf
/home/docs/checkouts/readthedocs.org/user_builds/pylossless/checkouts/latest/pylossless/pipeline.py:1041: RuntimeWarning: Converting data files to EDF format
mne_bids.write_raw_bids(
/home/docs/checkouts/readthedocs.org/user_builds/pylossless/checkouts/latest/pylossless/pipeline.py:1041: RuntimeWarning: EDF format requires equal-length data blocks, so 0.998046875 seconds of zeros were appended to all channels when writing the final block.
mne_bids.write_raw_bids(
Writing 'ds002778/derivatives/pylossless/sub-pd6/ses-off/sub-pd6_ses-off_scans.tsv'...
Wrote ds002778/derivatives/pylossless/sub-pd6/ses-off/sub-pd6_ses-off_scans.tsv entry with eeg/sub-pd6_ses-off_task-rest_eeg.edf.
Writing ICA solution to /home/docs/checkouts/readthedocs.org/user_builds/pylossless/checkouts/latest/examples/ds002778/derivatives/pylossless/sub-pd6/ses-off/eeg/sub-pd6_ses-off_task-rest_ica1_ica.fif...
Writing ICA solution to /home/docs/checkouts/readthedocs.org/user_builds/pylossless/checkouts/latest/examples/ds002778/derivatives/pylossless/sub-pd6/ses-off/eeg/sub-pd6_ses-off_task-rest_ica2_ica.fif...
Clean up¶
Total running time of the script: (0 minutes 51.433 seconds)