Skip to content

Weekly MOABB meeting

Jump to bottom
Sylvain Chevallier edited this page Mar 12, 2021 · 24 revisions

  • Thursday 3rd December

  • Thursday 10th December

    1. Open discussions:
      • Making a leaderboard, publicly available on the wiki, as in paper with code. What process to report score? see The Ladder paper
    2. Enhancements
  • Thursday 14th January
    1. Blocking issues:
    2. Enhancements
      • Adding SSVEP datasets: almost okay, 2 datasets ok. Need to update classes for MAMEM datasets (200+ electrodes)
      • Additional information in results (PR #127): almost ok, need review
      • Learning curve: evaluate accuracy while varying the number of trials. First draft.
      • Transfer learning: discussions in issue.
    3. Open discussions:
      • Google summer of code
      • Sebastien: run ML/autoML competitions, interest in transfer learning, few shot learning, cross-dataset, cross-hardware, cross-paradigm. Could be interesting to run a meta-learning challenge, in a RAMP like fashion maybe.
      • How to compare fairly different tasks, with different number of class, research questions close to meta-learning
  • Thursday 21st January
    1. Open discussion:
      • Morgan Hough: asking financial support from a company to support cloud computing time for processing data. Also computational power available to run evaluations soon.
      • New techniques in ML: evaluation on EEG
      • BCI society meeting on ML, check for recording available online
      • Adding support to affective computing, and emotion recognition, as more and more databases are available such as DEAP or GAMEEMO datasets (with GAMEEMO paper)
    2. Code update:
      • Additional columns PR # 127: last review and merge
      • Learning curve on various trial number: cross validation first, select subset of trials (in percentage or in number of sample per classes), repeat multiple times (permutation)
      • open a PR on MNE for FTP download
    3. Issues
      • Explain MOABB philosophy, regarding the idea to restrict the number of parameters to ease the comparison of algorithm. For example, the number of k-fold validation is set to 5 to ensure reproducible and accurate results.
      • Divyesh: open a new discussion for cVEP paradigm, as there is data available
      • How to start contributing: adding examples and running some pipelines to contribute on the leaderboard
  • Thursday 4th February
  • Thursday 11th February
  • Thursday 18th February
  • Thursday 25th February
  • Thursday 4th March
  • Thursday 11th March
    1. Open discussion:
      • On PR #132, we discussed about how to compare different classifiers when the number of samples is different on different datasets. Should we ensure that the same number of trials is used for each dataset or does that make sense to compare just a proportion of the total number of trials. One interesting alternative is to measure the EEG recording time for training, and not the number of samples. While it could be computed simply for MI and SSVEP, this is much a challenge for P300 due to overlaping stimulation.
      • A follow-up discussion on issue #146 regarding the use of poetry to handle dependencies and PyPi export. One major interest is that poetry takes care of the differences regarding dependencies for several plateform (linux, windows, osx). This is a important thing to ensure reproductibility. Also it is possible to handle separately dependencies for usage and for dev, as well as specific dependencies, like exotic libs or multiple choices.
      • If we push MOABB on PyPi, using poetry, we will need to set up a guideline regarding version number. We will follow a semantic versioning scheme.
      • We should discuss during next meeting how to handle datasets, having a distributed storage and a automated check, verifying that download url are up.
    2. Code update:
      • Following the pre-commit PR introducing black, we have now changed all the codebase to black formatting PR #147
      • Thanks to Mohammad Mostafa Farzan, we have corrected the issue #138 regarding h5py.
    3. Issues:
      • We have updated issue #121 about updating README page. The how to contribute section should mention that we use pre-commit with isort and black.
      • The documentation is not up to date and we should have a setup that allows to build it automatically.
      • There are still configuration regarding old CI (TravisCI) and a universal flag (indicating python 2 and 3 compatibility) in setup.cfg that should be removed.
      • We should correct the unit tests to allow a migration to python 3.8, as it work quite well with newer python version even if they are not officially supported.
Clone this wiki locally