MAINT: Test against installed package

[larsoner]

Hopefully shows problem from mne-tools/mne-bids-pipeline#825 and mne-tools/mne-bids#1206

cc @hoechenberger @drammock @sappelhoff

[hoechenberger]

Nice, thanks @larsoner

x-ref #12281

[hoechenberger]

@larsoner
I think it's worthwhile considering switching to a src/ layout to further reduce the chances of erroneously using mne in the CWD instead of the installed version. The current layout also caused the entire "VS Code cannot handle editable installs anymore" thing to go unnoticed.

Thoughts?

[larsoner]

It sounds like a pretty big change and a departure from how most scientific python projects (numpy/scipy/pandas etc., but there are exceptions like matplotlib) are organized AFAIK. I'd rather move slower than faster here.

[drammock]

I think it's worthwhile considering switching to a src/ layout to further reduce the chances of erroneously using mne in the CWD instead of the installed version.
It sounds like a pretty big change and a departure from how most scientific python projects (numpy/scipy/pandas etc., but there are exceptions like matplotlib) are organized AFAIK. I'd rather move slower than faster here.

I'm actually less swayed by the "what does numpy do" heuristic here, perhaps because there are exceptions (pydata-sphinx-theme is another one). For me the relevant question is "is it likely to continue causing problems for us?" We've hit two big-ish and annoying problems, and now have fixes for both of them. The biggest cost may be that changing from mne/ to src/ will require our devs to update their muscle memory for actions that some of them make tens or hundreds of times per day. So I don't want to incur that cost unless we can foresee future problems. Might be worth asking for opinions on the scientific python discord or discourse?

[hoechenberger]

Suggested change

	if [ ! -z "$CONDA_ENV" ]; then # use installed verison
	if [ ! -z "$CONDA_ENV" ]; then # use installed version

[hoechenberger]

@larsoner Are you sure this is actually testing what we want here?

I'd much rather ship our tests with the sdist, rm -rf the checked-out repository, and run the tests from site-packages/mne just to be absolutely sure… No?

[larsoner]

@larsoner Are you sure this is actually testing what we want here?

We aren't, and I'm trying to figure out how to make it work but it's not trivial it seems

I'd much rather ship our tests with the sdist

Agreed we should do this, I think we used to so it's weird to me to exclude them now

rm -rf the checked-out repository, and run the tests from site-packages/mne just to be absolutely sure… No?

Yes this could work... except that we don't ship the test files with the sdist (or wheel) for space reasons. So we'd need to copy over the test data files to the installed location first.

[hoechenberger]

Let's include the test files with the sdist, then

I'd much rather ship our tests with the sdist

Agreed we should do this, I think we used to so it's weird to me to exclude them now

I excluded the tests because most wouldn't work anyway without the testing files.

rm -rf the checked-out repository, and run the tests from site-packages/mne just to be absolutely sure… No?

Yes this could work... except that we don't ship the test files with the sdist (or wheel) for space reasons. So we'd need to copy over the test data files to the installed location first.

Let's move the testing files to an external dataset

[hoechenberger]

FWIW including the testing files in the sdist makes it grow from 14 to 55 MB on my system. Nothing I would consider outrageous these days anymore.

[larsoner]

See also #4926, feel free to reopen if you see a good path forward