docs: Settings Simplification ADR #36224

kdmccormick · 2025-02-05T18:00:41Z

https://github.com/kdmccormick/edx-platform/blob/kdmccormick/settings-adr/docs/decisions/0022-settings-simplification.rst

kdmccormick · 2025-02-10T22:22:41Z

@feanil , could you take a first pass? Once it looks good to you, I'll circulate it more broadly.

feanil

A few minor comments but nothing major. I can re-review once those are addressed.

docs/decisions/0022-settings-simplification.rst

kdmccormick · 2025-02-12T18:50:08Z

@regisb , your wisdom on this ADR would be invaluable.

docs/decisions/0022-settings-simplification.rst

kdmccormick · 2025-02-12T20:04:43Z

Sorry @feanil , forgot to push. Ready now.

feanil

Looks good to me now.

regisb

Reading this ADR, I found myself nodding and mumbling "yes, yes, ok..." all the way down. I used to tell people that there are just two things which are difficult in Open edX: static assets and settings. I'm grateful that you are tackling the latter after having greatly improved the state of the former.

My comments are mostly questions for my own understanding. As far as I'm concerned this is good to go.

regisb · 2025-02-13T13:32:48Z

docs/decisions/0022-settings-simplification.rst

+(Tutor). Specifically, production edx-platform configuration currently works
+like this:
+
+* ``lms/envs/tutor/production.py``...


This looks accurate, but it's slightly depressing to read how complex it is...

regisb · 2025-02-13T13:34:07Z

docs/decisions/0022-settings-simplification.rst

+setting, the preferred way to do so is to "make a Tutor plugin". This is a
+large amount of prior knowledge, boilerplate, and indirection, all required
+to simply do something which Django provides out-of-the-box via a custom
+``DJANGO_SETTINGS_MODULE``.


Again: depressing but true. As Tutor maintainers, we could certainly do a better job at explaining how to create such plugins: either via better documentation or plugin cookiecutters. But that's beyond the scope of this ADR.

regisb · 2025-02-13T13:42:30Z

docs/decisions/0022-settings-simplification.rst

+
+* ``openedx/envs/common.py``: Defaults shared between LMS and CMS (new!).
+
+  * ``lms/envs/common.py``: LMS default settings. Wherever possible,


Just to clarify:

we don't need a production.py settings file because common.py will be (mostly) production-ready, right?

and manage.py will default to loading common.py, right?

Yes and yes. I will clarify both.

regisb · 2025-02-13T13:49:29Z

docs/decisions/0022-settings-simplification.rst

+  (lms,cms)/envs/production.py unchanged through this process.
+
+* Develop (cms,lms)/envs/development based off of (cms,lms)/envs/common.py.
+  Iterate until we can run "bare metal" development server for LMS and CMS


Oh wow I'm really looking forward to that!

regisb · 2025-02-13T13:54:55Z

docs/decisions/0022-settings-simplification.rst

+  (cms,lms)/envs/common.py into openedx/envs/common.py. This may be iteratively
+  done across multiple PRs.
+
+* BREAKING (major, just common.py): Find the best production-ready defaults


Is this really going to be a breaking change if production.py remains unchanged in the process? In other words: do we consider that changes to common.py are breaking changes?

Well, since we don't have a well-defined edx-platform API contract, any non-additive change to any python object that isn't underscore-prefixed or literally called "internal" could be considered "breaking" 🥲.

But that's a rant for another day. Practically speaking, changes to common.py probably won't break anyone. So, I'll mark the common.py commits as feat!, but we won't be wringing our hands or issuing DEPR tickets until we're trying to change/remove production.py.

regisb · 2025-02-13T13:56:03Z

docs/decisions/0022-settings-simplification.rst

+* BREAKING (major): Deprecate and remove (cms,lms)/envs/devstack.py.
+  Tools (like Tutor and 2U's devstack) will either need to maintain local
+  copies of these modules, or "rebase" themselves onto
+  (lms,cms)/envs/development.py.


regisb · 2025-02-13T14:11:17Z

docs/decisions/0022-settings-simplification.rst

+Decision
+********
+
+This is our target edx-platform settings module structure:


This is a greatly simplified structure, and I'm all in favor of it. But I expect that it's going to create issues for some settings that derive from others.

To clarify, I'm talking about the following scenario:

common.py defines SETTING_BASE = ... and DERIVED_SETTING1 = foobar1(SETTING_BASE) and DERIVED_SETTING2 = foobar2(SETTING_BASE).

production.py modifies SETTING_BASE. DERIVED_SETTING1 and DERIVED_SETTING2 need to be re-defined.

This is for instance what currently happens with STATIC_ROOT_BASE, from which STATIC_ROOT, WEBPACK_LOADER are derived.

How do you expect we are going to resolve these cases? I see 3 possible approaches:

Be dumb about it and duplicate settings in common.py and production.py.

Somehow preserve the existing derive_settings mechanism. (by the way, are we going to keep it around?)

Refactor settings such that we don't have to duplicate settings.

Approach 3 would probably be the "right" one, but I expect we'll have to make this decision on a case-by-case basis.

(I'm sure you've thought about this but I don't see it reflected in this ADR. Or is it part of the "Remove redundant overrides" paragraph below?)

Great question, silly of me to omit that from the ADR.

Having looked through the modules with @feanil a lot lately, we feel that (1) would add lot more work for operators and (3) is just an infeasibly large amount of work and breaking changes. I don't love having bespoke tooling, but if we have to keep some bespokeness around, I think that derived_settings is pretty tame and well-defined. So I propose (2). Specifically, the three common.py files would have a lot of derived defaults. Lmk what you think.

Relatedly, in preparation for this, I have already made the derived_settings API a bit easier to work with: #36192

I'm OK with some bespoke tooling. Like I said, I expect that we might be able to use approach 3 for some settings. If anything, derive_setting will help us identify settings that should be simplified, and we can take on those further down the road.

I expect that we might be able to use approach 3 for some settings. If anything, derive_setting will help us identify settings that should be simplified, and we can take on those further down the road.

I agree. The dependencies between different settings will become much clearer and easy-to-untangle once we have converted all the production.py-special-cases into explicit common.py Derived settings.

regisb · 2025-02-13T14:14:09Z

docs/decisions/0022-settings-simplification.rst

+      derived from ``lms/envs/common.py``, and point their
+      ``DJANGO_SETTINGS_MODULE`` environment variable at this module.
+
+    * ``lms/envs/yaml.py``: (Possibly) An alternative to third-party


For the record, I expect that Tutor will not be using this module at all.

robrap

Thank you.

robrap · 2025-02-14T20:43:48Z

docs/decisions/0022-settings-simplification.rst

+
+OEP-45 declares that sites will configure each IDA's (indepently-deployable
+application's) Django settings with an ``<APPNAME>_CFG`` yaml file, parsed and
+loaded by a single upstream-provided ``DJANGO_SETTINGS_MODULE``. this contrasts


Suggested change

loaded by a single upstream-provided ``DJANGO_SETTINGS_MODULE``. this contrasts

loaded by a single upstream-provided ``DJANGO_SETTINGS_MODULE``. This contrasts

robrap · 2025-02-14T20:44:16Z

docs/decisions/0022-settings-simplification.rst

+application's) Django settings with an ``<APPNAME>_CFG`` yaml file, parsed and
+loaded by a single upstream-provided ``DJANGO_SETTINGS_MODULE``. this contrasts
+with the django convention, which is that sites override Django settings using
+a their own ``DJANGO_SETTINGS_MODULE``. The rationale is that all Open edX


Fixed typo and tried to make it clear that this is rationale that you are going to change...

Suggested change

a their own ``DJANGO_SETTINGS_MODULE``. The rationale is that all Open edX

their own ``DJANGO_SETTINGS_MODULE``. The rationale was that all Open edX

robrap · 2025-02-14T20:45:14Z

docs/decisions/0022-settings-simplification.rst

+loaded by a single upstream-provided ``DJANGO_SETTINGS_MODULE``. this contrasts
+with the django convention, which is that sites override Django settings using
+a their own ``DJANGO_SETTINGS_MODULE``. The rationale is that all Open edX
+customization can be reasonably specified in YAML; therefore, it is


Suggested change

customization can be reasonably specified in YAML; therefore, it is

setting customization can be reasonably specified in YAML; therefore, it is

robrap · 2025-02-14T20:48:42Z

docs/decisions/0022-settings-simplification.rst

+successfully uses only YAML files for configuration. Furthermore,
+upstream-provided ``DJANGO_SETTINGS_MODULE`` which loads these yaml files


Suggested change

successfully uses only YAML files for configuration. Furthermore,

upstream-provided ``DJANGO_SETTINGS_MODULE`` which loads these yaml files

successfully uses only YAML files for configuration. Furthermore, the

upstream-provided ``DJANGO_SETTINGS_MODULE`` which loads these yaml files

robrap · 2025-02-14T22:35:45Z

docs/decisions/0022-settings-simplification.rst

+
+    * and uses templates vars from Tutor configuration (``config.yml``),
+
+    * and invokes hooks from any enable Tutor plugins;


Suggested change

* and invokes hooks from any enable Tutor plugins;

* and invokes hooks from any enabled Tutor plugins;

robrap · 2025-02-14T22:57:58Z

docs/decisions/0022-settings-simplification.rst

+* BREAKING (major, just common.py): Find the best production-ready defaults
+  between both (lms,cms)/envs/production.py and Tutor's production.pys, and
+  "bubble" them up to (openedx,cms,lms)/common.py. Keep
+  (lms,cms)/envs/production.py unchanged through this process.


If the production.oy files remain unchanged, can you better explain the breaking change aspect of this, and what we expect to break or not break? Also, maybe note the action to be taken to workaround the breaking change, if applicable?

robrap · 2025-02-14T22:59:17Z

docs/decisions/0022-settings-simplification.rst

+  Iterate until we can run "bare metal" development server for LMS and CMS
+  using these settings.
+
+* BREAKING (major): Deprecate and remove (cms,lms)/envs/devstack.py.


[inform] I think 2U already handled this as part of the DEPR work for devstack. So for 2U, I think it will just be a question of when and if we can take advantage of dev.py.

robrap · 2025-02-14T23:01:56Z

docs/decisions/0022-settings-simplification.rst

+One alternative settings structure
+----------------------------------
+
+Here is an alternate structure would de-dupe any shared LMS/CMS dev & test


Suggested change

Here is an alternate structure would de-dupe any shared LMS/CMS dev & test

Here is an alternate structure that would de-dupe any shared LMS/CMS dev & test

robrap · 2025-02-14T23:02:30Z

docs/decisions/0022-settings-simplification.rst

+Here is an alternate structure would de-dupe any shared LMS/CMS dev & test
+logic by creating more shared modules within openedx/envs folder. Although
+DRYer, this structure would increase the total number of edx-platform files and
+potentially encourage more LMS-CMS coupling. So, will not pursue this


Suggested change

potentially encourage more LMS-CMS coupling. So, will not pursue this

potentially encourage more LMS-CMS coupling. So, we will not pursue this

robrap · 2025-02-14T23:06:16Z

docs/decisions/0022-settings-simplification.rst

+the distressing level of complexity that developers and operators face when
+working with the platform.
+
+Decision


Consider adding a sentence or two in this Decision section about the decision to adjust OEP-45 in a separate effort to better align with the decisions noted in this ADR. You can also reference the Consequences section for more details.

kdmccormick mentioned this pull request Feb 5, 2025

Simplify edx-platform settings #36215

Open

kdmccormick force-pushed the kdmccormick/settings-adr branch 3 times, most recently from ba09230 to 57f550a Compare February 10, 2025 22:21

kdmccormick requested a review from feanil February 10, 2025 22:22

kdmccormick marked this pull request as ready for review February 10, 2025 22:22

feanil reviewed Feb 11, 2025

View reviewed changes

kdmccormick requested a review from regisb February 12, 2025 18:49

kdmccormick mentioned this pull request Feb 12, 2025

Let's talk about OEP-45 (Configuring & Operating Open edX) openedx/open-edx-proposals#587

Open

kdmccormick requested a review from feanil February 12, 2025 18:51

feanil reviewed Feb 12, 2025

View reviewed changes

docs/decisions/0022-settings-simplification.rst Outdated Show resolved Hide resolved

docs/decisions/0022-settings-simplification.rst Outdated Show resolved Hide resolved

docs/decisions/0022-settings-simplification.rst Outdated Show resolved Hide resolved

kdmccormick force-pushed the kdmccormick/settings-adr branch from 57f550a to c8e03af Compare February 12, 2025 20:04

kdmccormick requested a review from feanil February 12, 2025 20:23

feanil approved these changes Feb 12, 2025

View reviewed changes

regisb approved these changes Feb 13, 2025

View reviewed changes

kdmccormick requested a review from robrap February 13, 2025 17:49

kdmccormick added 3 commits February 13, 2025 13:19

docs: Settings Simplification ADR

9908298

docs: Feanil's review comments

08c357e

docs: Clarify defaults and derived settings

7883fc4

kdmccormick force-pushed the kdmccormick/settings-adr branch from c8e03af to 7883fc4 Compare February 13, 2025 18:19

docs: Clarify default for ./manage.py

7847aab

robrap reviewed Feb 14, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: Settings Simplification ADR #36224

docs: Settings Simplification ADR #36224

kdmccormick commented Feb 5, 2025 •

edited

Loading

kdmccormick commented Feb 10, 2025

feanil left a comment

kdmccormick commented Feb 12, 2025

kdmccormick commented Feb 12, 2025

feanil left a comment

regisb left a comment

regisb Feb 13, 2025

regisb Feb 13, 2025

regisb Feb 13, 2025

kdmccormick Feb 13, 2025

regisb Feb 13, 2025

regisb Feb 13, 2025

kdmccormick Feb 13, 2025

regisb Feb 13, 2025

regisb Feb 13, 2025

kdmccormick Feb 13, 2025

regisb Feb 13, 2025

kdmccormick Feb 13, 2025

regisb Feb 13, 2025

robrap left a comment

robrap Feb 14, 2025

robrap Feb 14, 2025

robrap Feb 14, 2025

robrap Feb 14, 2025

robrap Feb 14, 2025

robrap Feb 14, 2025

robrap Feb 14, 2025

robrap Feb 14, 2025

robrap Feb 14, 2025

robrap Feb 14, 2025


		* ``openedx/envs/common.py``: Defaults shared between LMS and CMS (new!).

		* ``lms/envs/common.py``: LMS default settings. Wherever possible,

	loaded by a single upstream-provided ``DJANGO_SETTINGS_MODULE``. this contrasts
	loaded by a single upstream-provided ``DJANGO_SETTINGS_MODULE``. This contrasts

	a their own ``DJANGO_SETTINGS_MODULE``. The rationale is that all Open edX
	their own ``DJANGO_SETTINGS_MODULE``. The rationale was that all Open edX

	customization can be reasonably specified in YAML; therefore, it is
	setting customization can be reasonably specified in YAML; therefore, it is

		successfully uses only YAML files for configuration. Furthermore,
		upstream-provided ``DJANGO_SETTINGS_MODULE`` which loads these yaml files


		* and uses templates vars from Tutor configuration (``config.yml``),

		* and invokes hooks from any enable Tutor plugins;

	* and invokes hooks from any enable Tutor plugins;
	* and invokes hooks from any enabled Tutor plugins;

	Here is an alternate structure would de-dupe any shared LMS/CMS dev & test
	Here is an alternate structure that would de-dupe any shared LMS/CMS dev & test

	potentially encourage more LMS-CMS coupling. So, will not pursue this
	potentially encourage more LMS-CMS coupling. So, we will not pursue this

docs: Settings Simplification ADR #36224

Are you sure you want to change the base?

docs: Settings Simplification ADR #36224

Conversation

kdmccormick commented Feb 5, 2025 • edited Loading

kdmccormick commented Feb 10, 2025

feanil left a comment

Choose a reason for hiding this comment

kdmccormick commented Feb 12, 2025

kdmccormick commented Feb 12, 2025

feanil left a comment

Choose a reason for hiding this comment

regisb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robrap left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kdmccormick commented Feb 5, 2025 •

edited

Loading