DEP 0018 Dictionary-based EMAIL_PROVIDERS (ticket-35514)

[medmunds]

📄 Read the formatted draft DEP 0018, latest revision.

See ticket-35514#comment:24

Django ticket-35514 will implement a dictionary-based EMAIL_PROVIDERS setting, similar to CACHES, DATABASES, STORAGES and TASKS. The ticket was approved following discussion on the django-developers list and at DjangoCon Europe 2024 sprints.

The purpose of this DEP is to facilitate discussion and decisions on the proposed API and related deprecations.

[nessita]

Submitting my initial set of comments to see if there are any red flags.

[nessita]

Amazing work, super well laid out, clear and precise. Thank you @medmunds! I'm overall onboard with the proposal as a whole pending some minor fixes. I tried to provide guidance on every 🤔 callout.

Let me know how to progress this!

[nessita]

@medmunds I'm happy to be added as Shepherd, and I think we could use number 18 for this and putting it up for review once the small bits are taken care of. What do you think?

[medmunds]

@nessita thanks as always for the thorough review and thoughtful suggestions 🙌

I'll update to incorporate all the feedback. (I suspect this may unearth some new questions about fail_silently, but we're converging on a solution.)

I hadn't really thought about handling this as a real DEP; I was just borrowing the format and the repo for discussion. (And I was going to close it unmerged once the details were worked out, since the ticket is already approved.) But now that you suggest it, a DEP makes sense. Thanks for shepherding!

Once there's a new draft, I think it's important to get Jacob R's input. (Recognizing the enormous amount of time he's already put into this, I wanted to wait to @ him until there were fewer open questions.)

I'll also try to reach out directly to some third-party packages that would be affected (django-mailer, django-celery-email, django-ses, etc.), to make sure they're aware of the proposal and have a chance to weigh in.

And it would be helpful to get feedback from people who are actively using the SMTP EmailBackend in production, particularly on some of the deprecations. Maybe a forum post to re-introduce the feature? (My own production experience is with http-API-based backends like django-anymail and django-ses, plus a private fork of django-mailer for queuing and error recovery. So I may have some blind spots around things like get_connection() with username/password overrides.)

[nessita]

@medmunds On a related tanget, I just came across this PR that allows cache backends to "know" their alias. I haven't reviewed that but figured you may be interested in taking a quick look, in the interest of consistency? django/django#20469

[medmunds]

@medmunds On a related tanget, I just came across this PR that allows cache backends to "know" their alias. I haven't reviewed that but figured you may be interested in taking a quick look, in the interest of consistency? django/django#20469

Ouroboros! I've been citing the new-features discussion that led to that PR as justification here for providing alias on EmailBackends. I'll take a look at the PR.

[medmunds]

I've updated the DEP to incorporate @nessita's feedback. There are still a few open questions (including a few new ones), but I think it's ready for @jrief to take a look if he's available.

Based on discussion in this pr, the DEP now proposes moving fail_silently out of individual EmailBackends, instead handling it in EmailMessage.send(). This both resolves current inconsistencies in fail_silently behavior and avoids an API-shape mismatch that would complicate mail.providers[alias]. See New handling of fail_silently (and EmailBackend fail_silently deprecated) in the current draft.

The fail_silently change can (and likely should) be split out from implementing EMAIL_PROVIDERS. I'll start a forum discussion on it shortly.

[LilyFirefly]

I've given this a full read through and I didn't spot any red flags.

[tim-schilling]

I'm on board with this as is! Thank you @medmunds and @nessita for your work on this. I had a few nitpick comments, some formatting questions, and then provided my opinion around the 🤔 points.

I don't have any substantive requests for change, though if we engage other 3rd party package maintainers, they may have thoughts on the upgrade process.

[tim-schilling]

Thank you @medmunds and @nessita for pushing this forwards!

[codingjoe]

Why "providers"? I'd honestly prefer a technical term that is hyperscaler agnostic. We also don't do datastore_providers but DATABASES.

There are still people who run their own SMTP servers. They might still run different configurations, even on a single server.

I admit, this is highly subjective, but I thought I at least raise the concern.

[LilyFirefly]

Why "providers"? I'd honestly prefer a technical term that is hyperscaler agnostic. We also don't do datastore_providers but DATABASES.

EMAILS would fit the existing pattern: CACHES, DATABASES, STORAGES and TASKS.

[jrief]

Why "providers"? I'd honestly prefer a technical term that is hyperscaler agnostic. We also don't do datastore_providers but DATABASES.

Please check the discussion here: https://code.djangoproject.com/ticket/35514

We came up with this name during the sprints at Django Con Europe 2024 (Carton Gibson, Natalia Bidart, Jacob Rief).

[codingjoe]

Please check the discussion here: https://code.djangoproject.com/ticket/35514

We came up with this name during the sprints at Django Con Europe 2024 (Carton Gibson, Natalia Bidart, Jacob Rief).

In said thread, I see a multitude of people raising the same concern, but no resolution. And I don't warm up to the idea of it referring to "email service providers".

1. The term is not as equally common as ISP
2. It refers to a money-making entity and are not a technical term.

The closest technical term I can think of would be email relay. It is both more commonly used and technical.

I am really hesitant to open the doors for non-technical terms that describe institutions. IMHO, it sets a negative precedent and leaves a strange aftertaste.

[codingjoe]

Side note: every big provider, from AWS, Google, Twilio, or Azure, refers to their services as SMTP relays.

[nessita]

Why "providers"? I'd honestly prefer a technical term that is hyperscaler agnostic. We also don't do datastore_providers but DATABASES.

EMAILS would fit the existing pattern: CACHES, DATABASES, STORAGES and TASKS.

Thank you everyone for revisiting this. It's good to challenge decisions with fresh eyes.

We discussed the settings name at length, and the key distinction that is worth mentioning first is that settings like CACHES, DATABASES, STORAGES, and TASKS refer to concrete resources. This DEP is not about "an email" as a resource, but about configuring how emails are sent. So in that sense, in my view EMAILS is somehow misleading and inaccurate.

In practice, this setting represents a set of delivery strategies rather than a single mechanism. While SMTP may be used at the boundary, from the Django user's POV there are many valid approaches, including API-based backends that do not expose SMTP at all. In that sense, this setting represents a collection of configurations for different delivery strategies (SMTP, HTTP APIs, gateways, etc.), which do not map cleanly to a single "relay" concept.

"Relay" is also quite SMTP-specific and less approachable as a general term, whereas EMAIL_PROVIDERS aims to capture that broader abstraction in a way that aligns better with how users think about configuring email delivery in modern Django projects. EMAIL_SENDERS could also be worth considering but notably, not all "senders" actually send directly; some queue, log, or delegate, which is precisely why a more abstract term may be appropriate here.

Other options we explored:

• EMAIL_BACKENDS: there is already EMAIL_BACKEND (singular noun setting being deprecated), so it would be confusing. Also none of the other similar settings append _BACKENDS (except for AUTHENTICATION_BACKENDS but that's a list of dotted-path strings, not a dict of configurations, so it's a different pattern).
• EMAIL_DELIVERIES: could be confused with delivery options like "fast", "delayed", "bulk".
• EMAIL_TRANSPORTS: fairly accurate and protocol-agnostic but perhaps jargon-heavy.
• EMAIL_SERVICES: close to "providers" but signals more strongly third-party connotations, therefore less inviting to self-hosted setups.
• EMAIL_CONFIGS or EMAIL_CONFIGURATIONS: too generic I would say and just like _BACKENDS Django does not generally append _CONFIGS to settings. (Django only uses LOGGING_CONFIG to match the Python's name for dictConfig.)

Anyway, if EMAIL_PROVIDERS has to be changed, my second preference would be EMAIL_SENDERS: despite that not all backends actually send the emails, Django's own terminology uses send_mail, send_mass_mail, send_message, etc. We would then do this (the repetition I don't like though):

from django.core.mail import senders

senders.default.send_messages([...])

[LilyFirefly]

I like the simplicity of EMAILS and I'm not sure the nuanced distinctions between all the different possible names are actually important in the grand scheme of things. That said, this isn't blocking for me, so I'll leave it at that.

[medmunds]

As a perfectionist, I'm tempted to revisit the naming bikeshed. As a perfectionist with a looming 6.1 deadline, I'm inclined to stick with EMAIL_PROVIDERS.

In said thread, I see a multitude of people raising the same concern, but no resolution.

It looks like I represent fully 50% of that multitude. I initially expressed concern about the name, but in a later comment revised my position and supported EMAIL_PROVIDERS. (The other commenter was Adam Johnson, who never weighed in again.)

I think EMAIL_PROVIDERS will be descriptive, approachable and discoverable, even to someone not familiar with email technical jargon. It configures the entities that "provide email" to your Django application. That can certainly include your own SMTP servers. (I think the allusion to "email service providers" is useful for people already familiar with that term, but the setting is descriptive even if you don't know that. Also, the setting is not EMAIL_SERVICE_PROVIDERS, which I agree would be unnecessarily limiting—and problematic for a few other reasons. Btw, I mean "entity" in the generic computational object sense, not the corporate/legal sense.)

If we had to change it (which, again, I don't think is advisable if we want to land this in 6.1):

• I'd suggest MAILERS, configuring entities that mail (send email). The factory object would be django.core.mail.mailers. (See also the venerable pinax/django-mailer package. EMAILERS could work, but it's django.core.mail not .email. If not for the timing, I'd be +0 or +1 on MAILERS.)

• I'm -0 on EMAIL_SENDERS. "Sender" has a specific, conflicting meaning in the context of email (for those who are familiar with the technical jargon).

• I'm -0 (or maybe -1) on EMAILS, which I think is prone to confusion given the common meaning of "email." (And I'm aware of the TASKS and TEMPLATES parallels.) The factory would have to be something like django.core.mail.email_backends (emails would cause way too many naming conflicts with existing code, including Django's own tests).

• I'm -1 on EMAIL_RELAYS. I don't think it's approachable for users unfamiliar with SMTP jargon. And it's unnecessarily limiting: not all email backends interact directly with an SMTP relay. (Among the dozen-plus commercial ESPs I'm familiar with, only a few use "relay" to describe their HTTP sending APIs—and those few tend to be the more enterprise-y hyperscalers.)

• I'm -1 on the other options Natalia mentioned were considered, for exactly the reasons she described.

[jrief]

I would also stick with EMAIL_PROVIDERS – we had a lengthy discussion on the proper naming.

My second choice would be MAILERS or maybe EMAILERS (if that can even be considered a word).

[nessita]

I also like MAILERS, and from a chat I had with Jacob (Walls) he is in favor of that too. I also agree that we should prioritize having the feature ready and merged, and leave the rename for after (via sed/replace likely?). We could even do the rename after landing, we have a precedent for doing a similar thing for the querystring template tag.

I can take ownership of finalizing/driving the rename once the feature is complete (so we focus on that).

[codingjoe]

Stupid question: We are always talking about e-mails, right? Not SMS or RCS, right?

Because emails are ALWAYS transmitted via SMTP, even if you use a different protocol to speak to your relay. However, I understand that “relay” is a rather technical term used by DevOps folk.

Back to RCS, I mean, do we want to support RCS backends in the future? This would be super cool! Especially since the EU is enforcing client interoperability. How cool would sending iMessage or WhatsApp messages be natively inside Django 🤯

Certainly outside the scope for this poor proposal, but with this in mind:

MESSAGES??

Oh, and sorry for reopening the discussion. Naming things and cache invalidation remain the hardest things in computer science… 🤷‍♂️

[medmunds]

Back to RCS, I mean, do we want to support RCS backends in the future? This would be super cool! Especially since the EU is enforcing client interoperability. How cool would sending iMessage or WhatsApp messages be natively inside Django 🤯

Jacob R mentioned something similar in the ticket. Some sort of "integrated communications" (sorry for the marketing-speak) does seem like a natural extension of core.mail and contrib.messages. I was about to suggest prototyping it in a third-party package, then TIL django-notifs. Fwiw, they call their pluggable communications channels "providers."

Oh, and sorry for reopening the discussion. Naming things and cache invalidation remain the hardest things in computer science… 🤷‍♂️

It's a useful discussion. The DEP should probably mention the rationale for the naming (whatever we end up with) and outline the alternatives that were considered. (Also… is there one other hardest thing in computer science? I always lose count.)

[nessita]

Thank you everyone, so much, for your feedback and thoughtful comments. After serious deliberation and hands-on work seeing how the various options would look and work in the code, Mike and I will proceed with the rename of the settings EMAIL_PROVIDERS to MAILERS. The collection set will be renamed from providers to mailers.

[nessita]

This DEP will be updated in the next 7-10 days to reflect the recent (and minor other than the rename) changes following the implementation. Once the DEP text is updated, I'll propose the PR to move it to the approved folder.

[jacobtylerwalls]

@LilyFirefly do you have permission to merge? We can fix the MAILERS name change in a follow-up.

[medmunds]

I am in the process of updating the draft DEP for the renaming, and will open a new PR for that soon.