Currently the Django apps serves users MEDIA files by Django's FileResponse...
A better way is that the server used sendfile. A solution is https://pypi.org/project/django-sendfile2/ with a proper server. But Caddy doesn't support this, yet: caddyserver/caddy#3828