Obviously it will depend on what you're doing and one's tolerance for things many deem to be annoying. I've encountered people with incredibly high tolerance for slow and awkward workflows but, alas, I am not one of them.
If I had to call out one thing it would be that you can't do tests without having a database there. This results in incredibly slow tests for even the simplest things. I don't need to test database persistence every time I'm testing some domain logic. So maybe then don't do fat models and "map" the data from Django models to a domain layer? Well, congrats, you've just manually implemented a data mapper ORM, which is what SQLAlchemy is.
It works well for simple CRUD stuff, which is really useful. But it very quickly becomes a mess and a big ball of mud when you start to do more complicated things. IMO a db access layer or web framework should be completely independent of domain logic, but Django makes that really difficult.
> IMO a db access layer or web framework should be completely independent of domain logic, but Django makes that really difficult.
That is an issue/features in Django, depending on your view. You really don't get to do things the framework doesn't want. If you're trying to fight the ORM or any of the components in Django really, including the Django REST framework, you're going to lose and have a bad time.
There are certainly reason why you'd want to separate domain logic from the database access, but then Django isn't what you want. You're also going to miss out many of the things that makes Django easy to work with, like getting most CRUD operations for free.