I'm an embedded person, so clueless on this stuff. Are notebooks really used in prod? Outside of applications where notebooks are specifically useful to the goal of the system, ie. documentation,blog posts,textbooks with interactive/data driven elements? How would this even work? Can an ipynb file be mechanized in the same way a py file can?
Jupyter notebooks can be executed roughly like scripts by papermill. You can also save a .py version of the notebook without outputs using jupytext. We use these packages together where I work to basically auto-generate the start of notebooks for exploratory work after certain batch jobs. For dashboards only used by a small number of users, we’ve found voila occasionally useful. Voila turns notebooks into dashboards or web apps basically.
You generally shouldn’t put code you want to reuse in notebooks, but we haven’t found this to be much of a problem. And >3/4 of the people who’ve worked on our team don’t have software engineering backgrounds. If you set clear expectations, you can avoid the really bad tar pits even if most of your co-workers are (non-software) engineers or scientists 0-3 years out of college.
I think the issue actually goes beyond notebooks. It's just that notebooks are the method du jour for creating hacked-together code. But regardless of the tool chosen, hacked-together code does eventually leak into production. Mine has.
It's harder to breach that barrier in embedded, because someone has to approve of getting the hardware made.
In most organizations, there's a kind of no mans land between hacked code, and professionally written software. That's because the real SWE's are always overbooked, and for good reasons -- business and career wise -- should be working on the biggest projects. This leaves software needs that are urgent for somebody but not urgent enough to rise to the top of the priority list for the software team. The somebody could be an internal or external customer.
