Anyone else noticed that when editing a notebook in VS Code, one could lose unexecuted edits to a cell with accidental arrow key input (that defocuses the current cell and erases changes)? Too many times I have lost a paragraph worth of Markdown text midst typing, it’s kind of maddening.
Yep - it's very annoying but for me it only happens with markdown cells? So now I generally select all and copy now before leaving a markdown cell if I've edited it.
The other issue is the language server seems to get into a funk regularly - the completion stops working as does the display of docstrings so I regularly use the "Developer: reload window" feature to reset everything.
Also the plots view is poor - it's basically just an SVG you can magnify - there's no proper zoom or rotation for 3d plots.
I was also irritated by the "smart" environment discovery stuff - just let me specify which python install or vent I want to use. Don't make be google to see what names/locations I have to use in order for an environment to magically appear in the list.
Also the editor extension (I use vi mode) is ignored. And even saving the notebook doesn't work the way every other editor pane works in vscode.
Hmm... writing this list makes me wonder why I haven't looked at alternatives.
Fyi VS Code is working on[1] native first-class notebook support, which should make the experience of using a notebook much better. Notebooks right now stretch the bounds what extensions can do, particularly in that they can't use the VS Code's own text editing services, so things like vi mode aren't inherited.
(Disclaimer: I work on VS Code, but am not working on notebooks)
Cool. A while ago I wanted to report bugs wrt notebooks but the bug reporting flow asked me to turn off extensions, and after turning off extensions notebooks no longer work, so I didn’t even know where to report the bugs.
Jupyter Notebooks today are handled by the Python extension, which is a separate codebase and team from VS Code. You should file issues there: https://github.com/Microsoft/vscode-python
Yes, as conner4312 VS Code is investigating how to better incorporate a juypter notebook experience within it, natively. As has been pointed out here, there are a number of limitations that the python extension notebook support has run into. We have been working closely with the VS Code team to find solutions to these limitations. The custom editor work (https://github.com/microsoft/vscode/issues/77131) is one such mechanism. We plan on rolling out our use of that work soon after it is officially released (sometime in February). However, this will not address all of the problems mentioned in this thread. Those issues will be addressed as native jupyter support arrives in VS Code.
Here's some additional info:
- Find/replace. (Personally, this drove me NUTS! Just ask my team! :) ) That being said a simple find (ctrl-f) is avaiable in the current official release. Unfortunately, it does not include replace. While we recognize that this is a severe limitation for some, implementing a full find and replace solution would end up requiring the extension to duplicate most of the functionality provided by VS Code itself. Therefore, we currently planning to piggy back off of that work when it becomes available. If that winds up taking an inordinate amount of time, we will consider further stop gap work.
- The markdown loss issue has been addressed and will be in the February release.
- For the language server problem discussed above, we need more info. I would encourage you to enter a new issue at https://github.com/Microsoft/vscode-python. In the meantime, there are known related issues that will be fixed the Feb release. One in particular, was caused by the backend jupyter server inexplicably taking a long time to return us a full completion list. So when you don't see completion pop up, it may not be that it's broken, it may just be taking a long time. We are working around this issue for the February release. Also, if you're having docstring issues, you may find that turning off the python jedi language service in settings will help.
- Plots view. Yes, it is limited and honestly can cause rendering slowdowns. We are working through ways of addressing both problems. A good solution for supporting "proper" zoom will require adding ipywidget support overall, which is coming soon.
- "Smart environment stuff". I'm not following completely. Is this a general python extension issue or does it have to do with the jupyter kernel selection support we have for notebooks? Again, feel free to enter additional details at: https://github.com/Microsoft/vscode-python
- Supporting global editor extensions like "VI mode", isn't something that we can support without help from VS Code itself. One of the main reason's VS Code is pursuing native notebook support is to be able to handle issues exactly like this.
- Saving the notebook weirdness will be addressed in the February release. We were waiting for the custom editor support I referred to above to arrive.
Great response - I really appreciate the detail and also commend the tone you use. It makes me a little worried that I came across as too negative earlier.
To balance that: I haven't forgotten the "WOW WHAT IS THIS" feeling I experienced the first time opened an ipynb file from vscode when the notebook support kicked in and asking my colleagues "did you know about this?" as I excitedly started exploring the feature. Since then (and I'm a relative vscode noob), I've never been tempted to go back to working with notebooks through a browser. Despite my gripes, the overall experience is still miles better (in my opinion) than accessing notebooks through a browser. It's the future for sure.
And thanks for all those updates - I can't wait for the Feb release - and it's very reassuring that these issues are not only well recogonized but also are being aggressively addressed.
Regarding some of your feedback to the feedback:
- Re. the language server issue - it could well be caused by a slow down. I'll test that hypothesis the next time I experience it.
- The "smart environments stuff" is, as you say, actually a general python extension issue. But actually, I withdraw that complaint. I've just checked and I can point the extension at any python interpreter by editing settings.json - it's just the GUI which restricts you to selecting from a list of discovered interpreters.
Thanks again for your response and your engineering efforts in general.
I think environment selection would be considered a bug. I theory any python environment that's on your machine should be selectable in the UI. Could you enter an issue at that same location? I can for you, but I'd prefer that the issue comes directly from an outside user (sometimes we filter bugs based on from where they came).
> - The markdown loss issue has been addressed and will be in the February release.
Awesome!
> - For the language server problem discussed above...
I don’t think that’s Python-specific, I’ve encountered language server seemingly getting “stuck” in some state for almost any language I’ve used for more than a few minutes in VS Code, including Python, C/C++, Go, Rust, etc. I usually need to reload the window more than once per day.
> - "Smart environment stuff".
Probably talking about the general Python-mode Python selector. One can only choose from a list of detected Pythons; no file system dialogue available, and one can’t even paste in a full path unless it’s in the “sanctioned” list. I’ve cursed on this one too.
Microsoft have released another IDE with notebook support called Azure Data Studio. It uses the same underlying code base as VS Code but is targeted at data driven tasks such as connecting to databases.
Strangely, it uses a different jupyter notebook implementation to VS Code. I found it avoids some of the annoying bugs mentioned in another comment.
It also introduces a SQL kernel which I’ve found useful for organising my queries.
Whilst jupyter notebooks are often synonymous with python code, the languages are actually pluggable by using a different kernel. Microsoft have implemented an SQL kernel which means you can write SQL queries directly in a notebook.
Azure Data Studio is a proper fork of VS Code all up. This is why it was able to work around some of the limitations the Python extension is currently dealing with.
I am curious to see the general workflow expanded on. I hesitate to think there is a single correct way to keep a notebook. Seems flexibility is the key feature. And is why text is so well used.
I haven't even gotten into org mode, I just edit Python using emacs and it works great. I can evaluate parts of the code and jump to the interpreter when needed. I keep a matplotlib window open with plt.ion() when running locally, or use plt.savefig() when running remotely and have a qiv window open that automatically detects new images. Works great.
The only thing I would like is to be able to use the Python interpreter _within_ the context of a function sometimes, it's annoying when an error occurs inside a function and I can't access the local variables.
I like jupyter notebooks too, but one of the things stopping me is that I can't use emacs features for editing ;)
To interactively inspect within a function, you could use the debugger with `breakpoint()`. But it's not a regular Python interpreter, which annoys me having to learn the different syntax. You can also embed IPython interactively with:
import IPython; IPython.embed()
though IPython kinda doesn't clean up after itself, so it's a bit messier (your prompt is messed up afterwards).
Finally, you can:
import code; code.interact()
The downside of this is there's no history or autocomplete by default.
I have a custom function `embed()` [1] that calls `code.interact()`, but with history and autocomplete configured. This is the closest I got it to looking like a regular Python interpreter at an arbitrary place in the code.
Thank you! I didn't know about code.interact, but IPython.embed I have used in the past. My point was more that I would like a way to drop to an IPython prompt when my program crashes, I really don't like the solution of having to modify the code to insert a statement where I think my program is going to crash.
In any case I think there was a way to examine variables from up the stack of a stack trace or something like that.. bit too lazy to look it up right now. Regardless, working with a REPL in emacs is pretty great ;)
I think there is a way to do jupyter from emacs (or at least there was a way to do ipython notebooks), but I haven't used it extensively, just tested it once I think. I guess I'm pretty satisfied with the REPL and haven't found a need to have intermediate output during the running of a program.
What I do like about the matpotlib interactive approach is being able to watch the progress of a loop very easily, which is hard to do from a notebook, although there are some ways, but they are either hacky or require some pretty sophisticated things like custom widgets.
Yeah I considered extending it to automatically pop into the frame where the exception happened, but this is actually not as useful as it sounds, since the exception is usually much deeper in some other library and not in your code where you actually want to inspect variables. You would either need to declare the boundary between 'your' code and other code (by filepath or something), so the tool could know where it should run, or you would need to include commands to push and pop through the stack, which complicates things a bit and is on the path to a full debugger (so perhaps one should just use the debugger - as you've mentioned in the sibling comment).
So I just add a try: except around code known to be crashing and call my `embed()` function on except. This has served me well enough that I haven't bothered with anything else.
Plus, if I really want to inspect unplanned crashes, I would want to be able to get them when running, and not just developing, the app. So there might not even be a terminal - there are more complications than just 'not having to declare where you think it will crash', so I don't think I'll bother solving the more specific problem unless it would help me debug unexpected crashes in a broader context.
Update: this prompted me to look into it again, it's super easy, if you're running an IPython shell, after an error just type "%debug". Or add "--pdb" to your python-shell-interpreter-args.
This gives you a pdb prompt not an IPython prompt, but you can examine local variables where the exception was raised, and exit back to IPython with Ctrl-D.
I'm curious if there is an objective breakdown of emacs org mode and other methods. I'd suspect emacs penetration is not terribly high due to initial learning curve.
What I'm working on right now is a perfect example. Mission: to write a Python program that parses. a file format that has no documentation. Create a folder for the project along with a data directory inside that has some of the files I need to parse. VS Code is my tool of choice, with the Python extension. Use its ability to run Jupyter to play with opening the files and parsing their contents. Experiment with some Jupyter. Get something running. Take my code and paste it into a module. Save the module, which I install locally in develop mode. Import the module into the notebook. Try what I just did against the file to make sure everything works nicely together. Keep going until the file format is completely figured out. Go on to the next challenge.
Frankly I HATE web based notebooks. I do deep learning research, and I write a lot of complex functions to handle pre-processing. And with that there's a lot of debugging I need to do, particularly visual debugging of images. The interactivity of notebooks and the features of an IDE has made my programming experience much better.
I'm curious, what interactivity do you enjoy about notebooks that you find missing in IDEs? I find my situation to be quite the opposite. I love developing/writing code in Pycharm because apart from all the things it excels at (introspection, auto complete, linting, documentation, debugging, etc), it lets me easily prototype things in the interactive python interpreter (usually set to IPython), has an excellent variable viewer and even allows you to attach a debugger to your python interpreter and debug things on-the-fly.
I find myself able to prototype code a lot faster in that setting compared to a Notebook. Even with plots, plotting in Pycharm provides me an interactive plot I can zoom around in, which is not the default behavior in Jupyter. The only situations where I ever use a Notebook is when I need to share a Python "document" that isn't just a python script + results, but rather some form of extensive self-documenting code that could benefit from Markdown integration and co-location of output plots, etc. The other situation is some of their built in animation/interactive widgets (perhaps that is the interactivity you are referring to?). In most other cases, I've found it quite limiting compared to a good IDE like Pycharm/Code.
That being said, I think the efforts on Pycharm and Vscode's part to bridge this gap is commendable and quite interesting.
Microsoft Visual Studio level of Edit and Continue has never been well implemented in open source tooling (with the exception of certain Common Lisp implementations that predate even visual studio, I suspect Turbo Pascal might have this feature too but I have never really used it). This is why most people have to resort to hacks like notebooks and repls with bad UX that spend most of the time spewing errors at the user. It is starting to come back with Flutter, React, and all the livereload stuff but you need a ton of work to get it to work on Python.
If I'm being honest, I haven't really mucked around pycharm. My primary hate really is using a chome/firefox/safari browser window for the notebook interactivity.
Personally, I use Atom with Hydrogen backend, but I develop the files in sch a way that they tend to end up as working for both interactivity and for scripts. So say my train_keras_script.py works both to train when it's deployed on our GPU server, and works as my primary debugging script.
I tend to throw most of my results into a .png output files and .csv's, the latter of which I use for R analysis scripts (and yes I know R Studio has markdown, but I'm just stubborn I suppose ).
In terms of IDE bells and whistles, I think #1 is certainly autocomplete. I will admit that I can be somewhat of a crummy speller. When first learning to program, it caused me a lot of headache trying to debug. Simple syntax checking and autocomplete of variables (especially long variable names that are meant to be descriptive) made me really embrace programming professionally.
The IDE (I also use PyCharms) is great for editing code (search, refactoring, syntax highlight...)
Jupyter notebooks (or JupyterLab in my case) is great for prototyping and data visualisation.
In my work (data engineering and training deep neural nets) I also happen to use both. It's unwieldy, but I can't get rid of either one (pycharms or jupyter lab)
I feel like this is the only sane way to go about things. I really don't understand people using notebooks to actually develop code and train ML/deeplearning models. A real IDE like Pycharm is so much more powerful at debugging, refactoring and in-general, writing good code, structuring it, and not to mention, being able to version control it in a sane manner.
I keep trying VSCode and Jupiter Labs every so often and yet I keep going back to the old Jupiter Notebook web UI. It’s getting better but it’s still not as good. For example, just the way VSCode displays a Pandas dataframe when you do something like .head() just isn’t as nice as Jupiter Notebook’s output. Anecdotally I’ve been developing software for 2 decades from C, Java, all the web crap, using Vim, IDEs, etc. and I _still_ prefer Jupiter Notebooks (not Labs, it sucks) over anything in terms of just how quickly you can iterate. With Vim keystrokes plugin and Black plugin I feel very comfortable.
The problem with notebooks is that they are not searchable and tracking code changes and creating libraries from them is not straight forward.
My approach is to combine the strengths of both IDE and notebooks. I put my code into a package that I edit with PyCharm. I import that code with the `%autoreload 2` magic in Jupyther to load data and test my functions. This way I get data persistency and plotting capabilities of the notebook and editing, etc capabilities of the IDE.
I do something similar, but I also check in my notebooks with all the output cells cleared. It’s not perfect but I can see what changes between commits.
While most of my code is in my modules, I still like having the actual notebook versioned.
I'd compare it loosely to taking notes by hand on a tablet. There is benefit in both, when they are combined it can be a better tool than either alone.
Spyder has a Jupyter notebook plugin. I was able to get a notebook going and link the kernel to the IDE. The variable editor worked and I could debug code in Spyder and then paste it into the notebook. The notebook piece of it was good for graphics, widgets, markdown, etc.
Hard to say as I am not a power-user/coder by any standard least of all those of this forum. I've tried Python on and off over the years and the Spyder 3.x/Jupyter combo was the closest I could get to the kind of UI that Matlab offers: Variable inspector, debugger, persistent/docked plots, documentation, recordkeeping, etc. It wasn't clear whether the Spyder/Jupyter plugin was going to be maintained in future Spyder releases and the combo, although close, was not as smooth as I would have liked.
Ultimately, Matlab is a better fit for the things I want to do in technical computing. My employer offers it and Simulink with a wide variety of toolboxes and blocksets so I can get farther faster for the problems that interest me or that I am asked to solve. The new table datatype and the steadily improving Livescript make exploration and notetaking very easy. I've also been able to rely on Mathworks documentation to get up-to-speed in new areas.
I'm trying to implement these same features in my app, Repla, but without requiring an editor plugin, and instead having a view in a separate app that automatically updates via file-system events when you save. There's some more information here https://repla.app/live-coding.html
It's an approach that I think has somes pros and cons relative to integrating these features into an IDE. The main argument for moving them to a separate app, is that IDEs are already really complicated.
I'm not on macOS, so I can't try it right now, but I've bookmarked this and will keep an eye on it. Basically the report/output view of Iodide/Jupyter without the "editor" view? I was really hoping something like this would come along.
Yes, that's exactly right. Great to hear other people are also interested in this approach! The technology I developed has ended up being pretty flexible, you can read about it here (https://blog.repla.app/2020/01/13/what-is-repla/) and here (https://blog.repla.app/2020/01/13/repla-use-cases/) if you're curious. But yeah, that's exactly the use case I developed the technology for. I wanted a browser engine with a plugin system so that I could use to display output of systems like Jupyter and Observable HQ, while still editing the source code in my exiting text editor, the same way I edit other source code. I sort of discovered along the way that this system is also a nice way to run a local development web server, and that was a bit easier to implement, so that's what I've launched with first, the homepage has more info on that (https://repla.app/).
I'd like to support Linux and Windows eventually too, but it's easier to start with supporting just one platform while I'm just testing out whether there's other people interested in the idea.
I see this, and I cannot help but think, hey, one step closer to achieving the plain-text goodness magic of the R Notebook. But maybe I'm being too sassy. Its just that every time I pick up Jupyter I think, what is this clumsy thing? I know people love it, but that's how I feel. I wish pycharm had the in-line capabilities of the R Notebook, because I really like pycharm.
I don’t do data science and I really don’t understand what Jupyter notebooks are used for. Are they just a way to explore ideas with immediate feedback?
You can publish them on github or mybinder as persistent self documenting interactive repl sessions. Then others can see what the code did, the intermediate results, and can easily play with it.
(No "data science" here either, but into repl based instant feedback development & collaboratiove programming tools)
This misses a pretty stellar entrant in the field, Atom + Hydrogen. Works nicely with multiple kernels (I use R and Python), allowing notebook-like functionality across multiple languages.
You're right, I missed that, thanks for pointing it out. I wish it had ranked high enough for a comparison though, seems to solve some of the issues raised (though perhaps causes others).
