The Jupyter notebook development environment is very popular in the scientific Python community. With this tool you can write markdown and code in the same file, which can increase the readability of the program. This paradigm is known as literate programming.
An example of a Jupyter notebook including a section of markdown and Python code is shown in the following.
Literate programming is a very well regarded concept, formally discussed by respected researchers like Donald Knuth. At the same time, Jupyter notebooks are considered inefficient for serious software development. This controversy led to the famous “I don’t like notebooks” talk, responded by the “I like Jupyter notebooks” talk that generated quite some drama within the community.
Jupyter notebooks have been limited to explore data, small scripts, and educational materials. The recent introduction of nbdev, a python library, expanded the capabilities of notebooks. Nbdev allows the development and distribution of python packages using the literate programming style, as well as test, document, and develop technical files.
Nbdev allows you to tell a story with your code. It is a practical and powerful tool!
Nbdev has proven to be uselful in developing big and serious projects, like FastAi. Relying on nbdev and Jupyter notebook allows software development with quality and the benefits of the literate programming style.
Experience
My first experience with nbdev was weird. Working with Jupyter notebooks for software development didn’t feel natural, as expected, afterall it’s a new paradigm.
The software tool that I’m currently working on, Ipyannotator, is an open source framework for annotations with a Graphical User Interface (GUI) that works on top of Jupyter notebook. Ipyannotator development was possible due to the nbdev allowing us to test the software on notebooks and export the code as a Python library.
After 10 months of developing software with Jupyter notebook I feel that this has increased my productivity.
This software paradigm allowed me to keep my documentation up to date since the documentation is written right above my Python code block. This literate programming paradigm also reduces the mental lapse of going back to old code, making it easier to understand previous design decisions, since they are already documented.
Quarto
On July 28th, nbdev released its second version and incorporated another open source tool into its arsenal, Quarto. Quarto allows the creation of technical content using Python, R or Julia, using files such as Jupyter notebooks, markdown, R Notebooks and allowing to export these files as articles, websites, blogs, HTML, PDFs, EPub, etc.
Quarto is a complete solution to easily write any technical material with the tools you already know.
This tool is so awesome that I’m planning to migrate my personal blog from Jekyll to Quarto. It’s clean in design, usability, and ability to execute code embedded by text blocks, which fits perfectly any technical writing requirements.
Nbdev noticed how powerful Quarto can be. This addition helps with website development and documentation rendering. The first version of nbdev already contained some of these features, however, they were rudimentary and required several workarounds. In the second version nbdev added Quarto to its core.
Conclusion
Development with nbdev and Quarto can increase the productivity of many teams, helping to document and tell the story of your software. These tools also expand the possibilities of software development, as was the case with the already exemplified Ipyannotator.
Like any tool, it also has its tradeoffs. One of them is the lack of auto-complete in Jupyter notebooks, something that can be avoided by using the visual code extension. Another problem is that Jupyter notebook writes json files, a format that makes git conflict resolution difficult due to extra metadata alongside the code. Making conflict resolution difficult minimizes the chances of big teams working on the same software.
Fortunately, the addition of Quarto shows that nbdev already plans to use other files besides Jupyter notebooks in its development core. This enables large teams to develop software with literate programming, decreasing the main tradeoff.
If you are interested in trying nbdev you can check the following reference: nbdev+Quarto: A new secret weapon for productivity. If you’re curious about Jupyter notebooks code for software development just check Ipyannotator code.