robotframework-jupyterlibrary#
A Robot Framework library for automating (testing of) Jupyter end-user applications and extensions
pip |
conda |
docs |
demo |
actions |
---|---|---|---|---|
Using#
Write .robot
files that use JupyterLibrary
keywords… or use magics in
notebooks.
*** Settings ***
Library JupyterLibrary
Suite Setup Wait For New Jupyter Server To Be Ready jupyter-lab
Test Teardown Reset JupyterLab And Close
Suite Teardown Terminate All Jupyter Servers
*** Test Cases ***
A Notebook in JupyterLab
Open JupyterLab
Launch A New JupyterLab Document
Add And Run JupyterLab Code Cell
Wait Until JupyterLab Kernel Is Idle
Capture Page Screenshot
See the acceptance tests for examples.
Installation#
pip install robotframework-jupyterlibrary
Or
mamba install -c conda-forge robotframework-jupyterlibrary
Or (if you must):
conda install -c conda-forge robotframework-jupyterlibrary
Or see the contributing guide for a development install.
Free Software#
JupyterLibrary is Free Software under the BSD-3-Clause License. It contains code from a number of other projects:
-
Initial implementations of robot keywords
Some of its testing approaches (only distribtued in source form, not e.g. wheels) are also derived from other tools:
-
Initial implementation of kernel-under-test coverage instrumentation
Documentation Contents#
WHY#
…Jupyter?#
Jupyter clients are some of the most powerful pieces of technology users can run in their web browsers. By developing cross-client capabilities, either as kernel-level extensions, widgets, media types, or other confections, you are helping to advance fields of inquiry you might not even know exist.
…Acceptance Tests?#
Unit tests and strongly-typed languages are superb for rapid, confident iteration on even large codebases. But users will be installing Your Code next to an unknown number of Other People’s Code, and then write Their Code. If^H^H
When it breaks, they might not be able to tell that it’s the subtle interaction between these things. Testing All the Code together, as your user will use it, gives you greater confidence in your ability to ship.
…JupyterLibrary?#
Powered by Robot Framework and SeleniumLibrary, JupyterLibrary
allows you to:
write tests in concise, plain language
and extend this language to meet your needs
test multiple Jupyter clients
and multiple versions of them
run in multiple, real browsers (even at the same time)
and on multiple operating systems
view rich reports of your test results
but also compare your reports over time with machine-readable formats
generate screenshots to augment your documentation
INSTALL#
Installing JupyterLibrary
will bring along Robot Framework and SeleniumLibrary. Jupyter components, like notebook
, jupyterlab
and nteract_on_jupyter
, and browser executors (e.g. chromedriver
, geckodriver
) and various utilities (e.g. nodejs
) are up to you, depending on what you want to test. Here are some examples.
pip
#
pip install robotframework-jupyterlibrary
mamba
#
mamba install -c conda-forge robotframework-jupyterlibrary
conda
also works, usually, butmamba
is both faster and provides better error messages when things go wrong.
main
#
JupyterLibrary
is under active development, and is heavily invested in the conda
ecosystem, and related tools and mamba
and conda-lock
, because of the complexity of managing browser execution dependencies. But conda
and mamba
(rightly) make it hard to install Random Repos from the Internet, so you’ll need a bit of pip
, too.
Start with a sensible, activated base
like Mambaforge. Mixing the miniconda
or anaconda
distributions’ defaults
(e.g. anaconda.com
is a recipe for disaster, and may violate the terms of service).
Here’s a complete setup:
mamba create \
# using `--prefix=.venv` is also useful for having predictable file locations, but can confuse IDEs
--name testing-jupyter \
--channel conda-forge \
# CPython 3.8+ required, not tested with pypy, or (near) end-of-life CPython
python=3 \
jupyterlab \
robotframework-seleniumlibrary \
geckodriver \
# using a Long Term Support (LTS) Firefox is useful for avoiding "only works in Chrome"
firefox
Activate the environment:
source activate testing-jupyter
Install “hot” dependencies:
pip install \
# don't want any "surprises"
--no-deps \
# just to be sure
--ignore-installed \
git+http://github.com/robots-from-jupyter/robotframework-jupyterlibrary
Contributing to JupyterLibrary#
Get CONDA_EXE
#
Get Mambaforge
mamba install -c conda-forge doit
# optional meta-dependency
mamba install -c conda-forge conda-lock
Get the code#
git clone http://github.com/robots-from-jupyter/robotframework-jupyterlibrary
cd robotframework-jupyterlibrary
Doit#
Listing all the tasks#
doit list
Just run (just about) everything#
doit release
Lock Files#
After adding/changing any dependencies in
.github/env_specs
, the lockfiles need to be refreshed in.github/locks
and committed.
doit lock
Bootstrapping from no lockfiles requires an external provider of
conda-lock
. It may require runningdoit lock
a few times to get a stable set of environment solutions.
Reproducing CI failures#
By default, the doit
scripts use the lockfile most like where you are developing,
hoping for a better cache hit rate. On the same operating system, however, any of the
pre-solved lockfiles can be used, by specifying the RJFL_LOCKFILE
environment
variable.
For example, if linux-64
running python3.8
with jupyterlab 3
failed:
!/usr/bin/env bash
set -eux
RFJL_LOCKDIR=test/linux-64/py3.8/lab3 doit release
Or, in a bat
script:
@echo on
set RFJL_LOCKDIR=test/win-64/py3.8/lab3
doit release
This will recreate the test
environment with the specified lockfile, and repeat all
the steps.
Environment Variables#
A number of environment variables control how some of the doit
tasks function.
variable |
default |
note |
---|---|---|
|
|
a JSON array of tokens to pass to |
|
|
number of times to re-run failing tests |
|
|
where to start in the retry order |
|
|
which browser to use (only tested with FF) |
|
|
a custom |
|
|
skips a number of steps |
|
`` |
|
Releasing#
[ ] merge all outstanding PRs
[ ] start a release issue with a checklist (maybe like this one)
[ ] ensure
pyproject.toml#/project/version
has been increased appropriately[ ] ensure the
HISTORY.ipynb
is up-to-date[ ] validate on binder
[ ] validate on ReadTheDocs
[ ] wait for a successful build of
main
[ ] download the
dist
archive and unpack somewhere (maybe a freshdist
)[ ] create a new release through the GitHub UI
[ ] paste in the relevant
HISTORY
entries[ ] upload the artifacts
[ ] actually upload to pypi.org
doit publish
[ ] postmortem
[ ] handle
conda-forge
feedstock tasks[ ] validate on binder via simplest-possible gists
[ ] activate the version on ReadTheDocs
[ ] bump
pyproject.toml#/project/version
to next development version[ ] update release procedures
Appendix: Current doit
tasks#
doit
is used heavily in development and continuous integration.
binder Get to a basic interactive state.
build Build packages.
build:hash generate a hash file of all distributions
build:pypi build the pypi sdist/wheel
conda_build Build conda package.
conda_build:build use boa to build the conda package
conda_build:recipe update the conda recipe
docs Build HTML docs.
docs:rtd:env generate a readthedocs-compatible env
docs:sphinx build the docs with sphinx
env
env:docs create the local docs environment
env:lint create the local lint environment
env:meta create the local meta environment
env:test create the local test environment
js Javascript cruft.
js:yarn install nodejs dev dependencies
lab Start a jupyter lab server (with all other extensions).
lab:serve runs lab (never stops)
lint Lint code.
lint:black ensure python code is well-formatted
lint:prettier ensure markdown, YAML, JSON, etc. are well-formatted
lint:robocop ensure robot code is well-behaved
lint:robotidy ensure robot code is well-formatted
lint:ruff ensure python code is well-behaved
lint:ssort apply source ordering to python
lock Generate conda lock files for all the excursions.
lock:docs__linux-64 lock the docs environment for linux-64 []
lock:docs__osx-64 lock the docs environment for osx-64 []
lock:docs__win-64 lock the docs environment for win-64 []
lock:lint__linux-64 lock the lint environment for linux-64 []
lock:lint__osx-64 lock the lint environment for osx-64 []
lock:lint__win-64 lock the lint environment for win-64 []
lock:meta__linux-64 lock the meta environment for linux-64 []
lock:meta__osx-64 lock the meta environment for osx-64 []
lock:meta__win-64 lock the meta environment for win-64 []
lock:test__linux-64__py3_11__lab3 lock the test environment for linux-64 (ft. py3.11, lab3)
lock:test__linux-64__py3_11__lab4 lock the test environment for linux-64 (ft. py3.11, lab4)
lock:test__linux-64__py3_8__lab3 lock the test environment for linux-64 (ft. py3.8, lab3)
lock:test__linux-64__py3_8__lab4 lock the test environment for linux-64 (ft. py3.8, lab4)
lock:test__osx-64__py3_11__lab3 lock the test environment for osx-64 (ft. py3.11, lab3)
lock:test__osx-64__py3_11__lab4 lock the test environment for osx-64 (ft. py3.11, lab4)
lock:test__osx-64__py3_8__lab3 lock the test environment for osx-64 (ft. py3.8, lab3)
lock:test__osx-64__py3_8__lab4 lock the test environment for osx-64 (ft. py3.8, lab4)
lock:test__win-64__py3_11__lab3 lock the test environment for win-64 (ft. py3.11, lab3)
lock:test__win-64__py3_11__lab4 lock the test environment for win-64 (ft. py3.11, lab4)
lock:test__win-64__py3_8__lab3 lock the test environment for win-64 (ft. py3.8, lab3)
lock:test__win-64__py3_8__lab4 lock the test environment for win-64 (ft. py3.8, lab4)
publish Publish distributioons.
publish:pypi upload python sdist and wheel to PyPI
release Run the full set of tasks needed for a new release.
report Generate reports of test data.
report:cov:combine gather coverage
report:cov:html:rfjl generate coverage html
report:cov:html:rfsl generate coverage html
report:cov:html:se generate coverage html
report:cov:report emit coverage console report and check
report:robot:combine combine all robot outputs into a single HTML report
setup
setup:docs [docs] python development install
setup:lint [lint] python development install
setup:test [test] python development install
test (dry)run tests.
test:atest run acceptance tests with robot
test:dryrun pass the tests through the robot machinery, but don't actually _run_ anything
KEYWORDS#
Keywords are the the smallest unit of Robot Framework tasks and tests. The built-in Robot Framework documentation is pretty good, but for various reasons, are split out below.
click 🔎 in the bottom right to filter
Browser + Server#
JupyterLibrary
inherits all of the keywords of SeleniumLibrary, and then adds a few more, including the two most important ones:
Wait For New Jupyter Server To Be Ready
Terminate All Jupyter Servers
All the server keywords include Jupyter in the keyword name, and all of the client keywords are also dynamically loaded. A few screenshot convenience methods are also provided.
Clients#
The Jupyter client keywords are themselves defined in .robot
files, and are loaded dynamically. They all include the name of the client in the keyword name.
JupyterLab#
Jupyter Notebook#
Jupyter Notebook Classic#
Common#
A number of libraries are shared between multiple frontends, and so can use the same underlying keywords.
CodeMirror#
The real workhorse of the Jupyter code editing experience, CodeMirror 6 is used by JupyterLab and Notebook, while Notebook Classic uses CodeMirror 5.
MAGIC#
JupyterLibrary
provides a few lightweight IPython magics for its own testing purposes.
If you like writing and executing Robot Framework in a Jupyter kernel, you might like a more full-featured experience:
%reload_ext JupyterLibrary
The %%robot
magic runs a cell of code as you would write in a .robot
file. No funny stuff (by default).
%%robot -o _static
*** Tasks ***
Log Something
Log Something
🤖 making files in
/home/docs/checkouts/readthedocs.org/user_builds/robotframework-jupyterlibrary/checkouts/stable/docs/_static/_robot_magic_/b48ff0e77cff
🤖 running!
-
stdout.txt
============================================================================== Untitled b48ff0e77cff ============================================================================== Log Something | PASS | ------------------------------------------------------------------------------ Untitled b48ff0e77cff | PASS | 1 task, 1 passed, 0 failed ============================================================================== Output: /home/docs/checkouts/readthedocs.org/user_builds/robotframework-jupyterlibrary/checkouts/stable/docs/_static/_robot_magic_/b48ff0e77cff/output.xml Log: /home/docs/checkouts/readthedocs.org/user_builds/robotframework-jupyterlibrary/checkouts/stable/docs/_static/_robot_magic_/b48ff0e77cff/log.html Report: /home/docs/checkouts/readthedocs.org/user_builds/robotframework-jupyterlibrary/checkouts/stable/docs/_static/_robot_magic_/b48ff0e77cff/report.html
-
stderr.txt
empty
🤖 returned 0
The interactive help is pretty good.
%%robot?
Of note: you can specify extra arguments to robot.run
with -a
, the name of a local variable.
args = dict(include=["mytag:a"])
%%robot -a args -o _static
*** Tasks ***
Do thing A
[Tags] mytag:a
Log A
Do thing B
[Tags] mytag:b
Log B
Do thing AB
[Tags] mytag:a mytag:b
Log AB
🤖 making files in
/home/docs/checkouts/readthedocs.org/user_builds/robotframework-jupyterlibrary/checkouts/stable/docs/_static/_robot_magic_/0b0dd3cec340
🤖 running!
-
stdout.txt
============================================================================== Untitled 0b0dd3cec340 ============================================================================== Do thing A | PASS | ------------------------------------------------------------------------------ Do thing AB | PASS | ------------------------------------------------------------------------------ Untitled 0b0dd3cec340 | PASS | 2 tasks, 2 passed, 0 failed ============================================================================== Output: /home/docs/checkouts/readthedocs.org/user_builds/robotframework-jupyterlibrary/checkouts/stable/docs/_static/_robot_magic_/0b0dd3cec340/output.xml Log: /home/docs/checkouts/readthedocs.org/user_builds/robotframework-jupyterlibrary/checkouts/stable/docs/_static/_robot_magic_/0b0dd3cec340/log.html Report: /home/docs/checkouts/readthedocs.org/user_builds/robotframework-jupyterlibrary/checkouts/stable/docs/_static/_robot_magic_/0b0dd3cec340/report.html
-
stderr.txt
empty
🤖 returned 0
Running JupyterLibrary
#
The line below is a Markdown Cell… change it to a Code Cell to run it
%%robot
*** Settings ***
Documentation A nice task suite
Library JupyterLibrary
Suite Setup Wait For New Jupyter Server To Be Ready
Test Teardown Reset JupyterLab And Close
Suite Teardown Run Keyword And Ignore Error Terminate All Jupyter Servers
*** Tasks ***
A Notebook in JupyterLab
Open JupyterLab
Launch A New JupyterLab Document
Add And Run JupyterLab Code Cell print("hello" + " world")
Wait Until page Contains hello world
Capture Page Screenshot ran-code.png
With Widgets#
There is some more stuff coming with
%%robot
, but for now,ipywidgets.interact
can be used to quickly build UI around robot-generated artifacts
from pathlib import Path
from IPython.display import display, Image
ipywidgets = None
try:
import ipywidgets
except:
pass
if ipywidgets:
@ipywidgets.interact
def show_image(i=(0, 100)):
all_images = sorted(Path("_robot_magic_").rglob("*.png"), key=lambda p: p.stat().st_mtime)
if not all_images:
return
start = all_images[0].stat().st_mtime
i = min(len(all_images) - 1, i)
img = all_images[i]
delta = img.stat().st_mtime - start
display(f"[{round(delta)}s][{i} of {len(all_images)}] {img.name}", Image(img))
CI#
At first, you’ll want to write your tests locally, and test them against as many local browsers as possible. However, to really test out your features, you’ll want to:
run them against as many real browsers on other operating systems as possible
have easy access to human- and machine-readable test results and build assets
integration with development tools like GitHub
Enter Continuous Integration (CI).
Providers: Cloud#
Multi-Provider#
Historically, Jupyter projects have used a mix of free-as-in-beer-for-open source hosted services:
Each brings their own syntax, features, and constraints to building and maintaining robust CI workflows.
JupyterLibrary
started on Travis-CI, but as soon as we wanted to support more platforms and browsers…
Azure Pipelines#
At the risk of putting all your eggs in one (proprietary) basket, Azure Pipelines provides a single-file approach to automating all of your tests against reasonably modern versions of browsers.
JupyterLibrary
was formerly built on Azure, and looking through pipeline and various jobs and steps shows some evolving approaches…
Github Actions#
At the risk of putting all your eggs in one (proprietary) basket, if your code is on Github, Github Actions offers the tightest integration, requiring no aditional accounts.
JupyterLibrary
is itself built on Github Actions, and looking at the workflows offers some of the best patterns we have found.
Providers: On-Premises#
Jenkins#
If you are working on in-house projects, and/or have the ability to support it, Jenkins is the gold standard for self-hosted continuous integration. It has almost limitless configurability, and commercial support is available.
warnings-ng
can consume many outputs ofrobotframework
Approach: Environment management#
Acceptance tests need benefit from tightly-controlled, but flexibly-defined environments.
this repo uses (and recommends)
conda-lock
andmamba
to manage multiple environmentssimpler cases, such as pure-python projects, can use
tox
Approach: It’s Just Scripts#
No matter how shiny or magical your continuous integration tools appear, the long-term well-being of your repo depends on techniques that are:
simple
cross-platform
as close to real browsers as possible
easily reproducible outside of CI
Practically, since this is Jupyter, this boils down to putting as much as possible into platform-independent python (and, when neccessary, nodejs) code.
JupyterLibrary
uses doit to manage a relatively complex lifecycle across multiple environments with minimal CLI churn.
doit
has very few runtime dependencies, and works well with caching, etc.Environment variables are used for feature flags
aside from some inevitable path issues, environment variables are easy to migrate onto another CI provider
A small collection of development scripts, not shipped as part of the distribution, provide some custom behaviors around particularly complex tasks.
sometimes
doit
is too heavy of a hammer for delicate work
Approach: Single Test Script#
Having a single command that runs all unit, integration, and acceptance tests is a useful property of a project.
make
(or the more pythonicdoit
, used in this repo) make this most robustusually, all unit tests need to be re-run when any functional source, e.g.
*.ts
and*.py
acceptance tests often need to be run when almost anything changes, including
.css
, build configuration files, etc.
wrap
robot
execution in another toolfor example, jupyter-server-proxy launches
robot
from withinpytest
use
tox
for pure-python test management
Approach: Log Centralization#
After a full test run, it can be useful to combine many test results into a single, navigable page
in CI, download all the test result archives and put them together
rebot
can combine multiple runs, including retries, into a single HTML report
embed different kinds of results
pytest-html
can embed generated reportswhen embedding
robot
reports with screenshots, useSet Screenshot Directory EMBED
to make this easierother files like logs can also be embedded
create a single log aggregation HTML page
jupyterlab-deck generates and publishes a notebook/slideshow containing all of its logs
this is served as a JupyterLite site, so the underlying (semi-)machine-readable is also available to
Approach: Caching#
Most of the CI providers offer nuanced approaches to caching files. Things to try caching (it doesn’t always help):
packages/metadata for your package manager, e.g.
conda
,pip
,yarn
built web assets
Approach: Pay technical debt forward#
A heavy CI pipeline can become necessary to manage many competing concerns. Each non-trivial, browser-based robot test can easily cost tens of seconds. Some approaches:
use an up-front dry-run
robot
testthis can help catch whitespace errors in robot syntax
this usually costs $\frac{\sim1}{100}$ the time of running the full test
run tests in subsets, in parallel, and in random order with
pabot
requires avoiding shared resources, e.g. network ports, databases, logfiles
if neccessary, declare explicit dependencies with e.g.
DependencyLibrary
orpabot
’s#DEPENDS
Approach: Get More Value#
While the pass/fail results of a test are useful in their own right, acceptance tests can provide useful artifacts for other project goals.
gather additional coverage insrumentation
[x] client:
[x] jupyterlab-deck uses
istanbul
andnyc
to collect browser code coverage
[x] kernel and widgets:
[x] this repo gathers kernel coverage from JupyterLab-based tests iof its custom
%%robot
IPython magic[x] ipyforcegraph tests custom Jupyter widgets
[ ] serverextension: TODO
[ ]
.robot
suites: TODO
use generated screenshots
[ ] reporting: TODO
[ ] accessibility: TODO
[ ] documentation: TODO
[ ] PDF generation: TODO
[ ] revisit when supported by
geckodriver
LIMITS#
NotebookApp
vs ServerApp
#
Prior to JupyterLibrary 0.4.2
, Start New Jupyter Server
relied on backwards compatibility of jupyter_server
with notebook
, using e.g. --NotebookApp.token
to configure temporary credentials.
With jupyter_server>=2
, nbclassic
and various other newer packages vying for the CLI, this doesn’t always pick the correct tool, so several options are available:
explicitly setting the named
app_name
parameter when launching a serverexplicitly setting the
command
parameter will usually pick the correct servere.g.
jupyter-lab
should usually beServerApp
using the new keyword
Set Default Jupyter App Name
setting the
JUPYTER_LIBRARY_APP
environment variable, either from the CLI, or in CIenvironment
, will influence the default behavior
Press Keys
on MacOS/Chrome#
While SeleniumLibrary 3.3.0 added Press Keys
which can target non-inputs, as of chromedriver
version 2.45
the ⌘ key cannot be used. As this is the favored key for shortcuts, this means almost all of the client keyboard shortcuts just won’t work if you are trying to test on MacOS.
Workaround
If you are trying to
Press Keys
where the ⌘ key would be used, try to find a combination of simpler key combinations and mouse clicks.
HISTORY#
0.5.0#
Products under test |
Versions |
---|---|
Python |
|
CodeMirror |
|
Robot Framework |
|
Jupyter Notebook |
|
JupyterLab |
|
Jupyter Server |
|
0.5.0a0#
Products under test |
Versions |
---|---|
Python |
|
CodeMirror |
|
Robot Framework |
|
Jupyter Notebook |
|
JupyterLab |
|
Jupyter Server |
|
0.4.3#
Products under test |
Versions |
---|---|
Python |
|
Robot Framework |
|
Jupyter Notebook Classic |
|
JupyterLab |
|
Jupyter Server |
|
0.4.2#
Products under test |
Versions |
---|---|
Python |
|
Robot Framework |
|
Jupyter Notebook Classic |
|
JupyterLab |
|
Jupyter Server |
|
supports
robotframework
6drops support for
robotframework
3tests
jupyter_server
2to account for some deprecations, the app name may need to be set when starting a managed Jupyter/notebook server
the new keywords
Set Default Jupyter App
andGet Jupyter App Name
allow for changing auto-detection based on CLI commandan environment variable
%{JUPYTER_LIBRARY_APP}
(default:NotebookApp
) can be set toServerApp
for when combinations ofnotebook
,nbclassic
,jupyter_server
andjupyterlab
break autodetection.
0.4.1#
Products under test |
Versions |
---|---|
Jupyter Notebook Classic |
|
JupyterLab |
|
selenium
4.5 is now supportedGet WebElements Relative To
(and the singular) are now available as keywords
%%robot
magic ignores--pretty
ifrobot.tidy
is unavailableSome keywords now have type hints.
0.4.0#
Products under test |
Versions |
---|---|
Jupyter Notebook Classic |
|
JupyterLab |
|
Products under review |
Versions |
---|---|
Retrolab |
|
Voila |
|
Kernel launcher keywords are more lax to account for more-spefic names, e.g.
Python 3 (ipykernel)
Put all robot source code under formatting/linting by robotidy and robocop
The minimum Python has been raised to 3.7, replacing the now-EOL Python 3.6 in the test matrix
Python 3.10 replaces Python 3.8 in the CI test matrix
0.3.1#
Products under test |
Versions |
---|---|
Jupyter Notebook Classic |
|
JupyterLab |
|
Products under review |
Versions |
---|---|
JupyterLab Classic |
|
Voila |
|
Several JupyterLab keywords now accept an
${n}
argument to handle multiple documents on the page.Many JupyterLab keywords that wait for certain events can be configured with
${timeout}
and${sleep}
to suit.Properly pass library initialization options to
SeleniumLibrary
0.3.0#
Products under test |
Versions |
---|---|
Jupyter Notebook Classic |
|
JupyterLab |
|
Require SeleniumLibrary 4.5 and RobotFramework 3.2
Expanded support for newer Notebook Classic and JupyterLab versions in keywords
Dropped support for
nteract_on_jupyter
0.2.0#
Require SeleniumLibrary 3.3.0 and remove backport of
Press Keys
Start New Jupyter Server
now has a defaultcommand
ofjupyter-notebook
(instead ofjupyter
)Build Jupyter Server Arguments
no longer returnsnotebook
as the first argumentFix homepage URL for PyPI
Test on Chrome/Windows
0.1.0#
Initial Release