Dr. Jelena Mirkovic was invited to deliver a keynote at the IEEE sponsored workshop on Computer and Networking Experimental Research Using Testbeds (CNERT). Jelena spoke about "The Art and Science of Testbed Experimentation" to an audience of approximately 50 people from academia (faculty and students) and industry. As Figure 1 (below) from her talk illustrates, outsiders imagine tesbed experimentation as a sophisticatd and seamless process, where templates and tools are at the researcher's fingertips and can be easily combined into complex experiments. Researchers themselves view testbeds as environments where they can scientfically test their hypothesis, through careful parameter exploration in a realistic setting. However, what usually happens is that an experiment is built in an ad-hoc manner, over a long period of time, using a patchwork of tools and data. Such an experiment is very fragile and not reapeatable, sometimes not even by its own author(s). Dr. Mirkovic referred to this current state of testbed experimentation as "art," because each experiment is unique and hard to accurately reproduce by others.
Jelena also talked about the future of testbed experimentation, where we would have more science and less art (as it were). This future would enable testbed-assisted experiment design, where a researcher could start from an existing experiment, understand easily what pieces the experiment consists of, and then change some pieces to align the experiment to the researcher's unique needs. It would also enable testbed-assisted runs, where all the actions and data are automatically saved. Further, it would enable easy sharing, where all pieces of data and code are included and structred in a way to facilitate easy reuse. With such infrastructure, we could build repeatable, reproducible and reusable experiments and enable vertical development of the related science fields, such as cybersecurity, distributed systems and networking.
Dr. Mirkovic mapped a path forward, illustrated in Figure 2 (below). This path consists of several layers, where services at the lower layers become foundations for the upper layers. At the bottom is the new experiment representation, which focuses on the experiment behavior. This representation, called DEW (Distributed Experiment Workflow), talks about what the experiment does (scenario), how it does this (bindings) and where (requirements for the topology). Also essential are translators, which would produce DEW from the current scripts (e.g., bash, MAGI) and topologies (e.g., NS-2, Rspec) and generator, which would generate scripts and topologies from DEW. Jelena elaborated on how DEW's simple representation could provide input to other services, such as fault tolerance, experiment orchestration, monitoring and version control. On top of these foundational services, communities could build libraries and benchmarks, frameworks for hypothesis testing, identification of similar experiments, etc. Ultimately, achieving the goals of easy sharing and resuse, repeatable and reproducible experimentation and vertical development of experimental science.
