BilderSkript

Project Definition

Motivation and Goal

A lecturer usually provides a lot of supplemental course material such as a written script, slides, reading material, exercises.

However, students visiting the course’s lecture experience an additional channel, which helps them to sort and weight the supplemental material. The way how a lecturer presents the content will let attendees realize the weight and importance of certain parts of the content. Hence, it supports the creation of a red thread throughout entire supplemental material.

In contrast to full lecture recording, the BilderSkript approach condenses the recording to find those parts within a lecture which contribute to the understanding of the supplemental material.

Primarily, BilderSkript aims at course attendees, however, it could also be valuable for remote attentees.

BilderSkript has the following goals

• re-create partially the course experience
• support attenting students to follow-up on the content
• support students who have not attended all lessions to catch-up on the course
• improve the efficiency when working with the supplemental course material

Objectives

There are a couple of activities to perform

• record lectures as a sequence of wide angle still images
• train a computer to identify objects within still images
• define a metric of interestingness from the sequence of recognized objects
• extract interesting images and compile a lecture notebook from which ultimately creates the BilderSkript

Research Questions

BilderSkript relies on widely available open-source ML software. However, there are a couple of interesting research questions on the way.

• How to automate image preprocessing? - Image recording activity does not happen in a controlled environment, i.e. lecture rooms change, the light varies, the camera positions are not fixed (resulting in varying field of views), varying colors of background, cloths etc.
• Can we successfully transfer other object identfication models to lecture recordings? Object identfication utilizes labeled data from one or few lectures for training. Since labeling is laborious, we are interested to transfer trained models from other areas and apply for object identfication in our lecture recordings.

System Design

This use case diagram depicts BilderSkript’s main services. Note, the shown use cases do not imply an execution order.

The engineer performs the system setup, configures and trains the system for the user to be beneficial.

Pipelines

Pipelines are the fundamental building blocks of the BilderSkript system.

Pipeline Definitions

The term pipeline is heavily used in machine learning (ML). It generally refers to a sequence of steps to run in order to perform transformations. There are various kinds of pipelines, e.g. data pipelines, machine learning pipelines, deployment pipelines and others. Depending on the pipeline type, it takes a a certain ressource type as input and produces output ressources by the application of the pipeline’s transformation steps. For instance, a data pipeline takes data files as an input and prepares them to be fed into a machine learning algorithm of the ML pipeline.

BilderSkript stores pipelines in the pipelines volume usually accessed under /bilderscript/pipelines. They are encoded as snakemake makefiles for version control.

Snakemake

Snakemake is a workflow management system to implement data analysis pipelines.

Snakemake compares the input and output ressources. These ressources are files. If the modification date of any of the input files is newer than the output file, snakemake runs the shell command. This behavior is encoded as rules, which transform input files into output files. All rules of a pipeline are defined in a snakefile. We list some important snakemake commands.

Run the snakemake pipeline

snakemake <pipeline name>

Generate a report

snakemake <pipeline name> --report <snakefile.html>

Generate a summary table displaying the current state of input and output files.

snakemake <pipeline name> --summary

Before the snakefile is run, snakemake generates a DAG which shows the dependencies. This command visualizes the DAG.

snakemake <pipeline name> --dag | dot -Tpng > <pipeline name>.png

[data_prep.snakefile] Pipeline

The data preparation pipeline, data_prep.snakefile, comprises of two phases

1. Configure the pipeline’s parameters
   • config.yaml defines pipeline specific parameters
   • datasets.csv defines the data input and data-specific processing parameters
2. Run the pipeline on the images from the lecture recording
   

1. Configuration

The following activity diagram describes the steps to configure the data preparation pipeline. Configuration is stored in config.yaml and dataset.csv.

Precondition:

• builder and hugin-vnc container up and running
• at least one image from lecture recording available

Postcondition:

• images directories set
• .pto files created
• (optionally) crop specification defined

The parameter values from above need to be stored in the config.yaml and dataset.csv file. The activity diagram indicates to which file each parameter belongs to.

Commands

Spin up the hugin-vnc container.

docker-compose up -d hugin-vnc

Afterwards, use a vnc client to get Hugin’s display and create the .pto files. Modify the pipeline’s dataset.csv and config.yaml to tell the scripts where to look for the files. The activity diagram below displays the steps to create the files.

2. Run the Pipeline

Finally, the engineer starts the data prep pipeline and the pipeline processes the input images from the img_dir and places it in the out_dir.

Precondition:

• builder and hugin container up and running
• .pto files created
• img_dir and out_dir image directories set

Parameter values from above are sourced from the the dataset.csv file.

Postcondition:

• images processed placed in out_dir

Pipeline Start

The following commands start the pipeline. The activity diagram belows depicts the details of the pipeline run.

Spin up the builder and hugin container and get an interactive shell to the builder container.

docker-compose up -d builder hugin
docker exec -it bilderskript_builder_1 /bin/bash

Within the builder container run

cd BilderSkript/pipelines
./data_prep.sh

[doc.snakefile] Pipeline Generating the Blog

The pipeline doc.snakefile generates

1. UML diagrams using plantuml
2. the project’s website using hugo

Run the pipeline from within the builder container in /bilderskript/pipelines

snakemake doc

Labeling

This activity assigns names or descriptors to components within an image. Any supervised ML algorithm requires labeled data for training. Successfully trained on the labeled data, the ML algorithm is afterwards enabled to identify and name similar components in images which do not have labels.

Labeling is usually done manually. A user marks regions within images and assigns labels therewith identifying objects in the image. Labeled data is not questioned and provides the algorithm with ground truth data.

Problem

BilderSkript delivers images from various lectures. We will need to repeat the labeling process to find appropriate features which let the ML based object identification algorithm generalize sufficiently well in order to identify objects in recordings from other lectures.

Tool Support

Labeling objects within images has a long history in the computer vision research community. For ML-based products there are a couple of online services available which support labeling activties distributed across a crowd workers, integrate semi-automated quality checks and other functions when it come to large-scale applications.

Quora list not-complete list of labeling tools and services; some are comercial, some are open-source. Check out https://www.quora.com/What-is-the-best-image-labeling-tool-for-object-detection.

For BilderSkript we found the following open tools tools attractive and discuss them briefly.

• ImgLab: web based tool; Online service for immediate use available under https://imglab.in
• LabelImg: the classic one, often mentioned on towardsdatascience.com. Google finds approx. 45 hits. It is a desktop tool.
• Alp’s Labeling Tool (ALT): more than just a labeling tool; it’s a desktop tool for Windows as well as Linux / Ubuntu.
• Computer Vision Annotation Tool (CVAT): web based online tool. Works with videos, too. Apart from the web GUI, there is also a REST API to programmatically access the tool.

ImgLab to annotate Labels

BilderSkript utilizes the ImgLab for label annotations. The web based tool exports label annotation in multiple formats

• dlib XML
• dlib pts
• Pascal VOC: standardized, however, the annotation data export only works for the current image. One needs to proceed to next image to export the next image’s annotation.
• COCO: standardized, however, the exportet annotation text seems to be incomplete

The dlib XML format originates from the dlib toolkit containing various machine learning algorithms. Here is an example using the python API for using dlib tools.

The XML format is reduced to the very basic tags required. An example is shown below.

<?xml version='1.0' encoding='ISO-8859-1'?>
<?xml-stylesheet type='text/xsl' href='image_metadata_stylesheet.xsl'?>
<dataset>
<name>dlib face detection dataset generated by ImgLab</name>
<comment>
    This dataset is manually crafted or adjusted using ImgLab web tool
    Check more detail on https://github.com/NaturalIntelligence/imglab
</comment>
<images>	<image file='IMG_20191219_094800_001_flat_pc_resize_mirror.png'>
		<box top='36' left='119' width='278' height='208'>
			<label>slide</label>
		</box>
		<box top='31' left='-2' width='206' height='319'>
			<label>person</label>
		</box>
	</image>
	<image file='IMG_20191219_094802_002_flat_pc_resize_mirror.png'>
		<box top='44' left='1' width='204' height='304'>
			<label>person</label>
		</box>
	</image>
</images></dataset>

BilderSkript Labels

We create rectangular shapes and define the following labels to annotate the shapes:

Versioning and Repository Management

Pipeline Code and Data Versioning

The builder image takes care of versioning using [git]() and [dvc]().

Pipelines are commonly shared in various volumes.

Repository Management

The project’s repository is mounted as separate volume in the builder container. As a consequence, only makefiles in the project’s root dir are able to commit changes to the repository. Scripts in subdirectories are not aware that the project’s root is under git version control. The volumes are organized under the following two mount points.

Mount point: /BilderSkriptRepo

Scripts which perform pipeline and data versioning start under this mount point. It contains all files and directories from the project repository:

• .git/
• Dockerfiles/
• …
• docker_compose.yml
• README.md

Mount point: /bilderskript

Scripts which actually execute pipelines start under this mount point. It contains the following directories:

• docs/
• docs_site/
• images/
• scripts/
• src/

APPENDIX: Pipeline Interprocess Communication (IPC)

The distribution of pipelines across various docker containers requires interprocess communication between the docker containers. Let’s discuss the design for the BilderSkript pipelines.

IPC Control Problem

We have the builder container which controls the execution of all pipelines. A pipeline implements a sequence of processs executions, where some of the processes run in their respective containers. As an example consider the data preparation pipeline. The pipeline in the builder container initiates the image data preparation in the hugin container. Once the process in the hugin container completes the pipeline can continues with the next step.

Basically, the sequence diagram above depicts a synchronous remote procedure call (RPC). This is a classical function in many programming languages, e.g. Java RMI.

IPC control problem formulation:

Implement a synchronous RPC to start processes across containers and that seamlessly integrates into pipelined workflows.

Contraints / requirements:

• low effort for additional network setup
• low effort for access and securing the access control
• script based, preferrably in bash, as it makes it easy to integrate the RPC as shell command.

In the following sections, we discuss some IPC approaches for their qualification as synchronous RPC.

File-based IPC

A very simple, but effective approach is the use of files for exchanging messages between container processes. The process in container A writes a message character string into a file on a shared volume from which the process in container B can read the file’s content. In this scenario, a single file serves as a buffer between those processes. However, the file access needs to be coordinated, otherwise the processes would overwrite each other’s messages. Additionally, to get informed about new messages, the processes need to poll the commonly shared file.

Client / Server IPC

Process within different containers may communicate as networked client and servers. Subsequently, some options:

• REST or similar RPC-alike calls across the network
• SSH remote script execution
• Shared UNIX sockets

REST requests are easy to implement, addtionally, a webserver such as nginx or apache is required to receive the requests. Alternatively, one could implement a simple webserver using python and flask. SSH remote script execution needs accounts to login to the remote containers. One needs to enter a password before execution, which limits automation capabilities. Public key access can resolve this problem, but needs caution in handling public and private keys. Both options expose containers to the network. As a consequence, it requires network definition between containers and a firewall to block unauthorized access.

Finally, UNIX sockets appear to be an ideal approach to interprocess communication. Sockets are stored on a shared volume, therewith enabling a local access control. However, this type of interprocess communication is limited between containers on the same host.

Message Queues and Brokers

A message broker maintains a message queue (MQ), where distirbuted processes can register themselve to exchange messages without directly knowing each other.

Message brokers are a prefered way for decoupled distributed processes, in particular for long running processes executed asynchronously. However, the setup and managment of a MQ takes effort. Processes need to agree on the message format and containers need to reach the broker’s queue via the network.

Databases for IPC

Databases store information, which are accessible by clients. As such, this is related to the file-based IPC. Process information is organized in tables and table attributes store process states, e.g. process execution start or whether a process successfully completed.

Similar to file-based IPC, processes are required to poll the database to receive updates on the process state. Addtionally, the setup and maintenance of even simple DBMS is significant. A simple file based database, e.g. sqlite, is an attractive solution for this last issue. However, write and read access must be coordinated to avoid (orphaned) file locks.

Final Design Decision

As a result of the discussion, BilderSkript implements an interprocess communication using shared UNIX sockets. It combines the charm of file based approaches with the advantage of bi-directional communication. Tool support for scripting is very mature. The next section provides more details how this approach implemented.

Limitation. The use of UNIX limits IPC between containers on the same host. To overcome one may use socat to forward the UNIX socket to a TCP one. See this stackoverflow post to get an idea how this can be implemented. However, this would re-open the discussion on network access issues, but enables a migration path to evolve the local host approach to distributed network approach.

Further Ressources:

There are a couple of ressources to follow up on these issues.

• https://www.linode.com/docs/security/authentication/use-public-key-authentication-with-ssh/
• Dockerize an SSH service and ubuntu sshd
• sshpass and Docker secret
• Docker ssh considered evil

An interesting op-ed from Bozho’s tech blog is You Probably Don’t Need a Message Queue. It inspired some of the disussion above.