Typing heterogeneous dataflow graphs for static buffering and scheduling

Abstract

Interactive multimedia system usually represents computations as a dataflow graph – the patching model. Nowadays , dataflow graphs integrate multiple medias from various dedicated languages and systems, with multiple rates, for instance audio rate, video rate, control rate, but also the rate of FFT frames, as well as requirements on buffer sizes, or on what kind of data they consume or produce. We design a type system for the DSP extension of Antescofo that makes it possible to safely combine effects in such a heterogeneous graph. We show how to infer the types given known types in the graph, and how we use this type system to compute buffer sizes and to schedule the graph. Our approach is exemplified on an Antescofo program that does speed tracking of a foreground object in a video stream to drive an audio effect.

Publication
ICMC 2017 - 43rd International Computer Music Conference