1. Introduction
The development of electronic and digital technologies in the twentieth century saw the birth of many new musical instruments. Topics of exploration have been wide-ranging, from new types of synthesis to the topic at hand: spatialisation. Often moving in tandem – but not always unison – with the development of immersive fixed-media compositions, sound spatialisation instruments have charted their own path forward. For example, while Edgard Varèse’s Poème Electronique (1958) used a sophisticated system which routed pre-recorded audio to 425 individual speakers (Zvonar Reference Zvonar1999), Pierre Schaeffer and Pierre Henry’s relief desk (also known as the pupitre d’espace or potentiomètre d’espace) employed perpendicular circular electro-magnets to control real-time spatial diffusion for Symphonie pour un homme seul (1951) (Teruggi Reference Teruggi2007: 218).
Since then, the evolution of spatial technology has been articulated through numerous new instruments. The realm of analogue synthesisers saw the Buchla joystick-driven Model 204 Quad Spatial Director module, as heard in Morton Subotnick’s 1971 album Sidewinder (Bengler Reference Bengler2011: 15). Straddling the transition to the digital age, later iterations of Max Mathews’s Electronic Violin added spatial capabilities. The EV, a digital/acoustic augmented string instrument and the focus of this article, inherits spatial manipulation techniques pioneered by its predecessors, including Max Mathews’s Electronic Violin (Lindgren Reference Lindgren2022).
2. Related Work
2.1. Augmented stringed instruments
Of the many instruments developed since the advent of spatial audio technology, the Mathews Electronic Violin is pertinent to the EV as a forerunner in augmented stringed instruments. Spatial functionality was a later development in the evolution of his instrument, as his earliest publication on the topic describes ‘electrical signals from the four strings’ as being ‘summed and applied to a set of electrical resonances’ (Mathews and Kohut Reference Mathews and Kohut1973: 1621). While the rationale behind the discrete pickups was to mitigate intermodulation (UC San Diego Reference San Diego1982), it is unclear whether the design was envisioned with the eventual creation of immersive music in mind. Richard Boulanger describes the Electronic Violin’s sound localisation capabilities when writing of his 1985 composition Three Chapters from The Book of Dreams: ‘the discrete four-channel output of the instrument significantly directed the composition in a number of ways. Clearly it pointed the work toward a composed spatial component’ (Boulanger Reference Boulanger1986: 142).
Although not a common practice in augmented string development, three notable immersive instruments emerged in the years following. A decade after Mathews’s work, Dan Trueman and Curtis Bahn, of the duo Interface, experimented with the violin and bass. Trueman’s instrument, the BoSSA (Bowed-Sensor-Speaker-Array), was a ‘deconstructed violin’ split ‘into its primary constituent physical interfaces’ (Trueman Reference Trueman1999: 15); mainly, a fingerboard separated from the bridge. The SBass, by Bahn, was built upon a ‘5 string electric upright bass’ (Bahn and Trueman Reference Bahn and Trueman2001: 20) extended with many sensors, pickups and even a mouse trackpad. Synthesis is performed in Max/MSP before being sent to SenSAs (Sensor-Speaker-Arrays), custom-built 12-channel speaker arrays (Trueman et al. Reference Trueman, Bahn and Cook2000).
In contrast with the less-conventional diffusion methods of Mathews, Trueman and Bahn, the Augmented Cello (Freed et al. Reference Freed, Wessel, Zbyszyński and Uitti2006), developed at UC Berkley’s CNMAT (Center for New Music and Associated Technologies), uses Ville Pullki’s vector-based amplitude panning (VBAP) Max/MSP objects, which can accommodate a variety of speaker arrays (ibid.: 412–13). Two munger∼ granulation objects generate four audio channels (two stereo pairs), where each of the channels is panned 90 degrees apart. Using a rotary encoder at the base of the instrument, the player can rotate the granular sound field.
2.2. Development of spatial synthesis paradigms
As these technologies developed, spatialisation grew not only to address the placement of sound source, but also to explore the relationship between sound synthesis and space. Referred to as spatial sound synthesis, these methods ‘perform an individual spatialisation of single sound components at an early stage of the synthesis process’ (Coler et al. Reference Coler, Lepa and Weinzierl2020: 464). Some of these methods have been implemented to varying degrees in the EV.
2.2.1. Spatio-operational spectral synthesis
One of the earlier spatial sound synthesis inquiries experimented with ‘psychoperceptual interpretation of audio objecthood as a result of streaming theory’ (Topper et al. Reference Topper, Burtner and Serafin2002: 133). This technique breaks a signal into its constituent sine waves, placing each at a separate position in the sound field. Using a square wave, the study found that because of the ‘inherent limitations of the auditory system, the listener cannot readily decode the location of specific spectra, and at the same time can perceive the assembled signal’ (ibid.). In contrast to localisation afforded by spatial systems, this experiment reveals the compelling prospect of de-localisation.
2.2.2. Granular sound spatialisation
In 2008, a study explored dictionary-based methods applied to audio signals whereby ‘spatialisation can now be explored down to the microsound level of sonic structure, where individual spatial positions are assigned to every sonic grain’ (McLeran et al. Reference McLeran, Roads, Sturm and Shynk2008). Various methods of spatialisation described as scattering, convergence, divergence and panning, can be applied to each grain, creating the effect of motion in the sonic image.
2.2.3. Spatial modulation synthesis
Published in 2015, the concept of spatial modulation synthesis draws upon Stockhausen’s notion of the ‘unity between timbre, pitch, intensity, and duration’ (McGee Reference McGee2015: 247). In contrast to John Chowning’s investigation of this phenomena in frequency modulation, McGee asserts that ‘spatial parameters [can be] modulated to result in frequency and amplitude modulation of an input signal rather than controlling FM and AM directly’ (ibid.: 246). As with granular sound spatialisation, the incorporation of space within traditional synthesis paradigms has yielded notable results.
3. The EV
3.1. Overview
The initial impetus for the EV was to explore the convolution of a synthesised and acoustic string signal. Aside from the unusual body (Figure 1), the dimensions of the instrument are that of a 16-inch viola, rendering the instrument readily playable by any trained violist or violinist. A variety of string tunings have been tested: viola, cello and viola with octave-down A and D strings. Unlike other augmented strings which may feature a variety of buttons or knobs for extended software control, the EV eschews this, instead embracing Agostino Di Scipio’s concept of sound is the interface (Di Scipio Reference Di Scipio2003). The expressive nuance of a trained player is captured by the spectral analysis; each frequency bin acts as a control source, providing great detail in shaping the convolution processes. This approach prompts the player to focus entirely on performing the EV as they would a traditional instrument, without the need to divert attention to manipulate the synthesis process. A spectral centroid tracker can also modulate a variety of synthesis parameters. To facilitate parametric control, the instrument includes a preset system by which parameters can be changed using a Bluetooth foot pedal. So far, performances have used standard notation, with numbered boxes (similar to rehearsal numbers) indicating preset changes.
Figure 1. The EV with its external processing box.
Although the instrument has a solid body, the vibrating strings can be felt through the chinrest, providing the player with a tactile sensation of amplitude and frequency like that of an acoustic instrument. Headphones can be connected to the computer’s audio jack for monitoring, while the main output is routed through an external ADC. The instrument includes an external processing box for pre-amplification, A/D conversion, and computer interface, where the synthesis engine runs in Pure Data (Pd) (Figure 2).
Figure 2. The EV’s GUI in Pure Data: 1) current string displayed; 2) the three synthesisers; 3) four FFT convolution engines with settings for sample rate, overlap, window size, input swap and source select; 4) Karplus-Strong effect; 5) buffer freeze with settings for overlap, window size, playback rate and refresh; 6) granular delay; 7) CV matrix; 8) LFOs; 9) envelope generator; 10) mixer; 11) reverb; 12) ambisonics options.
3.2. Development
3.2.1. Beginnings
Over the past four years, the EV has undergone significant transformation. Three versions have been built, each used for the composition of new material. Rather than a fixed instrument, the EV embodies an evolving process, each version a manifestation of its ongoing development. Initially, three custom-built synthesisers (two frequency modulation and one phase distortion) and four distinct FFT convolution engines provided the user with convolution tools. In subsequent iterations, the introduction of modulation sources and a flexible routing scheme for both audio (mixer) and control voltage (matrix) allowed for more nuanced and complex sound synthesis. A preset system was introduced that facilitated the instantaneous change of parameters. This version of the instrument was used by the author to compose Etudes & Vignettes (Lindgren Reference Lindgren2023). At this point, the instrument’s output was mono, with stereo reverb and delay provided by third-party software.
3.2.2. Time-based effects and independent string parameters
Further development focused on incorporating time-based effects, enabling the sound to sustain without continuous input (playing); these included a buffer freeze effect, granular delay and reverb. In addition, each string was given the ability to use independent parameter settings (it was already polyphonic). Various synthesis processes and routings could be employed for different strings without the need for preset changes or fiddling with the GUI.
3.2.3. Spatial capabilities
A spatialisation engine was conceived to free the EV’s growing sonic palette from its limited mono sound field. The first implementation was 2D and had similarities to vector-based amplitude panning. Later, this was replaced with a 3D first-order ambisonics engine, followed by the inclusion of second and third orders. Ambisonics is a widely used spatialisation technology that supports any speaker configuration, from a standard stereo setup to a complex 3D multichannel array. Often, a greater number of speakers yields a more consistent representation of sound source, as phantom images may fail to form when relying on fewer speakers. Ambisonics can perform well on a variety of speaker arrangements, provided that the source representation satisfies compositional aims. The EV system includes a Python script which receives speaker coordinates as input, and outputs coefficients used by the spatialisation engine. The spatialisation engine – later released as ambiNilla – uses Pd’s vanilla signal rate objects to support smooth panning; this proved to be essential in experiments with spatial motion synthesis. Spatialisation can be controlled via a GUI joystick interface or a control voltage matrix; however (as described in section 3.1), performance parameters are primarily intended to be adjusted using the preset functionality, allowing the player greater freedom to focus on their performance. Panning automation can be controlled with matrix sources: an amplitude tracker, spectral centroid tracker, two LFOs and a custom-drawn envelope generator (Figure 2). Each string can be configured independently.
The granular delay utilises up to eight individual read heads to generate audio grains. These read heads can be positioned in space using a joystick and a spread parameter. The instrument does not yet support the independent spatialisation of each grain, as detailed by McLeran et al. This iteration of the instrument was used to compose Daughter of the Stars (Lindgren Reference Lindgren2024a, Reference Lindgren2024b).
3.2.4. Latest developments
Recent developments have further advanced the exploration of timbre within the spatial dimension through multivoice processing. Leveraging Pd’s updated multichannel capabilities, the EV’s software effects can synthesise up to eight voices simultaneously (resulting in a potential 324 simultaneous voices). For instance, a synthesiser can generate eight voices, each detuned a specified amount with parameters offset randomly. This, in turn, enables the FFT convolution engine to produce eight slightly varied mono signals, which then can be routed into the multichannel time-based effects (as previously mentioned). Each sound module has a spread parameter that adds up to 45 degrees of offset between these eight voices, allowing the module to render its output as wide or narrow within the sound field.
In addition, a modified Karplus-Strong synthesis engine was developed to explore basic physical modelling capabilities. Instead of a click as the exciter, the buffer receives a continuous audio stream routed from various sound modules via the mixer. The buffer length is controlled by the pitch detection algorithm. As with the other modules, the Karplus-Strong engine supports up to eight voices, allowing for dense timbral exploration and wide panning options.
3.2.5. Use as in external signal processor
At one point it was decided to experiment using the EV’s engine as a processor for other instruments, following in the footsteps of the many analogue synthesisers which also feature external signal processing capabilities (e.g., the Korg MS-20). The synthesis engine already featured four independent units (one for each string) and had access to a range of sound-processing algorithms. All that was required was to route external signals into the Pd patch and modify the pitch-detection algorithm. Additionally, an option to send a dry signal from one instrument to the other was established to enable the cross-convolution of two separate sources. This experiment provided an excellent opportunity to explore the software’s capabilities, including further spatial experimentation. The result was a new composition for HYPERCUBE (saxophone, guitar, piano and percussion): experiments from the n-space (Lindgren Reference Lindgren2024c).
3.3. Summary
All in all, each string of the EV can simultaneously spatialise: a dry signal, three synthesisers, four FFT convolution engines, reverb, granular delay and a Karplus-Strong algorithm. Any one of these can be subject to a buffer freeze effect and can support up to eight voices. Each module’s 3D spatial position can be automated via voltage control matrix sources: an amplitude tracker, spectral centroid tracker, two LFOs and a custom-drawn envelope generator. Much experimentation has been dedicated to exploring the sonic potential of this instrument, yet there is more to discover. A signal flow diagram is shown in Figure 3.
Figure 3. The EV’s signal flow.
4. Results of spatial experimentation
4.1. Spatial width and timbral detail
One of the motivations for incorporating spatialisation in the EV was to effectively display the varied layers of sound it generates. A perceptually narrow summation of many sources has its own creative uses; however, spatial width can allow various timbres to be separately panned so they may be appreciated individually. In Daughter of the Stars, the opening F♯ sonority is created by the phase distortion synthesiser running wet-only through the reverb, which then is sent to the granular delay operating with eight voices spread apart. The reverb is panned to the rear right and the delay to the rear left, resulting in nine voices spread over approximately 135 degrees. Compositionally, this results in a perceptual breadth of sound, which, at the 2024 premiere at the Experimental Media and Performing Arts Center (EMPAC) in Troy, New York (Sound Example 1), was markedly different from the two-channel version used in preparation for the performance.
Similarly, in ‘Aporia’, the last movement of experiments from the n-space, the piano is processed through three spatially wide modules to achieve a 360 degree spread of 24 separate voices: 1) an eight-voice synthesiser pitch-tracking the piano is convolved with its dry signal, resulting in eight voices spread across the front of the sound field; 2) the convolved piano is passed through an eight-voice granular delay, which is then buffer-frozen and spread over the left rear area of the sound field; and 3) the same convolved piano is passed through an eight-voice reverb, which is also buffer-frozen and spread over the right rear area of the sound field. Because each of these 24 audio streams is unique, the feeling of immersion is paired with a rich sonic density, conveying a sense of grandeur reflective of the compositional intent. When the guitar and saxophone enter later in the movement, an additional 29 unique voices are introduced across the sound field (Sound Example 2).
4.2. Width and narrowness as contrasting tools
The preceding movement in experiments from the n-space, ‘homilies of the nuclear priesthood’, opens with a spacious guitar solo, using much of the sound field. The piano continues with a melancholy motif after the guitar’s conclusion. In stark contrast to the sonic breadth of the guitar, the piano is voiced at only two narrow spatial locations: the dry signal panned to the front and a reverberated signal with a frozen buffer to the left. A wider reverb with eight voices was auditioned in the composition process, but it was decided that the narrow version both reflected a feeling of desolation intended by the composer and added contrast to the spatial width of the guitar (Sound Example 3).
A similar effect is used in Daughter of the Stars. The first half of the work features two wide and timbrally rich sounds: the F♯-based sound mentioned earlier and a sonorous melodic line played on the A string. This melodic line is representative of the main characters of the Native American folk tale on which the piece is based, and versions of it are repeated throughout the work. Its sound consists of ten voices spread over the front, front right and back left areas of the sound field. The middle of the work abruptly changes character and features a new roster of sounds intended to contrast with the rich sonic palette of the opening. In particular, one of these sounds is a thin frequency-modulated square wave with a duty cycle and spatial location that are intermittently shifted by a square wave LFO. As with the previous piano example, a singular narrow voice is used for the dry FM sound and another single voice for the reverb. Although a simple approach, the use of sonic breadth contrasted with narrowness has shown to be an effective tool in composing spatial music with the EV (Sound Example 1).
4.3. Spatial modulation synthesis
The central premise of ‘Les Espaces d’Abraxas’, the fourth movement of experiments from the n-space, was inspired by McGee’s notion of spatial modulation synthesis. It opens with a mysterious, dark, yet nasal, drone played by the saxophone and guitar (with an EBow) processed using three convolution algorithms. After a bowed vibraphone melody, the drone begins to slowly spin about the sound field, its texture producing a cracking sound, like a spitting fire. Over four minutes, the envelope generator causes an increase in both LFO’s speed and amplitude, which, in turn, control the X and Y spatial coordinates, thereby panning the drone in a circular motion. The envelope also increases the amplitude of the convolution, creating an intensity that parallels the drone’s rotational velocity. As the periodic rotation passes 20Hz, the listener can observe a psychoacoustic shift as the oscillation enters the audio realm and new sonic properties emerge; a fascinating new overtone spectrum envelopes the original tonal properties of the drone, an effect akin to an increase in modulator rate in FM synthesis (Sound Example 4). The encouraging results of this basic implementation of spatial modulation synthesis warrant further exploration.
4.4. Staging
In contrast to the complex trajectories and psychoacoustic effects that are possible with spatialisation, the staging, or dramatic positioning, of sound source can be compositionally impactful. For example, in Daughter of the Stars the sonorous melody represents the main characters of the folk tale, and the roaring F♯-based sound depicts a silver lake and blue mountains. The placement of the latter behind the audience suggests an expansive backdrop, while the positioning of the melody in the front establishes a sense of priority or focus within the narrative (Sound Example 1). Although invisible, spatial music can conjure soundscapes akin to the set design of a theatre production. Sharma and Schultz propose that the ‘alteration of spatial constellations and staging of entities create spaces (and sometimes more than one) in relation to a given place. We could say that loudspeaker arrays actually produce their own spaces within acoustic environments’ (Sharma and Schultz Reference Sharma and Schultz2017). While the author’s compositions tend to consider the sonic projection of the speaker similarly to video projection in the cinema (an ephemeral sonic mirage, intentionally decoupled from the real space), the differential between speaker output and room acoustic can potentially give rise to a multiplicity of spaces as described by Sharma and Schultz. The EV provides the performer/composer with the flexibility to navigate and balance this dynamic in accord with their creative intent.
5. Future work
Many explorations lie ahead in the pursuit of new music with EV. Acknowledging that constraints are often an important part of the creative process, progress thus far has sought to strike a balance between expansion of the instrument’s capabilities and the composition of new music. Although certain elements of the previously mentioned studies (section 2) have been implemented in the instrument, future work involves a more detailed exploration of that research.
5.1. Spatio-operational spectral synthesis
A long-desired feature of the EV is the inclusion of discrete summation oscillators because of their utility in avoiding (where undesirable) aliasing. However, their implementation would also allow for the addition of spatio-operational spectral synthesis (section 2.2.1). As spatial synthesis incorporates spatialisation into the early stages of the synthesis process, discrete summation oscillators could enable the spatialisation of individual frequency components, allowing for the creative exploration of non-localised sound within immersive composition, as described by Topper et al. (Reference Topper, Burtner and Serafin2002).
5.2. Granular sound spatialisation
Currently, each of the EV’s granular read heads (or voices) is panned at a distance set by a spread parameter. The option to individually spatialise each grain, as described by McLeran et al., will be explored further to achieve more precise and dynamic spatial effects in future implementations. Although the use of dictionary-based methods is outside of the EV’s current scope, the ability to apply the described spatialisation techniques (e.g., scattering, convergence, divergence, blur) to collections of grains is a promising avenue for future development, potentially enhancing the depth and versatility of the instrument’s spatial sound capabilities. This approach would require reworking the EV’s audio routing system; spatial location is currently applied post-synthesis, which prevents the use of discrete panning angles for each grain. To address this, a new audio routing scheme that integrates spatial location at the point of synthesis would be needed.
5.3. Physical modelling
The implementation of a modified Karplus-Strong algorithm (section 3.2.2) was the first step towards the exploration of physical models. The use of virtual acoustics has an interesting parallel with physical models as they both can simulate the physics (with a licence for artistic flexibility) of resonant bodies. This has an interesting correlation to the relationship between the performer of an acoustic instrument and the hall. Often, especially in unaccompanied string music, the performer considers the reverberation of the space to be one with the decay of their instrument’s ringing strings; the hall serves as an extension of the instrument’s body. Spatialising a physical model offers an opportunity to explore this relationship further, allowing the virtual space to become an integral part of the instrument itself.
5.4. Wave field synthesis
Although ambisonics has opened exciting new possibilities for the EV, the generation of artificial wavefronts using wave field synthesis offers a fundamentally different perceptual experience, bypassing reliance on interaural differences to create phantom image representations, as seen in ambisonics, VBAP and similar systems. Future work will involve the research and building of a 2D wave field synthesis rendering engine for the EV.
5.5. Considerations
Currently the EV’s engine employs a conventional signal chain paradigm: sound is first synthesised and then routed to a mixer for amplitude adjustment, bussing and panning. Many of the aforementioned spatial synthesis techniques would necessitate a different signal chain to be successfully implemented in the EV: one that incorporates panning as an integral component of the synthesis process. A potential path forward could involve supporting both signal chains, rather than privileging one over the other.
6. Conclusion
This article presents a new hybrid digital/acoustic stringed instrument, the EV, with a focus on both its ability to spatialise a variety of synthesis sources and its use as a versatile compositional tool. The EV has its precedent in a long history of spatialisation within instrument design, and specifically, in augmented string design. Building upon the experiments of its forerunners, the future developments of the EV include exploration of previously published spatialisation techniques in greater depth, aiming to utilise them for the composition of new music within the context of the EV’s unique design and capabilities.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/S1355771825100575
Open access
