PdParty User Guide

Version: 1.2.0
Date: 2018-10-15

PdParty is an iOS app that allows you to run Pure Data patches on Apple mobile devices using libpd. It is directly inspired by Chris McCormick's PdDroidParty and the original RjDj app by Reality Jockey. It takes a step further by supporting OSC and MIDI and by implementing the native Pd gui objects for a WYSIWYG patch -> mobile device experience:

This patch in Pure Data...

becomes this on iPhone...

and this on iPad.

Acknowledgments

This project has been supported by the CMU Frank-Ratchye STUDIO for Creative Inquiry, a visiting professorship at the DU Emergent Digital Practices Program, and my time at the ZKM | Hertz-Lab.

Table of Contents

Background

Dan Wilcox 2016

robotcowboy is the author's ongoing human-computer wearable performance project. Focusing on the embodiment of computational sound, robotcowboy was originally built in 2006-2007 as an MS thesis project using an industrial wearable computer running GNU/Linux & Pure Data, external stereo USB sound & MIDI interfaces, and various input devices including HID gamepads.

The original robotcowboy system hardware was gigged often, went on a 2 month tour of the United States in 2008, and lasted until the 2011 Pd Convention in Weimar. Around this time, Apple released the iPad 2 which featured a dual core processor and, most importantly, supported USB audio & MIDI interfaces. Seeking an option for new system hardware, the author began on and off development of an iOS application that could perform all of the tasks required for a live robotcowboy performance: run patches, full duplex stereo audio, MIDI, HID game controller support, & Open Sound Control communication.

With PdParty, the author now has a stable low latency mobile/wearable platform with a touchscreen, accelerometer, WiFi networking, and USB MIDI/audio. Here is a belt-based wearable setup using an iPhone, Camera Connection Kit, powered USB hub, Roland Edirol UA-25 USB audio interface, and a Behringer direct box (the latter two are built in the case on the left):

See the Pure Data 2016 conference paper for more details.

App Layout

TL;DR

There is a root settings screen and a patch/scene browser. Go to your patch and run it. Go back to update settings. Patches/scenes have on screen controls for the input volume, audio processing state (on/off), & recording.

Start Screen


Start screen on iPhone


Start screen on iPad, activate by pressing the "Patches" nav button.

This is the root of the app and is inspired by TouchOSC. Here you can launch the Patch Browser or change app settings.

This is also where you can enable the WebDAV server to access the app Documents folder for adding/updating patches.

Patch Browser

This is a simple "drill-down" file browser in the app Documents folder. Stuff in here is sandboxed between app versions. Simply navigate to patches or scene folders to run them. A "Now Playing" nav button will take you back to the Scene View for the currently playing patch/scene.

It only displays folders and supported file types. File type icons represent the supported files & scene folder types & certain scene types include a thumbnail image & author subtitle.

You can delete items by swiping right to show the delete button. Moving, copying, mass deletion, & renaming files/folders is available via the Edit button.

The default layout is:

Feel free to delete samples and tests. The libs folder contains abstractions needed by PdParty. This folder is exposed to allow you to update global abstractions as well as GPL-licensed abstractions which must be user upgradeable. If the libs folder is not found, PdParty falls back to its internal backup copy.

The all folders within the libs folder are automatically added to the PdParty search path so this can also be a location for centralized abstraction libraries.

Note: These default folders can be restored on the Settings screen. So if you accidentally remove everything, you're not out of luck!

Scene View

Running a patch/scene launches the Scene View:

Gui elements work similarly to those in the Pure Data gui, except now you have multitouch instead of the 10 foot pole called a mouse pointer. Also, Numberboxes can be incrementally scrolled using two fingers. Empty space is used for touch events if the patch/scene type supports them.

The desired aspect ratio is inferred from the patch canvas size and the Scene View is automatically rotated. Also, the device orientation is locked depending on if the Scene is in portrait or landscape. The exceptions to this are RjDj scenes which are portrait on iPhone & PdDroidParty scenes which are always landscape:

On iPad, however, RjDJ scenes can be both portrait or landscape:

On Screen Controls

Simple on screen controls inspired by the RjDj app are provided to change the audio input level, enable/disable audio dsp, start/stop recording, switch output between earpiece and speakers (iPhone only), restart the scene.

Patches, DroidParty, & PdParty scenes display these controls in a popover activated by the "Controls" nav bar button:

RjDj scenes always have a square background with the controls located below.

Optionally, a button to launch a Console view to display Pd prints for the current patch/scene can be added to the Controls popover. Enable to Console view in the PdParty Settings.

On iPhone, the speaker icon is added to allow for switching the audio output between the headset speaker (quiet) and the speaker-phone speakers (loud). This may useful to prevent feedback between input and output in certain scenes. For Rj scenes, this control is added as a switch on the scene's info view.

Recording Scene

There is also a special scene for playing back recordings (aka .wav files). There is a button for looping the playback and the slider controls the current playback position.

The cassette background is a placeholder for now. When metadata is added, it may be replaced by the scene icon, etc.

OSC Server Settings

Enable the OSC server and update its settings: ports, host (destination address), etc. The network IP address of the device itself is shown so you know where to send OSC messages to from another device. The server supports both IPv4 and IPv6.

Send via multicast by setting a multicast group in the Host field such as "224.0.0.0/24."

MIDI Settings

Enable CoreMIDI and optionally enable Virtual MIDI ports and/or Network MIDI with a macOS machine. There is also a convenience menu item to launch Bluetooth MIDI device discovery. Inputs & Outputs are refreshed when a MIDI device is connected or disconnected. PdParty supports a maximum of 4 Inputs and 4 Outputs.

As of PdParty 1.2.0, there are two MIDI port modes:

In Single (Simple) Device Mode, PdParty automatically connects to all detected MIDI devices. All devices share a single MIDI port aka channels 1-16.

In Multiple Device Mode, PdParty mimics the behavior of the Pure Data desktop application and assigns separate MIDI ports to each device:

Devices can be reordered to set their port number. When Multiple Device Mode is enabled, tap the Edit button and drag the device within the Input or Output table.

App Settings

App Behavior

These settings are mainly for live performance reasons where you don't want the device to go to sleep when you're trying to wow your audience. Obviously, keeping the screen on and audio running will drain your battery but in my tests so far it will still last for quite a while.

OSC Event Forwarding

These are useful options for patch creation & debugging. Basically, you can send accelerometer, touch, key*, and game controller events from the device to your computer while you work on your patch/scene in Pure Data. You can also receive live Pd prints from a running patch/scene in PdParty to make sure everything is working as expected.

The OSC server needs to be enabled and a patch/scene must be running in order for events to be streamed. The easiest option is to run an existing patch or upload an empty one you can run while sending events. Event sending will not work unless a patch or scene is being run.

Note: Most sensor events require the services to be enabled within the patch/scene before any events will be sent. Also, certain scene types do not support all events, please see "Scenes" in "Patching for PdParty" for more info.

* requires a usb/bluetooth keyboard

Audio Latency

This is pretty straight-forward and awesome thanks to CoreAudio. A lower audio latency will give you more responsive audio (nearer to "real time") at the cost of more CPU usage and lower battery life. Higher latencies are good for more simultaneous processing or where realtime response is not as critical (lots of delay, recording only, etc).

By default, the "Choose automatic latency" switch is enabled which tells the app to choose the best latency for your device. If this setting is disabled, you can set your desired buffer size using the radio buttons and you'll see the approximate latency.

Your mileage may vary depending on the device and the complexity of the patch or scene you're running. If you get audio clicks/dropouts, then you need to either increase the buffer size or decrease the processing complexity of your patch.

Copy Default Folders

These buttons allow you to copy the default libs, samples, and test folders into the app Documents folder, just as they were when you first ran PdParty. This is mainly there if you wish to re-copy or update any of the default folders.

Patching for PdParty

TL;DR

PdParty only shows gui objects (numbox, slider, etc) with send or receive names. It does not render your entire patch, so you need to create send and receives through the guis you want to appear on the device when you run your patch/scene.

Download the PdParty Composer Pack which contains the abstractions you need when composing on your computer as well as scene type templates.

Naturally, you can also download the PdParty source and open the test patches & examples to see how this is done: https://github.com/danomatika/PdParty/tree/master/res/patches

Detailed Instructions

Largely borrowed from PdDroidParty

  1. Create a new Pd patch that will contain your GUI objects like sliders, toggles, numberboxes etc. Place your main patch logic inside a subpatch and use the [soundinput] & [soundoutput] rjlib objects in place of [adc~] and [dac~] (soundoutput is required for the recording controls).

  2. PdParty will scale GUI objects to fit the screen of the device. Your patch should have the rough dimensions of a phone/tablet in portrait or landscape mode (e.g. 3:2 aspect ratio or e.g. 480x320 should usually work well). If it is not exact it, doesn't matter - the GUI elements will be scaled.

  3. Numberbox, Sliders, Radios, Toggle, Comment, Bang, Canvas, and VU are currently rendered by PdParty and are feature complete (yes, all the edit commands work!). Also, the PdDroidParty GUI abstractions are supported.

  4. All GUI elements should communicate with the main audio patches using send and receive only. You can usually set send and receive for each GUI by right clicking on the object and choosing 'properties' in Pd. Do not directly connect cables to the GUI elements as they won't work. It helps to keep the GUIs on their own in the main patch and have it include the logic of your patch as an abstraction or subpatch containing senders and receivers for interfacing with GUI elements. This is good patch design practice anyway as it is basically a model-view-controller methodology.

  5. Copy the patch and/or its containing directory and any needed abstractions to your iOS device using iTunes File Sharing, via WebDAV over your local network, or through "Open in..." from other apps:

    • iTunes File Sharing

      Plug in your iOS device and open iTunes. Select the device, choose the App tab, and choose PdParty in the File Sharing section. You should then see the contents of the PdParty Documents dir. You can drag and drop items onto this panel and/or use the "Add..." and "Save to..." buttons.

      Note: You can only see the top most level in the Documents folder and cannot enter subfolders. Sorry, that is simply how the iTunes file sharing system currently works.

    • WebDAV

      1. Enable the WebDAV server on the PdParty start screen on the device and connect to it using a file transfer program or the built in WebDAV support in some operating systems using the address below the WebDAV controls on the Start Screen. If you're using macOS or Linux on a local network, the *.local address should work, otherwise use the IP address (#.#.#.#). The server supports both IPv4 and IPv6.

        • macOS: Finder can mount WebDAV server folders: Go->Connect to Server... CMD+K. Login as "anonymous" (or anything, really) and leave the password blank:

        • Linux: both Nautilus (Gnome) & Konqueror (KDE) support mounting WebDAV folders, also FileZilla and other file transfer applications should work

        • Windows: newer versions of Windows Explorer support connecting to WebDAV folders, type the full address (including http://) into the address bar, also Cyberduck and FileZilla work nicely
      2. When the transfer is complete, navigate to the patch folder and run the patch. Don't forget to turn off the WebDAV server when you're done.

      3. Transfer performance may be low if you are transferring lots of files. If you experience this, try zipping the project folder and transferring the zip file. You can then unpack the zip in the PdParty browser by selecting it.
    • Open in...

      PdParty registers Pd patch *.pd files and Zip archives (*.zip, *.rjz, *.pdz) with iOS as supported types. This allows for each of these file types to be opened in PdParty from another application ie. Mail, DropBox, etc. When choosing "Open in..." via the Share button, the file(s) will be copied into the main PdParty Documents folder. Zip archives can then be unpacked by clicking on them in the Patch Browser.

Audio I/O

The audio device PdParty uses for input & output is the current device used by the system. If the device changes (something was plugged-in), PdParty will switch to this new device. There is currently no method for manually selecting audio routing.

Input & output are stereo (2 channel) by default. As of version 1.2.0, PdParty will attempt to use the maximum number of input or output channels supported by the current device, depending whichever is greater. As with desktop Pure Data, any extra input or output channels beyond those used by the current device are ignored. Note that the [soundinput] and [soundoutput] abstractions are stereo only, so you will need to roll your own i/o using [adc\~] & [dac\~] objects.

Scenes

PdParty also supports running "scenes" which are basically folders with a specific layout that are treated as a single entity for encapsulation and have certain event attributes:

Running a regular .pd patch (a Patch scene) is the same as running a PdParty scene.

Pure Data Compatibility

PdParty is built using libpd and can be compared to Pd-vanilla with the following externals:

It is highly recommended that you use a vanilla-based abstraction library like rjlib for expanded functionality.

When patching for PdParty (as with RjDj & PdDroidParty), it is recommended that you work with Pure Data vanilla versions 0.46+. If you are working with Pd-extended, disable all externals in order to help lessen the chance you inadvertently use an object that will not create in PdParty. I actually have separate copies of my Pd settings file, one for desktop development and another for pd-vanilla/libpd work.

expr

[expr], [expr~], & [fexpr~] are included with PdParty. As of Pd versions 0.47+, they are under the BSD license which is compatible with the Apple App Store licensing requirements. This was not the case with earlier versions of Pd/libpd.

Key events

[key] works with an external bluetooth or usb keyboard. [keyup] & [keyname] are not supported as there is currently no official way to intercept raw key events on iOS.

VU Meter

[vu] gui objects do not have a sending name in PdParty, so make sure you're patch doesn't rely on passing values through any meters.

MIDI

All of the midi objects ([notein], [ctlout], etc) work. Obviously you'll need to have a usb MIDI interface (through a USB hub connected to the Apple Camera Connection Kit) or using Network MIDI and macOS.


Midi test Pdparty scene

PdDroidParty Compatibility

PdParty currently supports:

* doesn't do much as audio has to be off for vibrate to happen on iOS

SVG widget styling support is planned, but not an immediate priority as there is no built-in svg handling on iOS and there doesn't seem to be a standout 3rd party SVG library.

[droidnetclient] & [droidnetreceive] are deprecated in PdDroidParty and therefore not supported. Use [netsend] & [netreceive] instead.

RjDj Compatibility

PdParty currently supports:

Currently, scene paging and metadata are not supported.

Testing has been done using the original RjDj composer pack as well as various RjDj scenes including:

Also, thanks to Joe White for providing a copy of the RjDj get_sensors.pd patch by Roman Haefeli, et al. which provided an overview of the extended rj sensor objects.

Events

PdParty returns the following events:

Note: RjDj scenes receive #touch, #accelerate, & #gyro events by default, DroidParty scenes do not receive any events, PdParty & Patch scenes receive all events. This is mainly for explicit compatibility. Extended RjDj sensor access is made via the [rj_loc] & [rj_compass] abstractions.

Accelerate, Gyro, & Magnet Control


PdParty accelerometer events

Reading accelerometer, gyroscope, and/or magnetometer events will affect battery life, so these must be manually started after the scene is loaded by sending messages to the internal #pdparty receiver:

Note: #touch & #accelerate events are automatically started for RjDj scenes for backward compatibility.

Loc (GPS) Control


Loc test PdParty scene

Loc events are essentially GPS location events, dependent on your device's sensors for accuracy (WiFi only, cell tower + GPS chip, etc). Additionally, speed & altitude events are generated when the location events are enabled.

Since running the GPS location service will affect battery life in most cases, it must be manually started and configured after the scene is loaded by sending messages to the internal #pdparty receiver:

It usually takes a few seconds to fix your position after enabling the location services.

Note: Loc events are available in PdParty & Patch scene types by default, while the presence of an [rj_loc] object enables them in RjDj scenes. These events work best on devices with multiple location sensors (iPhone) and may not work on some devices at all (iPad).

Compass Control


Compass test PdParty scene

A compass event is simply the orientation toward magnetic north with the top of the current UI orientation being at 0 degrees.

Like location events, the tracking the compass requires extra resources so it must be manually started by the scene after it is loaded by sending messages to the internal #pdparty receiver:

Note: Compass events are only available in PdParty & Patch scene types by default, while the presence of an [rj_compass] object enables them in RjDj scenes. Events work best on devices with a digital compass (phones) and may not work on some devices at all.

Timestamps

Timestamps are sent to the [r #time] receiver with the following argument format:

This is the same format that the RjDj [rj_time] object returns.

Timestamp events must be triggered manually by sending a message to the internal #pdparty receiver in your patches:

Recording

You can manually trigger recording via sending messages to the internal #pdparty receiver in your patches:

Note: Recording will only work if you are using the rjlib [soundoutput] patch instead of [dac~].

Opening a URL

You can launch a web view with a given url via sending a message to #pdparty:

url can be:

title is an open ended list of arguments that will be appended together and used as the navigation bar title, "URL" is used by default when there are no title arguments.

Local files are opened in a slide up web view within PdParty while all others are opened by the system in their respective apps: "http://" & "https://" in Safari, "maps://" in Maps, etc.

On iOS 10+, non-local URLs are opened asynchronously.

Game Controllers


Controller PdParty scene

Compatible iOS MiFi game controllers can be read in PdParty if your device supports them. If the controller uses Bluetooth, enable Bluetooth in your iOS settings and make sure the controller is paired to your device. Currently, iOS limits the number of simultaneous controllers to 4.

Controller events can be read via the [r #controller] receiver with the following format:

There is no direct control over enabling/disabling game controller support. This is handled by scene type detection as RjDj & DroidParty scenes do not use controller events.

Game controller button & axis names are based on the SDL 2.0 GameController enumerations where "dpup" refers to digital pad up, "dpleft" refers to digital pad left, etc. This format is compatible with the OSC messages sent by the joyosc desktop HID device to OSC event daemon.

Buttons: "a", "b", "x", "y", "leftshoulder", "lefttrigger", "rightshoulder", "righttrigger", "dpup", "dpdown", "dpleft", "dpright"

Axes: "leftx", "lefty", "rightx", "righty"

OSC

PdParty sends and receives OSC (Open Sound Control) messages internally between the PureData instance and the OSC server:

All of the PdParty events can be streamed over OSC, included Pd prints. The receive addresses are as follows:

Note: The argument number and types are equivalent with their receive counterparts, i.e. /pdparty/touch receives the same data as [r #touch].

See docs/composerpack/osc/osc-event-receiver.pd in the PdParty source repository for an event receiver you can use while patching & debugging on your computer:


osc-event-receiver.pd test patch

Also, try the tests/osc-test.pd test patch on your computer with the tests/pdparty/Osc scene on the device for a simple example on two-way communication:


OSC test PdParty scene

Note: PdParty utilizes the OSC library provided by Pure Data vanilla versions 0.46+, mainly the [oscparse] & [oscformat] objects.

Libs Folder

PdParty automatically adds any subfolders inside the libs folder in the PdParty Documents root folder to the current search path. Put any abstractions here if you want them to be globally accessible.

Also, this is where the default PdParty libraries are copied when the app is first run. This allows you to update the newer versions of the core lib patches. Be careful, as the rj patches, for instance, are required for running RjDj scenes etc. If you screw up / delete something, you can copy back the original files in the Settings screen.

PdParty URL Scheme

As of version 1.1.0, PdParty has a custom URL scheme "pdparty://" for opening PdParty from other apps on iOS. The domain portion of the URL is used as a message type and the following messages are supported:

To test, enter the following into the address bar in Safari and hit enter:

pdparty://

This should open PdParty. To open the all_pd_guis.pd test patch in the tests folder:

pdparty://open/tests/all_pd_guis.pd

Further message types may be added in the future.

TODOs