Rivendell in the cloud

I joined up with a Facebook group called Rivendell Open Source Radio Automation Users as a place to trade tips on using Rivendell. A question that comes up frequently is how Rivendell can be run in the cloud. Since I’ve been doing this for eight years or so I have a pretty good understanding of the challenges. I’ve mentioned some of it before but thought I’d go into more detail of my current setup.

I’m running Rivendell 2.19.2, the current version, and presently I’m not actually running it in the cloud though I could easily change this in a few moments. The magic that makes this happen is containerization. I have created my own Docker instance which installs everything I need. This container can be fired up virtually anywhere and it will just work.

Here’s a summery of my setup. In my container, I install CentOS 7. Then I pull in Rivendell from Paravel’s repos with a “yum install rivendell” command. Rivendell needs the JACK audio subsystem to run so I install Jack2 from the CentOS repos, too. To this I add darkice as an encoder, JackEQ for some graphical faders/mixers, a LADSPA-based amplifier module to boost gain, and of course Icecast2 to send the stream to the world.

Now, one of the problems with a CentOS-based setup is that CentOS tends to have fewer of the cool audio tools than distributions like Debian and Ubuntu have. These Debian-based distros are not officially supported with Paravel packages so you either have to hunt for your own Rivendell dpkgs or you build your own. I’ve found a few of these dpkgs mentioned on the Rivendell Developer’s mailing list but I’ve not had the time to make sure they’re up to date and meet my personal needs. Thus, for my personal setup you’ll find a few parts which I have compiled myself, rather than install from a package. A project for me to take on in my Copious Free Time is to create an entirely repo-based Docker container but I’m not there yet.

Rivendell needs a MySQL/MariaDB database to store its data. I rely on a non-containerized instance of MariaDB in my setup because I already use the database for other projects and didn’t want to create an instance solely for Rivendell.

So here’s how it all works.

Once Rivendell and JACK packages are installed, you’ll need to get Jack running first. Here’s the command line I use in my container for running jack:
/usr/bin/jackd -r -t2000 -ddummy -r48000 -p1024

For ease of connecting, you should run all audio parts under the same Linux user. I usually create an “rduser” user under Linux to take this role. Do not use the root user for this as anyone exploiting any flaws in Rivendell would have total control of your system! Good UNIX philosophy calls for only giving processes the minimum permissions needed to carry out their tasks.

IF you’ve started JACK as the rduser user, log into a terminal as that user and perform the “jack_lsp” command. This will show you the JACK channels that are available. Starting Rivendell using the “service rivendell start” or “systemctl start rivendell” should populate the JACK channel listing with the Rivendell channels. Make sure you edit the Rivendell config file at /etc/rd.conf to get Rivendell to start under the rduser user so that Rivendell and JACK can talk to each other.

Once you see Rivendell’s talking to JACK, you need to connect these virtual JACK channels so that your audio gets somewhere useful. In my case, I want to take the following path: Rivendell -> JACK -> JackEQ (with amp plugin) -> JACK -> darkice -> Icecast2 -> cloud-hosted Icecast2 on my VPS.

I connect Rivendell output channels to JackEQ’s inputs:
jackEQ:c.1-in-L
rivendell_0:playout_0L
jackEQ:c.1-in-R
rivendell_0:playout_0Rrivendell_0:playout_0R

You should see the VU meter in JackEQ’s channel 1 light up with audio from Rivendell. JackEQ’s master channel should also light up with audio. You can raise the levels here in JackEQ and your audio gain should be boosted. While you could theoretically pipe the Rivendell output straight to darkice through JACK, I’ve found in a virtual or dummy sound card Rivendell setup that the levels are too low and need a boost. If you try this and find that you need to boost it even further, check out the “Jack Rack” set of audio plugins for Linux. Several plugins are available to create a very professional sound. I highly recommend it.

So now that we’ve got the audio in JackEQ and nicely boosted, we need to connect it using JACK to our encoder, Darkice. I looked at many encoders when I was first getting started but few seemed to be as flexible and efficient as Darkice. As far as I know, no ready-built packages exist for Darkice. Fortunately it’s not difficult to compile once you have rounded up all the audio libraries needed for your particular setup. I use Ogg Vorbis and AAC+ streams for my stations and Darkice supports them well.

Here’s what my JackEQ – Darkice links look like in JACK (via the “jack_lsp -c” command):

darkice-29630:left
jackEQ:a.master-L
darkice-29630:right
jackEQ:a.master-R


Here’s an edited version of my darkice.cfg file:

# sample DarkIce configuration file, edit for your needs before using
# see the darkice.cfg man page for details

# this section describes general aspects of the live streaming session
[general]
duration = 0 # duration of encoding, in seconds. 0 means forever
bufferSecs = 5 # size of internal slip buffer, in seconds
reconnect = yes # reconnect to the server(s) if disconnected

# this section describes the audio input that will be streamed
[input]
device = jack # OSS DSP soundcard device for the audio input
sampleRate = 48000 # sample rate in Hz. try 11025, 22050 or 44100
bitsPerSample = 16 # bits per sample. try 16
channel = 2 # channels. 1 = mono, 2 = stereo

# this section describes a streaming connection to an IceCast2 server
# there may be up to 8 of these sections, named [icecast2-0] … [icecast2-7]
# these can be mixed with [icecast-x] and [shoutcast-x] sections
[icecast2-0]
bitrateMode = abr # average bit rate
format = vorbis # format of the stream: ogg vorbis
bitrate = 96 # bitrate of the stream sent to the server
server = <my_icecast_server>
# host name of the server
port = 8000 # port of the IceCast2 server, usually 8000
password=<my_password>
mountPoint=<my_mountpoint>
name = Neuse Radio OGG 96Kbps
# name of the stream
description = Let the music flow!
# description of the stream
url = http://<my_webserver>
# URL related to the stream
genre = Alternative # genre of the stream
public = yes # advertise this stream?
#localDumpFile = /tmp/dump.ogg # local dump file

# aacp low stream
## aac high stream
#[icecast2-2]
#bitrateMode = abr # average bit rate
#format = aac # format of the stream: ogg vorbis
#bitrate = 64 # bitrate of the stream sent to the server
#server = <my_icecast_server>
# # host name of the server
#port = 8000 # port of the IceCast2 server, usually 8000
#password=<my_password>
#mountPoint=<my_mountpoint>
#name = Neuse Radio – AAC
# # name of the stream
#description = Let the music flow!
# # description of the stream
#url = http://<my_webserver>
# # URL related to the stream
#genre = alternative # genre of the stream
#public = yes # advertise this stream?

## mp3 low stream
#[icecast2-1]
#bitrateMode = abr # average bit rate
#format = mp3 # format of the stream: ogg vorbis
#bitrate = 64 # bitrate of the stream sent to the server
#server = <my_icecast_server>
# # host name of the server
#port = 8000 # port of the IceCast2 server, usually 8000
#password=<my_password$gt;
#mountPoint=<my_mountpoint>
#name = Neuse Radio medium mp3
# # name of the stream
#description = Let the music flow!
# # description of the stream
#url = http://<my_webserver>
# # URL related to the stream
#genre = alternative # genre of the stream
#public = no # advertise this stream?
##localDumpFile = /tmp/dump.mp3 # local dump file

## mp3 hi stream
#[icecast2-1]
#bitrateMode = abr # average bit rate
#format = mp3 # format of the stream: ogg vorbis
#bitrate = 128 # bitrate of the stream sent to the server
#server = <my_icecast_server>
# # host name of the server
#port = 8000 # port of the IceCast2 server, usually 8000
#password=<my_password>
#mountPoint=<my_mountpoint>
#name = Neuse Radio – Alternative MP3
# # name of the stream
#description = Let the music flow!
# # description of the stream
#url = http://<my_webserver>
# # URL related to the stream
#genre = Alternative # genre of the stream
#public = yes # advertise this stream?
##localDumpFile = /tmp/dump-high.mp3 # local dump file

As you can see, you can define as many encoded streams as you wish. Just increment the name of the Icecast2-x sections as you go.

Now that you’ve got darkice configured, you’ll need to set up Icecast2 to distribute the stream darkice creates. Put your Icecast address and service password in your /etc/darkice.cfg file. I tried sending darkice connections from my local computer to an Icecast server on my hosted server but I found that darkice is really finicky about latency. If for some reason your Icecast server hiccups and darkice can’t send out the packets is creates then darkice begins to act really squirrely and eventually will crash. The best workaround for this is to run darkice and Icecast on the same machine, then have Icecast do the sending! If Icecast is set up to relay to a hosted Icecast instance you will avoid any latency issues with darkice.

After hosting my Rivendell in the cloud for a few years I opted to host it on a home server instead. I wanted to listen to my stream while I’m around the house but only send packets over my Internet connection when I have to. Thus, I have my hosted Icecast instance set up to pull a steam from my home Icecast server only upon demand. Here’s a snippet from my hosted Icecast server which shows how relaying is configured:

<relay>
<server><my_home_server></server>
<port>8000</port>
<mount><my_mount_point></mount>
<local-mount><my_mount_point></local-mount>
<on-demand>1</on-demand>
<relay-shoutcast-metadata>1</relay-shoutcast-metadata>
</relay>

My Docker container has no display of its own and any server in the cloud won’t have one, either. So how do you manage Rivendell when it’s headless? You create a virtual display in the form of VNC. I install a lightweight display manager called ICEWM, the Ice Window Manager. It’s not fancy but it works well for a virtual environment. You could now access your desktop by installing something like the tigervncserver but I prefer for security reasons only to run VNC when I need it. Thus, I ssh into my container and run the “x11vnc” command from the command line. I then port-forward my ssh session so that it connects with x11vnc on my container using the localhost:5900 port. Now I can manipulate Rivendell remotely! It’s nearly as good as running it locally but any latency you have between you and your hosted server will make audio editing challenging at best and impossible at worst.

How do you import audio? I use scp to copy my audio files to a folder on my hosted system. Then I open an ssh session there and run “rdimport” from there. Then I delete the uploaded audio once it’s imported.

How do you get around the latency issue so you can edit audio? This is best done by running a local Rivendell instance and setting up both instances to share the same /var/snd directory and the same MariaDB database. You can use NFS or SAMBA to share the /var/snd directory ,or if you are feeling extra geeky, you can make S3 buckets look like a filesystem using the fuse-s3 package. As for the database, you can either connect to the same database instance for both Rivendell instances or you can set up database servers on either end and configure them to replicate. Setting up MariaDB for replication is beyond the scope of this article but there are several good resources on the Internet that show how to do this.

How to connect between your home and your hosted server? I use a one-way VPN to get to my hosted server. WireGuard is my VPN of choice at the moment.

So there you go. Below is the Dockerfile I’m using. Note that you’ll have to supply your own binaries for the parts not already packaged, like the LADSPA plugin and JackEQ. Hopefully this is enough to get you going! I’ll refine this process further and post a followup someday. Enjoy!

FROM centos:7
ENV container docker
RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == \
systemd-tmpfiles-setup.service ] || rm -f $i; done); \
rm -f /lib/systemd/system/multi-user.target.wants/*;\
rm -f /etc/systemd/system/*.wants/*;\
rm -f /lib/systemd/system/local-fs.target.wants/*; \
rm -f /lib/systemd/system/sockets.target.wants/*udev*; \
rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \
rm -f /lib/systemd/system/basic.target.wants/*;\
rm -f /lib/systemd/system/anaconda.target.wants/*;
VOLUME [ “/sys/fs/cgroup” ]
CMD [“/usr/sbin/init”]

# Create rduser (password is rduser)
RUN adduser –create-home –groups wheel,audio rduser ; \
echo “rduser:rduser” | chpasswd ;\
mkdir -m 0750 /etc/sudoers.d && \
echo “rduser ALL=(root) NOPASSWD:ALL” >/etc/sudoers.d/rduser && \
chmod 0440 /etc/sudoers.d/rduser

# locale-gen en

# Install EPEL repos
RUN yum install -y epel-release; \
yum-config-manager –enable epel-testing

# Get repo GPG keys
RUN rpm –import https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL http://download.paravelsystems.com/CentOS/7/RPM-GPG-KEY-Paravel-Broadcast

# Install rivendell stuffs
RUN yum-config-manager –add-repo http://download.paravelsystems.com/CentOS/7/Paravel-Broadcast.repo ; \
yum install -y rivendell sudo lame faac libaacplus twolame libmad id3lib icewm jackd x11vnc openssh-server tigervnc-server-minimal tigervnc-server supervisor cronie ghostscript-fonts less; \
/usr/sbin/sshd-keygen

#COPY rdmysql.conf /etc/mysql/conf.d/rdmysql.cnf
#COPY rd.icecast.conf /etc/rd.icecast.conf
COPY rd.conf /etc/rd.conf
COPY supervisord.conf /etc/supervisord.d/supervisord.conf

# copy darkice binary and config
COPY darkice /usr/local/bin
COPY darkice.cfg /etc
COPY rlm_icecast2.conf /etc

USER rduser
COPY init.sh /home/rduser/init.sh
COPY icewm /home/rduser/.icewm
COPY xstartup /home/rduser/.vnc/xstartup

RUN sudo chown -R rduser.rduser /home/rduser ; \
echo ‘rduser’ | vncpasswd -f > /home/rduser/.vnc/passwd ; \
chmod 700 /home/rduser/.vnc ; \
chmod 600 /home/rduser/.vnc/passwd ; \
chmod +x /home/rduser/init.sh /home/rduser/.vnc/xstartup
USER root

COPY start /start
RUN chmod +x /start

#set proper timezone
RUN sudo rm /etc/localtime && sudo ln -s /usr/share/zoneinfo/America/New_York /etc/localtime

# Expose ssh, vnc
EXPOSE 22
EXPOSE 5900
# Set boot command
CMD /start

# Set permissions on /var/snd
RUN chown rduser /var/snd

# Add fftw-3.3.3* and libaacplus*.rpm to get darkice running (and cfg file, too)
#RUN mkdir /usr/local/src/rpms; chown rduser /var/snd
#COPY rpms/libaacplus-2.0.2-1.el7.centos.x86_64.rpm rpms/fftw*.rpm /usr/local/src/rpms/
#RUN sudo yum install -y /usr/local/src/rpms/*.x86_64.rpm
RUN mkdir /usr/local/src/rpms
COPY rpms/*.x86_64.rpm /usr/local/src/rpms/
RUN sudo yum install -y /usr/local/src/rpms/*.rpm && rm -rf /usr/local/src/rpms/*.x86_64.rpm

# add cron job to generate logs
COPY cron.rduser /var/spool/cron
RUN mv /var/spool/cron/cron.rduser /var/spool/cron/rduser && chown rduser.rduser /var/spool/cron/rduser && chmod 600 /var/spool/cron/rduser

# copy jackeq bin, oggimport and library
COPY jackeq /usr/local/bin/jackeq
COPY oggimport /usr/local/bin/oggimport
COPY dj_eq_1901.so /usr/local/lib/ladspa/

#Copy asound.conf for using alsa apps (linphone) with JACK
COPY asound.conf /etc/

# mount music dir
RUN mkdir /mnt/music