Archive for the ‘Programs’ Category

Gyroscope MIDI Controller

Tuesday, January 23rd, 2018

I made a program to send pitch-bend messages to Bawami (my MIDI synth) based on the strongest reading out of the X/Y/Z axes of the gyroscope on the GY-87 sensor board, via an Arduino. Gently moving the sensor makes for a really natural-feeling control for vibrato, allowing really subtle (or not-so-subtle) pitch changes.

[Watch in HD]

I was able to get readings from the board to Windows at a stable speed of 400 Hz, but to avoid spamming too many MIDI messages (a problem if sending them outside the computer to some hardware synth), the pitch-bends are “only” being sent at 100 Hz. =P

The GY-87 also has X/Y/Z accelerometers, but these were way too sensitive to orientation to be convenient to use as a controller. Gravity is always pulling down on one axis, so if you tilt the sensor then it massively overwhelms the readings that you actually want (the ones caused by moving the sensor around). The best use I could get from them was tracking the maximum difference between 2 points in time and sending that as a MIDI message, which basically just made it respond to vibrations (and only made positive numbers). The gyros naturally only detect changes, so the readings centre around 0 and go negative when turning in one direction and positive in the other, ideal for vibrato.

BaWaMI (revision 134)

Tuesday, December 5th, 2017

This is a tiny update which simply fixes the checkbox to enable/disable responding to MIDI channel coarse/fine tuning messages, on the “MIDI params” tab of the config window, so that it actually has an effect. Previously, Bawami always responded to those messages even if the checkbox was unticked.

You can grab this fixed version here (7.80 MB).

BaWaMI (revision 133)

Wednesday, November 29th, 2017

This is a big update which fixes a bunch of bugs, especially ones related to the PC speaker, and graphical mistakes. A new internal tuning system means Bawami now supports a big range of tuning messages (their effects can combine together!), plus there are a few new instruments and tweaks to existing ones.

Some of the MIDI Tuning Standard messages are quite advanced, and you’d typically use some other scale-related software to generate the SysEx messages rather than hand-crafting them, but they mean that Bawami can now play with tuning other than equal temperament, or different scales entirely (e.g. Arabic).

You can grab this new version from here (7.80 MB), and view details of all the changed stuff in the full post, below:

(more…)

BinToUTF8 – Public release

Thursday, May 4th, 2017

Because several people have asked for it, I’ve decided to release my program for converting any binary file to a valid UTF-8-encoded text file (and vice-versa). This is the program I made to be able to train the open-source neural network software “torch-rnn” on audio, even though it’s only designed to work with text, in these previous videos.

My program is a console-mode program, so it has no graphical interface, and it’s an EXE, so it’ll only run on Windows (and maybe Wine). It’s also slow, because I hadn’t had the pressure (from the idea of making it public) to optimize it until I suddenly decided to release it this evening. It comes with pseudocode and a technical description for any programmers who want to remake it to run on other OSes, though (they’re the same text files I linked to in the blog post for my first neural network video).

The download contains BinToUTF8.exe, which you can use yourself on the command prompt (run it from the command prompt without any parameters to see usage instructions). It also contains several batch files, which make it much more convenient to use – you only have to drag a binary or text file onto the batch file on Windows Explorer to automatically launch BinToUTF8.exe with the appropriate command line parameters.

A brief description is below, but make sure you read the included “info.txt” to find out what each batch file does and avoid accidentally overwriting any of your own files!

The program works by assigning a unique Unicode or ASCII character to each of the 256 possible byte values in your binary file. There are 2 modes for this:

  • Byte/Character Lookup (BCL) mode (recommended):

Characters are assigned on a “first-come, first-served” basis, meaning that bytes appearing near the beginning of the file will be assigned ASCII characters, and Chinese Unicode characters will only be used once no more ACSII characters are available. This is done to allow you to pass text from the start of the file to torch-rnn using torch-rnn’s -start_text parameter, which does not support Unicode characters. A utf8.bcl file is made when converting to text and is required when converting back to binary. This file is the lookup table for converting between bytes and Unicode characters which the program made when converting the binary file to text.

  • Non-BCL mode (default, not recommended for torch-rnn):

All bytes are converted to Chinese Unicode characters and none are converted to ASCII. This means the text file will be larger, but more importantly, you won’t be able to use any of this text with torch-rnn’s -start_text parameter. The conversion in this mode may be faster, and no utf8.bcl file is made or required.

Text files made using the BCL mode cannot be converted back to binary using the non-BCL mode, and vice-versa. To convert text back to binary correctly, you must use the same mode that you used when converting the original binary file to text.

You can download BinToUTF8 from here (19 KB). Now, have fun!

(By the way, if training torch-rnn on audio files, you should use an 8-bit audio encoding such as 8-bit PCM, U-law or A-law, to be kind to torch-rnn.)

BaWaMI (revision 132)

Sunday, April 23rd, 2017

This biggest update ever to my MIDI software synth contains dozens of bug- and crash-fixes, improvements to live MIDI input, and a big new feature for instruments called “multi-osc” (explained below), which many instruments now take advantage of! It’s stable when clicking “Apply” to restarting the sound system, which often caused crashes in the past, and there are a couple of new features to do with overriding controls. Also, one particular system file (included since a long time ago) is now correctly checked / set up when Bawami starts, which may fix Bawami not being able to start for some people. All these improvements mean that Bawami has grown to version 0.7!

The new “multi-osc” feature for instrument files allows one note to trigger more than one sound channel, massively improving the sound of some instruments. This opens the door to having a proper Fifths instrument, octave basses, octave-stacked strings, detuned Honkey Tonk, better organs and more! Of course, I updated lots of instruments to take advantage of this, and added new GS instruments whose sounds simply weren’t possible to generate before. Multi-osc is enabled by default, but can be disabled if you want to keep CPU usage as low as possible (if you really hate the new sound, you can replace all instrument files with those from the previous version, or have fun editing them yourself!).

You can grab this shiny new version from here (7.79 MB), and view the full post to see exactly what’s changed, below:

(more…)

Neural Network Tries to Generate English Speech (RNN/LSTM)

Saturday, December 24th, 2016

By popular demand, I threw my own voice into a neural network (3 times) and got it to recreate what it had learned along the way!

[Watch in HD]

This is 3 different recurrent neural networks (LSTM type) trying to find patterns in raw audio and reproduce them as well as they can. The networks are quite small considering the complexity of the data. I recorded 3 different vocal sessions as training data for the network, trying to get more impressive results out of the network each time. The audio is 8-bit and a low sample rate because sound files get very big very quickly, making the training of the network take a very long time. Well over 300 hours of training in total went into the experiments with my voice that led to this video.

The graphs are created from log files made during training, and show the progress that it was making leading up to immediately before the audio that you hear at every point in the video. Their scrolling speeds up at points where I only show a short sample of the sound, because I wanted to dedicated more time to the more impressive parts. I included a lot of information in the video itself where it’s relevant (and at the end), especially details about each of the 3 neural networks at the beginning of each of the 3 sections, so please be sure to check that if you’d like more details.

I’m less happy with the results this time around than in my last RNN+voice video, because I’ve experimented much less with my own voice than I have with higher-pitched voices from various games and haven’t found the ideal combination of settings yet. That’s because I don’t really want to hear the sound of my own voice, but so many people commented on my old video that they wanted to hear a neural network trained on a male English voice, so here we are now! Also, learning from a low-pitched voice is not as easy as with a high-pitched voice, for reasons explained in the first part of the video (basically, the most fundamental patterns are longer with a low-pitched voice).

The neural network software is the open-source “torch-rnn“, although that is only designed to learn from plain text. Frankly, I’m still amazed at what a good job it does of learning from raw audio, with many overlapping patterns over longer timeframes than text. I made a program (explained here, and available for download here) that substitutes raw bytes in any file (e.g. audio) for valid UTF-8 text characters and torch-rnn happily learned from it. My program also substituted torch-rnn’s generated text back into raw bytes to get audio again. I do not understand the mathematics and low-level algorithms that go make a neural network work, and I cannot program my own, so please check the code and .md files at torch-rnn’s Github page for details. Also, torch-rnn is actually a more-efficient fork of an earlier software called char-rnn, whose project page also has a lot of useful information.

I will probably soon release the program that I wrote to create the line graphs from CSV files. It can make images up to 16383 pixels wide/tall with customisable colours, from CSV files with hundreds of thousands of lines, in a few seconds. All free software I could find failed hideously at this (e.g. OpenOffice Calc took over a minute to refresh the screen with only a fraction of that many lines, during which time it stopped responding; the lines overlapped in an ugly way that meant you couldn’t even see the average value; and “exporting” graphs is limited to pressing Print Screen, so you’re limited to the width of your screen… really?).

BaWaMI struggles to play Arecibo by TheSuperMarioBros2 [Black MIDI]

Wednesday, September 14th, 2016

Here’s my MIDI software synth Bawami doing its best to even keep responding while trying to play TheSuperMarioBros2‘s black MIDIArecibo“. The left view shows how it’s processing every MIDI message. Not shown: About 5 minutes of Bawami loading the 12MB MIDI file hideously inefficiently (tempo changes make it even worse).

[Watch in HD]

This problem of my player stopping responding when maxed out is something I need to (re-)fix. I fixed this a long time ago (probably before releasing Bawami), but broke it again afterwards somehow, also a long time ago now… As always, the most recent version of Bawami can be download here (also check the most recently tagged posts to see recent changes).

TheSuperMarioBros2 have made a lot of great black MIDIs that are often fun to stress-test MIDI players with. You can see lots playing at their channel (they also provide download links for the MIDI files). However, Bawami’s loading of MIDIs is inefficient, so I’d recommend not trying to torture it with black MIDIs too much. I also suggest unticking “Loop” so that, if it stops responding during playback, it’ll eventually start responding again at the end.

Neural Network Learns to Generate Voice (RNN/LSTM)

Tuesday, May 24th, 2016

This is what happens when you throw raw audio (which happens to be a cute voice) into a neural network and then tell it to spit out what it’s learned. (WARNING: Although I decreased the volume and there’s visual indication of what sound is to come, please don’t have your volume too high.)

[Watch in HD]

This is a recurrent neural network (LSTM type) with 3 layers of 680 neurons each, trying to find patterns in audio and reproduce them as well as it can. It’s not a particularly big network considering the complexity and size of the data, mostly due to computing constraints, which makes me even more impressed with what it managed to do.

The audio that the network was learning from is voice actress Kanematsu Yuka voicing Hinata from Pure Pure. I used 11025 Hz, 8-bit audio because sound files get big quickly, at least compared to text files – 10 minutes already runs to 6.29MB, while that much plain text would take weeks or months for a human to read.

I was using the program “torch-rnn“, which is actually designed to learn from and generate plain text. I wrote a program that converts any data into UTF-8 text and vice-versa, and to my excitement, torch-rnn happily processed that text as if there was nothing unusual. I did this because I don’t know where to begin coding my own neural network program, but this workaround has some annoying restraints. E.g. torch-rnn doesn’t like to output more than about 300KB of data, hence all generated sounds being only ~27 seconds long.

It took roughly 29 hours to train the network to ~35 epochs (74,000 iterations) and over 12 hours to generate the samples (output audio). These times are quite approximate as the same server was both training and sampling (from past network “checkpoints”) at the same time, which slowed it down. Huge thanks go to Melan for letting me use his server for this fun project! Let’s try a bigger network next time, if you can stand waiting an hour for 27 seconds of potentially-useless audio. xD

I feel that my target audience couldn’t possibly get any smaller than it is right now…

EDIT: Because I’ve been asked a lot, the settings I used for training were: rnn_size: 680, num_layers: 3, wordvec_size: 110. Also, here are some graphs showing losses during training (click to see full-size versions):


Training loss (at every iteration) (linear time scale)


Training loss (at every iteration) (logarithmic time scale)


Validation loss (at every checkpoint, i.e. 1000th iteration) (linear time scale)


Validation loss (at every checkpoint, i.e. 1000th iteration) (logarithmic time scale)

For sampling, I simply used torch-rnn’s default settings (which is a temperature of 1), specifying only the checkpoint and length and redirecting it to a file. For training an RNN on voice in this way, I think the most important aspect is how “clear” the audio is, i.e. how obvious patterns are against noise, plus the fact that it’s 8-bit so it only has to learn from 256 unique symbols. This relatively sharp-sounding voice is very close to a filtered sawtooth signal, compared to other voices which are more breathy/noisy (the difference is even visible to human eyes just by looking at the waveform), so I think it had an easier time learning this voice than it would some others. There’s also the simple fact that, because the voice is high-pitched, the lengths of the patterns that it needs to learn are shorter.

EDIT 2: I have been asked several times about my binary-to-UTF-8 program. The program basically substitutes any raw byte value for a valid UTF-8 encoding of a character. So after conversion, there’ll be a maximum of 256 unique UTF-8 characters. I threw the program together in VB6, so it will only run on Windows. However, I rewrote all the important code in a C++-like pseudocode here. Also, here is an English explanation of how my binary-to-UTF-8 program works.

EDIT 3: I have released my BinToUTF8 Windows program! Please see this post.

BaWaMI (revision 131)

Sunday, April 10th, 2016

This is a small update which fixes OGG file rendering, and a couple of other superficial things.

  • Fixed tabbing order of controls on “Mod shape” tab of the config window.
  • Writing OGG files works again – I accidentally removed a file needed by the OGG encoder in revision 130. Users who extracted revision 130′s folder over revision 129′s would not have experienced the problem.
  • Stopped warning about buggy controls from being displayed on the “MIDI params” tab of the config window, since it’s no longer true.

You can grab the new version from here (7.87 MB)!

BaWaMI (revision 130)

Wednesday, March 2nd, 2016

This update lets Bawami write WAV files in paths containing non-ASCII characters (something long-overdue), and fixes a bunch of complicated interwoven bugs related to the handling of “bad” MIDI files and of percussive notes. There’s also a load of other fixes, including for portamento-related bugs, crashing in /translator mode, and visual fixes, and a couple of new instruments.

You can grab the latest version from here (7.81 MB), and see details of the changes below the page break:

(more…)