//Audio Comments!

It's half past 2 in the morning.

I sit at my desk mindlessly scrolling through X, the soft glow of the screen fills my dark room.

Filtering through shitposts and reposts, mind adrift, I suddenly see an abomination on my feed:

I could barely contain my laughter at the shear ridiculousness of the idea. It was the perfect fit for UHAW. I tapped on the reply symbol in the post and sent out my response...

It was mere hours later that ACK was recieved

I knew in my heart that I couldn't let IroncladDev down. Nor the legions of fans waiting for such a feature to appear in VSCode.

No more // TODO

just

// HEAR THIS.

Little did I know, just the kind of bullshit that awaited me. This is the story of how INFURIATING it is to add audio comments to VS Code.

Part 1: Inception

I knew that the first thing I had to do to make this work was figure out whether VS Code extensions supported inlaying custom HTML elements. That way, I could insert an audio player directly into a code file.

FUCK

Well looks like that's the end of that for this hack. See y'all next week...

But wait a minute.

I know damn well that VS Code has extensions with custom insets. I mean, look at Github Copilot!

That is CLEARLY an insert

I had to do more investigation. After browsing VS Code's Github repo, I found exactly what I was looking for.

Merged?

Well, well, well, it turns out that many people have been asking for such a feature, no doubt to implement audio comments as well. But to get access to the feature I wanted, I had to download another copy of VS Code called VS Code Insiders.

The ability to insert custom features directly into the text editor was not supported by, you know, the version EVERYONE uses.

But wait, there's more.

To actually get access to the parts of the API I needed. I also needed to use the Proposed API and manually add the appropriate dependency into my project. If you plan on implementing this yourself you need to read this.

Part 2: Electron

With everything I needed to get started. I got to work. I flew through writing the code for adding the audio inset. I mean, after all, HTML natively provides a prebuilt audio player, which made it simple to just insert the player at whatever line a comment would be in.

I figured the easiest way to add recording functionality would be to simply add a button next to this <audio> element for recording and a button to stop recording. The audio would then be saved somewhere locally and I could just take its file path and put it straight into the <audio> element as a source.

original inset design

Easy-peasy yeah?

HELL NO

Why? Because VS Code actually purposely makes this impossible. But I didn't know that at the time. I spent hours searching for ways to get the record button to actually record something from my microphone.

To save you some trouble, let me tell you now what took me 8 hours to figure out.

VS Code basically is built on top of the Electron framework to make its standalone application that everyone uses on their computers. As recently as 2022, for security reasons, the process responsible for running the webview elements that would need to be able to access my computer's microphone became sandboxed (a fancy term to just mean isolated).

No matter what kind of permissions I tried configuring in the code, I tried running the extensions as administrator, it didn't matter. The application refused to do what I told it to do. If you're bored one day or just want to torture yourself you can read more about it here

I had to figure out another way.

Instead of relying on a sandboxed Electron and recording audio comments directly in the inset, I decided to fight fire with fire. If VS Code's Electron wouldn't cooperate, I would just build my OWN ELECTRON APP.

If I could build an Electron App that uses some sort of IPC (Interprocess Communication) to talk to the VS Code extension, I can essentially outsource the recording and saving of the files to the Electron App and set up some sort of listener instead on my extension.

Using something like fs.watch() in my VS Code extension to monitor the folder where I save the audio recordings from the Electron App, I could just load the new file whenever the folder changes!

PERFECT!!!

(But was it going to go as smoothly as I had hoped?)

Part 3: MIME

Within just a couple of hours after deciding on the new strategy, I had gotten a basic Electron application up and running. The idea was simple:

To record an audio comment:

• A user types // on the line they want the comment to be one
• They press CMD/CTRL + SHIFT + X to start recording on the Electron app
• They talk about the code or whatever else for the comment
• They press CMD/CTRL + SHIFT + X again to end the recording and everything is handled seamlessly.
• The audio would automatically be loaded into the location of the // with an audio player to play the audio.

The way the Electron app worked was it would first register a global hotkey (The ctrl + shift + x described above) so that the user can start recording from anywhere, even when the Electron app was minimized. And then it would use the built-in MediaRecorder type to automatically convert and save the recorded audio into a audio_recordings folder.

app.on('ready', () => {
	globalShortcut.register('CmdOrCtrl+Shift+X', () => {
		console.log('Shortcut Pressed'); 
		//console.log('mainWindow:', mainWindow);
		mainWindow.webContents.send('toggle-recording');
	});
});

main.js

I held my breath as I booted up the extension and the paired Electron app at the same time.

Electron app with my extension active

I pressed CMD/CTRL + SHIFT + X and started speaking.

"BOOLEAN!"

I quickly re-pressed the hotkey combination to stop recording.

SUCCESS!

The appearance of the audio inset proved that not only did the hotkey work and save something, but that the VS Code extension was actually listening properly to the audio_recordings folder and successfully inset the appropriate element.

Amidst my celebratory dancing in my chair though, lay a sliiight bug.

why is the audio element greyed out?

"Well that's weird" I thought, I was pretty sure I hooked up the file path properly to the audio element. I went to check the audio file saved in audio_recording. A .WAV file, it played perfectly in my media player.

Surely, it was a fluke right? I could solve this issue in like 5 minutes.

I made multiple audio comments, NONE of them had a working play button.

I went through the code carefully, I checked the metadata in the audio files to make sure they were actually .WAV files, I reorganized where the <audio> element's source was from. I tried changing the audio file's reference URL to some encoded webview url using asWebviewUri(vscode.Uri.file(audioUrl)); (Which I DID need to do but wasn't the main problem)

I started to panic a bit, what if audio playback was ANOTHER thing VS Code didn't support?

After several hours of racking my head, as a last resort, I decided to do a little test:

I wanted to see if it was ALL audio files with this problem. I uploaded a random .WAV file from a different part of my computer.

Huh?

The play button wasn't grayed out...

Let me get this straight, the .WAV files I was creating *specifically for this project weren't working, but some random one I made 5 years ago in a completely different directory DID?

But why was I able to the play the .WAV files in Windows Media Player just fine? Why was the one .WAV file that I didn't record the only one that worked?

And that's where I learned about MIME types.

A media type (also known as a Multipurpose Internet Mail Extensions or MIME type) is a standardized nature and format of a document, file, or assortment of bytes.

Unfortunately for me, the MIME type natively supported by MediaRecorder isn't WAV. It's actually something you should specifiy as an argument in the MediaRecorder object, otherwise, you're gonna have a bad time.

navigator.mediaDevices.getUserMedia({ audio: true })
  .then(stream => {
      //THIS DOESN'T WORK, WE NEED TO SPECIFY A MIME TYPE LIKE
      //mediaRecorder = new MediaRecorder(stream, {mimeType: 'audio/webm'}) !!!!
      mediaRecorder = new MediaRecorder(stream);
      let audioChunks = [];
      mediaRecorder.ondataavailable = event => {
          audioChunks.push(event.data);
      };
      mediaRecorder.onstop = () => {
          //WE NEED TO CHANGE THIS TO {type: 'audio/webm'}
          const audioBlob = new Blob(audioChunks, { type: 'audio/wav' });
          const reader = new FileReader();
          reader.onload = function() {
              const buffer = Buffer.from(reader.result);
              const oggFilePath = path.join(__dirname, `audio_recordings/audio_${Date.now()}.WAV`);
              fs.writeFileSync(oggFilePath, buffer);
          };
          reader.readAsArrayBuffer(audioBlob);
      };
      mediaRecorder.start();
      statusElement.textContent = "Recording started";
  });

I found out the hard way that MediaRecorder didn't support .WAV after my Electron App shit the bed after passing audio/wav as the parameter. Which meant that I would need to first record the audio comments as something like .webm and then, because I was traumatized from all the shit .WAV caused me, I decided to convert that to a good ol'reliable .mp3 file.

I downloaded the ffmpeg library specifically meant to convert audio files and changed up my audio processing in my Electron App. Before I knew it, I finished the changes.

Part 4: Fame

I booted up the extension and the paired Electron app once again.

I pressed CMD/CTRL + SHIFT + X and started speaking.

"you dumb stupid piece of shit work this time"

I quickly re-pressed the hotkey combination to stop recording...

Holy crap.

It actually worked. I couldn't believe it. All the anger and frustration that had built up was finally diffused. Like air escaping a balloon, the fury rushed out of me and all I could feel was a sense of pride and accomplishment not unlike that of landing on the moon.

I was going to be famous!

Ecstatically, I filmed a short demo video. I couldn't wait to show this to everyone, especially IroncladDev and everyone on X waiting for someone to build it!

I went straight back to the post where I promised to build audio comments for real and uploaded the video as a reply.

As I sat there, fidgeting, imagining the immense fame and glory that was about to be hoisted upon me; I started thinking about my next steps:

What speech should I give at the red carpet event?

what color private jet to buy?

Hours pass. How many impressions was my reply getting? I checked and checked. 3 views, 4 views. No likes. No comments. I started to lose hope of my dreams.

All of a sudden, I get a notification on my phone. I leapt up and tapped on it.

It was from IroncladDev.

My moment had come. Fame and riches awaited. I opened up the reply to see what he said:

...

...

...

He didn't even like my post.

Code for this project can be found here!

Do you like unhinged content? Follow me on X! Or Buy me a coffee!