I like electronic music. If you go to some DJ’s shows, you can’t see what they are doing. You don’t know that they are actually live or not. Yes, music is good, but interaction between audience is much less then band’s performance, which is using various of real instruments

I wanted to make something that can control virtual instruments and interact with audience, too.  So I thought about using AR to make live performance more visual.

Above video is my result. I used ‘Processing’, ‘NyARToolkit’, and ‘Minim’. What I did is that I mapped each virtual instruments on each pattern, and when pattern is in the view of camera, instrument is played and box is popped up. The height of box is proportional to the volume of sound. I will show part of my code.

This is ‘setup’ part. I set camera, load patterns for AR recognition, load instrument files.

void setup(){
  String camId = "1";
  if(useExternalCam){
    camId = "0";
  }
  cam = new GSCapture(this, arWidth, arHeight,"0");
  size(arWidth,arHeight,OPENGL);
  cam.start();
  textFont(createFont("Arial", 80));
  displayScale = (float) width / arWidth;
  nya = new MultiMarker(this, arWidth, arHeight, camPara, NyAR4PsgConfig.CONFIG_DEFAULT);
  nya.setLostDelay(1);
  String[] patterns = loadPatternFilenames(patternPath);
  for (int i=0; i<numMarkers; i++) {
    nya.addARMarker(patternPath + "/" + patterns[i], 80);
    colors[i] = color(random(255), random(255), random(255), 160); // random color, always at a transparency of 160
    scaler[i] = random(0.5, 1.9); // scaled at half to double size
  }
  
  minim = new Minim(this);
  String[] audioNames = loadAudioFilenames(audioPath);
  for (int i=0; i<numAudio; i++) {
    audios[i] = minim.loadFile(audioPath +"/"+ audioNames[i],2048);
    ffts[i] = new FFT(audios[i].bufferSize(),audios[i].sampleRate());
    audioBitmap[i] = false;
  }
}

This is ‘draw’ part.

void draw(){
  if(!cam.available()) return;
  cam.read();
  background(0);
  image(cam, 0,0, width, height);
  if(!liveStarted) return;
  PImage cSmall = cam.get();
  cSmall.resize(arWidth,arHeight);
  nya.detect(cSmall);
  pushMatrix();
  drawBoxes();
  popMatrix();
  playAudio();
}

This is how I played ‘audio’. I could not use given repeat function in Minim library, because delay is occurred when I used that library. Also I need function for syncing virtual instruments in beat. It is really hard to manage little delays that computer make. So it is really hard to start virtual instrument at exact time that you want. For example, even though you put your pattern in camera view, instrument delays slightly, and this delay is critical in music. That is why I made additional function to sync virtual instruments in certain beat. It is ‘isSync’, which makes to start instrument at certain beat.

void playAudio(){
  for (int i=0; i<numAudio; i++) {
    if (!nya.isExistMarker(i)){
      if(audioBitmap[i]){
        audios[i].pause();
        audioBitmap[i] = false;
      }
      continue;
    }
    if(!audioBitmap[i] && isSync()){
        audios[i].play(0);
        audioBitmap[i] = true;
        audioStartTime[i] = millis();
        continue;
    }
    if(audioBitmap[i] && isRepeat(audioStartTime[i])){
       audios[i].pause();
       audios[i].rewind();
       audios[i].play();
       audioStartTime[i] = millis();
    }
  }
}

boolean isRepeat(int startTime){
   int loopTime = (repeatBeats*60/bpm)*1000;
   int curTime = millis();
   int numLoops = round((float)(curTime-startTime)/((float)loopTime));
   if(numLoops == 0){
     return false; 
   }
   int error = (curTime - startTime) - (numLoops*loopTime);
   if(error > -65 && error < 65){
     return true; 
   }
   return false;
}

boolean isSync(){
   int loopTime = (beatsPerLoop*60/bpm)*1000;
   int curTime = millis();
   int numLoops = round((float)curTime/((float)loopTime));
   int error = curTime - (numLoops*loopTime);
   if(error > -50 && error < 50){
     return true; 
   }
   return false;
}

This algorithm is not perfect, because I had hard time to record successful live. Still it is hard to sync between instruments. I might make another program using another language and structure.

In addition, the music that I used in this video is music that I made, so there is no copyright issue.

I found many helpful information in this site.