I’ve often wondered about the reverse. Get the finest grand piano action, like the ones used in the Fazioli, and create a minimal mechanical unit that has just that piece and the pedals (with the action mechanics that connect the left/right pedals to the keys) and then nothing else but a couple of sensors and a couple fast USB-C. The Kawai Novus series did an OK action and left pedal, but then they connect it to a whole furniture and their sound synthesis and vibrations are imperfect. With modern powerful GPU the sound should be possible to become near perfect and easy to tune to one’s preference.
Yes but without the need for the full furniture. One should be able to miniaturize everything else but the action. I think such compact designs will eventually appear but the difference in real action versus very good hybrid digital actions is of highest interest to musicians who typically have access to real pianos. Roland, Yamaha, Kawai, and others have been moving in that direction over the years and the top models by Yamaha and Kawai have passable actions though they still have a large piece of furniture around them.
A large part of the experience comes from the actual sound. GPUs don't matter. Mozart has some very good physically modeled pianos that respond very naturally. But to get that to sound like a real piano in a real space is a greater challenge than reproducing the feel of Fazioli.
I am no master pianist, but I imagine much of the feel of such an instrument comes from the physical weight of the keys, and the feel as they strike the strings. Although good quality keyboards have "weighted" keys, they don't feel exactly the same. I think it would be hard to reproduce this exactly.
The top-end Yamaha electric pianos have a near-complete replica of the action inside of them. If you adjust the volume exactly right, you almost replicate the feel of playing a real piano.
Real (proper) pianos also have latency between a finger pushing on a key and sound coming out. There's mechanical bits that need to move some distance before a string is struck or muted, and these bits do not have infinite acceleration -- they've got mass, momentum, and inertia.
Can these latencies not be balanced between techniques?
I'm no expert on synthesis or pianos, but my immediate thought is that the digital thing may need slowed down to match the latency of the analog thing.
You may have a point. Perhaps naively I imagined that if competitive gamers can handle it, they may be useful enough for piano. Currently the CPU software is lacking in quality (though improving over time) but has acceptable latency compared to grands (which also have latency due to the mechanism and the ___location of the sound source).
I had an idea for a similar project. Basically, a tabletop drum kit. In this case, you’d tap on a few objects on the table that make distinct sounds and then map that back to a virtual drum kit in your DAW using midi. I never got around to building it, but I am still keen on the idea.
That already exists in some forms, most notably perhaps in Dubler 2.
I’m also working on an open source version of something similar, albeit quite involved and aiming at professional real-time performance and drumkit augmentation (if all goes well to share on HN this month for the first time :))
> I had been playing with the idea of creating a browser-based virtual piano for when I'm travelling and don't have access to a real piano but have my laptop with me. The idea would be to point the webcam down at the table between me and the laptop, and play on the table as if a piano were there. Then use the mediapipe framework [1] to capture finger positions, and use those to update a virtual environment like the one you have here.
> I put it on hold due to the significant engineering required, but it seems you have already implemented (and open sourced!) the browser-based piano simulation component.
> A quick scan through your repo indicates that this is all implemented in Python. I see that you are using mujoco_wasm [2].
I'm surprised they actually exist instead of being a rendered design concept that I saw many years ago, and are apparently common and numerous enough that someone would write a listicle about them. They don't seem to cost more than a good regular keyboard either, but I've found reviews saying that the typing experience is about as uncomfortable as you'd expect from tapping on a hard surface.
You don't need to type on a hard surface (but anyway, most people tap away on their phone screens for hours in a day). You can type on an octopus if you want. Anything can be a projection surface keyboard with enough inspiration.
This is a delightful project. Congratulations on shipping it! I would be curious to see ways to do this where the piece of paper had lines so you could visually see on the paper which key was which.
Doing this would be tricky, though. On first thought, an object detection model that detects each key from a perspective, then saves the coordinates, may work. Then you would calculate the Intersection over Union between the finger annd each region. This may only work with a top-down perspective, which might be less practical in real-world use.
In any case, I always miss my keyboard when I am traveling. A project like this feels a lot more practical than the “portable” travel keyboards.
When I was in 3rd grade we learned to type on a “keyboard” that was printed on a sheet of paper. I suppose it’s a novel way to teach kids to type with a budget of zero.
I have a long password which I don't remember. I can type it on my laptop, but not recall it.
Last holiday season I unexpectedly needed to access my master keyring, but all I had with me was my phone. Which doesn't have a laptop-sized or -shaped keyboard.
No amount of conscious typing helped me remember. As a last-ditch effort, I found squared paper and a pen, drew a full-sized keyboard, and "typed" my password in on that.
It took a couple of attempts, but I did eventually reverse-engineer my own password from my own muscle memory by carefully observing what my hands are doing.
You should write it down. I had such a password that only my fingers knew, not my head. Then I went a couple of months without using it. When I wanted to type the password again, I had completely forgotten it. Without even knowing the first letter, I had no hope of guessing it.
When I was in high school in the 80s, there was a piano class at the high school and the students had non-sounding, non-moving plastic piano keyboards to practice on at their desks. Nowadays, they get 60-key keyboards with headphones.
I'm a pianist and have actually resorted to drawing a keyboard on paper in the past when I've been without a piano for extended periods (yes I'm addicted lol).
Although VR keyboards absolutely won't replace real pianos for obvious reasons they actually do have some use. They are certainly better than nothing and better than drawing a piano on physical paper. The accuracy of finger tracking still needs improvement but that will come and as haptics improves I'm sure a virtual piano one day might start to feel more realistic. Technology needs to catch up.
I've been able to memorise simple pieces with PianoVisions air piano during free play. While playing with songs within the app I've been able to remember some harder works.
Still connecting a midi piano is 100 times better no argument about that.but the virtual piano certainly has its place at the moment and will become stronger as tech improves.
I've previously used PianoVision, and it worked quite well. The concept seems similar, now augmented with VR. I anticipate that VR pianos will evolve to such an extent that they'll replace traditional pianos in many scenarios (like teaching/learning)
https://www.meta.com/experiences/5271074762922599/
> they'll replace traditional pianos in many scenarios
I'm not convinced. Totally VR pianos are obviously a bad idea just like totally VR keyboards for typing are rubbish for anything more complex than a wifi password.
And as for learning using VR - well, I'm not convinced by things like pianovision. How do you learn a piece with that? With sheet music I can play the music at my own speed, when and how I want. I can go back and play a measure a few times to experiment with timing. Speed up and slow down, and develop my own ear for how I want it to sound. If I'm feeling cheeky, I'll improvise extra notes in the song, or take some notes out if I'm not feeling it. At an instrument, sheet music is a polite suggestion.
I can imagine playing something like pianovision and getting good at pushing the buttons at the right time like guitar hero. But being a human shaped CD player feels like a very small part of what makes a musician.
I agree for pianos, because the mechanical feedback is part of what makes playing enjoyable.
I'm not as sure for keyboards in VR! There has been a lot of research on non-invasive brain computer interface (BCI), including predictive systems that guess what you want to type, instead of where you're actually typing (example from 8y ago [1]).
Simpler gesture recognition is already on the market (like this one, from 2020, for the apple watch [2]). And now bigger VR players are investing in the tech [3]. I expect useful brain interface to be integrated in common VR devices in a couple of generations.
Pianovision encourages connecting a midi piano to the headset. It also uses sheet music and has over 1000 works with finger numbers and sections. Virtual keyboards require haptics to get a lot better, still it is better than nothing at all.
A melodica is not good for learning piano obviously but is a really interesting musical instrument in and of itself.
Trying to replace the real instrument makes about as much sense as a paper computer keyboard for learning typing. If you really want to learn to type, you obviously are going to "invest" in a real computer keyboard.
If that investment is too much up front then you have already set yourself up for failure so what is the point?
This isn't to say that something like this can't make its own cool instrument, but as to learning the piano, no matter how good the tech is, its always going to be an imitation of the real thing. Which isn't to say that its useless for learning piano either.
What? Why would they? Digital piano sounds still can't reproduce the richness of dynamics and texture of an acoustic piano. Practising on a digital keyboard and then performing on a real piano is jarring enough, let alone using an instrument with zero haptic feedback. This is pure gimmick.
I'm glad you're finding ways of getting into music inexpensively. But my advice is: haptics are super important in music interfaces! Spend the extra effort to make something that you can satisfyingly press and which can give your fingers some mechanical feedback. Your playing will be sooooo much more rewarding.
Obviously in no way comparable, but you reminded me that many, many years ago, when I was taking typing in high school, I'd sit in class "typing" in mid-air (hands at my side) transcribing what the teacher was saying.
It was a little annoying to people around me, but ended up being very effective.
This is pretty cool work, love the concept. I am curious if there are fast, easy to use and efficient ways to also detect the full hand. Maybe by placing the camera from above would make it easier to process but at the cost of complicating the setup.
I think this idea would work better using 3D printed keys. It's possible to print flexible springs, and filament can be used as a hinge pin. The computer just needs to recognize which keys have moved.
yeah! but main motive was to make it as free as possible. Someone like me who does not have much of money to buy a piano but to play one see this as absolute win with trade offs ofc!
Mayuresh, I'd like to donate a keyboard instrument to you. Please choose between an AKAI MPK Mini and a Hohner Melodica and email me at my nick on gmail with a shipping address.
Also, if you consider this effort much cheaper than a $100 instrument, maybe I could hook you up with a programming job?
I suspect it's less about the music and more about the finger tracking, but to the extent it's about the music - billions of people have smartphones with cameras that would work with this code, but would unlikely be able to afford piano lessons or convince / inspire anyone to offer to purchase them an instrument without first demonstrating aptitude in some area. Definitely not intended at a dig at you and hope you manage to hook Mayuresh up with a keyboard instrument and a job!
You are absolutely right, my intention behind it was that you could really play the piano not on a small screen (like a phone) but like an actual sized piano. I didn't want to buy one because they were expensive and I am more like "free is the better" guy. So I decided to make one on paper. This being rather a prototype, I am currently working on designing and mapping keys, improving upon touch recognition. ( also I have mailed you )
