In progress
Point your phone at a page of sheet music and it tells you what is on it. You keep your physical books.
Why it exists
I am an adult beginner at the piano, with real books on the stand. The problem was not motivation, it was that I could not decode the page in front of me.
I fundamentally don’t know the basics of when I read the sheet, how do I remember what notes, especially if it’s shelves below. I want to be able to look at my phone at the side of the paper and then it help me.
How it will work
Step one
Photograph the page
Manual shutter, always. It takes a burst and keeps the sharpest frame.
Step two
Read what is there
The key, the chords, the note range, and whether it fits a beginner hand.
Step three
Explore or practise
Tap any note for a plain explanation, or follow a cursor at your own tempo.
What it will not do
- Set you a curriculum. There are no lesson ladders and no virtual coach.
- Replace learning to read music. It is a companion to it.
- Present a guess as a fact. When recognition is uncertain, it says so.
- Touch the network during recognition. There is no server at all.
Where it goes
A gentle lens for the books you already own. Further out, the same teaching layer on a headset, annotating the real page on the stand in front of you rather than a screen beside it.
White paper
- Platform
- Native iOS, Swift and SwiftUI, iPhone and iPad
- Capture
- Custom AVFoundation scanner, three frame burst, sharpest kept
- Recognition
- On-device optical music recognition, pure Swift with ONNX Runtime
- Text
- Apple Vision for titles and markings. Never for notation
- Engraving
- Verovio 6.1.0, bundled and fully offline
- Audio
- AVAudioEngine with a bundled piano SoundFont
- Network
- None in the recognition path. No server, no cloud AI
Recognition accuracy is benchmarked against a fixed set of pages from my own shelf. The method, the failure cases and the current numbers are in the full write up.
Buying it
Not yet. There is an open licensing question in one of the recognition components, and I will not sell something I am not certain I can hand over. The rest of the work, the teaching layer, the design system and the analysis engine, is mine outright. If you are interested, email support@picksley.ai and I will tell you where it stands.