I did some digging and found a solution:
First, you have to use OGV format video. I don't know the technical details, but that's the only thing that works in Node-Webkit, even though MP4 works in regular Chrome.
Since handbrake doesn't seem to support OGV, I used this free online service to convert a sample file. EDIT: Turns out that VLC can do this, too, which is a lot easier than relying on an online converter.
Second, import your video into the project.
Finally, set up your video as usual. Set the source to videoname.ogv.