I've just put a midi through a linux program called midicsv which outputs a midi as a csv file. The midi is 4 crotchets (middle C), D, E, F. Perhaps this output will help indicate what a (binary) MIDI file holds.

0, 0, Header, 1, 2, 480
1, 0, Start_track
1, 0, Copyright_t, "Unknown"
1, 0, Cue_point_t, "Created by Rosegarden"
1, 0, Cue_point_t, "http://www.rosegardenmusic.com/"
1, 0, Tempo, 500000
1, 0, End_track
2, 0, Start_track
2, 0, Title_t, ""
2, 0, Program_c, 0, 0
2, 0, Control_c, 0, 7, 100
2, 0, Control_c, 0, 10, 64
2, 0, Note_on_c, 0, 60, 100
2, 480, Note_off_c, 0, 60, 127
2, 480, Note_on_c, 0, 62, 100
2, 960, Note_off_c, 0, 62, 127
2, 960, Note_on_c, 0, 64, 100
2, 1440, Note_off_c, 0, 64, 127
2, 1440, Note_on_c, 0, 65, 100
2, 1920, Note_off_c, 0, 65, 127
2, 1920, End_track
0, 0, End_of_file

In particular, see there are Note On and Note Off events and there is increasing time (0, 480, 960, 1920). At "time=480" the previous note (60/middle C) is turned off and at "time=480" the next note (D) is turned on.