If you were to invite him to a home-prepared dinner and, say, a DVD (or some music you'd like him to hear, or a board game, or whatever), then he would already be in your flat and you wouldn't have that awkward moment of inviting him in "for a coffee". Then, watch for signals. With privacy, he may develop enough courage to make a move without your having to appear "loose" by explicitly inviting him in for "you know".

If that seems too forward, you might prevail on another couple to come to dinner as well, but privately arrange with them to leave earlier in the evening so that you will have him to yourself for a while. Then proceed as above...

Good luck, whatever you choose to do!