넘어가다 is going over an edge, and these lines use every version of that. Toppling over: push him and he goes down, go that way and you will trip. Passing by: if it gets late it is three or four o'clock. And letting something slide or being taken in, which is the sense in if we just decide to let it go then it is let go, and in the idea that saying you already know might get you off the hook.