If you have a topic you have a better chance of creating a thread between clips that seems more natural.
If you talk about 15 things with no natural transitions and cut out the dead space, every time you cut back you're on a different topic. It feels disjointed.
If you talk about 1 or 2 topics you have a related clip to cut back to most times. Then you can cut in the tangents in between major topics so that they are little asides and still fun, but not the entirety of the video.
Also, the more you guys do this together you will develop a natural rapport that helps keep things together and not broken up. You'll naturally build in segues that get around the issue. And you'll get better at using those segues and your second angle to cover the bigger cuts.
You obviously already know how to talk to one another, sense of humor, etc. You just need to finish transitioning that to a video setting where you have to force yourself to keep talking and linking topics.