Fun fact: the painful desire to breathe you feel when you are “out of breath” isn’t due to lack of oxygen, but too much carbon dioxide (CO2)! Likewise, the feeling of faintness, tingling, and dizziness when you hyperventilate isn’t due to too much oxygen [which hyperventilating can’t cause anyway], but too little carbon dioxide. In fact, most people’s bodies can’t detect low oxygen at all!
When you breathe in and hold your breath, you have plenty of oxygen (which keeps you from blacking out) and little CO2. Over time, you run out of oxygen, but your blood CO2 levels rise. Your brain actually uses blood CO2 as an indicator for blood oxygen, so when CO2 levels are too high you are signaled to take a breath. If you find it hard to believe that low oxygen isn’t what makes you want to breathe, consider carbon monoxide poisoning. It happens because you are breathing air with low oxygen, but the CO2 levels are the same and your body regulates your breathing as usual, so you die without ever realizing you were oxygen deprived. You will blackout without any warning.
When you exhale, you are forcing out the CO2. Why is holding your breath here harder? Because you’ve lowered your lung volume, the CO2 concentration of your lungs is now higher than your blood. Gasses diffuse from air to blood passively, from areas of high concentration to low. Lungs full of a breath of air have a higher oxygen concentration and lower CO2 concentration than your blood does, so oxygen flows into your blood and CO2 out of the blood. When the lungs become as concentrated as your blood with CO2, the CO2 no longer leaves your blood, but builds up until your next breath.
Recall that concentration is dependent on volume: empty lungs have lower volume, so fewer CO2 molecules are needed to get the same concentration as your blood. Thus, CO2 stops leaving your blood faster if the lungs are empty, meaning the CO2 levels in your blood reach critical faster and you need to breath sooner. Oxygen levels in your blood are likewise not rising fast enough (or at all, since you just exhaled), but it’s the rise in CO2 that makes you gasp.