The limitations on refining of crude stocks into multiple kinds of products is still true, but with some additional factors to be considered.
Both the construction of the refinery and the composition of the feed stock affect how much fuel of a given density can be separated out as a specific kind of fuel. In the earliest refineries, the more volatile components of the crude oil were simply boiled off and condensed as "gasoline" (and a few other "useful" chemicals) and the rest was separated in the same way into diesel and road tar. The ratio between gasoline and diesel (volatile fuels and heavy fuels) was pretty much fixed by how much of each kind of molecule was in the crude stock.
The discovery of catalysts that would "split" the larger molecules into smaller ones permitted an increase in the fraction of the crude that came out of the pipe as "gasoline" by "cracking" diesel into lighter components. Some of the diesel components reduced by cracking into gasoline were replaced by cracking heavier (road tar) into diesel, but the cracking is less efficient as the molecules get bigger so the overall the ratio of gasoline to diesel was greatly increased.
In that time period when "cat-cracked gasoline" was being developed, gasoline power for vehicles had several advantages over diesel. Compression ratios of 6:1 to about 8.5:1 permitted much lighter construction of the engines for gasoline, while the 18:1 - 22:1 compression ratios required for ignition of diesel required much heavier construction. For "standard construction" engines, the lower flame propagation speed for burning diesel also limited max RPMs.
Spark ignition in Otto cycle (gasoline) engines also allowed a much wider RPM range, with a higher ratio between Max Torque RPM / Max HP RPM, with reasonably good efficiency (for the time). The "wider engine speed ratio meant fewer gear ranges were required for useful transmissions.
Diesels of that era had a relatively narrow RPM range, and the ratio of Max HP RPM to Max Torque RPM was much smaller than for Otto cycle engines, so useful "highway speeds" required "many gears" (15 speed transmission are now common in diesel trucks, and 22 or even 27 speed are not rare) and good efficiency was only obtainable if the vehicle could run for long periods at relatively constant RPM and load. This difference largely dictated the preference for Otto cycle (gasoline) engines for stop-and-go applications typical of "civilian drivers" and Diesel cycle engines for "truckers." (Marine engines, and heavy equipment machines also commonly used Diesels. The load range and load variation requirements would have made Diesels a "logical" choice for aircraft, but the lower air density - and much lower ambient temperatures - made it almost impossible to achieve reliable compression ignition there, and reciprocating Diesels of the era had much lower power/weight ratios which made them unsuited for flight applications.)
New refineries built around the time that catalytic cracking became feasible were designed to produce a ratio between "gasoline" and "diesel" that was appropriate to the existing and expected future use. The predictions then all favored getting the maximum amount of gasoline, and selling off the diesel as a "semi-waste" product where possible.
In the US, vehicles were developed to take advantage of the greater availability of "light fuels" that the refineries were designed to produce, and diesel fuel was sold fairly cheaply to the few industries that used it.
Virtually NO NEW REFINERIES have been built in the US since the ratio between volatiles and nonvolatiles was set by refinery designers of 40 to 60 years ago. New environmental rules make it difficult to get the permits to build a new refinery, largely due to the extensive time required for the permitting process; but the NIMBY factor is probably the most significant barrier to new construction.
The need for new refinery designs to produce a higher percent of low volatility fuels is clear in the current US price differential between gasoline and diesel fuel. Diesel that once (within my memory) cost 30% to 60% as much as gasoline (in $/gallon) is now 30% more than for gasoline in the US (not including the additional "road taxes" imposed on the main highway users of diesel here).
Because all existing refineries produce a low "diesel fraction," any increased demand for diesel would likely produce a much more drastic increase in fuel price ("demand penalty") than the same percentage increase in demand for gasoline. Existing refineries cannot be modified to change the gasoline/diesel ratio by much, in any economically feasible ways.
Largely due to NIMBY, all "our" new refineries are, and likely will continue to be, "off shore."