Yes, there is mathematical theory for prioritizing projects. One result is the ranking theorem: If independent projects are ranked based on the ratio of benefit-to-cost, and selected from the top down until the budget is exhausted, the resulting project portfolio will create the greatest possible value (ignoring the error introduced if the portfolio doesn't consume the entire budget). This solution is useful because it clarifies key information needed to optimize project decisions: (1) the cost of each candidate project, and (2) dollar worth of the benefits to be derived if the project is conducted.
There are, as well, useful theories and methods for quantifying project benefits (for example, AHP, real options, decision analysis, and multi-attribute utility analysis). In addition, there are practical and effective methods for measuring and accounting for risk. Finally, there are mathematical methods more accurate than the ranking theorem that allow you to, among other things, optimally select from alternative project versions (e.g., different project funding levels), account for people and other resource limitations, compare projects that return benefits over different time periods, and make choices that achieve specified performance targets.
To learn the details, read my paper Mathematical Theory for Prioritizing Projects.