Empirical Risk Minimization

Ban and Rudin (2019) provide the formulation of the feature-based newsvendor problem as an empirical risk minimization (ERM) problem.

The objective is to learn a function that maps the feature vector to an order quantity, parameterized by . The optimization problem can be formulated as

This objective is called the empirical risk. Let and be the underage and overage for the -th observation, respectively. The optimization problem can be rewritten as

Quantile Regression

It is well-known that the newsvendor problem is closely related to a quantile of the demand distribution, called the critical ratio.

Huber et al. (2019) show that the ERM problem is equivalent to a quantile regression problem. Let be the quantile level. Let and be the actual and predicted values for the -th observation, respectively. The quantile loss function is defined as

In the newsvendor problem, the critical ratio corresponds to the quantile level . Therefore, we can use quantile regression to estimate the conditional quantile of demand given the features. The optimization problem can be formulated as

This is equivalent to minimizing the newsvendor cost, as shown below:

which is the newsvendor objective scaled by the constant . Hence minimizing the quantile loss with is equivalent to minimizing the newsvendor cost.

Successive Halving

Successive Halving was originally proposed by Karnin et al. (2013). Later, Jamieson and Talwalkar (2016) proposed to use this method for HPO.

Successive Halving requires a budget and a number of configurations . In the context of HPO for machine learning models, a configuration refers to a specific combination of hyperparameters. The budget can be defined as the total number of iterations that can be used. The performance of a configuration is evaluated by the loss function. In this algorithm, denotes the loss of configuration after iterations.

Let be the set of configurations. For each round , the number of configurations is halved, so that for , we have

and eventually .

The budget that is spent on a configuration in round is , and the total budget spent on configuration is . Since the number of configurations is given by , the total budget spent in round is , which is approximately a constant value .

To illustrate the Successive Halving algorithm, consider the following example. Let and , so that and ranges over .

0	2	2	16	8
1	4	6	8	4
2	8	14	4	2
3	16	30	2	1

Each round spends iterations, so the total budget is , as expected.

Hyperband

In the Successive Halving algorithm, it is required to specify and in advance. However, it is may be difficult to choose a good value for in practice. Hyperband (Li et al. 2016) can be considered as a extension of Successive Halving.

Note that the original paper writes this step as , rounding up only at the end; here we use , which floors the per-round budget first and matches the values in the paper’s Table 1 and its reference implementation.

The following example is adapted from Li et al. (2016). Let and , so that . The following table shows the values of and for each value of .

def hyperband(R, eta):
    """Return the (n_i, r_i) schedule for each bracket s."""
    s_max = 0
    while eta ** (s_max + 1) <= R:
        s_max += 1
    B = (s_max + 1) * R

    brackets = {}
    for s in range(s_max, -1, -1):
        n = B // R // (s + 1) * eta**s
        r = R // eta**s
        brackets[s] = [(n // eta**i, r * eta**i) for i in range(s + 1)]
    return brackets


R = 81
eta = 3
brackets = hyperband(R, eta)

s_max = max(brackets)
print("  i" + "".join(f"{f's = {s}':>12}" for s in range(s_max, -1, -1)))
print("   " + "".join(f"{'n_i':>6}{'r_i':>6}" for _ in range(s_max + 1)))
for i in range(s_max + 1):
    row = f"{i:>3}"
    for s in range(s_max, -1, -1):
        if i < len(brackets[s]):
            n_i, r_i = brackets[s][i]
            row += f"{n_i:>6}{r_i:>6}"
        else:
            row += " " * 12
    print(row)

  i       s = 4       s = 3       s = 2       s = 1       s = 0
      n_i   r_i   n_i   r_i   n_i   r_i   n_i   r_i   n_i   r_i
  0    81     1    27     3     9     9     6    27     5    81
  1    27     3     9     9     3    27     2    81            
  2     9     9     3    27     1    81                        
  3     3    27     1    81                                    
  4     1    81                                                

Hyperband Variants

Later works have proposed several variants of Hyperband. There are two main directions for improving Hyperband. One direction is to combine Hyperband with metaheuristics. The other direction is to combine Hyperband with Bayesian optimization.

Awad et al. (2021) proposed DEHB, a hyperparameter optimization method that combines differential evolution and Hyperband. Differential evolution (DE) is a classic evolutionary algorithm and Hyperband is a bandit-based method for hyperparameter optimization.

DEHB has been shown to outperform other hyperparameter optimization methods on a variety of benchmark problems. Eimer et al. (2023) discussed the hyperparameter optimization problem in the context of reinforcement learning (RL). They conducted a experimental study on the performance of various hyperparameter optimization methods, including random search, DEHB, and BGT. Their results show that DEHB has a good performance. Eggensperger et al. (2021) developed a collection of multi-fidelity benchmark problems for HPO, called HPOBench. Their results also show that DEHB has a good performance.

Another variant of Hyperband is BOHB (Falkner et al. 2018), which combines Hyperband with Bayesian optimization.